ISD

PROCESS GUIDE

Instructional Systems Development

Methodology

For:
AQP Curriculum Developers
and
Program Management

Vin Parker, MRAeS

AQP Learning & Development
2018

[Used by Permission]

1
Instructional Systems Development
Guide
ISD Guide Agenda:

 AQP-101 (brief refresher)

 ISD Process Overview
 ISD Process Considerations
1) ISD Training Philosophy
2) ISD Requirements
 ISD Methodology
 ADDIE Process Steps
• ANALYSIS
1) Training Needs Assessment (TNA)
2) Content Needs Analysis (CNA)
• DESIGN
• DEVELOPMENT
• IMPLEMENTATION
• EVALUATION
• REVISION
 ISD Documentation
 Considerations and Important Notes
2
AQP

‘AQP 101’

Refresher

3
AQP

Why AQP?

“AQP permits a greater degree of regulatory flexibility in the

approval of innovative pilot training programs.

Based on a documented analysis of operational requirements, a

certificate-holder under AQP may propose to depart from
traditional practices with respect to what, how, when, and where
training and testing is conducted.”
- AC 120-54A

4
AQP

Goal of AQP

“The primary goal of AQP is to achieve the highest possible

standard of individual and crew performance.

In order to achieve this goal, AQP seeks to reduce the probability

of crew-related errors by aligning training and evaluation
requirements more closely with the known causes of human
error.”
- AC 120-54A

5
AQP

Benefits of AQP:

 CRM – Integration of technical and behavioral skills

 Scenario-based – training and evaluation
 The ability to modify training curricula, media, and intervals
 Data-driven program improvement
 Crew evaluation as well as individual assessment
 Improved standardization across fleets and flight personnel
 Shift from programmed hours to proficiency-based training
 Access to innovative training ideas and research (industry)
 Opportunity to achieve more efficient training

6
AQP

Benefits of AQP:

FLEXIBILITY

7
AQP
Required Documents of AQP:

1. Application
2. Job Task List/Analysis (JTA)
3. Qualification Standards (replaces ‘Aircrew Certification Standards - ACS’)
4. Instructional Systems Development (ISD) Methodology
5. Implementation and Operations Plan (I&O)
6. Curriculum Outline(s)

Also:
Data and Program Reports
- FAA data submissions
- Periodic review(s)
- Annual (or end of reporting period)
8
Instructional Systems Development
(ISD)

AQP - ISD
Development and Modification
Process

Overview

9
Instructional Systems Development
(ISD)
What is ‘ISD’ anyway?!

There are several Systematic Training Program methodologies (STPM) to choose

from. The primary objective of a systematic program is to use data-informed
analysis to develop – implement – and improve the program itself.

In AQP, Instructional Systems Development (ISD) – is the most commonly used

systematic process for developing aviation training curriculums. However, ISD is
also used and applied in many different industries and work settings – such as
education, medical, military, even sports.

ISD uses adult learning theory, educational psychology, human factors

engineering, and cognitive psychology (just to name a few of the disciplines that
may support an ISD).

Quite simply: ISD is the methodology of how to develop and update training
curriculums; how to deliver the training; and how to improve the training.

10
Instructional Systems Development
(ISD)

What is ‘ISD’ anyway?!

In AQP, the ISD document itself is basically divided into two parts:

1) Curriculum Development
Describing the approach for using JTAs and Qual Standards as a baseline
to construct general training curriculums across all AQP courses.

2) Line Operational Simulation (LOS) Methodology

Describing the approach for developing LOS scenarios

11
Instructional Systems Development
(ISD)

ISD Regulations – Part §121 Subpart Y (900’s)

The ISD process must be followed for ‘developing or modifying Qualification

Standards and associated Curriculum Content, based on using a documented
analysis of the job tasks, skills, and knowledge required for job proficiency.’
[§121.907]

‘All AQP curriculums must be based on an instructional systems development

methodology.’
[§121.909]

(…That’s it!)

12
Instructional Systems Development
(ISD)

ISD Process Overview

The ISD training development and modification process follows a basic step-development
‘ADDIE’ approach that includes the following components:

1 - ANALYSIS 4 - IMPLEMENTATION

2 - DESIGN 5 - EVALUATION

3 - DEVELOPMENT 6 - REVISION (pro re nata)

13
Instructional Systems Development
(ISD)

ISD Considerations – for Development and Modifications

During the AQP development or modification process, two considerations must be

applied to the overall objective(s):

1) Training Philosophy
2) ISD Requirements

14
Instructional Systems Development
(ISD)

1. Training Philosophy

AQP training programs are designed to incorporate an adult education philosophy

that supports positive learning and enrichment through coaching and measurable
assessment of skills, knowledge, and behaviors. This philosophy also supports the
use of multiple training mediums that may include, but are not limited to: distance
learning through computer-based training (CBT); instructor-led (classroom); flight
training devices (FTD) such as flat-screen and full-flight simulator (FFS) automation
devices; and self-guided practice.

This approach includes data collection of specific task and behavioral performance to
primarily identify areas for program improvement through trend analysis.
Performance data analysis also may be used to identify opportunities for individual
enrichment and professional improvement.

15
Instructional Systems Development
(ISD)

Training Philosophy includes three elements:

1. Use of Advanced Qualification Program (AQP)

2. Training Objectives
3. Measurable Performance

16
Instructional Systems Development
(ISD)

A. Training Philosophy – Use of Advanced Qualification Program (AQP):

The first determination of a training philosophy is to simply declare the training

will be centered within the AQP-ISD structure.

The AQP seeks to integrate the training and evaluation of task performance,
knowledge, and cognitive skills at each stage of a curriculum. For ‘pass/fail’
purposes, crew members must demonstrate proficiency in scenarios that test
both technical and crew resource management (CRM) skills together.

The AQP is designed to include data collection strategies which are diagnostic of
cognitive and technical skills. Also implemented are procedures for refining
curricula content based on quality control data.

17
Instructional Systems Development
(ISD)

B. Training Philosophy – Objective(s):

The overall objective of training is to focus on continual enhancement of safety

by achieving the highest possible levels of individual and crew performance.
This is accomplished through data collection and trending analysis

18
Instructional Systems Development
(ISD)

C. Training Philosophy – Measurable Performance:

Skills, knowledge, and behaviors are measured through an election of ratable

tasks and behavioral indicators. Performance outcome and areas for program
improvement are determined through a combination of data-driven inputs,
from a range of internal and external data source programs.

Internal data inputs include AQP performance data, as well as operational data
(i.e., ASAP, FOQA, LOSA, Line Check, et cetera.).

External data inputs may include published FAA documents, industry trending,
and substantiated expert opinion.

19
Instructional Systems Development
(ISD)

2) ISD Requirements
1. Develop a job task listing.
2. Analyze that listing to determine essential skill and knowledge requirements
(either directly or by reference). This becomes a Job Task Analysis (JTA)
3. Determine which skill and knowledge requirements must be trained/tested.
4. Develop proficiency objectives that capture all training requirements.
5. Develop qualification standards that define acceptable operational performance
levels.
6. Develop tests that measure proficiency in skill and knowledge areas.
7. Provide instructional programs that teach and test training requirements.
8. Establish and maintain an audit trail of explicit links between task requirements,
training requirements, training and evaluation activities, and evaluation results.
9. Measure student performance against proficiency objectives and qualification
standards for all curriculums.
10. Revise the training program based on student performance levels on an ongoing
basis. This de-identified data (stored in the Performance/Proficiency Database)
will be collected and reported to the FAA on a regular basis.
20
Instructional Systems Development
(ISD)

Instructional Systems Development

(ISD)

Methodology

21
Instructional Systems Development
(ISD)

ISD Methodology

The AQP instructional systems development (ISD) methodology is a systematic

process of developing training curriculum and follows an ADDIE step-
development ISD model.

Each step of the ISD development, including the order of its application may be
modified or arranged to meet the needs of the training objectives.

22
Instructional Systems Development
(ISD)

The most complex AQP curriculums developed are those designed for pilot training, so
the descriptions and examples in most training documents are based primarily on the
development and maintenance of the pilot courses.

Therefore, the pilot procedures and methodologies will typically apply to the
development of all AQP curriculums, unless otherwise noted.

Other course development methodologies may differ from the pilot course
methodologies, and will be described by exception. Such as those for instructors or
evaluators, etc.

23
ISD Process

Using the ADDIE Process Steps

The ADDIE process (Analysis – Design – Development – Implementation - Evaluation)

generally follows a linear process, particularly during initial curriculum development.

The general activities associated with each of the ADDIE steps should always be
considered for any development or modification, but may not always apply, depending
on the activity content or application.

24
ISD Process
ISD
ADDIE Process

ANALYSIS

EVALUATE

25
ISD Process
ISD
ADDIE Process

ANALYSIS

REVISE

EVALUATE DESIGN

26
ISD Process
ISD
ADDIE Process

ANALYSIS

REVISE

EVALUATE DESIGN

REVISE

DEVELOP

27
ISD Process
ISD
ADDIE Process

ANALYSIS

REVISE

IMPLEMENT EVALUATE DESIGN

REVISE REVISE

DEVELOP

28
ISD Process
ISD
ADDIE Process

ANALYSIS

REVISE REVISE

IMPLEMENT EVALUATE DESIGN

REVISE REVISE

DEVELOP

29
ISD Process
ANALYSIS
The analysis phase of the ISD process includes two parts:

Part 1: Training Needs Assessment (TNA)

Part 2: Content Needs Analysis (CNA).

30
ISD Process
ANALYSIS – Part 1: Training Needs Assessment (TNA)
The analysis process must first identify the need (or problem), and determine if a
desired result outcome would be achieved through a training solution. This is
determined by completing a Training Needs Assessment (TNA).

In many cases, a problem or objective may not require a training solution or

modification to achieve a resolution. Therefore, a TNA must first be conducted to
determine if the applicable data analysis indicates a genuine need for training. If the
TNA process determines there is a need for training, analysis steps are required to
determine the specific task and behavior objectives to support design and
development.

31
ISD Process
ANALYSIS – Part 1: Training Needs Assessment (TNA)
PRACTICAL APPLICATION

When the only tool in your toolbox is a hammer, every problem looks like a nail.

When your only area of influence is training, every human performance shortfall
looks like a training issue. But human performance has at least six components.
Unless this shortfall consists of a lack of skill and knowledge, and nothing other than
a lack of skill and knowledge, then there may be more to the solution than training, if
training is even part of the solution at all!

32
ISD Process
ANALYSIS – Part 1: Training Needs Assessment (TNA)
PRACTICAL APPLICATION

When the only tool in your toolbox is a hammer, every problem looks like a nail.

When your only area of influence is training, every human performance shortfall
looks like a training issue. But human performance has at least six components.
Unless this shortfall consists of a lack of skill and knowledge, and nothing other than
a lack of skill and knowledge, then there may be more to the solution than training,
if training is even part of the solution at all!

-Dr. Doug Farrow

AQP Program Manager, FAA AFS-280 (ret.)
‘Human Performance Technology, ISD’
FAA Memo, 2001

33
ISD Process
Identified Needs and
Objectives

Which level of Operational Needs

or Objectives
RESULTS

evaluated
outcome applies?
(if any) Line Performance BEHAVIOR Levels of
Evaluated
Outcome*

Trained to
LEARNING
Proficiency
How do I
determine Needs
and Objectives?
Individual Needs REACTION

*Kirkpatrick’s Levels of Evaluation 34

ISD Process
Questions to Identified Needs and
Determine Needs & Objectives
Objectives

Identify: Operational Needs

What is the objective? or Objectives

What does the

individual(s) need to do Line Performance
Questions to
to reach the objective? Determine
Needs and Objectives
What activities, tools or Trained to
experiences will help
the individual learn?
Proficiency

What will motivate the

individual to Individual Needs
perform?

35
ISD Process
Training Needs Assessment (TNA) Process
Questions to Identified Needs and Does data analysis indicate training
Determine Needs &
Objectives
Objectives would improve the outcomes?

Identify: Operational Needs

RESULTS
What is the objective? or Objectives

What does the

individual(s) need to do Line Performance BEHAVIOR Levels of
to reach the objective? Evaluated
Outcome*

What activities, tools or Trained to

experiences will help LEARNING
the individual learn?
Proficiency

What will motivate the

individual to Individual Needs REACTION
perform?

*Kirkpatrick’s Levels of Evaluation 36

ISD Process

Does the data analysis indicate that…

… a training solution would achieve the
RESULTS
operational outcome or results?

… a training solution would provide what they

need to do to reach a behavior objective? BEHAVIOR Levels of
Evaluated
Outcome*

… a training solution would provide the

activities, tools, or experience they need to LEARNING
learn? (training)

… a training solution would provide the

REACTION
motivation to learn?

*Kirkpatrick’s Levels of Evaluation 37

ISD Process
Training Needs Assessment – Conclusion Flow Chart

Is a training solution needed

to achieve the objective?

No

Proceed with SMS or another

company process other than
training to determine a
mitigation solution

// END //

38
ISD Process
Training Needs Assessment – Conclusion Flow Chart

Is a training solution needed

to achieve the objective?

Yes

Would an existing training

curriculum, or partial curriculum,
achieve the objective?

Yes

Use the remaining ISD steps to

implement, and evaluate, an
EXISTING training solution

Proceed to
Analysis – Part 2:
Content Needs Analysis

39
ISD Process
Training Needs Assessment – Conclusion Flow Chart

Is a training solution needed

to achieve the objective?

Yes

Would an existing training

curriculum, or partial curriculum,
achieve the objective?

Yes No

Use the remaining ISD steps to Use the remaining ISD steps to
implement, and evaluate, an design, develop, implement, and
EXISTING training solution evaluate, a NEW training solution

Still Proceed to Part 2

Proceed to
Analysis – Part 2:
Content Needs Analysis

40
ISD Process
ANALYSIS – Part 2: Content Needs Analysis
The Content Needs Analysis (CNA) is used to determine specific measurable tasks or
behaviors that may be used within a training curriculum. The CNA process also
identifies the requirements of specific task factors, such as currency and criticality, as
well as Qualification Standards, demographics and media considerations. The CNA
provides the building blocks for the remaining design and development process

41
 ISD Process
Use the following items to complete a CNA:

 Develop or identify a job task listing

 Analyze job tasks to determine component or prerequisite skills and knowledge for each job task
 Examine student demographic data and perform instructor/evaluator assessments to determine
which skills and knowledge will be trained and/or tested in each curriculum
 Develop or identify proficiency objectives (TPO), to include the specification of real-world conditions
and standards, for all tasks, subtasks, and CRM components, as well as higher-level elements, where
appropriate
 Develop or identify enabling objectives (EO), to include the specification of academic conditions and
standards, where appropriate, for CRM components, behaviors, elements, and for skills and
knowledge, as required
 Assess criticality, currency, media requirements, and evaluation strategy for proficiency objectives,
as required
 Develop or identify Job Task Analysis and Qualification Standards documents for each duty position

Proceed to the next ISD Steps…

42
ISD Process
DESIGN
 Allocate proficiency objectives among major AQP curriculums: Indoctrination,
Qualification and Continuing Qualification.
 Append enabling objectives to their parent proficiency objectives.
 Cluster and sequence objectives into lessons according to media and method
considerations.
 Cluster and sequence lessons into modules, modules into segments, and
segments into curriculums.
 Develop a Curriculum Outline for each major AQP curriculum.
 Design the performance data collection and analysis system to support
crewmember, instructor and evaluator performance.

43
ISD Process
DEVELOPMENT
 Develop teaching and testing materials, activities, events and data collection
forms for crewmember, instructor and evaluator curriculums.
 Develop the data collection and analysis system.
 Train instructors and evaluators, using Instructor and Evaluator AQP curriculums,
to train and evaluate crewmember AQP curriculums.
 Conduct small group tryouts (SGT) of crewmember curriculums.
 Collect and analyze SGT student performance data.
 Report de-identified SGT data to the FAA.
 Revise materials, activities, events, and the data collection and analysis system as
applicable. Conduct additional small group testing, if required.

44
ISD Process
IMPLEMENTATION
 Implement curriculums for all crewmembers, instructors and evaluators.
 Collect and analyze performance data on all students, instructors and evaluators.
 Report de-identified data to the FAA.
 Revise materials, activities, events, and data collection and analysis system as
applicable.

45
ISD Process
EVALUATION
 Assessment and review all analyses, processes, products, data, reports and
activities completed as part of the analysis, design, development and
implementation activities.
 Analysis and review of all performance data on crewmembers, instructors and
evaluators.

46
ISD Process
REVISION (pro re nata)
 The revision process is incorporated as necessary, at any stage, throughout the
ISD process.

REVIEW - EVALUATE - REVISE

47
Instructional Systems Development

ADMINISTRATIVE

48
Instructional Systems Development
(ISD)

2) ISD Requirements
1. Develop a job task listing.
2. Analyze that listing to determine essential skill and knowledge requirements
(either directly or by reference). This becomes a Job Task Analysis (JTA)
3. Determine which skill and knowledge requirements must be trained/tested.
4. Develop proficiency objectives that capture all training requirements.
5. Develop qualification standards that define acceptable operational performance
levels.
6. Develop tests that measure proficiency in skill and knowledge areas.
7. Provide instructional programs that teach and test training requirements.
8. Establish and maintain an audit trail of explicit links between task requirements,
training requirements, training and evaluation activities, and evaluation results.
9. Measure student performance against proficiency objectives and qualification
standards for all curriculums.
10. Revise the training program based on student performance levels on an ongoing
basis. This de-identified data (stored in the Performance/Proficiency Database)
will be collected and reported to the FAA on a regular basis.
49
Instructional Systems Development
(ISD)

2) ISD Requirements

8. Establish and maintain an audit trail of explicit links between task requirements,
training requirements, training and evaluation activities, and evaluation results.

50
Instructional Systems Development
(ISD)
ISD Process Documentation

Throughout the ISD process, documentation of the ISD process should be

maintained to ensure each step of the ISD was followed. In some cases, the order of
the step process, or specific step elements may vary, depending on the development
or modification needs.

The ISD process may be conducted within, or as, an SMS safety risk management
process. In those cases, documentation of the ISD process may exist within either
the training documents, or the SMS documentation system.

Documentation of the ISD process supporting AQP curriculum development and

modification may be internal to the organization and is not necessarily required for
external publication or FAA submission, beyond the normal AQP communications.

However, ISD process documentation should accompany FAA submissions for

development or modifications of Qualification Standards or Curriculum Content .
ISD process documentation may also be used to support any other development or
modifications, as needed, at the discretion of the training management.
51
Instructional Systems Development
(ISD)
ISD Process Documentation

SUMMARY:

In most cases it will be prudent to document the specific actions and steps taken as a
record of the ISD process.

This helps the organization reference how qualification standards or a certain

element of curriculum content - was created, or changed.

When appropriate, ISD Process documentation should also be included as support

documentation for certain FAA approval submissions.

52
Instructional Systems Development
(ISD)
NOTES:

Program managers, curriculum, and courseware developers should use the ISD
ADDIE process steps in the order outlined to develop and/or modify the Qualification
Standards or Curriculum – unless otherwise appropriate.

It is important to understand that the ISD process doesn’t end after the initial
curriculum development process. AQP is a continually improving program and the
ISD process must be used throughout the life span of each AQP curricula.

The ISD step development process is intended to be flexible. Not all steps or items
listed are intended to be all-inclusive, nor does every step or item listed necessarily
apply to every program or curriculum development or modification. In many cases,
minor adjustments, concepts, or changes may occur through subject matter expert
(SME) discussions or development activities. The intention of the ISD methodology
and development step-process, is to provide a familiar guideline for developers to
follow, and ensure consideration of data-driven improvements to the AQP.

53
 Questions?

Vin Parker, MRAeS

AQP, Learning & Development
Pilot and FA Training Programs

54
Instructional Systems Development

NEXT:

LOS
Development and Modification

55