ICAO/ASPA Regional Seminar

TEM, LOSA & NOSS – Essential SMS Tools

Mexico City, Mexico

LOSA Data Analysis: From Diagnostic

Counts to Targets for Enhancement
James Klinect, PhD
The University of Texas / The LOSA Collaborative

The University of Texas

Human Factors Research Project
 Introduction
• Based on our LOSA experience, airlines are:
– Overdeveloped in data collection – too many tools

– Underdeveloped in the capability to analyze and use data in their

safety management systems

• LOSA uses TEM flight crew performance data to diagnose

system safety performance strengths and weaknesses

• Long term LOSA success is dependent on how data are

translated to meaningful findings that an airline can act on
Threat and Error Management Review
Threat and Error Management (TEM)

Everyday Operations
(Routine flights)

1 Manage operational complexity Threat Management

LOSA/TEM
2 Avoid/Manage their own errors Error Management
Measures

3 Avoid/Manage aircraft deviations Undesired Aircraft

State Management
 Threats Defined
• Threats – External events or errors that occur outside the influence of
the flight crew but require their attention to maintain adequate safety
margins

Environmental Threats Airline Threats

• Adverse Weather • Airline Operational Pressure
• Airport • Aircraft
• ATC • Cabin
• Environmental Ops Pressure • Dispatch / Paperwork
• Ground / Ramp
• Ground Maintenance
• Manuals / Paperwork
 Error Types and Codes

• Errors – Observable crew action or inactions that leads to

a deviation from “organizational” or “flight crew”
expectations
Aircraft Procedural Communication
• Aircraft Handling • Checklists • Pilot to Pilot

• Automation • Callouts • Crew to ATC

• Flight Controls • Briefings

• Systems / Radio / • SOP Cross-

Instruments verification

• Ground Navigation • Documentation

• PF/PNF Duty
 LOSA Undesired Aircraft States
• Undesired Aircraft States – Crew-error induced aircraft state that
increases risk and decreases safety margins

Vertical, lateral or speed deviations

Unnecessary weather penetration
Aircraft Handling
Unstable approach
Long, floated, firm or off-centerline landings

Runway/taxiway incursions
Ground Navigation Wrong taxiway, ramp, gate, or hold spot

Taxi above speed limit

Incorrect Aircraft
Automation, engine, flight control, systems, or weight/balance events
Configuration
LOSA Data Analysis
 LOSA Data Analysis
• LOSA provides two types of data:
1. Text data - Phase of flight narratives / TEM descriptions / in-flight
crew interviews about organizational issues

2. Numerical data - TEM categorical coding

• Three stages of LOSA data analysis

1. TEM organizational profiles (prevalence/mismanagement rates)

2. Drill-down analyses into LOSA narrative and categorical data

3. Targets for enhancement (Swiss cheese holes that need plugging)

 TEM Organizational Profiles
• Objective: Provide airlines with a general overview of TEM
performance on a typical day in flight operations

• Profiles are built on two primary types of data indices:

1. Prevalence – Percentage of flights with threat, error, or undesired
aircraft state

2. Mismanagement – Percentage of threats, errors, or undesired

aircraft states leading to flight crew error

• TEM organizational profiles are only a starting point in

LOSA data analysis
Example: Threat Organizational Profile
Threat Prevalence Threat Mismanagement
Raw
Threat Categories Prevalence Archive Mismanagement Archive
Counts
Index Average Index Average

ATC 61% 54% 8% 14% 27/342

Adverse Weather 61% 58% 6% 13% 18/304

Environmental Operational Pressure 51% 44% 2% 8% 4/228

Airport 13% 25% 5% 9% 2/42

Threat Prevalence Threat Mismanagement
Aircraft Percentage
Percentage of LOSA flights of threats linked to flight crew error
with a threat
52% 32% 9% 15% 27/290
(Malfunctions, MELs, and anomalies)
61% of flights had an ATC8% of ATC threats were mismanaged
threat
Cabin 18% 15% 5% 9% 2/65
(Archive average = 14%)
(Archie average = 54% of flights)
Airline Operational Pressure 18% 21% 5% 10% 3/60

Dispatch/Paperwork 12% 12% 2% 8% 1/44

Ground Maintenance 8% 8% 3% 8% 1/32

Ground/Ramp 8% 13% 4% 10% 1/27

 Primary LOSA Drill-Down Analyses

Event Description/Coding

Examines text descriptions and

coding of TEM events

Demographic

Examines city pairs, fleet, pilot

TEM flying, and flight time differences
Organizational
Profiles
Operational Context

Examines TEM performance in

context (phase of flight narratives)

TEM Process
Description/Coding
Examines flight crew management
of threats, errors, and undesired
aircraft states
 Example: Event Coding Drill-Down
What type of aircraft threats are most prevalent?

Threat Threat
Aircraft Threats Prevalence Mismanagement Raw Count
Index Index

MEL with operational implications 36% 8% 14/178

Malfunction unexpected by the flight crew 20% 13% 12/96

Automation event/anomaly 5% 6% 1/16

Total 52% 9% 27/290

 Example: Demographic Drill-Down
Are there fleet differences with aircraft threats?

Threat Threat
Fleet Raw Count
Prevalence Index Mismanagement Index

Fleet 1 56% 6% 3/52

Fleet 2 49% 9% 7/77

Fleet 3 61% 3% 2/70

Fleet 4 52% 6% 2/32

Fleet 5 47% 22% 13/59

Example: Event Description Drill-Down
LOSA Observation #21 Fleet Three Pilot Flying: First Officer Threat #1

Threat Description
APU amber fault light came on during taxi-out.

Threat Management Description

FE checked the operational manual and attempted to recycle the APU switch but
the light was still on. This was linked to a flight crew error of failing to run the
abnormal checklist (Quick Reference Handbook) and resulted in an undesired
aircraft state of operation with an unresolved MEL. Consequently, the crew
discussed the fault and diagnosed it as being caused by a slightly opened or not
flushed APU air inlet door. The crew took off with the light on where it remained on
until landing.

Phase of Flight: Preflight/Taxi Threat Type: Aircraft Malfunction

Threat Code: Aircraft malfunction unexpected by crew

Threat Outcome: Linked to Flight Crew Error (Procedural error – Failure to execute an
abnormal checklist)
Example: Operational Context Drill-Down
Predeparture/Taxi-Out Narrative
Observation #21 Fleet Three Pilot Flying: First Officer

It was a very early morning departure, i.e., even the observer's pick-up was at 3:30 AM local
time. All respective duties were done with everyone 'working–in–sync'. While the Flight
Engineer (FE) was still busy with his work/scan, the Captain (CA) offered to make a round of
drinks, which was accepted by First Officer (FO).
The ground crew completed the pushback but did not call for the CA to set the parking brakes.
After some time, the CA asked the ground crew if he wanted the brakes on, which was quickly
acknowledged with an affirmative answer. All checklists were read with the correct procedural
protocol with everyone verifying and cross-checking each switch position.

On taxi out, the FE noticed the Auxiliary Power Unit (APU) Fault light was ON (Aircraft
malfunction threat unexpected by the crew). The FE informed the rest of the crew. The CA
ordered the FE to check the ops manual and to recycle the switch. However, there was no luck
as the light was still on. Consequently, this led to the crew discussing this fault and it was
diagnosed as being caused by a slightly opened or not flushed APU air inlet door. Although all
crew had a consensus on this, the FE was still very much bothered with it and waited for it to
disappear all the way to entering the runway. No abnormal checklist was performed (flight crew
error) and the light remained on throughout the flight.
 Targets for Enhancement

Event Description/Coding

Examines text descriptions and

coding of TEM events

Demographic

Examines city pairs, fleet, pilot

Threat and Error flying, and flight time differences
Organizational Targets for
Profiles Enhancement
Operational Context

Examines TEM performance in

context (phase of flight narratives)

TEM Process
Description/Coding
Examines flight crew management
of threats, errors, and undesired
aircraft states
 Targets for Enhancement
• After reviewing the LOSA results, one question should come
to mind for safety managers

• What are the holes in the Swiss cheese that need plugging?

• Unfortunately – There is no software or magic data analyzer

that points to Swiss cheese holes and tells airlines what to do

• Suggested format for presenting LOSA results and targets:

– LOSA Finding
– Potential Swiss Cheese hole (Systemic and driven)
– Target for Enhancement
Simple Example: Findings to Targets
• LOSA Findings:
– 13% of all “aircraft malfunctions unexpected by the crew” threats
were mismanaged
– Drill-down analyses shows many flight crews failed to properly
reference the QRH – mostly occurring on Fleet #5
– Further analysis – Interviews with crews about QRH issues

• Potential Swiss Cheese Hole – Aircraft malfunctions and

QRH usage

• Sample Target for Enhancement - Improve the

management of aircraft malfunction threats that were
unexpected by the flight crew before the next LOSA
 Example: Error Profile
Error Error Mismanaged
Error Type
Prevalence Index Mismanagement Index Error/Error Count

System/Instrument/Radio 18% 34% 22/65

Automation 18% 24% 16/66

Manual Handling/Flight Control** 15% 77% 41/53

Ground Navigation 3% NC* 6/9

Callout 32% 5% 8/149

Checklist 20% 15% 13/88

SOP Cross-Verification 18% 14% 9/66

Briefing 15% 4% 2/55

PF/PNF Duty 10% 3% 1/35

Documentation 4% NC* 2/11

Crew to External 20% 7% 5/72

Pilot to Pilot 3% NC* 1/9

LOSA Case Study
 LOSA Case Study
• 1st LOSA results – Targets identified:
1. Improve energy management issues during descent/approach/land
• 18% of flights with a speed deviation undesired aircraft state

2. Lower the prevalence of unstable approaches

• 11% of flights with an unstable approach

3. Lower prevalence of checklist errors

• 50% of flights with a checklist error

4. Improve monitor/cross-checking performance

• 38% of errors went undetected (not responded to by the flight crew)

5. Improve ATC threat management

• 22% of ATC threats were mismanaged - linked to flight crew error
 LOSA Case Study Two
Targets 1st LOSA 2nd LOSA
1. Improve energy 18% of flights with
6% of flights
management during DAL speed deviations

2. Unstable approaches 11% of flights 4% of flights

3. Checklist errors 50% of flights 24% of flights

4. Monitor/cross-checking 38% of errors (crew fail 61% of errors (crew fail

performance to respond/undetected) to respond/undetected)

5. ATC threat management 22% mismanaged 13% mismanaged

Proactive safety change …….

Over one year, approx 182,500 flights, 4% rate = 7,300
unstable – much better than 11% rate = 20,075 unstable
 Concluding Remarks
• From a large sample of observations, LOSA generates a
“flight operations” snapshot of strengths (thick cheese slices)
and weaknesses (cheese holes)

• Some say LOSA is just an error counting exercise – They’re

wrong – it’s just the start in data analysis!

• LOSA allows airlines to measure the effectiveness of their

safety improvements across time (Target for enhancements)

• LOSA is just one SMS tool that allows operators to become

proactive in their safety efforts
Muchas Gracias