Sigma Metrics System in Medical Laboratory

Hanita Othman
(MD, DrPath (Chemical Pathology)
Hospital Cancelor Tuanku Muhriz, UKMMC
drhanita@ppukm.ukm.edu.my
 Agenda
• Concept of Six Sigma and Sigma Metrics
• Application of Sigma Metrics in Medical
Laboratory
• Sigma Metrics Verification (SVP) Program
• SVP Program – HCTM experience

Palembang Oct 2017

Sigma Metrics System in Medical Laboratory

Six Sigma Quality Management

System
 What is Six Sigma QMS
In the business world:
 Six Sigma is a
management strategy
that seeks to improve the
quality of process outputs
by identifying and
removing the causes of
defects (errors) and
minimizing variability in
manufacturing and
business processes.

Palembang Oct 2017

MEDICAL ERRORS
 Medical Errors

44%

PHYSICAL
HYPOTHESES/ LABORATORY FOLLOW
HISTORY EXAMINATIO
SYNTHESIS TESTS UP HARM
N

Schiff et al 2009; Diagnostic Error In Medicine-Analysis Of 583

Physicians-Reported Errors. Archives Int Med 169:1881-7
 How does Six Sigma fit into Healthcare TQM
• In healthcare, “quality” has been difficult to define
and measure
• Mostly focus on “performance”
How do we measure errors objectively in the lab?
• Quality and performance are related but not the
same!!
• Performance- measure how well we are doing
• Quality-is a measure of performance against
standard

Palembang Oct 2017

 Error in Medical Laboratory
Errors remain a challenge for laboratory performance
improvement
Phases in testing process

Pre Post
Analytical
analytical analytical

Error in testing process

62% 15% 23%

Palembang Oct 2017

 Error in Medical Laboratory
Phases in testing process Error (%)

Post
Pre analytical Analytical
analytical
(62%) (15%)
(23%)

Errors counted and reported as number / %

Performance are
evaluated IQC and EQA

Palembang Oct 2017

 Error in Medical Laboratory

Post
Pre analytical Analytical
analytical
(62%) (15%)
(23%)

• IQC and EQA to evaluate performance via precision and bias

• Not generally understood by clinicians and patients
• In instances where there are questions about laboratory
performance, there are no clear scaling data to present.
• This scaling data has to be an evidence-based analytical
quality specifications.

Palembang Oct 20172017

 How does Six Sigma fit into Healthcare TQM

• To measure errors objectively, Six Sigma provides principle

and methodology that can be widely applied to all laboratory
processes.

• Sigma-metrics are the key to making the quality measurable

and understandable

Palembang Oct 2017

 Six sigma concept in Healthcare TQM
Six sigma improves the entire quality management process:

• Requires that “tolerance limit” be define-to describe good quality

and poor quality/defect

• Provides a standard scale – Sigma scale- measuring the quality of any

process

• Identify the goal for how good quality should be – 6 sigma = world
class

Palembang Oct 2017

 Evolution in QC Practices
QC for future

5th Gen QC
4th Gen QC 90s
80s-90s Six sigma and
2nd and 3rd Gen QC 5th Gen QC
TQM and 4thGen
QC Define
1980s tolerance limit
Validation of
1st Gen QC Westgard method Set a goal for
multirules performance world class
1950s Introduction of Uniform ways
QC applied to
other rules for monitor to describe Q,
Levey- Jenning
error detection performance(QC defect,defect
Medical Lab rates, DPM
procedures
2SD-warning rule (rules,N) Frequency of
1930s 3rdGenQC
3SD-rejection appropriate to the QC around
Shewhart QC Use of diff QC method known and
rule
procedure for diff unknown
Industries analyzers Total allowable
error (TEa)

Automation
 Sigma metrics

In the laboratory:
Sigma metrics has been used
to monitor performance of
analytical system by
identifying biased and/or
imprecision and TEa and
translated into Sigma Value
which serve as benchmarking
for performance.

Palembang Oct 2017

 Six Sigma Scale
Sigma value indicates how often defects are likely to occur
The higher the sigma value – less likely the process will generate defect.

Average service or
World class
product

1σ 2σ 3σ 4σ 5σ 6σ
691 462 308 538 66 807 6210 233 3.4 dpm

G/B (%) G/B (%) G/B % G/B(%) G/B (%) G/B (%)
31/69 69/31 93.3/6.7 99.4/0.6 99.97/0.023 99.9997/
0.00034

Palembang Oct 2017

Applying sigma concepts to
Clinical Laboratory data
 Applying sigma concepts to
Clinical Laboratory data

• Define acceptability criteria: Total allowable error (TEa)

• For quantitative assays
 a) Determine imprecision
 b) Determine bias

 Determine Sigma value

 Sigma = (TEa - bias)/CV

Palembang Oct 2017

Sigma Metric
Sigma Metric and TEa

Bailey, CACMID
Strategies for selecting TEa

Bailey, CACMID
Total allowable Error (TEa)
Display of Sigma Metrics
Optimizing QC design
 Sigma Metric Application

• Sigma = (TEa - bias)/CV

• Strive for 6 sigma
• >4 sigma is ideal
• <3 sigma cannot be controlled with statistical QC
protocols

Palembang Oct 2017

Sigma Metrics System in Medical Laboratory

Application in Medical Laboratory

 Application in Medical Laboratory

• Evaluate assay quality and assist in selection of

analytical system
• Monitoring performance –multiple analyzers,
multiple sites
• Tool for Continuous quality improvement (CQI)
activity
• Formulate QC strategy and implication on cost

Palembang Oct 2017

Evaluate assay quality and assist in selection of
analytical system

• Studies by Lenters-Westra and Slingerland: compared 7 POC

devices for measuring HbA1c, using “secondary reference
measurement procedures” (from European Ref Lab for
Glycated Hb)
• Precision were established using CLSI EP05 protocol
• Accuracy were obtained from comparison with ref method
(CLSI EP09 protocol)
• The results were tabulated in Method Decision Chart

Palembang Oct 2017

 Result:
2 devices provide less than 2
sigma quality
2 devices provide less than 3-
sigma quality
1 device overlap the 3 sigma
line
2 devices overlap the 4 sigma
line

Sigma metric value may

assist in method validation
acceptance and selection
of analytical system.

Ref: Lenters-Westra E, Slingerland RJ. Three of 7 hemoglobin A1c point-of-care instruments

do not meet generally accepted analytical performance criteria. Clin Chem 2014;60:1062-
1072.
 Application in Medical Laboratory
Monitoring performance –multiple analyzers, multiple
sites
• Harold et al used sigma QC to verify and monitor performance
in a multi-instrument, multisite integrated health care
network of the Geisinger Medical Laboratory (GLM)

• Population: 2 hospital Labs and 7 regional clinic lab

• Sigma values were determined for:

– 33 Chemistry tests
– Coagulation and hematology tests

Palembang Oct 2017

Fig. 3. Monthly Sigma trend plots (dashboards) for selected chemistries (CHEM)
on Roche analytical platforms. ALK PHOS, alkaline phosphatase.

Ref: Harrison et al Using Sigma Quality Control to verify and monitor performance in multi-instrument,multisite
Integrated Health Care Network Clin Lab Med 2016
Alanine aminotransferase (ALT) Sigma by instrument type: 2012 to 2015.

Ref: Harrison et al Using Sigma Quality Control to verify and monitor performance in multi-
instrument,multisite Integrated Health Care Network Clin Lab Med 2016
Ref: Harrison et al Using Sigma Quality Control to verify and monitor performance in multi-instrument,multisite
Integrated Health Care Network Clin Lab Med 2016 (AIP)
 Application in Medical Laboratory
Monitoring performance –multiple analyzers, multiple sites, over
months and years

• Results of the study showed:

– Clear variation among analytes and analyzers
– Change in QC rejection rules for Sigma>4; 76.9% reduction
in QC rejection without affecting pt’s result and EQA
performance
• Conclusion:
– Sigma values enable the laboratory management to
monitor performance of the method, analyzers and staff
competency.
Ref: Harrison et al Using Sigma Quality Control to verify and monitor performance
in multi-instrument,multisite Integrated Health Care Network Clin Lab Med 2016
(AIP)
Tool for Continuous quality improvement (CQI)
activity

• Dr Navapun Charuruks (Bumrungrad Hospital,Thailand)

monitored performance of 28 chemistry assays quarterly
using sigma value

Palembang Oct 2017

Analytical performance based on Sigma metric
Tool for Continuous quality improvement (CQI)
activity

• Results of the study showed:

– Most analytes achieved the desired quality based on
sigma metric
– For analyte that showed deterioration on performance,
the root cause of the problems were identified and
necessary CA were taken
• Conclusion:
– Sigma metric can be used as a tool for CQI

Palembang Oct 2017

Sigma Metrics System in Medical Laboratory

Chemical Pathology Unit

Dept of Diagnostic Laboratory Services
Hospital Cancelor Tuanku Muhriz, UKMMC
EXPERIENCE
CHEMICAL PATHOLOGY UNIT
JPMD, HCTM UKMMC
Journey to Six Sigma QC practice
 Chemical Pathology Unit

• Workload : 2000 samples/day

• Test menu: Clinical Chemistry,
Hormones, TDM, ABG, osmolality,
HbA1c, Protein Electrophoresis,
UFEME, ESR.
• Staff strength: 5 SOs, 33 MLTs, 10
others
• Staff trained and competent.
• 99% tests offered are ISO 15189
accredited.
• Instrumentation: 2 high throughput
Chem analyzers, 2 high throughput IA
analyzers and TLA

Palembang Oct 2017

 Achievements

ISO 9001 Certification – 2003

ISO 15189 Accreditation – 2010

(99% tests accredited)

Westgard Sigma Verification of

Sigma-Metric Performance -
2015

Reference Centre for Abbott

i3600 TLA System-2015
 Evolution in QC Practices
1997 2003 2007

Traditional QC Electronic QC Electronic QC

1. n=2,R=1: All Assays 1. n=2,R=1: All Assays
1. n=2,R=1: All Assays
2. Manufacturer QC materials 2. Manufacturer QC range
2. Manufacturer QC materials
3. Manufacturer QC range 3. Third party QC materials
3. Manufacturer QC range
4. Any rules/no rules 4. Single QC rule, 12s, Multi
4. Single QC rule, 12s
QC rules

2010 2013 2015

Total Quality Management QC
Electronic QC 1. n=2 ,R=2: All Assays Six Sigma QC
1. n=2, R=2: All Assays
2. Third party QC materials 1. n,R, depends on sigma
2. Third party QC materials
3. Establish Lab QC range quality
3. Establish Lab QC range
4. Multi QC rules (12s,13s,22S..) 2. Sigma Westgard rules
4. Multi QC rules (12s,13s,22S..)
5. Use of UNITY real time 3. Use of UNITY Program
connectivity
Key Drivers to 6 Sigma QMS
 Key Drivers to 6 Sigma QC Management

• To acquire timeliness quality laboratory results;

• To meet the evolving needs of our patients;
• To be constantly challenged with the aim to maintain
high analytical quality in our clinical laboratory;
• To change “art of practice” to a “scientific process”
To meet the healthcare system demand:
↑product (TAT,Q), ↓waste, ↓ cost

Palembang Oct 2017

Westgard Sigma Verification Program (WSVP)
Journey to Sigma Verification Program

• July 2015
• Submitted performance data for Clinical Chemistry and
Urine assays from 2 ARCHITECT c8000 analyzers
• 34 analytes-21 chemistry tests/13 urine tests
• Laboratory QC Protocol
• 3 Quality officers-training & competency assessment
• Data were verified by Westgard QC Inc. in October 2015
 Distribution of Sigma performance for HCTM UKMMC
(34 clinical chemistry and urine assays)

%
85.3 % > 4 σ

Sigma value
Sigma Method Decision Chart

Normalized Sigma-method Decision Chart

5.10.2015
Verification by Westgard QC.Inc
 QC Practice
1997 2003 2007

Gen 1 :Traditional QC Gen 2: Electronic QC Gen 3: Electronic QC

1. n=2,R=1: All Assays 1. n=2,R=1: All Assays
1. n=2,R=1: All Assays
2. Manufacturer QC materials 2. Third party QC materials
2. Manufacturer QC materials
3. Manufacturer QC range 3. Manufacturer QC range
3. Manufacturer QC range
4. Any rules/no rules 4. Single QC rule, 12s, Multi
4. Single QC rule, 12s
QC rules

2015
5th Gen QC
1. n,R –depends on analytical
performance
2. Sigma Westgard rules 2013
2010 Gen 4 :Total Quality Management
3. Use of UNITY program
Gen 3: Electronic QC QC
1. 4. RCPA
n=2, R=2: All Assays 1. n=2 ,R=2: All Assays
2. Third party QC materials 2. Third party QC materials
3. Establish Lab QC range 3. Establish Lab QC range
4. Multi QC rules (12s,13s,22S..) 4. Multi QC rules (12s,13s,22S..)
5. Use of UNITY real time
connectivity
Westgard Sigma Verification
Program (SVP)
 Westgard Sigma Verification Program
• Introduced in April 2014
• An important distinction: WSVP is not a certification, not an
accreditation, not an award.
• The Westgard Sigma Verification Program standardizes quality
in the medical laboratory.
• Purpose:
– To encourage laboratories to define their own quality
requirements for intended use and to evaluate their own
performance relative to their goals for quality based on sigma
quality.
– To provide an evidence-based, data-driven approach to quantify
the quality being achieved by a laboratory method as well as
confirmation that the laboratory can routinely deliver that
quality by detecting medically important errors.

Palembang Oct 2017

 ISO 15189 vs SVP

ISO 15189 SVP

5.6.2.1 – Design QC procedure and Demand on the used of right QC

prove of implementing QC

5.3.1.1- Selection of instrument

The instrument and method is
5.5.1.1- select examination
performing not just adequately
procedures and performance
but at 4-sigma or more
specifications

4.1.2.4 - Require Lab to set quality Provides a set of TEa for use in
goals benchmarking the quality of test
 ISO 15189 vs SVP

In other words….SVP is not positioned at the high, abstract

altitude of "Quality System" - it's much more focused on the
performance.
The two concepts work hand in hand. You can't really have a
Quality System without actually having appropriate method
performance.
And if you want to get the right method performance, having a
Quality System is going to help you accomplish that much more
easily.
James O. Westgard

Palembang Oct 2017

Sigma Verification Program
Benefits and Impact
2015-2016
 Benefits of Westgard SVP program

• Change the QC practice “usual practice” to “evidence based

QC”
• Direct support- Westgard Inc-Dr James Westgard, Mr Sten
Westgard, Westgard QC team
• Improve knowledge and competency of technical personnel
• Universal benchmark for quality of analytical performance
• CQI of quality performance in analytical process

• Drawback:
– Cost

Palembang Oct 2017

 Impact of Westgard SVP
6σ QMS Performance improvement
comparison

QC frequency reduction

↓ cost ↓waste CQI

Reagent ,QC material

QC run
cost Performance of each
analyte, 2 analyzers
Initial-12 MO
Labour cost False rejection

Unnecessary
troubleshooting
 Impact of Westgard SVP
• Cost and waste reduction:
– Step 1: Determine QC rule(s) based on sigma metric value
– Step 2: Overlay new QC practice on old QC practice
• Old practice: Aug 2014-Sept 2015 (13 MO)
• Sigma QC : Nov 2015-Dec 2016 (13 MO)
– Step 3: Determine effectiveness: number of rejection and
false rejection and cost reduction

• CQI : Performance – compare sigma value initial and

current
– 2 Chem analyzers – C8000-1 (27 assays) and C8000-2 (44 assays)

Palembang Oct 2017

 Impact of Westgard SVP

6σ QMS Performance improvement

comparison

QC frequency reduction

↓ cost ↓waste CQI

Reagent ,QC material

QC run
cost Performance of each
analyte
Initial-6 mo
Labour cost False rejection

Unnecessary
troubleshooting
 Total Cost Reduction
MYR 95 030 OR 19.1 % reduction in total cost
600000

MYR 496 995

500000

MYR 401 965

400000

300000 19.1 %

200000

100000

0
pre implementation post implementation
 Impact of Westgard SVP

6σ QMS Performance improvement

comparison

QC frequency reduction

↓ cost ↓waste CQI

Reagent ,QC material

QC run
cost Performance of each
analyte
Initial-6 MO
Labour cost False rejection

Unnecessary
troubleshooting
 Overall performance of 61 Chemistry assays (serum & urine)
on Abbott C8000 analyzers between 2015 & 2016

40
35
2016 : 85.2 % assays ≥ 4 Sigma
35
31
2015 : 77.0 % assays ≥ 4 sigma
30

25
No

20 2015
15 2016
10
10 9
8
7 7
5
5 3
4
3

0
0
≥6 ≥5 ≥4 ≥3 ≥2 <2
Sigma value
 Comparison of performance of 27 chemistry assays
(serum) on C8000-1 between 2015 & 2016

16
14
2016 : 81% assays ≥ 4 Sigma
14 2015 : 74% assays ≥ 4 sigma
12
10
10

8 2015
No

6
6 2016
5
4
4 3 3 3
2 2 2
2
0
0
≥6 ≥5 ≥4 ≥3 ≥2 ≤2

Sigma value
 Comparison of performance of 22 chemistry assays
on C8000-2 between 2015 & 2016

12
11
2016 : 86.4 % assays ≥ 4 Sigma
10
9
2015 : 68.1% assays ≥ 4 sigma
8
7

6 2015
No

5
2016
4
3 3

2
2
1 1 1 1

0
0
≥6 ≥5 ≥4 ≥3 ≥2 <2

Sigma value
 Comparison of performance of 12 chemistry assays
(urine) on C8000-2 between 2015 & 2016

2016 : 91.6 % assays ≥ 4 Sigma

14 2015 : 100 % assays ≥ 4 sigma
12
12
10
10
No

8
2015
6
2016
4

2 1 1

0
≥6 ≥4
Sigma value
≥3
 Impact of Westgard SVP

6σ QMS Performance improvement

comparison

QC frequency reduction

By adopting and implementing

↓ cost ↓waste Six Sigma quality
CQI
control strategies, we were able to improve
laboratory
Reagent ,QC material operation efficiency by reducing the
QC run
cost
false QC rejection rate, which in turnPerformance
enabledofaeach
analyte
direct cost-saving benefit without compromising Initial-6 MO
the
Labour costdetection ofFalse
errors. These changes have
rejection

enabled us to continuously provide efficient,

high-qualityUnnecessary
service at a low cost.
troubleshooting
 Challenges

• Change of practice: from traditional to current

• Learning new concept-training and competency for
staff
• Not about achieving … but sustainability… strive to
be better…
• Continual improvement
 Summary

• Laboratories have to meet the current healthcare

system demand
• Sigma QMS provides universal benchmarking for the
performance of laboratory processes.
• Triple WIN:
– Easy for the lab
– Cheaper for the healthcare system
– Better for the patient

If nothing ever changed, there’d be no butterflies

unknown
………..behind the scene

Shahril Nazrul Steward

Ida
 Sigma Metrics System in Medical Laboratory
"Cobo-cobo ningkep belalang
Tapi belalang pinter terbang
Jangan lupo dateng ke Palembang
Kota Palembang lemak dipandang”
dr. Asep Zainuddin, S

Kota Palembang indah sekali,

Venice timur gelaran diberi;
Terima kasih menjemput kami,
Hati tertambat rasa-rasa mau mampir lagi….

Terima Kasih