3/23/2015

HOW TO MANAGE LAB ERROR &

PASS PROFICIENCY TESTING
Lt Col Ramil C. Codina, MS, MT(ASCP) COL Donald L. Taillon, MD (FCAP)
Diagnostics & Therapeutics Flight Commander at Tyndall AFB, FL Laboratory Medical Director at Ft. Steart, GA
ramil.codina@us.af.mil donald.l.taillon.mil@mail.mil

Disclaimer:

The views expressed are those of the
authors and do not reflect the official policy of 
the Department of Defense, Army, Air Force or 
the U.S. Government. 

1
 3/23/2015

GOALS

• Reinforce PT Basics
• Offer troubleshooting guidelines
• “Total QA Package”
• Proactive approach - prevention

OUTLINE

• Reasons for PT
• Basics of PT
• Troubleshooting
• Probability of Failure
• Recommendations – Prevention
of Failure

2
 3/23/2015

WHY DO PT?
• REGULATORY REQUIREMENTS
• CLIA’ 88 - the dreaded * and #
• QUALITY ASSURANCE TOOL
LOOKING BEYOND THE * and #
• Accuracy
• Precision
• Comprehensive PI program– QC, calibration,
maintenance, variances, patients

BIG BROTHER
• CLIA ’88, HCFA (now CMS) mandated
• All labs performing PPM, moderate
and/or high complexity testing
• Regulated vs. Non-regulated
• CAP requires PT for all analytes
• Semi-annual split sample testing

3
 3/23/2015

SURVEYS

• 2015 CMS Approved listing:

http://www.cms.gov/Regulations-and Guidance/
Legislation/CLIA/Downloads/ptlist.pdf

• CAP, ASCP, AAB, API etc…

• State Departments of Health
• American Academies FP-PT, CA
Thoracic Society, etc
• Others – commercially available

THE BASICS
• THE TESTING PROCESS
• Handling of PT material
• Performance limits and categories

• THE RESULTS -
INTERPRETATION
• Failures - * and #
• Looking beyond * and # (QA/QI)

4
 3/23/2015

TEST LIKE A PATIENT

• Testing personnel
• Read instructions carefully
• Check for clerical errors
• Duplicate Testing
• Attestation Statement
• Keep all records and samples

LIMITS
• TARGET VALUES
• PEER GROUP MEANS
• Size, variability, methodology

• ACCEPTABLE PERFORMANCE
• CLIA PT LIMITS (more to follow)
> Percent (Glucose +/- 10%)
> Quantity (Calcium +/- 1.0 mg/dl)
> SD (+/- 3 SD) 2.9 vs. 3.1

5
 3/23/2015

LIMITS – Converting PT Limits to CV

• Creatinine PT Limits: +/- 0.3 mg/dl or 15%, whichever is greater

• Survey C-01
> Mean = 1.5
– 15% x 1.5 = 0.22 (<0.3 mg/dl so use 0.3 mg/dl criteria)
– 1.5 + 0.3 = 1.8
> CV = SD/mean
> CV = 0.3/1.5 = 0.2 or 20% so, 1/3 CV = 6.7%

• Survey C-02
> Mean = 4.94
– 15% x 4.94 = 0.74 (>0.3 mg/dl so use 15% criteria)
– 4.94 + 0.74 = 5.68
> CV = SD/mean
> CV = 0.17/4.94 = 0.34 or 34% (>15%), 1/3 CV = 5%

GAUSSIAN DISTRIBUTION

6
 3/23/2015

CATEGORIES
• Current Event Performance
• Satisfactory 80% or better
• Unsatisfactory < 80%
• Except ABO Rh typing & compatibility testing

• Cumulative Performance
• Successful 80% for analyte or subspecialty
• Unsuccessful <80% for analyte or subspecialty
• Exception BB

CATEGORIES (cont.)
• Unsuccessful Performance –
Failure in 2 out of 3 challenges
<1> Means currently successful - At risk for next survey
<2> Means currently successful - At risk for next two
surveys
<3> Means currently unsuccessful - At risk for the next
three surveys

• Failure to return a survey is

considered a failure

7
 3/23/2015

UNSUCCESSFUL PERFORMANCE
• CMS, formerly HCFA
• Require a written response, action plan
and allow continued testing
• May implement sanctions such as
–Financial penalties
–Mandatory suspension
–Revocation of the lab’s certification

FAILURES – TROUBLE SHOOTING

*Unsatisfactory – initial investigation

• Consider suspending testing
• Clerical check
• Methodology codes
• CAP scanning error
• Improper reconstitution/sample
handling

8
 3/23/2015

TROUBLE SHOOTING

• QC – in range, shifts or trends,

history of CVs (time of testing)
• Maintenance records
• Calibration & Verification –
correct and up to date
• Reagents and controls – in date

9
 3/23/2015

TROUBLE SHOOTING
PATTERNS

• High/Low values – suspect

LINEARITY

• Only one out of many tests on

same instrument – suspect
REAGENT

TROUBLE SHOOTING
PATTERNS

• Multiple tests/PT samples on

same instrument - suspect
INSTRUMENT

• Several tests on same PT sample

-suspect
SAMPLE integrity

10
 3/23/2015

TROUBLE SHOOTING
• Retest PT sample
• Split sample testing
• Prove degradation
• Peer group is optimal
• Educational challenges
• CMS (HCFA) vs. patients

CODE [26]
[26] = Educational Challenge

Are not considered educational,

if you’re the patient

11
 3/23/2015

THE ASTERISK

There’s more to it

*
than meets the eye

CODE [26]
[26] = Educational Challenge

Hey, don’t sweat it. It’s

only a challenge…

Oh no, it’s the ER calling about that

Troponin in the patient with chest pain…

12
 3/23/2015

TROUBLE SHOOTING
• Understand the problem
• Understand Clinical Significance
• Consider assistance
• Manufacturer or Consultant
• Discontinue testing, if necessary
• Patients, patients, patients
• Document, document, document

“Doctor, your patient has a Glucose

of 225 mg/dL.”

13
 3/23/2015

ACCURACY
• BIAS or SYSTEMATIC ERROR
• INTER-LABORATORY - EXTERNAL QC
• PEER GROUP COMPARISON
• EVALUATE (even if no *)
• Consistently over or under target
• How much is too much?
• Examine QC and Calibration history
• Verify calibration or recalibrate

BIAS LIMITS
• CAP
• SDI +/- 1.5
• Westgard
• SDI 1.0
• Poor Man’s (Person’s) Guide
• < 20 % of HCFA (CMS) limits

14
 3/23/2015

PRECISION
• VARIABILITY or RANDOM ERROR
• INTRA-LABORATORY – INTERNAL QC
• COEFFICIENT OF VARIATION (CV)
• EVALUATE (even if no *)
• GOAL
• MAXIMUM CLIA LIMITS?
• RULE OF ONE THIRD

PROBABILITY OF FAILURE
• CV = CLIA LIMIT (Glucose +/- 10%)
• Lab would fail (2 out of 5 misses) 50% of time
• CV = 50% CLIA LIMIT (Glucose +/- 5%)
• Lab would fail 2% of time
• CV = 33% CLIA LIMIT (Glucose +/- 3%)
• Lab would not be expected to fail

• Assumes 0% bias
• CLIA limit represents TE (Glucose +/- 6 mg/dL or
10%, whichever is greater

Laessign-Ehrmeyer, et al. 1990

15
 3/23/2015

PROBABILITY OF FAILURE

Laessign-Ehrmeyer, et al. 1990

31

SIGNIFICANCE
• What error is acceptable for your lab?
• Random vs. systematic
• Instrumentation performance
characteristics
• Clinical significance
• It’s ultimately all about patient care
• Medical decision limits

16
 3/23/2015

RECOMMENDATIONS
• Reduce CV to 1/3 CMS PT limit
• Eliminate Bias or < 20% of CMS PT limit
• Thoroughly investigate PT Survey information, not
only failures
• Educational challenges
• Multiple instrument comparisons - intralab
• Other methodologies and caveats
• READ THE BOOKS (websites)!
• Time

RECOMMENDATIONS
• Incorporate PT into a
Comprehensive Quality Plan (PI)
• Quality Control
• Incidents/variances
• MFR recommendations
• Maintenance
• Calibration
• and most importantly Patients

17
 3/23/2015

Even with a perfect proficiency

survey, there is much to be
learned.

Are You Sure? (10 minute break)

18
 3/23/2015

Evaluate PT Surveys

CAP Troubleshooting Guide

www.cap.org

19
 3/23/2015

CAP Troubleshooting Guide – Table 2

www.cap.org

CAP TS Guide – cont. Table 2

www.cap.org

20
 3/23/2015

CAP Troubleshooting Guide – Table 3

www.cap.org

CAP TS Guide – cont. Table 3

www.cap.org

21
 3/23/2015

Examples – Cases

43

Case 1a - Transcription // Clerical Error

22
 3/23/2015

Case 1b/c – Transcription // Clerical Error

[20] = No appropriate target/response could not be graded
[28] = Response qualified with a “ > ” or “ < “ sign; or unable to quantitate

45

Case 2 – Bias

46

23
 3/23/2015

Case 2 Cont - Bias

47

Precision –
Method Decision Data
FIB LOW ABNORMAL A NORMAL C
n MLA 1400 ACL 9000 MLA 1400 ACL 9000 MLA 1400 ACL 9000
1 73 67.4 169 173 306 272
2 71.5 66.9 175 171 300 296
3 73 67.1 167 160 309 300
4 72.5 67.5 169 168 297 296
5 71.5 67 174 168 297 308
6 72.7 67 165 176 297 308
7 70.8 67.3 166 175 294 316
8 72 72.3 160 173 297 308
9 71.5 67.5 176 176 306 312
10 68.7 70.3 177 167 305 316
11 73.5 68.5 170 172 298 300
12 71.5 70.6 180 176 302 300
13 73 68 173 154 305 289
14 70.6 71.3 168 161 298 292
15 69.6 67.8 169 164 297 308
16 71.2 72 160 174 302 297
17 69.5 68.3 162 170 305 304
18 70.2 71 172 167 295 316
19 68.7 66.6 169 177 297 308
20 69.2 67.8 167 161 302 285

n 20 20 20 20 20 20
Mean 71 69 169 169 300 302
SD 1.5 1.9 5.4 6.4 4.3 11.3
CV 2.1% 2.7% 3.2% 3.8% 1.4% 3.8%
Bias 2.6 0 -1
Bias % 3.7% 0.1% -0.4%

48

24
 3/23/2015

Method Decision Chart

Method Decision Chart

Test Method: FIB-C @ 71 mg/dL CLIA: +/- 20% @150 mg/dL

Allowable Total Error (% ): 20.0 Operating point: +

Imprecision (CV): 2.10
Inaccuracy (bias% ): 3.70 Method Performance: Excellent

25
Observed Inaccuracy

20
(bias% )

15
poor
10
excellent
5 good fair
0
0 1 2 3 4 5 6 7 8 9 10 11
Observed Imprecision (CV)

Reference: Westgard, JO. Clin Lab Science. 1995;8:277-83.

Developed by: Jose Carlos Basques, Marcelo Basques, and Marcio Basques.

1/3 CLIA = 6.7% / 1/4 CLIA = 5% / 1/5 CLIA = 4% / 6 Sigma = 3.3%

www.westgard.com

49

Reportable Range - Linearity

• Analyte pCO2, mmHg

1. Is the line straight? 120
2. Is it visually linear? 100
3. What makes it linear?
4. Why? 80
5. It looks straight, passes
60
through zero, slope doesn’t
change, TE < TEa
40
6. AMR, CRR, Reportable
Range? (CAP requires 20
definition of each)
0
0 20 40 60 80 100 120

50

25
 3/23/2015

Method – Is the method the problem?

1. ACCURACY (Rhoads EP Evaluator): A Comparison of Methods
Experiment for the new I-Stats compared: pH, pCO2, PO2, Na, K, and Ca
with the old I-Stats. 42 specimens were analyzed in a uniformed “bin-box”
manner. All six analytes demonstrated a correlation coefficient (r) greater
than 0.975, non-significant Student’s T-test result, and acceptable regression
analysis. Bias was judged to be acceptable at the medical decision levels.
2. PRECISION (Westgard) A replication experiment was accomplishedto
estimate the imprecision of the new I-stats versus the old I-Stat. Three
levels of quality control materials that covered the medical decision points
were analyzed 21 times over a period of 10 days. CVs at all levels (x,y,and
z) were less than 1/3 CLIA Limits and judged to be acceptable by using a
method decision chart.
3. REPORTABLE RANGE (Rhoads EP Evaluator): A linearity experiment
was performed to estimate the new I-Stat’s accuracy, linearity, and
reportable range using an I-Stat linearity kit. Testing was performed in
duplicate and the average value was plotted. At least 5 samples were tested
for each analyte that encompassed low, mid and high values – see
attachment.
4. REFERENCE RANGE: Normally, we would analyze specimens
collected for physical exams or volunteers to verify the reference range. In
lieu of obtaining 21-41 arterial blood gasses from normal healthy individuals,
the medical director has approved the use of manufacturer’s ranges.
Codina, Taillon, Harms_SAFMLS 2005
51

REFERENCES
• Ehrmeyer, SS, Laessig, RH: The New Poor Man’s (Person’s,
2011). Guide to the Regulations. Madison, WI: R&S
Consultants, 2001.
• Publications by Laessig RH, Ehrmeyer SS, et al.:
> Clinical Chemisry 1990;36;1736-40

> Archives Pathology and Lab Med. 1992;116(7):770-76

• Laboratory Accreditation Program. College of American

Pathologists. Northfield, IL.
• U.S. Department of Health and Human Services. Medicare,
Medicaid and CLIA programs: Regulations implementing the
Clinical Laboratory Improvement Amendments of 1988
(CLIA). Final rule. Fed Regist 1992; 57:7002-186.

26
 3/23/2015

Questions

53

27