See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/319706625

Assessing precision, bias and sigma-metrics of 53 measurands of the Alinity ci

system

Article  in  Clinical Biochemistry · September 2017

DOI: 10.1016/j.clinbiochem.2017.09.005

CITATIONS READS

19 3,324

6 authors, including:

Sten Westgard Jörg Herzogenrath

Westgard QC, Inc. Abbott Laboratories
36 PUBLICATIONS   737 CITATIONS    4 PUBLICATIONS   70 CITATIONS   

SEE PROFILE SEE PROFILE

Anthony Orzechowski
Northwestern University
3 PUBLICATIONS   19 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Risk, Error and Uncertainty: Laboratory Quality Management in the Age of Metrology View project

All content following this page was uploaded by Sten Westgard on 17 February 2019.

The user has requested enhancement of the downloaded file.

 Clinical Biochemistry xxx (xxxx) xxx–xxx

Contents lists available at ScienceDirect

Clinical Biochemistry
journal homepage: www.elsevier.com/locate/clinbiochem

Assessing precision, bias and sigma-metrics of 53 measurands of the Alinity

ci system
Sten Westgarda,⁎, Victoria Petridesb, Sharon Schneiderb, Marvin Bermanb, Jörg Herzogenrathc,
Anthony Orzechowskib
a
Westgard QC, Inc, Madison, WI, USA
b
Abbott, Abbott Park, IL, USA
c
Abbott, Wiesbaden, Germany

A B S T R A C T

Assay performance is dependent on the accuracy and precision of a given method. These attributes can be
combined into an analytical Sigma-metric, providing a simple value for laboratorians to use in evaluating a test
method's capability to meet its analytical quality requirements. Sigma-metrics were determined for 37 clinical
chemistry assays, 13 immunoassays, and 3 ICT methods on the Alinity ci system.
Methods: Analytical Performance Specifications were defined for the assays, following a rationale of using CLIA
goals first, then Ricos Desirable goals when CLIA did not regulate the method, and then other sources if the Ricos
Desirable goal was unrealistic. A precision study was conducted at Abbott on each assay using the Alinity ci
system following the CLSI EP05-A2 protocol. Bias was estimated following the CLSI EP09-A3 protocol using
samples with concentrations spanning the assay's measuring interval tested in duplicate on the Alinity ci system
and ARCHITECT c8000 and i2000SR systems, where testing was also performed at Abbott. Using the regression
model, the %bias was estimated at an important medical decisions point. Then the Sigma-metric was estimated
for each assay and was plotted on a method decision chart. The Sigma-metric was calculated using the equation:
Sigma-metric = (%TEa − |%bias |)/%CV.
Results: The Sigma-metrics and Normalized Method Decision charts demonstrate that a majority of the Alinity
assays perform at least at five Sigma or higher, at or near critical medical decision levels.
Conclusion: More than 90% of the assays performed at Five and Six Sigma. None performed below Three Sigma.
Sigma-metrics plotted on Normalized Method Decision charts provide useful evaluations of performance. The
majority of Alinity ci system assays had sigma values > 5 and thus laboratories can expect excellent or world
class performance. Laboratorians can use these tools as aids in choosing high-quality products, further con-
tributing to the delivery of excellent quality healthcare for patients.

1. Introduction up and confirm the initial findings.

The Reproducibility Crisis is not confined to the research side of
According to a recent editorial by John P.A. Ioannidis [1], we are in medicine. There is actually an even more problematic issue concerning
the midst of a Reproducibility Crisis in medical research, which has the analytical performance of routine laboratory methods and assays –
generated a lack of confidence in the results produced by laboratory call it a Producibility Crisis. It is not only a challenge to reproduce
medicine. This is not, as some suspect, a problem with politics and findings of analytical performance, it is surprisingly difficult to get an
denial of science, but something worse – a true inability to reproduce initial set of data that's reliable in the first place. Typically, a short-term
many of the findings of scientific publications, meaning these novel study or abstract is published whenever a new method or measuring
findings and acclaimed breakthroughs may not be real. Reproducibility system is introduced, but rarely are these initial results expanded into
of scientific findings is increasingly difficult – today's great discovery is full-blown papers, nor are they routinely followed by longer term stu-
too often tomorrow's statistical fluke. The publication bias of many dies that demonstrate the routine performance in “real world” labora-
journals means they are highly motivated to showcase the most unusual tories. Many follow-up studies are dismissed from publication as being
and positive results, while neglecting to publish any papers that follow “merely confirmatory”, as if the verification of manufacturer claims was

⁎
Corresponding author.
E-mail address: westgard@westgard.com (S. Westgard).

http://dx.doi.org/10.1016/j.clinbiochem.2017.09.005
Received 4 July 2017; Received in revised form 10 September 2017; Accepted 10 September 2017
0009-9120/ © 2017 The Canadian Society of Clinical Chemists. Published by Elsevier Inc. All rights reserved.

Please cite this article as: Westgard, S., Clinical Biochemistry (2017), http://dx.doi.org/10.1016/j.clinbiochem.2017.09.005
S. Westgard et al. Clinical Biochemistry xxx (xxxx) xxx–xxx

a topic unworthy of scientific attention. acceptable performance level of a process, generates almost 67,000
Manufacturers have to face even more challenging aspects of the defects per million outcomes. As the Sigma-metric of a process rises, it
Producibility Crisis. First and foremost, there is an assumption that all becomes more efficient, reliable, and profitable. A Six Sigma process is
manufacturers should standardize and harmonize their methods so that nearly defect-free, inarguably the goal that healthcare laboratories
test results are comparable from competitor to competitor. Even for the strive to deliver to clinicians and patients.
simplest, well established tests, comparability has been disproven; Six Sigma has been used in industry for > 30 years, and has also
manufacturers are selling methods that have significantly different been used in healthcare for decades. Its use in the laboratory for
performance and results than their competitors [2]. Ideally, assays benchmarking analytical quality is more recent [4]. The Sigma-metric
would be traceable to a standard, or at least harmonized, but we un- assesses the analytical quality of a method by evaluating both the ob-
fortunately are not there yet. Even for assays that are standardized, the served accuracy and the precision. The combination of these two at-
matrix in which specimens are collected, endogenous and exogenous tributes into a Sigma-metric allows the laboratory to ensure test
substances and certain medical conditions can all contribute to differ- methods not only meet minimum standards but also contribute to high
ences among methods. Second, manufacturers must assure that the quality clinical outcomes. The Sigma-metric provides a simple objective
performance they claim in their marketing will actually be achieved value for laboratorians to use in evaluating test methods. Increasingly,
when customers implement their systems in daily routine operation. the Sigma-metric has been a common way to assess instruments and
Third, they must assure that across their own multiple instruments and methods.
different product lines, they still provide a comparable result. Given
today's consolidation of diagnostic manufacturers into an ever-smaller
handful of players, it is not uncommon for disparate instrument lines to 2. Materials and methods
fall under the same corporate banner. It is no longer guaranteed that by
going with Company B, laboratories will automatically have compar- Abbott Laboratories has introduced a new line of instruments into
able results across all of B′s instruments, because within Company B, the marketplace – the Alinity ci system [c is chemistry and i is im-
there may be instruments from diagnostic line C, and completely dif- munoassay]. The Alinity ci system allows random and continuous ac-
ferent instruments from a diagnostic line D (which was probably from a cess, as well as priority and automated retest processing.
company purchased, merged, and/or absorbed by the larger B con- The Alinity c system is a fully automated chemistry analyzer de-
glomerate). We have already seen that the major diagnostic manu- signed to generate up to 1350 tests per hour using photometric and
facturers have instruments that are not comparable with each other, potentiometric detection technology. The Alinity c system uses photo-
much less the competitor instruments [3]. metric detection technology to measure sample absorbance for the
Nevertheless, when laboratories evaluate a new instrument or quantification of analyte concentration. A calculated absorbance is
method for purchase, one of the most critical factors they consider is the generated based on the reaction mode of the assay (rate or endpoint)
analytical performance that is routinely exhibited by the device. Cost, and the calculated absorbance is measured using a calibration curve to
convenience, informatics, etc., are all important elements in the deci- generate a result. In addition, the Alinity c system uses potentiometric
sion to purchase an instrument, but the core competency of an instru- detection to quantify sample electrolyte concentration with an
ment must be its ability to produce the right result. Where the Integrated Chip Technology (ICT) module that contains Na +, K +, Cl-,
Reproducibility Crisis hits home for labs is the mismatch between and reference electrodes.
marketing claims and the analytical performance. This gap between The Alinity i system is a fully automated immunoassay analyzer
promise and performance could be so significant as to call into question designed to generate up to 200 tests per hour. The system utilizes
the fitness for purpose of the method or instrument for appropriate paramagnetic microparticles and chemiluminescent detection based on
clinical care. Acridinium labeled conjugate. Based on the assay format, the resulting
Without frequent and continuous publication of analytical perfor- relative light units (RLUs) of the chemiluminescent reaction have a
mance, the laboratory marketplace remains a blind arena, where ad- direct or indirect relationship with the amount of analyte (antigen or
vertising campaigns and marketing puffery provide a one-sided source antibody) present in the sample.
of information to the customers. Sigma-metrics were determined for 37 clinical chemistry assays, 13
Ideally, performance data on an instrument should be made avail- immunoassays, as well as for Chloride, Potassium, and Sodium (elec-
able early and often. At the beginning of a new instrument launch, the trolytes) on the Alinity ci system.
manufacturer should provide not only the typical poster or abstract, but The clinical chemistry assays included are shown in Table 1.
also a fully fleshed out analysis based on the deep experience of the The immunoassays included are shown in Table 2
development process. Every manufacturer collects a wealth of data in And as noted above, the Sigma-metric performance of the ICT
the course of verifying and validating the instrument performance and
assuring that the methods meet their design goals. Too often, this data Table 1
Chemistry analytes assessed on the Alinity c system.
is not shared with the laboratory community. But these results can be a
critical starting point, partly because the initial validation studies may Activated Alanine Non-activated Activated Aspartate Non-activated
be more thoroughly conducted by the manufacturer than the typical Aminotransferase ALT Aminotransferase AST
routine validation or verification protocols performed by an individual (ALT) (AST)
laboratory, and partly because when an instrument is truly new to
Albumin (BCG) Albumin (BCP) Alkaline phosphatase Amylase
market, few laboratories have had the chance to install and study its Calcium Carbon dioxide Cholesterol Creatinine
performance. kinase
Even when such data is provided, laboratories need a way to eval- Creatinine (regular) Creatinine CRP Digoxin
uate and compare performance. Six Sigma is a widely-accepted quality (enzymatic)
Direct bilirubin Direct LDL GGT Glucose
management technique, which at its essence focuses on the elimination Iron Lactate Lactic acid (plasma) Lithium
of defects, specifically the number of defects that occur per million dehydrogenase
outcomes of a process (DPMO, often simplified to DPM). On the short- Magnesium Microalbumin Phenytoin Phosphorous
term Sigma scale, a Six Sigma process has fewer than 4 defects per (urine)
Salicylate Total bilirubin Total protein Transferrin
million outcomes of the process while allowing for expected drifts and
Triglycerides Ultra HDL Urea nitrogen Uric acid
shifts that will occur over the life of the reagents and measuring system. Vancomycin
In contrast, a Three Sigma process, often considered the minimum

2
S. Westgard et al. Clinical Biochemistry xxx (xxxx) xxx–xxx

Table 2 there was only one published paper on the biologic variation of the
Immunoassay analytes assessed on the Alinity i system. analyte [12], so the performance specification was derived, via Fraser's
equations, from that data. This study did not seek to choose the smallest
Total β-hCG CA 125 II CA 15–3 CA 19-9XR
goals, for many of the “Ricos goals” for instance, cannot be achieved by
Cortisol Estradiol Free PSA Free T3 any methods or instruments currently on the market. Instead, the aim of
Free T4 Prolactin Total T4 TSH selecting performance specifications was to find challenging but prac-
25-OH Vitamin D
tical goals. This selection of goals is not without controversy, but this
logic used in this study has already been implemented in a global Sigma
Chloride, Potassium, and Sodium assays were also studied. verification of performance program [13]. Nevertheless, this specific
A Sigma-metric was estimated for each assay and was plotted on a selection of performance specifications may not concur with the pre-
method decision chart [5]. The sigma value was calculated using the ferred goals of some laboratories. The tables contain all the necessary
equation: Sigma-metric = (%TEa − |%bias |)/%CV. The TEa is the al- data for anyone to recalculate the Sigma-metrics using the goals of their
lowable total error, which provides the tolerance limits around the preference.
“true value” that an analytical method is trying to produce. As long as The imprecision for each assay was estimated via a precision study
analytical methods can confine their precision and bias to fall within conducted at Abbott using the Alinity ci system following the guidelines
the allowable total error, the test will produce very few medically im- of CLSI EP05-A2 [14]. At least 2 panels along with either 2–3 Tech-
portant errors (defects). nopath controls (single constituent controls for Microalbumin, multi-
Around the world, multiple sources of allowable total errors are constituent controls for all other clinical chemistry and ICT assays) or
available, most typically in the EQA/PT programs from different 2–3 single analyte controls (for immunoassays) were tested. The con-
countries or regions. However, the US CLIA PT criteria [6] are legally centrations of the controls and panels spanned the measuring interval of
mandated in the Federal Register, and these are accepted as valid al- the assays. Each control and panel was tested in replicates of 2–3 during
lowable total errors around the world through the influence of CAP 2 runs each day for at least 20 days, and an analysis of variance
accreditation. One drawback of the CLIA performance specifications is (ANOVA) of the data was performed to estimate the variance compo-
that they don't cover all analytes, so additional sources must be refer- nents [standard deviation (SD) and percent coefficient of variation
enced when creating a set of analytical goals for chemistry and im- (%CV)] for repeatability, between-run and a within-laboratory preci-
munoassay tests. As discussed in the limitations section, there is no sion of the assay. For each assay, the precision study was conducted
international standard for the use of allowable total error goals, and using a minimum of one Alinity system and one reagent lot. The within-
there is a healthy global debate over which goals are most appropriate. laboratory %CV at or near the medical decision level was used in the
For this study, primarily CLIA PT goals were utilized, with CAP PT goals Sigma-metric calculation, which are listed in Table 4.
[7] used as the second choice, Biological analytical goals, third choice, Bias (or in metrological terms, trueness) is much harder to estimate
Spanish “minimum” PT goals the fourth choice, Australian Royal Col- in an ideal manner. The preferred technique for establishing trueness is
lege of Pathologists of Australasia (RCPA) Allowable Limits of Perfor- to compare the laboratory performance against the performance on a
mance (ALP) the fifth choice, and German Rilibak goals the goal of last reference method or reference materials. However, for the vast majority
resort. Table 3 lists the source of goals for the tests. of methods, reference methods either don't exist or are not available at
In this study, CLIA goals were the primary choice, since they are an economically realistic price point. That is, it is beyond the financial
mandated by the US and carry the biggest penalty if they are not met reach of most labs and even most manufacturers to continuously
(failure to meet CLIA goals can result in a suspension of reimbursement compare their methods against reference methods. Thus, the vast ma-
from federal healthcare programs). For non-regulated CLIA analytes, jority of EQA/PT programs compare participant laboratory perfor-
CAP PT goals were consulted next, which are internationally recognized mance against that of all-group consensus means or instrument/man-
as a high standard for quality. If no practical goals could be found in ufacturer-specific group means.
CLIA or CAP, then the Ricos goals [8] were consulted. “Ricos goals” is a Urine specimens were collected for the Microalbumin assay and
popular term for performance specifications derived from a systematic plasma specimens were collected for the Lactic Acid assay. Serum
biologic variation database created by an expert group in Europe. If specimens were collected for all other assays. Specimens were pur-
these Ricos goals were unrealistic, then the Allowable limits of per- chased from a third-party vendor, and for most assays only these spe-
formance (ALP) from the Royal College of Pathologists of Australasia cimens were tested. For assays where the specimens did not adequately
(RCPA) EQA program [9] were used. If those RCPA goals were not cover the measuring interval, additional samples were prepared by
practical, a set of goals determined by a consortium of Spanish EQA spiking specimens with analyte stock solution to cover the high end of
programs, referred to as the “Spanish Minimum goals” [10] were con- the measuring interval or by diluting specimens to cover the low end of
sulted. Finally, if none of the goals from all previous resources were the measuring interval. Bias was estimated, following the principles of
deemed practical or appropriate, the German Rilibak goals [11] were the CLSI EP09-A3 [15] guideline, by testing between 100 and 200
consulted. Finally, for one analyte newer to the field, 25-OH Vitamin D, samples, over at least three days, with testing in duplicate at Abbott on
the Alinity c and ARCHITECT c8000 systems for clinical chemistry and
ICT assays and on the Alinity i and ARCHITECT i2000SR systems for the
Table 3 immunoassays. The ARCHITECT c8000 and i2000SR systems were
Source of allowable total errors for each type of test chosen as the comparator methods because they were the measuring
Assays Performance specifications used
systems upon which the assays were originally developed. Each assay
was tested using at least one Alinity and one ARCHITECT system. Using
3 ICT 3 CLIA Passing-Bablok or weighted Deming regression methods, the first re-
13 IA 9 Ricos plicate of the Alinity concentration was compared against the mean of
2 Spanish minimum (FT4, TT4)
1 Rilibak (βhCG)
the ARCHITECT results; these more sophisticated regression techniques
1 biological variation study (Vitamin D) help to identify the real bias versus the comparator to identify and
37 CC 26 CLIA minimize the effect of imprecision present during the comparison study.
4 CAP (CO2, LDL, Phosphorous, Transferrin) Using these regression models, the %bias was estimated near a medical
1 RCPA (Salicylate)
decision point, which are listed in Table 4.
5 Ricos (CRP, Direct Bilirubin, GGT, Lactic Acid, Microalbumin)
1 Spanish minimum (Digoxin) Performance of all methods was measured at multiple levels across
the measuring interval. Rather than generate Sigma-metrics at multiple

3
S. Westgard et al. Clinical Biochemistry xxx (xxxx) xxx–xxx

Table 4
Analytes assessed on the Alinity ci system.

Alinity c - Clinical Chemistry Assay Sigma Values

Analyte TEa TEa Source Level Bias CV Sigma
Activated-ALT 20% CLIA 113.47 U/L 0.77% 0.93% 20.61
Activated-AST 20% CLIA 43.98 U/L 2.99% 2.18% 7.80
ALT 20% CLIA 30.20 U/L 0.26% 2.83% 6.96
AST 20% CLIA 41.93 U/L 3.38% 1.45% 11.48
Albumin BCG 10% CLIA 3.01 g/dL 0.00% 0.92% 10.82
Albumin BCP 10% CLIA 2.91 g/dL 0.00% 1.11% 9.02
Alkaline Phosphatase 30% CLIA 187.25 U/L 3.44% 1.59% 16.71
Amylase 30% CLIA 114.32 U/L 0.00% 0.78% 38.51
Bilirubin, Direct 44.50% Ricos Desirable 0.40 mg/dL 1.01% 3.90% 11.16
Bilirubin, Total 20% CLIA 2.78 mg/dL 1.76% 2.00% 9.14
Calcium 9.72% CLIA 10.29 mg/dL 1.35% 1.00% 8.38
CO2 25% CAP 3SD approximate 22.34 mEq/L 3.10% 3.91% 5.60
Cholesterol, total 10% CLIA 154.99 mg/dL 1.27% 0.96% 9.06
Creatinine Kinase 30% CLIA 148.21 U/L 1.81% 1.07% 26.26
Creatinine 15% CLIA 2.00 mg/dL 1.94% 1.88% 6.93
Creatinine - enzymatic 15.5% CLIA 1.94 mg/dL 0.86% 0.97% 14.99
CRP vario WR 56.6% Ricos desirable 4.03 mg/L 1.91% 3.09% 17.72
Digoxin 20% Spanish EQA minimum 1.39 ng/mL 8.93% 1.39% 7.98
GGT 22.1% Ricos Desirable 71.42 U/L 2.82% 1.23% 15.72
Glucose 10% CLIA 126.86 mg/dL 0.22% 1.12% 8.76
HDL, Ultra 30% CLIA 51.77 mg/dL 0.93% 2.14% 13.60
Iron 20% CLIA 103.03 μg/dL 3.13% 1.17% 14.38
LDH 20% CLIA 127.98 U/L 1.35% 3.39% 5.50
Direct LDL 20% CAP 78.44 mg/dL 1.24% 1.60% 11.75
Lactic Acid (plasma) 30.4% Ricos Desirable 8.29 mg/dL 1.59% 2.12% 13.62
Lithium 20% CLIA 1.27 mmol/L 0.52% 1.78% 10.94
Magnesium 25% CLIA 2.22 mg/dL 0.01% 1.43% 17.53
Microalbumin (urine) 46.1% Ricos Desirable 32.00 μg/mL 2.74% 4.58% 9.46
Phenytoin 25% CLIA 13.85 μg/mL 0.35% 4.24% 5.82
Phosphorous 10.7% CAP 4.24 mg/dL 0.05% 1.83% 5.83
Protein, Total 10% CLIA 5.09 g/dL 0.56% 1.23% 7.65
Salicylate 10% RCPA 39.75 mg/dL 1.10% 1.00% 8.88
Transferrin 20% CAP 362.43 mg/dL 1.88% 1.29% 14.09
Triglycerides 25% CLIA 150.79 mg/dL 1.07% 0.87% 27.57
Urea Nitrogen 9% CLIA 38.82 mg/dL 1.19% 1.84% 4.24
Uric Acid 17% CLIA 2.49 mg/dL 1.23% 1.01% 15.59
Vancomycin 25% CLIA 34.60 μg/mL 5.20% 1.08% 18.35

Alinity i - Immuno-Assay Sigma Values

Analyte TEa TEa Source Level Bias CV Sigma
β-hCG 30% Rilibak 24.45 mIU/mL 0.08% 5.46% 5.48
CA 125 II 35.4% Ricos Desirable 39.33 U/L 5.54% 4.81% 6.20
CA 15–3 20.8% Ricos Desirable 33.19 U/mL 5.14% 2.91% 5.39
CA 19-9XR 46.03% Ricos Desirable 37.22 U/mL 9.62% 9.10% 4.00
Cortisol 22.8% Ricos Desirable 15.11 μg/dL 1.06% 2.66% 8.17
Estradiol 26.9% Ricos Desirable 187.52 pg/mL 9.77% 2.60% 6.58
Prolactin 29.4% Ricos Desirable 38.75 ng/mL 4.81% 2.79% 8.83
Free PSA 33.6% Ricos Desirable 0.39 ng/mL 7.36% 4.46% 5.88
Free T3 17% Ricos Minimum 6.15 pg/mL 5.43% 3.71% 3.12
Free T4 16% Spanish EQA minimum 1.16 ng/dL 4.26% 1.97% 5.94
Total T4 24% Spanish EQA minimum 4.18 μg/dL 1.96% 4.52% 4.88
TSH 23.7% Ricos Desirable 0.31 μIU/mL 4.48% 1.77% 10.86
25-OH Vitamin D 30% Biological Variation paper [11] 21.14 ng/mL 5.08% 3.48% 7.17

Alinity c – ICT Assay Sigma Values

Analyte TEa TEa Source Level Bias CV Sigma
ICT Chloride 5% CLIA 94.65 mmol/L 0.06% 0.75% 6.57
ICT Potassium 17.97% CLIA 2.78 mmol/L 0.56% 1.46% 11.95
ICT Sodium 3.57% CLIA 112.03 mmol/L 0.10% 0.82% 4.20

levels, the assessment process was streamlined by choosing the most normalized by taking each component of performance (imprecision and
critical medical decision levels for each assay. These levels are listed in bias) and dividing it by the allowable total error. This resulting chart
Table 4, and a summary of these key decision levels can also be found in provides a single visual “performance at a glance” through the plotting
a Sigma Verification program [13]. The practical impact of this focus is of the imprecision (as a percentage of the allowable total error) as the x-
to simplify the Sigma-metric assessment so that there is just one Sigma- coordinate, and the bias (as a percentage of the allowable total error) as
metric for each test. For some assays, a control and a panel were tested the y-coordinate, with each point representing the analytical perfor-
near the medical decision level. In such cases, the sample yielding the mance of one test. The plotted performance falls within different Sigma-
lowest sigma value was included in this summary. metric categories, so the laboratory can quickly grasp which tests are
In addition to calculating the Sigma-metric, the Normalized Sigma- World Class (or Six Sigma) quality, and which tests are unacceptable (or
metric Method Decision chart [16] was used to provide a graphic less than Three Sigma). Colloquially, these Method Decision charts are
summary of instrument performance. The Method Decision chart can be sometimes referred to as “bulls-eye” graphs, since the closer each test is

4
S. Westgard et al. Clinical Biochemistry xxx (xxxx) xxx–xxx

Alinity c Chemistry Assays (37 assays) Alinity c ICT Chloride, Potassium, and
100
2 Sigma Sodium
90 3 Sigma 100
Bias, % of TEa%

4 Sigma 2 Sigma
80 5 Sigma 90 3 Sigma
70 6 Sigma 4 Sigma
80

Bias, % of TEa%
Six Sigma Assays 5 Sigma
60 5 Sigma Assays
70 6 Sigma
4 Sigma and lower Six Sigma Assays
50 60 5 Sigma Assays
4 Sigma Assays
40 50
30 40
20 30
10 20
0 10
0 10 20 30 40 50
0
CV, % of TEa% 0 10 20 30 40 50

Fig. 1. Normalized Method Decision Chart for Alinity c Clinical Chemistry Assays.
CV, % of TEa%
Fig. 3. Normalized Method Decision Chart for Alinity c ICT Chloride, Potassium and
plotted to the graph's origin, the better the Sigma-metric, and the more Sodium Assays.

optimal the routine operation of that assay.

quality. This indicates that “Westgard Rules” will not be necessary to
use for most of the assays, and less onerous QC rules and possibly even
3. Results & discussion
fewer, less frequent controls will need to be run in order to provide
routine monitoring of the methods.
Table 4 lists all the analytes, their allowable total errors, the source
The favorable Sigma-metrics are a result of both low imprecision
of each allowable total error, the level at which data was collected and
and low bias. The Alinity ci system methods are engineered to high
performance was estimated (this is near the chosen critical medical
precision in relation to the performance required by CLIA. The Alinity ci
decision level), the bias, the CV, and the Sigma-metric. Figs. 1–3 display
system methods also compare well to the ARCHITECT system methods.
the Sigma performance on Normalized Method Decision charts.
This indicates that customers switching or upgrading from the previous
Of the 37 chemistry methods analyzed, 32 (86%) are Six Sigma
generation of instruments will see little disruption when they begin
quality, another 4 (11%) are Five Sigma quality, and just one other
using the Alinity ci system.
method is Four Sigma quality. No methods are below Three Sigma.
This study reveals performance across all methods by the Alinity ci
Ninety-seven percent (97%) of the methods analyzed here are Excellent
system is not perfect, but frankly no perfect instrument has yet been
or World Class quality.
created. The most challenging assays for these instruments are the as-
Of the 13 immunoassays studied, 6 (46%) were Six Sigma quality.
says where the goals set by CLIA are known to be extremely demanding,
Four methods (31%) were of Five Sigma quality. Two methods were
not just for ARCHITECT or Alinity, but for the diagnostic industry as a
Four Sigma quality and One method was at Three Sigma quality.
whole7.
Thus, > 76% of the assays were performing at Excellent or World Class
quality.
Of the ICT assays, Chloride and Potassium were Six Sigma quality, 4. Limitations
while Sodium was Four Sigma quality.
The calculated Sigma-metrics and the Normalized Six Sigma Method This study was conducted by the manufacturer at the manufacturer's
Decision chart showed a majority of methods (48 out of 53, > 90%) on facilities in Abbott Park, Illinois, and in Wiesbaden, Germany, by their
the Alinity ci system encompassing chemistry, immunoassay and elec- trained staff, under more controlled conditions than may be typically
trolytes performed at Five Sigma (Excellent) or Six Sigma (World Class) observed in a traditional laboratory. There are two possible drawbacks
to this. First, the staff in the Research and Development department
Alinity i Immunoassays (13 Assays) may be better trained than the technologists of “normal” laboratories.
100 Second, the instruments may be maintained, calibrated and operated
2 Sigma
90 3 Sigma
differently than would be typically seen in a customer laboratory. As
4 Sigma such, the performance observed in this study may be different from
Bias, % of TEa%

80 5 Sigma
then performance that might be seen in a customer site. Thus, perfor-
6 Sigma
70 Six Sigma Assays mance in a “real world” laboratory may not be as good as determined in
60
5 Sigma Assays this study. Only follow-up studies in customer sites will be able to tell if
4 Sigma and lower
this performance can be reproduced.
50
Another limitation is the performance specification used to bench-
40 mark the methods. There is a vigorous debate about which analytical
30 allowable total errors should be used in Sigma-metric calculations.
Currently there is no international consensus on which goals to use,
20
merely a suggested hierarchy created in Milan in 2014 [17]. But even
10 the Milan Hierarchy details only ideal considerations, and acknowl-
0 edges that in real-world laboratories, there is no one source of analy-
0 10 20 30 40 50 tical goals for all tests – that labs must choose appropriate practical
CV, % of TEa% goals for each individual test. The CLIA goals are often criticized as too
forgiving or large, even though the German Rilibak goals are in fact the
Fig. 2. Normalized Method Decision Chart for Alinity i Immunoassays.
largest set of goals in the world. Further, when CLIA goals are used as

5
S. Westgard et al. Clinical Biochemistry xxx (xxxx) xxx–xxx

Six Sigma goals, these performance specifications are still quite chal- References
lenging for many assays and laboratories. For many immunoassays,
there is even less of a consensus about which performance specifica- [1] The reproducibility wars: successful, unsuccessful, uninterpretable, exact, con-
tions to use; the desirable Ricos goals are quite large, and laboratories ceptual, triangulated, contested replication, Clin. Chem. 63 (2017) 5943–5945.
[2] H.C. Stepman, U. Tiikkainen, D. Stöckl, H.W. Vesper, S.H. Edwards, H. Laitinen,
may want to consider adopting the tighter optimum goals. As stated J. Pelanti, L.M. Thienpont, Measurements for 8 common analytes in native sera
earlier, individual laboratories are free to choose which allowable total identify inadequate standardization among 6 routine laboratory assays.
errors to use when calculating Sigma-metrics, particularly if they found Participating laboratories, Clin Chem. 60 (6) (2014 Jun) 855–863, http://dx.doi.
org/10.1373/clinchem.2013.220376 (Epub 2014 Mar 31).
the set of goals used in this study disagreeable. [3] B. Glibert, V. Bourleaux, R. Peeters, T. Reynolds, G. Vranken, Analytical perfor-
Additional limitations include the fact that the Alinity ci system is mance verification of the beckman coulter AU5800 clinical chemistry Analyser
technically related to the ARCHITECT system family and uses the same against recognized quality specifications reveals relevance of method harmoniza-
tion, Clin Lab. 62 (1–2) (2016) 57–72.
reagents, thus minimizing the bias between the two methods. For ex- [4] J.O. Westgard, Six Sigma QC Design and Control, 2nd edition, Westgard QC,
ample, factors such as interferences or matrix effects when comparing Madison WI, 2008 (First edition published in 2001).
two unrelated methods would not be comprehended in bias estimates [5] A method evaluation decision chart (MEDx chart) for judging method performance,
Westgard JO Clin Lab Sci. 8 (5) (1995 Sep-Oct) 277–283 Normalized Method
when comparing Alinity and ARCHITECT. The ARCHITECT is not a
Decision charts are explained at http://www.westgard.com/calculators/normcalc.
“gold standard”. Therefore, the bias estimated in this study is not a bias htm.
against a true reference method or reference material; that bias could be [6] CLIA proficiency testing criteria for acceptable analytical performance, federal
quite different. Further, the CLSI EP05 guideline recommends that register, 57 (40) (February 28, 1992) 7002–7186. Also unofficially posted at
https://www.westgard.com/clia.htm.
manufacturers evaluate imprecision across an extended period of time [7] College of American Pathologists Proficiency Testing. http://www.cap.org/web/
with 3 different reagent lots; this preliminary study was conducted over home/lab/proficiency-testing Participation is required to access the acceptability
20 days with one reagent lot. criteria.
[8] C. Ricos, V. Alvarez, F. Cava, J.V. Garcia-Lario, A. Hernandez, C.V. Jimenez,
As stated earlier, the Sigma-metrics summarized in the paper focus J. Minchinela, C. Perich, M. Simon, Desirable specifications for total error, im-
on one critical medical decision level for each assay. Performance at precision, and bias, derived from intra- and inter-individual biologic variation,
other levels across an assay's measuring interval can often vary. “current databases on biologic variation: pros, cons and progress.”, Scand. J. Clin.
Lab. Invest. 59 (1999) 491–500 Updated in 2014 and posted at https://www.
Also, the use of regression models to estimate bias does not always westgard.com/biodatabase1.htm.
represent the true bias at a single level, especially when the level is near [9] RCPA Allowable Limits of Performance for Biochemistry. http://www.rcpaqap.
one of the extremes of the measuring interval. For a better estimate of com.au/docs/2014/chempath/ALP.pdf accessed5/23/2017. Also unofficially
posted at https://www.westgard.com/rcpa-biochemistry.htm
bias at extremes of the measuring intervals, control materials or re- [10] Carmen Ricos, Francisco Ramon, Angel Salas, Antonio Buno, Rafael Calafell,
ference materials, etc., should be run at those levels. Jorge Morancho, Gabriella Gutierrez-Bassini, Josep M. Jou, Minimum analytical
quality specifications of inter-laboratory comparisons: agreement among Spanish
EQAP organizers, interdisciplinary expert committee for quality specifications in
5. Conclusion
the clinical laboratory, Clin. Chem. Lab. Med. 50 (3) (2012) 455–461. Also un-
officially posted at https://www.westgard.com/minimum-requirements.htm.
This use of Sigma-metrics and Method Decision charts provide a [11] Revision of the “guideline of the german medical association on quality assurance in
useful objective evaluation and comparison of the Alinity ci system. medical laboratory examinations – Rili-BAEK” (unauthorized translation), J. Lab.
Med. 39 (1) (2015) 26–69. Also unofficially posted at https://www.westgard.com/
Through these tools, laboratories can gain a comprehensive under- rilibak.htm.
standing of expected assay performance. The majority of Alinity ci [12] A. Viljoen, D.K. Singh, K. Farrington, P.J. Twomey, Analytical quality goals for 25-
system clinical chemistry assays and immunoassays had Sigma-me- vitamin D based on biological variation, J. Clin. Lab. Anal. 25 (2) (2011) 130–133,
http://dx.doi.org/10.1002/jcla.20446 (The Minimum goal for allowable total error
trics > 5. For the electrolyte assays on the ICT chip, Chloride and from this paper is used for this study's calculations).
Potassium had Sigma-metrics > 5, and Sodium had a Sigma-metric > [13] Westgard Sigma Verification of Performance Program, https://www.westgard.com/
4. The favorable Sigma-metrics are influenced by both well-engineered westgard-sigma-vp.htm.
[14] CLSI, Evaluation of Precision Performance of Quantitative Measurement Methods;
precision as well as strong comparability between the Alinity ci system Approved Guideline-Second Edition. CLSI Document EP05-A2, Clinical and Laboratory
and the ARCHITECT system. Standards Institute, Wayne, PA, 2004http://shop.clsi.org/method-evaluation-
Laboratorians can use these Sigma-metric assessment tools as aids in documents/EP05.html.
[15] Measurement Procedure Comparison and Bias Estimation Using Patient Samples;
choosing high-quality products, further contributing to the delivery of
Approved Guideline-Third Edition. CLSI document EP09-A3, Clinical and Laboratory
excellent quality healthcare for patients. The result is more efficient Standards Institute, Wayne, PA, 2013http://shop.clsi.org/method-evaluation-
instrument operation, more optimized laboratory workflow, and more documents/EP09.html.
[16] A method evaluation decision chart (MEDx chart) for judging method performance,
reliable test results, ultimately helping clinicians better diagnose and
Westgard JO Clin. Lab. Sci. 8 (5) (1995 Sep–Oct) 277–283.
treat their patients. [17] Sverre Sandberg, Callum G. Fraser, Andrea Rita Horvath, Rob Jansen,
Graham Jones, Wytze Oosterhuis, Per Hyltoft Petersen, Heinz Schimmel, Defining
Disclosures analytical performance specifications: Consensus Statement from the 1st Strategic
Conference of the European Federation of Clinical Chemistry and Laboratory
Medicine, Ken Sikaris and Mauro Panteghini Clin. Chem. Lab. Med. 53 (6) (2015)
Sten Westgard is an employee of Westgard QC, which provides 833–835.
consulting services to Abbott Diagnostics. Victoria Petrides, Sharon
Schneider, Marvin Berman, Jorg Herzogenrath, and Anthony
Orzechowski are employees of Abbott Diagnostics.

View publication stats