Statistical algorithms in Review Manager 5

Jonathan J Deeks and Julian PT Higgins

on behalf of the Statistical Methods Group

of The Cochrane Collaboration

August 2010

Data structure
Consider a meta-analysis of k studies. When the studies have a dichotomous (binary) outcome the
results of each study can be presented in a 2×2 table (Table 1) giving the numbers of participant
who do or do not experience the event in each of the two groups (here called experimental (or 1)
and control (or 2)).

Table 1: Binary data

Study i Event No event Total
Experimental ai bi n1i
Control ci di n2i

If the outcome is a continuous measure, the number of participants in each of the two groups, their
mean response and the standard deviation of their responses are required to perform meta-analysis
(Table 2).
Table 2: Continuous data
Group Mean Standard
Study i
size response deviation
Experimental n1i m1i sd1i
Control n2i m2i sd 2i

If the outcome is analysed by comparing observed with expected values (for example using the
Peto method or a log-rank approach for time-to-event data), then ‘O – E’ statistics and their
variances are required to perform the meta-analysis. Group sizes may also be entered by the review
author, but are not involved in the analysis.

Table 3: O minus E and variance

Variance of Group size Group size
Study i O minus E
(O minus E) (experimental) (control)
Zi Vi n1i n 2i

For other outcomes a generic approach can be used, the user directly specifying the values of the
intervention effect estimate and its standard error for each study (the standard error may be
calculable from a confidence interval). ‘Ratio’ measures of effect effects (e.g. odds ratio, risk ratio,
hazard ratio, ratio of means) will normally be expressed on a log-scale, ‘difference’ measures of

1
effect (e.g. risk difference, differences in means) will normally be expressed on their natural scale.
Group sizes can optionally be entered by the review author, but are not involved in the analysis.

Table 4: Generic data

Estimate of Standard error of Group size Group size
Study i
effect estimate (experimental) (control)
θˆ
i
SE θ ˆ { }i n 1i n2i

Formulae for individual studies

Individual study estimates: dichotomous outcomes
Peto odds ratio
For study i denote the cell counts as in Table 1, with n1i = ai + bi , n2i = ci + d i , and let
N i = n1i + n2i . For the Peto method, the individual odds ratios are given by
⎧Z ⎫
ORPeto ,i = exp ⎨ i ⎬ .
⎩ Vi ⎭
The logarithm of the odds ratio has standard error

{
SE ln ( ORPeto ,i ) = } 1
Vi
,

where Z i is the ‘O – E’ statistic:

Z i = ai − E [ ai ] ,
with
n1i ( ai + ci )
E [ ai ] =
Ni
(the expected number of events in the experimental intervention group), and
n n ( a + c )( b + di )
Vi = 1i 2i 2i i i
N i ( N i − 1)
(the hypergeometric variance of ai ).

Odds ratio
For methods other than the Peto method, the odds ratio for each study is given by
ad
ORi = i i ,
bi ci
the standard error of the log odds ratio being

SE {ln ( ORi )} =
1 1 1 1
+ + + .
ai bi ci d i

Risk ratio
The risk ratio for each study is given by
ai / n1i
RRi = ,
ci / n2i

2
the standard error of the log risk ratio being

SE {ln ( RRi )} =
1 1 1 1
+ − − .
ai ci n1i n2i

Risk difference
The risk difference for each study is given by
ai ci
RDi = − ,
n1i n2i
with standard error
ai bi ci di
SE { RDi } = + 3 .
n13i n2i

Empty cells
Where zeros cause problems with computation of effects or standard errors, 0.5 is added to all cells
( ai , bi , ci , d i ) for that study, except when ai = ci = 0 or bi = d i = 0 , when the relative effect
measures ORi and RRi are undefined.

Individual study estimates: continuous outcomes

Denote the number of participants, mean and standard deviation as in Table 2, and let
N i = n1i + n2i
and

si =
( n1i − 1) sd12i + ( n2i − 1) sd 2i2
Ni − 2
be the pooled standard deviation across the two groups.

Difference in means (mean difference)

The difference in means (referred to as mean difference) is given by
MDi = m1i − m2i ,
with standard error
sd12i sd 22i
SE {MDi } = + .
n1i n2i

Standardized difference in means (standardized mean difference)

There are several popular formulations of the standardized mean difference. The one implemented
in RevMan is Hedges’ adjusted g, which is very similar to Cohen's d, but includes an adjustment
for small sample bias
m − m2i ⎛ 3 ⎞
SMDi = 1i ⎜1 − ⎟,
si ⎝ 4 Ni − 9 ⎠
with standard error
Ni SMDi2
SE {SMDi } = + .
n1i n2i 2 ( N i − 3.94 )

3
Individual study estimates: O – E and variance
For study i the effect estimate is given by
Z
θˆ i = i ,
Vi
with standard error

{ }1
SE θˆ i =
Vi
.

The effect estimate is either of a log odds ratio or a log hazard ratio, depending on how the
observed and expected values were derived.

Individual study estimates: Generic method

As the user directly enters the intervention effect estimates and their standard errors no further
processing is needed. All types of intervention effects are eligible for this method, but it might be
most useful when intervention effects have been calculated in a way which makes special
consideration of design (e.g. cluster randomized and cross-over trials), are adjusted for other effects
(adjusted effects from non-randomized studies) or are not covered by existing methods (e.g. ratios
of means, relative event rates).

Meta-analysis methods
All summations are over i, from 1 to the number of studies, unless otherwise specified.

Mantel-Haenszel methods for combining results across studies

Odds ratio

The Mantel-Haenszel summary log odds ratio is given by

⎛ ∑ wMH ,i ORi ⎞
ln ( ORMH ) = ln ⎜ ⎟⎟ , (1)
⎜ ∑w
⎝ MH ,i ⎠
and the Mantel-Haenszel summary odds ratio by

ORMH =
∑ wMH ,iORi ,
∑ wMH ,i
where each study’s odds ratio is given weight
bi ci
wMH ,i = .
Ni
The summary log odds ratio has standard error given by
1⎛ E F +G H ⎞
SE {ln ( ORMH )} = ⎜ + + 2⎟, (2)
2 ⎝ R2 RS S ⎠
where
ai di bc
R=∑ ; S =∑ i i ;
Ni Ni

4
 E=∑
( ai + di ) ai di ; F =∑
( ai + di ) bi ci ;
2
N i N i2

G=∑
( bi + ci ) ai di ; H =∑
( bi + ci ) bi ci .
2
N i N i2

Risk ratio
The Mantel-Haenszel summary log risk ratio is given by
⎛ ∑ wMH ,i RRi ⎞
ln ( RRMH ) = ln ⎜ ⎟⎟ , (3)
⎜ ∑w
⎝ MH ,i ⎠
and the Mantel-Haenszel summary risk ratio by

RRMH =
∑ wMH ,i RRi ,
∑ wMH ,i
where each study’s risk ratio is given weight
ci ( ai + bi )
wMH ,i = .
Ni
The summary log risk ratio has standard error given by

SE {ln ( RRMH )} =
P
, (4)
RS
where
n n ( a + c ) − ai ci N i an cn
P = ∑ 1i 2i i 2i ; R = ∑ i 2i ; S = ∑ i 1i .
Ni Ni Ni

Risk difference
The Mantel-Haenszel summary risk difference is given by

RDMH =
∑ wMH ,i RDi , (5)
∑ wMH ,i
where each study’s risk difference is given weight
n1i n2i
wMH ,i = .
Ni
The summary risk difference has standard error given by
J
SE { RDMH } = , (6)
K2
where
a b n3 + c d n3 n n
J = ∑ i i 2i i 2 i 1i ; K = ∑ 1i 2i .
n1i n2i N i Ni

Test for heterogeneity

The heterogeneity test statistic is given by
(
QMH = ∑ wi θˆ i − θˆ MH ),
2

5
where θ̂ represents the log odds ratio, log risk ratio or risk difference and the wi are the weights

{ }
2
calculated as 1 SE θˆ i rather than the weights used for the Mantel-Haenszel meta-analyses.
Under the null hypothesis that there are no differences in intervention effect among studies this
follows a chi-squared distribution with k − 1 degrees of freedom (where k is the number of studies
contributing to the meta-analysis).

The statistic I2 is calculated as

⎧ Q − ( k − 1) ⎫
I 2 = max ⎨100% × MH , 0⎬
⎩ QMH ⎭
This measures the extent of inconsistency among the studies’ results, and is interpreted as
approximately the proportion of total variation in study estimates that is due to heterogeneity rather
than sampling error.

Inverse-variance methods for combining results across studies

Inverse-variance methods are used to pool log odds ratios, log risk ratios and risk differences as one
of the analysis options for binary data, to pool all mean differences and standardized mean
differences for continuous data, and also for combining intervention effect estimates in the generic
method. In the general formula the intervention effect estimate is denoted by θˆ i , which is the
study’s log odds ratio, log risk ratio, risk difference, mean difference or standardized mean
difference, or the estimate of intervention effect in the generic method. The individual effect sizes
are weighted according to the reciprocal of their variance (calculated as the square of the standard
error given in the individual study section above) giving
1
wi = .
( { })
2
SE θiˆ

These are combined to give a summary estimate

θˆ IV =
∑ w θˆ i i
. (7)
∑w i

with
{ }
SE θˆ IV =
1
. (8)
∑w i

The heterogeneity statistic is given by a similar formula as for the Mantel-Haenszel method:
∑ ( )
2
Q = w θˆ − θˆ
IV i i . IV

Under the null hypothesis that there are no differences in intervention effect among studies this
follows a chi-squared distribution with k − 1 degrees of freedom (where k is the number of studies
contributing to the meta-analysis). I2 is calculated as
⎧ Q − ( k − 1) ⎫
I 2 = max ⎨100% × IV , 0⎬ .
⎩ QIV ⎭

Peto's method for combining results across studies

The Peto summary log odds ratio is given by

6
 ln ( ORPeto ) =
∑V ln ( OR i Peto ,i ). (9)
∑V i

and the summary odds ratio by

⎧⎪ ∑ Vi ln ( ORPeto ,i ) ⎫⎪
ORPeto = exp ⎨ ⎬,
⎪⎩ ∑Vi ⎪⎭
where the odds ratio ORPeto ,i is calculated using the approximate method described in the individual
study section, and Vi are the hypergeometric variances.

The log odds ratio has standard error

SE {ln ( ORPeto )} =
1
. (10)
∑Vi
The heterogeneity statistic is given by
QPeto = ∑ Vi {( ln OR Peto ,i ) − ( ln OR ) } .
2
Peto
2

Under the null hypothesis that there are no differences in intervention effect among studies this
follows a chi-squared distribution with k − 1 degrees of freedom (where k is the number of studies
contributing to the meta-analysis). I2 is calculated as
⎧ Q − ( k − 1) ⎫
I 2 = max ⎨100% × Peto , 0⎬ .
⎩ QPeto ⎭

O – E and variance method for combining studies

This is an implementation of the Peto method, which allows its application to time-to-event data as
well as binary data. The summary effect estimate is given by

θˆ =
∑Vi θˆ i , (11)
∑Vi
where the estimate, θˆ , from study i is calculated from Z and V as for individual studies. The
i i i

summary effect is either a log odds ratio or a log hazard ratio (the user should specify which). The
effect estimate (on a non-log scale) is given by
⎧⎪ ∑ Vi θˆ i ⎫⎪
effect estimate = exp ⎨ ⎬,
⎩⎪ ∑ Vi ⎭⎪
and is either an odds ratio or a hazard ratio.

The effect estimate (on the log scale) has standard error
SE θˆ = {} 1
. (12)
∑ iV

The heterogeneity statistic is given by

(
QPeto = ∑ Vi θˆ i 2 − θˆ 2 . )
Under the null hypothesis that there are no differences in intervention effect among studies this
follows a chi-squared distribution with k − 1 degrees of freedom (where k is the number of studies
contributing to the meta-analysis). I2 is calculated as

7
 ⎧ Q − ( k − 1) ⎫
I 2 = max ⎨100% × Peto , 0⎬ .
⎩ QPeto ⎭

DerSimonian and Laird random-effects models

Under the random-effects model, the assumption of a common intervention effect is relaxed, and
the effect sizes are assumed to have a distribution
θi ∼ N ( θ, τ2 ) .
The estimate of τ2 is given by
⎧⎪ Q − ( k − 1) ⎫⎪
τˆ 2 = max ⎨ , 0 ⎬,
w (
⎩⎪ ∑ i ∑ i ∑ i ⎭⎪
− w 2
) w
where the wi are the inverse-variance weights, calculated as
1
wi = ,
{ }
2
SE θ ˆ
i

for log odds ratio, log risk ratio, risk difference, mean difference, standardized mean difference, or
for the intervention effect in the generic method, as appropriate.

For continuous data and for the generic method, Q is QIV . For binary data, either QIV or QMH may
be taken. Both are implemented in RevMan 5 (and this is the only difference between random-
effects methods under ‘Mantel-Haenszel’ and ‘inverse-variance’ options). Again, for odds ratios,
risk ratios and other ratio effects, the effect size is taken on the natural logarithmic scale.

Each study’s effect size is given weight

1
wi′ = .
{ }
2
SE θˆ i + τˆ 2
The summary effect size is given by

θˆ DL =
∑ wi′θˆ i , (13)
∑ wi′
and
{ }
SE θˆ DL =
1
. (14)
∑ wi′
Note that in the case where the heterogeneity statistic Q is less than or equal to its degrees of
freedom (k − 1) , the estimate of the between study variation, τˆ 2 , is zero, and the weights coincide
with those given by the inverse-variance method.

Confidence intervals
The 100(1 − α )% confidence interval for θ̂ is given by
{}
θˆ − SE θˆ Φ (1 − α 2 ) to θˆ + SE θˆ Φ (1 − α 2 ) , {}
where θ̂ is the log odds ratio, log risk ratio, risk difference, mean difference, standardized mean
difference or generic intervention effect estimate, and Φ is the standard normal deviate. For log
odds ratios, log risk ratios and generic intervention effects entered on the log scale (and identified

8
as such by the review author), the point estimate and confidence interval limits are exponentiated
for presentation.

Test statistics
Test for presence of an overall intervention effect
In all cases, the test statistic is given by
θˆ
Z= ,
SE θˆ()
where the odds ratio, risk ratio and other ratio measures are again considered on the log scale.
Under the null hypothesis that there is no overall effect of intervention effect this follows a standard
normal distribution.

Test for comparison of subgroups

The test is valid for all methods. It is based on the notion of performing a test for heterogeneity
across subgroups rather than across studies. Let θ̂ j be the summary effect size for subgroup j, with

{ }
standard error SE θ̂ j . The summary effect size may be based on either a fixed-effect or a
random-effects meta-analysis. For fixed-effect meta-analyses, these numbers correspond to above
equations (1) and (2); (3) and (4); (5) and (6); (7) and (8); (9) and (10); or (11) and (12), each
applied within each subgroup. For random-effects meta-analyses, these numbers correspond to
equations (13) and (14), each applied within each subgroup. Note that for ratio measures, all
computations here are performed on the log scale.

First we compute a weight for each subgroup:

1
wj = ,
{ }
2
SE θˆ j
then we perform a (fixed-effect) meta-analysis of the summary effect sizes across subgroups:

θˆ tot =
∑ w j θˆ j .
∑ wj
The test statistic for differences across subgroups is given by
∑ ( )
2
Q = w θˆ − θˆ
int j .j tot

Under the null hypothesis that there are no differences in intervention effect across subgroups this
follows a chi-squared distribution with S − 1 degrees of freedom (where S is the number of
subgroups with summary effect sizes).

I2 for differences across subgroups is calculated as

⎧ Q − ( S − 1) ⎫
I 2 = max ⎨100% × int , 0⎬ .
⎩ Qint ⎭
This measures the extent of inconsistency across the subgroups’ results, and is interpreted as
approximately the proportion of total variation in subgroup estimates that is due to genuine
variation across subgroups rather than sampling error.

9
Note. An alternative formulation for fixed-effect meta-analyses (inverse variance and Peto methods
only) is as follows. The Q statistic defined by either QIV or QPeto is calculated separately for each of
the S subgroups and for the totality of studies, yielding statistics Q1 , …, QS and Qtot . The test
statistic is given by
S
Qint = Qtot − ∑ Q j .
j =1

This is identical to the test statistic given above, in these specific situations.

10
Bibliography
Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-analysis. John Wiley
& Sons, 2009.

Breslow NE, Day NE. Combination of results from a series of 2x2 tables; control of confounding.
In: Statistical Methods in Cancer Research, Volume 1: The analysis of case-control data. IARC
Scientific Publications No.32. Lyon: International Agency for Health Research on Cancer, 1980.

Deeks JJ, Altman DG, Bradburn MJ. Statistical methods for examining heterogeneity and
combining results from several studies in a meta-analysis. In: Egger M, Davey Smith G, Altman
DG. Systematic Reviewes and Healthcare: meta-analysis in context. BMJ Publications (in press).

DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials 1986; 7: 177-
188.

Greenland S, Robins J. Estimation of a common effect parameter from sparse follow-up data.
Biometrics 1985;41: 55-68.

Greenland S, Salvan A. Bias in the one-step method for pooling study results. Statistics in Medicine
1990; 9:247-252.

Hedges LV, Olkin I. Statistical Methods for Meta-analysis. San Diego: Academic Press 1985.
Chapter 5.

Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analysis.
BMJ 2003; 327: 557-560.

Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of
disease. Journal of the National Cancer Institute 1959;22: 719-748.

Robins J, Greenland S, Breslow NE. A general estimator for the variance of the Mantel-Haenszel
odds ratio. American Journal of Epidemiolgy 1986; 124:719-723.

Rosenthal R. Parametric measures of effect size. In: Cooper H, Hedges LV (eds.). The Handbook
of Research Synthesis. New York: Russell Sage Foundation, 1994.

Sinclair JC, Bracken MB. Effective Care of the Newborn infant.Oxford: Oxford University Press
1992.Chapter 2.

Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta blockade during and after myocardial
infarction: an overview of the randomized trials. Progress in Cardiovascular Diseases 1985;27:335-
371.

11