Statistics in practice

How to perform a meta-analysis with R: a

practical tutorial
Sara Balduzzi ‍ ‍, Gerta Rücker ‍ ‍, Guido Schwarzer ‍ ‍

►► Additional material is Abstract and interpretation of meta-analysis when outcome

published online only. To view Objective  Meta-analysis is of fundamental importance data are missing and when small-study effects occur.
please visit the journal online
(http://​dx.​doi.​org/​10.​1136/​ to obtain an unbiased assessment of the available In this publication, we replicate these analyses in
ebmental-​2019-​300117). evidence. In general, the use of meta-analysis has been R using the packages meta7 and metasens.8
increasing over the last three decades with mental health In the following, we present R commands to:
Institute of Medical Biometry
and Statistics, Faculty of as a major research topic. It is then essential to well ►► Install R packages for meta-analysis.
Medicine and Medical Center – understand its methodology and interpret its results. In ►► Conduct a meta-analysis when the outcome of
University of Freiburg, Freiburg this publication, we describe how to perform a meta- interest is binary.
im Breisgau, Germany analysis with the freely available statistical software ►► Assess the impact of missing outcome data.
environment R, using a working example taken from the ►► Assess and account for small-study effects.
Correspondence to
Dr Sara Balduzzi, Institute of field of mental health.
Medical Biometry and Statistics, Methods  R package meta is used to conduct standard Methods
Faculty of Medicine and Medical meta-analysis. Sensitivity analyses for missing binary Before conducting a meta-analysis, the R packages
Center - University of Freiburg, outcome data and potential selection bias are conducted meta and metasens need to be installed,9 which
Freiburg im Breisgau 79085,
Germany; ​balduzzi@​imbi.​uni-​
with R package metasens. All essential R commands are include all functions to perform the analyses and
freiburg.​de provided and clearly described to conduct and report to create the figures presented in this publication.
analyses. install.packages(c(“meta”, “metasens”))
Received 6 September 2019 Results  The working example considers a binary Using this first R command, we would like to
Accepted 16 September 2019
Published Online First
outcome: we show how to conduct a fixed effect and mention three general properties of R commands.
28 September 2019 random effects meta-analysis and subgroup analysis, First, we use brackets in order to execute an R func-
produce a forest and funnel plot and to test and adjust tion, here install.packages(); the command install.
for funnel plot asymmetry. All these steps work similar packages (without brackets) would show the defini-
for other outcome types. tion of the function. Second, the argument of an R
Conclusions  R represents a powerful and flexible tool function can be another R function; here we use the
to conduct meta-analyses. This publication gives a brief function c() to combine the names of R packages.
glimpse into the topic and provides directions to more Third, function arguments are separated by commas
advanced meta-analysis methods available in R. that is visible in the call of function c(). RStudio
users can also install packages via the menu: Tools
->Install Packages…
The library function can be used to make the R
Introduction packages available in the current session:
Evidence synthesis is the basis for decision making library(meta)
in a large range of applications including the fields library(metasens)
of medicine, psychology and economics.1 System- A brief and very helpful overview of R package
atic reviews and meta-analysis as statistical method meta is provided by the R command help(meta).
to combine results are cornerstones to yield an Among other things, the overview mentions that we
unbiased assessment of the available evidence.2 In can use the s​ ettings.​meta function to define default
general, the use of meta-analysis has been increasing settings for the current R session. In this publica-
over the last three decades with mental health as tion, we want to print all results with two signifi-
a major research topic, for example, in Cochrane cant digits: s​ ettings.​meta(​digits=​2) .
reviews.3 The dataset used in this paper comes from a
The freely available statistical software envi- Cochrane review evaluating the effect of halo-
ronment R4 (https://www.​r-​project.​org) and the peridol in the treatment of symptoms of schizo-
commercial software Stata5 provide the largest phrenia.10 It has already been used to assess the
collection of statistical methods for meta-analysis. impact of missing data on clinical outcomes11 and
RStudio (https://www.​ rstudio.​
com/) is a popular as working example in Chaimani et al.6 It consists
© Author(s) (or their
and highly recommended integrated development of 17 trials comparing haloperidol and placebo, the
employer(s)) 2019. No
commercial re-use. See rights environment for R providing menu-driven tools for outcome of interest is clinical improvement and the
and permissions. Published plotting, a history of previous R commands, data chosen measure of association is the risk ratio (RR),
by BMJ. management and the installation and update of R with an RR larger than 1 meaning that haloperidol
To cite: Balduzzi S, Rücker G, packages. is better than placebo.
Schwarzer G. Evid Based An introduction to meta-analysis with Stata has The information available for each trial, along
Ment Health been published in Evidence-Based Mental Health6 with the name of the first author (author) and the
2019;22:153–160. with specific focus on the challenges in the conduct year of publication (year), is:
Balduzzi S, et al. Evid Based Ment Health 2019;22:153–160. doi:10.1136/ebmental-2019-300117    153
 Statistics in practice

Figure 1  Printout of metabin function.

►► The number of participants who responded in the haloper- users can use the menu to import a dataset: File ->Import
idol arm (resp.h) and in the placebo arm (resp.p). Dataset ->From Text (base)… In RStudio, by default, the
►► The number of participants who failed to respond in the name of the R dataset is identical to the filename without the
haloperidol arm (fail.h) and in the placebo arm (fail.p). extension. Accordingly, the name of the dataset must be set to
►► The number of participants who dropped out, for which the ‘joy’ in the import wizard in order to run the subsequent R
outcome is missing, in either arm (drop.h, drop.p). commands.
The dataset contained in ref 10 is available in online supple- A new R object joy is created that can be viewed by simply
mentary file 1 and can be imported with the following R typing: joy .
command, given it is stored in the working directory of the R Alternatively, we can see the data in spreadsheet format:
session: ​joy=​read.​csv(“​Joy2006.​txt”) . View(joy) .
The function ​read.​csv has several arguments; however, we Or get an overview of the structure of the data (eg, class,
only specify the name of the text file with the dataset. RStudio dimension, listing and format of variables): str(joy) .
154 Balduzzi S, et al. Evid Based Ment Health 2019;22:153–160. doi:10.1136/ebmental-2019-300117
 Statistics in practice

Figure 2  Forest plot showing the results of fixed effect and random effects meta-analysis (available case analysis).

Some studies have participants with missing information due default method to calculate the fixed effect estimate is Mantel-
to drop outs (variables drop.h and drop.p). Later, we want to Haenszel.13 The inverse variance weighting could be used for
conduct a subgroup analysis of studies with and without missing pooling by specifying method=“Inverse”, which was used in
data and therefore add a new variable with this information to Chaimani et al.6 The method to estimate the between-study
the dataset: variance in the random effects model can be specified with the
joy$miss=ifelse((joy$drop.h+joy$drop.p)==0, c(“Without argument m ​ ethod.​tau; in this example we chose the method by
missing data”), c(“With missing data”)) . Paule and Mandel,14 which is a recommended method for binary
In general, it is recommended to add all variables that will be outcomes.15
used in analyses to the dataset before conducting a meta-analysis. The result can be viewed by typing m.publ or ​print(​m.​publ).
Note, we access single variables in a dataset by using the dollar The second command could be extended to fine tune the printout
sign. by using additional arguments (see h ​ elp(​print.​meta)).
A comprehensive description of R features for meta-analysis We use the following command to generate the forest plot:
can be found in Schwarzer et al.9 ​forest(​m.​publ, sortvar=year, prediction=TRUE, ​label.​left =
“Favours placebo”, ​label.​right = “Favours haloperidol”) .
Fixed effect and random effects meta-analysis Only the first argument providing the meta-analysis object
The outcome of interest, that is, clinical improvement, is binary m.publ is mandatory. The argument sortvar orders the studies
and the brief overview provided by help(meta) reveals that the according to the values of the specified variable, in this case by
appropriate R function is metabin. increasing year of publication. The argument prediction=TRUE
m.publ=metabin(resp.h, resp.h + fail.h, resp.p, resp.p + fail.p, indicates that a prediction interval16 should be shown in the
data=joy, studlab=paste0(author, “ (”, year, “)”), ​method.​tau = forest plot. The arguments ​label.​left and ​label.​right specify labels
“PM”) printed at the bottom of the forest plot to simplify its interpreta-
This command creates a new R object, named m.publ, which tion. A vast number of additional arguments exists to modify the
is a list containing several components describing the meta-anal- forest plot (see ​help(​forest.​meta)).
ysis that can be accessed with minimum input by the user. By
default, the RR is used in metabin as the effect measure, and it
is not necessary to specify this explicitly (which could be done Assessing the impact of missing outcome data
setting the argument sm=“RR”). In order to understand if the results of studies with missing data
The first four arguments of metabin are mandatory and define differed from studies without missing data, a subgroup analysis
the variables containing the number of patients who experienced can be done through the command:
a clinical improvement and the number of randomised patients ​m.​publ.​sub = ​update(​m.​publ, byvar = miss, ​print.​byvar =
(for which we have the information), for the experimental arm FALSE) .
and the control arm, respectively. The function performs both The meta-analysis object m.publ is updated and saved in a new
fixed effect and random effects meta-analysis,12 using the dataset object ​m.​publ.​sub. The argument byvar indicates the grouping
joy (argument data). The argument studlab defines study labels variable miss, added above to the dataset. The argument ​print.​
that are printed in the output and shown in the forest plot, here byvar is used to suppress the printing of the variable name in the
as the name of the first author and the publication year. The subgroup label.
Balduzzi S, et al. Evid Based Ment Health 2019;22:153–160. doi:10.1136/ebmental-2019-300117 155
 Statistics in practice

Figure 3  Forest plot showing the subgroup analysis by presence of missing data (available case analysis).

To explore the impact of missing outcome data on the results, variance meta-analysis instead of the SEs from the available case
several imputation methods have been proposed,11 17 which are meta-analysis.
available in the function metamiss of the metasens package. All other methods are based on the Informative Missingness
In metamiss, the number of observations with missing Odds Ratios (IMOR), defined as the odds of an event in the
outcomes must be provided for the two treatment groups (in missing group over the odds of an event in the observed group11
our example the variables drop.h and drop.p). The imputation (eg, an IMOR of 2 means that the odds for an event is assumed
method is specified with argument ​method.​miss, and may be one to be twice as likely for missing observations).
of the following: For m​ ethod.​miss=“IMOR”, the IMORs in the experimental
“GH” method by Gamble and Hollis.18 (argument IMOR.e) and control group (argument IMOR.c) must
“IMOR” based on group-specific Informative Missingness be specified; if both values are assumed to be equal, only argu-
Odds Ratios (IMORs). ment IMOR.e has to be provided. For all other methods, the
“0” imputed as no events, (i.e., 0) – default method (Imputed input for arguments IMOR.e and IMOR.c is ignored as these
Case Analysis (ICA)-0). values are determined by the respective imputation method (see
“1” imputed as events (ie, 1) (ICA-1). table 2 in ref 11). It must be specified whether the outcome is
“pc” based on observed risk in control group (ICA-pc). beneficial (argument ​small.​values=“good”) or harmful (​small.​
“pe” based on observed risk in experimental group (ICA-pe). values=“bad”) when best or worst case scenarios are chosen,
“p” based on group-specific risks (ICA-p). that is, if argument m
​ ethod.​miss is equal to “b” or “w”.
“b” best case scenario for experimental group (ICA-b).
“w” worst case scenario for experimental group (ICA-w).
For example, the following command will impute missing data Assessing and accounting for small-study effects
as events: mmiss.1 = metamiss (​m.​publ, drop.h, drop.p, ​method.​ Sometimes small studies show different, often larger, treatment
miss = “1”) . effects compared with the large ones. The association between
The method by Gamble and Hollis18 is based on uncertainty size and effect in meta-analysis is referred to as small-study
intervals for individual studies, assuming best and worst case effects.19
scenarios for the missing data. Inflated SEs are calculated from The first step to assess the presence of small-study effects is
the uncertainty intervals and then considered in a generic inverse usually to have a look at the funnel plot, where effect estimates
156 Balduzzi S, et al. Evid Based Ment Health 2019;22:153–160. doi:10.1136/ebmental-2019-300117
 Statistics in practice

Figure 4  Summary risk ratios according to different assumptions about mechanism of missingness.

are plotted against a measure of precision, usually the SE of the Several tests, often referred to as tests for small-study effects
effect estimate. This can be done through the function ​funnel.​ or tests for funnel plot asymmetry, have been proposed to assess
meta with the meta-analysis object as input: f​ unnel(​m.​publ) . whether the association between estimated effects and study
An asymmetric funnel plot indicates that small-study effects size is larger than might be expected by chance.20 22 These tests
are present. Publication bias, though the most popular, is only typically have low power, meaning that even when they do not
one of several possible reasons for asymmetry.20 A contour-en-
support the presence of asymmetry, bias cannot be excluded.
hanced funnel plot may help to distinguish if asymmetry is due
Accordingly, they should be used only if the number of included
to publication bias, by adding lines representing regions where a
test of treatment effect is significant.21 In order to generate such studies is 10 or larger.20 The Harbord score test, where the test
a contour-enhanced funnel plot, the argument c​ontour.​levels statistic is based on a weighted linear regression of the efficient
must be specified: f​unnel(​m.​publ, ​contour.​levels = c(0.9, 0.95, score on its SE,23 is applied in this example: ​metabias(​m.​publ,​
0.99), col.contour = c (“darkgray”, “gray”, “lightgray”)) . method.​bias = “score”) .
Balduzzi S, et al. Evid Based Ment Health 2019;22:153–160. doi:10.1136/ebmental-2019-300117 157
 Statistics in practice

Figure 5  Funnel plot and various methods to evaluate funnel plot asymmetry.

As usual, the first argument is the object created through The trim-and-fill method, 1) removes/“trims” studies from
metabin, while the argument m ​ ethod.​bias specifies the test to the funnel plot until it becomes symmetric 2) adds/“fills” mirror
be used: “score” is for the Harbord test. The metabias function images of removed studies (ie, unpublished studies) to the orig-
provides several other methods for testing funnel plot asym- inal funnel plot and 3) calculates the adjusted effect estimate
metry (see help(metabias)). based on original and added studies. The function to apply is
Once the presence of asymmetry in the funnel plot has been trimfill: t​ f.​publ = t​ rimfill(​m.​publ) .
detected, it is also possible to conduct sensitivity analyses to With this command, a new object is created, named t​f.​publ.
adjust the effect estimate for this bias. Different methods exist22; The results can be shown by typing t​f.​publ or s​ ummary(​tf.​publ)
in this paper, we report the more established trim-and-fill to get a brief summary, and a corresponding funnel plot can be
method24 and the method of adjusting by regression.25 Another created: ​funnel(​tf.​publ) .
more advanced method is the Copas selection model,26 which A specific implementation of the “adjusting by regression”
we do not treat here. method, called “limit meta-analysis”, is described in detail in
158 Balduzzi S, et al. Evid Based Ment Health 2019;22:153–160. doi:10.1136/ebmental-2019-300117
 Statistics in practice
ref 25. The underlying model, motivated by Egger’s test,20 is an poor responses were not seen, and the treatment effects in these
extended random effects model with an additional parameter studies were underestimated. For example, two studies (Beasley
alpha representing possible small study effects (funnel plot asym- and Selman) with a larger number of missing observations in the
metry) by allowing the treatment effect to depend on the SE. placebo group have rather small treatment estimates.
More explicitly, alpha is the expected shift in the standardised In order to better understand how missing data might have
treatment effect if precision is very small. The model provides an influenced the results, more appropriate methods are provided
adjusted treatment effect estimate that is interpreted as the limit by metamiss making different assumptions on the missing-
treatment effect for a study with infinite precision. Graphically, ness mechanism. The results for the random effects model are
this means adding to the funnel plot a curve from the bottom presented in figure 4. Overall, results are rather similar with RRs
to a point at the top which marks the adjusted treatment effect ranging from 1.90 to 2.64 for the extreme worst and best case
estimate. The corresponding function is limitmeta: ​l1.​publ = ​ scenarios. All sensitivity analyses still suggest that haloperidol
limitmeta(​m.​publ) . is better than placebo in improving the symptoms of schizo-
A new object, named ​l1.​publ, is created. As usual, the results phrenia, indicating that missing outcome data are not a serious
can be viewed by typing its name. The funnel plot can be created problem in this dataset.
using the command: ​funnel(​l1.​publ) .
Assessing and accounting for small-study effects
Results The funnel plot is shown in figure 5 (panel A). The fixed effect
R commands for meta-analysis and sensitivity analyses have model is represented by a dashed line on which the funnel is
been described in the previous section. In order to produce the centred, while the random effects model estimate is indicated
figures in this publication, we slightly modified some of the R by a dotted line. As in our example, both estimates are similar;
commands introduced before and had to run some additional they cannot be well distinguished. The funnel plot clearly looks
computations. All R commands used to perform the analyses in asymmetric; however, based on the contour-enhanced funnel
this section—including R code for the figures—can be found in plot (figure 5, panel B), publication bias seems not to be the
the online supplementary file 2. dominant factor for the asymmetry as most small studies with
large SEs lie in the white area corresponding to non-signifi-
Fixed effect and random effects meta-analysis cant treatment estimates. The Harbord test is highly significant
The metabin function printout is displayed in figure 1 containing (p<0.001), supporting the presence of small-study effects.
all basic meta-analysis information (individual study results, The trim-and-fill method added nine studies to the meta-anal-
fixed effect and random effects results, heterogeneity informa- ysis (figure 5, panel C), leading to an adjusted random effects
tion and details on meta-analytical method). The forest plot estimate RR=1.40 (0.83–2.38) suggesting a non-significant
showing the results of both fixed effect and random effects benefit of haloperidol compared with placebo.
meta-analysis considering the available cases is given in figure 2. The result of the limit meta-analysis is shown in figure 5,
For both models, the diamonds presenting the estimated panel D. The grey curve indicates some funnel plot asymmetry:
RRs and confidence limits do not cross the line of no effect, it begins (at the bottom) with a considerable deviation from the
suggesting that haloperidol is significantly more effective than random effects estimate caused by the small studies and balances
placebo. However, this result should be taken with caution: the this by approaching a point (at the top) left to the random effects
prediction interval, incorporating the between-study heteroge- estimate, representing the adjusted estimate RR=1.29 (0.93–
neity, crosses the line of no effect, revealing that placebo might 1.79), again covering the line of no effect.
be superior to haloperidol in a future study.
Some CIs in the forest plot do not overlap, and the test of Discussion
heterogeneity (p=0.004) also suggests the presence of hetero- Meta-analysis is a fundamental tool for evidence-based medi-
geneous results. The heterogeneity statistic I2 is 54%, indicative cine, making it essential to well understand its methodology
of moderate heterogeneity; its CI ranges from 21% to 74%, and interpret its results. Nowadays several software options are
denoting potentially unimportant to substantial heterogeneity available to perform a meta-analysis. In this paper, we aimed
(ref 13, section 9.5.2).13 As expected, the CI for the summary to give a brief introduction on how to conduct a meta-analysis
estimate from the random effects model is wider compared with in the freely available software R using the meta and metasens
the one from the fixed effect model, but the two results differ packages, which provide a user-friendly implementation of
only slightly in terms of magnitude. meta-analysis methods. The meta package has been developed
by the last author to communicate meta-analysis results to clin-
Assessing the impact of missing outcome data ical colleagues in the context of Cochrane reviews.
The forest plot for the subgroup analysis by presence of missing For illustration, we used an example with a binary outcome
data in the studies is shown in figure 3. Though both subgroups and showed how to conduct a meta-analysis and subgroup anal-
give significant results, studies without missing data report a ysis, produce a forest and funnel plot and to test and adjust for
larger haloperidol effect compared with the studies with missing funnel plot asymmetry. All these steps work similar for other
data. Not all CIs for the subgroup estimates include the respec- outcome types, for example, R function metacont can be used for
tive overall effect (in a negligible way for the random effects continuous outcomes. Additionally, we conducted a sensitivity
model), and the test for subgroup differences under the random analysis for missing binary outcomes using R function metamiss.
effects model displayed in the forest plot support the visual detec- In our example, all sensitivity analyses for missing data resulted
tion, suggesting that missing data might have some impact on the in similar results supporting the benefit of haloperidol over
results (p=0.03). A reason may be that patients randomised to placebo despite very different assumptions on the missingness
placebo are more likely to withdraw from the study because of mechanism. However, the evaluation of funnel plot asymmetry
lack of efficacy compared with the patients randomised to an revealed a small-study effect that—according to the contour-en-
active treatment. If these patients were lost to follow-up, their hanced funnel plot—cannot be attributed to publication bias.
Balduzzi S, et al. Evid Based Ment Health 2019;22:153–160. doi:10.1136/ebmental-2019-300117 159
 Statistics in practice
While all sensitivity analyses adjusting for selection bias resulted 6 Chaimani A, Mavridis D, Salanti G. A hands-on practical tutorial on performing meta-
in non-significant treatment estimates, we would not like to analysis with Stata. Evid Based Ment Health 2014;17:111–6.
7 Schwarzer G. meta: an R package for meta-analysis. R News 2007;7:40–5.
interpret these results too much as clinical heterogeneity could 8 Schwarzer G, Carpenter JR, Rücker G. metasens: advanced statistical methods to
be another explanation for the small-study effect.22 A deeper model and adjust for bias in meta-analysis. R package version 0.4-0; 2019.
knowledge of the condition under study, of the treatment and 9 Schwarzer G, Carpenter JR, Rücker G. Meta-Analysis with R. Springer international
the settings in which it was administered in different trials could publishing, 2015. https://www.​springer.​com/​de/​book/​9783319214153
help to identify the probable reason for asymmetry in the funnel 10 Joy CB, Adams CE, Lawrie SM, et al. Haloperidol versus placebo for schizophrenia.
Cochrane Database Syst Rev 2006;(4):CD003082.
plot. 11 Higgins JPT, White IR, Wood AM. Imputation methods for missing outcome data in
In this publication, we could only provide a brief glimpse into meta-analysis of clinical trials. Clin Trials 2008;5:225–39.
statistical methods for meta-analysis available in R. The inter- 12 Nikolakopoulou A, Mavridis D, Salanti G. Demystifying fixed and random effects meta-
ested reader can see this publication as a starting point for other analysis. Evid Based Ment Health 2014;17:53–7.
13 Deeks JJ, Higgins JPT, Altman DG. Cochrane handbook for systematic reviews of
(more advanced) meta-analysis methods available in R. An over-
interventions 2008:243–96.
view of R packages for meta-analysis is provided on the website 14 Paule RC, Mandel J. Consensus values and weighting factors; 1982: National Institute
https://​cran.​r-​project.​org/​view=​MetaAnalysis. We would like to of Standards and Technology.
only briefly mention two R packages from this list. R package 15 Veroniki AA, Jackson D, Viechtbauer W, et al. Methods to estimate the between-study
metafor27 is another general package for meta-analysis, which in variance and its uncertainty in meta-analysis. Res Synth Methods 2016;7:55–79.
16 Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-
addition provides methods for multilevel meta-analysis28 as well
analysis. J R Stat Soc Ser A Stat Soc 2009;172:137–59.
as multivariate meta-analysis.29 R package netmeta30 implements 17 Mavridis D, Chaimani A, Efthimiou O, et al. Addressing missing outcome data in meta-
a frequentist method for network meta-analysis and is as of today analysis. Evid Based Ment Health 2014;17:85–9.
the most comprehensive R package for network meta-analysis. 18 Gamble C, Hollis S. Uncertainty method improved on best-worst case analysis in a
binary meta-analysis. J Clin Epidemiol 2005;58:579–88.
Contributors  SB drafted the manuscript and performed analyses; GR and GS 19 Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power
critically revised the manuscript and reviewed the analyses. of statistical tests and prevalence in the literature. J Clin Epidemiol 2000;53:1119–29.
20 Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and
Funding  The authors have not declared a specific grant for this research from any
funding agency in the public, commercial or not-for-profit sectors. interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials.
BMJ 2011;343:d4002.
Competing interests  None declared. 21 Peters JL, Sutton AJ, Jones DR, et al. Contour-enhanced meta-analysis funnel plots
Patient consent for publication  Not required. help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol
2008;61:991–6.
Provenance and peer review  Not commissioned; externally peer reviewed. 22 Rücker G, Carpenter JR, Schwarzer G. Detecting and adjusting for small-study effects
in meta-analysis. Biom J 2011;53:351–68.
ORCID iDs 23 Harbord RM, Egger M, Sterne JAC. A modified test for small-study effects in meta-
Sara Balduzzi http://​orcid.​org/​0000-​0003-​1205-​1895
analyses of controlled trials with binary endpoints. Stat Med 2006;25:3443–57.
Gerta Rücker https://​orcid.​org/​0000-​0002-​2192-​2560
24 Duval S, Tweedie R. A nonparametric “Trim and Fill” method of accounting for
Guido Schwarzer http://​orcid.​org/​0000-​0001-​6214-​9087
publication bias in meta-analysis. J Am Stat Assoc 2000;1:89–98.
25 Rücker G, Schwarzer G, Carpenter JR, et al. Treatment-effect estimates adjusted for
References small-study effects via a limit meta-analysis. Biostatistics 2011;12:122–42.
1 Borenstein M, Hedges LV, Higgins JPT, et al. Introduction to meta-analysis. Chichester: 26 Copas JB, Shi JQ. A sensitivity analysis for publication bias in systematic reviews. Stat
John Wiley & Sons, Ltd, 2009. Methods Med Res 2001;10:251–65.
2 Rothstein HR, Sutton AJ, Borenstein MH. Publication bias in meta-analysis: prevention, 27 Viechtbauer W. Conducting Meta-Analyses in R with the metafor Package. J Stat
assessment and adjustments. Chichester, UK: John Wiley & Sons, 2005. Softw 2010;36:1–48.
3 Davey J, Turner RM, Clarke MJ, et al. Characteristics of meta-analyses and their 28 Assink M, Wibbelink CJM. Fitting three-level meta-analytic models in R: a step-by-step
component studies in the Cochrane database of systematic reviews: a cross-sectional, tutorial. Quant Method Psychol 2016;12:154–74.
descriptive analysis. BMC Med Res Methodol 2011;11:160. 29 van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis:
4 R Core Team. R: a language and environment for statistical computing. Vienna R multivariate approach and meta-regression. Stat Med 2002;21:589–624.
Foundation for Statistical Computing; 2019. https://www.​R-​project.​org 30 Rücker G, Krahn U, König J, et al. netmeta: network meta-analysis using frequentist
5 StataCorp. Stata statistical software: release 16. College Station, TX: StataCorp LLC, methods. R package version 1.1-0, 2019. Available: https://​CRAN.​R-​project.​org/​
2019. package=​netmeta

160 Balduzzi S, et al. Evid Based Ment Health 2019;22:153–160. doi:10.1136/ebmental-2019-300117

© 2019 Author(s) (or their employer(s)) 2019. No commercial re-use. See
rights and permissions. Published by BMJ.