A brief introduction to

meta-analyses
Integrating the results of multiple studies

José M. Mestre Ph&D

 SUGGESTED READING
• Reading materials:
Higgins JPT, Green S (editors). Cochrane Handbook for Systematic
Reviews of Interventions Version 5.1.0 [updated March 2011]. The
Cochrane Collaboration, 2011. Freely available from: www.cochrane-
handbook.org

• For working with Comprehensive Meta-analysis software:

Video tutorials freely available at:
http://www.meta-analysis.com/pages/videotutorials.php

ACOSTUMBRENSE A LEER, AL MENOS, EN INGLÉS. ES EL LENGUAJE DOMINANTE EN LAS CIENCIAS. ESTAS TRANSPARENCIAS SERÁN EN INGLÉS AL MENOS

2
 THE STEPS IN A META-ANALYSIS
1. What are meta-analyses and why are they important?
2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses

4 0. Why are meta-analyses important?

0. Why are meta-analyses
important?
 THE STEPS IN A META-ANALYSIS

1. What are meta-analyses and why are they

important?
2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses
0. Why are meta-analyses important?
 THE STEPS IN A META-ANALYSIS
1.- What are meta-analyses and why are they important?
The explosion of science
Traditional and systematic reviews (including meta-analyses)
Advantages of systematic reviews
Problems of systematic reviews
2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses
0. Why are meta-analyses important?
The explosion
of
Science
0. Why are meta-analyses important?
 THE EXPLOSION OF SCIENCE

• Exponential increase in research

• Including the number of randomized trials
(RCT) examining the effects of treatments
• In 1965: 39 randomized trials, in 2013: 16,054.
Now the tripple
• Studies do not result in the same outcomes/ Ojo: esto
es lo que mas afecta a la psicoterapia

0. Why are meta-analyses important?

ACCUMULATION OF RANDOMIZED TRIAL PUBMED

400000

350000

300000

250000

200000

150000

100000

50000

0. Why are meta-analyses important?

NEW META-ANALYSES INPUBMED
8000

7000

6000

5000

4000

3000

2000

1000

0. Why are meta-analyses important?

Traditional and systematic
reviews

0. Why are meta-analyses important?

 SYSTEMATIC
TRADITIONAL OR REVIEW
‘NARRATIVE’REVIEW
• Clear objective with predefined eligibility
• Written by expert criteria for studies
• Selection of research is • Explicit, reproducable methodology
often unclear • Systematic search with attempt to identify all
• Summary of results is not studies
systematic • Assessment of validity of the included studies
• Systematic synthesis of findings
• The value of the conclusions
can not be verified
• A meta-analysis is a systematic review in which
• Emphasis on authority the results of individual studies are statistically
• Little transparancy integrated into one outcome

0. Why are meta-analyses important?

 ADVANTAGESOFMETA-ANALYSES

• More power by combining individual studies

• More precize and accurate estimates of effect
sizes
• Inconsistencies across studies can be analyzed
• Differences between subgroups of studies can
be examined
• Publication bias can be tested

0. Why are meta-analyses important?

 HOWCAN META-ANALYSES BEUSED?
• Doctors and therapists: decisions and guidelines
• Policy makers: laws and payments
• Patients: individual decisions
• For researchers:
• Generate new research questions
• Prevent methodological limitations
• Estimating sample sizes

0. Why are meta-analyses important?

CRITICISMS OF SYSTEMATICREVIEWS
• A series of small studies are not the same as one large study
• Sources of bias can not be controlled by the method
• Garbage in, garbage out
• Combining apples and oranges
• Agenda-driven bias and researcher allegiance

0. Why are meta-analyses important?

 KEY POINTS
• Because of the exponential growth of research there is a need to
integrate the results of multiple studies
• Traditional reviews are not systematic and transparant; they can
not solve the need for integration of research
• Systematic reviews have a reproducable methodology
• In a meta-analysis the results of individual studies are statistically
integrated into one (more precise) outcome
• Systematic review have many advantages for professionals,
patients, policy makers and researchers
• But there are also risks: bias, garbage in and out, and combining
apples and oranges
• The Cochrane Collaboration has a leading role in this field

23 0. Why are meta-analyses important?

 THE STEPS IN A META-ANALYSIS
1. What are meta-analyses and why are they important?
2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses

Step 1. Defining research questions

RESEARCH QUESTION:PICO

• Participants
• Interventions
• Comparisons
• Outcomes

Step 1. Defining research questions

 From: Higgins & Green; Cochrane Handbook, 2008

CRITERIA FOR ‘TYPES OFPARTICIPANTS’

• How is the disease/condition defined? BPD: borderline personality
disorder
• What are the most important characteristics that describe these people
(participants)? Emotional disregulation Uespecially, anger)
• Are there any relevant demographic factors (e.g. age, sex, ethnicity)?
(females teenagers)
• What is the setting (e.g. hospital, community etc)? families
• Who should make the diagnosis? A clinician psychologist!!!
• Are there other types of people who should be excluded from the review
(because they are likely to react to the intervention in a different way)?
Other externalizing disorders
• How will studies involving only a subset of relevant participants be
handled? RCT: randomized controlled trials
Step 1. Defining research questions
 CRITERIA FOR ‘TYPES OFINTERVENTIONS’
• What are the experimental and control (comparator) interventions
of interest
• Does the intervention have variations (e.g. dosage/ intensity,
mode of delivery, frequency, duration, therapists)?
• Are all variations to be included (for example is there a critical dose
below which the intervention may not be clinically appropriate)?
• How will trials including only part of the intervention be handled?
• How are trials including the intervention of interest combined
with another intervention handled?

From: Higgins & Green; Cochrane Handbook, 2008

Step 1. Defining research questions
 CRITERIA FOR ‘TYPES OFOUTCOMES’
• Main outcomes: essential for decision-making (usually emphasis on
patient-important outcomes)
• Primary outcomes help to reach a conclusion about the effects
(beneficial and adverse)
• Secondary outcomes include the remaining main outcomes
• Ensure that outcomes cover potential as well as actual adverse
effects.
• Consider outcomes relevant to all potential decision makers,
including economic data.
• Consider the type and timing of outcome measurements.

From: Higgins & Green; Cochrane Handbook, 2008

Step 1. Defining research questions
 EXAMPLE1.
• The effects of psychotherapy on depressive symptoms compared
to no treatment for women with postpartum depression

• P : women with postpartum depression

• I : psychotherapy
• C : no treatment
• O : depressive symptoms

Step 1. Defining research questions

 THE STEPS IN A META-ANALYSIS
1. What are meta-analyses and why are they important?
2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses

Step 2. Searching bibliographical databases

Selecting bibliographical
databases

Step 2. Searching bibliographical databases

THE MOST IMPORTANT BIBLIOGRAPHICALDATABASES
• Pubmed
• Psycinfo
• Cochrane Central Register of Controlled Trials
(CENTRAL)

• For web-addresses: see the online Cochrane Handbook

(http://handbook.cochrane.org)
Step 2. Searching bibliographical databases
OTHER BIBLIOGRAPHICALDATABASES
• National and regional databases, for example:
• Latin America: LILACS
• China: Chinese Biomedical Literature Database (CBM)
• India: indMED
• Subject specific databases, for example:
• International pharmaceutical abstracts
• Bibliomap (Health promotion)
• Global health (www.cabi.org)
• Nursing (CINAHL)
• Educational Resources Information Center (ERIC)
• AgeLine

For web-addresses: http://handbook.cochrane.org)

Step 2. Searching bibliographical databases
 MORE BIBLIOGRAPHICALDATABASES
• Citation indexes:
• ISI web of knowledge (Thompson Reuters)
• Scopus (Elsevier)
• Google Scholar
• Dissertations and theses
• ProQuest
• www.theses.com (Great Britain and Ireland)
• www.dissonline.de (Germany)
• Grey literature
• Conference proceedings (BIOSIS database)
• SIGLE (closed, but available on http://opensigle.inist.fr)
• National Technical Information Service (www.ntis.gov) with reports of
government-sponsored research

Step 2. Searching bibliographical databases

OTHER WAYSTO IDENTIFY STUDIES FORINCLUSION

• Handsearching or Tables of content

• Other reviews and guidelines
• DARE (database of abstracts of reviews of
effects)
• www.guideline.gov
• Ongoing studies: trial registers
• Contacting experts

Step 2. Searching bibliographical databases

THE SEARCH
PROCESS
What do you
search?

Search plan:

Adapt (if  search question(PICO)

necessary)  Choose search terms Exec ute
 Develop search strategy

Evaluation

Step 2. Searching bibliographical databases

FINDINGTHEBALANCEBETWEENSENSITIVITY AND PRECISION
• Higher comprehensiveness (sensitivity) reduces precision (more non-
relevant articles)

More
Result
s
Fewer

Narrow Broad

Search
strategy
Step 2. Searching bibliographical databases
DEVELOP A SEARCH STRATEGY
• Translate PICO to search strings
• Use text words as well as key words (MeSH terms)
 Text words are words used in the title, abstracts and other
fields
 Key words (MeSH terms) indicate a taxonomy for the
purpose of indexing articles
 keywords, synonym, variants (behavior / behaviour)

• Look in relevant papers, reviews which key words

they use
• Each database uses its own system of key words:
 MeSH (Medical Subject Heading) is used in PubMed
 EMTREE is used in Embase
• Ask search specialists in library for help!
Step 2. Searching bibliographical databases
 WORKING WITH BOOLEANOPERATORS
• Boolean operators that can be used include
 AND
 OR
 NOT

• Use “AND” for the key concepts (for example: “depression”

and “psychotherapy” and “randomized trial”) a)
• Use “OR” for synonyms (for example: “generalized anxiety”
OR “worrying”)
• Use “NOT” to exclude major categories of studies (for
example: “depression” AND “treatment” NOT
“antidepressant”)

a) Theseexamples are only for illustrative purposed and do

not contain fully functioning search strings
Step 2. Searching bibliographical databases
TRUNCATION, WILDCARDS AND PROXIMITYOPERATORS

• Truncation: *
 For example: random* identifies random, randomized, randomised, randomly, etc
• Wildcard: ?
 For example: m?n identifies man, men, min, mun, etc
• Proximity operators: ADJ
 For example: “depression adj3 disorder” returns articles where depression and
disorder are within 3 words of each other in any order. "adj3" means "adjacent
within 3 words”

Step 2. Searching bibliographical databases

USING SEARCHFILTERS

• Search filters
• www.york.ac.uk/inst/crd/intertasc/
• Simple search filter for randomized trials in
Pubmed:
randomized controlled trial [pt] OR
controlled clinical trial [pt] OR
randomized [tiab] OR randomly [tiab]

Step 2. Searching bibliographical databases

COCHRANEHIGHLYSENSITIVE SEARCHSTRATEGYFOR IDENTIFYINGTRIALS:
SENSITIVITY-MAXIMIZING VERSION
# 1 randomized controlled trial [pt] #2
controlled clinical trial [pt]
#3 randomized [tiab]
#4 placebo [tiab]
#5 drug therapy [sh] #6randomly
[tiab]
#7 trial [tiab]
#8 groups [tiab]
#9 #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8
#10 animals [mh] NOT humans [mh] #11 #9 NOT
#10

Step 2. Searching bibliographical databases

9
MeSHtree
 FOR
EXAMPLE
The effects of cognitive behavior therapy for major depression in
adults compared to waiting list control:

("depressive disorder, major"[MeSH Terms] OR "depressive

disorder"[MeSH Terms])
AND
(Cogniti* and (therapy OR treatment OR intervent*)) AND
(randomized controlled trial [pt] OR controlled clinical trial [pt] OR
randomized [tiab] OR randomly [tiab])

58 Step 2. Searching bibliographical databases

RESULTS(9 JULY2014)
SAVERESULTS
SEND RESULTSBYEMAIL
SAFE FILE FORENDNOTE
 KEY POINTS

• Search at least Pubmed, PsycInfo, Cochrane

database
• Find a balance between sensitivity and precision in
your search strategy
• Ask help from a librarian
• Use MeSH terms, boolean operators,
truncation, wildcards and proximity
operators

63 Step 2. Searching bibliographical databases

 THE STEPS IN A META-ANALYSIS

1. What are meta-analyses and why are they important?

2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses

Step 3. Selection of studies

 THE STEPS IN A META-ANALYSIS

1. What are meta-analyses and why are they

important?
2. Defining research questions for meta-analyses:
PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
 Selection of studies
 Data extraction
 Assessing methodological quality and bias
4. Calculating and pooling effect sizes
5. Examining heterogeneity
6. Reporting and publishing meta-analyses
Step 3. Selection of studies
HANDLING THERESULTSOFTHESEARCHES

• Save the results of searches in files (that

can be handled by the software you use)
• Enter results of searches in a bibliographical
software package, such as Endnote or
Reference Manager
• Remove duplicates
• Document your findings (you need to report
all this later in your paper): dates, search
strings, numbers, number of duplicates
removed, number of remaining abstracts

Step 3. Selection of studies

 WORKING THROUGH THE SEARCHRESULTS
• The resulting list of abstracts (duplicates removed) has to
be read by preferably two independent researchers
• Disagreements should be discussed and when no
solution is found a third (senior) researcher should be
involved
• In this phase a decision is made about retrieval of the full-
text of studies, not about in/exclusion of studies
• When there is doubt about possible retrieval: retrieve the
full-text

Step 3. Selection of studies

 SELECTING STUDIES FORINCLUSION
• Then retrieve the full-text of the selected papers through your library
• Usually a period of waiting (because not all papers are readily
available, and copies have to be ordered)
• When all papers are available the selection can start
• Again two independent researchers (with a third when needed)
• Make a list of inclusion and exclusion criteria
• Read all the papers carefully
• Decide about in- and exclusion (although this may change later in the
process)
• The data extraction can start!

Step 3. Selection of studies

DATATO BE RETRIEVED FROMSTUDIES

• Characteristics of the
studies
• Quality assessment

• Data to calculate effect http://www.facthum.com/

sizes que-significa-tamano-del-
efecto/

Preferrably by two independent reviewers

Step 3. Selection of studies

 CHARACTERISTICS OF THESTUDIES
Participants
 Disorder
 Severity
 Recruitment

Interventions
 Type
 Sessions
 Format
Comparisons
 Control group characteristics
Outcomes: these are part of ES calculation
Step 3. Selection of studies
Assessing methodological quality and
bias

Step 3. Selection of studies

 OVERVIEW

1. Why is assessment of risk of bias important?

2. Lists to assess quality and risk of bias
3. Risk of bias assessment tool (Cochrane Collaboration)
4. Analysing the influence of quality
5. Example: quality of studies on psychotherapy for adult depression

Step 3. Selection of studies

 WHY IS ASSESSMENTOFRISKOFBIAS IMPORTANT?

• Bias: a systematic error, or deviation from the truth, in results or

inferences
• Quality is not the same as bias
• Assessment of ‘risk of bias’ is important because it gives an
indication of the “deviation from the truth” of the included trials
and the pooled results

Step 3. Selection of studies

 HOWTO ASSESS QUALITY OFTRIALS

• Moher et al. (1995) identified 25 scales and 9 checklists

for quality
• Most are based on “generally accepted criteria”
• Number of items range from 3 to 34
• No generally accepted list available!
• Assessment of risk of bias is more straightforward

Step 3. Selection of studies

 SOME QUALITY CRITERIA INTRIALS
• (Correct) Randomisation?
• Concealment of randomisation
• Baseline differences between conditions?
• Was the patiënt blinded?
• Was the outcome assessor blinded?
• Was care-provider blinded?
• Was compliance acceptable?
• Was drop-out acceptable?
• Were outcome assessments at the same time?
• Intention-to-treat analyses?

Step 3. Selection of studies

 IMPORTANT SOURCES OFBIAS
1. Selection bias: systematic differences between
characteristics of the conditions; proper randomization:
 sequence generation
 allocation sequence concealment
2. Performance bias: blinding of study participants and
personnel: often not possible in psychotherapy research
3. Detection bias: Blinding of outcome assessors
4. Attrition bias: systematic differences between groups in
withdrawals from a study: intention-to-treat analyses.
5. Reporting bias: systematic differences between reported
and unreported findings.
6. Other biases
7. In psychotherapy research: Researcher allegiance
 COCHRANE RISK OF BIAS ASSESSMENTTOOL

Six criteria: Three possibilities to each

1. Adequate sequence generation criterion:
2. Allocation concealment • Low risk of bias
3. Blinding • High risk of bias
4. Incomplete outcome data • Unclear risk of bias
addressed
5. Selective reporting
Done by two independent
6. Other potential threats to validity researchers
 REPORT THE RISK OF BIAS
• Separately for each study
• Aggregated for all studies that are included
• Graphical representation

From: Kok et al., 2014

 FOR EXAMPLE: QUALITY AND
PSYCHOTHERAPY
• 115 controlled studies on psychotherapy for adult depression
• Risk of bias and quality criteria (Chambless & Hollon, 1998; Cochrane)
(1) diagnostic criteria for a depressive disorder
(2) treatment manual
(3) training of therapists
(4) treatment integrity was checked
(5) intention-to-treat analyses
(6) minimal level of statistical power (N>50)
(7) randomization conducted by independent party
(8) assessors of outcome were blinded
• Only 11 studies met all criteria!

Cuijpers et al., Psychol Med 2010

DIFFERENCEBETWEENHIGH-QUALITY ANDOTHERSTUDIES
N d NNT
All studies
- High-quality 16 0.22 8.06
- Other 162 0.75 2.48

CAU
- High-quality 9 0.22 8.06
- Other 34 0.51 3.55

Placebo
- High-quality 7 0.23 7.14
- Other 27 0.55 3.31

Cuijpers et al., Psychol Med 2010

KEY POINTS

• Work through the abstracts and selection of studies

preferrably with two researchers
• Make a clear overview of in/exclusion criteria to guide this
process
• Assessment of validity of studies is of vital importance
• The Cochrane risk of bias tool is a good instrument
THE STEPS IN A META-ANALYSIS

1. What are meta-analyses and why are they important?

2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses

Step 4. Calculating and pooling effect sizes

THE STEPS IN A META-ANALYSIS
1. What are meta-analyses and why are they important?
2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
 Calculating effect sizes
 Pooling of effect sizes
 Software for conducting meta-analyses
 Interpreting the forrest plot
 Interpreting the pooled effect size

5. Examining heterogeneity
6. Reporting and publishing meta-analyses

Step 4. Calculating and pooling effect sizes

EFFECT SIZES BASED ON CONTINUOUS
OUTCOMES
• From the two groups in each study that are compared you
need:
 Mean (usually at post-test)
 Standard deviation
 N

• Then you can calculate Cohen’s d:

Mcontrol – M experimental
Cohen’s d =
SDpooled

• Hedges’ g is Cohen’s d after adjustment for small sample bias

(pooled SD is calculated differently)

Step 4. Calculating and pooling effect sizes

WHAT IS COHEN’S D?
• The difference between 2 groups in terms of standard deviations

Treatment Control

-5 -4 -3 -2 -1 0 1 2 3

Step 4. Calculating and pooling effect sizes

WHAT IF SD IS NOT REPORTED IN APAPER?

• 95% Confidence interval around M is: M ± 1,96 SE

• SE= SD/√N = √(S/N)
• Values of p, t, F can also be used to calculate effect size
• Pre-post effect sizes: you need correlation between pre
and post

Step 4. Calculating and pooling effect sizes

 DATATO BE CONVERTED TO EFFECT SIZES
Study Outcome Mther SDther Nther Mctr SDctr Nctr

Study 1 Instr A 100 10 90 120 12 95

Instr B 50 6 90 60 9 95

Study 2 Instr A 80 15 30 95 17 26

Instr C 15 3 30 17 4 26

Study 3 Instr D 44 5 56 49 7 59

Step 4. Calculating and pooling effect sizes

COHEN’S D NOT EASY TO INTERPRET: CONVERSION TO NNT
d NNT d NNT
0,1 18 0,21-0,23 8
0,11 16 0,24-0,27 7
0,12 15 0,28-0,32 6
0,13 14 0,33-0,40 5
0,14 13 0,41-0,52 4
0,15 12 0,53-0,74 3
0,16 11 0,75-1,37 2
0,17-0,18 10 >1,38 1
0,19-0,20 9
NNT= numbers-needed-to-be-treated (how many patients have to be
treated for one more positive outcome than in control group

Methods in: Kraemer & Kupfer, Biol Psychiatry 2006

79 Step 4. Calculating and pooling effect sizes
EFFECT SIZES BASED ON DICHOTOMOUSOUTCOMES
Event No event Total

Therapy a b (a+b)
Control c d (c+d)
Odds Ratio (OR) = (a*d)
(b*c)

Relative Risk (RR) = a / (a+b)

c / (c+d)

Risk difference (RD) = risk in therapy – risk in control

= a / (a+b) – c / (c+d)

80 Step 4. Calculating and pooling effect sizes

 Software for conducting
meta-analyses

81 Step 4. Calculating and pooling effect sizes

 AVAILABLESOFTWARE PACKAGES

• STATA
• SPSS and SAS, (with the meta-analysis macros
for SAS and SPSS, by David B. Wilson; available
from:
http://mason.gmu.edu/~dwilsonb/ma.html
• Review manager (Cochrane Collaboration;
available from:
http://tech.cochrane.org/revman/download
• Comprehensive Meta-analysis

82
 COMPREHENSIVEMETA-
ANALYSIS
• Trial version available from:
http://www.meta-analysis.com/index.php
• Easy to use (cut and paste from spreadsheets)
• While most analyses are available

83
 COMPREHENSIVE META-ANALYSIS:
VIDEO TUTORIALS
• Video tutorials available at:
http://www.meta-analysis.com/pages/videotutorials.php
• Data entry
• Means
• Proportions
• Correlations
• Basic analysis
• Means
• Proportions
• Correlations
• Advances analysis
• Subgroups
• independent subgroups within studies
• Multiple outcomes
84
 WORKING WITH COMPREHENSIVE
META-ANALYSIS

• Download trial version from:

http://www.meta-analysis.com/pages/demo.php
• For working with Comprehensive Meta-analysis software:
Detailed video tutorials freely available at:
http://www.meta-analysis.com/pages/videotutorials.php

115 Faculty of Psychology and Education

 Interpreting the forest plot

86 Step 4. Calculating and pooling effect sizes

 THE FORRESTPLOT

• Excellent summary of meta-analysis

• Summary of each individual study (effect size +
95 % CI)
• Pooled outcome
• Examine “outliers” (outside pooled effect size)

87 Step 4. Calculating and pooling effect sizes

 THE FORRESTPLOT:ANEXAMPLE

From:Griffiths et al., World Psychiatry, 2014

88 Step 4. Calculating and pooling effect sizes
 THE FORRESTPLOT:ANEXAMPLE

Effect size

From:Griffiths et al., World Psychiatry, 2014

89 Step 4. Calculating and pooling effect sizes
 THE FORRESTPLOT:ANEXAMPLE

95% CI

From:Griffiths et al., World Psychiatry, 2014

90 Step 4. Calculating and pooling effect sizes
 THE FORREST PLOT: AN
EXAMPLE

Pooled effect size

From:Griffiths et al., World Psychiatry, 2014

91 Step 4. Calculating and pooling effect sizes
 THE FORRESTPLOT:ANEXAMPLE

Outlier

From:Griffiths et al., World Psychiatry, 2014

92 Step 4. Calculating and pooling effect sizes
 From: Cuijpers et al., Am J Psychiatry 2008
93 Step 4. Calculating and pooling effect sizes
 Interpreting pooled effect
sizes

94 Step 4. Calculating and pooling effect sizes

 SMALL, MODERATE, LARGE
EFFECTSIZES?
According to Cohen (1988):
 Small: 0.2
 Moderate: 0.5
 Large: 0.8

According to Lipsey and Wilson (1993) (based on

estimate of multiple meta-analyses):
 Small: <0.32
 Moderate: 0.33-0.55
 Large: >0.56

95 Step 4. Calculating and pooling effect sizes

 SIGINIFICANCE AND RELEVANCE
• It has been suggested that d=0.5 is the cut-off for
clinical relevance (NICE, 2004)
• But: Effect size does not say anything about
clinical relevance
• It is still a statistical measure
• For example:
• An effect size of d=0.1 on years of survival is a highly
relevant effect
• But an effect of d=0.1 on for example social skills is
not considered relevant

Cuijpers, Depress Anx 2014

96 Step 4. Calculating and pooling effect sizes
 KEYPOINTS

• Cohen's d indicates the difference between the treated

and the control at post-test in terms of standard deviation
• Many other statistics can be pooled in meta-analyses
• Pooling means the statistical integration of the result of
multiple studies into one overall effect size
• The forest plot gives a good summary of a meta-analysis
• Effect sizes d of 0.2 are small, 0.5 moderate and 0.8 large

151
 THE STEPS IN A META-ANALYSIS

1. What are meta-analyses and why are they important?

2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses

98 Step 5. Examining heterogeneity

 THE STEPS IN A META-
ANALYSIS
1. What are meta-analyses and why are they important?
2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
 What is heterogeneity?
 Examining the causes of heterogeneity
 Publication bias
6. Reporting and publishing meta-analyses

99 Step 5. Examining heterogeneity

 What is heterogeneity?

100 Step 5. Examining heterogeneity

 WHAT ISHETEROGENEITY
• Statistical heterogeneity is the variability in the
intervention effects being evaluated in the different
studies (Cochrane Handbook)
• It manifests itself in the observed intervention
effects being more different from each other
than one would expect due to random error
(chance) alone.
• Statistical heterogeneity is not the same as
clinical heterogeneity
• Clinical heterogeneity refers to the variability
in the participants, interventions and
outcomes studied.

101 Step 5. Examining heterogeneity

 HETEROGENEITY
No heterogeneity:
> All studies are exact replications
> Effect sizes are divided around the “true” effect size
> Fixed effects model

Heterogeneity:
> There are differences between studies we can not explain
> Random effects model
> Try to find explanation!
> Outliers, subgroup-analyses, meta-regression

102 Step 5. Examining heterogeneity

 EXAMINING WHETHERTHEREISHETEROGENEITY

1. Visual inspection of the forrest plot

2. Test for homogeneity (Q) (is it significant?)

3. Quantifying heterogeneity (I2) (in
percentages)

103 Step 5. Examining heterogeneity

 IS THERE HETEROGENEITY: 1. VISUALINSPECTION OF
THE FORRESTPLOT

• Look at the results in the forest plot

• Look at point estimates
• Overlap of confidence intervals: outliers

104 Step 5. Examining heterogeneity

 PSYCHOTHERAPY VERSUS PLACEBO FOR ADULT
DEPRESSION

I2=0%; 95% CI: 0–58% Cuijpers et al., Psychol Med 2014:

105 Step 5. Examining heterogeneity
Psychotherapy
for depression in
older adults

Cuijpers et al., Maturitas 2014

Psychotherapy
for depression in
older adults

Cuijpers et al., Maturitas 2014

Psychotherapy
for depression in
older adults

I2= 80; 95% CI: 73–85

Cuijpers et al., Maturitas 2014

 IS THERE HETEROGENEITY: 2. TESTOFHOMOGENEITY

• Tested hypothesis: Effects are homogenuous in

all trials (or: the effects are equal)
• Q-test indicates whether there is significant
heterogeneity
• Q-test is not very attractive: depends on size and
number of studies
• In many meta-analyses the number of effect
sizes is small: suggesting no heterogeneity

109 Step 5. Examining heterogeneity

 IS THERE HETEROGENEITY: 3) QUANTIFYING
HETEROGENEITY (I2)
• I2 is the proportion of the total variance that can
be explained by heterogeneity
 I2 = 0% no heterogeneity
 I2 = 25% low heterogeneity
 I2 = 50% moderate heterogeneity
 I2 = 75% high heterogeneity
• It is important to calculate 95% confidence interval
around I2 (Ioannidis et al., 2007)
• Heterogeneity is often small or zero and not significant,
but when 95% confidence intervals are broad, it is
uncertain whether there is heterogeneity

110 Step 5. Examining heterogeneity

 PSYCHOTHERAPY VERSUS PLACEBO FOR ADULT
DEPRESSION

I2=0%; 95% CI: 0–58% Cuijpers et al., Psychol Med 2014:

111 Step 5. Examining heterogeneity
 Examining the causes of
heterogeneity: moderator
analyses

112 Step 5. Examining heterogeneity

 MODERATOR
ANALYSES
• Examine the association between characteristics
of studies and the effect size
• Such associations may explain heterogeneity
• Always indirect evidence (an association is only
correlational)
• Three types of moderator analyses
1. Subgroup analyses
2. Bivariate metaregression analyses
3. Multivariate metaregression analyses

113 Step 5. Examining heterogeneity

 EXAMINING
PUBLICATIONBIAS
• Direct examination in trial databases:
 For
example: Antidepressants (Turner et al., Nw Engl J
Med 2008): Effect size dropped from d=0.41 to d=0.31
• Statistical methods based on the symmetry of the
funnel plot:
 Visualinspection of the funnel plot (no test)
 Some tests of the symmetry (resulting in a p-value):
o Begg and Mazumdar Rank Correlation Test
o Egger's Test
 Imputing missing data: Duvall & Tweedie’s trim & fill
procedure

114 Step 5. Examining heterogeneity

 PUBLICATIONBIAS?UNADJUSTED

From: Cuijpers et al., Br J Psychiatry 2010

115 Step 5. Examining heterogeneity
 PUBLICATIONBIAS?ADJUSTED

116 Step 5. Examining heterogeneity

 PUBLICATIONBIAS: RESULTSIN PSYCHOTHERAPY FOR
ADULTDEPRESSION

• Unadjusted: d=0.67 (95% CI: 0.60~0.75)

• Adjusted: d=0.42 (95% CI: 0.45~0.57)
• Imputed studies: 51
• Egger’s, Begg & Mazumdar test: p<0.001
• Also publication bias in subgroups

117 Step 5. Examining heterogeneity

 KEY POINTS
• Heterogeneity is the variability in the intervention effects in the
different studies
• The level of heterogeneity is important for the interpretation
of the pooled effect size
• Heterogeneity can examined through moderator analyses
• Possible publication bias can be assessed through the asymmetry of
the funnel plot

182
 THE STEPS IN A META-ANALYSIS

1. What are meta-analyses and why are they important?

2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses

119 Step 6. Reporting and publishing meta-analyses

 THE STEPS IN A META-ANALYSIS

1. What are meta-analyses and why are they important?

2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses
 The PRISMA Statement
 Structure for a publication about a meta-analysis

120 Step 6. Reporting and publishing meta-analyses

 The PRISMA Statement

121 Step 6. Reporting and publishing meta-analyses

 THE PRISMA
STATEMENT
• Preferred Reporting Items for Systematic Reviews
• and Meta-Analyses (PRISMA)
• PRISMA focuses on ways in which authors can ensure
the transparent and complete reporting of systematic
reviews and meta-analyses
• www.prisma-statement.org
• Checklist and flowchart

122 Step 6. Reporting and publishing meta-analyses

188
 THE ITEMS FROMPRISMA

124 Step 6. Reporting and publishing meta-analyses

 THE ITEMS FROMPRISMA(CONTINUED)

125 Step 6. Reporting and publishing meta-analyses

 Structure for a publication
about a meta-analysis

126 Step 6. Reporting and publishing meta-analyses

 STRUCTURE OFAPAPER

• Introduction
• Methods
• Results
• Discussion
• Tables and figures

127 Step 6. Reporting and publishing meta-analyses

 STRUCTURE OF A
PAPER
• Introduction
 explain the background,
 the importance of the problem,
 earlier (meta-analytic) research,
 why this new meta-analysis is needed,
 end with the research question (PICO)

• Methods
• Results
• Discussion
• Tables and figures

128 Step 6. Reporting and publishing meta-analyses

 STRUCTURE OF A
PAPER
• Introduction
• Methods
 Identification (searches) and selection of studies (in/exclusion
criteria
 Data extraction and quality assessment
 Analyses (which effect size, how was it calculated, pooling, the
model used, heterogeneity, publication bias, subgroup and
metaregression analyses)
• Results
• Discussion
• Tables and figures

194 Step 6. Reporting and publishing meta-analyses

 STRUCTURE OF A
PAPER
• Introduction
• Methods
• Results
 Selection and inclusion of studies (PRISMA flowchart)
 Characteristics of included studies, including quality/validity, table
with selected characteristics
 Outcomes: pooled effect sizes
 Other analyses, subgroup and metaregression analyses,
publication bias

• Discussion
• Tables and figures

195 Step 6. Reporting and publishing meta-analyses

 STRUCTURE OF A
PAPER
• Introduction
• Methods
• Results
• Discussion
 Summary of main results
 What does this add to existing knowledge
 Implications for research and practice
 Future research
 Limitations
 Conclusion

• Tables and figures

131 Step 6. Reporting and publishing meta-analyses

 STRUCTURE OF A
PAPER
• Introduction
• Methods
• Results
• Discussion
• Tables and figures
•  Figure. Flowchart of inclusion of studies (PRISMA flow diagram)
•  Figure. Forrest plot
•  Table. Selected characteristics of included studies
•  Preferrably: table summarizing meta-analyses

132 Step 6. Reporting and publishing meta-analyses

 STRUCTURE OF A
PAPER
• Introduction
• Methods
• Results
• Discussion
• Tables and figures
 Figure. Flowchart of inclusion of studies (PRISMA flow diagram)
 Figure. Forrest plot
 Table. Selected characteristics of included studies
 Preferrably: table summarizing meta-analyses

198 Step 6. Reporting and publishing meta-analyses

From: Moher et al., Plos Med 2009
 STRUCTURE OF A
PAPER
• Introduction
• Methods
• Results
• Discussion
• Tables and figures
•  Figure. Flowchart of inclusion of studies (PRISMA flow diagram)
•  Figure. Forrest plot
•  Table. Selected characteristics of included studies
•  Preferrably: table summarizing meta-analyses

135 Step 6. Reporting and publishing meta-analyses

 FORRESTPLOT

Cuijpers et al., Psychol Med 2014:

136 Step 6. Reporting and publishing meta-analyses
 STRUCTURE OF A
PAPER
• Introduction
• Methods
• Results
• Discussion
• Tables and figures
•  Figure. Flowchart of inclusion of studies (PRISMA flow diagram)
•  Figure. Forrest plot
•  Table. Selected characteristics of included studies
•  Preferrably: table summarizing meta-analyses

137 Step 6. Reporting and publishing meta-analyses

TABLE. SELECTED CHARACTERISTICS OF INCLUDED STUDIES
Study Recruit- Target Condi- N For- Validity Coun-
ment group tions mat try
One, 1999 Clinical Adults 1. CBT 49 Ind − + − − + + USA
2. PST 51 In
3. CAU 48
Two, 2006 Comm. Elderly 1. IPT 65 Ind + + + + + + NL
2. WL 63
Three, 2013 Clinical Adults 1. PST 24 Grp + − + − + + USA
2. BAT 22 Grp
3. CAU 23
Four, 1995 Comm. Elderly 1. CBT 78 Grp − + − + − − China
2. WL 82
Five, 2003 Comm. Cancer 1. CBT 16 Ind + + + + + + UK
2. IPT 18 Grp
….

138 Step 6. Reporting and publishing meta-analyses

 STRUCTURE OF A
PAPER
• Introduction
• Methods
• Results
• Discussion
• Tables and figures
•  Figure. Flowchart of inclusion of studies (PRISMA flow diagram)
•  Figure. Forrest plot
•  Table. Selected characteristics of included studies
•  Preferrably: table summarizing meta-analyses

139 Step 6. Reporting and publishing meta-analyses

 PREFERRABLY: TABLE SUMMARIZING
META-ANALYSES
N g (95% CI) I2 (95% CI) p NNT

All studies 12 0.25 (0.14-0.36) 0 (0-58) 7

Only HAM-D-17 10 0.34 (0.21-0.46) 0 (0-62) 5
Only BDI 7 0.30 (0.13-0.46) 0 (0-71) 6

Subgroup analyses
Type of therapy
- CBT 5 0.33 (0.16-0.50) 0 (0-79) 0.23 5
- Other 7 0.20 (0.06-0.33) 1 (0-71) 9
Recruitment
- Clinical 6 0.31 (0.13-0.50) 30 (0-72) 0.38 6
- Other 5 0.21 (0.06-0.36) 0 (0-75) 8

140 Step 6. Reporting and publishing meta-analyses

 KEY POINTS

• The PRISMA Statement is a useful tool for complete and

transparant reporting of meta-analyses
• A paper on a meta-analyses can be be structured in a
format comparable with any other scientific paper

206
 THE STEPS IN A META-ANALYSIS

1. What are meta-analyses and why are they important?

2. Defining research questions for meta-analyses: PICO
3. Searching bibliographical databases
4. Selection of studies and retrievement of data
5. Calculating and pooling effect sizes
6. Examining heterogeneity
7. Reporting and publishing meta-analyses

207 Step 6. Reporting and publishing meta-analyses

RCT
 MUCHA SUERTE Y QUE DISFRUTÉIS DE UN MARAVILLOSO
APRENDIZAJE
https://www.facebook.com/TreatMeEU/

CA16102 - European Network on

Individualized Psychotherapy
Treatment of Young People with
Mental Disorders

https://www.treat-me.eu/

STSM and ITN coordinator