Systematic Review

and

Meta Analysis
 Understand the terminologies
• Prevalence
• Odds ratio
• Risk Ratio
• Standard Deviation and Standard Error
• 95% CI
• Statistical Significance
 Event No-event Total
Treatment a b a+b
Control c d c+d
Total a+c b+d N

⚫ Odds-ratio=

⚫ Risk ratio=

⚫ Proportion or prevalence= Number of cases

= (a+c)
examined/ Population

N
Statistical Significance from CI

CI should not contain NULL

Standard Deviation
Standard Error

Sample-1 Sample-2
Mean Mean

Population
Mean

Sample-3 Sample-4
Mean Mean
Evidence pyramid

Meta analysis
 History

Archibald Cochrane, an epidemiologist, published an

influential book in 1972 (Effectiveness and Efficiency)
criticized our collective ignorance about the effects of
health-care.

“It is surely a great criticism of our profession that we have

not organized a critical summary, by specialty or
subspecialty, updated periodically, of all relevant
randomized controlled trials”
What is a Systematic Review?
• It is a detailed, systematic and transparent means of gathering,
appraising and synthesizing evidence to answer a well-defined question.

• In basic terms, a systematic review is a protocol-driven, comprehensive

literature review, usually designed to answer a specific clinical question.
What is meta analysis?
Quantitative approach for systematically combining results of previous
research to arrive at conclusions about the body of research.

• Quantitative : Numbers
• Systematic : Methodical
• Combining: Putting together
• Previous research: What's already done
• Conclusions: New knowledge
 Study eligibility
• Study eligibility is assessed according to pre-defined eligibility criteria
related to the study itself such as,
• Study design
• Study population
• Exposure/s and outcome/s of interest
• Language
• Year of publication.
• Usually two reviewers assess each study for eligibility to reduce errors
and bias.
1. Specify Problem
2. Search for and Identify
Studies
3. Enter studies into database
4. Select Studies for Review
5. Review Studies
6. Develop Coding Scheme
7. Abstract / Code Studies
8. Select Effect Size Statistic
9. Transform and Weight Effec
t Sizes
10. Assess heterogeneity
11. Assess Bias
 Develop Coding Scheme

⚫ Coding instruments-coding sheets and

codebooks, are designed for a specific
research synthesis
⚫ Reflect the types of interventions, outcome
variables, and other data that are commonly
reported in the relevant literature
⚫ Thorough knowledge of the literature they intend
to
synthesize
⚫ Coded data
(a) methodological and substantive features,
(b) study quality,
(c) intervention descriptors, and
Methodological and substantive
features

⚫ Source of the study,

⚫ Year of publication,
⚫ Type of research design, and
⚫ Characteristics of the
authors/investigators (e.g.,
author’s discipline, educational
credentials).
Data extraction

• Data extraction is performed according to a standardized form

• Usually two reviewers extract data for each study for eligibility to
reduce errors and bias.
• Data extraction for observational studies might be less straight
forward than for RCTs because multiple analyses may have been
conducted (e.g.unadjusted and adjusted, with analyses adjusting for
different sets of potential confounders), and each observational study
design will have different data to be extracted.
 Data Collection

• The list of data to be extracted should be decided a priori.

• A data extraction form should be used so that the same data are
extracted from each study and missing data are clearly apparent.

• To be sure that data extraction is accurate and reproducible, it should be

performed by at least two independent readers.

• Disagreement between readers could be solved by agreements or by a

third reviewer
Collected data includes:

• Study characteristics (year and journal of publication, number of

patients in each arm, treatments performed, duration of follow-up)

• Sample demographics (age, % males or females)

• Sample characteristics (traditional CV risk factors - % hypertensive

pts, % diabetic pts, % dyslipidemic pts, % smokers – concomitant
treatments, comorbidities, etc)

• Outcome data (all-cause death, CV death, MI, stroke, etc)

 Assessment of Study Quality

• Quality of evidence in studies ↓; Quality of MA ↓ leading to

incorrect results
• Method of evaluating quality of evidence
• Newcastle-Ottawa Scale (NOS),
• National Toxicology Program (NTP) Office of Health Assessment and
Translation systematic review framework,
• Strengthening the Reporting of Observational Studies in Epidemiology
(STROBE) checklist,
• Critical Appraisal Skills Program (CASP) checklist
Intervention categories and
relevant outcome measures
⚫ Specific to a study
⚫ Driven by clear research questions
⚫ Difficult to ensure inclusion of all relevant
variables so that data are not missed
⚫ Difficult to limit the variables in accordance
with the time and financial resources of the
investigators
 Outcome
measures
Binary outcome
⚫ Proportion : No. of people experiencing an event/no.
Examined
⚫ Control event rate(CER) : Proportion of events in the
control group
⚫ Experimental event rate (EER): Proportion of
events in the Experimental group

Continuous outcome
⚫ Mean difference= Mean of an outcome in the expt
group-mean of the outcome in the control group
=Mean before-after difference(for BA
studies)
⚫ Correlation
⚫ Response ratio= Mean in the experimental
Mean in the controls
 Binary effect
measures
Event No-event Total
Treatment a b a+b
Control c d c+d
Total a+c b+d N

⚫ Odds-ratio=

⚫ Risk ratio=

⚫ Proportion or prevalence= Number of cases

= (a+c)
examined/ Population

N
⚫ Comparable
⚫ Standardized numeric scale for
evidence across disparate studies
⚫ Amenable to calculation of standard
error
⚫ Must not be a direct function of sample
size
 Example of Different Outcomes

Study 1 (Prevalence Study):

A cross-sectional study conducted in India reports that 30% of adults aged 40–
60 years have diabetes.

Study 2 (Odds Ratio Study):

A case-control study from the USA finds that individuals with obesity have an
odds ratio (OR) of 2.5 for developing diabetes compared to individuals with a
normal weight.
 Example: The prevalence of smoking is the variable of interest.

Study Smokers Total Participants Prevalence

Hypertension 200 500 40
Diabetes 150 1000 15
Obesity 125 500 25
COPD 210 300 70

Combines diverse studies to focus on a common secondary variable

(smoking prevalence), allowing for valuable insights despite differing
primary objectives.
 Weights of Studies

The weights of studies in a meta-analysis indicate how much each

study contributes to the overall pooled effect size.

For Binary Outcomes

For Continuous Outcomes
Heterogeneity

Differences in results between studies.

When heterogeneity exists between studies, it is important to

understand why as this will alter the conclusions drawn by the review.
Heterogeneity in meta-analysis

• Heterogeneity in meta-analysis refers to the variation in study

outcomes between studies.
• The I² statistic describes the percentage of variation across studies
that is due to heterogeneity rather than chance.
• While determining what constitutes a large I2 value is subjective, the
following rule-of thumb can be used: < 40% may be low. 30-60% may
be moderate. 50-90% may be substantial. 75-100% may be
considerable.
Assess heterogenity
•Study design (inclusion criteria, treatment,
duration)
•Study quality (randomisation, blinding etc)
•Individual level (prognostic factors)
•Outcomes (chance results)
•inadequate sample size
• different patient follow-up
•different statistical analysis
•different reporting
•different patient response
Classical Test
To test the heterogeneity between i=1,2,3….k
studies
k
Cochrane’s
wi ( y  y) 2

~ k
2

Q= i
1•
Low Power for small k
•Large power for large
k

I 2 statistic = 100% x (Q -df)/Q

Percentage of variation due to heterogeneity

• Does not depend on

I-
Squar
e⚫ is interpretable
⚫ is easy to calculate
⚫ is ‘size invariant’ (unlike the Q test)
⚫ number of studies in the meta-analysis
⚫ is ‘scale invariant’ (unlike τ2)
⚫odds ratio / mean difference / hazard
ratio etc is based only on estimates and
weights from each
study
⚫ Where heterogeneity is found, investigate
reasons for it
⚫ Use sensitivity or subgroup analyses (defined
a priori)
⚫ Regress important covariates to examine
effect
⚫ Random effects model widens CIs but does
not
explain heterogeneity
◦ Graphical methods
◦ Subgroup analysis
◦ Sensitivity analysis
◦ Meta-regression
 Analysis-Choice of models
Fixed effects model – Homogeneity I2 is small
Did the treatment produce any benefit on an average in
the studies at
hand?

Assumption:
•Studies use identical methods, patients, and
measurements;
•Produce identical results;
•Differences are only due to within-study variation.

Random effects model – Heterogeneity- Significant Q

Will the treatment produce benefit ‘on an average’?

Assumption:
• Studies are a random sample from the universe of all
possible studies
• Differences occur both due to between-study and
 Data Synthesis

• Once the data have been extracted and their quality and validity
assessed, the outcomes of individual studies within a systematic
review may be pooled and presented as summary outcome or effect
Primary outcome: Basis for meta-analysis
Forest Plot
 Graphical
exploration

I-squared =
92%

Meta-analysis on efficacy of BCG vaccination for

TB
Subgroup
analysis:example

Egger et al. Systematic reviews in health care. London: BMJ

Checking for publication Bias
• Studies with significant results are more likely
◦ to be published
◦ to be published in English
◦ to be cited by others
◦ to produce multiple publications
• Including only published studies can introduce publication bias
• Most reviews do not look for publication bias
• Methods for detecting publication bias:
• Graphical: funnel plotasymmetry
• Tests: Egger test,Rosenthal’s Fail-safe N [all have low power]
 Examples of Publication Bias
1.Positive Results Bias:
2.Selective Outcome Reporting:
Only the statistically significant outcomes are reported in the published paper,
while nonsignificant outcomes are omitted.
3.Language Bias:
Studies with significant results are published in high-impact English-language
journals, whereas studies with nonsignificant results are published in low-
impact, non-English journals.
4.Time-Lag Bias:
Studies with significant findings are published quickly, while studies with null
results face delays in the peer-review or publication process.
 Real-Life Example
In the 1980s, selective serotonin reuptake inhibitors (SSRIs)
like Prozac were hailed as highly effective antidepressants.
However, unpublished trials with null or negative results were
later revealed, indicating that the drugs' efficacy had been
overestimated due to publication bias.
 Funnel plot
• The easiest way to analyze a funnel plot is
to look at the shape of the graph
• Does it resemble a symmetrical inverted
funnel?
• Small trials should scatter widely across
the bottom of a funnel plot with the
spread narrowing as the trial size increases
• If small ‘negative’ trials are missing from
the meta analysis there will be a gap in the
bottom right (or left) of the funnel
 When can meta-
analyses mislead?
⚫ When a meta-analysis is done outside of a systematic
review
⚫ When poor quality studies are included or when
quality issues
are ignored
⚫ When small and inconclusive studies are included
⚫ When inadequate attention is given to heterogeneity
Indiscriminate data aggregation can lead to inaccurate
conclusions
⚫ When reporting biases are a problem
◦ Publication bias
◦ Time lag bias
◦ Duplicate publication bias
M e t a X L
• Add-in for meta-analysis in
Microsoft Excel for Windows.

• It supports all major

meta-analysis methods,
plus, uniquely, the inverse variance
Heterogeneity and quality effects models.

• Output is in table and

graphical formats.
• Binary (relative risk, odds ratio,
risk difference, prevalence)

• continuous (weighted mean, mean

difference, Cohen’s d, correlations)
methods,

• rates, ratio and rate difference

effect sizes.
Interface
 Dos and Don'ts while doing the data entry

• Alphabets and Numeric should not be combined

• Don’t enter as per your wish/ Convenience
• Don’t enter everything what you had done in data extraction form

• Find the appropriate Input Template

• Enter the Study name
• Only numeric data for outcome
How to input Data
Lets have a quick look at
What to note down
• Effect Size:

• Heterogeneity Statistics:

• I 2 : Percentage of variation due to heterogeneity rather

than chance.

• Q: Statistical significance of heterogeneity

• Weights of Studies:

• P-Value:
• Significance level for the pooled effect size and
heterogeneity test.
• Subgroup Analysis Results (if applicable):

• Publication Bias Assessment

Funnel Plot