Meta-analysis

8 steps in conducting a meta-analysis (Hansen et al., 2022)

Step 1. Define research question
 Two hurdels:
o Number of publications: balance manageability and relevance
o Similar meta-analysis already exist
 Solution:
 consider moderators or mediators of the relationship
 replicate prior meta-analysis by testing their finding with an updated sample of primary
studies or newly developed methods
Step 2. Literature search
 Search strategies
1. keyword search in electronic databases1, see (Gusenbauer & Haddaway, 2020) for 34 databases
2. prior meta-analyses/reviews on the same/related topic: lists of included studies
3. track forward citations of influential studies
4. unpublished or undetected studies: conference, listservs, personal contact with leading scholars,
databases (e.g., ProQuest Dissertation and Theses, AllAcademic.com, osf.co, data.gov)
 Specific guidance on documenting search procedures (e.g., Harari et al., 2020; Higgins &
Thomas, 2019)
o Describe and justify the variations in search terms across databases
o Moher et al. (2009, p. 266) including a flow diagram depicting the “numbers of studies
screened, assessed for eligibility, and included in the review, with reasons for exclusions at
each stage.”

1
At least use two different databases (DeSimone et al., 2020), to prevent biased results due to the scope or journal coverage of
one database.
Gusenbauer & Haddaway (2020) point out that Google Scholar is not appropriate as a primary search engine due to a lack of
reproducibility of search results.
 o Report the order of application of exclusion rules and the number of studies remaining after
each step
o
 Level of analysis
o Explicitly specify the level(s) (i.e., individual? team? organization? and/or industry?) of interest
for the phenomena under investigation and how your search strategy yield appropriate data.
o Research has demonstrated the fallacy of generalizing across levels (Glick, 1985; Rousseau,
1985).
 study inclusion criteria and sample composition
o principle: inclusion of all available studies (Steel et al., 2021)2
o the screening process should be conducted stepwise:
 1.remove duplicate citations from different databases
 2.abstrat screening to exclude clearly unsuitable studies
 3.full-text screening
o A graphical tool to document the sample selection process: PRISMA (Moher et al., 2009; Page
et al., 2021)
o exclusion criteria:
 use unsuitable variable measurement
 not report usable effect size
 multiple studies by the same authors using similar datasets (if they don’t differ
sufficiently the sample characteristics or variables used, only one of these studies should
be included, see (Wood, 2008)for a detection heuristic)

Step 3. Effect size

 two most common: correlation coefficient (r) and standardized mean difference (d)
 conversion: convert different effect sizes to the chosen primary effect size
o conversion formulae (Hedges et al., 2009; Lipsey & Wilson, 2001)
o online effect size calculator for meta-analysis: https://www.campbellcollaboration.org/escalc/

Step 4. Four analytical methods

2
Not just limite the studyies published in renowed academic journals to ensure the quality of dinings.
1. Univariate meta-analysis
 Report a weighted mean effect size for the relationship or intervention
 Limitation: examine boundary conditions of the relationship of interest
2. Meta-regression analysis
 Aim: investigate the heterogeneity among observed effect sizes by testing multiple potential
moderators simultaneously
 Further reading:
o Tipton et al. (2019): outline the technical, conceptual, and practical developments of meta-
regression,
o Gonzalez-Mulé & Aguinis (2018): provide an overview of methodological choices and
develop evidence-based best practices for future meta-analyses in mgmt using meta-
regression.
3. Meta-analytic structural equation modeling (MASEM)
 Aim: simultaneously investigate the relationships among several constructs in a path model.
 Application:
o test several competing theoretical models again each other;
o identify mediation mechanisms in a chain of relationships.
 Procedure (Cheung & Chan, 2005):
o Step1. derive a pooled correlation matrix that includes the meta-analytical mean effect
sizes for all variable combinations;
o Step2. Use this matrix to fit the path model.
 Further reading:
o Cheung (2015a) and Jak (2015): provide an overview of these approaches in their books
with exemplary code. For datasets with more complex data structures.
o Wilson et al. (2016): develope a multilevel approach that is related to the TSSEM approach
in the second step.
o Bergh et al. (2016): discuss nine decision points and develop best practices for MASEM
studies.
 4. Qualitative meta-analysis
 Aim: identify influential variables or patterns and derive a meta-causal network.
 Further reading:
o Levitt (2018): provides a guide and discusses conceptual issues for conducting qualitative
meta-analysis in psychology, which is also useful for mgmt researchers.
Step 5. Software
 Programs purely dedicated to meta-analysis: Comprehensive Meta-Analysis (Borenstein et al.
2013); RevMan by The Cochrane Collaboration (2020)
 Excel: Meta-Essentials (Suurmond et al., 2017)3

3
The workbooks can be downloaded from: https://www.erim.eur.nl/research-support/meta-essentials/.
  R: metafor package4 (Viechtbauer, 2010)
 SPSS, SAS: existing macros5

Step 6. Coding of effect size

 Design a coding sheet
o Should specify any coding rules or parameters used (e.g., how nonindependent effect sizes
were handled, coding procedures used for primary studies that report multiple effect sizes or a
range of sample sizes in a table)
o Should use at least two independent coders, report intercoder agreement, and report how
disagreements were resolved (Russo, 2007)
 Inclusion of moderator or control variables
o The decision should be based on strong (theoretical) rationales about how these variables can
impact the investigated relationship.
o Havranek et al. (2020): provide a list of recommended variables to code as potential
moderators.
 Treatment of multiple effect sizes in a study
o Case1. Use multiple variables for the same construct within a single primary study, resulting in
multiple relevant effect sizes
 Researcher can either (randomly) select a single value calculate a study average, or use
the complete set of effect sizes (Bijmolt & Pieters 2001; López-López et al. 2018)
 Nonindependence of effect sizes
o Effect sizes computed using non-independent samples can distort summary effect sizes and
their confidence intervals.
4
The tutorials can be found via: https://www.metafor-project.org/doku.php.
metafor does currently not provide functions to conduct MASEM. For MASEM, users can, for instance, use the package metaSEM
(Cheung, 2015b).
5
The macros of David B. Wilson can be downloaded from: http://mason.gmu.edu/~dwilsonb/.
The macros of Field and Gillet (2010) can be downloaded from:
https://www.discoveringstatistics.com/repository/fieldgillett/how_to_do_a_meta_analysis.html.
 o Types of non-independent samples
 When the degree of dependence is known, multivariate meta-analyses can be used to
model this dependence
 Unkown dependence:
 Three-level meta-analysis
o Examples include
 overlapping samples across different studies
 different time points within the same study
 different treatment groups compared to a common control group, or
 different measures of the same underlying construct (e.g., multiple measures of job
satisfaction)
o approaches to handling these dependencies in the analysis (see Chapter 24 in Borenstein et
al., 2009; Cheung, 2014; Hedges et al., 2010; Schmidt & Hunter, 2015; Tanner-Smith & Tipton,
2014)
 Aggregation approach: average the dependent effect sizes into a single effect size when
multiple effect sizes represent the same construct (procedures see e.g., S. F. Cheung &
Chan, 2004)
 Elimination approach: select only one effect size per study
 Shifting the unit of analysis (Cooper, 2010):

Step 7. Analysis
 Outlier analysis
o Aim: ensure the robustness of the meta-analytical findings.
o should explicitly report and justify how outliers were identified and addressed
o Ways to identify outliers (Aguinis et al., 2013; Viechtbauer & Cheung, 2010)
 sample-adjusted meta-analytic deviancy statistic (Beal, Corey, & Dunlap, 2002), and
 analogs to ordinary regression diagnostics (Viechtbauer & Cheung, 2010).
o Ways to deal with outliers
  the leave- one-out analyses (Viechtbauer, 2010), or
 removing outliers in groups (Schmidt & Hunter, 2015)
o Whether to keep or omit the outliers depends on:
 Outliers can be a valid representation: Representing a different population,
measure, construct, etc.
 Provide the basis to infer potential moderators (Aguinis et al., 2013)
 Outliers can indicate invalid research (e.g., construct overlap), invalid measures,
or typing errors.
 Advisable step: compare the results both with and without outliers and base the
decision on whether to exclude outlier observations.
 Consider both size and leverage (Viechtbauer & Cheung, 2010) of the
outlier
 Publication bias
o Methods to examine publication bias:
 Funnel plot (Stanley & Doucouliagos 2010): can help detect publication bias and outlier;
 Statistical procedures:
 subgroup comparisons between published and unpublished studies
 rank correlation test (Begg & Mazumdar, 1994)
 cumulative meta-analysis (Borenstein et al. 2009)
 the trim and fill method (Duval and Tweedie 2000a, b)
 Egger et al.’s (1997) regression test
 fail- safe N (Rosenthal 1979)
 selection models (Hedges & Vevea 2005; Vevea & Woods 2005).
 Recommend “triangulation approach”: use multiple tests to examine it (Harrison et al.,
2017; Kepes et al., 2012)
 Availability bias
o i.e., a subset of effect size is not accessible for reasons other than publication bias (e.g.,
studies published in other languages)
 o There are several different approaches to detecting availability bias (Rothstein et al., 2005),
and the methods continue to evolve (Banks et al., 2012; O’Boyle et al., 2014)
 drift analysis (Banks et al., 2012)
 funnel plots (Peters et al., 2008)
 trim-and-fill analysis (Duval & Tweedie, 2000)
o should report and justify the methods used for any analyses conducted; consider evidence
from multiple indices of availability bias (DeSimone et al., 2021)
 Psychometric adjustments (corrections) for statistical artifacts
o i.e., effect size estimates may be adjusted to account for the influence of various artifacts
o Common psychometric adjustments:
 Sampling error (which is addressed by weighting primary study effect sizes)
 attenuation due to measurement error (i.e., imperfect measurement reliability)
 report information about primary study reliability estimates and type(s) of
reliability used
 reliability estimates (e.g., alpha, retest, interrater) are not interchangeable
(DeSimone et al., 2021)
 range restriction
 ?
o should report information about each adjustment:
 which adjustments were used (e.g., predictor reliability, criterion reliability, range
restriction),
 the order in which adjustments were made,
 whether the meta-analysts considered potential dependencies between artifacts (Ko ̈hler
et al., 2015),
 whether adjustments were applied at the local level (i.e., to individual primary studies;
see Viswesvaran et al., 1996) or the global level (i.e., to the summary effect size; see
Oswald & McCloy, 2003).
o Should report both observed (unadjusted) and adjusted effect sizes and variance estimates
  Moderator analysis
 Sensitivity analysis
o it can be applied to decisions (e.g., the treatment of missing data, the identification of outliers)
made at any stage of the meta-analysis (e.g., coding, data, analysis)
o authors can conduct sensitivity analysis and report what the results would look like if different
decisions had been made.
o Open-access tools are available to facilitate the conduct of sensitivity analyses (see Field et al.,
2020)
 Meta-analytic model6
o Random-effects (RE) model
 Assume that the variance in the distribution of observed effects is attributed to
 1) within-study variance (i.e., sampling error), and
 2) between-study variance (i.e., differences of true effect size in the
superpopulation)
 CI about mean effect size is wider than using CE model
 Always the more appropriate choice in organizational science research (Aguinis et al.,
2011)
 Two assumptions must be satisfied (could seen as limitations):
 random-studies assumption: The observed effect sizes have been randomly
sampled from a superpopulation of ture effect sizes
 superpopulation7 normality assumption: The superpopulation of ture effect sizes is
normally distributed
o requre at least 20 primary-level studies (Field, 2015)
o Fixed-effect or common-effect (CE) model
 Assume that the variance in the distribution of observed effects is attributed solely to
 1) within-study variance (i.e., sampling error)

6
The decision of model is based on 1) the goal of meta-analysis and 2) the nature of the primary-level studies included in the
review. Some meta-analysis assessed Q (i.e., variance in true effect sizes). but Q’s significance level should not be used as a
decision-making tool for choosing among RE, CE, and VC models (Aguinis et al., 2011).
7
Superpopulation:
 Appropriate to use when each of the primary-level studies included in a meta-analysis is

functionally identical8
 This condition rarely exists in organizational research, could occur in biological,
medical, and health sciences
 Advantages
 Can be used with as few as 2 studies (Borenstein et al., 2009, p.363)
o Varying coefficient (VC) model
 Advantages
 Do not assume equality of effect sizes across studies
 Do not rely on random-studies and superpopulation normality assumptions
 Availability of CI for assessing the magnitude of interaction effect
 Can be used with as few as 2 studies (Borenstein et al., 2009, p.363)
 Limitations
 Provide inferences only to a subset of the superpopulation
 in organization science, RE and VC models are almost always preferred to the CE model;
RE methods are preferred over the VC methods if the random-studies and
superpopulation normality assumptions can be met.
Step 8. Report results
 Checklist for quantitative meta-analysis reporting standards (DeSimone et al., 2021; Page et al.,
2021)
o Table reporting results should include:
 Mean effect sizes, standard errors, confidence or credibility intervals 9 around summary
effect sizes, the number of observations, study samples included 10, heterogeneity
measures

8
Identitical = include samples of participants from exactly use the sample population, the same researchers, research design,
and measures for IV and DV
9
DeSimone et al. (202) recommend report prediction interval as well.
10
If the meta-analytic sample is rather small, a forest plot should be provided (Hansen et al., 2022).
  Essential elements include (a) the citation, (b) the computed effect size, and (c)
information necessary to calculate the sampling variance of the effect size (e.g., sample
size and related information, such as sample size per group for Cohen’s d or Hedges’s g)
 Including heterogeneous types of effect sizes and statistics (e.g., p values, t tests, odds
ratios) requires additional assumptions and computational procedures
 When effect size transformations are used, reviewers can suggest that authors report the
original effect size (and type) for each primary study, any transformations used, and any
differences between studies that analyzed the focal relationships in different ways
 present this information in a table, appendix, or online supplement to ensure it is
accessible to readers
o Authors can also graphically display effect size distributions (DeSimone et al., 2021) using
formats such as a funnel plot or forest plot (Anzures-Cabrera & Higgins, 2010; Kepes et al.,
2012)
 Forest plots can depict a great deal of information about an effect size distribution,
including summary effect size magnitude, heterogeneity, and moderation.
o Results displayed in the tables and figures must be explained verbally in the results and
discussion sections
o Dicussion section: must include identified heterogeneity and important moderators, future
research directions, and theoretical relevance.
 Checklist for qualitative meta-analysis reporting standards (Levitt et al., 2018)
 Missing data
o Should provide sufficient detail and justification to inform readers about how missing
information was handled
 missing reliability or range restriction values may be estimated using an artifact
distribution (Schmidt & Hunter, 2015)
 missing elements of a correlation matrix may be imputed in meta-analytic structural
equation modeling (Cheung, 2000)
o ensure that meta-analysts handle missingness consistently and appropriately
o primary studies provided incomplete information (Higgins & Thomas, 2019) read later
Reference
Aguinis, H., Gottfredson, R. K., & Joo, H. (2013). Best-practice recommendations for defining, identifying,
and handling outliers [outlier]. Organizational Research Methods, 16(2), 270-301.
Andersson, U., Cuervo-Cazurra, A., & Nielsen, B. B. (2014). From the Editors: Explaining interaction effects
within and across levels of analysis. Journal of International Business Studies, 45(9), 1063-1071.
https://doi.org/10.1057/jibs.2014.50
DeSimone, J. A., Brannick, M. T., O’Boyle, E. H., & Ryu, J. W. (2021). Recommendations for reviewing meta-
analyses in organizational research. Organizational Research Methods, 24(4), 694-717.
Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic
reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other
resources. Research Synthesis Methods, 11(2), 181-217.
Hansen, C., Steinmetz, H., & Block, J. (2022). How to conduct a meta-analysis in eight steps: a practical
guide. Management Review Quarterly, 72(1), 1-19. https://doi.org/10.1007/s11301-021-00247-4
Harrison, J. S., Banks, G. C., Pollack, J. M., O’Boyle, E. H., & Short, J. (2017). Publication bias in strategic
management research. Journal of Management, 43(2), 400-425.
Hedges, L. V., Higgins, J. P. T., Rothstein, H. R., & Borenstein, M. (2009). Introduction to meta-analysis.
Chichester, UK.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research
findings (2nd ed.) [key]. Sage.
Kepes, S., Banks, G. C., Mcdaniel, M., & Whetzel, D. L. (2012). Publication Bias in the Organizational
Sciences. Organizational Research Methods, 15(4), 624-662.
https://doi.org/10.1177/1094428112452760
Levitt, H. M., Bamberg, M., Creswell, J. W., Frost, D. M., Josselson, R., & Suárez-Orozco, C. (2018). Journal
article reporting standards for qualitative primary, qualitative meta-analytic, and mixed methods
research in psychology: The APA Publications and Communications Board task force report [qua].
American Psychologist, 73(1), 26.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. SAGE publications, Inc.
Marcus, J., & Le, H. (2013). Interactive effects of levels of individualism–collectivism on cooperation: A
meta‐analysis. Journal of Organizational Behavior, 34(6), 813-834. https://doi.org/10.1002/job.1875
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & Prisma Group*, t. (2009). Preferred reporting items for
systematic reviews and meta-analyses: the PRISMA statement [PRISMA]. Annals of internal medicine,
151(4), 264-269.
Murphy, K. R. R., Craig J. (2017). Mend It or End It: Redirecting the Search for Interactions in the
Organizational Sciences. Organizational Research Methods, 20(4), 549-573.
https://doi.org/10.1177/1094428115625322
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff,
J. M., Akl, E. A., & Brennan, S. E. (2021). The PRISMA 2020 statement: an updated guideline for
reporting systematic reviews [PRISMA]. BMJ, 372.
Steel, P. D., & Kammeyer-Mueller, J. D. (2002). Comparing meta-analytic moderator estimation techniques
under realistic conditions. Journal of Applied Psychology, 87(1), 96.
Viechtbauer, W., & Cheung, M. W. L. (2010). Outlier and influence diagnostics for meta‐analysis [outlier].
Research Synthesis Methods, 1(2), 112-125.
Wood, J. A. (2008). Methodology for dealing with duplicate study effects in a meta-analysis. Organizational
Research Methods, 11(1), 79-95.