Guide to an Overall Critique of a QuaNtitative Research Report

ASPECT OF THE CRITIQUING QUESTIONS

REPORT
Title  Is the title a good one, succinctly suggesting key variables and the
study population?
Abstract  Does the abstract clearly and concisely summarize the main
features of the report (problem, methods, results, conclusions)?
Introduction
Statement of the  Is the problem stated unambiguously, and is it easy to identify?
problem  Does the problem statement build a cogent and persuasive
argument for the new study?
 Does the problem have significance for nursing?
 Is there a good match between the research problem and the
paradigm and methods used? Is a quantitative approach appropriate?
Hypotheses or  Are research questions and/or hypotheses explicitly stated? If not,
research questions is their absence justified?
 Are questions and hypotheses appropriately worded, with clear
specification of key variables and the study population?
 Are the questions/hypotheses consistent with the literature review
and the conceptual framework?
Literature review  Is the literature review up-to-date and based mainly on primary
sources?
 Does the review provide a state-of-the-art synthesis of evidence on
the research problem?
 Does the literature review provide a solid basis for the new study?
Conceptual/theoretical  Are key concepts adequately defined conceptually?
framework  Is there a conceptual/theoretical framework, rationale, and/or map,
and (if so) is it appropriate? If not, is the absence of one justified?
Method  Were appropriate procedures used to safeguard the rights of study
Protection of participants? Was the study subject to external review by an
participants’ rights institutional review board/ethics review board?
  Was the study designed to minimize risks and maximize benefits to
participants?
Research design  Was the most rigorous possible design used, given the purpose of
the research?
 Were appropriate comparisons made to enhance interpretability of
the findings?
 Was the number of data collection points appropriate?
 Did the design minimize biases and threats to the internal,
construct, and external validity of the study (e.g., was blinding used,
was attrition minimized)?
Population and  Was the population identified and described? Was the sample
sample described in sufficient detail?
 Was the best possible sampling design used to enhance the
sample’s representativeness? Were sample biases minimized?
 Was the sample size adequate? Was a power analysis used to
estimate sample size needs?
Data collection and  Are the operational and conceptual definitions congruent?
measurement  Were key variables operationalized using the best possible method
(e.g., interviews, observations, and so on) and with adequate
justification?
 Are the specific instruments adequately described and were they
good choices, given the study purpose and study population?
 Does the report provide evidence that the data collection methods
yielded data that were high on reliability and validity?
Procedures  If there was an intervention, is it adequately described, and was it
properly implemented? Did most participants allocated to the
intervention group actually receive it? Was there evidence of
intervention fidelity?
 Were data collected in a manner that minimized bias? Were the
staff who collected data appropriately trained?
Results  Were analyses undertaken to address each research question or test
Data analysis each hypothesis?
 Were appropriate statistical methods used, given the level of
measurement of the variables, number of groups being compared, and
so on?
 Was the most powerful analytic method used? (e.g., did the
analysis help to control for confounding variables)?
 Were Type I and Type II errors avoided or minimized?
Findings Was information about statistical significance presented? Was
information about effect size and precision of estimates (confidence
intervals) presented?
 Are the findings adequately summarized, with good use of tables
and figures?
Are findings reported in a manner that facilitates a meta-analysis,
and with sufficient information needed for EBP?
Discussion  Are all major findings interpreted and discussed within the context
Interpretation of the of prior research and/or the study’s conceptual framework?
findings Were causal inferences, if any, justified?
 Are the interpretations consistent with the results and with the
study’s limitations?
 Does the report address the issue of the generalizability of the
findings?
Implications/  Do the researchers discuss the implications of the study for clinical
recommendations practice or further research—and are those implications reasonable
and complete?
Global Issues  Is the report well written, well organized, and sufficiently detailed
Presentation for critical analysis?
In intervention studies, was a CONSORT flow chart provided to
show the flow of participants in the study?
 Was the report written in a manner that makes the findings
accessible to practicing nurses?
Researcher credibility  Do the researchers’ clinical, substantive, or methodologic
qualifications and experience enhance confidence in the findings and
their interpretation?
Summary assessment  Despite any identified limitations, do the study findings appear to
be valid—do you have confidence in the truth value of the results?
 Does the study contribute any meaningful evidence that can be
used in nursing practice or that is useful to the nursing discipline?
Guide to an Overall Critique of a QuaLitative Research Report

ASPECT OF THE REPORT CRITIQUING QUESTIONS

Title  Was the title a good one, suggesting the key phenomenon
and the group or community under study?
Abstract  Does the abstract clearly and concisely summarize the
main features of the report?
Introduction  Is the problem stated unambiguously and is it easy to
Statement of the problem identify?
 Does the problem statement build a cogent and persuasive
argument for the new study?
 Does the problem have significance for nursing?
 Is there a good match between the research problem on the
one hand and the paradigm, tradition, and methods on the
other?
Research questions  Are research questions explicitly stated? If not, is their
absence justified?
 Are the questions consistent with the study’s philosophical
basis, underlying tradition, conceptual framework, or
ideological orientation?
Literature review  Does the report adequately summarize the existing body of
knowledge related to the problem or phenomenon of interest?
 Does the literature review provide a solid basis for the new
study?
Conceptual underpinnings  Are key concepts adequately defined conceptually?
 Is the philosophical basis, underlying tradition, conceptual
framework, or ideological orientation made explicit and is it
appropriate for the problem?
Method  Were appropriate procedures used to safeguard the rights
Protection of participants’ of study participants? Was the study subject to external
rights review by an IRB/ethics review board?
  Was the study designed to minimize risks and maximize
benefits to participants?
Research design and research  Is the identified research tradition (if any) congruent with
tradition the methods used to collect and analyze data?
 Was an adequate amount of time spent in the field or with
study participants?
 Did the design unfold in the field, giving researchers
opportunities to capitalize on early understandings?
 Was there evidence of reflexivity in the design?
 Was there an adequate number of contacts with study
participants?
Sample and setting  Was the group or population of interest adequately
described? Were the setting and sample described in
sufficient detail?
 Was the approach used to gain access to the site or to
recruit participants appropriate?
 Was the best possible method of sampling used to enhance
information richness and address the needs of the study?
 Was the sample size adequate? Was saturation achieved?
Data collection  Were the methods of gathering data appropriate? Were
data gathered through two or more methods to achieve
triangulation?
 Did the researcher ask the right questions or make the right
observations, and were they recorded in an appropriate
fashion?
 Was a sufficient amount of data gathered? Were the data
of sufficient depth and richness?
Procedures  Were data collection and recording procedures adequately
described and do they appear appropriate?
  Were data collected in a manner that minimized bias or
behavioral distortions? Were the staff who collected data
appropriately trained?
Enhancement of  Did the researchers use strategies to enhance the
trustworthiness trustworthiness/integrity of the study, and was the description
of those strategies adequate?
 Were the methods used to enhance trustworthiness
appropriate and sufficient?
 Did the researcher document research procedures and
decision processes sufficiently that findings are auditable and
confirmable?
 Is there evidence of researcher reflexivity?
Results  Were the data management and data analysis methods
Data analysis sufficiently described?
 Was the data analysis strategy compatible with the
research tradition and with the nature and type of data
gathered?
 Did the analysis yield an appropriate “product” (e.g., a
theory, taxonomy, thematic pattern, etc.)?
 Did the analytic procedures suggest the possibility of
biases?
Findings  Were the findings effectively summarized, with good use
of excerpts and supporting arguments?
 Do the themes adequately capture the meaning of the data?
Does it appear that the researcher satisfactorily
conceptualized the themes or patterns in the data?
 Did the analysis yield an insightful, provocative, authentic,
and meaningful picture of the phenomenon under
investigation?
Theoretical integration  Are the themes or patterns logically connected to each
other to form a convincing and integrated whole?
  Were figures, maps, or models used effectively to
summarize conceptualizations?
 If a conceptual framework or ideological orientation
guided the study, are the themes or patterns linked to it in a
cogent manner?
Discussion  Are the findings interpreted within an appropriate social or
Interpretation of the findings cultural context?
 Are major findings interpreted and discussed within the
context of prior studies?
 Are the interpretations consistent with the study’s
limitations?
 Does the report support transferability of the findings?
Implications/recommendations  Do the researchers discuss the implications of the study for
clinical practice or further inquiry—and are those
implications reasonable and complete?
Global Issues  Was the report well written, well organized, and
Presentation sufficiently detailed for critical analysis?
 Was the description of the methods, findings, and
interpretations sufficiently rich and vivid?
Researcher credibility  Do the researchers’ clinical, substantive, or methodologic
qualifications and experience enhance confidence in the
findings and their interpretation?
Summary assessment  Do the study findings appear to be trustworthy—do you
have confidence in the truth value of the results?
 Does the study contribute any meaningful evidence that
can be used in nursing practice or that is useful to the nursing
discipline?