Medical Teacher, Vol. 27, No. 1, 2005, pp.

71–75

Appraising qualitative research articles in

medicine and medical education

LUC CÔTÉ & JEAN TURGEON

Department of Family Medicine, Faculty of Medicine, Laval University, Quebec, Canada

SUMMARY Although qualitative research is gaining recognition quantitative research. However, the application of appraisal
in medicine and in medical education, most clinical teachers do not criteria in qualitative research is somewhat different because
Med Teach Downloaded from informahealthcare.com by University of Zuerich Zentrum fuer Zahn Mund und on 08/14/13

know how to perform a critical appraisal of articles in these fields. it has its own assumptions and end purposes.
This article describes a grid for the critical appraisal of qualitative
research articles so that clinical teachers are in a better position to
evaluate this type of research and to teach the critical appraisal Introduction
of it. The grid is comprised of 12 items with an explanation and Item 1: The issue is described clearly and corresponds to the current
suggestions for additional reading for each item. state of knowledge (Morse & Field, 1995). The introduction
describes the issue being studied and states the research
Introduction question or objective. Using reasoning that is logical,
progressive and understandable, the author demonstrates
Recently, we emphasized the need for more qualitative the importance of the subject and the relevance of his choice
research, especially because it looks at the complex health of the qualitative approach. The issue must be placed in the
issues that physicians often face (Turgeon & Côté, 2000). context of the current state of knowledge; through his survey
The purpose of this article is to present a new grid designed of the literature, the author sheds new light on the issue by
to help clinical teachers better understand this research taking stock of what other researchers have written (theories)
For personal use only.

approach and, especially, to help them appraise articles or done (previous studies) in relation to this issue or similar
in medicine and in medical education. It will also be useful issues. Qualitative research studies are often exploratory, that
to practitioners who are teaching the critical appraisal is to say, their purpose is to provide a better understanding
of scientific publications in their respective settings. Lastly, of ill-defined subjects or even to examine from a new
it may serve as a guide for those who are involved in perspective a problem for which there is as yet no satisfactory
qualitative research protocols and articles. response.

Why a new grid? Item 2: The research question and objectives are clearly stated and
are relevant to qualitative research (Griffiths, 1996). It goes
By searching the literature, we located guidelines for authors without saying that the research question (and objectives)
(Rowan & Huston, 1997; Reid & Leduc, 1998), and must be clearly formulated and flow logically from the issue.
appraisal grids (Cobb & Hagemaster, 1987; Polit & As a general rule, qualitative studies strive to describe
Hungler, 1995; Rowan & Huston, 1997; Devers, 1999; in greater depth the how and why of phenomena, whereas
Seale, 1999; Giacomini & Cook, 2000a; Giacomini & Cook, quantitative studies are used to measure a phenomenon for
2000b; Mays & Pope, 2000). However, the accompanying generalizing the results or testing a hypothesis. Verbs such as
notes generally referred to more than one item at a time and explore, describe and understand are often used in describing
are not oriented to medical education. This grid: the objective of a qualitative research study. According to the
 is short and fairly simple for clinical teachers, residents research question, a number of topics may be relevant to
and medical students to use; qualitative research studies: the relational processes
 is written in qualitative language that is as easy as possible associated with care (e.g. the doctor–patient relationship)
for individuals who are not experts in qualitative research and the decision-making processes; an understanding of
to understand; specific health issues (e.g. non-compliance with treatment), a
 follows the usual structure of a research article in a medical medical education issue (e.g. the process by which students
journal. choose role models, the career paths of clinical teachers, and
the process of learning the doctor–patient relationship).
For each grid item explanations, clinical or pedagogical
examples, and suggestions for additional reading are given.
Methods

Presentation and explanation of the grid While the issue reflects the conceptual phase of the study, the
methods refer to the operational phase, i.e. the way in which
The grid is comprised of 12 items, each of which is divided
into five sections: introduction, methods, results, discussion,
Correspondence: Luc Côté, Office of Medical Education, Faculty of Medicine,
and conclusion (Figure 1). The reader will note that the Vandry Building, Office 3358-A, Laval University, Ste-Foy, Quebec, Canada,
general appraisal criteria are basically the same as for G1K 7P4; email: luc.cote@fmed.ulaval.ca

ISSN 0142–159X print/ISSN 1466–187X online/00/000071-75 ß 2005 Taylor & Francis Group Ltd 71
DOI: 10.1080/01421590400016308
 L. Côte´ & J. Turgeon

Yes ± No

Introduction

1. The issue is described clearly and corresponds to the current state of knowledge.
2. The research question and objectives are clearly stated and are relevant to
qualitative research (e.g., the process of clinical or pedagogical decision-making).

Methods

3. The context of the study and the researchers’ roles are clearly described (e.g. setting in
which the study takes place, bias).
4. The method is appropriate for the research question (e.g. phenomenology, grounded
theory, ethnography).
5. The selection of participants is appropriate to the research question and to the method
Med Teach Downloaded from informahealthcare.com by University of Zuerich Zentrum fuer Zahn Mund und on 08/14/13

selected (e.g. key participants, deviant cases).

6. The process for collecting data is clear and relevant (e.g. interview, focus group, data
saturation).
7. Data analysis is credible (e.g. triangulation, member checking).

Results

8. The main results are presented clearly.

9. The quotations make it easier to understand the results.

Discussion

10. The results are interpreted in credible and innovative ways.

11. The limitations of the study are presented (e.g. transferability).
For personal use only.

Conclusion

12. The conclusion presents a synthesis of the study and proposes avenues for further
research.

Figure 1. Grid for the critical appraisal of qualitative research articles in medicine and medical education.

the researcher answered his research question. In any departure of influential persons) and all other information
research article, we expect the author to describe the methods needed to understand the phenomenon (e.g. any major
that he used clearly and precisely and to demonstrate their organizational conflicts that emerged over the course
ability to answer the research question. Even though the of the study).
researcher must choose a methodology based on the purpose Another aspect of qualitative research is the role that it
of the study and the time and resources at his/her disposal, gives the researcher, i.e. the proximity that he/she has to the
realism and rigour must always inform this process. setting and the study participants. Qualitative theory
In the section on methods, five things must be considered: questions the notion of objectivity, arguing instead that
the context of the study and the role of the researchers, the researchers have their own reasons for choosing one type
method chosen, the selection of participants, data collection, of research over another. Thus, no research situation is ever
and data analysis. completely ‘neutral’ and it is always interpreted by the
researcher through the lens of his/her own perception of the
Item 3: The context of the study and the researchers’ roles are results, whether or not they have been quantified. Unlike
clearly described (Devers, 1999). Qualitative research makes quantitative research, in which the researcher ‘distances’
it possible to study complex phenomena in their natural him/herself from the subjects so as not to introduce bias into
context. In fact, an in-depth understanding of phenomena data collection, qualitative researchers often work ‘with’ their
cannot be achieved outside the context in which they occur. participants. For example, a qualitative researcher studying
Even though some contexts appear to be similar (e.g. the interdisciplinary collaboration in an emergency room would
faculty of medicine at University X and at University Y), be more likely to go to the setting to observe the professionals
each context is still unique, as are the experiences of the at work and to meet with them in individual or group
individuals involved. The context must be clearly and interviews.
adequately described so that the reader is able to properly In a qualitative research article, the researcher–setting
understand the phenomenon being studied. The reader must relationship needs to be described, given that they will
be provided with meaningful information concerning the inevitably influence each other. This will provide the reader
characteristics of the setting and the individuals involved with the information he/she needs to determine whether the
(e.g. the types of educational approaches used, the arrival or researcher has a good understanding of the setting and if

72
 Appraising qualitative research articles

he/she is sufficiently involved to truly understand the study be noted that, in research using the grounded theory method,
phenomenon. It is also essential for determining whether his/ situations, individuals or groups are chosen in successive
her role could influence the collection and analysis of data. stages based on the information that the researcher is looking
Each role involves potential bias. for (Strauss & Corbin, 1998).

Item 4: The method is appropriate for the research question Item 6: The process for collecting data is clear and relevant
(Morse, 1994). Another criterion of scientific robustness is (Britten, 1995; Brown, 1999). In addition to knowing
the appropriateness of the method chosen to answer the specifically how the data were collected, the reader must be
research question. While many qualitative research methods able to determine whether the process used by the researcher
exist, the most commonly used methods in the field of health to collect these data was adequate and realistic. The
are phenomenology, grounded theory and ethnography. researcher must, therefore, describe this process in sufficient
The boundaries between these methods are often blurred detail. In qualitative research, two of the main data collection
and they are often combined; consequently, the researcher techniques are individual interviews (semi-structured and
must be very careful to avoid any confusion. unstructured) and group interviews such as focus groups.
Med Teach Downloaded from informahealthcare.com by University of Zuerich Zentrum fuer Zahn Mund und on 08/14/13

If the purpose of the study is to understand the meaning Depending on the study goals, some techniques are more
or significance of a phenomenon based on the experiences appropriate than others. For example, if the purpose of the
of those living with it (e.g. life after myocardial infarction, study is to understand why hypertensive patients do not
being a PBL tutor), phenomenology is the method of choice. follow their doctor’s recommendations, semi-structured
Phenomenology focuses on the experience of one or individual interviews would most probably be preferable to
more individuals, documented over the course of individual focus groups since they allow individuals, especially the shyer
interviews. ones, to express themselves more easily. On the other hand,
The grounded theory method begins with no theory if the purpose of the study is to better understand why
or concept. Data are collected first and then conceptual physicians chose a certain type of practice, focus groups
categories or theory arise from the data. This method is often might be appropriate since the group synergy and interaction
used to study a process (e.g. student’s choice of learning might generate information that would be difficult to collect
strategies) or to analyse interactions (e.g. clinical supervision through individual interviews. If the purpose of the study is to
during residency). The theory is usually generated through understand how teachers actively involve students in their
individual or group interviews. learning activities, an observation technique might be used
For personal use only.

Lastly, the strength of the ethnographic method lies and then rounded out with individual interviews.
in understanding the nature of certain cultural elements of To study certain issues (e.g. the doctor–patient relation-
a group (notions, representations, beliefs) based on the point ship), it is useful to use more than one source of information
of view of members of this group, on observations of how (e.g. physicians and patients). This is a form of data
they function or an analysis of various types of relevant triangulation. It is also possible to use more than one
documents. An educational researcher might be interested in technique for collecting data (e.g. individual interviews with
how surgeons describe good clinical instruction in the physicians and patients and direct observation of medical
operating room. consultations) or more than one method of data analysis.
This is a form of method triangulation (Begley, 1996). Lastly,
Item 5: The selection of participants is appropriate to the research data collection must be conducive to data saturation, i.e. to the
question and to the method selected (Sandelowski, 1995). In this greatest possible understanding of the study topic and
section of the research article, the researcher convinces to the researcher’s conviction that adding more participants
the reader that, to the extent possible, he/she selected the would not add any new data that would contribute to this
participants who are the most likely to assist in answering his/ understanding.
her research question. Often, relatively restricted samples
are painstakingly selected, based on the study goals and the Item 7: Data analysis is credible (Miller & Crabtree, 1994;
methodology used. The size of the sample is not as important Devers, 1999). In qualitative research, data collection and
as its quality. Participant selection must allow for diversifica- analysis is often an iterative process in which the researcher
tion of the data, i.e. it must capture various points of view goes back and forth from one step to the other so that they
or representations of the study phenomenon. enrich each other.
In the field of health, the researcher will usually select In order for data analysis to be credible, the researcher
individuals or groups as his/her unit of analysis, although must demonstrate that it is precise, consistent and exhaus-
he/she may also work from various types of documents (e.g. tive. The analytical process must be described with enough
interview videos, files, etc.). The most common technique for detail to enable the reader to determine whether the process
selecting participants (Patton, 1990a) is the theoretical is credible. As indicated above, credibility refers to the overall
(or intentional) sample based on the researcher’s judgement methodological quality of the study. In qualitative research,
in choosing individuals that he/she feels will be relevant this refers to internal validity.
because of their characteristics (wealth of opinions on, or Even though there are several strategies for analysing
experience with, the subject) and the goals of the study. qualitative data (Miles & Huberman, 1994), they usually
These individuals become key participants. The inclusion of refer to inductive inference of thematic content, i.e. the
extreme or deviant cases, i.e. individuals who stand out process of creating categories based on an analysis of
because of a particular experience or viewpoint (e.g. teachers statements by participants. In practice, this type of analysis
who are either very satisfied or very unsatisfied with their involves the following steps: transcribing the interview
work) is very useful, even essential, to some studies. It should material in full, reading the material several times, selecting

73
 L. Côte´ & J. Turgeon

the units of significance or units of meaning, identifying Lastly, in order for the discussion to be credible, the
general themes, and categorizing and classifying the data. researcher must discuss all of the relevant results, including
For the grounded theory method, this method includes results that were unexpected and results that did not
a specific step of codification and analysis, i.e. constant correspond to the main explanations of the phenomenon
comparison analysis (Strauss & Corbin, 1998). For ethno- being studied (i.e. negative or contradictory results).
graphic and phenomenological studies, the researcher
must describe how the notes taken during field observations Item 11: The limitations of the study are presented (Morse,
were processed. 1999). A good researcher neither under- nor overestimates
Researchers can also analyse their data by combining the strength of his/her study. Often, due to logistical and time
qualitative data from content analysis and quantitative constraints, the study has limitations (for example, in terms
data, for example, the frequency of various statements. of the selection of participants). The researcher must mention
This combination can be useful because it combines two these without over-or underemphasizing them.
perspectives. Clarity, precision, and relevance are criteria In qualitative research, the researcher must discuss the
used to evaluate these types of analyses. A word of caution: transferability of the study results (external validity).
Med Teach Downloaded from informahealthcare.com by University of Zuerich Zentrum fuer Zahn Mund und on 08/14/13

numerical data do not comprise a criterion of robustness. This refers to the concept of the generalization found in
It all depends on the study question! quantitative research. In fact, one of the main criticisms of
Regardless of the analytical procedure used, credibility qualitative research is that it produces results that cannot
is enhanced if the data are analysed by more than one be generalized. In response to this criticism, qualitative
researcher (researcher triangulation). In addition, many researchers argue that their participants and contexts are
authors recommend submitting the analyses to participants painstakingly selected for the very purpose of data diversifica-
for their feedback (member checking). tion and saturation. In-depth analysis of one or more
aspects of an issue from several points of view produces
new knowledge (processes, concepts, and theories). As in
Results
quantitative research, it is this new knowledge that is
Item 8: The main results are presented clearly (Mays & Pope, generalizable. The reader must first ask him/herself whether
1995). Usually, the amount of data generated from the study issue and the interpretation of data are applicable to
a qualitative study is considerable. Often, several hundred his/her own context or to contexts that he/she feels are
pages of interviews and notes taken by the researcher similar.
For personal use only.

are analysed. The researcher must present, in a way

that is understandable to the reader, the results that
Conclusion
he/she feels are most relevant, theoretically and
practically, to his/her research question. Where there are Item 12: The conclusion presents a synthesis of the study and
illustrations or tables, they must be clear and contribute to proposes avenues for further research (Bordage, 1989; Devers,
the reader’s understanding of the themes, categories or 1999). In this section, the researcher presents the key
concepts. messages of his/her study. This is not simply a list of the
various elements of the study, which would be a summary.
Item 9: The quotations make it easier to understand the results The researcher also discusses the benefits of his/her study by
(Morse, 1994). When qualitative research results are proposing new avenues for research or describing the study’s
presented, it is recommended that several participants be implications for clinical practice, for example.
quoted. A reasonable number of short, clear quotations
make the results easier to understand and more credible.
Conclusion
Usually, the researcher presents the point(s) of view most
likely to help the reader understand the study results and We hope that this grid will provide clinicians and teachers
those that support the emergence of a new concept. with a better understanding of qualitative research and its
unique criteria for robustness. However, a cautionary note is
in order. A grid is one way of systematically organizing
Discussion
information but it does not provide a detailed description of
Item 10: The results are interpreted in credible and innovative ways the epistemological values or challenges of qualitative
(Patton, 1990b; Frankel, 1999; Grbich, 1999). Interpreting research. Remember that we cannot evaluate qualitative
the results means explaining them and giving them meaning research in the same manner as quantitative research, nor
in terms of the purpose of the study and, if applicable, within should we.
the theoretical framework used. In addition to proposing
plausible interpretations, i.e. interpretations that demonstrate
some consistency between the results and the meaning that Practice points
he/she has attributed to them, the researcher should add to
our knowledge of the subject through new theoretical or  Qualitative research is relevant to the study of various
practical interpretations. For example, the researcher might issues in medicine and medical education.
come up with an explanation not offered in previous studies  Its credibility is based on the clarity and rigour of the
or he/she might make recommendations for implementing research process.
the results. The researcher must make the connection  Using specific criteria, it is possible to appraise the
between the study results and the current state of knowledge value of each part of a qualitative research article.
to emphasize the innovative nature of his/her interpretations.

74
 Appraising qualitative research articles

Note of the panel discussion at the conference ‘Exploring qualitative research

in general practice’, Family Practice, 13(Suppl. 1), pp. S27–S30.
A previous version of this paper was published in French in Pédagogie MAYS, N. & POPE, C. (1995) Rigour and qualitative research, British
Médicale (2002, 3, pp. 81–90). Medical Journal, 311, pp. 109–112.
MAYS, N. & POPE, C. (2000) Quality in qualitative health research,
In: C. Pope & N. Mays. Qualitative Research in Health Care (London,
British Medical Journal Books).
References
MILES, M.B. & HUBERMAN, A.M. (1994) Qualitative data analysis
BEGLEY, C.M. (1996) Triangulation of communication skills in (Thousand Oaks CA, Sage Publications).
qualitative research instruments, Journal of Advanced Nursing, 24, MILLER, W.L. & CRABTREE, B.F. (1994) Qualitative analysis: how to begin
pp. 688–693. making sense, Family Practice Research Journal, 14, pp. 289–297.
BORDAGE, G. (1989) Considerations on preparing a paper for publication, MORSE, J.M. (1994) Designing funded qualitative research, In:
Teaching and Learning in Medicine, 1, pp. 47–52. N.K. Denzin & J.M. Morse. Disseminating Research/Changing Practice,
BRITTEN, N. (1995) Qualitative interviews in medical research, British pp. 59–75 (Thousand Oaks, CA, Sage Publications).
Medical Journal, 311, pp. 251–253. MORSE, J.M. & FIELD, P.A. (1995) Principles of conceptualizing a
BROWN, J.B. (1999) The use of focus groups in clinical research, qualitative project, In: Qualitative Research Methods for Health
Med Teach Downloaded from informahealthcare.com by University of Zuerich Zentrum fuer Zahn Mund und on 08/14/13

In: B.F. Crabtree & W.L. Miller (Eds) Doing Qualitative Research (2nd Professionals 2nd edn (Thousand Oaks, CA, Sage Publications).
edn) pp. 109–124 (Thousand Oaks, CA, Sage Publications). MORSE, J.M. (1999) Qualitative generalisability, Quality Health Research,
COBB, A.K. & HAGEMASTER, J.N. (1987) Ten criteria for evaluat- 9, pp. 5–6.
ing qualitative research proposals, Journal of Nursing Education, 26, PATTON, M.Q. (1990a) Qualitative Evaluation and Research Methods
pp. 138–143. (Newbury Park, CA, Sage Publications).
DEVERS, K.J. (1999) Part II: How will we know ‘good’ qualitative research PATTON, M.Q. (1990b) Qualitative analysis and interpretation,
when we see it? Beginning the dialogue in health services research, In: Qualitative Evaluation and Research Methods. pp. 428–436
Health Service Research, 34, pp. 1153–1188. (Newbury Park, CA, Sage Publications).
FRANKEL, R.M. (1999) Standards of qualitative research, In: B.F Crabtree POLIT, D.E. & HUNGLER, B.P. (1995) Evaluating research reports,
& W.L. Miller (Eds) Doing Qualitative Research, pp. 333–346 In: Nursing research. Principles and Method, pp. 574–588 (Philadelphia,
(Thousand Oaks, CA, Sage Publications). J.B. Lippincott).
GIACOMINI, M.K. & COOK, D.J. (2000a) Are the results of the study valid? REID, A.J. & LEDUC, C.P. (1998) What we want, Canadian Family
Users’ guide to the medical literature, XXIII: Qualitative research Physician, 44, pp. 1063–1070.
in health care, Journal of the American Medical Association, 284, ROWAN, M. & HUSTON, P. (1997) Qualitative research articles: informa-
pp. 357–362. tion for authors and peer reviewers, Canadian Medical Association
GIACOMINI, M.K. & COOK, D.J. (2000b) What are the results and how Journal, 157, pp. 1442–1446.
For personal use only.

do they help me care for my patients? Users’ guide to the medical SANDELOWSKI, M. (1995) Sample size in qualitative research, Research in
literature, XXIII: Qualitative research in health care, Journal of the Nursing and Health, 18, pp. 179–183.
American Medical Association, 284, pp. 478–482. SEALE, C. (1999) The Quality of Qualitative Research, pp. 189–192
GRBICH, C. (1999) Qualitative Research in Health. An Introduction (London, Sage Publications).
(London, Sage Publications). STRAUSS, A. & CORBIN, J. (1998) Basics of Qualitative Research (Thousand
GRIFFITHS, F. (1996) Qualitative research: the research questions it can Oaks, CA, Sage Publications).
help answer, the methods it uses, the assumptions behind the research TURGEON, J. & CÔTÉ, L. (2000) Qualitative research in family medicine, an
questions and what influences the direction of research. A summary inevitable development, Canadian Family Physician, 46, pp. 2171–2172.

75