October 2011 FC/Evaluation/2011-03pr

Qualitative Program Evaluation Methods

J. Mitch Vaterlaus, M.S., Graduate Extension Assistant
Brian J. Higginbotham, Ph.D., Associate Professor and Family Life Extension Specialist

Co-published in The Forum for Family and Consumer Issues, March 2011

Abstract has been defined as the process of “making sense”

Evaluation is an important component of of data gathered from interviews, on-site
refining programs and documenting impacts. observations, documents, etc., then “responsibly
Evaluation aids the profession as a whole and presenting what the data reveal” (Caudle 2004,
assists Extension faculty in meeting promotion 417). The major difference between qualitative and
requirements. Qualitative methods are commonly quantitative approaches lies in their epistemological
used in evaluations in order to explore specific foundations (Bamberger et al. 2006). In other
facets of programs and to give voice to participants’ words, the approaches differ in what constitutes
experiences. These methods provide in-depth knowledge, how knowledge is acquired, and how it
information that can assist Extension faculty in is used. Ragin (1994, 93) explains, “Most
enhancing the quality of their programs. This quantitative data techniques are data condensers.
review highlights differences between quantitative They condense data in order to see the big picture.
and qualitative evaluation methods. The elements, Qualitative methods, by contrast, are best under-
processes, and limitations of qualitative evaluation stood as data enhancers. When data are enhanced, it
methodology are detailed. In addition, specific is possible to see key aspects of cases more clearly.”
guidelines are provided for increasing the The underlying assumptions of qualitative
trustworthiness of qualitative evaluations. methods are closely related to Cooperative
Extension’s mission of understanding and meeting
Introduction people’s needs at the local level (U.S. Department
Extension professionals may not feel they of Agriculture 2010). For Extension administrators
have the time, resources, or expertise for conducting and faculty, qualitative program evaluations can
advanced statistical analyses (Higginbotham, enhance understanding of their participants’
Henderson & Adler-Baeder 2007). There also may experiences (Bamberger et al. 2006). This is done
be concern that quantitative methodologies will not through techniques that give voice and articulate
provide practical and in-depth information often participant perspectives (Bodgan & Biklen 1998).
needed for program improvement. Extension faculty Qualitative analyses are often used in large-scale,
with these concerns should consider the possibilities rigorous, and formal program evaluations.
of qualitative research. However, they can also be used in the pilot studies,
“Qualitative research” is a title that small budget projects, ad hoc, and quick-turnaround
represents a broad family of methods (Bamberger, endeavors that many Extension faculty undertake
Rugh & Mabry 2006; Bodgan & Biklen 1998). It (Bamberger et al. 2006; Caudle 2004).
 This review highlights the following issues lists (Berg 1998). Some things to consider in
for Extension faculty who may be interested in collecting data through interviews include the
using qualitative methods in program evaluation: following:
• The research question • Confidentiality. Just as in other types of
• Qualitative data collection research, participants may expect their
• Qualitative data analysis answers to be confidential. Depending on
• Quality in qualitative evaluation the requirements of the researchers’ Human
• Challenges and considerations in qualitative Subjects Review Board, confidentiality may
evaluation even be required. If data are published, real
names of people should be replaced with
The Research Question pseudonyms (Corbin & Strauss 2008).
Research questions are different in • Interviewing not intervening. Collecting
quantitative and qualitative methodologies (Corbin information from an interview can bring up
& Strauss 2008). Qualitative research questions are sensitive topics. Depending on the state,
used to seek understanding of phenomena that are interviewers may have to follow reporting
not fully developed, where quantitative methods are laws (e.g., abuse reports). It can be helpful
used to test hypotheses. In qualitative research, the to have a list of resources on hand during an
research question leads the evaluator into the data interview in case a request comes up (e.g.,
where the issue can be explored. Qualitative local therapists, women’s shelters, etc.).
research questions are broader than quantitative However, interviewers should remember
research questions but should be specific enough to they are acting in the role of “researcher”
tell the reader what is being investigated. For and not as a “therapist” or “detective.”
example, “What do male participants say about their • Reciprocal process. Interviewing makes the
marital relationships after completing a marriage interviewer an active part of the research
enrichment course?” The question identifies the process (Corbin & Strauss 2008). An
topic (marital relationships), the period in time interviewer should be aware of his or her
(after program completion), and the perspective of biases, paradigms, and belief systems. The
interest (men who participated in a marriage interviewer should not lead participants to
enrichment course). With qualitative research, the desired or preconceived conclusions nor use
perspective of interest can be individuals, families, non-verbal language to reinforce or
groups, or organizations (Corbin & Strauss 2008). discourage certain responses (e.g., nodding,
rolling eyes, etc.).
Qualitative Data Collection • Recording. Audio recording is often used in
Once a research question has been interviewing (Creswell 2007). The audio
formulated then data can be collected from recording can then be transcribed. This
appropriate sources. A particular strength of allows for the inclusion of direct quotes in
qualitative research is the variety of data sources final reports, which can support themes and
that can be used including face-to-face interviews, results from the overall study. Appropriate
phone interviews, focus groups, videos, permission must be granted from the
observation, diaries, or historical documents participants in order to record or videotape
(Corbin and Strauss 2008). Interviews are the interview.
commonly used in qualitative program evaluations • Questions. The questions used in the
(Bamberger et al. 2006). interview should be open-ended questions or
Qualitative interviewing is typically semi- conversational prompts (Kaplan & Saccuzzo
structured. The interviewer has a focus but is also 2009). For example, “Tell me about your
afforded flexibility (Bamberger et al. 2006). In experience participating in this program.”
semi-structured interviews the interviewer generally Open-ended questions keep the interaction
has a list of questions and discussion prompts, but flowing; closed-ended questions halt the
the order in which they are asked can vary in each interview. An example of a closed-ended
interview. The interviewer may ask additional question would be, “Did you like
questions and probe beyond the questions on their participating in this program?”
 • Cultural competence. The language and depending on the type of data that is collected (see
culture of the person being interviewed Berg 1998; Corbin & Strauss 2008; Creswell 2007).
should be taken into consideration
(Bamberger et al. 2006). If possible, Quality in Qualitative Evaluation
participants should be interviewed in their The quality of qualitative research rests on
own language. Careful attention should how the data are gathered and analyzed (Tracy
always be given to interpretations (Caudle 2010). “Trustworthiness” is a common term in
2004). Teaming with a representative of the qualitative research and is closely related to the
culture may assist in making culturally term “validity” in quantitative research (Marshall &
competent translations (Bamberger et al. Rossman 2011). This term refers to the credibility,
2006). transferability, dependability, and objectivity of the
• Sampling. Purposive sampling is often used research (Marshall & Rossman 2011; Schwandt
in qualitative methodology because the 2007). Increasing the trust-worthiness of the study
focus is more on understanding than it is on increases the likelihood that evaluation results will
generalizability (Creswell 2007). Quota warrant publication. A few suggestions for
sampling is one technique that can lessen the increasing trustworthiness include
effects of sampling bias (Bamberger et al. • Triangulation. This concept refers to cross-
2006). For example, five members who checking the data (Shwandt 2007).
attended the entire program and five Triangulation reduces the potential
members who attended only part of the same systematic bias that can occur with using
program could be interviewed. The type of only one data source, method, or procedure
sampling procedure largely depends on the (Maxwell 2008). Triangulation can be done
perspective of interest in the research through the use of multiple data sources
question (e.g., anyone who participated in (e.g., facilitators, participants, and
the program vs. only those who completed observations), multiple methods of data
the program). This procedure can also be collection (e.g., individual interviews, focus
used to gain understanding from different groups, and diaries), multiple data collectors
genders, ethnicities, ages, etc. (e.g., more than one interviewer), multiple
data collection points (e.g., same person
Qualitative Data Analysis interviewed several times over a defined
Generally, qualitative findings are generated time period), multiple theories (e.g., using
through inductive processes—from detailed theories from multiple disciplines), and
information to general themes (Bamberger et al. using a mixed-methods approach (e.g.,
2006). The most common qualitative analytic collaborating with a quantitative researcher
technique is thematic analysis. Thematic analysis on the evaluation) (Bamberger et al. 2006;
involves: Creswell 2007; Tracy 2010).
• Viewing the data several times as a whole • Theory. Theory may emerge from
(e.g., reading and re-reading the qualitative inquiry, although this is generally
manuscripts). not the primary purpose (Bamberger et al.
• Identifying patterns and themes (e.g., 2006). Qualitative results are not generally
finding common statements or ideas that used for confirmation of existing theories,
appear repeatedly). but can provide additional support for them.
• Reorganizing the data (e.g., coding the data Existing theory can be used to guide
according to the themes identified). qualitative research (Malterud 2001).
This type of data analysis requires attention Published qualitative studies often use
to detail and simultaneously being able to consider theoretical frameworks to provide
the data as a whole. Depending on the number and justification for the methodologies that are
length of interviews, this process can be very time used (Corbin & Strauss 2008). Theoretical
consuming. There are several variations of thematic frameworks can also provide explanations
approaches (Bodgan & Biklen 1998). There are also and deeper understanding when interpreting
other analysis techniques that can be used the qualitative results.
 • Validation. This is the process of checking to the population at large, but rather as descriptions,
with participants concerning the accuracy of notions, or theories applicable within a specified
the data and interpretations (Creswell 2007; setting.” The sampling technique and rigor of the
Tracy 2010). It is also called “member data collection influence the scope of the
checking.” Selected representatives of the generalizability or transferability of the findings.
sample are given opportunities to review, The results from qualitative studies provide in-depth
prior to dissemination, copies of the and rich information that can lead to new
transcribed data (manuscripts, with hypotheses, theory, and directions in programming.
confidentiality requirements completed) and Before presenting or submitting an article based on
the results section (e.g., containing the qualitative data, Extension faculty should consider
themes drawn from the interviews). the scope and purpose of the research to make sure
the evaluation will make a meaningful impact on
Challenges and Considerations in the field (Tracy 2010).
Qualitative Evaluation Publishing qualitative results is one way to
Qualitative evaluation does not come contribute to the progression of Extension. The
without challenges. The beginning qualitative trustworthiness of the data is critical because
researcher may feel overwhelmed by the time and academic journals attempt to publish rigorous
expertise required to complete qualitative findings. Some academic journals do not publish
evaluations (Corbin & Strauss 2008). Many of the qualitative research but some journals exclusively
procedures and terminologies used within publish qualitative research (e.g.,
qualitative inquiry are very different than http://qrj.sagepub.com/). The Forum for Family and
quantitative research (Malterud 2001). Consumer Issues and Journal of Extension regularly
As with any evaluation, Extension faculty publish articles that use qualitative methods. Lists
must carefully make a plan to complete the of journals that are receptive to qualitative methods
evaluation in light of their other responsibilities and can be found online (see
time constraints. Organization and documentation is http://www.slu.edu/organizations/qrc/QRjournals.ht
particularly important when working with large data ml). Reviewing qualitative articles from these
sets (e.g., transcripts, recordings, field notes) journals can lead to a greater understanding of
(Bogdan & Biklen 1998; Caudle 2004). Research qualitative procedures and terminologies.
procedures should be documented and accepted best
practices should be followed to ensure quality and Conclusion
trustworthiness. Planning the entire process from Extension faculty are generally required to
the onset can also increase the coherence in the publish articles in order to meet tenure promotion
design and procedures (Maxwell 2009). The plan requirements (Schwab 2003). They are also
should include realistic time frames for conducting expected to provide quality research-based
interviews, transcribing, coding, and writing. programming (U.S. Department of Agriculture
Participants may feel uncomfortable with 2010). It is possible for Extension faculty to
the less-structured nature of qualitative interviews. accomplish both of these purposes through the
Consideration should be given in the procedures to evaluation of their programs. Qualitative evaluation
build rapport and to ensure participants’ may serve as a less intimidating way to contribute
confidentiality. Extension faculty may need to to professional literature and meet promotion
identify areas of qualitative inquiry that they may requirements. It does not require an advanced
need to read more about or seek mentorship from a knowledge of statistics and can be done at a scale
more experienced qualitative researcher. and scope to match each agent’s budget, interests,
When data is collected and analyzed, and need. Furthermore, steps can be taken to insure
researchers should use caution in discussing the quality of the results and to enhance the
implications and generalized findings. The trustworthiness of the process. When done well,
foundational purposes of qualitative research are qualitative research can provide valuable insights
different than quantitative research. Malterud (2001, that can be used to improve programs locally while
486) explained, “The findings from a qualitative also influencing related programming efforts more
study are not thought of as facts that are applicable
broadly (see Higginbotham, Henderson & Adler- Malterud, K. 2001. “Qualitative research:
Baeder 2007). Standards, challenges, and guidelines.” The
Lancet 358(9280): 483-488. doi:
References 10.1016/S0140-6736(01)05627-6
Bamberger, M., J. Rugh, & L. Mabry. 2006. Real Marshall, C., & G. Rossman. 2011. Designing
World Evaluation: Working under budget, qualitative research. Thousand Oaks, CA:
time, data, and political constraints. Sage.
Thousand Oaks, CA: Sage. Maxwell, J.A. 2009. “Designing a qualitative
Berg, B. 1998. Qualitative research methods for the study.” In L. Bickman & D.J. Rog (eds.)
social sciences. Boston: Allyn and Bacon. Applied Social Research Methods. Thousand
Bodgan, R.,& S. Biklen. 2007. Qualitative research Oaks, CA: Sage. 214-253.
for education: An introduction to theories Ragin, C. 1994. Constructing social research.
and methods. Boston: Pearson. Thousand Oaks, CA: Pine Forge Press.
Caudle, S.L. 2004. “Qualitative data analysis,” in Schwab, C. 2003. “Editor’s Corner: The scholarship
J.S. Wholey, H.P. Hatry, & K.E. Newcomer of extension and engagement: What does it
(eds.) Handbook of practical program mean in the promotion and tenure process?”
evaluation. San Francisco: Jossey-Bass. The Forum for Family and Consumer Issues
417-438. 8(2).http://ncsu.edu/ffci/publications/2003/v
Corbin, J., & A. Strauss. 2008. Basics of qualitative 8-n2-2003-may/editors-corner.php
research. Thousand Oaks, CA: Sage. Schwandt, T.A. 2007. “Judging interpretations.”
Creswell, J. W. 2007. Qualitative inquiry and New Directions for Evaluation 114:11-25.
research design: Choosing among five Tracy, S.J. 2010. Qualitative quality: Eight “big-
approaches. Thousand Oaks, CA: Sage. tent” criteria for excellent qualitative
Kaplan, R., & D. Saccuzzo. 2009. Psychological research. Qualitative Inquiry 16:837-851.
testing: Principles, applications, and issues. United States Department of Agriculture. 2010.
Belmont, CA: Wadsworth. Extension.
Higginbotham, B., K. Henderson, & F. Adler- http://www.csrees.usda.gov/qlinks/extension
Baeder. 2007. Using research in marriage .html
and relationship education programming. http://ncsu.edu/ffci/publications/2011/v16-
Forum for Family and Consumer Issues n1-2011-spring/index-v16-n1-Marchr-
12(1).http://ncsu.edu/ffci/publications/2007/ 2011.php
v12-n1-2007-spring/higginbotham/fa-4-
higginbotham.php

Utah State University is committed to providing an environment free from harassment and other forms of illegal discrimination based on race, color,
religion, sex, national origin, age (40 and older), disability, and veteran’s status. USU’s policy also prohibits discrimination on the basis of sexual orientation in
employment and academic related practices and decisions.
Utah State University employees and students cannot, because of race, color, religion, sex, national origin, age, disability, or veteran’s status, refuse to hire;
discharge; promote; demote; terminate; discriminate in compensation; or discriminate regarding terms, privileges, or conditions of employment, against any person
otherwise qualified. Employees and students also cannot discriminate in the classroom, residence halls, or in on/off campus, USU-sponsored events and activities.
This publication is issued in furtherance of Cooperative Extension work, acts of May 8 and June 30, 1914, in cooperation with the U.S. Department of
Agriculture, Noelle E. Cockett, Vice President for Extension and Agriculture, Utah State University.