Survey Research Methods

Jennifer S Albrecht, PhD

Associate Professor
Department of Epidemiology and Public Health
University of Maryland School of Medicine
August 12, 2021
 What is a A method of collecting data from a
sample of individuals in a
survey? standardized way to gain information
and insight into a topic.

amstat.org 2
 Survey = the
methodology, the
whole process
Survey vs.
Questionnaire
Questionnaire = the
instrument (many
might refer to this part
as the survey)

3
 High Quality Surveys
• Topic/research question is clearly defined
• Sample is identified and selected according
to pre-defined criteria
• Data collection process is systematic
• Information is gathered by asking questions
of individuals (questionnaire/interview)
• Results are generalizable to the population
of interest
4
 Purpose of Surveys
• To understand individuals
• To understand households
• To understand social units
• Most are directed to a specific administrative,
commercial, or scientific purpose

5
 Well Known Surveys
• Demographic and Health Surveys
– https://dhsprogram.com/
• National Center for Health Statistics Surveys
and Data Collection Systems
– https://www.cdc.gov/nchs/surveys.htm#tabs-1-1
• Unites States Census Bureau
– https://www.census.gov/
 Designing a Survey
• Questions to ask
1. What
2. Who
3. How
4. When
5. Where

7
 What is your topic
of interest?

WHAT What is your

research question?

What are your

primary variables?
8
Most Common Topics of
Surveys
• Opinions
• Attitudes
• Behaviors
• Facts

9
Example: Topics of Health Surveys
• Environmental characteristics
– Political, cultural, and social factors
• Health system characteristics
– Operational and cost issues
• Population of interest
– Health status and relationship to utilization
and expenditures

10
 Research Question
• Most important step
• Study design flows from research question
– Cascade of questions
• What domains/constructs will you
capture?

11
 Conceptual Models are Helpful
• Organize ideas and identify constructs and
variables relevant to your specific study
question
• Identify relationships among variables
– Predictors and Outcomes
– Confounders
– Effect Modifiers
– Acknowledge omitted variables
12
 Organize your thoughts about study variables

Exposure Mediators

Outcome

Image from Edmund Seto, PhD 13

 Study Design
• Research question drives study design
• Observational
– Cross-sectional (one point in time)
– Case control
– Cohort
• Experimental
– Quasi-experimental
– Randomized Controlled Trials

14
 Who is your target population?

How will you identify and

WHO select the participants in the
sample?

Who will be compared?

15
 Why Sample?
• Not feasible to work with entire population
– Too expensive/time consuming
• Oversampling – selecting sample participants
in a proportion that is (usually) larger than
that in population
– Permits inference on small populations
– Weights used to bring back ‘natural’ distributions
 Target Population
Target

http://www.socialresearchmethods.net/kb/sampterm.php 17
 Sampling
• Random
– Simple
– Systematic
– Stratified
– Clustered
• Non-random
– Purposeful
– Convenience
– Volunteer
– Snowball
18
 Random Sampling

• Simple Random Sample

– Every unit of the
population has an equal
chance of being included
• Systematic Random Sample
– Select a starting point on
a list and then take every
nth person afterwards

19
 Stratified Random Sampling

• The entire population of

interest is divided up into
homogenous subgroups and
those strata are sampled
randomly
– E.g. Participants’ ages in
decades: 50-59, 60-69,
70-79, etc.

20
 Clustered Random Sampling

• The entire population is

divided into
heterogeneous clusters
and a sample is drawn
from each cluster
– ICUs are the cluster
• Sample providers
randomly from
each ICU

21
 Non-Random Sampling
• Biased samples, not generalizable to target
population
• Purposive
– Qualitative research
– Driven by need to include heterogeneous
perspectives or targeted at specific people
 Non-Probability Sampling
• Convenience
– Sampling based on ease of access.
– E.g. friends, family, people in waiting room.
• Volunteer
– Flyers, phone calls, mailers
• Snowball
– Participants refer friends/family
 Will you measure
your variables of
interest?
HOW
Will you collect
your data?

24
 Questionnaire Variables
• Single or multiple variables?
• How do you measure?
– Vocabulary
Don’t just make something up!
– Intelligence
– Well-being Review existing, validated
questionnaires.

25
 Optimal Way of Finding Content
• Review literature for surveys on the construct
and in the population you are interested in
– http://www.healthmeasures.net/explore-
measurement-systems/promis
– https://www.cdc.gov/nchs/dhcs/index.htm
• Existing measures:
• Known psychometric properties
• Known sensitivity to change
26
Psychometric Properties
Not valid Not valid
Not reliable reliable

valid valid
Not reliable reliable
 Psychometric Properties
• Reliability
– Internal consistency
– Test-retest
– Inter-/Intra-rater
reliability
• Validity (More Important)
– Content
– Criterion
– Construct

28
 Reliability
Internal Consistency
• Cronbach’s Alpha (or Correlation Coefficients)
• Are the items in your scale measuring the
same concept?
– Function of inter-item covariance and
number of items
– Aim is for coefficient value close to 1.00
(>0.7 is acceptable)

29
 Reliability
Test-Retest (Stability over time)
• Correlation of scores between time points

correlation
Measurement coefficient Measurement
Time 1 Time 2

30
 Reliability
Inter-Rater Reliability
• The extent to which two data collectors agree (or
get the same responses)
• Kappa coefficient (compare observed to expected
agreement)
• Training to standardize administration of
questionnaires to limit variation between data
collectors

correlation
Data Gatherer coefficient Data Gatherer
A B
31
 Validity
Content Validity
• How well does an instrument measure a
theoretical construct
– Based on appropriate selection of the items
– Item selection based on:
• Review of the literature
• Items from similar instruments
• Expert panel
• Feedback from the target population
32
 Validity
Criterion Validity
• Extent to which the instrument
predicts/agrees with a criterion measure
(similar concept) or “gold standard”

33
 Validity
Construct Validity
• Theory driven
• Examines the extent to which the instrument
demonstrates hypothesized relationships with
other concepts/constructs
– Essential for more abstract concepts such as
attitudes and psychological distress
– Factor Analysis

34
 Sensitivity to Change

• Important when the test/instrument is

intended for use in a clinical trial to assess
therapeutic change
– Pre-post analysis where we compare
within groups t-test

35
Designing your own
Questionnaire
 Generate questions

Pilot test
Process
Refine

Evaluate
 Generating Questions
• Start with large pool
– Drawn from existing questionnaires
– Expert opinion
• Phrasing the question
– Consider reading level (usually aim for max 8 th
grade)
– Positive vs. negative phrasing
– Open vs. closed ended
• Ordering of questions – first vs later…
38
 Open Ended Questions
• More time
• Less constrictive: allows responses beyond a small
collection - minority views
• Difficult to code and analyze
– Qualitative analysis
• Less influenced by researcher because not providing
discrete answers
• Can lead to future, improved closed ended questions

39
 Closed Ended questions

• More constrictive but easier to code and

analyze
• Allow for ranking, rating
• Agree vs. disagree – strength
• Quantitative analysis

40
Level of Measurement

41
 Likert Scales (Ordinal)
1. Strongly disagree
2. Disagree
3. Neither agree nor disagree (neutral)
4. Agree
5. Strongly agree
• Ideal to have at least 5 ordinal responses
• Best practice to be balanced
• Some people think you should force a choice
by eliminating #3
42
43
 Pilot Testing Questions
• Pilot test questions in focus group of 6-10
people from the population of interest
– Help identify concepts of interest
– Early review of questions/language
• Revise questions and pilot again

44
 Variability in Response
• Critical to analysis. If everyone selects the same
answer, provides limited information
• What do you think about this course? (check one)
– It is the worst course I have ever taken
– It is somewhere between the worst and best
– It is the best course I have ever taken
• Respondents are likely to choose the second
category – no variability
Designing Your Own
Questionnaire
Content must be
evaluated with
hundreds of
participants to
establish
reliability and
validity

46
Data Collection
 Data Collection Methods
• Mail, telephone interview, in-person
interview are historically most common
• Online and/or mobile data collection
becoming much more popular –
especially since Covid!

48
 Data Collection Methods
• Personal interviews
– Higher response rates, ideal for open-ended
questions, but more socially threatening,
time consuming
• Telephone interviews
– More flexible design, perhaps less
threatening, but lower response rate

49
 Data Collection Methods
• Self-administered questionnaires
– Lower response rate, questions must be
less complex (less explanation), ideal for
closed-ended questions, can be less
threatening (more privacy)
• Computer Adaptive Testing (CAT)
– PROMIS®

50
 WHEN
• Timing of data collection
– Seasonal variations?
– Temporal variations?
– Cyclical events?

51
 Will you recruit
your participants?
WHERE
Will you interview
participants?

52
 Closing Comments
• Development of a questionnaire and
conducting survey research is…
– A major undertaking
– Requires a systematic approach
– Requires a skillset not commonly taught in
research or clinical training programs

53
 Closing Comments
• Educate yourself
– Existing questionnaire strengths &
weakness
– Psychometric methods and analyses
• Find collaborators with skillset that will
augment your own

54
Questions

55
 Reference Material
General texts on survey design/development/methods
1. Groves, RM, Fowler, FJ, Couper, MP, Lepkowski, JM, Singer, E, and Tourangeau. Survey
Methodology (2nd edition), John Wiley and Sons, Hoboken, NJ.
2. Aday, LA, Cornelius, LJ. Designing and Conducting Health Surveys: A Comprehensive Guide,
3rd ed. Jossey-Bass, San Francisco, CA. 2006.
3. Dillman, DA, Mail and Internet Surveys, The Tailored Design Method, 2 nd ed., New York: John
Wiley & Sons, 2000.
4. Levy, PS, Lemeshow, S. Sampling of Populations: Methods and Applications, 3 rd ed. John
Wiley & Sons, New York, 1999.
5. Ruel, E. 100 Questions (and Answers) about Survey Research. SAGE Publications, Thousand
Oaks, CA, 2019.
6. Robinson, SB, Leonard, KF. Designing Quality Survey Questions. SAGE Publications, Thousand
Oaks, CA, 2019.
Psychometric texts
1. Nunnally JC, Bernstein IH. Psychometric Theory, 3rd ed. McGraw-Hill, Inc., New York. 1994.
2. Bond TG, Fox CM. Applying the Rasch Model: Fundamental Measurement in the Human
Sciences, 2nd. ed. Lawrence Erlbaum Associates, Mahwah, NJ. 2007.
3. Wright BD, Stone MH. Best Test Design: Rasch Measurement. Mesa Press, Chicago, IL. 1979.
4. Wright BD, Masters GN. Rating Scale Analysis. Chicago: MESA Press, 1982.

56
http://www.nidcr.nih.gov/Research/DER/BSSRB/PowerPointPresentations/default.htm