Evidenced Based Medicine and Research Methods

Observational Studies
STUDENT HANDOUT
Aims of session

 To critically appraise an observational epidemiological study (cohort study) using a systematic

approach.
 To consider the strengths and weaknesses of an observational study design in relation to a
clinical question.

Learning objectives

 To consider the strengths and weaknesses of observational epidemiological studies when

answering clinical questions.
 To consider the role of bias and confounding in interpretation of the findings of an
observational epidemiological study.
 To interpret and explain the findings of observational epidemiological studies.
 To consider the applicability of an observational epidemiological study to a clinical question.

Key Learning Points

 Epidemiological observational studies take advantage of naturally occurring phenomenon to

investigate associations between exposures and outcomes. The exposures under study are not
under the control of the researcher. This is in contrast to experimental epidemiological studies
such as randomised controlled trials (RCTs).
 Observational epidemiological studies are used to: generate hypotheses for further research;
answer clinical questions about the frequency of disease or risk factors; answer clinical
questions about whether exposures cause harm or benefit.
 When an association is observed between an exposure and an outcome first ask if the
observation can be explained by chance, bias or confounding. The Bradford Hill criteria can be
used to help decide if the observed association represents a cause and effect relationship.
 A well designed cohort study is considered the most valid (trustworthy) observational study
design to examine the association between environmental/lifestyle/iatrogenic exposures and
disease outcome. Case control studies are considered less valid evidence for a cause and effect
relationship between exposures and outcomes. However the choice between a cohort study and
a case control study may be dictated by pragmatic considerations such as the time period
between exposure and development of outcomes and how rare the outcome of interest is.
 There are a number of issues concerning study subject selection, measurement of exposures
and outcomes that need to be considered as potential sources of bias and confounding when
assessing whether the findings of a cohort study are trustworthy, and if the findings are
applicable to a specific clinical situation.
 Systematic Reviews are useful to summarise a body of observational evidence and explore the
effects of bias and confounding across multiple primary studies.

EBM&RM Observational Studies Page 1

Clinical scenario

Carol Jones is a 53 year old woman who presents to you (her GP) with recent onset (3-4 months) of
sweating, anxiety, loss of libido, problems with concentration and irregularities with her menstrual
cycle. She has wondered for a while if she is menopausal and has come to you now because the
symptoms have started to interfere with her ability to work. You organise some tests to rule out
alternative explanations for her symptoms but consider that the most likely diagnosis is the onset of
menopause.

You recall that there has been a lot of debate over the years about whether HRT treatment is
associated with an increased risk of cardiovascular disease. You want to know ‘What are the effects
of HRT on cardiovascular disease in post menopausal women?’

Background

Menopause occurs as part of the normal ageing process in women, usually between 45 and 55 years
of age. Iatrogenic causes of menopause include hysterectomy and total abdominal hysterectomy
with bilateral salpingo-oophorectomy. Oestrogen therapy was first used for the treatment of
menopausal symptoms in the 1930s.

Oestrogens are known to be associated with physiological processes that could have favourable
effects on Cardiovascular Disease (CVD). These include lowering low-density lipoprotein cholesterol
levels, lower lipoprotein (a) levels and increased high-density lipoprotein cholesterol levels (HDLs).
However oestrogen is also observed to increase inflammatory proteins such as C-reactive protein
which may have an adverse effect on CVD.

In the 1980s, accumulating evidence from observational studies of women prescribed HRT for
menopausal symptoms suggested that cardiovascular disease may be reduced in these women. In
1992, on the basis of this accumulating observational evidence, the American Medical Association
published guidelines promoting the use of HRT as a preventive treatment in menopausal women (4)

EBM&RM Observational Studies Page 2

Pre SGT Instructions

PRIOR to the small group tutorial consider the questions below:

(i) What are the components of the clinical question arising from the clinical scenario?

Population: Women ( 45-55 yrs) with menopausal symptoms

Intervention / Exposure: HRT

Comparator: No intervention/placebo

Outcome: Frequency of CVD events

(ii) Consider potential sources of confounding that are relevant to the investigation of the
relationship between Hormone Replacement Therapy (HRT) and cardiovascular disease:

Hint: A confounder is a factor associated with the exposure and independently also associated with
the outcome

Exposure: Hormone Replacement

Therapy

Outcome: Cardiovascular Disease

Confounders:

 Age
 Obesity
 Smoking
 Physical activity
 Socioeconomic status
 HTN
 Family history
 Alcohol

EBM&RM Observational Studies Page 3

Use the CASP cohort study checklist to critically appraise two cohort studies: Grodstein et al and
Hunt et al available on the CANVAS page. Make sure you add comments in the CASP checklist in
response to the HINTs provided for each question to facilitate discussion during the SGT.

-Hunt et al. Mortality in a cohort of long-term users of hormone replacement therapy: an updated
analysis. British Journal of Obstetrics and Gynaecology 1990. (1) AVAILBLE ON CANVAS

For the Hunt paper read pages 1080-1082 (up to the results section) and table 1. Work through
questions 1-10 of the CASP checklist. For CASP questions 7 and 8 use the result for ‘Ischaemic heart
disease’ in table 1.

-Grodstein et al. A prospective, observational study of postmenopausal Therapy and Primary

Prevention of Cardiovascular Disease. Annals of Internal Medicine 2000. (2) AVAILABLE ON CANVAS

For the Grodstein paper read pages 933-936 including the results for ‘Risk for Major Coronary
events’ and Table 2. Work through questions 1-10 of the CASP checklist. For CASP questions 7 and 8
use the multivariate adjusted relative risk for ‘Current hormone use’ in table 2.

EBM&RM Observational Studies Page 4

DURING the small group tutorial:

PART 1

(a) In small groups you will discuss and compare your answers to the CASP checklist questions
1-10 for each of the Hunt and Grodstein papers.
(b) The whole group will discuss and compare answers to questions 1-10.

PART II

(a) In small groups or as a whole group you will consider an extract from the results of a
systematic review of observational studies: Humphrey et al. Postmenopausal Hormone
Replacement Therapy and the Primary Prevention of Cardiovascular Disease. Annals of
Internal Medicine 2002.(3) CONTAINED IN THESE SGT NOTES
(b) Based on your discussion from Part II (a) you will consider CASP questions 11 and 12.

EBM&RM Observational Studies Page 5

SGT Instructions

Part 1: Critical Appraisal of Cohort studies

In breakout groups discuss and compare your answers to questions 1-10 of the CASP checklist for
the Hunt and Grodstein cohort studies.

Questions 1 and 2: Did the studies address a clearly focused issue and were the cohorts recruited
in an acceptable way?

 Think PICO

Question 3: Was the exposure accurately measured to minimise bias?

Information bias arises because of systematic differences in measurement between groups being
compared (exposed and unexposed; with or without the outcome of interest or over time).

Objective measures reduce the potential for measurement bias compared to subjective measures
(interviewer bias; social acceptability bias). Measurement of exposure should be blind to
knowledge /measurement of outcomes and vice versa.

EBM&RM Observational Studies Page 6

Question 4: Was the outcome accurately measured to minimise bias?

Questions 5(a) and 5(b): Have the authors identified all important confounders and have they
taken account of the confounding factors in the design and/or analysis?

Confounders should be measured during recruitment and then perform appropriate analysis to
account for them e.g. stratification, standardisation or using suitable regression models.

One of the benefits of randomisation in experimental studies is that it ensures known and unknown
confounders are equally distributed across comparison groups.

Questions 6(a) and 6(b): Was the follow up of subjects (a) complete enough and (b) long enough?
Complete enough – the majority of people in the trial need to be followed up.

Long enough – is time long enough.

EBM&RM Observational Studies Page 7

Questions 7 and 8: What are the results and how precise are the results?
-What is the effect measure (risk ratio or risk difference)?
-What is the direction of effect (more of less of the outcome)
-What is the size of the effect?
-How precise is the estimate of effect (CI)
-Is the effect size statistically significant?

In addition to the ‘hints’ provided in the CASP checklist consider whether the size and precision of
effect likely to be clinically significant? Consider whether the outcome is indirect (eg blood
pressure, cholesterol level) or direct (eg myocardial infarction; stroke). Consider whether the lowest
estimate of effect (lowest CI) is still large enough to be clinically important.

EBM&RM Observational Studies Page 8

Question 9: Do you believe the results?
Use the Bradford Hill Criteria to help with this question.

-Is the observed effect large?

-Is the effect likely to be due to chance?

Is a p value reported? Does the CI cross 1 for a ratio measure or 0 for a risk difference?

-Is the effect likely to be due to bias or confounding? Consider your answer to questions 2, 3 and 4
for bias and 5 for confounding

-Did exposure precede outcome

-Is there a dose response relationship between exposure and outcome?

(HINT Look at table 2 in the Grodstein paper).

-Is it biologically plausible that the exposure could cause the outcome

Question 10: Can the results be applied to the local population?

EBM&RM Observational Studies Page 9

SGT Part 2: Interpretation of a SR of observational studies

The objectives of part II of this SGT are to

 Interpret a Forest plot

 Appreciate the potential effects of confounding in observational studies
 Appreciate the potential role of systematic reviews in investigating heterogeneity
 Help to answer questions 11 and 12 from the CASP checklist:
-Do the results of the Hunt and Grodstein cohort studies fit with other available evidence?
-What are the implications (of the cohort studies and SR) for practise?

Concurrent with the publication by the American Medical Association guidelines promoting the use
of HRT as a preventive treatment in menopausal women, and based on the growing body of
observational research suggesting HRT may protect against Cardiovascular Disease (CVD), two
Randomised Controlled Trials (RCTS) were begun. The RCTs were designed to address specifically the
effectiveness of HRT for preventing cardiovascular disease. Note this is distinct for the effectiveness
of HRT for reducing symptoms associated with the menopause.

The first RCT was conducted in women with established CVD to investigate whether HRT could
prevent disease progression (secondary prevention) (5). The second RCT was conducted in women
with no prior history of CVD, to investigate if HRT was effective for the primary prevention of CVD
(6). The secondary prevention trial demonstrated no difference between women taking HRT and
those not taking HRT in further CVD events over a 4 year period (5). The primary prevention trial
demonstrated no difference between women taking HRT and those not taking HRT in overall CVD
after 8 years of follow up for women without uteri taking unopposed oestrogen (6). For the
subgroup of women with intact uteri taking combined oestrogen and progesterone the trial was
stopped after 5 years because of an observed increase in CVD (6).

In the light of conflicting results from observational studies suggesting a protective effect of HRT for
the development of CVD and RCT evidence suggesting no or increased risk of CVD for women taking
HRT, a systematic review was undertaken to examine the relationship between HRT and the primary
prevention of cardiovascular disease (3). The review objectives included the investigation of whether
bias and confounding might explain discordant findings between observational studies and
randomised controlled trials. Eleven case control studies, 9 cohort studies and 1 RCT were included
in the review which included the Grodstein cohort study appraised above. The Hunt cohort study
was considered by the review authors but was not included in the review meta-analysis on the basis
of poor quality, (no controlling for confounding factors, inclusion of a non-representative cohort of
women and poorly defined outcome measures).

EBM&RM Observational Studies Page 10

The figure below is a Forest plot from the systematic review illustrating the results for one of the
outcomes measured: incidence of cardiovascular disease.

Note: the forest plot combines case control studies, cohort studies and one small RCT. It is not
considered good practice to combine studies of different designs in a meta-analysis as differences in
methodological quality introduce variation (heterogeneity). However we are using the meta-analysis
to illustrate the effect of confounding across different measures of exposure and across different
study types rather than producing a precise summary estimate of effect size across studies.

The comparator (unexposed) group for the derivation of summary measures in the Forest plot is ‘never use’ of
HRT.

Humphrey L, Chan NKS, Sox HC. Postmenopausal Hormone Replacement Therapy and the Primary
Prevention of Cardiovascular Disease. Ann Intern Med. 2002;137:273-284.(3)

EBM&RM Observational Studies Page 11

Question 11: With reference to the Forest plot from the SR (fig 4). Do the results of the Grodstein
and Hunt studies fit with other available evidence?

 No

11(a) Are the outcome measures reported in the Grodstein and Hunt studies the same as the
outcome measure reported in the Forest plot?

11(b) Take the study by Pfeffer (1978) and explain the box and whisker plot

11(c) What do the circles at the bottom of the plot represent?

- Socioeconomic status can have an influence ( HRT might not have such an effect by itself )
so needs to be considered in CVD mortality

11(d) Comment on the differences in summary estimates that are SES (socioeconomic status)
adjusted and non SES adjusted. What does this illustrate?

EBM&RM Observational Studies Page 12

Conclusion

Questions 12: What are the implications for practice, regarding the use of HRT and CVD?

Early observational evidence suggesting HRT had a beneficial effect on CVD outcomes was almost
certainly an effect of confounding by social class. Women of higher socio economic status are more
likely to seek out HRT and are also more likely to have a reduced risk of CVD as a result of healthy
behaviours (non smoking, physically active, better diet). One of the benefits of systematic reviews is
that they allow researchers to investigate the effects of the quality of evidence, including the effects
of confounding, on estimates of effect.

Since publication of the Humphrey systematic review (3) experimental evidence has accrued
showing that the use of HRT in younger women (50–59 years) or in early postmenopausal women
(within 10 years of menopausal onset) has a beneficial effect on the cardiovascular system, reducing
coronary diseases and all-cause mortality [6-10]. Furthermore, a large controlled trial from Denmark
(reported in 2012) has demonstrated that healthy women taking combined HRT for 10 years
immediately after menopause have a reduced risk of heart disease and death from heart disease
[11]. However HRT is specifically associated with an increase in risk of venous thromboembolism
and thromboembolic stroke, which may be clinically significant in older women and women with
pre-existing CVD.

What would you advise Mrs Jones about taking HRT and the associated risks of CVD?

EBM&RM Observational Studies Page 13

References and additional reading:

References to papers for critical appraisal

1. Hunt K, Vessey M, McPherson K. Mortality in a cohort of long-term users of hormone replacement

therapy: an updated analysis. British Journal of Obstetrics and Gynaecology. 1990:97; 1080-1086.

2. Grodstein F, Manson J, Colditz GA, Wilwt WC, Speizer FE, Stampfer MJ. A Prospective,
Observational Study of Postmenopausal Hoemoine Therapy and Primary Prevention of
Cardiovascular Disease. Ann Intern Med. 2000;133:933-941

3. Humphrey L, Chan NKS, Sox HC. Postmenopausal Hormone Replacement Therapy and the Primary
Prevention of Cardiovascular Disease. Ann Intern Med. 2002;137:273-284.

Additional references and further reading

4. American Medical Association. Guidelines for counseling postmenopausal women about
preventive hormone therapy. American College of Physicians. Ann. Intern. Med. 1992, 117, 1038–
1041
5. Hulley, S.; Grady, D.; Bush, T.; Furberg, C.; Herrington, D.; Riggs, B.; Vittingho_, E. Randomized trial
of estrogen plus progestin for secondary prevention of coronary heart disease in postmenopausal
women. Heart and Estrogen/progestin Replacement Study (HERS) Research Group. JAMA 1998, 280,
605–613.
6. Rossouw, J.E.; Anderson, G.L.; Prentice, R.L.; LaCroix, A.Z.; Kooperberg, C.; Stefanick, M.L.; Jackson,
R.D.; Beresford, S.A.; Howard, B.V.; Johnson, K.C.; et al. Writing Group for the Women’s Health
Initiative Investigators. Risks and benefits of estrogen plus progestin in healthy postmenopausal
women: Principal results from the Women’s Health Initiative randomized controlled trial. JAMA
2002, 288, 321–333.
7. Manson, J.E.; Chlebowski, R.T.; Stefanick, M.L.; Aragaki, A.K.; Rossouw, J.E.; Prentice, R.L.;
Anderson, G.; Howard, B.V.; Thomson, C.A.; LaCroix, A.Z.; et al. Menopausal hormone therapy and
health outcomes during the intervention and extended poststopping phases of theWomen’s Health
Initiative randomized trials. JAMA 2013, 310, 1353–1368
8. Salpeter, S.R.; Walsh, J.M.; Greyber, E.; Salpeter, E.E. Brief report: Coronary heart disease events
associated
with hormone therapy in younger and older women. A meta-analysis. J. Gen. Intern. Med. 2006, 21,
363–366.
9. Salpeter, S.R.; Walsh, J.M.; Greyber, E.; Ormiston, T.M.; Salpeter, E.E. Mortality associated with
hormone replacement therapy in younger and older women: A meta-analysis. J. Gen. Intern. Med.
2004, 19, 791–804.
10. Boardman, H.M.; Hartley, L.; Eisinga, A.; Main, C.; i Figuls, M.R.; Cosp, X.B.; Sanchez, R.G.; Knight,
B. Hormone therapy for preventing cardiovascular disease in postmenopausal women. Cochrane
Database Syst. Rev. 2015
11. Rossouw, J.E.; Prentice, R.L.; Manson, J.E.; Wu, L.; Barad, D.; Barnabei, V.M.; Ko, M.; LaCroix, A.Z.;
Margolis, K.L.; Stefanick, M.L. Postmenopausal hormone therapy and cardiovascular disease by age
and years since menopause. JAMA 2007, 297, 1465–1477.
11. Schierbeck, L.L.; Rejnmark, L.; Tofteng, C.L.; Stilgren, L.; Eiken, P.; Mosekilde, L.; Køber, L.; Beck
Jensen, J.E.E_ect of hormone replacement therapy on cardiovascular events in recently
postmenopausal women: Randomized trial. BMJ 2012, 345, e6409.

EBM&RM Observational Studies Page 14