Evaluation in Healthcare

Education

Dr. Duaa Amr

Hafez
 Learning objectives

By the end of the session, you will be able to:

• Describe the differences between the terms evaluation and assessment.
• Distinguish between five basic types of evaluation: process, content, outcome, impact,
and program.
• Discuss characteristics of various models of evaluation.
• List the major barriers to evaluation.
• Examine methods for conducting an evaluation.
• Explain the variables that must be considered in selecting appropriate evaluation
instruments for the collection of different types of data.
• Identify guidelines for reporting the results of evaluation.
 Introduction

Evaluation is defined as a
systematic process that
judges the worth or value of
something.
 Evaluation vs. assessment

Assessment Evaluation

• The process of assessment • The process of evaluation

focuses on initially gathering, involves gathering,
summarizing, interpreting, summarizing, interpreting,
and using data to decide a and using data after an activity
direction for action has been completed to
• Assessment data might be determine the extent to which
called the “input.” an action was successful.
• Could be unplanned • Evaluative data might be
called “output.”
• Must be planned
 Determining the focus of evaluation

For which audience is the evaluation being conducted?

• Primary vs. general audience

For what purpose is the evaluation being conducted?

• Keeping the purpose singular and focused on the audience helps avoid the frequent tendency to attempt to do too
much in one evaluation.

Which questions will be asked in the evaluation?

• Questions to be asked must be directly related to the purpose for conducting the evaluation, must be specific, and
must be measurable.

What is the scope of the evaluation?

• Scope considers the extent of what is being examined

Which resources are available to conduct the evaluation?

• Resources include time, expertise, personnel, materials, equipment, and facilities.
Evaluation Models
 Types of evaluation
High Low

Impact

Outcome
Time & Cost (summative) Frequency

Content

Process
(formative)
Low High
 Process (formative) evaluation

The purpose of process evaluation is to make necessary adjustments to

an educational activity as soon as they are identified.

It is a component of the ongoing education cycle of assessment,

planning, and implementation.

Process evaluation helps the nurse anticipate and prevent problems

before they occur or identify problems as they arise.

Resources usually are less costly and more readily available for process
evaluation than for the other types of evaluation,
 Content evaluation
The purpose of content evaluation is to determine whether learners have acquired the knowledge or skills taught during the
learning experience.

It takes place immediately after the learning experience to answer the guiding question, “to what degree did the
learners learn what they were taught?” Or “to what degree did learners achieve preset behavioral objectives?”

The purpose of content evaluation is to focus on how the teaching–learning process affected immediate, short-term
outcomes

The scope of content evaluation is limited to a specific learning experience and to specifically stated objectives for
that experience.

Data are obtained from all learners involved in a specific class or group teaching session.

Resources used to teach content can be evaluated as to how well that content was learned.
 Outcome (summative) evaluation
The purpose of outcome evaluation is to determine the effects of teaching efforts.

It measures the changes that result from teaching and learning.

It occurs after teaching has been completed or after an educational program has been carried out.

It measures more long-term change that “persists after the learning experience”

Resources required for outcome evaluation are costly and complex

It requires knowledge of how to establish baseline data, greater expertise to develop measurement and data collection strategies, more time to
conduct the evaluation, and the ability to collect reliable and valid data for comparative purposes after the learning experience has occurred.
 Impact evaluation

The purpose of impact evaluation is to determine the relative effects of education on the institution
or the community.

The purpose of impact evaluation is to obtain information that will help decide whether
continuing an educational activity is worth its cost

The scope of impact evaluation is broader, more complex, and usually more long term than
that of process, content, or outcome evaluation.

It focuses on a goal for learning.

Because impact evaluation requires many resources, including time, money, and research expertise,
this type of evaluation is usually beyond the scope of the individual nurse educator.
 Total program evaluation

The purpose of total program evaluation is to determine the extent to which

all activities for an entire department or program over a specified time meet
or exceed the goals originally established.

The scope of program evaluation is broad, generally focusing on overall

goals rather than on specific learning objectives.

Given its scope, total program evaluation is complex, usually focusing on the
learner and the teacher and the educational activity
Designing the evaluation
 Evaluation methods

Which types of data will be collected?

•People
•Educational program
•Environment

What data will be collected and from whom?

•Stakeholders (all or representatives)
•Direct vs. Indirect

How, when, and where will data be collected?

•Observation
•Interview
•Questionnaire or written examination
•Record review
•Secondary analysis of existing databases

Who will collect the data?

•Educator, learners
 Evaluation instruments

Conduct a literature search for evaluations similar to the evaluation that is

being planned

Once potential instruments have been identified, each instrument must be

carefully critiqued

The instrument must measure the performance being evaluated

An appropriate instrument should have documented evidence of its reliability

and validity
 Barriers to evaluation

Lack of clarity
• To overcome a potential lack of clarity, the nurse educator must identify all five components and make them
available to those conducting the evaluation.

Lack of ability
• To resolve lack of knowledge enlisting the assistance of individuals with needed expertise through
consultation or contract (if funds are available), through collaboration, or indirectly through literature review.

Fear of punishment or loss of self-esteem

• The first step in overcoming this barrier is to realize that the potential for its existence may be close to 100%
• The second step in overcoming the barrier of fear or threat in being evaluated is to remember that “the
person is more important than the performance or the product”
• A third step in overcoming the fear of being evaluated on the outcomes of education is to point out
achievements, if they exist, or to continue to encourage effort if success at learning has not been achieved.
• Lastly, people must be reassured that their privacy will be protected, otherwise they may be unwilling to be
evaluated.
 Conducting the evaluation

How smoothly an evaluation is implemented depends primarily on how

carefully and thoroughly that evaluation was planned and how carefully
the instruments for data collection were selected or developed.

The following three methods are likely to add to a successful achievement

of the process:

• Conduct a pilot test first.

• Include extra time to complete all the evaluation steps.
• Keep a sense of humor throughout the experience.
 Analyzing and interpreting data collected

The purposes for conducting data analysis are:

(1) To organize data so that they can provide meaningful information
(2) To provide answers to evaluation questions.

Qualitative Quantitative

• All qualitative data are at the nominal level of • Quantitative data can be at the nominal,
measurement ordinal, interval, or ratio level of
• Qualitative data are summarized, or themed, measurement.
into categories of similar comments. • The first step consists of organizing and
• Each category or theme is qualitatively summarizing the data
described by directly quoting one or more • The next step is to select the statistical
comments that are typical of that category. procedures appropriate for the type of data
• These categories then may be quantitatively collected to answer the questions asked during
described using descriptive statistics such as the planning phase of the evaluation.
total counts and percentages.
 Reporting evaluation results

Be audience focused.
• Always begin an evaluation report with an executive summary or an abstract that is no longer
than one page.
• Present evaluation results in a format and language that the audience can understand and use
without additional interpretation.
• Present results in person as well as in writing.

Stick to the evaluation purpose.

• Keep the main body of an evaluation report focused on information that fulfills the purpose for
conducting the evaluation.

Use data as intended.

• Maintain consistency with actual data when reporting and interpreting findings
• A discussion of any limitations of the evaluation.
Questions?

Thank you!