Step by Step- Evaluating Violence and Injury Prevention Policies

Brief 1: Overview of Policy Evaluation

This brief provides a definition of policy evaluation, including a
description of the evaluation framework used and a definition "Policy evaluation uses a
of policy. It also presents information about the general value range of research methods to
and potential challenges of conducting policy evaluation. systematically investigate the
effectiveness of policy interventions,
Defining Policy implementation and processes, and
The CDC definition of "Policy" is "a law, regulation, procedure, to determine their merit, worth,
administrative action, incentive or voluntary practice of or value in terms of improving the
governments and other institutions." Policies generally operate
1
social and economic conditions of
at the systems level and can influence complex systems in ways different stakeholders."3
that can improve the health and safety of a population. A policy
approach can be a cost-effective way to create positive changes
in the health of large portions of the population. There are
several types of policy, each of which can operate at different levels (national, state, local, or organizational) 2
Legislative policies are laws or ordinances created by elected representatives. Regulatory policies include rules,
guidelines, principles, or methods created by government agencies with regulatory authority for products or
services. Organizational policies include rules or practices established within an agency or organization.

What Is Policy Evaluation?

Policy evaluation applies evaluation principles and methods to examine the content, implementation or impact
of a policy. Evaluation is the activity through which we develop an understanding of the merit, worth, and
utility of a policy.

Figure 1. Steps in the CDC Framework for

CDC Evaluation Framework
Evaluation in Public Health 4

7
While there are a variety of different approaches to evaluation, this
set of briefs utilize the six-step CDC Framework for Evaluation in
' Engage
Public Health, as shown in Figure 1. 4
�keholders

The Framework outlines an ongoing process comprising six steps

reUse nd
j
Describe the of program evaluation. 2 These six steps are also applicable to
[:
are Lessons
Standards policy evaluation and provide a guide for implementing a thorough
Learned
Utility
evaluation. The remaining briefs each discuss the implementation
Feasibility
Propriety of one or more of these steps. Figure 2 illustrates which steps are
Justify Accuracy Focus discussed in each brief.
the Evaluation
Conclus ions
Design Briefs 2, 6, and 7 discuss the steps as they apply to all types of
policy evaluation, whereas Briefs 3, 4, and 5 discuss Step 3 (focusing
the evaluation design) as it applies to the specific types of policy
evaluation.

1 CDC, Office of the Associate Director for Policy. (2011). Definition of policy. PDF available upon request; please contact ADpolicy@cdc.gov
2 CDC, Office of the Associate Director for Policy. (2011). Definition of policy. PDF available upon request; please contact ADpolicy@cdc.gov
3 Her Majesty's Treasury. (2011). The magenta book: Guidance for evaluation. London, UK: Author. Retrieved from http://www.hm-treasury.gov.
uk/data_magentabook_index.htm
4 CDC, Office of the Associate Director for Program. (2012, September). A framework for program evaluation. Retrieved from http://www.cdc.gov/
eval/framework/index.htm

National Center for Injury Prevention and Control

Figure 2. The Briefs in Relation to the Steps in the CDC Evaluation Framework
(1) Engaging (2) Describing (3) Focusing (4) Gather ( 5) Justify (6) Ensure Use
Stakeho Iders the Program or the Evaluation Credible Conclusions and Lessons
Policy Evidence Learned
Brief 2 • • •
Brief 3 • • •
Brief 4 • • •
Brief 5 • • •
Brief 6 • •
Brief 7 •

Standards for Conducting Evaluation

The Framework also includes the following four categories of standards for conducting evaluation to help guide
choices along the process:

• Utility: Who wants the evaluation results and for what purpose?
■ Feasibility: Are the evaluation procedures practical, given the time, resources, and expertise available?

• Propriety: Is the evaluation being conducted in a fair and ethical way?

• Accuracy: Are approaches at each step accurate, given stakeholder needs and evaluation purpose?

Policy Evaluation Versus Program Evaluation

Although policy evaluation and program evaluation have many similarities, there are some important
differences as well. Some of these differences include:

■ The level of analysis required (e.g., system or community level for policy evaluation; program level for
program evaluation).
• The degree of control and clear "boundaries" may be more challenging with policy evaluation.
• The ability to identify an equivalent comparison community may be more challenging with policy evaluation.
• The scale and scope of data collection may be greater with policy evaluation.
• Policy evaluation may require increased emphasis on the use of surveillance and administrative data.
• The type and number of stakeholders involved may differ.

Why Is Policy Evaluation Important?

Developing and implementing policy strategies is important in addressing injury and violence prevention at the
population level. Although policy has been used effectively in some areas of injury and violence prevention,

2 Step by Step- Evaluating Violence and Injury Prevention Policies

policy strategies in some areas lack a sufficient evidence base. Policy evaluation, like all evaluation, can serve
important purposes along the entire chain of the policy process, including1 :

■ Documenting policy development. ■ Demonstrating impacts and value of a policy.

■ Documenting and informing implementation. ■ Informing an evidence base.
■ Assessing support and compliance with existing ■ Informing future policies.
policies. ■ Providing accountability for resources invested.

Evaluation Within the Policy Process

It is important to understand how policy evaluation fits into the larger policy process. Understanding this
context provides an increased understanding of why policy evaluation is critical to advancing the field of policy.
Although there are many theories regarding the policy process and mechanisms of policy change, the policy
change process is often conceptualized in several key stages as depicted on the top row of Figure 3. 2 Evaluation
is an integral part of each step in the policy process. Although these steps are laid out in a row, in reality, the
steps are circular in nature. The three main types of evaluation, shown in the bottom row of Figure 3, each
focus on a different phase of the policy process5 : policy content evaluation, policy implementation evaluation,
and policy impact evaluation. Figure 3 illustrates the relationship between the main stages of the policy
process and the three types of evaluation.

■ Evaluating Policy Content: Does the content clearly articulate the goals of the policy, its implementation
and the underlying logic for why the policy will produce intended change? Evaluating the development of a
policy helps to understand the context, content, and implementation.
■ Evaluating Policy Implementation: Was the policy implemented as intended? The implementation of a
policy is a critical component in understanding its effectiveness. Evaluation of policy implementation can
provide important information about the barriers to and facilitators of implementation and a comparison
between different components or intensities of implementation.
■ Evaluating Policy Impact: Did the policy produce the intended outcomes and impact? Within injury
prevention, the intended impact may be a reduction in injuries or severity of injuries. However, it is
important to evaluate short-term and intermediate outcomes as well.
The type of evaluation selected depends on many factors, and often more than one type of evaluation will
be needed. Each type of evaluation can provide valuable information for the planning and interpretation of
the other types of evaluation (content, implementation, and impact) in addition to uncovering unintentional
consequences. 6 However, it is critical for each evaluation to be focused so the most appropriate design and
methodology is selected. 1 The team can develop an overarching set of evaluation questions and then select
specific evaluation questions and methods for each particular phase. Appendix A and Appendix B provide
examples of the planning, implementation, and dissemination of a policy evaluation.

5 Brownson, R. C., Royer, C., Chriqui, J. F., & Stamatakis, K. A. (2009). Understanding evidence-based public health policy. American Journal of
Public Health, 99, 1576-1583.
6 MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowksi, K., & Turner, K. (2001). Introduction to program evaluation for comprehensive
tobacco control programs.Atlanta, GA: Centers for Disease Control and Prevention. Retrieved from http://www.cdc.gov/tobacco/tobacco_
control_programs/ surve ill ance _eval uation/evaluation _manua1/pdfs/eval uation. pdf

3 Step by Step- Evaluating Violence and Injury Prevention Policies

 Figure 3. Policy Development Phases and Types of Evaluation

Problem Policy Strategy and Policy Policy Policy

Identification Analysis Development Enactment Impie mentation

Implementation Impact
Content Evaluation
Evaluation Evaluation

Policy Evaluation Challenges

While all evaluations encounter challenges, some are particularly relevant to policy evaluation, and some
of these are listed below. Many of these challenges can be easily addressed by using an appropriate design,
indicators, and methods. Specific solutions to many of these challenges are provided in Briefs 2-7. In addition,
Appendix C provides an overview of the challenges and potential solutions when conducting policy evaluation.

Potential Policy Evaluation Challenges

■ Lack of resources or clear responsibility for ■ Lack of strong evide nee base to support policy
evaluation
■ Fear of evaluation and lack of familiarity with ■ External and contextual factors such as
policy evaluation methods economic conditions or public awareness
■ Lack of "control" over policy implementation ■ Access to appropriate data
■ Rapid pace of policy ■ Lack of appropriate measures
■ Political scrutiny and desire for quick production ■ Difficulty in identifying appropriate comparison
of results communities

Additional Resources
The Magenta Book: Guidance for Evaluation (Her Majesty's Treasury). Provides general and technical
guidance on policy evaluation. Available at http://www.hm-treasury.gov.uk/data_magentabook_index.htm

CDC Evaluation Page: http://www.cdc.gov/eval/

Appendix A: Policy Impact Evaluation Example - Child Restraint Law Expansion

Appendix B: Restricting Hours of Alcohol Sales to Prevent Excessive Alcohol Consumption and Related lnj uries

Appendix C: Challenges and Potential Solutions to Policy Evaluation

Appendix D: Glossary

4 Step by Step- Evaluating Violence and Injury Prevention Policies