Evaluating Supported

Your Program Employment

U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES

Substance Abuse and Mental Health Services Administration
Center for Mental Health Services
www.samhsa.gov
 Evaluating Supported
Your Program Employment

U.S. Department of Health and Human Services

Substance Abuse and Mental Health Services Administration
Center for Mental Health Services
Acknowledgments

This document was produced for the Substance Abuse and Mental Health Services Administration
(SAMHSA) by the New Hampshire-Dartmouth Psychiatric Research Center under contract number
280-00-8049 and Westat under contract number 270-03-6005, with SAMHSA, U.S. Department
of Health and Human Services (HHS). Neal Brown, M.P.A., and Crystal Blyler, Ph.D., served as
the Government Project Offcers.

Disclaimer

The views, opinions, and content of this publication are those of the authors and contributors and
do not necessarily refect the views, opinions, or policies of the Center for Mental Health Services
(CMHS), SAMHSA, or HHS.

Public Domain Notice

All material appearing in this document is in the public domain and may be reproduced or
copied without permission from SAMHSA. Citation of the source is appreciated. However, this
publication may not be reproduced or distributed for a fee without the specifc, written
authorization from the Offce of Communications, SAMHSA, HHS.

Electronic Access and Copies of Publication

This publication may be downloaded or ordered at www.samhsa.gov/shin. Or, please call

SAMHSA’s Health Information Network at 1-877-SAMHSA-7 (1-877-726-4727) (English
and Español).

Recommended Citation

Substance Abuse and Mental Health Services Administration. Supported Employment: Evaluating
Your Program. DHHS Pub. No. SMA-08-4364, Rockville, MD: Center for Mental Health Services,
Substance Abuse and Mental Health Services Administration, U.S. Department of Health and
Human Services, 2009.

Originating Offce

Center for Mental Health Services

Substance Abuse and Mental Health Services Administration
1 Choke Cherry Road
Rockville, MD 20857

DHHS Publication No. SMA-08-4364

Printed 2009
Evaluating Your Program

Evaluating Your Program shows quality assurance team members

how to evaluate the effectiveness of your Supported
Employment program. It includes the following: Supported
n A readiness assessment; Employment
n The Supported Employment Fidelity Scale;
n The General Organizational Index; and
n Outcome measures that are specifc to your program.
You will also fnd instructions for conducting assessments and tips
on how to use the data to improve your program.

For references, see the booklet The Evidence.

This KIT is part of a series of Evidence-Based Practices KITs created
by the Center for Mental Health Services, Substance Abuse and
Mental Health Services Administration, U.S. Department of Health
and Human Services.

This booklet is part of the Supported Employment KIT that includes

a DVD, CD-ROM, and seven booklets:

How to Use the Evidence-Based Practices KITs

Getting Started with Evidence-Based Practices

Building Your Program

Training Frontline Staff

Evaluating Your Program

The Evidence

Using Multimedia to Introduce Your EBP

What’s in Evaluating Your Program

Why Evaluate Your Supported Employment Program . . . . 3

Conduct a Readiness Assessment. . . . . . . . . . . . . . . . . . . 5

Supported
Conduct a Process Assessment . . . . . . . . . . . . . . . . . . . . 7 Employment
Monitor Outcomes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Using Data to Improve Your Program . . . . . . . . . . . . . . 19

Appendix A: Cover Sheet—Supported Employment

Fidelity Scale and
General Organizational Index . . . . . . . . . . 25

Appendix B: Supported Employment Fidelity Scale

and Score Sheet . . . . . . . . . . . . . . . . . . . . 29

Appendix C: Supported Employment Fidelity Guide . . . . 37

Appendix D: General Organizational Index

and Score Sheet . . . . . . . . . . . . . . . . . . . . 47

Appendix E: General Organizational Index Protocol . . . . 53

Appendix F: Outcomes Report Form . . . . . . . . . . . . . . . 65

Appendix G: Instructions for the Outcomes Report Form . .69

Appendix H: Assessor Training and Work

Performance Checklist. . . . . . . . . . . . . . . . 77
 Evaluating Your Program

Why Evaluate Your Supported Employment Program

Key stakeholders who are implementing To answer the frst question, collect
Supported Employment (SE) programs process measures (by using the SE
may fnd themselves asking two questions: Fidelity Scale and General Organizational
n Has SE been implemented as planned?
Index). Process measures capture how
services are provided. To answer the
n Has SE resulted in the expected second question, collect outcome
outcomes? measures. Outcome measures capture the
Asking these two questions and using the results or achievements of your program.
answers to help improve your SE program
are critical for ensuring the success of As you prepare to implement SE, we
your SE program. strongly recommend that you develop a
quality assurance system using both
process and outcome measures to monitor
and improve the quality of the program
from the startup phase and continuing
through the life of the program.

Evaluating Your Program 3 Why Evaluate Your Supported Employment Program?

Why you should collect Why you should collect
process measures outcome measures

Process measures give you an objective, structured While process measures capture how services are
way to determine if you are delivering SE services provided, outcome measures capture the program’s
in the way that research has shown will result in results. Every mental health service intervention
desired outcomes. Process measures allow agencies has both immediate and long-term consumer goals.
to understand whether they are providing services In addition, consumers have goals for themselves,
that are faithful to the evidence-based practice which they hope to attain by receiving mental
model. Programs that adhere closely to the SE health services. These goals translate into outcomes
model are more effective than those that do not and the outcomes translate into specifc measures.
follow the model. Adhering to the model is called
fdelity. Consumer outcomes are the bottom line for mental
health services, like proft in business. No successful
Collecting process measures is an excellent way to businessperson would assume that the business was
diagnose program weaknesses while helping to proftable just because employees work hard.
clarify program strengths. Once SE programs reach
high fdelity, ongoing monitoring allows you to test
local innovations while ensuring that programs do
not drift from the core principles of the evidence- Why develop a quality
based practice.
assurance system
Process measures also give mental health
authorities a comparative framework to evaluate the In your mental health system, you should develop a
quality of SE programs across the state. They allow quality assurance system that collects not only
mental health authorities to identify statewide process measures, such as those on the SE Fidelity
trends and exceptions to those trends. Scale and General Organizational Index, but also
outcome measures such as those specifed above to
show the effect of SE. Developing a quality
assurance system will help you do the following:

Research Has Shown That You Can n Diagnose your program’s strengths and
Expect These Outcomes weaknesses;
n Formulate action plans for improving your
n Higher rates of competitive employment program;
n More work hours n Help consumers achieve their goals for recovery;
n Higher wages and

n Improved symptoms n Deliver mental health services both effciently

and effectively.
n Improved self-esteem

n Increased consumer satisfaction with fnances

(Bond et al., 2001; Mueser et al., 1997)

Why Evaluate Your Supported Employment Program? 4 Evaluating Your Program

 Evaluating Your Program

Conduct a Readiness Assessment

Let’s assume that administrators and SE Answering these questions will help you
leaders have read Building Your Program. generate an ongoing to-do list (or
Your new SE team has completed implementation plan) to guide your steps
Training Frontline Staff. How do you in implementing SE. Your answers will
know if you are ready to begin providing also help you understand the components
SE services to consumers? of SE services that are in place and the
work that still remains.
The Readiness Assessment on the next
page will help quality assurance team
members, advisory group leaders, and SE
leaders track the processes and
administrative tasks required to develop
an SE program.

Evaluating Your Program 5 ConductAppendix

a Readiness
A: Cover
Assessment
Sheet
Readiness Assessment

Check any areas that you feel you do NOT completely understand.

q Which practitioners will provide SE? q How will you generate SE referrals?
q Who will supervise employment specialists and direct q How will you advertise SE to consumers, families,
the SE program? and others?

q What are the roles of SE leader and employment q What are your procedures for completing Vocational
specialists? Profles?

q Who will provide benefts counseling? q What are your procedures for completing Individual
Employment Plans?
q What is the size of employment specialists’ caseloads?
q How will the SE team communicate with other
q What is the size of the SE leader’s caseload?
treatment team members?
q What is the SE supervisory structure (for example,
how often does the SE leader meet with employment
q How will families or other supporters be involved
in SE?
specialists and the agency director)?
q How does the SE team relate to advisory groups?
q How will you supervise your employment specialists?
q How will you measure your program’s fdelity to the
q What are the admission criteria for your program?
SE model?
q What is your referral process?
q How does the system for collecting consumer outcome
data work?

Note areas where you still are unclear or have questions. Arrange to speak to an expert consultant or experienced SE leader.

Appendixa A:
Conduct Readiness
Cover Sheet
Assessment 6 Evaluating Your Program
 Evaluating Your Program

Conduct a Process Assessment

In addition to the Readiness Assessment, Once your SE program has achieved high
you should conduct your frst process fdelity to the evidence-based model,
assessment before you begin providing employment specialists may tailor the
any SE services. By doing so, you will program to meet individual needs of the
determine whether your agency has core community. If you continue to use process
components of SE in place. During the evaluations along with outcomes
frst 2 years of implementing your monitoring, you will be able to understand
program, plan to assess your program the extent to which your changes result in
every 6 months. your program’s departure from model
fdelity and whether the changes positively
After your program has matured and or negatively affect consumers.
achieved high fdelity, you may choose to
conduct assessments once a year. Agencies
that have successfully implemented SE
indicate that you must continue to
evaluate the process to ensure that you do
not revert to previous practice patterns.

Evaluating Your Program 7 Conduct

Appendix
a Process
A: Assessments
Cover Sheet
 Who can conduct process assessments?
How to use process measures
We recommend enlisting two assessors to conduct
Two tools have been developed to monitor how SE your process assessment. Data collected by two
services are provided: assessors simultaneously increases the likelihood
that information will be reliable and valid.
n The SE Fidelity Scale; and
n General Organizational Index. Agencies who have successfully implemented SE
You may administer both tools at the same time. programs have taken different approaches to
identify assessors. Some agencies train SE Advisory
The SE Fidelity Scale has 15 program-specifc Committee members as assessors and rotate the
items. Each item is rated on a 5-point scale, ranging responsibility of completing assessments. Others
from 1 (meaning not implemented) to 5 (meaning have pre-existing quality assurance teams and
fully implemented). The items assess whether the simply designate members of the team to complete
SE program is provided as the SE model prescribes. the assessments. In other cases, the mental health
authority has designated staff to conduct
The General Organizational Index is a second set of assessments.
process measures that has been developed. In
contrast to fdelity scales, which are specifc to each Assessments can be conducted either internally by
evidence-based practice, the General Organizational your agency or by an external review group.
Index can be used when implementing any External review groups have a distinct advantage
evidence-based practice. It measures agency-wide because they use assessors who are familiar with
operating procedures that have been found to affect the SE model but, at the same time, are
agencies’ overall capacity to implement and sustain independent. The goal is to select objective and
any evidence-based practice. competent assessors.

For the SE Fidelity Scale and General Although we recommend using external assessors,
Organizational Index, see Appendices B and D. You agencies can also use fdelity scales to rate their own
can also print these forms from the CD-ROM in SE programs. The validity of these ratings (or any
the KIT. ratings, for that matter) depends on the following:

About the process measures that are included in the KIT

Quality assurance measures have been developed and are included in all Evidence-Based Practices KITs. The
SE Fidelity Scale has one of the longest histories. Developed and described by Bond, Becker, Drake, and
Vogler (1997), it was formerly known as the Individual Placement and Support (IPS) Fidelity Scale. It has
demonstrated discriminant and predictive validity and has been widely adopted by many state and local
agencies throughout the United States and internationally.

The scale has been found to differentiate between well-established programs following the SE model (or IPS)
and other types of vocational programs (Bond et al., 2001). Regarding predictive validity, one study found a
strong correlation (.76) between fdelity, as rated by the SE Fidelity Scale and competitive employment rates
(Becker et al., 2001).

The General Organizational Index, developed by Robert Drake and Charlie Rapp, is a newly developed scale.
This scale has undergone multiple revisions based on feedback gathered during the 3-year pilot testing of the
KIT materials.

Appendixa A:
Conduct Process
CoverAssessments
Sheet 8 Evaluating Your Program
n The knowledge of the person making the ratings; n Observing an SE treatment team and supervisory
meeting; and
n Access to accurate information pertaining to the
ratings; and n Conducting a chart review.
n The objectivity of the ratings. Collecting information from multiples sources
helps assessors more accurately capture how
If you do conduct your assessments using internal
services are provided. A day-long site visit is the
staff, beware of potential biases of raters who are
best way to learn this information.
invested in seeing the program look good or who do
not fully understand the SE model. It is important
for ratings to be made objectively and that they be The following suggestions outline steps in the
based on hard evidence. assessment process.

Circumstances will dictate decisions in this area,

but we encourage agencies to choose a review
process that fosters objectivity in ratings, for
example, by involving a practitioner who is not Before the process assessment
centrally involved in providing SE services. Only
people who have experience and training in n n n Prepare your assessment questions
interviewing and data collection procedures The SE Fidelity Guide in Appendix C
(including chart reviews) should conduct provides questions that you may use to
assessments. Additionally, assessors need to collect information during your
understand the nature and critical ingredients of assessment visit.
the evidence-based model.
A detailed protocol has also been
If your agency chooses to use a consultant or trainer developed to help you understand each
to help implement your SE program, involving that item on the General Organizational Index
person in the assessment process will enhance the and the rationale for including it, guidelines
technical assistance you receive. Whichever for the types of information to collect, and
approach you choose, we encourage you to make instructions for completing your ratings.
these decisions early in the planning process. For a For the General Organizational Index
checklist to help evaluate assessors’ training and Protocol, see Appendix E.
work performance, see Appendix H.
Use the SE Fidelity Guide and General
How to conduct process assessments Organizational Index protocol to help
prepare the questions that you will ask
A number of activities take place before, during, during your assessment visit.
and after a process assessment. In general,
assessments include the following: While we expect that quality assurance
teams will select which outcome measures
n Interviewing administrators, the SE leader, meet your agency’s needs, you should use
employment specialists, benefts specialists, the SE Fidelity Scale and General
consumers, and families; Organizational Index in full. Collecting data
for all the items on these scales will allow
n Interviewing other agency staff (such as case your agency to gain a comprehensive
managers, therapists, psychiatrists, or nurses); understanding of how closely your SE
n Shadowing employment specialists; services resemble the evidence-based model.

Evaluating Your Program 9 Conduct

Appendix
a Process
A: Assessments
Cover Sheet
n n n Create a timeline for the assessment n Roster of employment specialists—
List all the necessary activities leading up to (roles, full-time equivalents [FTEs]);
and during the visit and create a timeline n Roster of SE consumers for each
for completing each task. Carefully employment specialist;
coordinating efforts, particularly if you have
n Roster of SE consumers who have left
multiple assessors, will help you to complete
the program in the past 6 months;
your assessment in a timely fashion.
n Copy of the agency’s SE brochure or
mission statement; and
n n n Establish a contact person
n Total number of consumers that the SE
Have one key person in the SE program program served in the past 6 months.
arrange your visit and communicate
beforehand the purpose and scope of your Reassure the SE leader that you will be able
assessment to people who will participate to conduct the assessment, even if all of the
in interviews. Typically, this contact person requested information is unavailable.
will be the SE leader. Indicate that some information is more
critical (for example, number of
Exercise common courtesy and show employment specialists and SE consumers)
respect for competing time demands by than other information.
scheduling well in advance and making
reminder calls to confrm interview dates Tell the contact person that you must
and times. observe an SE treatment team meeting and
group supervision meeting during your visit.
These are important factors in determining
n n n Establish a shared understanding with when you should schedule your visit.
the SE team

The most successful assessments are those

n n n Alert your contact person that you will
in which assessors and the SE team share
need to sample 10 charts
the goal of understanding how the program
is progressing according to evidence-based From an effciency standpoint, it is
principles. If administrators or employment preferable that the charts be drawn
specialists fear that they will lose funding or beforehand, using a random selection
look bad if they don’t score well, then the procedure. There may be a concern that
accuracy of the data may be compromised. the evaluation may be invalidated if
The best agreement is one in which all employment specialists handpick charts or
parties are interested in learning the truth. update them before the visit. If you both
understand that the goal is to learn how the
program is implementing services, this is
n n n Indicate what you will need from less likely to occur.
respondents during your visit

In addition to the purpose of the Additionally, you can further ensure

assessment, briefy describe what random selection by asking for 20 charts
information you need, who you must speak and randomly selecting 10 to review. Other
with, and how long each interview will take options include asking the SE program for
to complete. The visit will be most effcient a de-identifed list of consumers (that is,
if the SE leader gathers beforehand as much with names removed) and using the list to
of the following information as possible: choose 10 charts to review.

Appendixa A:
Conduct Process
CoverAssessments
Sheet 10 Evaluating Your Program
 If the program only has one employment n The second and third forms are score
specialist with fewer than fve consumers sheets for the two scales. They help
on their caseload, then review the charts you compare assessment ratings from
for all SE consumers. one time period to the next. They may
also be useful if you are interested
in graphing results to examine your
n n n Clarify reporting procedures
progress over time.
With the appropriate people (agency
For the SE Fidelity Scale and General
administrators, the mental health authority,
Organizational Index instruments, cover
or the SE leader), clarify who should
sheet, and score sheets, see Appendices A,
receive a report of the assessment results.
B, and D. You can also print these forms
Recipients may include the following:
from the CD-ROM in the KIT.
n Agency administrators;
n Members of the agency’s quality
assurance team; During your assessment visit
n Members of the SE Advisory
Committee; n n n Tailor your terminology

n The SE leader; To avoid confusion during your interviews,

tailor the terminology you use. For
n Employment specialists; and example, an SE program may use member
n Consumers and families. for consumer or clinician for practitioner.
Every agency has specifc job titles for
Assessors should also clarify how the particular staff roles. By adopting the local
agency would like the report to be terminology, you will improve
distributed. For example, assessors may communication.
mail or fax the report and follow up to
discuss the results in a meeting or by
n n n Conduct your chart review
conference call.
It is important that your chart review is
conducted from a representative sample of
n n n Organize your assessment materials charts. When you begin your chart review,
Three forms have been created to help note whether your sample refects both
you conduct your assessment: consumers who are already working and
those who are in the process of searching
n The frst form is a cover sheet for
for a job. You should also note whether
the SE Fidelity Scale and General
your sample includes consumer charts
Organizational Index, which is
from each employment specialists’
intended to help you organize your
caseload. If your random sample is not
process assessment. It captures general
representative in this manner, consider
descriptive information about the
supplementing your sample with selected
agency, data collection, and community
charts that will increase its
characteristics.
representativeness.

Evaluating Your Program 11 Conduct

Appendix
a Process
A: Assessments
Cover Sheet
 Within each chart, examine the referral, n n n If discrepancies between sources occur,
Vocational Profle, and Individual query the SE leader
Employment Plan. If more than one The most common discrepancy is likely to
treatment plan exists (such as a clinical occur when the SE leader’s interview gives
treatment plan and Individual a more idealistic picture of the team’s
Employment Plan), review both to see if functioning than the chart and
similar vocational goals are refected in observational data. For example, on the SE
each plan. Fidelity Scale, Community-Based Services
(Item 8), assesses the amount of time
Review recent Progress Notes to employment specialists spend providing
understand the amount and type of contact SE services in the community. The chart
employment specialists have with the review may show that consumer contact
consumers on their caseloads. If Progress occurs largely in the offce, while the SE
Notes are not integrated into consumer leader may indicate that employment
charts, then ask if employment specialists specialists spend most of their time
have any additional fles that you may working in the community.
review.
To understand and resolve this
In some cases, a lag may exist between discrepancy, the assessor should ask the SE
when a service is rendered and when it is leader by saying:
documented in the consumer’s chart. To
get the most accurate representation of Our chart review shows 50 percent of
services rendered when you sample chart consumer contact is offce-based, but your
data, try to gather data from the most estimate of community-based services is
recent time period in which 75 percent. Would you help us understand
documentation is completed in full. the difference?

To ascertain the most up-to-date time Often the SE leader can provide
period, ask the SE leader, employment information that will resolve the
specialists, or administrative staff. Avoid discrepancy.
getting an inaccurate sampling of data
where offce-based services might be
n n n Before you leave, check for missing data
charted more quickly than services
rendered in the feld. Fidelity scales should be completed in full,
with no missing data on any items. Check
in with the SE leader at the end of the visit
to collect any additional information you
may need.

Appendixa A:
Conduct Process
CoverAssessments
Sheet 12 Evaluating Your Program
After your assessment visit

n n n Follow up n n n Complete scales independently

It is important to collect any missing data If you have two assessors, both should
before completing your rating. If necessary, independently review the data collected and
follow up on any missing data (for example, rate the scales. They should then compare
by calling or sending an e-mail). This would their ratings, resolve any disagreements,
include discussing with the SE leader any and devise a consensus rating.
discrepancies between data sources that you
notice after you’ve completed the visit.
n n n Complete the score sheets
Tally the item scores and determine which
n n n Score your scales level of implementation was achieved.
Use the SE Fidelity Guide and General
Organizational Index Protocol in
Appendices C and E to score the SE
program. If you assess an agency for the
frst time to determine which components
of SE the agency already has in place, some
items may not apply. If an item cannot be
rated, code the item as “1.”

Evaluating Your Program 13 Conduct

Appendix
a Process
A: Assessments
Cover Sheet
 Evaluating Your Program

Monitor Outcomes
Unlike process measures which must be n Competitive employment;
used in full to comprehensively understand
n Independent living;
how services are provided, you must decide
which outcome measures will be most n Educational involvement;
informative for your SE program. Initially, n Self-esteem; and
your outcomes monitoring system should
be simple to use and maintain. Complexity n Satisfaction with fnances.
has doomed many well-intended attempts These few outcomes refect the primary
to collect and use outcome data. goals of SE. Specifcally, goals of SE are to
help consumers move forward in their
One way to simplify is to limit the number process of recovery and become integrated
of outcome measures used. Select your in the community by obtaining competitive
outcome measures based on the type of employment. For this reason, it is
information that will be most useful to your important for you to capture the outcome
agency. Based on the research literature, of competitive employment in a way that is
we suggest that you monitor a core set of most useful for your program.
outcomes such as the following:

Evaluating Your Program 15 Monitor Outcomes

 What Is the Consumer Outcomes Monitoring Package?

Sponsored in part by the Substance Abuse and Mental Health Services Administration (SAMHSA), the
Consumer Outcomes Monitoring Package (COMP) was designed by a team at the School of Social Welfare,
University of Kansas. This computer application allows agencies to choose from a pre-established list of
outcomes developed for each evidence-based practice. Data may be entered for the chosen outcomes, and
reports can be generated quarterly or monthly. The COMP also allows agencies to view their outcomes data
using a variety of tables and graphs.

The designers of COMP tried to make the computer application as easy and as fexible to use as possible.
You may access COMP through the Web. Agencies can download the computer application and print out
Installation Instructions and a User Manual, which provides defnitions and forms.

To download COMP:

n Go to http://research.socwel.ku.edu/ebp

n Click on the link to the download page.

n Click the links to download the Installation Instructions and a User Manual.
n Follow the instructions to install the application.

For data to be useful, they must be valid. That is, Develop procedures
the data must measure what they are supposed to
measure. Thus, the outcomes must be few and Agencies may choose to develop the outcomes
concrete for employment specialists to focus on key portion of their quality assurance system from
outcomes, to understand them in a similar way, and scratch or use existing outcomes monitoring
to make their ratings in a consistent and error-free systems. A number of electronic evaluation
fashion. programs are available to help you develop
comprehensive, integrated, user-friendly outcome
To enhance validity, we recommend using simple monitoring systems. Examples include the
ratings initially (such as, Did the consumer hold a following:
competitive job in this quarter?), rather than more
detailed ones (such as, How many hours during this n Publicly available tools such as the Consumer
quarter did the consumer work competitively?). Outcomes Monitoring Package (see the next
Limiting your outcome measures to concrete page), and Decision Support 2000+ Online
measures will also allow you to collect data from (www.ds2kplus.org); or
employment specialists.
n Various commercially available products.
When deciding whether to use an existing
outcomes monitoring package or to design your
own, it is important to keep your organization’s
capabilities in mind. The system must not create
undue burden for employment specialists, and it
must provide information to them that is useful in
their jobs.

Monitor Outcomes 16 Evaluating Your Program

The system should ft into the workfow of the Computer software that allows for data entry and
organization, whether that means making ratings on manipulation (such as Microsoft Access, Excel, or
paper, using the COMP computer application, or Lotus) makes tabulating and graphing data easier
developing your own outcomes monitoring than if it is done by hand. A computerized system
package. Start with whatever means are available for data entry and report generation presents a
and expand the system from there. In the clear advantage and it may be the goal, but do not
beginning, you may collect data with a simple wait for it. Feedback does not have to come from a
report form and you can report hand-tallied sophisticated computer system to be useful. It is
summaries to employment specialists. more important that it is meaningful and frequent.
For a sample Outcomes Report Form, which is an
example of a simple, paper-based way to collect
participation and outcome data regularly, see
Appendix F. For instructions for using the
Outcomes Report Form, see Appendix G.

Expanding Your Outcome Measures

Once you have established your core outcomes consumers and families can give employment specialists
monitoring system, learned how to routinely collect valuable feedback.
data, and are accustomed to using it to improve
your SE program, you will be ready to expand your We recommend the following surveys for collecting
outcomes measures. Consider adding other SE information from consumers and families:
outcome measures such as the types of
employment positions, number of hours worked n The Mental Health Statistics Improvement Program
(MHSIP) Consumer Satisfaction Survey at www.
per week, number of weeks worked, wages, place
mhsip.org
of employment, date job started/ended, and reason
for job termination. n Recovery measurement instruments such as those
described in Measuring the Promise: A Compendium
Furthermore, consider asking consumers and of Recovery Measures, Volume II, available through
families for input on how to improve your SE http://www.tecathsri.org
program, both practically and clinically. Consumers
and families are important informants for agencies
It is diffcult to obtain a representative sample of
that are seeking to improve outcomes. Agencies
consumer and family respondents since mailed surveys
may want to know the following:
are often not returned and interviews may only be done
n If consumers and families are satisfed with with people who are cooperative and easy to reach.
their services; Samples that are not representative may be biased.

n How services have affected their quality of life; Avoid bias in your consumer and family data by using a
and variety of mechanisms to conduct your assessments. For
example, consider combining feedback collected
n Whether consumers believe the services are
through surveys with that obtained from focus groups.
helping them achieve their recovery goals.
Another option is to hire a consultant to conduct
qualitative interviews with a small group of consumers
While collecting data from consumers and families
or families.
requires more staff time than the information that
may be reported quickly by employment specialists,

Evaluating Your Program 17 Monitor Outcomes

How often should you collect How should you identify data collectors?
outcomes data?
Agency administrators or mental health authorities
Plan to monitor the outcomes for consumers in may assign the responsibility for collecting
your SE program every 3 months and share the outcomes data to the following:
data with your SE team. Collecting data at regular
n The SE leader;
and short intervals will enhance the reliability of
your outcomes data. While we recommend that you n Members of the SE advisory committee;
design a system for collecting outcomes early in the n The quality assurance team;
implementation process, SE programs should not
expect to see the desired results until the SE n Independent consultants, including consumers
program is fully operational. Depending on and family members; and
resources available to your program, this may take n Other staff.
anywhere from 6 to 18 months to accomplish.
Unlike collecting process measures, collecting
outcome measures does not require a daylong
assessment process. Many standard outcome
measures, such as competitive employment and
educational involvement, will be information that
employment specialists can report from their daily
work with consumers.

It is important to develop a quick, easy, standardized

approach to collect outcomes data. For example,
create a simple form or computer database that
employment specialists can routinely update.

Monitor Outcomes 18 Evaluating Your Program

 Evaluating Your Program

Using Data to Improve Your Program

As you develop a quality assurance system, n n n Create reports from
SE leaders and employment specialists your assessments
will weave it into the fabric of their daily For your process data, in addition
routines. Process assessments will give you to completing the SE Fidelity
a window into the demanding work done Scale, General Organizational
every day. Outcome reports will give you Index, and score sheets, assessors
tangible evidence of the use and value of should write a report explaining
services, and they will become a basis for their scores. The report should
decisionmaking and supervision. include the following:

At some point, your SE team members n An interpretation of the results

of the assessment;
may wonder how they did their jobs
without an information system as n Strengths and weaknesses of
they come to view it as an essential the SE program; and
ingredient of well-implemented
n Clear recommendations to
evidence-based practices. help the SE program improve.

Evaluating Your Program 19 Using DataAppendix

to ImproveA:Your
Cover
Program
Sheet
 The report should be informative, factual, n n n Use tables and graphs to understand
and constructive. Since some process your outcomes data
measures assess adherence to the evidence- After the frst process and outcomes
based model at both the agency and assessments, it is often useful to provide a
program staff levels, remember to target visual representation of a program’s progress
recommendations to administrators, SE over time. We recommend that you use
leaders, and employment specialists. tables and graphs to report the results.
When summarizing outcomes data, start
with simple, easy-to-read reports. Then let By graphing your SE fdelity score, you have
experience determine what additional a visual representation of how your SE
reports you need. You can design your program has changed over time. For an
reports to give information about individual example, see Figure 1. For your process
consumers, a single employment specialist’s data, you may simply graph the results using
caseload, or the program as a whole. For a spreadsheet and include this in your report.
example, reports generated for individual
When your program shows greater fdelity
consumers may track the consumer’s
over time, the graph will display it and
participation in specifc SE services and
reinforce your efforts. Additionally, as you
outcomes over time. You could enter these
can see in Figure 1, the graph allows you
reports in consumers’ charts and they could
to quickly compare one team to another. In
be the basis for discussions about
this example, Team A struggled in the frst
consumers’ progress.
6 months. Understanding Team A’s
progress compared to Team B’s allowed
the teams to partner and share strategies.

Figure 1. SE Fidelity Over Time

75 ■
▲
70 ■ ■
■
65 ▲
60 ▲ ▲ Team A
■ Team B
55
50 ▲
■

45
▲
40
6 6 06 6 06 6 7 7 07 7 7
b -0 r-0 - g-0 t- c-0 b-0 r-0 - g-0 t-0
Fe Ap Jun Au Oc De Fe Ap Jun Au Oc

Date of Assessment

Note: 66 – 75 = good implementation

56 – 64 = fair implementation
55 and below = not evidence-based practice

Appendix
Using DataA:
toCover
Improve
Sheet
Your Program 20 Evaluating Your Program
 Consequently, Team A improved Example 2: Movement tables
dramatically over the next 6-month period.
Tables that track changes in consumer characteristics
Another feature of graphing assessment (called movement tables) can give you a quick
scores is to examine the cutoff score for reference for determining service effectiveness. For
fair (56) or good (66) implementation. Your example, Table 2 compares consumers’ employment
program can use these scores as targets. status between two quarters.

Table 2: Sample Movement Table

Here are three examples of tables and graphs
To FY ’06 Qtr 3
that can help you understand and use your
outcomes data. Unemployed Part-time Full-time Total
employment employment

From:
FY
Unemployed 2 1 3 6
Example 1: Periodic summary tables ‘01
Part-time
Qtr:
employment
3 8 3 14
2
Periodic summary tables summarize your outcomes data
Full-time
each quarter and address these kinds of questions: employment
1 3 100 104

n How many consumers participated in our SE Totals 6 12 106 124

program during the last quarter?
Above the diagonal
n How many consumers worked competitively during
Below the diagonal
the last quarter?
Within the diagonal
n How did the hospitalization rate for those
participating in SE compare to the rate for
consumers in standard treatment? To create this table, the data were collapsed into
Agencies often use this type of table to understand the three broad categories. The vertical data cells
consumer participation or to compare actual results with refect the employment status for consumers for the
agency targets or goals. These tables are also frequently beginning quarter. The horizontal data cells refect
used to describe agencies’ services in annual reports or the most recent quarterly information. The
for external community presentations. employment status categories are then ordered
from the least desirable (unemployed) to the most
Table 1: Sample Periodic Summary Table of Enrollment
desirable (full-time employment).
in Evidence-Based Practices
Not Eligible but Enrolled Percent
eligible NOT in EBP of eligible
service consumers The data in this table are presented in three colors.
enrolled
The purple cells are those above the diagonal, the
Supported 0 30 60 67 pink cells are those below the diagonal, and the
Employment
white cells are those within the diagonal. The data
cells above the diagonal represent consumers who
Assertive 30 25 90 78 moved into a more desirable employment status
Community
Treatment between quarters. As you can see, one consumer
moved from unemployed to part-time employment,
This agency provides both Supported Employment (SE) and three consumers moved from unemployed to full-
Assertive Community Treatment (ACT). The SE team time employment and three consumers moved
identifed 90 consumers for the program. Of those, 60 from part-time to full-time employment. These
receive SE, while 30 consumers are eligible but receive seven consumers (6 percent of the 124 consumers
another service. Consequently, 67 percent of consumers in the program) moved to a more desirable
who are eligible for the SE program currently participate in employment status between quarters.
the program.

Evaluating Your Program 21 Using DataAppendix

to ImproveA:Your
Cover
Program
Sheet
The data reported in the diagonal cells ranging consumers, caseloads, or programs for comparison.
from the upper left quadrant to the lower right Figure 2 presents an example of a longitudinal plot
refect consumers who remained in the same comparing critical incidents for one SE team over
employment status between quarters. As you can an 11-month period.
see, two consumers were unemployed for both
quarters of this report, eight consumers remained This plot reveals that with the exception of private
in part-time employment, and 100 remained in full- psychiatric hospitalizations, all other critical
time employment. These 110 consumers (89 incidents appear to be going in a positive direction
percent of the 124 consumers in the program) (that is, there is a reduction in incidence).
remained stable between quarters.
Longitudinal plots are powerful feedback tools
because they permit a longer range perspective on
The cells below the diagonal line represent
participation and outcome, whether for a single
consumers who moved into a less desirable
consumer or a group of consumers. They enable a
employment status between quarters. Three
meaningful evaluation of the success of a program,
consumers moved from part-time employment to
and they provide a basis for setting goals for
unemployed, one moved from full-time
future performance.
employment to unemployed and three moved from
full-time employment to part-time employment.
These seven consumers (6 percent of the 124
consumers in the program) experienced some
setbacks between quarters. The column totals show Figure 2. Sample Longitudinal Plot for Monthly Frequency
of Negative Incidents for Consumers
the number of consumers in a given employment
16
status for the current quarter, and the row totals
✖
show the prior quarter. 14
✖
12 ✖ ✖
You can use movement tables to portray changes in ✖
10 ✖
outcomes that are important to consumers,
✖ ✖ ✖
supervisors, and policymakers. The data may 8 ✖
stimulate discussion around the progress that ✖
6 ▲
consumers are making or the challenges with which ▲ ▲
they are presented. 4 ◆ ▲ ▲
◆ ■ ▲ ◆ ◆ ▲ ▲ ▲
2 ◆
■ ■ ▲ ◆
■ ◆ ■
Example 3: Longitudinal plots ■ ◆ ■ ▲ ◆
■ ■ ◆
0 ■ ◆ ■
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov
A longitudinal plot is an effcient and informative 2007
way to display participation or outcome data for
more than two successive periods. The goal is to ■ Incarcerated
view performance in the long term. You can use a ▲ In-Patient SA
longitudinal plot for a consumer, a caseload, a ✖ Priv. Psych. Hosp.
◆ State Psych. Hosp.
specifc EBP, or an entire program. A single plot
can also contain longitudinal data for multiple

Appendix
Using DataA:
toCover
Improve
Sheet
Your Program 22 Evaluating Your Program
n n n Share your results community, staff from your mental health
The single factor that will most likely authority, and consumers and family
determine the success of an information advocates can be valuable. Through these
system is its ability to give useful and timely channels, you may develop support for the
feedback to key stakeholders. It is fne to SE program, increase consumer
worry about what to enter into an participation, and raise private funds for
information system, but ultimately its worth your agency.
is in converting data into meaningful
information. For example, the data may Sharing results internally
show that 20 consumers worked in a
competitive job during the past quarter, but Agencies may distribute reports during all
it is more informative to know that this staff and manager-level meetings to keep
represents only 10 percent of the consumers staff across the agency informed and
in the SE program. engaged in the process of implementing SE.
Agencies with successful SE programs
For information to infuence practice, it highlight the importance of developing an
must be understandable and meaningful, understanding and support for the
and it must be delivered in a timely way. In evidence-based model across the agency.
addition, the quality assurance system must
tailor the information to suit the needs of Additionally, integrating consumer-specifc
various users and to answer their questions. reports into clinical charts may help you
monitor consumers’ progress over time.
Reporting consumer-specifc outcomes
Sharing results with employment specialists information at the treatment team
After each assessment, dedicate time during meetings also helps keep the team focused
a supervisory meeting to discuss the results. on consumers’ vocational goals.
Numbers that refect above average or
exceptional performance should trigger
Sharing results with consumers
recognition, compliments, or other rewards.
and families
Data that refect below average
performance should provoke a search for Agencies may highlight assessment results
underlying reasons and should generate in consumer and family meetings.
strategies that offer the promise of Increasing consumers’ and families’
improvement. By doing this regularly, SE understanding of the SE program may
leaders will create a learning organization motivate them to participate in the
characterized by adaptive responses to treatment process and build trust in the
information that aim to improve consumer-provider relationship.
consumer outcomes.
Also, sharing results may create hope and
enthusiasm for your SE program. Sharing
Sharing results with your SE Advisory
information motivates people and
Committee or quality assurance team
stimulates changes in behavior. Sharing the
You may also use this information to keep results of your assessments with a variety of
external stakeholders engaged. Sharing stakeholders is the key to improving
information with vested members of the your program.

Evaluating Your Program 23 Using DataAppendix

to ImproveA:Your
Cover
Program
Sheet
 Evaluating Your Program

Appendix A: Cover Sheet—

Supported Employment Fidelity Scale
and General Organizational Index

Evaluating Your Program 25 Appendix A: Cover Sheet

Cover Sheet: Supported Employment Fidelity Scale and
General Organizational Index

Assessors’ names: ___________________________________________ Today’s date: _____/___/_____

___________________________________________

___________________________________________

Program name (or program code): __________________________________

Agency name: ___________________________________________

Agency address: ______________________________________________________________________________

Street

______________________________________________________________________________
City State ZIP code

SE leader or contact person: _________________________________________________________________________

Names of the employment specialists: __________________________________________________________________

Telephone: ( ___) _____–_______ E-mail: _________________________________________________

Sources used for assessments: q Chart review: Number reviewed: _______

q SE treatment team observation

q SE supervisory meeting observation

q SE leader interview

q Employment specialist interviews: Number interviewed: ____

q Consumer interviews: Number interviewed: ____

q Family member interviews: Number interviewed: ____

q Other staff interviews: Number interviewed: ____

q Brochure review

q Other ______________

Number of employment specialists: _______

Number of current SE consumers: _______
Number of SE consumers who left the program in the past 6 months: _______
Number of consumers served in the past 6 months: _______

Funding source: ______________________________________________________________________________

Agency location: q Urban

q Rural

Date program was started: _____/_____/______

Contact with local or state Vocational Rehabilitation agencies: q None
q Minimal
q Regular

Evaluating Your Program 27 Appendix A: Cover Sheet

 Evaluating Your Program

Appendix B: Supported Employment Fidelity Scale

and Score Sheet

Evaluating Your Program 29 Appendix B: Fidelity Scale and Score Sheet

Supported Employment Fidelity Scale
Ratings / Anchors
Criteria 1 2 3 4 5
Staffng

1. Caseload: A ratio of 81 or A ratio of 61 to A ratio of 41 to A ratio of 26 to A ratio of 25 or

more consumers 80 consumers 60 consumers 40 consumers fewer consumers
Employment specialists manage
per employment per employment per employment per employment per employment
caseloads of up to 25 consumers.
specialist, or specialist specialist specialist specialist
Cannot rate due to
no ft

2. Vocational services staff: Employment Employment Employment Employment Employment

specialists provide specialists provide specialists provide specialists provide specialists provide
Employment specialists provide
nonvocational nonvocational nonvocational nonvocational only vocational
only vocational services.
services such as case services such as case services such as case services such as case services
management 80% or management about management about management about
more of the time, or 60% of the time 40% of the time 20% of the time
Cannot rate due to
no ft

3. Vocational generalists: Employment Employment Employment Employment Employment

specialists provide specialists maintain specialists provide specialists provide specialists carry
Each employment specialist
only vocational caseloads but 1 aspect of the 2 or more phases out all phases of
carries out all phases of
referrals to other refer consumers to vocational service of vocational service vocational service
vocational service including
vendors or programs, other programs for but not the entire
engagement, assessment, job
or vocational service service
development, job placement,
job coaching, and follow-along Cannot rate due to
supports. no ft

Organization

1. Integration of rehabilitation Employment Employment Employment Employment Employment

with mental health treatment: specialists are part specialists attend specialists have specialists are specialists are
Employment specialists are of a vocational treatment team several contacts attached to 1 attached to 1
program, separate meetings once per with treatment or more case or more case
part of the mental health
from the mental month team members each management management
treatment teams with shared
health treatment. month and attend treatment teams treatment teams
decisionmaking. They attend
No regular direct 1 treatment team with shared with shared
regular treatment team meetings
contact with mental meeting per month decisionmaking; decisionmaking;
(not replaced by administrative
health staff, only attend weekly attend 1 or more
meetings) and have frequent
telephone or 1 face- treatment team treatment team
contact with treatment team
to-face contact per meetings meetings per week
members.
month, or and have at least 3
consumer-related
Cannot rate due to
case manager
no ft
contacts per week

2. Vocational unit: Employment Employment Employment Employment Employment

Employment specialists function specialists are not specialists have the specialists have the specialists form a specialists form a
as a unit rather than a group of part of a vocational same supervisor but same supervisor vocational unit and vocational unit with
practitioners. They have group unit, or do not meet as a and discuss cases discuss cases between group supervision at
supervision, share information, group between each other; each other; they least weekly; provide
Cannot rate due to
and help each other with cases. they do not provide provide services for services for each
no ft
services for each each other's cases other’s cases and
other’s cases backup and support
for each other

Evaluating Your Program 31 Appendix B: Fidelity Scale and Score Sheet

Supported Employment Fidelity Scale
Ratings / Anchors
Criteria 1 2 3 4 5
3. Zero-exclusion criteria: Consumers are Some eligibility Some eligibility All consumers are All consumers are
No eligibility requirements such screened out criteria, or criteria, or eligible and services encouraged to
as job readiness, lack of substance based on formal or Vocational staff who Vocational staff
are voluntary. participate, and
abuse, no history of violent informal eligibility Referral sources are
make consumer who provide the Several sources
behavior, minimal intellectual requirements, or limited
referrals to other vocational service (self-referral, family
functioning, and mild symptoms Case managers frst vocational programs screen referrals members, self-help
screen referrals, or screen referrals groups, etc.) solicit
referrals
Cannot rate due to
no ft

Services

1. Ongoing, work-based Vocational evaluation Consumer Assessment occurs in Most of the Vocational
vocational assessment: is conducted before participates in a sheltered setting assessment is based assessment is
Vocational assessment is an job placement with a prevocational where consumers on brief, temporary ongoing and occurs
ongoing process based on work emphasis on offce- assessment at the carry out work for job experiences in in community jobs.
experiences in competitive jobs. based assessments, program site (e.g. pay the community that Minimal testing may
standardized tests, work units in a day are set up with the occur but not as a
intelligence tests, and program) employer prerequisite to the
work samples, or job search. Aims
at problem-solving
Cannot rate due to
using environmental
no ft
assessments
and considering
reasonable
accommodations

2. Rapid search for competitive First contact with an First contact with First contact with First contact with First contact with
jobs: employer about a an employer about an employer about an employer about an employer about
The search for competitive jobs competitive job is a competitive job a competitive job a competitive job a competitive job
occurs rapidly after program typically more than is typically at more is typically at more is typically at more is typically within 1
entry. 1 year after program than 9 months and than 6 months and than 1 month and month after program
entry, or within 1 year after within 9 months after within 6 months after entry
program entry program entry program entry
Cannot rate due to
no ft

3. Individualized job search: Employer contacts About 25% of About 50% of About 75% of Most employer
Employer contacts are based are based on employer contacts employer contacts employer contacts contacts are based
on consumers’ job preferences decisions made are based on job are based on job are based on job on job choices,
(relating to what they enjoy and unilaterally by choices which choices which choices which which refect
their personal goals) and needs the employment refect consumers’ refect consumers’ refect consumers’ consumers’
(including experience, ability, specialist. These preferences and preferences and preferences and preferences and
how they affect a good job and decisions are usually needs rather than the needs rather than the needs rather than the needs rather than
setting match) rather than the driven by the nature job market job market job market the job market
job market (that is, what jobs are of the job market, or
readily available). Cannot rate due to
no ft

4. Diversity of jobs developed: Employment Employment Employment Employment Employment

Employment specialists provide specialists provide specialists provide specialists provide specialists provide specialists provide
job options that are in different options for either the options for either options for either options for either options for either
settings. same types of jobs the same types of the same types of the same types of the same types of
for most consumers, jobs, e.g., janitorial, jobs, e.g., janitorial, jobs, e.g., janitorial, jobs, e.g., janitorial,
e.g., janitorial, or or jobs at the same or jobs at the same or jobs at the same or jobs at the same
jobs at the same work settings, about work settings, about work settings, about work settings, less
settings, or 75% of the time 50% of the time 25% of the time than 10% of the
time
Cannot rate due to
no ft

Appendix B: Fidelity Scale and Score Sheet 32 Evaluating Your Program

Supported Employment Fidelity Scale
Ratings / Anchors
Criteria 1 2 3 4 5
5. Permanence of jobs developed: Employment Employment Employment Employment Virtually all
Employment specialists provide specialists usually do specialists specialists specialists competitive
competitive job options that have not provide options provide options provide options provide options jobs offered by
permanent status rather than for permanent, for permanent, for permanent, for permanent, employment
temporary or time-limited status. competitive jobs, or competitive jobs competitive jobs competitive jobs specialists are
about 25% of the about 50% of the about 75% of the permanent
Cannot rate due to
time time time
no ft

6. Jobs as transitions: Employment Employment Employment Employment Employment

All jobs are viewed as positive specialists prepare specialists help specialists help specialists help specialists help
experiences on the path consumers for a consumers fnd consumers fnd consumers fnd consumers end jobs
of vocational growth and single lasting job, another job 25% another job 50% another job 75% when appropriate
development. Employment and if it ends, will of the time of the time of the time and offer to help
specialists help consumers end not necessarily help them all fnd
jobs when appropriate and then them fnd another another job
fnd new jobs. one, or
Cannot rate due to
no ft

7. Follow-along supports: Follow-along Follow-along Follow-along Follow-along Most working

Individualized, follow-along supports are supports are time- supports are time- supports are ongoing consumers are
supports are provided to nonexistent, or limited and provided limited and provided and provided to less provided fexible,
employer and consumer on to less than half to most working than half the working follow-along
Cannot rate due to
a time-unlimited basis. of the working consumers consumers supports that are
no ft
consumers individualized and
Employer supports may include
ongoing
education and guidance.
Consumer supports may include
crisis intervention, job coaching,
job counseling, job support
groups, transportation, treatment
changes (medication), and
supportive networks (friends and
family).

8. Community-based services: Employment Employment Employment Employment Employment

Vocational services such as specialists spend specialists spend specialists spend specialists spend specialists spend
engagement, job fnding, 10% of time or less 11–39% of time in 40–59% of time in 60–69% of time in 70% or more of time
and follow-along supports are in the community, community community community in community
provided in community settings. or Cannot rate due
to no ft

9. Assertive engagement Employment Employment Employment Employment Employment

and outreach: specialists do not specialists make 1 specialists make 1 or specialists make specialists make
Assertive engagement and provide outreach to contact to consumers 2 contacts as part of multiple contacts multiple contacts
outreach (telephone, mail, consumers as part of as part of initial initial engagement as part of initial as part of initial
community visits) are conducted initial engagement engagement or or within 1 month engagement and engagement and at
as needed. or to those who to those who stop when consumers at least every least monthly on a
stop attending the attending the stop attending the 2 months on a time-unlimited basis
vocational service, vocational service vocational service time-limited basis when consumers
when consumers stop attending the
or Cannot rate due
stop attending the vocational service,
to no ft
vocational service and
Employment
specialists
demonstrate
tolerance of different
levels of readiness
using gentle
encouragement

Evaluating Your Program 33 Appendix B: Fidelity Scale and Score Sheet

Score Sheet: Supported Employment Fidelity Scale

Agency name: ___________________________________________ Today’s date: _____/___/_____

Assessors’ names: ___________________________________________

___________________________________________

Data Source
Assessor 1 Assessor 2 Consensus Circle the data
sources used
Staffng

1 Caseload VL, MIS, DOC, INT

2 Vocational services staff MIS, DOC, INT

3 Vocational generalists VL, MIS, DOC, INT

Organization

Integration of rehabilitation with

1 VL, MIS, DOC, INT
mental health treatment

2 Vocational unit MIS, INT

3 Zero-exclusion criteria DOC, INT

Services

1 Ongoing work-based assessment DOC, INT

2 Rapid search for competitive jobs DOC, INT, ISP

3 Individualized job search DOC, INT, ISP

4 Diversity of jobs developed DOC, INT, ISP

5 Permanence of jobs developed DOC, INT, ISP

6 Jobs as transitions VL, DOC, INT, ISP

7 Follow-along supports VL, DOC, INT

8 Community-based services VL, MIS, DOC, INT

9 Assertive engagement and outreach VL, MIS, DOC, INT

Total score

Items not rated due to insuffcient data

66-75 = Good implementation VL = Vocational Logs (Employer logs or other SE fles)

56-65 = Fair implementation MIS = Management Information System
55 and below = Not SE DOC = Document review: chart review, agency policy and procedures
INT = Interviews with consumers, families, employment specialists, other staff
ISP = Individualized Service Plan (Individualized Employment Plan)

Evaluating Your Program 35 Appendix B: Fidelity Scale and Score Sheet

 Evaluating Your Program

Appendix C: Supported Employment Fidelity Guide

Evaluating Your Program 37 Appendix C: Fidelity Guide

Supported Employment Fidelity Guide

Discussion guide for interviewing SE leaders, employment specialists, and other agency staff

Ask questions such as the ones suggested below to collect information that you may use to rate the items on
the SE Fidelity Scale. The questions are not meant to be used as a structured interview, so feel free to put
them into your own words and to ask them in any order. The goal is to collect accurate information that
refects the daily practice in the SE program.

Staffng

Caseload size Vocational services staff

n Does each employment specialist have a discreet n Do any employment specialists provide services
caseload? other than vocational, such as case management,
day programming, or residential services?
n How many consumers does each employment
specialist (full-time equivalent) have on his or n For how much of their time do they provide
her caseload? nonvocational services?
n How often does the employment specialist meet
with each person on the caseload?
n Approximately how long do consumers stay on
Vocational generalists
the caseload? When is a consumer removed from
n Do different employment specialists provide
the caseload?
different aspects of the vocational service?
For example, one employment specialist does
only job development or one does only job
support. Which aspect of the vocational process
does each provide?

Evaluating Your Program 39 Appendix C: Fidelity Guide

 Organization

Integration of rehabilitation with Vocational unit

mental health treatment
n Do employment specialists have the same
n Do employment specialists interact with case supervisor?
managers about their mutual consumers?
n Do employment specialists meet as a group
n In what situations do they interact and for supervision? If so, how often?
how regularly? Do they meet in person
n Do employment specialists provide services
or by telephone?
for one another’s consumers?
n Are employment specialists assigned to work
with specifc case managers or case management
teams?
Zero-exclusion criteria
n Do they participate in shared decisionmaking
about consumer services? Who (staff) makes the n What are the criteria to be eligible to receive
fnal decision? Supported Employment services?
n Where are case managers’ offces located in n Who makes referrals?
relation to the Supported Employment offce?
n Who conducts the screening?
n Do provisions exist to make sure no one
is excluded?
n What is the rate of referral?

Appendix C: Fidelity Guide 40 Evaluating Your Program

 Services

Ongoing, work-based vocational assessment Individualized job search

n Does the program include vocational evaluation n How is it decided which jobs are identifed
procedures? in the job search? Who makes these decisions?
What information is it based on?
n What type of assessment procedures do you use
and in which settings? n How has the nature of the job market affected
the type of jobs consumers have obtained?
n Must certain assessment procedures be
completed before getting a competitive job, such
as, testing, prevocational work adjustment?
n How much preplacement assessment is done?
Diversity of jobs developed

n How much time is spent on vocational n Do employment specialists ever suggest to

assessment? consumers that they work at the same job
setting as other consumers? What percentage
of consumers work in the same job settings?
Rapid search for competitive job n Do employment specialists ever suggest
to consumers that they obtain the same type
n What is the average length of time between of job as other consumers? What percentage
when a consumer begins the program and of consumers have the same type of work?
the frst contact with a competitive employer?
What is the range of time?
n What is the program’s philosophy about when Permanence of jobs developed
to start the job search? Must consumers take
any steps in the program before starting to look n What percentage of the jobs that employment
for a job? specialists suggest to consumers are permanent,
competitive jobs?
n Do employment specialists ever suggest jobs that
are temporary, time-limited, or volunteer? If so,
how often?

Evaluating Your Program 41 Appendix C: Fidelity Guide

Jobs as transitions Community-based services

n Do employment specialists help consumers fnd n Where do employment specialists spend most
another job when one ends? of their time?
n What percentage of consumers who have n What percentage of their time is spent outside of
ended jobs have been provided help in fnding the mental health facility? (Ask the employment
another job? specialist to review how he or she spent time
over the last couple of days to determine
n What are reasons an employment specialist
location of services.)
would not help a consumer fnd another job
when one has ended? (for example, consumer
was fred due to poor attendance, problems with
substance abuse.) Assertive engagement and outreach

n Do employment specialists provide any

outreach if consumers do not engage or drop
Follow-along supports
out of services?
n Do employment specialists provide follow-along n What kinds of outreach are provided? How often
supports to consumers and employers? What are outreach attempts made? Is there a time
kind of supports? limit to providing outreach if a consumer stops
attending? If so, what is the time limit?
n What percentage of working consumers has
follow-along supports provided?
n Is there a time limit for providing supports?

Appendix C: Fidelity Guide 42 Evaluating Your Program

Discussion guide for interviewing consumers and family members

The interviews with consumers and family members should take an informal, conversational tone. Use the
questions below to guide the conversation. You don’t have to read the questions verbatim. If the respondent
has diffculty understanding the questions, then reword them as needed.

In the wording of the questions below, we use you/your family member to indicate the different wording that
you might use depending on the participant. When you speak with family members, feel free to use the
consumer’s frst name or the relationship to the respondent (for example, your son) instead of the phrase,
your family member. Also, make any necessary grammatical changes.

Beginning the conversation

Thank you for taking the time to meet with me Also, what we talk about today is confdential and
today. anonymous. That means that we will not collect any
information that could identify you and we will not
My name is _______________ (YOUR NAME) and include any identifying information about you in
I work for _____________ (YOUR AGENCY the results of our review. The only exception to this
NAME). is if you report an incident of child abuse during
our conversation or if you indicate that you may be
We’re working with _____________ (NAME OF about to hurt yourself or someone else. In these
SE PROGRAM) to learn more about how it cases, we’ll discuss the situation and I may be
provides services for consumers who are interested required by law to report it.
in working.
The risk to participating is that you might feel
We will use the information to help provide better uncomfortable with some of the questions. If
employment services to people in this program. there’s a question you don’t want to answer, you
We’ll meet for about 15 to 30 minutes to talk about don’t have to answer it, and we can stop at any time.
the services (you/your family member) have The beneft to participating is that the information
received and what you’ve thought of them. you provide can help us improve employment
services offered by this agency.
Your participation in the interview is voluntary; you
don’t have to do this if you don’t want to. Your Do you have any questions before we begin?
decision to participate will not affect the services
(you/your family member) receive(s).

Evaluating Your Program 43 Appendix C: Fidelity Guide

 Questions

n Please tell me about the ____________________ n Have (you/your family member) gotten a job
(NAME OF SE PROGRAM). since (you/your family member) entered the
program? How many jobs?
What kinds of services have (you/your family
member) received from them? What kinds of n What has been most helpful to (you/your family
things have they helped (you/your family member) about this program? What has been
member) with? least helpful?
How long have (you/your family member) been n How did (you/your family member) fnd out
involved in this program? about the _______________ (NAME OF SE
PROGRAM) program?
n What is the name of the person (you/your family
member) meet(s) with most often to talk about
your interest in working? _______
(Note to Interviewer: Note frst name and insert Integration of rehabilitation with mental
health treatment
for “EMPLOYMENT SPECIALIST NAME”
as indicated below.)
n Do (you/your family member) talk about (your/
n How often do (you/your family member) meet his/her) interest in working with other staff at
with _______ (EMPLOYMENT SPECIALIST this agency?
NAME)?
n Do (you/your family member) meet with a
How long do (you/your family member) mental health worker or other practitioner?
usually meet with _______ (EMPLOYMENT A psychiatrist? Anyone else?
SPECIALIST NAME)? How much time
n How have they helped (you/your family member)
do (you/your family member) usually
in terms of (your/his/her) interest in working?
spend together?
n Does _______________ (EMPLOYMENT
n What do (you/your family member) do with
SPECIALIST NAME) talk with these other staff
_______ (EMPLOYMENT SPECIALIST
members about (your/his/her) interest in
NAME) when (you/your family member)
working? Tell me about that.
are together?
n Does _______________ (EMPLOYMENT
n Is _______ (EMPLOYMENT SPECIALIST
SPECIALIST NAME) talk with other
NAME) helpful? If so, how? If not, how not?
professionals outside of this agency, such as
n In general, is the program helpful to (you/your Vocational Rehabilitation, about (your/his/her)
family member)? If so, how? If not, why not? interest in working or going to school?
Are there ways that the program could better
help (you/your family member)?

Appendix C: Fidelity Guide 44 Evaluating Your Program

Zero-exclusion criteria Rapid search for competitive job

n When (you/your family member) frst told them n Once (you/your family member) told someone in
that (you/your family member) were interested the program that (you/your family member)
in the program, were (you/your family member) wanted to work, how long was it before (you/
able to get into the program as soon as (you/your your family member) started meeting with
family member) wanted? __________ (EMPLOYMENT SPECIALIST
NAME) or another staff person to talk about
How long was that?
working?
If (you/your family member) couldn’t get in
n Once (you/your family member) started meeting
when (you/your family member) wanted, why
with __________ (EMPLOYMENT
did (you/your family member) have to wait?
SPECIALIST NAME) to talk about working,
n Did staff members encourage you to think about how long was it before (you/your family member)
working or going to school even before you actually started looking for a job?
expressed an interest in work or school?

Individualized job search

Ongoing, work-based vocational assessment
n When (you/your family member) began meeting
n When (you/your family member) entered the with ___________ (EMPLOYMENT
program, did (you/your family member) need to SPECIALIST NAME), how did the two of you
take any tests before (you/your family member) decide which jobs to search for?
could look for a job? If so, tell me about them.
Did (you/your family member) like the way that
n When (you/your family member) entered the decisions were made? Why or why not?
program, did (you/your family member) have to
Were other people who know you, like other
take a job (you/your family member) didn’t want
family members or other staff workers, asked
in order to “get ready” for the type of job (you/
about jobs that you might like?
your family member) did want?
If so, tell me about that. Did (you/your family member) make the fnal
Were you asked to work in the day program or decision about which jobs to apply for?
sheltered workshop before being helped to fnd a
regular job?
Diversity of jobs developed

n Have the jobs (you/your family member) have

gotten been at the same place that other people
receiving Supported Employment services are
working? If so, tell me about that.
n Do people who receive services from this
program end up getting the same types of jobs
or do they get different types of jobs? Tell me
about that.

Evaluating Your Program 45 Appendix C: Fidelity Guide

Permanence of jobs developed Jobs as transitions

n When (you/your family member) worked with n When (you/your family member) had a job that
_______________ (EMPLOYMENT ended, did ____________ (EMPLOYMENT
SPECIALIST NAME) to get a job, did (you/your SPECIALIST NAME) or other staff help
family member) work toward jobs that were— (you/your family member) fnd a new job?
n Permanent, so (you/your family member) could If so, how did he or she help (you/your family
stay as long as (you/your family member) member)?
wanted? Tell me about that.
If not, why didn’t he or she help (you/your
n Temporary, so that (you/your family member) family member)?
had to leave the job at some point? Tell me
about that. n Did (you/your family member) like the jobs that
(you/he/she) got? Why or why not?
n In a competitive workplace where non-mental
health consumers also worked? Tell me If (you/your family member) did not like a job,
about that. did (you/your family member) tell _________
(EMPLOYMENT SPECIALIST NAME)?
n Part of a sheltered workshop or agency work
program so that (you or your family member) What happened then?
were supervised by a mental health staff
person? Tell me about that.
n Volunteer work, that is, work that (you/your Follow-along supports
family member) didn’t get paid for? Tell me
about that. n Once (you/your family member) got a job, did
_______________ (EMPLOYMENT
SPECIALIST NAME) or other staff members
still help (you/your family member)? How?
Community-based services
n Did he or she help (you/your family member)
n Where do (you/your family member) and with the job or help (you/your family member)
_______________ (EMPLOYMENT keep the job?
SPECIALIST NAME) usually meet?
n What kinds of things did he or she do to help
n Where else have (you/your family member) met (you/your family member) keep the job?
and how often?
n Was _______________ (EMPLOYMENT
SPECIALIST NAME) or other staff member
able to help (you/your family member) as long
Assertive engagement and outreach as (you/your family member) wanted, or, once
(you/your family member) got a job, could he or
n Have (you/your family member) ever decided she only keep helping (you/your family member)
that (you/your family member) didn’t want to for a certain period of time? Tell me about that.
be in the Supported Employment program
anymore?
n Did (you/your family member) tell staff?
What happened then?
(Note to Interviewer: Ask only of consumers
who report having more than one job.)

Appendix C: Fidelity Guide 46 Evaluating Your Program

 Evaluating Your Program

Appendix D: General Organizational Index

and Score Sheet

Evaluating Your Program 47 Appendix D: General Organizational Index and Scoresheet

General Organizational Index

1 2 3 4 5

G1. Program philosophy No more than 2 of 5 sources 3 of 5 sources 4 of 5 sources All 5 sources
1 of 5 sources show clear show clear show clear show clear
Committed to clearly articulated shows clear understanding understanding understanding understanding
philosophy consistent with specifc understanding of program of program of program and commitment
evidence-based model, based on these of program philosophy philosophy philosophy to program
5 sources: philosophy philosophy for
n SE leader specifc EBP

n Senior staff (for example, executive

director, psychiatrist) OR OR OR OR

n Team members providing the EBP All sources All sources Sources mostly Sources mostly
n Consumers and families receiving EBP have numerous have several aligned to aligned to
major areas of major areas of program program
n Written materials (such as, brochures) discrepancy discrepancy philosophy, but philosophy,
have 1 major area but have 1 or 2
of discrepancy minor areas of
discrepancy

*G2.Eligibility or consumer 20% of 21–40% of 41–60% of 61–80% of More than 80%

consumers receive consumers receive consumers receive consumers receive of consumers
identifcation
standardized standardized standardized standardized receive
screening screening screening screening standardized
All consumers with serious mental
and/or agency and agency and agency and agency screening
illnesses in the community support
DOES NOT systematically systematically systematically and agency
program; crisis consumers, and
systematically tracks eligibility tracks eligibility tracks eligibility systematically
institutionalized consumers are screened
track eligibility tracks eligibility
to determine if they qualify for EBP
using standardized tools or admission
criteria consistent with EBP. Also, agency
systematically tracks number of eligible
consumers.

*G3. Penetration Ratio .20 Ratio .21 – .40 Ratio .41 – .60 Ratio .61 – .80 Ratio > .80

Maximum number of eligible consumers

served by EBP, as defned by the ratio:
Number of consumers receiving EBP
Number of consumers eligible for EBP

* These two items coded based on all consumers with serious mental illnesses at the site or sites where EBP is being implemented; all other
items refer specifcally to those receiving the EBP.

Total number of consumers in target population

Total number of consumers eligible for EBP % % eligible:

Total number of consumers receiving EBP Penetration rate

Evaluating Your Program 49 Appendix D: General Organizational Index and Scoresheet

 1 2 3 4 5

G4. Assessment Assessments are Pervasive Pervasive 61%-80% of More than 80%
completely absent defciencies in 2 defciencies in 1 consumers receive of consumers
Full standardized assessment of all or completely of the following: of the following: standardized, receive
consumers who receive EBP services. non-standardized high-quality standardized,
n Standardization n Standardization
Assessment includes: assessments at high-quality
n Quality of n Quality of least annually assessments; the
n History and treatment of medical;
assessments assessments information is
psychiatric; substance use disorders
n Timeliness n Timeliness OR comprehensive
n Current stages of all existing disorders
across all
n Comprehen- n Comprehen- Information is
n Vocational history assessment
siveness siveness defcient for 1
n Any existing support network domains and
or 2 assessment
updated at least
n Evaluation of biopsychosocial risk domains
annually
factors

G5. Individualized treatment plan 20% of 21 – 40% of 41– 60% of 61– 80% of More than 80%
consumers consumers consumers consumers of consumers
For all EBP consumers, an explicit, EBP serves EBP serves EBP serves EBP serves EBP serves
individualized treatment plan exists have explicit have explicit have explicit have explicit have explicit
related to the EBP that is consistent with individualized individualized individualized individualized individualized
assessment and updated every 3 months treatment plans, treatment plans, treatment plans, treatment plans, treatment plans
related to EBP, related to EBP, related to EBP, related to EBP, related to EBP,
updated every updated every updated every updated every updated every
3 months 3 months 3 months 3 months 3 months

OR

Individualized
treatment plan
updated every
6 months for
all consumers

G6. Individualized treatment 20% of 21–40% of 41–60% of 61–80% of More than 80%
consumers EBP consumers EBP consumers EBP consumers EBP of consumers EBP
All EBP consumers receive individualized serves receive serves receive serves receive serves receive serves receive
treatment meeting goals of EBP individualized individualized individualized individualized individualized
services meeting services meeting services meeting services meeting services meeting
goals of EBP goals of EBP goals of EBP goals of EBP goals of EBP

G7. Training 20% of program 21–40% 41–60% 61–80% More than 80%
staff receive of program of program of program of program
All new team members receive standardized staff receive staff receive staff receive staff receive
standardized training in EBP (at least a training annually standardized standardized standardized standardized
2-day workshop or equivalent) within training annually training annually training annually training annually
2 months after hiring. Existing team
members receive annual refresher
training (at least 1-day workshop
or equivalent).

G8. Supervision 20% of EBP 21–40% of EBP 41–60% of EBP 61–80% of EBP More than
practitioners practitioners practitioners practitioners 80% of EBP
SE team members receive structured, receive receive weekly receive weekly receive weekly practitioners
weekly supervision (group or supervision structured, structured, structured, receive
individual format) from a team consumer- consumer- consumer- structured weekly
member experienced in particular centered centered centered supervision,
EBP. Supervision should be consumer- supervision supervision supervision focusing
centered and explicitly address EBP on specifc
model and its application to specifc OR OR OR consumers,
consumer situations. in sessions that
All EBP All EBP All EBP
explicitly address
practitioners practitioners practitioners
EBP model and its
receive informal receive monthly receive
application
supervision supervision supervision 2
times a month

Appendix D: General Organizational Index and Scoresheet 50 Evaluating Your Program

 1 2 3 4 5
G9. Process monitoring No attempt Informal process Process monitor- Process Standardized
at monitoring monitoring is ing is defcient monitoring is comprehensive
Supervisors and SE leaders monitor process is made defcient on 1 of process
used at least on 2 of these 3
process of implementing EBP these 3 criteria: monitoring
annually criteria:
every 6 months and use the data occurs at least
n
to improve program. Monitoring n Comprehensive every 6 months
Comprehensive
involves a standardized approach, for and and is used to
and standardized
example, using fdelity scale or other standardized guide program
comprehensive set of process indicators. n Completed
n Completed improvements
every 6 months
every 6 months
n Used to guide
n Used to guide
program
program
improvements
improvements

OR

Standardized
monitoring done
annually only

G10. Outcome monitoring No outcome Outcome Standardized Standardized Standardized

monitoring occurs monitoring outcome moni- outcome outcome
Supervisors and program leaders occurs at least 1 monitoring occurs monitoring
toring occurs at
monitor outcomes for EBP consumers time a year, but at least 2 times a occurs quarterly.
least 1 time a
every 3 months and share data with EBP results are not year. Results are Results are
team members. Monitoring involves
year. Results are
shared with team shared with team shared with EBP
standardized approach to assessing a shared with team
members members members
key outcome related to EBP, such as, members
psychiatric admissions, substance abuse
treatment scale, or employment rate.

No review or no QA committee has Explicit QA review Explicit QA review Explicit review

G11. Quality Assurance (QA)
committee been formed, but occurs less than occurs annually occurs every 6
no reviews have annually months by QA
Agency has QA committee or
been completed group or steering
implementation steering committee
OR committee for
with an explicit plan to review EBP
EBP
or components of the program every QA review is
6 months. superfcial

G12.Consumer choice about Consumer- Few sources agree Half of the Most sources All sources agree
service provision centered services that type and sources agree agree that type that type and
are absent frequency of EBP that type and and frequency frequency of EBP
All consumers receiving EBP services (or practitioners services refect of EBP services services refect
are offered choices; EBP team members
frequency of EBP
make all EBP consumer choice refect consumer consumer choice
consider and abide by consumer services refect
decisions) choice
preferences for treatment when offering consumer choice
and providing services.
OR
Agency fully
embraces
consumer
choice with
one exception

Evaluating Your Program 51 Appendix D: General Organizational Index and Scoresheet

Score Sheet: General Organizational Index

Agency name: ___________________________________________ Today’s date: _____/___/_____

Assessors’ names: ___________________________________________

___________________________________________

Assessor 1 Assessor 2 Consensus

G1 Program philosophy

G2 Eligibility or consumer identifcation

G3 Penetration

G4 Assessment

G5 Individualized treatment plan

G6 Individualized treatment

G7 Training

G8 Supervision

G9 Process monitoring

G10 Outcome monitoring

G11 Quality Assurance (QA)

G12 Consumer choice regarding service provision

Total mean score

Appendix D: General Organizational Index and Scoresheet 52 Evaluating Your Program

 Evaluating Your Program

Appendix E: General Organizational Index Protocol

Evaluating Your Program 53 Appendix E: General Organizational Index Protocol

 Sources of information:
General Organizational
Overview:During the site visit, be alert to indicators
Index Protocol of program philosophy consistent with or inconsistent
with the EBP, including observations from casual
conversations, staff and consumer activities, etc.
The General Organizational Index Protocol explains
Statements that suggest misconceptions or reservations
how to rate each item of the index. In particular, it
about the practice are negative indicators, while
provides the following: statements that show enthusiasm for and understanding
n A defnition and rationale for each item; and of the practice are positive indicators.

n A list of data sources most appropriate for The intent of this item is to gauge the understanding
each fdelity item (for example, chart review, of and commitment toward the practice. It is not
program leader, practitioners, consumers, necessary that every element of the practice is currently
and family interviews). in place (this is gauged by the SE Fidelity Scale), but
rather whether all those who are involved are
When it is appropriate, a set of probe questions is committed to implementing high fdelity SE.
provided to help you elicit the critical information
needed to score the fdelity item. These probe The employment specialists rated for this item are
questions were specifcally generated to help you limited to those implementing this practice. Similarly,
collect information from respondents that is the consumers rated are those receiving the practice.
relatively free from bias, such as social desirability.
1. SE leader, senior staff, and employment
specialist interviews
Decision rules will help you score each item
correctly. As you collect information from various n Ask interviewees to briefy describe the program.
sources, these rules will help you determine the
n “What are the critical ingredients or principles
specifc rating to give for each item. of your services?”
n “What is the goal of your program?”

G1. Program Philosophy n “How do you defne SE?”

Defnition: The program is committed to a clearly 2. Consumer interview

articulated philosophy consistent with the n “What kind of services do you receive from
specifc evidence-based practice (EBP), this program?”
based on the following fve sources:
n Using a layperson’s language, describe to the
n SE leader; consumer or family the principles of SE.
n Senior staff (such as executive director, Probe if the program offers services that refect
psychiatrists); each principle.

n Employment specialist; n “Do you feel that employment specialists are

competent and help you address your problems?
n Consumers and family members; and
n Written materials (such as brochures). 3. Written material review

n Does the site have written materials on SE?

Rationale: In psychiatric rehabilitation programs that
truly endorse EBPs, staff members at all n Does the written material articulate program
levels embrace the program philosophy philosophy consistent with SE?
and practice it in their daily work.

Evaluating Your Program 55 Appendix E: General Organizational Index Protocol

Item response coding: The goal of this item is not to quiz by the provider agency or service area. If
every employment specialist to determine if they can the agency serves consumers at multiple
recite every critical ingredient. Rather, the goal is to sites, then assessment is limited to the site
gauge whether the understanding is generally accurate or sites that are targeted for the EBP.
and not contrary to the EBP. For example, if a senior
staff member says, “Most of our consumers are not If the target population is served in discrete
work-ready,” then that would be a red fag for the programs (such as case management,
practice of Supported Employment. residential, or day treatment), then
ordinarily all adults with serious mental
If all sources show evidence that they clearly illnesses are included in this defnition.
understand the program philosophy, code the item
as “5.” For a source type that is based on more than Screening will vary according to the EBP.
one person (for example, team member interviews) The intent is to identify all who could
determine the majority opinion when rating that source beneft from the EBP. Although the
as endorsing or not endorsing a clear program program is intended to be offered to
philosophy. Note: If no written material, then count that consumers at the point they express interest
source as unsatisfactory. in working, all consumers are presumed to
be eligible and should be invited to receive
Difference between a major and minor area of discrepancy SE services.
(needed to distinguish between a score of “4” and “3”): An
example of a minor source of discrepancy for Assertive In every case, the agency should have an
Community Treatment (ACT) might be larger explicit, systematic method for identifying
caseload sizes (such as 20 to 1) or some brokering of the eligibility of every consumer. Screening
services. An example of a major discrepancy would be if typically occurs at admission; programs that
the team seldom made home visits or if the psychiatrist are newly adopting an EBP should have a
was uninvolved in the treatment team meetings. plan for systematically reviewing consumers
who are already active in the agency.

Rationale: Accurately identifying consumers who

G2. Eligibility/Consumer Identifcation would beneft most from the EBP requires
routine review for eligibility, based on
Defnition: For EBPs implemented in a mental health criteria consistent with the EBP.
center: All consumers in the community
support program, crisis consumers, and
Sources of information:
institutionalized consumers are screened
using standardized tools or admission
1. SE leader, senior staff, and employment specialist
criteria that are consistent with the EBP.
interviews

For EBPs implemented in a service area: n “Describe the eligibility criteria for your
All consumers within the jurisdiction of the program.”
service area are screened using standardized
n “How are consumers referred to your program?
tools or admission criteria that are consistent How does the agency identify consumers who
with the EBP. For example, in New York, would beneft from your program? Do all new
county mental health administrations are consumers receive screening for substance abuse
responsible for identifying consumers who or SMI diagnosis?”
will be served by Assertive Community
Treatment (ACT) programs. n “What about crisis (or institutionalized)
consumers?”
The target population refers to all adults n Request a copy of the screening instrument used
with serious mental illnesses (SMI) served by the agency.

Appendix E: General Organizational Index Protocol 56 Evaluating Your Program

2. Chart review:Review documentation of screening Numerator: The number receiving the service is based
process and results. on a roster of names that the SE leader maintains.
Ideally, this total should be corroborated with service
3. County mental health administrators (where applicable): contact sheets and other supporting evidence that the
If eligibility is determined at the service-area level identifed consumers are actively receiving treatment.
(such as the New York example), then interview the As a practical matter, agencies have many conventions
people responsible for this screening. for defning active consumers and dropouts, so that it
may be diffcult to standardize the defnition for this
Item response coding:This item refers to all consumers item. Use the best estimate of the number actively
with SMI in the community support program or its receiving treatment.
equivalent at the sites where the EBP is being
implemented; it is not limited to consumers who Denominator: If the agency systematically tracks eligibility,
receive EBP services only. Calculate this percentage then use this number in the denominator. (See the rules
and record it on the fdelity rating scale in the space listed in G2 to determine the target population before
provided. If 80 percent or more of these consumers using estimates below.) If the agency doesn’t track
receive standardized screening, code the item as “5.” eligibility, then estimate the denominator by multiplying
the total target population by the corresponding
percentage based on the literature for each EBP.

G3. Penetration According to the literature, the estimates for EBP KITs
available at this writing should be as follows:
Defnition: Penetration is defned as the percentage
of consumers who have access to an EBP
n Supported Employment — 60%;
as measured against the total number
of consumers who could beneft from n Integrated Treatment for Co-Occurring
the EBP. Numerically, this proportion Disorders — 40%;
is defned by:
n Illness Management and Recovery — 100%;
Number of consumers receiving an EBP
n Family Psychoeducation — 100%
Number of consumers eligible for the EBP (some kind of signifcant other); and
n Assertive Community Treatment — 20%.
As in the preceding item, the numbers used
in this calculation are specifc to the site or
Example for calculating denominator:
sites where the EBP is being implemented.
Suppose you don’t know how many consumers are
Rationale: Surveys have repeatedly shown that people eligible for Supported Employment (that is, the
with SMI often have a limited access to community support program has not surveyed
EBPs. The goal of EBP dissemination is not consumers to determine those who are interested). Let’s
simply to create small exclusive programs, say the community support program has 120 consumers.
but to make these practices easily accessible Then you would estimate the denominator to:
within the public mental health system.
120 X .6 = 72

Sources of information:
Item response coding:Calculate this ratio and record it on
The calculation of the penetration rate depends on the Fidelity Scale in the space provided. If the program
the availability of the two statistics defning this rate. serves more than 80 percent of eligible consumers,
code the item as “5.”

Evaluating Your Program 57 Appendix E: General Organizational Index Protocol

 2. Chart review
G4. Assessment
n Look for comprehensiveness of assessment
Defnition: All EBP consumers receive standardized, by looking at multiple completed assessments
high-quality, comprehensive, and timely to see if they address each component of the
assessments. comprehensive assessment every time an
assessment is performed.
Standardization refers to a reporting
n “Is the assessment updated at least yearly?”
format that is easily interpreted and
consistent across consumers.
If more than 80 percent of
Item response coding:
consumers receive standardized, high-quality,
High quality refers to assessments that
comprehensive, and timely assessments, code
provide concrete, specifc information that
the item as “5.”
differentiates between consumers. If most
consumers are assessed using identical
words or if the assessment consists of
broad, noninformative checklists, then G5. Individualized Treatment Plan
this would be considered low quality.
Defnition: For all EBP consumers, an explicit,
Comprehensive assessments include individualized treatment plan exists (even
the following: if it is not called this) related to the EBP
n History and treatment of medical, that is consistent with assessment and
psychiatric, and substance use disorders; updated every 3 months.

n Current stages of all existing disorders; Individualized means that goals, steps
n Vocational history; to reaching the goals, services and
interventions, and intensity of involvement
n Any existing support network; and
are unique to this consumer. Plans that
n Evaluation of biopsychosocial risk are the same or similar across consumers
factors. are not individualized. One test is to place
Timely assessments are those updated a treatment plan without identifying
at least annually. information in front of supervisors
to see if they can identify the consumer.
Rationale: Comprehensive assessment or re-
assessment is indispensable in identifying Rationale: Core values of EBP include individualizing
target domains of functioning that may services and supporting consumers’ pursuit
need intervention, in addition to of their goals and progress in their
consumers’ progress toward recovery. recovery at their own pace. Therefore,
treatment plans need ongoing evaluation
Sources of information: and modifcation.
1. Program leader, senior staff, and employment
specialist interviews Sources of information:

n “Do you give a comprehensive assessment Note: Assess this item and the next together; that is,
to new consumers? What are the components followup questions about specifc treatment plans
that you assess?” with questions about the treatment.

n Ask for a copy of the standardized assessment

form, if available, and have practitioners go
through the form.
n “How often do you re-assess consumers?”

Appendix E: General Organizational Index Protocol 58 Evaluating Your Program

1. Chart review (treatment plan)
G6. Individualized Treatment
Using the same charts as examined during the EBP-
specifc fdelity assessment, look for documentation of Defnition: All EBP consumers receive individualized
specifc goals and consumer-based, goal-setting process: treatment meeting the goals of the EBP.

n “Are the treatment recommendations consistent Individualized treatment means that

with assessment?” steps, strategies, services, interventions,
n “What evidence is used for a quarterly review?” and intensity of involvement are focused
on specifc consumer goals and are unique
2. SE leader interview for each consumer. Progress Notes are
often a good source of what really
“Describe the process of developing a treatment plan. goes on. Treatment could be highly
What are the critical components of a typical treatment individualized, despite the presence
plan and how are they documented?” of generic treatment plans.

3. Employment specialist interview An example for a low score on this item for
n When feasible, use the specifc charts selected Assertive Community Treatment (ACT):
above. Ask practitioners to go over a sample If most of the Progress Notes are written
treatment plan. by day treatment staff who see consumers
n “How do you come up with consumer goals?” 3 to 4 days per week, while the ACT team
[Listen for consumer involvement and sees consumers only about once per week
individualization of goals.] to issue their checks.

n “How often do you review (or follow up on) Rationale: The key to the success of an EBP is
the treatment plan?”
implementing a plan that is individualized
and meets the goals for the EBP for each
4. Consumer interview
consumer.
n “What are your goals in this program?
How did you set these goals?” Sources of information:

n “Do you and your employment specialist 1. Chart review (treatment plan)
together review your progress toward achieving Using the same charts as examined during the EBP-
your goals?” [If yes, “How often? Please describe specifc fdelity assessment, examine the treatment
the review process.”] provided. Limit the focus to a recent treatment plan
related to the EBP. Judge whether an appropriate
5. Team meeting and supervision observation, treatment occurred during the time frame indicated
if available by the treatment plan.
Observe how the treatment plan is developed.
Listen especially for discussion of assessment, 2. Employment specialist interview

consumer preferences, and individualization When feasible, use the specifc charts selected above.
of treatment. Do they review treatment plans? Ask practitioners to go over a sample treatment plan
and treatment.
Item response coding: If more than 80 percent of EBP
consumers have an explicit individualized treatment plan 3. Consumer interview
that is updated every three months, code the item as “5.”
“Tell me about how this program is helping you meet
If the treatment plan is individualized but updated only
your goals.”
every 6 months, code the item as “3.”
If more than 80 percent of EBP
Item response coding:
consumers receive treatment that is consistent with
the goals of the EBP, code the item as “5.”

Evaluating Your Program 59 Appendix E: General Organizational Index Protocol

G7. Training G8. Supervision

Defnition: All new employment specialists receive Defnition: Employment specialists receive structured,
standardized training in the EBP (at least a weekly supervision from practitioners
2-day workshop or its equivalent) within 2 experienced in the particular EBP.
months after they are hired. Existing The supervision can be either group or
employment specialists receive annual individual, but CANNOT be peers-only
refresher training (at least a 1-day supervision without a supervisor. The
workshop or its equivalent). supervision should be consumer-centered
and explicitly address the EBP model and
Rationale: Practitioner training and retraining are how it applies to specifc consumer
warranted to ensure that evidence-based situations. Administrative meetings and
services are provided in a standardized meetings that are not specifcally devoted
manner, across employment specialists and to the EBP do not ft the criteria for this
over time. item. The consumer-specifc EBP
supervision should be at least 1 hour
Sources of information: long each week.

1. SE leader, senior staff, and employment specialist

Rationale: Regular supervision is critical not only
interviews
for individualizing treatment, but also
n “Do you provide new employment specialists for ensuring the standardized provision
with systematic training for SE?” [If yes, of evidence-based services.
probe for specifcs: Mandatory or optional?
Length? Frequency? Content? Group or Sources of information:
individual format? Who trains? In-house
1. SE leader, senior staff, and employment
or outside training?]
specialists interviews
n “Do employment specialists receive refresher
Probe for logistics of supervision: length, frequency,
trainings?” [If yes, probe for specifcs.]
group size, etc.
2. Review training curriculum and schedule, if available n “Describe what a typical supervision session
looks like.”
Does the curriculum appropriately cover the critical
ingredients for [EBP area]? n “How does the supervision help your work?”

3. Team member interview 2. Team meeting and supervision observation, if available

n “When you frst started in this program, did you Listen for discussion of [EBP area] in each case
receive a systematic and formal training for SE?” reviewed.
[If yes, probe for specifcs: Mandatory or
optional? Length? Frequency? Content? Group 3. Supervision logs documenting frequency of meetings
or individual format? Who trains? In-house or
outside training?] Item response coding:If >80 percent of emplyment
specialists receive weekly supervision, code the item
n “Do you receive refresher trainings?” as “5.”
[If yes, probe for specifcs.]
If more than 80 percent of SE team
Item response coding:
members receive at least yearly, standardized training
for SE, code the item as “5.”

Appendix E: General Organizational Index Protocol 60 Evaluating Your Program

G9. Process Monitoring G10. Outcome Monitoring

Defnition: Supervisors and SE leaders monitor the Defnition: Program leaders and administrators
process of implementing the EBP every 6 monitor the outcomes of EBP consumers
months and use the data to improve the every 3 months and share the data with
program. Process monitoring involves a employment specialists in an effort to
standardized approach, for example, using improve services. Outcome monitoring
a fdelity scale or other comprehensive set involves a standardized approach to
of process indicators. assessing consumers.

An example of a process indicator would Rationale: Systematic and regular collection of

be systematic measurement of how outcome data is imperative in evaluating
much time case managers spend in program effectiveness. Effective programs
the community instead of in the offce. also analyze such data to ascertain what
Process indicators could include items is working and what is not working and
related to training or supervision. use the results to improve the quality of
The underlying principle is that services they provide.
whatever is being measured is related to
implementing the EBP and is not being Key outcome indicators for each EBP are
measured to track billing or productivity. discussed in the EBP KITs. A provisional
list is as follows:
Rationale: Systematic and regular collection of
n Supported Employment —competitive
process data is imperative in evaluating
employment rate;
program fdelity to EBP.
n Integrated Treatment for Co-Occurring
Sources of information: Disorders — substance use (such as the
Stages of Treatment Scale);
1. SE leader, senior staff, and employment specialist
interviews n Illness Management and Recovery —
hospitalization rates, relapse prevention
n “Does your program collect process data plans, medication compliance rates;
regularly?” [If yes, probe for specifcs.
Frequency? Who? How (using SE fdelity n Family Psychoeducation
scale vs. other scales)? etc.] — hospitalization and family burden; and

n “Does your program collect data on consumer n Assertive Community Treatment —

service use and treatment attendance?” hospitalization and housing.

n “Have the process data affected how your

Sources of information:
services are provided?
1. SE leader, senior staff, and practitioner interviews
2. Review of internal reports and documentation, n “Does your program have a systematic method
if available for tracking outcome data?” [If yes, probe for
If evidence exists that standardized
Item response coding: specifcs: How (computerized vs. chart only)?
process monitoring occurs at least every 6 months, code How often? Type of outcome variables?
the item as “5.” Who collects data?]
n “Do you use any checklist or scale to monitor
consumer outcome (such as the Substance
Abuse Treatment Scale)?”

Evaluating Your Program 61 Appendix E: General Organizational Index Protocol

 n “What do you do with the outcome data? Sources of information:
Do your employment specialists review the data 1. SE leader interview
regularly?” [If yes, “How is the review done (for
example, cumulative graph)?”] n “Does your agency have an established team
or committee that is in charge of reviewing
n “Have the outcome data affected how your the components of your [EBP area] program?”
services are provided?” [If yes, “How?”] [If yes, probe for specifcs. who, how, when, etc.]

2. Review of internal reports and documentation,

2. QA committee member interview
if available
n “Please describe the tasks and responsibilities
If standardized outcome monitoring
Item response coding:
of the QA committee.” Probe for specifcs:
occurs quarterly and results are shared with SE team, purpose, who, how, when, etc.
code the item as “5.”
n “How do you use your reviews to improve
the program’s services?”

G11. Quality Assurance If the agency has an established QA

Item response coding:
group or steering committee that reviews the EBP or
Defnition: The agency’s quality assurance (QA) components of the program every 6 months, code the
committee has an explicit plan to review item as “5.”
the EBP or components of the program
every 6 months. The steering committee for
the EBP can serve this function.
G12. Consumer Choice About Service
Good QA committees help the agency in Provision
important decisions, such as penetration
goals, placement of the EBP within the Defnition: All consumers who receive EBP services
agency, and hiring and staffng needs. QA are offered a reasonable range of choices
committees also help guide and sustain consistent with the EBP; employment
the implementation by — specialists consider and abide by consumer
preferences for treatment when they offer
n Reviewing fdelity to the EBP model; and provide services.
n Making recommendations
for improvement; Choice is defned narrowly in this item to
refer to services provided. This item does
n Advocating and promoting the EBP not address broader issues of consumer
within the agency and in the community; choice such as choosing to engage in self-
and destructive behaviors.
n Deciding on and keeping track of key
outcomes relevant to the EBP. To score high on this item, it is not suffcient
that a program offers choices. The choices
Rationale: Research has shown that programs that must be consonant with EBP. So, for
most successfully implement EBPs have example, a program implementing
better outcomes. Again, systematically Supported Employment would score low
and regularly collecting process and if the only employment choices it offered
outcome data is imperative in evaluating were sheltered workshops.
program effectiveness.
A reasonable range of choices means
that employment specialists offer realistic
options to consumers rather than

Appendix E: General Organizational Index Protocol 62 Evaluating Your Program

 prescribing only one or a couple of choices n Specifc goals
or dictating a fxed sequence or prescribing
n Daily living skills to be taught
conditions that a consumer must complete
before becoming eligible for a service. n Nature of medication support
n Nature of substance abuse treatment
Examples of relevant choices by EBPs
(current at this writing) Rationale: A major premise of EBP is that consumers
Supported Employment are capable of playing a vital role in
managing their illnesses and in making
n Type of occupation progress towards achieving their goals.
n Type of work setting Providers accept the responsibility for
getting information to consumers so that
n Schedules of work and number of hours
they can more effectively participate in
n Whether to disclose treatment.
n Nature of accommodations
Sources of information:
n Type and frequency of followup supports
1. SE leader interview

Integrated Treatment for n “Tell us what your program philosophy

Co-Occurring Disorders is about consumer choice. How do you
incorporate consumers’ preferences in
n Group or individual counseling sessions
the services you provide?”
n Frequency of dual disorders treatment
n “What options exist for your services?
n Specifc self-management goals Give examples.”

Family Psychoeducation 2. Employment specialist interview

n Consumer readiness for involving family n “What do you do when a disagreement occurs
n Whom to involve between what you think is the best treatment
for consumers and what they want?”
n Choice of problems and issues to address
n “Describe a time when you were unable to abide
by a consumer’s preferences.”
Illness Management and Recovery

n Selection of signifcant others 3. Consumer interview

to be involved
n “Does the program give you options
n Specifc self-management goals for the services you receive?”
n Nature of behavioral tailoring n Are you receiving the services you want?”
n Skills to be taught
4. Team meeting and supervision observation
Assertive Community Treatment
Look for discussion of service options and consumer
n Type and location of housing preferences.
n Nature of health promotion
5. Chart review (especially treatment plan)
n Nature of assistance
with fnancial management Look for documentation of consumer preferences
and choices.

Evaluating Your Program 63 Appendix E: General Organizational Index Protocol

Item response coding:If all sources support that type and
frequency of EBP services always refect consumer
choice, code the item as “5.” If the agency embraces
consumer choice fully, except in one area (for example,
requiring the agency to assume representative
payeeships for all consumers), then code the item
as “4.”

Note: Ratings for both scales are based on current

behavior and activities, not planned or intended
behavior.

The standards used for establishing the anchors for the

fully implemented ratings were determined through a
variety of expert sources as well as empirical research.

Appendix E: General Organizational Index Protocol 64 Evaluating Your Program

 Evaluating Your Program

Appendix F: Outcomes Report Form

Evaluating Your Program 65 Appendix F: Outcomes Report Form

Outcomes Report Form

Quarter q January, February, March Year ______

q April, May, June
q July, August, September
q October, November, December
Reported by __________________________
Agency __________________________ Team _________________________________________________________________

About the consumer

Consumer ID _________________________ Discharge date ____/____/____ Date of birth ____/____/____

q Male Ethnicity _______________________________________________

q Female Primary diagnosis _______________________________________

What was the consumer’s evidence-based service status on the last day of the quarter?

Unknown Not Eligible Eligible Enrolled

Integrated Treatment for Co-Occurring Disorders q q q q

Supported Employment q q q q
Assertive Community Treatment q q q q
Illness Management and Recovery q q q q
Family Psychoeducation q q q q

Number of
In the past 3 months, how often has the consumer… Number of days
incidents

Been homeless?

Been incarcerated?

Been in a state psychiatric hospital?

Been in a private psychiatric hospital?

Been hospitalized for substance abuse reasons?

Evaluating Your Program 67 Appendix F: Outcomes Report Form

In the past 3 months, how many days was the consumer What was the consumer’s educational status on the last day
competitively employed? (Use 0 if the consumer has not been of the quarter? Check one.
competitively employed.)
q Not applicable or unknown
______ Days
q No educational participation

q A vocational/educational involvement
Was the consumer competitively employed on the last day
of the reporting period? q Pre-educational explorations

q Yes q Working on GED

q No
q Working on English as Second Language

q Basic educational skills

What was the consumer’s stage of substance abuse q Attending vocational school, vocational program,
treatment on the last day of the quarter? Check one. apprenticeship, or high school

q Not applicable q Attending college: 1-6 hours

q Pre-engagement q Attending college: 7 or more hours

q Engagement q Other (specify) __________________________________________

q Early persuasion
What is the consumer’s highest level of education? Check one.
q Late persuasion

q Early active treatment

q No high school

q Late active treatment

q High school diploma or GED

q Relapse prevention
q Some college

q Associates degree
q In remission or recovery
q Vocational training certifcate

What was the consumer’s living arrangement on the last q Bachelor of Arts or Bachelor of Science
day of the quarter? Check one. q Master’s degree or Ph.D.

q Not applicable or unknown

q Psychiatric hospital

q Substance abuse hospitalization

q General hospital psychiatric ward

q Nursing home

q Family care home

q Living with relatives (heavily dependent for personal care)

q Group home

q Boarding house

q Supervised apartment program

q Living with relatives (but is largely independent)

q Living independently

q Homeless

q Emergency shelter

q Other (specify) __________________________________________

__________________________________________

Appendix F: Outcomes Report Form 68 Evaluating Your Program

 Evaluating Your Program

Appendix G: Instructions for the Outcomes Report Form

Evaluating Your Program 69 Appendix G: Instructions for the Outcomes Report Form
Instructions for the Outcomes Report Form

Before you fll out the Outcomes Report Form, become familiar with the defnitions of the data elements
to provide consistency among reporters.

General data Evidence-based service status

Quarter: Check the time frame for the reporting What was the consumer’s evidence-based service status
period. Check the appropriate
on the last day of the quarter?
boxes according to these defnitions:
Year: Fill in the current year.
Eligible: Does the consumer meet the participation
Reported by: Fill in the name and title of the person
criteria for a specifc EBP? Each EBP has
who completed the form.
criteria for program participation that
should be used to determine eligibility.
Agency: Identify the agency name.

Enrolled: Is the consumer participating in a particular

Team: Write the team name or number.
EBP service or has the consumer
participated in the EBP in the past period?
Note: Aggregate data about eligibility and
About the consumer enrollment can be used to determine the
percentage of eligible consumers who
Consumer ID: Write the consumer ID that is used at your received services.
agency, usually a name or an identifying
number. This information will be accessible
only to the agency providing the service.
Incident reporting
Discharge date: If the consumer has been discharged
For the following outcomes, record the number of days
during this report period, fll in the
and number of incidents that the consumer spent in each
discharge date.
category during the reporting period.
Date of birth: Fill in the consumer’s date of birth
(example: 09/22/1950). Categories:

Gender: Check the appropriate box. Been homeless: Number of days that the consumer
was homeless and how many times
Ethnicity: Fill in the consumer’s ethnicity. the consumer was homeless during
the reporting period. Homeless refers
Primary Write the Diagnostic and Statistical to consumers who lack a fxed, regular,
diagnosis: Manual of Mental Disorders (DSM) and adequate nighttime residence.
diagnosis
. Been Number of days and incidents that
incarcerated: the consumer spent incarcerated in jails
or in other criminal justice lock-ups.

Been in a state Number of days and incidents that the

psychiatric consumer spent hospitalized primarily for
hospital: treatment of psychiatric disorders in a state
psychiatric hospital.

Evaluating Your Program 71 Appendix G: Instructions for the Outcomes Report Form
Been in Number of days and incidents that the n Engagement: Contact with an assigned case
a private consumer spent hospitalized primarily manager or counselor, but does not have
psychiatric for treatment of psychiatric disorders regular contacts. The lack of regular contact
hospital in a private psychiatric hospital implies lack of a working alliance.

Been Number of days and incidents that the n Regular contacts with a case
Early persuasion:

hospitalized consumer spent hospitalized primarily manager or counselor, but has not reduced
for substance for treatment of substance-use disorders, substance use for more than a month.
abuse reasons: including both public and private hospitals Regular contacts imply having a working
whose primary function is treating alliance and a relationship in which substance
substance-use disorders. abuse can be discussed.
n Engaged in a relationship with
Late persuasion:
a case manager or counselor, is discussing
Competitive employment substance use or attending a group, and
shows evidence of reducing use for at least
In the past 3 months, how many days was the consumer
one month (fewer drugs, smaller quantities,
competitively employed? Competitive employment
or both). External controls (such as
means working in a paid position (almost always
Antabuse) may be involved in reduction.
outside the mental health center) that would
be open to all community members to apply. n Early active treatment: Engaged in treatment, is

Competitive employment excludes consumers discussing substance use or attending a group,

working in sheltered workshops, transitional has reduced use for at least one month, and is
employment positions, or volunteering. It may working toward abstinence (or controlled use
include consumers who are self-employed but without associated problems) as a goal, even
only if the consumer works regularly and is paid though consumer may still be abusing.
for the work. n Engaged in treatment,
Late active treatment:
has acknowledged that substance abuse is
a problem, and has achieved abstinence (or
Stage of substance abuse treatment controlled use without associated problems)
but for less than 6 months.
What was the consumer’s stage of substance abuse
Record the
treatment on the last day of the quarter?
n Engaged in treatment,
Relapse prevention:

consumer’s stage of substance abuse recovery, has acknowledged that substance abuse is
according to the following nine categories: a problem, and has achieved abstinence (or
controlled use without associated problems)
n No history of substance
Not applicable: for at least 6 months. Occasional lapses, not
abuse disorder. days of problematic use, are allowed.
n Pre-engagement: No contacts with a case In remission or recovery: No problems related
n
manager, mental health counselor, or to substance use for more than one year and
substance abuse counselor. is no longer in any type of substance
abuse treatment.

Appendix G: Instructions for the Outcomes Report Form 72 Evaluating Your Program
Living arrangement family members provide substantial care,
consider such things as taking medication,
What was the consumer’s living arrangement on the last using transportation, cooking, cleaning,
These data give your agency an
day of the quarter? having control of leaving the home, and
ongoing record of the consumer’s residential status. managing money. If consumers are unable
to independently perform most daily living
n Not applicable or unknown
functions, consider family members to be
n Psychiatric hospital: Those hospitals, both providing substantial care.
public and private, whose primary function n Group home: A residence that is run by staff
is treating mental disorders. This includes who provide many functions (shopping, meal
state hospitals and other freestanding preparation, laundry, etc.) that are essential to
psychiatric hospitals. living independently.
n Those hospitals,
Substance-use hospitalization:
n Boarding house: A facility that provides a place
both public and private, whose primary to sleep and meals, but it is not seen as an
function is treating substance use disorders. extension of a mental health agency nor
n Psychiatric
General hospital psychiatric ward: is it staffed with mental health personnel.
wards located in general medical centers These facilities are largely privately run and
that provide short-term, acute crisis care. consumers have a high degree of autonomy.
n Nursing home:Facilities that are responsible for n Supervised apartment program: Consumers
the medical and physical care of consumers live (fairly independently) in an apartment
and have been licensed as such by the state. sponsored by a mental health agency.
In determining whether someone fts
n Family care home: Consumers live in single-
this category, look at the extent to which
family dwellings with nonrelatives who
mental health staff have control over
provide substantial care. Substantial care
key aspects of the living arrangements.
is determined by the degree to which
Example characteristics of control include
nonrelatives are responsible for the daily
the following:
care of consumers. Such things as medication
management, transportation, cooking, n The mental health agency signs the lease.
cleaning, restrictions on leaving the home, n The mental health agency has keys to the
and money management are considered. house or apartment.
Nonrelatives may have guardianship
responsibilities. If consumers are unable n Mental health agency staff provides onsite
to do most daily living tasks without the day or evening coverage.
aid of caretakers, consider caretakers n The mental health agency mandates that
to be providing substantial care. consumers participate in certain mental
n Lives with relatives (heavily dependent for personal health services — medication clinic, day
care): Consult consumers and relatives program, etc., to live in the house or
about how much family members are apartment.
responsible for the daily care of consumers.
An important distinction between this status Note: Consumers who receive only case
and supervised apartment program is to management support or fnancial aid are NOT
ask, “If the family were not involved, would included in this category; they are considered
the consumer be living in a more restrictive to be living independently.
setting?” In assessing the extent to which

Evaluating Your Program 73 Appendix G: Instructions for the Outcomes Report Form
n Lives with relatives (but is largely independent): Educational status
An assignment to this category requires
having information from consumers and What was the consumer’s educational status on the last
families. The key consideration relates to the These data provide your agency
day of the quarter?
degree to which consumers can perform most with an ongoing record of the consumer’s
tasks essential to daily living without being educational status.
supervised by family members.
n Not applicable or unknown
n Living independently: Consumers who live
n No educational participation: Consumer is not
independently and are capable of self-care,
including those who live independently with participating in educational activities.
case management support. This category n These are
Avocational/educational involvement:
also includes consumers who are largely organized classes in which consumers enroll
independent and choose to live with others for consistently and expect to take part for
reasons unrelated to mental illness. They may the purpose of life enrichment, hobbies,
live with friends, a spouse, or other family recreation, etc. These classes must be
members. The reasons for shared housing community based, not run by the mental
could include personal choice related to health center. Classes are those in which
culture or fnancial considerations. anyone could participate, not just consumers.
n Homeless: Consumers who lack a fxed, regular,
If any of these activities involve college
and adequate nighttime residence. enrollment, use the categories below.

n Emergency shelter: Temporary arrangements due n Consumers in this

Pre-educational explorations:

to a crisis or misfortune that are not specifcally status are engaged in educational activities
related to a recurrence of the consumer’s with the specifc purpose of working toward
illness. While many emergency shelters provide an educational goal. This includes consumers
emotional support, the need for emergency who attend a college orientation class with the
shelter is due to an immediate crisis unrelated goal of enrolling, meet with the fnancial aid
to the consumer’s mental illness. offce to apply for scholarships, or apply for
admission to enroll. This status also includes
n Other:Those who complete the form consumers who attend a mental health center-
should clearly defne this status in the sponsored activity focusing on an educational
space provided. goal (for example, campus visits with a case
manager to survey the location of classrooms;
meetings with the case manager and college
staff to secure entitlements).
n Working on General Educational Develpoment (GED):
Consumers who are taking classes to obtain
their GED dilpoma.

Appendix G: Instructions for the Outcomes Report Form 74 Evaluating Your Program
n Working on English as Second Language: n Consumers
Attending college: 1 to 6 hours.
Consumers who are taking classes who attend college for 6 hours or less per
in English as a Second Language term. This status continues over breaks, etc.,
in a community setting. if consumers plan to continue enrollment.
n Consumers who are
Basic educational skills:
This status suggests that consumers regularly
taking adult educational classes focused attend college and includes correspondence,
on basic skills, such as math and reading. TV, or video courses for college credit.
n Attending college: 7 or more hours. Consumers
n Attending vocational school or apprenticeship,
vocational program or high school: Consumers who attend college for more than 7 hours per
who are – term. This status continues over breaks, etc.,
if consumers plan to continue enrollment.
n Participating in community-based
vocational schools; Regular attendance with expectations
of completing course work is essential
n Learning skills through an apprenticeship, for assignment to this status.
internship, or in a practicum setting;
n Other:Those who complete the form
n Involved in on-the-job training to acquire should clearly defne this status in the
more advanced skills; space provided.
n Participating in correspondence courses
which lead to job certifcation; and
n Young adults attending high school.

Evaluating Your Program 75 Appendix G: Instructions for the Outcomes Report Form
 Evaluating Your Program

Appendix H: Assessor Training and

Work Performance Checklist

Evaluating Your Program 77 Appendix H: Assessor Training

Assessor Training and Work Performance Checklist

Assessment date ____/____/____

Assessor’s name ____________________________________________________________ ______________________

First Middle Initial Last Title

Agency visited ____________________________________________________________

Agency address __________________________________________________________________________________________________

Street

__________________________________________________________________________________________________
City State ZIP code

EBP assessed __________________________________________________________________________________________________

Assessor qualifcations

Yes

q 1a. Data collection and skills: Assessor’s skills are evidenced by his or her prior work experience, credentials,
or supervisor’s observations.

q 1b. EBP knowledge: Assessor’s knowledge is evidenced by his or her prior work experience, credentials,
or passing a knowledge test on a specifc EBP.

q 1c. Training: Assessors receive at least 8 hours of systematic training on chart review, interviewing techniques,
and process assessment.

q 1d. Shadowing: Assessors complete at least 1 assessment with an experienced assessor before the frst offcial
process assessment.

q 1e. Practice rating: Assessors co-rate as practice before being offcial assessors and agree exactly
with an experienced assessor on ratings for at least 80 percent of items.

____/5 Subtotal

Evaluating Your Program 79 Appendix H: Assessor Training

 Data Collection

q 2a. Contact and scheduling: With contact person, assessors identify a date convenient to site, explain
purpose of the assessment, identify information to be assembled ahead of time, and develop specifc
schedule of interviews and assessment activities.

q 2b. Number of assessors: Two or more assessors are present during the assessment visit and independently
rate all items. If agency is working with a consultant, assessor may join with consultant to conduct
assessments.

q 2c. Time management: Suffcient time is allotted and all necessary materials reviewed (2 days for
2 assessors).

q 2d. Interviewing: Interview all the sources stipulated in the protocol (for example, interviews with the
program leader, team members, and consumers).

q 2e. Completion of documents: Complete score sheet, cover sheet, and any other supplemental documents
relating to the agency.

q 2f. Documentation supporting rating: Each assessor provides written documentation for evidence
supporting the rating for each item (such as marginal notes).

q 2g. Chart selection and documentation: Chart selection follows guidelines provided in the protocol
(for example, appropriate type and number of charts). Assessors note discrepancies (such as
chart unavailability).

q 2h. Chart review: Both assessors review all charts and rate them independently.

q 2i. Resolution of discrepancies: When a discrepancy exists between sources (such as charts and SE
team members), assessors follow up with an appropriate informant (typically the SE leader or relevant
staff members).

q 2j. Independent ratings: No later than 1 day after the assessment, assessors independently complete scales
before discussing ratings.

____/10 Subtotal

Post-assessment visit

q 3a. Timely consensus: Within 5 working days after the assessment, assessors discuss their ratings
to determine consensus ratings, identifying any followup information needed. A third assessor
(for example, supervisor) may be consulted to resolve diffcult ratings.

q 3b. Inter-rater reliability: Raters agree exactly on ratings for at least 80 percent of the items. Sources
of unreliability are discussed with supervisor and strategies developed to reduce future unreliability.

q 3c. Follow up on missing data: If followup calls are needed to complete an item, information obtained
within 3 working days.

____/3 Subtotal

Appendix H: Assessor Training 80 Evaluating Your Program

 Comprehensive report writing

q 4a. Documentation of background information:

n List recipients of report in the header (usually the agency director and SE leader;
add others by mutual agreement).

n Summarize time, place, and method.

n Provide background about scale.

q 4b. Site and normative fdelity data: Provide a table with item-level (consensus) scores, along with
normative data (if available). Normative data include both national and state norms. In this table, provide
comparative site data from prior assessments. On second and later assessments, provide a graph of global
fdelity ratings over time for the site (trend line).

q 4c. Quantitative summary: Provide narrative summary of quantitative data. List strengths and weaknesses.

q 4d. Score interpretations:

n Interpret overall score

n Include other pertinent observations

n Provide overall summary

n Provide opportunity for site to comment and clarify

q 4e. Report editing: If agency is working with a consultant, consultant may write report. Assessor and
supervisor review draft of the report before it is submitted to the agency.

____/5 Subtotal

Report submission and followup

q 5a. Timely report: Report sent to agency director within 2 weeks of visit.

q 5b. Follow up on report: If agency is working with a consultant, consultant discusses report with designated
agency staff within 1 month of assessment.

____/2 Subtotal

Quality control

q 6. Quality control: Supervisor reviews assessments and gives feedback, as necessary, to assessors.
Depending on skill level of assessors, supervisor periodically accompanies assessors on assessment
for quality assurance purposes.

____/1 Subtotal

____/27 Total — Add the subtotals.

Evaluating Your Program 81 Appendix H: Assessor Training

DHHS Publication No. SMA-08-4364
Printed 2009
24770.0409.7765020404