Evaluation of Training

Chapter 4

Effectiveness
A relative term
Effectiveness is determined with respect to the achievement of a goal or a set of goals Must be determined with respect to the goals of the program or programs being examined

Training Effectiveness
Training effectiveness refers to the benefits that the company and the trainees receive from training Training outcomes or criteria refer to measures that the trainer and the company use to evaluate training programs

Effectiveness of Training Programs

T&D Program can be effective in meeting some goals
staying within budget increasing a participants skills

and be ineffective in meeting others

Improving customer satisfaction

How do you ensure effectiveness?

Training and HRD Process

Assessment A ssess needs Design Implementation E valuation P rioritize needs Define objectives Develop lesson plan Develop/ acquire materials S elect trainer/ leader S elect methods and techniques S chedule the program/ intervention Deliver the HRD program or intervention S elect evaluation criteria

Determine evaluation design

Conduct evaluation of program or intervention

Interpret results

Purpose of HRD Evaluation

HRD evaluation:
The systematic collection of descriptive and judgmental information necessary to make effective training decisions related to the selection, adoption, value, and modification of various instructional activities.

HRD Evaluation
Both descriptive and judgmental information may be collected
Descriptive information provides a picture of what is happening or has happened Judgmental information communicates some opinion or belief about what has happened

HRD Evaluation
Evaluation involves the systematic collection of information
According to a predetermined plan to ensure that the information is appropriate and useful

Evaluation is conducted to help make informed decisions about particular programs and methods

HRD Evaluation Can Help

Determine whether a program is accomplishing its objectives Identify the strengths and weaknesses of HRD programs Determine the cost-benefit ratio of an HRD program Decide who should participate in future HRD programs Identify which participants benefited the most or least from the program Gather data to assist in marketing future programs Establish a database to assist management in making decisions

Training Evaluation
Training evaluation refers to the process of collecting the outcomes needed to determine if training is effective Evaluation design refers to from whom, what, when, and how information needed for determining the effectiveness of the training program will be collected

Reasons for Evaluating Training

Companies are investing millions of dollars in training programs to help gain a competitive advantage Training investment is increasing because learning creates knowledge which differentiates between those companies and employees who are successful and those who are not

Training evaluation provides the data needed to demonstrate that training does provide benefits to the company.

Formative Evaluation
Formative evaluation evaluation conducted to improve the training process Helps to ensure that:
the training program is well organized and runs smoothly trainees learn and are satisfied with the program

Provides information about how to make the program better

Summative Evaluation
Summative evaluation evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program May also measure the return on investment (ROI) that the company receives from the training program

Why Should A Training Program Be Evaluated?

To identify the programs strengths and weaknesses To assess whether content, organization, and administration of the program contribute to learning and the use of training content on the job To identify which trainees benefited most or least from the program

Why Should A Training Program Be Evaluated?

To gather data to assist in marketing training programs To determine the financial benefits and costs of the programs To compare the costs and benefits of training versus non-training investments To compare the costs and benefits of different training programs to choose the best program

The T&D Evaluation Process

Conduct a Needs Analysis

Develop Measurable Learning Outcomes and Analyze Transfer of Training

Develop Outcome Measures

Choose an Evaluation Strategy

Plan and Execute the Evaluation

Evaluation Frameworks
Model
Kirkpatrick (1967, 1987, 1994)

Training Evaluation Criteria

Four levels: Reaction Learning Job Behavior Results

Evaluation Frameworks
Model
CIPP (Galvin, 1983)

Training Evaluation Criteria

Four levels: Context Input Process Product

Evaluation Frameworks
Model
Brinkerhoff (1987)

Training Evaluation Criteria

Six stages: Goal Setting, Program Design, Program Implementation, Immediate Outcomes Intermediate or Usage Outcomes Impacts and Worth

Evaluation Frameworks
Model
Kraiger, Ford, & Salas (1993)

Training Evaluation Criteria

Classification scheme that specifies three categories of learning outcomes cognitive skill-based affective Evaluation measures appropriate for each category of outcomes

Evaluation Frameworks
Model
Holton (1996)

Training Evaluation Criteria

Identifies five categories of variables and the relationships among them: Secondary Influences Motivation Elements Environmental Elements Outcomes Ability/Enabling Elements

Evaluation Frameworks
Model
Phillips (1996)

Training Evaluation Criteria

Five levels: Reaction and Planned Action Learning Applied Learning on the Job Business Results Return on Investment

Saratoga Institute Approach

Elements
1. Training satisfaction 2. Learning Change

Hallmarks
The degree to which participants are satisfied with the training they have received The actual learning that has occurred, with pre-and-post-course instruments The on-the-job change in behavior as a result of the training program Did the change in behavior positively affect the organization?

3. Behavior Change

4. Organizational Change

Kirkpatricks Framework

Reaction
Did trainees like program? Did trainees think it valuable?

Learning
Did they learn what objections said they should learn?

Job Behavior
Did they use learning back on job?

Results
Has HRD improved organizations effectiveness?

Kirkpatrick and Industry

Most organizations do not collect information on all four types of outcomes About one-third of organizations use Kirkpatricks model Some feel it only measures after training Others feel it is more of a taxonomy of outcomes

Brinkerhoffs Six Stages

Goal Setting:
What is the need?

Program Design:
What will work to meet the need?

Program Implementation:
Is it working, with the focus on the implementation of the program?

Immediate Outcomes:
Did participants learn?

Intermediate or Usage Outcomes:

Are the participants using what they learned?

Impacts and Worth:

Did it make a worthwhile difference to the organization?

Classification of Learning Outcomes

Category Learning Constructs Focus of Measurement Potential Evaluation Methods

Cognitive
Verbal knowledge Declarative knowledge

Outcomes
Amount of knowledge Accuracy of recall Speed, accessibility of knowledge Recognition and recall tests
Power tests Speed tests

SOURCE: K. Kraiger, J. K. Ford, & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, table 1, 323. Copyright 1993 by the American Psycho logical Association. Adapted with permission.

Classification of Learning Outcomes

Category Cognitive Knowledge organization Mental Models Learning Constructs Focus of Measurement Outcomes Similarity to ideal Interrelationships of elements Free sorts Structural assessment Potential Evaluation Methods

SOURCE: K. Kraiger, J. K. Ford, & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, table 1, 323. Copyright 1993 by the American Psycho logical Association. Adapted with permission.

Classification of Learning Outcomes

Category
Cognitive Cognitive strategies Self insight Metacognitive skills

Learning Constructs

Focus of Measurement
Outcomes Self-awareness Self-regulation

Potential Evaluation Methods

Probed protocol analysis Self-report Readiness for testing

SOURCE: K. Kraiger, J. K. Ford, & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, table 1, 323. Copyright 1993 by the American Psycho logical Association. Adapted with permission.

Classification of Learning Outcomes

Category
Skill Based
Compilation Composition Proceduralization

Learning Constructs

Focus of Measurement
Outcomes
Speed Fluidity of performance Error rates Chunking Generalization Discrimination Strengthening

Potential Evaluation Methods

Targeted behavioral observation Hands-on testing Structural situational interviews

SOURCE: K. Kraiger, J. K. Ford, & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, table 1, 323. Copyright 1993 by the American Psycho logical Association. Adapted with permission.

Classification of Learning Outcomes

Category
Skill Based Automaticity Automatic processing Tuning

Learning Constructs

Focus of Measurement
Outcomes Attentional requirements Available cognitive resources

Potential Evaluation Methods

Secondary task performance Interference problems Embedded measurement

SOURCE: K. Kraiger, J. K. Ford, & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, table 1, 323. Copyright 1993 by the American Psycho logical Association. Adapted with permission.

Classification of Learning Outcomes

Category
Affective Attitudinal Target objective

Learning Constructs

Focus of Measurement
Outcomes Attitude direction Attitude Strength Accessibility Centrality conviction

Potential Evaluation Methods

Self report measures

SOURCE: K. Kraiger, J. K. Ford, & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, table 1, 323. Copyright 1993 by the American Psycho logical Association. Adapted with permission.

Classification of Learning Outcomes

Category
Affective Motivation Motivational disposition

Learning Constructs

Focus of Measurement
Outcomes Mastery versus performance orientations Appropriateness of orientation

Potential Evaluation Methods

Self report measures

SOURCE: K. Kraiger, J. K. Ford, & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, table 1, 323. Copyright 1993 by the American Psycho logical Association. Adapted with permission.

Classification of Learning Outcomes

Category
Affective Motivation Self-efficacy

Learning Constructs

Focus of Measurement
Outcomes

Potential Evaluation Methods

Perceived performance capability Goal Setting Level of goals Complexity of goal structures Goal Commitment

Self-report measures Free recall measures Free sorts

SOURCE: K. Kraiger, J. K. Ford, & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, table 1, 323. Copyright 1993 by the American Psycho logical Association. Adapted with permission.

Shortcomings of Kirkpatricks Model of Training Evaluation

Lack of explicit causal relationships among the different levels Lack of specificity in dealing with different types of learning outcomes Lack of direction concerning which measures are appropriate to assess which outcome measures

Expanded Framework of Kirkpatrick Model

Reaction
Perceived usefulness/utility
What was the perceived relevance/usefulness of this training?

Post-training attitudes
How well did trainees like training?

Expanded Framework of Kirkpatrick Model

Cognitive learning
How much did trainees learn from the training? Post-training learning
How much learning does the trainee demonstrate immediately after training?

Retention
How much learning does the trainee demonstrate back on the job?

Expanded Framework of Kirkpatrick Model

Behavior
What behavior change occurred as a result of training? Training performance
How well can trainees demonstrate the newly acquired skills at the end of training?

Transfer performance
How well can trainees demonstrate the newly acquired skills back on the job?

Expanded Framework of Kirkpatrick Model

Results
What tangible outcomes or results occurred as a result of training? What was the return on investment (ROI) for this training?
(See ROI and utility sections below; this is Phillips Level 5)

What was the contribution of this training program to the community/larger society?

Outcomes Used in Evaluating Training Programs

Cognitive Outcomes

Skill-Based Outcomes

Affective Outcomes

Results

Return on Investment

Outcomes Used in Evaluating Training Programs:

Cognitive Outcomes
Determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, or processes emphasized in the training program Measure what knowledge trainees learned in the program

Skill-Based Outcomes
Assess the level of technical or motor skills Include acquisition or learning of skills and use of skills on the job

Outcomes Used in Evaluating Training Programs:

Affective Outcomes
Include attitudes and motivation Trainees perceptions of the program including the facilities, trainers, and content

Results
Determine the training programs payoff for the company

Outcomes Used in Evaluating Training Programs:

Return on Investment (ROI)
Comparing the trainings monetary benefits with the cost of the training direct costs indirect costs benefits

Good Outcomes: Relevance

Criteria relevance the extent to which training programs are related to learned capabilities emphasized in the training program Criterion contamination extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions Criterion deficiency failure to measure training outcomes that were emphasized in the training objectives

Criterion deficiency, relevance, and contamination:

Outcomes Measured in Evaluation

Outcomes Related to Training Objectives

Outcomes Identified by Needs Assessment and Included in Training Objectives

Contamination

Relevance

Deficiency

Good Outcomes (continued)

Reliability degree to which outcomes can be measured consistently over time Discrimination degree to which trainees performances on the outcome actually reflect true differences in performance Practicality refers to the ease with which the outcomes measures can be collected

Training Program Objectives and Their Implications for Evaluation:

Objective Learning Transfer

Outcomes
Reactions: Did trainees like the program? Did the environment help learning? Was material meaningful? Pencil-and-paper tests Performance on a work sample Skill-Based: Ratings by peers or managers based on observation of behavior

Cognitive: Skill-Based:

Affective: Results:

Trainees motivation or job attitudes Did company benefit through sales, quality, productivity, reduced accidents, and complaints? Performance on work equipment

A Stakeholder Scorecard to Training Evaluation

SENIOR MANAGEM ENT CONTRIBUTIONS Measure 1 Measure 2 Measure 3 INDUCEMENTS Measure 1 Measure 2 Measure 3

TRAINEES CONTRIBUTIONS Measure 1 Measure 2 Measure 3 INDUCEMENTS Measure 1 Measure 2 Measure 3 TRAINING

TRAINEES MANAGER S CONTRIBUTIONS Measure 1 Measure 2 Measure 3 INDUCEMENTS Measure 1 Measure 2 Measure 3

TRAINERS CONTRIBUTIONS Measure 1 Measure 2 Measure 3 INDUCEMENTS Measure 1 Measure 2 Measure 3

Major Goal of Training Evaluation

Making HRD programs investments
Leading to measurable payoffs in the future

Two practical Methods

Evaluation of training costs
Return on Investment (ROI)

Utility analysis

Types of Cost Analysis

Cost-benefit analysis
comparing monetary costs of training to benefits received in non-monetary terms
improvements in attitudes, safety, and health

Cost-effectiveness analysis
financial benefits accrued from training
increases in quality and profits reduction in waste and processing time

Return on Investment (ROI)

Most common business ratio for determining performance ROI = Results Training Costs If ROI < 1, training costs more than benefits accrues If ROI > 1 benefits accrue Greater the ratio, the better the benefit

Types of Costs in Training

Direct Costs Directly associated with delivery of learning activities Course materials reproduced or hired Instructional aids Equipment rental Travel Food Instructors salary & benefits

Types of Costs in Training

Indirect Costs Incurred in support of learning activities but not directly Instructor prep Clerical & admin support Course materials already distributed and therefore not recoverable if program cancelled Marketing the program

Types of Costs in Training

Development Costs Development of videotapes, DVDs, CBI Design of program materials Piloting the program Any necessary redesign after piloting

Types of Costs in Training

Overhead Costs Not related directly to any training program but essential for operating effort Maintaining equipment Heat, light Cost of dedicated resources not in use for specific program

Types of Costs in Training

Compensation for participants Salaries and benefits paid to participants for the time in a training program Individual data not available, but HR should provide average for all participants

Increasing ROI Credibility through Training

Use conservative cost estimates
Error on high side

Find reliable estimate sources Explain all assumptions and techniques used to calculate costs Rely on hard data whenever possible Use the Balanced Scorecard shown earlier

Training Cost Analysis

Calculate direct costs Calculate indirect costs Calculate development costs Determine overhead costs Determine compensation for participants Sum total costs Divide by number or trainees to get cost per participant

Calculating ROI on Training

ROI = Return = Operational Results Investment Training Costs

Goal of Using Cost-Benefit Analysis

Put HRD on equal footing as other managers Language of business is money Results must be quantifiable Results need to be expressed statistically Need to
Demonstrate expected gains of HRD programs Compete with needs of other managers for equipment, facilities, personnel, etc.

Increasing Managerial Acceptance for Training

Involve senior management in determining the utility model and procedures to be used Train HR professionals and managers in the details of utility analysis Offer an explanation of the components of the utility model
Focus on utility information as a communication tool to aid in decision making

Increasing Managerial Acceptance for Training Involve management in arriving at estimates Use credible and conservative estimates Admit that the results of utility analysis are often based on fallible but reasonable estimates Use utility analysis to compare alternatives, rather than to justify individual programs

Impediments to Effective Training

There are many impediments, which make a training program ineffective, these are as follows: q q q q q q q Management commitment might be lacking or be uneven Educational institutions award degrees but graduates lack skills Aggregate spending on training is inadequate Large-scale Poaching of trained workers No help to workers displaced because of downsizing Employers & B Schools must develop closer ties Organized labor can help.

Impediments to Effective Training

Employees Performance on present job Employee identified for Training Training Program

Measurement of Actual Performance

Employee for the new assignment

Stability for the new assignment

Employee after Training

Measurement of Training Performance

Fig: Model for Feedback on Training

Factors That Influence the Type of Evaluation Design

Factor Change potential Importance Scale Purpose of training Organization culture Expertise Cost Time frame How Factor Influences Type of Evaluation Design Can program be modified? Does ineffective training affect customer service, product development, or relationships between employees? How many trainees are involved? Is training conducted for learning, results, or both? Is demonstrating results part of company norms and expectations? Can a complex study be analyzed? Is evaluation too expensive? When do we need the information?