The “Most Significant Change”

Technique

Nur Hidayati & Maaike Esselink

“Most Significant Change”

2
 What will we learn?

• Introductions
• Expectations, ground rules
• MSC and M&E
• MSC steps and practical exercises
• Significant change stories analysis
• How to use MSC in M&E system
• Training Evaluation

3
 Assignment 1 - Monitoring and Evaluation –
what is it and how do we do it?
 Individual
1.1. What is Monitoring?
1.2. What is Evaluation?
1.3. What do you monitor and evaluate in a programme you
are working on?
1.4. Do you experience any challenges in monitoring and/or
evaluating your program using your indicators so far?

Write your answers down and discuss in the class in the plenary

4
 Qualitative vs quantitative monitoring

Quantitative Qualitative

• Focus on measurement • Focus on questioning

• Closed questions • Open questions
• About ‘proving’ • About learning
(accountability) • Hard to aggregate
• Easy to aggregate • Inductive
• Deductive • Dynamic
• Static • Goal displacement is not an
• Goal displacement can be a issue
problem
5
 Most Significant Change Technique

• Form of qualitative, participatory M&E

• Based on ‘stories’ of significant change
• ‘Monitoring without indicators’
• Developed by Davies 1996 - Bangladesh
• Now used in numerous development programs and in the
public sector
• NOT a stand-alone approach to M&E

6
 Why use MSC?

• Monitor and evaluate complex project

• Identify changes
• Improve staff’s skills in observation and analysis
• Identify strategic direction
• Understand project better and improve ownership
• Provide learning process
• Improve accountability

7
 Purpose of MSC in M&E

• Facilitate improvement by focusing direction of work (what to

achieve and how)

• Contributes to summative evaluation:

– Information about unexpected outcomes
– Performance information on very best success stories
– Can inform criteria used to judge projects

8
 Impact MSC

• Staff gained more fully shared vision

• Process boosted their morale – ownerships of project
• Beneficiaries, staff, collaborators sitting together and
interpreting qualitative data & casting evaluative
judgements
• Project committees became better at conceptualising
impact

9
 Why stories?

• People tell stories naturally

• Stories can deal with complexity and context
• People remember stories
• Stories can carry hard messages /un-discussabled
• But stories not known for accuracy/truth

10
 What is Significant?
• Relative
• Important to story tellers
• A central part of MSC to surface individual values
• NOT necessarily about HUGE or BIGGEST changes
• Can be positive or negative
• Difficult to get

How to get? (probing, local language, list of changes)

11
 Steps of MSC
1. How to start and raise interest?
2. Establishing ‘domains of change’
3. Defining the reporting period
4. Collecting and writing stories of change
5. Selecting stories of change
6. Feeding back the results of the selection process
7. Verification of stories
8. Quantification
9. Secondary analysis and meta-monitoring
10. Revising the system
12
 Basics of what MSC involves
1. Determine sorts of change to monitor (domain of
changes)
2. Collect of significant change stories
3. Systematic selection of the most significant changes by
beneficiaries, stakeholders and/or staff
4. ‘Search’ for project impact (regular and in-depth
discussion) – part of content analysis
5. Communication results of selection process (feedback)
6. Monitor and review the process

13
 Establish Domains of Change

• ‘Newspaper’ look
• Categories of SC stories:
a. Changes in quality of live
b. Changes in participation in development activities
c. Changes in sustainability of organizations
d. Any other changes

14
Hand-out 1: SSQ project

15
16
 Planning the use of domains

1. Will they be used at all?

– or just a general “most significant change”
2. Who will select the domains to be used?
3. What will the domains be?
– Must use simple and clear words, that fit within the
MSC question
4. Will you ask for negative changes?
– or “changes” – regardless whether positive or negative?
5. Must the MS changes be caused by the development
initiative?
– or significant changes regardless of cause?
17
 Assignment 2 –
Developing Domains of Change
 In groups of 5
Work with the case of the SSQ program

2.1. List as many as possible domains of change

2.2. Choose domain(s) of change to be used in the story
collection process
2.3. Why do you select these domains of change?
2.4. If you will implement MSC, whom will you involve in
selecting domains of change?

Write your answers down and discuss in the

18 plenary
 Collecting SC stories

1. Whose stories will be collected?

2. Who will collect these stories?
– The same people who are in the MSC stories?
– Or others outside the story? Who and why?
3. How will they be recorded?
– Note book, audio tape, video, or…?
– How much time will this take?
– And how and where will they be stored? Why?

19
 Stories Collection: the who and how?
Who are the story tellers?
- Program participants/beneficiaries
- Project staff
- Project partners
- Stakeholders
How to collect stories?
- Individual interview
- Focus group discussion (FGD)
- Staff project writes stories they know about (activity journal)
- Individual writes their own story (beneficiaries journal)
- Video (recorded story) 20
 The core of MSC

• A question:
– “In your opinion what was the most significant change that
took place in ….over the last … months?”
• [describe the change and explain why you think it is
significant]
• Re-iteration of the same kind of question
– “Which of these SC stories do you think is the most
significant of all?”
• [describe the change and explain why you think it is
significant]

21
 Whose stories to collect?

• Depends on the Domains of Change

• Stories from those directly involved can be the most
powerful, but they can be more difficult to collect
• May be best to start with fieldworkers who are in contact
with them
• Also depends on whose perspective you want to make more
visible and open to discussion: staff or clients?

22
 Funder meeting

State meetings

flow of feedback
stories

Region 1 Region 2 Region 3 Region 4

Story tellers
23
 The voice of the story-teller

• Important to get the story…….

• You may need to re-phrase the question
(“examples of....” “a time when…..”)
who?
where?
when?
what happened?
why significant?

24
 What to document in a story? (1)
• Information on who is the story teller, who collect the story
and when Description of the story – changes that happened

• Significance (to the story teller) of the events described in the

story

• The process how changes happened

• Use a story collection form

25
 What to document in a story? (2)
1. Beginning: Things were bad
2. Middle: Then the project came
3. Ending: Then things got better

To understand the significance of the change from ‘bad’ to ‘better’, the

two situations need to be described.

For instance, if somebody tells you that they used to have to walk to
school, but now it’s much easier because they can take a bus, what
does it actually mean?
•How long did they have to walk?
•What were the implications of this for their life?
•How is the situation different now that they take the bus?
•What difference has this made beyond the time saved?
•What are they able to do with the extra time?
•This extra detail helps the reader to understand the ‘significance’ of the change
26
 What to document in a story? (3)
The story contains three central parts and ends by explaining
why the story teller believes it to be significant
PLUS:
WHY WAS
Beginning Middle End THIS
What was the situation What happened? + SIGNIFICANT?
before the change (the nature and type of The situation after.
happened? support provided) What difference did it
(captures the
(background/
make? perspective of
context e.g. before
the storyteller
they participated in xxx
project or received
and why they
Support) value the
change)

27
 Probing for a story

• Initial question is basis:

“Looking back over the 3 months, what do you think was the most
significant change in the attitudes of health staff to young people?”

• In reality: ask additional probing questions to ‘capture’ the

full story

28
 Ethics

• Explain how the stories will be used, at the time of collection

• Consider the time cost to unpaid participants
• Consider the interests of 3rd parties mentioned in stories
• Take special care if stories are to go in media
• Allow anonymity but maintain verifiability!
• Use consent tick boxes on MSC forms, verbal consent with
witnesses

29
 Example of Story of Change

Hand-out 2

30
 Who writes SC stories

• ‘Beneficiaries’ write their own stories directly

• Hired people to interview beneficiaries
• Fieldworkers write down unsolicited stories they have
heard from beneficiaries
– Or they set up interviews/focus groups to elicit stories
• Through group discussion
– Of beneficiaries - self-organised or facilitated
– Of field workers

31
 How to write MSC stories (1)

• At least:
• Who documented the story (name, position location,
date)
• Description of what happened (who, what, where,
when)
• story form, describing the sequence of events
• Explanation of why is it significant
• bring out the meaning of the story
• Use quotes to give the voice of the story teller
• Give a title to the story
32
 How to write MSC stories (2)

• Optional:
– Headline
– Recommendations (from storyteller)
– Length: Long enough to be verifiable, short enough for a
group to read many stories
• Beware:
– Over-specifying what is needed.
• Fragmenting the story, losing the plot!

33
 Assignment 3 - Making a story collection
guideline
 Work in pair

3.1. Develop MSC core questions for the story collection

guideline using the domains of change as agreed upon in the
plenary session
3.2. Prepare a story collection guideline based on the
questions made in 3.1.
Please use the backside of this paper or a separate paper

Discuss the guidelines in class to come up with a final story

collection guideline 34
 Assignment 4 - Significant Change story
collection and story writing
 Trio with 2 or 3 training participants and 1 resource person from the
SSQ project. Roles:
 1 training participant: interviewer
 1 training participant: note taker
 1 training participant: observer
 Resource person from SSQ: interviewee

4.1. Conduct an interview with one resource person as assigned, using the
story collection form just developed
4.2. Write a story of change using a laptop in couples after the interview
4.3. Verify the content of the story with the resource person from SSQ
after the story has been written
Reflect on the interviewing process in the plenary
35
 Reflection on interview process
What was challenging What was rewarding

Interviewer

Interviewee/story teller

36
 Reflection on story writing process
What was What was rewarding
challenging

Preparation
During the writing Process
Finalizing the story

37
 Assignment 5 -
Tips and Tricks for collecting and writing
stories
 Groups of 2/3 (same group during story collection)

5.1. Reflect on the process of collecting a story from the SSQ

resource person using the MSC story collection guideline?
5.2. Reflect on the process of writing the story of the SSQ
resource person using the MSC story collection guideline?

Write your answers separately on coloured paper and report

back in the plenary
38
 Story Selection

• Why selection
• Selection process
• Reason for choice
• Selection committee roles and structure
• Mini workshop

39
 Why selection?

• The heart of MSC

• Encourage discussion
• Part of analysis
• Reduces volume of stories in systematic way

40
 How to do story selection?
• Focus group discussion (facilitated discussion)
• Questions: which one is the most significant change and why?
• Transparent – always document reasons and process in each
level of selection
• Various way to arrive to a final decision:
(a)Negotiation
(b)Score/ranking
(c)Using criteria of selection
(d)Voting

41
 Stories Selection process
1. Use a diagram to show flow of stories and feedback
– Who will be involved?
– Who will send selected stories to who?
– How many levels of story selection?
– How many stories go to and from each level?
– Who participates in each level?
– And explain these choices

2. How many person-days will the process take?

– Each selection round, on average?
– From start to finish - reporting, selection, feedback cycle?
42
SC Stories Selection Process

43
Using a layers of elected representatives, or
layers of existing authorities?

44
Or parallel processes of selection?

45
 Selection committee roles

Two roles:
1. Reporting
2. Monitoring/reflection

46
 Selection committee and structure
Issues:
• Composition of selection committee (representativeness,
gender, etc)
• Consider practicalities of involving partners/beneficiaries
• Staff have difficulties to be neutral from stories they collected
• Structure and timing of the collection/selection

Discussion:
How to deal the above issues?

47
 Group Exercise: Mini Workshop
 Class into 3 groups

Setting: Story Selection Committee of program staff

Divide roles for: one facilitator to lead the selection process, one note taker,
one time keeper and discussion participants

• Read all stories: make sure all participants know the content of the
stories
• Clarify unclear information in the stories together
• Select the story reflecting the Most Significant Change by discussion
using the story selection form provided
• NOTE: Make sure notes of the process of selection are made

Reflect on the selection process and outcome in the plenary

48
 Providing Feedback

• Why important
• Who provide feedback to and what sort of feedback
• How to provide feedback

49
 Why Feedback?

• Often overlooked but essential

• To ensure findings from MSC is shared as part of M&E
• Facilitates learning and for improvement
• Improving accountability
• Opportunity to give acknowledgement

50
 What sort of feedback?

• Results of selection
• Comments on stories
• Recommendations for action
• How we have used the stories

51
 To whom feedback is provided?

• Community (beneficiaries/target groups)

• Story tellers
• Story collectors and selection committee
• Partners
• Stakeholders

52
 How to provide feedback?
• Verbal or written
• Various ways:
a. via meetings
b. Email
c. Newsletters/printed materials
d. Reports
e. Exhibitions (photos, movies/film, etc)
f. Arts (drama, mimes, etc)

Just be creative!

53
 Stories Verification

• Which stories will be verified? Any?

• By who?
• And why?
– Check and correct facts?
– Check participants’ interpretations of the event?
– Develop into a detailed case study for organisational
learning?

Recommended only for selected stories

54
 Assignment 6 - Using the results of MSC
 Groups per table
Discuss on the following questions:
6.1. How will the MSC stories be used in your
organization/program:
Internally?
By beneficiaries/partners?
By donors?
By others?

6.2. Which of these uses is most important for your

organisation/program? Why?
Report on your discussion and answers in the plenary
55
 Analysing SC Stories

• How and where will records be kept of all SCs that are
collected?

• What questions are important to ask about the whole set of

stories?

• How SC stories should be analysed and who will do the

analysis?

• Who will use the results of the analysis?

56
 MSC Analysis: What is the difference

• Primary analysis:
– What participants do when they select MSc stories. This is
the core of MSC
• Secondary analysis
– Further analysis of MSC stories by other means
– May or may not involve original participants

57
 Primary Analysis

• What are the type of changes?

• Select the MOST significant change story
• Always document reasons for choices in selection process
• Analysis positive and negative changes (if any)
• In what level changes happen? (individual, community,
organization, etc)

58
 Two types of secondary analysis
• Meta-monitoring:
– Tracking who provides the stories, and who does not
– Whose stories get selected, or not
• Content analysis
– Looking into the content of the stories
• Selected and/or unselected stories
• Basic Quantification, what can be quantified?
• Against logic model (or Theory of Change)
• Is the changes ‘one-off’ or often happen?
• How the changes happen and how we observe them?
E.g. the use of Progress Marker (see hand outs)
59
 Assignment 7 –
Hierarchical Card Sorting
 Groups per table

• The group sorts stories into 2 piles based on the most

important difference and document the reasons why they do so
• Each of these piles are sorted into another 2 piles and again the
reasons for the sorting process is documented.
• Continue this process of sorting to end up with story(ies) which
cannot be sorted anymore. Document the reason why sorting is
not possible anymore (Question: Is it necessary?)

Make notes of all reasons for sorting and of the process

60
 A definition of content analysis…

“Content analysis has been defined as a systematic,

replicable technique for compressing many words of text
into fewer content categories based on explicit rules of
coding”
http://pareonline.net/getvn.asp?v=7&n=17

61
 Why do it?
• Meta Monitoring:
– Find out how to improve coverage of stories.
• Diversity of sources matters
• Content Analysis:
– Track specific issues of concern without changing the
ongoing MSC process
– Identify wider trends that may not emerge through MSC
selection process

62
 Resources

www.mostsignificiantchange.com

63
 Meta-Monitoring: key tasks
• Record
– Who collects each story, from whom
– Who was involved in each selection process
• Even their “votes” for specific stories?
– Types of MSC stories of interest e.g neg

64
 Introducing MSC in an Organization

• Key resources to learn from

– Build on prior experience
• Finding champions and drivers
• Pre-testing
• Training
• Considering duration and sustainability

65
 Champions and Drivers

• Who will ‘authorise’ the use of MSC

– and who will enthuse others?

• Who will push and pull to make it work?

– answer questions about the method
– help design the proposed use of MSC
– help plan or run training
– monitor and guide the MSC process

66
 Pre-testing MSC

• Advisable in large organisations or projects or ‘new’

alliances
• Enables early identification and resolution of problems
• Creates enthusiasm/support
• Helps customise MSC process

67
 Duration and Sustainability
• MSC does not have to be used for ever!
– E.g.
• When used as part of an evaluation or planning
process
• When a project or action research process ends
• If you do want to sustain its use then…
– Refresher training
– System for on-going adaptation in order to meet current
needs
– Dedicated resources and people

68
 Think about your context

• Who might be a good champion in your organisation, or even

outside it?
• Who could drive the use of MSC
• What sort of training would be needed?
• What sort of a pre-test would be useful?
• How long will you need to use MSC?

69
 Assignment 8 - Using MSC in your project
 Groups of 5 participants

You are interested in using MSC in your programme/organisation

8.1. What issues need to be considered for implementation of MSC in your
programme/organization?
8.2. How you will implement MSC in your programme/organisation? List
all steps
8.3. Discuss the use of domains

Make notes of your discussions. Share observations and queries in plenary

Make notes of all reasons for sorting and of the process

70
 MSC steps (full version)

1. How to start and raise interest?

2. Establishing ‘domains of change’
3. Defining the reporting period
4. Collecting and writing stories of change
5. Selecting stories of change
6. Feeding back the results of the selection process
7. Verification of stories
8. Quantification
9. Secondary analysis and meta-monitoring
10. Revising the system

71
 What will be M&E
measured
Outputs - Quantitative and qualitative data
- Monthly reporting
- Output indicators
- Information traceability

Outcomes - Quantitative and qualitative data

- Design monitoring technique and tools
Most - Semi-annual or annual reporting
significant - Sampling method Qualitative analysis
change - Outcome indicators Learning
- Information traceability
Impact Evaluation; Triangulation among M&E data (output,
outcome and MSC data) to identify evidence for
changes happened and possible impacts on the
long run
 Assignment 9 - MSC in the M&E of your
organisation
 Groups of 5

9.1. Discuss how timing and integration of MSC in your

organisation’s/program’s M&E framework might work
9.2. Where do you expect there will be most need for adjustment of your
M&E framework? Why?
9.3. Where have you already made changes to the design? Why?
9.4. Draft your plan of action on integrating MSC in your
organisation’s/program’s M&E framework

Use a flipchart to draw and/or explain the integration

Make notes of your discussions.
Share observations and queries in plenary and present your plan of action
73
 More info for Assignment 9
Timing and Integration

1. How often will SC stories be collected?

– Weekly, monthly, quarterly, yearly…
– Pros and cons for most choices
2. How will MSC related to existing systems?
– How will it build on existing P,M&E system?
– What extra requirements will there be?

74
 Tuning the system

• Possible features to modify

– Domains
– Frequency of collection
– Structure of selection processes
– Who participates
– What happens to the SC stories
– Other?

75
 Key resources to learn from

• MSC Guide (2005) at

http://www.mande.co.uk/docs/MSCGuide.htm

• http://www.mande.co.uk/docs/MSCGuide.pdf

• MSC Mailing list is at

http://groups.yahoo.com/group/mostsignificantchanges
Including Files section

 Google “Most Significant Change”

76