INSTITUTE OF DISTANCE AND LEARNING,

UNIVERSITY OF MUMBAI, SANTACRUZ(E)

MA Education Programme Open Book

Cover Page

Full Name of the Student: - Khade Arati Kashinath

Academic Year AND Semester: - 2022 - 2023

Admission Application No: - 160793

Email ID: - annumskhan66@gmail.com

Paper Name: - OPEN AND DISTANCE LEARNING

Paper Code: - EC6

Language of writing assignment: - English

Assignment Submission Date: - 05-11-2022

Total Page No: - 13

Signature of the Student: -

33
 EC6: OPEN AND DISTANCE LEARNING
1) Evaluate the types of Open and Distance Learning Evaluation and Explain each
evaluation type critically.

INTRODUCTION:

It is so fitting that we end our exploration of distance education with a lesson about evaluation!
You probably know from any of the design models you chose for your final project there is an
element of evaluation that should be worked into your design. As your chapter and the
additional article point out, the term “evaluation” is sometimes (incorrectly) used
interchangeably with “assessment” and “research.” It is best to keep the terms separate as their
purposes are different. Using the term “assessment” works well when
discussing learner achievement and may be used in an evaluation. Evaluation, in the context
of learning, performance, and change initiatives, can be defined as a form of inquiry that seeks
to address critical questions concerning how well a program, process, product, system or
organization is working. The results are designed to benefit a variety of stakeholders for them
to use for decision-making purposes and, if shared throughout an organization, may lead to
organizational learning. Three types of evaluation that are performed to provide a variety of
results are formative, summative and developmental. (We will focus on Formative and
Summative):

Evaluation in the sense of "the systematic investigation of the worth or merit of some object"
has several meanings within education. Its most common purpose is to grade and classify
students, to sort out the successes from the failures. However, here we are concerned with
"Programme Evaluation", where the focus is on how well a particular educational programme,
curriculum or teaching method works, how it might be improved and how it compares with
alternatives.

34
Open and Distance Learning (ODL) system is a system wherein teachers and learners need not
necessarily be present either at same place or same time and is flexible in regard to modalities
and timing of teaching and learning as also the admission criteria without compromising
necessary quality considerations. ODL system of the country consists of Indra Gandhi National
Open University(IGNOU), State Open Universities (SOUs), Institutions and Universities
offering education and includes Correspondence Course Institutes (CCIs) in conventional dual
mode universities. This is becoming more and more significant for continuing education, skill
updating of in service personnel and for quality education of relevance to learners located at
educationally disadvantageous locations.

MEANING OF EVALUATION

Evaluation is an integral part of the instructional process and involved in the following three
steps.

a) identifying and defining the intended learning outcomes

b) constructing and selecting texts and other evaluation tools relevant to the specified outcomes

c) using the evaluation results to improve learning and teaching

Evaluation is an inseparable component of teaching-learning activity where judgments need to

be made. The teaching and learning activities are the continuous process linked with the
instructional decisions to promote learners’ learning. The main concern of evaluation in DE is
to judge the effectiveness of instruction which relate to the causes of quality of DE. In majority
cases evaluation report is considered for taking a decision on the issue on which the evaluation
is done.

Teachers/instructors involve in educational system tend to ascertain the output of an

educational programme. Output is counted in terms of test results. It presupposes results are
expressed in quantitative indices, such as scores or marks. For obtaining those scores a device
consisting of a set of tasks, called a ‘test’. Tests may consist of assignments, tem-end question
papers, interviews, group discussions, projects, and so on. Thus from the instructional point of
view evaluation may be defined as a systematic process of determining the extent to which
instructional objectives are achieved by learners

35
Wottawa and Thierue (1990) made an attempt to explain the concept of evaluation which can
serve different purposes. They are:

i) Evaluation has something to do with valuation

ii) ii) Evaluation serves to help in planning and deciding and thus has something to do
with assessment and valuation of alternative ways of acting.

iii) iii) Evaluation is oriented towards aims and purposes. It primarily has the aim of
checking practical measures, of improving them or of making decisions concerning
them.

iv) iv) Evaluation measures reflect the current state of techniques and research
methods.

COURSE EVALUATION IN DE

DE courses are designed and developed under ideal conditions. The reason for adopting
this method of development is to improve learners’ study conditions since they are at a
distance place. The evaluation of a course is necessary to know its effectiveness and
learners’ reaction to that course. And, whether the materials suits to the learners’ actual
study conditions or not? For evaluating a course the evaluator needs the feedback or
information from different audiences; such as, learners, course writers, buyers/users, and
other stakeholders, etc.. The evaluator(s) will collect the information/feedback by using
various tools and techniques to the audiences. Course evaluation is a continuous process to
maintain the quality of course from time to time, and changes made in the design of learning
materials (instructional design and curriculum design) if it is required. The aim of course
evaluation is to identify whether a course is well managed, well presented, and up to date.
When undertaken periodically, evaluation helps statements to be made about the
appropriateness, worth, coherence and relative balance of the learning situations provided
in relation to their intended outcomes. Thus the purpose of course evaluation is to improve
the quality and effectiveness of the teaching and learning that takes place

36
PROGRAMME EVALUATION IN DE

Programme evaluation is a management tool. It is a time bound exercise that attempts to access
systematically and objectively the relevance, performance and success of ongoing and
completed programmes and projects. Programmes must be evaluated to decide if the
programmes are actually useful to the learners and whether they are achieving their stated
objectives.

Programme evaluation is about carefully

collecting information on a programme or
some aspects of a programme in order to
make necessary decisions. Programme
evaluation can include different types of
evaluation, such as needs assessment,
accreditation, cost effectiveness, formative,
and summative evaluations. Evaluation is
undertaken to guide decision-makers and/or programme managers/coordinators, in order to
provide information on, whether the underlying theories and assumptions used in programme
development were valid; or to see which of them worked and which did not, and why.
Evaluation commonly aims to determine the relevance, efficiency, effectiveness, impact and
sustainability of a programme or project.

The key elements that can be examined in programme evaluation are;

• the inputs that were selected and considered for the development of a programme

• the process that were used for the production of a programme

• the products and out puts that were achieved

• the outcomes/impacts affected to the learners. Before elaborating further the concept
‘programme evaluation’, let us understand what is a ‘programme’ in the context of DE.

37
What is a Programme?

A programme is a set of activities designed to produce certain desired effects or outcomes to

meet some set of standard or norm, a recognized social need or to solve an identified social
problem. For example, philosophy programmes aim at to develop rational and logical thinking
of learners where they can understand various arguments made by different schools of thought.
A programme on Income Taxation guides learners to know the rules and regulations of tax
benefit, tax deduction, and individual’s benefit.

Programmes are of three types. These are;

a) Degree programmes

b) Diploma programmes

c) Certificate programmes

Degree programme are of usually three years but extended to some years more in distance
education set up. This is so because distance learners are heterogeneous in character and they
have equally other responsibilities, such as, office work, family responsibility, etc. in addition
to complete their enrolled programme in DE institutions. They are part time and adult learners.
In the similar way, diploma and certificate programme though are of one year and six months
respectively but can be extended to a few years more in DE set up. After running a particular
programme in a few years in the DE institution, it needs evaluation for the reasons to see how
the programme performs, whether the expected outcomes of the programme are achieved or
not, whether target learners have any suggestions and or comments on this programme, and
how it can be effectively delivered to the target population with all ramifications.

With these understanding, it will be comfortable for you to understand the differences between
‘Evaluation in an Educational Programme (EIEP)’ and ‘Evaluation of an Educational
Programme (EOEP)’. We will discuss these two concepts and bringing out their differences in
detail. Evaluation as an essential component of DE has implications both in terms of evaluating
learners’ progress and attainment, and evaluating the effectiveness of the whole programme.
The former refers to ‘evaluation in DE’ and the later to ‘evaluation of DE’ programmes.

38
Evaluation in an Educational Programme (EIEP)

The concept ‘evaluation’ in the present form enlarges its scope and domain. It encompasses
the whole educational programme, which includes three essential components, and these are;

a) educational objectives

b) learning experiences

c) evaluation procedures These three components must support one another to achieve the
desired effectiveness of the programme. The function of evaluation procedures is presented
below in the graphical format.

In this evaluation, the evaluators are to examine to what extent the pre stipulated learning
objectives of a programme have been achieved in terms of learners’ academic achievement.
Further, they will examine programme suitability, programme activities, and the improved
effectiveness of a programme.

The evaluation lies in three stages of a programme.

i) at the beginning of a programme

ii) during the programme
iii) at the end of a programme

At the initial stage, the programme planners use protests to study the existing knowledge
of learners about the proposed programme before they enter it, so that learning experiences,
appropriate to the concerned learners can be suitably designed and effectively presented.
At the mid process stage, they monitor the learners’ academic progress by providing

39
feedback to them. At the end of a programme, they assist in measuring the learning
outcomes of learners.

Within the process-product relationship in an educational programme, the process of

evaluation may help in

• Diagnosing weakness in learning

• Predicting learners’ aptitudes and abilities

• Selecting suitable persons for a course or career through entrance tests

• Guiding learners’ abilities through tests/examinations

• Providing guidance for course choice and subject choice within a course

• Evaluating the effectiveness of the whole programme

Evaluation of an Educational Programme (EOEP)

Evaluation of an educational programme is understood in wider context in comparison to

evaluation in an educational programme. In this case, the evaluators analyse the related
components, process, and measure the variables associated with the programme. In a sense,
evaluate the programme as a whole. The report of this evaluation is considered for
accountability and cost benefit information. It gives an overall picture of a programme’s
running status and present market demand

PROCESS AND TYPES OF EVALUATION IN DE

To evaluate a programme in DE, one needs to adopt the appropriate and effective
methodology. While evaluating a programme, the evaluator should not refrain from the
following tasks.

a) assessing the learning outcomes at different levels

b) examining learners’ view on the programme

c) feedback on the quality of the programme

d) market demand and impact of the programme

e) worth of opting the programme

40
Thus, the evaluation report is to be considered seriously and worth for decision making.
There are four types of evaluation found in an educational programme. These are

i) Document evaluation
ii) Formative evaluation
iii) Summative evaluation
iv) Impact evaluation.

i) Document Evaluation

It provides a descriptive record of the evaluation process. It supplies a contextual record of

all evaluation events to the institution. Further, it facilitates the development of a
benchmark system. It helps in the revision of a programme. This evaluation is used for two
purposes. First, documentation of programme and lesson development, second; giving
support to other types of evaluation. In the case of first, the examples are; evaluating
learners’ records block instruction, anecdotal records, and programme reviews both
informal and formal, and evaluation of academic counsellors. In the case of second, the
evaluation will take a stipulated time period. It implies that without proper documentations,
other types of evaluation would not be feasible in DE context.

ii) Formative Evaluation

It is a continuous activity. This evaluation is used for the purpose of improving the
components associated in teaching and learning process at a distance. It helps to modify
and improve the quality of the programmes while they are at the development stage. It
includes reviewing all instructional materials, assessment tools and techniques, accuracy of
instructional contents, ensuring all instructional objectives are covered through the
instruction, monitoring learners’ performance and learning strategies. The purpose of
formative evaluation is not to measure the effectiveness of a programme, but to identify
any changes it may require which would improve the programme. Thus, it is stated that
formative evaluation is the systematic collection of information for the purpose of making
informed decisions about designing and improving educational programmes.

Formative evaluations are usually performed to improve a product or a program and are
often performed by members of the staff. These internal formative evaluations are
designed, throughout the process to improve or refine what is being evaluated. The results
of a formative evaluation can be used to help an organization come closer to a goal of a

41
product or program. Instructor feedback and target audience member feedback are helpful
in this process.

iii) Summative Evaluation

It is used to judge the effectiveness of a programme. While doing that it focuses on the
quality of distance teaching components and effectiveness of the support systems provided
to the learners of the programme. It validates all the courses of a programme. An evaluator
while doing the summative evaluation of a programme, following considerations is must.

a) have the programme achieved as it was intended or expected?

b) how it can be revised in future?

c) does the programme stand as a model for other programmes?

d) Is the cost appropriate for the programme? In short, summative evaluation reports on the
effectiveness of the curriculum of a DE programme.

Summative evaluations are different from formative evaluations as they are usually
performed to decide if a product or program is worth keeping, if it needs changes or an
overhaul or even if it has merit. Its results have more of a final judgment of a program that
is mature or has already been completed. Unlike the formative evaluation the results are
not necessarily designed to improve what is being evaluated; instead the results may help
stakeholders decide if the product or program should stay in existence, or if it “passed” or
“failed.” There are various types of summative assessments such
as monitoring and auditing where some fine-tuning can occur, outcome which can be an
ongoing evaluation schedule for an organization program participants, and impact which
focuses on the participants of a particular program.

42
iv) Impact Evaluation

It focuses on the preparation of the design of a programme in such a manner that learners’
after completion of a programme uses their knowledge at their workplace. If a learner does
make any difference after completion of the programme towards the real world situations,
then the instruction has a very little value. Impact evaluation is a continuous process. This
form of evaluation includes surveys and questionnaires, formal evaluation of research
projects, etc. The report of this evaluation can be used to improve the curriculum,
instructional contents, and assessment methods.

Developmental evaluation is a form of long-term evaluation where the evaluator is more

closely linked to the stakeholders during the development of a program and is actually a
member of the design team. Feedback is provided in real-time as the project progresses.
There is a wonderful metaphor from one of the readings “When the cook tastes the soup,
that’s formative; when the guests taste the soup, that’s summative.” An organization
can implement all three types of evaluations depending on their need and where they are in
the program or product’s design or implementation. Looking more closely at Kirkpatrick’s
evaluation approach (discussed in Chapter 12) he provides four levels of evaluation
outcomes: In the chart below each level is named, the purpose is described and specific
challenges or limitations of each level area listed. As a designer, you will want to consider
what outcome you are looking for? What do you want know about your
program/training/lesson? Do you just want to know if the learners “liked it?” Do you want
to evaluate if any learning has occurred? How will you do that? Who will the results serve?
Being an Evaluator can be its own profession, but to be able to evaluate your own lesson,
training, program will be a valuable tool for you, your learners and whatever organization
you are working with.

Routine evaluation: Routine evaluation is best done as a part of management process. This
approach keeps the faculty/schools in constant touch with the students and the materials,
resulting in an appropriate and timely feedback to the schools, course writers and academic
counsellors. It will eventually feed into the process of course maintenance which can be
carried out through supplementary materials without wasting resources. In addition to
instructional materials, assignments and the term-end examination question papers also are
evaluated.

43
 There are many data sources available when performing an evaluation. The most commonly
used four are (a) observation, (b) survey, and (c) interview. One or all can be used when
evaluating. Below is a small glance at some advantages, disadvantages, real-world
examples and factors influencing the choice to use it.

Factors
Data Real-World
Advantages Disadvantages influencing
Source Example
this choice

The observer must

First hand When a process
be qualified and
documentation is or a procedure
understand the
valuable data is being
product and/or
which, when evaluated, such
process
added to the as how a
completely in When a
participant’s data, participant
order to be able to process
could prove to be interacts with a
accurately needs to be
quality data computer
observe. The evaluated.
sources. program or
observer’s
Observations can application or
presence could To evaluate
allow evaluation an employee
Observation negatively affect in real-time
data to be using a new
the data versus after
collected during tool in the
collections. Also the fact.
real events environment of
if there are not
occurring in real- the workplace
enough The need to
time. They can be or the flow of a
observations, the observe in
videotaped or procedure or
data may not be context.
photographed practice, then
helpful and if
also. This can be observation
there are too
referred to more might be a
many, this could
than once if good data
take an evaluation
necessary. source.
over time/budget.

Surveys allow an Often the answers If an evaluation The number

evaluator the received do not needs to be of
opportunity to provide the done quickly, participants
receive data from information for a company
a large number of desired. The skill that has offices Their
participants and of the evaluator or all over the locations
possibly over a the one who country a
Survey
large designs the survey could be Time-table
geographical questions is a best choice restrictions
area. necessary in order option. A
to obtain relevant computer Privacy or
Also, if the data. Also, people program flexibility
participants might may be burned out designed to for the
be influenced by on surveys and organize, participant

44
 the presence of then participation calculate,
others, this allows numbers will lag. evaluate the
them a certain data could be
privacy of time used to help in
and place. the process and
meet deadlines
to gain as much
relevant data as
possible.

There may be a
bias that affects
the data based on
the questions
asked, how they
This allows for a
were asked.
deeper response.
Sometimes the A real-world
Allows the
participant might example might
participant to
be responding to be when a rich
expand, if
the interviewer description is
necessary on their
and not the needed such as
responses which
question so the customer
is sometimes
data is not as satisfaction or a Deeper
preferred to a
accurate as user-experience responses
survey. This also
possible. This is that would needed.
Interview allows the
another data require details
interviewer to
source that and Clarification
narrow in on the
requires a highly explanations in needed.
major topics of
skilled interviewer order to fully
interest.
who can stay on understand
Interviewers can
track, cover all the their reaction to
meet with more
bases, redirect the the program or
than one person
participant and process being
as in a focus
know when to evaluated.
group. This can
probe deeper. It is
save time.
also time
consuming which
might push the
time-table and
budget.

45