Assessing the effects

of ICT in education

Indicators, criteria and benchmarks

for international comparisons
edited by Friedrich Scheuermann and Francesc Pedró

ORGANISATION
FOR ECONOMIC
CO-OPERATION
AND DEVELOPMENT
Assessing the effects of ICT in education
Indicators, criteria and benchmarks for international comparisons

Edited by Friedrich Scheuermann and Francesc Pedró

 Europe Direct is a service to help you find answers
to your questions about the European Union

Freephone number (*) :

00 800 6 7 8 9 10 11

(*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls
may be billed.

European Commission
Joint Research Centre

Legal notice
The opinions expressed and arguments employed herein do not necessarily reflect the
official views of the OECD or of the governments of its member countries or those of JRC
or the European Commission .
Neither the European Commission nor any person acting on behalf of the Commission is
responsible for the use which might be made of this publication .
A great deal of additional information on the European Union is available on the Internet .
It can be accessed through the Europa server (http://europa.eu/) .

This document has been produced with the financial assistance of JRC .

Cataloguing data can be found at the end of this publication .

Luxembourg : Publications Office of the European Union , 2009

ISBN 978-92-79-13112-7
doi : 10.2788/27419 .

European Union/OECD, 2009

Reproduction is authorised provided the source is acknowledged .

Printed in France

PRINTED ON WHITE CHLORINE - FREE PAPER

2
Contents

Introduction 5
Friedrich Scheuermann and Francesc Pedró

Chapter I — Context and general reflections

In search of the sustainable knowledge base:
multi-channel and multi-method? 13
Øystein Johannessen

Addressing the complexity of impact — A multilevel approach

towards ICT in education 21
Ola Erstad

Chapter II — State of the Art

Monitoring in education: an overview 41
Willem Pelgrum

What do we know about the effective uses of information and

communication technologies in education in developing countries? 61
Michael Trucano

Chapter III — Conceptual frameworks

A framework for understanding and evaluating the impact
of information and communication technologies in education 69
Katerina Kikis, Friedrich Scheuermann and Ernesto Villalba

ICT to improve quality in education —

A conceptual framework and indicators in the use of information
communication technology for education (ICT4E) 83
Marcelo Cabrol and Eugenio Severin

A conceptual framework for benchmarking the use and assessing

the impact of digital learning resources in school education 107
Beñat Bilbao-Osorio and Francesc Pedró

3
 Chapter IV — Case studies
Assessing new technological literacies 121
Edys S. Quellmalz

The impact of ICT in education policies on teacher practices

and student outcomes in Hong Kong 143
Nancy Law, Yeung Lee and H.K. Yuen

Indicators on ICT in primary and secondary education:

results of an EU study 165
Willem Pelgrum
Impacts of ICT use on school learning outcome 189
Heeok Heo and Myunghee Kang
ICT impact data at primary school level: the STEPS approach 199
Roger Blamire

4
Introduction

Despite the fact that education systems have been heavily investing in technol-
ogy since the early 1980s, international indicators on technology uptake and use
in education are missing. For more than 25 years education systems have been
able to design and implement policies in this domain without those indicators, so
the question is: why start discussing them now? Is the information available not
good enough?

Why now?
The existing international indicators still mirror the first policy priorities of the
early 1980s: securing student access to computers and the Internet in schools.
Indicators such as ratios of students per computer or percentage of schools with
broadband access, although still a concern in some countries, do not yet provide
the most relevant information for today’s policy in the field: how is technology
used in schools? Is this use truly supporting the emergence of the learning envi-
ronment that a knowledge-based society requires?

Certainly, knowledge economies and societies would greatly benefit from a

broader set of internationally comparable indicators. These could monitor
progress in ICT uptake and unveil important information about use, ranging from
issues such as frequency to purpose. If carried out in an international compara-
ble framework they will become an important tool for benchmarking policies and
practices across countries and over time.

Our increasingly technology-rich world raises new concerns for education while
also expecting schools to become the vanguard of knowledge societies. Firstly,
technology can provide the necessary tools for improving the teaching and
learning process, opening new opportunities and avenues. In particular, it could
enhance the customisation of the educational process, adapting it to the particu-
lar needs of the student. Secondly, education has the role of preparing students
for adult life, and therefore it must provide students with those skills necessary
to join a society where technology-related competencies are becoming increas-
ingly indispensable. The development of these competencies, which are part of
the set of the so-called ‘21st century competencies’, is increasingly becoming
an integral part of the goals of compulsory education. Finally, in a knowledge
economy driven by technology, people who do not master these competencies
may suffer from a new form of digital divide that may affect their capacity to fully
integrate the knowledge economy and society.

5
 Introduction

Because of these reasons , most countries have undertaken significant invest-

ments to enhance the role of technology in education recently, after some years
of less activity immediately after the implosion of the Internet bubble . Many
would say that the incorporation of technology in education has lost its status
as policy priority number one, although for a number of political reasons invest-
ments have not been stopped . In many respects , the principle of ‘ build it and
they will come’ seems to have taken root, and education systems keep investing
in technology based on the belief that, sooner or later, schools and teachers will
adopt it and benefit from it. The question that arises then is whether or not these
new investments are paying off; is this investment in technology within educa-
tion systems managing to fulfil expectations?

New policy concerns, increased need for evidence

and indicators

Ironically, what countries have been investing in this field has hardly been the
subject of any comparison . Therefore , countries can hardly claim that they are
investing significantly in this . But even more important than the amount of effort
invested , what really presses for an evidence -based policy debate about tech-
nology in education is the emergence of new policy concerns. At least some of
them , and the corresponding policy discussions , could benefit from more solid
and comparable evidence: the emergence of a second digital divide, the need to
promote the broad set of 21st century skills , and the still unfulfilled experience of
promoting radical change in the provision of school education .

First, recent evidence has unveiled that the digital divide in education goes
beyond the issue of access to technology. A new second form of digital divide
has been identified : the one existing between those who have the right compe-
tences and skills to benefit from computer use , and those who do not . These
competences and skills are closely linked to the economic, cultural and social
capital of the student. This has important implications for policy and practice.
Governments should make an effort to clearly convey the message that compu-
ter use matters in the education of young people and they should do their best
to engage teachers and schools in raising the frequency of computer use to a
relevant level . Such an increase could not only be a clear indication of teachers’
and schools ’ engagement with the development of 21st century skills and com-
petencies , but it could also report gains in educational performance . In addition ,
schools should be reminded that they have a crucial role in the development of
the cultural capital that will allow students to bridge the emerging second digital
divide .

Second , the changing needs of economic and social development require a wide
range of new skills and competencies , known as the 21st century competencies .
These are considered key enablers of responsible citizenship in a knowledge-
based and technology-pervaded economy. For instance , the recommendation
of the European Parliament and the Council on key competences for lifelong
learning defines a framework of eight competences considered important for
the knowledge society. Digital competence is highlighted as one of the eight key

6
 Introduction

competences. In 2007 the Council identified a framework of 16 core indicators

for monitoring progress in the field of education. ICT skills are a core indicator
in this framework. Technology is hence expected to play an increasing role in
education in the coming years.

Last but not least, there is the pending issue of whether or not today’s teach-
ing and learning experience in schools matches what could be expected from
a knowledge society. The question is not which technology leads to increased
productivity in education, but which new technology-supported methodologies
improve student performance over traditional ones, if any at all, and which other
factors intervene. Previous calls have already been made in order to investi-
gate the explicit relationships among technology, instructional strategy, psycho-
logical processes and contextual factors. The almost infinite array of methodo-
logical possibilities makes this kind of investigation extremely difficult, but not
impossible, provided that there is sufficient effort devoted to the accumulation
and dissemination of the resulting knowledge base. Such a task might appear
overwhelming, particularly as the technological frontier is constantly changing.
However, it is worth the effort. And policymakers and researchers cannot be in a
position to monitor what is truly going on in schools unless critical indicators about
intensity, purpose and context of use of technology in education are available.

A truly international effort

Therefore it is relevant to assess and compare how education systems are deal-
ing with technology integration in schools — particularly in terms of securing
and improving access, enhancing a wide range of educational and managerial
uses, and monitoring the effects and impacts on the development of critical
technology-related skills and competencies. Such a comparison is not possible
in the absence of appropriate indicators which, at the moment, are missing in
the international collections already available.

Both the European Commission and OECD have recognised the need for reli-
able indicators in the area of technology in education. OECD has raised this
issue in the context of the recently published report Beyond Textbooks. Digital
Learning Resources in the Nordic Countries. It highlights the need for a compre-
hensive approach to indicators on technology in education and the difficulties
associated with their development and data collection. The same need has also
emerged during the analysis of the relationship between technology use and
educational performance drawing on PISA 2006 data, which will be published by
CERI in 2009. The European Commission has initiated several studies intended
to summarise existing and available information in the field.

Other international organisations, such as Unesco, the World Bank and the Inter-
American Development Bank, share similar needs and are willing to cooperate
in this process. An inter-agency seminar carried out in Korea in July 2009 (1),
provided an excellent opportunity to compare priorities and agree on the need
to explore further synergies.

(1) see http://go.worldbank.org/DJTDITWI40

7
 Introduction

What this volume adds to the discussion

It is within this context that the present volume has to be understood . The con-
tributions included stem from an international expert meeting which took place
in April 2009 when the Centre for Research on Lifelong Learning (CRELL), in
cooperation with CERI , organised and hosted an international expert meeting
on the issue of benchmarking technology use and effects in education . The
workshop specifically aimed at constructing a framework to look at the relevant
domains and interdependence between components related to ICT in educa-
tional processes from a holistic perspective (2) .

This book is organised into four different sections . The first one looks into the
context of ICT impact assessment in education . This chapter addresses the
political context and includes reflections about the assessment needs at an
international level . Øystein Johannessen follows a policy perspective . He dis-
cusses the challenge of developing benchmarks and the need to incorporate
a multi -faceted approach which takes into account the complexity of issues to
consider when setting up a knowledge- base on ICT in education . In his article ,
Ola Erstad maintains the need for a broadened understanding by policymakers
of impact and outcomes . Based on experience gained in Norway, he suggests
a multilevel approach and tries to identify key indicators of impact for all the dif-
ferent levels addressed .

The second chapter is about the state of the art of ICT impact assessment. A
conceptual overview on educational monitors is provided by Willem Pelgrum ,
who introduces the various dimensions and challenges of ICT assessment and
methodologies issues in international comparative monitoring . Michael Trucano
then presents conclusions from the World Bank series of ‘ knowledge maps’ about
ICT in education . Despite a variety of useful resources, he identifies important
gaps and a lack of reliable impact evidence in order to better support the effec-
tive integration of ICT in developing countries .

Conceptual frameworks are discussed in Chapter 3 in order to agree on a gen-

eral common understanding about aspects to take into account when assessing
the effects and impact of ICT in education especially for comparative purposes
at country level . A conceptual framework should provide an orientation for any
kind of measurement required in the decision-making process and act as a ref-
erence which is flexible and adaptable to specific purposes of studies to be car-
ried out. It should also provide a holistic view and support the setting of standard
orientations when defining the evaluation methodology and selecting appropri-
ate instruments for measurement. The framework developed by Katherina Kikis ,
Friedrich Scheuermann and Ernesto Villalba for the Joint Research Centre of
the European Commission aims to contribute to a systematic approach on how
to identify the use of ICT and its effects on all different levels and stages . A simi-
lar approach is then presented from Marcelo Cabrol and Eugenio Severin , which
is currently being implemented in projects of the Inter-American Development
Bank in Latin America and the Caribbean . Finally, Beñat Bilbao and Francesc

(2) Contributions are published at the CRELL website (http://crell.jrc.it/workshopimpact.htm) .

8
 Introduction

Pedró discuss the conceptual approach proposed by the OECD for looking into
the impact of digital learning resources and benchmarking the use in school
education .

A series of reflective case studies are presented in Chapter 4. One important

aspect of ICT impact assessment is to be clear on what is to be assessed at
the individual level and to think about appropriate ways of measurements .
Technology use and critical thinking and problem-solving approaches (‘ new
literacies ’ ) are discussed by Edys S. Quellmalz in the context of assessment
design and implementation . She looks at current approaches in assessment
and underlines the need to reach consensus about what is to be measured . ICT
implementation policies in education in Hong Kong are then analysed by Nancy
Law, Yeung Lee and H. K. Yuen in terms of their impact on teaching and learn-
ing processes. They also present an interesting research design and concepts
of information literacy assessment. Willem Pelgrum then reports about moni-
toring scenarios and sets of indicators on the use and impact of ICT in primary
and secondary education . His work is based on the results of a study carried
out in the European Union which can be seen as a further step to implement
mechanisms for regular ICT implementation monitoring at a European level . A
theoretical framework of various factors affecting ICT use in education is pre-
sented by Heeok Heo and Myunghee Kang . This framework had been embed-
ded in a nationwide investigation in Korea . Findings clearly indicate that a better
understanding of the real impact can only be achieved if more consideration is
given to the use of ICT in informal learning . In addition , results from a compara-
tive analysis in the European countries on ICT in primary education are then
described by Roger Blamire . The approach was based on an analytical frame-
work allowing an examination of the impact on three different levels : on learning
and learners , on teachers and teaching and on primary school development
plans and strategies . Altogether these cases help to better understand the need
for comprehensive studies of the complex interactions between various types of
ICT implementation and its effects , including other factors to take into account
which have not yet been addressed by existing studies .

The aim of this book is to provide a basis for the design of frameworks , the
identification of indicators and existing data sources as well as gaps in areas
where further research is to be initiated . The contributions clearly demonstrate
that there is a need for the development of consensus around widely accepted
approaches , indicators and methodologies. In this context more harmonisation
of existing survey approaches would be desirable. Therefore , this collection of
articles follow the intention of both organisations , the OECD and the European
Commission , to foster international cooperation with other relevant international
organisations and to serve as a starting point for common reflection on ways to
assess how ICT is used in education .

Without such an assessment, it is virtually impossible to make any progress

in the direction of understanding better how the actual pedagogies are trans-
formed and which policies , both at national and local levels , are making a dif-
ference . Only a truly international comparative effort can provide the necessary
evidence . And even if the contributions in this book show a vast diversity of

9
 Introduction

perspectives , at least they point in the right direction . Even more important than
getting the hard evidence is to make significant progress in understanding the
worth of technology in education and in how to measure progress . This book has
to be seen as a serious attempt to touch base and , as such , has to be taken as
the beginning of a journey. The sooner we start walking the better.

Friedrich Scheuermann Francesc Pedró

10
CHAPTER
CONTEXT AND GENERAL REFLECTIONS

In search of the sustainable knowledge base:

multi - channel and multi -method ?

Addressing the complexity of impact —

A multilevel approach towards ICT in education

11
In search of the sustainable knowledge
base: multi- channel and multi - method ?
Øystein Johannessen
Ministry of Education and Research , Norway

Abstract

This article discusses the need for developing an open, flexible and international
knowledge base for ICT in education, in which joint development of benchmarks
can play a key role for addressing complexity, multi-stakeholder interests and
international comparisons. The needfora multi-channelandmulti-methodapproach
is elaborated. The article is written from the point of view of a policymaker.

Introduction developing a sustainable knowledge

base should be discussed . The second
ICT (information and communication part focuses on the issue of what we
technologies ) in education lives a life have learnt from R & D with regard
at the crossroads between evidence- to the effects of ICT in education . In
based policymaking , learning and the the third part, I describe the concept
fast-changing world of technology. of the multi-channel and multi-method
Key stakeholders (politicians , parents , knowledge base , before I finish with
teachers , school leaders) demand some remarks on the issue of a sys-
evidence of the impact of ICT derived temic approach to benchmarks and
from research , monitoring and evalu- other critical components of a knowl-
ation . The challenge for policymakers edge base for ICT in education .
is ( in collaboration with the research
community and the educational com- This article is written from the point of
munity) to develop a sustainable view of a policymaker.
knowledge base for ICT in educa-
tion , in which key indicators and other
What is the rationale
sources of information are identified ,
which enables better insight into the behind the focus on ICT in
use and effects of ICT for learning . I education?
have chosen to discuss the issue of
developing benchmarks for ICT in ICT in education has , in recent years ,
education , because benchmarks are emerged as a policy area . Many coun-
embedded in the evolving knowledge tries have developed ICT strategies ,
base in this field . either as separate strategies or as
strands embedded in national strate-
This article is structured in four parts . gies for education or for the devel-
In the first part, I describe the policy opment of the information society at
backdrop , within which the issue of large in the country. The strategies

13
 Chapter I - Context and general reflections

and their underlying rationales share principles . Our educational systems

many common features . should bear in mind that ICT should be
an integral part of learning , in order to
Kozma ( 2008) has identified impor- provide learners from families with a
tant reasons for investing in ICT for low socio- economic status with neces-
education . sary digital skills for learning , work and
life in order to avoid digital divides .
• To support economic growth mainly
by developing human capital and ICT is not integrated in education for
increasing the productivity of the its own sake. A proper integration of
workforce . ICT in key policy priorities in differ-
• To promote social development by ent countries can be a productive
sharing knowledge, fostering cultural approach in order to secure ICT as
creativity, increasing democratic a mainstream part of education. In
participation , improving access to Norway, ICT is not subject to a sepa-
government services and enhancing rate strategy; it is rather embedded
social cohesion . in the national curriculum and linked
• To advance education reform , i.e. to overall political priorities stated by
major curriculum revisions , shifts in the government: quality of learning ,
pedagogy or assessment changes . higher completion rates and students’
• To support educational management well-being and mastery.
and accountability, with an empha-
sis on computer- based testing and
the use of digital data and manage- What have we learnt from
ment systems . R& D?

These features relate the issue of ICT We have been through a period in
in education to its function in a broader, which politicians and policymakers
societal context. The role of ICT in have focused on the need for estab-
education must also be linked to edu- lishing credible proof for the return on
cational needs . In many countries , the investments in ICT. This has resulted
role of ICT is linked to issues of educa- in a search for causal relationships
tional attainment and the importance between ICT and educational quality,
of ICT for advancing robust learning i.e. learning outcomes . As the OECD
strategies on the side of the students . (2008) has pointed out, this has been
A second area is ICT as a tool for the difficult to achieve because of the
support of personalisation strategies in lack of large-scale , longitudinal stud-
teaching and learning . ICT can also be ies and a lack of methodologies that
used to increase visualisation and var- can capture the complexity of ICT and
iation in many subjects . As a greater other elements influencing educational
proportion of our homes are linked to quality.
the Internet , the role of ICT in home/
school access is now being exploited . One of the most significant studies to
Many children start to use ICT at an date is the ImpaCT2 report from 2002
early age, and the home and the family ( Harrison et al . , 2002 ) . The study
are, in many cases , an arena for the shows that ICT leads to statistically
initial acquisition of digital skills . Thus , significant improvements in some sub-
education has a role to play in further- jects , whereas there are no significant
ing these skills , based on pedagogical improvements in other studies. The

14
 In search of the sustainable knowledge base

OECD, through its work on the PISA Multi-channel:

studies, has been able to demonstrate
interesting correlations between home ICT in education covers
access and use of ICT on the one hand a wide spectrum
and PISA score on the other hand. The
relation between ICT use at school The first pillar of my approach to a sus-
and PISA score is fare more complex. tainable knowledge base is the reali-
So far, these correlations have not sation that ICT in education covers a
been explained. The study ‘E-learning wide spectrum — both thematically
Nordic’ (Ramboll Management, 2006), and along the administration–peda-
which looks at the perceived impact of gogy axis. This is a consequence of
ICT, shows that all stakeholders (stu- the incremental integration of ICT into
all domains of education.
dents, parents, teachers, principals)
believe that ICT can have a positive
Kozma (2008) has highlighted this in
impact on teaching and learning.
his work, and he acknowledges that
The studies and reports mentioned ICT strategies in many countries cut
across diverse fields.
above represent a plethora of studies.
The European Schoolnet shows in its
• Infrastructure development is nec-
metastudy on impact studies (EUN, essary in order to ensure access to
2006) that there are a number of stud- schools, networks and resources for
ies, also related to patterns of use learning.
across the technological spectrum. • Teacher training, both initial and in-
Impact studies cover a wide spectrum service, is a prerequisite for the abil-
between the search for causal relation- ity of education to use ICT in learn-
ships between ICT and educational ing processes.
attainment on the one hand and stud- • Technical assistance is needed both
ies looking at the perceived impact of in the administrative as well as in
ICT on the other hand. the pedagogical domain.
• Curricula and pedagogical
The focus of some studies has been approaches may have to be changed
on causality and on quantitative issues in order to cater for educational
regarding ICT use. It is time to review change with ICT.
critically whether we have been asking • Content development is necessary
the right research questions. In its first in order to facilitate the interactive
report on ICT and PISA score (OECD, potential ICT can offer in the teach-
2004), the OECD states: ing and learning process.

‘It is the quality of ICT usage, rather than In my opinion, a multi-channel

necessarily the quantity, that will deter- approach to the knowledge base is
mine the contribution that these technolo- necessary in order to be able to ask
gies make to students’ outcome.’ the right questions and to grasp the
plethora of issues related to ICT in
Instead of looking for causality, we education. Let me elaborate on a few
need to ask how we can improve and issues.
optimise the use of ICT in teaching and
learning, and in doing so we also need • It is necessary to continue the moni-
to listen to the voices of the learners toring of infrastructure development.
and the practitioners. Although many countries have

15
 Chapter I — Context and general refiections

developed a superb infrastructure, ences within a gender as between

access to ICT is still an issue in many genders.
European countries. This is truly the • Digital learning resources (DLR)
case if you look at access issues on are characterised by complexity —
a global scale. The same goes for a crossroads between pedagogy,
the need for monitoring the evolving technology, IPR and the market-
patterns of use. We need to be able place. This is an area which, in my
to assess the speed of uptake of dif- opinion, has been under-assessed,
ferent technologies for learning as and we need a stronger focus both
well as assessing the degree of vari- on benchmarking of digital learn-
ation across the spectrum of learn- ing resources as well as a research
ing technologies. A particular chal- agenda for DLR and learning.
lenge with regard to monitoring the • For PISA (2003) and PISA (2006),
patterns of use is the high degree of follow-up analysis based on ICT data
technological and cultural diversity has been undertaken. In future, the
that is to be expected in many coun- ICT analysis of PISA should be rep-
tries around the globe. licated and improved, and the ICT
• Gender issues are visible. PISA familiarity questionnaire should be
data show that although the gap updated in order to keep up with the
between genders is closing, there evolving use of ICTs for learning.
are still interesting differences to • Few countries have developed good
be found with regard to patterns of methodologies for assessing digital
use. A fairly new dimension regard- skills among students. Such meth-
ing gender issues is that it might odologies should be developed both
be just as important to study differ- within and across subjects.

Figure 1: Pupil use of digital content, computer games, mobile phones and office programs — seventh grade,
ninth grade and VK1, where ‘daily’ and ‘weekly’ have been merged (in percentages).

16
 In search of the sustainable knowledge base

Some countries are monitoring both development is to be able to capture

access and use of ICT. The Norwegian the complexity of the learning proc-
ITU Monitor (Arnseth et al . , 2007) is ess . In my view, we need to further
a biannual monitor that assesses the explore the potential of ethnographic
status with regard to ICT in Norwegian research and so- called test-bed stud-
schools . The following figure shows ies . However, a downside to these
an example of patterns of use among approaches is that they are consum-
Norwegian students . ing both in terms of time and money.

The list of topics shows that there are In the last couple of years , we have
many phenomena in ICT and learn- seen projects in several countries
ing that should be monitored and aiming at capturing the voices of the
assessed through a variety of chan- learners . One example of this is the
nels , but is this enough ? In the next digital generation project, funded
chapter I will elaborate on the need for by the MacArthur Foundation pro-
a multi -method approach in order to gramme for digital media and learn-
ensure a sustainable and systemically ing . The project conveys how children
coherent knowledge base . develop engagement, self- directed
learning , creativity and empowerment
through the use of digital media. Our
Multi -method approach educational systems need to develop
to the knowledge base our ability to listen to and reflect on
the voices of the learners in order to
A consequence of the increased focus understand how digital media influ-
on evidence-based policymaking is ence the lives and learning of our chil-
that national authorities need to move dren . This topic will be addressed in
away from anecdotal and unsystem- the second half of the OECD new mil-
atic evidence of how ICT is being lennium learners’ project.
used in education and how it impacts
teaching and learning . Such a change Digital media play a much bigger
of focus highlights requirements of role in the lives of our students today
methodology and validity. The multi- than before . A Norwegian report from
tude of issues at hand , which I have 2008 (Arnseth et al . , 2008) shows that
described in the preceding chapter, more than nine out of 10 adolescents
and the need for diverse approaches aged 16 to 19 use social media , and
indicates that building a sustainable three out of four use social media on
and flexible knowledge base requires a daily basis . This raises the question
a combination of quantitative and of whether only ICT use in schools
qualitative methods . Furthermore , a should form the basis of our under-
system of indicators and other input to standing of digital media and learning .
the knowledge base must be flexible We may have to broaden the scope
enough to allow for changing patterns and include out-of-school use of dig-
of use and the emergence of new ital media , given the extensive home
technologies for learning . use of digital media . This would also
acknowledge the fact that the home
An important question is whether the of youngsters is the first arena for the
methods are good enough , and if there acquisition of digital skills , albeit an
is room for improvement. A well- known informal , but nevertheless important
challenge in educational research and arena .

17
 Chapter I - Context and general reflections

Another important consideration

A system of benchmarks
regarding the benchmarking of ICT
As I wrote in the introduction to this in education is related to the search
paper, benchmarking is an integral for precision and validity. Given the
part of the knowledge base national complexity of education , underlying
authorities , and the research and edu- research- based concepts and models
cational community must develop . will inevitably reach a high level of
Developing a system of benchmarks is sophistication . Herein lies a danger.
The models can be too ambitious in
an exercise that requires careful plan-
ning and solid reflections on the selec- their strive for perfection , and it is
tion and usability of benchmarks. important to realise that the concepts
and models behind benchmarks must

As a point of departure for discussion , find an equilibrium between simplicity

it is possible to distinguish between and complexity, because, by the end
different types of benchmarks for ICT of the day, they should meet the needs
in education . I have divided them of policymakers and practitioners .
into first , second and third order
benchmarks .
Systemic challenges
• First order benchmarks are typically related to development
related to access to ICT. This could ofbenchmarks
be pupil : PC ratio and broadband
access . The development of benchmarks does
not happen in a vacuum ; it serves
• Second order benchmarks try to
capture in what ways and to what purposes related to decision-making ,
informed choices and the need for a
extent ICT is used in teaching and
deeper understanding of ICT in edu-
learning . These benchmarks can
cation and its development. It is, how-
cover a wide range of use patterns
ever, difficult to know with great preci-
and learning technologies , and they
sion what we are looking for, because
should capture both teachers ’ and
ICT is embedded in pedagogical
students’ use of ICT for learning .
practice . This is especially pertinent
• Third order benchmarks should
for the development of third order
cover the impact of ICT in teaching benchmarks .
and learning . Benchmarks should
be related to learning outcomes and Another systemic challenge is related
learning strategies . to the trend in recent years that educa-
tion has evolved into an arena for solv-
Development of benchmarks should ing many problems society as a whole
pay attention to the need for research and younger cohorts in particular are
and development in order to meet facing . The educational community
demands for validity and methodo- may at times feel that it is under siege .
logical rigour. Many countries have Thus , development and utilisation of
elaborated benchmarks of the first and
benchmarks that represent an admin-
second order, but it has proved diffi- istrative burden should be carried out
cult to develop solid third order bench- with great caution .
marks . Further research efforts should
therefore be directed at the develop- A particular advantage related to
ment of such benchmarks . benchmarks is that they are well

18
 In search of the sustainable knowledge base

suited for international comparisons . of political validity. By political valid-

However, so far little work has been ity I mean (in the context of discuss-
done to develop an agreed interna- ing benchmark development) that the
tional framework for benchmarking choice of benchmarks should not only
ICT in education . It should be in every- be directed by methodological per-
one’s interest to develop an interna- spectives , it should also pay attention
tional benchmarking framework . This to the needs of key stakeholders in
could be done in a joint OECD–EU education when it comes to the choices
collaboration . One important consid- of benchmarks . As such , developing
eration is to agree on common topics benchmarks should take place at the
for benchmarking, and it is in my opin- crossroads between policy, practice
ion vital to make sure that a sufficient and research . Methodological validity
spectrum of issues is addressed . ensures that we can trust the informa-
Digital learning resources are a good tion we get from benchmarks , political
case for benchmarking development, validity ensures that stakeholders in
because DLR has a high degree of politics and society get the information
complexity, they are important for the they need .
quality of learning and there is too little
evidence on the impact of DLRs .
The road ahead?
Developing a framework for bench-
marking is a challenge that cannot be Benchmarking can play a role in
solved by one party alone . It is vital to developing an open knowledge base
ensure that such a framework should for ICT in education . International
be developed in a triangular collabora- collaboration is necessary for such a
tion between researchers , policymak- venture because of complex issues ,
ers and practitioners . The notion of a wide spectrum of stakeholders and
‘methodological validity’ is important the need for agreed frameworks for
in research and benchmarking . When international comparisons . By the end
it comes to benchmarking of ICT and of the day, the knowledge base should
the issue of power of definition of what be there to guide us in informed
we are looking for to benchmark, it is choices for the benefit of current and
in my opinion interesting to combine future cohorts of learners . Because
methodological validity with the notion they deserve it!

References

Arnseth , H. C. , Hatlevik , O. , Kløvstad , V. , Kristiansen , T. and Ottestad , G. ( 2007) .

ITU Monitor 2007. Oslo : ITU .

Arnseth , H. C. , Bucher, T. , Enli , G. , Hontvedt , M. , Kløvstad , V. , Maasø , A. and

Storsul , T. (2008). Nye nettfenomener: Staten og delekulturen. Oslo : ITU and
University of Oslo .

European Schoolnet (EUN) (2006). The ICT impact report: a review of studies of
ICT impact on schools in Europe. Brussels : European Schoolnet .

19
 Chapter I — Context and general refiections

Harrison, C. et al. (2002). ImpaCT2: the impact of information and communica-

tion technologies on pupil learning and attainment. Coventry: Becta.

Kozma R. B. (2008). ‘Comparative analysis of policies for ICT in education’, in:

J. Voogt and G. Knezek (eds), International handbook on information technol-
ogy in primary and secondary education. New York: Springer.

OECD (2004). Are students ready for a technology rich world? What PISA stud-
ies tell us. Paris: OECD.

OECD (2008). New millennium learn-ers: a project in progress. Paris: OECD.

Ramboll Management (2006). E-learning Nordic 2006: impact of ICT on educa-

tion. Copenhagen: Ramboll Management.

The George Lucas Educational Foundation, The digital generation project.

http://www.edutopia.org/digital-generation

The MacArthur Foundation, Digital media and learning. http://www.mac

found.org/dml/

20
Addressing the complexity of impact —
A multilevel approach towards ICT
in education
Ola Erstad
University of Oslo, Institute of Educational Research

Abstract
Within research on ICT and school development there is an increased understand-
ing of the complexity involved in such processes. However, the focus on indicators
and the impact of ICT in education from a policy perspective have been oriented
towards a more narrow understanding of impact and outcomes, especially on the
individual level. This article argues for the need for a multilevel approach towards
ICT in education in order to fully understand the impact of such technologies in
the education system. In the first part, some theoretical reflections on change and
the research on impact are presented. In the second part, some examples will be
described, mostly from a Norwegian setting, and in the last part, some key indica-
tors of impact on different levels will be discussed.

Introduction In recent years, there has been a ten-

dency to argue that complexity is an
The most important point I have issue in itself in studying knowledge
learned by studying the impact of ICT practices (Law and Mol, 2002) or stud-
(information and communication tech- ies on ICT, development and schools
nologies) on Norwegian education (Engestrøm, Engestrøm and Suntio,
during the last 10 years is the com- 2002; Thomson, 2007). In order to
plexity and multilevel aspects of such fully understand or assess the effects
innovations. The challenge is not so of ICT in education we need to know
more about how ICT operates on dif-
much to develop indicators for ICT in
ferent levels, and what we are really
education as such. At present there
measuring on which levels. It is crucial
are several available frameworks of
that we synthesise the research with
indicators, about implementation of
a holistic perspective in order to lay a
ICT in educational settings, about dig- foundation for further development in
ital literacy, about leadership and so this area (Sutherland, Robertson and
forth. The challenge is rather to study John, 2009). In this article, the argu-
different levels and domains at the ment is built around the need to look
same time, and to bring different sets at the bigger picture in order to create
of indicators together into one strategy sustainable developments throughout
in order to assess the broad scope of our education systems, and under-
impact of ICT on education. stand ICT as a catalyst for change

21
 Chapter I - Context and general reflections

on different levels . This creates and people work together and relate
challenges for the development of to each other, as a globalising proc-
indicators of the impact of ICT in edu- ess (Castells , 1996). Education is also
cation since several sets of indicators thought of in a more distributed way
need to be developed and different by using these technologies for edu-
methods must be used . The objective cational purposes , such as in compu-
would be to build a model that looks ter supported collaborative learning
at how different levels and dimensions (CSCL) .
work together to create conditions for
The challenge , and the complex-
change and the integration of ICT in
ity, rests on how these levels and
educational practice .
perspectives relate to each other.
This is a challenge of educational
Understanding change research in general , but especially
when trying to understand the mech-
A major challenge for developments anisms involved in the educational
within technology and education today use of ICT. In the research literature
is to grasp the complexity of such devel- there is now a greater consciousness
opments . In general , there has been towards multilevel analysis (Van Dijk,
a tendency to simplify the research 2009) and more holistic approaches
approaches and understanding of towards learning and school devel-
how digital technologies might have opment (Hakkarainen , Palonen ,
an impact on schools and educational Paavola and Lehtinen , 2004 ; Arnseth
outcomes (Cuban , 1986, 2001 ; Erstad , and Ludvigsen , 2006 ; Sutherland ,
2004) , and evidence of the impact of Robertson and John , 2009) . As David
ICT on educational practice has mainly Olson has pointed out in his book
been drawn from small -scale case Psychological theory and educational
studies (Condie and Munro , 2007). reform (2003):
Both policymakers and researchers
The problem, I believe, is that the theo-
have created expectations towards the
ries that gave us insight into children’s
impact of information and communica-
understanding , motivation , learning and
tion technologies on student learning,
thinking have never come to terms with
which has not gained strong support schooling as an institutional practice with
in the research literature (ibid . ) . Much its duties and responsibilities for basic
research has been oriented towards
skills , disciplinary knowledge , grades ,
the new possibilities and limitations standards , and credentials ... What is
created by the implementation of digital required , then , is an advance in our under-
technologies into educational settings standing of schools as bureaucratic insti-
(De Corte, Verschaffel , Entwistle and tutions that corresponds to the advances
van Merrienboer, 2003). Again , other in our understanding of the development
research and development initiatives of the mind . ( D. Olson , 2003 :x–xi )
have been more directed towards the
institutional framework of school devel- Olson argues that the challenge is to
opment and the use of ICT ( Krumsvik, combine different levels in our under-
2009) . In later years , there has also standing and analysis of key charac-
been a growing interest for networks , teristics of how schools function as
both online and offline (Veugelers and
learning organisations , and also the
O’Hair, 2005). The argument goes
that digital technologies have created conditions for changes of activity at
different levels .
a new situation for how organisations

22
 Multilevel approach to address complexity

In his classic book The new meaning norms , division of labour and commu-
of educational change (1991 ), Michael nities of practice . The relation between
Fullan presents a broad framework on these factors is defined as an activity
different levels and involving different system, and within an organisation and
actors in understanding educational between organisations there might be
reform and school development. Also several activity systems that relate to
in his later book Change forces (1993), each other in different ways .
he addresses the real complexity
of dynamic and continuous change , The complexities of knowledge crea-
showing the challenges this implies tion and knowledge building have
been an issue within research com-
both for peoples ’ mind-sets and for
mechanisms defining educational munities dealing with CSCL, studying
practices . This has made the research how collaborative and distributed ways
community understand that change of working using different technologi-
was not an event that occurred in such cal applications stimulate knowledge
a way that a ‘ before ’ and ‘ after’ could building among learners . This can be
be recognised and measured ; rather, seen in the developmental work done
he defined change as a process . by Marlene Scardamalia and Carl
Bereiter in Canada (Scardamalia and
In recent years , this has been taken up Bereiter, 2006). Knowledge building ,
by other researchers trying to develop and the technological platform that has
models to study and also to create been developed ( Knowledge Forum) ,
interventions into educational prac- aim for collective cognitive responsibil-
tices in order to work towards school ity among learners . Collective respon-
development. This represents a move- sibility refers to a condition in which
ment away from traditional models responsibility for the success of the
of change based on organisational group is distributed across all mem-
theory such as Senge or Nonaka and bers rather than being concentrated on
Takeuchi , towards models trying to the leader. Collective cognitive respon-
grasp the complexity of change proc- sibility refers to taking responsibility to
esses through the activities involved . know what needs to be known on the
The most important perspective cognitive level in addition to the more
for studying change processes in tangible practical aspects .
schools in recent years has been
activity theory, or more specifically Networking is a broad conceptualisa-
cultural -historic activity theory (CHAT) tion based on global perspectives on
(Engeström , 1987 ) . This has grown social development, but which also
out of the intellectual work done by relates specifically to the role of edu-
the Russian psychologist Vygotsky, in cation in moving towards knowledge
the 1920s and 1930s , and later on by societies and the role of networking
Leontjev. The focus of this perspec- in such processes . As an example , in
tive is on activity as the unit of analy- the Unesco report Towards knowledge
sis and mediation between actors and societies (2005), the concept of learn-
certain cultural tools . Yrjö Engeström ing is closely tied to innovation and
has then expanded this model beyond networking . Credé and Mansell ( 1998 )
the person and the tools by introduc- have also shown how this thinking on
ing a larger framework of factors that knowledge societies and networking
are part of developmental processes is fundamentally based on identifying
on different levels , such as rules and new ICT opportunities .

23
 Chapter I - Context and general reflections

In his literature review on ‘whole school An upscaling of activities has brought

change’ , Thomson emphasises that: about a need for development of indi-
‘ The ways in which we think about the cators that capture the more systemic
school also impact on what counts developments of ICT in education ,
as change . There are two important and how that transcends to the micro
aspects to thinking about change in level of teaching and learning by teach-
schools: ( 1 ) understanding the school ers and students : not how we change
as an organisation , and (2) under- single schools in the way they work
standing that change will be multi- with ICT, but rather how all schools and
layered’ (2007: 15) . In his presentation the school system as such experience
of a framework for change, he focuses changes by implementing and using
on two important themes : ‘the timing ICT.
of, and time for, whole school change ’ ,
and ‘ a supportive framework’ . One example is the national cur-
riculum in Norway, from 2006 , which
The impact of ICT has become a key defines ‘the ability to use digital tools’
factor in many studies in understand- and digital competence as a basic
ing how new technologies both might skill throughout the curriculum . In
be a catalyst and a driving force for this way, the Ministry of Education
change processes in themselves , and and Research has placed a strong
also an element that supports change emphasis on ICT as part of learning
within organisational settings . All this activities in schools . ICT should be an
points towards a stronger emphasis integrated part of learning activities
on multilevel approaches studying among all students , at all levels of pri-
change and the impact of ICT on dif- mary and secondary education and in
ferent levels within the same analysis . all subjects . This also challenges how
schools are organised .
Systemic impact, The focus on ICT and digital compe-
curriculum developments tence in the new national curriculum

and future competencies builds on former plans and docu-

ments . At the same time, it points
Such broader conceptions of change towards future competencies , what
are important in developing an under- are also termed as 21st century skills
standing of key factors influencing (www.21stcenturyskills.org). Theimpor-
educational practices (see examples tant implication for the discussions
below). The last decade has been in this article is the commitment this
associated with an upscaling of activi- implies for teachers and students to
ties using ICT in educational settings . use ICT much more broadly in the
From small groups of students and learning activities in schools . In this
teachers , we have seen a rise in the way, a stronger push mechanism is
way ICT has been implemented across created for school leaders and teach-
the curriculum . The consequences in ers to work towards capacity building
many countries have been that whole on school development and the use of
school communities use such tech- ICT in order to fulfil the challenges of
nologies in different activities, and that the new curriculum . Important national
these developments have an impact objectives related to the new national
on a national level through curriculum curriculum can be summarised as
developments . follows:

24
 Multilevel approach to address complexity

•
a focus on how ICT can contribute to
Two examples
an increased quality in teaching and
learning ; Example 1 - PILOT (project
•
an increased use of new ICT-based
innovation in learning,
means for cooperation and inter-
organisation and technology)
change of knowledge and experi-
ence at all levels of the educational PILOT was the largest and most exten-
system; sive project in Norway related to the
•

a broad access to learning materi- pedagogical use of ICT in schools during

the years 2000–04 . The project was
als and the development of new and
initiated by the Ministry of Education
varied forms of learning in order to
and Research , and a national agency
stimulate activity, independence and
(ITU) was responsible for coordinating
cooperation; the research work and research com-
an increased focus on students' crit-
•

munities involved in the project.

ical reflection with respect to the use
of ICT in teaching and learning and This project was part of upscaling of
in society in general ; activities on a national level using new
•
an increased focus on how to avoid digital technologies , from a few inno-
creating digital divides . vative teachers and schools towards
whole school communities and includ-
Such curriculum developments also ing many schools . Some 120 primary
point to the need for multilevel analy- and secondary schools in nine regions
sis of the ways we study the impact of Norway took part in this four-year
of ICT on education . And it brings up
research and development project
a future observation of and an orien- based on interventions concerning the
educational use of ICT and developing
tation towards the competencies stu- a framework within whole school set-
dents need today and in the future and tings . The aim of the project was: ‘to get
that our school system needs to take the participating schools to develop the
into consideration . Competencies are pedagogical and organisational oppor-
here understood on different levels , tunities afforded by the use of ICT, and
not only seeing competency as an to develop and spread new knowledge
individual ability, but also on the col- on this subject’. The research design
lective level and the school level . In was structured with a quantitative
a Norwegian context, we have had part (pre-post) and a qualitative part
different projects and strategies in (during) .
developing indicators on these differ-
In the initial phase , infrastructure
ent levels , also trying to define what
and technological challenges were in
is called ‘ the digital competent school ’
focus . However, in the second part of
or ‘ digital maturity’ . the project, the focus was much more
on various pedagogical approaches
Before I move on to some reflections
to education . This was due in part to
on a multilevel approach to indicator the fact that the use of technology had
development on ICT in education , I become more common in everyday life
want to give two examples from my at many of the schools , and in part to
own research where such a multilevel the fact that technology could not be
approach has become apparent. used as a helpful aid until the proper

25
 Chapter I — Context and general refiections

conditions had been established. In • Technological problems dominated

other words, the schools spent time the project during the first year, but
restructuring the school day so that were then resolved for most schools.
they could benefit from the educational
opportunities that ICT represented. At the school level
A number of the regions reported a
• Those schools which worked holis-
positive impact on the pupils’ learning
achievement with respect to academic tically achieved the best results in
performance, motivation for learning terms of school development and
and changes of the subject content ICT was also more integrated into
pedagogical practices.
through the use of digital learning
• There were divisions and conflicts in
resources.
the teaching staff at most schools,
but there were major variations in
Results from this project showed that
schools handled the challenges of relation to how this was handled by
change and the introduction of ICT as the school leaders.
a new object in very different ways. • Over the course of the period,
Four typologies of schools were identi- a majority of the PILOT schools
fied according to two dimensions, one attained a larger contact network
going from unsystematic versus sys- vis-à-vis the local community.
tematic in the way school communities • The significance of easing the tran-
worked towards school development, sition between the school levels was
and another going from being devel- documented.
opment oriented in the school culture
towards being dominated by resist- School administrators
ance towards change (Erstad, 2004). • PILOT as a project involving the
whole school community was chal-
Findings on different levels lenging for school administrators.
• The majority of principals reported
Use of technology that the school had initiated changes
in activities in the school organisa-
• Writing activities of pupils and teach- tion due to the integration of ICT,
ers increased. such as no longer using paper for
• There are differences between pupils sending out messages and instead
and teachers in relation to how they putting them on the local network.
use ICT. Many teachers do not rec-
ognise the pedagogical opportuni-
ties that the technology affords. Pupils and teachers
• Pupils and teachers experienced an • PILOT focused on the importance
increase in their competencies in of professionalising the teaching
using ICT. profession.
• Use of digital portfolios provided • Pupils want a teacher who is a clear
many pedagogical opportunities, for academic and pedagogical leader
example in connection with parent/ even though ICT is used more
teacher meetings. extensively.
• Pupils, teachers and head teachers • The majority of teachers were uncer-
were positive towards the use of ICT tain about the pedagogical use of
in teaching throughout the project. ICT.

26
 Multilevel approach to address complexity

• Teachers believed that ICT has a to the advancement of knowledge and

positive effect on pupils’ perform- experiences are important . Of course ,
ance, that it creates more flexibil- the challenges for optimal function of
ity and differentiation , and that this such networks are huge and it might
tendency was amplified during the be difficult to find the right balance
course of PILOT. between a strong leadership for devel-
• After the introduction of ICT, teach- opment and stimulating initiatives
ers experienced a positive change in among participants where leadership
their work day that intensified during is more invisible . Networks are by
the PILOT period . definition decentralised , which makes
• There is often a small group of leadership and division of responsibil-
enthusiastic teachers running the ity and labour a challenge . The main
activities . Activists are important . focus is on the role technology has in
supporting and building networks for
Sustainability learning .

• The school leaders reported that An important aim of the programme

they would continue the restructur- has been diffusion of innovations to
ing efforts and ICT work after PILOT a large number of schools, through
had finished .
small funds and incentives . In the
•

Learning communities help create different reports during the last four
a basis for and support change years , teachers and school leaders
processes .
report that the economic funds have
• In the majority of schools, the PILOT not been the most important incentives
activities gained a stronger local for participating . Rather it is the possi-
foundation .
bility of working with others in building
capacities that make both each school
Example 2 -

Networks of but also the collective efforts in each

learning network stronger.

The Ministry of Education decided

in 2004 to establish a national pro- Starting up
gramme for school development and The first year of the programme was
ICT called ‘ networks of learning’ , in all dominated by a lot of insecurity,
regions of Norway. It was structured unclear definitions of responsibility on
with 10 schools in each network, from different levels (locally, regionally and
primary to upper secondary levels , nationally) and technologies that did
and with one teacher training college not work optimally between schools .
leading each network. About 600 dif- After the first year, the participating
ferent schools were involved in the schools became more experienced ,
project until June 2009 when the pro- and the division of labour and
gramme ended . responsibility was made clearer, which
created a platform to define a new
Organising by networks is an alter- phase of more strategic development.
native to a hierarchical and rational The intention of the programme was
goal -oriented approach , where the to build up capacities for learning
main aim is to develop the collective and networking that could be further
competence in the group of members . developed after the programme
Strategies for collaboration, develop- ended , implying a model for expansive
ments of trust and support in addition learning and knowledge building . By

27
 Chapter I — Context and general refiections

using a strategy of reflection on action, Diversity of network models

networks have been able to learn from A qualitative study was done towards
the challenges and tensions during the end of the programme, doing inter-
the first phase, for example in the views with different participants in
way networks have become more several networks. This study shows a
focused in their work, concentrating broad diversity of experiences across
on certain aspects of technology use
different networks (Skogerbø, Ottestad
and educational perspectives, instead and Axelsen, 2007), both related to the
of trying to be too broad in their way networks work with different issues,
approach. and to the different ways networks are
organised. The development process of
Experiences by school leaders the networks became more focused and
meaningful for the participants when
Arecent report (Eliassen, Jøsendal and each network defined a specific issue or
Erstad, 2008), based on a survey done theme to concentrate on. For example,
among school leaders in the partici- some schools focused specifically on
pating schools, shows that the overall multimodal texts and how teachers and
impression is that there is a very posi- students could use specific technol-
tive attitude among both school lead- ogy within different subject domains.
ers and school communities towards Others looked at how schools in a net-
working together in networks in this work could use a learning management
way to build capacity for change. The system (LMS) to support collaboration.
school leaders further reported the In this way, the networks also gained a
following. clearer idea of the possible potential of
using ICT for certain purposes, which
• The experiences of working more increased the reported time spent with
closely with the teacher-training using ICT at these schools.
colleges was inspiring and created
better conditions for school devel- An interesting outcome so far has
opment because they had someone been to see how networks organise
from outside their own community to themselves in different ways, often
follow them over time and give feed- based on local interests and expe-
back on activities both online and riences. Some keep a hierarchical
offline. model where the teacher training col-
• Participation in this programme lege in the network is taking the lead.
increased the amount of discussions Others are organised in a much more
about educational issues, on school horizontal way, with different schools
development and the use of ICT. contributing in different ways and
• In general they have positive experi- taking responsibility, without any spe-
ences of working with other schools, cific overall leaders. One success cri-
mainly in smaller networks (mini- teria for many networks has been the
networks) between teachers from development of mini-networks within
different schools or with one or two the larger network. In this way, teach-
other school communities. ers within science education could
• The use of ICT both for networking develop their own network based on
and in educational settings improved, their interests and needs, or principals
but not as much as expected at the could have their own network. These
beginning of the programme. mini-networks have shown interesting

28
 Multilevel approach to address complexity

developments of knowledge building, special planning. In some mini-

focusing on how to build experiences networks, online collaboration has
and knowledge together over time. worked better because they have a
more focused approach and a clearer
The working method chosen in most understanding of why they use
networks was a combination of meet- online resources for networking.
ings where participants met face to
• Schools that already had experi-
face, and online collaborative efforts.
The physical meetings turned out to ence with using ICT reported that
be very important for the networks, they felt that they gave more than
they got in return. This is due to
because they got time to discuss and
the way networks were organised,
reflect together and to bring up tensions
where schools with more experi-
and problems in the developmental
ence in using ICT should work with
process at the schools, as part of the
schools with less experience in this
expansive learning processes. The
area, but which might have experi-
teachers and school leaders reported
ence in other areas that they could
that these meetings had an important
bring to the collaboration.
function to make the networks evolve
as communities of learning. • Commitment of school leaders and
school owners to make sure of sus-
tainability over time.
Limitations and challenges
A meta-evaluation of experiences and
activities
challengeswiththiskindofdevelopment
show that there are important Dimensions of indicators
work involving many actors on different So how might these examples and
levels of the education system. This the discussion above help in devel-
also indicates that this programme has oping a multilevel approach on indi-
limitations related to the initial objec- cators about the impact of ICT on
tives and ambitions of the programme education? Most importantly, what is
from policy level. Some important chal- described above shows the necessity
lenges have been the following. of understanding ICT and its impact
on education on different levels. The
• To get teacher training colleges synergy of different levels is the basis
to become more development ori- for change and development in both
ented. Many of these colleges have projects, where ICT is both a catalyst
huge challenges in keeping up with for change and a new cultural tool for
developments within schools, espe- enhancing student learning.
cially on using ICT.
• Many teachers report lack of enough This implies a higher degree of com-
time to follow up development work plexity in developing indicators.
as intended. The way schools are However, the results from studies like
organised and the daily duties of the ones mentioned above show that
teachers make development efforts schools that define ICT as important
come on top of everything else. on different levels of the organisation
• Almost all networks have reported and have a strategy for how the whole
difficulties in keeping up activities school should orient itself towards
between face-to-face meetings. the use of ICT are more successful
Online activities to stimulate devel- in using ICT for educational purposes
opment work are difficult without than other schools.

29
 Chapter I — Context and general refiections

Below, I present some key compo- In her book Literacy for sustainable
nents that are important as sets of development in the age of information
indicators to measure the impact of (1999); Naz Rassool presents an over-
ICT on education. Again, I will mainly view of different debates on literacy
build on projects and developments in in recent decades. Her point is that
Norway. Perspectives on digital litera- research perspectives on technology
cies/competencies are seen here as and literacy need to reconceptualise
something that frames these sets of power structures within the informa-
indicators, something aggregated that tion society, with an emphasis on ‘com-
relates to all indicators in one way or municative competence’ in relation to
another. democratic citizenship. Digital technol-
ogies create new possibilities for how
people relate to each other, how knowl-
Across levels: digital edge is defined in negotiation between
literacy as the framing actors and how it changes our concep-
tion of learning environments in which
Digital literacy relates to both an ability actors make meaning. Empowerment
to operate technological applications is related to the active use of differ-
and to use technology to accomplish ent tools, which must be based upon
personal and collective needs. In this the prerequisite that actors have the
sense, it raises important questions competence and critical perspective on
about new digital divides in the popula- how to use them for learning. Literacy,
tion, between the ones who know how seen in this way, implies processes of
to operate the technology and the ones inclusion and exclusion. Some have
who do not, and between the ones who the skills and know-how to use them for
use the technology to gain relevant personal development, but others do
knowledge for education and the ones not. Schooling is meant to counteract
who use it for other purposes. such cultural processes of exclusion.

One report on conceptualising digital/

This implies that we constantly have ICT literacy often referred to is Digital
to ask the more general question of transformations: a framework for ICT
what it means to ‘read’ and ‘write’ in Literacy (ETS, 2002) written by a team
a culture, and thereby how we learn of experts for the Educational Testing
(Pahl and Rowsell, 2005). In their Service in the USA. In this report, they
Handbook of literacy and technology: identified some key concepts of what
in
transformations a post-typographic they called ICT literacy. One interpre-
world, David Reinking et al. (1998) tation of such key concepts can be the
present several perspectives on how following (my elaboration based on
the development of digital technolo- ETS). (See Figure 1).
gies changes conceptions of text, of
readers and writers and ultimately of This consists of more general com-
literacy itself. This implies that digital petencies (communicate, create,
literacy relates to changes in traditional access, information handling, critical/
cultural techniques such as reading analytical) that are not connected to
and writing, and yet meanwhile opens specific subjects in school or specific
up new dimensions to what it means technologies. They can be taught and
to be a competent reader and writer are not only related to what is learned
in our culture, and the institutions that in school settings, but also to situa-
support these processes. tions outside the school.

30
 Multilevel approach to address complexity

Basic skills
Be able to open software, sort out and save information on the com-
puter and other simple skills in using the computer and software.

Download Be able to download different information types from the Internet .

Search Know about and be able to get access to information .

Be able to orient oneself in digital networks, learning strategies in

Navigate
using the Internet.

Be able to organise information according to a certain classification

Classify
scheme or genre.

Be able to compare and put together different types of information

Integrate related to multimodal texts .

Be able to check and evaluate if one has got the information one
Evaluate seeks to get from searching the Internet. Be able to judge the qua-
lity, relevance, objectivity and usefulness of the information one has
found . Critical evaluation of sources.

Communicate
Be able to communicate information and express oneself through
different mediation means .

Be able to take part in net-based interactions of learning, and take

Cooperate advantage of digital technology to cooperate and take part in
networks.

Be able to produce and create different forms of information as

Create
multimodal texts, make web pages and so forth . Be able to develop
something new by using specific tools and software. Remixing diffe-
rent existing texts into something new.

Figure 1. Key concepts of ICT literacy ( my elaboration based on key concepts in the ETS report)

Other frameworks have used ‘ dig- prise the use of multimedia technology
ital competence’ as an overall term . to retrieve , assess , store , produce ,
One example is the working group present and exchange information ,
on key competences of the European and to communicate and participate in
Commission, ‘ Education and training networks via the Internet.’ ( European
2010’ . This programme identifies dig- Commission , 2004 , p. 14). Digital
ital competence as one of the eight competence in this framework encom-
domains of key competencies, defin- passes knowledge, skills and attitudes
ing it as ‘ the confident and critical use related to such technologies .
of information society technologies
for work, leisure and communication . As shown in this section , there are dif-
These competencies are related to log- ferent frameworks to relate to in our
ical and critical thinking , to high -level understanding of digital literacy/com-
information management skills and to petence which relate to different levels
well - developed communication skills . and issues . However, the key challenge
At the most basic level , ICT skills com- is to go deeper into the implications of

31
 Chapter I - Context and general reflections

increased use of new technologies in A more fruitful approach would be to

educational practices . study impact on different levels and look
at co - variation between levels . This will
give a broader and richer understand-
Different levels combined ing of impact that is also closer to the
experiences of schools .
Most often , impact has been related to
the individual level . The interest, espe- One way of defining indicators on dif-
cially among policymakers , has been ferent levels is to describe them on
in student outcomes when using ICT. macro , meso and micro levels . Two
However, this is not as easy to detect of the levels of indicators mentioned

as it might seem. It has been prob- in Figure 2 are on the macro level
lematic to define clear effects and out- (national , local). The meso levels
would be the institutional and learning
comes , first of all because it is difficult
environments . The micro levels focus
to isolate the effects of ICT itself since
on teacher and student practices and
most schools change many aspects of outcomes (collective and individual).
their teaching and learning practices Below is an attempt to bring together
when they start to use ICT. different levels and different contexts

National
dispositions

Individual
Local
use and
dispositions
outcome

Impact of
ICT on
education
Collective
Institutional
collaboration
framing
and outcome

Learning Teacher
environment education

Figure 2: Different levels of understanding the impact of ICT in education

32
 Multilevel approach to address complexity

where ICT plays a role for education curriculum as of 2006. From a former
and learning. situation where ICT was mentioned
as a tool that might be integrated into
the classroom, the new curriculum
Indicators and levels states that ICT has to be used in all
For each level, a set of indicators is of subjects and on all levels of compul-
relevance, and for some levels indi- sory schooling. There has thus been
cators of impact are well established, a marked impact on the curriculum.
while for others the development of • Infrastructure/access: In most coun-
indicators has been limited. tries during the last decade there has
been a prime focus on making com-
Different levels and indicators also puters and Internet connections avail-
imply different methods of collecting able to educational institutions. This
information on the possible impact of has partly been a national responsi-
ICT on education. Monitoring of impact bility by ministries and other national
can be done in several ways as a com- agencies, and is expressed in different
bination of quantitative and qualitative national documents and action plans.
methods. Some countries have also adopted
instruments to monitor progress in
National level this area, which specify the ratio of
Impact on a national level deals with computers and Internet access per
key factors of importance for how ICT students and teachers. A critique has
is implemented in the school system in surfaced in recent years about the
different countries. This is most of all focus on implementation of technol-
related to the ways countries define ogy in the education system for too
ICT as of importance in educational much technological determinism.
development. This is to go beyond the • Standardisation: Many countries
policy slogans about the importance of have started work on standardisation
of technological solutions. The ISO
ICT in itself and a technological deter-
minism, and focus more on the con- standard has been implemented in
crete steps taken by policymakers in several European countries for the
different countries. The methods used coordination of technological devel-
for such indications of impact could opments and to make use more
be analysis of policy documents and accessible across different technolo-
monitoring through national surveys gies and platforms. This has become
an important part of technological
of developments within the education
system. Some key indicators on this strategies on national levels, as an
level are as follows. indication of developments within
ICT and education systems.
• Curriculum development: In many • Digital learning resources: National
countries, ICT is mentioned in curric- initiatives to stimulate the production
ulum documents, but it differs in what of digital learning resources have
way and to what extent. In most coun- been important, yet problematic, in
tries, curricula are important in the many countries. As such, they are
way they frame the education system an important indicator of progress
and the practices taking place within on a national level, because they
these systems. For example, in my are important for how teachers
own country (Norway), digital literacy and students use ICT in education.
has been written into the national Publishing companies have invested

33
 Chapter I - Context and general reflections

in technological developments to tures , which are important especially

develop different learning resources to secure the use of ICT among
beyond the book. Yet investments teachers .
have not always made a profit and
such companies are often reluctant Institutional level
to make the necessary investments . •

Leadership: On the institutional level ,

This has also raised issues about
the leadership at the school is impor-
public and private collaborations to tant in creating the setting for ICT
develop such resources on a sys- use . This of course relates to the
tems level for education .
• Use: Some countries have instru-
implementation strategies developed
by national and local authorities , but
ments to follow the actual use of ICT
on different levels within the educa-
also to how the leadership gives
direction to certain developments .
tion system . This is to get a national This also concerns how the school
overview of implications of invest- and the leadership at the school
ments and implies a set of indicators
make the strategies for school devel-
to be developed at a national level
opment with the use of ICT explicit. It
to map how ICT is used on different often varies how the school leader-
levels and subjects in order to com-
ship manages to develop strategies
pare and see developments .
that have real implications on a prac-
tical level . Another indicator concern-
Local level
ing leadership could be how schools
•

Strategies : Important on a local level use ICT as an administrative tool .

is the extent to which local authori- • School culture : Each school is differ-
ties develop strategies, expressed ent from another due to differences
in different kinds of documents , to in leadership, the teacher community,
give a direction for the implementa- the local community of the school ,
tion and use of ICT in education . It the student population and so forth .
varies a lot as to how well such docu- School culture relates to the daily
ments and local policies are devel- life of each school . The school cul-
oped and used . Some are too vague ture influences the way ICT is imple-
and contain unrealistic intentions and mented and used in the school . As
visions ; others have clear objectives shown in the PILOT project above,
and implementation plans . some schools see ICT as a catalyst
• Infrastructure/access : Even though for change while others are much
there are national policies concern- more sceptical towards ICT.
ing the implementation of infrastruc- • Collaboration : This could be an indi-
ture , it varies to what extent this cation of the ways teachers collabo-
is followed up on a local level . It is rate and share experiences in order
therefore necessary to develop indi- to build up competencies in using
cators that track the implementation ICT. Collaboration could also be
of infrastructure on a local level . between schools , between school
•

Support: Another important aspect leaders in a community, or between

concerning impact on a local level students nationally and internation-
is support structures , both for imple- ally. The point is that this is often an
mentation of technology and guide- indication of how schools use ICT as
lines for use . Local authorities have a tool for collaboration .
•

been important in many countries Reorganisation : An indication of impact

in developing such support struc- on the institutional level also relates to

34
 Multilevel approach to address complexity

the extent to which schools start to resources are used within the learn-
reorganise their practices due to the ing environment.
implementation of new technologies . • Assessment : To what extent assess-
For example , that the introduction of ment procedures are changed . How
laptops makes it difficult to uphold a teachers and students use sum-
traditional classroom setting . mative and/or formative ways of
assessment .
Teacher education level
• Teachers ’ ICT competence : To what Collective level
extent teacher education has imple- • Collaborative work: This point is an
mented courses and strategies
indication of how the use of ICT might
towards the increased competence stimulate more collaborative work
of teachers in using ICT is an impor- among students , and that project
tant part of educational development work becomes more prevalent in
and change . This could be seen as schools .
ICT literacy indicators for teacher •

Sharing content: To what extent stu-

education , and of how teachers are dents and teachers upload content
prepared to face the challenges in produced in schools to the Web and
their practice as teachers .
•
sharing it with others. Or the extent
Teaching methods : This point relates
to which they reuse content that they
to the training of teachers in differ-
find on the Web as part of their own
ent methods of using ICT and digital
learning activities .
resources . This implies a change
within teacher training colleges in the
Individual level
way the teaching profession might be
performed using ICT. • Outcomes : Different indications of
• Written strategies : For schools , the outcomes of ICT use on the indi-
teacher training colleges also need vidual level , both in a summative and
written strategy documents that give a formative way related to learning .
direction and indications of change . •

Knowledge building, problem solving :

The ways in which ICT stimulates
Learning environment level knowledge building and problem
• ICT use: The ways ICT is actually solving among students , assessed
used within learning environments . by performance assessment.
•

Flexibility: At school level , the tradi- • ICT competencies : The differences in

tional classroom might be changed ICT competencies among students ,
into a more flexible understanding the digital divide.
of learning spaces and rooms, big or
small , which are used for learning . These are just some examples of indi-
The technology might push for this . cators that might be thought of on dif-
• Online/offline: Learning environ- ferent levels . Some indicators overlap
ments might also be thought of as on different levels ; others are unique
a combination of face-to-face offline for specific levels . When we have this
interaction , and online environments more holistic view of indicators on dif-
for learning activities . This also indi- ferent levels , we might see better how
cates an opening up of the learning they are important in different ways on
environment to the outside world . different levels . Some of these levels
• DLRs used : This concerns the and indicators are directed towards
extent to which digital learning preconditions for use of ICT, some

35
 Chapter I — Context and general refiections

towards the framing of such use and avoiding reducing ICT in education to
some towards the actual use and a question of whether students learn
outcomes of such use. Indicators on better now than before. Change and
national and local levels are primarily outcome is about the system of edu-
preconditions for use in the way they cation and how students learn is con-
create the platform and the basics for nected to teachers’ competencies
use by providing the technology. The in this area, about the assessment
framing relates to the institutional level, system, about the available digital
teacher education and the learning learning resources and so forth.
environment, which create conditions Policymakers can develop strategies
for how ICT will be used in educational for systems of indicators and col-
settings, while the collective and indi- lection of such data that will provide
vidual aspects relate more directly to them with the necessary tools for
the use of ICT itself and to outcomes creating capacity for further develop-
of such use. ment within this area.
• Practice: In order to stimulate use
Implications of ICT in educational practice, we
need a better understanding of the
In specifying indicators of ICT in edu- interrelationship between different
cation, the argument in this article has levels, and how each of them might
been to draw different levels together strengthen or hinder changes within
in order to get a fuller and wider under- educational practices. It is the impact
standing of the role of ICT in our edu- on the practical level that is of impor-
cation system. As stated, this is not an tance, but that level is dependent
easy task, but the risk of reducing the on developments on other levels,
complexity of impact of ICT on our edu- like school leadership, digital learn-
cation system is that we only see a part ing resources, curriculum develop-
of the picture, and that we do not see ment and so forth. Teachers and
how things are interconnected. students need a framework that
stimulates change and development.
Such a multilevel approach has implica-
Perspectives on digital/ICT literacies,
tions for policy, practice and research.
for example, have real implications
• Policy: Policymakers need to take into on a practical level in the way this
consideration how the system levels term applies to certain learning objec-
interconnect with the practice levels tives using ICT. In addition, it relates
in their understanding of impact. My directly to several other levels.
impression is that policies within this • Research: There is a need for more
area have moved beyond simple research that manages to grasp the
technological determinism, believ- complexity of the matters mentioned
ing that technology itself will create above. One example given in this
change, towards an awareness of article is activity theory developed
the complexity involved in drawing by Yrjö Engeström, but we need
up policies for ICT in education. Still, more development in this area to be
the understanding of impact is often able to develop analytic concepts
drawn towards simple outcomes and research tools that can help us
on the individual level. A multilevel research such a multilevel approach
approach might give a more realistic to the impact of ICT on educa-
understanding of how impact is inter- tion better than we are able to at
related on different levels, thereby present.

36
 Multilevel approach to address complexity

References
Arnseth, H. C. and Ludvigsen, S. (2006). ‘Approaching institutional contexts:
systemic versus dialogic research in CSCL’, International Journal of Computer-
Supported Collaborative Learning, Vol. 1, No 2.
Balanskat, A., Blamire, R. and Kefala, S. (2006). The ICT impact report: a review
of studies of ICT impact on schools in Europe. Brussels: European Schoolnet,
European Commission.
Castells, M. (1996). The rise of the network society, the information age: economy,
society and culture, Vol. I. Cambridge, MA: Blackwell.
Condie, R. and Munroe, B. (2007). The impact of ICT in schools — a landscape
review. London: Becta Research.
Credé, A. and Mansell, R. (1998). Knowledge societies … in a nutshell: information
technology for sustainable development. Report for the UN Commission on Science
and Technology for Development and the International Development Research
Centre. Retrieved at www.idrc.ca/openbooks/858-9/, on 30.07.2009.
Cuban, L. (1986). Teachers and machines: the use of classroom technology since
1920. New York: Teachers College Press.
Cuban, L. (2001). Oversold and underused: computers in the classroom. Cambridge,
MA: Harvard University Press.
De Corte, E., Verschaffel, L., Entwistle, N. and van Merienboer, J. (eds) (2003).
Powerful learning environments: unravelling basic components and dimensions.
Amsterdam: Pergamon.
Eliassen, E., Jøsendal, J. S. and Erstad, O. (2008). Ledelse av Lærende nettverk.
(Leadership of Networks for Learning) ITU. Oslo: University of Oslo.
Engeström, Y. (1987). Learning by expanding: an activity–theoretical approach to
developmental research. Helsinki. Retrieved 30.07.2009 at http://lchc.ucsd.edu/
MCA/Paper/Engestrom/expanding/toc.htm.
Engeström, Y., Engeström, R. and Suntion, A. (2002). ‘Can a school community
learn to master its own future? An activity-theoretical study of expansive learning
among middle school teachers’, in: G. Wells and G. Claxton (eds) Learning for life
in the 21st century. Oxford: Blackwell.
Erstad, O. (2004). ‘PILOTer for skoleutvikling’ (PILOTs for school development. Final
and summary report of the PILOT project. 1999–2003). Report No 28. Oslo: ITU.
Erstad, O. (2005). Digital kompetanse i skolen (Digital literacy in the school). Oslo:
Universitetsforlaget (University Press).
ETS (2002). Digital transformation: a framework for ICT literacy. Princeton, NJ:
Educational Testing Service.
European Commission. (2004). Key competences for lifelong learning: a European
reference framework. Directorate-General for Education and Culture. Retrieved
on 30.07.2009 from http://europa.eu.int/comm/education/policies/2010/doc/basic-
frame.pdf

37
 Chapter I — Context and general refiections

Fullan, M. G. (1991). The new meaning of educational change. London: Cassell.

Fullan, M.G. (1993). Change forces: probing the depths of educational reform.
London: Falmer Press.
Goodlad, J. I. and Anderson, R. H. (1984). A place called school. Blacklick, OH:
McGraw-Hill.
Gray, J., Hopkins, D., Reynolds, D., Wilcox, B., Farell, S. and Jesson, D. (1999).
Improving schools: performance and potential. Buckingham: Open University
Press.
Hakkarainen, K., Palonen, T., Paavola, S. and Lehtinen, E. (2004). Communities of
networked expertise. Amsterdam: Elsevier.
Kozma, R. B. (ed.) (2003). Technology, innovation and educational change: a global
perspective. Eugene, OR: ISTE Publ.
Krumsvik, R. (ed.) (2009). Learning in the network society and the digitized school.
New York: Nova Science Publishers.
Law, J. and Mol, A. (2002). Complexities: social studies of knowledge practices.
Durham, NC: Duke University Press.
Olson, D. (2003). Psychological theory and educational reform. Cambridge:
Cambridge University Press.
Pahl, K. and Rowsell, J. (2005). Literacy and education: understanding the new
literacy studies in the classroom. Thousand Oaks, CA: Sage.
Rassool, N. (1999). Literacy for sustainable development in the age of information.
Clevedon: Multilingual Matters Ltd.
Reinking, D., McKenna, M. C., Labbo, L. D. and Kieffer, R. D. (eds) (1998). Handbook
of literacy and technology: transformations in a post-typographic world. Mahwah,
NJ: Lawrence Erlbaum.
Scardamalia, M. and Bereiter, C. (2006). ‘Knowledge building: theory, pedagogy,
and technology’, in: R. K. Sawyer (ed.) The Cambridge handbook of the learning
sciences. Cambridge: Cambridge University Press.
Skogerbø, M., Ottestad, G. and Axelsen, H. K. (2007). Lærende nettverk — fra
informasjonsutveksling til kunnskapsdannelse? (Learning networks — from infor-
mation sharing to knowledge building?) Report. Oslo: ITU.
Sutherland, R., Robertson, S. and John, P. (2009). Improving classroom learning
with ICT. London: Routledge.
Thomson, P. (2007). Whole school change: a review of the literature. London:
Creative Partnerships.
Van Dijk, J. (2009). ‘Outline of a multilevel approach of the network society’, paper
presented at the annual meeting of the International Communication Association,
Sheraton New York, New York City, NY. Retrieved 30.07.2009 at http://www.allaca-
demic.com/meta/p12672_index.html
Veugelers, W. and O’hair, M. J. (eds) (2005). Network learning for educational
change. Buckingham: Open University Press.

38
CHAPTER
CHAPTER
STATE OF THE ART

Monitoring in education : an overview

What do we know about the effective uses

of information and communication technologies
in education in developing countries?

39
Monitoring in education: an overview
Willem Pelgrum
EdAsMo

Abstract
In this article, a description of educational monitoring will be provided. This consti-
tuted the background for a study about monitoring ICT in primary and secondary
education in the EU (1) (see Chapter IV: Indicators on ICT in primary and second-
ary education). First the function of monitoring for policymaking will be described,
showing that educational monitors in general can have different functions, and
the concepts of policy goals, indicators, instruments and data will be introduced.
A distinction can be made between international, national and school monitor-
ing. This is followed by a description of the main steps involved in designing and
conducting international comparative educational monitors, sketching a number
of dilemmas for which solutions need to be sought. This is followed by a review of
methodological issues in international comparative monitoring.

Functions of monitoring many different methods can be used

for collecting observations. Qualitative
Monitoring can be defined very broadly and quantitative methods can be dis-
as ‘the act of periodically/continu- tinguished. In this study, the main
ously observing something’. The act focus is on quantitative methods that
of observation will be called ‘assess- allow for comparisons between coun-
ment’ further on and hence regular tries and, hence, imply statistical gen-
assessment equals monitoring. An eralisations to the educational system
educational monitor is thus ‘assess- at large.
ment of education and how it is devel-
oping over time’. This definition is fairly A distinction can be made between
neutral and could, in certain situa- national and international monitors.
tions, when explicit targets are set, be National educational monitors are
translated into ‘assessment of educa- meant to draw conclusions about
tion in order to determine if standards changes that take place in educational
are met’. Educational monitoring can systems over time, which implies that
be focused on many different charac-
the observations are collected in
teristics of education, such as input,
such a way that they are comparable
processes and learning outcomes and
over time. International comparative
educational monitors offer possibilities
to interpret the state of the art and/
(1) This study was financed (at a cost of or changes over time in one country
EUR 122 200) by the European Commission.
Contract EACEA-2007-3278. Opinions presented
with reference to changes in other
in this chapter do not reflect or engage the countries, provided that the measures
Community. © European Commission that are used are internationally

41
 Chapter II State of the art

comparable between countries and different settings , for instance at the

over time . international (worldwide , regional ),
national , school and even individual
A fairly recent development is school level .
monitoring , whereby schools keep
track of their developments (some- Given the purpose of our study, we
times in comparison with other will further focus mainly on the inter-
schools ) for evidence-based school national level and will describe below
policymaking . A full multilevel monitor in more detail each of the steps that
would be a system in which interna- are distinguished in Figure 1 , in par-
tional , national and school monitoring ticular in terms of what is required in
are integrated . each of these steps , which concepts
are relevant and which questions and
dilemmas will be faced .
Monitoring in general can be con-
ceived as regular assessments that
are part of a cyclic policy process 1. Policy goals
that consists of a number of steps , as
shown in Figure 1. Monitoring implies Whereas national monitors are focused
a regular repeat of step 2 (Figure 2) . on policy goals that are relevant for one
country’s stakeholders , the group of
Figure 2 concerns a very general stakeholders is larger for international
model that can be applied in many monitors and the participating countries

Policy documents

1. Policy
goals/intended Research literature
curriculum

2. Assessment Primary indicators

3. Evaluation and
reflection

4. Diagnosis Secondary indicators

5. Interventions

Figure 1 : Steps in evidence-based policy cycle

42
 Monitoring in education

Monitor

Assessment 1 Assessment 2 Assessment 3

Evaluation Evaluation

time

Diagnosis Intervention Diagnosis Intervention

Figure 2: Monitoring as a regular repeat of assessments

need to decide first on which common that have a connection to the Internet’ .
goals a monitor should be focused . An However, when the goal statements
example of a common goal might be are fairly global , as is often the case in
‘To connect all schools to the Internet’ . international consensus -building proc-
A dilemma in establishing common esses (e.g. ‘ provide all students with
goals is that some goals may be highly access to the Internet’ ) , a number of
relevant in some countries (e.g. those different indicator definitions may be
which are just starting to connect to the needed ( e.g. number of Internet con-
Internet) , but not or not yet relevant in nected computers per 100 students ,
other countries (e.g. those which have connection speed, etc.) .
already realised this goal). We will call
this ‘ goal disparities’ . A serious problem in defining indicators
concerns their ‘ comprehensiveness ’ ,
What can also happen is that certain which is the extent to which they ade-
common goals have a short ‘lifetime’ , quately cover the domain that is implied
so that they were perhaps highly rel- by the goal statements . Monitors can
evant in a certain time period , but potentially have quite serious (unin-
were no longer so later on (for ex- tended ) conservative impacts on edu-
ample because the goals have been cational policymaking if the compre-
reached) . In relation to ICT particularly, hensiveness is low. This can occur if,
where rapid technological develop- for instance, they do not cover relatively
ments are taking place , this is an issue new competencies , but rather focus on
of special concern (in this respect the traditional competencies of students .
notion of ‘ life expectancy’ of indicators For example , suppose that the use of
becomes relevant) . ICT leads to a slight decrease in math-
ematics skills (for which an indicator is
Once common goals have been available) , because as a result of stu-
established , indicators for monitor- dents’ autonomous working less con-
ing the progress towards these goals tent can be covered . If, at the same
need to be defined. If goal statements time, a high increase in communication
are very concrete , as in the example and studying skills (for which no indi-
above , this may be relatively easy to cators are defined) occurs , this posi-
do, such as ‘the percentage of schools tive effect would remain unnoticed and

43
 Chapter II State of the art

there would be a chance that ICT use issues and constraints that need to be
in mathematics would be discouraged . considered when designing an inter-
In this respect, the notion of ‘ holistic national comparative assessment.
monitoring ’ is relevant. International Firstly, as the instruments are adminis-
comparative assessments may have tered to educational actors in schools
a big impact on education . Recently a (school leaders , teachers , students
consortium of Cisco, Intel and Microsoft and sometimes parents) a serious
concluded that, in order to reform edu- constraint is the amount of time that
cation , the current prevailing inter- can be asked from each respondent
national comparative assessments to answer the tests/questionnaires .
would have to be changed . Increasing the amount of time will lead
to lower response rates , which then in
For practical reasons , the number of turn would affect the quality of national
indicators that can be addressed in an statistical estimates that are based on
assessment is limited (see point 2) . the collected data . As the number of
Therefore establishing priority needs questions that can be included in ques-
is an essential aspect of step 1 . tionnaires is limited , this in turn has
implications for the number of intended
An important distinction in Figure 1 is indicators that can be included . Initial
between ‘ primary’ and ‘ secondary’ indi-
priority decisions can be made on the
cators (sometimes also called respec-
basis of a priori response time esti-
tively key indicators and background
mates . Further, during the process of
or explanatory indicators). Primary
indicators are those that are featured operationalisation and piloting (when
as the main focus of an assessment; response -time estimates can be col-
for instance when it concerns PISA or lected ) it may appear that the number
of intended indicators needs to be fur-
IEA-TIMSS- PIRLS , primary indicators
ther reduced .
concern the test results in mathemat-
ics, science and/or reading , which An important issue concerning the
are usually the first to be featured operationalisation of intended indi-
when statistical reports from these
international monitors are released .
cators concerns costs . Developing
completely new indicators is a time-
Secondary indicators are used to
consuming process , because empiri-
throw further light on the test results , cal evidence needs to be collected
for instance by examining difference
regarding the comparability, statistical
in outcomes between sub-populations
quality and interpretability of the new
in countries (e.g. boys and girls ) or for measures .
analysing how the differences between
countries can be explained . After the data are collected , ‘ indicator
statistics’ can be calculated . For exam-
2. Assessment ple , when an indicator definition might
be ‘ use of ICT ’ , one of the indicator
An international comparative assess- statistics might be ‘percentage of stu-
ment consists of collecting data in rep- dents using ICT daily at school ’ . If the
resentative national samples on the same intended indicator was included
basis of instruments (usually question- in earlier assessments , another indi-
naires and tests) that contain opera- cator statistic might be ‘increase of
tionalisations of the intended indica- daily use of ICT at school between
tors ( from step 1 ) . There are several 2000 and 2009 ' .

44
 Monitoring in education

3. Evaluation system , the need may arise to try to

find out what the potential reasons are
and reflection that could lead to interventions aimed

Once the indicator statistics are at realising improvements. Existing

available , the interpretation of the international comparative monitors
outcomes can start. This would be usually include quite a number of sec-
quite a straightforward process in ondary indicators that are intended
cases where the common goals and to be used for explaining the differ-
ences between countries and between
intended indicators were phrased in
schools and students within coun-
operational terms (e.g. ‘we expect that
at least 90 % of students are using tries . A common experience among
ICT daily in school’ ) , which for obvi- researchers involved in the process of
finding causes is that the set of second-
ous reasons is hardly ever the case.
ary indicators is too limited to answer
Hence , this is usually a very tedi-
ous process , in which many different concrete why questions that are posed
after the data have been collected , and
groups of stakeholders have their say.
hence this often does not result in con-
It is not uncommon to observe that
national researchers present well- crete suggestions for policy interven-
qualified interpretations covering both tions that could lead to improvement . A
more fundamental problem is that the
strengths and weaknesses , while after
collected data do not allow for cause-
the publication of the international
reports , the media and certain groups effect analyses . At best they can result
of stakeholders interpret the outcomes in strengthening or weakening par-
ticular beliefs about cause and effects .
like Olympic league tables , whereby a Therefore some countries occasion-
place lower than the top three is quali-
fied as bad and reason for serious ally conduct additional research in
order to find out whether handles can
policy concern and action . This is the
phase where the highest risk exists be found for improvement . In the past,
of eliciting conservative effects (see one country (the Netherlands ), scoring
above), but many other potential falla- low on international reading tests , con-
cies exist that could lead to unjustified ducted in -depth analyses on the read-
interpretations of the statistics. ing methods used in schools and con-
cluded that these were no longer up
Evaluation and reflection may lead to to date. A change of reading methods
generating concerns and questions took place and later it appeared that
about potential causes of what has the international ranking had consid-
been observed . In general , it appears erably improved , which strengthened
that in countries ranking high on the the belief that the reading methods
primary indicators , not many initiatives were among the potential causes of
for follow- up activities will occur, while low performance . This is an example of
in countries that rank low, questions qualitative follow-up of the international
will be raised about potential causes assessment .
of this outcome .

5. Interventions
4. Diagnosis
Throughout the world , many examples
Once the primary indicator statistics are available of policy actions that
have raised concerns about the exist- were undertaken as a result of the
ence of weaknesses in the education outcomes of international comparative

45
 Chapter II — State of the art

assessments. It seems safe to This may, in particular, be the case

infer, on the basis of the continuing when it concerns ICT indicators.
increase of participating countries in
international comparative educational The main steps underlying the design
monitors (from 20 in the IEA studies and execution of a monitor can be
in the 1980s to over 60 in the current summarised as follows.
IEA studies), that policymakers are
becoming more aware of the potential 1. Establishing common objectives
benefits of international comparative 2. Defining indicators
educational monitors for evidence- 3. Operationalising indicators
based policymaking. (= instruments)
4. Drawing samples of respondents
It should be noted that interventions do 5. Collecting data
6. Presenting descriptive results
not necessarily need to be top-down:
if schools in a country could see how 7. Generating questions for
diagnosis
they perform on the primary indicators
8. Analysing data
(by means of school monitoring) and
9. Making recommendations for
make inferences about the existence interventions
of potential weaknesses and their
10. Making recommendations for
likely causes, these initiatives might revised/new indicators.
be designed and generated at school
level. This approach is advocated in
some EU countries. Participation of EU
The policy cycle that is sketched above
countries in full-scale
may help to illustrate several functions international comparative
that international comparative moni- educational monitors
tors may have, such as: description
(mirror), accountability, benchmark- Despite the needs for monitoring,
ing, enlightenment, understanding resulting from benchmarks that were
established after the open method of
and cross-national research. Some
of these functions (such as bench- coordination was introduced follow-
ing the Lisbon 2000 summit, the EU
marking, monitoring, understanding
has no system in place for a full-scale
and cross-national research) can be
regular monitoring of quantitative
explicitly addressed by the research
indicators of student skills in primary
design, while other functions are more and secondary education. It there-
or less collateral (mirror, enlighten-
fore needs to rely on data collected
ment). Monitors can help in the proc- through other international organisa-
ess of evidence-based policymaking tions, mainly the IEA (International
by which decisions are based on facts Association for the Evaluation of
rather than rhetoric. In this sense, Educational Achievement) and the
monitors are also conceived as navi- OECD (Organisation for Economic
gation tools. However, one should Cooperation and Development). Both
also be aware of potential resistance organisations conduct regular inter-
to participate in international compar- national comparative assessments
ative monitors, as these may be per- measuring (among other things) stu-
ceived as leading to undesirable influ- dents’ skills in mathematics, science
ences on educational policymaking. and reading.

46
 Monitoring in education

The IEA has existed for over 50 years . Intentions may be formally legislated
As a non-governmental organisa- in syllabi , examination standards or
tion , it conducts large-scale quantita- in the words of the IEA ‘ intended cur-
tive assessment in mathematics , sci- ricula ’ . These constitute the basis for
ence, reading, civic education and ICT, guiding many educational processes ,
amongst other things . The core studies such as the content of the textbooks ,
(in mathematics, science and reading) teaching and learning activities in
take place roughly every four years schools , the content of (in-service or
and , since 2000 , the assessments pre-service) teacher training , etc. An
have also been conducted roughly analysis of these intentions is usually
each four years . In 2011 , a combined the basis for designing international
assessment of mathematics , science comparative assessments that are
and reading will take place. The OECD currently run by international organi-
PISA assessment was conducted for sations , such as OECD ( PISA) and
the first time in 2000 and is run every IEA (TIMSS , PIRLS) . These analyses
three years . The core performance may be based on extensive curricu-
domains are mathematics , science and lum analyses (IEA) or expert opinions
reading . The latest assessment took about what the important life skills
place in 2009 and is expected to be are that students need to acquire in
reported by the end of 2010. The next schools ( OECD) . The outcomes of
assessment is scheduled for 2012 . such analyses constitute the basis for
developing the content specifications
Since 2000 , the majority of EU coun- for the instruments that are used to
tries have participated in the OECD measure educational outcomes (e.g.
assessments (PISA) and/or IEA in the cognitive domain , such as math-
(TIMSS and PIRLS, respectively ematics , science and reading , but also
mathematics/science and reading) at affective , e.g. learning motivation) ,
the primary and/or secondary educa- whereas on the other hand these
tion level .
content specifications can also be
used for measuring the opportunities
that schools offer to students to learn
Core areas for monitoring these contents . Educational moni-
For monitoring educational progress , toring that would only be focused on
at least three main core areas need to these three core concepts would allow
be considered , namely: educational actors to make a limited
number of inferences , such as :
• intended learning outcomes ;
•

opportunities to learn (OTL); • for national monitors :

• competencies/attitudes of students . whether intentions , OTL and out-

comes are changing over time ,
Definitions of intended outcomes whether discrepancies exist
are needed for steering educational between intentions and OTL ,
processes that result in OTL , which -

whether inequities exist between

in turn are supposed to influence the sub - populations of students and
competencies and attitudes of stu- how these are changing over
dents . Moreover, these definitions are time;
needed to be able to construct tests • for international monitors :

for measuring the extent to which the the same as for national moni-
intentions are realised . torsbut with enhanced possi-

47
 Chapter II — State of the art

bilities to interpret the national — the ICT-related competencies of

observations with reference students;
to what is happening in other • instrumental concepts, such as:
countries. — the competencies of teachers
Although such inferences are impor- about ICT (technical ICT literacy)
tant as a first step towards understand- and the use of ICT (pedagogical
ing educational progress, they would ICT literacy),
offer insufficient handles for undertak- — the number of hours scheduled
ing policy actions for remediation. for learning about ICT,
— the availability and quality of ICT
Therefore it is necessary in educa- learning materials.
tional monitors to also address con- When ICT is conceived as a transversal
cepts that are (politically) malleable issue, the concepts mentioned above
and which relate to areas that are could be considered all instrumental.
believed to influence OTL and out-
comes. For these concepts (earlier we
referred to secondary indicators), an Methodological issues in
almost endless variety of candidates
could be generated, such as:
international comparative
monitoring
• competencies of teachers;
• number of hours in the timetable International comparative educational
scheduled for certain OTL areas; monitors (later in this section abbrevi-
• availability and quality of learning ated to ICEMs) are designed in such
materials; a way that high-quality data are col-
• instructional methods applied; lected that allow for generalisations to
• school organisation and quality of the defined national target populations
leadership; and for comparisons between coun-
• class climate; tries. Several aspects that need to
• examination standards. be considered in the design of these
monitors are described in the sections
For the study that formed the basis of below.
this chapter, the question was how ICT
fits into the picture sketched above. Measurement
ICT can be conceived as a transversal
issue as well as a subject area. In order to be able to make statements
about the concepts (and derived indi-
When ICT is a subject area (such as cators) underlying the assessment,
mathematics and science in existing measures are needed that can be used
international comparative monitors) for statistical generalisations. With
the previous concepts could be trans- regard to measurement, a main dis-
lated into, for example: tinction that can be made is between
‘what’ and ‘whom’ is measured.
• core concepts, such as:
— the intentions (formally legislated ‘What’ refers to the constructs that are
or informally adhered to) with materialised in instruments. Typically
regard to ICT literacy, in ICEMs, which are targeting stu-
— the OTLs for learning about and dents, the following types of instru-
learning with ICT, ments are distinguished.

48
 Monitoring in education

• Context instruments: for collecting collection , in order to determine

information about school external which items constitute the best test.
conditions ( e.g. funding , regula- Psychometric analyses are con-
tions , curriculum) . For instance, to ducted to check if the items fit in
what extent do curricula prescribe intended scales .
the use of ICT? • Translation verification . All items are
• School instruments : containing originally in the English language .
questions about school character- They are translated into national
istics (e.g. organisation , manage- languages — which can be many, as
ment, school policies). For instance , in South Africa where in IEA-PIRLS
how many computers are available (reading literacy) a translation into
in schools or what is the vision of 12 languages is needed . It is cru-
school leaders about desirable ped- cially important that the translation
agogical approaches using ICT? matches the international version as
• Teacher instruments : containing well as possible . The quality of the
questions about instructional prac- translations is checked by involving
tices . For instance , to what extent professional translators .
•

do teachers use ICT for testing Lay- out verification . As even the lay-
students? out of tests and questionnaires may
• Student instruments: tests for meas- influence the responses of the test-
uring achievement and student ees , the national lay-out of tests and
questionnaires about activities and questionnaires is checked at the
background . For instance , how often international coordination centres to
did you use a computer for learning determine whether any deviations
mathematics? can be discovered .

The question as to who is measured

Development ofinstruments
relates to the issue of populations and
The construction of international
samples , which is discussed in the
instruments is usually a very time-con- next section .
suming activity in which many steps
are specified that are all intended to
improve the quality of the assess- Populations and samples
•
ments . Some of these activities are Population definitions
described briefly below. The purpose of international assess-
ments is to provide good national
• Involvement of international experts . estimates of the indicators that have
At the start of international assess- been defined for students , schools
ments , committees of experts with a and/ or teachers . The challenge is to
good reputation in the areas that are define populations in such a way that
tested are formed . These commit- they are comparable across countries .
tees review items in order to guaran- This is a complex task that cannot
tee that they represent the content always be solved to the complete sat-
area . isfaction of all participants . The IEA
• National experts are involved in and OECD use different approaches
judging proposed items in terms of for defining populations . In IEA stud-
their fit with national curricula . ies , the definitions are grade- based ,
• Pilot testing is conducted on roughly which means that within each educa-
double the amount of items that is tion system a particular target grade
actually needed for the main data is chosen that corresponds to an

49
 Chapter II — State of the art

international population definition. countries grade repetition occurs fre-

Most IEA studies are focused on stu- quently, resulting in large age-varia-
dent populations at three levels in the tion between countries.
education system: primary education,
lower secondary education and upper The approach of the OECD to defining
secondary education. Definitions that student populations is different from
were used in TIMSS2003 (for what in the IEA: it is age-based. In PISA2003,
most countries constitutes the primary the definition was ‘all students who are
and lower secondary level) were, for aged between 15 years 3 months and
example: ‘all students enrolled in the 16 years 2 months at the time of the
upper of the two adjacent grades that assessment, regardless of the grade
contained the largest proportion of or type of institution in which they are
13-year-old students at the time of enrolled and of whether they are in full-
testing and all students enrolled in the time or part-time education’ (OECD,
upper of the two adjacent grades that 2004). This definition has several dis-
contained the largest proportion of advantages when it comes to the com-
9-year-olds. These correspond to the parability of populations: the number
eighth and fourth grade in practically of years in school may differ between
every country’ (Mullis et al., 2004). countries, and students are at differ-
ent grade levels. A practical problem
The main reason for choosing a grade- is the collection of data from teach-
based definition is that in IEA assess- ers about their instructional practices
ments, teachers and student data are which can be linked to the students. To
linked and hence IEA is targeting data some extent, this can be overcome by
collection in intact classes. It should asking students to provide information
be noted that linkage is also possible about their teachers.
when targeting individual students, but
more complicated: for many teachers, An example may illustrate the prob-
their current reference point for their lem of differences between countries
instructional activities is still an intact in terms of population characteristics.
class. This is typical for the traditional The results of PISA2003 showed that
organisation of teaching and learn- the scores of Danish students were
ing in schools. If the current reform moderate (roughly 36 score points
trends in education (which call for under the top for mathematics) as
more individual learning trajectories compared to other countries, which
and multidisciplinary team teaching) resulted in some consternation among
are implemented on a large scale, the stakeholders in Denmark, particularly
target class approach may need to be because the Danish education system
changed. is believed to be one of the best in the
world. Certainly the expenditures on
Grade-based definitions do not nec- education are quite high (for second-
essarily result in comparable popula- ary education, 35 % of GDP, which
tions across countries. The national is among the highest in the world).
definitions may still result in large vari- However, when comparing the charac-
ability with regard to characteristics teristics of the populations of students
that impact the interpretation of the from countries in PISA2003 it appears
assessment outcomes, for instance: that Danish students were, in compari-
the number of years that students are son with students from other countries,
in school may differ, while in some one year less in school, because of a

50
 Monitoring in education

later entry age into compulsory edu- means of m ± 0.1s (where ‘ m’ is a

cation . From TIMSS1995, Pelgrum mean estimate and ‘ s’ is its esti-
and Plomp (2002) estimated that one mated standard deviation ) and for
year of schooling can result in score percentages : p ± 5 % ( where ‘ p’ is a
point differences that varied between percentage estimate) .
13 and 44. Hence , one may wonder
•
Participation rates : criteria are
if the position of Denmark in the inter- defined for participation rates that
national tables might be attributable should be reached in order to con-
to the deviating educational career of sider a sample acceptable .
Danish students . Another complication
of using age- based samples is that the These standards have implications ,
logistics of data collection are much amongst others , for the reporting
more complex (resulting in a heavier of the outcomes . Flags are applied
burden on NRCs and schools , causing for samples that are considered not
higher costs and risks of higher non- too far below standards . The results
response) . of some countries are ‘flagged’ and
shown ‘ below the line ’ which means
The population definitions also have that the sample quality is considered
implications for the definitions of teach- to be insufficient . It also happens
ers and school populations that is , that the results of some countries
to which populations the results can are excluded from the international
be generalised . This is reflected in reports , which occurred with the
the way the results are presented, for Netherlands in PISA2000 . An example
example: of rules for flagging from TIMSS2003
is provided by Martin et al . (2004) , see
• IEA : percentage of students by their http://timss.bc.edu .
school’s report of teachers ’ involve-
ment in professional development Data collection and quality
opportunities in mathematics and
control
science (TIMSS2003 , Exhibit 6.6. ) ;
• OECD: percentage of students in The collection of data is a crucial
schools where the principals report phase in any ICEM . The purpose is
that mathematics teachers were that a high percentage of the sampled
monitored in the preceding year respondents answer the question-
through the following methods naires and/or tests as accurately and
(Figure 5.17, PISA2003); completely as possible. Any loss of
•

Sampling data or inaccuracies (such as unread-

able answers) will result in lower data
Once the population definitions for quality and fewer possibilities for pro-
each country are settled , samples can ducing good estimates of population
be drawn . These samples need to be parameters . There may be many rea-
of high quality in order to warrant good sons why data gets lost, such as the
estimates for the whole population . following .
Therefore , international assessments
apply sampling standards, which cover • Questionnaires may not reach
a number of aspects . respondents , for instance because
of failing mail services , wrong or
•

Accuracy: the population param- unreadable addresses , sloppy

eters should have an accuracy for administration in schools , etc.

51
 Chapter II State of the art

• Answers to questions or test items the instruments . This can mean that
may be unreadable or conflicting whole teams are busy for a consider-
(e.g. more than one answer) . able amount of time with :
• The materials are not correctly
•

returned , for example because of checking returned questionnaires

wrong addresses , failing mail serv- and tests for completeness and
ices , wrong handling at the data readability;
collection institute or sloppiness at •

contacting schools to get hold

schools ( sometimes materials were of missing materials or to clarify
completed but returned one year unreadable answers ;
after data collection) . •

reminding schools by (e-)mail or

phone to return the materials ;
In order to minimise data loss as much •

informing schools about the disas-

as possible , rigorous procedures trous effects when they, on second
are nowadays implemented in most thoughts ( after an initial agreement
ICEMs , that are all documented in to participate) , are inclined not to
manuals and software programs as is participate : sometimes the data for
shown for instance in the TIMSS2003
a whole country are excluded from
technical report (see http://timss . the international reports .
bc.edu/ for more details ) .
For planning a period for data collec-
In particular, when achievement tests
tion , it is important to try to avoid over-
are used , it is of crucial importance
lap with other time-consuming and
that the test administration takes place
competing activities in school , such as
in a very controlled manner in order
the weeks before the school holidays ,
to avoid the test scores being biased
when everyone is busy with end -of-
downwards or upwards . This requires
term activities , or, in some countries ,
the following , for instance .
the periods in which the final examina-
• Cheating should be avoided . tions are taking place .
• Students need to be motivated to
answer the test -
this is particu- Data collection is one of the biggest
larly important because quite often budget items for national teams ,
students will perceive the test as because it is time consuming and
low- stake as it will not have conse- requires quite high expenditures for
quences for their grades in school . materials (printing , mailing ). Hence,
• Use of tools such as calculators one would expect that considerable
or other aids should be standard- budget reductions might be possible
ised when the data are collected electroni-
this is not always possible ,
because in some countries certain cally, via online data collection (ODC) .
aids are always allowed while this is ODC was not feasible for a long time ,
not the case in other countries . This because respondents (schools , teach-
may have serious consequences ers and/or students ) did not have
for the interpretation of differences access to ICT, the Internet or were not
between countries . competent enough to use these facili-
ties . The IEA SITES2006 was the first
Nowadays many countries have to ICEM to apply ODC on a large scale.
spend substantial budgets in order A feasibility test of ODC , conducted in
to guarantee the proper return of two groups of respondents , randomly

52
 Monitoring in education

allocated to a paper version and an More research is needed to investigate

ODC version , revealed the following . and try out these possibilities . This
requires staging and cooperation at
The analyses showed that the results the international level , involving differ-
obtained from the two modes of data col- ent partners that are still working inde-
lection are comparable , although there pendently (such as national and inter-
are some differences and issues to be national assessment organisations ) .
taken into consideration . One of the most

important issues is the level of drop-out Data entryand file building

in web - based questionnaires . Despite a
The purpose of data entry is to enter
higher missing item rate in web-based
the answers from respondents accu-
questionnaires , this method appears to
rately in data files . This should prefer-
provide a reliable data collection method
ably be done by highly qualified key
when compared to equivalent paper-
punchers . But, as this is a human
based questionnaires . ( Brečko and
activity, failures are possible (this also
Carstens , 2006)
holds for optical mark reading , where
human interventions to solve ambigu-
SITES2006 was a study that only
used instruments at school and ous responses are needed) . Such fail-
ures may, if undiscovered , have a huge
teacher level . The sample sizes for
impact on the statistical estimates .
these categories of respondents are
usually relatively small in ICEMs and, There are several tools used to dimin-
hence , the efficiency profit is much ish data- entry failures :
less than when ODC can also be used
for students in those assessments that •

use of data- entry programs that con-

administer tests and/or questionnaires tain immediate checks when data
to students . Feasibility tests of ODC are entered , such as :
for large- scale student assessments valid codes for categorical
still need to be tried out . variables ,
valid ranges ( e.g. for the school
Additional advantages of applying sizes , number of computers,
ODC in the future might be : etc. );
• data checks that consist of , for
•

negligible costs for data entry (see example :

next section ) ; ID checks : every identification
• tailored testing ; number for respondents in the
•

performance testing , such as testing data files should be unique ,

via simulations , practical laboratory linkage checks: every student
skills , communication competen- should be linkable to a unique
cies , etc .; teacher and unique school ,
•
possibility to provide more direct inconsistency checks: the
and timely feedback to respondents answer to a filter question should
(which may increase willingness to not be in conflict with subsequent
participate); questions;
•

more continuous and periodic moni- • data analytical checks in the national
toring for large samples of schools centre as well as the international
and possibilities for school self data management centre, such as:
evaluation ; distribution of answers should
•

more possibilities for diagnosis . be plausible when compared

53
 Chapter II State of the art

with other statistical sources in ators ( NRCs ) and the international

a country, e.g. from other inves- coordinating centre ( ICC) — quite
tigations or national census often suspicious statistics are dis-
statistics , covered by comparing univariate
-

the ‘ behaviour’ of variables statistics and sometime even at

should be plausible, e.g. an unu- a late stage by inspecting the out-
sually high correlation between comes of the analyses , for instance
two variables in one or a few as a result of an undiscovered trans-
countries may be suspicious and lation error ( e.g. ‘don’t mind ’ trans-
could point to errors in the data, lated as ‘ don’t like’ in SITESM1 ) .
— examining differential item-func-
tioning , e.g. relative high p-val- Errors in the data are not always caused
ues in one country as compared
by data entry failures . They may also
with other countries could poten-
tially ( but not necessarily) point result from printing errors or national
to flaws in the translation of test adaptations in questionnaires .
items.
When potential problems are discov- Due to all these checking and valida-
tion steps , it can take some time to
ered in the data, it is often necessary to
go back to the returned questionnaires produce international data sets that
and/ or tests to find out what was actu- are ready for further processing and
ally entered by the respondents . analysis . However, it is usually not
until the first official report is published
It goes without saying that by applying (about a year after data collection)
ODC , many of the problems that result that the data sets are considered to
from human failures during data entry be in their final shape and ready for
can be avoided . However, ODC is not public access . This is because , as
a panacea for getting error-free data , mentioned above , even at a late stage
because : errors in the data can be detected .

respondents may accidentally hit The purpose of data processing is to

wrong keys – it is not known whether produce statistics that were envisaged
this is maybe more likely than acci- when conceptualising and designing
dentally hitting a wrong answer in a the ICEM . These statistics may be :
printed questionnaire (e.g. for a filter
question); • univariate and based on one vari-
• for open questions requiring the able (e.g. a percentage of students
specification of a number (e.g.
having a computer at home) or com-
number of students in school) ,
posed on a set of variables (e.g. a
respondents may accidentally write
mean number of possession from a
a wrong number.
set of 10 in students’ homes ) ;
• bivariate , for instance breakdowns
Part of the procedure for checking and
cleaning data is: of such test scores for boys and girls
or correlations , e.g. between score
•

inspection of national univariate stat- on a like - math - scale and the math-
istics by each participant; achievement score ;
•

inspection of international univari- • multivariate, e.g. structural models

ates by national research coordin- that are fitted on the data .

54
 Monitoring in education

A persistent problem in data process- random samples , this can be simply

ing is how to handle missing data, such calculated by dividing the standard
as when respondents (intentionally or deviation of a statistic by the square
accidentally) have not answered a root of the number of cases . As was
question . For example , the following explained in the section about popu-
cases may occur. lations and sampling , ICEMs are not
normally based on random samples of
•

Missing should be interpreted as 0 . students , but are rather so-called clus-

Although respondents are explicitly ter-samples : first schools are selected
instructed to write a zero if that is the and then students within schools .
answer to particular open questions , As the units in these clusters usu-
this does not always happen , and ally resemble each other, the effect
bias can be introduced in the statis- of this approach is that less accurate
tical estimates . estimates can be made . Hence , using
•

Missing to be interpreted as neu- statistical tests from standard software

tral , such as when response scales such as SPSS is not correct and there-
are used without a neutral answer fore appropriate dedicated procedures
category. need to be developed for each ICEM
•

Missing by design , such as when separately.

matrix sampling (2) is used .

Missing data that result from design Data analysis

are often replaced by imputed values . The purpose of data analysis in gen-
Most imputations take place via regres- eral is to find answers to several types
sion analyses in which a large number of questions , such as the following .
of variables are used to predict scores
•

on the variable that contains missing 'Why’ questions , e.g. ‘Why are the
codes . Once the regression weights achievement scores in certain coun-
are known , these are used to ‘ predict’ tries low?’ , ‘ Why are the scores
the score for the missing answers . on emerging- practice indicators in
some countries much higher than in
How to handle other missing data other countries ?’ .
requires a close inspection of the data , • Questions about hypothesised rela-
because , as argued above, hypoth- tionships : e.g. ‘ Is the availability of
eses need to be generated about what ICT related to the extent that emerg-
missing may mean . ing pedagogical practices exist in
schools?
A very important step in data process- •
Exploratory questions : ‘Which
ing is the calculation of appropriate school factors are associated with
standard errors for the statistics that the existence of emerging pedagog-
are produced . A standard error is an ical practices ?
estimate of the sampling inaccuracy. It
is used to describe the so-called confi- In the current international descrip-
dence interval for statistical estimates tive reports , variables can be found
of population parameters . For simple that could be of interest for further
analysing the data , e.g. breakdowns
(2) Matrix sampling means that a sample of of achievement scores by different
questions is administered to a (sub)sample of groups of students, those having com-
respondents. puters at home , low, medium and high

55
 Chapter II State of the art

social welfare index, etc. The current riori research questions . This in itself is
reports , particularly the PISA reports , not a fundamental problem , but rather
also contain initial results of more in- the lack of a coherent and long-term
depth analyses . research agenda is, or in the words of
Martin et al . (2004): ‘ more work needs
However, these analyses do not offer to be done to identify the most fruitful
more than a first approach to the analy- variables to capture the dynamic proc-
sis of the data. For a comprehensive esses that take place within schools
analysis , the ‘behaviour’ of a large set and to understand how national and
of variables needs to be taken into cultural contexts interact with other
account , which is often done by fitting factors to influence how education is
models on the data (confirmatory, that transmitted and received’ .
is , based on an a priori hypothesised
structure ; or exploratory and aimed at Reporting
generating a posteriori hypotheses,
As argued earlier in this chapter, an
which is more common: by trying out
important step in any ICEM is the valu-
many different models and by deter-
ation of the results . ICEM reports offer
mining which model fits the best).
a rich variety of statistics that can help
Examples of statistical programs for
the participants to judge the results
modelling are LISREL and AMOS
for their country. In ICEMs this is usu-
(part of the SPSS package). As the
data often have a multi -level character ally a relative judgment, that is, coun-
try statistics are valued on the basis
(school- , teacher- and student-level) ,
of comparisons with other countries .
so called hierarchical linear modelling
A danger in interpreting the statistics
(HLM) programs are also used . Finding may be that too much of an atomistic
appropriate models that fit the data well
approach is used (focusing on one or
is a time -consuming process , which
a few subject areas) rather than trying
often takes place after the first descrip-
to value an education system from a
tive ICEM reports have been published .
holistic perspective .
It should , however, be noted that the
OECD included quite a lot of multivari- However, it can be observed that once
ate analyses in the PISA reports .
the final report has been released ,
absolute judgments also enter the
Sometimes special issues of journals
or dedicated books are devoted to
scene , e.g. some people claiming that
despite the high score of a country in
secondary analyses of the assess- fact the quality of maths achievement
ment data ( e.g. Robitaille and Beaton ,
is very low. This happened recently
2002) . However there is a lack of up-
in the Netherlands , when a group of
to-date meta-analyses , showing which researchers from the Freudenthal
analyses have been done over the Institute for Science and Mathematics
years and which results have been
Education concluded that , despite the
reported . Such an activity is important,
high international ranking, the level
among other reasons because it is not of achievement in the PISA tests was
yet very well understood why some very low.
variables are highly intercorrelated
in some countries but not in others .
Also , as mentioned before , quite often Secondary analyses
constraints of studies do not allow for ICEMS result in huge data sets
enough variables covering the a poste- (50 countries with on average

56
 Monitoring in education

5 000 students per country is not • Data analysis: international com-

uncommon) that are nowadays easily parative data sets nowadays offer a
accessible for several purposes. The wealth of opportunities to investigate
background documents on design how certain measures behave under
and methodological issues (sampling, different circumstances. Questions
technical standards, psychometrics) include: ‘do attitude measures from
also reflect how researchers in the field Japanese and UK data show the
apply theoretical insights from educa- same underlying dimensions?’.
tional methodology. These data can be • Substantive questions: international
of value for examining and illustrating comparative assessments typically
several methodological topics and for cover a broad range of topics. For
conducting substantive research after instance, the tests for measuring
the data have been archived, includ- student achievement may con-
ing the following.
tain hundreds of questions cover-
ing a large part of the mathematics
• Conceptualisation (concepts and domain. Detailed examination of
indicators): every interested person
these items may reveal much more
can have access to instruments, con-
than the overall test statistics which
ceptual frameworks and data and,
are published in the international
hence, can be involved in reflecting
reports.
about the choices that were made in
particular assessments.
Whereas in earlier days the use of
• Questionnaire development: by criti-
international databases was compli-
cally examining questionnaires that
cated (often the data files which were
have been used in international
assessments, forming hypotheses stored on tape did not even fit on hard
about the strong and weak points and disks of mainframe computers) nowa-
analysing the data to find evidence days anyone with a relatively simple
for these hypotheses, much can be laptop can download the data bases
learned about issues that concern and conduct analyses. Such analyses
questionnaire development. usually require an in-depth under-
• Sampling: several issues are worth standing of technical details, such as:
examining and discovering in the
international data files. • which sampling weights are avail-
— Is the accuracy of the population able in the data files and how these
estimates comparable to theo- should be used;
retical expectations? • how to calculate standard errors in a
— Do education systems where correct way, taking into account the
streaming occurs have higher sampling design of the studies;
intra-class correlations than sys- • how to apply the so-called plausible
tems where this is not the case? values that are stored in the files.
• Data collection: international com-
parative assessment projects over However, when carefully studying
the past 30 years have developed a the technical documentation and
whole set of tips and tricks for col- user guides that are available for the
lecting high-quality data from large ICEMs, secondary analyses are pos-
samples of students, teachers and sible for almost everyone with some
schools in a country. affinity for statistics.

57
 Chapter II State of the art

PISA data can be explored online (3) . use of the Internet as reported by
Japanese students (see Chapter 4) .
•

A tool that may be useful in doing sec- Self- ratings : Quite often in inter-
ondary analyses on IEA data is the national as well as national ICT
IEA International Database Analyzer monitors , instead of using objective
(IEA IDB Analyzer) , a plug-in for SPSS standardised tests , students and/
that helps to correctly handle data and or teachers are asked to rate their
which can be found at http://www.iea. own ICT competencies . Although
nl /iea_software.html . such measures may be fine as indi-
cators of self-confidence , they are
often used as proxies for real com-
Additional methodological
petences . Such use is unwarranted ,
considerations for monitoring as self- ratings are prone to bias
ICTin education (Stromsheim, 2002; Ross, 2006).
In addition to the general methodologi- • Teacher perceptions : Some assess-
cal issues that were (not exhaustively) ments in the past included percep-
reviewed above , when monitoring ICT tions of teachers regarding the
the following additional issues should impact of ICT on , for instance, moti-
be considered . vation and skills of students . The
validity of such measures is highly
•

Terminology: In questionnaires used questionable and the ratings are

for collecting data with regard to ICT prone to wishful thinking . Hence , in
indicators , the term ‘ computer’ is future assessments , such measures
often used as a stand- in for the more should only be used as an indicator
of teachers ’ attitudes towards ICT.
general term ‘ICT’ . Given the techno-
logical developments in recent years
itadequately
is questionable whether ‘computer’
covers the current tech-
Summary and conclusions
nology options . For example , when In the previous sections , a number of
students in primary education are key terms were introduced that play a
asked whether they use computers role in monitoring . A key term for this
during lessons and where they only study is the word ‘indicator’ , which
use interactive whiteboards , one does not have an unequivocal defini-
may wonder whether they recognise tion . Literally it means ‘ an indication of
this device as a computer. In this something that is not directly observa-
case, the use of the word ‘ computer’ ble’ . Indicators may be categorised in
may lead to downwards biased esti- terms of global descriptions of rather
mates of ICT use during lessons at broad areas , or more concrete defini-
school . The same problem applies tions . For instance , a broad area con-
to questions like ‘ Do you use a com- cerns ICT infrastructure in education .
puter for accessing the Internet? ’ It Many different indicator definitions
is quite likely that students who use may be distinguished within this area ,
mobile phones for accessing the such as ‘ Quantity of available PCs in
Internet would answer ‘ no’ to such
schools ’ . Such definitions are guiding
question . Maybe this could be an the development of instruments to col-
explanation for the extremely low lect data , which consist of response-
codes delivered by respondents and
(3 ) For PISA 2003: http://pisaweb.acer.edu.au/ stored in data files . Once these data
oecd_2003/ oecd_pisa_data_s2.php. are available , several statistics may

58
 Monitoring in education

be calculated in order to provide a are available . Hence , these termino-

quantitative estimate for the targeted logical distinctions were relevant for
indicator. Hence these will be called the purpose of our study.
‘ indicator statistics’ in this study. For
instance , examples of statistics for The distinction between primary and
the indicator definition just mentioned secondary indicators was also intro-
might be : mean number of comput- duced and the problem was men-
ers per school in each country or the tioned of defining appropriate second-
median number of computers . Another ary indicators before the questions for
relevant statistic might be the mean the phase of diagnosis are generated .
number of students per available com- It was pointed out that in order to avoid
puter in a country’s education system . undesirable impacts on educational
decision-making , holistic monitoring is
It is conceivable that data are avail- needed. It was also argued that multi-
able for which no indicator statistics level monitoring may be an important
exist, which can quite often happen option for the future. Several potential
in international comparative assess- advantages of online data collection
ments due to space limitations in the were mentioned that may play a role
final reports . Indicator areas may also in further discussions about a future
exist for which no indicator definitions EU ICT monitor.

References

Brecko, B. N. and Carstens , R. (2006) . Online data collection in sites 2006:

paper versus web survey. Do they provide comparable results? Paper pre-
sented at the Second International Research Conference : Proceedings of the
IRC- 2006 , Amsterdam .

Martin , M. O. , Mullis , I. V. S. , Gonzalez, E. J. and Chrostowski , S. J. (2004) .

TIMSS 2003 international science report. Findings from IEA’s trends in interna-
tional mathematics and science study at the fourth and eighth grades. Chestnut
Hill : TIMSS & PIRLS International Study Center, Boston College.

Pelgrum , W. J. and Plomp , T. (2002) . ‘ Indicators of ICT in mathematics: status

and co - variation with achievement measures’ , in : D. F. Robitaille and A. E.
Beaton (eds) , Secondary analyses of the TIMSS data. Dordrecht: Kluwer
Academic Publishers .

Ross , J. A. (2006) . ‘The reliability, validity and utility of self-assessment’ , Practical

assessment, Reasearch and Evaluation, Vol . 11 , No 10 , 2–13 .

Stromsheim , J. (2002) . ‘Information and communication technology ( ICT) com-

petencies of pupils and teachers’ (unpublished) .

59
What do we know about the effective
uses of information and communication
technologies in education in developing
countries?
Michael Trucano ( 1 )
World Bank , ICT Education and Social Sector Innovation Specialist, infoDev

Executive summary
infoDev maintains a series of knowledge maps that attempt to document what is
known — and what is not known about ICT use in education. These knowledge
maps reveal that, despite a decade of large investment in ICT to benefit educa-
tion in OECD countries, and increasing use of ICT in education in developing
countries, important gaps remain in our knowledge . In addition, there appears
to be a dearth of useful resources attempting to translate what is known to work
and not work in this field for policymakers and donor staff working on education
issues in developing countries, especially those issues related to ‘education for all’
and other education-related millennium development goals. A lack of reliable data
related to the impact of ICT on learning and achievement in developing countries,
as well as a lack of useful indicators and methodologies to measure such impact,
hampers policy guidance in this area. A mismatch also exists between methods
used to measure the effects of ICT use in education in developing countries, and
type of learning styles and practices that the introduction of ICT is meant to pro-
mote, or at least facilitate.

Despite a lack of reliable impact evidence, recent infoDev surveys of World Bank
support for ICT components in projects in its education portfolio, and country-level
surveys sponsored by infoDev of ICT use in education in Africa and the Carib-
bean, document tremendous growth in the use of — and demand for — ICT in the
education sector. This mismatch between weak evidence and growing use raises
many questions about the nature of ICT-related investments in the education sec-
tor in developing countries.

(1 ) NB: The findings , interpretations and conclusions expressed herein are entirely those of the author
and do not necessarily reflect the view of infoDev, the donors of infoDev, the International Bank for
Reconstruction and Development/World Bank and its affiliated organisations, the Board of Executive
Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the
accuracy of the data included in this work.

61
 Chapter II State of the art

What do we know Widely accepted , standard method-

ologies and indicators to assess the
about the effective impact of ICT in education do not
uses of information exist.

and communication • A disconnection is apparent between

the rationales most often presented
technologies in education to advance the use of ICT in educa-

in developing countries? tion (to introduce new teaching and

learning practices and to foster ‘21st
infoDev maintains a series of ‘ knowl- century thinking and learning skills’ )
edge maps’ outlining what is known and their actual implementation
and what is not — about the use of (predominantly for use in computer
information and communication tech- literacy and dissemination of learn-
nologies ( ICT) in education . These ing materials) .
knowledge maps reveal that, despite a
decade of heavy investment in ICT to Key findings: Costs
benefit education in OECD countries , •

Very little useful data exists on the

and increasing use of ICT in educa-
cost of ICT in education initiatives ,
tion in developing countries , signifi-
especially related to total cost of
cant gaps remain in our knowledge . In
ownership and guidance on how to
addition , there appears to be a dearth conduct cost assessments .
of useful resources for policymakers
and donor staff working on education
issues in developing countries , iden- Key findings: Current
tifying what is known to work -

and
implementation ofICT in
not to work -

in this field, especially education

in support of ‘education for all ’ (EFA)
and other education - related millen- • Interest in and use of ICT in educa-
nium development goals (MDGs ) (see tion appears to be growing , even in
Trucano , 2005) . the most challenging environments
in developing countries .
The knowledge maps, which are used
to help guide discussions between Keyfindings: Policy lessons
donors and governments exploring learned and best practices
the use of ICT in the education sector,
investigate 10 topics (impact of ICT on • Best practices and lessons learned
learning and achievement, monitoring are emerging in a number of areas ,
and evaluation , equity issues , costs , but, with few exceptions , they have
current projects and practices , spe- not been widely disseminated nor
cific ICT tools , teaching and ICTs , con- packaged into formats easily acces-
tent and curriculum, policy issues , and sible to policymakers in developing
school -level issues) . The key findings countries , and have not been explic-
are divided into four major themes . itly examined in the context of the
education - related MDGs .

Key findings: Impact While much of the rhetoric about (and

• The impact of ICT use on learning rationale for) using ICT in education
outcomes is unclear, and open to has focused on the potential for chang-
much debate . ing the teaching-learning paradigm , in

62
 Effective uses in developing countries

practice ICTs are most often used in evaluation studies of key initiatives
education in less developed countries like NEPAD e- Schools , is a first step
(LDCs) to support existing teaching in a larger, ongoing , systematic and
and learning practices with new (and , coordinated initiative to track develop-
it should be noted , often quite expen- ments in technology use in the educa-
sive) tools . While impact on student tion sector to help inform a wide vari-
achievement is still a matter of reason- ety of stakeholders interested in the
able debate , a consensus seems to topic as they seek solutions to larger,
have formed that the introduction and more fundamental educational and
use of ICT in education can help pro- development challenges in the years
mote and enable educational reform , ahead.
and that ICT is a useful tool to both

motivate learning and promote greater Keyfindings

efficiencies in education systems and
• ICT use in schools in Africa and the
practices .
Caribbean is growing rapidly (from
an admittedly low base).
Surveys of ICT use in This growth is largely the result of
‘bottom up’ initiatives , often facili-
education in developing tated by civil society organisations .
countries: what is actually Barriers to use include high costs
happening ? (especially of connectivity) , poor
infrastructure , insufficient human
Research teams supported by infoDev resource capacity, high costs , a
and coordinated by the Commonwealth variety of disincentives for use and
of Learning (COL) and others are inadequate or insufficient policy
seeking to document the major devel- frameworks .
opments in each country in Africa (see
•

Farrell et al . , 2007a, 2007b , 2007c) The process of adoption and diffu-

and the Caribbean (see Gaible , 2007) sion of ICT in education in Africa is
related to technology use in education in transition and widely variable.
in order to create the first consolidated A marked shift seems to be emerg-
look at this fast-changing sector in ing from a decade of experimenta-
these regions and provide preliminary tion in the form of donor-supported ,
answers to three broad questions . NGO-led , small-scale pilot projects
towards a new phase of systemic
• How is ICT currently being used in integration informed by national
the education sector, and what are government policies and multi-
the strategies and policies related to stakeholder-led implementation pro-
this use? cesses . ‘ This shift from projects to
• What are the common challenges policies , and the more systematic
and constraints faced by countries development that that implies , would
in this area ? not be possible without the growing
• What is actually happening on the commitment to ICT in education
ground , and to what extent are on the part of government leaders
donors involved ? across the continent’ (Farrell/Isaacs ,
2007) .
infoDev and its partners hope that
release of the results from these • ICT use in education in the
surveys , and related monitoring and Caribbean , and the context of its

63
 Chapter II State of the art

use, varies only within a limited and project leaders should think in
range . terms of combinations of input factors
ICT use in schools in the region is that can work together to influence
primarily centred on basic ICT liter- impact . Coordinating the introduction
acy instruction and computer use . of computers with national policies
and programmes related to changes
in curriculum, pedagogy, assessment
Planning for ICT use in and teacher training is more likely to
education in developing result in greater learning and other
countries: a way forward outcomes’ (Wagner, 2005) .
for policymakers The process of integrating ICT into
As an aid to education policymakers educational systems and activities can
in developing countries under tre- be ( and typically is ) arbitrary, ad hoc
mendous pressure -

from parents , and disjointed , as evidenced through

recent infoDev surveys of ICT use in
vendors , business , technology advo-
cates , etc. — to provide schools with education in the 75 developing coun-
a variety of ICT, infoDev, Unesco and tries ( Farrell et al. , 2007a , 2007b,
others partners have developed and 2007c, Trucano, 2007) . Such adhoc-
utilised an ICT- in- education toolkit racy often results in ineffective, unsus-
tainable and wasteful investments . On
as part of policy consultations in
26 countries (see Haddad , 2007 ) . the other hand , a comprehensive set
Feedback from toolkit users consist- of analytical , diagnostic and planning
ently states that provisioning ICT for tools , such as those promoted through
use in schools , no matter how hard the ICT in education toolkit , can ‘ force
and expensive initially, is the easiest a certain discipline on the process .
and cheapest element in a series of The use of tools does not make policy
formulation ‘ scientific’ and ‘ rational ’ .
policy choices that ultimately could
make ICT use sustainable and/or ben- Nor will it replace the political/organi-
eficial for learners . Indeed , the appro- sational nature of policy formulation’
priate and effective integration of ICT ( Haddad, 2007).
in schools to impact teaching and
learning practices is much more com- That said , it is clear that current tools
plicated . The proliferation of ICT use available to help aid policymakers
outside the school especially the make informed decisions about
growing use of mobile phones — has technology choices for schools are
yet to impact in any meaningful way on quite primitive . Reasonable minds can
the use of ICT within formal education argue over what is meant by ‘impact’
systems . To help guide policy choices and ‘ performance ’ , but substituting
around technology use and choice in belief for scientific inquiry does not
education in developing countries , a seem to be a particularly responsible
more robust set of shared indicators course of action . The power of ICT
and evaluation methodologies must as an enabler of change — for good ,
be developed and tested in real- as well as for bad -

is undeniable.
world circumstances . As discussed in However, the use of ICT in education
infoDev’s Monitoring and evaluation of in many developing countries ,
ICT in education projects: a handbook especially the ‘ poorest of the poor’ ,
for developing countries, ‘ evidence is associated with high cost and
to date suggests that policymakers potential failure . Simply wishing away

64
 Effective uses in developing countries

the existing local political economy of developing country policymakers —

the way technology is implemented and their partners in the international
and supported in schools does not community — can make wiser and
mean that it actually goes away. With more sustainable choices in deploying
more rigorous analysis and evidence ICT to enhance access to, and quality
of impact, and better decision tools, of, education at all levels.

References
Farrell, Glen and Isaacs, Shafika (2007a). Survey of ICT and education in Africa:
a summary report, based on 53 country surveys. Washington, DC: infoDev/
World Bank. Available at http://www.infodev.org/en/Publication.353.html, last
accessed on 22.09.2009.

Farrell, Glen, Isaacs, Shafika and Trucano, Michael (2007b). The NEPAD
e-Schools demonstration project: a work in progress (a public report).Washington,
DC: infoDev/World Bank; Vancouver, British Columbia: Commonwealth of
Learning.

Farrell, Glen, Isaacs, Shafika and Trucano, Michael (eds) (2007c). Survey of ICT
and education in Africa: Vol. 2: 53 country reports. Washington, DC: infoDev/
World Bank. Available at http://www.infodev.org/en/Publication.354.html, last
accessed on 22.09.2009.

Gaible, Edmond (2007). Critical review and survey of ICT in education in the
Caribbean. Washington, DC: infoDev/World Bank.

Haddad, Wadi (2007). ICT in education toolkit for policymakers, planners

and practitioners (version 2.0). Washington, DC: infoDev/World Bank; Paris:
Unesco.

Trucano, Michael (2007). ICT components in World Bank education projects

(2001–04).Washington, DC: infoDev/World Bank.

Trucano, Michael (2005). Knowledge maps: ICTs in education. Washington, DC:

infoDev/World Bank.

Wagner, Daniel et al. (2005). Monitoring and evaluation of ICT in education

projects: a handbook for developing countries. Washington, DC: infoDev/World
Bank.

65
CHAPTER
CHAPTER
CONCEPTUAL FRAMEWORKS

A framework for understanding and evaluating

the impact of information and communication
technologies in education

ICT to improve quality in education -

A conceptual
framework and indicators in the use of information
communication technology for education (ICT4E)

A conceptual framework for benchmarking the use

and assessing the impact of digital learning resources
in school education

67
A framework for understanding and
evaluating the impact of information and
communication technologies in education
Katerina Kikis, Friedrich Scheuermann and Ernesto Villalba

During the last decades, considerable resources have been invested in hardware,
software, connections, training and support actions under the scope of improving
the quality of teaching and learning. A major tenet of the policies that supported
the introduction of information and communication technologies (ICT) in educa-
tion was that they can become catalysts for change. Undoubtedly, some countries
have made considerable progress in bringing networked ICT into education and
made it possible for teachers and learners to use them on a daily basis. In many
other cases, however, implementation policies have not been a consequence of
systematic analysis and reflection. As a consequence, we still know little about
the impact and effectiveness of ICT in education. To close this gap, the Center
for Research on Lifelong Learning based on benchmarks and indicators (CRELL)
established a research project on measuring ICT performance and effectiveness
in education. The project explores the effects of ICT on learning outcomes aiming
at stimulating debate on educational policy needs. This paper presents the first
step in the process. It presents a conceptual framework to guide the analysis for
orienting work activity towards the study of ICT effectiveness.

Integrating ICT in education tant consequences in the articulation

of educational policies . The identified
The integration of ICT in education gap in assessing the impacts of ICT is
is affecting educational systems in especially unsatisfying for policymak-
multiple ways . Likewise , ICT use in ing stakeholders who aim at defin-
education influences the private life ing evidence-based strategies and
of all educational actors in the sense regulatory measures for effective ICT
that these are engaged in innovative implementation and efficient use of
practices which require new meth- resources .

odologies , techniques and attitudes .

Most studies carried out , however, do Emerging technologies (e.g. smart-
not provide clear information about boards , mobile devices) stimulate the
the multifaceted effects and impact of change in contextual conditions for
ICT on the learner and learning . There learning . Computer equipment and
are still unanswered questions about software are becoming increasingly
the impact of technology in the short available inside educational estab-
and long terms on learning and how lishments as well as in private house-
it has affected simple and complex holds -

not only for school -related

learning tasks. In turn , this has impor- activities of young people, but also for

69
 Chapter III — Conceptual frameworks

learning at all stages in life. Instruc- In many cases, in the context of school
tional practices are changing due to education, the massiveness of gov-
new possibilities to access and share ernment top-down ICT-related pro-
information, new roles and peda- grammes and reforms implied that poli-
gogical paradigms. Furthermore, we cymakers were expecting schools to
observe new ways of learning in the change sooner rather than later. Unlike
context of new educational software books or blackboards, digital technolo-
applications and tools provided, dig- gies tend to age and even become
ital resources available, etc. (see, for unusable within just a few years.
example, Redecker, 2009). This justi- Furthermore, technology changes very
fies once more the need to study the fast and even if older technology is
effects of ICT at different levels and still usable it can be incompatible with
to examine implications for the indi- new digital products and services or
vidual and society. More insights into be unsuitable for their full exploitation.
the multifaceted effects are needed Overall, this top-down approach has
to enable us to conduct cost-benefit had its own risks because the heavy
studies in an appropriate manner investments could pay back only if
and to react to necessary changes by schools were ‘ready enough’ in to start
updating national curricula, design- immediately using ICT productive
ing teacher training programmes ways. The massiveness of the pro-
and revising adequate school and grammes and reforms introduced also
implied that the changes anticipated
classroom implementation, keeping
in mind that ICT is often a catalyst were envisaged to take place not just in
some or even in the majority, but in all
for change but does not itself deter-
schools within a system. The reformers
mine the direction of change. There
is a lack of comprehensive studies probably pushed ahead because they
of the complex interactions between wanted to minimise the risk of creat-
ing inequalities among schools which
various types of ICT implementation make heavy use of ICT and those that,
and the effects of other factors such for one reason or another, do not. The
as school-based interventions, socio-
scenario, however, that assumed that
economic status and expenditure. It all schools would start using ICT in
appears that, firstly, we are in need of productive ways as soon as the teach-
instruments which will allow assess- ers and the pupils put their hands on it
ing and monitoring the state of use
was not very realistic. What was more
and changes affected. Secondly, we plausible was that the top-down pro-
need to identify the various sources grammes and reforms would gradu-
and gaps in a systematic manner in ally help more and more teachers and
order to determine data available and pupils alter their teaching and learning
desired. There are a number of ambi- practices. According to this scenario,
tious initiatives to explore the scope the early adopters who used ICT prior
of influencing factors already car- to the implementation of massive top-
ried out (see, for example, Ramboll down programmes and reforms will
Management, 2006; Underwood et al., soon be joined by an early majority, and
2007). They provide a good basis for the sceptics, what Rogers (1995) called
going one step further and designing the ‘late majority’, will eventually follow
a systematic approach to identify the them. As teachers and pupils convert
use of ICT and its effects on all differ- from being non-users to regular users
ent levels and stages concerned. of ICT for teaching and learning, they in

70
 Understanding and evaluating the impact

parallel learn how to use them in opti- how teachers and pupils actually use
mal ways , i.e. as they learn something ICT (utilisation indicators) , what the
new, they learn new ways to learn . In outcomes are of their use (outcome
other words, according to this scenario, indicators ) , and , more recently, what
ICT will ‘ penetrate’ and change schools the impact is of their use on school
in successive stages . learning (learning impact indicators) .
Utilisation indicators often measure
how often teachers and students use
Indicators for monitoring ICT for school teaching and learning ,
the integration of ICT what they use and for what purposes
in education (for example, what kind of software
they use for subject teaching and learn-
Such ‘ outside-inside mentality’ is also ing ), and how they use it (for exam-
evident in widespread approaches ple, whole-classroom teaching , group
to the evaluation of the integration work, individual work , etc.) . Outcome
and impact of ICT in school life . At indicators often focus on the attitudes
national and cross- national level , a of teachers and pupils towards ICT,
widespread approach to evaluation and their confidence and skills in using
is through indicators . Indicators , as ICT. They also start to focus on wider
defined by Unesco (2003), are meas- ‘strategic’ practices such as the use
uring devices to assess or evaluate of ICT for lifelong learning and pro-
materials , methods , an intervention , a fessional development , and assess-
programme or a project on the basis ment of actual ICT skills is starting
of adopted assumptions on what is rel- to be developed in some areas. It is ,
evant . Many countries worldwide have however, much less common to use
adopted quantitative and qualitative indicators to measure the impact of
indicators of the degree of integration the use of ICT on pupils’ attainment in
of ICT into schools and some of them core curriculum subjects .
have even established annual surveys
to monitor progress in this area . The development and use of indica-
tors is popular among policymak-
Input indicators are the most widely ers because they provide them with
used type of indicators , something a wealth of easy-to-use information .
that reflects the priorities of national However, it is important to bear in
policies , which commonly focus first mind that the use of indicators has
on building a minimum level of ‘frame- its limitations: generally, indicators
work conditions ’ in schools . The great- provide support to assess a current
est emphasis has been placed on state , but usually do not cover other
input indicators regarding national important issues , such as reasons
policies and the regulatory frame- for not using ICT; mental effects on
works , expenditure , teacher training , learner and learning , etc. Moreover,
the inclusion of ICT in school curricula, comparative surveys typically only
ICT infrastructure in schools and the provide a snapshot of a given situa-
access of ICT equipment by teachers tion at a very specific moment in time.
and pupils at home. As ICT gradually Furthermore, the choice of mainly
becomes an integral part of schools ‘ input’ indicators is often driven by
and elsewhere , and many teachers political priorities and the philosophy
receive training in ICT, the interest and concerns of the bodies , often
has shifted towards issues concerning government supported , issuing such

71
 Chapter III Conceptual frameworks

studies . Therefore , the indicators tend strong policy support for the creation of
to focus on areas where there has been monitoring tools in education . In 2007 ,
a recent policy initiative and they tend the Commission published the coher-
to ignore other areas which , although ent framework of indicators (European
highly relevant, are not included in the Commission , 2007b) . This communica-
current policy agenda or may reveal tion established 16 indicators that were
disturbing policy failures . For ex- adopted by the European Council and
ample, the use of the ratio between can be used to monitor Member States
pupils and computers and the ratio in the achievement of the Lisbon goals
between teachers and computers as
in education and training , one of which
input indicators draws a picture which
is ‘ ICT skills’ . In the current state , there
may be quite different from the pic-
ture which would result if the teacher: is a necessity to place this indicator
within a wider context of ICT use and
pupil ratio was also included as a third
indicator. From a wider perspective, integration. Likewise, other European
the indicators approach often reflects programmes , such as i2010 , aim at
the wider top-down , outside-inside promoting the positive contribution of
mentality that was adopted through ICT in the economy, society and qual-
the implementation of massive pro- ity of life . There is a need to have a
grammes and reforms . In a way, it is framework that will allow evaluating
a consistent part of a wider top-down the impact of ICT for this purpose, par-
policymaking culture which assumes ticularly its contribution in educational
that the starting points for generating settings .
school change are the actions of poli-
cymakers (Kollias and Kikis , 2005) .
Existing comparative data
From a European perspective , the
The OMC , as well as the trend of
development and use of indicators is
highly relevant, especially for the devel- proposing knowledge-based policies ,
opment of monitoring policies estab- requires reliable data and information
lished by the European Union . The for policymakers to enable the moni-
Lisbon strategy set up the open method toring of policies . Data needs to be
of coordination (OMC) in education comparable in order to allow for mutual
and training (among other fields). This learning between countries. In prin-
implies that Member States agreed to be ciple , it is possible to group potential
monitored in a series of issues to allow sources and instruments for assess-
for mutual policy learning . In 2002, five ing the ICT effect at a comparative
benchmarks were established as the
level into three different categories :
average level to achieve by 2010 and
several indicators were proposed for • data collected by international
monitoring purposes . In addition , the bodies ( Eurostat, World Bank ,
recent emphasis on evidence- based Unesco, OECD);
policies in education (see European • international surveys , (such as PISA,
Commission , 2007a) (1) also provides a TIMSS , PIRLS , SITES , TALIS) ;
• thematic studies (e.g. ‘ Study of the
(1) European Commission (2007) . ‘Towards impact of technology in primary
more knowledge- based policy and practice
schools ’ ( STEPS) 2009 , carried out
in education and training' . SEC (2007) 1098 .
Luxembourg : Office for Official Publications of by European Schoolnet and Empirica
the European Communities . for the European Commission) .

72
 Understanding and evaluating the impact

Additionally, national experiences and obsolete relatively fast and have to

studies are a good source of informa- be replaced , which makes it difficult to
tion. These , however, do not allow track changes over time . The survey
for comparison across countries in a is mainly directed to assessment of
straightforward manner. These consti- ICT and Internet use in the working-
tute ‘ case studies ’ and could be used age population and thus has limited
as ‘lessons learned’ . For the present value for education . The ‘ ICT usage
paper, the focus remains on the com- in enterprises’ survey only retrieves
parative , international sources of information on the so- called ‘ core
information . sectors’ of the economy, which means
that services such as education are
The data compiled by international not covered by the survey. ISS , there-
bodies might be instrumental in pro- fore, can be used to provide a picture
viding the context for the effects of of the context in which the effects of
ICT in education . Any effect has to ICT in education can be assessed but
necessarily be related to the context would need to be adapted for allowing
where it has appeared . In this regard , the study of ICT effects in education .
several international bodies collect
information on ICT infrastructure . Studies concerning education at a
The OECD , for example , publishes comparative level are carried out by
the Communications outlook and the the OECD and IEA on a regular basis .
Information technologies outlook every Their main focus is on the assessment
two years . These two publications pro- of student achievement in different
vide an overview of the situation in the competences: reading , mathematics
telecom market. They contain plenty and science . These further concern
of information on Internet availabil- themselves with investigation of ICT
ity and infrastructure as well as the use in education . PISA is probably
dynamics in industries supplying IT the best known survey of this type.
goods . Eurostat also provides a good It has had important political impact
amount of statistics through the infor- and results in PISA are used within
mation society statistics survey ( ISS). the OMC to monitor progress towards
ISS is carried out in two main surveys the Lisbon objective (the percentage
pertaining to ‘ ICT usage in enterprises’ of low-skilled readers is used as one
and ‘ ICT usage in households and of the five benchmarks agreed by the
individuals’ . The aggregate numbers Council in 2002) . PISA has a specific
can be obtained by breakdowns of age module on ICT. The module has been
group , sex, educational level , employ- modified in each of the three rounds
ment situation and region. However, of PISA (2000, 2003, 2006) and will
the information provided is limited . In probably have a different version in
terms of e -skills , for example, it is only 2009. It strives to gather information
possible to obtain the percentage of from 15-years-olds (the PISA target
people who report to have done tasks group) on the use they make of com-
of the type ‘ installed a new device’ or puters and their self-reported capac-
‘written a computer program’ in the ity for doing certain computer tasks .
last three months , in the last year or In 2004 , the OECD published a report
never. Despite the efforts of Eurostat specifically looking into PISA and ICT:
in keeping up with the pace of change Are students ready for a technology-
and adapting to new developments rich world? The report mainly looks
in ICT, some of the items become into the effects of use of ICT in student

73
 Chapter III Conceptual frameworks

performance. But it lacks information and comparison between countries

on how the computer has been used has to be done cautiously. Trucano
and in what way because of the limi- (2005) also reviews a series of stud-
tations of the ICT module question- ies on ICT impact in schools . He also
naire . TIMSS and PIRLS , carried out concludes that the impact of ICT use
under the auspices of the International on learning outcomes is unclear and
Association for the Evaluation of calls for the need for more ‘widely
Education ( IEA), also have specific accepted methodologies and indica-
information on the use of ICT. In tors to assess the impact on educa-
TIMSS, for example , information on tion ’ (Trucano, 2005, p . 1 ) . In a similar
the use of ICT is linked to subject, line , Cox and Marshall ( 2007) point
and , therefore , it is more possible to out that studies and indicators on ICT
explore the impact of the educational do not reflect sound effects .
use of ICT on student performance.
But we have no information on how They maintain that this relates mainly
the computer has been used . to three aspects :

In terms of thematic studies , there are opposing views on ICT and educa-
a number of initiatives looking specifi- tion ;
• different perspectives on/goals
cally into aspects of ICT in education .
Empirica (2006), in a study financed for innovation in learning/learning
by the European Commission , contexts ;
•

explores the access and use of ICT in missing planning strategies for edu-
European schools in 2006. It presents cational change .
information for 25 EU Member States ,
Norway and Iceland , but it does not Current approaches for evaluating ICT
look into student results so it is not in education are often only focused on
possible to study this important aspect a few aspects , such as input, utilisa-
of ICT impact. Another relevant study tion and outcome/impact. By the use
is SITES , which , like TIMSS , is under of indicators , they can assess how
the auspices of the IEA. The survey the input (e.g. monetary, infrastruc-
explores the use of computers in ture , resources ) relates to the impact.
teaching through sampling teachers , These models may apply for sev-
principals and ICT responsibility in eral purposes , but come too short to
schools . It does not look into student assess the integration of ICT in poli-
achievement , but it does look at the cies and curricula, particularly because
perceived impact on ICT in students they often use a snapshot, one time
from the teacher’s perspective . and one level approach . Furthermore ,
evaluation has to care about different
states in the implementation process
The impact of ICT and analyse changes in the culture
in education of the school system — at the micro
level (pupils ) as well as at the meso
Balanskat et al . (2006) reviewed sev- (school ) and macro (curriculum/attain-
eral studies on the impact of ICT on ment targets ) level . Therefore , a con-
schools in Europe . They conclude that ceptual framework is needed to look
the evidence is scarce and compa- into the various dimensions of ICT use
rability is limited . Each study uses a and to discuss possibilities to measure
different methodology and approach , the effects of use of electronic media

74
 Understanding and evaluating the impact

in education . Such an orientation aims 10-year research project (2) , identifies

at constructing a framework to look at five phases of technology integration
the relevant domains and interdepend- into schools (see Dwyer et al . , 1991 ) .
ence between components related to These phases , as described by a more
ICT in educational processes from a recent report on school technology
holistic perspective . This paper pro- and readiness prepared by the CEO
vides a first attempt at an innovative Forum on Education and Technology
approach to the study of the impact ( CEO , 1999 , p . 14); are : entry, adop-
of ICT/ ICT innovation in learning . It tion , adaptation , appropriation and
will further provide a multidimensional invention . These models focus on
framework for analysis which can what teachers and pupils actually do
locate heterogeneous indicators from when they use ICT in schools , some-
different studies and data sources . thing that the ‘ indicators’ approach
This provides a coherent structure to deals with only in a limited way ( for
guide the exploration of data and the example , a common utilisation indi-
map of complex relationships . cator is the ‘ average hours of weekly
use for teaching’) . The above models ,
when used in combination with indica-
Evaluating different tors such as those described earlier,
stages of implementing ICT may offer outcomes of more explana-
in education and levels tory power regarding the integration
of ICT in education . They may also
of evaluation offer a more solid basis for develop-
ing models and other instruments to
One of the shortcomings of many study the capacity of educational sys-
indicator approaches is that they are tems to absorb ICT- related pedagogic
measuring an ‘instance’ within a wider innovations . For example , review-
historic process , but they are never ing the above technology integration
exhaustive , and by being unavoidably phases in relation to what we defined
‘ selective’ they can create an incom- as ICT- related pedagogic innovations
plete picture of the integration of ICT in schools, one can identify the ACOT
into educational systems . A powerful model’s phases of ‘ appropriation’ and
approach to the study of the degree of ‘ invention ’ as those offering the most
integration of ICT in education makes promising potential for the diffusion of
use of such indicators within develop- pedagogic ICT-related innovations in
mental models of integration of ICT schools .
in education . Such models attempt to
describe potential successive phases A more recent effort to use indicators
through which teachers and students within a model of ICT integration in
gradually adopt and use ICT. For education was made in the context
example , for the context of school of a project carried out by Unesco’s
education , the levels of technol- Institute of Information Technology (3)
ogy implementation (LoTi) proposed in 2001. The ‘ Morel’s matrix’ (4) , which
by Moersch ( 1995) identifies seven was adopted as an instrument for
technology implementation levels in
schools : (a ) no use, (b) awareness , (c) (2) See http://www.apple.com/education/k12/
exploration , ( d) infusion , (e) integra- leadership/ acot/library.html
tion, (f) expansion and (g) refinement. (3 ) See http://www.iite.ru/
Another model , abstracted from the (4 ) This matrix was named after Prof. Raymond
Apple Classroom of Tomorrow (ACOT) Morel from Switzerland , who developed it .

75
 Chapter III Conceptual frameworks

evaluating the degree to which ICT and emphasis of domains studied may
has been integrated in an educational vary depending on which of these
system , is based on the assumption levels are taken into consideration :
that this process progresses through macro , meso and micro levels . The
four distinct successive phases : (a) macro level refers to aspects at the
emerging , (b ) applying , (c) integrat- highest level of aggregation . At this
ing and ( d ) transforming . Unesco has level , indicators would refer to global
further developed this approach to or national socioeconomic character-
help schools determine their stage istics related to the use and integra-
of progress in implementing ICT. tion of ICT in education . In a way,
Variations of the matrix have been the macro level could be seen as the
used in comparative studies of ICT specific ICT context where meso and
implementation at various levels of micro levels are situated . The meso
education ( see Unesco , 2003a) . As level refers to aspects at the institu-
with the ACOT model , the transition tional level (school , organisations ,
from one phase of ICT implementa- universities , etc. ) . The meso level
tion to another in the above matrix
refers to aspects related to an inter-
presupposes the emergence and dif- mediate level that shaped the relation-
fusion of several types of innovations . ship between micro and macro level
Pedagogic innovations are implicitly aspects . The micro level refers to the
assumed to be the driving force in that
individual ; it portrays individuals in
they are sine qua non for any other their use of ICT.
innovation to have a meaningful impact
on school teaching and learning .
These levels present different focuses
and relate to each other in that lower
A somewhat newer version of the
levels are integrated (‘belong’) into
‘ stages’ approach is exemplified in
higher levels (an individual is in a
‘ e-maturity’ models (see , for example ,
Butt and Cebulla , 2006 ; Underwood school , a school is in a region , a region
and Dillon , 2004 ; Underwood et al . , is in a country, etc. ). These three levels
2007) . Such models focus on what determine the type of indicators that we
teachers and pupils actually do when might use within each of the domains .
Some indicators at the macro level
they use ICT in schools , something
that the indicators approach deals or meso level might just be aggrega-
with only in superficial ways . When tions of micro level data . For example ,
such models are used to guide evalua- the percentage of those reporting the
tion , in combination with the indicators use of computers for instruction in a
approach , this may offer outcomes of country is the result of the aggregation
more explanatory power regarding the of individual (micro level ) teachers ’
integration of ICT in education . They answers . If we were to analyse these
may also offer a more solid basis for data at the micro level (the impact of
developing models and other instru- ICT in a specific individual/teacher, for
ments to study the capacity of educa- example), the aggregate level indica-
tional systems to absorb ICT- related tor would serve to contextualise his/
pedagogic innovations . her answers . Some indicators , on
the other hand , might be exclusively
Besides the different stages , there are of a specific level , as for example the
several levels to be considered when existence of a national policy to have
studying the effects of ICT. Indicators all school materials digitalised .

76
 Understanding and evaluating the impact

Conceptual framework that , contrary to the specific models , a

conceptual framework acts as a refer-
All in all , we can say that learning ence which is flexible and adaptable
practices and teaching for a vari- to the purpose of a study to be car-
ety of obvious reasons need to be ried out . To take an example: if we
assessed in different ways . New tools want to study if technology is having
and instruments are required to moni- a positive impact on educational per-
tor both achievements and progress formance , a framework would help us
made in the context of ICT, but there to identify the various domains in the
is no clear position yet on adequate given context to be looked at (such
indicators , instruments and scales for as ICT availability and devices used ,
measurement. A conceptual frame- pedagogies applied in which subject
work would help to alleviate this defi- areas , etc. ) and possible perspectives
cit . There is a ‘ need for a thorough , to be taken into account (school level ,
rigorous and multifaceted approach to individual level , etc.). This is important
analysing the impact of ICT on educa- for ensuring that all relevant aspects
tion and students’ learning ’ (Cox and are considered and that a systematic
Marshall , 2007; also Kikis and Kolias , approach is followed that is transparent
2005; Aviram and Talmi , 2004). Of and comprehensible for the stakehold-
interest here is that , as early as 1997 , ers involved . It provides a holistic view
Collins pointed out that ‘ research into and supports the setting of standard
the contribution of ICT to students’ orientations when defining the evalua-
thinking and acting reflects the social tion methodology and selecting appro-
and epistemological beliefs of the priate instruments for measurement.
research community’ . This has seri- In more complex evaluation settings ,
ous implications for evidence-based when conclusions are to be based on
policies . Major policy analyses that a combination of surveys conducted
encompass a wide range of settings by different research teams world-
and look for commonalities and differ- wide , it would , ideally, also contrib-
ences as a result of systemic condi- ute to a coherent common approach
tions are often missed from most of the to the identification of phenomena to
previous research agendas . Currently be analysed and their evidence- based
conducted meta-analyses on ICT interpretation in the light of a common
and attainment suggest that the most understanding of aspects to be stud-
robust evidence of ICT use in enhanc- ied . In the case of the assessment of
ing learning was from those studies ICT effects in education , this is to the
that focused on specific uses of ICT benefit of more effective valorisation
(Cox and Marshall , 2007, p. 60) . of evaluation studies carried out and
better quality of analytical work.
The purpose of a conceptual framework
should be to provide an orientation for A conceptual framework could further-
any kind of measurement required in more act as the basis for the design
the decision - making process. A frame of monitoring tools aimed at informing
work serves as the basis for modelling policy on the emerging trends , their
an appropriate assessment approach effects and their implications for cur-
and the design of methodologies and rent or future education . It is therefore
instruments . It connects to all aspects oriented towards medium- and long-
of empirical enquiry. When drafting a term policies and benchmarks defined
framework, we would therefore expect for ensuring effective integration into

77
 Chapter III Conceptual frameworks

society. A framework can facilitate the tation of instruments and data sources
construction of models to explain ICT that are further analysed and reported
effects in education , and for the adap- ( see Figure 1 ) .

Monitoring ICT impact in education for policy making

Context Resources
Population Rationale

Learning culture Socioeconomic factors

Conceptual framework Assessment Reporting

Policy goals,
priorities
Domains Analytical Analysis
levels

Indicators Assessment Evaluation

instruments

Stages Data Review and

sources reflection

Figure 1 : Overall monitoring frame

Conceptual framework for studying ICT effects

Domains Indicators Stages Policy priorities

Input
/output, performance, process e.g. European Union:
Levels 1. Improvingequityin
education and training
Macro Meso Micro 2.Promoting
efficiencyin
Examples Examples Examples Examples e.g.Morel'smatrix educationandtraining
Implementation National policiesfor eLearning IntentionsofICT 3.Makinglifelonglearning
Policies
strategies ICTimplementation strategies in school uses in courses areality
4.Keycompetences
Transforming

ICTpenetration AvailabilityofLANs Privateaccess

Resources ICT availability amongyoung people
Integrating

ineducation in school
/class toICT
Emerging

5.Modernising school
Applying

Extentofcurricula ICT-related Level ofrequiredfor education

Curriculum ICT-related courses coursesoffered
adaptation teaching
-learning 6.ModernisingVET
ICT-related services for ICT in schoolsfor UseofCMSfor Internet-
delivered 7.Modernising higher
Organisation teachers, students organ,purposes classmanagement assignments education
8.Employability
Extent of ICTuse ICT implementation Pedagogicaluse of Teacher'useof
Teaching in school education ICTin classroom ICTforteaching Indicators
:

Extent ofICT-related
eg creativity, ICTskills
Students ICT-enhanced ICT-relatedlearning
Learning activities ICTuse learninginclass activities athome

Figure 2 : Framework for evaluating ICT in education

78
 Understanding and evaluating the impact

A conceptual framework is given in is the use of content/learning man-

Figure 2 for further discussion which agement systems for educational
takes into account the political con- purposes .
•

text of European education . It covers Teaching practices : This domain

several domains relevant to specific characterises the use of ICT for
EU policy priorities . However, policy teaching activities , pedagogical
goals /priorities are presented here as practices , etc.
•

an example and could be adapted to Learning : Like the definition pro-

any other policy priority which might vided above , ‘ Learning’ focuses on
be dominant in other countries . The the use of ICT by the learner (stu-
framework is divided into domains , dent, etc. ) .
indicators and stages .
It is possible to find specific indicators
The domains identified by the concep- for each of the domains that describe
tual framework here represent the rel- the state of the domain and that vary
evant areas of study. When assessing from context to context and case to
the effects of ICT in education , such case. For example , in the domain
domains should cover the complete referring to resources , one possible
range of analytical constructs to be aspect to look at would be ‘ ICT avail-
studied in the context of the integration ability’ . As indicated above, the spe-
and use of ICT in education . Ideally, cific indicators to look at here would
each domain should be exclusive be determined partially by the level of
and not overlap with other domains . analysis ( macro , meso or micro) to be
Based on the literature review carried undertaken . As such , at macro level ,
out between 2007 and 2008 relating it would be possible to use indicators
to European projects, case studies such as ‘ broadband penetration’ , ‘ ICT
and research reports , the following six availability in the country’ or ‘ percent-
dominant blocks were identified in the age of educational software sales in
research discussions . a country’ among others . At the meso
level , indicators would be slightly dif-
• Policies: By this term we understand ferent and would refer specifically to
any type of strategies relating to the school contexts (or to another meso
implementation of ICT and their level entity that would be in focus).
effective use . This could take place In our example , possible indicators
at a national policy level as well as would include ‘the presence of LAN in
at an institutional level , such as in schools’ or ‘ the percentage of schools
universities , schools , etc. reporting having educational software’ .
• Resources : This domain refers to At the micro level , indicators would
the ICT infrastructure in terms of refer to individuals in relation to the
hardware, software, network capaci- availability of ICT, for example individ-
ties and any type of digital resources uals reporting on having educational
used for teaching and learning . software at home and uses made .
• Curriculum : By ‘ curriculum ' we under-
stand the level of ICT integration in Furthermore , the present framework
the curriculum , including courses on permits the identification of the ICT
how to use ICT effectively. maturity stages . Each of the different
•

Organisation: This term refers to indicators identified would have certain

organisational measures to imple- levels that would suggest a specific
ment ICT and its use . One example stage of ICT maturity. As such , continu

79
 Chapter III Conceptual frameworks

ing with our example , ICT resources in interventions and provide a basis for
schools might have reached a certain further decisions . The framework pre-
degree that would allow for a ‘trans- sented in this paper builds a compre-
forming ’ stage (let’s say all schools in hensive model for the analysis of ICT
a country have an adequate supply of effects into the educational process
ICT tools) . However, other indicators , from various levels and perspectives .
for example relating to curriculum , It establishes a structure for reflecting
might not be as advanced , or have on relevant indicators . The framework
no teachers trained in the pedagogi- takes into account different levels of
cal use of ICT. These latest indica- analysis allowing therefore for dif-
tors would denote an emerging state . ferentiation in scope . The framework
Under this scope , the framework pro- further introduces different stages of
vides a holistic picture of the range of implementation . This allows policy-
aspects related to ICT. makers to acquire a holistic view on
policy changes and the effects these
It is important to note that the differ- have on different actors within the
ent indicators would have a different educational system . A holistic view is
degree of aggregation depending on an essential aspect for policy evalua-
the analysis that we will want to draw tion because it can disclose the matu-
from it (see Figure 1 ) . The framework rity of the implementations of policies .
provides the pre-stage for the analysis ,
allowing stakeholders to see the rele- In brief, the paper proposes that in
vant aspects in a holistic picture before order to deepen our analysis of the
a specific analysis is carried out. As impact of ICT on education , we need
such , individual reporting of the number to shift our attention from technology
of computers at home , for example , per se to processes and skills teach-
can be aggregated at the national level ers and learners are currently apply-
to analyse country-specific patterns in ing . This will allow us to identify and
relation to use and possession , or can explore conditions and factors that
be used at the individual level to carry are shaping the way ICT is used in
out studies on the use and possession education . Under this perspective ,
of ICT by individuals in relation , for we need to shift from approaches
example, to their age . Our framework that exclusively monitor macro level
permits the review results of the analy- aspects to an integrated model where
sis in light of the ‘greater scenery of the three different levels are consid-
ICT’ within a given setting. This facili- ered in conjunction . Such a compre-
tates the consideration of aspects not hensive approach to the study of ICT
specifically accounted for in the origi- effects and their impact on education
nal level of analysis, but which might needs to be considered in a coher-
play an important role in understand- ent manner. The proposed framework
ing the results . allows for the integration of differ-
ent levels and types of data sources .
Outlook It is important to bear in mind that
there appears to be a need to reflect
Conceptual frameworks are impor- beyond pure observations and evalu-
tant tools for orienting and evaluating ate more concretely institutional con-
policy decisions . They offer policymak- texts of learning (schools , university,
ers dimensions for consideration when etc. ) , learning situations and teaching
evaluating the effectiveness of policy processes to determine under which

80
 Understanding and evaluating the impact

circumstances ICT-based activities degree of qualitative interpretation.

can enhance learning and improve It is highly recommended that the
skills. Due to the complexity involved actors engaged in the process define
in mapping factors/variables on to one the scope for evaluation and on such
another, the evaluation of the causes bases interpret the results.
of the observed impacts requires a

References
Aviram, R. and Talmi, D. (2004). The impact of ICT in education: the three
opposed paradigms, the lacking discouse. At: http://www.elearningeuropa.info/
extras/pdf/ict_impact.pdf (accessed 25.4.2009).

Balanskat, A., Blamire, R. and Kefala, S. (2006). A review of studies of ICT

impact on schools in Europe. Brussels: European Schoolnet.

Butt, S. and Cebulla, A. (2006). E-maturity and school performance: a secondary

analysis of COL evaluation data. London: National Center for Social Research.

CEO Forum on Education and Technology (1999). Professional development:

a link to better learning (Year Two Report). Washington, DC: CEO Forum on
Education and Technology.

Cox, M. J. and Marshall, G. M. (2007). ‘Effects of ICT: do we know what we

should know?’, Education and Information Technologies, Vol. 12, 59–70.

Cuban, L. (1986). Teachers and machines: the classroom use of technology

since 1920. New York: Teachers College Press.

Dwyer, D. C., Ringstaff, C. and Sandholtz, J. H. (1991). ‘Changes in teachers’

beliefs and practices in technology-rich classrooms’, Educational Leadership,
Vol. 48, No 8, 45–52.

Kollias, A. and Kikis, K. (2005). Pedagogic innovations with the use of ICT: from
wider visions and policy reforms to school culture. Barcelona: Universitat de
Barcelona.

Moersch, C. (1995). ‘Levels of technology implementation (LoTi): a framework for

measuring classroom technology use’, Learning and Leading with Technology,
Vol. 23, No 3, 40–42.

Ramboll Management (2006). E-learning Nordic 2006: impact of ICT on educa-

tion. Copenhagen: Ramboll Management.

Redecker, C. (2009). ‘Review of learning 2.0 practices: study on the impact

of Web 2.0 innovations on education and training in Europe’, JRC Scientific
and Technical Reports, EUR 23664. Retrieved 10.08.2009, from http://ftp.jrc.es/
EURdoc/JRC49108.pdf.

Rogers, E. M. (1995). Diffusion of innovations. New York: Free Press.

81
 Chapter III Conceptual frameworks

Sarason , S. ( 1990) . The predictable failure of educational reform. San Francisco ,

CA : Jossey-Bass .

Trucano , M. (2005) . Knowledge maps: ICT in education, Washington, DC:

infoDev/World Bank.

Underwood , J. and Dillon , G. (2004) . ‘ Maturity modelling : a framework for cap-

turing the effects of technology’ , Technology, Pedagogy and Education, Vol . 13,
No 2 , 213–225 .

Underwood , J. , Baguley, T. , Banyard , P. , Coyne, E. , Farrington - Flint, L. and

Selwood , I. (2007) . Impact 2007: Personalising learning with technology.
Coventry: Becta .

Unesco (2003) . Developing and using indicators of ICT use in education.

Unesco (2003a). Towards policies for integrating information and communica-

tion technologies into education .

82
ICT to improve quality in education —
A conceptual framework and indicators
in the use of information communication
technology for education (ICT4E)
Marcelo Cabrol and Eugenio Severin (1)
Inter-American Development Bank (2)

1. Introduction in ICT, its penetration into the struc-

tures of production, knowledge man-
The use of information and commu- agement, communication and culture,
nication technologies (ICT) in educa- the demand for new skills and compe-
tion is no longer optional. A substantial tencies and the loss of importance in
change in society and individuals (3) others. In addition, there has been a
has occurred thanks to development change in ways of approaching and
understanding the world and devel-
(1) We express our thanks for the revisions opment of new industries. For all
and corrections by Carla Jiménez (IADB), as these reasons, schools, countries and
well as comments and suggestions by Juan regions are compelled to develop new
Enrique Hinostroza, Claudia Peirano, María initiatives that incorporate ICT tools in
Paz Domínguez, Francesc Pedró, Friedrich
teaching and learning, so that educa-
Scheuermann, Seong Geun Bae and Michael
Trucano. tion systems can succeed in linking
(2) The Inter-American Development Bank the new demands of the knowledge
Technical Notes encompass a wide range of society with the new characteristics of
best practices, project evaluations, lessons learners (4).
learned, case studies, methodological notes
and other documents of a technical nature that
are not official documents of the bank. The Some education systems in Latin
information and opinions presented in these America have overcome the chal-
publications are entirely those of the author(s), lenge of access to education and are
and no endorsement by the Inter-American now confronting the demand for qual-
Development Bank, its Board of Executive
Directors or the countries they represent is ity improvement; some systems face
expressed or implied. significant challenges in attempting to
(3) ‘In this technological environment, computers include all children in the learning proc-
have become an integral part of our societies ess; others require more radical solu-
and our lives, transforming such diverse matters
as the way we work and relax, how businesses
operate, the conduct of scientific research, (4) ‘Economic theory describes three factors
and the ways governments govern. They are that can lead to increased productivity: capital
integrating into other technologies — in cars, deepening (that is, the use of equipment that is
phones and many other things that used to be more productive than earlier versions), higher-
‘low-tech’. There is every reason to suppose that quality labor (a more knowledgeable workforce
the pace of technological change will continue that is more productive), and technological
though we cannot say precisely in which forms innovation (the creation, distribution and use of
and directions.’ (OECD, 2008) new knowledge).’ (Kozma, 2008)

83
 Chapter III Conceptual frameworks

tions to support student learning while Inadequate assessment of the incor-

designing strategies within already poration of ICT initiatives in education
fragile institutions . In any case , prop- is in many cases a result of intuitive
erly implemented ICT projects offer and unsound development but also
an alternative for implementation and relates to the lack of specific tools
impact on student learning (5 ), espe- that would give confidence to meas-
cially with new millennium learners ure these impacts adequately sepa-
(NML) (6) . rate from a myriad of other variables
present in educational processes ,
Nevertheless , most current evalua- which are dynamically affected by the
tions have no conclusive information introduction of ICT.
to inform decision - makers on how
ICT can improve the quality of edu- It is very likely that this lack of instru-
cation ( 7) . The lack of clarity about ments is a natural consequence of the
different options and impact areas of emerging development in this process .
ICT use in education is an obstacle Considering that the personal com-
for the development of successful puter has been in existence for only
projects . 30 years , and that the first computers
that came to some schools did so only
about 20 years ago , it is only logical
that we still have many unanswered
questions about how ICT can achieve
the best contribution towards improv-
(5) ‘ All the studies reviewed have identified
ing the quality of education .
a range of important wider benefits of ICT on
learning . These include the positive impact of In fact , recent literature has drawn
ICT on student motivation and skills, independent attention to the infovation phenom-
learning and teamwork. Increased motivation enon in educational practices incor-
leads to more attention during lessons which can
be exploited by the teacher. Aspects for more porating ICT, with the caveat that so
individualised learning were described in a variety far the greatest amount of experience
of ways . Students learn more independently, at has been limited to 'computerisation'
their own pace and according to their needs . of processes and practices , which
They also take more responsibility for their
continues to repeat the same actions
own learning process . As seen , ICT can benefit
likewise academically strong and weak students of the past, but now with the support
and students with special needs.’ ( Balanskat et of computers and other technological
al . , 2006) devices . The predictable consequence
(6) ‘ In times when a lot of emphasis is put on is that impact on results will be quite
the effectiveness of teaching , more attention
limited (8) .
should be devoted to the changes occurring in
pupils as they increasingly become NML. Their
emergence claims for a reconsideration of ICT- The use of ICT in the context of dis-
based educational innovations putting pupils’ ruptive innovation and comprehen-
new attitudes and expectations , as well as sive intervention regarding the above
transformed competences at the very centre.’
(Pedró , 2006)
(7) The exercise to establish a ‘knowledge map’ (8) ‘ Many of today's schools are not teaching the
developed by the World Bank infoDev ( Michael deep knowledge that underlies innovative activity.
Trucano , 2005) showed how, beyond the large But it is not just a matter of asking teachers to
investments made in many countries to use teach different curriculum , because the structural
ICT in education systems , data to support the configurations of the standard model make it very
affirmation of its role in improving education are hard to create learning environments that result
limited and debatable . in deeper understanding .' (Sawyer, 2008).

84
 Improving quality in education

practices is probably a better progno- the focus ( 11 ) . Lastly, the project offer-
sis for changing results . Nevertheless , ing is so vast that there is no common
even less data , assessments in stud- framework that can be both flex-
ies are available on this issue (9) . ible and broad enough to include the
diverse nature , contexts and different
This document presents a general stages of projects .
conceptual framework to support the
design , implementation , monitor- The main hypothesis of the framework
ing and evaluation of projects where is that the goal of all education projects
information and communication tech- is to improve student learning , regard-
nologies have been incorporated to less of whether they are children or
improve education quality. adults . The goal expected and meas-
ured in these projects should then be
One of the main challenges in the use impact(s ) on learning and changes
of ICT in education is the lack of indica- brought about by implementation and
tors that offer clear criteria and objec- enabling such learning .
tive information to allow policymak-
ers to make the proper decisions (10) . Learning outcomes can be broadened
Projects have not always considered by putting children at the centre of the
rigorous evaluation processes and learning process . It is necessary to
in those instances where they have , consider improvements in students'
ICT impact on learning has not been involvement in and commitment to
learning as the initial result. This plays
a direct role in curricular learning

(11 ) See , for example, the conclusion of the

World Bank evaluation of an ICT programme
in Colombia : ‘The main reason for these (poor)
results seems to be the failure to incorporate
the computers into the educational process .
Although the programme increased the number
of computers in the treatment schools and
(9 ) ‘Schools should use computers and related provided training to the teachers on how to use
technologies to help students who are poorly the computers in their classrooms , surveys of
served , or not served at all , by the current both teachers and students suggest that teachers
technology of education that is, by the schools did not incorporate the computers into their
most of us grew up with . In addition , elementary curriculum’ ( The use and misuse of computers
and secondary students ought to use computers, in education evidence from a randomized
the Internet and other digital tools directly, experiment in Colombia, Barrera-Osorio and
not necessarily through a school . In these Linden , 2009) . Also in the Enciclomedia Project
ways , schools , students and families will help in México: ‘ no significant differences were found
promising computer-based technologies grow in the knowledge skills , implementation and
and improve. The schools can pay a huge price evaluation of content among children who used
for not changing in time to accommodate new Enciclomedia and those who did not have such
technologies .’ ( Christensen et al . , 2008) equipment. Even children from 6th grade who
(10 ) The World Summit on the Information Society did not use that technology had a better result by
(WSIS) concluded that: ‘ We must develop a reaching 1.48 over 1.23 points over those who
realistic plan for evaluating results and setting did have such a tool , Whereas in the application
benchmarks ( both qualitative and quantitative) of content learned , those first gained 2.15 points
at the international level , through comparable to 2.11 for those who did have this tool . Those
statistical indicators and research findings to 5th grade students without Enciclomedia were
monitor the implementation of the action plan best evaluated with 1.83 points on 2 of their
goals and objectives , taking into account national classmates with this equipment’ , Libro Blanco
circumstances . ’ (WSIS 2005) Enciclomedia, ILCE, 2007 .

85
 Chapter III Conceptual frameworks

improvement and can be observed involved , can be affected by the devel-

in the participation and continuance opment of the project (13 ) .
of students in the learning process ,
and with improvement in teaching Application of this framework and
practices and learning processes indicators at different levels of edu-
as well . It also takes into account cation systems (national or subna-
that these changes in practices and tional) aims to provide a holistic and
improvements are directly linked to the integrated vision of ICT incorporation
impact and the development of either in order to support decision -making
general skills or ' 21st century skills ' , regarding actions that can or should
including an understanding of ICT skill be made based on the available
acquisition ( 12) . information , taking into considera-
tion all necessary areas or domains
The monitoring and evaluation proc- (inputs) .
ess should be considered more care-
fully and rigorously as a substantial At the specific project level, use of
component of each project, much diverse quantitative and qualitative
more than it has been thus far, to methodologies for data collection and
account for such impacts . Monitoring observation will provide a set of indica-
tors . This evaluation will allow meas-
and evaluation processes must be
incorporated as an integral part of the urement of the project's efficiency
process itself. Review of key informa- and monitoring of its development
tion before (baseline), during the proc- by those carrying out the project and
ess (monitoring) and at the end of the other stakeholders , making it easier to
project (final evaluation) is fundamen- determine best practices and promote
tal to the proposed framework. The the development of new initiatives for
use of ICT in education areas .
use of indicators to measure the sys-
tem’s level of development and matu-
ration will be an indispensable tool This framework has been developed
for making policy decisions based on taking into account empirical informa-
solid data and targeted knowledge . tion available from past Inter-American
Development Bank experience and
The proposed framework identifies from other experts in ICT education
five domains (inputs) that should be project implementation .
considered in an education system or
in each specific project, its planning Considering that every ' ICT in educa-
tion' project implements different lines
processes and products , and those
of action , the framework is broad in
processes that, though not directly
nature , allowing different variables to
be reviewed and selected (like a road-
(12) In the case of students from low-income
families , the flexibility of schools is even smaller.
Wealthier schools attract the best teachers, (13) Since computer availability alone will not
leaving the least prepared teachers to schools have an impact, policymakers and project leaders
in poor and remote areas. [...] Consequently, should think in terms of combinations of input
these systems perpetuate social inequalities , factors that can work together to influence learning .
lose excellent students as victims of boredom , Coordinating the introduction of computers with
increase the cost of education through the high national policies and programmes related to
dropout and repetition rates, and pass the cost changes in curriculum, pedagogy, assessment
of training their graduates to employers or other and teacher training is more likely to result in
systems . (Haddad and Drexler, 2002) widespread use and impact. (Kozma, 2005)

86
 Improving quality in education

map) depending on direct or indirect from one ICT education project to

involvement in the project and how be compared to other projects (ICT-
they could be affected by it. related or not) in order to evaluate the
efficiency of the investment.
Regardless of the variables and com-
ponents included in the project, the This document should be considered
goal (and objectives) should be linked a working paper within the conceptual
to the improvement of learning and framework, which will be improved
its implementation should take into through the development of new
account monitoring and evaluation projects and continually updated due to
mechanisms linked to the objectives. the constantly changing nature of ICT
A good evaluation will allow results education processes and products.

87
88
framewo al
Concepturk
2.

Definition
projects
(I
education
in
iThe CT4E
ICT
of
evaluation
and
onitoring
mplementation
,mconceptual
design
the
for
framework
)is
.framework
table
following
the
in
presented

Final
:student
goal
learning

Development
Inputs products
and
Processes stages Impact
Physical
Infrastructure specs
tech
and
layout
ICT Intermediate Final
ICT Implementation
process
Connectivity Access
use
and 3.
achievement
Student
scores
Test
Resources curriculum
ICT development
Curriculum assessment
()Curriculum
Chapter III - Conceptual frameworks

Contents Learning
organization
Tools Resources
availability practices
in
Change
1.
.systems
Inf Systems
use Pedagogical
practices
Students
practices
Training Curriculum Teachers
performance Emerging
(innovation
)practices
skills
ICT ICT
experience
E
for
ICT training
ICT 2.
involvement
Student
Enrollment
Support Pedagogy Help
desk Promotion
Technical Education
support
system Applying Retention

Emerging
Integrating
Transforming

Attendance
Management
Administration School
organization Attitudes
dissemination
Information Management
systems Expectations 3.
skills
Student
Incentives Incentives
plans thinking
Critical
Regulations Legal
framework Problem
resolution
Community
involvement Community
expectations
and
attitudes innovation
and
Creativity
Communication
Political
Sustainability visibility
and
Priority Collaboration
Financial Budget ICT

Baseline Monitoring Final

evaluation
 Improving quality in education

As shown in the table, the framework change that can be demonstrated in

includes the following elements. students learning.
• Student learning, as the main
goal of all project implementation. 2. Impact
Students must be considered the
direct beneficiaries of any ICT4E
Results
initiative, regardless of whether they 1. Practices
are children or adults.
The use of ICT in education implies
• The Inputs refer not only to project
the reasonable expectation that modi-
lines of action but also to fac-
fications in teaching methodologies
tors that could be affected by its
and student learning processes will
implementation.
• The Processes and products occur (14).
are those elements that will be
ICT offers a unique opportunity for
modified by the project and should
access and knowledge construction.
demonstrate the results of the
implementation. In order to achieve effective, compre-
• The projects’ Impact and the con- hensive use of ICT in education devel-
ditions that allow such outcomes opment of new learning practices,
are measured broadly with different strategies and methodologies must
variables. be put into place (15). A review of the
• Development stages: four stages literature indicates that, in instances
are described which will impact the where ICT has been incorporated
design, implementation and evalua- as an additional tool to ‘maintain the
tion of the projects. status quo,’ educational impacts are
• The process of Monitoring and scant or non-existent.
evaluation includes different
sources of data and information. This is an important field for innovation,
where ICT4E plays an important
The
workelements
are described
included
below.
in the frame- catalysing role. The link between

(14) ‘When learning scientists first went into

1. Student learning classrooms (Sawyer, 2006), they discovered
that most schools were not teaching the deep
Student learning is the purpose and knowledge that underlies knowledge work. By
main goal of an education system’s the 1980s, cognitive scientists had discovered
that children retain material better, and are able
actions and must remain so regarding to generalise it to a broader range of contexts,
use of ICT in educational processes. when they learn deep knowledge rather than
surface knowledge, and when they learn how to
In each specific project, students are use it in real-world social and practical settings.
direct beneficiaries, so the expected Thus, learning scientists began to argue that
standard model schools were not aligned with the
results should be directly linked to the knowledge economy.’ (Benavides et al., 2008)
learning that the project explicitly aims (15) ‘Measuring changes in learning and
to impact or which will be indirectly teaching processes is a time-consuming task,
impacted by the project’s action. but one which may yield valuable results.
Knowing how educational technology changes
The project’s impact (positive, nega- teaching practices, as well as the ways in which
students learn, is fundamental for evaluating its
tive or no change) and its effective- effectiveness and for developing better tools.’
ness will depend on evidence of (Balanskat et al., 2006)

89
 Chapter III Conceptual frameworks

teaching and learning practices Impact

and the growing daily interaction of
students with digital , multimedia and 3. Student achievement
interactive environments make this A country’s education curriculum
a key aspect of the framework and determines the knowledge and skills
an important element to connecting that students should achieve for each
projects with expected results (16) . grade as well as tasks required of
teachers and schools . The first area
2. Student involvement where impact is evident in ICT4E
One of the fundamental components projects is in learning associated with
a specific school subject or topic, or
of educational processes is student how the curriculum content is divided
commitment. Although it may be obvi-
according to learning aims or expected
ous, the motivation and ongoing par-
competencies for each student.
ticipation of students are necessary
for project success . Furthermore ,
student motivation and enthusiasm in Typically, this impact has been evalu-
ated in subjects such as language ,
activities have a positive impact, not
mathematics and science , since these
only with respect to potential learn-
are the subjects evaluated in most
ing results and development of new
standardised tests (focus groups or by
competencies but also to the learning
census ) and, therefore, data are avail-
environment, in stakeholder expecta-
able in many countries (e.g. standard-
tions and results for student promotion
ised tests such as TIMMS and PISA) .
from one level to another. These proc- Even though these instruments have
esses also generate change regard- had a small , limited field of meas-
ing the motivation and expectations of
urement to date (limited to only cer-
parents and teachers . Both are inter-
tain skills and content) , studies have
twined with student motivation and
revealed positive but moderate corre-
expectations , resulting in the ongoing lations between ICT projects and test
development of learning , results.

Data on attendance , repetition , pro-

motion and drop-out rates are usually There are some challenges in coun-
tries that do not have national tests or
available and facilitate the performance
participate in international standard-
of straightforward impact analysis .
ised tests . In these cases the project
Measuring motivation requires other
could develop ad -hoc standardised
instruments which , when applied cor-
tests to be administered before , during
rectly, can yield important information
and after the project implementation
about the effects of ICT4E projects .
(baseline and evaluation ) or among
groups that do or do not participate in
(16) ' One of the fundamental lessons to be learnt the project (control and comparison
from European , North American and Australian
experiences over the last 20 years has been groups).
that those responsible for helping people learn
must be confident in the appropriate use of new A lack of rigorous studies in this
technologies if the process is to be successful . area has made it harder to prove
Introducing new technologies into places of the reasonable expectation that a
learning should involve a fundamental shift
whereby the role of teachers becomes less country’s investment in ICT projects
didactic and more that of facilitating individual can improve learning in different
learning processes .' ( Unwin , 2005) subjects . Therefore it remains to be

90
 Improving quality in education

seen whether this impact is significant, thinking and problem solving; creativ-
and , if so , on what subjects. More ity and innovation; and communication
important yet is the lack of clarity as and collaboration . Development of ICT
to what impacts can be reasonably competences is also considered .
expected in projects according to their
stage of development or maturity. Until now, evaluation has not been
particularly exact and has been mostly
This task is especially complex conducted through qualitative studies,
because the introduction of ICT into
interviews and perception surveys that
education processes is often accom- collect information/data on the vision
panied by modifications in teaching of students , or through structured
methodologies . In fact, this is what is observation exercises . Nevertheless,
intended ; with the introduction of ICT,
more objective tools will be developed
old methodologies could have little or over time that will allow for more rigor-
no impact . ous evaluation exercises .

Evidently both people and govern-

One of the components of the OECD
ments reasonably expect that use of
new millennium learners project is
ICT in education (usually a complex
developing ICT competencies for a
and expensive process) will improve
student learning , and this needs to be
working definition framework and
tools for evaluation . Another initiative
proven empirically.
working towards similar objectives is
the alliance supported by CISCO , Intel
4. Skills and competences and Microsoft and a group of univer-
It is fairly common to point out that ICT sities and international institutions :
use in education has an impact on the ‘Transforming education : assessing
development of new skills and com- and teaching 21st century skills’ .
petencies in students. These compe-
tencies have often been described as Information and communications
‘21st century skills’ due to their impor- technologies are instruments that are
tance in a knowledge society age ( 17) . a regular part of a range of work and
development opportunities . Even a
There is extensive literature describing basic understanding of ICT use can
these competencies and it is therefore result in opportunities for access and
easy to consolidate a group of gen- growth , both personally and profes-
eral competencies required by stu-
sionally, which can make the difference
dents that will eventually develop fully in a country’s overall development.
with the use of ICT. They have been
grouped into three major areas : critical
ICT skills and competencies are a
clear objective in any project involving
(17) 'To participate in this global economy and the use of ICT in education ; therefore it
to improve their standard of living , students will
need to leave school with a deeper understanding
is necessary to evaluate the effective-
of school subjects , particularly science , ness of each project. To perform these
mathematics and technology. They will need tasks , standardised tests will be used
skills necessary to respond to an unbounded but alongside IDB’s own validated test
uncertain 21st century to apply their knowledge
to evaluate student ICT skills before ,
to real -world situations, to think critically, to
collaborate, to communicate , to solve problems, during and after implementation of
to create and continue to learn .' (Kozma, 2008) activities in primary education .

91
 Chapter III Conceptual frameworks

3. Development stages stage of the project (emerging , apply-

ing , integrating and transforming) and
Clearly, the type of projects to inform the expected outcome with
develop and evaluate (as well as the results indicators .
impacts expected) will depend on the
respective stage of development in For example , you can generically
the use of ICT in and the educational
describe these steps for each domain
context where each project will be considered in the general framework ,
applied ( 18) . in the table .

The development stage reached The table operates in practice as

through incorporation of ICT into
a section for reading the indicators
education systems is strongly cor- present in a system or project, which
related with the type and depth of allows for ascertaining maturity or
potential changes in application con- stage of development.
texts . Thus , the intensity of use and
the impact increase to the extent that
efforts toward incorporation are sus- Once this section has been applied
tained over time . to each system or project, reading
it may provide criteria for decision-
making regarding the domains regis-
Following Morel’s Matrix (2001 ) , four
tering greater or less progress and ,
project phases are proposed which
therefore , the kind of priorities that
are vital in the project’s design , imple-
could lead the development of new
mentation , follow- up and evaluation
actions .
steps , and in the follow- up of compara-
ble education systems . Therefore , by
analysing the indicators described in Definition of development stages is
the ‘ Processes and products’ column , directly related to reasonable expec-
you can determine the development tations for the impact that ICT has on
educational systems , particularly with
(18) 'Countries which are presented in the respect to learning , skills and student
initial stages of ICT incorporation in education competences . It is therefore possible
have different assessment needs than those
to enter into the table below some
who already have a long tradition of use. For
example , initially it is important that teachers examples of the kind of results that
and students have access to software and can be found in education systems
hardware and that they have acquired basic or in project target groups . Analysis
skills in computer science . Countries which
of indicators will therefore depend on
are at the initial stages of ICT incorporation
in education have different assessment needs each stage of development .
than those who already have a long tradition
of ICT use . For example , initially it is important Until now, limited and partial invest-
that teachers and students have access to
ments in ICT (implying very small
software and hardware and that they have
acquired basic skills in computer science. In changes in inputs) were rarely
the case of countries at more advanced stages, expected to involve changes that
other considerations such as management can quickly translate into new and
of educational innovations , changes in improved skills and competences in
educational curricula and other organizational
changes in schools , and ongoing support students . Applying this framework
and training for staff are more important" . has allowed us to recognise that the
(Manual for the production of statistics on the achievement of significant impacts is
information economy, UNCTAD , 2008) the result of a development process

92
 Improving quality in education

that requires a broad vision, compre- tation of curriculum content in ICT or

hensive, integrated implementation other subjects (in the use of ICT).
and development time in order to
exhibit genuine impact. b. Content: Digital or analog material
aimed at teaching and learning with
technology tools, e.g. encyclopedias,
4. Domains or inputs
manuals, textbooks, books, guides,
Domains or inputs considered in videos and hypertext.
project design and evaluation include
the following: c. Tools: Software development or
support initiatives for development of
1. Infrastructure teaching and learning processes; e.g.
a. Physical: Initiatives associated productivity applications, virtual simu-
with provision of infrastructure nec- lators and modeling.
essary for the use of and access to
d. Information systems: Aimed at
ICT, e.g. laboratories, libraries and
furniture. supporting implementation and distri-
bution of management and education
b. Equipment: Equipment planned information systems at the school,
for the project or considered part of country and regional levels, as well as
the project (even if not conceived as those that allow monitoring of educa-
a direct part of the project) includes tional projects and their stakeholders,
computers, printers, projectors and the including curriculum, pedagogies and
conditions included in the purchase possible models of use (19).
and use of those items, e.g. guarantee
and service support.
3. Human resources
c. Connectivity: Access to Internet a. Teacher training: Initial and in-
and networks that allow their use service training associated with the
for education purposes; bandwidth adoption, adaptation and updating of
access, connection stability and tech- curriculum and practices for the inte-
nologies that facilitate better online gration of ICT into education.
traffic and provide privacy protection
filters for content accessed by stu- b. ICT competences: Training activi-
dents. Implementation of a reliable ties for the acquisition and/or certifi-
local network structure that is safe and cation of specific ICT skills, general
accessible. education, and productivity and com-
munication tools.
d. Support: Activities aimed at admin-
istration, maintenance and repair of (19) ‘Clearly, compared to the traditional structure
equipment as well as problem-solving of the Internet, with few transmitters and many
related to project activities and techni- receivers, a new platform begins to be adopted
cal support for users. where web applications are easy to use and
allow for many transmitters, many receivers and
a significantly higher information exchange rate.
2. Contents Some of the most common resources are having
an impact in teaching models based on online
a. ICT curriculum: Initiatives linked technologies such as blogs, wikis and others.’
to the implementation and/or adap- (Cobo Romany and Kuklinski, 2007)

93
94
Emergence Application Integration Transformation

Infrastructure Isolated
administrative
for
PCs ,
laboratories
Computer Computer
in
networks for
available
platforms
Diverse
processes
,restricted
to
access .
access
Internet
broadband classrooms
and
laboratories ,
learning
and
communication
Chapter III

and
computers
students
for administrator
or
Educator with
combination
in
used communication
-based
web
educators
. prepared
technical
provide
to |other
(c
,devices
ameras ,
services
collaboration
and
support
. scanners
,etc.
).C ontinuous -managed
self
learning
for
computers
to
access systems
.Local
highly
staff
students
.
educators
and specialised
and
support
in
Lstaff
networks
.Wireless
development
solutions
ocal
.
support
in
specialised

Contents not
does
Curriculum Curriculum
|Curriculum
account
into
takes
contemplates comprehensively
Curriculum
exclusively
account
into
take the
ICT
of
development
basic .
ICT
of
use
inclusive
all ICT
of
use
the
incorporates
the
.Office
ICT
of
use competencies
.Educational Educational
and
contents bas
-akuilding
nowledge
Conceptual frameworks

automation
educational
and portals
digital
to
access
with applications
and
enriched options
.Advanced
strategy
games
.CDs
applications resources
the
support
that .
practices
specific
to
adapted of
development
the
for
or
with
software
local curriculum
.E-mail
web
and Basic
content
for
applications collaboration
and
content
content
educational search
.
available
services reconstruction
and
creation
of stakeholders
diverse
.among
)e.g.
pedagogy
eacher-
entred
eacher
(.-c|Tencyclopedias .
objects
learning
and
teaching experimentation
for
Platforms
centred
.
pedagogy Collaborative
-c,sentred
tudent .
resources
of
publication
and
.
pedagogy pedagogy
-centred
:Student
-tollaborative
hinking
c,critical
experiential
.

Human
resources Training
to
according General
ICTs
in
training sInitial
- ervice
in
and Peer
,
networks
learning
individual
.No
interests through
-service
in
teacher associated
with
training -managed
self
continuing
pedagogical
the
for
support and
with
curriculum
the
||
local
training
.No
programmes education
systems
.Peer
integration
.
ICT
of ICT
pedagogical
for
support educational
in
ICT
for
uses online
and
networks
.
integration .Tofraining
classroom
the .
collaboration
local
the
in
support
for
staff
pedagogical
of
integration
.
ICT
 Administration Pragmatic
on
based
view Practical
on
based
view aiming
view
Holistic ,innovative
Proactive
individual
.No
interests the
new
of
adoption processes
integrate
to view
generate
to
aiming
for
the
pedagogical
support technologies
.Information by
incorporating developments
allow
that
integration
.
ICT
of technology
.Complex
technologies
of
,administration ,better
new
for
systems
some
,but
systems
are
they information
interconnected ,recording
information
and
not
.Isolated
,interconnected systems
for
technology communication
.C ommunity
partial
the
of
involvement recording
-critical
system and
solutions
seeking
actively
organised
.
community .
communication
and collaborative
the
in
engaged
incorporation
Regular building
.
knowledge
shared
of
into
community
the
of
formal
and
processes
.
communications

Policies Causistic
experimental
and ICTs
of
development
Limited broad
Development
,of educational
Development
of
development
ICT
isolated
of centralised
plans
,on
based policies
comprehensive
ICT plans
take
that
policies
and
initiatives
W
. ithout
policies concentrated
.
decisions covering
domains
of
set
the holistically
account
into
ICT
over
allocated
budgets
or Partial
,generic
that
policies with
,
levels
depth
similar strategies
their
with
together
.There
term
long
the
no some
account
into
take allowing
for
areas
flexible ,aand
components
llowing
legal
the
to
adjustments components
depth
various
at -dependent
context
specific specific
their
for
areas
broad
,nor
framework
specific
are .Sbudgets
levels
-thort
erm .Merm
adaptations
-t edium .
context
into
inclusion
incentives
.
considered
being specific
with
a( ssociated budgets
.Legal
guaranteed over
the
Inclusive
budgets
projects
).Indirect
generic adjustments
facilitating long
.Legal
term
framework
adjustments
legal
the
to incorporation
their
and
ICT
of new
to
adapted
completely
framework use
.Incentive
education
in requirements
.Incentives
and
t
( elecommunications systems
into
integrated associated
system's
the
with
education
).Pilot
plans predefined
educational overall
.
achievements
learning
specific
programmes
for achievements
.
incentives
.
Improving quality in education

95
96
Emergence Application Integration Transformation

Practices Predominance
,
vertical
of classes
that
Teacher
-centred t entred
classes
c
;- he
Student Lifelong
learning
expository
.Classes
classes the
incorporate
sporadically of
role
the
assumes
teacher environment
;teachers
Chapter III

centred
and
teacher
the
on use
school
some
into
ICT
of , ctively
tutor
and
apresenter and
continually
students
/her
.Ihis
knowledge
as
CT activity
its
with
beginning accompanying
and
proposing creation
the
in
collaborate
content
training
specific regular
.
planning
curricular use
who
students
of
work
the of
communication
and
.Students
students
the
for access
regular
have
Students their
in
collaboratively
ICT knowledge
.Emphasis
accessing
difficulties
have to
,but
technologies
seldom .This
work
rather
is
use
school on
the
and
investigation
.technologies
use
for connect
their
with
them of
context
the
in
intensive ,
projects
of
development
experience
school
. low
substantially
but
school with
increasing
the
proposed
the
and
it
of
outside and
actor
each
of
autonomy
.
activities abundant
platforms
of
use
and
communication
for
Conceptual frameworks

collaboration
.

Student
involvement Passive
students
of
attitude Passive
students
of
attitude Active
students
the
of
attitude ,autonomous
Proactive
regarding
.L
learning
ow regarding
.Moderate
learning learning
.High
regarding .
attitude
life
entire
throughout
or
expectations
moderate regarding
the
expectations expectations
regarding High
regarding
expectations
regarding
of
impact
the impact
their
on
school
of their
personal
and
learning their
that
role
the
and
future
studies
the
in
lives
their
on lives
generate
future
the
in ,though
achievements
not education
it.
in
plays
.
future motivations
.
school
of
outside their
to
connected
explicitly
.
experience
school

Learning
results None impact
Low Medium
impact High
impact

competencies
and
Skills
None impact
Low Medium
impact High
impact
 Improving quality in education

c. Use of ICT for education : stood as the ownership level with the
Training initiatives for the specific success and objectives of those lead-
use of ICT in educational ing the project) .
contexts (20) .
b . Budget : Long-term budget needed
d. Pedagogical support: Efforts to for operational continuity and devel-
provide educational support and follow opment of complementary initiatives
up for participants , guidance or tutor- required for the project’s success .
ing service developed for implementa-
tion of proposed activities. c . Legal framework: Actions to adjust
and adapt the rules and regulations to
4. Management enhance and improve the impact of
the initiative and minimise the risks .
a. Administration : Structures and
Includes measures to improve the
strategies for system and project
safety and security of minors , regula-
management and administration for all tions associated with industries and
levels considered (school , province,
copyright protection
country and region) as well as the rela-
tionship with other institutional stake-
d. Incentives: Plans and programmes
holders associated with the project
designed to (positively or negatively)
e.g. strategic allies and donors .
underscore beneficiary commitment
b . Information dissemination : and the results of the project expected
by its participants .
Activities aimed at providing informa-
tion about project results , strategies
and actions and involving all potential 5. Processes and products
interested stakeholders and benefici-
aries of the project. Processes and products being pro-
posed to allow the framework to sup-
c. Community involvement : How port the design , implementation and
scope , strategies and actions are monitoring of specific projects devel-
communicated . How all actors con- oped to incorporate the use of ICTs for
cerned and potentially affected by the educational purposes .
project’s development are involved .
Actions that promote (and allow for) For example , listed below are some of
the active participation of community the products and processes that may
members and families in the develop- typically be considered as part of these
ment (and as direct beneficiaries) of projects and whose observation and
the project . monitoring will reveal how each contrib-
utes to achieving the expected results .
5. Policy
1. Infrastructure
a. Planning : The project’s prior-
ity (short or long term) in the context a. Amenities : Specific references
of other initiatives , plans , projects or about the technical characteristics
actions , including visibility ( under- of the equipment. The relationship
between product characteristics and
(20) Particularly important here is Unesco's work specific reasons why the equipment
in the development of the use of ICT in education was selected ; distribution and the final
and its standards for teachers . characteristics of the equipment as it

97
 Chapter III — Conceptual frameworks

is implemented. Also included here curriculum, pedagogical approach(es)

is the connection with other existing at the institutional level as well as
equipment indirectly related to the knowledge management strategies.
success of the project. Characteristics
and conditions of connectivity. c. Availability of resources: Levels
b. Implementation process: Des- of access to educational resources
from direct and indirect beneficiaries;
cription of project logistics, location and
equipment distribution. Additionally, whenever possible underscore rele-
specifics on the procedure for equip- vance and importance with respect to
ment selection, purchasing, distribu- project objectives.
tion and integration/implementation in
projects. Also included are references d. Access and use: The opportu-
to the investment made in the context nity for and simplicity of access to the
of the project essential to its success, information and management systems
such as classrooms or buildings (even by the beneficiaries (direct or indirect),
when they have not been a project- whenever possible, provide their rel-
specific component), as well as calen-
evance to and the quality of the pro-
dars and systems in use by ICT users
and their availability. posed objectives.

c. Helpdesk: Describes systems 3. Human resources

installed to lend support to indirect
a. Teacher performance: Describes
and direct project beneficiaries in
the event of technical and pedagogi- teacher background information perti-
cal difficulties. It will provide the user nent to the project: e.g., performance,
rate, response time, mechanism used, planning activities, student:teacher
most common difficulties, the best- ratio, performance evaluation and
rated responses and other indica- incentives.
tors describing support available to
participants. b. ICT experience: Previous experi-
ence with ICT in educational use, both
2. Resources in and outside the classroom.
a. Curriculum development: Work
developed to connect curriculum to the c. Models for educational use:
learning goals and project objectives Characteristics of ICT training to stake-
associated with ICT4E. Inclusion of ICT holders in order to capitalise on the
in the curriculum at the different levels use of ICT in educational contexts.
as a competency or as cross-cutting or
vertical content, learning goals specifi- d. Education support system:
cally proposed by the stakeholders. Mechanisms aimed at motivating and
lending support to the work of different
b. Learning organisation: Des-
stakeholders involved in the project,
cription of how learning activities are
structured and organised, including such as tutoring or assistantships for
how the curriculum is developed (inte- teachers, personal or online support
grated or separated from other the- plans, training resources, mutual com-
matic areas), how often and at what munication among peers and guides
time of day ICT is integrated into the for families.

98
 Improving quality in education

4. Management ations. Any difficulties with the proce-

a. School organisation: The way the dure and future financing plans should
project is integrated into the overall be described. The expenses entailed
institutional scheme of the school, how by the project should be noted, speci-
many hours each teacher spends on it fying one-time purchases as well as
and systems aimed at organising and recurring purchases that will therefore
supervising the project’s operation. be part of the project in the future.
Mechanisms recommended to secure
b. Management systems: Insti- funding in the future. For long-term
tutional framework, systems and implementation, the project’s strengths
mechanisms implemented by the and weaknesses and how the project
itself plans to address them. This will
project, or that the project modifies and
include the total ownership cost as
impacts and which allow for follow-up
of project activities and objectives. proposed by GESCI (21).

c. Systems use: Opportunity for and c. Priority and visibility: The position
of those responsible for the project as
simplicity of access to the information
well as project objectives and the pro-
and management systems by the ben-
motion of such activities.
eficiaries (direct or indirect), whenever
possible stating relevance to and qual- d. Legal framework: Description of
ity of the proposed objectives.
regulations associated with project
d. Community attitudes and expec- implementation.
tations: Actions involved in the
e. Incentives plans: Programme or
project’s implementation aiming to
incentive plans associated with the
include the initiative in its develop-
project’s beneficiaries and objectives.
ment context, introduction of partici-
pants (direct or indirect) to the project,
communication with those involved in 6. Evaluation
the project who facilitate the project’s
The Conceptual Framework is not
implementation. Also describe how
proposed as an evaluation model, nor
the project considers the impact on
does it develop specific assessment
the community, particularly regarding
instruments. It should work as a guide
students’ families.
to consider the elements involved in
ICT for education projects. The evalu-
5. Sustainability ators using the Conceptual Framework
a. National (subnational) plans: should then apply and develop the
Displays the existence or lack of adequate evaluation models and instru-
national plans that comprehensively ments depending on the context.
maintain and describe the use of ICT
in education systems, linking them to 1. Baseline
each other and to the rest of the goals The data that inform the processes and
and policies, and to the development products before the project’s imple-
strategies as well. mentation and by which the project
impact can be measured.
b. Budget: Different budget sources
and procedures that are directly or (21) Global e-school and communities initiative
indirectly involved in the project’s oper- http://www.gesci.org/

99
 Chapter III — Conceptual frameworks

The baseline is concerned with data of pertinent information for monitoring

that allow for identification of indicator and decision-making.
status at the system level upon start-
ing the application or before project 3. Evaluation
implementation. From these initial This process involves comprehensive
data (sum zero), system progress or review of a project, its achievements,
project action impact will be meas- progress and difficulties, and estab-
ured, once they are implemented. lishes its impact vis-à-vis proposed
The baseline should take into con- objectives. Evaluation is conducted
sideration the systems level, a broad at project completion or at the end of
set of indicators that facilitates pre- a given phase of the project’s imple-
cise analysis of ICT incorporation mentation, and its purpose is to meas-
status. At the project level, you should ure actions and the strategy proposed
select those indicators that explicitly against the results obtained, and to
impact the project’s objectives, includ- monitor its relationship with and impact
ing those linked to student learning. on system indicators.
Wherever possible, however, the data
for all processes should be taken into Along these lines, impact made on all
consideration to facilitate documenta- areas, processes and products must
tion of unforeseen impacts. be taken into consideration and not
only the ones where the project has
2. Follow-up and monitoring implemented actions.
When applied at the system level, Evaluation is a process that is crucial
steady action is required that may be to every project and should be consid-
implemented to ascertain changes ered an essential component at the
occurring due to various actions aimed outset of project design. Whenever
at incorporating ICT into education
possible, efforts should be made to
systems. Periodic application (annual,
have evaluation conducted by an
biannual or as frequently as possible)
external entity unassociated with the
aims to shed light on the decision-
project’s direct or indirect executors,
making of policymakers.
to achieve objectivity and impartial-
At the project level, relevant data ity. Whenever possible, experimen-
design in the intermediate steps of the tal evaluation methods should be
project’s implementation will inform favoured to complement other data
progress and steer the project toward sources to produce more solid, reli-
its proposed objectives, allowing for able results.
early problem detection and correction
in learning.
3. Indicators for ICT
Technically precise periods (e.g. in education
monthly, quarterly, biannually) may
be formally set for development of the Application of the conceptual frame-
monitoring phase, according to spe- work on a set of indicators is proposed
cific project characteristics, but tech- as an exercise to facilitate a compre-
nological models may also be estab- hensive view at the system level (sub-
lished enabling permanent feedback national, national, regional or global)
to project administrators in the form and at the project level as well.

100
 Improving quality in education

Regarding use in monitoring systems , proposes to develop ; consequently, it

we propose creating an index based is defined ad hoc .
on a set of indicators to help describe
The methodological proposal for
the respective system . When apply-
applying the indicators in the context
ing indicators at the project level , this of this conceptual framework and its
set of indicators lends support to and associated indicators is comprised of
organises the project evaluation proc- five instances :
ess, but in no case is it completely
exhaustive, since this process involves 1. System index
many other variables .
At the systems level , the IDB pro-
For purposes of organising the indica- posal is to consider all or the greatest
tors and associating them to the pro- number of indicators possible from
among those proposed, in order to
posal framework, we have considered
achieve the most complete view pos-
the need for input, process and impact
sible of the development status for the
indicators , depending on the data type incorporation of ICT into education .
you want to describe and its scope
of application . Nonetheless , process This set of indicators, to the extent that
indicators are applied exclusively at it is possible to obtain complete, up-to-
the project level and refer specifically date information, allows us to create
to the components that each project one or more indices accounting for

System
index

Selection
Evaluation of actions

(policy)

Follow-up Selection
and of relevant

monitoring indicators

101
 Chapter III Conceptual frameworks

the status of progress in the incorpo- ering is to be carried out periodically

ration of ICT into education , allowing at regular intervals , depending on the
us to determine the system’s phase of availability of data at each level :
development and the areas in which it
is more or less advanced . • before the project begins: building
the baseline;

2. Selection ofactions (policy) • mid -term measurement: data gath-

ering at the halfway point, while a
Based on the data provided by the indi-
project is being implemented . Allows
cators, policymakers can make better
you to determine impacts over the
informed , more complete decisions
medium term and to take steps if
that place a high priority on achieving
necessary;
specific impacts .
• end of project measurement: gath-
ering information upon completion
3. Selection ofrelevant indicators of the intervention ; facilitates quan-
At the project level , it will not be fea- tification of changes in indicators
sible for all defined input indicators to during the project implementation
be modified by an ICT for education period . At this point, the status of all
project . Therefore , at the outset of a the input indicators is ascertained .
project of this type, the first task to These indicators provide informa-
undertake would be to establish which tion about the impact attributable
of the proposed set of indicators are to the project and about changes
feasible for this initiative to impact . observed in the overall status of the
Furthermore , process indicators are system undergoing intervention .
created that will enable follow- up and
monitoring of project development. A fourth instance of data gathering is
advisable , whenever possible :
Aside from the above , the methodo-
logical proposal suggests that all input •

follow- up measurement: gathering

indicators must be measured each
information one or two periods after
time or taken into consideration . This
the respective project is completed .
is done for two reasons . The first is
This allows evaluation of the status
because a project may, in practice, pro- of the situation over the medium
duce impacts unforeseen in its original
term , after the project has ended . At
design , and it is advantageous to be this point, drops may be observed
able to ascertain and quantify them . in some indicators due to lack of
The second reason has a systemic
funding for recurring expenses , for
or public policy aspect to it: becoming
example .
acquainted with all of a country’s edu-
cational indicators will afford a project Process indicators required for the
executor a broader view of the global
project to report should also be
impact of any country’s different edu-
defined . Reports on these indicators
cation projects and of its status at dif-
will be of utmost value to the project
ferent points in time .
executor because this facilitates rigor-
ous monitoring of project implementa-
4. Follow-up and monitoring tion and provides the opportunity to
According to the methodological pro- make suggestions and , if necessary,
posal at the system level , data gath- to propose remedial measures .

102
 Improving quality in education

At the outset of the project, it is advis- educational systems and the support
able to agree on a timetable for sub- of empirical evidence on how to opti-
mitting reports on these indicators . mally capitalise on ICT potentials .
Perhaps not all of these indicators will
be relevant to all of the processes . This ICT alone will not make the difference .
means agreement must be reached We are confident that no technological
among the parties regarding which device will solve the enormous chal-
indicators will be used for each project lenges facing the education systems
management plan and what reporting seeking to meet today’s demands .
intervals will be observed . We are not facing a technological
challenge, but an educational chal-
5. Impact evaluation lenge ( 22) . We know that training
people , a country’s human capital , is a
The final evaluation of a project may
take into consideration a broad set of complex process involving a myriad of
variables with which ICT must interact
tools , models and indicators to report
on results . According to the proposal dynamically to produce the changes
required .
presented herein , we suggest taking
into account how project results have
enabled modification of indicators of We acknowledge that we are facing a
challenge that is both vast and new,
the system where they were intro-
duced , in terms of impact. These indi- but which also changes at speeds
cators were established in the defini- heretofore unseen . Therefore , we
tion of the general indicators and in expect this proposal will undergo
the selection of specific indicators rel- continual revisions , adjustments and
evant to project action . reformulations . We present it with the
humility of an individual who explores
unknown lands without the benefit of
In this way, definition of the indicator
allows us to set goals for the project, certainties or necessary tools , but with
which under the same terms of the the urgency of having to move forward
with determination .
indicator it proposes to change.
Therefore , for each relevant indicator,
the project impact evaluation presents Currently we are preparing the pro-
its respective status before the posal for indicators that reflect and
intervention , the status targeted by the complement the scale proposed in
intervention (goal ) and the percentage this conceptual framework . To accom-
of the goal achieved .
(22) ‘Today’s classroom teachers need to be
prepared to provide technology-supported
4. Conclusions learning opportunities for their students . Being
prepared to use technology and knowing how
and next steps that technology can support student learning
have become integral skills in every teacher’s
We have worked on this conceptual professional repertoire . Teachers need to
framework and indicators proposal in be prepared to empower students with the
the belief that information and com- advantages technology can bring . Schools and
munication technologies ( ICT) can classrooms , both real and virtual , must have
teachers who are equipped with technology
indeed make an important contribution
resources and skills and who can effectively
to improving the quality of education , teach the necessary subject matter content while
but they demand a much more rigor- incorporating technology concepts and skills .’
ous , comprehensive incorporation into (Unesco , 2008)

103
 Chapter III — Conceptual frameworks

plish this task, we are considering a allow us to apply it to our specific con-
very important proposal that Unesco texts. We are still striving to improve it,
UIS has already developed including in collaboration with experts and other
over 50 indicators, which we are com- agencies and international organisa-
plementing with additional indicators tions. It is now being implemented in
covering all the areas proposed. currently operating bank-supported
projects in Latin America and the
We are making this seminal work Caribbean for the purpose of aligning
available to those who wish to col- definitions, specifying and testing indi-
laborate in its continual improvement cators and building new instruments
and to the development of tools that for its implementation.

References
Balanskat, A., Blamire, R. and Kefala, S. (2006). The ICT impact report: a review
of studies of ICT impact on schools in Europe. Brussels: European Schoolnet.

Barrera-Osorio, F. and Linden, L. L. (2009). The use and misuse of computers

in education: evidence from a randomized experiment in Colombia. Washington,
DC: World Bank.

Becta, (2006). ‘The Becta review: evidence on the progress of ICT in education’.
Accessed at: http://becta.org.uk/corporate/publications/documents/The_Becta_
Review_2006.pdf.

Benavides, F., Dumont, H. and Istance, D. (2008). The search for innovative
learning environments, on innovating to learn, learning to innovate. Paris:
OECD.

Cobo Romani, C. and Pardo Kulinski, H. (2007). Planeta Web 2.0. Inteligencia
colectiva o medios fast food. Grup de Recerca d’Interaccions Digitals. Universitat
de Vic. Flacso México. Barcelona / México DF.

Christensen, C. M., Horn, M. B. and Johnson, C. W. (2008). Disrupting class:

how disruptive innovation will change the way the world learns. McGraw-Hill,
2008.

Hepp, P., Hinostroza, J. E., Laval, E. and Rehbein, L. (2004). Technology in

schools: education, ICT and the knowledge society. Washington, DC: World
Bank. (http://www1.worldbank.org/education/pdf/ICT_report_oct04a.pdf).

Kozma, R. (2008). ICT, education reform, and economic growth: a conceptual

framework.

Kozma, R. (2005). ‘Monitoring and evaluation of ICT for education impact: a

review’, in: M. Trucano (ed.), Monitoring and evaluation of ICT in education
projects. infoDev, World Bank.

OECD, CERI (2008). Trends shaping education. Paris: OECD.

104
 Improving quality in education

Partnership for 21st Century Skills (2009), P21 framework definitions

document.
Pedró, F. (2006). The new millennium learners: challenging our views on ICT
and learning. Paris: OECD, CERI.

Sawyer, K. (2008). Optimising learning implications of learning sciences research

on innovating to learn, learning to innovate. Paris: OECD.

Trucano, M. (2005). Knowledge maps: ICTs in education. infoDev/World Bank,

Washington, DC.

Unwin, T. (2005). ‘Capacity building and management in ICT for education’,

in: M. Trucano (ed.), Monitoring and evaluation of ICT in education projects.
infoDev.

Unesco (2008), ICT competency standards for teachers. Unesco.

Unesco (2003). Performance indicators for ICT in education. Bangkok: Unesco.

http://www.unescobkk.org/index.php?id=1109

Unesco (n.d.). Indicators for assessing ICT impact in education. Bangkok:

Unesco. http://www. unescobkk.org/index.php?id=662

Unesco Institute for Statistics (UIS) (2005). ICTs and education indicators: sug-
gested core indicators based on meta-analysis of selected international school
surveys. WSIS Phase II, Tunis. Available at: http://www.itu.int/ITUD/ict/partner-
ship/material/ICT_Education_Paper_Nov_2006.pdf.

Unesco Institute for Statistics (UIS) (2008). Proposal for internationally compa-
rable core indicators on ICTs in education.

World Bank (2004). Monitoring and evaluation: some tools, methods and
approaches. http://www.worldbank.org/oed/oed_approach.html.

105
A conceptual framework
for benchmarking the use and assessing
the impact of digital learning resources
in school education
Beñat Bilbao -Osorio and Francesc Pedró
OECD Centre for Educational Research and Innovation

The comparative study of information and communication technologies (ICT) in

school education has primarily focused on investments in infrastructure, equip-
ment and the resulting ratios per pupil, as well as on in-service teacher training
and, lately, on the incentives and barriers for classroom use. Less attention has
been paid to the development and publication of digital learning resources (DLR)
as a mean to increase the added value that ICT can bring about for teaching and
learning. In some countries, governments have started to subsidise programmes,
repositories and networks focusing on DLR. However, until now, little empirical
evidence has existed on the dimensions and impact of these policies, both on
their capacity to foster the development of DLR and on their final effects on the
teaching and learning processes.

The OECD Centre for Educational Research and Innovation (CERI) has recently
completed a project intended to bridge this knowledge gap by reviewing and eval-
uating the process of innovation involved in policies and public as well as private
initiatives designed to promote the development, distribution and use of DLR for
the school sector. Among its final outputs (23), this project includes the delivery of
a conceptual framework for the creation of a system of indicators related to the
development, use and effects of DLR.

This chapter presents the resulting initial proposal. It aims at shedding more em-
pirical light on the theoretical and policy debate about the effects of technology-
enhanced learning in schools. In this respect, the chapter sets the scene for the
ongoing policy debate and then discusses the lack of empirical evidence. Then it
outlines the objectives of the CERI proposal and describes its main components.
The final section comments on what the next steps will be in the process of defin-
ing and compiling the appropriate indicators.

(23) The main report is published as CERI-OECD (2009), Beyond textbooks: digital learning resources
as systemic innovation in the Nordic countries. Paris : OECD .

107
 Chapter III Conceptual frameworks

The policy debate OECD already reported that education

policymakers saw enormous potential
Investment in ICT as a means to raise for ICT to transform education . In 1999 ,
educational standards and create the limited available data on trends in
opportunities for every child has ICT investment and use were headed
attracted attention from both policy- up sharply (OECD , 1999) . Around that
makers and academic researchers . time , an OECD conference warned
Based on the belief that ICT could about the urgency of ‘ bridging the
make learning much more efficient, digital divide’ ( OECD , 2000). In 2004 ,
and encouraged by the findings that PISA data confirmed the exponential
new technologies had accounted for growth in the presence of ICT in educa-
much of the productivity gains in the tion ( OECD , 2004) . In just three years ,
workplace during the advent of the between 2000 and 2003 , student- per-
new economy (Lindbeck and Snower, computer ratios dropped by more than
2000) , many governments explored half in most countries (and by a factor of
the role that ICT could play for edu- 4 to 5 in those that were lagging). While
cation , although this interest started less than a third of secondary schools
to fade away in some countries after had Internet access in 1995 , it was vir-
the 2001 technology crisis and a grow- tually universal by 2001. Although there
ing awareness that the investments
are no internationally comparable data
seemed not to produce any relevant on current expenditure on educational
accountable effects .
ICT hardware and software , there are
Nevertheless , the benefits of the use
signs of unmet demand for additional
investment, particularly in the areas of
of ICT have been largely echoed
hardware upgrading and availability of
among policymakers and claims like
‘ ICT can make a significant contribu- digital content or learning resources.
According to the most recent PISA
tion to teaching and learning across all
data, a lack of adequate computer soft-
subjects and ages , inside and outside
the curriculum’ ( UK DfES , 2003) have ware for instruction is cited by school
been common . Four main sources of principals as an important hindrance to
educational benefits have been identi- science instruction (OECD , 2007)
fied with the use of ICT: (i) preparation However, there seems to be little
of students for the knowledge econ-
empirical evidence (24) about the
omy; (ii ) improvement of educational final benefits associated with these
performance ; (iii) struggle against investments in terms of use of ICT in
the ‘ digital divide’ between students;
schools and their impacts throughout
(iv) improved quality in the teaching the educational system , and claims
and learning processes , allowing for
a customisation of the educational of ‘ unfulfilled promises’ have opened
processes .
an academic and policy debate about
whether the considerable investment

Based on these expected benefits , sig- in ICT pays off in any obvious way.
nificant government investments have
been made available . Between 1998 (24) Recently a number of studies have aimed at
and 2002 , ICT expenditure in the UK analysing the impacts of ICT in education . The
analytical works at SITES , E- Nordic Learning ,
almost doubled in secondary schools Becta or the OECD’s PISA reports (based on
and multiplied by three in primary 2003 and 2006 results) are the main experiences
schools . Equally, 10 years ago , the in the field .

108
 Benchmarking and impact assessment

Lack of empirical evidence rich methodologies . This would seem

to be the outcome of the two system-
about the effects of DLR, atic reviews of literature conducted
and ICT more broadly, recently, which conclude that ‘ in gen-
on education eral and despite thousands of studies
about the impact of ICT use on stu-
Computers in education are gener- dent attainment, it is difficult to meas-
ally used in two broad contexts: (i ) ure and remains reasonably open to
to provide computer skills training , debate’ ( infoDev, 2005) , and also that
which teaches students how to use ‘ some studies reveal a positive cor-
computers ; (ii) to provide technol- relation between the availability of
ogy enhanced learning (TEL) , which computer access or computer use and
uses computers to enhance teach- attainment, others reveal a negative
ing and learning methods, strategies correlation , whilst yet others indicate
and activities in the whole curriculum . no correlation whatsoever between
While there is a clear case for the the two’ ( Kozma, 2006) .
use of ICT for enhancing the compu-
ter skills of students, the role of TEL Experiments can only attempt to deter-
is more controversial (Machin et al . , mine how effective ICT is in teach-
2006) . There is neither a strong and ing specific school subjects , due to
well - developed theoretical case nor the multitude of compartmentalised
much empirical evidence supporting methodologies to be found in a single
the expected benefits accruing from school , and even in lines or different
the use of ICT in schools , as different groups of students studying the same
studies find mixed results (Kirkpatrick subject, albeit with different teach-
and Cuban , 1998) . ers. Consequently, the experiments
designed to date compare the educa-
Apparently, there seems to be no con- tional attainment of a group of students
clusive evidence. On the one hand , using an ICT- rich teaching methodol-
studies carried out by Becta (2002) ogy with the achievement of another
and Machin et al . (2006) find a posi- group with similar characteristics being
tive effect on the use of ICT and edu- taught using traditional methods .
cational attainment , and , on the other
hand , the research carried out by However, an in -depth analysis of the
Fuchs and Woessman ( 2004), Leuven available knowledge base shows that
et al . (2004) or Goolsbee and Guryan school attainment only improves if cer-
(2002) find no real positive effect tain pedagogical conditions are met.
between the use of ICT and educa- This is the conclusion reached by Kulik
tional results once other factors , such (2003) , who used the measurement
as school characteristics or socio- of the effects found by eight different
economic background , are taken into meta- analyses covering 335 studies
account. There is a generalised belief before 1990 and 61 controlled experi-
that, overall , the ‘ no significant differ- ments whose outcomes were pub-
ence’ phenomenon, documented on lished after 1990. Most of the studies
many occasions in the case of dis- carried out in the 1990s concluded that
tance learning, also emerges in school stimulation programmes have positive
education . According to this , there is effects when used to enhance reading
insufficient evidence to affirm either and writing capabilities and that, albeit
the superiority or inferiority of ICT- less frequently, they have a clearly

109
 Chapter III Conceptual frameworks

positive effect on maths and natural the development of DLR and to the
and social sciences. Indeed, ‘ simply development of content production .
giving students’ greater access to both Although many big private publishing
computers and Internet resources companies have entered the market
often results in improved writing skills’ . of developing DLR and have acknowl-
The assessments of primary school edged their potential , until recently
pupils using tutorials to improve their they have regarded this market as
writing increased significantly in this unattractive and major investments
field . Even very young primary school have not been made. A possible
pupils using computers to write their explanation for this may lie in the
own stories ended up improving their role that private publishers play in the
marks in reading . In short, there is a development of school content, either
positive correlation between the fre- in analog or digital form . Commercial
quent use of word processors and publishers have traditionally played a
improved writing -related capabilities . key role in developing and distributing
printed learning material. However,
Much less attention has been paid when it comes to DLR, they seem to
both by researchers and policymakers find that the market may not be ready
to the actual determinants of ICT use to use this type of resource yet , mainly
in school and their impacts in different due to the lack of infrastructure , teach-
dimensions of the educational system . ers’ skills or cultural factors . Therefore ,
For a long time, as noted above, ICT they may lack the necessary incentive
investments have been channelled to develop this kind of material . At the
towards the construction of an ICT same time , the lack of readily avail-
infrastructure in schools , and most able DLR of sufficient quality can also
available resources have been devoted affect the motivation and attitudes of
to the acquisition of ICT equipment, teachers towards DLR and ICT more
i.e. computers , and of Internet access broadly, and the need to invest in ICT
connections , e.g. broadband net- infrastructures . On the whole , a vicious
works . While this investment is a clear circle appears when the lack of signifi-
pre- requisite to foster the use of ICT in cant teacher demand proves a disin-
schools , it can also be regarded as a centive to publishers’ offers , which in
necessary but not sufficient condition turn affects demand negatively, and
to assure its use , if other factors are where all the determinants are closely
not simultaneously born in mind . More intertwined .
precisely, factors such as the compe-
tences and attitudes of teachers to In addition to private publishers , stu-
use ICT or the availability of DLR have dents and teachers have also started
also been identified as key factors to producing DLR by themselves , partly
explain the degree of use of ICT by along the lines and rationale which are
teachers and students . successfully inspiring the production
and use of open educational materi-
While teachers ’ attitudes and compe- als in higher education (25) . There has
tences in respect of ICT have been been a shift in the paradigm where
widely recognised as a key factor both teachers and students were only
(Williams et al ., 1998) and signifi-
cant public investments have aimed (25 ) See CERI -OECD , Giving knowledge for free:
at enhancing these competences , open educational resources in higher education .
much less attention has been paid to (Paris , OECD , 2007) .

110
 Benchmarking and impact assessment

users of learning material, and they data sources and the possibility (or
are now also producing content mate- not) of linking different datasets.
rial
selves and
which they
that
exchange
is regarded
amongbythem-
their 4. To highlight possible options to
generate the missing data. As a
peers as ‘very important’. The mate- result of the analysis of the data
rial of these ‘user-producer’ teachers already available, data gaps will be
and students is increasingly important identified and different strategies
and will continue to be so as Web 2.0 and tools to develop the required
applications become more available data will be suggested.
generally. However, until now, its study
has also been somehow neglected in Definition
traditional studies.
While there is a clear and practical
interest to track the availability and use
Objectives of the of DLR, there is an even greater inter-
conceptual framework est in understanding the causes driv-
ing the development and use of DLR,
The overall aim of this proposal is to and the impacts they generate on the
bridge this analytical gap in the study teaching and learning processes,
of DLR and deliver a conceptual because the lessons learnt can be
framework for developing indicators used to refine our understanding of the
that could trace and benchmark the incentives and barriers regarding the
development, use and effects of DLR. broader use of ICT to enhance school
education. An analytical framework
More precisely, the objectives of this capable of identifying and explaining
proposal are as follows. these factors, their interrelations and
their impacts would allow analysts to
1. To provide a holistic conceptual enhance their knowledge about the
framework for the development of use of DLR and ICT more broadly, and
these indicators. This model would to provide evidence-based policy rec-
map the different factors affect- ommendations for policymakers.
ing the development and use of
digital learning resources, and However, at the moment, the lack of
a holistic conceptual framework that
their impacts on the educational
takes into account all the intervening
system.
factors and their possible interrelation-
2. To define and construct a number
of key indicators that would allow ships, and the lack of available data
have prevented the development of
comparing and benchmarking more robust results allowing to moni-
across different countries of the tor and evaluate the role that different
progress in the production, avail- sources of ICT investment, including
ability, use and impacts of DLR in investments in DLR, play in the use
schools. of ICT and in the teaching and learn-
3. To identify existing relevant ing processes and the educational
sources and collect the available attainment of students. This lack of
data. Based on the different fac- empirical evidence has also affected
tors described on the conceptual the necessary political support for any
framework, to identify what data further investments and has increased
are already available in different the feeling among stakeholders of

111
 Chapter III Conceptual frameworks

' unfulfilled promises’ related to the use ments produces a specific output in the
of ICT in the educational system . form of available computers or Internet
access (for the case of ICT infra-
In light of the information gathered in structure) , digital learning resources
the OECD project on DLR during the or enhanced teachers’ ICT compe-
interviews conducted with a number tences . The combination of these out-
of stakeholders (i.e. departments of puts would influence the actual use
education , teachers , headmasters , of DLR and ICT more broadly, in a
students , local and regional govern- particular moment in the educational
ments and publishers) and a review of system . However, rather than claiming
the existing literature on comparative
a linear and causal relationship , the
research and recent practices , an ana-
model intends to reflect the complex
lytical framework is proposed below. nature of the interaction between each
This framework aims to account for
of these factors and the actual use of
the factors affecting the development ,
use and impacts of DLR , as well as ICT/DLR. For instance , higher levels
of ICT/DLR use could also stimulate
for the complexity of the interrelation-
ships between these factors . Figure 1 higher levels of ICT/DLR investments .
presents a visual representation of
this framework . In addition to these three main direct
investment variables , a number of
The proposed model presents a ‘ environmental factors ’ would also
number of investment measures on affect the levels of DLR/ICT use and
the left- hand side of the chart that therefore should be included in the
are interrelated . Each of these invest- model . These variables relate to the

Investment policies Policies' results

ICT infrastructure

Student performance:
Academic performance
Digital competences

DLR
USE OF ICT

Perceived satisfaction in

teaching and learning

process

Teachers' ICT competences

ICT environment

Meso: Macro:
Micro
:
Teacher Curriculum
Socio
commitment ICTnational
economic
School responsive
factors
ness
leadership

Figure 1 : Analytical framework for assessing the development, use and impacts of DLR

112
 Benchmarking and impact assessment

overall ICT environment in the coun- variables is not unidirectional , and

try that may push for or against the therefore higher levels of technologi-
use of ICT in society in general , and cal competences , better academic
in the educational system in particular. performances or higher levels of sat-
Particular attention has to be paid to isfaction in the teaching and learning
the fact that very different factors can processes could also influence higher
be brought into the picture. The degree ICT/DLR uses , triggering a virtuous
of public policy influence on these fac- upwards circle that would move within
tors could largely differ both in scope the whole model .
and impact depending on the nature of
these factors . Teachers’ commitment The relationships between the differ-
to the use of ITC in classes , for exam- ent variables in this model are hypo-
ple , is a key variable that affects the thetical and their existence ( or not)
final use of DLR or ICT in schools , and should be investigated empirically,
that would be the result of a mix of fac- should data become available .
tors such as policies to promote ICT
in schools and teachers ’ attitudes and
The variables
convictions regarding the role of ICT in
the teaching and learning processes . The model described above presents
Pupils ’ expectations would be another a number of variables and hypotheti-
variable that could significantly affect cal relationships between the vari-
the use of DLR and ICT and that ables that would need to be tested .
could be far from being affected by This section briefly presents the dif-
public intervention . These factors are ferent variables . As presented , this
somehow the ‘ soil ’ where the DLR/ section only identifies the variables
ICT investments are ‘ seeded’ and that and provides some initial suggestions
could be a determinant in obtaining for their definition and measurement.
the desired ‘ fruit’ . The difference in scope of these defi-
nitions would therefore affect the type
As a result, policymakers are con- of data that would be required . These
fronted with a policy dilemma in terms variables , classified according to their
of what to do: invest in infrastructure , nature and role in the proposed model ,
DLR, teaching competences (in which are as follows .
ones , and how much ? ) and/or in
improving the ICT environment (how, 1. Direct investment variables :
and how much ?) in order to obtain the These are the different sources of
desired results in terms of enhancing investment where a clear connec-
the ICT/DLR use . tion between the initial investment
and the actual results accruing from
Finally, the model suggests that the them can be identified . The model
use of ICT/DLR could have a final identifies three investment types ,
impact on the educational system by closely intertwined between them .
allowing students to achieve higher
•

educational attainment, developing ICT infrastructure: This vari-

stronger digital competences and able deals with the investment
improving the perceived satisfaction in equipment (computers , white-
in the teaching and learning proc- boards , laptops, projects) and
esses. However, as happened previ- network connections . A number
ously, the relationship between the of clear outputs can also be

113
 Chapter III Conceptual frameworks

observed as a direct result of teachers more competent and

these investments : the number eventually having a positive atti-
of computers by students or tude towards ICT and using ICT
the number of computers with in school . The input investment
(broadband ) Internet connection would be the resources devoted
by student are just a few exam- towards teacher training and ICT.
ples of this type of variable . The output measure , however,
•

Digital learning resources (DLR): could differ and allow for differ-
There have been many defini- ent definitions and measures . On
tions of DLR. In this project (26) , the one hand , an easy and direct
it has been pointed out that DLR measure could be the number of
‘ can refer either to any resource teachers trained in the system .
used by teachers and students On the other hand , a more com-
for the purpose of learning , or plex measure could relate to the
to only resources particularly attitudes and changes in atti-
designed to be used in learning tudes of the trained teachers
settings . It is both a strength and towards the use of ICT/DLR.
a weakness of the former defini-
tion that it is very general it 2. Outcomes : An intermediate out-
can refer to anything from a stone come can be linked and traced
or a feather, to Encyclopedia back to the initial investment vari-
Britannica or advanced data- ables , but can also be influenced
bases , as long as it is used for by some external factors .
learning . The second definition is
more limited and hence easier to •
Use of ICT/DLR: The amount
use. But it excludes open learning and nature of the different uses
resources like online newspaper of DLR and ICT. This broad vari-
articles , most computer games , able could be broken down into
and applications such as Google different categories and create
Earth ’ . Moreover, it would not a typology of different types of
take into account the production ICT/DLR uses according to the
of DLR carried out by individual different categories of DLR, for
teachers and shared within a example . Equally, a classification
closed system or intranet exclu- of the use by subject and class
sively. As a result, it is important group would also provide more
to note that this definition and information that could be useful
measurement would be a stricter when analysing its relationship
approximation of the overall DLR with the investment variables .
concept and therefore any con-
clusions about the availability 3. Impacts: These are the final objec-
and role of DLR should be han- tive that the initial investments aim
dled very carefully. at . The model identifies two main
•

Teachers ’ ICT competences: This types of possible impacts .

variable relates to those invest-
•

ments aiming at making the Student performance: The use

of ICT and DLR could have an

(26) Please refer to the full OECD report Beyond impact on the student’s per-
Textbooks. DigitalLearningResources inthe Nordic formance that could go in two
Countries for a more thorough definition of DLR. directions:

114
 Benchmarking and impact assessment

Development of ICT compe- ing this process , including the

tences (or ‘21st century com- always fuzzy concept of quality.
petences ’): The definition However, a ‘ subjective’ measure-
of ICT competences could ment of the changes in the proc-
be restricted to the effective ess by the different stakeholders
use of the ICT infrastruc- could be a way to get around this
ture , i.e. use of a computer initial difficulty.
or the Internet , or it could
have a broader scope , where 4. Environmental factors : These
students would be able to variables , although they cannot be
use, search , understand directly controlled by direct govern-
and even produce different ment investment, have a very clear
content in a digital support impact in the capacity of the direct
in order to obtain or show a investments to achieve the desired
better understanding of par- results . They are ‘the soil’ where
ticular subjects . In the latter, the different investments ( ‘ the
specific definitions of com- seeds ’ ) are planted .
petences should be devel-
oped and appropriate tests •
Teachers ’ commitment to ICT:
should be put in place in The teachers ’ commitment and
order to measure and evalu- determination to use ICT and
ate the achievement of these
DLR in their schools is one key
competencies . variable that may explain differ-
Academic performance in ences in the levels of investment
basic subjects: The use of ICT in schools and also in the actual
in learning different subjects use of ICT/ DLR by the teachers .
could have an impact in the
actual academic attainment
This is particularly true in decen-
tralised systems , where teach-
of students in these differ-
ers count on a large degree of
ent subjects . Analysing these
autonomy. Also , research has
results and comparing them shown the relevance of leader-
before and after the use of
ship in schools in this domain .
ICT/DLR would be important •
Socioeconomic factors: Socio-
to establish any causal rela-
tionship between the two . economic background , age and
•

Improved or new teaching and gender of students have been

learning process: The use of pointed out in the literature as
DLR and ICT could also improve being key factors that may influ-
or bring about new processes ence not only their learning
of both teaching and learning , expectations but also the degree
making it more interesting for stu- and scope of the actual use of
dents and teachers , and improv- ICT/DLR (outcome variable) ,
ing the communication between and also influence decisively the
the different stakeholders . students ’ educational attainment
Having an ‘ objective’ measure of (impact variable). Therefore , any
‘ improved’ process could be very study that aims at drawing causal
difficult as it would require a clear relationships between the vari-
definition and measurement of ables should take these factors
all the different aspects affect- into account.

115
 Chapter III Conceptual frameworks

•
In addition to these variables , nities , but in itself access to DLR does
it is important to note that the not imply automatically granting better
model also identifies a very broad educational processes . How DLR is
variable that somehow affects used , in the wider context of other
all the different variables in the
intervening factors , is the critical vari-
model . able and not much is known about
• The overall ICT environment. This
it yet. This points out the need for a
variable aims at explaining the
overall societal attitude towards
clear understanding not only of the
intervening factors but, in particular, of
the use of ICT, not only in the edu-
their interrelationships .
cational system , but more broadly
in all aspects of life . This broad
variable would include : CERI’s ongoing work in this domain
ICT responsiveness: ICT is addressing some of these issues in
readiness and acceptance in close cooperation with other interna-
the overall society influence tional agencies . The main objective is
the pressure and demand to enlarge the number and quality of
for the inclusion of ICT in the indicators about access to and use of
educational system as well as ICT in education . In so doing , the main
the attitudes of both teachers activities that will be carried out are as
and students towards the use follows .
of ICT. Possible measures
of this responsiveness could 1. Redefinition and refinement
be the penetration of ICT in of the model : A validation of the
homes or in firms . model should be carried out. More
National curriculum: The
precisely, this activity would (re- )
inclusion of the obligation to define and identify new factors ,
use ICT/DLR in the national
map the hypothetical relationships
curricula , directly or indirectly between the variables and revisit
(by way of mentioning them
the scope of the model . This refine-
in the definition of expected
ment of the model would allow
pupils’ competences), may
be a variable that may explain building the necessary consensus
difference across countries in in order to develop internationally
the use of ICT/DLR , and also agreed and comparable indicators .
may be a factor affecting the
levels of ICT/DLR investments 2. Redefinition of variables : Alter-
in the educational system . native definitions for the variables
are available , with differences in
scope and nature . A commonly
Conclusions and next steps
agreed redefinition of the variables
Benchmarking DLR use and effects in would then be necessary.
education is not an easy task . To start
3. Evaluation of available data:
with , in the complex context of a class-
room , it is difficult to isolate the DLR Based on the agreed model , an
contribution from other elements that evaluation of the existing data
intervene in the educational process . sources and the possibility to link
Access to quality DLR is certainly an different datasets in a coherent
enabler of better educational opportu- manner should be carried out .

116
 Benchmarking and impact assessment

4. Data needs assessment: Based different levels of depth . The com-

on the agreed model and the data plexity and cost to obtain the data
already available , a data needs should match the utility and a con-
assessment should be carried out . sensus should be reached when
As mentioned in the definition of the defining the variables and devel-
variables of the conceptual model , oping the necessary methods to
the data needs can be defined in obtain the required new data .

References

Becta ( 2002) . ‘ ImpaCT2 — The impact of information and communication tech-

nologies on pupil learning and attainment’ . Coventry: Becta .

Empirica (2006) . ‘Benchmarking access and use of ICT in European schools

2006: final report from head teacher and classroom teacher surveys in 27
European countries , August 2006. Bonn .

Fuchs , T. and Woessman , L. (2004) . ‘ Computers and student learning: bivari-

ate and multivariate evidence on the availability and use of computers at home
and at school ’ , Brussels Economic Review/Cahiers Economiques de Bruxelles,
Editions du Dulbea , Université libre de Bruxelles , Department of Applied
Economics (Dulbea), Vol . 47, No 3–4 , 359–85 .

Johnson , L. , Levine, A. and Smith , R. (2009). The 2009 horizon report. Austin ,
TX : New Media Consortium .

Kausar, T. , Choudhry, B. N. and Gujjar, A. A. (2008). ‘A comparative study to

evaluate the effectiveness of computer assisted instruction ( CAI) versus class
room lecture ( RL) for computer science at ICS level’ , Turkish Online Journal of
Educational Technology — TOJET, October 2008 , Vol . 7, Issue 4, Article 2 .

Kozma, R. (2003). ‘Technology and classroom practices: an international study’ ,

Journal of Research on Technology in Education, Vol . 36 , No 1 , 1–14 .

Kozma, R. B. (2008) . ‘ Comparative analysis of policies for ICT in educa-

tion . Center for Technology in Learning’ , SRI International (to appear in the
International Handbook on Information Technology in Education) .

Kulik, J. (2003) . ‘The effects of using instructional technology in elementary and

secondary schools : what controlled evaluation studies say’ . Menlo Park, CA :
SRI International .

Lee , M. G. (2003). Comparative analyses of ICT integration initiatives in Korean ,

German and American educations , Mediapädagogik ( 10.1.2003).

Machin , S. , McNally, S. and Silva , O. (2006). ‘ New technologies in schools : is

there a pay off’ ? London : Centre for Economic Performance ; Bonn : Institute for
the Study of Labour.

117
 Chapter III — Conceptual frameworks

OECD, (1999). OECD science, technology and industry scoreboard 1999, Paris:
OECD.

OECD (2000). OECD Forum 2000. Paris: OECD.

OECD (2007). Giving knowledge for free: the emergence of open educational
resources. Paris: OECD.

OECD (2009). Working out change: systemic innovation in vocational education

and training. Paris: OECD.

Ramboll Management (2006). E-learning Nordic 2006 – Impact of ICT on educa-

tion. Copenhagen: Ramboll Management.

118
CHAPTER
CASE STUDIES

Assessing new technological literacies

The impact of ICT in education policies on teacher

practices and student outcomes in Hong Kong

Indicators on ICT in primary and secondary

education : results of an EU study

Impacts of ICT use on school learning outcome

ICT impact data at primary school level :

the STEPS approach

119
Assessing new technological literacies
Edys S. Quellmalz
WestEd , Technology Enhanced Assessment and Learning Systems

Abstract

As technologies and contexts of their use increase, characterizations of 21st cen-

tury skills have grown beyond operation of computer productivity tools to encom-
pass individuals’ use of the Internet, specialized software, and facility with hand-
held and wireless devices. ‘New literacies ’ have expanded to refer to expertise in
the use of a range of digital media and information and communication technolo-
gies exercised in academic and applied settings to solve a range of problems
(Quellmalz & Haertel, 2008). This paper addresses: ( 1) distinguishing features of
the multiple frameworks for ICT, 21st century skills, and new literacies; (2) alterna-
tive assessment designs and prototype student assessments of new literacies,
(3) evidence-centered design methods for establishing technical quality, and
(4) features of coherent, balanced assessments of new literacies across class-
room, district, state, national and international levels.

Introduction As technologies and contexts of their

use increase , characterisations of 21st
Information and communication tech- century skills have grown beyond oper-
nologies ( ICT) permeate school , work, ation of computer productivity tools
personal and civic activities. Their to encompass individuals’ use of the
prevalence speaks to the centrality of Internet , specialised software and facil-
these powerful , transformative tools in ity with handheld and wireless devices .
all walks of life. Policymakers through- ‘ New literacies ’ have expanded to refer
out the world recognise the significance to expertise in the use of a range of
of technologies for economic, civic and digital media and information and com-
global progress , along with the con- munication technologies exercised in
comitant need for coherent educational academic and applied settings to solve
policies to promote and implement skills a range of problems (Quellmalz and
characterised as ‘ new literacies ’ , 21st Haertel , 2008) .
century skills, information and commu-
nication technology skills and techno- Technologies are increasingly rec-
logical literacy (ISTE, 2007; Partnership ognised as transforming schooling
for 21st Century Skills, 2005; Kozma , as a result of their capacity to extend
McGhee , Quellmalz , and Zalles , 2004) . students ’ opportunities to access

121
 Chapter IV — Case studies

rich repositories of knowledge and to level tests of new literacies are not
engage in deep, extended problem aligned and articulated, the assess-
solving. Large-scale national and inter- ment systems will not be balanced and
national studies are providing evidence the validity of inferences about student
that technologies are truly changing performance will be compromised.
and improving schools by enriching cur-
ricula, tailoring learning environments,
offering opportunities for embedding
‘New literacy’ assessments
assessment within instruction and pro- Currently, there are multiple frame-
viding collaborative tools to connect works for assessing technology use
students, teachers and experts locally and 21st century critical thinking and
and globally (Kozma, 2003; Law, problem-solving processes. In one
Pelgrum and Plomp, 2008). Despite view, ICT assessment is of technology,
the pervasiveness of technology, there such as the international computer
are few traditional large-scale tests driving licence and technology profi-
or curriculum-embedded, formative ciency tests in some states in the USA.
measures that directly measure new lit- These tests measure the facts and pro-
eracies (Burns and Ungerleider, 2002; cedures needed to operate common
Quellmalz and Kozma, 2003). Internet and productivity tools, while
the content or the academic or applied
The quest for tests of students’ profi- problem and context are deliberately
ciencies with these 21st century skills selected to be familiar background
is hindered by a number of persistent
knowledge (Venezky and Davis, 2002;
issues. There are myriad definitions of Crawford and Toyama, 2002). The
information and communication tech- cognitive processes addressed in 21st
nologies and technological literacy century skills frameworks such as prob-
knowledge and skills. The contexts in lem solving, communication, collabora-
which ICT should be taught and tested tion, innovation and digital citizenship
vary widely. The extent to which the are not targeted by tests of technology
knowledge and skills about technolo- operations.
gies to be used within a domain-based
problem or context can be distinguished In a second view, ICT and 21st century
from the domain-specific knowledge frameworks emphasise learning with
and skills required is ambiguous technology by presenting test problems
(Bennett, Jenkings, Persky and Weiss, and items that integrate measurement
2003; Quellmalz and Kozma, 2003). of technology operations, strategic use
Methods for designing 21st century of technology tools to solve problems
assessments and for documenting their and subject matter knowledge and
technical quality have not been widely processes through carefully designed
used. Finally, a critical issue facing sets of tasks and items related to com-
the promotion of 21st century learn- plex academic and real world prob-
ing is that assessments of ICT should lems. This is the most prevalent view in
be coherent across levels of educa- 21st century ICT frameworks.
tional systems (Pellegrino et al., 2001).
Coherence must start with common or In a third view, testing is implemented
overlapping definitions of the knowl- by technology. Assessments by tech-
edge and skills to be assessed as nology simply use technical infrastruc-
new literacies. If the designs of inter- tures to deliver and score tests that
national, national, state and classroom are designed to measure other content

122
 New technological literacies

and skills in subjects such as maths ing technical quality, and (iv) features
and reading. These test designs aim of coherent, balanced assessments of
to reduce or eliminate the demands new literacies across classroom, dis-
of the technology, treating it as an trict, state, national and international
irrelevant construct. Equivalence of levels.
paper-based and technology-based
forms is the goal. Technology-based
tests are increasing rapidly in large- Features of new literacy
scale state, national and international assessment frameworks
testing, where technology is being
embraced as a means to reduce the Different specifications of knowl-
costs and logistics of assessment func- edgeandskills:Numerousframeworks
tions such as test delivery, scoring and have been developed by international,
reporting. Technology-based tests typi- national, state and professional organi-
cally assume that supportive technol- sations to specify the important char-
ogy tools such as calculators or word acteristics of new technology-based
processors are irrelevant to the content literacies, variously named ICT literacy,
constructs being tested and therefore 21st century skills and technological lit-
not to be measured separately. Since eracy. These frameworks differ in the
these types of testing programs seek range of technologies included, the
comparability of paper and online types of processes assessed in their
tests, the tests tend to present static use and the types of contexts of prob-
stimuli and use traditional constructed- lems in which the technologies will be
response and selected-response item applied. The frameworks differ in their
formats. For the most part, these con- focus on common Internet and produc-
ventional online tests remain limited tivity tools such as browsers, graphing
to measuring knowledge and skills tools,word processorsand presentation
that can be easily assessed on paper. tools and inclusion of more advanced,
Consequently, they do not take advan- specialised tools such as visualisa-
tage of technologies that can measure tions, simulations and domain-specific
more complex knowledge structures datasets and software. The frameworks
and extended inquiry and problem solv- differ in their relative emphases on the
ing included in 21st century ICT frame- operation of technology tools in con-
works. In short, a technology delivered trast to the use of the tools along with
and scored test of traditional subjects 21st century skills for solving problems
is not an assessment of 21st century and achieving goals in practical or aca-
ICT skills and should not be confused demic domains. Common processes
as one. often include accessing, organising,
representing, analysing, evaluating,
This paper focuses on assessments synthesising, communicating and col-
of technology and assessments with laborating (ISTE, 2007; Partnership for
technology, not assessments by tech- 21st Century Skills, 2005). The forth-
nology. It addresses: (i) distinguishing coming 2012 framework for the US
features of the multiple frameworks National Assessment of Educational
for ICT, 21st century skills and new Progress (NAEP) for Technological
literacies; (ii) alternative assessment Literacy has expanded the conceptu-
designs and prototype student assess- alisation of the kinds of technologies
ments of new literacies; (iii) evidence- and contexts of their use even further
centered design methods for establish- by integrating engineering design and

123
 Chapter IV - Case studies

ICT technological literacy frameworks . to complete tasks and items using

The NAEP technological literacy frame- technology. Background knowledge
work specifies three major assessment from life experience will present task
areas : technology and society, engi- demands different from tasks requiring
neering design and systems and ICT knowledge from academicsubjects . For
(see http://naeptech2012.org). The assessments of students’ ability to use
addition of engineering design to ICT technologies in a range of academic
frameworks incorporates knowledge and practical problems , assessment
and skills about how technologies are frameworks must be explicit about the
developed as well as how they are used . areas , complexity and familiarity of
The 2012 NAEP technological literacy content in assessment tasks or items
framework, which will shape the next and if that knowledge will be scored in
decade of designs of the assessments addition to processes and operations .
that serve as ‘ the nation’s report card’ ,
deliberately specifies a wider range of The next section describes a coordi-
technology products and processes nated assessment framework devel-
than those called out in ICT frame- oped in an international project that
works . Technologies in the designed aimed to provide a cross-cutting set
world include those in contexts such as of knowledge and skills that could be
transportation , energy, agriculture and used to test ICT literacy in academic
health , as well as information and com- or applied contexts .
munication technologies ( ITEA, 2000) .
A coordinated framework
Specifications for assessment vs.
curriculum : A critical issue in the for the design of ICT
assessment of the new technological assessment
literacies is the distinction between
curriculum and assessment frame- The purpose of the project to develop
works . The standards specified by a coordinated ICT assessment frame-
ISTE, ITEA and national frameworks work was to integrate measurement
set goals for promoting technology of technology use , ICT strategies and
understanding and use . These stand- subject matter. Development of the
ards aim to shape curriculum , instruc- coordinated ICT assessment frame-
tion and assessment. In contrast , a work was one component of a three-
national or international assessment year study funded by the National
framework may limit the content and Science Foundation (NSF) (Quellmalz
skills specified in the framework to what and Kozma, 2003). The project goals
can be directly tested and reported in were to develop a coordinated ICT
large -scale , on-demand assessments . framework and to design ICT per-
Thus , extended projects , collaboration formance assessments that could
and teamwork or creativity are unlikely be administered as a national option
to be tested in systematic, replicable in an international study that was
ways on the large-scale tests , but can planned for the third module of the
be promoted and potentially assessed Second International Technology in
at the classroom level . Education Study (SITES) . (SITES was
funded by the International Association
The role of domain knowledge: for the Evaluation of Educational
Another issue is the role of knowledge Achievement, or IEA. ) The framework
about topics and contexts required was intended to guide the develop-

124
 New technological literacies

DOMAIN

CONTENT PROBLEM-
Graphing tools Simulations
KNOWLEDGE Spreadsheets Visualisations SOLVING
Databases Modeling tools DEMANDS
Word processors
rich- lean Web browsers complex simple
Presentation tools

ICT TOOLS
Design products Access and organise

Critically evaluate Represent and transform

Communicate Plan
ICT
Collaborate
STRATEGIES Analyse and interpret

REASONING WITHIN DOMAIN

USING ICT STRATEGIES

Figure 1. Coordinated ICT assessment framework

ment of performance assessments of that could shape the coherent col-

ICT that could be used across a range lection of evidence across studies of
of technology use in school subjects students ’ abilities to use ICT in aca-
documented in IEA SITES Modules 1 demic domains . The cross- cutting
and 2 ( Kozma , 2003) . framework laid out the knowledge
and skills to be assessed . It served as
To these ends , a working group of the first component of an evidence-
international experts in ICT represent- centered assessment design for ICT
ing Chile , Finland , Norway, Singapore ( Mislevy and Haertel , 2006) . Figure 1
and the United States was formed . presents a model of the coordinated
The group aligned standards docu- ICT assessment framework .
ments that specified important tech-
nology proficiencies with those that The circle depicts the subject matter
focused on mathematics and science domains — the content and processes
(since NSF was the funding agency) of the disciplines of science and math-
and the role of technology within ematics addressed in the NSF project .
those domains . To create the coor- Other academic domains in social
dinated ICT assessment framework , science and the humanities were not

the descriptions and classifications of included , although the generic frame-

problem solving and inquiry from the work could be applied to domains
maths and science frameworks were other than maths and science . The
incorporated into the more general left side of the circle represents the
categories of information processing , declarative knowledge of the domain ,
knowledge management and com- which can vary from content-lean ,
munication categories in the technol- factual knowledge to content- rich ,
ogy proficiency frameworks . From schematic knowledge composed of
these frameworks , the project team interrelated concepts and principles
culled common categories of ICT use (Baxter and Glaser, 1998) . The right

125
 Chapter IV - Case studies

side of the circle represents the proc- chosen or required . This framework
ess dimension , in which problem -solv- was designed to focus on generaliz-
ing demands of an assessment can able ICT strategies , rather than on
range from simple , procedural knowl- discrete , often changing , features of
edge for routine problems to complex, technology tools .
strategic knowledge for nonroutine
problems . Within the problem space ,
New literacies assessment
learners use ICT strategies to inte-
grate technologies into the problem- designs
solving activities . The ICT strategies
include: taking advantage of the capa- The NSF project involved design of
bilities of technologies to understand prototype performance assessments
and plan how to approach a problem ; that the international study could use
accessing and organising information to test problem- based reasoning using
and relevant data ; representing and technology. The project used a modu-
transforming data and information ; lar design approach that aimed to :
analysing and interpreting information
•

and data ; critically evaluating the rele- provide common , credible , techni-
vance , credibility and appropriateness cally sound measures of standards
of information , data and conclusions ; related to technology use , reasoning
communicating ideas, findings and with information and communica-
arguments ; designing products within tion outcomes addressed in a wide
constraints ; and collaborating to solve range of technology programs and
complex problems and manage infor- classrooms ;
•

mation . These strategies align with apply and extend an assessment

current versions of 21st century skills . design framework with modular
components that could provide
The figure deliberately portrays these ‘templates’ or task models for new
ICT strategies as non -linear and itera- or modified assessments address-
tive . Thus , planning may be needed ing similar outcome areas ;
•

to find relevant digital information and provide preliminary evidence about

data at the outset of a task and again , the technical quality of the general
at a later stage of the task, to decide design approach and function of the
what to vary in the test of a model . prototype assessments .
Various technologies can support col-
laboration throughout the problem- The modular design was intended to
solving activities. support flexible reuse of component
tasks . First, the modules could be
Technology tools appear in the center based on an ICT strategy, technology
of the problem space in a ‘tool kit’ . tool , subject-matter of the problem or
Internet , productivity and specialised complexity level . Second , modules
tools such as simulations or visuali- could be independent of each other so
sations may be chosen to accomplish that they could be inserted or deleted
multiple ICT strategies . Factual and without disrupting the flow of an inves-
procedural knowledge required for tigation or problem-based assessment
operation of specific tools or classes task. Third , the modular approach would
of tools can vary according to the permit extraction of separate score
affordances of particular tools and reports for domain knowledge, strat-
the basic or more advanced features egies and/or technology use . These

126
 New technological literacies

Strategy
Module Sample questions/tasks ICT strategy Sample tools
component
1 Given data in text message Plan Analyse problem. Spreadsheet
of 4 years of hare and lynx strategies and Choose Table
population data, describe the procedures. appropriate tools. E-mail
problem . Collaborate Integrate others'
Given data for more years by to solve data.
collaborators , describe the problem .
problem.
2 Type in a search to find how hare Access Formulate a search Web browser
and lynx populations are related . information query. Table
Look through these three sites. and data. Conduct search . Search box
Take notes and cite sources. Organise Enter information Search results
Copy and paste information . information in table or notes . Web directory
Pick which search might be and data. Evaluate quality of Web pages
better. Critically search results. Table
Are these good search results? evaluate. Contribute Word
Send suggestions to collaborator. Collaborate. feedback. document
E-mail
3 Enter the 25 years of population Represent Display data in one Spreadsheet
data into a spreadsheet. and transform format, convert to Table
Create another way to look at the data and a different form . Graph
pattern . information . Record and read
What is the relationship in 2003 ? Analyse and data.
What trends do you see ? interpret data. Identify and
What do you predict will happen explain trends .
in 5 years? Make predictions.
4 Run the model with given Analyse data . Read graphs. Modelling tool
settings. Interpret data . Infer trends. Word
What are the populations in 2002 Make predictions. processor
and 2005? Explain
What do you predict will happen predictions .
in 2008?

Increase the lynx population.

What do you think will happen?
Run the model . Explain
5 Plan your recommendation and Plan Specify position Web form
presentation . argument. Identify relevant Word
Compose your presentation Communicate evidence. processor
using information and pictures findings and Present Tables
from websites , data. supported recommendation , Graphs
Present argument. argument. relevant data Graphics
and information Presentation
in coherent tool
argument.
6 Critique recommendation from Critically Critique position, E-mail
another team (with inaccurate evaluate evidence, support Word
data) by explaining if you agree arguments . explanation , processor
with the recommendation , the organisation .
appropriateness of their data and
information, their support for the
recommendation .

Table 1 : ICT assessment scenario: predator-prey

Problem: Parks are being overrun by hares. The governmentshould reintroduce lynx.
Science and math content: Familiar or given.

127
 Chapter IV - Case studies

affordances of the modular design for • Module 1 assesses ICT planning

ICT performance tasks would permit strategies through questions and
custom design and adaptive assess- tasks for analysing the problem by
ment. The next section presents exam- examining data on hare and lynx
ples of prototype ICT assessments of populations , while selecting from
and with technology that use the modu- a set of technology tools . Module
lar design approach . 1 assesses collaborative planning
through tasks and questions in which
the student uses e- mail to examine
Prototype tasks for
hare and lynx population data sent
assessing new literacies by virtual team members . Evidence
of skills in operating the technology
Assessment oftechnology: This first
tools is a by- product of students’ use
prototype was designed to illustrate
of the tools in the problem-solving
modules that could be developed for tasks .
the planned SITES 2 Module 3 optional
performance assessments . This proto- • Assessment of strategies for using
type , funded by the National Science technology to access and organ-
ise information is tested in Module
Foundation , was an assessment of
2 in a series of tasks in which the
technology, aimed to test student pro-
ficiencies using common Internet and student formulates a search query,
productivity tools to solve problem- gathers information and data from
based tasks drawing upon familiar sci- web pages and organises them in
ence and maths content. The proto- a table. Critical evaluation , tested
type was designed to be appropriate throughout the modules, is assessed
for administration to 13 -year- olds . The by questions on the credibility of
predator–prey ICT assessment pre- information from a web report pro-
sented a driving authentic problem: duced by a fur trading company and
‘Should lynx be reintroduced into a by questions on the effectiveness of
web search results .
Canadian park?’ This problem was an
example of predator–prey problems • Module 3 assesses the ICT strate-
that have been addressed in curricula gies for using technologies to rep-
ranging from the upper elementary to resent and transform information
university levels . For this prototype, and data . Questions and tasks ask
the science and mathematics required students to convert data sent in an
were well -taught, well-learned mate- e- mail text message by virtual col-
rial . Table 1 outlines the sequence of laborators to data on a spreadsheet
tasks . and then transform the data into a
graph .
Questions and tasks within modules in • Module 4 tests the ICT strategies
this prototype were designed to cap- for using technologies for analysis
ture student responses dynamically and interpretation of information
as students employed ICT strategies and data . Questions and tasks ask
to accomplish a subtask by using vari- students to read specified data pre-
ous technologies . First, the problem sented in tables and graphs and to
is presented and hypothetical student interpret trends .
team members from another school • Module 5 tests analysis and interpre-
who will be virtual collaborative part- tation by using a modelling tool that
ners are introduced . displays the pattern of hare and lynx

128
 New technological literacies

populations . Students answer ques- Assessments with technology :

tions about output of the model at solving complex science and math-
specified years, predict trends , and ematics problems using advanced
manipulate population values in the learning tools: A second assessment
model to test predictions . design goal in the NSF ICT assessment
• In Module 6 , uses of ICT strate- project was to draw on the coordinated
ICT assessment framework and the
gies and technologies for planning
a presentation and communicating modular design approach to fashion
findings and results are tested .
prototype performance assessments
for the secondary school level that
would tap transformative uses of learn-
Figures 2 to 9 illustrate the modules .
ing with technology in advanced sci-
The predator–prey modules were ence and mathematics (e.g. visualisa-
tions, modelling , specialised software).
designed to permit flexibility in inter-
The prototypes addressed assessment
national administrations in which
targets for science concepts , ICT strat-
the assessment would be a national
egies and the use of technology tools .
option . The modules such as using a The prototypes were designed to serve
spreadsheet or modelling tool could as classroom-level models for teach-
be removed if students had not had ers and evaluators to assess student
experience with these tools , but the learning at the secondary level in inno-
flow of the problem-solving task would vative technology-supported curricula
not be disrupted . in which students had the opportunity

Too many hares !

• Read the following passage.
Park rangers in Canada have observed that The park service hired Dr. Kloss at the Arctic
too many hares in the parks are causing Research Institute to investigate. Dr. Kloss wants
problems. They eat small plants that other teams of students made up of students from
animals depend on for food. Some park different schools to help. You'll be workingwith
rangers suggest that the government bring in two other students , Filo from York and Kari from
morelynx (which eat hares) and help reduce Ottawa.
thepopulation.
Forthis project you will use technology to solve
the problem should the government bring in more
lynx?

Figure 2 : Predator-prey ICT assessment — Problem introduction

129
 Chapter IV - Case studies

Collect information, take notes

and cite sources
Data / information to collect:

•Whendoes the hare population start to go down?

•What are reasons that can cause it to go down?
Canadian Lynx 回日

>>>
Canadian lynx forowe Mal

>> The lynx and hare

>>>Predator-prey cycles Canadian
Lynx
Lynx
canadensis

Thelynx,initsfourspecieshasbyfarthe largest rangeof

Key Facts
Roldes

anyofthefelidsand inasmuchinhabitsthe continentson

Page

either side oftheAtlanticOcean

. IntheNewWorld, the
Canadian lynx is foundinbothCanadaandAlaska, aswell
BodyLenghon 900-1100 asinsome ofthe northem states oftheUSA,where
changesinhabitatand overhunting havereducedits
Weight --10 extent. LynxfoundinNewfoundlandaresometimes
accreditedasasubspecies that ofLc.subolanus
LiterSize-24эчки
Asincommonwithlynx inEurope andAsiatheCanadian

Submit

Figure 3 : Predator-prey ICT assessment -Web search task

Access & Organize Information / Data

Analyze & Interpret – infer trends & patterns

Data from the last 4 years

Hereis the data for the number of hares in the parks over the last four years. Last
year(2002) there were about 95,000 hares. The year before that (2001) there were
about80,000. In 2000, there were 25,000. And in 1999, there were only about
1,000 hares.

Organize the data and describe the population trend.

Pick a tool to use:

Word processor Spreadsheet Presentation

2 Finished with task

Figure 4: Predator-prey ICT assessment ·Select a tool to organise information

130
 New technological literacies

Figure 5: Predator–prey ICT assessment — Organise data

Figure 6: Predator–prey ICT assessment — Transform data from table to graph

131
 Chapter IV - Case studies

Use a model to predict outcomes

Applet

Initial Number ofHares

low high

InitialNumber ofLynx

low high

setup go

29

time
(years)

Hare and Lynx Populations Pens

Many Hare
Lynx
population

Few

0.0 time(
years) 50.0

Figure 7 : Predator-prey ICT assessment Usea model

Create a presentation
Dr. Kloss would like you to prepare a short presentation (3 pages / slides) to the park service
with your recommendations what to do about about the hare and lynx populations.
Be sure your presentation is clearly organized and includes:
•A statement of the problem
•Your group's recommendation
•Information, data and explanations to support your recommendation

Presentation Word processor

2 Finished with task

Figure 8: Predator-prey ICT assessment Select a tool to develop presentation

132
 New technological literacies

to work with the types of technological technology use . Figure 9 presents a

tools used by professionals . screen shot of the module .

One prototype was designed to test Another prototype designed according

the ability of high school physics stu- to the modular design approach tested
dents to apply the laws of motion to a student model for secondary stu-
solve an authentic problem ( design- dents to solve an applied problem by
ing a motorway car crash barrier) with using a widely available commercial
a widely used commercial modelling visualisation tool , ArcView. The tar-
tool , ‘ Interactive Physics’ . The tar- geted knowledge and skills included :
geted knowledge and skills included : (a) science and maths knowledge ;
(a) physics concepts related to force, (b) inquiry skills for planning and con-
mass , acceleration and velocity; ducting investigations, analysing and
(b) ICT inquiry strategies for planning/ interpreting data and communicating
recommendations ; and (c) technology
design , conduct of investigations ( run-
ning the simulation ), analysis and inter- use. The task design involved pres-
entation of the problem ( Which states
pretation (of acceleration and velocity
meet requirements to apply for solar
graphs ), evaluation of possible design
solutions and communication of a
power funds?); accessing , analysing
and combining visualisations of dif-
recommendation ; and (c) technology ferent types of data (for solar energy) ;
proficiencies related to using the mod-
interpreting data; and presenting a
elling tool , graphing tool and presenta- recommendation . Evidence of student
tion tools . The task design consisted of learning consisted of scores for the
a series of modules in which students three outcome areas and their com-
planned their design , iteratively pre- ponents . Figure 10 presents a screen
dicted and tried out designs using the shot of the assessment .
simulation , interpreted results , evalu-
ated a proposed design and devel- The three prototypes developed in the
oped a presentation for their recom- NSF project described above used the
mended design . Evidence of student coordinated framework for the design
learning was provided by scores for of ICT assessments to illustrate how a
student work related to physics knowl- modular design approach can shape
edge, the component inquiry skills and assessments of new literacies that can

FreewayOverpass
EraseMeterValues

Run Reset

CrushBarrier InertialImpactBarriers

CrashZone

100metercrash zone

Figure 9: High school physics assessment using simulation software

133
 Chapter IV — Case studies

Figure 10: High school science and math assessment using visualisations

vary foregrounding of competencies in nology. These testing programs are

use of the technology tools, in use of capitalising on the capacities of tech-
21st century ICT cognitive strategies, nology to support logistical assess-
or domain knowledge and skills. The ment functions including test devel-
next section examines contemporary opment, delivery, adaptation, scoring
designs of technology-based assess- and reporting. A new generation of
ments and their potential for providing assessments, however, is attempting
evidence of student learning of new to move beyond logistical supports of
literacies advocated in 21st century testing by technology to reformulating
and ICT skill frameworks. task and item formats to test 21st cen-
tury thinking and reasoning processes
with technology in order to overcome
Towards the next many of the limitations of conventional
generation of assessments testing practices.
of new literacies
In 2006, the Programme for
A new generation of technology-ena- International Student Assessment
bled assessments is transforming how (PISA) conducted a pilot of computer-
testing is done and what gets tested based assessment in science which
(Quellmalz and Pellegrino, 2009). used animations and simulations of
An increasing number of large-scale phenomena such as energy .ow in a
tests are embracing testing by tech- nuclear reactor to test science skills

134
 New technological literacies

that could not be tested in the paper- Recommendations for 21st century ICT
based booklets. In 2009, PISAincluded assessments are turning from a pri-
electronic texts to test reading. Since mary emphasis on summative goals to
2005, the US state of Minnesota has methods for assessing new literacies
administered computer-based state within school curricula. Assessment
science tests in grades 5, 8 and 11. designs are seeking to harness tech-
These science tasks present anima- nology to measure understanding
tions and simulations of laboratory of complex and dynamic phenom-
experiments and phenomena such as ena that were previously difficult to
the water cycle. In the USA, in 2011, assess by conventional means. In the
the national assessment of educa- domains of reading and written com-
tional progress (NAEP) for writing, position, ICT tools such as web brows-
word processing and editing tools will ers, word processors, editing, drawing
be used in the computer-administered and multimedia programs can support
test for grade 8 and grade 12 students reading and writing processes. These
to compose essays. same tools can expand the cognitive
skills that can be assessed, including
The large-scale tests described above accessing and finding relevant infor-
are assessments of subject matter mation, integrating multiple sources of
knowledge and processing skills, i.e. information, planning, drafting, com-
assessments of learning with tech- position and revision.
nology. Data is not collected on how
well technologies are used, nor of the These assessments of learning with
number of 21st century skills used, technology can vary along a continuum
such as collaboration or multimedia from static to animated and dynamic
presentations. In fact, these subject displays of information, data and phe-
area tests are designed to minimize nomena and from static to interactive
the requirements for knowing how to ways for students to solve problems
operate particular technology tools. and enter responses (Koomen, 2006).
At the beginning of the continuum
Large-scale assessments of the new would fall technology-based assess-
technological literacies that directly ments by technology intended to repli-
test and report on the spectrum of 21st cate paper counterparts. Assessments
century ICT skills are not yet availa- that would fall at a midpoint on the
ble. A 2003 ICT feasibility test by PISA continuum may permit students to con-
was conducted with a small sample of struct tables and graphs or they may
students in Japan, Australia and the present animations of science experi-
USA. The study pilot was an assess- ments or phenomena, such as chemi-
ment of technology which tested a set cal reactions, for students to observe.
of ICT skills for access, management, Assessments presenting dynamic
integration and evaluation. Modules simulations that allow students to
included uses of web (select relevant interact by manipulating multiple vari-
reliable site, search), desktop (email, ables would be placed at the most
database) and e-learning (science transformative end of the continuum.
simulation) environments. Scored ICT Technology-enhanced assessments
proficiencies related to students’ abili- can offer the following benefits.
ties to correctly use the technologies.
A full-scale ICT assessment was not • Present authentic, rich, dynamic
funded by PISA. environments.

135
 Chapter IV - Case studies

Support access to collections of strategy use as reflected by informa-

information sources and expertise . tion selected , numbers of attempts and
• Present phenomena difficult or time allocation . Such work involves
impossible to observe and manipu- reconceptualizing assessment design
late in classrooms . and use and tying assessment more
•

Represent temporal , causal , directly to the processes and contexts

dynamic relationships ‘in action ’ . of learning and instruction .
• Allow multiple representations of
stimuli and their simultaneous inter- Assessments of new literacies at
actions (e.g. , data generated during the classroom level : The systematic ,
a process). direct assessment of new literacies
• Allow overlays of representations , in classrooms remains rare . Although
symbols . students may be taught to use common
• Allow student manipulations/investi- and advanced tools , teachers tend not
gations , multiple trials . to have specific technological literacy
• Allow student control of pacing , standards to meet nor testing methods
replay, reiterate . to gather evidence of student skill in
• Make student thinking and reason- using the technologies . Teachers are
ing processes visible . typically left on their own to figure out
•

Capture student responses during how to integrate technology into their

research , design , problem solving . curricula . The state of practice for
• Allow use or simulations of a range of assessing new literacies integrated
tools (Internet, productivity, domain- into instructional activities remains in
based) . its infancy.

Across the disciplines , technologies The advent of the 2012 NAEP

have expanded the phenomena that Tecnological Literacy probe will provide
can be investigated, the nature of argu- as set of examples of new literacies
mentation and the use of evidence . in areas of Technology and Society,
The area of science assessment is per- Engineering Design and Systems ,
haps leading the way in exploring the and ICT. In the USA , assessments of
presentation and interpretation of com- 21st century skills and technological
plex, multi-faceted problem types and literacy standards are required for all
assessment approaches. Technologies students by grade 8; however, states
are being used to represent domains , may report achievement on a state
systems , models and data, and their test or from school reports . School
manipulation , in ways that previously reports may be based on teacher
were not possible. Dynamic models reports that may, in turn , be based
of ecosystems or molecular structures on questionnaires or rubrics judging
help scientists visualise and communi- students’ use of ICT in project work.
cate complex interactions. This move Most teachers do not have access to
from static to dynamic models has classroom assessments of 21st cen-
changed the nature of inquiry among tury skills or professional development
professionals and the way that aca- opportunities to construct their own .
demic disciplines can be taught and Moreover, the lack of technical qual-
tested . Moreover, the computer's ability of teacher-made and commercially
ity to capture student inputs permits developed classroom assessments is
collecting evidence of processes such well documented (Wilson and Sloan ,
as problem-solving sequences and 2000) . Even more of a problem is the

136
 New technological literacies

lack of clarity for teachers on how to reflection and action , and teachers
monitor student progression on the actually making adjustments to their
development of 21st century skills , not instruction based on the assessment
only tool use , but ways to think and results ( 1 ) . Technologies are well-
reason with the tools . Teachers need suited to supporting many of the data
formative assessment tools for these collection , complex analysis and indi-
purposes . vidualised feedback and scaffolding
features needed for the formative use
The UK ICT Stage 3 assessment pro- of assessment (2) . However, for the
gramme represented an attempt to most part, technology-based assess-
provide teachers with assessments ments that provide students and
to check and monitor their students’ teachers with feedback on perform-
operation of ICT tools ( National ance on the subject matter tasks and
Assessment Agency, 2008). In a 2007 items do not also provide feedback on
pilot of an ICT test, modules on use of students ’ use of embedded technol-
websites, databases , graphs , images ogy tools such as graphs , tables or
and presentations were administered visualisations.
and teachers received feedback on
The next section describes assess-
where students’ proficiencies fell on
a continuum of operational tasks . ments being developed by WestEd
Teachers were then expected to help in a SimScientists project funded
their students become more proficient by the National Science Foundation
with the ICT tools. A major challenge (Quellmalz, Timms and Buckley,
reported from the 2007 pilot was that 2009) . The project is studying the use
teachers viewed the time required to of science simulations for end - of- unit,
prepare students to take the exams summative , benchmark purposes and
for curriculum embedded formative
as time taken away from their regular
instruction . This finding supports the purposes . The project assesses com-
need for assessments of 21st century plex science learning with technology.
ICT strategies and operations that are Students use a range of technology
designed as assessments of learning tools and inquiry skills to investigate
with technology. science problems that relate to under-
standing increasingly complex levels
For direct assessments of new lit- of grade-appropriate models of sci-
eracies knowledge and strategies to ence systems . Assessment targets
are integrated knowledge about a sci-
become integrated into classroom
formative assessment practices , new ence system and inquiry skills aligned
literacies assessments must be sys- with 21st century skills such as analy-
sis , evaluation and communication .
tematically designed and subjected to
technical quality screening . The form- Although the project does not directly
ative use of assessment has been assess students’ use of technology
tools or their abilities to select appro-
repeatedly shown to significantly ben-
priate tools for a task , this paper offers
efit student achievement ( Black and
Wiliam , 1998). Such effects depend
on several classroom practice factors , ( 1 ) Black, P. , Harrison, C., Lee, C., Marshall, B.,
and William , D. ( 2004) . Phi Delta Kappan 86 , 8 .
including alignment of assessments
(2 ) Brown , J., Hinze, S., and Pellegrino,
with standards and frameworks , qual-
J.W. ( 2008 ). In : 21st century education,
ity of the feedback provided to stu- T. Good (ed .) , Sage, Thousand Oaks . CA, Vol . 2,
dents , involvement of students in self- Chap . 77 , 245–255 .

137
 Chapter IV — Case studies

suggestions for how such assess- and algae by observing animations of

ments could be augmented with the interactions between and among
tasks, items and feedback to promote organisms in the lake. The assess-
21st century ICT strategies such as ments then present sets of simulation-
tool selection and use or collaborative based tasks and items that focus on
research. By providing formative feed- students’ understanding of the emer-
back and further scaffolding on the gent behaviours of the dynamic eco-
use of technologies as they are used system by conducting investigations
during subject matter problem solving, with the simulation to predict, observe
the assessments can encompass new and explain what happens to popula-
literacies and lessen teachers’ percep- tion levels when numbers of particular
tions that technological fluency poses organisms are varied. In a culminating
additional, irrelevant burdens. task, students write a report of their
findings about the lake ecosystem.
Figure 10 presents a screen shot
of tasks in a SimScientists assess- In a companion set of curriculum
ment designed to provide evidence embedded assessments, the techno-
of middle school students’ under- logical infrastructure identifies types
standing of ecosystems and inquiry of errors and follows up with feed-
practices. Students are presented back and graduated coaching. In the
with the overarching problem of pre- assessment screen shown, feedback
paring a presentation and report to is provided if the student’s investiga-
describe the ecology of a lake for an tions saved do not show organisms
interpretive centre. They investigate existing for the specified amount of
the roles and relationships of the fish time. Levels of feedback and coaching

Figure 11: SimScientists assessment screenshot — Using a model to conduct investigations about population
dynamics

138
 New technological literacies

progress from identifying that an error design represent a best practice in

has occurred and asking the student to the field of assessment ( Mislevy and
try again , to showing results of inves- Haertel , 2008) . Assessments of new
tigations that met the specifications . In literacies , then , should specify an
the task shown , additional evidence assessment framework with these
could be collected on technological three components . The framework
literacy. The system could score how should identify the 21st century ICT
well students are able to vary values domain knowledge and processes
for the number of organisms while that define the constructs to be meas-
using the simulation , use the graph ured . These would include students ’
inspector to examine the graphs and declarative knowledge about technol-
tables , and save and enlarge views of
ogy tools , such as their purposes and
graphs of multiple experiments . Such
additions would allow assessment of features , and students ’ procedural
technology, i.e. students’ understand- knowledge, or proficiency for operat-
ing of how and when to use the tech- ing particular technology tools . The
nology features in the simulations , as 21st century ICT domain would also
well as assessment of learning out- define strategies such as information
comes with technology. processing , knowledge management,
problem solving and communication ;
each of these are strategies that indi-
Developing new literacies viduals must draw on to make use of
assessments with technical technologies to address significant,
quality recurring problems in general , applied
contexts and in academic disciplines .
The report, Knowing what students For new literacy assessments aiming
know (Pellegrino et al . , 2001 ), sum- to measure technology use and also
marises the implications for assess- to measure academic knowledge and
ment of decades of research in the sci- skills , the framework would need to
ences of cognition and psychometrics. specify, test and report separately
The report characterises the assess- 21st century thinking and reasoning
ment of the knowledge and skills that strategies including collaboration and
individuals possess as a process of communication , use of desk top or
reasoning from evidence . The reason- e-learning tools and domain knowl-
ing required to make inferences about
edge and processes.
an individual’s knowledge and skills is
best developed through the specifica-
The second component of evidence-
tion of an assessment argument that
centred design for the assessment
connects three components including:
(a) the specific knowledge and skill of new literacies would then specify
the features of assessment tasks and
constructs in the particular domain (s)
items that would elicit observations of
to be measured; (b) the features of
assessment activities that will require achievement of the 21st century ICT
examinees to use that knowledge and and domain knowledge and skills of
those skills ; and (c) the data derived interest. The types of assessment tasks
from student responses that will count and items would represent the types of
as evidence of the level of knowledge fundamental contexts , problems and
and skills demonstrated . These three activities in which examinees use tech-
components of evidence -centered nology in school and applied settings.

139
 Chapter IV — Case studies

The third component of evidence-cen- coherence will increase the validity of

tred design would specify: (a) the evi- inferences from the assessments and
dence of student learning that needs to increase the likelihood that informa-
be extracted from student responses tion about student performance can be
to the assessment tasks and items; used to describe and promote skilled
(b) how the responses will be scored; use of technologies in significant aca-
and (c) the details of the statistical demic and applied tasks.
models needed to calibrate items and
create proficiency estimates and reports
of students’ knowledge and skills.
Summary
The development of assessments of
By shaping large-scale and classroom new literacies is in its early stages.
assessments according to this princi- Multiple frameworks, contexts and
pled assessment design approach, new points of view both invigorate and com-
literacies assessments can initiate the plicate design efforts. Educators differ
process of documenting technical qual- as to whether or not technology should
ity by describing a systematic design be assessed as a distinct domain or
process. Further technical quality evi- should be integrated into assessments
dence gathered during cognitive labs within academic disciplines (Quellmalz
of students thinking aloud as they solve and Kozma, 2003). Expert panels need
assessment tasks and items would to reach consensus on the knowledge
provide evidence of construct validity. and skills that constitute new literacy
The psychometric data from analyses skills and how those skills align with the
of student performance on tasks and knowledge and skills in subject matter
items would provide further evidence of frameworks and standards. Research
technical quality. Since the process of is needed on how to design tasks that
documenting technical quality requires integrate the use of technologies into
considerable expertise, some of the subject matter tests and how to directly
assessment resources made available test, extract and report the skill with
to teachers for classroom formative which technologies are operated and
assessment should have such proc- strategically used. Experts need to
esses and data documented. identify the features and functions of
technologies that are relevant to aca-
demic and 21st century constructs of
Multilevel, balanced interest as well as those features that
assessment systems need to be controlled because they
interfere with performance on tar-
Discussions of the need for assess- geted knowledge and skills. Studies
ments of 21st century ICT competen- are needed to examine student per-
cies increasingly recognise the need formance on items and tasks in which
for the articulation of large-scale and technology is assumed to enhance or
classroom-based assessments. A key hinder performance.
feature in creating a multi-level, bal-
anced system is the use of common Work with technology-based assess-
design specifications that can operate ments that scaffold learning and
across classroom, district, state and performance in complex tasks while
national levels (Quellmalz and Moody, adapting to student responses is also
2004). Deliberately designing assess- in its early stages. Research on ways
ments at different levels to assure their that these adaptive modules can serve

140
 New technological literacies

as formative and summative assess- ICT assessment frameworks , princi-

ments is greatly needed . Changes pled assessment designs , exemplary
in scaffolding could be features that assessments and evidence of their
are varied in the assessment tasks . validity. In the 21st century students
Research would examine how changes will need to become facile users of
in the scaffolding levels of assessment technologies , and 21st century edu-
task designs relate to student perform- cators will need to be able to define ,
ance . Such efforts would provide the target, measure and promote students’
field with interdisciplinary 21st century progress on these new literacies .

References

Baxter, G. P. , and Glaser, R. (1998). ‘The cognitive complexity of science per-

formance assessments ’ , Educational Measurement: Issues and Practice, Vol .
17 , No 3 , 37–45 .

Black, P. and Wiliam , D. ( 1998) . Inside the black box: raising standards through
classroom assessment. London : King’s College .

Bennett, R. E. , Jenkins , F. , Persky, H. and Weiss, A. (2003). Assessing complex

problem solving performances , Assessment in Education, Vol . 10, 347–373 .

Burns , T. C. and Ungerleider, C. S. (2002) . ‘ Information and communication

technologies in elementary and secondary education’ , International Journal of
Educational Policy, Research, and Practice, Vol . 3 , No 4 , 27–54 .

Crawford , V. and Toyama, Y. (2002). Assessment of student technology pro-

ficiency and an analysis of the need for technology proficiency assessments:
a review of state approaches. Paper presented at the annual meeting of the
American Educational Research Association , New Orleans , LA .

International Society for Technology in Education (ISTE) . (2007) . National edu-

cational technology standards for students: connecting curriculum and technol-
ogy. Eugene, OR.
International Technology in Education Association (ITEA). (2000). Standards for
technological literacy.

Koomen , M. (2006). The development and implementation of a computer-based

assessment ofscience literacy in PISA 2006. Paper presented at the annual meet-
ing of the American Educational Research Association , San Francisco , CA.
Kozma, R. (2003). Technology, innovation, and educational change: a global
perspective . Eugene , OR: International Society for Technology in Education .

Kozma, R. , McGhee , R. , Quellmalz , E. and Zalles , D. (2004). ‘ Closing the

digital divide : evaluation of world links ’ , International Journal of Educational
Development, Vol . 24 , No 4 , 361–381 .

141
 Chapter IV - Case studies

Law, N. , Pelgrum , W. J. and Plomp , T. (eds) (2008) . Pedagogy and ICT use in
schools around the world: findings from the IEA SITES 2006 study. Hong Kong :
Comparative Education Research Center.

Mislevy, R. J. and Haertel , G. D. (2006). ‘Implications of evidence-centred design

for educational testing’ , Educational Measurement: Issues and Practice, Vol . 25,
No 4 , 6–20 .

National Assessment Agency (2008). Report on the 2007 key stage ICT test
pilot . London : Qualifications and Curriculum Authority.

Partnership for 21st Century Skills (2005). Assessment of 21st century skills:
the current landscape. Tucson , AZ. Available at: http ://www.21stcenturyskills .
org/images/stories/otherdocs/Assessment_Landscape.pdff ..
Pellegrino , J. , Chudowsky, N. and Glaser, R. (2001 ) . Knowing what students
know: the science and design of educational assessment. Washington , DC :
National Academy Press .

Quellmalz , E. S. and Haertel , G. D. (2008). ‘Assessing new literacies in science

and mathematics ’ , in : D. J. Leu , Jr. , J. Coiro , M. Knowbel and C. Lankshear
(eds ) , Handbook of research on new literacies. Mahwah , NJ : Erlbaum .
Quellmalz, E. S. and Moody, Mark (2004) . Models for multi-level state science
assessment systems. Report commissioned by the National Research Council
Committee on Test Design for K- 12 Science Achievement.
Quellmalz, E. S. and Pellegrino, J. W. (2009). ‘Technology and testing’ . Science,
Vol . 323 , 75–79 .

Venezky, R. L. and Davis , C. (2002) . Quo vademus? The transformation of

schooling in a networked world. Paris : OECD . Available at http://www.oecd.org/
dataoecd/48/20/2073054.pdf

Wilson , M. , and Sloan , K. (2000) . ‘ From principles to practice : an embed-

ded assessment system’ , Applied Measurement in Education, Vol . 13 , No 2,
181–208 .

142
The impact of ICT in education policies
on teacher practices and student
outcomes in Hong Kong
Nancy Law, Yeung Lee and H. K. Yuen
University of Hong Kong

1. Introduction ducted as part of the evaluation of the

effectiveness of the implementation of
A major theme running through edu- the first and second ICT in education
cation policy recommendations (Catts masterplans (Law et al., 2007).
and Lau, 2008; CERI, 2001; European
Council, 2000; OECD, 2005; Unesco, This paper begins with an overview
2008) and policy initiatives (US of the three ‘IT in education’ strate-
Department of Education, 1996; CDC, gies (EMB, 1998, 2004; EDB, 2008)
2001; Singapore MOE, 1997, 1998) in launched in Hong Kong to highlight
many parts of the world is the impor- the policy foci and the changes in em-
tance for education to prepare its citi- phases that have taken place over
zenry for life in the 21st century. This time. It then summarises the changes in
has brought about changes in the teaching practice and ICT use in Hong
school curriculum as well as plans for Kong schools between 1998 and 2006
the integration of IT (1) in the teach- based on findings from international
ing and learning process to foster the comparative studies of ICT in educa-
development of 21st century skills in tion. The design and key results from
students. Is there evidence that these the evaluation study of students’ infor-
education policy initiatives impact on mation literacy skills is then described.
The paper ends with a discussion of the
how teaching and learning take place
in schools, and even more importantly, links between education policy, teach-
on students’ learning outcomes? In ing practice and students’ outcomes as
revealed by the findings.
this paper, we explore this question
in the context of the policy initiatives
that have taken place in Hong Kong 2. ICT in education policies
since 1998, when the first ‘IT in educa-
tion’ masterplan was launched (EMB,
in Hong Kong since 1998
1998), drawing on the data that have The Hong Kong Government announ-
been collected over the period 1998 ced its first ICT in education policy in
to 2006 from international and local November 1998 with its ‘Information
evaluation studies, with a particular technology for learning in a new era:
focus on an evaluation study of stu- Five-year strategy’ (EMB, 1998), as
dents’ information literacy skills con- an integral part of the policy goal for
Hong Kong to become ‘a leader, not
(1) IT and ICT are used interchangeably to refer a follower, in the information world
to information and communication technology. of tomorrow’, which was a statement

143
 Chapter IV - Case studies

made in the inaugural policy address of a knowledge-based society was

of Mr Tung Chee Hwa , the first Chief launched (EC, 2000). This curriculum
Executive after the return of Hong reform had a major impact on the for-
Kong’s sovereignty to China in 1997 mulation of the second ‘ IT in education ’
(EMB, 1998; p. i). The vision of this first policy — Empowering Learning and
policy was to help students develop an Teaching with Information Technology
understanding of the pervasive impact (EMB, 2004). This document formu-
of ICT on their daily lives and society as lated the goal ‘to transform school
a whole, as well as higher order think- education from a largely teacher-cen-
ing skills and abilities to seek, evalu- tred approach to a more interactive
ate, organise and present information . and learner-centred approach ’ (EDB ,
The document indicates the need for 2004, p . i) as the ‘paradigm shift’
schools to undergo a paradigm shift targeted .
for the policy to be implemented suc-
cessfully, though it does not elaborate The vision of this second policy was
on the nature of the shift. It highlights to encourage the effective use of ICT
four important missions in order to as a tool for enhancing learning and
achieve this vision . teaching to ‘ prepare the younger gen-
eration for the information age, turning
1. Access and connectivity — to pro- schools into dynamic and interactive
vide students and teachers with learning institutions , and fostering col-
adequate and equitable access to laboration among schools, parents and
IT facilities and access to informa- the community’ ( EDB , 2004 , p. 10).
tion worldwide . The document used a somewhat dif-
2. Teacher enablement to assist ferent rhetorical language . Instead
teachers ’ migration to the new of missions , this document identified
teaching mode . seven strategic goals :
3. Curriculum and resource support —
to meet the target of having 25 % of 1. empowering learners with ICT;
the school curriculum taught with 2. empowering teachers with ICT;
the support of IT. 3. enhancing school leadership for
4. Fostering a community-wide cul- the knowledge age ;
ture to coordinate all stakehold- 4. enriching digital resources for
ers within and outside the school learning;
sector (school management, teach- 5. improving ICT infrastructure and
ers , students , parents , the busi- pioneering pedagogy using ICT;
ness sector and other community 6. providing continuous research and
bodies) to take up their new roles development ;
in ' IT in education ' in a collabora- 7. promoting community-wide support
tive manner in implementing the and community building .
policy.
These seven goals have a much
It is important to note that it was only stronger educational focus and reflect
in 2000 , two years after the launch different priorities and a more compre-
of the first five-year strategy, that the hensive set of strategies compared
comprehensive curriculum reform ini- with the missions contained in the first
tiative to renew the school curriculum policy. Empowering learning is identi-
with the goal of preparing the younger fied as the policy goal while the other
generation for meeting the challenges six are strategic goals . There is an

144
 Teacher practices and student outcomes

underpinning assumption in this docu- This third policy is clearly a turnaround

ment that the process of IT implemen- in the developmental direction taken
tation involves innovation , the nature by the first two . There is avoidance
of which is not only technological , of any indication that there are value
but also pedagogical. It is within this judgments in deciding how and what
framework that enhancing school technology is used and that the vision
leadership such that principals and leadership of the school matters .
and key personnel in schools under- It is a policy document in name with-
stand better the nature and process out having to play the role of a policy
of change required and continual without having to set a policy directive ,
research and development were given with the least possibility of stimulating
important strategic considerations in any debate or controversy. This policy
this second policy. was also released with an extremely
low profile . There was no formal
The second policy was planned to pro- launch and no media publicity. It is not
vide strategic guidance for three years possible to pinpoint what might have
in view of the fluidity in the technology caused such change and discontinuity
and education arenas . The third ‘ IT in in policy. However, there was a major
education ’ policy document -

'Right change in the top-level leadership in

technology at the right time for the right the Education Bureau at the time this
task' was released in 2008 ( EDB , policy was drafted and approved , and
2008 ) . As indicated by the title , IT is the key people who led the curriculum
perceived as purely instrumental in this reform launched in 2000 had stepped
document ; it does not see the need to down .

identify what is ‘right’ ; and the focus is

at the ‘task’ level rather than at the level
of an overarching curriculum/educa-
3. Teaching practice and
tional goal . Instead of identifying mis- ICT use in Hong Kong
sions ( as in the first policy) or goals ( as schools ( 1998 to 2006)
in the second policy) , this third policy
identified six strategic actions . Hong Kong took part in all three
modules of the Second Information
1. Provide a depository of curricu- Technology in Education Study
lum -based teaching modules with (SITES ) conducted under the aus-
appropriate digital resources. pices of the International Association
2. Continue to sharpen teachers ' ICT for the Evaluation of Educational
pedagogical skills . Achievement (IEA) . The first module,
3. Assist schools in drawing up and SITES- M1 (study homepage at http ://
implementing school- based ICT in www.mscp.edte.utwente.nl/sitesm1 ) ,
education development plans . focused on describing the status of
4. Enable schools to maintain effec- ICT and its use in schools through a
tive ICT facilities . survey of principals and technology
5. Strengthen technical support to coordinators , with data collection con-
schools and teachers . ducted at the end of 1998. Details of
6. Collaborate with non - governmen- the design and findings from this study
tal organisations to raise informa- are reported in Pelgrum and Anderson
tion literacy of parents and launch (1999) . This study collected informa-
parental guidance programmes on tion on the percentage of schools
e -learning at home . having ICT available for use for

145
 Chapter IV - Case studies

instructional purposes within formal or study are reported in Law, Pelgrum

informal educational settings , as well and Plomp (2006) .
as the extent to which principals per-
ceive ‘emergent’ practices in teach- As the first ‘ IT in education ’ strategy
ing and learning were present in their in Hong Kong was only launched in
schools . Emergent practices were November 1998, computers were not
defined as those practices designed used much for instructional purposes
towards developing students’ lifelong except for the teaching of computing-
learning abilities. These are generally related subjects in the curriculum .
Data collection for SITES - M1 was
more student-centered , open - ended
conducted at the end of 1998. The
learning and teaching activities with
characteristics not commonly found in student–computer ratios in primary
traditional classrooms . These ‘ emer- and secondary schools in Hong Kong
gent’ characteristics include the fol- were 53.3 and 35.7 respectively,
which were rather low levels of hard-
lowing , and namely that students:
ware provisions among the participat-
• develop abilities to undertake inde- ing countries at the time (Pelgrum and
pendent learning ; Anderson , 1999). Use of computers for
• learn to search for, process and instructional purposes in non-comput-
present information ; ing subjects was extremely rare . The
• SITES 2006 teacher survey results
are largely responsible for control-
showed 70 % of mathematics teach-
ling their own learning progress ; ers and 82 % of science teachers in
• learn and/or work during lessons at
Hong Kong reported having used ICT
their own pace ;
• with the sampled grade 8 classes that
are involved in cooperative and/or
they taught in that school year, which
project -based learning;
• determine for themselves when to
was among the highest percentage
take a test .
reported in the participating countries .
This finding indicates that in terms of
classroom adoption , the government
The third SITES module , SITES 2006 ,
strategies have achieved noticeable
was designed as a survey of schools success .
and teachers to examine the kinds of
pedagogical practices adopted in dif- Obviously, use is not the only criterion
ferent countries and the use of ICT in
for gauging success in policy imple-
them . In this module, the principals mentation . If ICT use were to support
were also asked the same question on students ’ development of 21st century
their perception of the extent to which skills , it matters whether learning was
emergent practices were present in still organised as traditional teacher-
their schools . The mathematics teach-
centered instruction or lifelong learn-
ers and science teachers surveyed in ing oriented as characterised above ,
this study were also asked about the and how ICT was actually used in
frequency with which different kinds of classroom settings . The SITES 2006
teaching and learning activities (tradi- teacher survey results indicate that
tional as well as lifelong learning ori- the pedagogical orientation of Hong
ented ones) took place in their class- Kong teachers was among the most
rooms and whether ICT was used in traditional among the 22 participat-
those activities . Details of the design ing educational systems ( Law and
and findings from the SITES 2006 Chow, 2008a ) . Further, the extent of

146
 Teacher practices and student outcomes

pedagogical deployment of ICT was of the second ‘IT in education’ policy

slightly greater for traditional peda- (EMB , 2004). The goal was to find out
gogical activities than lifelong learning whether students were able to make
oriented ones . On the other hand , the effective use of ICT to tackle learn-
strong traditional pedagogical orienta- ing tasks in the school curriculum at
tion should not be interpreted as one a level that is not normally achievable
of the policy outcomes. In fact, the without the appropriate use of tech-
SITES- M1 findings show that princi- nology. In commissioning this project,
pals in Asian countries , including Hong the EDB was interested in methodo-
Kong , generally reported much lower logical innovation in assessment
levels of presence of emerging prac- that will assess not only technical
tice in their schools compared with operational skills , but also students ’
their counterparts in other participat- problem-solving and lifelong learning
ing countries (Pelgrum and Anderson , skills . Hence the focus was less on the
1999) . The results from the principal psychometric qualities of the evalua-
survey in SITES 2006 show a remark- tion indicators but more on exploring
able ‘swing ’ in the percentage of prin- new ways of assessing new kinds of
cipals reporting a lot of presence of outcomes . In this section , we will elab-
emerging practice compared with the orate on the conceptual framework
same statistic reported in 1998 taken in this study with respect to IL,
with countries like Norway, Slovenia the key principles underpinning the
and Denmark reporting big decreases design of the assessment tasks and
while big increases were observed a brief description of the assessment
in Asian education systems such as instruments , the technology platform
Hong Kong, Japan , Thailand and used to conduct the assessment and
Singapore , and in some others such the sampling design for the study.
as Israel and Italy. Findings from the
two SITES studies seem to indicate 4.1. The conceptual framework
that there has been a move towards
In the curriculum reform document
more emerging , lifelong learning ori- launched in 2000 (EC , 2000) , IL is
ented pedagogical practices in Hong identified as one of the important skills
Kong classrooms over the period 1998 for the 21st century. In this study, the
to 2006 , though practices as a whole concept of IL encompasses much more
are still very traditional because of the than simply technical competence
cultural and historical background of and includes the cognitive abilities to
the schools .
identify and address various informa-
tion needs , critically evaluate informa-
4. Assessing students’ tion and apply the learning gained in
the solution of real -life open -ended
information literacy problems . Furthermore , just as prob-
skills Research design
—
lem solving requires not only generic
‘problem-solving skills’ but also exper-
In 2006 , the Centre for Information tise in the relevant content knowledge
Technology in Education of the as well as in the selection and use of
University of Hong Kong (CITE) was tools appropriate to the problem con-
commissioned by the Education text, IL is also subject-matter depend-
Bureau (EDB) to conduct a study ent. It is considered entirely possible
on students’ information literacy (IL) that the level of IL achievement of a
skills as part of the overall evaluation student may be different in different

147
 Chapter IV - Case studies

proficiency
Technical
inter l
activearni
IT for e ng

Mathematics education

Information Chinese language Literacy

education

Science education
proficiency
Cognitive

c
ri ll
ne
ITge ski

Figure 1 : Conceptual framework of IL in the evaluation study

subject domains . Hence , the assess- day life at school , at home and at work .
ment of IL should also take account Literacy, numeracy, problem-solving
of the domain context. Figure 1 is a and spatial/visual literacy demonstrate
diagrammatic representation of the these proficiencies. Technical profi-
conceptual framework underpinning ciency refers to basic knowledge of
this study on how IL develops in the hardware, software applications, net-
context of learning within school cur- works and elements of digital technol-
riculum subjects . ogy. These proficiencies are developed
through acquiring generic technical IT
In this framework, IL encompasses skills and applying them for interac-
both cognitive and technical profi- tive learning within the corresponding
ciency. Cognitive proficiency refers to subject learning contexts in everyday
the desired foundation skills of every- learning and teaching practices.

148
 Teacher practices and student outcomes

4.2. Identifying indicators ther developed a rubric with four levels

of performance: novice, basic, profi-
for assessing students’ cient and advanced. Rubrics are scales
information literacy of performance that can be used to
In order to identify appropriate indi- judge the quality of students’ perform-
cators for evaluating the impact of ance based on the descriptive criteria
ICT in the learning of specific sub- provided (Popham, 2003). Rubrics
ject disciplines, several major frame- are considered appropriate for use in
works developed in different countries this study as they can be used across
for the assessment of ICT literacy a broad range of subjects in assess-
have been carefully reviewed (EMB, ing both the process and product of
2005; ETS, 2003; MCEETYA, 2005). students’ learning (Moskal, 2000).
Somewhat different terminologies are Moreover, in assessing complex com-
used in these documents, but the set petences, rubrics providing specific
of skills these refer to are largely simi- objective criteria for different levels of
lar. Further, these all adopt a process- performance offer a way to provide the
driven approach in identifying indica- desired validity in the grading process
tors for IL. Two important features of (Morrison and Ross, 1998; Wiggins,
the ETS framework make it most ame- 1998). The IL rubric developed in the
nable for our adoption in the present present study is modified from NCREL
study: its focus on IL as exhibited in (2003), which has been validated by
complex tasks that resemble real-life Lee (2009) for use in assessing stu-
situations, and the fact that it was dents’ IL outcomes as indicated by their
designed for use in online assess- performance in learning activities in the
ment. Table 1 presents the details of classroom and through their authentic
the seven dimensions of IL compe- learning products. Some examples of
tence in the ETS framework. the rubrics and their application in the
context of specific tasks in the PAs
In implementing the IL framework for developed in this study will be given
assessing students’ outcomes, we fur- later in this paper.

Define Using ICT tools to identify and appropriately represent information needs

Access Collecting and/or retrieving information in digital environments

Manage Using ICT tools to apply an existing organisational or classification scheme to in-
formation

Integrate Interpreting and representing information, such as by using ICT tools to syn-
thesise, summarise, compare and contrast information from multiple sources

Create Adapting, applying, designing or inventing information in ICT environments

Communicate Communicating information properly in its context (audience and media) in

ICT environments

Evaluate Judging the degree to which the information satisfies the needs of the task
in ICT environments, including determining authority, bias and timeliness of
materials

Table 1: The seven dimensions of IL in the ETS framework adopted in this study (Source: ETS, 2003, p. 18)

149
 Chapter IV - Case studies

The number of tasks that assess

4.3. The performance
achievement for each of the dimen-
assessment tasks and their
sions may also vary across the dif-
design considerations ferent PAs , depending on the sub-
The evaluation study was to be con- ject disciplines with respect to their
ducted at two levels , grade 5 and subject nature .
grade 8 , with three sets of online per- • For each PA, general guidelines
formance assessment (PA) tasks at will be given at the beginning of
each level : technical IL , mathemati- the assessment to the students for

cal IL and Chinese language IL at answering the questions . Besides ,

grade 5 and technical IL , science IL the approximate completion time for
and language IL at grade 8. As it was each main question is indicated at
expected that students’ technical IL the end of the question in each PA .
competence might differ widely for
students within the same educational
4.4. Example performance
level , it was decided that the techni-
assessment items
cal IL assessment tasks at both levels
should be the same to allow for com- illustrating the IL dimension
parison across the two age groups . they assess
Hence , a total of five sets of online
Some examples of assessment items
performance assessment (PA) tasks, drawn from the technical PA, math-
answer keys and scoring rubrics were
ematic PA and science PA are given
developed for this study.
below in this section to illustrate how
the dimensions of IL are assessed in
Each set of PA was designed accord-
the different subject areas.
ing to the following criteria .

• The contexts for the tasks within 4.5. Developing and using
each PA are relevant to students ’
rubrics to assess students’
daily life experiences and hence
present authentic scenarios . performance
• With the exception of the technical PA, As described in Section 4.2 , we have
the PAs were designed to be relevant developed for each IL dimension a
for and appropriate to the curriculum generic set of assessment rubrics (i.e.
at the respective subject and grade descriptive criteria) for identifying per-
levels. formance at the four different levels :
• Each PA was designed to be com- novice , basic, proficient and advanced .
pleted in 45 minutes . Based on these generic rubrics , a set
• The full score for each PA was 50 . of task-specific scoring rubrics was
• The score for each question was developed for each assessment item in
approximately proportional to the each of the PAs . Table 2 presents the
time allocation for its completion . scoring rubric for item 3.1 in the science
• Each PA was designed such that PA (see Figure 6). The item asked stu-
the totality of tasks within the PA will dents to construct a classification dia-
provide assessment on all the seven gram for a set of plants and animals .
IL dimensions . However, the levels There are two IL dimensions involved
of achievement required for satis- for the satisfactory completion of this
factory task completion may differ task: manage (apply an existing organ-
across the different IL dimensions . isational or classification scheme for

150
 Teacher practices and student outcomes

An item to assess the

Planning a trip 'define' dimension. Figure 2
shows an item in the
technical PA designed to
assess students' ability
to define their information
needs . Here , students
are asked to plan a trip
for their grandfather and
grandmother to visit Hong
You and your classmates are asked to form a group of Kong . It asked the students
three to do a project about ' Planning a trip for your to define appropriate
grandfather and grandmother' . It is the first time that keywords for searching
your grandfather and grandmother visit Hong Kong and the ' discover Hong Kong'
they would like to go to some special and distinctive website . The assessment
scenic spots in Hong Kong. You and your groupmates
are required to suggest 2 scenic spots in Hong Kong and criteria are related to
make a presentation for the whole class to present why whether students can
you have chosen these scenic spots . identify the appropriate
Question 1.2 keywords or not .
What are the keywords which you have used for you
searching?

Figure 2: An item in the technical PA designed to assess the 'define' dimension

An item to assess the

A day trip to the Hong Kong 'access ' dimension. Figure

3 shows an item in the
Ocean Park
mathematics PA designed
to assess students' ability
to access information
effectively. In this item ,
students are asked to use
a search engine to retrieve
correct fares for adults and
children to visit the Hong
Kong Ocean Park. The
assessment criteria are
Question 1 related to whether students
John and his family would like to visit the Hong Kong can access relevant and
Ocean Park this Saturday. Below is the information about correct information or not.
his family. They do not have Smart Fun Annual Pass .
Please go to the Internet, find out the ticket prices and
help him to calculate the total amount for the whole
family's general admission for one day at the park. (10
mins)

Figure 3 : An item in the mathematics PA designed to assess the ʻaccess' dimension

151
 Chapter IV — Case studies

An item to assess the

‘manage’ dimension. Figure
4 shows an item in the
technical PA designed to
assess students’ ability
to manage information
effectively. This item
asked the students to
edit the information in a
Word document according
to six given formatting
requirements. Students were
also provided with a sample
text formatted according
to those six requirements
for their reference. The
assessment criteria are
based on the number of
changes that students can
make correctly.

Figure 4: An item in the technical PA designed to assess the ‘manage’ dimension

An item to assess the

‘integrate’ dimension. Figure
5 shows an item in the
mathematics PA designed
to assess students’ ability
to integrate information
effectively. In this item,
students are asked to
manipulate an interactive
applet to observe changes in
the area of a rectangle with
the different length–width
configurations formed
by a piece of string of
fixed length. Students
are then asked to deduce
the maximum area of
a rectangle that can be
enclosed by the piece of
string. The assessment
criteria are based on the
comprehensiveness of the
students’ manipulations
and observations, and the
correctness of the students’
interpretations.
Figure 5: An item in the mathematics PA designed to assess the ‘integrate’ dimension

152
 Teacher practices and student outcomes

An item to assess the

Endangered Species 'create' dimension. Figure 6

shows an item in the science
Question 2
Some new plants and animals have been sent to the
PA designed to assess
Kadoorie Farm for adding to their collection. Your students ' ability to effectively
classmates used a digital camera to take pictures of create representations of
these new species as shown below.
Among the animals , there is an endangered species information . In this item,
found in Hong Kong. An endangered species is a species students were asked to
whose number is so small that it is at the risk of
use electronic resources to
extinction. Can you identify which species it is and what
kind of habitat it needs? Use the links on the right hand create a classification chart
side or other websites to look for information about
with four categories for nine
endangered species in Hong Kong. (5 mins)
species and also include
both the names and photos
of those species in the chart.
The assessment criteria are
Common Muntjac Cock Cuora Galbinifrons based on the complexity of
the chart created .

Sago Palm Romer's Tree Frog Choloepus Hoffmanni

Grantham's Camellia BirdNet Fern Golden Agouti

Question 3.1
Please construct a classification diagram :
i. Classify the above new animals and plants into four
suitable categories. Please make reference to the
" Nature Walk Self-guided Map " to help the farm staff
decide where to place the above newcomers .

Figure 6: An item in the science PA designed to assess the 'create' dimension

An item to assess the

'communicate ' dimension.
Figure 7 shows an item in
Discussion
the technical PA designed
Question 4 to assess students' ability
You would like to discuss about the scenic spots you to communicate information
suggested with your classmates through the discussion effectively. This item asks
forum. In the forum, you may discuss with your students to share and
classmates why you suggest these two scenic spots discuss their suggestions
and give opinions to those scenic spots suggested by on their choice of scenic
your classmates. (5 mins)
spots for their grandparents
using a discussion forum .
The assessment criteria
Click here to start the discussion
are based on whether the
students can post their ideas
and give responses to their
peers or not.
Figure 7: An item in the technical PA designed to assess the ‘communicate' dimension

153
 Chapter IV - Case studies

An item to assess the

Question 4-7 'evaluate ' dimension .

Part II Exploring Ecosystems Figure 8 shows an item in

the science PA designed to
Using a Simulation Programme assess students' ability to
communicate information
effectively. In this item ,
students have to run and
observe the behaviour of an
ecological simulation , and
then propose a guideline
for protecting the pond
ecosystem . Hence they
need to be able to evaluate
Doing ecological experiments may bring danger to the
environment. Therefore, scientists often use the challenges to the pond
simulations to find out how changes will affect different ecology based on their
species . Now you are going to use one simulation observations of the simulation
programme to find out more about a pond ecosystem (
池塘生態系統). There are totally four questions in the as well as what they have
simulation programme . (20 mins) learnt from other information
Please read the instruction carefully once the simulation sources they read . The
is started. assessment criteria are
based on whether students'
7.4 Use the 'chat room' to discuss with your classmates and
generated guidelines applied
draw up for the visitors one guideline to protect the pond
ecosystem from any destructive behaviour done by the to the whole ecosystem and
visitors. Give explanations for this guideline . wether sufficient reasons
were given .
Figure 8: An item in the science PA designed to assess the ‘evaluate' dimension

the information ) and create (adapting , scoring of the PA tasks requires expert
applying , designing or inventing infor- judgment based on a thorough under-
mation in ICT environments). Hence standing of the scoring rubrics . A train-
two scoring rubrics are necessary for ing workshop including an inter-coder
assessing these two aspects of the stu- moderation and discussion of discrep-
dents’ performance . The scoring rubric ant scoring was conducted before the
shown in Table 2 is for scoring perform- formal scoring took place . The inter-
ance in the create dimension only. The coder reliabilities for the scoring were
specific skill pertaining to the create 0.95 in mathematics , 0.99 in Chinese
dimension in this task is the ability to language at grade 5, 0.96 in Chinese
use an advanced tool to create a well- language at grade 8 , 0.95 in science
and 0.98 in the technical PA for both
structured chart. The scoring criteria
grades 5 and 8 .
and an illustrative sample of students’
work for each level of performance are
also provided in Table 2 . 4.6. Challenges encountered in
thedesign ofperformance
Experienced teachers were recruited
assessment tasks
to score the students’ performance
based on the students’ responses to in this study
the questions as well as the products We encountered serious challenges
they created for the assessment. The in the design of the PA tasks . A com

154
 Teacher practices and student outcomes

prehensive literature review conducted Assessment of IL in subject-specific

at the start of the study revealed that contexts was only found for science
most of the reported empirical work on (e.g. Quellmalz et al., 1999; Quellmalz
assessment of IL was in the area of and Kozma, 2003). Hence the develop-
assessing technical IL (e.g. ETS, 2003; ment of PA for technical IL and in differ-
Lennon et al. 2003; Jacobs, 1999). ent subject domains using the same IL

IL dimension
and specific Performance Scoring Illustrative sample of students’work
IL skill level criteria at this level
assessed
Create – able Advanced Able to use
to use an an advanced
advanced tool (diagram
tool to create function,
a well- Excel or other
structured drawing tool)
chart to create a
chart with
at least 2
levels of
hierarchical
structure
Proficient Able to use
an advanced
tool (diagram
function
or other
drawing tool)
to create a
chart with
1 level of
hierarchical
structure
Basic Able to use Or
a simple
tool (table)
to create a
classification
table

Novice Unable to
create a
classification
chart

Table 2: A sample scoring rubric for item 3.1 in the science PA

155
 Chapter IV - Case studies

framework is a pioneering attempt we specific subjects because of the nature

have made in this study. There are a of the learning tasks and the discipline .
number of challenges we faced in the Examples of these are exploratory
design of the PA tasks that have not geometry tools in mathematics and
been satisfactorily resolved . Two of the simulation tools for exploring the out-
challenges with important methodologi- come of different scenarios in science .
cal implications are reported below. In the present study, in assessing ‘eval-
uation’ skills in science , students were
One of the methodological challenges required to run the simulation program
in comparing students’ IL perform- and make observations of how the
ance across subject areas is the task ecology changes and then discuss with
dependence of the level of IL perform- their peers to propose a guideline for
ance required for the most satisfac- protecting the pond ecosystem (see
tory completion of a task along a spe- Figure 8 for task detail) . However, for
cific dimension . For example , an item evaluating the same dimension in the
assessing the ‘create’ dimension in the technical PA, students only need to
mathematics PA as shown in Figure 5 critically evaluate whether the retrieved
(question 3.1 ) only asks students to use information was related to the topic ,
an interactive program to create three without the need for using any sub-
rectangles and record their lengths ject specific tool . It is also not possible
and widths . The level of performance to isolate the effect of the students’
required for the most satisfactory com- subject matter knowledge on their IL
pletion of this task is at the basic level performance .
only. On the other hand , question 3.1 in
the science PA (See Figure 6) assess-
4.7. Sampling and
ing the same dimension (create)
required students to create a classi- administration of the
fication diagram . For the mathemat- assessment
ics item , it simply required students to A two-stage sample design was used in
follow the instruction to create an arti-
this study. First, 60 primary schools and
fact . On the other hand , the science PA
60 secondary schools were sampled
required a higher level of competence from each of these two populations of
for satisfactory task completion since schools using stratified random sampling
they need to determine the shape of
based on the achievement banding (2)
the chart and how many hierarchical and size of the schools . Then one intact
levels they need for the chart. In other
class (at grade 5 for primary schools
words , the levels of skills and com-
and at grade 8 for secondary schools)
petences in the ‘create’ dimension in is randomly selected from each sam-
the science PA are higher than those
required in the technical PA. Matching
(2) All students in publicly funded primary schools
the levels of performance required for
need to take territory-wide assessment for the
all dimensions on two different PAs is
purpose of secondary school placement. Primary
particularly difficult if the task contexts schools can be categorised into three student
in both have to be authentic . achievement groups (referred to in Hong Kong
as achievement banding), high , medium and low,
based on the performance of their students in
Another challenge for ‘ comparability’
these assessments . Secondary schools can also
across subject domains is that there be categorised into three achievement bands
are some digital tools and their usage based on the assessment performance of the
which are core IL performance only for students they admit into secondary 1 .

156
 Teacher practices and student outcomes

pled school to take part in the IL assess- 5. Students’ information

ment. As both the assessment of IL and
online performance assessment were literacy outcomes —
totally new to schools, it was not easy to Impact of eight years of
get schools to voluntarily agree to take ICT in education policy
part in the study. At the end , 40 primary
and 33 secondary schools took part in in Hong Kong
the study (after replacement). A total of
We analysed the results of the assess-
1 320 grade 5 students and 1 302 grade
ment by computing the percentage
8 students took part in the main data
score obtained by the students for the
collection in this study. Two pilot studies items in each dimension . In the follow-
had been conducted before the main
ing section , we will first report on the
data collection . The first pilot study was
technical IL achievement of the grade
to ensure the validity of the instruments 5 and grade 8 students to examine
and the second pilot study was to try out the differences between them . Next ,
the logistic arrangements of the main a summary of the results across the
study. three PA at the grade 8 level is pro-
vided . A comparison of the students ’ IL
To ensure that the assessment
achievement across the three domains
conducted in this study was fair and (technical , Chinese language and sci-
valid , it was necessary for students in ence) is then given , taking account of
all schools to have access to a uniform
the challenges mentioned and of the
computing environment. As a result of limitations of the study, as pointed out
the implementation of the first ‘ IT in in section 4.6 .
education’ strategy, all schools in Hong
Kong had been equipped with at least
one computer laboratory with broad- 5.1 . Students’ performance in
band Internet access . However, the the technical IL PA
differences in hardware and software
Figures 9a and 9b shows the boxplots
infrastructure and configurations were of the school means of students ’
still extremely wide between schools outcomes on the seven IL dimensions
in Hong Kong . It was also administra- in the technical PA at grades 5 and 8
tively not feasible for the study team to respectively. These results show that
install the same software environment at both levels performances in the
in the computer laboratories of the dimensions of ‘ define ’ , ‘access’ and
sampled schools. After exploring pos- ‘ manage ’ were rather high while the
sible alternatives , we decided on the poorest performances were observed
use of a remote server system the in the dimensions of ‘ communicate’
Microsoft Windows Terminal Server and ‘ create’ . The results also showed
(WTS) as the most suitable tech- that grade 8 students had significantly
nology platform for the administration better performance than grade 5
of the IL performance assessments in students with respect to all seven IL
our context . Students worked on the indicators , which is not surprising .
PA tasks in the computer laboratories
in their own schools , which acted as On the other hand , there were huge
dumb terminals . All assessment task- differences in students ’ IL outcomes
related computations and manipu- between schools at both levels . The
lations were in fact carried out and largest dispersions were found in the
saved on the WTS . dimensions of ‘define’ and ‘ manage' .

157
 Chapter IV - Case studies

The very large inter-school differences 5.2. Grade 8 students’

lead to some rather surprising obser-
vations . Firstly, the mean achievement performance in the Chinese
of the best-performing primary school language PA
was almost the same or higher than the Figure 10 shows the boxplots of the
median school mean of the second- school means of students’ outcomes on
ary schools for all the IL dimensions the seven IL dimensions in the Chinese
with the exception of ‘ manage’ . On language PA for grade 8 students . It can
the other hand , the lowest- performing be seen that performance was best for
secondary schools had means below the dimensions ‘ define’ , ‘ access ’ and
the median of all the school means ‘manage’ and significantly worse for the
for the primary schools except for the ‘ evaluate’ dimension . Further, compared
‘ evaluate’ dimension . with the performance in the technical PA,
it can be seen that the medians of the
Results also showed that there were school mean achievement levels across
significant differences between schools the different IL dimensions are much
in terms of students ’ levels of IL com- more similar (the ‘ evaluate’ dimension
petences in technical proficiency. This being an outlier in this respect) .
seems to indicate that school experi- On the other hand , the between-
ences matter in contributing to the IL school differences in mean student
outcomes of students and that the achievement remain huge across
number of years of schooling and cog- schools , indicative of similarly huge
nitive maturity contribute less to stu- differences across schools in terms of
dents’ IL outcomes compared with the students’ opportunities to learn IL skills
curriculum experiences of students . in the Chinese language subject.

100-

80-

60-

137

40-
137

20-

135129

123

0-

Define Access Manage Intergrate Create Communicate Evaluate Total

Figure 9a : Boxplots of the school means of grade 5 students' IL performance in the technical PA across the 40
primary schools

158
 Teacher practices and student outcomes

100-

80-

60-

40

232 211

212

20-

define access manage integrate create communicate evaluate total

Figure 9b : Boxplots of the school means of grade 8 students' IL performance in the technical PA across the 33
secondary schools

100-

80-

60-

40-

223

20-

0-

define access manage integrate create communicate evaluate total

Figure 10: Boxplots of the school means of grade 8 students' IL performance in the Chinese language PA across
the 33 secondary schools

159
 Chapter IV - Case studies

100-

80-

60-

223

209
40- 223 。
*
234
223

20-

232

01 8
212

define access manage integrate create communicate evaluate total

Figure 11 : Boxplots of the school means of grade 8 students' IL performance in the science PA across the
33 secondary schools

5.3. Grade 8 students’ 5.4. Summary ofthe findings

performance in the science PA from the IL performance
Figure 11 shows the boxplots of the assessment study
school means of students ’ outcomes
Overall , we find that most students have
on the seven IL dimensions in the
some basic technical skills in operating
science PA for grade 8 students . It can
the computer, using the basic functions
be seen that performance is better in in the Office suite of applications and
the ‘ access ’ and ‘ define ’ dimensions
surfing the web. Student competence
and poorest in the ‘ evaluate’ dimension . in lower-level IL skills such as defining
Similar to the other assessment results
and accessing information are highest
reported earlier, the between-school
while performance in the dimensions
differences in terms of the mean
‘integrate’ , ‘ create’ , ‘communicate ’ and
student achievement were very large . ‘ evaluate’ were poor. Student perform-
The variation in student performance ance was found to be poorest for items
across the seven dimensions as
requiring the use of digital tools specific
reflected by the medians of the school
to the subject discipline, e.g. explora-
means differ much more widely than
tory geometry tools in mathematics
that found in the Chinese language and simulations in science . The inter-
PA but is somewhat smaller than that
school differences in achievement also
in the technical PA. However, when
tend to be very wide for such items . We
comparing students’ results among the also find that schools with high achieve-
five PAs , science has the lowest mean
total score .
ment banding do not necessarily have

160
 Teacher practices and student outcomes

higher overall student IL achievement on teachers’ pedagogy (Law, 2008 ) ,

levels . In fact, some newer schools which in turn also influences the per-
with medium student achievement ceived impact of ICT on students ’
banding known for their engagement in learning outcomes ( Law and Chow,
curriculum and pedagogical innovation 2008b ) . Further, in -depth analyses
showed higher student IL achievement of the SITES 2006 and SITES- M1
than some schools well-known for their findings indicate that system level
excellent general academic achieve- policy impacts on teachers ’ peda-
ment. These findings indicate that IL gogical practice orientation and ICT
achievement in the subject areas is not use ( Law, Lee and Chan , in press) .
only dependent on students’ achieve- The analysis of the policy changes in
ment levels in the specific subject area Hong Kong, both in terms of the first
assessed , but also on how ICT has two ‘ IT in education’ strategies and
been integrated into the curriculum by the overall school curriculum reform
teachers in their classrooms . There which started in 2000 , has resulted in
was wide variation within schools and a stronger lifelong learning orienta-
between schools in terms of student IL tion in pedagogical practices in Hong
achievement, indicating that both stu- Kong classrooms . The various inter-
dent background and learning experi- national and local studies indicate
ence in school matters . that the policy initiatives have brought
about positive (though yet still small )
6. Conclusion progress in realising the goal of lev-
eraging the use of ICT to prepare stu-
What have the first two ‘ IT in education ’ dents for life in the 21st century. The
strategies in Hong Kong achieved? The apparent change in policy direction in
studies reviewed in this paper indicate the third strategy is hence somewhat
that some basic measures of infra- worrying . It has lost the strong focus
structure and teacher use have been on pedagogy and fostering of school
achieved in all publicly funded schools leadership for ICT use in schools to
in Hong Kong . There have been some support curriculum innovation , which
changes in pedagogy, but pedagogi- have been found to be most important
cal innovation integrated with ICT use for achieving the educational poten-
is still rare and not well integrated with tial of ICT. Another concern is the
use of ICT tools specific to subject absence of any mention of research
areas. Students have generally gained and development as a strategic goal
some basic IT operational skills but are in the third strategy. The continuing
very poor in tackling the more complex support for local ICT- related research
tasks involving information literacy skills initiatives as well as Hong Kong’s
in ‘integration’ , ‘evaluation’, ‘create’ and participation in the SITES studies
‘communicate’ . The findings also indi- have provided valuable data and
cate that learning experience in school findings to inform policy and prac-
matters in terms of students’ IL achieve- tice . It is hoped that the absence of
ment and that there is still a long way mention is not an indication that such
ahead between students’ ICT use in support will not be forthcoming in the
classrooms and nurturing 21st century third strategy. The study on perform-
skills in Hong Kong . ance assessment of students ’ IL skills
reported earlier in this paper is only
Analyses of the SITES 2006 data indi- a preliminary study, and should be a
cate that school leadership impacts priority area for further research .

161
 Chapter IV — Case studies

References
Catts, R. and Lau, J. (2008). Towards information literacy indicators. Paris:
Unesco. Retrieved 17.08.09 from http://www.uis.unesco.org/template/pdf/cscl/
InfoLit.pdf

CERI (2001). What schools for the future. Paris: OECD.

Curriculum Development Council (CDC) (2001). Learning to learn — The way

forward in curriculum. Hong Kong: Hong Kong Government Printing.

European Council (2000). Lisbon European Council 23 and 24 March 2000,

Presidency conclusions. Lisbon: European Parliament. Retrieved 17.08.08 from
http://www.europarl.europa.eu/summits/lis1_en.htm.

EC (Education Commission) (2000). Learning for life, learning through life:

reform proposals for the education system in Hong Kong, Hong Kong: Education
Commission. Hong Kong SAR Government. Retrieved 17.08.09, from http://
www.edb.gov.hk/FileManager/EN/Content_4079/overview-e.pdf

EDB (2008). Right technology at the right time for the right task. Hong Kong:
Education Bureau (EDB), Hong Kong SAR Government.

EMB (1998). Information technology for learning in a new era: five-year strat-
egy 1998/99 to 2002/03. Hong Kong: Education and Manpower Bureau, Hong
Kong SAR Government. Retrieved 17.08.09 from http://www.edb.gov.hk/index.
aspx?langno=1&nodeID=425.

EMB(2004).Empoweringlearningandteachingwithinformationtechnology.Hong
Kong: Education and Manpower Bureau, Hong Kong SAR Government. Retrieved
17.08.09 from http://www.edb.gov.hk/index.aspx?langno=1&nodeID=2497.

Educational Testing Service (ETS) (2003). Succeeding in the 21st century: What
higher education must do to address the gap in information and communication
technology proficiencies. Princeton: NJ.

Jakobs, K. (1999). Information technology standards and standardization: A

global perspective. IGI Global.

Law, N. and Chow, A. (2008a). ‘Pedagogical orientations in mathematics and

science and the use of ICT’, in N. Law, W. J. Pelgrum and T. Plomp (eds),
Pedagogy and ICT in schools around the world: findings from the SITES 2006
study, 121–179. Hong Kong: CERC and Springer.

Law, N. and Chow, A. (2008). ‘Teacher characteristics, contextual factors, and

how these affect the pedagogical use of ICT’, in: N. Law, W. J. Pelgrum and T.
Plomp (eds), Pedagogy and ICT in schools around the world: findings from the
SITES 2006 study. Hong Kong: CERC and Springer.

162
 Teacher practices and student outcomes

Law, N. , Lee , M. W. and Chan , A. (in press) . ‘ Policy impacts on pedagogical

practice and ICT use: an exploration of the results from SITES 2006’ . Journal of
Computer Assisted Learning.

Law, N. , Yuen, A., Shum , M. and Lee, Y. (2007). ‘ Final report on Phase (II) study
on evaluating the effectiveness of the ‘Empowering learning and teaching with
information technology’ strategy (2004/07)’ . Retrieved 23.07.09 from http://www.
edb.gov.hk/index.aspx?langno=1&nodeID=6363

Lee, Y. (2009). ‘An investigation of 21st century skills in innovative pedagogi-

cal practices using technology’ . Unpublished doctoral dissertation , University of
Hong Kong .

Lennon , M. , Kirsch , I. , Von Davier, M. , Wagner, M. and Yamamoto , K. (2003).

‘ Feasibility study for the PISA ICT literacy assessment. Report to network A’ , a
joint project of ACER, ETS, NIER. Paris : OECD . Retrieved 17.08.09 from http://
www.pisa.oecd.org/dataoecd/35/13/33699866.pdf

Morrison , G. R. and Ross , S. M. (1998) . ‘ Evaluating technology-based proc-

esses and products’ , in : R. S. Anderson and B. W. Speck (eds), Changing the
way we grade student performance: classroom assessment and the new learn-
ing paradigm, 69–77 . San Francisco, CA: Jossey-Bass .

Moskal , B. M. (2000). ‘Scoring rubrics: What, when , and how?’ Practical

Assessment, Research and Evaluation Vol . 7 , No 3. Retrieved 23.07.06 from
http://pareonline.net/getvn.asp?v=7&n=3

NCREL. (2003). ‘ enGauge 21st century skills: literacy in the digital age’ [elec-
tronic version] . Retrieved 08.01.06 from http://www.ncrel.org/engauge/skills/
indepth.htm

OECD . (2005). ‘ Definition and selection of key competencies — Executive sum-

mary’ [ electronic version] . Retrieved 16.06.06 from http ://www.deseco.admin .
ch/bfs /deseco/en/index/02.html

Pal , L. A. (2001 ) . Beyond policy analysis — Public issue management in turbu-

lent times. (2nd ed) . Scarborough (Ontario): Nelson .

Pelgrum , W. J. and Anderson, R. E. (eds). ( 1999). ICT and the emerging para-
digm for life -long learning. Amsterdam : IEA.

Popham , J. W. (2003). Test better, teach better: the instructional role of

assessment. Alexandria , Virginia : Association for Supervision and Curriculum
Development.

Quellmalz , E. , Schank , P. , Hinojosa , T. and Padilla, C. ( 1999). ‘ Performance

assessment links in science’ , Practical Assessment, Research and Evaluation ,
Vol . 6 , No 10 , 1998–1999 .

163
Quellmalz, E. and Kozma, R. (2003). ‘Designing assessments of learning with
technology’, Assessment in Education: Principles, Policy and Practice, Vol. 10,
No 3, 389–407.

Singapore Ministry of Education (1997). Masterplan for IT in education: 1997–

2002. Retrieved 17.08.09 from http://www.moe.gov.sg/edumall/mpite/index.
html

Singapore Ministry of Education (1998). Learning to think: thinking to learn.

Singapore: Author.

Unesco (2008). ICT competency standards for teachers: Policy framework.

Paris: Unesco.

US Department of Education. (1996). Getting America’s students ready for the

21st century: meeting the technology literacy challenge. Washington, DC: US
Government Printing Office.

Wiggins, G. (1998). Educative assessment: designing assessments to inform and

improve student performance. San Francisco, CA: Jossey-Bass Publishers.
Indicators on ICT in primary
and secondary education:
results of an EU study
Willem Pelgrum
EdAsMo

Introduction 2. In which areas are indicators

needed?
The study ‘ Indicators on ICT in edu- 3. Which international comparative
cation ' was run under the auspices of data are available and what are the
the European Commission . The study data gaps?
was run from November 2008 until 4. Which actions could be under-
October 2009 (1 ) . The main purposes taken for ensuring that monitor-
of the study were the following . ing of European benchmarks and
international comparisons on edu-
1. To identify a set of indicators that cational progress will take place in
are relevant for enabling the regu- the future?
lar monitoring of the use and impact
of ICT in primary and secondary Each of these questions will be
education . addressed in the subsequent sections .
2. To describe scenarios for moni-
toring ICT in education in the
European Union . 1. Policy issues
As explained in Chapter II , educational
The study was focused on the 27 EU
monitoring is primarily a tool for
Member States , the three candidate
countries and the countries from the
policymaking and , hence , a first step
in the process of exploring scenarios
European Economic Area. This group
will hereafter be referred to as ‘ EU+’ .
for monitoring ICT in education in
the EU + consisted of analysing the
In line with the main steps for moni-
intentions of policymakers with regard
toring that are described in Chapter II : to this area . A distinction will be made
Monitoring in education : an overview. between common objectives and
The main questions addressed in this
common goals/topics . The common
study were the following .
objectives were inferred from EU
policy documents reflecting common
1. What are the policy issues regard-
ICT- related objectives that originate
ing ICT in education?
from the Lisbon strategy and follow- up
declarations . For all the EU + countries
(1 ) This study was financed (at a cost of that were targeted in this study, policy
EUR 122 200) by the European Commission . documents were collected from
Contract EACEA-2007-3278. Opinions presented
in this chapter do not reflect or engage the several sources , for instance official
Community. European Commission policy documents issued by ministries ,

165
 Chapter IV — Case studies

reports available through the EUN LexUriServ.do?uri=OJ:L:2006:394:00

Insight project and/or articles about 10:0018:en:PDF), eight main compe-
national policies regarding ICT written tency areas were distinguished:
by researchers in a recent book edited
by Plomp et al. (2009). A qualitative • communication in the mother tongue;
analysis of these documents was • communication in foreign languages;
conducted, which resulted in a list of • mathematical competence and
policy topics. basic competences in science and
technology;
In the next sections, these topics will • digital competence;
be summarised and the issues under- • learning to learn;
lying these topics will be described. • social and civic competences;
This will constitute the basis for forming • sense of initiative and
an impression of the current relevance entrepreneurship;
of indicator domains for these topics • cultural awareness and expression.
in the targeted group of countries for
which a survey was conducted among These areas will be further referred to
ICT policy experts from the EU+ coun- as ‘the EU core competency areas’.
tries (see Section 1.2). Although most of these competency
areas can be considered more or less
1.1. EU policy topics regarding traditional (as they always tended to
be featured in national educational
ICT in primary and goals of countries), some of these
secondary education (such as ‘digital competence’, ‘learn-
At the EU level, several initiatives ing to learn’ and ‘sense of initiative
were taken to promote the use of ICT and entrepreneurship’) are believed to
in education. With regard to ICT, one be essential for the information soci-
common EU objective resulting from ety, but also an underlying expectation
Lisbon was that ‘schools and train- can be observed that ICT is a crucial
ing centres, all linked to the Internet, facilitator for acquiring and maintain-
should be developed into multi-pur- ing competencies in these areas.
pose local learning centres accessible ‘Learning to learn’ can be conceived
to all, using the most appropriate meth- as a basic skill underlying the abil-
ods to address a wide range of target ity for lifelong learning, and, hence,
groups; learning partnerships should against this background it is relevant
be established between schools, to observe that in the Council’s con-
training centres, firms and research clusions on a strategic framework for
facilities for their mutual benefit’. This European cooperation in education
objective implies, for instance, that and training for the period until 2020
100 % of schools should have access (‘ET2020’), the importance of lifelong
to the Internet. learning is reiterated (http://www.con-
silium.europa.eu/uedocs/cms_data/
In follow-up declarations, these objec- docs/pressdata/en/educ/107622.pdf).
tives have been further elaborated in
more specific terms. In a recommenda- The common objectives are not explicit
tion of the European Parliament and of in terms of performance expectations
the Council of 18 December 2006 on and ICT-related opportunities to learn.
key competences for lifelong learning It seems fair to infer that the underly-
(http://eur-lex.europa.eu/LexUriServ/ ing assumption is that countries are

166
 Indicators of ICT in education

expected to implement opportunities 5. Connectedness, e.g. national and/

for students that lead to improvements or international cooperation, pub-
in these core competency areas. With lic–private partnerships.
regard to ICT, ‘access for all’ implied in 6. Teacher training, e.g. teacher com-
relation to these areas could be inter- petencies, pedagogical drivers
preted as ‘opportunities for students licence.
in school to use ICT for learning’. 7. Support, e.g. the way technical
However, concrete targets need to be and/or pedagogical support is
further defined. made available.
8. Transversal issues, e.g. equity,
The implications for our study are that,
financing, safety.
when it concerns students, competen-
cies and attitudes will mainly refer to
In the next section, policy issues that
these EU core competency areas.
are underlying these topics will be
described.
1.2. National ICT-related policy
topics for primary and 1.2.1 Infrastructure
secondary education Infrastructure as a topic is very broad.
An analysis of ICT-related policy docu- It covers sub-topics such as hardware
ments from the targeted group of EU+ and software, which are still quite
countries was undertaken. All policy broad, as policy concerns with regard
documents collected were read and to hardware cover a further wide range
coded, and the topics that were cov- of topics, as is the case with software.
ered in these documents were listed. The overall picture regarding policy
This resulted in a long list of cat- issues with regard to infrastructure
egories that were classified in terms resulting from the analysis of policy
of main topics, sub-topics, sub-sub
documents can be summarised as
topics, etc. follows.
The main topics that resulted from this
Afirst observation is that ICT infrastruc-
analysis were as follows.
ture is still an important topic for policy
concerns. This topic is addressed
1. Infrastructure: this concerns issues
in almost all documents and can be
such as hardware and software
and sub-issues such as access to considered a crucial condition for the
the Internet, broadband connec- use of ICT: ICT infrastructure should
tions and open-source software. be present before ICT can be used.
2. Curriculum and content: this In the early days of the introduction of
covers issues such as pedagogical computers in education, a shortage of
approach (e.g. autonomous learn- hardware and/or software was often
ing), content (e.g. development of mentioned by educational practition-
methods), assessment (e.g. portfo- ers as a major obstacle for integrating
lios, digital drivers licence). ICT in teaching and learning.
3. Outcomes, e.g. competencies, dig-
ital literacy. The policy documents refer to the
4. School leadership, e.g. change intention to improve the current infra-
management. structure, namely:

167
 Chapter IV — Case studies

• equipping classrooms with fast present a vision of how they would

Internet connections (e.g. Austria, use ICT in their schools.
Belgium);
• providing interactive white boards to In Belgium and other countries, the
schools (e.g. UK, Denmark); access of students (and even the local
• standardising systems and software community) to ICT infrastructure out-
(e.g. UK); side school hours is stimulated.
• providing laptops for teachers (e.g.
UK); With regard to software, it can be
• improving buildings (e.g. Cyprus); noted that the developement and
• ensuring own e-mail addresses maintenance of high-quality software
for students and teachers (e.g. for education has been a challenge
France). since the first micro-computers were
installed in schools around the mid-
It is interesting to note that some 1980s. Recent initiatives concern the
countries seem to take more initiatives creation of educational portals offering
than others regarding the provision of open content (via Internet-reachable
new equipment. For instance, the UK databases containing educational
made substantial investments, while content in many different forms) and
countries like Denmark (pilot project) the promotion of using open-source
and Sweden were more hesitant. software which, in principle, can be
attuned to the needs of the users (a
In some countries, equipping schools common problem in education is that
no longer seems to be a policy pri- teachers do not like pre-cooked con-
ority, for instance in Norway where tent which they cannot change). In the
‘there are no national programmes or UK, the curriculum online programme
initiatives for introducing new devices (see http://www.dfes.gov.uk/curriculu-
in schools’. This last observation is monline/) was launched in 2003 and
important, because it illustrates that provided every teacher and school
countries are in different stages of with e-learning credits that they could
introducing ICT in education. This will spend on approved ICT resources
have consequences for the monitoring purchased through the website.
needs of these countries.
1.2.2 Curriculum and content
Several policy strategies are in place
for allocating and financing equipment A curriculum allows governments to
in schools, namely: regulate (formally and prescriptively
or less formally) educational proc-
• lump sum financing in the esses in order to influence outcomes
Netherlands; of learning. Educational practitioners
• in Estonia, a school must submit a often mention the time that is needed
statement indicating how it is cur- for realising the existing curriculum as
rently using ICT in its teaching and a major obstacle for implementing ICT
learning programmes; the school in teaching and learning. In the UK,
must also detail how it will use the ‘improving the quantity and quality of
new equipment; e-learning is irrelevant, however, if it
• in Slovakia, the schools have to pre- is not done within the context of cur-
pare a project proposal in which they riculum development’.

168
 Indicators of ICT in education

The policy documents analysed often mention is made of ‘learning design

(in more than 50 % of the documents) packages that would enable teachers
refer to curriculum measures that were in all sectors to build their own individ-
planned in order to promote the inte- ual and collaborative learning activi-
gration of ICT. Frequently mentioned ties around digital resources’ .
are the intention to integrate ICT in
school subjects and the development In some countries , the intention is
of methods for ICT-assisted learning . explicitly formulated that ICT should be
a daily part of student learning activi-
A major distinction that can be made ties (e.g. Belgium , Estonia ) , which is
is between learning about ICT ( ICT as an example of an explicit educational
an object) and learning with the help policy objective dealing with promot-
of ICT (ICT as a tool ) . Whereas in ing the opportunities of students to
some countries the acquisition of ICT learn with and/or about ICT.
skills is organised via a separate sub-
ject, in other countries it is assumed 1.2.3 Outcomes
that these skills can be acquired From the policy plans a clear expec-
via the traditional subject areas (for tation is transmitted , phrased in differ-
instance , in some German states , it is ent terms and with different degrees
integrated in media education , while in of explicitness, but with an underlying
other countries , in particular the new strong conviction that the use of ICT
member countries , a separate infor- in education can improve access to
matics subject exists). Some docu- teaching and learning opportunities ,
ments are very explicit about the issue help to enhance the quality of teach-
of separate ICT subjects , for instance ing and learning, improve learning out-
Belgium where ‘the policy underpin- comes and promote positive reform of
ning the plan is to incorporate ICT into education systems .
different courses rather than to intro-
duce a specific ICT- related course’ . However, these expectations are
global in character. The past decades
Next to the expectation that ICT can have witnessed a search for getting a
improve outcomes of learning in tra- better insight into what impact may be
ditional subject areas , a number of expected from applying ICT in educa-
policy documents also mention that tion . It is still unclear how to answer
ICT can help to implement new ways questions, such as the following .
of learning whereby the students (with
the help of ICT) acquire more control 1. What are the basic functional e-lit-
and responsibility for their own learn- eracy skills that students should
ing processes and outcomes . For master when they leave compul-
instance , digital portfolios are con- sory education?
ceived as a tool that can help to keep 2. In which content areas can most
track of learning activities and prod- added value be expected when
ucts resulting from these activities . ICT is applied?
3. Has the use of ICT in the past dec-
Making explicit links between digital ades improved the competencies of
instructional materials and curriculum our students in core subject areas?
goals (the Netherlands) is conceived as Are students better prepared for life-
helping the teacher to choose appro- long learning (in terms, for instance ,
priate ICT applications . In the UK, of motivation to learn , analysing

169
 Chapter IV — Case studies

their own shortcomings, setting out and secondary education should also
learning trajectories, self-assess- pay attention to ICT-related competen-
ment, problem solving, etc.)? cies, sometimes combined with media
literacy (among others, Germany). For
In this respect, a recent ‘knowledge this purpose, ICT can be a separate
mapping’ exercise conducted by the subject or integrated in other subject
World Bank’s infoDev Group (Trucano, areas. Several countries made a delib-
2005) is relevant. It revealed that, erate choice for either one of these
despite decades of large investment in models. However, countries differ with
information and communication tech- regard to what is included in the ICT
nologies to benefit education in OECD competencies. Some countries have
countries and despite the increasing examinations to establish these com-
use of ICT in education in develop- petencies, such as the junior compu-
ing countries, data to support the per- ter driver’s licence. A prerequisite is
ceived conviction on the benefits from that all students have the opportunity
ICT are limited and evidence of effec- to use ICT during their schooling or at
tive impact is very elusive or debat- home. The latter refers to, for example,
able. These findings highlighted vari- disadvantaged students in second-
ous knowledge gaps and recognised ary education (United Kingdom). The
the need for internationally accepted efforts countries undertake to include
standards, methodologies and indica- ICT in the curriculum fall under the
tors to better measure the real benefits umbrella of the more general goal of
of ICT in education. bridging the digital gap by providing all
citizens with opportunities to acquire
This lack of good quality and unques- basic ICT skills and skills to use all
tionable data, in addition to the kinds of ICT services.
absence of standardised guidelines
for establishing relevant and compa- Another goal is that students are
rable indicators, hinders the ability of well prepared for the labour market.
policymakers to make informed deci- Governments of some countries initi-
sions or to demonstrate greater volun- ated programmes to promote access
tarism towards the integration of ICT to computers and the Internet at
into their education systems. home. Students get an extra oppor-
tunity to use a computer and to learn
The above is not meant to claim that no with computers. It is not only students
research has yet been done regarding who benefit from these programmes,
these questions. Many research and but also their families.
meta-studies have been conducted
over the past decades. Most of these
1.2.4 School leadership
studies, however, do not deal with
changes in the total education system, For a long time (since the introduction
and, therefore, when policymakers of the first micro-computers in educa-
have to take policy initiatives for the tion) the issue of school leadership
educational system at large, they was not featured in many policy plans.
often stand with empty hands. The However, it seems that (probably as a
policy documents that were analysed result of diffusion of research results
offer the following exposé regarding regarding mechanisms that play a role
(expected) outcomes. Objectives of in successful educational changes)
new or revised curricula for primary awareness is increasing that school

170
 Indicators of ICT in education

leaders may be important gatekeep- ing the real world to enter the school
ers and facilitators in the implementa- more easily. The walls of the school
tion of ICT. and the classrooms are no longer dif-
ficult blockades for integrating real-life
Weextracted the following observations components in the learning process .
from the policy documents . School lead- There is also a growing awareness
ers need appropriate training in a new that ICT innovations within schools
kind of management in which ICT is a cannot be realised without the help of
permanent factor from now on in their
the outside world and that the help of
strategy. In the UK, tools are provided outside colleagues and even business
to help school leaders to assess how firms is needed .
well their organisation uses ICT. These
tools help to modernise the school
In the policy documents , we find this
management (Austria) . In Belgium , reflected in several examples in almost
school leaders have to develop their
all European countries . Most important
ICT policy instead of using an imposed are the links between schools and pri-
policy document made by the gov- vate partners (business companies).
ernment . School leaders also have
Several companies in the field of ICT,
to do this in Germany. The reason is
such as Apple , Intel and Microsoft,
that they can describe their vision but
also become aware of what is needed
are involved in partnerships. For the
schools in the respective countries ,
to achieve this vision and the impact
the public–private partnerships involve
on teaching and learning . Norwegian
training of teachers , development
schools are required to develop an ICT of ICT- related educational materi-
plan . An ICT policy document is not
required in Sweden though local stake- als (including e-learning and portals),
holders ask school leaders to have infrastructure (hardware and access to
the Internet) and support and/or fund-
one. Each school has to make a quality
ing . Most of the public–private partner-
report every year. This report includes
plans for how to improve . Schools in ships are taking place at national level ,
Malta have included their ICT policy in but some are regionally based , as in
their school development plan . One of France . In several European countries ,
the topics that has been identified in the projects have been set up to establish
a link between school and the school
research literature as important when
it concerns school leadership is the environment. These projects vary in
development of a common vision on their goals : enabling students to learn
ICT that is shared by all stakeholders at home or in hospital , informing par-
ents of the achievements of their chil-
in the school (and preferably consist-
ent with the vision from stakeholders dren and to have contact with teachers ,
outside the schools, such as ministry, increasing digital literacy of other family
inspectorate, parents) . This topic is members (including parents) or provid-
hardly addressed in the policy docu- ing access to the Internet at home .
ments that were analysed . An excep-
tion is the UK where BECTA aims ‘to 1.2.6 Teacher training
deliver a vision for ICT in schools ’ .
Before teachers can apply ICT in
their lessons , they first need to know
1.2.5 Connectedness
what ICT is and how it may be used
ICT can help to open the school to the for improving instructional processes .
world as well as vice versa by allow- Hence they need to be trained . It is dif

171
 Chapter IV — Case studies

ficult to contradict a statement like this, also paid to the training of librarians in
but it is even more difficult and quite the field of ICT.
often impossible to realise adequate
continuous staff development activi- Within the framework of ICT projects,
ties for all teachers in an education programmes have been set up for the
system. Since the early days of ICT in in-service training of teachers, among
education, policy solutions have been others the MoNES programme in
tried in order to train teachers ade- Poland, the KK-foundation in Sweden,
quately but the complaints about the the POCTI programme in Portugal,
FOR TIC in Italy, Infovek in Slovakia
lack of teachers’ competencies and
and OPE.fi in Finland. In many coun-
confidence remained and hence the
tries, the teacher training institutes are
search for adequate solutions (that
involved in the in-service training of
are also payable) is continuing. Many
teachers. One would expect that, next
promising initiatives were taken, and
to the in-service training of teachers,
applied in small contexts, but were
the European countries consider the
probably not upscaleable.
pre-service training of teachers as an
The current policy issues that were important issue. However, only a few
inferred from the policy documents are policy (related) documents state this
summarised below. In all European issue. In Belgium, the institutes for
countries, the in-service training of teacher training have to pay attention to
teachers is a policy issue. Training the ICT competencies of their students
programmes and other arrangements by setting new attainment targets and
have been set up to organise the train- goals, not only for the basic ICT skills
ing of teachers. Teachers are offered but also for skills related to using ICT in
opportunities to learn how to use ICT the teaching-learning process.
for their own use and how to use it in the
teaching-learning process. An example 1.2.7 Support
is Hungary, where ‘teacher training is
At the beginning of the computer era,
beginning to concentrate on ICT-based
technical support in schools was impor-
educational methodology, with particu-
tant. Teachers did not have sufficient
lar emphasis on how to make optimal
ICT knowledge to solve hardware and
use of educational technology in the
software problems. At that time, hard-
classroom’. Several countries have for- ware and software in schools were less
mulated ICT competencies for teach- reliable. Nowadays, technical support
ers, including the didactical skills to use is provided quite often by professionals
ICT in the classroom (‘using ICT as a or well-trained staff in schools, espe-
pedagogical tool’). In some countries, cially in secondary education.
the training results in a certificate or the
European computer driving licence. In Even more important than techni-
Lithuania, for example, the basic mod- cal support is pedagogical support
ules of the European computer driving needed by teachers when applying
licence have been extended with addi- ICT in teaching and learning. The
tional modules specifically related to the policy documents that have been
use of ICT in schools, as in Denmark, reviewed show that teachers may
where the pedagogical computer driv- have difficulties in implementing ICT
ing licence has been developed. In in the teaching-learning process and
some countries, such as the United that they need support to accomplish
Kingdom and Lithuania, attention is this task. Mostly, the support is pro

172
 Indicators of ICT in education

vided by agencies outside the school . Therefore , courses in basic ICT skills
In Sweden , ‘ Schoolnet offers many are set up or people are given access
different services, functioning as an to ICT facilities after office hours .
information centre , a library and a Activities take place within the frame-
news agency. Schoolnet provides a work of digital literacy for all , narrowing
platform for the development of new the digital divide and lifelong learning .
educational approaches opened up by A policy goal in Finland is that all citi-
the Internet and new multimedia tech- zens have ‘ opportunities and the basic
nologies . ’ In Portugal , the Ministry of capabilities to use electronic services
Education relaunched the Nónio pro- (e-services) and content’ .
gramme to broaden the ICT compe-
tence centres network to support all Special programmes are aimed at cer-
school groups in the country. Hence , tain groups in society: disadvantaged
there are indications of a change of children in (secondary) education ,
emphasis from technical support to students who are ill , young sportsmen
pedagogical support. This is , among and sportswomen , young migrants or
other things , reflected in the role of certain regions in a country. Several
school ICT coordinators , which in programmes also focus on parents and
some countries is no longer limited to other groups (elderly persons , disabled
technical support. Educational sup- persons ). The programmes provide
port, including in-service training , is training in basic skills, access to (broad-
a task of the coordinator. In Catalonia
band) Internet, computers at home or
(Spain) , ‘ a new job description for ICT digitalisation of (learning) materials .
coordinators in schools (with specific Disabled persons are often faced with
regard paid to the new breed of tech- ill- adjusted standards and extra costs
nical support services), reforming in- for hardware . This limits their access to
service teacher training and setting up the knowledge society.
new pedagogical support services for
ICT using personnel from pedagogical Documents from Sweden and Portugal
resource centres’ has been created .
state that there is no specific pro-
In some countries (e.g. Portugal) , the gramme in these countries .
function of ICT coordinator does not
exist and , hence , the teachers have to Financing
organise the technical and pedagogi- Governments (mostly ministries of
cal support in their schools . education ) in several EU countries
purchase the hardware , software
1.2.8 Transversal issues
and access to the Internet and/or
In the documents , there are a number they finance the training of teachers .
of recurring issues that can be consid- Sometimes local governments are
ered transversal , as they cut through involved too, as in Poland .
the categories that were described
above . A number of these issues are Initially, hardware was financed by
reviewed below. grants and sponsors in Slovakia
because the government had not yet
Equity set up an information technology pro-
Almost all countries have the policy that gramme. Programmes were later set up
all citizens should have equal opportu- to give schools access to the Internet.
nities in society. It is expected that the By participating in European projects ,
use of ICT can foster these chances . schools received equipment.

173
 Chapter IV — Case studies

Safety such expectations certainly exist. For

In policy documents, two aspects are instance, one of the many possible
distinguished regarding safety of ICT conceptualisations of expected rela-
use: first is the protection of children tionships is shown in Figure 1, which
against harmful content; second the contains most of the issues that were
critical evaluation and use of sources. identified in the document analysis
For instance, in Sweden, a programme and can be summarised as follows.
was set up to raise the awareness of The ICT learning opportunities of stu-
children, parents and educators with dents have a (hypothesised) impact
regard to the first aspect. Other coun- on the competencies and attitudes
tries have started similar initiatives. that they acquire. These opportunities
are believed to depend on the peda-
In Malta, a portal has been devel- gogical practices of teachers (which
oped to protect schools from inappro- in turn depend on the extent to which
priate content and it also offers links the teachers are trained) and availa-
to useful educational websites. The bility and access to ICT infrastructure,
Greek school network has a protec- which is a crucial condition for creating
tion policy for students. In many coun- ICT-OTL at school. On the other hand,
tries, protection is often offered by the these opportunities are determined by
government through providing filter- what students learn outside school.
ing techniques, information on how to
Policymakers can influence these con-
use the Internet or a telephone line to
ditions via curricula, but countries differ
report illegal information. Campaigns
in the extent to which the curricula can
have been launched to increase citizen
be prescriptive. As the use of ICT is an
awareness. educational change, the role of school
leaders is important as well as the
Monitoring
availability of teacher training facilities
In order to be able to evaluate ICT for getting acquainted with the techni-
policy in education, monitoring of the cal and pedagogical aspects of ICT.
implementation takes place in quite a
number of countries. The fact that the policy documents are
not specific with regard to expected
The monitoring focuses, among other ICT-OTL and impact is not surprising,
things, on infrastructure, competen- on the one hand, as it is currently still
cies, integration in the teaching-learn- too early to take policy decisions for
ing process, perceptions, attitudes and the education system at large, since
needs. it is not yet known what works and
what does not beyond the borders
1.2.9 From policy issues to of small-scale pilots, case studies,
conceptual framework experiments and the like. On the other
hand, although policy documents are
The policy topics mentioned above usually not very specific, given the
may be conceived as concepts that large investments with regard to ICT
can constitute the basis for a concep- infrastructure in education, one would
tual framework in which expectations expect more explicit expectations to
about interdependencies can be made be formulated. However, if clear policy
explicit. Although the policy docu- expectations are lacking, one may
ments are not usually very explicit wonder what implications this may
about cause and effect expectations, have for EU monitoring.

174
 Indicators of ICT in education

Curriculum and content

School leadership Teacher training

Teacher competencies
Pedagogical practices

Infrastructure

Outside school experiences

Support
OTL-ICT

Outcomes/attitudes

Home background

Figure 1 : The main concepts for monitoring ICT use and impact

1.2.10 Implications for described in Section 1.2.1 , these need

to be further specified in order to be
monitoring the use and
useful for drawing up indicator defini-
impact of ICT in the EU tions and operationalisations .
As mentioned in Chapter 2 , in order
to be able to make inferences about 1.3. Perceived relevance
whether progress is being made with
of indicator areas
regard to educational outcomes , poli-
cymakers need monitors that show — A panel of 54 national ICT policy
on the basis of reliable and valid quan- experts from the EU + countries were
titative indicators -

to what extent invited to give their opinion about the

expected changes are taking place over need for international comparative
time . Although the analysis of policy ICT indicators in 55 indicator areas
documents does not immediately lead that covered the policy topics men-
to identifying common goals , they offer tioned in Section 1.2. The question-
a first step for delineating goal domains naire was administered online in the
that can (in principle) be further defined period May/June 2009. Responses
and used for an exploration among ICT were received from 76 % of the invited
policy experts from the EU + group of experts from 26 countries (which is
countries . The approach for this explo- 93 % of the countries with which active
ration is described in Section 1.3 . Next communication channels were estab-
to finding empirical evidence for the lished ) . The data were processed with
relevance of indicator domains , the SPSS - 16. If more than one response
common objectives (resulting from was received from a country, the
the Lisbon strategy) can (in principle) responses were weighted so that in
be used as a basis for more concrete the end result each country had the
indicator definitions . However, as same weight. In the following sections ,

175
 Chapter IV - Case studies

the results from this survey will be cational matters and in particular what
summarised . to monitor, how extensively and how
frequently. Nevertheless , the ratings
Firstly, a description will be given of can be used for a first priority list which ,
the extent to which the respondents when it eventually comes to monitoring
experienced in general a need for com- ICT in the EU , can be further revised
parative indicators on ICT in education . in subsequent negotiations between
Next, an overview will be given of the countries , taking into account too areas
areas for which the highest needs were other than the ones considered in our
expressed . study.
Several caveats should be taken into
1.3.1 The need for comparative
account when using the ratings pre-
sented in the next sections for setting indicators in general
indicator priorities . Firstly, the descrip- A first question for which opinions were
tions for each area were quite general solicited from the panel members con-
and hence more concrete indicator cerned the need for international com-
elaborations could elicit different indi- parative monitoring in the EU of ICT in
cator needs , as usually is the case : the education . From Figure 2 we can infer
more concrete a proposal , the less con- that, among the panel members , there
sensus may be expected among panel was a high consensus . Slightly more
members . Also , one should take into than 50 % of the respondents are defi-
account that the ratings concern sub- nitely sure that this need exists , while
jective estimates of panel members , another 38 % think that this is the case
which do not necessarily reflect the depending on the kind of indicators .
opinions of national educational actors Hence, altogether, a large majority
involved in decision-making about edu- (92 % ) indicated that there is a need for

Yes
,definitely

Yes
, depending on the kind
ofindicators

Maybe, if certain conditions are met

No

Unlikely

0 10 20 30 40 50 60

Percentage of countries

Figure 2 : The need for ICT monitoring in general

176
 Indicators of ICT in education

international comparative monitoring of be given by taking 60% agreement

ICT in education. In only one country about high needs as the threshold for
did the panel member have the opinion selecting indicator areas. The indicator
that this need did not exist. Apparently, areas which 60% or more of the panel
the panel members felt confident about members qualified as highly needed is
their capacity to rate, because the shown in Table 1.
answer category ‘Don’t know’ was not
used at all. It could be inferred from The issues ‘Connectedness’,
this poll that, throughout Europe, there ‘Curriculum and content’ and ‘Infra-
is a need for international comparative structure’ did not contain topics for
monitoring of ICT in primary and/or which 60 % or more of the panel mem-
secondary education. bers expressed a high need. The fact
that ‘Curriculum and content’ was not
rated as highly needed by many panel
1.3.2 High priority indicator areas members is a bit surprising, as it is often
Keeping in mind the caveats expressed argued that the curriculum is an impor-
earlier in this chapter, an indication tant handle for introducing educational
of priority areas for monitoring may change.

Table 1: Indicator areas per topic with high need above 60 %

Percentage
Indicator areas
high need
a. Opportunities to learn with and/or about ICT
Activities where students use ICT for learning in the 5 EU key competency
areas (literacy in reading, mathematics and science, language skills, ICT skills 73 %
and learning to learn skills)
The extent to which students use ICT for cooperation and/or communication 65 %
Activities where students use ICT in general at school 60 %
The extent to which students use ICT at school 60 %
b. Competencies and attitudes of students
The ability of students to solve assignments that require the use of ICT in the
5 EU key competency areas (literacy in reading, mathematics and science, 66 %
language skills, ICT skills and learning to learn skills)
The ability of students to use ICT for learning to learn (goal setting, self eva-
64 %
luation, management of learning, self evaluation)
c. ICT support
The extent to which pedagogical support is available for teachers (for lesson
61 %
preparation, class management issues, assessment procedures, etc.)
d. Teacher training
Pedagogical ICT competencies of teachers 82 %
Ability of teachers to build their own individual and collaborative learning
68 %
activities around digital resources
Ability of teachers to locate digital content resources that fit their curriculum
62 %
targets
Application of innovative forms of assessment 61 %
e. School leadership
Competencies of the school leadership to manage ICT-related innovations 63 %

177
 Chapter IV - Case studies

2. ICT- related data ments from the OECD and/or IEA

which have been conducted since
available in regular 2000 , all questionnaires from these
assessments from the studies were collected and mapped
IEA and OECD on the list of policy topics that were
described in Section 2.2 . The ICT-
In order to determine which data related data which are available in
and instruments with regard to ICT the existing data sets are listed in
were available in the regular assess- Table 2 below.

Table 2 : ICT- data available in data bases from IEA and/or OECD

Infrastructure

Primary education •

Secondary education
Number of computers available for Availability of computer software at
instruction ( school leader) students ' home
Number of Internet computers at school Availability of computer at students'
Shortage of computers for instruction in home
general ( perceived by school leaders) Access to Internet at students' home
Shortage of computers for instruction in Access to Internet in general (teachers)
mathematics/science (school leaders) Access to Internet for mathematics/
Access to Internet in general (teachers) science (teachers)
Access to Internet for mathematics/ Shortage of computers for instruction
science (teachers) mathematics/science (teacher)
Computers available in classroom and/or Computers available for mathematics/
elsewhere (teacher) science (teacher)
Computers available for educational Shortage of software for mathematics/
purposes (teacher) science (teacher)
Computers available for mathematics/
science (teacher)
Availability of computer at students'
home

Use reported by students

•

Primary education •

Secondary education
Computer use in general Computer use in general
Computer use at school Computer use at school
Computer use outside school Computer use in mathematics
Internet use outside school Computer use outside school
Use of computers for communication Use of Internet outside school
purposes Use of Internet at school for:
•

Downloading music
Collaboration
Use of computers for:
•

Playing computer games

•

Writing stories or reports

•

Spreadsheets
Graphical software
•

Programming
•

Downloading
•

Searching information
•
Communication

178
 Indicators of ICT in education

Use as reported by teachers

•

Primary education •

Secondary education
Use for searching information on Use in mathematics for:
Internet •

Exploration
Use in mathematics for: •
Practice
•

Exploration •

Searching information
Practice •

Analysis
•

Searching information Use in science for:

Use in science for: •

Experiment
•

Experiment •
Practice
•
Practice •

Searching information
•

Searching information •

Analysis
•
Simulation •
Simulation

Use for reading:

•

Use of computers
•
Use of software
•

Writing stories
•
Use of Internet for collaboration

Competencies

Secondary education
Self- ratings by students with regard to :
•

Using anti-virus software

•

Programming
•

PowerPoint presentation
•

Multimedia presentation
•

Downloading a file
•

Sending a file
•

Downloading music
•

E-mailing
Designing web pages

Support

Primary education •

Secondary education
Availability of educational support Shortage of support for mathematics/
(perceived by school leaders) science ( perceived by teachers)
Shortage of technical support ( perceived
by school leaders)
Person who is providing educational
support

In addition to the above , the OECD For the purpose of our study available
databases also contain data about the statistics about students ’ use of ICT
years of experience in computer use and infrastructure were extracted from
that students had at the time of data the available data bases . The statis-
collection . tics that are included in the final report
of this project are listed in Table 3 .

179
180
available
databases
Table
:Indicator
3
from
calculated
statistics

Primary
education Secondary
education

PISA
TIEA
PIEA
TIEA

Area Short
label c
+ountry
EU
per
Statistic 2001
2000
2007
2006
2003

-OTL
ICT overall
Use

Overall students
all
at
computers
used
having
% L L L L L L L

Frequent overall
weekly
students
computers
using
% L L
Chapter IV - Case studies

writing
For students
writing
for
computers
using
% L L L

Information
students
%
retrieval
information
for
computers
using L L L L

Collaboration students
collaboration
for
computers
using
% L L

Spreadsheets students
%
spreadsheets
using L L L

Programming for
students
%
computers
using
programming L L L

c-m/ailing
Ehatting students
%
for
computers
using L L L L L
/ce-mhatting
ailing

Educational students
%
software
educational
using L L
software

school
at
Use

Overall overall
computers
school
having
used
sat
%tudents
L L L L L L L L L

Frequent students
weekly
school
at
computers
used
having
% L L L L L

computers
in
school
at
students
used
having
Mathematics
%
overall L L
mathematics
overall

Schoolwork mathematics
for
students
computers
used
having
% L L
and
schoolwork
science
 school
outside
Use

Overall computers
school
outside
used
having
%
studentsL L L L
overall

Frequent shaving
school
outside
computers
used
%tudentsL L
weekly

frequent
Internet daily
school
outside
Internet
students
used
having
% L L L L

learning
in
cscoreonfidence
scale
-on
mathematics
'sAverage
Competencies
/Lelf
iking L L L L
Attitudes '
mathematics

mathematics
Average
'score
‘valuing
scale
on L L

science
Liking -celf
learning
onfidence
sAverage
‘in
scale
on
score L L L L
science
'

'valuing
science
Average
scale
on
score L L

Infrastructure
School

Educational number
of
Distribution
)o(5intervals
average
f S S S S S S S S S
per
100
computers 100
computers
per
students

Internet
computers (5intervals
)oDistribution
of
number
average
f S S S
per
students
100 students
100
per
computers
Internet

Home

Computer their
homes
students
in
computers
having
% L L L L L L L L L
availability
at
home
access
having
sInternet
%tudents
availability L L L L

Support Support S S S S
Available
educational
support
Indicators of ICT in education

provided
data;S=chool
eacher
sTtLlearner

181
 Chapter IV — Case studies

For space considerations, in the next students at grade 4 primary education

section only a limited number of statis- level ever used a computer at all. This
tics listed in Table 2 are shown. indicator is based on questions shown
in Box 1. A first observation from
2.1. Statistical indicators from Figure 3 regards the data gaps with
regard to the coverage of EU+ coun-
existing assessments: what tries and the incomplete time series.
do they show?
For most countries for which data
The review of the available interna-
existed from 2007 the conclusion
tional comparative ICT indicators on
seems warranted that nearly all stu-
students’ use of ICT and infrastructure
dents in primary education had used
revealed the following.
a computer at least once. Steady
• Many data gaps exist, for instance increases occurred from 2001 (some-
for some EU+ countries data are times exceptional as in Latvia). The
completely lacking and for many cross-study trends have face validity
others the time series since 2000 to the extent that an expected steady
are not complete. increase is indeed observed.
• Some indicators have reached the An illustration of an observation that
end of their lifetime.
requires further in-depth research
• Some results are unexpected and
concerns the statistics for Italy where
more in-depth validity research is
the percentage in 2007 is lower than
needed. in 2003 (a similar phenomenon was
• International comparative data (and
observed in the household survey
associated measurement instru-
from Eurostat). This could point to,
ments) regarding the core areas that
although not necessarily, incompara-
should be the focus of monitoring
ICT do not exist. bility of samples.

Below, a few statistics illustrating the Another interesting observation is that

observations mentioned above will this indicator has reached the end of
be shown for primary education. The its lifetime, because it is close to the
ceiling of 100 % (already even by
reader is referred to the final report for
2001 in some countries). This is due
a more comprehensive description.
to the global character of the indicator
An example of data gaps as well as (‘whether computers were used ever’)
lifetime can be observed in Figure 3 which had value in the early days of
which looks at the question of whether the introduction of computers, but

Box 1: Source of indicator

Source: PIRLS2001, TIMSS2003, TIMSS2007

Question: Do you ever use a computer (do not include Nintendo, Gameboy or other
TV/video game computers)?

Answers: Yes, no

Calculation: Percentage of yes answers

182
 Indicators of ICT in education

Figure 3: Percentage of grade 4 students having ever used a computer

Percentage students having used computers at all, grade 4

100

80

60

40

20

0
UK
AT BEfl BG CY CZ DK FR DE EL HU IS IE IT LV LT NL NO RO SK SI SE TR MK UKS JP US
E
PIEA2001 60 59 75 95 85 62 77 91 11 75 58 53 93 92 42 54 78 97 44 60 97 95 94
TIEA2003 59 68 88 5959889788 93 88
96 98 89799999
TIEA2007 89 98 99 89 93 67 93 93 96 98 99 99 94 97

Sources: PIEA2001: the IEA PIRLS assessment (reading) conducted in 2001. TIEA2003 and TIEA2007: the IEA TIMSS assessment
(mathematics and science) conducted in 2003 and 2007. For the meaning of country acronyms, see Annex A.

currently it is more appropriate to of computers by students in grade 4

zoom in on the intensity of use of substantially increased between 2001
ICT in general by students. Hence,
and 2006, particularly in countries
in Figure 4 below, the percentages
that joined the EU more recently (for
of students are shown who indicated
that they used computers at least instance, Bulgaria, Latvia, Lithuania).
weekly. The calculations are based In other countries (e.g. the Netherlands
on a questionnaire item shown in and UK), this statistic is reaching a
Box 2. ceiling and, hence, future statistics
From Figure 4 one may infer that, can be better expressed in terms of
in some countries, the weekly use daily use of computers, perhaps with

Box 2: Source of indicator presented in Chart 2

Source: PIRLS2001, PIRLS2006

Question: How often do you use a computer in each of these places?

At home, at school, other place.

Answers: Every day or almost every day, once or twice a week, once or twice a
month, never or almost never

Calculation: Percentage students answering every day or almost every day or once or
twice a week on use at home or use at school or use at another place

183
 Chapter IV - Case studies

Figure 4: Percentage of grade 4 students using computers weekly overall

Percentage students using computers weekly overall , grade 4

100
Percentage

80

60

40

20

0
BEF
BEf UK UK
AT BG CY CZ DK FR DE EL HU IS IE IT LV LT LU NL NO PL RO SK SI ES SE TR MK US
r ES

PIEA2001 4651 63 78 72 52 66 84 964 44 41 8678 33 42 67 83 37 46 92 88 89

PIEA2006 76 84 78 78 90 84 75 8695 89 83 83 71 95 89 86 66 79 84 80 89 77 98 95 93

Sources: PIEA2001 and PIEA2006: the IEA PIRLS assessment (reading) conducted in 2001 and 2006. For the meaning of country
acronyms, see Annex A.

a further differentiation towards the principle , there are ample opportuni-

number of hours per day. ties to learn with and/or about tech-
nology. The question is whether stu-
The few examples above concerned dents , in general (both inside as well
students ’ use of ICT irrespective of the as outside school), use computers
context (inside or outside school ) and for school work. This question has
the indicators show that most students been addressed in TIMSS2007 ( see
are engaged with ICT and hence , in Box 3) .

Box 3 : Source of indicator presented in Figure 5

Source: TIMSS2007

Question: How often do you use a computer for your schoolwork (in and out of
school) ?
In mathematics
In science

Answers : Every day, at least once a week, once or twice a month , a few times per
year, never

Calculation: Percentage students answering ‘every day' or 'at least once a week' or
'once or twice a month'

184
 Indicators of ICT in education

Figure 5: Monthly use in general for mathematics and science schoolwork, grade 4

Percentage of grade 4 learners using computers

for mathematics and science schoolwork
100
Percentage

80

60

40

20

0
AT CZ DK DE HU IT LV LT NL NO SK SI SE UKE UKS JP US

■Mathematics 17 39 53 32 26 27 19 35 58 50 27 34 29 54 55 30 37

Science 21 40 35 39 30 35 31 49 24 26 31 38 21 50 41 31 34

Sources: TIEA2007: the IEA TIMSS assessment (mathematics and science) conducted in 2007. For the meaning of country acro-
nyms, see Annex A.

The statistics in Figure 5 show that noted that ubiquitous use of ICT in
in most countries large groups of pri- schools is still rare . One may wonder
mary school students do not seem to whether this should be judged nega-
encounter opportunities for learning tively. Rather, the question emerges
mathematics and science with the help ‘ and so what’ ? As long as it is not
of computers (either inside or outside known whether students’ skills are
school ) . This not only points to the exist- seriously hampered by a lack of ICT
ence of digital divides in the population use in schools , this question cannot
of students, but also to underuse of ICT be answered . Hence , a plea should
in areas where many good examples of be made for measuring the extent
ICT applications exist . to which students lack skills which
evidently can be improved by more
2.2. Reflections about available sophisticated use of ICT in teaching
and learning . For planning future mon-
data
itoring , this implies that the focus (as
From the previous sections one may used to be the case in the past) should
infer that certain indicators have shift from monitoring ICT- related con-
reached the end of their lifetime. This ditions ( as was , for example , the case
is , for instance , the case concerning in SITES2006) to ICT- related student
the use of ICT on a daily basis by stu- outcomes . This implies substantial
dents . This indicator has witnessed investments in designing adequate
major changes since the start of the instruments . With political will this
current millennium , and it clearly should be possible: if mankind is able
shows that ICT is used in the daily to create instruments to measure the
life of students . However, it was also characteristics of distant planets , it

185
 Chapter IV - Case studies

is certain that, with adequate invest- The profit for countries consist of being
ment , it should be possible to offer able to use measures that have rela-
educational actors the instruments to tively high quality and are extensively
observe what is happening in educa- tested , whereas where other countries
tional practices . use the same measures , compara-
tive data also become available with-
out the need for a heavy international
3. Recommendations overhead .

Indicators for ICT- related student

It is recommended that studies are
outcomes will have to be developed .
undertaken in which the characteris-
International organisations (the EU ,
OECD, Unesco) could stimulate this tics and impact of existing ICT- related
development through their regular school monitors are investigated .
research programmes . A first step
It is recommended that interna-
could be to generate frameworks for
ICT use in the most important core tional organisations coordinate their
competency areas and to create for efforts to develop a vision regarding
each of these areas item banks con- the future of monitoring educational
taining concrete performance tasks change (of which ICT is one compo-
that are perceived as relevant by a nent) . For the EU , a key question is
substantial number of countries . If, whether this monitoring will be run
in the short term, the development fully under the auspices and control
of concrete performance tasks is too of the Commission addressing the EU
complex, it is advised to focus at first core competency areas .
on definitions of these tasks and to
monitor the extent to which students This would be a vision for the long
have opportunities (in and outside term ( 10–15 years) which could set
school ) to acquire the competencies the scene developing appropriate
required by these tasks. In relation solutions for organisational , financial
to this , it is recommended that inter- and methodological issues . Several
national organisations coordinate elements that have been dealt with
the development and elaboration of in this chapter (and Chapter II ) could
frameworks for monitoring . For the be part of such a vision , such as
developers of indicators for the other (a ) capitalising on highly innovative
areas , it is recommended that the indi- forms of monitoring (through online
cator definitions are tuned to the com- data collection and authentic tasks) ,
petency frameworks . (b) holistic and multi-level monitor-
ing ( e.g. including school monitoring)
It is recommended that international and (c) tailored monitoring allowing
organisations stimulate the creation for flexibility according to the indicator
and use of a worldwide instrument needs of countries . Part of this vision
bank containing measures that can be would be to sketch the responsibilities
used for assessing the development and roles of the different international
of ICT in education . Substantial pri- organisations involved in regular inter-
orities could be based on the overview national comparative assessments . In
provided in Table 1. Incentives might the short term, the EU (but maybe this
for instance consists of co-financing is also applicable to APEC and other
national projects in which measures organisations) could embark on exist-
from this instrument bank are used . ing assessments that are run by OECD

186
 Indicators of ICT in education

and IEA in order to explore which wrong with the students’ skills for which
desirable indicators can be included in ICT could offer solutions? ’ .
these assessments and which options
are feasible for guaranteeing an ade- An implication of our study is that, in
quate geographical coverage of the years to come , intense efforts need
EU Member States . to be undertaken to define 21st cen-
tury skills , and the opportunities that
schools should offer to students to
4. Summary learn with and about ICT. This calls for
and discussion international cooperation , as it implies
a substantial investment in the devel-
This article started with questions about opment of new curricula and assess-
monitoring ICT in education . It seems ment methods , which would probably
thatclearly a need for monitoring ICT in outstrip the manpower and financial
education exists . But what then should capacities of individual countries .
be monitored? The main policy issues What then is the role of the European
were identified in this article and the Commission to ensure that appropri-
existence of international comparative ate and efficient methods for moni-
ICT indicators was reviewed . It was toring will ultimately be in place? In
argued that what ultimately counts in this respect many potential actions
education are the skills and perform- could be considered of which the
ances of students . The overarching most prevalent ones were presented
question is : Are students well enough in Section 1.3.2 . Still , the future tra-
prepared during compulsory schooling jectory is paved with uncertainties as
to adequately function in the informa- much internal EU and external nego-
tion society? The answer, as implied by tiation with third parties will be needed
the previous sections , is that we do not before a workable operational plan
have sufficient international compara- can be made. Nevertheless , the mes-
tive data available to address this ques- sage appearing from our study is that
tion . At the moment we are inclined to the Commission has a very important
monitor conditional factors , but this potential role in stimulating and facili-
leaves open the question : ‘What is tating these future developments .

References

Plomp, T. , Anderson , R. E., Law, N. and Quale , A. (eds) . (2009) . Cross national
policies and practices on information and communication technology in educa-
tion (2nd ed .) . Greenwich , CT: Information Age Publishing .

Trucano , M. (2005) . Knowledge maps: ICT in education. Washington DC :

infoDev/World Bank .

187
 Chapter IV — Case studies

Annex A. Target countries and country abbreviations

EU+ countries

Austria AT France FR Netherlands NL

Belgium BE Croatia HR Norway NO

Bulgaria BG Hungary HU Poland PL

Cyprus CY Ireland IE Portugal PT

Czech Republic CZ Iceland IS Romania RO

Germany DE Italy IT Sweden SE

Denmark DK Lithuania LT Slovenia SI

Estonia EE Luxembourg LU Slovakia SK

Greece EL Latvia LV Turkey TR

Spain ES FYR Macedonia MK United Kingdom UK

Finland FL Malta MT

Non-EU countries

Japan JP USA US

Acronyms used in some charts showing indicator statistics

Belgium (Flemish) BEfl UK (England) UKE

Belgium (French) BEfr UK (Scotland) UKS

188
Impacts of ICT use on school
learning outcome
Heeok Heo
Sunchon National University, Korea

Myunghee Kang
Ewha Womans University, Korea

I. Introduction is posed on the imperative for the

effective use of digital technologies in
At the start of the 21st century, human education .
society is facing an information and
communication revolution , resulting Many efforts have been made to
in the advent of new technologies . adopt information and communication
Computers and network technology technologies (ICT) to promote learn-
have influenced a range of societal ing excellence in various educational
and cultural aspects of life as well settings . At national and institutional
as individual experiences . People in levels , educational policies and regula-
modern societies have different life- tions have been established to support
styles , thinking styles , ways of work- the educational use of ICT. In schools
ing and new communication patterns and classroom settings , teachers and
compared to previous societies. This school administrators are attempt-
has been well proven by a variety of ing to find the best ways to use ICT
research findings in human and social technology for their teaching and stu-
science studies. Many enquiring dents’ success . However, accomplish-
scholars and practitioners have made ments that are convincingly the result
an effort to discover the effects of of the direct causal impact of ICT use
technologies on individuals’ lifestyles are not always easily identifiable . It
and communication modes. It may is even hard to ascertain the impact
be assumed that different lifestyles of ICT use in a simple way, because
result in different learning styles and many other factors besides ICT itself
outcomes . Some authors claim that might influence the ICT use in the indi-
digital technologies could be powerful vidual’s genuine growth in education .
transformational tools in individuals’ Suppose that a 10th grader performed
learning and growth . Even commer- better in mathematics after using ICT
cial videogames could have a positive in maths classes for a certain period
impact on cognitive development and of time. Of course ICT is an important
skills . Some other studies present the tool for the student to improve his/ her
negative influence of technology use maths performance , but there might
on human behaviours (Meyo , 2009) . be other factors improving the per-
Even though there are inconsistent formance , such as the way in which
findings on the impact of advanced he/she uses ICT, learning contents,
technologies in human life , no doubt teachers’ support, etc. In spite of all

189
 Chapter IV — Case studies

the limitations, salient studies to dem- guide further investigations into the
onstrate the impact should be carried implications of the findings.
out to promote successful educational
implementation. II. ICT use in school
Currently,thereareasignificantnumber settings
of initiatives assessing and monitoring
the quality of ICT use and its impact School experience was formerly a
on education. SITES (the second critical resource for humans to obtain
information technology in educational knowledge and skills in their lives, but
study), sponsored by the International other sources and methods are now
Association for the Evaluation of available to access new information
Educational Achievement (IEA), is an and to interact with people in today’s
exemplary study, which identifies and knowledge-based society. Computer
describes the educational use of ICT and Internet technologies will probably
across 26 countries in the world. The open a door that will make human life
study collected data from different different. While, in the late 20th cen-
stakeholders, and compared and inter- tury, students asked questions to their
preted the results based on the rela- teachers when they had a question in
tionships of various factors affecting a perplexing situation, children in the
the educational use of ICT (Pelgrum 21st century might choose access to
and Anderson, 1999; Kozma, 2003). the Internet first and use information
The OECD has also emphasised the search engines like Google for solv-
need for clarifying the effects of ICT ing their questions and problems.
use comparing PISA results. European However, school is still an essential
Schoolnet published a technical report environment for individuals’ experi-
to provide comprehensive information ences on the road to success and to
on the impact of digital technologies promote better adjustment in a soci-
on learning and teaching using inter- ety. That is why many educational
national evidence (Balanskat, Blamire practitioners and policymakers pay
and Kefala, 2006). attention to utilising ICT for improving
education in school settings.
In the meantime, the Korean Ministry
of Education, Science and Technology There are three major uses of ICT
(MEST) has the opportunity to work in school education (Taylor, 1980;
on the impact studies of ICT use on Smaldino, Lowther and Russell, 2008;
educational performance in coopera- White, 1997), as follows.
tion with the OECD. For better under-
standing of the relationships of ICT First, ICT is used to improve teaching
use and educational performance, and learning — this includes the use
this paper will provide a theoretical of application software as a teaching
mapping of various factors affecting and learning tool. Teachers can use
ICT use in education by using a con- ICT for presenting information to their
ceptual framework, which was a part learners, for assessing and monitor-
of the findings of the Korean study, ing learners’ achievements and for
and a summary of key findings of a their own professional development.
nationwide investigation conducted Learners may use ICT for getting
in Korea. Constructing a conceptual access to new information, augment-
framework is a useful way to connect ing existing knowledge, sharing what
all aspects in a study, and then it may they have learned with others, work

190
 ICT use and school learning outcome

ing on school projects with peers and include the mastery of content knowl-
acquiring new knowledge and skills . edge , basic skills and attitudes as well
as core competencies needed in this
The second use is to enhance admin-
modern society. On the teachers’ side ,
istrative productivity — such adminis-
educational performance might refer
trative services as grading and keep- to teaching competencies , pedagogi-
ing records in schools are vital for cal content knowledge and teachers’
tracing a student’s learning history roles in the learning processes and
and monitoring each student’s per- outcomes . For educational administra-
formance . The automated administra-
tors , educational performance relates
tive services using ICT are beneficial to drop - out rate , underachievement in
to all stakeholders in schools .
school work , entrance rates to higher
Third , ICT is used to build information lit- education , reputation ratings from
eracy — the school curriculum includes stakeholders outside of schools and

ICT as a learning object for students . so forth . The learner’s performance , in

The ultimate goal of ICT education most cases , will be a key component
is to develop ICT skills for problem- to assess educational performance in
solving in real life. The main contents school settings. That is why we, first,
may include computer architecture and need to clarify the impact of ICT use on
cyber ethics . ICT is an indispensable educational performance in learning
tool for people living in this society. and from the learners’ point of view.
Teachers who have ICT skills can effec-
tively prepare teaching materials using III . Conceptual
computers and present complex ideas framework of ICT
better than those who have fewer ICT
skills . Students who have ICT skills can
use and educational
also be successful in their learning and performance
achieve greater outcomes than others
who have fewer ICT skills . Constructing a conceptual framework
and indicators is a good starting point
The irreversible influence of ICT will for investigating a complicated phe-
eventually revolutionise the way we nomenon , and then providing inte-
learn and teach but the revolution grated perspectives , even though the
may be not remarkable viewed over a process has some limitations (Kikis ,
short time. In particular, the changes Scheuerman and Villalba , 2009) . In
in educational settings are very slow. this paper, a conceptual framework
It is also hard to determine the posi- indicates various factors that pro-
tive influence of ICT use in educational foundly influence both ICT use and
performance in schools , because the educational performance of learn-
assessing the impact is complex, and ers . This framework was generated as
lots of factors affect the processes and a result of comprehensive literature
outcomes of ICT use (White , 1997) . reviews and expert reviews . As shown
Educational performance in school in Figure 1 , the factors are classified
settings can be interpreted in various into three levels surrounding ICT use
ways . From the perspective of learn- and educational performance: the
ers, educational performance may classroom setting (micro level ), the
refer to learning achievement and out- school and local community (meso
comes obtained from the prescribed level ) and regional and national enti-
learning contents and activities. These ties (macro level ) .

191
 Chapter IV - Case studies

Macro level
Mesolevel
Micro level

Teacher practice
(Methods/roles/collaborations)

ICT Educational
use Curriculum goals and contents performance

Students (SES, experience with technology/activities/

produces/roles/communications)

Teacher (Ed. background/innovation history/experience with tech./norms)

classroom factors (organisation/size/type and arrangement of tech . facil.)

School types and location/school organisation/local culture/intended curriculum

staff development/ICT infrastructure/technical support/innovation history

Economic forces/cultural norms/ed. goals and problems/ed. funding/

curriculum standards/teacher standards/ICT policies/ICT infrastructure

Figure 1 : The conceptual framework of ICT use and educational performance

ICT use and its impact on educational portable devices , such as cellular
performance may be influenced by phones , are included in ICT use as
various factors such as the personal well . Individuals may use ICT in their
attributes of teachers and students , daily lives, and their use may have a
and curriculum and teaching practices considerable influence on personal
at the micro level . At the meso level , the performance . The following three
school environment and its surround- dimensions are employed to clarify the
ing factors may affect the use of ICT patterns and frequency of ICT use .
in educational practice . At the macro
level , ICT use and educational per- Places in ICT use
formance may be influenced by socio- Place in ICT use is divided into two cat-
cultural norms , economic forces and
egories , in -school and out-of-school ,
technological advances . This paper based on the location where learners
focuses on understanding the effect of use ICT. Most education in schools
ICT use on educational performance
focuses intensively on preparing stu-
at the micro level and controls meso
dents to acquire academic skills and
and macro level variables as con-
life competencies . On the other hand ,
stants either by random selection or
interest has recently been growing in
by setting research boundaries .
ICT use for informal learning outside
of schools and the idea that students
ICT use can benefit from the extracurricular
use of ICT. There is no doubt that
ICT is characterised as a networked individuals spend much time using
computer that can process and com- ICT in daily life . They make use of
municate information in this study. ICT for finding information , shopping
However, stand -alone computers and for commercial goods online , chatting

192
 ICT use and school learning outcome

In school

Out of school Social context

Individual context

Learning Entertainment

Figure 2 : The dimensions of ICT use

with others and playing online games . a project for solving a problem can
Along with these digital lifestyles , ICT use software to present ideas and
use by children and youths might have thoughts . A social context refers to a
some influence on their thinking and setting in which two or more learners
learning styles in schools . use one computer together, or in which
a learner works with friends to perform
collaborative tasks online . Such tools
Purposes ofICTuse
as wikis , blogs and bulletin boards
The category of purposes of ICT use might be used by learners to interact
indicates a set of classifications for the
with others . For example , students
reason of ICT use and the intentional-
could use a wiki for the collaborative
ity of learning, which include learning development of a project.
and entertainment. Learners may use
ICT for their learning needs , such as
obtaining knowledge, solving complex Educational performance
problems and acquiring new skills .
Experiences that learners have with- The meaning of educational perform-
out any specific intention of learning ance is vague and diverse depending
may be categorised as entertainment. on domains , despite the long history of
research and attention from academia
ICT also creates new entertainment
environments in which learners can as well as practitioners . Based upon
previous studies , educational per-
socialise with friends and play games .
formance may be conceptualised as
a futuristic concept that encompasses
Contexts in ICTuse
not only the traditional concept of edu-
Learners may work individually or cation but also the extended version
socially with peers while using ICT. In of human learning . The educational
an individual context, learners use ICT performance of learners is defined as
alone without collaborating with others . the processes and results of perform-
For instance , a student involved in ance , which are revealed internally

193
 Chapter IV - Case studies

and externally through the integration Traditional educational taxonomies

of essential knowledge , skills and atti- emphasised cognitive categories with
tudes , and the continuing construction less, if any, emphasis on the affective
of experiences with ICT use . and sociocultural dimensions . As the
world evolves into a more post-mod-
To make operational definitions of ern society, however, where multiple
complex educational performance, the voices are heard , its citizens , includ-
study suggests a two-dimensional tax- ing the younger generation , should be
onomy model , which is composed of sensitive to socio cultural performance .
six cells within the two dimensions: ( 1 ) The presence of ubiquitous computing
three performance domains (cognitive , technology connected in a global net-
affective and sociocultural) by (2) two work will also accelerate sociocultural
behaviour levels ( internal , external). dynamism .
This model utilises the approaches
of Bloom’s taxonomy of educational These categories are assumed to lie
objectives and Krathwohl’s taxonomy along a continuum from internalised
of affective competencies . It also (or centripetal ) behaviour, to external-
puts more emphasis on socio cultural ised (or centrifugal) behaviour. The
aspects and less on psycho motor continuum underlying the behaviour
aspects than other approaches . levels is assumed to be the orientation
of performance; that is, internal com-
The performance dimension contains petencies are believed to be oriented
three categories : cognitive , affective more toward the learners themselves ,
and socio cultural . These three cat- while external competencies relate
egories are assumed to be mutually more to the world and others outside .
independent and , at the same time , In the new millennium , learners are
to be critical for learners in the future . expected to be more participatory and

Internal
competencies

External
competencies

Cognitive Affective Socio-cultural

domain domain domain

Figure 3 : Conceptualisation of educational performance in learners' perspectives

194
 ICT use and school learning outcome

active practitioners who will contribute ance begins with open-mindedness

to the betterment of the community toward uncertainty. Members should
and the world. To live as active prac- also be equipped with global com-
titioners, learners should understand munication skills such as foreign
the cognitive, affective and sociocul- language proficiency and cross-cul-
tural aspects of the world to make it a tural understanding.
more liveable place. Recent epistemo- • Sociocultural-external competency:
logical perspectives such as those of If one fully recognises the presence
Leontev’s activity theory and Lave and of others and acquires communica-
Wenger’s situated cognition theory tion skills, then one may be ready to
also confirm this internal-to-external collaborate with others to make the
developmental orientation. community a better one. Assuming
proactive roles, such as those of
The following descriptions briefly
leadership, performing social serv-
explain the six cells constructed by ices and maintaining strong ties with
three performance domains and two others in a community are some
behavioural levels. exemplary behaviours.
• Cognitive-internal competency: This
refers to the individual’s internal abil- IV. Findings from
ity to select and gather information,
and construct knowledge. a nationwide
• Cognitive-external competency: Cogni- investigation in Korea
tive-internal competency is mani-
fested as useful tools for transform- Based upon the conceptual framework
ing the individual’s situated life- presented briefly in the previous chap-
world. Effective problem solving is a ters, a nationwide investigation was
relevant example. conducted in Korea. Measurement
• Affective-internal competency: To scales were developed for measuring
live as an independent and mature the types of ICT use and educational
member of many overlapping com- performance of learners in the investi-
munities, a learner should have a gation. The scales consisted of 42 test
set of internal values to recognise items in ICT use and 33 items in edu-
the importance of oneself as well cational performance using the self-
as of others. Individuals should also reporting method with a 4-point Likert
be able to appreciate social norms scale. Expert reviews and pilot tests
such as the importance of honesty were conducted for validation. During
and integrity. the expert review, the experts evalu-
• Affective-external competency: Mature ated and provided comments on both
individuals are those who act in conceptual frameworks and measure-
accordance with their own true ment scales quantitatively and quali-
values in adverse as well as favour- tatively. Item analyses, reliability tests
able situations. Self-efficacy, goal- and validity tests were employed to
setting and perseverance are a few optimise the measurement tools for
examples. ICT use and educational performance
• Sociocultural-internal competency: through three pilot tests. The nation-
As future societies will be more wide investigation was conducted
socially diverse, individuals need among 1 071 first-year high school stu-
to tolerate and appreciate one dents (15-year-olds) during the three
another. This sociocultural perform- weeks at the end of a school term.

195
 Chapter IV - Case studies

The overall interpretation of the results ( Ferguson , 2007). However, participa-

in the investigation indicates that ICT tion in online communities as an activ-
use and educational performance ity outside schools positively affected
were significantly connected . ICT sociocultural competencies rather than
use has a positive influence not only the other two competencies .
on cognitive competencies enhanced
through traditional education systems , Third , ICT use in individual contexts
but also on affective and sociocultural
resulted in a more positive influence
competencies required for individuals on learners ’ educational perform-
in future societies . The findings from
ance than using it in social contexts .
the investigation are summarised as
When individuals use ICT for their
follows ( Kang , Heo, Jo, Shin , Seo and
Shin , 2008 ) . learning outside schools , it possibly
enhances the cognitive , affective and
First , using ICT outside school influ- sociocultural competencies of their
ences an individual’s educational educational performance . Using ICT
performance more than using it in in social contexts also has a small
school settings . In most cases , learn- positive impact on their educational
ers can get access more conveniently performance . Collaborative learning
in homes and commercial computer outside of schools as a learning activ-
rooms outside schools than in schools .
ity in a social context may enhance
Schools still provide limited access educational performance . It indicates
to learners . Teachers , probably, are
that collaborative learning while solv-
responsible for ICT use in both class
hours and after classes . Most activi- ing real problems, and working on
ties using ICT in class hours are to authentic projects must be included
present information to students by for better learning . Web 2.0 tools , one
teachers . Few opportunities may be of the recent technologies , have been
provided to students to use computers widely used in many situations and are
except for special needs . It suggests expected to provide more opportuni-
that we should rethink how to use ties for sharing ideas and cooperating
ICT in schools and integrate learners’ among individuals in social contexts .
experiences in informal settings into
school learning . It is evident that ICT use affects learn-
ers’ educational performance posi-
Second , when individuals use ICT for tively, but its impact is mainly on the
their learning rather than entertain- cognitive development within their
ment, it generates a positive impact on educational performance . This study
educational performance . It means that assumed that the integration of cogni-
such activities as playing games and tive , affective and sociocultural com-
listening to music may not enhance
petencies is important for individuals
educational performance much , even to be successful in current and future
though some studies on the educa-
tional use of games reported its posi- society. Even though ICT use did not
influence affective and sociocultural
tive impact on learning outcomes . As
an investigation on types of games that competencies much , more attention
most individuals use for their enter- should be paid to possible methods for
tainment reports , violent games may using ICT for developing those compe-
spread more than other types of games tencies in and out of school settings .

196
 ICT use and school learning outcome

V. Conclusion ance . Sometimes , individuals use ICT

in personal contexts (home, cafés and
The biggest challenge in assessing pupils’ houses ) more than in schools
ICT impact on learners’ educational and then those experiences can affect
performance is to identify the distinctive ICT use in schools in some ways .
influence of ICT use on it. As mentioned
earlier, educational performance is a Most activities in a school setting
vague concept and difficult to define might be predetermined by teachers
and to measure , and various attributes and through national standards , but all
of learners and complicated features of experiences and activities outside of
the external environment surrounding schools cannot be estimated precisely.
learners might affect their performance When another new world where cyber-
in the present and future. However, it is space and physical space are com-
not a matter of whether the impact of ICT bined in one space opens, the apparent
use can be exactly measured. We need distinction between formal and informal
to pay more attention to what and how
learning may disappear. Since this is
to measure , and to make interpretations
the case , ICT use in informal learning
to promote better performance . The
that happens to learners unintention-
following aspects should be taken
into account in possible further impact
ally should be paid more attention by
studies on ICT use in education . educational practitioners .

Third, the quantitative and qualitative

First , more interest needs to be taken
approaches in assessing and inter-
in making connections and studying
preting the impact of ICT use in edu-
the relationships among various fac- cation should be combined for the
tors that influence ICT use in educa-
tion . This paper elucidates diverse comprehensive understanding of this
factors on three levels , referring to emerging phenomenon . While the
school settings and the related supra quantitative approach answers best
systems . Among them , some factors to problems requiring a description of
relate directly to learners’ performance trends or an explanation of the relation-
and some others indirectly. It will lead ships among variables , the qualitative
to the construction of another frame- approach will address questions refer-
work for comprehensive interpretations ring to the exploration of little-known
and future development . situations or a detailed understanding
of a central phenomenon (Creswell ,
Second , ICT use in informal learning 2008) . Unknown factors affecting ICT
must be examined for a better under- use in education may be found through
standing of ICT use in learners’ perform- qualitative methods of evaluation .

References

Balanskat, A. , Blamire, R. and Kefala, S. (2006). The ICT impact report: a review
of studies of ICT impact on schools in Europe. Retrieved 01.08.09 , from http ://
insight.eun.org/shared/data/pdf/impact_study.pdf
Creswell , J. W. (2008) . Educational research: planning, conducting, and evalu-
ating quantitative and qualitative research (3rd ed . ). Upper Saddles River, NJ :
Prentice Hall .

197
 Chapter IV — Case studies

Ferguson, C. J. (2007). The good, the bad and the ugly: a meta-analytic review
of positive and negative effects of violent video games, Psychiatric Quarterly,
78(4), 309–316.
Kang, M., Heo, H., Jo, I, Shin, J., Seo, J. and Shin, S. (2008). The new mil-
lennium learners and educational performance: the 2nd year report. Technical
report. KERIS.
Kang, M., Kim, D., Lee, I and Heo, H. (2007). The new millennium learners and
educational performance. Background paper of CERI-KERIS international expert
meeting on ICT and educational performance. Cheju Island, South Korea.
Kang, M., Kim, D., Lee, I, Heo, H., Seo, J. and Shin, S. (2007). The new mil-
lennium learners and educational performance: the 1st year report. Technical
report. KERIS.
Kikis, K., Scheuerman, F. and Villalba, E. (2009). A framework for understand-
ing and evaluating the impact of information and communication technologies in
education. A paper presented at the international expert workshop on assessing
the effects of ICT in education — Indicators, criteria and benchmarks for inter-
national comparisons in Ispra, Italy.
Kozma, R. B. (2003). Technology, innovation, and educational change: a global
change. Eugene, OR: International Society for Technology in Education.
Lim, Cher Ping (2006). The science and art in integrating ICT in Singapore
schools. Singapore: iT21 (Singapore) Pte Ltd.
Meyo, M. J. (2009). Video games: a route to large-scale STEM education?
Science, 323, 79–82.
Pedró, F. (2006). The new millennium learners: challenging our views on ICT
and learning.
Pelgrum, W. J. and Anderson, R. E. (1999). ICT and the emerging paradigm for
lifelong learning: a worldwide educational assessment of infrastructure, goals
and practices. Enschede: Printpartners Ipskamp.
Smaldino, S. E., Lowther, D. L. and Russell, J. D. (2008). Instructional technology
and media for learning (9th ed.). Upper Saddles River, NJ: Pearson Prentice Hall.
Taylor, R. (1980). The computers in the school: tutor, tool, tutee. New York:
Teachers College Press.
White, J. N. (1997). Schools for the 21st century. Harpenden: Lennard
Publishing.

198
ICT impact data at primary school level :
the STEPS approach
Roger Blamire
European Schoolnet, Brussels ( 1 )

Introduction English . There are gaps in what is known

about other countries . No doubt some evi-

The relationship between information dence exists and efforts should be made

and communication technologies ( ICT) to identify it and ensure it is translated . If

and improved teaching and learn- it does not exist , efforts should be made
ing has increasingly been the focus to support transnational studies to ensure
of interest for education policymak- good coverage and reliable results .
ers , researchers and other education (Balanskat, Blamire and Kefala, 2006)
stakeholders after two decades of ICT
investment and integration in schools The ‘ Study of the impact of technology
across Europe . What impact or dif- in primary schools’ (STEPS) sought to
ference can ICT make in education close this gap and to provide a more
systems ? How can ICT be a motor for balanced and comprehensive picture
improvement, progress , educational of the impact of ICT on primary educa-
change and innovation ? The inter- tion . The study was commissioned by
relationship between policy, practice the European Commission Directorate-
and research has likewise become General for Education and Culture (2)
an important focus within the area of and undertaken jointly by European
evidence-based policymaking . Schoolnet (EUN) and Empirica GmbH
between January 2008 and June
The ICT impact report — a review of 2009. Empirica was responsible for
studies on the impact of ICT in educa- the LearnInd survey of 30 000 teach-
tion produced by European Schoolnet ers and head teachers in 27 European
countries for the Directorate- General
in the framework of the European
Commission’s ICT cluster -
revealed for the Information Society and Media
considerable gaps in what is known at (Empirica, 2006): this provided quanti-
a European level about the impact of tative evidence on the access and use
ICT in schools . of ICT in European schools in 2006
generally in primary and secondary
Evidence or access to evidence on the education . Based on the experience of
impact of ICT in schools is unevenly both organisations in the field and the
spread across Europe . Many of the find- application of different approaches and
ings relate to the United Kingdom and to methods ( quantitative and qualitative)
England in particular. They are mostly in
(2) This study was financed (at a cost of
EUR 232545) by the European Commission .
(1 ) This paper draws on longer studies in STEPS Contract EACEA-2007-3278. Opinions presented
written by the author, A. Balanskat, T. Hüsing, in this chapter do not reflect or engage the
W. Korte , B. van Oel and L. Sali . Community. © European Commission .

199
 Chapter IV - Case studies

for gathering and analysing develop- Approach

ments in ICT in education , European
Schoolnet and Empirica worked in The methodological challenge was
a complementary way to paint a rich considerable . Strategy and impact
portrait of the impact of ICT on primary were the two underlying concepts of
education . the STEPS study. They can be seen
as the two ends of a chain : a strat-
The main purpose of STEPS was to egy is always designed with the aim
produce a comparative analysis of the of having impact. Strategies and poli-
main strategies for the integration of cies are shaped at several levels , and
ICT in primary schools in the EU-27 , this makes policy implementation and
Iceland , Liechtenstein and Norway,
evaluation a difficult task, especially
their impact and future development
because they involve attitudinal and
perspectives . The study aimed to iden-
work process changes . How do we
tify the impact of ICT at three levels: on know whether it was the interven-
learning and learners, on teachers and
tion that made the impact without
teaching and on primary school devel-
taking other factors into account? Can
opment plans and strategies . It sought
to identify the main drivers and ena-
change attributed to an ICT strategy be
isolated from other factors? How was
blers for effective and efficient use of
ICT, and to propose recommendations policy implemented in practice? How
on the integration of ICT in education do we measure impact? As Gordezky
etal . note :
for policymakers and stakeholders .

The challenge was considerable: to Changing a large complex school system

identify commonalities across 209 866 is a messy business. Results from change
schools (3 ) offering primary-level edu- efforts are often unpredictable , show up
cation in the 30 countries surveyed , in ways that are difficult to quantify, and
ranging from 14 in Liechtenstein to can lead to counterintuitive and undesir-
55 329 in France . Moreover, compul- able consequences . (Gordezky, Marten
sory schooling in the countries cov- and Rowan , 2004)
ered begins between the ages of four
and seven and most primary schools A number of strategic layers play a
are managed, funded and governed role when looking at the implementa-
by the local municipal councils and so tion of ICT. Strategies can be found
data tend to be held locally and are from societal level all the way down to
not always available . an individual teacher making strategic
decisions on when and how to use ICT.
The final report amounted to some These levels include , first, society at
66 separate reports totalling over large and how it tackles ICT; second ,
1 000 pages . In the following sections, the education system (including policy
the approach and main findings are targets and the main actors) . The third
outlined .
and fourth layers are formed by gov-
erning bodies ( e.g. regional or local
(3) A primary school is defined as one that authorities ) and by individual schools .
educates children between the ages of four A final layer is the ‘ end-user’ : often
and 11. The figures do not include private schools
or kindergartens . A number of countries have
the teachers , but also the learners
all -age schools or combine primary and lower themselves . These end - users develop
secondary schools in one school . strategies to comply with national ,

200
 ICT at primary school level

regional and local requirements ; and learners ) . Evidence came from five
of course to satisfy their own targets . sources :

Impact can be described as the over- a policymaker survey in the 30

all achievement of an intervention on countries to provide an overview of
these domains within the educational policy approaches to ICT in primary
system and can be described by a education ;
•

variety of qualitative and quantitative an analysis of quantitative data from

indicators such as ‘improvements in over 18 000 teachers and head
national tests’ or ‘improved learning in teachers interviewed for the 2006
schools ' depending on the policy target. LearnInd ICT benchmarking survey
It is the end point of an intervention (Korte and Hüsing , 2006) ;
•

involving input, process , output and a review and analysis of the evidence
outcome . Isolating the variable which from over 60 research studies pub-
actually causes the impact is problem- lished in more than 20 countries ;
atic in education . Within STEPS , the • 250 responses to a school survey
following definition of impact was used: seeking qualitative insight into the
‘ a significant influence or effect of ICT impact of national strategies in
on the measured or perceived quality schools , and the identification of
of (parts of) education ’ . The study was good practices via self-reporting ;
based on the assumption that not all • 25 case studies documenting the
impacts are positive or intended , that good practices identified .
not all policies are implemented as
planned and that classroom practices Policysurvey
are hard to change (see McLaughlin , The policy survey was the main tool
2005) . Although evidence about effec- for deepening knowledge of national
tive strategies has been identified , and regional strategies and was in
policies are generally shaped to local three parts:
contexts and practices take a long
time to change. Years of ICT impact •

general information about the char-

studies confirm this complex picture . acteristics of the primary school
ICT impacts cannot always be meas- system ( ranging from the number of
ured through test scores — sometimes schools , curriculum , teachers ’ pay
no gain in test scores can be found and conditions to school govern-
and no direct link can be established ance) and emerging policy trends
between an ICT intervention and and priorities ;
improved attainment. One solution in • the use of ICT in primary schools ,
this study was to look at impact not covering ICT resourcing , teacher
only in attainment (hence the broad skills development and ICT support,
definition of impact) but also to look at the place of ICT in teaching and
how ICT improves processes of teach- learning;
ing and learning within the school . • ICT policy for primary schools ,
including ICT in education policy,
A multi - perspective approach was examples of strategies and good
adopted for STEPS, taking into practice.
account evidence from stakeholders
(policymakers , teachers and head The policy survey was completed
teachers ) , research and site visits between July 2008 and March 2009 .
to schools (including interviews with National correspondents (in most

201
 Chapter IV — Case studies

cases nominated by ministries of • technical infrastructure in schools,

education) gathered information on including computer equipment and
national or regional policy contexts, Internet connectivity;
often translating documents only • the use of ICT in class and for edu-
available in the local language. This cational purposes;
was supplemented by information • ICT competence of teachers;
from other STEPS sources (LearnInd • barriers to ICT use as perceived by
data, school surveys and the lit- teachers and head teachers.
erature review) and by data in the
public domain (EUN insight country The results of the LearnInd data anal-
reports (4), Eurydice (5)). ysis were presented and discussed in
Report 2, ‘LearnInd data results and
The results of the policymaker survey analysis’. Summaries of the data anal-
were analysed and presented in ysis per country can be found in the 30
Report 1, ‘Policy survey results and country briefs.
analysis’, providing an overview and
comparison of policies and types of Literature review
strategies. Summaries of national pol- The main scope of the literature review
icies were also included in 30 country was qualitative rather than quantita-
briefs. tive, in order to ensure sufficient cov-
erage from participating countries. The
Teacher survey aim was to identify and summarise in
English recent studies (up to four per
Quantitative data in the LearnInd sur-
country) that gave important insights
veys used standardised interviews with
in the field and to include countries
head teachers and class teachers (a
where information access has so far
random sample) in 27 European coun-
proven to be difficult due to language
tries collected in 2006. The sample
barriers and fragmented research as
was split between primary, lower sec- revealed by the ICT impact report.
ondary and upper secondary schools,
but STEPS concentrated on the results The appointment of committed key
of primary schools only. In total, 12 379 experts from existing partner networks,
interviews with classroom teachers and from a wide geographical area (north,
6 449 interviews with head teachers of south, east and western Europe) and
schools which offer primary education especially in those countries where
were carried out. until now information had been unob-
tainable, enabled important studies in
The use of ICT in European primary those countries to be identified, and,
schools was measured using the fol- most importantly, to make the results
lowing criteria: of these studies more widely known.
Summaries of research in each country
• teachers’ attitudes and motivation were presented in the country briefs.
with regard to ICT, including per-
ceived impact of ICT;
School survey
The STEPS school survey aimed to
(4) http://insight.eun.org/ww/en/pub/insight/misc/
country_report.cfm gather examples of the integration of
(5) http://eacea.ec.europa.eu/portal/page/portal/ ICT in primary school daily activities
Eurydice/Overview/OverviewByCountry and to obtain a snapshot of current

202
 ICT at primary school level

views of teachers on ICT use and • include the voice of teachers , pupils
impact in their school . The survey and school leaders ;
consisted of an online questionnaire • complement the evidence base
with both closed and open questions by an in-depth investigation and
in nine languages . observation .

In total , 25 contrasting schools in

Case studies
13 countries were selected for a case
The purpose of the case studies was study visit. The case study (written by
to find out more about effective use an evaluation team) followed a fixed
of ICT and enablers or barriers at dif- format . At school , teacher and learner
ferent levels of the education system . levels, the reporters were asked to
The case studies sought to show how highlight impact, enablers and bar-
the strategies of policymakers , schools riers . All the case studies were ana-
and teachers impacted on teaching lysed in terms of themes , issues and
and learning. typologies and presented in Report 5 ,
‘Case study analysis’ .
The case studies were designed to
show the richness of implementation
and also to describe a number of typ- Key findings
ical situations in sometimes quite dif-
ferent schools and contexts . In most
An analytical framework was devel-
oped early in the project and used for
cases , the visits were related to a
the integrated analysis and presenta-
specific application of ICT or a project
tion of the overall findings . The frame-
which had been identified by ministries
of education or schools themselves as
work visually captures key elements
and represents them in a logical and
demonstrating good practice. concise way. The analytical frame-
Within STEPS , the case studies work is built around a core of teachers ,
learners and the school as a whole .
helped to :
The framework helps to describe the
• visualise what happened in the context in which ICT is introduced and
classrooms ; implemented .

Learners
Teachers
School
Education system
Society

eb eb eb eb eb
ICT STRATEGY ICTSTRATEGY ICTSTRATEGY ICT STRATEGY &ATTITUDE ICTATTITUDE
RESOURCES RESOURCES RESOURCES RESOURCES RESOURCES
SUPPORT SUPPORT SUPPORT SUPPORT
CURRICULUM CURRICULUM SUBJECT MATTER SUBJECT MATTER
ASSESSMENT ASSESSMENT ASSESSMENT ASSESSMENT
PLANNING &ADMIN PLANNING & ADMIN PLANNING OF LEARNING
COMMUNICATION &COLLAB. COMMUNICATION &COLLAB. COMMUNICATION& COLLAB.
TEACHING/PEDAGOGY LEARNING
COMPETENCES COMPETENCES
b = barrier MOTIV./CONFID./SELF-ESTEEM MOTIV./CONFID./SELF-ESTEEM
e=enabler ATTAINMENT/TESTSCORES

IMPACT

Figure 1 : Analytical framework on context of ICT introduction and implementation

203
 Chapter IV - Case studies

The model consists of five levels: soci- vidual needs , although schools find it
ety, education system , school , teach- hard to isolate the contribution of ICT
ers and learners . These levels repre- to test scores .
sent where strategies , enablers and
barriers can be found . The framework However, research suggests that there
reads from left to right, representing is a discrepancy between children’s
under- use of ICT at school and their
not only a hierarchical flow but also a
flow from strategy to impact . more frequent and often more sophis-
ticated use at home . Although a range
A synthesis report was compiled taking of digital skills are acquired outside
into account the results of the five con- school informally, some basic compu-
tributory reports described above. It ter skills are not .
presented key findings , conclusions
and recommendations for future work. ICT increases motivation,
The key findings are summarised confidence and engagement
below, together with suggestions for in learning
further investigation . They are grouped
under four headings: impact on learn- Some 87 % of teachers say that pupils
ers and learning , impact on teachers are more motivated and attentive
and teaching , impact on schools and with ICT according to the LearnInd
planning and system-wide findings . data . Much of the research suggests
that ICT has a positive impact on stu-
Impact on learners and learning dent attendance , behaviour, motiva-
tion , attitudes and engagement, that
ICT improves children’s guided , active and enquiry-based
knowledge, skills and tasks with ICT are particularly motivat-
ing , and that technology enables finer
competences
differentiation and personalisation . A
There is a broad consensus among large-scale comparative study shows
primary teachers about the positive that pupils participate more actively in
impact of ICT on learners and learn- learning when ICT is used . Teachers
ing. Research shows that a range of in the school survey felt strongly that
skills and competencies are acquired ICT is a means of overcoming low
by the use of ICT: digital , communi- motivation , social diversity and dis-
cation , language (first and second), engagement. In the case studies ,
social and cognitive skills . Teachers there are examples of schools using
interviewed in the LearnInd survey ICT to improve links between learning
note a positive impact on basic skill inside and outside school and involve
acquisition (reading , writing and arith- parents . ICT also impacted on group
metic) through the use of ICT and processes and collaborative learning .
research echoes this finding . UK
research shows that English , maths Assessment can be
and science test scores improve with more sophisticated and
ICT, and a Hungarian study shows that individualised
ICT- rich constructivist learning envi-
ronments improve learning outcomes , ICT- based assessment systems used
especially for disadvantaged children . in some case study schools give more
Many case studies highlight how ICT sophisticated feedback to teachers ,
helps children understand the subject parents and pupils on their perform-
they are studying and caters for indi- ance, e.g. through the analysis of

204
 ICT at primary school level

Figure 2: United Kingdom: motivated confident and engaged independent learners

test scores . Virtual learning environ- -

subjects where resource devel-

ments enable the individual track- opment by individual teachers is
ing of progress and help identify difficult and/or costly.
the next ‘learning step’ , so enabling • Almost all aspects of assessment :
pupils themselves to detect errors and developing effective tools to meas-
shortcomings . Achievement can be ure ICT skills ; enabling ICT deploy-
recorded in e- portfolios . ment by students within the assess-
ment process ; e-assessment ; etc.
•

Areas for further investigation Development of indicators on suc-

cessful use of ICT in relation to dif-
•

Longer-term study of the impact of fering learning tasks and contexts.

ICT on improving learning achieve- •

Understanding the feasibility, costs

ment , also taking into account the and benefits of personalised learn-
effects of differing learning styles . ing .
• How quality assurance and inspec-
tion regimes are developing to take
Impact on teachers and teaching
full account of ICT developments .
•

Subject-specific investigation of ICT Most teachers use ICT and are

impact and potential , especially in :
— key priority subjects , in particu- ‘ICT-optimistic’
lar mathematics , science and Some 75 % of primary teachers (and
technology; their pupils) use computers in class

205
 Chapter IV — Case studies

Figure 3: Spain: after-school on-site training, responsive to needs, with a pedagogical expert on hand

according to the LearnInd data: from There is little to no correlation between

around 90 % in the Nordic countries to impact-optimism and levels of school
approximately 35 % in Greece, Latvia equipment, sophistication of use or
and Hungary. Teachers find that ICT even teacher skills. There is a cluster
supports in equal measure a range of of countries with high skill levels and
learning and teaching styles, whether high expectations as to ICT impact:
didactic or constructivist, in passive the United Kingdom, the Netherlands,
activities (exercises, practice) and Cyprus and Malta.
in more active learning (self-directed
learning, collaborative work). The ICT is pedagogically
research shows that rich constructivist
under-used
learning environments improve learn-
ing outcomes, especially for learners Despite the high levels of reported
from disadvantaged areas. Teachers in classroom use mentioned above,
some countries (the United Kingdom, according to some studies teach-
Cyprus, the Netherlands, Portugal ers use ICT more for administration,
and Poland) are more optimistic about organisation and planning. They also
ICT than others (Sweden, France and indicate that teachers are aware of the
Austria). Nevertheless, a significant potential benefits of ICT for students,
minority (21 %) consider that using have a positive perception of ICT in
computers in class does not in itself terms of supporting active autono-
have significant learning benefits. mous learning and creating authentic

206
 ICT at primary school level

tasks, but lack the pedagogical vision ways ICT specifically can enhance
to integrate ICT effectively in teaching. teaching and learning.
The research shows that ICT can pro- • Developing fully integrated models
mote new pedagogical approaches, of ICT-supported learning delivery
but only if ICT is fully integrated into which provide examples and tem-
subject lessons. In the Nordic coun- plates to guide local development.
tries, teachers in primary schools • The environment and conditions for
more often regard ICT as supporting continuing professional develop-
their pedagogy than teachers in sec- ment for teachers in relation to ICT.
ondary schools. • Improving interoperability in the
interests of maximum exchange,
Quality training increases deployment and sharing of teaching
materials.
teachers’ motivation and digital
and pedagogical skills Impact on schools and ICT
Teachers responding to the good prac- planning
tice survey consider that using ICT
improves their motivation and teaching Children’s access to
skills. We know from the policy survey
that the 30 countries are investing in
technology is improving
developing teacher ICT skills; but that Analysis of the 2006 LearnInd data
in a significant number of countries reveals that almost all primary schools
teachers entering the profession may use computers, with at least 88 %
have little formal training in using ICT of schools in each country having
in teaching. Researchers have drawn Internet access and with an average
some worrying conclusions about the of eight Internet computers per 100
effectiveness of continuing professional pupils. However, there are huge vari-
development in ICT: that teachers have ations in ICT infrastructure and con-
failed to acquire the desired level of nectivity across and within countries.
ICT skills for classroom instruction and The computer-to-pupil ratio ranges
that training has not translated into from Luxembourg (23 computers per
gains in pupil learning. Research sug- 100 pupils), Denmark and Norway
gests that teachers adapt more easily (18), the United Kingdom (16) and the
to new technologies through a step-by- Netherlands (15) to much lower fig-
step approach with minimal disruption, ures for Latvia, Lithuania and Poland
and that on-site is preferable to off- (6) and Greece and Slovakia (5).
site training. Training courses failed to
match needs and lack the pedagogical According to figures provided for the
and practical dimension, according to policy report, the computer-to-pupil
the analysis of responses to the policy ratio now ranges from 3.1 to 32 per
survey. The survey also indicates that 100 pupils and eight countries have
reliable technical back-up and inspiring more than 14 computers per 100 pupils
pedagogical support for teachers are (representing over 50 000 schools).
often missing. Some 72 % of the study’s 209 866 pri-
mary schools have broadband and in
20 countries over two thirds of primary
Areas for further investigation schools have broadband. Interactive
• Pinpointing sound pedagogy and whiteboard provision ranges from very
understanding whether and in what few (e.g. Finland, Norway) to near

207
 Chapter IV — Case studies

saturation (the UK, where all primary ICT makes administration accessible to
schools have at least one). Denmark, wider groups through a web interface
Estonia and Norway have the highest and school records are more easily
levels of virtual learning environments maintained, exchanged and updated.
that offer access from outside school. However, research indicates that
Smaller primary schools are disadvan- school ICT plans tend to concentrate
taged in terms of equipment, accord- more on infrastructure than on how ICT
ing to research, yet case studies show can be used to enhance teaching and
that the benefits for schools in small learning, and this can actually work
communities are considerable. against innovation (as found in some
case studies). Virtual learning environ-
Whole school ICT integration ments are becoming more widespread,
but are used more for administration
and leadership matter than for learning. Research shows that
ICT integration in subjects and class- sufficient time is needed to assimilate
rooms is the key to changing teach- virtual learning environments. However,
ing practices, according to research once introduced, they are increasingly
— and the school leader’s support used by teachers.
is crucial in cases where primary
schools are free to integrate ICT in Areas for further investigation
the curriculum. The policy survey sug-
gests that countries with high levels • The economics of ICT investment, at
of ICT favour dispersion into class- both micro (e.g. optimal initial capi-
rooms. Some 68 % of primary schools tal and human resource investment
at school level) and macro (e.g.
have computers in classrooms, rather
than in computer labs, according to the relative effectiveness of local,
the LearnInd data. This is the case in regional and national investment)
more than 90 % of primary schools levels.
in Luxembourg, Slovenia, the United • Exchange of best practices in reach-
ing remote and disadvantaged
Kingdom, the Netherlands, Cyprus
and Ireland. In contrast, there are communities.
10 countries with computers in class- • How ICT in school management can
rooms in fewer than 50 % of schools support and facilitate the teacher’s
(Cyprus, Estonia, Greece, Italy, Latvia, role and the quality of the educa-
Lithuania, Hungary, Poland, Slovakia tional experience.
and Spain). In these countries, the • Models for managing and supporting
majority of primary schools use com- ICT development and use at school
puters for education in dedicated com- level.
puter labs.
Primary schools systems
ICT improves administration
and access to information Strategies for ICT tend
to feature infrastructure and
Schools have incorporated ICT into
management tasks and ICT is increas-
teachers’ digital competence
ingly used by teachers for administra- Responses to the policy survey indi-
tion and planning. In several case stud- cate that all 30 countries have or
ies, school-wide planning improved have recently had at least one ICT
with the help of ICT. This is because policy or initiative affecting primary

208
 ICT at primary school level

schools, usually aimed at improving of schools and computer science is

infrastructure and digital competence taught separately in 97 %.
among teachers; and less frequently
targeted at the supply of digital learn- ICT responsibilities within the
ing resources, pedagogical reform or
leadership. From the 74 policies, pro- system can be unclear
grammes and projects analysed in the In most countries, ICT is part of gen-
study, strategies range from a system- eral education policy and there is also
wide intervention including ICT to spe- a specific ICT policy for all schools,
cific projects focused on, for example, but no specific policy for ICT in pri-
equipment, e-safety or teacher educa- mary schools. In countries where
tor ICT training — and with the locus
of control running from central govern- ICT has long been used in primary
schools, policies seem to make fewer
ment control to high levels of school
explicit references to ICT; and so ICT
autonomy and responsibility. ICT in
schools is still a topic that arouses could be said to be pervasive and a
controversy; and where the debate given. Responsibilities can be unclear
involves the general public, the con- according to the policy survey: while
cerns tend to be about e-safety, primary schools have increasing
according to the policy surveys. autonomy as public sector services
become decentralised, ICT responsi-
Digital competence usually bility in the system varies and is some-
features in the curriculum times unclear. Hardware provision is
is in often a national or municipal respon-
Digital competence the pri- sibility, but not maintenance, techni-
mary school curriculum in 22 of the cal or pedagogical support. This can
30 countries, according to the policy
leave schools in some confusion.
survey, either integrated across sub-
jects (in 15 countries) or taught as a
Areas for further investigation
separate subject (in 11 countries).
LearnInd data show that teaching ICT • Understanding ways in which
as a separate subject, computer sci- national and regional strategies can
ence, varies across Europe: ranging address the aim of improving the
from being taught in nearly all schools quality of education.
in Latvia, Poland and Hungary to very • Whether there are differential impacts
few in Finland (19 %) and Austria depending on whether ICT skills are
(9 %). There is little evidence from the
taught separately or through integra-
LearnInd data to suggest that teaching tion in the general curriculum.
computer science as a separate sub-
• Creating a flow of information on
ject implies placing less importance
on ICT in other subjects. There are, future visions for ICT in education
however, exceptions to this obser- (e.g. emerging new technologies,
vation: in the United Kingdom, ICT integration, networking, mix of school-
is used in most subjects in 94 % of based and home-based learning).
schools; but at the same time compu- • Rates of investment in ICT in educa-
ter science is taught separately in only tion:howhavetheydevelopedinrecent
52 % of schools. In Latvia, conversely, years, what are the current trends and
ICT is used in most subjects in 42 % is investment sustainable?

209
 Chapter IV - Case studies

Some early conclusions – nology can then be evaluated in terms

of its contribution to these wider policy
a personal perspective aims . Until recently, policy measures
The conclusions and recommenda- to encourage the use of ICT have
tions arising from STEPS are still tended to focus on improving infra-
under discussion at the time of writ- structure and developing teacher
ing. A synoptic report, conclusions competence in ICT. From that narrow
and recommendations , the five con- perspective it is more difficult to justify
tributory reports , 30 country reports the investment . In some recent edu-
and 25 case studies are to be pub- cation policies and initiatives , ICT is
lished online (see http ://steps.eun . invisible , either because it is a given
org ) , together with a paper describing or perhaps because it is perceived
the methodology in detail . as problematic . Yet the evidence
What is certain from the evidence is
suggests that the impact of ICT on
schools , teachers and learners can
that teachers are at the heart of ICT
increase the effect of other initiatives ,
success in Europe’s primary schools .
They are positive about ICT but can for example reducing learner drop-
be frustrated by external (and some out, efficiency gains, key competence
internal ) inhibitors . Teachers need development, improved teaching and
ongoing appropriate training and school autonomy.
quality support driven by pedagogy
not technology, good digital learn- Although the studies reviewed in
ing resources and room for initiative STEPS provided a generally positive
and risk-taking . School leaders and picture of ICT impact, information is
municipalities (depending on school patchy and tends to focus on inputs .
governance arrangements) would More research is needed into the
benefit from guidance in the use of impact of ICT on learning outcomes ,
ICT in organisational change and the and in other sectors , such as second-
use of tools for whole-school self- ary education, and to identify transfer-
evaluation . able interventions . More international
cooperation on regular benchmark-
Likewise , it is clear that primary school ing and lessons learned , definitions
children are excited about technol-
and methodologies would help to
ogy; they are competent with ICT in assess the return on investments in
many (but not all) respects and are at technology in education , and enable
home with technology, using it exten-
teachers , school leaders and policy-
sively outside school . This should be
shapers to make sound decisions . As
more actively exploited by schools,
Michael Trucano of the World Bank
but sensitively (it is their technology
and their free time), and ensuring that recently said :
critical gaps like children’s media lit-
eracy are covered . It is necessary to have new types of
evaluation in place and new monitor-
The value and contribution of ICT ing indicators . The impact of ICTs on
as an enabler for more general edu- learning and future employment is still
cational policy visions , reforms and debatable , precisely because there is
objectives could be more explicitly no standard methodology. (Trucano ,
stated in policies . The impact of tech- 2009) .

210
 ICT at primary school level

References

Balanskat, A. , Blamire, R. and Kefala, S. (2006). The ICT impact report: a review
of studies of ICT impact on schools in Europe. Brussels : European Schoolnet.

Empirica (2006) . Benchmarking access and use ofICT in European schools 2006:
final report from head teacher and classroom teacher surveys in 27 European
countries. Download at http://europa.eu.int/information_society/eeurope/i2010/
docs/studies/final_report_3.pdf

Gordezky, R. , Martens , K. and Rowan, S. (2004). Influencing system-wide change

at the Toronto District School Board. Download at http ://thresholdassociates.
com/successes/pdf/Futuresearch.pdf

Korte , W. and Hüsing, T. (2006). LearnInd: benchmarking access and use of ICT
in European schools. Bonn : Empirica .

McLaughlin , M. (2005). In : A. Lieberman (ed . ), The roots of educational change:

international handbook of education change. Dordrecht: Springer.

Trucano , M. (2009) . Speech at ‘ Reinventing the classroom’ seminar,

15 September 2009 , Washington DC .

211
European Commission Joint Research Centre

Assessing the effects of ICT in education - Indicators , criteria

and benchmarks for international comparisons

Luxembourg : Publications Office of the European Union , 2009

2009 211 pp. 17.6 x 25 cm

ISBN 978-92-79-13112-7
doi : 10.2788/27419

Price (excluding VAT) in Luxembourg : EUR 15

How to obtain EU publications

Publications for sale:

• via EU Bookshop ( http://bookshop.europa.eu );

• from your bookseller by quoting the title , publisher and/or ISBN number;
•

by contacting one of our sales agents directly. You can obtain their contact details
on the Internet ( http://bookshop.europa.eu) or by sending a fax
to +352 2929-42758 .

Free publications:
• via EU Bookshop (http://bookshop.europa.eu );
• at the European Commission's representations or delegations . You can obtain their
contact details on the Internet (http://ec.europa.eu) or by sending a fax
to +352 2929-42758 .
 78-09-991
EN
LB
C
--
The mission of the Joint Research Centre is to provide customer-driven scientific and technical support for
the conception, development, implementation and monitoring of European Union policies. As a service of the
European Commission, the Joint Research Centre functions as a reference centre of science and technology
for the Union. Close to the policy-making process, it serves the common interest of the Member States, while
being independent of special interests, whether private or national.

CRELL
Centre for Research
on Lifelong Learning

Price (excluding VAT) in Luxembourg: EUR 15

10.2788/27419
ISBN 978-92-79-13112-7

Publications Office 9789279 131127