See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/268204134

The Role of Large Scale Assessments in Research on Educational

Effectiveness and School Development

Article · January 2013

DOI: 10.1007/978-94-007-4629-9_7

CITATIONS READS

39 558

1 author:

Eckhard Klieme
DIPF - Leibniz Institute for Research and Information in Education
381 PUBLICATIONS   10,357 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Studie zur Entwicklung von Ganztagsschulen (StEG) View project

DaTa - Determinants of Teaching Quality View project

All content following this page was uploaded by Eckhard Klieme on 04 December 2015.

The user has requested enhancement of the downloaded file.

 The Role of Large Scale Assessments
in Research on
Educational Effectiveness and School Development

Eckhard Klieme

German Institute for International Educational Research (DIPF)

Frankfurt am Main, Germany

International Conference
“The Role of International Large Scale Assessments
in Educational Policy”

ETS, Princeton, March 18, 2011

 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Overview
1. LSA: Goals and limitations of (international) Large Scale Assessments
2. EER: Goals and design of educational effectiveness studies
3. EER  LSA:
Designing LSA based on educational theory and research
> OTL > Quality of instruction
4. LSA  EER:
How LSA may contribute to our knowledge of educational effectiveness
and school development
> Exploration > Hypothesis testing > Understanding context
5. LSA + EER:
Enriched (panel) designs integrating LSA and EER:
> Explaining student growth (German Study on Language Instruction)
> Explaining school development (PISA School Panel Germany)
6. Conclusions
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

1. (International) Large Scale Assessments

1959
• 1959 - Pilot Twelve-Country Study • 1998 - IALS: International Adult Literacy Survey (OECD,
• 1964 - First International Mathematics Study 1998 StatCan, ETS, UNESCO) – Wave 3
• 1970 - Study of Reading Comprehension • 1998-2000: SACMEQ: Southern Africa Consortium for
• 1970 - Study of Literature Education Monitoring Educational Quality
• 1970 - First International Science Study • 1999 - Trends in International Mathematics and Science
• 1970 - Study of French as Foreign Language Study
• 1970 - Study of English as Foreign Language • 1999 - CivEd: Civic Education Study (IEA)
• 1999 – SITES-M2: Second Information Technology in
• 1970 - Study of Civic Knowledge Education Study Module 2 (IEA)
• 1980 - SIMS: Second International Mathematics Study • 2000 - PISA: Programme for International Student
(IEA) Assessment (OECD)
• 1981 - CES: Classroom Environment Study • 2003 - PISA 2003: Programme for International Student
• 1983 - SISS: Second International Science Study (IEA) Assessment (OECD)
• 1986 - PPP: Pre-Primary Project (IEA, 1986-2003) • 2003 - ALL: Adult Literacy and Life Skills Survey
• 1988 - International Assessment of Educational Progress (StatCan, ETS, OECD, UNESCO)
(ETS): IAEP I • 2006 - ALL: Adult Literacy and Life Skills Survey
• 1990 - International Assessment of Educational Progress (StatCan, ETS, OECD, UNESCO)
(ETS): IAEP II • 2006 – SITES 2006: Second Information Technology in
• 1990 - International Reading Literacy Study Education Study 2006 (IEA)
• 1994 - IALS: International Adult Literacy Survey (OECD, • 2008 – TEDS-M: Teacher Education and Development
StatCan, ETS, UNESCO) – Wave 1 Study in Mathematics (IEA)
• 2009 - ICCS 2009: International Civic and Citizenship
• 1995 - Third International Mathematics and Science Education Study (IEA)
Study • 2010 - PIAAC: Programme for the International
• 1996 - IALS: International Adult Literacy Survey (OECD, Assessment of Adult Competencies (OECD)
StatCan, ETS, UNESCO) – Wave 2 • 2011 – PIRLS 2011: Progress in International Reading
• 1996 - CivEd: Civic Education Study (IEA) Literacy Study 2011 (IEA)
• 1997 – SITES-M1: Second Information Technology in • 2011 – TIMSS 2011: Trends in International Mathematics
Education Study Module 1 (IEA) and Science Study 2011 (IEA)
• 2013 – ICILS: International Study of Computer and
1998
Information Literacy
2011

(graphical display with permission from Knut Schwippert, Univ. of Hamburg)

3
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

LSA provide Indicators for policy making

1. describing state and change of systems
2. setting and monitoring benchmarks, both inter- and intranationally
3. identifying strengths and weaknesses, success and challenge
4. communicating to the public and the professionals

With regard to
- Functioning of the educational system (inputs, processes)
- Effectiveness: Output (achievement, certificates) and/or
outcome (yield: competencies, educational pathways)
- Equity: Distribution of input, process, and outcome by (e.g.)
gender, SES, immigration background, region
- Efficacy: Outcome (value-added ?) per investment
- Production function: Relationsship between input-process-outcome
at the individual, classroom/teacher, school, system level
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Example: Monitoring Gender Inequity in

Reading:
Gender differences in favor of girls (effect sizes, d)
in PIRLS 2001 and PISA 2003

0,6
Geschlechterunterschied der

0,5
Lesekompetenz*

0,4

0,3

0,2

0,1

0
ITA FRA CZE DEU HUN NLD CAN USA ISL TUR GRC NOR SWE NZL

IGLU 2001
PIRLS (Ende Grundschulzeit)
2001 PISA
PISA2003 (15-Jährige)
2003
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

LSA databases allow for sophisticated

analyses in educational research

Example:

Between-School differences in grading standards

depending on centralized vs. decentralized exit exams

Data from German extension to PISA 2000 (about 100 schools per state)

Method: decomposing variance in PISA-outcomes (Math, reading, science)

by school, grade level, school X grade level
within school types, states
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Bavaria, High track: Northrhein-Westfalia, High track:

Standardsicherung im Gymnasium Standardsicherung im Gymnasium
Central exams No central exam
LAND: 5 BY LAND: 8 NW

Notenstufe
Notenstufe

Restvarianz

Einzelschule
Restvarianz
Einzelschule Note * Einzelschule

Bavaria, Low track:

Standardsicherung im Bildungsgang Hauptschule
No central LAND: exam 5 BY
Variance in PISA-scores
attributable to Notenstufe

Schools
Grade levels (1-6) Restvarianz
Einzelschule
Interaction school X grade
Note * Einzelschule
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Limits of Large Scale Assessments:

Reverse Causality on the classroom level

Professional educators, in their daily practice, need to monitor student

development, and change treatments accordingly.

Examples: assigning tasks, forming groups for collaborative learning, giving

feedback, deciding on grade retention/promotion …

 Effects of these treatments cannot be estimated from cross-sectional data

only, without knowing prior achievement and other factors that drive treatment
assignment
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Limits of Large Scale Assessments:

Reverse Causality (IEA Classroom
Environment Study, Anderson et al. 1989)
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Limits of Large Scale Assessments:

Reverse Causality on the School level (OECD 2007)
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

2. Educational Effectiveness Research (EER)

Creemers & Kyriakides

(2008): The dynamic model
of educational effectiveness
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

4 stages of Educational Effectiveness Research

(Creemers, Kyriakides & Sammons, 2010)

1. Showing that „school matters“: estimates of variation

2. Searching for factors associated with student outcomes
e.g., Edmonds (1979) 5 factors:
leadership, expectations, emphasis on basic skills,
orderly climate, evaluation
see also Bryk et al. (2010): „essential supports“
3. Deveploment of theoretical models
4. Analysing the complex nature of educational
effectiveness;
links with the study of school improvement
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Stage 3: CIPO-Model (Scheerens/Bosker 1997)

CONTEXT
School structure, Curriculum, pedagogical traditions and orientations,
teacher education, budgeting and regulation, socio-economic and cultural context
INPUT PROCESS OUTPUT
Teacher-student-rate, Quantity of instruction, School School
qualification of teaching curriculum, leadership, teacher level
staff, student population, cooperation and collaboration,
parent commitment professional development, cohesion,
school culture (norms and values),
school climate, internal and external
evaluation
Students per class, Instructional quality: Classroom
teacher competencies - opportunity to learn level
- clear, well-structured classroom
management,
- supportive, student-oriented
classroom climate,
- cognitive activation with
challenging content
SES, social and cultural time invested, self regulation, Individual
capital, family support, motivation and interest, self concept, level
gender, language and learning strategies
migration background,
general intellectual ability,
pre-knowledge
Stage 4: Mediating and moderating variables
School psychological/social
developement
t1 t2

Learning environment Learning process Indiv.

(opportunities) (take up) Cond.

t1 t2
Teacher Achievement
School psychological/social
developement
t1 t2

Learning environment Learning process Indiv.

(opportunities) (take up) Cond.

t1 t2
Teacher Achievement
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Challenges to Educational Effectiveness

Research
• The adaptive nature of educational processes
• The complexity of mediating processes
• The importance of moderating variables
• The weakness of distal effects, especially school effects
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Challenges to Educational Effectiveness

Research
• The adaptive nature of educational processes
• The complexity of mediating processes
• The importance of moderating variables
• The weakness of distal effects, especially school effects

• The dynamic functioning of educational institutions

• The incoherence and instability of effect sizes
• The fundamental difference between status (at a given
moment), individual gains, and organizational change
(over time)
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

3 Types of „value-added“ measure (n=273 schools)

Mean Change in mean
achievement gain gain over 2 yrs
within 1 year

Adjusted means
(adjusted for SES,
gender, immigration
status &
compositional
effects):
-> Stability = .70

Mean
achievement gain
within 1 year:
-> Stability = .31
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

3. How EER can inform LSA: TALIS

Professional Teacher classroom
competence practice
(knowledege and Structuring, Classroom
beliefs) Student orientation, Level
content knowledge, Enhanced Activities environment
Time on task*,
pedagogical content disciplinary Teacher-
knowledge, climate based
criteria:
Student Studen
learning out- Self efficacy
General pedagogical Teacher‘s School level come
knowledge and beliefs: professional environment
Job
activities School climate satisfaction*
Beliefs about the nature Cooperation among
of teaching and learning: staff: collaboration for
direct transmission vs teaching vs
constructivist. professional
development

Teacher background:
professional training / School background and processes Student
experience (e.g., Leadership:) background

OECD - TALIS: section on teacher beliefs and teaching practices

 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

3. How EER can inform LSA: PISA

(I) General (i.e., domain-independent) trend variables
General input variables:
• Student-level inputs (grade, gender, socio-economic background= parental
education and occupation/family wealth/educational resources/cultural
possessions, migration data = immigration status / heritage language/ age on
arrival in country, family support)
• School-level contexts and inputs (community size, resources, qualifications of
teaching staff)
General process variables:
• School-level processes (decision-making, admission policies, assessment and
evaluation policies, professional development, student-teacher-relations, parental
involvement)
• Instructional processes (learning time, disciplinary climate, teacher support)
General outcome variables:
• General non-cognitive outcomes – Commitment to learning (behavioural: truancy;
personal goal: educational aspirations; motivational: learning engagement,
affective: sense of belonging)

(II) Domain-specific trend variables

• Domain-specific cognitive outcomes (math, science, reading literacy)
• Domain-specific non-cognitive outcome variables (strategies and metacognition,
domain-related beliefs, self-related beliefs, motivation)
• Domain-specific process variables (Opportunity to learn, instructional quality,
system- and school-level support)

(III) Thematic extension variables (extensions within individual cycles), e.g.

• International options (e.g. in PISA 2012, educational career/second language
learners; ICT literacy)
• Context variables for additional domains (e.g. ICT-related experiences relevant for
computer-based problem solving)
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Opportunity to learn:
The „observable structure“ of teaching
(Schmidt & McKnight 1995; Schmidt & Maier 2009)

• Content Coverage
• Content Exposure Variables:
considering time and depth of teaching
• Content Emphasis Variables:
e.g., lower vs.higher order skills
• Quality of Instructional Delivery Variables:
classroom teaching practices
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Quality of instruction: Empirical findings (Brophy,2000)

2. Opportunity to learn
4. Establishing learning orientations
5. Coherent content
11. Goal-oriented assessment

1. A supportive classroom climate

10. Co-operative learning

3. Curricular alignment
6. Thoughtful discourse
7. Practice and application activities
8. Scaffolding students’ task engagement
9. Strategy teaching
12. Achievement expectations
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Quality of instruction: Empirical findings (Brophy,2000)

2. Opportunity to learn
4. Establishing learning orientations
5. Coherent content
Opportunity to learn:
11. Goal-oriented assessment

Students learn more when most of the

1. A supportive classroom climate
available time is allocated to
10. Co-operative learning activities and the
curriculumrelated
classroom management system
3. Curricular alignment
emphasizes maintaining their
engagement
6. Thoughtful in those activities.
discourse
7. Practice and application activities
(Brophy 2000, p. 10)
8. Scaffolding students’ task engagement
9. Strategy teaching
12. Achievement expectations
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Quality of Instruction:
Theoretical foundations
Process-Product-Research; Behavioral learning theory
 Classroom Management, Clarity & Structure

Humanistic pedagogy and psychology;

Self determination Theory (Deci & Ryan)
Supportive climate & Structured learning environment

Cognitive Theory (e.g. Brown 1997, Mayer 2004);

concepts from (moderate) constructivism
 Cognitive Activation & Deep Content
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Quality of Instruction:
The „latent“ structure of teaching

Pianta & Hamre (2009): CLASS framework

• Classroom organization
• Emotional support
• Instructional support

Tschannen-Moran, M. & Woolfolk Hoy, A. (2001):

Ohio teacher efficacy scales (OSTES)
• Efficacy for classroom management
• Efficacy for student engagement
• Efficacy for instructional strategies
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Quality of Instruction:
The „latent“ structure of teaching
Quality of instruction Mediation Effects
(opportunities provided) (take up)

Classroom management, (Time on task)

clarity & structure Knowledge and
understanding
Cognitive Activation & (Depth of
deep content processing,
reflection)

Supportive Climate (self Motivation

determination)

(Klieme, Pauli & Reusser, 2009; Klieme, Schümer & Knoll, 2001)
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

4. How LSA can inform EER

> exploring & generating hypotheses

 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

2nd order factors of classroom practice

based on high-inference video-ratings (Clausen,Klieme & Baumert 2002

(TIMSS-Video 1994 Germany: national sample, 100 + 86 lessons)

Classroom Management Supportive climate Cognitive Activation

Effective treatment of Social orientation: Teacher’s ability to motivate

interruptions „teacher takes care of his students:
„teacher intervenes students‘ problems“ „can present even abstract
immediately, before Teachers diagnostic competence content in an interesting
disturbance may evolve“ with regard to social behavior manner “
Clarity of rules Individual reference norm in Errors as opportunities
Interruptions (-) evaluation Demanding tasks
Waste of time (-) Rate of interaction (-) Practicing by repetition (-)
Monitoring Pressure on students (-)
Time on task
Teacher Unreliability (-)
Clarity and structuredness of
the Instruction
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

4. How LSA can inform EER

> exploring & generating hypotheses

> testing hypotheses

 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

PISA 2000: School-level effects by country

… on Reading Literacy: … on Interest in Reading:
Discipline > Relations Relations > Discipline

40,00 0,20
Impact on literacy: relations, school level

Impact on interest: relations, school level

30,00

0,15
20,00

10,00

0,10

0,00

-10,00
0,05

-20,00

-20,00 -10,00 0,00 10,00 20,00 30,00 40,00 0,05 0,10 0,15 0,20
Impact on literacy: discipline, school level Impact on interest: discipline, school level
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

4. How LSA can inform EER

> exploring & generating hypotheses

> testing hypotheses

> understanding systemic and cultural effects

 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

PISA 2000:: School- vs individual level effects

of teacher-student-relations on reading literacy

20,00
Impact on literacy: relations, individual perception

UK Norway
15,00 Island
Danemark
Finland Australia
New Zealand
Sweden
USA
10,00
Irland

OECD
Liechtenstein
5,00 South Korea
Switzerland
Austria Mexico
Germany Netherlands
Luxembourg

0,00 Brasil
Belgium
Italy

Hungary
-5,00

-40,00 -30,00 -20,00 -10,00 0,00 10,00 20,00

Impact on literacy: relations, school level
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

5. Combining LSA and EER: longitudinal designs

> understanding school and classroom effects

on growth in student motivation & achievement

> understanding school change over a period

of several years
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Two Paradigms (Reynolds 2005, p243)

School effectiveness School improvement

Looking for Evaluation of school
generalizable results reform, „bottom up“
Focus: Outcomes Focus: Organizational
processes
Schools as static Schools as dynamic
organizations organizations
Quantitative Quantitative
Longitudinal on the Longitudinal on the
student level school level
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

5.1 Longitudinal Design at the individual level: The

German Study on Language Instruction (DESI)
T1: 09-10/2003 T2: 05-07/2004
Language Individual and School & instructional Motivation and Language
competencies family context of teaching perception of competencies
in German & in background and learning instruction in German & in
English English
N = 10.543 N = 10.543
Achievement Student Principal questionnaire Student Achievement
tests questionnaire questionnaire tests
Motivation & Parent Questionnaire for staff Teacher
perception of questionnaire teaching German and questionnaire
instruction English at school (assessed
Student N = 210 classes)
questionnaire
11/2003 – 05/2004:
Observation of English lessons by videotaping
in 105 classrooms

German Study on Language Instruction (Klieme et al., 2008)

 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Modeling individual, school and staff characteristics:

2-level SEM with Mplus

Background variables and predictors Criteria

Individual level: Yield in competencies: T2
Gender German achievement, total
Cognitive ability test: KFT-N English achievement, total
SES: HISEI Increases in competencies: T1-T2
German as first language German: writing/w.composition
Attending bilingual lessons English: C-Test
School level:
School types
cognitive, social and linguistic
composition of students
School with bilingual classes
Estimation of student achievement
Cooperative practices
Focus on language competencies
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Relations between school processes (department staff),

and student outcomes
(Steinert/Hartig/Klieme 2007; Klieme/Steinert/Hochweber 2010)

Estimation of
students´ language
competencies by staff

.1
2
n.s.

Cooperation Students´
.32 among staff teaching n.s. competencies in
German German, total

n.s.

s.
n.
Focus on students´
language competencies
by staff

Modellfit: χ2 = 158.713; df = 80; CFI = .97; RMSEA = .01; SRMRbetween = .08; R2between = .86.
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Relations between school processes (department staff),

and student outcomes
(Steinert/Hartig/Klieme 2007; Klieme/Steinert/Hochweber 2010)

Estimation of students´
language
competencies by staff

n.
s.
n.s.

Cooperation Increases in students´

.32 among staff teaching n.s. competencies in
German writing (German)

n.s.

2
.3

Focus on students´
language
competencies by staff

Modellfit: χ2 = 151.044; df = 80; CFI = .92; RMSEA = .01; SRMRbetween = .07; R2between = .21.
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

Relations between school processes (department staff),

and student outcomes
(Steinert/Hartig/Klieme 2007; Klieme/Steinert/Hochweber 2010)

R2 = .07

Estimation of
Students´ Language
Competencies by Staff
.0
6* 3
.2

R2 = .92
R2 = .08

German Writing Cooperation among German Writing

at the Beginning -.29* Staff teaching -.01 at the End
of Grade 9 German of Grade 9

.0
9 R2 = .01 8*
.0
Focus on Students´
Language
Competencies by Staff

Modellfit: χ2 = 641.061; df = 116; CFI = .92; RMSEA = .02; SRMRbetween = .16.

 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

5.2 Longitudinal Design at the school level: The

German PISA School Panel (n=369 schools)
(Hochweber/Steinert/Gomolka/Klieme 2010)
change :
+· HISEI
-- migration
+ discipline

· school form slope

mean 2000 :
· HISEI
· migration intercept
school level
· discipline

2000 = 0
2003 = 1
reading
student level
· HISEI
competency
· migration
· discipline
 Klieme: The Role
Klieme: Was istof
guter
LSA (Mathematik-)
in ResearchUnterricht? GDM 2011,
on Educational Freiburg and School Development
Effectiveness

6. Conclusions
- As a tool for policy making, LSA need not provide
scientific explanations; descriptive validity is sufficient !
- However, in order to cover indicators that are relevant for
professionals and policy makers, LSA should build on
factors that have been shown elsewhere to have an
impact on educational outcomes and change (!).
- (I)LSA can inform EER by generating and testing
hypotheses, and especially by exploring system/culture
effects.
- Causal inference requires either matching on the school
level, or antecedent measures on the student level.
- Longitudinal designs (student/school panels) significantly
enhance the power of LSA studies, but: BQs and tests
have to be aligned with the analytical perspective
- There is still a lack of theoretical foundations for
inferences on system effects and school change.
View publication stats