The synthesis of a unified pedagogy for the design and

evaluation of e-learning software for high-school computing.

Vol 1 of 3 - Thesis

by

Peter Yiatrou

A thesis submitted in partial fulfilment for the requirements for the degree of Doctor
of Philosophy at the University of Central Lancashire

June / 2019
STUDENT DECLARATION FORM
1. Concurrent registration for two or more academic awards

I declare that while registered as a candidate for the research degree, I have not been a registered
candidate or enrolled student for another award of the University or other academic or professional
institution.

2. Material submitted for another award

I declare that no material contained in the thesis has been used in any other submission for an academic
award and is solely my own work.

3. Collaboration

Where a candidate’s research programme is part of a collaborative project, the thesis must indicate in
addition clearly the candidate’s individual contribution and the extent of the collaboration. Please state
below:

_______Not Applicable_________________________________________________________

4. Use of a Proof-reader

The following third-party proof-reading service was used for this thesis:_London Proofreaders________
in accordance with the Policy on Proof-reading for Research Degree Programmes and the Research
Element of Professional Doctorate Programmes.

A copy of the confirmatory statement of acceptance from that service has been lodged with the Research
Student Registry.

Signature of Candidate ______________________________________________________

Type of Award ___ Doctor of Philosophy ________________________________

School ___Computing, Engineering and Physical Sciences_____________

 ABSTRACT
In recent decades, several countries1 have applied a resurgent focus to high-school computer science in
the hope that it will provide the foundations for strong and innovative national IT sectors. The UK is one
example undertaking this transition, and experiencing the associated challenges placed on the teaching
community. In parallel, recent years have seen a trend towards enriching education with digital resources,
specifically e-learning software. This study offers a practical contribution to the computer science teaching
community by supporting the provision of e-learning software, and hence the increased use of e-learning
in high-school teaching. However, it recognises that there remains a concern over the inconsistent
pedagogical quality of many e-learning implementations. To safeguard the pedagogical quality of e-
learning software, this study offers a research contribution by defining: (1) a comprehensive set of
pedagogical heuristics to inform the design of e-learning software; (2) an associated e-learning evaluation
protocol to guide the evaluation and selection of e-learning software for use in schools; and in doing so,
(3) contributes to the under-researched area of high-school computing pedagogy. The proposed pedagogy
synthesises a vast body of knowledge on learning theories into a comprehensive, yet accessible, set of
heuristics. These heuristics supplement existing literature by focusing more tightly and in depth on
pedagogy, rather than usability. The pedagogy synthesises the following learning theories: constructivism,
social constructivism, connectivism, and cognitive load, and additionally gives pedagogical focus to VARK
learning styles, ARCS 2 motivational design, collaborative learning, gamification, and computational
thinking. The e-learning evaluation protocol builds upon existing best practice in evaluation procedures
but is unique in its characteristics and focus. The study follows a rigorous three phase mixed methods
exploratory design in which the e-learning pedagogy and evaluation protocol were explored and
iteratively developed in concert with input and evaluation from education experts and teachers. In
parallel, practice-based input was secured via student usage of prototype e-learning software designed in
adherence to the pedagogy. The findings of this research offer preliminary validation of the
appropriateness and comprehensiveness of the e-learning pedagogy, and the final phase demonstrates
statistically significant learning increases based on student usage of the e-learning software prototype.
Additionally, this research offers preliminary validation of the reliability and validity of the evaluation
protocol. Building on the findings of this research, several possibilities are outlined to further empirically
establish this research, or develop it further into new avenues.

1
Such countries include, but are not limited to Australia, Belgium, France, India, Israel, Italy, New Zealand, Sweden,
South Africa, Russia, the United Kingdom, and the United States.

2
ARCS acronym stands for Attention, Relevance, Confidence and Satisfaction.

i
 TABLE OF CONTENTS
Abstract ......................................................................................................................................................... i

Table of Contents ......................................................................................................................................... ii

List of Tables ...............................................................................................................................................viii

List of Illustrative Material ..........................................................................................................................xii

Acknowledgements ................................................................................................................................... xvii

Glossary of Terms ..................................................................................................................................... xviii

1 Introduction ......................................................................................................................................... 1

1.1 Research Aims............................................................................................................................. 2

1.2 Research Contributions .............................................................................................................. 2

1.3 Research Question and Hypotheses ........................................................................................... 2

1.4 Research Context ........................................................................................................................ 3

1.5 Challenges and Limitations ......................................................................................................... 3

1.6 Thesis Organisation .................................................................................................................... 4

2 Literature Review ................................................................................................................................ 5

2.1 Digital Technology and the Economy ......................................................................................... 5

2.1.1 Digital Technology is Ubiquitous ............................................................................................ 5

2.1.2 The Economic Benefit to the UK............................................................................................. 7

2.1.3 The Digital Divide.................................................................................................................... 7

2.1.4 Globalisation and National Competitiveness ......................................................................... 8

2.1.5 Demand for IT Professionals .................................................................................................. 9

2.1.6 It Starts with Education ........................................................................................................ 10

2.2 Digital Technology and Education ............................................................................................ 11

2.2.1 The Call for Reform in High-School Computing Education ................................................... 11

2.2.2 Academically Rigorous Computing and Computer Science GCSEs ....................................... 13

2.2.3 Impact on the Teaching Community .................................................................................... 14

2.2.4 What has Changed in Five Years? ......................................................................................... 15

2.2.5 Related Research into High-school Computing, E-learning and Pedagogy .......................... 17

ii
 2.2.6 International Comparison .................................................................................................... 19

2.3 E-learning Software .................................................................................................................. 20

2.3.1 The Status of E-learning ....................................................................................................... 20

2.3.2 Quality of E-learning Software and the Need for E-learning Heuristics ............................... 23

2.3.3 E-learning Evaluation Protocols............................................................................................ 25

2.4 Learning Theories and Other Inputs into the Pedagogy ........................................................... 28

2.4.1 What are Learning Theories? ............................................................................................... 28

2.4.2 Choice of Learning Theories and Inputs into the Pedagogy ................................................. 28

2.4.3 Constructivism ...................................................................................................................... 31

2.4.4 Social Constructivism............................................................................................................ 32

2.4.5 Collaborative Learning.......................................................................................................... 33

2.4.6 Connectivism ........................................................................................................................ 35

2.4.7 VARK Learning Styles ............................................................................................................ 36

2.4.8 ARCS Theory of Motivation .................................................................................................. 39

2.4.9 Cognitive Load Theory .......................................................................................................... 41

2.4.10 Gamification ..................................................................................................................... 43

2.4.11 Computational Thinking ................................................................................................... 44

3 Research Methods ............................................................................................................................. 47

3.1 Chapter Brief ............................................................................................................................. 47

3.2 Theoretical Foundations of the Research Design ..................................................................... 47

3.2.1 Research Paradigm ............................................................................................................... 47

3.2.2 What is Mixed Methods Methodology and Why was it Used? ............................................ 48

3.2.3 The High-level Structure of the Mixed Methods Design ...................................................... 50

3.2.4 Validity, Reliability and Trustworthiness .............................................................................. 51

3.2.5 Action Research Design ........................................................................................................ 54

3.2.6 Incremental and Iterative Development Methodology ....................................................... 55

3.3 Introducing the Three Phases of Research ............................................................................... 57

3.4 Phase 1: Pilot Study .................................................................................................................. 59

3.4.1 Objectives ............................................................................................................................. 59

iii
 3.4.2 Phase 1 Participants ............................................................................................................. 59

3.4.3 Pilot of Research Methods ................................................................................................... 60

3.4.4 Phase 1 Research Activities .................................................................................................. 61

3.4.5 Interpretation ....................................................................................................................... 65

3.4.6 Phase 1 – Ensuring Validity and Trustworthiness ................................................................. 66

3.5 Phase 2: Elaboration via Action Research ................................................................................. 67

3.5.1 Objectives ............................................................................................................................. 67

3.5.2 Action Research Cycle .......................................................................................................... 68

3.5.3 Phase 2 Participants ............................................................................................................. 68

3.5.4 Phase 2 Action Research Activities ....................................................................................... 69

3.5.5 Interpretation ....................................................................................................................... 74

3.5.6 Phase 2 – Ensuring Validity and Trustworthiness ................................................................. 74

3.6 Phase 3: Final Experiment......................................................................................................... 75

3.6.1 Objectives ............................................................................................................................. 75

3.6.2 Phase 3 Participants ............................................................................................................. 75

3.6.3 Phase 3 Experiment Design .................................................................................................. 76

3.6.4 Phase 3 Research Activities .................................................................................................. 77

3.6.5 Data Collection, Analysis and Reporting .............................................................................. 81

3.6.6 Interpretation ....................................................................................................................... 85

3.6.7 Phase 3 – Ensuring Validity and Trustworthiness ................................................................. 86

3.7 Research Methods for the Development of the E-learning Evaluation Protocol ..................... 87

3.7.1 Phase 1 Procedures: Identifying the Need for an E-learning Evaluation Protocol ............... 87

3.7.2 Phase 2: Piloting the E-learning Evaluation Protocol ........................................................... 88

3.7.3 Phase 3: Usage of the E-learning Evaluation Protocol ......................................................... 89

3.8 Chapter Summary ..................................................................................................................... 94

4 Synthesis of the E-Learning Pedagogy ............................................................................................... 96

4.1 Chapter Brief ............................................................................................................................. 96

4.2 Versions of the E-learning Pedagogy ........................................................................................ 96

4.3 Intended Audience and Usage .................................................................................................. 96

iv
 4.4 Pedagogical Coverage ............................................................................................................... 97

4.5 A Non-Prescriptive Pedagogy ................................................................................................... 98

4.6 Further Support for the Selection and Usage of Heuristics .................................................... 102

4.6.1 Educational Benefits Matrix ............................................................................................... 102

4.6.2 Heuristic Interrelationships ................................................................................................ 103

4.7 Discussion of Heuristics .......................................................................................................... 106

4.7.1 Constructivism .................................................................................................................... 106

4.7.2 Social Constructivism.......................................................................................................... 114

4.7.3 Collaborative Learning........................................................................................................ 119

4.7.4 Connectivism ...................................................................................................................... 123

4.7.5 VARK Learning Styles .......................................................................................................... 126

4.7.6 ARCS Theory of Motivation ................................................................................................ 132

4.7.7 Cognitive Load .................................................................................................................... 138

4.7.8 Gamification ....................................................................................................................... 148

4.7.9 Computational Thinking ..................................................................................................... 153

4.8 Chapter Summary ................................................................................................................... 155

5 Synthesis of the E-Learning Evaluation Protocol ............................................................................. 156

5.1 Chapter Brief ........................................................................................................................... 156

5.2 Versions of the E-learning Evaluation Protocol ...................................................................... 156

5.3 Phase 1: The Origins of the E-learning Evaluation Protocol ................................................... 156

5.4 Characteristics of the E-learning Evaluation Protocol ............................................................ 157

5.4.1 Intended Usage .................................................................................................................. 157

5.4.2 Focus on Overall Pedagogical Quality, not on Individual Usability Issues .......................... 159

5.4.3 Evaluate Content Quality as well as Pedagogical Quality ................................................... 159

5.4.4 Not all Heuristics or Learning Objectives are Applicable or are of Equal Importance ....... 160

5.4.5 Number of Evaluators ........................................................................................................ 161

5.4.6 Experience of Evaluators .................................................................................................... 161

5.4.7 Protocol Steps for E-learning Evaluation ............................................................................ 162

5.4.8 The Need for Quantifiable Evaluation ................................................................................ 164

v
 5.4.9 Uniqueness of the E-learning Evaluation Protocol ............................................................. 170

5.5 Chapter Summary ................................................................................................................... 171

6 Implementation and Results ........................................................................................................... 172

6.1 Chapter Brief ........................................................................................................................... 172

6.2 Research Instruments and Materials ...................................................................................... 172

6.2.1 Pedagogy Evaluation Feedback Document ........................................................................ 172

6.2.2 E-learning Software Prototype ........................................................................................... 173

6.2.3 VARK Learning Styles Instrument ....................................................................................... 174

6.2.4 Survey Instrument .............................................................................................................. 175

6.2.5 Technical Specification Document ..................................................................................... 176

6.2.6 Phase 3 - Pre-test / Post-Test ............................................................................................. 176

6.2.7 Phase 3 - Research Protocol Confirmation Document ....................................................... 177

6.3 Phase 1 Pilot Findings ............................................................................................................. 177

6.3.1 Feedback into Research Methods and Experiment Protocol ............................................. 177

6.3.2 E-learning Pedagogy – Change of Focus ............................................................................. 178

6.3.3 E-learning Software Development – Change of Focus ....................................................... 178

6.4 Teacher and Educational Expert Feedback on the E-learning Pedagogy ................................ 179

6.4.1 Phase 1: Teacher and Expert Feedback on E-learning Pedagogy ....................................... 179

6.4.2 Phase2-Cycle1: Teacher and Expert Feedback on E-learning Pedagogy ............................ 180

6.4.3 Phase2-Cycle2: Teacher and Expert Feedback on E-learning Pedagogy ............................ 181

6.4.4 Phase 3: Teacher and Expert Feedback on E-learning Pedagogy ....................................... 182

6.4.5 Supplementary Teacher Feedback on the E-learning Pedagogy from the Phase 3 Workshop
184

6.5 Phase 1 and 2 – Findings from Student Usage of E-learning Software Prototype .................. 185

6.6 Phase 3 Results ....................................................................................................................... 197

6.6.1 Phase 3 Participants – Merging of Groups ......................................................................... 199

6.6.2 Student Perception of the E-learning Software ................................................................. 200

6.6.3 Response on the Educational Components used in the E-learning .................................... 208

6.6.4 Response on Perceived Difficulty ....................................................................................... 214

6.6.5 Motivation and Preferences ............................................................................................... 220

vi
 6.6.6 Learning Styles and Multi-modal Approach (VARK) ........................................................... 227

6.6.7 Student Learning Performance .......................................................................................... 230

6.7 E-learning Evaluation Protocol Results ................................................................................... 249

6.7.1 Phase 1: Identifying the Need for an E-learning Evaluation Protocol ................................ 249

6.7.2 Phase2-Cycle2: Piloting the E-learning Evaluation Protocol............................................... 250

6.7.3 Phase 3: Usage of the E-learning Evaluation Protocol ....................................................... 252

6.8 Chapter Summary ................................................................................................................... 253

7 Discussion ........................................................................................................................................ 258

7.1 Chapter Brief ........................................................................................................................... 258

7.2 Globalisation, Digital Ubiquity and the Need for a Strong IT Sector ...................................... 258

7.3 E-learning Software is Part of the Solution ............................................................................. 259

7.4 Contribution to a Small Body of Research and Establishing Research Rigour ........................ 261

7.5 International Contribution ...................................................................................................... 262

7.6 Synthesising an E-learning Pedagogy ..................................................................................... 262

7.7 Pedagogy Feedback as Evidenced by Usage of E-learning Software ...................................... 267

7.8 Phase 3 Learning Performance ............................................................................................... 279

7.9 E-learning Evaluation Protocol ............................................................................................... 281

8 Conclusion, Contribution and Future Work ..................................................................................... 285

8.1 Research Contribution ............................................................................................................ 286

8.2 Future Work for the E-learning Pedagogy .............................................................................. 287

8.3 Future Work for the E-learning Evaluation Protocol .............................................................. 288

9 References ....................................................................................................................................... 289

vii
 LIST OF TABLES
Table 1: Comparison of global internet penetration and the UK ................................................................. 6

Table 2: Further analysis of Sentance and Selby's classification of research into computer science
education in schools from 2005-2014 ........................................................................................................ 18

Table 3: Definitions of learning .................................................................................................................. 28

Table 4: Indicative listing of learning theories ........................................................................................... 29

Table 5: The Worldview underlying this research study ............................................................................ 48

Table 6: Summary of main research instruments and instrumentation used in the study ........................ 58

Table 7: Phase 3 - student participants ...................................................................................................... 75

Table 8: Evaluation protocol workshop agenda (sourced from workshop introductory presentation) .... 89

Table 9: Evaluation protocol workshop - participant background, qualifications and experience ............ 90

Table 10: Five versions of the e-learning pedagogy ................................................................................... 96

Table 11: Final set of e-learning pedagogical heuristics........................................................................... 101

Table 12: Educational benefits matrix ...................................................................................................... 103

Table 13: E-learning heuristics based on constructivism ......................................................................... 106

Table 14: E-learning heuristics based on social constructivism ............................................................... 114

Table 15: E-learning heuristics based on collaborative learning .............................................................. 119

Table 16: Collaborative technologies to be supported in a collaborative learning environment. ........... 121

Table 17: E-learning heuristics based on connectivist principles ............................................................. 123

Table 18: E-learning heuristics based on VARK learning styles ................................................................ 126

Table 19: E-learning heuristics based on ARCS theory of motivation ...................................................... 132

Table 20: E-learning heuristics based on cognitive load theory ............................................................... 138

Table 21: E-learning heuristics based on gamification ............................................................................. 149

Table 22: E-learning heuristics based on computational thinking ........................................................... 153

Table 23: Four versions of the e-learning evaluation protocol ................................................................ 156

Table 24: Keller’s IMMS Reliability estimate. ........................................................................................... 175

Table 25: Phase 1 teacher and expert feedback on e-learning pedagogy ............................................... 180

Table 26: Phase2-Cycle1 teacher and expert feedback on e-learning pedagogy..................................... 181

viii
Table 27: Phase2-Cycle2 teacher and expert feedback on e-learning pedagogy..................................... 182

Table 28: Supplementary teacher feedback on the e-learning pedagogy from the Phase 3 workshop .. 184

Table 29: Phase 1 and 2 student participants .......................................................................................... 185

Table 30: Phase 1 and 2 summary findings for usability and navigation ................................................. 187

Table 31: Phase 1 and 2 summary findings for learning styles and multi-modal learning ....................... 187

Table 32: Phase 1 and 2 summary findings for learning object ranking .................................................. 188

Table 33: Phase 1 and 2 summary findings for understanding and the zone of proximal development. 189

Table 34: Phase 1 and 2 summary findings for visual learning material .................................................. 189

Table 35: Phase 1 and 2 summary findings for text learning material ..................................................... 190

Table 36: Phase 1 and 2 summary findings for activities and active learning .......................................... 191

Table 37: Phase 1 and 2 summary findings for video learning material .................................................. 191

Table 38: Phase 2 summary findings for computational thinking ............................................................ 192

Table 39: Phase 1 and 2 summary findings for collaborative learning .................................................... 192

Table 40: Phase 1 and 2 summary findings for audio learning material .................................................. 193

Table 41: Phase 2 summary findings for authentic learning vs avoiding overload .................................. 193

Table 42: Phase 1 and 2 summary findings for connectivism and the web ............................................. 194

Table 43: Phase 1 and 2 summary findings for motivation ...................................................................... 195

Table 44: Phase 2 summary findings for gamification ............................................................................. 195

Table 45: Phase 2 summary findings for recreating books ...................................................................... 196

Table 46: Phase 3 - Summary of student perception of e-learning software .......................................... 198

Table 47: Phase 3 - Summary of student learning performance .............................................................. 199

Table 48: Phase 3 - Descriptive statistics for usability results .................................................................. 201

Table 49: Phase 3 - Descriptive Statistics for mixed usability responses ................................................. 206

Table 50: Phase 3 - The impact of the different educational components on perceived understanding 209

Table 51: Phase 3- Educational components impact on perceived learning, and perceived interest and
enthusiasm ............................................................................................................................................... 214

Table 52: Phase 3- Difficulty level of the educational content in the prototype e-learning software ..... 215

Table 53: Phase 3 - Student response on whether they needed to supplement the e-learning software.
.................................................................................................................................................................. 217

Table 54: Phase 3 - Alternate form reliability for survey instrument ....................................................... 220

ix
Table 55: Phase 3 - Internal consistency for survey instrument .............................................................. 220

Table 56: Phase 3 - Participant responses on the prototype’s effect on overall enthusiasm and interest in
computing ................................................................................................................................................ 221

Table 57: Phase 3 - IMMS summary results for Attention, Relevance, Confidence and Satisfaction ...... 223

Table 58: Phase 3 - Paired Sample T-Tests for ARCS sub-categories........................................................ 226

Table 59: Phase 3 IMMS Cronbach's alpha results ................................................................................... 227

Table 60: Phase 3 - Kolmogorov-Smirnov results for VARC sub-categories ............................................. 228

Table 61: Phase 3 - Paired Sample T-Test for VARK sub-categories. ........................................................ 229

Table 62: Phase 3 - Related Samples Wilcoxon Signed Rank test for VARK sub-categories. ................... 229

Table 63: Phase 3 - Average duration of engagement with the four levels of e-learning software ........ 231

Table 64: Phase 3 - Confidence Intervals for the median learning duration for the e-learning software 233

Table 65: Phase 3 - Summary of student responses to collaborative learning and assignment activities
.................................................................................................................................................................. 235

Table 66: Phase 3 - Descriptive statistics for %Pre-Test, %Post-Test and %Change. ............................... 236

Table 67: Phase 3 - Descriptive statistics for %Pre-Test, %Post-Test and %Change (outliers removed) . 239

Table 68: Phase 3 - Paired Sample T-Test for %Post-Test and %Pre-Test ................................................ 240

Table 69: Phase 3 - Paired Sample T-Test for %Post-Test and %Pre-Test using Bootstrap ...................... 240

Table 70: Phase 3 - Confidence Interval for Median %Change between %Post-Test - %Pre-Test ........... 241

Table 71: Phase 3 - Model for GCSE Computer Science (2016 specification) grade boundaries ............. 241

Table 72: Phase 3 - One sample T-Test of %Pre-Test towards Grade F threshold ................................... 242

Table 73: Phase 3 - One sample T-Test of %Pre-Test towards Grade F threshold using Bootstrap ......... 242

Table 74: Phase 3 - One sample T-Test of %Post-Test towards Grade C threshold ................................. 243

Table 75: Phase 3 - One sample T-Test of %Post-Test towards Grade C threshold using Bootstrap ....... 243

Table 76: Phase 3 - Tiered correlation results for %Post-Test vs. learning times .................................... 244

Table 77: Phase 3 - Tiered correlation results for %Change vs. learning times ....................................... 244

Table 78: Phase 3 - Tiered correlation results for KS4Prediction vs. learning times ................................ 245

Table 79: Phase 3 - Tiered correlation results for aural modality vs. %Post-Test, Level-3 Time, and Level-4
Time.......................................................................................................................................................... 245

Table 80: Phase 3 - Tiered correlation results for kinaesthetic modality vs. Learning Times .................. 245

Table 81: Phase 3 - Tiered correlation results for IMMS Relevance vs. Learning Times .......................... 246

x
Table 82: Phase 3 - Tiered correlation results for IMMS Satisfaction vs. Learning Times ........................ 246

Table 83: Phase 3 - Tiered correlation results for IMMS Overall vs. Learning Times ............................... 247

Table 84: Phase 3 - Tier 2 correlation results for %Post-Test and the survey instrument ....................... 247

Table 85: Phase 3- Tier 1 correlation results for %Change and the survey instrument ........................... 248

Table 86: Workshop findings on e-learning evaluation protocol ............................................................. 253

xi
 LIST OF ILLUSTRATIVE MATERIAL
Figure 1: IoT, cloud computing, big data and artificial intelligence - the new drivers of the ICT ecosystem
...................................................................................................................................................................... 6

Figure 2: UK overall and sub-index rankings for Networked Readiness Index 2016 .................................... 9

Figure 3: Comparison between the number of IT professionals and applicants to IT-related HE courses,
2002 -2010 (UK domicile) ........................................................................................................................... 11

Figure 4: Number of school papers by country of first author ................................................................... 19

Figure 5: The fundamental processes of learning ...................................................................................... 29

Figure 6: Cognitive theory of learning ........................................................................................................ 41

Figure 7: The high-level structure of the phased research ........................................................................ 51

Figure 8: The incremental and iterative development model. .................................................................. 56

Figure 9: The three phases of research. ..................................................................................................... 57

Figure 10: Phase 1 - mixed methods convergent design ............................................................................ 62

Figure 11: Action research cycle................................................................................................................ 68

Figure 12: Phase 2 - mixed methods convergent design............................................................................ 71

Figure 13: Phase 3 - frequency distribution of Key Stage 4 predictions. .................................................... 76

Figure 14: Phase 3 mixed methods convergent design .............................................................................. 78

Figure 15: Mapping of learning theories to Illeris' three dimensions of learning ...................................... 98

Figure 16: Legend for interrelationship matrix ........................................................................................ 104

Figure 17: Heuristics interrelationship map ............................................................................................. 105

Figure 18: Gradual transition from worked examples to practice problems ........................................... 110

Figure 19: Example of contiguity - Illustration of hydraulic braking system ............................................ 140

Figure 20: Overload of visual channel with presentation of screen text and graphics. ........................... 141

Figure 21: Overload of visual channel with graphics explained by words in audio and screen text. ....... 141

Figure 22: Social actions drive social engagement. .................................................................................. 151

Figure 23: Intended usage of pedagogical heuristics and evaluation protocol ........................................ 158

Figure 24: The composition of Educational Value .................................................................................... 160

Figure 25: High-level flow of e-learning evaluation ................................................................................. 163

xii
Figure 26: Response scale for adherence of e-learning software to this heuristic. ................................. 169

Figure 27: Response scale for level of importance. ................................................................................. 170

Figure 28: Adjusted response scale for level of importance. ................................................................... 170

Figure 29: Phase summary of teacher and education expert feedback on e-learning pedagogy ............ 179

Figure 30: Thematic coverage of results in Phase 1 and 2 ....................................................................... 186

Figure 31: Phase 3 - All things considered, the e-learning software is easy to use. ................................. 202

Figure 32: Phase 3 - The e-learning software is reliable (i.e. does not contain bugs or errors). .............. 202

Figure 33: Phase 3- The learning content in the e-learning software was represented in a clear and
understandable way. ................................................................................................................................ 203

Figure 34: Phase 3- The various instructions and prompt messages are understandable. ...................... 203

Figure 35: Phase 3 - The graphical parts (symbols, logos, diagrams, pictures and illustrations etc.) of the e-
learning software are meaningful. ........................................................................................................... 204

Figure 36: Phase 3 - The e-learning software gave accurate feedback in response to my interactions. . 204

Figure 37: Phase 3 - The navigation and program controls of the e-learning software are logically arranged
and consistent. ......................................................................................................................................... 205

Figure 38: Phase 3- It is easy to use the navigation and program controls of the e-learning software. .. 205

Figure 39: Phase 3- Sometimes I felt that I didn’t quite understand what the e-learning software was doing.
.................................................................................................................................................................. 206

Figure 40: Phase 3 - I found errors (bugs) in the e-learning software that were difficult to recover from.
.................................................................................................................................................................. 207

Figure 41: Phase 3- The e-learning software felt speedy and responsive to my interactions. ................ 207

Figure 42: Phase 3 - The text material in the e-learning software helped me understand the subject matter.
.................................................................................................................................................................. 210

Figure 43: Phase 3 - The videos in the e-learning software helped me understand the subject matter. 210

Figure 44: Phase 3 - The visual material in the e-learning software helped me understand the subject
matter. ..................................................................................................................................................... 211

Figure 45: Phase 3 The collaborative activities (forum discussions, group or pair work) helped me
understand the subject matter. ............................................................................................................... 211

Figure 46: Phase3 - The practice activities (problem-solving) in the e-learning software helped me
understand the subject matter. ............................................................................................................... 212

Figure 47: Phase 3- The quiz questions (assessment activities) in the e-learning software helped me
understand the subject matter. ............................................................................................................... 212

xiii
Figure 48: Phase 3 - The ‘extend your knowledge’ learning material in the e-learning software helped me
understand the subject matter. ............................................................................................................... 213

Figure 49: Phase 3 - The audio material in the e-learning software helped me understand the subject
matter. ..................................................................................................................................................... 213

Figure 50: Phase 3- Overall, at what difficulty level do you believe the learning material represented in the
e-learning software was at? ..................................................................................................................... 215

Figure 51: Phase 3- Overall, at what difficulty level do you believe the practice activities (problem-solving)
were at? ................................................................................................................................................... 216

Figure 52: Phase 3 - Overall, at what difficulty level do you believe the quiz questions (assessment
activities) were at? ................................................................................................................................... 216

Figure 53: Phase 3 - I supplemented, or needed to supplement, the learning material in the e-learning
software with further textbook reading. ................................................................................................. 218

Figure 54: Phase 3 - I asked, or wanted to ask, my teacher for support in understanding the learning
material in the e-learning software. ........................................................................................................ 218

Figure 55: Phase 3 - After completing the four levels of the e-learning software, I was confident that I
would be able to pass a test on it. ........................................................................................................... 219

Figure 56: Phase 3- I would prefer using the internet and the web to support my learning of the computing
subject rather than this e-learning software. .......................................................................................... 220

Figure 57: Phase 3- It is more interesting to use the e-learning software to learn computing than the
textbooks.................................................................................................................................................. 221

Figure 58: Phase 3 - I could use the e-learning software for independent study to learn computing. .... 222

Figure 59: Phase 3- The e-learning software has increased my overall enthusiasm and interest in
computing. ............................................................................................................................................... 222

Figure 60: Phase3 - IMMS ARCS summary results (mean) ....................................................................... 223

Figure 61: Phase 3 - IMMS Attention frequency distribution .................................................................. 224

Figure 62: Phase 3 - IMMS Relevance frequency distribution ................................................................. 224

Figure 63: Phase 3 - IMMS Confidence frequency distribution................................................................ 225

Figure 64: Phase 3 - IMMS Satisfaction frequency distribution ............................................................... 225

Figure 65: Phase 3 - IMMS overall motivation frequency distribution .................................................... 226

Figure 66: Phase 3 - VARK modal preferences for students ..................................................................... 228

Figure 67: Phase 3- The use of different methods to represent the same learning content helped my
understanding. ......................................................................................................................................... 230

xiv
Figure 68: Phase 3 - Duration of student engagement with Level-1 ........................................................ 231

Figure 69: Phase 3 - Duration of student engagement with Level-2 ........................................................ 232

Figure 70: Phase 3 - Duration of student engagement with Level-3 ........................................................ 232

Figure 71: Phase 3 - Duration of student engagement with Level-4 ........................................................ 232

Figure 72: Phase 3 - Frequency Distribution for %Pre-Test...................................................................... 237

Figure 73: Phase 3 - Frequency Distribution for %Post-Test .................................................................... 237

Figure 74: Phase 3 - Frequency Distribution for %Change. ...................................................................... 238

Figure 75: Phase 3 - Group-2 evaluation of support for learning objectives ........................................... 283

xv
 NOTE ON NAVIGATION
The digital version of this document contains internal navigation:

1. Figures and Tables are cross-referenced using standard practice, standard font and link
accordingly.
2. The Table of Contents, List of Tables, and List of Illustrative Material are provided using standard
practice, and link accordingly.
3. References to the Appendices within the body of the thesis do not link to the Appendices
documents, they are for reference only.
4. The heuristics are cross-referenced and link accordingly, using the heuristic number (only) and
standard font.
5. A Glossary of Terms is provided for terminology that may potentially be uncommon. The first
genuine use of the term is marked as a hyperlink (blue underline) and provides a link back to the
term in the glossary. Subsequent use of the term is treated as normal.

xvi
 ACKNOWLEDGEMENTS
To my wife and children: Elena, Nikita-Maria, and Theodore, thank you for your patience in this long
journey.

A special thank you to my supervisory team who each contributed in different ways, each adding special
and unique value in guiding this study:

1. Professor Irene Polycarpou, my Director of Studies (DoS), who throughout remains the academic
calm in the storm; always available with support and practical sound advice, whilst
simultaneously ensuring that I remain the driver of the research, even when I went down the
occasional rabbit hole.
2. Professor Janet Read, whose wealth of experience in educational research and doctoral
supervision enabled her to interject as a radical change agent, ultimately to the benefit of the
study.
3. Assistant Professor Maria Zeniou, whose experience in educational research proved to be
invaluable in challenging me to reflect more deeply and improve the research.

Special thanks to Professor Charalambos Vrasidas, an ex-colleague and now friend, for his input to the
pedagogy as an educational expert.

Finally, my deep gratitude to all the teachers and students who devoted their time and energy to this
study, without which, this research would not exist.

xvii
 GLOSSARY OF TERMS

Glossary Term Description

Accommodation In relation to constructivist principles, accommodation is the process

by which existing schemas must be altered to cope with new
experiences that contradict the existing mental model.

Active Processing According to cognitive load theory, human learning occurs when the
appropriate cognitive processes are engaged to mentally organise
incoming auditory and visual sensory information and integrate it with
existing knowledge so that it can be stored in, and recalled from, long
term memory.

Assimilation In relation to constructivist principles, assimilation is the process by

which new information reinforces an existing schema, and the schema
is augmented with new information.

Behaviourism Behaviourism is a systematic approach to the understanding of human

or animal behaviour; it assumes that behaviour is a consequence of
individual history. Behaviourism gives specific focus to positive
reinforcement and punishment, together with the individual's current
motivational state and controlling stimuli.

Collaboration Collaboration involves a joint group endeavour to solve a problem; all

group members contributing to the same task.

Collaborative Learning A collaborative learning environment is a software system that offers

Environment (CLE) various tools and services that support learners in working and
learning together.

Cognitive Load Theory (CLT) CLT explains how incoming information from eyes and ears is
transformed into knowledge and skills in human memory. It proposes
that learners do not passively receive incoming information, but
instead undertake active cognitive processes that organise the
incoming information into logical structures and integrate it with
existing knowledge for long term recall.

Constructivism Constructivist approaches recognise a real-world that sets limits on

our experiences, but proposes that there is no uniformly perceived
single reality; in fact, each person’s perception of reality is a mental
construct founded on interpretation of their interactions with the
world. An individual’s reality is therefore based on their existing

xviii
 Glossary Term Description
experience and understanding, which is in turn used to make sense of
their current perception of events.

Cooperation Cooperation involves the division and assignment of tasks within the
group to solve the problem.

Deep Learning As opposed to surface learning, in which learning material is passively

memorised with a primary aim of passing assessments; deep learning
is learning where there is a vigorous interaction with the learning
material to truly understand it and integrate it with previous
experience and knowledge. Meaning it is integrated into existing
mental schemas in the learner’s long-term memory.

Dual Channels Humans have separate channels for processing visual and auditory
material.

Equilibrium In relation to constructivist principles, equilibrium is the process of

arriving at a stable state where there is no longer conflict between new
knowledge and existing mental schemas.

Extrinsic Not forming part of or belonging to a thing, or originating from the

outside. Typically, something originating outside of a thing and acting
upon that thing (Merriam-Webster 2018a).

Far Transfer Far transfer is the application of skills and knowledge learned in one
situation to a different situation. It builds upon deep learning and
requires learners to adjust the underlying principles they have learnt
for use in a new scenario or new problem.

Gamification Gamification is the use of game design elements in non-game

contexts. It does not focus on creating fully-fledged games, but instead
uses game dynamics, mechanics, and frameworks to increase
pleasure, fun, motivation, and influence behaviour.

Generation Y Generation Y refers to the specific generation born between the 1980s
and 2000; this term was given to this generation since they succeed
Generation X. They are defined by several characteristics, one of the
most notable being that they were born and raised in an emerging
world of technology, arguably making them inherently knowledgeable
of digital technologies.

Heuristic A heuristic is a specific rule-of-thumb or argument derived from

experience

xix
 Glossary Term Description

Higher Order Thinking Higher order thinking theorises that some types of learning are more
valuable but require more cognitive processing and are more difficult
to teach and learn. According to Bloom’s taxonomy analysis,
evaluation and synthesis are thought to be of a higher order as
compared to remembering, understanding and applying facts and
concepts.

Intrinsic Belonging to the essential nature or constitution of a thing or

originating and included wholly within an organ or part (Merriam-
Webster 2018c).

Key Stage State schools in England, Wales, Northern Ireland and the British
Territory of Gibraltar adhere to targets set by the National Curriculum.
The National Curriculum is divided into key stages according to pupil
age and the school years the key stage is planned to be taught in.

Mental Model Mental models are our internal symbolic representation of external
reality. They explain our thought process about how something works
in the real-world, and shape our behaviour and approaches to solving
problems.

Metacognition Metacognition is "cognition about cognition"; in this context, it relates

to thinking about one's own thinking process such as study skills,
memory capabilities, and the ability to monitor learning. It is a self-
awareness of our own cognitive processes and the understanding of
how to regulate those processes to maximise learning.

Mindful Activity Mindful activity is activity in which the learner is in direct contact with
real or virtual objects, and is encouraged to manipulate them to think,
hypothesise and test their hypothesis.

Multi-Modal Multi-modal approaches combine a mixture of approaches and

teaching methods to offer balanced modal coverage.

Learning Theory Learning theories are conceptual frameworks that describe how
humans acquire new, or modify or reinforce existing knowledge,
behaviour, skills, values, or preferences.

Limited Mental Capacity At any given time, humans can actively process only limited
information in each channel; material that exceeds this threshold may
enter working memory but will not be processed and encoded into
long term memory.

xx
 Glossary Term Description

Orienting Reflex Orienting Reflex is an organism's immediate response to a change in

its environment; that change is not sudden enough to elicit the startle
reflex, but is a novel or significant stimulus. This initial response makes
the organism temporarily more sensitive to the stimulation.

Part-task Instruction Traditional teaching methods take a part-task approach which breaks
the syllabus down into small parts that teach topics and sub-topics;
these are, in turn, followed by frequent (relatively small) practice
activities. This approach gradually builds knowledge and skills in the
learner.

Pedagogy Pedagogy focuses on the theory and practice of education, more

specifically the study and practice of how best to teach and assess.

Problem Manipulation As part of active learning, students are encouraged to engage in

Environment (PME) mindful activity. A PME supports such activity and should have a low
floor in terms of ease of entry and a high ceiling in terms of features
and functionality that learners can eventually master. In a computing
context, such an environment should allow students to model and run
simulations, look-under-the-hood on existing solutions, employ trial
and error, implement designs, and test and debug solutions.

Progressive Disclosure Progressive disclosure is an instructional technique used to reduce

cognitive load by disclosing the minimal learning material required and
releasing more information progressively, thereby avoiding learners
being overwhelmed. This technique can also be used to create
curiosity and maintain suspense by not providing all the necessary
material in one go.

Reflective Practice Reflective practice is the capacity to reflect (think deeply or carefully)
on our actions or thought processes to develop insight that, in turn,
enables improvement. It is argued that experience alone does not
necessarily lead to learning; deliberate reflection on experience is
essential.

Schema Schemas are the mental constructs that organise and categorise our
skills, and knowledge and understanding of the world.

Sensation-seeking Reflex Sensation seeking is a personality trait with a biological basis defined
by the "seeking of varied, novel, complex, and intense sensations and

xxi
Glossary Term Description
experiences, and the willingness to take physical, social, legal, and
financial risks for the sake of such experience”(Zuckerman 1994, p.27).

xxii
 CHAPTER ONE

1 INTRODUCTION
Digital technology penetrates all aspects of our lives and is fundamentally interwoven in the fabric of
society in developed countries. It is no longer a ‘nice to have’, it is a necessity; a lack of access and
knowledge in using digital technology leaves those without at a disadvantage economically, personally,
and in terms of future opportunities. This is true at both an individual level and at a national level, as
nations compete within the global economy.

The rapid innovation brought about by digital technology is driving globalisation and is a major
competitive factor between countries. So much so that nations compete on the strength of their
information technology (IT) sectors. Closely following the national imperative to invest in their IT sector is
the recognition that at the heart of a country’s IT sector is a well-educated and skilled workforce; yet these
skills are in global short supply.

The UK is one country actively engaged in this global competition. Around the turn of the last decade
(2010), although the UK’s IT sector was in good standing, there were deep concerns regards its inability
to meet the demand for IT professionals, affecting the economy and global competitiveness. This in turn
led to several high-profile critiques of UK computing education, which led to a well-publicised body of
inquiry and political bombast that UK computing education (especially in high-schools) was in crisis.

Significant reform was introduced to the computing curriculum in schools, and the information and
communications technology (ICT) curriculum was disapplied in 20123 and replaced with a renewed focus
on computing and computer science. This reform created considerable impact on the UK’s high-school
teaching community. In the context of this research, the most important being:

1. A shortage of qualified specialist teachers,

2. A lack of Continued Professional Development (CPD) and subject-specific pedagogy, and
3. A concern over often inadequate technical teaching resources, of which e-learning software is
included.

In parallel to these events, a natural by-product of the ubiquitous nature of technology is that it also
permeated into education; recent decades have seen a growing movement towards technology-enhanced
learning. There has been rapid growth of e-learning usage in industry, and to a good extent in schools.
Therefore, in view of the challenges faced in computing education in schools, one approach to support
teachers and high-school computing is the increased usage of e-learning software to supplement
instruction and aid individual learning.

3
Meaning ICT remained a national curriculum subject at all four key stages until the new national curriculum came
into force in September 2014. However, the legal requirement for schools to adhere to the specific national
curriculum programmes of study, attainment targets and statutory assessment arrangements was removed.

1
 CHAPTER ONE

In the context of this research, e-learning software is considered: software that delivers content and
instructional methods to facilitate learning, ideally in an engaging and interactive manner that promotes
active learning. The e-learning software should cover a complete topic or portion of the curriculum (e.g.
algorithms, programming, data representation, networks, etc.).

Several learning benefits are reported from the use of e-learning software, in terms of learning
performance and motivation. However, at the foundation of this research is the recognition that there
remains inconsistency in the quality of existing software; typically, in the areas of pedagogical design and
alignment with educational needs.

1.1 Research Aims

Considering the aforementioned context this research aims to support the increased use of e-learning
software in high-school computer science, and to simultaneously safeguard the pedagogical quality of the
software. It does this by defining a comprehensive set of pedagogical heuristics for use by teachers or
other educators in the design, or evaluation and selection, of e-learning software for high-school
computing. The heuristics can also be used by instructional designers engaged in the design of computing
e-learning software.

1.2 Research Contributions

The objective of this study is to synthesise a wide body of disparate learning theory knowledge into a
holistic pedagogy for e-learning that concentrates on high-school computing. In addition, the pedagogy
must communicate the essence of this wide body of pedagogical knowledge in a detailed, yet
understandable and accessible form, to a (primarily) teacher audience.

Building on the e-learning pedagogy is an e-learning evaluation protocol that offers structure and
guidance to teachers or other educators in the evaluation and selection of e-learning software for use in
their teaching.

1.3 Research Question and Hypotheses

This study seeks to answer, ‘whether there are improved student assessment results from e-learning
software that is designed in adherence with the pedagogical heuristics developed in the study’. This leads
to the following experiment hypothesis, which is answered in the final phase:

1. H0: The use of e-learning software designed in adherence to the pedagogical heuristics presents
no difference in student learning performance.
2. H1: The use of e-learning software designed in adherence to the pedagogical heuristics presents
improved student learning performance.

2
 CHAPTER ONE

1.4 Research Context

The recent events and challenges in the UK serve as a backdrop for this research, and motivate this study’s
support for computing education and computer science teachers through the difficult transition from ICT
to computer science. However, the UK is by no means unique; the transition of high-school curriculums
to a more computer science focus has been undertaken by several countries, with varying results and a
cross section of similar challenges. This gives a clear indication of the transferability of this study in an
international context.

1.5 Challenges and Limitations

It should be remarked that research in a school context, with in-service teachers and school pupils, was
the defining challenge of this research. Much of the research had to be scheduled within the school
academic year avoiding busy periods, exam periods and holidays, and trying to fit into the general
availability of both the teachers and students. In particular in the final phase, the study had to be
scheduled according to the GCSE scheme of work, or risk being delayed to the following year.

The grooming and preparation of schools and their teachers to participate took many months, typically
starting in the previous academic year and on numerous occasions leading to teachers withdrawing at the
last minute due to their pre-existing school workload. Hence throughout the study it was a constant
challenge to secure teachers and students to participate.

Section 3.4.2.1 discusses in detail the relative limitations in transferability of the student sample in the
qualitative exploratory phases. Additionally, it should be noted that as is common in computing there was
a significant gender imbalance (only one female student was involved in the exploratory phases). With
reference to section 3.6.2.2 the gender imbalance remained in the final quantitative phase but was
broadly aligned with the overall GCSE computing population.

As discussed in section 8.2, this research does not propose e-learning as a replacement for high-school
teachers. Hence by design, it does not consider a control group by which students are taught by traditional
didactic methods vs e-learning software designed in adherence with the pedagogy. An alternate control
group was considered with students using a different e-learning software which is purportedly equivalent
to the e-learning software prototype. However, in consideration of the quantitative priority in Phase 3
and a student sample of 66, this was rejected in favour of safeguarding statistically significant results and
confirming improved learning performance based on the e-learning prototype. The use of alternate
control groups is further discussed as future work in section 8.2.

Although not a challenge or limitation per se, the ethical implications of the research work were addressed
under the UCLan ethical approval process, by the Science, Technology, Engineering, Medicine and Health
(STEMH) ethics committee. The three-phase research methodology secured separate ethics approval for
each phase, and the inclusion of underage participants was managed and approved with due
consideration. Although not limited to these areas, special consideration was given to: how information

3
 CHAPTER ONE

was given to schools (head teacher), teachers, parents and students to secure their informed consent;
how withdrawals were managed; the appropriateness of research protocols, instruments and materials
for the underage student participants; how participant information was anonymized; and how research
data was stored, protected and archived.

1.6 Thesis Organisation

This thesis is presented in three volumes; the thesis body (volume 1) and the appendices (volumes 2 and
3). The next chapter presents the academic foundation of the study (literature review), giving focus to:
the underlying problem domain and international context; the research shortfall this study addresses - in
terms of e-learning heuristics, evaluation protocols, and pedagogy for high-school computing; the current
status and concerns relating to e-learning software; and the learning theories underpinning the pedagogy.

Later chapters elaborate in detail on the: three phase mixed methods research design; the synthesis of
the e-learning pedagogy; the synthesis of the e-learning evaluation protocol; the results from all three
phases of the study; the discussion and interpretation of those results; and the conclusion, contribution
and future works springing from this study.

4
 CHAPTER TWO

2 LITERATURE REVIEW
2.1 Digital Technology and the Economy
In 2012, the UK seemed to be in good standing with a global ICT Development Index (IDI) 4 position of 8th
(ITU 2013). However, in the 2011 MacTaggart lecture, Eric Schmidt (Google Chief Executive) lamented the
UK’s failure to build upon its long history in innovation. He expressed the opinion that the UK needed to
start at the very beginning with education, and to reignite children’s passion for science. This was echoed
by the 2012 figures from e-skills UK which showed that, from 2002 to 2010, while applicants for higher
education (HE) courses increased by 51%, applicants for single subject IT-related courses decreased by
28% (e-skills, 2012). At the time, this led to a well-publicised body of inquiry, consisting of various reports,
analyses and a good measure of political rhetoric, that asserted that computing education in the UK was
in crisis. This section explores the economic portion of that body of enquiry, contrasting it with current
digital drivers and indicators in the UK and globally.

2.1.1 Digital Technology is Ubiquitous

The prevalence and ubiquitous nature of Information and Communication Technology (ICT) in developed
countries and its impact on recent generations is well documented (DCMS & DBIS 2009; Halse et al. 2009;
Dutta & Mia 2011; Baller et al. 2016).

“Few today would go back willingly to a world without the Internet and its many associated developments.
For many young adults, conceiving of such a world may even be impossible. ICT, and the internet
particularly have already changed the world dramatically, and all indications point to an even higher rate
of transformation of our lives over the next decade.” - (Dutta & Mia 2011, p.29) World Economic Forum.

Five years later, in 2016, the World Economic Forum was heralding the “… Fourth Industrial Revolution.
Processing and storage capacities are rising exponentially, and knowledge is becoming accessible to more
people than ever before in human history.”(2016, p.v).

One quantifiable metric of global technology dependency is worldwide internet usage. As reported by the
International Telecommunications Union (ITU 2014; ITU 2017a) and the UK Office of National Statistics
(2013; Prescot 2017) internet penetration continues to grow in both developing and developed countries,
and the UK is ahead of most developed countries (refer to Table 1).

Whilst internet access varies depending on household composition, it is important to note that 98% of UK
households with children had an internet connection in 2017 (Prescot 2017). Also there is a growing trend,

4
The ITU Development Index (IDI) is a benchmark of ICT development in 176 economies; it measures 11 indices
that are subdivided into three categories: access; use; and skills.

5
 CHAPTER TWO

both globally and in the UK, of accessing the internet ‘on the go’ via mobile devices; in 2017 73% of UK
adults accessed the internet from a mobile device (Prescot 2017).

Internet Penetration (Households)

Developing Developed UK

2014 32% 78% 83%

2017 42.9% 84.4% 90%

Table 1: Comparison of global internet penetration and the UK
Adapted from (Office of National Statistics 2013; ITU 2014; Prescot 2017; ITU 2017a)

As early as 2009, the Digital Britain Report (DCMS & DBIS 2009) from the Department for Business
Innovation and Skills (BIS) and the Department for Culture, Media and Sport (DCMS) gave a realistic
representation of how technology underpins the UK in areas such as: personal, business and mass media
communications; personal and commercial transport; energy and utility infrastructure; the financial
industry; high street credit/debit card transactions; online purchases; digital living based on new digital
services; new ways of working (telecommuting); digital and self-publishing; education; healthcare; online
banking, job searches and recruitment; and e-government. These areas remain relevant and have all
intensified; more recently the ITU reflects a more holistic perspective of an ICT ecosystem in which the
Internet of Things (IoT), cloud computing, big data, and artificial intelligence are tied to every aspect of
our lives (ITU 2017b). Refer to Figure 1.

Figure 1: IoT, cloud computing, big data and artificial intelligence - the new drivers of the ICT ecosystem
Source: (ITU 2017b, p.95)

6
 CHAPTER TWO

2.1.2 The Economic Benefit to the UK

At the turn of this decade (2010), the magnitude of the positive impact that digital technology had on the
UK economy was quantified in various metrics:

• IT and digital communication sectors accounted for nearly £1 in every £10 that the whole
economy produced each year (DCMS & DBIS 2009);
• The IT & telecoms industry accounted for 8% (£75 billion) of the UK’s total Gross Value Added
(GVA). (e-skills UK 2012);
• The economic contribution of the internet in 2009 was worth an estimated £100 billion to the
UK economy (Boston Consulting Group 2010, p.5).

Although the impact that digital technology has on the UK economy has increased, more recent reporting
does not give such a clear-cut snapshot. In 2017, the Office of National Statistics reported that 77% of UK
adults bought goods or services online, an increase from 53% in 2008. Furthermore, in 2017, 93% of adults
who had bought online in the last 12 months had done so from UK online vendors.

As reported by the World Economic Forum (Baller et al. 2016), the estimated value of globalisation in
2016 was US$73.5 trillion. This is apportioned to the nations participating in the global economy, of which
the UK is a major contributor, and an estimated 75% of those benefits are captured by companies in
traditional industries.

2.1.3 The Digital Divide

Although digital technology is argued to be ubiquitous, it is only ubiquitous for those that have access and
the skills to use it. The digital divide exists on a global level between countries, and on a national level
within the populations of each country. In 2017, the ITU (2017a; 2017b) continued to report on the global
digital divide; the proportion of households with Internet access was reported to be 84.4% for developed
countries, 42.9% for developing countries, and only 14.7% for the least developed countries.
Furthermore, on a 10-point scale, the gap between the lowest and highest performing countries in the
ITU index, grew to 8.02 points. Most importantly, the ITU reported that there was a strong correlation
between a country’s economic and ICT development, which seemed to be a self-sustaining phenomenon.

In the UK, PricewaterhouseCoopers (PwC) (2009) estimated the total potential economic benefit from
getting everyone in the UK online as being in excess of £22 billion. Later, in 2015, the Tinder Foundation
(Oliver Hogan, Colm Sheehy 2015, p.4) cite a yearly benefit of £358 million for individuals, and £243 million
for the government in additional revenue. Furthermore, taking a 10-year view up to 2025 they estimated
an NPV5 of £14.3 billion for the investment of equipping the nation with basic digital skills.

5
Net Present Value

7
 CHAPTER TWO

2.1.4 Globalisation and National Competitiveness

Building on the discussion of the digital divide, globalisation has introduced fierce competition, and led to
the situation where the strength of a country’s IT sector is critical to its global economic survival. Decisions
relating to policy and investment in digital technology are now identified as being of national importance.
At the start of this decade, the Royal Academy (2009) and Oxford Economics (2011) raised alarm bells that
the global shifting of skills and workforce was leading to investment wherever the IT skills were strongest,
and that countries with inadequate skills would not be able to compete. At that time, e-skills UK (2012)
was also reporting that firms in emerging economies were investing aggressively in digital technology, and
the number of firms increasing their investment by over 20% was twice as high compared to Europe.

In fact, in 2012 the UK was rated highly with respect to IT investment and utilisation, and was actually in
a relatively strong position globally. However, the overriding stance from the government and semi-
governmental agencies was that there was no room for complacency, and that the UK must work even
harder to reach the levels of its Nordic neighbours.

Since then, the UK policy stance has proven to be justified by the findings of the World Economic Forum
and the ITU. In 2016, the World Economic Forum (Baller et al. 2016) reported that firms would face
increasing pressure to innovate continuously. Additionally, the report parallels the growth of globalisation
with the development of the commercial internet and quantifies the growth of globalisation since 1980
from US$11.1 trillion to US$73.5 trillion.

The ITU (2017b) report that the UK was one of six European countries in the top ten IDI rankings. This was
attributed to “high levels of ICT development as a result of high levels of investment in ICT infrastructure,
high-quality networks, and high levels of take-up of services by consumers” (ITU 2017b, p.35). It was noted
that these countries also stand towards the top of the rankings for gross national income (GNI) per capita
and other economic indicators.

On the surface, the UK’s lack of complacency has paid dividends both economically and in terms of their
global ICT rankings: the World Economic Forum ranked the UK 15 th in their Networked Readiness Index 6
(NRI) in 2011 (Dutta & Mia 2011), and 8th in 2016 (Baller et al. 2016); whilst the ITU ranked the UK 8th in
2012 (ITU 2013), and 5th in 2017 (ITU 2017b).

However, the UK’s lack of complacency must continue since the World Economic Forum reports relatively
poor rankings in terms of affordability and with significance to this research: skills (refer to Figure 2).

6
The Networked Readiness Index is reported by the World Economic Forum. It assesses the networked readiness of
139 economies across 53 individual indicators that are subdivided into four categories: environment; readiness;
usage; and impact.

8
 CHAPTER TWO

60 53
50
40

Rank
30 24
20
20 16
8 10 11
10 5 5 5 5
0

Skills
Infrastructure

Affordability

Governement Usage

Social impacts
Individual Usage

Business Usage

Economic impacts
Networked Readiness

Business and innovation

Political and regulatory
Index (Overall)

environment

environment

Environment Readiness Usage Impact

Index

Figure 2: UK overall and sub-index rankings for Networked Readiness Index 2016
Adapted from (Baller et al. 2016)

2.1.5 Demand for IT Professionals

It is well established that a skilled workforce is at the heart of any country’s IT Sector, that those skills are
globally in short supply, and that the skills shortage will intensify in the foreseeable future (Thomas et al.
2007, p.14). For the UK to continue to enjoy the aforementioned economic benefits and remain
competitive in the global economy, it must continue to focus on resolving the skills gap. In 2012, set
against a backdrop of increasing demand for IT professionals versus the drop in higher education IT
applicants, this was an area of concern. E-skills UK (2012) quantified this demand in terms of job vacancies;
in 2011, there were approximately 116,000 advertised vacancies for technology (IT & Telecoms)
professionals during each of the four quarters as compared with 82,000 during the whole of 2009.
Furthermore, predicted growth rates up to 2020 showed the IT professional workforce was expected to
grow at 1.62% per annum, nearly double the predicted growth rate for UK workers overall. These findings
were substantiated by a Confederation of British Industry (CBI) survey which showed that businesses
continued to report a shortage of people with science, technology, engineering and maths (STEM) skills
(CBI 2011). Furthermore, among science, engineering and IT firms, more than 84% viewed the number
and quality of STEM graduates as a priority concern (CBI 2011).

In recent years, despite a significant government focus and a shakeup within the high-school computer
science curriculum, a turnaround in the UK digital skills shortage has not occurred. The Tech Partnership,
the successor to e-Skills UK, reports that on average, 163,000 vacancies for digital specialists were
advertised across the UK during each quarter of 2015; this was an increase on the 2011 figure of 116,000
per quarter (TechPartnership 2016). Using the latest workforce estimates from the Office of National

9
 CHAPTER TWO

Statistics in conjunction with internal forecasts developed in association with Experian, the Tech
Partnership (2016, p.1), also reported that:

• 1.8m (6%) of the UK workforce were working in the technology sector – 1.3m (74%) as technology
specialists;
• Between 2015 and 2025 the number of people working as technology specialists was forecast to
increase by approximately 28% (2.8% per annum), to 1.65m people;
• By comparison, in the same period, the UK workforce was expected to grow only by around 7%.

The Tech Partnership (2016) forecasted that total demand, including growth and replacement of industry-
leavers, was estimated to be 1.71m gross job opportunities between 2015 and 2025. Considering the high
educational demands of the industry, a significant portion would need to be filled by those graduating
through education. However, analysis by the Tech Partnership (2015) of data from the Higher Education
Statistics Agency (HESA) for the period 2013/14 showed that of 777,600 qualifiers from UK Higher
Education Institutes, only 28,400 (3.6%) had followed technology courses. Of these, 19,600 qualifiers were
UK residents, and amongst these qualifiers, 13,700 had gained an undergraduate degree. This does not
reflect well on the UK’s capacity to meet future demand. Commenting on demand in 2015, the Tech
Partnership (Sambell 2016) also reported that: 42% of employers recruiting technology specialists were
already struggling to fill their vacancies, and that these job vacancies amounted to around £2 billion of
lost GVA to the UK.

2.1.6 It Starts with Education

In relation to the significant value of the IT sector to the UK economy, the substantial demand for IT
professionals, and the skills shortage, it becomes a national imperative to protect the UK’s global
competitiveness in the IT sector. It was argued by the Royal Society and various other sources that this is
“dependent upon a home-grown, reliable and growing supply of highly skilled graduates with significant
deep knowledge in Computing disciplines”(2012, p.24).

The objective was to inspire future talent to pursue technology-related careers by fostering a pipeline of
IT talent in the education system, starting from primary school onwards.

There are a number of contributing factors to the lack of uptake into Higher Education IT courses, and its
associated impact on the IT skills shortage. Those factors, the measures taken by the UK government and
the Department for Education (DfE) since 2012, and their relative impact are investigated in the next
section.

10
 CHAPTER TWO

2.2 Digital Technology and Education

2.2.1 The Call for Reform in High-School Computing Education

Computing and ICT education has suffered from the turn of the millennium, both in declining student
numbers and enthusiasm for the subject area. The 2012 figures from e-Skills UK show that from 2002 to
2010 applicants for higher education courses increased by 51%; comparatively, during the same period
applicants for single subject IT-related courses decreased by 28% (refer to Figure 3) (e-skills UK 2012,
p.83).

Figure 3: Comparison between the number of IT professionals and applicants to IT-related HE courses, 2002 -2010
(UK domicile)
(e-skills UK 2012, p.84)

In parallel, computing education at Key Stage 4 (GCSE) and Key Stage 5 (A-Level) showed a significant
decline in student numbers. Analysis from e-Skills UK showed an overall decline of 43% in Computing and
ICT A-Levels, between 2003 to 2011 (2012: 94). Likewise, there was a decline of 70% from 2005 to 2011
in GCSE ICT courses (2012: 95). It should be noted that at that time, there was no GCSE in Computing, and
student uptake in A-Level ICT7 was significantly higher than A-Level Computing.

This decline in student numbers viewed in light of section 2.1’s discussion of globalisation, the UK
economies dependency on digital technology, the skills gap, and ongoing demand for technology
professionals prompted a public call to arms from the BCS, e-Skills UK, The Royal Society, Google,
Microsoft and the DfE to reform high-school computing education.

7
The content of the Computing A-level is more relevant to Computer Science and IT Higher Education and IT-related
employment than ICT, which is more focused on end-user skills.

11
 CHAPTER TWO

A note of caution must be given, the negative impact of the drop-off in domestic IT HE applicants in
relation to the increasing demand for new IT professionals is important, but it should not be overstated.
IT-related HE is an important source of talent for the IT sector, but it is not the only source. IT employers
actively encourage graduate intake from degrees other than computing or IT, since this affords a healthier
workforce mix. Results from the Higher Education Statistics Agency (HESA) for 2010 (e-skills UK 2012,
p.86) indicate that of the UK domicile graduates taking up IT & Telecoms roles within six months of leaving
university, 50% were from computing, IT and telecoms-related courses and 50% were from other
disciplines. This reaffirms the point that IT employers actively encourage graduate intake from other
degrees and provides a tentative indicator that some computer science and IT graduates gain employment
in other areas.

Additionally, an alternate rational explanation for the trend reversal in applications to computer science
HE is the dot-com crash in 2001, which corresponds with the start of the slump (The Royal Society 2012).

It must also be considered that the link between Key Stage 4 and Key Stage 5 education with HE uptake is
multifaceted and dependent on the complex interplay of factors such as:

• The incentives given to schools to offer certain subjects (The Royal Society 2012; Computing at
School Working Group 2012; Wolf 2011; DfE 2011);
• The number and diversity of vocational qualifications offered at the time (Ofsted 2011; Wolf
2011; The Royal Society 2012);
• The pre-requisites in place for HE courses, and the value placed on high-school qualifications
(Computing At School Working Group 2009; The Royal Society 2012; Computing at School
Working Group 2012).
• The preparatory value of high-school qualifications versus the potential overlap they may have
with the 1st year of HE courses (The Royal Society 2012; Campbell & McCabe 1984; Franklin 1987;
Rountree et al. 2004); and
• Public perception of particular high-school qualifications (e-skills UK 2009; The Royal Society
2012).

Despite the above, in public and political media, the root cause of the crisis in computing education was
laid squarely on the negative perceptions of ICT at school, poisoning young people’s attitude towards
computing. It was viewed that ICT teaching in UK schools and colleges was deteriorating and had shifted
its focus to the basics of digital literacy. Educational reform was called for in computing education at Key
Stage 4 and 5, specifically to refocus away from ICT (digital literacy) towards computer science, with one
central initiative being to create a new GCSE in Computing (e-skills 2008; Computing At School Working
Group 2009; DCMS & DBIS 2009; e-skills UK 2009; Schmidt 2011; Gove 2012; The Royal Society 2012).

“To bridge the gap in core skills needed for competitiveness, it is essential to encourage students in
schools and further education to develop these crucial skills. The proposed introduction of a GCSE in

12
 CHAPTER TWO

Computing, the revision of the ICT curriculum at A-level and the introduction of the Diploma have already
been raised as important steps in the right direction.” (The Royal Academy of Engineering 2009, p.21)

Academically the GCSE in ICT had a strong bias towards IT user skills, meaning it was not directly relevant
to A-Level Computer Science, or IT-related HE and careers. The need for ICT qualifications was and is not
in debate; it is simply that they hold a different purpose to qualifications in computer science. The
exclusive focus on ICT meant that at that time school pupils had, in real terms, fewer opportunities to
learn computing than they had been offered 20 years earlier. It was argued that the lack of an alternative
to ICT qualifications at Key Stage 4 led to the profound demotivation of those students expecting a
computer science focus (CPHC 2009), and that not enough was being done in schools to promote
computer science as a degree choice (Mellors-Bourne 2008).

These findings are what led to the existing ICT curriculum being disapplied in 2012, and the announcement
of a new focus on Computing and Computer Science GCSEs. At the time, OCR 8 had been piloting a GCSE
in Computing, which was being piloted in 2010 and rolled out nationally from September 2011.

2.2.2 Academically Rigorous Computing and Computer Science GCSEs

As outlined previously, one factor affecting the uptake of a qualification is government incentives
associated with that qualification. One incentivising mechanism is the English Baccalaureate (EBacc) which
is a performance measure for schools (BCS 2012). It is awarded where students attain a grade C or above
at GCSE level in five core subject areas: English, Mathematics, History or Geography, a language and the
Sciences, of which Computer Science was now included.

On 11 January 2012 at BETT9, the Secretary of State for Education Michael Gove MP (2012) signalled the
intention that if Computer Science GCSEs were created that were academically rigorous, intellectually
challenging and offered genuine practical value, then they would be considered for the EBacc. As of
December 2013, GCSE Computing and Computer Science qualifications from OCR (2012), AQA10 (2013)
and Edexcel11 (2013) were accredited for inclusion in EBacc. The inclusion of Computing and Computer
science GCSEs in the EBaccs is significant since it: combats the previous negative perception of ICT GCSE
by rebranding the Computer Science GCSE as a subject worthy of EBaccs; signals the academic rigour of
computer science qualifications; and incentivises schools to promote the computer science qualifications.
The latter point is clearly reflected in the fact that, as of 2013 (DfE 2014, p.2), 55.6% of teaching time was
spent on EBaccs subjects.

8
Oxford Cambridge and RSA (OCR) is a qualification awarding body.
9
British Educational Training and Technology Show
10
Assessment and Qualifications Alliance (AQA) is a qualification awarding body.
11
Pearson Edexcel is a qualification awarding body.

13
 CHAPTER TWO

2.2.3 Impact on the Teaching Community

The shift towards more rigorous academic subjects, such as computing and computer science, created
significant change and impact within the UK high-school teaching community; with relevance to this
research, the most important being a:

Shortage of qualified specialist teachers: A variety of sources raise the critical concern that computing
and computer science are subjects largely new to secondary schools, and that due to strong demand
within industry there is a lack of qualified teachers to teach these subjects (DCMS & DBIS 2009; CAS 2011;
CAS 2013; The Royal Academy of Engineering 2009). The Royal Society (2012), using data from the
inaugural Schools Workforce Census conducted by the DfE in November 2010, highlighted that of 18,400
ICT teachers in England only 35% had a relevant qualification 12, of which only 25% had both a relevant
first degree and a teacher training qualifications. In their report: Shut down or restart? The way forward
for Computing in UK schools, they recommend that the government set targets for the number of
computer science and IT specialist teachers, and monitor recruitment against these targets.

Lack of Continued Professional Development (CPD) and subject-specific pedagogy: The existing
challenges in Continued Professional Development for teachers were further compounded by the
significant pressure to re-skill existing teachers towards the new computing curriculum (Sentance et al.
2012; The Royal Society 2012; Pachler 2010). In particular, there was concern over “a lack of support for
CPD that deepens subject knowledge and subject-specific pedagogy in this area, particularly in Computer
Science” (The Royal Society 2012, p.9). The Royal Society (2012) recommended that the government set
a minimum level of subject-specific CPD for computing teachers that deepened subject knowledge and
subject-specific pedagogy.

Concern over often inadequate technical teaching resources: There have been mixed perspectives
regarding technical teaching resources. In 2009, there were positive responses that access to web
resources were estimated to yield an increase of approximately ¼ of a GCSE grade in each subject (DCMS
& DBIS 2009, p.31). The British Computer Society (BCS 2012) commented that there were suitable
educational resources available that enable teachers to effectively teach the concepts of computer
science. The Computing at Schools working group (2012) commented that there were fantastic free
resources available. Ofsted (2011) reported that in the schools surveyed up to 2011, there was growing
use of ICT equipment and materials to enhance learning, and that these investments were changing
learning, with a positive response from students. Ofsted reported that “All the schools visited had made
investment in ICT infrastructure, equipment and resources during the period of the survey with notable
increases in mobile devices, laptops and virtual learning environments.” (2011, p.46). However, according
to e-Skills UK (2009), only a few years earlier, the majority of teachers were complaining that they had to

12
The Royal Study defined relevant qualifications to be those with JACS codes of G400-G700 plus G900. These
codes included degrees in computer science, information systems, software engineering, artificial intelligence and
other mathematical and computing degrees.

14
 CHAPTER TWO

pull together their own materials or adapt existing resources; that there were insufficient resources to
keep students engaged; and not enough time given to teachers to investigate new technologies. In 2012,
the Royal Society (2012) commented that technical teaching resources were often inadequate and
recommended to the government that appropriate technical resources be made available in all schools
to support the teaching of computing.

2.2.4 What has Changed in Five Years?

The Royal Society was instrumental in calls to reform computing education in UK schools, and in giving
recommendations toward those reforms; this was embodied in their 2012 report: Shut down or restart?
The way forward for Computing in UK schools. Five years later, they published a follow-up report that
monitored what progress had been made; this report is called - After the reboot: Computing education in
UK schools (Royal Society 2017b). In the latter report, the Royal Society maintained that “Data and digital
technologies promise revolutionary transformational changes across the full range of industry sectors and
spheres of life” (2017b, p.6). However, that high-school computing education in the UK remained patchy
and fragile.

In the time since the last report, the subject of computing, encompassing computer science, digital
literacy, and information technology (IT) has become mandatory in schools from ages 5 to 16, and has
been established as a foundational subject alongside English, Mathematics and the sciences (Royal Society
2017b; Crick 2017). However, the transition has not been easy, and there remains some way to go.

Shortage of qualified specialist teachers: Previously, a critical concern was a lack of qualified teachers to
teach computing and computer science principles; the Royal Society (2017b) reported that from 2012 to
2017 England met only 68% of its recruitment target, and in their teacher survey, 44% of secondary school
teachers only felt confident teaching the earlier portions of the curriculum, which have less focus on
computer science. The Royal Society summarised that teachers were working extremely hard, but that
the “majority of teachers are teaching an unfamiliar school subject without adequate support” (2017b,
p.6).

Lack of Continued Professional Development and subject-specific pedagogy: Previously, CPD was
envisaged as a mechanism to bridge the gap in subject-specific knowledge and pedagogy. However, the
Royal Society (2017b) reported that that the government had taken a passive role in setting minimum
levels of CPD, resulting in a large difference in the CPD hours undertaken by computing teachers. In
Scotland, this had been set at 35 hours per year, although it is not required to be subject specific. Whereas,
in response to their teacher survey, 26% of English respondents indicated that they undertook zero hours
of CPD and 40% had less than nine hours (Royal Society 2017b, p.73). In the five years since the original
report, the focus of providing subject-specific pedagogy to school teachers has evolved into the realisation
that research is needed to identify pedagogy to teach computer science in schools. The significance of this
realisation is discussed in more detail in section 2.2.4.1.

15
 CHAPTER TWO

Concern over often inadequate technical teaching resources: The Royal Society (2017b) report that
schools have not received ring-fenced funding for technical resources for computing. However, there are
several new, free software resources available, and several low-cost hardware resources. The relative
increase in technical resources has in turn caused a challenge to teachers in choosing appropriate
classroom resources: “A common refrain in the responses from teachers was that they felt overwhelmed
by the abundance of resources and were often unsure of their suitability.”(Royal Society 2017a, p.77). The
Royal Society emphasises the importance of appropriate technical teaching resources, but acknowledges
there is a complex landscape of resources available that must align with the curriculum and support the
varying knowledge levels of teachers. Citing Falkner and Vivian (2015), Kafai and Vasudevan (2015), and
Rich, et al. (2017), Waite (2017) comments that the resources available to teachers typically focus on
coding and content, rather than problem-solving and pedagogy; Waite goes further in calling for an audit
of the pedagogical foundations of technical resources currently recommended to teachers.

Considering recent sources, the three key factors that prompted this research in late 2013 remain evident,
and in some cases have intensified. Furthermore, a highlighted concern of the lack of appropriate
pedagogy towards teaching computing in schools gives further impetus to this study.

2.2.4.1 Lack of Research in Computing Education in Schools

In the UK, the new computing curriculum has also resulted in a resurgent research focus on computing
education in schools, and in particular the pedagogy for teaching computing in schools (Sentance & Selby
2015; Royal Society 2017a; Crick 2017; Waite 2017). The Royal Society (2017a) contends that:

1. General pedagogical approaches do not tackle the challenges faced by computing teachers and
specific computing pedagogies are essential to support effective teaching;
2. Much of the existing research in computing education is focused on higher education; therefore,
research is required to ascertain and give evidence of what is effective pedagogy at school level;
3. The relative newness of computing as a subject means associated pedagogies are not as mature
as in other subjects; and
4. Theories of learning and instruction should be fundamental to any programme of computing
education research.

Sentance and Selby (2015), Waite (2017), and Crick (2017) give support to the Royal Society’s assertions
that there is a need for pedagogy for teaching computing in schools, and that the majority of existing
research has focused on higher education. Sentance and Selby’s (2015) classification of research into
computer science education in schools from 2005 to 2014 illustrates this aptly. Concentrating on seven
journals focused on computing education, they initially identified 2749 papers, and after various academic
quality checks, they catalogued 2225 papers. From this base, they identified 420 papers relating to school-
aged children and 121 related to teachers (in-service or pre-service). From these 541 papers, only 66 were
identified as having pedagogy as a major theme.

16
 CHAPTER TWO

An additional area of concern is that literature reviews from Lye and Koh (2014), Sentance and Selby
(2015), Waite (2017), and Crick (2017) show concern that most of the existing research is not conducted
in a classroom context, and lacks methodological rigour in terms of quality and transferability; typically
with regard to sampling, sample size, statistical significance and claimed impacts. Sentance and Selby
(2015) go further in commenting that these concerns are not restricted to school/teacher research, but
are indicative of the whole field.

2.2.5 Related Research into High-school Computing, E-learning and

Pedagogy
As outlined in the previous section, in 2015 Sentance and Selby reported a classification of research into
computer science education in schools from 2005-2014 (Sentance & Selby 2015). Dr Sentance provided
access to a version of the research classification database for analysis in this study, this analysis is
summarised in Table 2.

Filter NUM of Comment

Papers

None 2,522 All papers included in the research classification

database.

major theme = pedagogy 58 None of the 58 papers related to pedagogy for

computing education in schools give a holistic view of
then
pedagogy. Instead, their focus is on specific aspects of
focus = school age students or
pedagogy, such as: teaching programming; teaching
teachers
databases, programming with robots; problem-based
learning, visualisation, computational thinking,
unplugged, handling failure, algorithms, games,
collaborative learning, etc.

major theme = pedagogy 432 Two papers have a minimal semblance to the focus of
this research in that their focus is on the identification
and use of pedagogical patterns for the teaching of
computer science.

major theme = pedagogy 8 None of the papers give a holistic view of pedagogy for
computer science e-learning. Instead, their focus was
then
on specific aspects of pedagogy, such as: video casting,
minor theme = e-learning
visualisation, using videos in programming, online
forums as a collaboration tool, use of tablets in the

17
 CHAPTER TWO

Filter NUM of Comment

Papers
classroom, classroom management, and collaborative
learning.

Major theme = e-learning 89 The overriding focus of the identified research is on

specific aspects of e-learning. Although, one paper
focuses on pedagogical patterns, and one focuses on
pedagogy within e-learning; however, the latter
focuses specifically on pedagogy for virtual classrooms.

major theme = e-learning 4 The four identified papers focus on: learning objectives;
e-portfolios; algorithms; and outreach.
then

focus = school age students or

teachers

major theme = e-learning 4 The four identified papers focus on: virtual classrooms;
collaborative learning; and programming e-books.
then

minor theme = pedagogy

Table 2: Further analysis of Sentance and Selby's classification of research into computer science education in
schools from 2005-2014

As Table 2 indicates, there is a severe scarcity of research into computing pedagogy, which is further
diminished when considering a school context. This then becomes almost non-existent when considering
computing e-learning and schools, or computing e-learning and pedagogy.

The pedagogy literature review (Waite 2017) commissioned by the Royal Society as an addendum to their
2017 report (Royal Society 2017a) concentrates on research in the past 10 years and initially identified
over 700 papers, which were eventually reduced to 86 papers focused on computing pedagogy relevant
to schools (although not necessarily carried out in a classroom context). That literature review, and the
literature review carried out by Crick (2017), give support to the constructivist and social constructivist
principles espoused in this research. Notably, no research was discussed that focused on a comprehensive
and holistic pedagogy towards computing. Instead, the research was on specific aspects of pedagogy.
However, the underlying aspects of pedagogy that were discussed give strong support to the heuristics
outlined in this research, since they focus on: scaffolding; kinaesthetic learning; Use-Modify-Create
strategy; problem and project-based learning; using demonstrations and tutorials; social aspects of
learning; computational thinking; guided discovery; managing cognitive load; worked examples;
demonstrating and exemplifying thinking; collaborative learning (programming); and collaborative
learning through pupil (peer) support.

18
 CHAPTER TWO

The above analysis of recent literature reviews gives a strong indication that research into a
comprehensive and holistic set of pedagogical heuristics for teaching, or the design of e-learning software
for high-school computer science, is currently an under-researched area. However, the recent literature
reviews by Crick (2017) and Waite (2017) gives positive feedback that the heuristics considered in this
research study are aligned with pertinent research in this area.

2.2.6 International Comparison

Although the focus of this research is primarily the UK, several countries are engaged in introducing or re-
evaluating their computing curriculum in schools. An initial indicator of this can be derived from an
analysis of the location of recent research in this field. The classification of computing education research
(2005 to 2014) carried out by Sentance and Selby (2015) show that: USA; Israel; Germany; UK; Austria;
Slovakia; Switzerland; Italy; Lithuania; and Poland are active in this research area (refer to Figure 4).

Figure 4: Number of school papers by country of first author

(Sentance & Selby 2015, p.11)

In addition, there are various international comparisons (CAS 2011; Grover & Pea 2013; Snyder 2012;
Hazzan et al. 2008; Sturman & Sizmur 2011; Crick 2017) that show that the concerns and challenges
outlined in the UK are common to a number of countries. Grover & Pea (2013, p.40) give a succinct
summary: “… 21st-century literacy is gaining momentum globally. Israel has long boasted an exemplary
mandatory high school CS curriculum. Countries such as Russia, South Africa, New Zealand, and Australia
have already made room for CS in the K–12 curriculum. More recently, the United Kingdom has piloted
programs to teach computing to all schoolchildren following a bold 2012 policy charter from the Royal
Society.”

There are several national updates and discussions on the status of the computing curriculum in schools.
The US at a national level (Stephenson et al. 2005; Wilson et al. 2010; CSTA 2013; ACM et al. 2016), and
state level (Ryoo et al. 2013; Ericson et al. 2016; Guzdial et al. 2014), New Zealand (Bell et al. 2010;

19
 CHAPTER TWO

Murugesh et al. 2010; Bell 2014; Bell et al. 2012), Australia (Egger et al. 2012; Falkner & Vivian 2015),
Israel (Hazzan et al. 2008; Armoni & Gal-Ezer 2014; Gal-ezer & Stephenson 2014), Germany (Brinda et al.
2009; Hubwieser et al. 2011), France (Baron et al. 2014), Italy (Bellettini et al. 2014), Sweden (Rolandsson
& Skogh 2014), Belgium (Martens & Hofkens 2013), Russia (Khenner & Semakin 2014), and India (Raman
et al. 2015) are all in varying stages of similar initiatives to give greater prominence to the high-school
computer science curriculum. These countries range between those with a well-established computer
science curriculum in schools, to those in their infancy; each country inheriting their own individual set of
difficulties, but with a common thread of challenges in transitioning teachers to a new curriculum,
ensuring teachers have both subject and pedagogical knowledge to support their teaching, and the
relevant teaching resources to support student instruction. It therefore follows that the findings of this
research are broadly transferable to such initiatives in other nations.

2.3 E-learning Software

In view of the challenges faced in computing education outlined in section 2.2, one approach to support
teachers and high-school computing is the increased usage of e-learning software (technical resources).

There are varying definitions of e-learning; however, in the broadest sense, e-learning is electronic
learning, and as such utilises electronic technology to facilitate the learning process. Historically, the
communication medium for e-learning has ranged through television, videotape, CD-ROM, DVD, etc.
(Ozkan & Koseler 2009; Shee & Wang 2008). However, in recent years, it has become synonymous with
the distribution of course material and content through computer networked technology (i.e. the
internet) (Welsh et al. 2003). The implementation of e-learning can vary in complexity to include “a
number of integrated tools and services for teaching, learning, communicating, and managing learning
material” (Ardito et al. 2006), and citing the European Commission, Alptekin and Karsak (2011) emphasise
the improved quality of learning via multimedia technologies and services to support collaboration.

Within the context of this study, e-learning is regarded as software that delivers content and instructional
methods to facilitate learning, ideally in an engaging and interactive manner. Furthermore, in the context
of high-school computing, such software should cover a complete topic or non-trivial portion of
curriculum (e.g. algorithms, programming, data representation, networks, etc.). This differentiates e-
learning software from the multitude of elemental digital resources found online that can be used
individually to support a small fraction of the computing curriculum.

2.3.1 The Status of E-learning

Investment in, and the growth of e-learning usage, have been widely reported as two of the most rapidly
expanding areas of education and training (Gilbert et al. 2007; Anderson 2008; Jang 2009; Ozkan & Koseler
2009; Zaharias & Poylymenakou 2009; Jung 2011); furthermore, e-learning is touted as a new paradigm
for modern education (Sun et al. 2008). According to the 2017 State of the Industry Report (ATD Research

20
 CHAPTER TWO

2017) from the Association of Talent Development (ATD), e-learning accounted for 45% of training
delivery in 2016, of which the most popular type of technology-based learning was self-paced e-learning.

The picture in school environments, whilst positive, is a little more complex to interpret; various studies
report on national / government initiatives to promote e-learning and the broad general support for e-
learning in schools (Gilbert et al. 2007; Boulton 2008; Jang 2009; Liu et al. 2011; Pardamean & Suparyanto
2014).

The US Department of Education’s 2004 National Educational Technology Plan is cited (Livingstone 2012;
Pardamean & Suparyanto 2014) in relation to the widespread implementation of e-learning in K-12
education, such as: 15 US states providing some form of virtual schooling to supplement regular classes;
hundreds of thousands of students taking advantage of e-learning in 2004; and about 25% of all K-12
public schools offering some form of e-learning or virtual school instruction. Citing various sources, Kim
et al. (2014) report that: in the academic year 2007/8 in the US, over 1 million school students took online
courses; that in the following five years it was estimated that 5 million students would take online courses;
and that in 2011, all but one state had virtual schools. More recent reporting from the Evergreen
Education Group (2017) is not directly comparable, but do not seem to support the growth predictions
from Kim et al. (2014); instead, for 2016, the Evergreen Education Group reports: 24 US states with virtual
schools; 934,968 course enrolments; and 511,251 students taking online classes.

Focusing on the UK, Livingstone (2012) cites BECTA13 in reporting that in the academic year 2008/9, UK
schools spent some £880 million on ICT, and that almost half of all primary school pupils used digital
resources at least once a week; this falls to one in ten in secondary schools.

In 2006, the European Commission reported that most schools in Europe were in the early phase of ICT
adoption with patchy provision and use, and hence could not report significant improvements in learning
and teaching. Nevertheless, a small percentage of schools were able to effectively embed ICT into their
curriculum, thereby transforming teaching and learning across a wide range of subject areas. Based on
the latter, the ICT Impact Report (Balanskat et al. 2006, pp.3–4), directly recounts a number of quantitative
benefits of ICT adoption and e-learning, such as:

1. “ICT impacts positively on educational performance in primary schools (with varying effect per
subject)”;
2. “Use of ICT improves attainment levels of school children in English - as a home language - (above
all), in Science and in Design and Technology between ages 7 and 16, particularly in primary
schools”;
3. “Schools with higher levels of e-maturity demonstrate a more rapid increase in performance
scores than those with lower levels”; and

13
British Educational Communications and Technology Agency

21
 CHAPTER TWO

4. “Schools with good ICT resources achieve better results than those that are poorly equipped”.

Additionally, the report offers a number of qualitative findings, that are summarised below:

1. Pupils, teachers and parents consider that ICT has a positive impact on pupils’ learning;
2. Pupils’ subject-related performance and basic skills (calculation, reading and writing) improve
with ICT, according to teachers;
3. Teachers are becoming more and more convinced that the educational achievements of pupils
improve through the use of ICT;
4. Academically strong students benefit more from ICT use, but ICT also serves weak students; and
5. Teachers in Europe state that pupils are more motivated and attentive when computers and the
internet are used in class.

In their 2009 report: The impact of digital technology (Underwood 2009), BECTA reported a growing body
of national and international evidence demonstrating the positive impact of digital technologies on
learning outcomes. They reported improvements in attainment at Key Stage 1 and 2 due to ICT, and the
following improvements in secondary school (2009, p.3):

1. “The equivalent to a term’s additional progress in KS3 science”;

2. “An average gain in GCSE science equivalent to 52,484 students moving from grade D to C.
Improvements to the overall percentage of pupils 5+ A*- Cs at GCSE in the year after broadband
introduction”; and
3. “After controlling for KS3 results, the availability of a computer at home significantly positively
associated with Key Stage 4 test scores. That association amounted to around 14 GCSE points
(equivalent to 2 GCSE grades)”.

They also reported a wider outcome that classes with online learning (either completely online or
blended) on average yielded stronger learning outcomes than traditional face-to-face learning.

A meta-analysis of online learning studies commissioned by the US Department of Education (Means et

al. 2009), reported slightly more cautious benefits as compared to the European Commission and BECTA
reports. Analysing studies from 1996 through to July 2008, Means et al. identified that there was a small
number of published studies relating to K–12 students; therefore, caution was advised in generalising to
the K–12 population. They reported that:

1. “Students who took all or part of their class online performed better, on average, than those
taking the same course through traditional face-to-face instruction.” (Means et al. 2009, p.xiv)
2. “Instruction combining online and face-to-face elements had a larger advantage relative to purely
face-to-face instruction than did purely online instruction.” (Means et al. 2009, p.xv)

However, again Means et al. cautioned that the observed advantage of online learning may have been a
product of the treatment conditions and not the instructional delivery. Likewise, the increased effect size

22
 CHAPTER TWO

of blended learning may have also been caused by aspects of the treatment and control conditions, such
as: curriculum material; aspects of pedagogy; and learning time.

Other research has been even more cautious in relation to e-learning; in the early days it was noted that
consumers were blinded by the possibility of technological innovation into assuming e-learning would
implicitly be educationally effective (Reeves 1994). In fact, at that time, e-learning material was frequently
of poor to moderate quality, often an electronic transposition of existing material, and often overly
focused on aesthetics to the detriment of usability and pedagogy (Reeves 1994; Wesson & Cowley 2003;
Dringus 1995; Ardito et al. 2004). Lanzilotti and Ardito (2006) and Boulos et al. (2006) comment that
despite the large number of available e-learning systems, there were concerns over quality. Although
there are a variety of contributing factors, Rovai (2002), Zaharias and Poylymenakou (Zaharias &
Poylymenakou 2009), Boyle et al. (2010), and Alptekin and Karsak (2011) all show concern over the high
dropout rates in distance learning, and postulate that a major contributor to this is the poor quality of e-
learning applications (Zaharias & Poylymenakou 2009).

E-learning research focused in schools mirrors many of the aforementioned concerns; Underwood (2009)
and Boulton (2008) report concerns over poor quality outdated e-learning material that was not aligned
with the expectations of the pupils of the day. Additionally, they linked the effectiveness of e-learning
with the alignment of technology affordances and pedagogy.

The students level of e-maturity is a recurring theme with school-focused research, showing that students
need to be trained in the skills to effectively participate in e-learning, and supported throughout
(Underwood 2009; Boulton 2008; Jang 2009; Sendall et al. 2008).

Boulton’s (2008) study on GCSE ICT e-learning showed concerns over poor quality e-learning material, and
a lack of preparation and support from the school, leading to students lacking the skills and motivation to
participate in e-learning. Ultimately, these factors led to low satisfaction from the students, and a high
dropout rate.

Lastly, Condie and Munro (2007), Livingstone (2012), and Kim et al. (2014) take a more critical view of the
evidence towards improved pupil attainment through e-learning/ICT, commenting that it is inconsistent
(and unconvincing). Condie and Munro (2007, p.4) remark that “it does appear that, in some contexts,
with some pupils, in some disciplines, attainment has been enhanced.” However, the overriding concern
is that much of the evidence is taken from small-scale studies lacking the rigour and scale by which
generalisations can be drawn.

2.3.2 Quality of E-learning Software and the Need for E-learning

Heuristics
Despite the aforementioned caution and scepticism, there remains a strong support for e-learning, but it
is tempered with a more realistic view. Although e-learning software has become mainstream, a concern
still remains that what is delivered can often fall short (Alonso et al. 2005; Alonso et al. 2009; Chan &

23
 CHAPTER TWO

Robbins 2006). Unsurprisingly, content quality is the most important concern (Sun et al. 2008). Alonso et
al. (2005, p.218) protest that “there is a serious dysfunction between the profusion of technological
features that are put forward and the shortage or non-existence of teaching principles for e-learning.”
Chan and Robbins (2006) advise that e-learning software “require(s) an understanding of educational
pedagogy and instructional design and demand(s) a considerable amount of planning and
preparation”(p.496). Echoing this sentiment, Hadjerrouit (2010) argues there is often a lack of
“pedagogical usability” in existing e-learning software, and a lack of alignment with education needs and
standards.

More recent studies in New Zealand (Murugesh et al. 2010), Morocco (Abderrahim et al. 2013), and
Australia (Falkner & Vivian 2015) evaluated e-learning resources used within their respective high-school
curricula; they reported evaluation results ranging from broadly positive to concerns over resources of
low or medium quality. In the UK, the Royal Society (2017a) has called for greater support to be given to
teachers in the selection and usage of suitable quality computing resources, and in their pedagogy
literature review recommended, as a high priority, an audit of the pedagogical foundations of e-learning
resources currently recommended in the high-school computing curriculum (Waite 2017).

There are several factors that contribute to the quality, and educational effectiveness, of e-learning.
Quality is defined by the Oxford English Dictionary as “The standard of something as measured against
other things of a similar kind; the degree of excellence of something.” Jung (2011) argues that the majority
of studies on e-learning quality are from the perspective of e-learning providers, assessors, governments
and professionals, but do not focus on learners or learning. This is peculiar since quality is also defined by
various authors as “fitness for purpose” (Campbell & Rozsnyai 2002; Harvey & Green 1993; Vlăsceanu et
al. 2004; Woodhouse 2004) and the purpose of e-learning software is learning. Therefore, in the context
of this research on e-learning software, quality focuses on the standard and degree of excellence of the
pedagogy that underpins the learning process. There is a wide body of research that contends that
pedagogy is one of the most critical underpinning factors in e-learning, and that, as already outlined, this
factor is often lacking (Govindasamy 2001; Wesson & Cowley 2003; Ardito et al. 2004; Zenios et al. 2004;
Lanzilotti & Ardito 2006; Ardito et al. 2006; Sun et al. 2008; Anderson 2008; Hsu et al. 2009; Ozkan &
Koseler 2009; Underwood 2009; Zaharias & Poylymenakou 2009; Alptekin & Karsak 2011; Ofsted 2011;
Zaharias & Koutsabasis 2012; Abderrahim et al. 2013; Kim et al. 2014; Falkner & Vivian 2015; Waite 2017;
Alonso et al. 2005).

There are a variety of texts on e-learning instructional design; the research of this thesis supplements
existing literature by providing an accessible set of heuristics for e-learning pedagogy. Heuristics are
specific rules-of-thumb or arguments derived from experience. They are consolidated and condensed to
provide the essence of e-learning pedagogy to the teacher audience. Existing e-learning heuristics are
extended to focus more tightly and in-depth on pedagogy, due to the aforementioned concerns on e-
learning quality and, in particular, the specific concerns on the underpinning pedagogy of some e-learning
resources. In addition, a more in-depth focus on pedagogy is necessary since originally, research on e-

24
 CHAPTER TWO

learning heuristics focused almost entirely on usability with little reflection of the complexities of the
learning process (Dringus 1995; Parlangeli et al. 1999; Wesson & Cowley 2003; Dringus & Cohen 2005;
Mehlenbacher et al. 2005; Hsu et al. 2009; Zaharias & Poylymenakou 2009). However, research has
matured, and it is now accepted that the value of e-learning software directly relates to its pedagogical
value. This is reflected in a body of research that builds upon Nielsen’s usability heuristics, and
supplements it with pedagogical usability heuristics (Reeves et al. 2002; Squires & Preece 1999). However,
there still seems to be a sense that pedagogy is an adjunct to usability (or other factors) instead of being
the primary focus of the heuristics. Nokelainen (2006) concludes that pedagogical aspects for designing
or using digital learning material are much less frequently studied than technical ones, and at that time,
the studies that existed were theoretical in nature and had not undergone a process of empirical testing.
Gilbert et al. (2007) contend that there are a number of implicit and explicit frameworks designed to
inform e-learning practice; however, the components of these frameworks cover a variety of contributing
factors, of which pedagogy is treated as one of many factors. Their discussion of pedagogy also reflects
that researchers have focused on specific pedagogical aspects of e-learning, rather than a holistic
pedagogy for e-learning; focus areas include: online communities, e-assessment, judging text on screen,
asynchronous collaborative learning, e-learning dialogues, social dimensions of online learning, etc.
Dabbagh (2005) offers the closest conceptual alignment with the objectives of this study, by offering
pedagogical models for e-learning that are presented as a theory based framework for e-learning design.
Dabbagh’s study does not specify the framework to the level of detail specified in this research and is
limited to a theoretical perspective; nonetheless, it holistically covers a number of pedagogical areas also
covered in this study, such as: promoting authentic learning; supporting role-playing; promoting
articulation and reflection; promoting collaboration and social negotiation; supporting multiple
perspectives; supporting modelling and explaining; and providing scaffolding.

The literature review carried out in this study reflects that despite the acknowledgement that pedagogy
is critical for e-learning design, there is a scarcity of e-learning heuristics that give adequate focus to
pedagogy. This is further supported by the fact that the overwhelming majority of e-learning heuristics
are dual purpose; they are intended for design and also for the evaluation of e-learning software. The
review of e-learning evaluation heuristics in section 2.3.3 also clearly shows the bias towards usability at
the expense of pedagogy. This scarcity of pedagogical heuristics is further evidenced by the review of
literature presented in section 2.2.4.1; which shows a lack of pedagogy defined for high-school computing.
Considering these factors, at the time of writing (January 2018) this study has not identified an equivalent
holistic set of heuristics focused on e-learning software for high-school computing.

2.3.3 E-learning Evaluation Protocols

As is the norm, the pedagogical heuristics and criteria defined in this study are dual purpose: to guide the
pedagogical design of e-learning software, and to serve as the basis of a protocol to guide the evaluation
of such software (Ardito et al. 2006; Masip et al. 2011).

25
 CHAPTER TWO

Following on from the previous section, the bias towards usability, often neglecting pedagogy, is further
evident in the development of e-learning evaluation protocols. The need for usability in software is
indisputable, and springing from this need, the evaluation of educational software has largely been
focused on usability. Lanzilotti and Ardito (2006) acknowledge this point and go one step further in
echoing the widely held sentiment that usability in e-learning software is particularly important: “If an e-
learning system is not usable, the learner spend more time learning how to use the software rather than
learning the contents”(2006, p.42). This study unequivocally agrees with the need for usability in e-
learning software and proposes it as a mandatory prerequisite for learning, but usability is not enough.
Analogous to Herzberg's (1987) motivation-hygiene theory, usability can be viewed as a hygiene factor
whose absence will cause dissatisfaction and hinder learning, but in itself it does not stimulate learning.
Pedagogy is the motivational factor that encourages learning; therefore, e-learning software must be
effective in meeting the intended pedagogical objectives.

Squires and Preece (1996; 1999), Parlangeli et al. (1999), Zaharias et al. (2002), Wong et al.(2003),
Lanzilotti and Ardito (2006), Zaharias and Poylymenakou (Zaharias & Poylymenakou 2009), Zaharias and
Koutsabasis (2012) and Abderrahim (2013) all concur that the evaluation of e-learning software deserve
special attention and that such an evaluation must integrate the assessment of the educational quality
of the software. Moreover, Lanzilotti and Ardito (2006), citing Parlangeli et al. (1999), Squires and Preece
(1999) and Wong et al.(2003), argue that typically the e-learning evaluation criteria are stated vaguely,
leaving the evaluation measurement open to subjective interpretation. Grützner et al. propose that e-
learning software development must consider “content and instructional issues; management; technical
and graphical issues…” (2004, p.946). Considering Grützner et al.’s broad classification, this study
considers that a foundation of usability must already exist to resolve potential technical and graphical
issues. This frees the proposed evaluation protocol to focus more specifically on educational content and
instructional design (pedagogy). According to Squires and Preece (1996) the separation of usability
evaluation and educational evaluation, as outlined in this research, is somewhat artificial since “the
design of an interface supports certain styles of interaction which may or may not support the intended
learning tasks”(1996, p.16). There is merit to this argument; however, considering the relative glut of
research on usability heuristics and evaluation, and the lack of focus on educational value and pedagogy,
the research presented in this thesis has purposely restricted its focus to the latter.

Reeves (1994) was one of the original advocates for comparative evaluations, and the development of
criteria for evaluating computer-based education. More recently, this need is reflected in a school context,
by the evaluation of e-learning resources carried out by Murugesh et al. (2010), Abderrahim et al. (2013),
and Falkner and Vivian (2015). These e-learning evaluations highlight the need for a rigorous e-learning
evaluation protocol that gives focus to pedagogy.

Murugesh et al. (2010), carried out an evaluation of online and offline resources to support New Zealand’s
high-school transition towards a more computer science focus. They evaluated the resources in

26
 CHAPTER TWO

accordance with the learning objectives of the three levels of the curriculum; however, it is important to
note that there is no mention of any pedagogical evaluation.

In contrast, Abderrahim et al. (2013) express a focus on pedagogy in their evaluation of commercially
produced digital education resources for the Moroccan education system. They formed their evaluation
as a 20-question checklist that gives focus to pedagogy and some focus to content appropriateness, but
does not offer an encompassing protocol to guide the evaluation. The evaluation process is intended to
provide quantifiable results, but the checklist is defined at a very high-level, and without supporting
evaluation criteria, so that any results are open to interpretation. Examples of these checklist questions
include (Abderrahim et al. 2013, p.32)14:

1. Is the information presented relevant?

2. Structuring the resource she promotes its use in an pedagogical context?
3. Does the product include stimuli likely to promote learning?
4. Does the tool present activities creating interactions between learners?
5. Is there a match between the audience, content and objectives?

Falkner and Vivian (2015) were commissioned by the Australian government to undertake an evaluation
of the educational quality of online computer science resources, and to ascertain their suitability for K-12
classroom learning and teaching. Their Quality Assessment Framework was developed in consultation
with the Australian Department of Education and Training and gave a partial focus on pedagogy, including
such aspects as: authenticity, accuracy, currency, consideration of prior learning, scaffolding and learning
styles, provision of open-ended tasks, reflection, community, and collaboration. In addition, the
evaluation was intentionally targeted towards computer science, and mapped to the Australian
curriculum. However, ultimately, the evaluation was again marred by several limitations. The evaluation
was grouped into four principal areas, with some description of underlying evaluation elements, but was
not specified at a level of detail to support a clearly objective evaluation. Furthermore, a rigorous protocol
to guide the evaluation steps was not presented, there was limited coverage of pedagogical principles and
the measurement of evaluation elements was coarsely defined as: No, Somewhat, and Yes.

These recent studies reflect a need for the evaluation of digital education resources; but the inherent
limitations of each of the evaluations also reflect the need for a rigorously defined evaluation protocol
focused on pedagogy. Considering the UK’s (Waite 2017) recent call to evaluate the pedagogical
foundation of their high-school digital education resources, this need is likely to continue.

The literature review outlined in this section, acts as a foundation and is elaborated in Chapter Five, which
discusses the synthesis of the e-learning evaluation protocol.

14
Grammar is intentionally not corrected on the checklist items.

27
 CHAPTER TWO

2.4 Learning Theories and Other Inputs into the Pedagogy

The e-learning pedagogy developed in this study, aims to provide a set of heuristics and guiding criteria,
for the design and evaluation of e-learning software (including the supporting use of a collaborative
learning environment), for use in the instruction of high-school computer science. It is therefore natural
that these heuristics are based on research and theories of how we learn best.

2.4.1 What are Learning Theories?

There are a variety of definitions for what learning is, and what learning theories are. According to the
Merriam-Webster (2017) dictionary, a theory is “a plausible or scientifically acceptable general principle
or body of principles offered to explain phenomena.” In this case, the phenomena we seek to explain is
learning. Likewise, with reference to Table 3, Alan Pritchard (2009, p.2) gives a selection of definitions for
learning, that each touch upon the different aspects of learning.

A change in behaviour as a result of experience or practice

The acquisition of knowledge

Knowledge gained through study

To gain knowledge of, or skill in, something through study, teaching, instruction or experience

The process of gaining knowledge

A process by which behaviour is changed, shaped or controlled

The individual process of constructing understanding based on experience from a wide range of sources
Table 3: Definitions of learning

A working definition for learning theories can therefore be formed as:

“A scientifically acceptable general principle or body of principles to explain how humans gain knowledge
of, or skill in, something through study, teaching, instruction or experience.”

2.4.2 Choice of Learning Theories and Inputs into the Pedagogy

The seemingly simple and instinctive process of learning is actually very complex. There is a significant
body of research into learning theories, e-learning and STEM education. However, this significant body of
knowledge is somewhat overwhelming; there are complementary and competing learning theories, and
differing research into the best implementation of these theories in technology (Illeris, 2009). Although
it has a more commercial perspective, the website https://www.learning-theories.com/ is adequate in
giving a conservative indicator of the size of the learning theory space, it lists over 90 distinct learning
theories. An extract of some of the more well-known theories is listed in Table 4.

28
 CHAPTER TWO

ADDIE Model of Instructional Discovery Learning (Bruner) Model of Hierarchical

Design Complexity
Distributed Cognition (Hutchins)
ARCS Model of Motivational Montessori Method
Elaboration Theory (Reigeluth)
Design (Keller) (Montessori)
Emotional Intelligence
Bloom’s Taxonomy (Bloom) Multimodality (Kress)
(Goleman)
Classical Conditioning (Pavlov) Multiple Intelligences Theory
Experiential Learning (Kolb)
(Gardner)
Cognitive Apprenticeship
Expertise Theory (Ericsson,
(Collins et al.) Online Collaborative Learning
Gladwell)
(Harasim)
Cognitive Dissonance
Flow (Csikszentmihalyi)
(Festinger) Operant Conditioning (Skinner)
Game Reward Systems
Cognitive Load Theory (Sweller) Psychological Behaviorism
Gamification in Education (Staats)
Cognitive Theory of Multimedia
Learning (Mayer) Gestalt Theory (von Ehrenfels) Situated Learning Theory (Lave)

Communities of Practice (Lave Grit (Duckworth, Matthews, Social Constructivism

and Wenger) Kelly, Peterson) (Vygotsky)

Connectivism (Siemens, Intrinsically motivating Social Network Analysis (Scott,

Downes) instruction (Malone) Prell)

Constructivism (Piaget) Maslow’s Hierarchy of Needs Stage Theory of Cognitive

(Maslow) Development (Piaget)
Table 4: Indicative listing of learning theories

Although not a learning theory, the work of Knud Illeris (2003; 2009), in providing a framework in which
learning theories can be contextualised, is valuable. Illeris posits that learning is based on two processes:
an external interaction process and an internal acquisition process. These are then modelled across the
dimensions of content, incentive and environment (refer to Figure 5). This framework is necessary since
many learning theories deal with only one of these processes, and therefore give an incomplete picture
of how learning occurs.

Figure 5: The fundamental processes of learning

(Illeris 2009, p.9)

29
 CHAPTER TWO

The first learning process is an external “Interaction” process between the learner and their social, cultural
or material environment. This interaction with the external environment provides the initial stimulus for
learning and allows the learner to integrate into communities and society in general. The interaction
process involves action, communication and cooperation, and takes varying forms such as perception,
transmission, experience, imitation, activity and participation.

The external stimulus then feeds into an internal psychological process of elaboration and acquisition.
This internal process focuses on managing the learning content and maintaining the incentive (motivation)
to devote mental energy to learn. The content dimension focuses on identifying meaning and building
understanding from the incoming information in such a way that it increases the learner’s overall abilities.
However, such processes require significant cognitive effort, so there is a need for an incentive dimension
that provides and directs the mental energy to support learning. This dimension aims for mental
equilibrium and involves the learner’s motivation, volition and emotions.

Whilst academics debate the relative merits and weaknesses of certain learning theories, or the
superiority of other theories (Bell 2011; Kop & Hill 2008; Siemens 2006; Verhagen 2006; Vogel-Walcutt et
al. 2011; Vrasidas 2000), this study aligns with Illeris (2009) and Anderson (2008) in asserting that the
various learning theories are not mutually exclusive. Aspects of various learning theories can be
synthesised to create a pedagogy that supports the whole learning process. As a theoretical foundation,
the pedagogy proposed in this thesis synthesises the following well established learning theories and one
learning style theory.

1. Constructivism,
2. Social Constructivism,
3. Connectivism,
4. VARK Learning Styles,
5. Cognitive Load Theory, and
6. ARCS Model of Motivational Design

As previously mentioned, these theories were selected primarily so that their synthesis can provide
comprehensive coverage across the three dimensions of content, incentive and environment.
Additionally, their selection was due to their maturity, and the availability of research that discusses the
theory, its effective implementation, and offers evidence of the theory’s positive effect on learning and
motivation. Furthermore, consideration was given to contemporary theories (e.g. connectivism), which
give more focus to technology and our current digital society. Finally, in keeping, with a focus on e-
learning, technology, and high-school computer science, the pedagogy was also informed by research in
the areas of:

1. e-learning best practice,

2. collaborative learning,
3. gamification, and

30
 CHAPTER TWO

4. computational thinking.

The following subsections explore at a high-level these learning theories and the additional pedagogical
inputs. This is then followed by Chapter Four, which gives further information on how these theories are
synthesised into a holistic set of heuristics.

2.4.3 Constructivism
Traditional objectivist (behaviourist) approaches contend that there is one true and correct reality, which
can be taught to a learner. In contrast, constructivist approaches recognise a real-world that sets limits
on our experiences, but proposes that there is no uniformly perceived single reality; in fact, each person’s
perception of reality is a mental construct founded on interpretation of their interactions with the world
(Brooks & Brooks 1999; Jonassen 1999; Vrasidas 2000; Pritchard 2009). An individual’s reality is therefore
based on their existing experience and understanding, which is in turn used to make sense of their current
perception of events. The mental constructs that organise and categorise our skills, and knowledge and
understanding of the world are called schema (Brooks & Brooks 1999).

In this light, constructivists view learning as the result of active mental construction, where new
information is built into an individual’s current mental models (schema). Piaget (1969), one of the
founding fathers of constructivism, proposes that external experience can reinforce existing mental
models or contradict them, leading to the following:

• Assimilation – the process by which new information reinforces an existing schema, and the
schema is augmented with new information.
• Accommodation – the process by which existing schemas must be altered to cope with new
experiences that contradict the existing mental model.
• Equilibrium – the process of arriving at a stable state where there is no longer conflict between
new and existing knowledge.

Constructivist principles are claimed to offer several learning benefits (Hattie 2009), including:

• Better and deeper understanding since it emphasises authentic and active learning (Kolb 1984;
Brooks & Brooks 1999; Jonassen 1999; Fadel & Lemke 2008; Melero et al. 2012; Hattie 2009);
• An increase in motivation since students feel more in control, and are closer to their perceived
reality (Kolb 1984; Brooks & Brooks 1999; Hidi & Harackiewicz 2000; Palmer 2005; Bertacchini et
al. 2012; Melero et al. 2012);
• An increase in social skills since constructivism focuses heavily on interacting, perceiving and
interpreting the world, often through dialogue (Brooks & Brooks 1999; Jonassen 1999; Karagiorgi
& Symeou 2005; Melero et al. 2012); and
• A focus on the big picture, promoting problem-solving and higher-order thinking (Brooks &
Brooks 1999; Jonassen 1999; Jonassen & Hung 2008; Hadjerrouit 2005; Liu et al. 2011).

31
 CHAPTER TWO

However, there are also counter-arguments that counsel a measure of caution (Perkins 1991; Perkins
1999; Brooks & Brooks 1999; Hadjerrouit 2005; Karagiorgi & Symeou 2005; Siemens 2008; Kirschner et al.
2006; Vrasidas 2000; Vogel-Walcutt et al. 2011; So 2002), and raise concern over the complexity of
implementing a curriculum using constructivist approaches, and the challenge for teachers to transition
from instructivist to constructivist approaches. There are warnings over the learner’s ability to construct
their own knowledge, and that the perceived lack of guided instruction in constructivist approaches can
allow learners to create their own “private universe” to explain complex phenomena. This in turn calls
into question the accountability that what students learn is aligned with expected learning outcomes.
Additionally, there are significant concerns over the additional cognitive load expended by students within
constructivist approaches (Vogel-Walcutt et al. 2011). In a comparative analysis of objectivist and
constructivist approaches, Vrasidas (2000) acknowledges that some learners may find constructivist
approaches intimidating and complex, and in some learning contexts, didactic instruction may be more
appropriate. Such concerns are to some extent valid if considering extreme constructivist positions, in
which students have almost unlimited discretion to structure their own learning. It is for this reason that
the proposed pedagogy takes a moderate constructivist strategy (Vrasidas 2000; Karagiorgi & Symeou
2005; Alonso et al. 2009) in which constructivist values are translated into more concrete instructional
design principles, or in the case of this study, into specific e-learning heuristics.

It is important to note that in the US, the national teacher associations have endorsed constructivist lesson
design and instructional practices, and that constructivist approaches underlie the US National Research
Council’s proposal that active learning is adopted in schools (Brooks & Brooks 1999; Crick 2017).
Furthermore, a recent literature review commissioned by the Royal Society to summarise what is known
about pedagogies for teaching computing in schools, recognises that: constructivist principles have had
a wide-ranging impact on learning theories and teaching methods; have been an underlying theme of
many education reform movements across the world; and have been proposed as a suitable pedagogy for
computer science, and in particular for computer programming (Crick 2017). This advocacy of
constructivist approaches is with the full recognition that there remains debate regarding the tension
between constructivist exploration and the controlled progression of the teaching of more complex
concepts (Waite 2017). It is with recognition of this tension and debate, and moderate constructivist
approaches, that these heuristics take a moderate line in which students should construct their own
knowledge but should also be given instructional guidance and support via the e-learning software and
their teachers.

2.4.4 Social Constructivism

As its name implies, social constructivism is closely related to constructivist principles; both theories are
based on knowledge being actively constructed. However, constructivism focuses on knowledge
construction “in the head of the learner” (intra-psychological), while social constructivism proposes that
knowledge is created through social interaction (inter-psychological), often with a more knowledgeable
other (Vrasidas 2000; Barbour & Rich 2007; Pritchard 2009). Although each theory has its own perspective

32
 CHAPTER TWO

on exactly when and where knowledge is constructed, they are complementary (Cobb 1994). An
individual’s current knowledge forms the basis of their contribution to the dialogue. Then through this
dialogue, new shared knowledge is constructed and is then assimilated by the individual learners, which
in turn feeds into the next cycle of social interaction (Dillenbourg et al. 1995; Kop & Hill 2008).

Vygotsky (1978; 1986), one of the earliest proponents of social constructivism, suggested two key
components in learning: “language” and “scaffolding”. According to Kop and Hill (2008) Vygotsky
postulated that self-talk (language) allowed children to work through complex problems by externalising
them as a form of self-guidance and self-direction. This in turn forms the basis of learners’ externalising
problems through language and social interaction. Language becomes the form for communicating ones
existing knowledge and entering into dialogue with others to solve a problem.

The second critical component within social constructivism and Vygotsky’s work, is scaffolding; the
process by which supportive interactions between the learner and a more knowledgeable other enable
the student to achieve something beyond his or her existing capabilities. The aforementioned knowledge
or activity that is just beyond the learner's current independent efforts is in the learner’s Zone of Proximal
Development (ZPD) (Vygotsky 1978).

Working at the ZPD stretches the learner’s existing capabilities and encourages them to the next higher
level of learning, which is postulated by Vygotsky (1978) to improve students developmentally15. The focus
on social interaction promotes higher order thinking by encouraging learners to develop, evaluate, and
appreciate multiple perspectives on an issue. Furthermore, the underlying principles and collaborative
nature of social constructivism also leads to greater student engagement, and learning that is more
meaningful.

2.4.5 Collaborative Learning

Closely aligned with social constructivism is collaborative learning. There is a significant body of research
that discusses and broadly supports collaborative learning (Johnson et al. 1984; Roger & Johnson 1988;
Jonassen 1999; Vrasidas 2000; Zenios et al. 2004; Nuutila et al. 2005; Palmer 2005; Black & Wiliam 2009;
Hattie 2009); this in turn has been extended into the realm of technology-assisted collaborative learning
(Lebow 1993; Squires & Preece 1999; Lou et al. 2001; Mayes & de Freitas 2004; Karagiorgi & Symeou
2005; Anderson 2008; Sun et al. 2008; Tutty & Klein 2008; Nicholas & Wan 2009; Papanikolaou &
Boubouka 2010; Hadjerrouit 2010; Kang et al. 2010; Clark & Mayer 2011; Jenkins et al. 2012).
Furthermore, it is argued that technology-assisted collaborative learning is all the more pertinent in the
current knowledge society, in which people engage in teamwork that is information- and technology-rich,
and that this trend is likely to continue in the future (Johnson & Johnson 1996; Knight et al. 2006; Siemens
2008; Kang et al. 2010).

15
Vygotsky postulates a link and interchange between a student’s learning and their development (physical/mental
maturation).

33
 CHAPTER TWO

Historically, it has been argued that the focus of traditional schooling on individual student abilities and
knowledge actually inhibits effective learning by creating barriers to social learning (Scardamalia &
Bereiter 1996), but in recent years there has been an increased willingness to consider collaborative
learning in a school context (Dabbagh 2005; Knight et al. 2006; Tseng & Tsai 2007; Nicholas & Wan 2009;
Slavin 2011; Bertacchini et al. 2012; Egger et al. 2012; Jenkins et al. 2012; Ryoo et al. 2013; Crick 2017;
Waite 2017). In particular, collaborative learning has garnered research support in computer science
education (Alonso et al. 2005; Nuutila et al. 2005; Beck & Chizhik 2008; Papanikolaou & Boubouka 2010;
Egger et al. 2012; Jenkins et al. 2012; Melero et al. 2012; Ryoo et al. 2013; Waite 2017).

It is proposed that learning most naturally occurs not in isolation, but by students working together.
Discussion allows them to share information, summarise points, verify and test their knowledge, and
debate their opinions. It helps the student understand the views which they disagree with, and consider
multiple perspectives when solving problems (Cunningham 1991; Lebow 1993; Dabbagh 2005). In
addition, by focusing groups of students on the common objective of solving a problem, it gives them a
common impetus to stimulate their learning and through the discussion, debate, contradiction and
construction necessary in solving the problem, it arguably helps them reach their full potential (Roger &
Johnson 1988; Dillenbourg et al. 1995; Johnson & Johnson 1996; Jonassen 1999; Vrasidas 2000; Nuutila
et al. 2005; Williams et al. 2008).

In the computer science field, the terms “cooperation” and “collaboration” are often used to indicate a
subtle difference; cooperation involves students’ dividing and assigning tasks within the group to solve
the problem, whereas collaboration involves a joint group endeavour to solve the problem (Dillenbourg
et al. 1995). This pedagogy uses the term collaborative to cover both cooperative and collaborative
learning; however, it should be noted that the typical focus is on collaborative learning (as defined
previously).

Originally, Moore (1989) proposed three types of interaction in distance learning: student-teacher,
student-student, student-content. This was followed by Hillman et al. (1994) who proposed a student-
interface interaction. More recently, Dron (2007) proposed four additional types of interaction: group-
content, group-group, learner-group, and teacher-group.

In relation to collaborative learning and social constructivism, this pedagogy focuses on the following
interaction types: student-teacher, student-student, student-content, group-group, learner-group, and
teacher-group. The overriding objective is that the collaborative learning environment (CLE) software will
give learners access to shared information and the tools for learners to work collaboratively and to
construct shared knowledge.

According to Slavin “there is a fair consensus among researchers about the positive effects of cooperative
learning on student achievement” (2011, p.345). However, not all research findings place the same value
on collaborative/cooperative learning, the best circumstances to use it, and the most effective
collaborative practices (Lou et al. 2001; Tutty & Klein 2008; Kirschner et al. 2009; Slavin 2011; Clark &
Mayer 2011). Although, when used with appropriate preparation and support, and with a clear

34
 CHAPTER TWO

instructional goal, it can lead to educational benefits, such as: improved student achievement; improved
long-term retention and higher-order thinking; and increased student motivation. Furthermore, according
to Kirschner et al. (2009), where task/learning complexity is high, dividing the processing of information
across students gives further benefit since it allows information to be distributed across a larger reservoir
of cognitive capacity. But the inverse must also be considered, where the cognitive load exerted by the
task/learning is low, the recombination and coordination inherent in collaborative learning cause an
unnecessary overhead.

2.4.6 Connectivism
One of the most influential proponents of connectivism, George Siemens (2004; 2006; 2008), proposes
that connectivism is a learning theory for the digital age, and positions it as a successor to existing
theories. Siemens(2006) and Downes (2006) contend that existing learning theories are no longer
adequate in the face of our improved understanding of learning, the increasing pace of knowledge growth,
the ubiquity of technology, the unmet expectations of recent generations (Generation Y, Millennials, etc.),
and the great complexification of knowledge.

Connectivism posits that learning can occur outside of people and that in fact learning is the network;
citing Siemens and Downes forum responses, Kop and Hill (2008) summarise that the creation of
connections between information nodes signifies knowledge, and that the ability to recognise patterns
within complex networks, and to traverse these information nodes, comprises learning.

Despite Siemens’ protestation that constructivism and social constructivism do not consider learning
outside of people and that they are outdated, he sees a small measure of alignment with constructivism
in that constructivist “classrooms which emulate the “fuzziness” of this learning will be more effective in
preparing learners for life-long learning” (Siemens 2004, p.2). In addition, the closer link between
connectivism and social constructivism is well-documented (McLoughlin & Lee 2008; Anderson 2008;
Wang et al. 2014; Bell 2010; Siemens 2004).

Connectivism seems to intuitively reflect the realities of our digitally saturated world and the necessity of
lifelong learning (typically, outside of formal institutions), and as such has a number of practitioners that
support and adopt connectivist approaches (Couros 2009; Downes 2005; Chiong & Jovanovic 2012;
McLoughlin & Lee 2008; Rochefort & Richmond 2011; Wang et al. 2014; Bessenyei 2008). Connectivists
argue that the focus of existing learning theories does not satisfactorily reflect learning in today's
interconnected world, and that there is an educational paradigm shift away from students being taught
by teachers, typically in a classroom context. This paradigm shift is rooted in the new opportunities and
challenges brought about by the exponential growth and complexity of information available on the
internet, new possibilities for people to communicate on global networks, and the ability to aggregate
different information streams.

Balancing popular support, there is academic debate on whether connectivism has the rigour to be a true
learning theory, whether it is supported by an empirical body of evidence, and whether it replaces more

35
 CHAPTER TWO

established theories (Bell 2011; Kop & Hill 2008; Siemens 2006; Verhagen 2006). However, what is clear
is that the contemplation of connectivism does add value in the development of new learning paradigms
(Bell 2011; Kop & Hill 2008) and in particular, in the development of this pedagogy.

In accordance with Siemens (2004) the main principles of connectivism are as follows:

1. Learning and knowledge rests in diversity of opinions.

2. Learning is a process of connecting specialised nodes or information sources.
3. Learning may reside in non-human appliances.
4. Capacity to know more is more critical than what is currently known.
5. Nurturing and maintaining connections is needed to facilitate continual learning.
6. Ability to see connections between fields, ideas, and concepts is a core skill.
7. Currency (accurate, up-to-date knowledge) is the intent of all connectivist learning activities.
8. Decision-making is itself a learning process (i.e. choosing what to learn and its meaning in the
current context).

Extreme connectivist positions as discussed by Kop and Hill (2008), Siemens (2008), and Bessenyei (2008)
related to Illich’s (1971) educational vision of de-schooling and ‘community webs’ are not considered in
the e-learning pedagogy. Instead, the previous eight principles are considered as the basis of the heuristics
outlined in section 4.7.4.

2.4.7 VARK Learning Styles

Learning styles are a relatively new set of theories that propose that each of us have our own personal
preferences on how we learn. These personal preferences relate to the best way in which we accept
incoming information, process it and then demonstrate learning. Campbell et al. (1996) define learning
styles as a certain specified pattern of behaviour according to which an individual approaches the learning
experience. Likewise, Dunn et al. (1981) define learning styles as the way in which the individual takes in
new information and develops new skills. Writing in the book Contemporary Theories of Learning (2009),
Howard Gardner aligns with constructivist principles in advising that students are not an empty vessel to
be filled with knowledge. Instead, due to their biological and cultural backgrounds, personal histories, and
experiences, they possess different kinds of minds, with different strengths, interests, and modes of
processing information. This initially complicates the teaching process and can lead to ineffectual
teaching, but once considered and understood, it can actually lead to more productive teaching that
reaches more students on a deeper level.

Research from psychology and education spanning over four decades support the overall proposition of
learning styles. This research has in turn spawned a wide range of inventories which try to define and
assess the various learning style dimensions (e.g., Canfield & Lafferty, 1974; Dunn, Dunn, & Price, 1987;
Honey & Mumford, 1982; Kolb, 1984). In their widely-cited critical review of learning styles, Coffield et al.

36
 CHAPTER TWO

(2004) identify 71 models of learning styles, and expound that it is a growing body of research that is still
producing ideas and a proliferation of instruments.

The widespread and popular appeal of learning styles amongst educators in schools and higher education
stems from the simple recognition that not all students respond equally well to the same instructional
approaches, and that at face value, learning styles provide a tool for educators and their students to better
align on what works best (Dunn et al. 1981; Frank Coffield et al. 2004; Weiler 2005; Fleming & Baume
2006; Franklin 2006; Leite et al. 2010; Popescu 2010; Newton 2015). However, in spite of the appeal and
widespread use of learning styles, it remains a complex and often disputed area of education research,
split between those who:

1. acknowledge the research debate, but offer broad support to learning styles (Dunn et al. 1981;
Weiler 2005; Fleming & Baume 2006; McCartney et al. 2007; Illeris 2009; Othman & Amiruddin
2010; Popescu 2010);
2. offer a balanced critical analysis of shortcomings and applicable use of learning styles (Johnson
& Johnson 1996; Klein 2003; Frank. Coffield et al. 2004; Frank Coffield et al. 2004; Clark & Mayer
2011; Fleming 2012);
3. explore the complexity of the research area, in terms of varying research quality, multiple
inventories and their instrument validity, and find a relative lack of robust empirical evidence
towards learning benefits (Johnson & Johnson 1996; Frank Coffield et al. 2004; Fleming & Baume
2006; Kirschner et al. 2006; Leite et al. 2010; Popescu 2010; Clark & Mayer 2011; Fleming 2012);
and
4. offer perspective pieces that aggressively challenge the educational value and use of learning
styles, branding them as a ‘neuromyth’ (Franklin 2006; Newton 2015).

This pedagogy concentrates on one of the better known and well used learning style models; Neil
Fleming’s VARK model focuses on the preferred mode of perceiving information within a learning context.
In recent years, Fleming and Baume (2006) and Fleming (2012) have clarified that the VARK model does
not constitute a full learning styles model in that it focuses only on the students’ and educators’ preferred
modes of communication, and does not address a wider set of learning behaviours and preferences.

The VARK model is based on four perceptual modes: Visual (V), Aural (A), Read-Write (R) and Kinaesthetic
(K). According to Fleming and Baume (2006), Othman & Amiruddin (2010) and Fleming (2012), the VARK
Model is based on the original VAK sensory modalities. Fleming and Baume (2006, p.5) explain that “some
students had a distinct preference for the written word whilst others preferred symbolic information as in
maps, diagrams, and charts.” Since these two preferences are not always aligned in the same person, a
second 'visual' modality for read-write learners was introduced in VARK. The VARK modalities are outlined
below:

1. Visual (V) - Visual learners prefer graphical and symbolic ways of representing information.
2. Aural (A) - Auditory learners prefer to learn from listening.

37
 CHAPTER TWO

3. Read-Write (R) - Read-Write learners prefer to learn through information displayed as words.
4. Kinaesthetic (K) - Kinaesthetic learners prefer learning that connects them to experience and
reality.

According to Allen et al. (2011) the VARK model has three basic principles:

1. Each person can learn, but may do so differently, irrespective of their level or ability.
2. Student motivation increases when their learning preferences are accommodated.
3. It is better to present new material using the learner’s preferred mode of perception.

The students’ preferred learning style(s) are identified by a short multiple-choice questionnaire that
places them in several situations within their experience and asks them to specify their preferred
action(s); this in turn indicates their favoured modal preference(s) (V, A, R, K). For each question, the
respondent can select one or more options, or can even omit questions where they find no suitable
option.

As discussed earlier, the research on learning styles contains some debate on whether a focus on learning
styles leads to genuine educational benefits and, if such a focus is to be taken, what the best
implementation is. Hence, once the students’ modal preferences are identified, different educational
strategies can be employed. At a high-level, there are two broad approaches: adaptive or multi-modal.

In the adaptive (matching) approach, the curricula and pedagogy are tailored to the different modal
preferences of the students, thereby increasing their motivation and learning performance. However, the
complexity and effort involved in implementing such an approach should not be underestimated;
providing individualised curricula and pedagogy for several high school classes, managing the varying
performance, whilst still providing meaningful feedback is incredibly challenging. From the outset,
Fleming (1992, p.138) the author of VARK, acknowledges this challenge and advises “it is simply not
realistic to expect teachers to provide programs that accommodate the learning style diversity present in
their classes.” Coffield et al. (2004) also reassert the point that the large scale adoption of matching
instruction towards individual learning styles is unrealistic given the demands it would place on teachers
and trainers.

It is, though, posited that current and future technologies can greatly support this process by catering to
the different modal preferences (Zapalska & Brozik 2006; Condie & Munro 2007; Gardner 2009; Popescu
2010). Advanced e-learning software that uses data analytics to build student profiles can in turn use
artificial intelligence approaches to offer different entry points and individualised educational experiences
based on affective status, past performance and modal preferences (TSIRIGA & VIRVOU 2004; Virvou et
al. 2005; Brusilovsky & Millán 2007; Schiaffino et al. 2008). However, Coffield et al. again prescribe
caution:
“It should be noted that the potential of ICT to support individualised instruction has not been fully
evaluated. However, the key point is that individualised instruction is not likely to work if it means more
unsupported individual learning.” (Frank Coffield et al. 2004, p.133)

38
 CHAPTER TWO

It is argued that the biggest danger with the adaptation (matching) approach, and one of the largest
sources of contention in learning styles research, is the potential for pigeon-holing or stereotyping
learners, which ultimately damages their learning and development (Dunn et al. 1981; Klein 2003; Frank
Coffield et al. 2004; Franklin 2006; Condie & Munro 2007; Pritchard 2009; Fleming 2012; Newton 2015).
Prevailing research opinion is that learning styles should be considered indicative, not diagnostic, and
considered in relation to the learning context and content, not rigidly tailored to specific student learning
styles.

In order to counter such problems, the majority of learning style theorists (Dunn et al. 1981; Felder &
Silverman 1988; Fleming & Mills 1992; Honey & Mumford 2000; Klein 2003; Frank Coffield et al. 2004;
Fleming & Baume 2006; Zapalska & Brozik 2006; Pritchard 2009; Leite et al. 2010; Birch et al. 2011;
Fleming 2012) promote the idea that learners should develop a repertoire of styles, so that an awareness
of their own preferences and abilities should not bar them from working to acquire those styles which
they do not yet possess. A person can have multiple dominant modal preferences, and in any case, the
less favoured modal channels should not be ignored. Instead the more encompassing approach is to use
the learning styles classification as a self-reflection tool, by which both teachers and students can
understand their own natural tendencies and, armed with that knowledge, actively manage their teaching
and learning.

It is typical for teachers to teach according to their preferred modal channel, not their students. It follows
that this needs to be broadened to provide learning experiences that accommodate all modal
preferences. Likewise, e-learning software should be developed to accommodate all modal preferences.
The multi-modal approach thereby combines a mixture of approaches and teaching methods that support
all modal preferences. The students’ improved self-awareness then allows them to actively choose the
instructional style that best fits their individual modal preference. This pedagogy therefore focuses on the
multi-modal approach.

2.4.8 ARCS Theory of Motivation

As discussed in section 2.4.2, learning requires significant mental effort; therefore, students need to be
suitably motivated to devote the mental energy to learn. There is a significant body of research
recognising that motivation is as an important factor and precursor to learning (J. Keller 1987; J. M. Keller
1987a; J. M. Keller 1987b; Hidi & Harackiewicz 2000; Illeris 2003; Alonso et al. 2005; Palmer 2005; Long
2007; Anderson 2008; Cook et al. 2009; Shroff & Vogel 2009; Hadjerrouit 2010; Ejiwale 2012; Herman
2012; Kim et al. 2014; Royal Society 2017a).

Motivation is a characteristic inherent within each learner, but must be fostered and enhanced by the
learning material and instructional design of the lesson (Cook et al. 2009). Its impact on learning is
complex and underpinned by a multifaceted interplay of factors such as: the students’ level of self-efficacy
and self-regulation; situational and individual interest; extrinsic and intrinsic motivation; and performance
and mastery goals, etc. (Schunk 1991; Hidi & Harackiewicz 2000; Lepper et al. 2005; Palmer 2005;

39
 CHAPTER TWO

Anderman & Dawson 2011). Lepper et al. (2005) maintain that there is over a half a century of research
into intrinsic motivation and its impact on learning, but assert with concern that:

“The lower levels of intrinsic motivation for older versus younger children reported here are troubling. Not
only do children seem to be losing their enjoyment of the learning process itself but the systems of extrinsic
incentives and constraints that American schools employ to keep students on track do not effectively
compensate for the declines in intrinsic motivation.” (2005, p.193)

In contrast, e-Learning and technology-enhanced learning are often viewed as inherently motivational,
arguably in many instances due to temporary novelty value; but in addition, they also offer motivational
affordances that can help make learning more engaging. However, Keller & Suzuki (2004) warn it is
important that these motivational affordances are considered deeply within the instructional design of
the e-learning, otherwise it still runs the danger of being a novelty to learners that quickly loses its appeal.
Furthermore, there are counteractive motivational forces to be considered since despite the popularity
and global growth of e-learning and online learning delivery (Elaine & Seaman 2011; ATD Research 2014;
Ambient Insight Research 2015; ATD Research 2017), there remains historic challenges in stimulating and
maintaining learner motivation in e-learning, and in particular in distance learning (Zvacek 1991;
Rowntree 1992; Visser 1998); this is most evident in the high drop-out rates in independent distance
learning (Lee & Choi 2011; Levy 2007; Liyanagunawardena et al. 2013). This in turn has led to a significant
volume of research into the importance of learner motivation in e-learning, as well as the technological
affordances and pedagogical theory that support it (Phipps & Merisotis 2000; Huffaker & Calvert 2003;
Keller & Suzuki 2004; Virvou et al. 2005; Ardito et al. 2006; Connolly & Stanfield 2006; Condie & Munro
2007; Boulton 2008; Zhang & Bonk 2008; Hsu et al. 2009; Jang 2009; Papastergiou 2009; Shen et al. 2009;
Zaharias & Poylymenakou 2009; Alsumait & Al-Osaimi 2010; Corbalan et al. 2010; Jung 2011; Lee & Choi
2011; Liu et al. 2011; Muntean 2011; Stuchlikova & Benkovska 2011; Bertacchini et al. 2012; Fletcher et
al. 2012; Livingstone 2012; de Freitas & de Freitas 2013; Morrison & DiSalvo 2014).

In particular, this pedagogy focuses on John Keller’s ARCS model (1987a; 1987b; 2008; 2006). For many
years, Keller has been developing and testing a systematic process for analysing learner motivation and
designing motivational techniques focused on the following areas: Attention, Relevance, Confidence and
Satisfaction (ARCS). Keller (1987a), defines the ARCS categories as:

1. Attention: Capturing the interest of learners; stimulating the curiosity to learn,

2. Relevance: Meeting the personal needs/goals of the learner to affect a positive attitude,
3. Confidence: Helping the learners believe / feel that they will succeed and control their success,
4. Satisfaction: Reinforcing accomplishment with rewards (internal and external).

The ARCS model is based on a comprehensive review and synthesis of motivational research; its
conceptual validity has been confirmed in several research papers (Visser & Keller 1990; Small & Gluck
1994; Means et al. 1997), and it has been successfully applied and validated in e-learning instruction

40
 CHAPTER TWO

(Chyung et al. 1999; Song & Keller 2001; Cook et al. 2009; Lim et al. 2009; Zaharias & Poylymenakou 2009;
Corbalan et al. 2010; Yacob et al. 2012).

2.4.9 Cognitive Load Theory

Cognitive Load Theory (CLT) considers the human cognitive architecture and processes, and their inherent
limitations, in explaining how incoming information from eyes and ears are transformed into knowledge
and skills in human memory (Sweller et al. 1998). There is well established support for the consideration
of cognitive load in: instruction, computer science instruction, and e-learning (Moreno 2004;
Mehlenbacher et al. 2005; Kester et al. 2006; Kirschner et al. 2006; Harskamp et al. 2007; Anderson 2008;
Tutty & Klein 2008; Lim et al. 2009; Hadjerrouit 2010; Clark & Mayer 2011; Morrison et al. 2014) and a
wide body of research on approaches to reduce and optimise cognitive load (Sweller et al. 1998; Pollock
et al. 2002; Kalyuga & Ayres 2003; Klein 2003; Mayer & Moreno 2003; Moreno 2004; Sweller 2004; Kester
et al. 2006; Kirschner et al. 2006; Harskamp et al. 2007; Moreno & Mayer 2007; Fadel & Lemke 2008;
Kirschner et al. 2009). As outlined by Clark and Mayer (2011) CLT proposes that learners do not passively
receive incoming information, but instead undertake active cognitive processes that take in relevant
information, organise it into logical structures, and integrate it with existing knowledge for long term
recall. CLT is based on three main principles:

1. Dual Channels - Humans have separate channels for processing visual and auditory material.
2. Limited Mental Capacity – At any given time humans can actively process only limited
information in each channel; material that exceeds this threshold may enter working memory
but will not be processed and encoded into long term memory.
3. Active Processing – Human learning occurs when the appropriate cognitive processes are
engaged to mentally organise incoming auditory and visual sensory information and integrate it
with existing knowledge so that it can be stored in and recalled from long term memory.

The above principles and learning process are represented in Figure 6.

Figure 6: Cognitive theory of learning

Adapted from (Mayer & Johnson 2008, p.381)

41
 CHAPTER TWO

This model of learning leads to three types of cognitive processing.

1. Essential (intrinsic) processing is the processing of the core learning material, and is dependent
on the inherent complexity of the material. The correct application of guidance, scaffolding and
segmenting of learning material can support it.
2. Extraneous processing is processing that does not support the instructional objectives and is
created by poor instructional design and the inclusion of irrelevant or duplicate learning material.
3. Generative (Germane) processing is processing aimed at the development of new schemas or
integration into existing schemas, and the automation of these schemas. It is aimed at deeper
understanding (and far transfer) of the core material by motivating the learner to make sense of
the material and integrate it into their schemas so that it can be recalled and applied in the
appropriate circumstance.

Considering CLT, an appropriate instructional strategy is therefore to:

1. Support the learner in selecting the important information in the lesson (signposting).
2. Support the learner in managing the limited capacity of working memory by:
a. Minimising extraneous cognitive load by a careful focus on the core learning material.
b. Optimising essential processing by the careful management of the complexity of the
core learning material.
3. Maximise generative cognitive processing by providing an opportunity for non-trivial practice
activities, guided instruction, and authentic (relevant) examples and problems. This enables
integration into existing schemas, and the hooks to allow for retrieval at a later stage.

CLT’s positive impact on learning performance (retention) is well established; however, it is also
contended that judicious use of Cognitive Load approaches improve germane processing, hence deep
learning and far transfer (Sweller et al. 1998; Mayer et al. 2001; Moreno & Mayer 2002; Mayer & Moreno
2003; Mayer et al. 2003; Moreno 2004; Moreno & Mayer 2005; Harskamp et al. 2007; Moreno & Mayer
2007; Mayer & Griffith 2008; Mayer & Johnson 2008; Rourke & Sweller 2009; Skudder & Luxton-Reilly
2014).

In many respects constructivism and CLT model the same learning processes, but from differing
perspectives. Both focus on the active construction of knowledge and skills as schemas. But constructivism
focuses on principles of authenticity and active learning and how new learning is assimilated or
accommodated in schemas. Whereas, CLT takes an information processing perspective and focuses on
the efficiency of the learning process and the different types of memory used during learning. In spite of
their potential similarities, it is well recognised that there lies a tension between constructivist principles
and CLT (Kirschner et al. 2006; Anderson 2008; Alonso et al. 2009; Vogel-Walcutt et al. 2011), in particular
between discovery-based learning (Problem-Based Learning) and CLT (Sweller & Cooper 1985; Sweller et
al. 1998; Moreno 2004; Sweller 2004; Moreno & Mayer 2005; Schraw et al. 2006; Moreno & Mayer 2007;
Lim et al. 2009). This is addressed in the proposed pedagogy and the underlying formulation of heuristics,

42
 CHAPTER TWO

by focusing on moderate constructivist approaches, explanatory feedback and scaffolding, and guided
discovery16.

In relation to collaborative learning, Kirschner et al. (2009, p.31) argue that “learning by an individual
becomes less effective and efficient than learning by a group of individuals as task complexity increases.”
Therefore, where learning material is intrinsically complex and demanding, then collaborative learning
may be appropriate since it “allows information to be divided across a larger reservoir of cognitive
capacity”. The inherent complexity of the learning material must be carefully considered since for less
complex learning the coordination, recombination and collaborative learning processes add an
unnecessary cognitive overhead. Irrespective of this, the use of collaborative learning must always be
structured, appropriately prepared, and guided to reduce extraneous cognitive processing.

In relation to multi-modal learning, CLT’s dual channel principle offers partial support for multi-modal
learning via visual and aural channels; however, full multi-modal learning via all four VARK modalities is
considered to create a high extraneous cognitive load.

In relation to learner motivation, Keller and Suzuki (2004) cite research from Chang and Lehman (2001) in
asserting that motivational approaches, in particular related to the relevance dimension, can be aligned
with CLT; however, it should be noted that unrestrained motivational approaches can negatively impact
cognitive load.

Overall, a central theme within the pedagogy is that the use of pedagogical heuristics in all other areas
must be considered in light of their potential impact on cognitive load.

A final consideration within CLT is the prior knowledge and expertise of the learner (i.e. existing schemas);
several of the approaches within CLT support low-knowledge learners, but may not support or may even
impede high-knowledge learners. This is called the expertise reversal effect (Pollock et al. 2002; Kalyuga
& Ayres 2003; Sweller 2004; Kalyuga 2005; Harskamp et al. 2007; Lim et al. 2009). This considers that the
level of CLT support must be tailored to the level of expertise of the learner and should typically be
gradually faded.

2.4.10 Gamification
According to Deterding et al., gamification is “the use of game design elements in non-game contexts”
(2011, p.9); in this case the context is education. Gamification does not focus on creating fully fledged
games, but instead uses game dynamics, mechanics, and frameworks to increase pleasure, fun,
motivation, and to influence behaviour. In the context of this research, it is to increase student motivation
and ultimately to increase learning performance.

16
As oppose to pure discovery-based learning.

43
 CHAPTER TWO

Gamification has been a trending topic in recent years with significant commercial interest and usage, and
a significant growth in academic papers (Deterding et al. 2011; Deterding 2012; IVETIC & Petrović 2012;
Deterding et al. 2013; Hamari et al. 2014; Morrison & DiSalvo 2014; Iosup & Epema 2014; Pirker et al.
2014). Despite the growing number of papers published on gamification, there remains a lack of clarity
on what the gaming affordances proposed in gamification are and how to effectively design a gameful
system (Deterding et al. 2013; Deterding et al. 2011). Based on a significant literature review, and meta-
analysis of 24 empirical studies, Hamari et al. (2014) suggest that the most common gaming affordances
are points, leaderboards, achievements/badges, levels, stories, clear goals, feedback, rewards, progress
and challenge. These are taken as the basis of the gamification heuristics proposed in this research.
Additionally, Hamari et al. (2014), and various other studies (Papastergiou 2009; Deterding et al. 2011;
Deterding 2012; Nicholson 2012; IVETIC & Petrović 2012; Decker & Lawley 2013; Deterding et al. 2013;
Stott & Neustaedter 2013; Haaranen et al. 2014; Iosup & Epema 2014; Morrison & DiSalvo 2014; Pirker et
al. 2014), suggest that the empirical evidence for gamification is mixed, although overall is positive.

Gamification has significant potential, but unfortunately, suffers from a tarnished reputation; all too
often, the complexity of well-designed games are simply reduced to their more superficial elements (Stott
& Neustaedter 2013). Deterding argues that the current stock implementation of gamification focuses on
the least important parts of games, such as points, badges and leaderboards, and adds them to mundane
user activities (2012). As quoted by Deterding, Elizabeth Lawley, a professor of interactive games and
media, counsels that when a comprehensive set of gaming affordances are considered and implemented
properly, "gamification can help enrich educational experiences in a way that students will recognize and
respond to" (2012, p.17). Based on their analysis, Stott and Neustaedter go one step further and advise
that “the underlying dynamics that make games engaging are largely already recognized and utilized in
modern pedagogical practices, although under different designations” (2013, p.1). This point is recognised
in this research, ensuring the integration of a comprehensive set of game affordances into the proposed
pedagogical heuristics.

2.4.11 Computational Thinking

Jeannette Wing, one of the key proponents of computational thinking (CT), defined computational
thinking as involving “solving problems, designing systems, and understanding human behaviour, by
drawing on the concepts fundamental to computer science” (2006b, p.33). In 2011, she further clarified
that computational thinking is “the thought processes involved in formulating problems and their solutions
so that the solutions are represented in a form that can be effectively carried out by an information-
processing agent.” (Cuny, Snyder, Wing, 2010, cited in, Wing 2011, p.60). It is important to note that the
term information-processing agent does not necessarily mean a computer system; computational
thinking does not inherently lead to computer solutions. It is more focused on reformulating seemingly
difficult real-world problems that are complex, messy and partially defined into a form that is tractable
and can be solved (Wing 2006b; Wing 2006a; Barr & Stephenson 2011; Lee et al. 2011; Zhou 2011; Grover
& Pea 2013; Curzon et al. 2014). It does this by borrowing concepts and mental tools from computer

44
 CHAPTER TWO

science such as abstraction, generalisation, iteration, recursion, decompositions, modelling, automation

and algorithms. The aim is not to get humans to think like computers, but to marry the rigour and
transparency of computing techniques with the ingenuity, imagination and creativity of humans. In recent
years, computational thinking has gained momentum and received significant publicity (Wing 2006b;
Wing 2006a; Guzdial 2008; Barr & Stephenson 2011; Wing 2011; Gouws et al. 2013; Grover & Pea 2013;
Curzon et al. 2014; Lye & Koh 2014; Selby et al. 2014).

Wing argues that computational thinking will be a fundamental skill used by everybody by the middle of
the 21st Century, and that it should be added to every child’s analytical ability in the same way as reading,
writing and arithmetic. This opinion is being taken seriously and computational thinking is being
integrated into the national curriculum of several nations. For example, in the UK (Brown et al. 2014):

“A high-quality computing education equips pupils to use computational thinking and creativity to
understand and change the world.” (DfE 2013, p.151)

In the United States:

“the approach to problem-solving generally described as CT is a recognizable and crucial omission from
the expertise that children are expected to develop through routine K–12 Science and Math education
(although CT has finally been mentioned, albeit briefly, in the 2012 NRC K–12 Science Education
framework).” (Grover & Pea 2013, p.4)

It is argued that use of computers and computational thinking have revolutionised research and
innovation in science, engineering and other disciplines. The Royal Society gives a number of examples of
where computational thinking is radically influencing other disciplines: the Human Genome Project, the
Large Hadron Collider, predicting global climate change, analysing the spread and control of infectious
diseases, the Airbus fly-by-wire system, and the pharmaceutical sector (The Royal Society 2012).

However, there needs to be some caution since there remains a variety of academic and pedagogic
interpretations of what computational thinking is, what differentiates it from other thinking skills, how
best to teach it, and what can realistically be expected from our children (Guzdial 2008; Hazzan et al. 2008;
Barr & Stephenson 2011; Lee et al. 2011; Wing 2011; Grover & Pea 2013; Selby & Woollard 2013; Curzon
et al. 2014; Lye & Koh 2014; Selby et al. 2014). This lack of a consistent and coherent definition is causing
some negative repercussions since the misleading message propagated by popular media and received
by parents and teachers is that computational thinking equates with coding (Selby et al. 2014). Another
area of concern is that currently there is not a wealth of empirical evidence to link computational thinking
to improved problem-solving or far transfer to other problem domains. Hemmendinger (2010) notes that
perhaps computer scientists are, to some extent, guilty of arrogance and overreaching in that many of the
elements claimed to be part of computational thinking have been around for a while, and are not within
the sole domain of computer science. The pedagogical heuristics proposed in this research will not focus
on these academic debates, but instead concentrate on the elements of computational thinking proposed

45
 CHAPTER TWO

by the UK Department for Education (DfE), AQA, EDEXCEL and OCR award bodies, and the barefoot
computing initiative.

46
 CHAPTER THREE

3 RESEARCH METHODS
3.1 Chapter Brief
This chapter first outlines foundation concepts in research design and methodologies, and discusses the
underlying rationale for the three-phase mixed methods approach (section 3.2). The remainder of the
chapter then describes the research design used in this study, specifically:

1. The three-phase mixed methods research design,

2. The procedures used in each phase and cycle of the study,
3. The controls put in place to ensure validity, reliability and trustworthiness,
4. The teacher, student and education expert participants used in each phase and cycle of the study,
5. The instruments and materials used in each phase and cycle of the study,
6. The data collection, analysis and reporting used in each phase and cycle of the study, and
7. The research methods used in the development of the e-learning evaluation protocol.

3.2 Theoretical Foundations of the Research Design

The aim of this study is to synthesise a holistic pedagogy for the design and evaluation of e-learning
software for high-school computing, and then to test its impact on student learning and motivation. In
keeping with this aim, the initial focus of the research was on the exploration of learning theories and
their synthesis into an e-learning pedagogy, followed by a quantifiable assessment of the pedagogy’s
impact on student learning and motivation. Based on these prerequisites, the study was designed with
three phases, in which each phase used a mixed methods approach, but with a different qualitative-
quantitative mix. With reference to relevant literature, the following sub-sections discuss the underlying
theoretical rationale for the three-phase mixed methods approach.

3.2.1 Research Paradigm

Each research study falls within a paradigm continuum ranging between the two extremes of positivism
(quantitative) and interpretivism (qualitative). A central tenet of quantitative research is the objective
(unbiased) reporting of a single reality, whilst qualitative research views that there is no single reality, but
instead each person has their perception of reality (Lincoln & Guba 1985; Guba & Lincoln 1989; Field &
Hole 2003; Creswell 2007; Creswell & Plano Clark 2010; MacKenzie 2013). Qualitative research embraces
the view that each participant has their own interpreted reality, and the possibility that the researcher’s
own interpretation may colour the research findings. It accounts for this by actively communicating the
sets of beliefs, basic assumptions, way of thinking, and framework that inform the conduct and writing of
the research study. Creswell (2007; 2010) defines this as the underlying paradigm or worldview that
governs the research; he characterises it in accordance with the following worldview elements: ontology,
epistemology, axiology, methodology, and rhetoric. The specific stance that this research takes for each
worldview element is outlined in Table 5.

47
 CHAPTER THREE

Worldview Element Research Stance

Ontology Singular and multiple realities are considered; primarily a single reality, but in some
cases viewed through multiple perspectives.
What is the nature of
reality?

Epistemology In Phase 1 and 2, a prolonged engagement is used to build a close relationship with
participants, but professional impartiality is used to collect data from participants
What is the relationship
(participants are not treated as full collaborators).
between the researcher
and that being In Phase 3, a short engagement is planned with an increased focus on impartiality and
researched? objectivity.

Axiology Depending on the research phase and the nature of the data (qualitative or
quantitative), there are multiple stances used; meaning the inclusion of potentially
What is the role of
biased and unbiased perspectives.
values?

Methodology Both quantitative and qualitative data are collected and combined.

What is the process of

research?

Rhetoric Typically, the style of writing is more formal in nature, more aligned with a quantitative
study; it avoids the typical qualitative writing style based on thick first-person literary
What is the language of
descriptions. However, the voice of the participants is carried through in the wide use
research?
of direct quotes.

Qualitative and quantitative terminology is used appropriately.

Table 5: The Worldview underlying this research study

In keeping with Creswell and Plano Clark’s (2010) description of four high-level worldviews: post-positivist,
constructivist, participatory, and pragmatist, this research study is broadly aligned with the pragmatist
worldview. Pragmatism is a philosophy oriented towards “what works” in practice (Creswell & Plano Clark
2010) and is commonly associated with mixed methods research (Yvonne Feilzer 2010; Creswell & Plano
Clark 2010). According to Johnson and Onwuegbuzie (2004) citing Dewey (1948), it focuses on practical
consequences and real-world phenomena and problems. It can take many forms since pragmatists insist
on the freedom to choose the methods, techniques and procedures most appropriate to the problem
under study and the research needs (Johnson & Onwuegbuzie 2004).

3.2.2 What is Mixed Methods Methodology and Why was it Used?

This study combined elements of qualitative and quantitative research approaches into a mixed methods
research design. Creswell and Plano Clark (2010) give a comprehensive definition of what mixed methods
is: “as a methodology, it involves philosophical assumptions that guide the direction of the collection and
analysis and the mixture of qualitative and quantitative approaches in many phases of the research
process. As a method, it focuses on collecting, analysing, and mixing both quantitative and qualitative data
in a single study or series of studies. Its central premise is that the use of quantitative and qualitative
approaches, in combination, provides a better understanding of research problems than either approach
alone.” (2010, p.4)

48
 CHAPTER THREE

Mixed methods is particularly suited to this study since it provides compelling support for studies in which:
exploratory findings need to be generalised; results need to be explained; results need to be triangulated
to increase validity; and one data source is insufficient to give a complete picture (Creswell & Plano Clark
2010; Johnson & Onwuegbuzie 2004).

Inherently, the use of both qualitative and quantitative research methods provides a more comprehensive
understanding of the problem domain. This was particularly relevant in the first two research phases of
this study, which were exploratory in nature, and in which both qualitative and quantitative methods
provided a detailed understanding of how the pedagogical heuristics (embodied in the prototype
software) performed within a school context.

The results of one method were used to elaborate, illustrate or clarify the results of the other method. As
discussed in later sections, this research used observation studies, survey instruments and focus groups
in which the observation study and the survey instrument were used to inform the focus group; and often
qualitative data from the survey instrument and focus group were used to explain or illustrate quantitative
data from the observation study or survey instrument.

Furthermore, the use of both approaches offset the weaknesses of one with the strengths of the other.
Arguably, the reductionist nature of quantitative research is weak in representing the full richness of the
problem domain and in understanding its context. Furthermore, it lacks depth by not linking back to the
voice of the participants (Creswell & Plano Clark 2010). In this research, the focus groups and survey
instruments where used to good effect in identifying the underlying opinion and voice of the student
participants. In contrast, qualitative research is arguably flawed due to the personal interpretations made
by the researcher, the potential for bias created by this, and the difficulty in generalising to a larger
population (Field & Hole 2003; Creswell & Plano Clark 2010; Lazar et al. 2010). The issue of potential
researcher bias is mitigated by marrying quantitative results with qualitative findings and ensuring they
are aligned.

It is also important to note that qualitative and quantitative approaches are best suited to answering
different research questions; for example, qualitative methods are best suited to the development of
theory, such as the pedagogical heuristics in this study, whilst quantitative methods are more suitable for
adding an additional confirmation and potentially generalising the findings (Creswell & Plano Clark 2010;
Field 2013). Hence, Phase 3 was designed specifically to confirm the exploratory phases and generalise
the findings to the wider student population.

Additionally, by combining both qualitative and quantitative views, this research could focus on questions
that could not be answered by either approach individually; more specifically, when the qualitative and
quantitative views did not concur it guided further research to understand why. This was particularly
significant in the exploratory phases of this research since further investigation in later phases or cycles
was prompted where the qualitative and quantitative views diverged.

A further reason to use mixed methods is that results from the different methods can be triangulated to
corroborate each other, which in turn leads to results that are considered to have greater validity (Bryman

49
 CHAPTER THREE

2006; Borrego et al. 2009). Hence, the qualitative and quantitative findings from the observation studies,
focus groups, survey instruments and other research instruments were triangulated to gain a
comprehensive and valid set of results.

Additionally, it is common practice to use qualitative research to inform the instrument development for
a later quantitative phase (Creswell & Plano Clark 2010). This approach was used in this research by the
iterative improvement of the survey instrument and e-learning prototype for final use in Phase 3.

One final benefit of mixed methods research is that it is ‘practical’ in nature; it allows the researcher the
freedom to use all methods possible to address the research problem (Creswell & Plano Clark 2010;
Yvonne Feilzer 2010; Johnson & Onwuegbuzie 2004).

However, the use of mixed methods is not without its drawbacks. Johnson and Onwuegbuzie (2004), and
Creswell and Plano Clark (2010) outline some key challenges, such as: the researcher is required to have
skills in both qualitative and quantitative approaches, and how to mix them effectively; the use of mixed
methods can demand extensive time, resources, and effort from the researcher; and there still remains a
potential for philosophical objections or a lack of understanding of the value of mixed methods from the
research audience. The first two points have been challenges faced in this study; in particular, the
increased demand on time, resources and effort have been significant factors.

3.2.3 The High-level Structure of the Mixed Methods Design

Creswell and Plano Clark (2010) build upon various seminal works to provide a primer on how to align a
mixed methods design with specific research objectives. For example: whether the research design will
be fixed or emergent; choosing a particular mixed methods design; and determining the timing, level of
interaction, and priority of quantitative and qualitative strands within a research phase.

Fixed and/or Emergent - As with many research designs, this study falls between being a fixed and an
emergent design. It was fixed since the three phases were predetermined and planned at the start of the
research process, and the research methods within each phase were implemented largely as planned.
However, it was also emergent since the details of the design of the second and third phases emerged
based on the findings of the previous phases.

Exploratory Sequential Design - The overall phased approach utilised an exploratory mixed methods
design; the exploratory design was based on sequential phases of qualitative theory development
followed by a final phase of quantitative research to test the theory on a different and larger sample. As
typified by theory development exploratory design, the initial qualitative phases took priority (Morgan,
1998; Morse, 1991). Due to the shift in paradigms between phases, the worldview within the research
shifted from a more interpretist worldview in Phases 1 and 2 towards post-positivism in Phase 3.

Convergent Parallel Design - Each phase of the study followed a mixed methods convergent parallel
design, otherwise known as a triangulation design. As outlined by Morse, the purpose of the convergent
design is “to obtain different but complementary data on the same topic” (1991, p. 122) to better
understand the research topic. The qualitative and quantitative data were collected mainly concurrently,

50
 CHAPTER THREE

but separately. In Phase 1 and 2 the weight of qualitative data was more than the quantitative data, hence
it was given priority; this trend was reversed in Phase 3. The two data sets were collected and analysed
separately and independently from each other, using typical quantitative and qualitative procedures.
After the data were analysed, the results were merged and interpreted to identify to what extent and in
what ways the two sets of results converge, diverge or relate to each other. This approach enabled the
corroboration between the two types of data, and therefore increased the inherent validity of findings.

Figure 7 summarises the above, showing the exploratory sequential design, based on sequential phases
of qualitative theory development (i.e. the pedagogical heuristics) followed by a final phase of
quantitative research to test the theory on a different and larger sample.

Exploration Confirmation

Develop: Phase 1 Prototype

Pedagogical Theory
Research Instruments Interpret
E-Learning Software QUAL + quan

Refine: Phase 2 Action Research

Pedagogical Theory
Research Instruments Interpret
E-Learning Software
QUAL + quan

Refine: Phase 3 Experiment

Emergent Design Pedagogical Theory
Research Instruments Interpret
E-Learning Software
QUAN + qual

Emergent Design

Figure 7: The high-level structure of the phased research

The findings from the convergent parallel design used in each phase informed the refinement of the
pedagogical heuristics, research instruments, and e-learning software tool used in the next phase. Finally,
at the end of Phase 3, the quantitative and qualitative strands were combined in a final interpretation
activity to discuss and draw final research conclusions.

3.2.4 Validity, Reliability and Trustworthiness

The differences and philosophical debate on what constitutes validity in quantitative and qualitative
research is widely and passionately discussed in research literature (Maxwell 1992; Golafshani 2003;
Morse & Barrett 2008; Guba & Lincoln 2005; Borrego et al. 2009; Guba & Lincoln 1989; Creswell 2007;
Creswell & Plano Clark 2010; Lincoln & Guba 1985), but is beyond the scope of this study. This research
aligns with the direction set by Lincoln and Guba (1985) in their seminal work Naturalist Inquiry. It is
argued that whilst quantitative research focuses on internal validity, external validity, reliability, and
objectivity, these criteria are less suitable for qualitative research due to the fundamental paradigmatic
differences between the two approaches. Instead, a set of parallel criteria focused on trustworthiness are
discussed for qualitative research; these are credibility, transferability, dependability and confirmability.

Where the research had a quantitative priority, the validation strategies used were associated with
Internal Validity, External Validity (generalisable), Reliability and Objectivity (Field & Hole 2003; Lazar et
al. 2010; MacKenzie 2013).

51
 CHAPTER THREE

1. Internal Validity refers to the extent to which the experimental effect observed is due to the test
conditions; it attempts to protect the experiment results from being corrupted by some other
unintended factor. In the context of this research, it means that the Phase 3 experiment
accurately measures student learning and identifies whether the pedagogical design of the e-
learning software influenced learning and/or motivation. Most importantly, this should be
achieved without any undue influence on the dependent variable (learning) by some other
confounding factor.
2. External Validity (Generalisable) refers to the extent to which the experiment results can be
generalised to other people and other situations (i.e. a wider population). In the context of this
research, it means that the results have a wider use beyond the sample to infer results to the
overall GCSE Computer Science population.
3. Reliability refers to the consistency, stability and predictability of the research over time. It is a
prerequisite for validity since unreliable research methods cannot lead to valid research results.
Reliability ensures that an experiment is replicable and can be reproduced and confirmed as
accurate multiple times.
4. Objectivity refers to how the research is protected from value or bias; it is typically achieved by
the utilisation of methods and instruments that mitigate the possibility of human bias distorting
quantitative results.

Where the research had a qualitative priority, the validation strategies used were associated with
credibility, transferability, dependability and confirmability.

1. Credibility attempts to establish that the constructed realities of the research participants, and
the realities represented by the researcher and attributed to these participants match (Guba &
Lincoln 1989; Creswell 2007). In relation to this research, this means that the feedback on the
pedagogy collected from the research participants is accurately understood and reflected by the
researcher.
2. Transferability - The small sample size means that the findings from Phase 1 and 2 cannot be
considered generalisable to a wider population; however, they can be considered transferable
to a similar context. The important point is that the description necessary to identify that context
is provided to enable another researcher interested in making a transfer to reach a conclusion
about whether it is possible (Lincoln & Guba 1985).
3. Dependability is parallel to the quantitative concept of reliability and focuses on the consistency,
stability and predictability of the research and qualitative data over time (Creswell 2007; Lincoln
& Guba 1985). However, it is important to note that this excludes intentional methodological
changes brought about by an emergent design. Guba and Lincoln (1989, p.242) maintain that
methodological changes “need to be both tracked and trackable (publicly inspectable), so that
outside reviewers of such an evaluation can explore the process, judge the decisions that were
made, and understand what salient factors in the context led the evaluator to the decisions and
interpretations made.”

52
 CHAPTER THREE

4. Confirmability is parallel to the quantitative concept of objectivity and confirms that the data,
interpretations, and outcomes of the inquiry are based on the contexts and participants of the
inquiry, and not biased or inventions of the researcher. According to Guba and Lincoln (1989)
this means that research data can be tracked to its source and that the logic used to reach
subsequent interpretations is transparent, coherent and corroborated by source data.

3.2.4.1 Validity and Reliability of the Survey Instrument Design

Due to the importance of survey instruments in almost all fields of research (Litwin 1995; Krosnick 1999;
Creswell & Plano Clark 2010; Lazar et al. 2010; Saris & Gallhofer 2014), it is worthwhile briefly reviewing
some of the concepts used to establish validity and reliability in survey instrument design. This is
particularly relevant in this study due to the central thread the survey instrument plays in all phases.

Unfortunately, according to (Litwin 1995; Krosnick 1999; Lazar et al. 2010; Saris & Gallhofer 2014) the
design and testing of a valid and reliable survey instrument is no trivial task. It is for this reason that an
initial step is to try to identify an existing, established instrument that is pre-validated, and that has a good
fit with the study in question (Litwin 1995; Lazar et al. 2010).

If no such instrument exists, then a new survey instrument must be designed specifically for the study.
Considering (Litwin 1995; Krosnick 1999; Creswell & Plano Clark 2010; Lazar et al. 2010; Saris & Gallhofer
2014), to ensure unbiased, good quality data, the design of the survey instrument should consider aspects
such as: pilot testing, test-retest (intra-observer) reliability, alternate form reliability, internal reliability,
inter-observer reliability, face validity, content validity, criterion validity, and construct validity. As
discussed in sections 6.2.3.1 and 6.2.4.1, this study focuses on (as defined by Litwin (1995)):

1. Pilot testing (pretesting) – One of the most important steps in the development of a new survey
instrument is to test the instrument with a smaller sample to identify unanticipated issues in the
instrument’s form, delivery and analysis. This includes, but is not limited to: instrument delivery;
instrument structure, such as length, numbering, skip flow, etc.; item language, spelling,
grammar and respondent appropriateness; and data coding and analysis.
2. Test-retest reliability – This establishes how reproducible the survey instrument’s data are; it
can be achieved by having the same respondents complete the survey (or survey item) at two
different points and seeing how consistent the responses are.
3. Alternate form reliability – This tries to mitigate the practice effect, by which respondents
become familiar with survey items and respond based on memory; it can be achieved by using
differently worded items to measure the same attribute.
4. Internal consistency reliability - To increase reliability, multiple survey items can be used to
measure different aspects of the same concept. However, it is critical that these items do
accurately measure the same underlying concept; hence the consistency of their measurement
must be ascertained.

53
 CHAPTER THREE

5. Face Validity - The most basic review of validity is typically a cursory review of the survey
instrument by untrained individuals. As a weak validity measure, this is typically skipped in favour
of content validity.
6. Content Validity – Focuses on how well the survey instrument measures the constructs it sets
out to measure; it is carried out by reviewers who have knowledge or expertise in the subject
matter.

Inter-observer reliability, criterion validity and construct validity are not addressed directly in this study
and hence, for brevity, are not discussed in this brief review.

3.2.5 Action Research Design

Within the overarching mixed methods design, Phase 2 follows an action research design. O’Brien(2001)
describes action research in simple terms: it is “learning by doing” – a group of people identify a problem,
think about how to fix it, take some action to fix it, see how successful they were, and, if not satisfied, try
again. Put in such terms, it gives the impression that action research lacks rigour; this is not the case
Susman and Evered (1978) detail a five-phase cyclical process, which forms the basis of the action research
cycle used in this research:

1. Diagnosing,
2. Action Planning,
3. Action Taking,
4. Evaluating, and
5. Specifying Learning.

It is also important to note that action research is not just about problem-solving; to remain valid as
research a theoretical foundation must be present as the basis of the intervention action. According to
Baskerville and Wood-Harper (1996), action research is a well-established, practice-based approach that
merges research and praxis, thus producing especially relevant research findings. Levy (2003) outlines the
concept of praxis as being informed, committed action that gives rise to knowledge rather than just
successful action. The reference to “informed” means it is a social collaborative activity that involves the
significant contribution of the research participants. The reference to “committed” means it is a planned
and intentional action. And finally, the objectives of action research go beyond a successful local action,
but must also result in the generation of new knowledge. In the context of this research, the new
knowledge is the definition and elaboration of pedagogical heuristics for e-learning. This is aligned with
the long history of action research in educational research; in particular, McNiff and Whitehead (2005)
argue that action research has much to offer in the generation of quality educational theory.

Additionally, mixed methods and action research fit well together; a meta-analysis of 108 Mixed Methods
Action Research (MMAR) studies between 1999 and 2012 showed that 35% were in education (Ivankova
2014). Ivankova (2014) argues that mixed methods provide a sound and pragmatic research approach to
support action research, and in particular provide a solid scientific methodological framework for it.

54
 CHAPTER THREE

Although action research is often viewed as a qualitative reflective approach, there is a growing body of
literature discussing the compatibility and harmonious use of mixed methods and action research
(Hinchey 2008; Mills 2011; Koshy et al. 2011; James et al. 2007).

Specifically, Mills (2011) asserts that mixed methods can be used to good effect in educational action
research when researchers need to enhance observations and qualitative narratives with quantitative
student performance data. In addition, Greenwood and Levin (2007) argue that action researchers need
to be knowledgeable of quantitative and qualitative data collection strategies in order to address complex
social problems (such as learning).

It should be noted that when deciding to follow an action research design, there was an internal
deliberation (which is not entirely resolved) on whether this research was more allied with action research
or design-based research. There is significant commonality and overlap between the two methods;
design-based research shares many characteristics with action research, in that: it is practice-based;
focused on real-world settings; follows iterative cycles; has a strong background in education research;
and is often used to extract shareable theories (The Design-Based Research Collective 2003; David 2007;
Barab & Squire 2004; Sandoval & Bell 2004; Wang & Hannafin 2005). However, arguably, design-based
research has a stronger focus on design principles and the development of an educational environment
or artefact; whereas in this research, the e-learning software is used purely as a prototype test tool. This
study does not dogmatically follow action research since, in action research, the research topic typically
has a local origin, and the research is often driven by the study participants, one of which is typically the
principal researcher. In this research, the research objectives have a strong local resonance, but did not
originate in the schools used in the study; additionally, whilst there was a strong collaborative
relationship, ultimately the study was governed by a researcher external to the research locations. In this
respect this study was more aligned with design-based research (Pardo-ballester & Rodríguez 2009; Wang
& Hannafin 2005). In a final analysis, action research was chosen due to its comparative maturity, the
rigour it adds to the research design, the relative richness of research that documents the steps to use in
action research, and its extensively cited usage with mixed methods.

3.2.6 Incremental and Iterative Development Methodology

As previously outlined, an e-learning software prototype was developed to embody the e-learning
heuristics in a tangible form for student usage, testing and learning. An incremental and iterative
development model (Larman & Basili 2003; Cockburn 2008; Ruparelia 2010) was used to design and
develop the e-learning software across the three phases of the research. As summarised by Cockburn
(2008):

1. Incremental Development: focuses on delineating different parts of the e-learning software

which are developed at different times or rates and integrated as they are completed. Within
each phase/cycle of the study and the associated releases of the e-learning prototype, different
aspects of the computer science curriculum, heuristics and parts of the software prototype were
focused on. This is discussed in more detail in sections 6.2.2 and Figure 30 (section 6.5)

55
 CHAPTER THREE

2. Iterative Development: based on previous evaluation cycles, iterative development was used to
revise and improve the different parts of the e-learning prototype. The evaluation cycles
gathered student feedback on the e-learning prototype to improve the heuristics, but this was
also fed back into the design/development process to improve the next version of the software
prototype.

Based on this methodology, the different versions of the e-learning software prototype are outlined in
section 6.2.2, and the student feedback based on the various versions of the software are reported in
sections 6.5 and 6.6. Figure 8, visually represents the incremental and iterative development process.

Figure 8: The incremental and iterative development model.

Adapted from (Ghahrai 2018)

For each phase/cycle the planning stage identified at a high-level what educational content (parts of
computer science curriculum) and pedagogical heuristics were to be included; this planning scope was
then specified in terms of requirements.

In keeping with standard design practice for educational software, before each development iteration
these requirements were analysed and transformed into a detailed design which combined the
pedagogical heuristics and the educational material, to represent them in a mixture of images and text in
the storyboard documents. The storyboards documented a clear description of the instructional design,
educational content and presentation before committing to software development.

During the implementation stage a combination of Captivate 8 (e-learning), Visual Studio 12 Web (Web
and XML) and Photoshop CS6 were used to develop SCORM 2004 (3rd Edition) compliant e-learning
software.

Prior to student usage of the e-learning software during the observation study, the researcher tested the
prototype software according to the following 3-step test approach:

1. Unit testing on the development environment using multiple web-browsers.

2. End-to-end testing of the cloud-hosted version of the software using multiple web-browsers.

56
 CHAPTER THREE

3. End-to-end testing of the cloud-hosted version of the software using multiple web-browsers,
from within the school’s computer lab.

As discussed in sections 2.3.2 and 2.3.3, it is important to reiterate that usability is a mandatory pre-
requisite for e-learning software. Although this study has purposely taken a restricted focus towards
pedagogy, usability was taken into consideration in the development of the e-learning software. After
Phase 1 an external expert usability evaluation was carried out, and this feedback was incorporated into
Phase 2. In parallel, a professional graphical designer provided their services to ensure the Graphical User
Interface (GUI) had a baseline of aesthetic appeal and more importantly consistency across graphical
elements. This approach ensured a foundation of usability that would not detract away from the
pedagogical aspects of the e-learning software.

3.3 Introducing the Three Phases of Research

Building upon section 3.2.3, this study is divided into three phases; Phase 1 and Phase 2 have a qualitative
priority. Phase 1 acted as a pilot study and Phase 2 employed action research methods in two cycles of
study. The rationale for the first two phases was to initially work in-depth, using a significant literature
review and involving a smaller number of students, teachers and education experts to inform the
synthesis of the pedagogical heuristics. This approach iteratively and progressively refined the heuristics
and e-learning software test tool.

Phase 3 had a quantitative priority, involving a larger student sample, to identify the impact of the
heuristics (as embodied in the e-learning software test tool) on student learning and motivation, and to
generalise the study findings to the wider student population. This research design is represented
diagrammatically in Figure 9.

Figure 9: The three phases of research.

57
 CHAPTER THREE

As an initial orientation, Table 6 provides a high-level summary of the instruments and instrumentation 17
used in this research, their primary purpose, the data type of information collected, and the phase(s) they
were used in.

Instrument / Purpose Phases Data Types

Instrumentation
Pedagogy Evaluation Used to collect feedback from 2 &3 Qualitative feedback - Open
Feedback Document education experts and teachers on the text
pedagogical heuristics.
Quantitative ratings -Interval
Where necessary, this was followed by data
one-to-one interviews to clarify
feedback.

VARK Learning Styles A survey instrument used to Identify 1,2, & 3 Quantitative categorisation –
Instrument student modality preferences. Nominal data

E-learning Software An iteratively refined software test tool 1, 2, & 3 Quantitative – Usage metrics
Prototype used to embody the e-learning from the Learning
heuristics into a tangible form for Management System (LMS).
student usage, testing and learning.

Observation Study Observation of student usage of the e- 1&2 Qualitative Observation

learning software prototype. notes

The observation study was

documented informally in handwritten
notes and transcribed later after the
observation.

Survey Instrument An iteratively refined survey 1, 2, & 3 Quantitative rating – Interval

instrument used in all phases to collect data
student rating and opinion on the e-
Qualitative opinion – Open
learning software prototype. Feedback
text
was then extrapolated towards the
pedagogical heuristics.

Focus Group Informed by the observation study and 1&2 Qualitative opinion – Video
survey results in each phase and used recording
to collect in-depth qualitative feedback
from the students on the e-learning
software prototype. Feedback was
then extrapolated towards the
pedagogical heuristics.

Pre-Test / Post-Test A GCSE Computer Science test used to 3 Quantitative – Ratio data
measure student learning performance
before and after using the e-learning
software.
Table 6: Summary of main research instruments and instrumentation used in the study

The remainder of this chapter discusses:

1. The three phases of the research, specifically their objectives, participants and research
activities;

17
The use or application of instruments (as for observation, measurement, or control) (Merriam-Webster 2018b).

58
 CHAPTER THREE

2. Data collection, analysis and reporting; and

3. The procedures and controls put in place to satisfy the validity, reliability and trustworthiness of
the study.

3.4 Phase 1: Pilot Study

3.4.1 Objectives
The primary objective of Phase 1 was to set a strong foundation for the research study; this was achieved
in terms of:

1. Piloting the research methods and protocol,

2. Developing the first draft of the pedagogical heuristics, and
3. Developing a working e-learning software prototype, which was in turn used for evaluation
purposes.

3.4.2 Phase 1 Participants

3.4.2.1 Exploratory Phases: Student Participant Overview

The participants for each phase of the study are discussed in the relevant sections. However, before
discussing Phase 1 and Phase 2, it is important to note that these two phases have a qualitative priority;
therefore, a small sample18 of GCSE students (15/16 years old) from a local private school in Cyprus was
chosen. Participant recruitment was undertaken via a presentation to both the students and their
guardians, accompanied by information letters and consent forms.

The sample size is the same for both the qualitative and quantitative strands of these phases; this aids
correlation, but does not allow for statistically significant research results. This is not a concern since this
is not an objective of these phases.

Using the same participants in the qualitative and quantitative strands of research is recommended by
Creswell and Plano Clark (2010), who advise that when the purpose is to corroborate, directly compare,
or relate two sets of findings about a topic, the individuals who participate in the qualitative sample should
be the same as those who participate in the quantitative sample.

Ideally, in such a study, purposeful sampling would be undertaken in which participants are intentionally
selected due to their inherent capacity to contribute to the understanding of the research problem;
specifically, a combination of Maximum Variation and Typical Case sampling would be optimal. However,
due to constraints on appropriate student participants in the research location (Cyprus), all research
participants were taken from two GCSE ICT classes in the 1st (year-4) and 2nd year (year-5) of the GCSE. It

18
Eight students were originally recruited; however, the number of participants varied depending on the phase and
research activity. This is discussed in more detail in the relevant sections.

59
 CHAPTER THREE

is important to note that since the students were taken from a private school, they are all high-performers;
based on mock exam results for GCSE ICT, all the year-5 students were A* students, and based on internal
school exams, the three year-4 students had two A grades and a B grade. It should also be noted that
although unavoidable, the recruitment of GCSE ICT students may not fully represent GCSE Computing
students; furthermore, the ICT GCSE syllabus does not cover in detail algorithms and computational
thinking 19 . Overall, this means that student responses may not be fully representative of the GCSE
Computer Science population. This point is partially mitigated by the fact that Phase 1 and Phase 2 do not
measure learning performance, and do not aim for statistically significant research results. The qualitative
student feedback on the e-learning software and the underlying pedagogy remains valuable, but has some
limitations on its transferability to the wider GCSE population.

3.4.2.1.1 Phase 1 Students

The recruitment process secured eight students to participate in Phase 1; these participants were taken
from the 1st year GCSE ICT class (year-4). As discussed in the previous section, all students were high
performers. The initial recruitment process established the Phase 1 student sample; however, thereafter
attendance was voluntary and based on student availability: eight students attended the observation
study, seven of those students responded to the survey, and six of those students attended the focus
group.

3.4.2.2 Teachers and Educational Experts

Five education experts were involved in the evaluation of the pedagogical heuristics in Phase 1, all of
whom were senior university academics in the fields of child computing interaction, computer science,
education and educational media, and curriculum, pedagogy and assessment.

One GCSE ICT teacher, with an educational background in computer science and over five years of
teaching experience, also provided evaluation feedback on the pedagogical heuristics.

3.4.3 Pilot of Research Methods

As part of good research practice (MacKenzie 2013), it is important to undertake pilot testing to get an
early assessment of the research methods, experiment protocols and data collection and analysis
approaches to be used. As a pilot stage, Phase 1 allowed the early identification of limitations in the:

1. Research methods,
2. Participant selection,
3. Survey instrument,
4. Observation protocols,
a. Participant instructions,

19
The fact that the ICT GCSE syllabus does not cover in detail algorithms and computational thinking is positive
since it avoids unintentional influence on student opinion from ongoing learning within the GCSE.

60
 CHAPTER THREE

b. Participant training,
c. Experiment preparation,
d. Environment and technical checks,
e. Researcher behaviour (professional neutrality), and the
5. E-learning software design and development processes.

The feedback and lessons learnt in this area were incorporated into the research methods and experiment
protocols of later phases, and are documented in section 6.3.1

3.4.4 Phase 1 Research Activities

The Phase 1 research is broken down into the following activities.

1. Literature review of learning theories.

2. Research on GCSE Computer Science curriculum – focus on algorithms.
3. Synthesis of pedagogical heuristics into an e-learning pedagogy.
4. Evaluation of pedagogical heuristics from teacher and education experts (refer to section 6.2.1).
5. Design of e-learning Software – Storyboards (refer to section 3.2.6).
6. Develop and test e-learning software prototype (refer to section 3.2.6).
7. Secure ethics approval for research in schools.
8. Secure informed consent from head-teacher, teachers, students and parents.
9. Distribute VARK learning styles instrument (refer to section 3.4.4.3).
10. School-based observation of student usage of e-learning software prototype (refer to section
3.4.4.4).
11. Distribute survey instrument (refer to section 3.4.4.5).
12. Undertake focus group (refer to sections 3.4.4.6).
13. Merge and interpret results from different research strands.
14. Provide feedback back to participants (member checking) to verify that researcher interpretation
is correct.

A mixed methods procedural diagram is shown in Figure 10, which gives a visual representation of points
4 and 9 to 13.

61
 CHAPTER THREE

Phase 1 Mixed Methods Convergent Design

Procedures Observation of E- Procedures Procedures

* 5 Education Experts & 1 Teacher * N students used software at Survey Instrument & * N students
Evaluation of Pedagogy Learning Software
* High-level guidance questions school and at home. VARK Questionnaire * Online VARK questionnaire and
QUAL usage
provided to evaluators * Observation of school software QUAN+qual student survey instrument
data collection QUAL+quan
* Evaluation and constructive usage. data collection administered
data collection
feedback on Pedagogy.
Product Product
Product * Observation notes * Quantitative VARK scores for
Summary feedback and/or in place * Usage statistics from SCORM each person.
comments within the pedagogy Cloud. * Quantitative survey scores
document from each evaluator. * Student answers on SCORM * Qualitative answers to open
Cloud and Teamie. questions

Procedures Procedures Procedures

* Summarise into high-level * Reflect on observation notes, * Summarise into descriptive
QUAL feedback points. QUAL+quan student answers and usage QUAN+qual statistics.
Data analysis * Detailed corrections to Data analysis statistics. Data analysis * Create themes for open
Pedagogy document. answers
Product * Correlate between VARK,
Product * Clean observation notes Survey and open questions.
* List of feedback areas for the * Summarised usage statistics
pedagogy from SCORM Cloud. Product
* Corrections to pedagogy * Student answers on SCORM * VARK learning style preference
document. Cloud and Teamie. per person and summarised for
group.
* Descriptive statistics for survey.
* Open answers categorised and
useful quotes identified.

Procedures
* Correlate the 4 strands of Procedures
results and findings. Merge the Results * N students
Focus Group
QUAL * Update focus group questions
Product based on survey results.
data collection
* Intermediate merged results * Semi-structured focus group
and discussion
Product
* Video
* Focus Group transcript

Procedure Product Procedures

*Consider how merged results Interpretation *Discussion and Phase 1 Results * Summarise focus group
produce a better understanding – thesis chapter QUAL responses into major themes.
how do results converge and * List of Improvements to Data analysis * Useful quotes identified.
diverge. research methods and research
*Evaluate if further research is instruments. Product
necessary to understand *List of Improvements and gaps * Summary of major themes
divergent results. areas in Pedagogical Heuristics identified in focus group.
*List of Improvements for E-
Learning prototype.

Figure 10: Phase 1 - mixed methods convergent design

62
 CHAPTER THREE

3.4.4.1 Synthesis of Pedagogical Heuristics into an E-learning Pedagogy

The first step in Phase 1 was to undertake a comprehensive literature review resulting in the initial e-
learning pedagogical heuristics for GCSE Computing. The literature review is outlined in section 2.4 and
the synthesis of the e-learning pedagogy is discussed in Chapter Four.

3.4.4.2 Evaluation of Pedagogical Heuristics from Teacher and Education Experts

A central component of this research was the involvement of teachers and education experts in the
evaluation and feedback on the e-learning pedagogy. The composition of the pedagogy evaluation group
in Phase1 is outlined in section 3.4.2.2. The evaluator feedback was guided by the following factors:

1. Appropriateness of principles for 15/16-year olds,

2. Appropriateness of principles for computer science,
3. Feasibility of principles for teaching in schools,
4. Gaps in coverage, or weak areas that need to be improved,
5. Suggestions of best practice in implementing the principles,
6. Principles that you may disagree with altogether, and
7. New principles that are not considered in the current version of pedagogy.

Evaluators were given up to two months to review the e-learning pedagogy document, and feedback was
provided in writing either by email and/or inline comments and corrections within the document. Where
the feedback was unclear to the researcher, then one-to-one interviews were used to clarify.

The evaluation feedback was analysed and incorporated into the research at the start of Phase 2. This
pedagogical feedback is summarised in section 6.4.1.

3.4.4.3 Distribution of the VARK Learning Styles Instrument

The VARK learning styles instrument was used in Phase 1 to collect data on students’ preferred modalities.
The instrument was administered to the student participants before the observation study; it was
distributed via email to the participants’ research email accounts. The emails contained a link to an online
survey tool by which the survey was administered. Pseudonyms (protecting anonymity) were used
throughout the process, from distribution all the way to interpretation of results.

Data from this instrument was collated automatically via the online survey tool, and exported in comma-
separated value (CSV) format. This was then imported into a spreadsheet program to categorise each
student’s response according to the Visual, Aural, Read-Write and Kinaesthetic modalities. Each student’s
VARK profile was then summed to show the overall modal preferences of the group.

This approach for distribution, data collection, and analysis was unchanged between Phase 1 through to
Phase 3.

Further information about the VARK learning styles instrument is outlined in section 6.2.3

63
 CHAPTER THREE

3.4.4.4 Observation Study

After the first e-learning software prototype was developed (refer to section 6.2.2.1), an observation
study was organised by the researcher to observe student usage. This was organised on school premises
in the school’s ICT lab; the observation started at 13:30, after the students had completed their school
day and had time for a short break. Ten minutes were allocated to an introduction and organising the
participants; then the student participants used the e-learning software prototype for a period of 1 hour
and 10 minutes, allowing the research investigator to observe the direct usage of the software. The
researcher provided minimal guidance and support (e.g. forgotten login credentials, resuming an e-
learning session if prematurely exited, slow connectivity and video streaming issues etc.).

At the end of the observation, the students were asked to progress and complete the remaining sections
of the e-learning software asynchronously from home.

Throughout the observation, no research instrument was used. However, prior to the observation, a
suggested environment checklist (refer to section 6.2.5) was distributed by email for the student
participants to check their home environment and for the school to check their ICT lab for compatibility.

Notes from this observation study were documented informally by the researcher in handwritten notes
and transcribed later after the observation.

The student answers posted onto the SCORMCloud platform and Teamie collaborative learning
environment were reviewed for: correctness, whether they were attempted, the number of attempts and,
where available, the duration taken to answer.

High-level usage patterns were also collected from the SCORMCloud platform showing the number of
accesses and the duration of each access to the learning material.

This information was then correlated and reflected upon to identify any major themes that required
further investigation in the focus group.

3.4.4.5 Distribution of Student Survey Instrument

After the observation study, but before the focus group, a student survey instrument was administered
to student participants to collect feedback on their use of the e-learning software prototype. The online
survey instrument was distributed according to the same controls and procedures as the VARK learning
styles instrument.

Data from this instrument was collated automatically via the online survey tool, and exported in comma-
separated value (CSV) format. This was then imported into a spreadsheet program for further analysis,
and summarised in descriptive statistics and bar charts showing student feedback frequencies. The
responses to open questions were thematically grouped and evaluated in terms of their value as quotable
comments. A preliminary analysis of the student responses was used to inform the questions in the focus
group.

Further information about the student survey instrument is discussed in section 6.2.4

64
 CHAPTER THREE

3.4.4.6 Focus Group

After the Phase 1 observation study, a focus group was held in which student participants had a facilitated
discussion where they elaborated on the key themes identified in the observation study and the survey
results. The focus group was loosely guided by a PowerPoint presentation that showed each e-learning
screen and listed (visible only to the facilitator) a set of pertinent questions identified during the
observation study, and the survey results. The presentation and questions were not used verbatim;
instead, the screens were used to quickly refresh the memory of the participants, and key questions were
used based on the flow and progress of the focus group.

The focus group duration was 75 minutes and was facilitated by the researcher in the school’s ICT Lab. On
this day the projector was not available to show the screens of the e-learning software prototype;
however, paper screen prints were used for this purpose. The focus group was recorded on video and
later transcribed into a summary format that gave significant detail, but did not transcribe word for word
everything that was said. Student participants are visible in the video but are not referred to by name; the
transcript only refers to pseudonyms. The original video is kept for archive purposes, but is encrypted in
a separate network location.

The focus group transcript was analysed thematically to identify any key themes raised in the discussion,
and evaluated in terms of the value of student responses as quotable comments.

3.4.5 Interpretation
The findings from the e-learning pedagogy review, observation study, VARK learning styles instrument,
student survey instrument and focus group were examined holistically to understand whether the
different research strands converged or diverged, to identify areas requiring further investigation and
areas that gave direction to Phase 2.

The feedback to the research methods is summarised in section 6.3, and the teacher / expert feedback
on the e-learning pedagogy are reported in summary form in 6.4.1, and elaborated thematically in full
form in Vol 2 Appendix A.

Since Phase 1 and Phase 2 are exploratory phases, the findings from student usage of the e-learning
software prototype are merged and reported together. These findings are reported in summarised form
in section 6.5, and in full form in Vol 2 Appendix B; they consolidate the mixed methods strands
(observation, survey and focus group) into a single picture. The findings are predominantly qualitative in
nature, have a small sample size, and are a collection of open and ordinal data (Likert and Ranking);
therefore, a statistical analysis, including significance tests, could not be carried out. However, descriptive
statistics are presented in Vol 2 Appendix B; the descriptive statistics give a basic quantification of results
in the form of bar charts of student feedback frequencies. These are then followed by textual descriptions
that link to open responses from either the questionnaire or focus group, and interpret the result findings.

65
 CHAPTER THREE

3.4.6 Phase 1 – Ensuring Validity and Trustworthiness

The basis of the pedagogical heuristics on well-established learning theories, with a wide pedigree of
supporting research, gives them a strong foundation and means their appropriateness is likely. However,
this is not enough; additionally, the involvement of an experienced teacher and education experts in
evaluating the heuristics encourages their further refinement, and gives an additional verification of their
suitability for computer science and the GCSE age group. Finally, student feedback of the e-learning test
tool gives further feedback as to the fitness of the heuristics for GCSE Computing.

Since Phase 1 has a qualitative priority, the validation strategies used for the aforementioned inquiry are
associated with trustworthiness; meaning a focus on credibility, transferability, dependability and
confirmability. These validation strategies were discussed in general terms in 3.2.4. Before discussing how
these strategies were employed in Phase 1, it should be noted that closely related to the discussion of the
trustworthiness of Phase 1 research methods are the sections discussing the reliability and validity of the
research instruments used (refer to sections 6.2.3.1 and 6.2.4.1).

Creswell (2007) suggests that of eight commonly used procedures to establish trustworthiness, qualitative
researchers should employ at least two. In Phase 1 of the research three of the procedures are used:
triangulation, member checks, and preparing for an inquiry audit.

3.4.6.1 Credibility

A key approach towards establishing the credibility of research findings is the triangulation of multiple
and different sources (Creswell 2007; Lincoln & Guba 1985). This is a central approach within Phase 1
since findings from the observation study, the survey instrument and the focus group are all combined to
form a single picture.

Additionally, a second fundamental approach towards establishing the credibility of research is the use of
member checks. In qualitative terms, this is the process of verifying hypotheses, data, themes, and
interpretations with the research participants that provided them (Creswell 2007; Guba & Lincoln 1989).

This approach was implemented informally by regularly summarising and representing back to the
participants the researcher’s understanding during that interaction, typically during the observation study
and the focus group.

This was implemented more formally by officially presenting back to the research participants the findings
attributed to them, and requesting their confirmation that it is a true account. In Phase 1, this was
originally planned to be accomplished by a presentation of research findings back to the research
participants. Unfortunately, the restricted availability of participants meant this was accomplished by
distributing a report of the findings, which were commented on or confirmed by the participants.

3.4.6.2 Transferability

As discussed previously in section 3.4.2.1, transferability is limited to some extent by the use of high-
performing students from a private school. However, in general, Phase 1 remains quite transferable since:

66
 CHAPTER THREE

1. ICT GCSE level students were used for the research,

2. The research was carried out in typical learning environments; i.e. in an undisturbed ICT school
lab, and later completed in the student’s home environment,
3. All material, instruments and protocols are available for reuse, and
4. The participant feedback is relatively unambiguous and unlikely to be tainted by a biased
researcher account.

3.4.6.3 Dependability and Confirmability

One approach to secure dependability and confirmability is the inquiry audit (Creswell 2007; Lincoln &
Guba 1985); this is conceptually similar to a fiscal audit, and allows for an external auditor to examine
both the process and products of the inquiry to assess whether the findings, interpretations, and
conclusions are accurate and supported by the underlying data.

The researcher will not explicitly request an inquiry audit for this research, but the research was
performed according to detailed methods and a rigorous approach to data collection, data analysis, and
report writing that would facilitate an inquiry audit if one was requested by the supervisory team or the
university.

Lincoln and Guba (1985) align with Halpern’s (1983) research on inquiry audits, and suggest six audit trail
categories; these are used in this research, partially in Phase 1 since it was a pilot study, and more
rigorously in Phase 2. These categories are:

1. Raw data.
2. Data reduction and analysis products.
3. Data reconstruction and synthesis products.
4. Process notes.
5. Materials relating to intentions and dispositions.
6. Instrument development information.

3.5 Phase 2: Elaboration via Action Research

3.5.1 Objectives
The objective of Phase 2 was to build on the findings from Phase 1, and to further refine and elaborate
the e-learning pedagogy and prototype software via an action research methodology. An action research
approach was chosen since it links theory and practice, achieving both practical and research objectives
(Susman 1983). The practical focus lies in the iterative development of the e-learning software and the
research focus on the elaboration and evaluation of the pedagogical heuristics.

67
 CHAPTER THREE

3.5.2 Action Research Cycle

As outlined previously in section 3.2.5, action research follows a cyclical five-stage process; the specific
implementation of this study is represented in Figure 11.

Figure 11: Action research cycle

Two cycles of evaluation and update were included in this phase. A small number of GCSE Computing
teachers and students were recruited from local high-schools to evaluate the e-learning software and the
underlying pedagogy. With each cycle of action research, the e-learning pedagogy was updated based on
the constructive input of experienced teachers, education experts and ongoing literature review. Then,
aspects of the pedagogy were represented in the e-learning software for evaluation purposes. This was
undertaken in a similar way to Phase 1, through storyboard creation followed by development updates to
the e-learning prototype (discussed previously in section 3.2.6). The student evaluations of the e-learning
software were collected via a combination of direct observation of software usage, online survey
instrument, and associated focus groups.

3.5.3 Phase 2 Participants

3.5.3.1 Teachers and Educational Experts – Cycle 1

In Phase 2-Cycle 1, the education expert in the field of curriculum, pedagogy and assessment withdrew
from the study. Additionally, the teacher from Phase 1 withdrew due to maternity leave; she was then
replaced by a novice teacher from a new school. At the time of the study, the teacher had less than one
year of experience in teaching GCSE Computer Science.

In summary, the four remaining education experts and one teacher evaluated the Phase2-Cycle1
pedagogical heuristics.

68
 CHAPTER THREE

3.5.3.2 Students – Cycle 1

The Phase 1 recruitment was supplemented with a Phase 2 recruitment 20 , in which a further eight
students were recruited from year-4 (1st year of ICT GCSE); however, thereafter attendance was voluntary
and based on student availability: all students participated in the observation study, four of those students
responded to the survey, and three of those students attended the focus group.

Previous student participants from Phase 1 now moved to year-5 (2nd year of the ICT GCSE): six students
participated in the observation study, the same six students responded to the survey, and four of those
students attended the focus group. It should be noted that the year-5 students had worked with the
previous version of the e-learning software.

3.5.3.3 Teachers and Educational Experts – Cycle 2

In Phase2-Cycle2, the teacher from Phase2-Cycle1 withdrew from the study, but was replaced by a teacher
with over ten years of experience in Key Stages 3, 4, and 5 computer science, and with additional
experience as a moderator for OCR GCSE Computing.

One of the education experts was unable to give feedback within the requisite timeline; therefore, three
education experts in the fields of computer science, education, and educational media provided their
feedback in Phase2-Cycle2.

3.5.3.4 Students – Cycle 2

In cycle 2, all student participants from cycle 1 were given the opportunity to attend the study; however,
considering the voluntary basis and student availability, Phase2-Cycle2 research results are based on four
students from year-5 (2nd year of the ICT GCSE) and three students from year-4 (1st year of ICT GCSE). Both
the year-5 and year-4 students had worked with the previous version of the e-learning software. From
year-5, four students attended the observation study, three of those students responded to the survey,
and the same three students attended the focus group. From year-4, three students attended the
observation study, two of those students responded to the survey, and the same two students attended
the focus group.

3.5.4 Phase 2 Action Research Activities

Each Phase 2 research cycle was broken down into the following activities. Many of the activities and
instruments in Phase 2 are similar to those in Phase 1; hence, where appropriate, previous sections are
referenced.

Specify Learning (Update Pedagogy)

1. Literature review of learning theories.

20
As discussed in section 3.4.2.1; participant recruitment was undertaken via a presentation to both the students
and their guardians accompanied by information letters and consent forms.

69
 CHAPTER THREE

2. Research on GCSE Computer Science curriculum – focus on algorithms and computational

thinking.
3. Refine the pedagogical heuristics (refer to section 3.5.4.1).
4. Evaluation of pedagogical heuristics from teacher and education experts (refer to section
3.5.4.2).

Diagnosing

5. Specify what aspects of the pedagogical heuristics are to be evaluated in this cycle.

Planning Action

6. Secure ethics approval for Phase 2 research in schools.

7. Secure informed consent from headteacher, teachers, students and parents.
8. Plan and schedule the activities of the observation study with schools.

Taking Action

9. Update the instructional design of the e-learning Software – Storyboards (refer to section 3.2.6)
10. Update and test the e-learning software prototype (refer to section 3.2.6).

Evaluating Action

11. Distribute the VARK learning styles questionnaire. This is the same as Phase 1 (refer to section
3.4.4.3).
12. School-based observation of the usage of e-learning software (refer to section 3.5.4.3).
13. Distribute survey instrument. This is the same as Phase 1 (refer to section 3.4.4.5).
14. Undertake focus group (refer to section 3.5.4.4).
15. Merge and interpret results from different research strands.
16. Provide feedback back to participants (member checking) to verify that researcher interpretation
is correct.

A mixed methods procedural diagram is shown in Figure 12, which gives a visual representation of points
4 and 11 to 15.

70
 CHAPTER THREE

Phase 2 Mixed Methods Convergent Design

Procedures Procedures Procedures

* n Education Experts & n Teacher Observation of E- * n students used software at * n students
Survey Instrument &
Evaluation of Pedagogy * Pedagogy evaluation instrument Learning Software school and at home. * Online VARK questionnaire and
VARK Questionnaire
QUAL provided to evaluators usage * Observation of school software student survey instrument
QUAN+qual
data collection * Evaluation and constructive QUAL+quan usage. administered
data collection
feedback on Pedagogy. data collection
Product Product
Product * Observation notes * Quantitative VARK scores for
Completed Pedagogy evaluation * Usage statistics from SCORM each person.
instrument and/or in place Cloud. * Quantitative survey scores
comments within the pedagogy * Student answers on SCORM * Qualitative answers to open
document from each evaluator. Cloud and Google Apps for Ed. questions
* Updates into Inquiry Reflection
Procedures Journal. Procedures
* Summarise into high-level * Summarise into descriptive
QUAL feedback points. Procedures QUAN+qual statistics.
Data analysis * Detailed corrections to * Reflect on observation notes, Data analysis * Create themes for open
Pedagogy document. QUAL+quan student answers and usage answers
Data analysis statistics. * Correlate between VARK,
Product Survey and open questions.
* List of feedback areas for the Product
pedagogy * Clean observation notes Product
* Corrections to pedagogy * Summarised usage statistics * VARK learning style preference
document. from SCORM Cloud. per person and summarised for
* Updates into Inquiry Reflection * Student answers on SCORM group.
Journal. Cloud and Google Apps. * Descriptive statistics for survey.
* Updates into Inquiry Reflection * Open answers categorised and
Journal. useful quotes identified.
* Updates into Inquiry Reflection
Journal.
Procedures
* Correlate the 4 strands of
results and findings. Merge the Results

Product Procedures
* Intermediate merged results * N students
Focus Group * Update focus group questions
and discussion
QUAL based on survey results.
* Updates into Inquiry Reflection
data collection * Semi-structured focus group
Journal.

Product
Procedure Product * Video
*Consider how merged results *Discussion and Phase 2 Results * Focus Group transcript
produce a better understanding – Interpretation thesis chapter
how do results converge and * List of Improvements to
diverge. research methods and research
Procedures
*Evaluate if further research is instruments.
* Summarise focus group
necessary to understand *List of Improvements and gaps
QUAL responses into major themes.
divergent results. areas in Pedagogical Heuristics
Data analysis * Useful quotes identified.
*List of Improvements for E-
Learning prototype.
Product
* Updates into Inquiry Reflection
* Summary of major themes
Journal.
identified in focus group.

Figure 12: Phase 2 - mixed methods convergent design

71
 CHAPTER THREE

3.5.4.1 Phase 2 Synthesis of Pedagogical Heuristics

Based on the findings of Phase 1, a focused literature review was undertaken resulting in the refinement
of existing heuristics and the addition of new e-learning pedagogical heuristics for GCSE Computing. This
formed the basis of the Phase2-Cycle1 e-learning pedagogy.

Based on the findings of Phase2-Cycle1, a focused literature review was undertaken resulting in the
refinement of existing e-learning pedagogical heuristics for GCSE Computing. This formed the basis of the
Phase2-Cycle2 e-learning pedagogy.

As previously outlined, the pedagogy literature review is outlined in section 2.4 and the e-learning
pedagogical heuristics are discussed in Chapter Four.

3.5.4.2 Evaluation of Pedagogical Heuristics from Teacher and Education Experts

The composition of the pedagogy evaluation group in Phase2-Cycle1 is outlined in section 3.5.3.1.
Evaluators were given up to two months to review the e-learning pedagogy document and feedback was
provided in the Pedagogy Evaluation Feedback document (refer to section 6.2.1) and/or inline comments
and corrections within the pedagogy document. Where the feedback was unclear to the researcher, then
one-to-one interviews were used to clarify.

The evaluation feedback was analysed and incorporated into the research at the start of Phase2-Cycle2.
This pedagogical feedback is summarised in sections 6.4.2, and reported in full detail in Vol 2 Appendix A.

For Phase2-Cycle2, a similar approach was undertaken, which informed Phase 3. The composition of the
pedagogy evaluation group in Phase2-Cycle2 is outlined in section 3.5.3.3. This pedagogical feedback is
summarised in section 6.4.3 and reported in full detail in Vol 2 Appendix A.

3.5.4.3 Observation Study

For each Phase 2 cycle, after the e-learning software prototype was updated, an observation study was
organised by the researcher to observe student usage of the e-learning software. The e-learning software
test tool is discussed in section 6.2.2, and the specific versions used in cycle 1 and cycle 2 are discussed in
sections 6.2.2.2 and 6.2.2.3, respectively.

Based on lessons learnt from Phase 1, a more detailed technical environment specification (refer to
section 6.2.5) was distributed to the schools and the participants prior to the observations. This was to
ensure participants used the e-learning software in an appropriate environment setup. Likewise, a more
rigorous set of technical checks were undertaken in each school’s ICT lab to ensure their adherence to the
technical environment specification.

Due to a school scheduling conflict, the two student groups participating in Phase 2 could not be
scheduled for a single observation study, thus two observation studies took place. However, both
observations were organised on school premises in the same ICT lab; the observations started at 13:30,
after the students had completed their school day and had time for a short break. Fifteen minutes was

72
 CHAPTER THREE

allocated to an introduction and organising the participants; then the student participants used the e-
learning software prototype for a period of 1 hour and 10 minutes, allowing the research investigator to
observe the direct usage of the software.

At the end of the observations, the students were asked to progress and complete the remaining sections
of the e-learning software asynchronously from home.

To minimise any potential bias and differences between the groups, instructions and basic training was
provided to each group of participants via a standard template; thereafter, the researcher provided
minimal guidance and support.

Notes from each observation study were documented informally in handwritten notes, and transcribed
later after each observation.

The student answers posted onto the SCORMCloud platform and the Google Apps for Education
collaborative learning environment were reviewed for: correctness, whether they were attempted,
number of attempts and, where available, the duration taken to answer.

High-level usage patterns were also collected from the SCORMCloud platform showing the number of
accesses and the duration of each access to the learning material.

This information was then correlated and reflected upon to identify any major themes that required
further investigation in the focus group.

3.5.4.4 Focus Group

The Phase 2 focus groups followed a similar approach to that detailed in Phase 1; refer to section 3.4.4.6.
Specific details of the Phase 2 focus groups are as follows:

1. Phase2-Cycle1: This cycle was broken into two focus groups since the year-5 group had previous
experience of the e-learning software.
a. Year-5 Group: The focus group duration was 88 minutes and was facilitated by the
researcher in the School’s ICT Lab. The projector was available to show the screens of
the e-learning software.
b. Year-4 Group: The focus group duration was 70 minutes, was facilitated by the
researcher in the School’s ICT Lab. The projector was available to show the screens of
the e-learning software.
2. Phase2-Cycle2: The focus group duration was 75 minutes and was facilitated by the researcher
in the school’s ICT Lab. Year-4 and year-5 students were merged into a single focus group. The
projector was not available to show the screens of the e-learning software, but paper-based
screen prints were used instead.

The focus group transcripts were analysed thematically to identify any key themes raised in the discussion,
and evaluated in terms of the value of student responses as quotable comments.

73
 CHAPTER THREE

3.5.5 Interpretation
As with Phase 1, the findings from the e-learning pedagogy review, observation study, VARK
questionnaire, student survey and focus group were examined holistically to understand whether the
different research strands converge or diverge, to identify areas requiring further investigation and areas
that give direction to the remainder of the study.

The teacher / expert feedback on the e-learning pedagogy are reported in summary form in 6.4.2 and
6.4.3, and elaborated thematically in full form in Vol 2 Appendix A.

As discussed in section 3.4.5, the Phase 1 and 2 findings from student usage of the e-learning software
prototype are merged and reported together. The findings are reported in summarised form in section
6.5, and in full form in Vol 2 Appendix B.

3.5.6 Phase 2 – Ensuring Validity and Trustworthiness

The approaches to trustworthiness utilised in Phase 1 were also used in Phase 2, for a discussion of these
approaches please refer to section 3.4.6. However, in Phase 2 some additional factors and activities
further strengthen the credibility, dependability and confirmability of the study.

Considering the researcher’s involvement with some participants from Phase 1 onwards and the
remaining participants in two cycles of action research, the strengthening of Credibility is achieved via
prolonged engagement and persistent observation (Creswell 2007; Guba & Lincoln 1989; Lincoln & Guba
1985). Prolonged engagement builds trust with participants, which in turn helps in identifying and
considering distortions and misinformation that might creep into the study. Additionally, the persistent
observation of participants helps to identify characteristics of the situation most pertinent to the area of
study, and to allow a follow up on those areas in more detail.

In Phase 2, an additional activity to maintain a reflective journal was undertaken; this is an approach used
often in action research (O’Brien 2001; Taylor 2006) and supports all aspects of trustworthiness, but
specifically is a major technique in supporting confirmability (Lincoln & Guba 1985).

The reflective journal is a kind of diary of the research progress and includes a variety of information about
the researcher and the thinking that underpins the direction of the research and any methodological
decisions. This information can then potentially be used by an inquiry auditor to get a better
understanding of the rationale used in the research and, in particular, whether the researcher’s biases
have influenced the research findings.

There is no widely accepted format or structure for a reflective journal; however, based on suggestions
from Taylor (2006) and Lincoln and Guba (1985), the researcher included a subset of the following
information:

1. Date,
2. Daily schedule and logistics of the study,
3. Personal Diary providing an opportunity for catharsis and reflection, and

74
 CHAPTER THREE

4. Methodologic log in which methodological decisions and accompanying rational are

recorded (refer to section 3.4.6.3).

3.6 Phase 3: Final Experiment

3.6.1 Objectives
Whereas the previous mixed methods phases had a qualitative priority, Phase 3 used a convergent parallel
mixed methods design with a quantitative priority. As previously outlined, the purpose of this phase was
to further validate the findings of the first two phases by confirming them using quantitative methods,
and attempting to generalise them to a wider population. Specifically, this phase measured whether
adherence to the e-learning pedagogy in the design of e-learning software influenced student learning
and/or motivation.

3.6.2 Phase 3 Participants

3.6.2.1 Teachers and Educational Experts

In Phase 3, the teacher from Phase2-Cycle2 withdrew from the study, but was replaced in the e-learning
pedagogy review by another experienced computer science teacher.

Two education experts in the fields of computer science, and education and educational media provided
their feedback on the Phase 3 pedagogical heuristics.

3.6.2.2 Students

The Phase 3 study comprised two UK comprehensive schools offering the Computer Science GCSE (2016
Specification). At the time of the study (January 2017), student participants were starting the 1 st year of
the GCSE. Table 7 summarises the Phase 3 student participants.

School Participants Male Participants Female Participant

School 1 34 34 0

School 2 32 21 11

Total 66 55 11
Table 7: Phase 3 - student participants

Student participants are evenly split between the schools and the overall number of female participants
(17%) is broadly aligned with national statistics published by the Department for Education (DfE) (2017),
reflecting 20% of students entered in the 2015/2016 Computer Science GCSE were female.

The participating schools were self-selecting, based on the teachers’ interest in participating in the
research. However, the self-selection process did not extend to the student participants; after providing
informed consent, all students undertaking the 1st year of the Computer Science GCSE participated in the
study. Therefore, there is a random cross-section of student ability as represented by the teachers’ Key

75
 CHAPTER THREE

Stage 4 (KS4) predictions shown in Figure 13. KS4 predictions range from 1 (A*), 8 (A) to 4 (E-F) with a
mean between a Standard and Strong Pass (M=4.29 SD=1.50) and median of Standard Pass (Mdn=4).

(DFE 2017)

Figure 13: Phase 3 - frequency distribution of Key Stage 4 predictions.

Considering these sample characteristics, the Phase 3 student participants are deemed representative of
the wider GCSE Computer Science population.

3.6.3 Phase 3 Experiment Design

In Phase 3, quantitative and qualitative methods were used to collect and analyse data, and the results
were integrated into a single view; as such, the final experiment is classified as a mixed methods
triangulation approach (Borrego et al. 2009).

The quantitative experiment measured whether there were improved student assessment results from e-
learning software that is designed in adherence with the pedagogical heuristics synthesised in this
research. This leads to the following experiment hypothesis:

• H0: The use of e-learning software designed in adherence to the pedagogical heuristics presents
no difference in student learning performance.
• H1: The use of e-learning software designed in adherence to the pedagogical heuristics presents
improved student learning performance.

The e-learning software prototype developed in this research study was critically evaluated against the
heuristics to identify its pedagogical value. This evaluation provides a quantifiable measure of the e-
learning Software’s adherence to the pedagogical heuristics (refer to section 6.2.2.4).

Before the experiment execution period began, the participating high-school teachers administered to
their students a standardised pre-test to act as a baseline for learning. Directly after the experiment, the
same test was administered as a post-test. Statistical analysis of pre- and post-test results was undertaken
to identify whether learning occurred, and if it was statistically significant. Furthermore, parametric

76
 CHAPTER THREE

statistical analysis was used to theorise the potential learning performance possible within the Computer
Science GCSE from use of the software prototype.

The quantitative analysis aimed to give statistically significant evidence that the use of e-learning software
with increased adherence to the pedagogical heuristics presents an improved student learning
performance. However, this does not give clear insight into the underlying reasons for the results; this
insight is given by the online distribution of the student survey instrument. This survey instrument
collected both quantitative and qualitative data. This was further supplemented by the online distribution
of the VARK questionnaire, previously used in Phases 1 and 2. The mixture of quantitative and qualitative
data sources enabled the triangulation of student feedback with assessment performance, thereby
putting the quantitative performance results in context.

3.6.4 Phase 3 Research Activities

1. Literature review of learning theories.
2. Research on GCSE Computer Science curriculum – focus on algorithms and computational
thinking.
3. Refine the pedagogical heuristics.
4. Evaluation of pedagogical heuristics from teacher and education experts (refer to section
3.6.4.2).
5. Update the instructional design of e-learning software – Storyboards (refer to section 3.2.6).
6. Update and test e-learning software prototype (refer to section 3.2.6).
7. Secure ethics approval for research in schools.
8. Secure informed consent from headteacher, teachers, students and parents.
9. Distribute VARK learning styles instrument (refer to section 6.2.3).
10. Administer student pre-test (refer to section 6.2.6 and 3.6.4.4).
11. School-based usage of e-learning software (refer to section 3.6.4.3).
12. Administer student post-test (refer to section 6.2.6 and 3.6.4.4).
13. Distribute the survey instrument. This is the same as Phase 1 (refer to section 3.4.4.5).
14. Merge and interpret results from different research strands.

A mixed methods procedural diagram is shown in Figure 14, which gives a visual representation of points
4 and 9 to 14.

77
 CHAPTER THREE

Phase 3 Mixed Methods Convergent Design

Procedures Procedures Procedures

* n Education Experts & n Teacher Experiment Usage of E- * n students undertake pre-test. Survey Instrument & * n students
Evaluation of Pedagogy * Pedagogy evaluation instrument * n students used software at * Online VARK questionnaire and
Learning Software VARK Questionnaire
QUAL provided to evaluators school and undertake post-test. student survey instrument
QUAN QUAN+qual
data collection * Evaluation and constructive administered
data collection data collection
feedback on Pedagogy. Product
* Usage statistics from SCORM Product
Product Cloud. * Quantitative VARK scores for
Completed Pedagogy evaluation * Student answers on SCORM each person.
instrument and/or in place Cloud and Google Apps for Ed. * Quantitative survey scores
comments within the pedagogy * Updates into Inquiry Reflection * Qualitative answers to open
document from each evaluator. Journal. questions
* Quantitative scores from Pre
and Post-Test

Procedures Procedures Procedures

* Summarise into high-level * Summarise, learning duration, * Summarise into descriptive
QUAL feedback points. QUAN and Pre- and Post-Test results QUAN+qual statistics.
Data analysis * Detailed corrections to Data analysis into descriptive statistics. Data analysis * Create themes for open
Pedagogy document. * Parametric statistical analysis to answers
identify whether pre-test and * Correlate between VARK
Product post test results indicate learning. questionnaire, student survey
* List of feedback areas for the Note: Non-Parametric tests will and open questions.
pedagogy be used if assumptions for * Parametric statistical analysis
* Corrections to pedagogy parametric tests are not met. comparing the student
document. Product perception of the E-Learning
* Updates into Inquiry Reflection * Parametric statistical analysis software under test.
Journal. relating to student learning.
* Final Pedagogical Heuristics * Summarised usage statistics Product
document from SCORM Cloud. * VARK learning style preference
* Student answers on SCORM per person and summarised for
Cloud and Google Apps. group.
* Updates into Inquiry Reflection * Descriptive statistics per group
Procedures Journal. for student survey.
* Correlate the 3 strands of * Open answers categorised and
results and findings. Merge the Results useful quotes identified.
* Results of parametric statistical
Product analysis comparing the student
* Intermediate merged results perception of the E-Learning
and discussion software under test.
* Updates into Inquiry Reflection * Updates into Inquiry Reflection
Journal. Journal.

Procedure Product
*Consider how merged results *Phase 3 results chapter
produce a better understanding – Interpretation * Thesis research findings
how do results converge and discussion.
diverge.
*Evaluate if further research is
necessary to understand
divergent results.

Figure 14: Phase 3 mixed methods convergent design

78
 CHAPTER THREE

3.6.4.1 Phase 3 Synthesis of Pedagogical Heuristics

Based on the findings of Phase2-Cycle2, a focused literature review was undertaken resulting in the
refinement of existing e-learning pedagogical heuristics for GCSE Computing. This formed the basis of the
Phase3 e-learning pedagogy.

As previously outlined, the pedagogy literature review is outlined in section 2.4, and the e-learning
pedagogical heuristics are discussed in Chapter Four.

3.6.4.2 Evaluation of Pedagogical Heuristics from Teacher and Education Experts

The composition of the pedagogy evaluation group in Phase3 is outlined in section 3.6.2.1. The evaluation
of the pedagogical heuristics followed the same procedures (refer to section 3.5.4.2) and instrument
(refer to section 6.2.1) as used in Phase2-Cycle2.

Based on the pedagogy’s mature state, the feedback was minimal, but was incorporated into the final
version of the e-learning pedagogy document. This pedagogical feedback is summarised in section 6.4.4,
and the final e-learning pedagogy document is in Vol 2 Appendix D.

3.6.4.3 School-based Use of E-learning Software

A key component of the Phase 3 research was the e-learning software usage experiment. After the four
levels of the Phase 3 e-learning software prototype were developed (refer to sections 6.2.2 and 6.2.2.4),
they were provided to two UK comprehensive schools to be incorporated into their teaching of the GCSE
Computer Science topics of algorithms and computational thinking.

The experiment was organised on school premises in each school’s ICT lab; however, to ensure uniformity
across experiment sites a detailed technical environment specification was distributed to the schools and
used to check and confirm environment conformance (refer to section 6.2.5).

The experiment was facilitated by the teachers associated with each school; however, to minimise any
potential bias and differences between the groups, the same instructions and training was provided to
the teachers. Additionally, a Phase 3 Research Protocol Confirmation document was distributed to the
teachers ahead of the study; this documented the steps and activities to be followed during the study
(refer to section 6.2.7). The teachers were requested to provide minimal additional guidance to students,
and to not provide additional learning material or teaching.

In keeping with heuristic 19: Optimise essential processing by segmenting learning material and providing
pre-training, to ensure the students were able to effectively use the software, a quick tips sheet was
provided which outlined e-learning software functionality. This was further supplemented by tutorial
videos in the e-learning software.

This experiment was not treated as an observation study; hence, there are no experiment research notes.
The student answers posted onto the SCORMCloud platform and Google Apps for Education were

79
 CHAPTER THREE

reviewed for: whether they were attempted, number of attempts and, where available, the duration
taken to answer.

High-level usage patterns were also collected from the SCORMCloud platform showing the number of
accesses and the duration of each access to the learning material.

The e-learning software contained approximately five hours of learning material; both schools completed
the teaching engagement in two weeks of in-class and homework sessions. School1 used four in-class
sessions and School2 used five in-class sessions; an analysis of student learning time is presented in
section 6.6.7.2. The teachers returned the Research Protocol Confirmation document to confirm the
research procedures were followed, and notified that the experiment and test results were not impacted
by any exceptional circumstance. Finally, at the end of the session, the students undertook the post-test.

3.6.4.4 Pre- and Post-Test

As outlined previously, before the experiment execution period began, the participating high-school
teachers administered to their students, under exam conditions, a pre-test. Exam conditions are defined
by the following test controls:

• 35 minutes are allocated to answer all questions.

• The examination environment is quiet, comfortable, and without any notable distractions.
• Students are seated equidistantly spaced to avoid copying.
• Teacher acts as invigilator to avoid students using unfair means (copying, etc.).
• Student Instructions are:
o Use black ink or ballpoint pen. Use pencil only for drawing.
o Answer all questions.
o You must answer the questions in the spaces provided.
o If you make a mistake, cross through the incorrect answer.
o Do all rough work in this book. Cross through any work that you do not want to be
marked.
o You are free to answer questions that require a coded solution in whatever format you
prefer as long as your meaning is clear and unambiguous.
o You must not use a calculator.

This pre-test provided a baseline for learning performance. Directly after the experiment period, the same
test was administered under the same exam conditions as a post-test. This ensured that the pre- and post-
test results are directly comparable. The test instrument is discussed in detail in section 6.2.6.

As discussed in 6.2.6.1, to reduce the possibility of the participants remembering the test between pre-
and post-tests, there was a minimum 2-week delay between the pre- and post-tests. However, to reduce
the potential for unintended learning, the time between pre-test, experiment, and post-test was kept
minimal (less than three weeks). Finally, before the post-test, students communicated whether they had

80
 CHAPTER THREE

accessed other learning material on the subject matter, thereby explicitly counteracting the issue of
unintended learning.

3.6.5 Data Collection, Analysis and Reporting

In Phase 3 there was an increased focus on quantitative results and statistical analysis. As standard
practice in quantitative analysis, a codebook was created to keep a meticulous log of any pre-treatment,
variable coding, decisions, and statistical tests taken in Phase 3 (Pallant 2001; Litwin 1995). For reference,
this codebook is included in Vol 3 Appendix M.

This section provides a high-level discussion of some of the keys decisions and approaches taken in the
Phase 3 statistical analysis. Also, during Phase 3, certain decisions were taken during the statistical
analysis that may at a first view seem to be non-standard. This section also discusses the theoretical
foundation for those decisions.

3.6.5.1 Parametric vs Non-Parametric Tests

Statistical procedures are broadly split into two groups: parametric and non-parametric tests. Parametric
tests require ratio or interval data and have a number of assumptions that should be met, in particular
relating to the distribution of the data. Non-parametric tests are applicable to all four measurement
scales, are less restrictive in their assumptions, and are distribution free (MacKenzie 2013; Field 2013;
Pallant 2001). The consensus is that by virtue of their superior power 21, as long as their assumptions are
reasonably met, parametric tests should be preferred. However, there is academic debate on what to do
when the assumptions for parametric tests are not met; MacKenzie (2013) offers three options:

1. Proceed with parametric tests.

2. Correct the violation by cleaning or transforming the data, and then proceed with parametric
tests.
3. Proceed with non-parametric tests.

Some researchers argue that parametric tests are reasonably robust to assumption violations (Box 1953;
Norman 2010), and this can offset the reduction in power from using non-parametric tests. Field (2013)
offers a balanced discussion on data transformation and cleaning, in that it can be a valuable tool in certain
scenarios, but can also add an additional layer of complexity which, if ill-considered, can negatively impact
the statistical model being used. Also, in the worst case, data transformation and cleaning could be viewed
as a convenient tool to manipulate the data to fit the research objective. The third option, MacKenzie
(2013) argues, is incorrectly becoming a de-facto standard: to jump straight to non-parametric tests. He
offers a convincing review of HCI research in which researchers have moved straight to non-parametric
tests even on interval and ratio data, without considering the impact on reduced statistical power. Citing

21
Power meaning a greater ability to draw the right statistical conclusions i.e. rejecting the null hypothesis where
appropriate, or not if there is insufficient basis to do so.

81
 CHAPTER THREE

two examples (Perugini et al. 2007; Munteanu et al. 2006), MacKenzie (2013) discusses the praiseworthy
approach, to consider both parametric and non-parametric analysis and adjust accordingly if there is any
disparity.

Field (2013) also offers another partial solution, which is underpinned by the prevalent usage of statistical
software and computational power: the Bootstrap procedure. The Bootstrap option overcomes issues of
non-normal sample data; it estimates the sampling distribution by treating the sample data as a
population. Field (2013, p.199) succinctly discusses the approach: “The sample data are treated as a
population from which smaller samples (called bootstrap samples) are taken (putting each score back
before a new one is drawn from the sample). The parameter of interest (e.g., the mean) is calculated in
each bootstrap sample.” This process can be repeated (i.e. 1000 times) to build an accurate confidence
interval for the parameter under analysis.

Building on MacKenzie (2013) and Field (2013) this research will often report results from parametric and
non-parametric tests, and make extensive use of the bootstrap option. In summary, the following strategy
was adopted when research data did not fully meet the assumptions for parametric tests:

1. Parametric tests were preferred for ratio and interval variables,

2. Where assumptions were not met for parametric tests; typically, due to a non-normal
distribution, then:
a. The bootstrap option was employed, and
b. Appropriate non-parametric tests were employed in parallel to confirm the parametric
and bootstrap tests.

3.6.5.2 Correlation analysis

Within Phase 3, there was extensive use of correlation analysis to identify relationships between
experiment variables. The Pearson product-moment correlation coefficient (parametric) and the
Spearman rank correlation coefficient (non-parametric) are two widely used tests for the correlation
between two variables. There is some debate on the comparative power of each of the tests (de Winter
et al. 2016), and various guidelines on which test is best suited in a given scenario (Pallant 2001; Field
2013; de Winter et al. 2016; Fowler 1987). However, the consensus is that Pearson’s r can be impacted by
a non-normal distribution (how big an impact is in debate) and is strongly impacted by outliers (Pallant
2001; Field 2013; de Winter et al. 2016). Whilst Spearman’s Rho is distribution-free, has minimal impact
from outliers, but can be strongly affected by small sample sizes (Pallant 2001; Field 2013). Pallant (2001,
p.133) warns that “in small samples n=30, you may have moderate correlations that do not reach
statistical significance at the traditional p< .05.”

To compensate for this, Pallant (2001) recommends that you still need to report the statistical significance,
but the focus should be more on the strength of the relationship. Field (2013) offers similar guidance,
and goes further in recommending a focus on the Bootstrap Confidence Interval (2013).

82
 CHAPTER THREE

In summary, the correlation analysis in Phase 3 was undertaken using Spearman’s Rho, including outliers.
The correlation coefficient used to measure the size of the effect was evaluated in terms of Cohen’s
(Cohen 1988; Pallant 2001; Field 2013) small, medium and large effect, where:

• Small r=.10 to.29

• Medium r = .30 to .49
• Large r = .50 to 1.0

Significance is reported, but is not used as cut-off; instead, emphasis is placed on the strength of the effect
and the bootstrap confidence interval. Correlation results are reported in two tiers that reflect the
strength of the result:

1. Tier 1: A correlation effect of ±.1 or more, and p < .05

2. Tier 2: A correlation effect of ±.1 or more, and the Bootstrap Confidence Interval (CI) does not
cross over the 0 threshold (i.e. CI is between -.09 and 1 or .09 and -1)

3.6.5.3 Treatment of Outliers

For the purposes of this study, outliers are defined as scores that are 1.5 x Interquartile Range. They were
identified by a combination of histograms and boxplots. As a general approach, outliers were first verified
as not being transposition errors. Additionally, they were investigated against other variables to try to
identify an explanation of the result. Examples of variables that were investigated against outliers include:

1. Duration of engagement with the e-learning software,

2. Pre-Test / Post-Test results,
3. Open question responses,
4. Experience with e-learning software and their use of the internet, and
5. Key Stage 4 predictions.

According to Cousineay & Chartier (2011), there is academic discussion and various recommendations on
how to deal with outliers, but there is no universally accepted approach. Although outliers can have a
potentially significant impact on parametric tests (Field 2013), the removal of unexplained outliers can
cause an underestimate of the true variability of the data, and can negatively affect representativeness
(de Winter et al. 2016; Pallant 2001). Furthermore, alternate, approaches to transform outlier data can
also have an unintended impact on the statistical analysis (Cousineay & Chartier 2011).

In summary, this research primarily reports descriptive statistics including outliers, but also reports, as a
reference, descriptive statistics without outliers. This approach indicates the potential impact of the
outlier(s) and gives input on whether subsequent statistical tests should be carried out, with or without
outliers.

3.6.5.4 Descriptive Statistics

The direction set in the previous sections means that often parametric and non-parametric tests are run
in parallel. As such, descriptive statistics will typically cover both mean and median, and their associated

83
 CHAPTER THREE

measure of spread. Meaning descriptive statistics will take the form of summary tables and histogram
diagrams that focus on mean, median, interquartile range, upper and lower limits, standard error, and
standard deviation.

In addition, skewness and kurtosis will be reported, as well as results from the Kolmogorov-Smirnov test;
this will give clarity on the distribution of the data, and feed into decisions on which statistical test takes
priority and are reported.

3.6.5.5 Pre-Test Post-Test Analysis

The pre-, post-test statistical analysis focuses primarily on the following variables: Key Stage 4 Indicator,
Level-1 Duration, Level-2 Duration, Level-3 Duration, Level-4 Duration, Level-3 and -4 Combined Duration,
All Levels Combined Duration, %Pre-Test, %Post-Test, and %Change.

Duration variables were exported from SCORMCloud or summed, based on the exported values. Key Stage
4 Indicators were provided directly by the teachers. %Pre-Test and %Post-Test are the percentage values
of the agreed moderated marks for each of the students, and %Change is the difference between %Post-
Test and %Pre-Test.

Prior to any statistical analysis, the following data cleaning steps were carried out on pre-, post-test data:

1. Removal from pre-test / post-test of all students who did not get a result for both tests. This step
ensures two controlled test results are recorded that can be used to measure learning
performance.
2. Removal from pre-test / post-test of all students who answered yes in the survey instrument to
using other learning materials. This step ensures that the post-test measures learning derived
from e-learning software and CLE activities, and avoids the potential for bias brought about by
unintended additional learning.
3. Removal of all students from pre-test / post-test who spent zero time on both Level-3 and Level-
4. The material tested in the pre-, post-test is directly related to Level-3 and Level-4; the inclusion
of students who had spent zero time on these levels would include an unintentional bias to the
learning performance results.

The pre-, post-test variables were first summarised by descriptive statistics. Where the assumptions for
parametric testing were met, then a Paired Sample T-Test was carried out between %Post-Test and %Pre-
Test.

Where the assumptions for parametric testing were not met, then the same Paired Sample T-Test was
carried out with Bootstrapping, and a Wilcoxon Signed-Ranks Test was carried out to verify.

One-Sample T-Tests were carried out on %Pre-Test and %Post-Test to map each of them towards the
grade boundaries for the Computer Science GCSE (2016 Specification).

Statistical inference is reported to a 95% confidence interval in order to be able to generalise towards the
wider GCSE Computer Science population.

84
 CHAPTER THREE

3.6.5.6 Survey Instrument Analysis

As in previous phases, the data from the student survey instrument was collated automatically via the
online survey tool, and exported in comma-separated value (CSV) format. This was then imported into a
spreadsheet program for data preparation and finally transposed to a statistical package for further
analysis.

The survey instrument responses were reported as descriptive statistics. Since the instrument focuses
primarily on the students’ perception of the e-learning prototype, there was no expectation to generalise
to the wider GCSE population, and no expectation of a normal distribution. However, several variables
from the survey instrument, including the IMMS responses, were used in correlation analysis against
%Pre-Test, %Post-Test and %Change variables.

The responses to open questions were thematically grouped and evaluated in terms of their value as
quotable comments.

As discussed in section 6.2.4.1, analysis was performed to verify alternate-form and internal consistency
reliability in the Phase 3 survey instrument.

3.6.5.7 Instructional Materials Motivation Survey (IMMS) Analysis

As standard practice, the IMMS instrument was reported on the motivational sub-categories and not the
36 individual questions (J. M. Keller 1987b; Keller 2006; Cook et al. 2009). Motivation sub-categories of
attention, relevance, confidence and satisfaction were:

1. reported as descriptive statistics,

2. assessed on whether the inter-group results had a statistically significant difference, and
3. assessed for internal reliability using Cronbach’s alpha.

3.6.5.8 VARK Learning Styles Analysis

In Phase 3, the approach used for data collection and the preliminary analysis of VARK learning styles
responses remains the same as is described in 3.4.4.3. However, in addition, the responses were imported
to a statistical analysis package to assess whether the inter-group results for the modal sub-categories
have a statistically significant difference.

3.6.6 Interpretation
As with the previous phases, in Phase 3, the two data sets were collected and analysed separately and
independently from each other using typical quantitative and qualitative procedures. After the data had
been analysed, the results were merged and interpreted to identify to what extent, and in what ways, the
two sets of results converged, diverged or related to each other. In Phase 3, the interpretation was based
primarily on quantitative results, and qualitative data was used to give an underlying explanation of these
results.

85
 CHAPTER THREE

After the Phase 3 results were analysed (refer to section 6.6), there followed a final interpretation and
discussion of how the findings of Phase 1 and Phase 2 converged or diverged from the results of Phase 3,
and what had been learnt from the combination of results from all three phases (refer to Chapter Seven).

3.6.7 Phase 3 – Ensuring Validity and Trustworthiness

Since Phase 3 had a quantitative priority, the validation strategies used for the aforementioned
experiment are associated with Internal Validity, External Validity (Generalisability), Reliability and
Objectivity.

To ensure that the above measures of validity were achieved, several controls were implemented to try
to rule out alternative explanations of the experiment results, ensure the experiment was objective, can
be repeated and can be generalised to the wider population in question.

1. Research instruments and procedures were documented and available for review, audit and
reuse in a repeat of the experiment. Reliability and validity of research instruments is discussed
in sections 6.2.3.1, 6.2.4.1, and 6.2.6.1.
2. As part of the development of the research instruments and procedures, they were reviewed by
other experienced researchers and vetted as part of the ethics approval process, thereby
ensuring there is no inherent bias.
3. Experiment procedures were standardised by the use of standard experiment instructions and
training. Additionally, the research procedures were confirmed by teachers in the Research
Protocol Confirmation document.
4. The experiment environment was generalisable to the student’s typical learning environment
since it is their School’s ICT lab.
5. The ICT labs were confirmed quiet, comfortable, clean and tidy and without any notable
distractions.
6. A feedback process was instigated in case any distracting event occurs during experiment
execution.
7. The learning measurement instrument (pre- and post-test) was accurate and targeted at the right
level since it was based on examination material from the relevant examination boards.
8. Learning was measured accurately by using the same test for the pre-test and for the post-test,
but allowing a minimum of two weeks between to give students time to forget the test. It should
be noted that the test was problem-based, therefore difficult to memorise, the students were
not given the correct answers and were not informed they would take the same test as a post-
test.
9. To reduce the potential for unintended learning the time between pre-test, experiment and post-
test was kept minimal (no more than 3 weeks).
10. Before the post-test, students communicated whether they had accessed other learning material
on the subject matter.

86
 CHAPTER THREE

11. The student sample was selected from GCSE Computer Science students and was representative
of the target population (refer to section 3.6.2.2).
12. The use of a non-trivial sample (n = 66) helps to improve confidence that the experiment is
generalisable to the overall population.

3.7 Research Methods for the Development of the E-learning

Evaluation Protocol
At the outset of this research, the development of an e-learning evaluation protocol was not envisaged
as a research objective. However, the Phase 1 Pilot identified a clear need for such an instrument for use
in both formative and summative assessment. It was at this point that the development of an e-learning
evaluation protocol became a secondary research objective.

The development of the e-learning evaluation protocol was loosely synchronised with the qualitative
research methods and timelines used to develop the e-learning pedagogy; they intersected in Phase 1,
Phase2-Cycle2 and Phase 3. This intersection was due to:

1. The evaluation protocol being based on the pedagogical heuristics; hence, it was updated in
parallel to their evolution.
2. Furthermore, the e-learning software prototype was used as a test tool for the piloting and usage
of the evaluation protocol.

3.7.1 Phase 1 Procedures: Identifying the Need for an E-learning

Evaluation Protocol
During the Phase 1 e-learning design activity (storyboard creation), it became apparent that even with the
best of intentions, it is not realistic to create e-learning software that gives a consistent and
comprehensive coverage of all the heuristics. This raised the view that each e-learning software design /
development would have a specific pedagogical coverage that would need to be assessed in some way.

In response to this need, an initial Phase 1 e-learning evaluation rubric was hastily constructed, and was
used by the researcher to assess the Phase 1 e-learning prototype (Algorithms V04a). The rubric and the
evaluation results are outlined in section 6.7.1.

The rubric was constructed based on the perceived needs at the time, was not informed by a significant
literature review or research, and in retrospect was quite rudimentary in nature. However, as with all
good pilots, it served as a strawman to be deconstructed. The Phase 1 evaluation rubric added significant
value in identifying shortcomings and highlighting the limited research in this area; the latter point was
eventually confirmed in Phase 2.

87
 CHAPTER THREE

3.7.2 Phase 2: Piloting the E-learning Evaluation Protocol

3.7.2.1 Objectives

Considering that Phase 1 identified the need for an e-learning evaluation protocol and served as a mock-
up to deconstruct, Phase2–Cycle2 focused on the development and pilot of the 1 st stable version of the
e-learning evaluation protocol.

3.7.2.2 Phase 2 – Cycle 2 Participants

Three education experts were involved in the review of the e-learning evaluation protocol in Phase 2;
these were the same experts involved in the review of the e-learning pedagogy. All the experts were
senior university academics in the fields of child computing interaction, computer science, and education
and educational media.

3.7.2.3 Phase 2 Research Activities

1. A literature review of heuristic evaluation protocols.

2. Develop the e-learning evaluation protocol (refer to section 3.7.2.3.1).
3. Evaluation of the e-learning evaluation protocol by education experts (refer to section 3.7.2.3.2).
4. Pilot the e-learning evaluation protocol using the Phase2–Cycle2 e-learning prototype (refer to
section 3.7.2.3.3).
5. Merge and interpret results from different research strands (Points 3 and 4).

3.7.2.3.1 Phase 2 – Cycle 2 - E-learning Evaluation Protocol

The Phase2–Cycle2 e-learning evaluation protocol was developed in response to the high-level findings
from Phase 1, a significant literature review on heuristic evaluation protocols, and based on the GCSE
Computer Science E-Learning Pedagogy v1.5 document.

The evaluation protocol describes both the evaluation process and acts as the evaluation review template
in which evaluators provide their feedback.

3.7.2.3.2 Evaluation of the E-learning Evaluation Protocol by Education Experts

The composition of the evaluation group that evaluated the e-learning evaluation protocol is outlined in
section 3.7.2.2. The feedback was provided in writing as inline comments within the document; where
necessary, the written comments were followed by one-to-one discussions for further explanation.

The evaluation feedback was analysed and incorporated into the research at the start of Phase 3. The
education expert feedback on the e-learning evaluation protocol is summarised in section 6.7.2.1.

3.7.2.3.3 Pilot of the E-learning Evaluation Protocol

The practical usage of the e-learning evaluation protocol was piloted by the researcher by evaluating
Level-1 and Level-2 of the prototype software. The primary objective of this pilot was not to secure an

88
 CHAPTER THREE

accurate evaluation of the software, but to identify the effectiveness of the protocol and instrument, and
identify any practical areas of improvement. The pilot feedback was analysed and incorporated into the
research at the start of Phase 3. The pilot feedback on the e-learning evaluation protocol is summarised
in section 6.7.2.2.

3.7.2.4 Data Collection, Analysis and Reporting

The qualitative findings from the education experts and pilot were analysed thematically and holistically
on whether they concurred. These findings were used to inform a focused literature review and the
update of the Phase 3 e-learning evaluation protocol.

3.7.3 Phase 3: Usage of the E-learning Evaluation Protocol

A workshop was used as the vehicle for the Phase 3 study on the usage of the e-learning evaluation
protocol. This is not a standard approach for an e-learning evaluation activity, although ultimately was
effective, with some limitations. The medium of a workshop was chosen to give back to the teaching
community, in this manner attracting the greatest number of teachers to participate in the research study.
Considering the workshop was scheduled in term-time and would necessitate the teachers’ absence from
school, it was necessary to keep it to one day. In addition, the need to secure research findings meant
that two end-of-day focus group sessions had to be accommodated in an already busy schedule. These
focus group sessions are not normally part of the e-learning evaluation protocol. The planned agenda for
this workshop is represented in Table 8, which was presented to the participants as part of the workshop
introduction.

Table 8: Evaluation protocol workshop agenda (sourced from workshop introductory presentation)

3.7.3.1 Objectives

The Phase 3 workshop focused on the realistic usage of a mature version of the e-learning evaluation
protocol by in-service teachers. The primary objective was to secure teacher feedback on the protocol:

89
 CHAPTER THREE

specifically, any areas of improvement and whether it is feasible for use in schools. A secondary objective
was to secure indicators of its validity and reliability across groups.

3.7.3.2 Phase 3 Participants

3.7.3.2.1 Phase 3 - Education Experts

Following on from the Phase2-Cycle2 evaluation protocol review, two education experts remained
involved in the review of the e-learning evaluation protocol in Phase 3; they were senior university
academics in the fields of computer science, and education and educational media.

3.7.3.2.2 Phase 3 - Teachers

The Phase 3 workshop was publicised via a flyer and university communications to private schools and
educational institutions in which high-school computing, and specifically the Computer Science GCSE,
were part of their curriculum. The schools then volunteered appropriate teachers to attend the workshop;
the workshop was attended by thirteen teachers representing a total of five private schools and
educational institutions.

Of the thirteen teachers attending the workshop, nine participated in the evaluation of the e-learning
software. The evaluation was a full evaluation of both content and pedagogical quality; hence, the nine
teachers were selected based on being double experts, with knowledge and expertise in teaching and
pedagogy, and additional knowledge and expertise in the domain of computer science.

The nine teachers were split into two groups and remained for the evaluation-debrief meeting and the
workshop focus group. Please refer to Table 9 for a summary of these teachers’ background, qualification
and experience.

Table 9: Evaluation protocol workshop - participant background, qualifications and experience

90
 CHAPTER THREE

3.7.3.3 Phase 3 Research Activities Relating to the Evaluation Protocol

1. A literature review of heuristic evaluation protocols.

2. Refine the e-learning evaluation protocol (refer to section 3.7.3.3.1).
3. Evaluation of the e-learning evaluation protocol by education experts (refer to section 3.7.3.3.2).
4. Workshop
a. Secure informed consent from teachers to participate in research activities during the
workshop.
b. Train teachers on the pedagogical heuristics (refer to section 3.7.3.3.3).
c. Outline the procedure for the e-learning evaluation (refer to section 3.7.3.3.4).
d. Individual teacher evaluation of the e-learning software (refer to section 3.7.3.3.5).
e. Group debrief meeting on evaluation results (refer to section 3.7.3.3.6).
f. Focus groups to secure research findings (refer to section 3.7.3.3.7).
5. Merge and interpret results from different research strands.

3.7.3.3.1 Phase 3- E-learning Evaluation Protocol

The Phase 3 e-learning evaluation protocol was developed in response to the Phase2–Cycle2 feedback
from the education experts, the findings from the pilot evaluation of Level-1 and Level-2 software, and a
focused literature review on heuristic evaluation protocols. The evaluation protocol was drafted based on
GCSE Computer Science E-Learning Pedagogy v1.7b(CLEAR).

The evaluation protocol describes both the evaluation process and acts as the evaluation review template
in which evaluators provide their feedback.

3.7.3.3.2 Evaluation of the E-learning Evaluation Protocol by Education Experts

The composition of the evaluation group that evaluated the e-learning evaluation protocol in Phase 3 is
outlined in section 3.7.3.2.1.This feedback was provided in writing as inline comments within the
document; where necessary, the written comments were followed by one-to-one discussions for further
explanation.

The evaluation feedback was minimal and incorporated into the final version of the e-learning evaluation
protocol. This education expert feedback on the e-learning evaluation protocol is summarised in section
6.7.3.1. For reference, the final version of the e-learning evaluation protocol (based on Phase 3 findings),
is included in Vol 3 Appendix N.

3.7.3.3.3 Train Teachers on the Pedagogical Heuristics

The workshop on the design and evaluation of e-learning software commenced with an introductory
session, followed by training on the pedagogical heuristics. These sessions were led by the researcher and
guided by the following presentations:

1. Workshop Introduction: A 6 slide presentation that outlined: the research foundations of the
workshop, the aims of the workshop, the benefits to the attendees, and the workshop agenda.

91
 CHAPTER THREE

2. Summary of the Pedagogical Heuristics: A 53 slide presentation that outlined: what pedagogical
heuristics are, their interrelationship, their educational benefits, and a summary of each of the
21 heuristics (40 including sub-heuristics).

3.7.3.3.4 Outline the Procedure for the E-learning Evaluation

After the teachers were trained on the pedagogical heuristics, they received training on the e-learning
evaluation protocol. This session was guided by the following presentation:

• Summary of e-learning evaluation protocol: A 15 slide presentation that outlined: what

educational value is, the necessary teacher expertise for the evaluation, why groups are best for
evaluations, the evaluation procedure, how to provide evaluation results, the response scales,
the e-learning software being evaluated, and the educational setting acting as the basis of the
evaluation.

3.7.3.3.5 Individual Teacher Evaluation of the E-learning Software

An integral part of the e-learning evaluation protocol is the hands-on evaluation of the software. After
receiving training on both the pedagogical heuristics and the evaluation procedures, the teachers were
given access to the Level 3 Orange - Algorithms and Flowcharts V0.3 software to undertake the evaluation.
The teachers accessed the software via SCORMCloud; due to time limitations, they had approximately 1
hour and 15 minutes to evaluate the software and record their responses in the online evaluation
response instrument.

In normal circumstances, the e-learning evaluation protocol document is also a feedback template that
allows evaluators to record their evaluation results in a structured and consistent manner. However, due
to the time constraints imposed by the workshop, a mechanism was necessary to consolidate the
evaluation results from all group members within a 15-minute window, before the group debrief sessions.
In response to this constraint, an online survey instrument was developed and used to collect the
respondents’ quantitative feedback on the level of importance of each heuristic and learning objective,
and the level of support the software gave to each heuristic and learning objective. For reference, the
Evaluation Response Collection Instrument is included in Vol 3 Appendix P.

The researcher was present throughout the evaluation activity and gave minimal guidance to ensure the
teachers could progress with the evaluation in the time allotted. There was no disruption to the activity.

3.7.3.3.6 Group Debrief Meeting on Evaluation Results

The overriding objective of the evaluation process is a single evaluation report that fairly represents the
aggregated findings of the set of evaluators. In support of this objective, a debrief session was run by an
unbiased facilitator whose role was to encourage effective discussion, and accurately document the
group’s consensus in the final report. Specifically, the aim was to reach a consensus in the areas of:

1. Quantitative evaluation of the importance of applicable heuristics and learning objectives,

92
 CHAPTER THREE

2. Quantitative evaluation of the level of support the software provides to the applicable heuristics
and learning objectives, and
3. Qualitative comments on pedagogical issues and improvement recommendations.

After the group report was finalised, it was then shared with the evaluators either for their feedback or
their confirmation that it was accurate.

As discussed in 3.7.3.3.5, prior to the group debrief session, all the individual evaluation responses were
consolidated into summary snapshots for each learning objective and heuristic, which in turn were used
to structure and guide the debrief session.

It is not a mandatory pre-requisite for the debrief session, but in support of the research efforts:

1. The Group 1 debrief session was recorded on video and later transcribed.
2. The Group 2 debrief session was recorded in audio and later transcribed, although due to poor
audio quality, the recording has some intermittent sections which could not be transcribed.

Standard procedures for the secure handling of audio and video material were followed.

3.7.3.3.7 Focus Group

Although not part of the evaluation process, at the end of the workshop two focus groups were planned
in which the teacher participants were to have a facilitated discussion to give their feedback on the day.
Specifically, to give their feedback on their personal and their school’s experience with the use and
evaluation of e-learning software, and feedback on the pedagogical heuristics and the e-learning
evaluation process they had used.

To ensure consistency between both focus groups, a set of guiding questions was developed and used.
The questions were created by the researcher and reviewed by an education expert prior to finalisation
and usage. For reference, the focus group guidance questions are included in Vol 3 Appendix R.

The planned duration of the focus groups was one hour; however due to an overrun during the day,
Group-2 completed the focus group in 20 minutes, and for Group-1, it was decided to continue with the
evaluation debrief session. The teachers in Group-1 agreed to answer the focus group questions from
home using an alternate survey instrument that contained the same guidance questions.

Five teachers were planned to attend the Group-1 focus group, of which four responded on the alternate
survey instrument. Five teachers were planned to attend the Group-2 focus group; all the teachers
attended, but one teacher had to leave early (after the first question) from the discussion.

The Group-2 focus group was recorded in audio and later transcribed, although due to poor audio quality
the recording has some intermittent sections which could not be transcribed. Standard processes for the
secure handling of audio and video material were followed.

93
 CHAPTER THREE

3.7.3.4 Data Collection, Analysis and Reporting

The Phase 3 research data collected in relation to the e-learning evaluation protocol is primarily
qualitative in nature. Observations from the Group-1 debrief session, transcripts from the Group 2 debrief
session and focus group, and the Group-1 survey response were all analysed thematically, and
triangulated to give a holistic interpretation.

Quantitative findings from the Evaluation Response Collection Instrument are compared with the final
debrief results and compared between Group 1 and Group 2 to give indicators on the validity and
reliability of the evaluation protocol.

These findings are presented in summary form in section 6.7.3.2 and full form in Vol 2 Appendix C, and
were used to update the final version of the e-learning evaluation protocol.

3.8 Chapter Summary

This chapter outlines the magnitude and interconnectedness of the mixed methods exploratory design
used in this research. It is a roadmap to the research activities, and gives reference in the appendices to
key instruments and materials used in the research. The following were discussed in detail:

1. The theoretical foundations underpinning the choice of research methods,

2. Discussion of the exploratory mixed methods design, which is divided into three phases, each
with a different qualitative-quantitative mix:
a. Phase 1: Pilot,
b. Phase 2: Elaboration via Action Research,
c. Phase 3: Final experiment to confirm the qualitative findings of Phase 1 and 2, and
generalise to the wider GCSE Computer Science population.
3. The use of a convergent parallel design (triangulation design) for each of the above phases,
4. Aligning research controls to the mixed methods design, by incorporating controls to ensure
validity and reliability, but additionally implementing procedures for the parallel qualitative
concept of trustworthiness and its underlying criteria of credibility, transferability, dependability
and confirmability,
5. A breakdown of the teacher, student and education expert participants used in each phase and
cycle of the study,
6. The apparatus used in each phase and cycle of the study, and a focus on the validity and reliability
of key instruments such as the VARK instrument, the survey instrument, the IMMS instrument
and the Pre-, Post-Test,
7. The procedures used in each phase and cycle of the study,
8. The data collection, analysis and reporting used in each phase and cycle of the study, and the
approaches to triangulate between the different strands of research data in each phase,
9. A detailed focus on the quantitative analysis used in Phase 3 and reference to the Phase 3
codebook.

94
 CHAPTER THREE

Finally, there is a compartmentalised discussion of the research methods for the development of the e-
learning evaluation protocol.

95
 CHAPTER FOUR

4 SYNTHESIS OF THE E-LEARNING PEDAGOGY

4.1 Chapter Brief
This chapter focuses on the synthesis of the e-learning pedagogy; previously, section 2.3.2 and 2.3.3
outlined the need to focus on pedagogy for e-learning software and the limitations of existing research.
Section 2.4 discussed what learning theories are, and provided a summary review of the learning theories
and other inputs that are at the foundation of this e-learning pedagogy. These are: constructivism, social
constructivism, collaborative learning, connectivism, VARK learning styles, ARCS motivational design,
cognitive load theory, gamification, and computational thinking. This chapter builds on that material to
discuss the underlying characteristics of the pedagogy and describe the individual pedagogical heuristics.

4.2 Versions of the E-learning Pedagogy

As discussed in Chapter Three, the e-learning pedagogy was iteratively developed and refined through
the three phases of this research study. Five versions of the e-learning pedagogy were developed and
released; these are outlined in Table 10.

Phase Date Pedagogy Appendices

Phase 1 04/02/2015 GCSE Computer Science E-Learning Included in pedagogy document.

Pedagogy v0.7

Phase 2-Cycle 1 01/12/2015 GCSE Computer Science E-Learning Included in pedagogy document.
Pedagogy v1.2

Phase 2-Cycle 2 05/06/2016 GCSE Computer Science E-Learning Computer Science E-Learning
Pedagogy v1.5 Pedagogy Appendices v0.3

Phase 3 04/10/2016 GCSE Computer Science E-Learning Computer Science E-Learning

Pedagogy v1.7b(CLEAR) Pedagogy Appendices v0.4 (CLEAR)

Final 26/03/2018 GCSE Computer Science E-Learning Computer Science E-Learning

Pedagogy v1.8 Pedagogy Appendices v0.5
Table 10: Five versions of the e-learning pedagogy

These pedagogy releases form the basis of the evaluation results presented in section 6.4.

4.3 Intended Audience and Usage

As discussed in Chapter Two, this pedagogy builds on the increasing integration of technology-enhanced
learning in education by supporting the use of e-learning software within a high-school environment. The
research aims to support high-school computing with a comprehensive set of pedagogical heuristics that
can be used by teachers or other educators in either the design or evaluation of e-learning software for
computer science.

The learning theories discussed in this pedagogy are not focused on any specific subject discipline;
however, holistically the focus of the pedagogy is towards computer science. It follows that, overall, the
pedagogy is designed to be more appropriate for computer science or other Science, Technology,

96
 CHAPTER FOUR

Engineering and Mathematics (STEM) subjects, and therefore potentially less appropriate for other
subject areas such as languages, arts, social studies and the humanities. These heuristics supplement
existing literature on instructional design and extend existing e-learning heuristics since these heuristics
focus more tightly and in-depth on pedagogy rather than usability. It is important to consider that:

1. The heuristics are not intended to instruct good pedagogy to teachers.

2. The heuristics are offered as guidance to teachers and instructional designers in the design or
selection of e-learning software appropriate for high-school computer science education.
3. The heuristics are not intended to be implemented as a mandatory checklist.
4. The heuristics are intended to be used as a toolset from which the correct tools are selected by
a teacher or instructional designer based on the specific learning material, intended audience
and intended learning objectives.
5. Although not explicitly considered in this pedagogy, usability is recognised to be critical to e-
learning software, and is a mandatory prerequisite to be considered in conjunction with
pedagogical quality.

4.4 Pedagogical Coverage

The overriding objective of this pedagogy is to holistically consolidate and condense the essence of e-
learning pedagogy in a manner accessible to the teacher audience. The first key factor in this objective is
to comprehensively embody the full learning process. In consideration of Illeris’ framework for learning
(section 2.4.2), Figure 15 maps the various learning theories to their primary area of educational influence.
It is evident that there is comprehensive coverage of the two processes of external interaction and an
internal acquisition, and the three dimensions of content, incentive and environment.

It should also be noted that these theories are taken as a theoretical foundation for the heuristics and not
prescribed in their extreme form. There is a concerted effort to find commonality and overlap between
the theories so as to develop a cohesive set of pedagogical heuristics that can be used holistically together
with minimal conflict.

97
 CHAPTER FOUR

Figure 15: Mapping of learning theories to Illeris' three dimensions of learning

4.5 A Non-Prescriptive Pedagogy

The proposed pedagogy avoids being reductionist and prescriptive; it is intended to be a toolset by which
teachers and instructional designers can use the tools (heuristics) based on their judgement and in
consideration of the educational setting. However, such judgement must be adequately informed; it is for
this reason that the pedagogy gives detailed information on design and evaluation criteria, potential
benefits, how the heuristics interrelate, and potential challenges in implementation. Each heuristic
contains the following information:

• ID: A unique heuristic identification number.

• Title: The title of the heuristic.
• Description: A brief description that summarises the heuristic.
• Learning Theory: An indicator of whether the heuristic has been derived from a specific learning
theory.
• Pedagogical Area: An indicator of which area of instruction the heuristic relates to.
• Design and Evaluation Criteria: A set of criteria that can be used either as design guidelines to
implement the heuristic in e-learning software, or as evaluation guidelines to evaluate the
adherence of existing e-learning software to the heuristic.
• Education Benefits: Outline of the potential benefits associated with the successful
implementation of this heuristic.
• Potential Challenges: Outline of some of the potential challenges that could be faced when
implementing the heuristic.
• Related Heuristics: Outline of other heuristics that interrelate with this heuristic, either in
positive support or potentially in contradiction.

98
 CHAPTER FOUR

The final e-learning pedagogy is included in Vol 2 Appendix D; however, for easy reference, below is an
example of how each heuristic is documented:

ID Heuristic Title:

1 Use authentic educational material, examples and activities.

Learning Theory: Constructivism Pedagogical Area: Authenticity

Description:

Authentic learning represents learning material in a manner that focuses on the context of when the
knowledge and skills will be used. It allows the learner a closer tie to reality and a better understanding
of the relevancy of the material and its true value. This in turn leads learners to take greater ownership
of their learning, achieve a deeper understanding, and increase knowledge transfer to the real-world.

Design and Evaluation Criteria:

The learning material and activities in the e-learning software and associated CLE should reflect a
meaningful subset of the following characteristics of authenticity:

1. Provide contextual authenticity by:

a. Exploring the real-world dimensions of a task,
b. Providing realistic background information that surrounds the learning or problem,
c. Simulating real-life complexities and occurrences, or
d. Using practices and tools used by experts in the field under study.
2. Provide cognitive authenticity by engaging the learner in activities which present the same
type of intellectual challenges as those in the real-world.
3. Provide activities that are intrinsically motivating, that learners are encouraged to solve.
4. Provide learning and activities that are personally relevant or interesting to the learner.
5. Provide learning and activities that are not artificially constrained.
6. Provide the technical affordance for teachers and instructional designers to easily change
text and visual learning material to be more authentic and personalised to their students.

Educational Benefits:

The use of authentic educational material, examples and activities leads to the following education
benefits:

1. Learners more successfully assimilate / accommodate the learning into their own mental
models (schemas).
2. There is an increased likelihood that learners will take ownership of their learning or the
resolution of a problem activity.
3. Learners more effectively grasp the meaning and value of the learning material and the
context in which the knowledge or skills can be used in the real-world.

99
 CHAPTER FOUR

4. Learners are more motivated by authentic or personally relevant learning material.

5. Authentic learning enables deeper learning and improved far transfer.

Potential Challenges:

It is important that authentic learning ties back to reality; but it should not model reality in an ultra-
realistic manner that overwhelms the learner or constrains the learning to a narrow context. Instead,
it is better to give greater focus to cognitive authenticity.

Although authentic learning most intuitively fits into whole-task instruction, special care should still be
given to examples and problems within part-task instruction to maintain a reasonable level of
authenticity where possible.

Related Heuristics: Authentic learning offers strong support in the following areas:

• Ensuring the currency of learning material (Heuristic 1.1)

• Reflective practice and making expert and learner thinking processes
explicit (Heuristics 2, 3)
• Integrating learning into long term memory (Heuristic 5)
• Problem-based learning and computational thinking (Heuristics 4, 4.1,
6, 6.1, 6.2)
• Practice activities and kinaesthetic learning (Heuristics 7, 16.4)
• Social and collaborative learning, situated learning through mobile
devices and networked learning (Heuristic 9, 11, 11.1, 12)
• Learner motivation and gamification (Heuristics 13, 14, 14.2, 14.4, 15,
15.1)

However, it is important to consider that authentic learning can increase

cognitive load and conflict with heuristics 18 and 19, relating to the optimisation
of essential processing, and focusing only on learning material that directly
supports learning objectives.

With reference to Table 11, the final e-learning pedagogy contains 21 heuristics, and an additional 19 sub-
heuristics.

ID Heuristic Title

1 Use authentic educational material, examples and activities.

1.1 Ensure the currency of learning material.

2 Prompt reflective practice to support learning.

3 Make expert and learner thinking processes explicit.

4 Use problem-based learning (PBL) to facilitate learning.

4.1 Use worked examples to support problem-based learning.

100
 CHAPTER FOUR

ID Heuristic Title

5 Integrate learning into long-term memory by using authentic examples, and non-trivial practice and
problems.

6 Support problem-solving through computational thinking.

6.1 Build a foundation for computational thinking.

6.2 Exemplify computational thinking in problem-solving activities.

7 Distribute well-designed practice activities across the lesson to support learning.

7.1 Provide explanatory feedback to practice activities to promote learning.

8 Provide scaffolding to advance learning progress.

9 Use social-interaction to increase learning and promote higher-order thinking.

10 Engage learners in a challenge; target learning towards the zone of proximal development (ZPD).

11 Use collaborative learning activities.

11.1 Support collaborative and situated learning via mobile devices.

12 Develop and nurture networks to support learning.

13 Use constructivist approaches to increase intrinsic motivation in the learner.

14 Use the concepts of Attention, Relevance, Confidence and Satisfaction (ARCS) to attain and sustain
learner motivation.

14.1 Use “Attention” grabbing strategies to increase learner motivation.

14.2 Explain the “Relevance” of the learning material to increase motivation.

14.3 Build “Confidence” to increase learner motivation.

14.4 Build “Satisfaction” to increase learner motivation.

15 Use gamification to increase motivation and learning performance.

15.1 Integrate gamification elements tightly within existing learning processes.

15.2 Build extrinsic gamification elements on top of existing learning processes.

16 Use multi-modal learning approaches.

16.1 Support visual modal preference.

16.2 Support aural modal preference.

16.3 Support read-write modal preference.

16.4 Support kinaesthetic modal preference.

17 Integrate words and graphics together, instead of words alone.

17.1 Apply contiguity by aligning words (audio or screen text) with corresponding graphics.

17.2 Representing words as audio, on-screen text or both.

18 Avoid adding learning content that does not directly support your instructional goal.

19 Optimise essential processing by segmenting learning material and providing pre-training.

20 Use a conversational style in screen text and audio narration.

21 Provide restricted navigational control in the e-learning software.

21.1 Provide consistent navigational elements and signposts for learning.

Table 11: Final set of e-learning pedagogical heuristics.

101
 CHAPTER FOUR

4.6 Further Support for the Selection and Usage of Heuristics

Considering the substantial number of heuristics and sub-heuristics, and the level of detail of each
heuristic, the teacher audience was given further support in the selection and usage of the heuristics by
the provision of summary visual material on how heuristics interrelate, and their potential benefits and
dis-benefits. This is facilitated via an:

1. Educational Benefits Matrix

2. Heuristics Interrelationship Map
3. Heuristics Interrelationship Matrix

4.6.1 Educational Benefits Matrix

The educational benefits matrix provided in Table 12 indicates the learning benefits that can potentially
be achieved by the successful implementation of each heuristic. This matrix is offered as a high-level visual
guidance to teachers and instructional designers on what heuristics can be focused on when targeting
certain educational benefits. It should be noted that implementation of a heuristic does not guarantee
that the educational benefit will be realised, and does not preclude that other educational benefits will
be realised. The educational benefits are categorised into five main areas:

1. Learning Performance (Surface Learning): Learning performance is an immediate measure of

the students’ ability to recall learning material and successfully pass assessments closely linked
to that learning material. In the context of this research, this relates to the students’ ability to
pass GCSE Computing assessments and exams.
2. Deep Learning: As opposed to surface learning, in which learning material is typically passively
memorised with a primary aim of passing assessments, deep learning is learning where there is
a vigorous interaction with the learning material to truly understand it and integrate it with
previous experience and knowledge. Meaning it is integrated into existing mental schemas in the
learner’s long-term memory.
3. Far Transfer / Higher Order Thinking: Far transfer is the application of skills and knowledge
learned in one situation to a different situation. It builds upon deep learning, and requires
learners to adjust the underlying principles they have learnt for use in a new scenario or new
problem. It requires higher order thinking, which in terms of Bloom’s revised taxonomy
(Krathwohl 2002) includes analysing, evaluating and creating.
4. Motivation: Motivation is a theoretical construct used to explain the reason(s) one has for acting
or behaving in a particular way, and relates to the general desire or willingness of someone to
do something. In the context of this research, it focuses on the incentive (motivation) to devote
mental energy to learning.
5. Cognitive Load: At any given time, humans can actively process only limited information;
material that exceeds this threshold may enter working memory but will not be processed and
encoded into long term memory. It is therefore important that the cognitive processes that

102
 CHAPTER FOUR

organise incoming information into logical structures, and integrate it with existing knowledge,
are supported by optimising the learning material to reduce cognitive load.

d
e

g
tion

Loa
anc

kin
Ord er /
hin
tiva
orm

ve
niti
her ransf
er T
ing

Mo
erf

Cog
arn
gP

sed
T
p Le

Far
rnin

uce
rea
Dee

Red
Lea

Hig

Inc
ID Heuristic Title
1 Use authentic educational material, examples and activities.
1.1 Ensure the currency of learning material.
2 Prompt reflective practice to support learning.
3 Make expert and learner thinking processes explicit.
4 Use problem based learning (PBL) to facilitate learning.
4.1 Use worked examples to support problem based learning.
Integrate learning into long-term memory by using authentic examples, and non-trivial practice
5
and problems.
6 Support problem solving through Computational Thinking.
6.1 Build a foundation for Computational Thinking.
6.2 Exemplify Computational Thinking in problem solving activities.
7 Distribute well-designed practice activities across the lesson to support learning.
7.1 Provide explanatory feedback to practice activities to promote learning.
8 Provide scaffolding to advance learning progress.
9 Use social-interaction
Engage to increase
learners in a challenge; learning
target and towards
learning promotethehigher-order thinking.
zone of proximal development
10 (ZPD).
11 Use collaborative learning activities.
11.1 Support collaborative and situated learning via mobile devices.
12 Develop and nurture networks to support learning.
13 Use constructivist approaches to increase intrinsic motivation in the learner.
Use the concepts of Attention, Relevance, Confidence and Satisfaction (ARCS) to attain and
14
sustain learner motivation.
14.1 Use “Attention” grabbing strategies to increase learner motivation.
14.2 Explain the “Relevance” of the learning material to increase motivation.
14.3 Build “Confidence” to increase learner motivation.
14.4 Build “Satisfaction” to increase learner motivation.
15 Use gamification to increase motivation and learning performance.
15.1 Integrate gamification elements tightly within existing learning processes.
15.2 Build extrinsic gamification elements on top of existing learning processes.
16 Use multi-modal learning approaches.
16.1 Support visual modal preference.
16.2 Support aural modal preference.
16.3 Support read-write modal preference.
16.4 Support kinaesthetic modal preference.
17 Integrate words and graphics together, instead of words alone.
17.1 Apply contiguity by aligning words (audio or screen text) with corresponding graphics.
17.2 Representing words as audio, on-screen text or both.
18 Avoid adding learning content that does not directly support your instructional goal.
19 Optimise essential processing by segmenting learning material and providing pre-training.
20 Use a conversational style in screen text and audio narration.
21 Provide restricted navigational control in the E-Learning software.
21.1 Provide consistent navigational elements and signposts for learning.

Table 12: Educational benefits matrix

4.6.2 Heuristic Interrelationships

Teachers are further supported in their selection and usage of heuristics via a clear understanding of how
the heuristics interrelate and reinforce each other and, in some cases, how some heuristics counteract
each other. This is achieved at a detailed level by the ‘Related Heuristics’ section of each heuristic, and is
reflected in a more accessible format via the interrelationship map. With reference to Figure 17, the
interrelationship map provides a high-level indicator of some of the key relationships between

103
 CHAPTER FOUR

pedagogical areas and heuristics. Heuristics that are highly related are grouped into boxes reflecting a
pedagogical area. Boxes that are enclosed or overlap reflect a strong positive relationship. Green
relationship lines reflect a potentially positive relationship between pedagogical areas or specific
heuristics. Red relationship lines reflect a potentially counteractive relationship between pedagogical
areas or specific heuristics.

Heuristic interrelationships are visually represented in more detail by the interrelationship matrix which
maps all heuristics against each other and represents relationships according to the colour coded legend
in Figure 16.

Figure 16: Legend for interrelationship matrix

Due to the unavoidable size (40 x 40 heuristics) and complexity of this matrix, it is more effectively used
as a reference resource. The interrelationship matrix is included for reference in Vol 3 Appendix X.

It should be noted that the pedagogy and sub-deliverables, such as the educational benefits matrix,
interrelationship map and interrelationship matrix, distil an extensive literature review on e-learning
pedagogy, but by definition this is bounded, and does not guarantee educational outcomes. It offers
guidance and recommendation on what heuristics, if employed correctly, give the best likelihood of
achieving the targeted educational outcomes.

As summary tools, it should also be noted that, if there is any lack of clarity or seemingly counterintuitive
information reflected in the educational benefits matrix, interrelationship map or interrelationship
matrix, then reference should be made to the specific section of the heuristic description, which provides
detailed explanatory information.

104
 CHAPTER FOUR

Figure 17: Heuristics interrelationship map

105
 CHAPTER FOUR

4.7 Discussion of Heuristics

The remainder of this chapter builds on the literature review in section 2.4 to provide an outline of the
learning theories and their specific influence on the pedagogical heuristics. As discussed in the previous
section (4.6.2) the interrelationships between heuristics is a significant area which is discussed in detail in
the pedagogy. However, for brevity it is not described in detail in this chapter, it is described in detail for
each heuristic in Vol 2 Appendix D (section 9) and can be viewed in summary form in Figure 17 (previous
page).

4.7.1 Constructivism
Building on the review of constructivism presented in section 2.4.3, this section further discusses
constructivist principles and elaborates on how they are distilled into the e-learning heuristics outlined in
Table 13.

ID Heuristic

1 Use authentic educational material, examples and activities.

2 Prompt reflective practice to support learning.

4 Use problem-based learning (PBL) to facilitate learning.

4.1 Use worked examples to support problem-based learning.

5 Integrate learning into long-term memory by using authentic examples, and non-trivial practice and
problems.

7 Distribute well-designed practice activities across the lesson to support learning.

7.1 Provide explanatory feedback to practice activities to promote learning.

13 Use constructivist approaches to increase intrinsic motivation in the learner.

Table 13: E-learning heuristics based on constructivism

4.7.1.1 Heuristic 1: Use Authentic Educational Material, Examples and Activities

Authentic learning represents learning material in a manner that focuses on the context of when the
knowledge and skills will be used. It allows the learner a closer tie to reality and a better understanding
of the relevance of the material and its true value. This in turn leads learners to take greater ownership
of their learning, a deeper understanding, and increased knowledge transfer to the real-world.

As outlined by Jonassen (1999, p.221) “Nearly every conception of constructivist learning recommends
engaging learners in solving authentic problems.” However, in the context of this study, the following
authors (Jonassen 1999; Jonassen & Hung 2008; Fadel & Lemke 2008; Karagiorgi & Symeou 2005;
Hadjerrouit 2005; Squires & Preece 1999; Gilbert et al. 2007; Jang 2009; Brooks & Brooks 1999) express
support for authentic learning. Additionally, the synthesis of their work gives direction that authentic
learning and activities should reflect several of the following characteristics: contextual authenticity,

106
 CHAPTER FOUR

cognitive authenticity, intrinsically motivational, personally relevant or interesting, and not artificially
constrained or abstract. Additionally, Jonassen (1999) cautions that it is important that authentic learning
ties back to reality, but it should not model reality in an ultra-realistic manner that overwhelms the learner
or constrains the learning to a narrow context.

The use of authentic learning most notably leads to the more successful integration of learning into mental
models, enabling deeper learning and improved far transfer. Furthermore, authentic or personally
relevant learning material is intrinsically motivational to the learner: increasing the likelihood that
learners will take ownership of their learning, and more effectively grasp the meaning and value of the
learning material and the context in which the knowledge or skills can be used in the real-world. However,
it is important to consider that authentic learning can increase cognitive load and conflict with the
optimisation of essential processing and focusing only on learning material that directly supports learning
objectives.

4.7.1.2 Heuristic 2: Prompt Reflective Practice to Support Learning

Reflective practice is the careful contemplation of one’s own thinking processes, actions and beliefs that
in turn support further learning; it is an important part of the constructivist knowledge-building process.
Learners typically do not reflect on their learning unless guided to do so; therefore, the e-learning
software and CLE should provide reflective prompts and associated activities. These typically take the
form of questions or discussions that stimulate the imagination, theory creation, further thinking, further
questions or meta-cognitive thinking.

As discussed in section 2.4.3, constructivist principles focus less on transferring information to learners
and more on facilitating their knowledge construction; the latter is greatly supported by the
encouragement of reflective practices. Jonassen (1999) recommends reflection on the monitoring and
analysis of learning performance (strategies and progress) and learning comprehension.

The following authors (Humphreys et al. 1997; Brooks & Brooks 1999; Jonassen 1999; Aleven & Koedinger
2002; Mayer et al. 2003; Slack et al. 2003; Moreno & Mayer 2005; Hubbs & Brand 2005; Barbour & Rich
2007; Moreno & Mayer 2007; Williams et al. 2008; Chen et al. 2009; Denton 2012) express support for
reflective practice. Additionally, the synthesis of their work gives guidance that reflective practice should
incorporate several of the following characteristics: provide reflective prompts within the learning
material; include these prompts when students are challenged in accommodating new learning; reflective
activities should lead to further iterations of activity, feedback and observation based on the reflection;
provide learners with collaborative reflection activities focused on recent learning events; and involve
learners in providing peer feedback or assessment.

Implementation of the above reflective practices can potentially lead to the following learning benefits:
improved learning performance; deeper understanding and integration into mental models; and
increased mental flexibility, thereby improving far transfer. In addition, collaborative peer reflection, in
particular in a peer feedback context, improves learning motivation and learning performance.

107
 CHAPTER FOUR

4.7.1.3 Heuristic 4: Use Problem-based Learning (PBL) to Facilitate Learning

Problem-based learning, in contrast to part-task instruction, focuses on the bigger picture and begins with
an authentic problem or work assignment which drives the learning process in trying to solve the problem.
Working on the problem takes the form of a guided discovery that integrates into the process the
necessary knowledge and skills to solve the problem, and arguably results in a richer, more challenging
learning experience. PBL has a strong foundation in medical teaching (Jonassen & Hung 2008; Koh et al.
2008; Schmidt et al. 2009), but has also garnered wider support (Jonassen 1999; Jonassen & Hung 2008;
Brooks & Brooks 1999; Buzzetto-more & Alade 2006; Lim et al. 2009; Liu et al. 2011; Melero et al. 2012;
Nasr & Ramadan 2008) in other educational disciplines. Williams et al. (2008, p.327), discuss PBL, in a
high-school computing context:
“The conclusion of the project evaluation was that a problem-based learning methodology is appropriate
for use in technology education, both to achieve the goals of technology education and to give students
experience in a realistic group work environment.”

Additionally, Zapalska and Brozik (2006) emphasise that the active learning inherent in PBL is aligned with
the multi-modal learning styles defined by the VARK method (which is also a central theme within the e-
learning pedagogy defined in this study).

Considering the findings of (Lebow 1993; Jonassen 1999; Jonassen & Hung 2008; Buzzetto-More & Alade
2006; Nasr & Ramadan 2008; Williams et al. 2008; Liu et al. 2011; Clark & Mayer 2011; Melero et al. 2012;
Lim et al. 2009; Sweller et al. 1998; Schraw et al. 2006; Moreno & Mayer 2005), the following guidelines
for PBL are identified:

1. Select an appropriately complex and ill-structured problem that considers: learning objectives,
learner ability, subject knowledge, and appropriate challenge, without creating cognitive
overload;
2. Provide a suitably rich problem representation that describes the problem and its context,
without being overly prescriptive and reducing the learner’s need to think;
3. Include multiple problems / case studies to initiate the learning process and give an authentic
context; this supports learners in building the necessary mental models for the current problem,
and in far transfer learning;
4. Integrate a problem-based learning flow to guide the students in the approach to tackle the
problem; such macro-scaffolding reduces the cognitive load on the learners. Depending on the
size and complexity of the problem, an abridged version of the PBL flow can be undertaken in
iterative cycles;
5. Support and educate students in the metacognitive processes needed in problem-based
learning. Either teachers or the e-learning software needs to educate and support learners in the
relevant metacognitive processes that enable the planning, monitoring and delivery of the
problem solution;

108
 CHAPTER FOUR

6. Provide a collaborative environment to support the social interactivity inherent in problem-

solving. PBL is inherently based on social theories of learning and on the interaction of learners
in negotiating a solution to the problem. Therefore, in the context of e-learning, appropriate
collaborative tools should be put in place; and
7. Provide a Problem Manipulation Environment. The kind of mindful activity necessary in PBL
requires a problem manipulation environment which supports the learners in investigating,
hypothesising and testing their hypothesis.

The appropriate implementation of such guidelines can potentially lead to the following learning benefits
typically associated with PBL (Jonassen & Hung 2008; Koh et al. 2008; Schmidt et al. 2009; Williams et al.
2008; Nasr & Ramadan 2008; Lim et al. 2009; Buzzetto-more & Alade 2006; Melero et al. 2012):
comparable or improved learning performance as compared with part-task learning; deeper learning and
improved higher-order thinking skills; superior far transfer of learning; increased intrinsic motivation; and
superior interpersonal skills and student satisfaction as compared to traditional instruction. However, it
must be recognised that PBL and pure discovery-based learning has been critiqued as negatively impacting
cognitive load (Sweller & Cooper 1985; Moreno 2004; Moreno & Mayer 2007; Sweller 2004).
Furthermore, PBL is still a significant shift in educational philosophy that to some extent remains out of
step with current educational and assessment practices in high-schools. In accordance with (Lebow 1993;
Jonassen 1999; Jonassen & Hung 2008; Buzzetto-More & Alade 2006; Williams et al. 2008; Nasr &
Ramadan 2008; Liu et al. 2011; Melero et al. 2012), a number of challenges must be considered and
overcome in PBL: significant pedagogical and instructional design experience is necessary to define and
scaffold a problem that engages the learner’s thinking processes without causing cognitive overload; the
representation of the problem must be suitably rich in describing the necessary information, but not
overly prescriptive; during PBL, new and unfamiliar roles and responsibilities are placed on both the
teacher and the students; and in relation to the pedagogy defined in this study, the guided discovery
inherent in this approach must carefully consider and balance with the structured e-learning environment.

4.7.1.4 Heuristic 4.1: Use Worked Examples to Support Problem-based Learning

The step-by-step demonstration of how to perform a task or solve a problem (worked example) is an
approach advocated in cognitive load theory (Sweller & Cooper 1985; LeFevre & Dixon 1986; Chi et al.
1989; Chandler & Sweller 1991; Sweller et al. 1998; Moreno 2004; Sweller 2004; Schworm & Renkl 2007;
Hattie 2009; Rourke & Sweller 2009; Anderman & Dawson 2011; Clark et al. 2006), although it is also
considered in scaffolding and active learning principles within constructivism (Anthony 1996; Jonassen
1999). Worked examples are considered within the K12 Computer Science Framework (ACM et al. 2016).
Additionally, Skudder and Luxton-Reilly (2014) report that the use of worked examples in computer
science is widely used yet is a comparatively under-researched area. Meanwhile, Gray et al. (2007) offer
some cautious success for the use of worked examples in teaching programming concepts, but not the
holistic design process inherent in program-based solutions.

109
 CHAPTER FOUR

Findings from the following authors (Sweller & Cooper 1985; LeFevre & Dixon 1986; Chi et al. 1989;
Schworm & Renkl 2007; Rourke & Sweller 2009; Anderman & Dawson 2011; Clark et al. 2006; Sweller
2004; Jonassen 1999) have informed the guidelines towards faded worked examples. In particular, the
work of Clark and Mayer (2011) has significantly influenced the following characteristics of worked
examples: gradually transition from worked examples to problems (as reflected in Figure 18); promote
self-explanation of worked examples; where appropriate, selectively include instructional explanation of
worked examples; and support far-transfer by using examples that provide the same underlying principles
in different contexts.

Worked Example Completion Completion Assigned

Example 1 Example 2 Problem

Step 1 Step 1 Step 1 Step 1

Step 2 Step 2 Step 2 Step 2

Step 3 Step 3 Step 3 Step 3

Worked by Learner

Worked Example

Figure 18: Gradual transition from worked examples to practice problems

Adapted from (Clark et al. 2011)

The aforementioned research claims the following potential benefits from the use of worked examples:
reduction in cognitive load, increased learner preference towards worked examples over traditional
instruction; improved learning performance, deeper learning; and when used in conjunction with multiple
case studies, improved far transfer. However, after consideration of the expertise reversal effect22, this
pedagogy recommends faded worked examples (Sweller et al. 1998; Kalyuga & Ayres 2003; Sweller 2004).

4.7.1.5 Heuristic 5: Integrate Learning into Long-term Memory by using Authentic

Examples, and Non-trivial Practice and Problems

To facilitate deep learning, students must integrate new learning material into existing schemas in their
long-term memory. This allows learning to move beyond memorisation and fact recall, and enables the

22
Worked examples become redundant, and can even hinder deep learning, when learners reach a sufficient level
of experience.

110
 CHAPTER FOUR

more flexible application of knowledge and skills to scenarios not explicitly covered in the learning
material.

This heuristic is informed by constructivism, social constructivism and cognitive load theory; it
consolidates the heuristics outlined below to give focus to deep learning and far transfer.

• 1: Use authentic educational material, examples and activities.

• 2: Prompt reflective practice to support learning.
• 3: Make expert and learner thinking processes explicit.
• 4: Use problem-based learning (PBL) to facilitate learning.
• 4.1: Use worked examples to support problem-based learning.
• 6.2: Exemplify computational thinking in problem-solving activities.
• 9: Use social interaction to increase learning and promote higher-order thinking.
• 16.1: Support visual modal preference. Guideline 10, support for visualisation.

Refer to the aforementioned heuristics for further information.

4.7.1.6 Heuristic 7: Distribute Well-designed Practice Activities across the Lesson to

Support Learning

A keystone principle within constructivism is that of active learning, whereby students actively engage
their mental processes in practice exercises, reflective observation, theoretical conceptualisation and
experimentation. This heuristic partially addresses the concept of active learning by the distribution of
well-designed practice activities across the lesson to support learning. This encourages the mindful
activity expounded by Jonassen (1999) and aligns with Hadjerrouit (2005), Buzzetto-more and Alade
(2006), Hattie (2009), Moreno & Mayer (2007), Brooks and Brooks (1999), and Boud (2000) that
instruction and learning is redesigned so that learning, practice and assessment are interwoven (and
cannot be separated), with a focus on authentic and meaningful, learning and activities.

Findings from Clark and Mayer(2011), Sloboda et al. (1996), Tuffiash et al. (2007), Ericsson (2006), and
Kellogg and Whiteford (2009), give clear evidence that practice activities that are deliberate, focused,
persistent, and with appropriate feedback lead to increases in intrinsic motivation and learning
performance. This leads to the following guiding characteristics for practice activities:

1. They should integrate into the learning experience to support and solidify learning, instead of
being used as a summative assessment. This integration has further significance since these
activities should be guided by the same heuristics intended for learning content (i.e. authentic
and meaningful, preferably not shallow activities to recognise or reiterate facts);
2. As outlined by Clark and Mayer (2011) the number and distribution of practice activities should
be carefully considered in relation to intended learning objectives, and whether there is a focus
on improved learning performance, automaticity, exam preparation, long term retention or
extended learning; and

111
 CHAPTER FOUR

3. As outlined by (J. M. Keller 1987a; Voelkel 2013; Brooks & Brooks 1999; Clark & Mayer 2011),
practice activities should provide variety and intrinsic motivation to maintain optimum
engagement (this relates to Heuristics 13 and 14).

As mentioned earlier, the benefits of giving a significant focus to practice activities is improved learner
motivation and improved student performance. Furthermore, giving practice activities extended focus,
improves efficiency and speed of execution (automaticity). However, it must be considered that practice
can improve student performance almost indefinitely to an elite status, but with diminishing returns.

4.7.1.7 Heuristic 7.1: Provide Explanatory Feedback to Practice Activities to Promote

Learning

A vital aspect in securing the benefits from practice activities is the provision of explanatory feedback,
which in turn gives a further opportunity to promote learning. This is widely cited (Sadler 1989; Hattie
2009; Black & Wiliam 1998; Moreno & Mayer 2005; Moreno & Mayer 2007; Clark & Mayer 2011; Nicol &
Macfarlane-Dick 2006) as a significant factor influencing learning achievement. Explanatory feedback as
outlined in this pedagogy shares many of the underlying principles of formative assessment as discussed
by (Voelkel 2013; Nicol & Macfarlane-Dick 2006; Taras 2005; Pachler et al. 2010). Additionally, when
viewed through the lens of Sadler (1989), Black and Wiliam (1998), Boud (2000), and Nicol and
Macfarlane-Dick (2006), there is additional alignment with constructivist principles in that the explanatory
feedback should empower and support students as self-regulated learners, who have in mind the learning
goals and, with some input, can self-regulate their performance towards those goals. Nicol and
Macfarlane-Dick (2006), and numerous other authors, cite Sadler (1989) in outlining three knowledge
areas essential for students to benefit from feedback:

1. Knowledge of what good performance is (i.e. what is the goal or standard being targeted);
2. Knowledge of how current performance compares to targeted performance;
3. Knowledge of how to bridge the gap between current and target performance.

Student knowledge of what the applicable learning goals and good performance are (Point 1), is discussed
in heuristic 14.3 - guideline 1a. The above authors, and more specifically the following authors (Hattie
2009; J. M. Keller 1987a; Moreno & Mayer 2007; Black & Wiliam 1998; Clark & Mayer 2011), give basis to
the following characteristics of explanatory feedback (that address Points 2 and 3):

1. Provide feedback that tells the learner whether the answer is correct or incorrect,
accompanied by a succinct explanation.
2. The explanation should provide cues, reinforcement or information on how to successfully
complete a task or achieve learning goals.
3. Feedback should be provided at, or just above, the level where the student is learning.
4. Position the feedback in close proximity to both the question and answer, so the learner can
see all together.
5. Feedback should focus on the task or task process and not on the learner.

112
 CHAPTER FOUR

6. Emphasise progress feedback that shows improvement over time.

7. In more complex problem-solving activities, that include multiple steps, it is important to
provide step-wise feedback.

Such feedback should then be used to complete the feedback loop (Sadler 1989) by giving students the
opportunity to produce improved work in consideration of the received feedback.

As outlined previously, the positive influence of explanatory feedback (formative assessment) on learning
performance is well cited, and is further evidenced by improved learning performance as compared to
corrective feedback (Mayer et al. 2003; Moreno 2004; Moreno & Mayer 2005; Clark & Mayer 2011).
Additionally, students express a preference towards explanatory feedback as being more helpful (Moreno
2004; Clark & Mayer 2011) and when applied successfully it can positively influence student engagement
(Nicol & Macfarlane-Dick 2006).

4.7.1.8 Heuristic 13: Use Constructivist Approaches to Increase Intrinsic Motivation

in the Learner

Intrinsic motivation stems from an interest or enjoyment in the learning or the activity itself, and
originates within the individual rather than relying on external incentives. Motivation is important in
giving the learner incentive to devote the mental energy to learn. A number of constructivist principles,
such as whole-task and problem-based learning, authentic learning, active learning, mindful activity, etc.,
are shown to be intrinsically motivational and therefore supportive of the learning process.

This heuristic is primarily informed by constructivism and social constructivism learning theories; it
consolidates the heuristics outlined below to give focus to intrinsic motivation.

• Focus on whole-task learning and problem-based learning (PBL): refer to heuristics 4, 6, and 6.2;
• Focus on authentic educational material, examples and activities: refer to heuristics 1, and 1.1;
• Focus on social and collaborative learning to improve student engagement: refer to heuristics 9,
and 11;
• Enable learners to manage and take responsibility for their own learning: refer to heuristics 3,
and 14.3;
• Convey the importance of the learning activity to the learner: refer to heuristic 1 and 14.2;
• Focus on active learning that encourages the student to actively engage their mental processes:
refer to heuristics 2, 4, and 7; and
• Provide a problem manipulation environment that supports mindful activity: refer to heuristic 4
- guideline 9, and 6.2-guideline-4.

Refer to the aforementioned heuristics for further information.

113
 CHAPTER FOUR

4.7.2 Social Constructivism

Building on the review of social constructivism presented in section 2.4.4, this section further discusses
social constructivist principles, and elaborates on how they are distilled into the e-learning heuristics
outlined in Table 14.

ID Heuristic

3 Make expert and learner thinking processes explicit.

8 Provide scaffolding to advance learning progress.

9 Use social-interaction to increase learning and promote higher-order thinking.

10 Engage learners in a challenge; target learning towards the zone of proximal development (ZPD).
Table 14: E-learning heuristics based on social constructivism

4.7.2.1 Heuristic 3: Make Expert and Learner Thinking Processes Explicit

According to Vygotsky (1978), humans’ capacity for language is instrumental in their ability to overcome
impulsive action, plan a solution to a problem prior to execution, and to master their own behaviour.
Language is used for self-guidance, self-direction and external communication. However, despite these
internal regulating and guiding capabilities, students often learn something without being clear on the
rationale behind it, when to do it, how to gauge progress or whether the approach is working. The e-
learning software and teachers (via the CLE) should make invisible mental processes explicit for the
learner to understand and thereby implement. Likewise, learners must undertake activities to clarify and
reflect on their underlying thinking and the rationale for their actions in resolving a problem (i.e.
metacognitive thinking).

With a basis of the following research, (Vygotsky 1978; Schoenfeld 1987; Jonassen 1999; Boud 2000; Slack
et al. 2003; Schraw et al. 2006; Barbour & Rich 2007; Anderson 2008; Fadel & Lemke 2008; Kop & Hill
2008; Chen et al. 2009; Illeris 2009; Denton 2012), the below guidelines were synthesised to make expert
and learner thinking processes explicit:

1. Make solution steps and underlying thinking explicit during problem-solving. During problem-
solving activities, the e-learning software and teachers (via the CLE) should model overt and
covert performance by articulating the plan, implementing it and evaluating the cycle. Decisions,
process steps and the underlying thinking and rationale for these decisions or steps should be
made explicit via a running commentary, and / or visualisation (with accompanying signposting).
2. Focus attention on expert behaviour. In line with kinaesthetic learning, expert behaviour should
be demonstrated, and the desired behaviour signposted to focus learner attention.
3. Promote learner reflection on their own thinking processes. Focus should be expanded beyond
finding the final solution to also include opportunities for learners to comment on their progress
and articulate their thinking in working on a problem. To promote further self-reflection,
teachers can provide feedback or students can enter a pair or group reflection in which they can
discuss, justify and comment on theirs and others’ thinking processes.

114
 CHAPTER FOUR

Expert thinking can be made explicit by the provision of a running commentary of the underlying thinking
and rationale for decisions, this commentary can be implemented in audio, video, animation or text in the
e-learning software. The CLE can be used by the teacher to demonstrate expert behaviour and thinking.
Additionally, the learners can undertake activities on the CLE to reflect on their own thinking processes.

By explicitly exposing thinking processes in varying contexts, the learner is able to model them,
incorporate them in their own thinking, and reflect on their own thinking processes. This in turn leads to
a comparative improvement in: immediate learning performance, deep learning, far transfer, and more
robust problem-solving skills. However, it must be considered that making invisible thinking processes
explicit is a challenging task that requires a high level of metacognitive skills in the teacher, or instructional
designer designing the e-learning software. In addition, it also creates a secondary layer of metacognitive
learning that, if not considered with measure, may overload or distract students from the primary topic.

4.7.2.2 Heuristic 8: Provide Scaffolding to Advance Learning Progress

Scaffolding is the process by which a teacher or other guiding figure (including the e-learning software or
more knowledgeable students) provide additional instructional assistance, coaching, guidance, or
prompting that supports a student’s learning process so they can accomplish learning or problem-solving
that is normally beyond their unassisted efforts.

The research of (Sweller & Cooper 1985; Jonassen 1999; Vrasidas 2000; Karagiorgi & Symeou 2005;
Buzzetto-More & Alade 2006; Schraw et al. 2006; Kop & Hill 2008; Lee et al. 2008; Chen et al. 2009;
Pritchard 2009; Caballé et al. 2010; Vogel-Walcutt et al. 2011; Melero et al. 2012) was considered in this
heuristic and the synthesis of the following guidelines to support scaffolding:

1. Choose appropriate scaffolding or combination of scaffolding approaches for the situation:

a. Macro-scaffolding – Provides high-level guidance on how to approach a learning task or
problem activity. It can give a general workflow to be followed in order to accomplish
the desired learning outcomes.
b. Micro-scaffolding - Provides scaffolds for solving detailed actions that focus on specific
learning activities.
c. Automated Scaffolding – Automated scaffolding is macro or micro-scaffolding that is
incorporated in the instructional design and is delivered by the e-learning software.
d. Social-scaffolding – Macro or micro-scaffolding provided by some form of social
interaction with a teacher or other guiding figure.
2. Choose a variety of scaffolding techniques, including:
a. Use explanatory feedback to support student learning (Heuristic 7.1).
b. Use screencasts that focus on detailed actions on specific learning activities.
c. Use simulations, animations and other visualisation tools to help learners construct
their mental images and visualise activities (Heuristic 16.1 guideline 10).
d. Pre-plan for students to request scaffolding on some low-level tasks by including a
"Hint" button that can be used when needed.

115
 CHAPTER FOUR

e. Provide worked examples that gradually transition to full problems (Heuristic 4.1).
f. Present related case studies - providing case studies with different contexts but the
same underlying principles supply the learner with a rich set of experiences to guide
them in resolving the current problem.
g. Articulate Reasoning – An expert model’s problem-solving or other skills which is used
to articulate the implicit reasoning and decision-making involved in each step of a
process (Heuristic 3).
h. Behavioural Modelling – Expert behaviour is modelled by demonstrating how to
perform certain activities and signal important steps (Heuristic 3).
3. Gradually remove scaffolding support as learners advance and develop their own learning
strategies.

The correct use of scaffolding can provide the following educational benefits: support learners in reaching
their full potential (i.e. their ZPD) and the expected learning outcomes; reduce cognitive load during
constructivist and whole-part instruction, thereby mitigating the potential for cognitive overload in these
approaches; and improve student engagement and satisfaction. However, scaffolding support should be
gradually removed as learners advance and develop their own learning strategies, otherwise it may
frustrate advanced learners or impede the progress of learners to more advanced levels.

4.7.2.3 Heuristic 9: Use Social-Interaction to Increase Learning and Promote Higher-

Order Thinking

As outlined briefly in section 2.4.4, according to Vygotsky (1978, p.57), “Every function in the child's
cultural development appears twice: first, on the social level, and later, on the individual level; first,
between people (inter-psychological), and then inside the child (intra-psychological).” Whilst this assertion
is disputed by some, social constructivism is well recognised since it builds upon cooperative and
collaborative learning and reflective learning practices to emphasise the importance of social interactions
in shaping the learner’s knowledge construction. It supports learners in reaching a higher level of learning
than can be achieved individually. In addition, the social interaction of individuals can often lead to
learning that is greater than the sum of the individuals, and can ultimately result in a shared understanding
inherently derived from the learning community.

The research of (Vygotsky 1978; Dillenbourg et al. 1995; Jonassen 1999; Squires & Preece 1999; Boud
2000; Vrasidas 2000; Karagiorgi & Symeou 2005; Schraw et al. 2006; Barbour & Rich 2007; Kop & Hill 2008;
McLoughlin & Lee 2008; Anderson 2008; Pritchard 2009; Shroff & Vogel 2009; Bell 2011; Caballé et al.
2010; Chai & Lim 2011; Chiong & Jovanovic 2012; Clark & Mayer 2011; Melero et al. 2012) is considered
in this heuristic, and underpins the synthesis of the following guidelines to support social learning:

1. Use social interaction to foster learners’ ability to develop an opinion and evaluate their
opinion in relation to other peoples’ opinions. The e-learning software and collaborative
learning activities (supported by teachers) should give an increased focus on social
(collaborative or cooperative) activities that foster students’ ability to develop an opinion,

116
 CHAPTER FOUR

evaluate their own and other peoples’ opinions, justify their thinking, and openly discuss their
interpretation.
2. Provide a suitable learning environment and context for social interaction. The activities in
point 1 always involve an open dialogue, typically within a community of learners. The
instructional design of the e-learning software and collaborative activities should provide an
authentic context, typically within a problem-based learning (PBL) approach in which learners
cooperate and learn together to solve a non-trivial authentic problem.
3. Provide expert knowledge, behaviour and guidance. Social interaction is not restricted to only
other learners; it is also facilitated by a more knowledgeable other. This can often take the
form of a cognitive apprenticeship where teachers (the experts) work alongside students (the
apprentices) within a situated context. The e-learning software and/or the teacher (via the CLE)
act as the expert to demonstrate appropriate behaviour, and offer scaffolding to support the
apprentices in working on a problem or activity.

Social constructivist learning interactions can potentially offer the following educational benefits:

1. The primary benefit is to promote higher-order thinking by encouraging learners to develop,

evaluate, and appreciate multiple perspectives on an issue, this can ultimately lead to a shared
consensual meaning, or if not, at least individual learning that is informed by other perspectives.
2. The underlying principles and collaborative nature of social constructivism also lead to greater
student engagement and learning that is more meaningful.
3. Although not explicit in the instructional objectives, social constructivism also allows the sharing
of learner workload through cooperative learning, and the building of consensus through
collaboration.
4. Social scaffolding approaches can help reduce cognitive load, thereby mitigating the potential
for cognitive overload.

However, it is important to note that there remains a significant need for individual learning; not all
learning should be made social, and not all learning material is suitable for social learning. Additionally,
there is significant effort involved in devising successful collaborative work, forming effective groups, and
maintaining productive social dialogue and learning. Also, some learners, such as those with a dominant
read-write modal preference (Heuristic 16.3), are not as responsive to social learning as others.
Furthermore, although recent literature suggests the trend is changing, it should also be noted that, to
some extent, social learning remains out of step with current educational and assessment practices in
schools.

4.7.2.4 Heuristic 10: Engage Learners in a Challenge; Target Learning Towards the
Zone of Proximal Development (ZPD)

Vygotsky postulates that a true measure of a student’s capabilities is not addressed by focusing on their
actual developmental level, but that “what children can do with the assistance of others might be in some

117
 CHAPTER FOUR

sense even more indicative of their mental development than what they can do alone” (1978, p.85). This
theory is elaborated in the Zone of Proximal Development (ZPD). The ZPD is a theoretical space of
understanding which is just above the level of understanding of a given individual, and which can only be
reached with support. It is the necessity of this support by others that explains the importance of social
interaction for learning development. It should be noted that the role of the more knowledgeable other
is not a role reserved only for the teacher; it is often taken by a more capable peer.

The research of (Vygotsky 1978; Vrasidas 2000; Karagiorgi & Symeou 2005; Barbour & Rich 2007; Pritchard
2009; Melero et al. 2012) is considered in this heuristic and the synthesis of the following guidelines to
support the Zone of Proximal Development:

1. Educational material should progressively increase in challenge and maintain the learner at the
upper limits of their current learning capacity. The level of educational material and activities
should progressively increase in difficulty as the learner progresses, thus encouraging the
students to work at the upper limits of their current learning capacity.
2. Teachers should be able to easily update text and visual learning material to be more
challenging. The e-learning software should provide the technical affordance for teachers and
instructional designers to easily update text and visual learning material to be more challenging
to their student audience.
3. Learners should be supported with scaffolding and access to more knowledgeable others. In
order to avoid feelings of disappointment, learners must have comprehensive scaffolding
support (Heuristic 8) and access to more knowledgeable others who can give assistance in
reaching the next learning level, or when the learning process is blocked.
4. Learners should have access to a learning community. A collaborative learning environment
(CLE) should be provided to support the community of learning and facilitate social interactions
between learners and more knowledgeable others. Within the community of learning, there will
be multiple overlapping ZPDs and the social interaction between students of different ZPD levels
will drive an overall improvement in levels of understanding.
5. Provide learning content and activities that adapt to the learner’s current abilities and
progress. Adaptive e-learning software can automatically adjust to the learner’s current
knowledge and skill level and progress to material, activities and questions that are more
challenging, or alternatively revisit previous material if the learner has not adequately mastered
it.

The purported educational benefit of the ZPD is that it motivates learning that stretches the learner’s
existing capabilities, and encourages them to the next higher level of learning. Vygotsky theorised that
child development (physical and mental maturation) are not independent of learning, and that by focusing
on the ZPD, the child’s mental development is encouraged.

In an idealised situation, the teacher or more knowledgeable other must recognise the individual learner’s
current state of understanding and plan accordingly to stimulate and support learning at the next higher
level. This is a very difficult task for teachers and instructional designers to achieve with a class of students.

118
 CHAPTER FOUR

This poses the risk when setting progressively more challenging learning material that some learners may
be left behind or become disappointed. This should be mitigated by the use of scaffolding and
individualised support via the collaborative learning environment.

Another approach is to implement adaptive learning into the e-learning software; however, this too must
be considered in relation to the significant additional effort in creating supplementary educational
content, multiple adaptive learning pathways, and the complexities and additional development of the
software.

4.7.3 Collaborative Learning

Building on the review of collaborative learning presented in section 2.4.5, this section further discusses
collaborative learning principles and elaborates on how they are distilled into the e-learning heuristics
outlined in Table 15.

ID Heuristic

11 Use collaborative learning activities.

11.1 Support collaborative and situated learning via mobile devices.

Table 15: E-learning heuristics based on collaborative learning

4.7.3.1 Heuristic 11: Use Collaborative Learning Activities

Knowledge construction and integration in schemas (in long-term memory) happens on an individual
level; however, it is evident that other people affect the learning process, and that arguably learning most
naturally occurs not in isolation, but when students work together. Collaborative learning capitalises on
other people’s knowledge, skills and resources, allowing learners to monitor one another’s work, share
information, summarise points, verify and test their knowledge, and debate their opinions.

With a basis of the following research (Roger & Johnson 1988; Johnson & Johnson 1996; Jonassen 1999;
Lou et al. 2001; Mayes & de Freitas 2004; Mitchell & Savill-Smith 2004; Alonso et al. 2005; Dabbagh 2005;
Karagiorgi & Symeou 2005; Nuutila et al. 2005; Palmer 2005; Gilbert et al. 2007; Anderson 2008; Beck &
Chizhik 2008; Tutty & Klein 2008; Black & Wiliam 2009; Kirschner et al. 2009; Nicholas & Wan 2009;
Papanikolaou & Boubouka 2010; Clark & Mayer 2011; Slavin 2011; Chiong & Jovanovic 2012; Denton 2012;
Ryoo et al. 2013; Waite 2017) this heuristic, and the following guidelines, were synthesised to maximise
the benefits of collaborative learning. The instructional design of the e-learning software and collaborative
activities should incorporate a subset of the following guidelines:

1. Ensure Social Interdependence of the group. In collaborative learning, incentives (feedback

and/or grading) should be based on a synthesis of individual and group outcomes, aligned to
ensure that each team member participates and supports the learning of the rest of the team.
2. Design the collaborative activity based on intended learning outcomes. The desired outcome
of the collaboration must be considered from the outset, and the activity designed accordingly.
The outcome of collaborative learning typically falls into two main areas; either improved

119
 CHAPTER FOUR

individual learning or an increase in the quality of the project deliverable(s). However, since
individual achievement does not necessarily correlate with group achievement or the quality of
group deliverables, it is critical to decide from the outset what the expected outcome from the
collaboration is.
3. Design the collaborative activity to ensure the quality of the collaborative dialogue. The design
of collaborative activities must ensure substantive contributions by all team members. Poor
instructional design can lead learners to shallow or non-participation in collaborative activities.
Adapt instructional material to promote interdependence. Avoid difficult activities that have
incentives that encourage one or two individuals to take on most of the work, or activities that
are too easy, unstructured or not designed to promote discussion or true collaboration. Example
activity types include: collaborative problem-solving; cooperative problem-solving; peer
instruction, scaffolding and feedback; brainstorming, discussion and debate; collaborative
reflection; collaborative editing; and pair programming.
4. Provide structure and support for collaborative activities
a. Clearly communicate objectives and deliverables for the collaborative activity.
b. Provide relevant instructions, guidance, problem description and context information
for the collaboration; at least enough information to start further enquiry.
c. Consider assigning roles to the group members to provide structure to the process (e.g.
solution creator, reviewer, moderator, etc.). The roles can then be switched to ensure
equal coverage.
d. Consider providing macro-scaffolding for the collaborative process. This can take the
form of high-level guidance on how to approach the collaborative activity, and can give
a general workflow to be followed in order to accomplish the desired learning
outcomes.
e. If necessary, teachers may need to intercede and act as a facilitator for the
collaboration.
5. Consider team size and composition. Team composition should typically either be
heterogeneous teams consisting of high and low prior knowledge learners, or homogeneous
teams consisting of high or medium prior knowledge learners. Avoid homogeneous teams
consisting of low prior knowledge learners. Factors to consider include:
a. The inherent complexity of the learning activity or task.
b. The learning material or nature of the task, which may impose on the group size.
c. With increased group size comes a theoretical increase in the range of abilities,
expertise, skills, and cognitive capacity.
d. With increased group size comes an increased overhead in effectively managing and
coordinating the group.
e. The shorter the period of the collaborative activity, the smaller the group should be.
f. If individual learning is the outcome goal, then consider pair work.

120
 CHAPTER FOUR

g. If creative problem-solving is the outcome goal, then consider a larger group of three to
five team members.
6. Prepare students for collaborative activities. Ensure the team have sufficient social skills to
manage conflict and that interactions are directed towards rational debate. This can be
accomplished by pre-training or by setting appropriate guidelines.
7. Monitoring and Intervening. The e-learning software and CLE must offer the technical
affordance for teachers to observe group performance in collaborative activities and their
subsequent deliverables, and the ability to provide support to the group in the task or in the
collaborative process.

To support collaborative and social learning, the e-learning software should be supplemented by a
collaborative learning environment (CLE). Students using the e-learning software should also be registered
on the CLE and the e-learning software will then direct learners to the CLE to undertake collaborative
activities. A synthesis of the following research (Jonassen 1999; Buzzetto-More & Alade 2006; Gilbert et
al. 2007; Anderson 2008; Siemens 2008; Couros 2009; Nicholas & Wan 2009; Halse et al. 2009; Boyle et
al. 2010; Caballé et al. 2010; Clark & Mayer 2011; Denton 2012; Jenkins et al. 2012) recommends that the
CLE should implement a number of the following collaborative technologies (refer to Table 16):

Blog and Micro-blogs Voice Over Internet Protocol (VOIP) e-Portfolios

Video Conferencing E-Mail Social Networks

Text Chats (Internet Relay Chat) Discussion (Message) Boards Wikis

Online Conferencing Cloud Office Applications Cloud Integrated Development

Environments
(Virtual Classrooms / Breakout
Rooms)

Virtual Environments
Table 16: Collaborative technologies to be supported in a collaborative learning environment.

In line with the following research (Roger & Johnson 1988; Lebow 1993; Johnson & Johnson 1996; Vrasidas
2000; Barbour & Rich 2007; Tseng & Tsai 2007; Beck & Chizhik 2008; Sun et al. 2008; Tutty & Klein 2008;
Barker et al. 2009; Cole 2009; Halse et al. 2009; Kirschner et al. 2009; Papanikolaou & Boubouka 2010;
Kang et al. 2010; Slavin 2011; Zenios 2011; Chiong & Jovanovic 2012), the benefits of collaborative learning
include:

1. Under certain conditions, collaboration can substantially improve student achievement.

2. It can improve long-term retention and support higher-order thinking, meta-cognition and
problem-solving by encouraging learners to develop, evaluate, and appreciate multiple
perspectives on an issue.
3. It can lead to greater student engagement and learning that is more meaningful.
4. It allows the sharing of learner workload through cooperative learning and building consensus
through collaboration.

121
 CHAPTER FOUR

5. Cooperativeness is positively related to indices of psychological health such as: “emotional

maturity, well-adjusted social relations, strong personal identity, and basic trust in people”
(Johnson et al. 1984, p.22).

However, it should be noted that collaborative learning attempts to instigate certain forms and types of
interactions that encourage learning, but there is no guarantee that improved learning or any of the other
benefits of collaborative learning will occur. Johnson & Johnson maintain that “Many groups are
ineffective, and some are even destructive. Almost everyone has been part of a group that has wasted
time and produced poor work” (1996, p.793), and that such groups are typically categorised by a number
of dynamics such as “social loafing, free riding, group immaturity, uncritical and quick acceptance of
members’ dominant response, and group-think” (1996, p.793). Other authors, such as (Tutty & Klein 2008;
Williams et al. 2008; Kirschner et al. 2009; Nicholas & Wan 2009; Papanikolaou & Boubouka 2010; Lee &
Choi 2011; Zenios 2011; Chiong & Jovanovic 2012), have also noted sub-optimal conditions for
collaborative learning. This means that the teacher effort in implementing guidelines one to seven are
significant, as are the efforts of the students working collaboratively.

Furthermore, as outlined in heuristic 9, there remains a significant need for individual learning, not all
learning should be made collaborative and not all learning material is suitable for collaborative learning.
Additionally, some learners, such as those with a dominant read-write modal preference (Heuristic 16.3),
are not as responsive to collaborative learning as others.

An important factor is that technology-enhanced collaborative learning can offer limited value in a
classroom context; depending on the learning context and technology used, often students can more
naturally collaborate in person.

As previously commented, it should be remembered that to some extent, collaborative learning remains
out of step with current educational and assessment practices in schools.

4.7.3.2 Heuristic 11.1: Support Collaborative and Situated Learning via Mobile
Devices

Arguably, we are currently living through a paradigm shift from education in formal settings towards
education that extends beyond the classroom to become more situated, personal, collaborative and
informal. This paradigm shift is supported by the explosion of mobile devices, their significantly enhanced
capabilities, pervasive wireless networks and cloud computing that enable communication, collaboration
and sharing of information resources almost anywhere.

This heuristic and its guidelines build upon and extend the research in heuristic 11, by focusing on mobile
learning (m-learning) research (Mayes & de Freitas 2004; Anderson 2008; Shen et al. 2009; Caballé et al.
2010; Chai & Lim 2011). To support m-learning the following guidelines should be considered:

1. The instructional design of the e-learning software and collaborative learning activities should
support asynchronous usage (self-directed learning).

122
 CHAPTER FOUR

2. The implementation of the e-learning software should be developed considering technologies

that are commonly accessible on mobile devices (e.g. web browsers, HTML5, audio and video
playback and recording, etc.).
3. The instructional design and implementation of the e-learning software should support
“responsive design”, enabling intrinsic support for the form factor and characteristics of mobile
devices.
4. The instructional design and implementation of the e-learning software should be enhanced
to take advantage of learning opportunities only available on mobile devices (e.g. location
awareness, device to device communication, gyroscopic control, etc.).

There are three main benefits of supporting mobile devices for e-learning:

1. Today’s mobile devices are ubiquitous; enhancing the e-learning software for use with mobile
devices immediately increases the access points available for the software.
2. By design, mobile devices excel at multimedia presentation, communication and collaboration.
3. Learning can happen anywhere and at any time; the inherent mobility of mobile devices means
student learning is no longer confined to the classroom and can become more situated; it can
happen almost anywhere either formally or in an informal context.

Although there is a general increase in the size of mobile phones and a trend towards larger size phablets,
there remains a concern that the relatively restricted form factor of mobile phones and tablets may pose
some restrictions on the educational content and inhibit true active engagement, thereby increasing the
risk that m-learning may lead to more passive information consumption.

As noted in the parent heuristic, an important factor is that technology-enhanced collaborative learning
offers limited value in a classroom context in which students can much more naturally collaborate in
person. However, there are some specific classroom uses that can add educational value, such as
anonymous student questions or instant polling functions that allow immediate feedback on the lesson.

It should also be noted that to some extent, m-learning is out of step with current educational and
classroom practices in high schools, and as such may not be an immediate priority to be supported by e-
learning software.

4.7.4 Connectivism
Building on the review of connectivism presented in section 2.4.6, this section further discusses
connectivist principles and elaborates on how they are distilled into the e-learning heuristics outlined in
Table 17.

ID Heuristic Title

1.1 Ensure the currency of learning material.

12 Develop and nurture networks to support learning.

Table 17: E-learning heuristics based on connectivist principles

123
 CHAPTER FOUR

4.7.4.1 Heuristic 1.1: Ensure the Currency of Learning Material

The currency of information and learning material is strongly related to the constructivist principle of
authenticity; hence, it is positioned as a sub-heuristic. The nature of information is that it is constantly
changing; therefore, the overriding aim is that by focusing on the currency of learning material (nodes),
accuracy and validity must be re-evaluated, which in turn leads to a re-evaluation of existing knowledge
and the possibility to learn more.

The process by which students refresh their knowledge is a learning process that cannot be automated,
but can be supported by the e-learning software and collaborative learning environment employing the
following criteria: learning material and activities are up-to-date and easily editable so they can be kept
current; the e-learning software should act as a focal point that recommends other learning resources
(nodes on the learning network) that are current; technology 23 is used to keep up-to-date; and other
people can provide up-to-date information via the CLE.

The criteria in this heuristic provide support for Connectivist principles 1, 2, 3, 6 and 7 (refer to section
2.4.6). However, it should be noted that they do so by following a moderate connectivist approach; as
discussed by Siemens (2008) citing Bonk (2007), teachers, and in this case the e-learning software, can act
as a ‘concierge directing learners to resources or learning opportunities’. Again, in alignment with Siemens
(2008), connectivist principles 4, 5 and 7 are supported by the pedagogy by the acknowledgement and
recommendation that this heuristic requires critical and metacognitive skills that students may not
inherently have; therefore, additional support and instruction from teachers will be required. The
changing role of teachers and the significant need for learner critical and metacognitive thinking is well
documented within connectivism (Siemens 2004; Siemens 2008; Couros 2009; Downes 2006; Boud 2000;
McLoughlin & Lee 2008; Kop & Hill 2008) and other writings (Jonassen 1999).

It is theorised by connectivists that following these underlying principles leads to learning that is aligned
with our current digital society and the NET generation. More specifically, that learning is no longer
constrained by one book or one perspective that may become outdated; instead, through connectivism,
the learner has access to a variety of existing nodes that are kept up-to-date or replaced by new nodes
and new contributors. This gives the learner the opportunity to experience learning that is refreshed and
kept up to date. To some extent, this heuristic is out of step with current educational practice in which
there is a single set syllabus and set texts that are to be examined. It is clear that in the current educational
climate this cannot be abandoned; nonetheless, there remains room for students to search and explore
new information that may be more current.

23
Push and pull technologies can help support the process of refreshing one’s knowledge by delivering changing
learning material. One such technology is Rich Site Summary (RSS) web feeds, which deliver regularly changing web
content to whoever wants to subscribe.

124
 CHAPTER FOUR

4.7.4.2 Heuristic 12: Develop and Nurture Networks to Support Learning

Connectivists propose that knowledge lies in a diversity of opinions, and that this knowledge resides in a
network of interconnected entities called nodes. These nodes can be almost anything with learning value,
such as individuals, groups, systems, fields, ideas, or communities, but for the most part the focus lies on
humans and digital resources. Considering the eight underlying principles of connectivism and the more
moderate connectivist approach already discussed, we arrive at the following heuristic criteria: the e-
learning software should act as a focal point that recommends other learning resources (nodes on the
network) through web links; the e-learning software should promote collaborative learning interactions
with other students and/or teachers; the learning process should be cyclical in nature 24; the e-learning
software and its constituent learning objects should act as nodes on the network 25 ; the e-learning
software can make use of information and other learning resources found on the network 26; and the
learning network, where possible, can be developed into a learning community.

With reference to the creation of a learning community, it should be noted that the e-learning software
should promote collaborative learning interactions (group assignments, collaborative blogging, peer
reviews, etc.) in the CLE, thereby introducing the students to the learning community. However, forming
and maintaining a learning community places significant extra requirements on both teachers and those
acting as facilitators, as well as the learners. With this in mind, the work of Rovai (2002), Brown (2001),
and Blas & Poggi (2007) have been synthesised into a set of guidelines for community building, which is
included in the appendices of the e-learning pedagogy. It should also be noted, as discussed in the
previous connectivist heuristic, the need for critical and metacognitive skills is also required in this
heuristic.

The key benefits of connectivism and the use of learning networks is that students are less constrained
by didactic teaching approaches; instead, they have increased freedom to explore and assimilate learning
from a variety of other resources, such as other people, web pages, blogs, educational videos, animations,
simulations, etc. The exploration of other nodes and networks containing potentially different
perspectives and information allow the students to build their knowledge actively. Furthermore, if there
is a focus on collaborative learning, or the learning network is developed into a learning community, then
there is potential for increased learning performance, improved higher order thinking, and increased
learner motivation and satisfaction.

It should be noted that connectivist principles are not appropriate for all learners and should be
considered with care to not negatively impact cognitive load and the structured environment of the e-
learning software.

24
The learner joins the network to gather information, will make sense of the information they find, update their
understanding, and will later reconnect to the network to share these realizations and new understanding.
25
The e-learning software should be composed of SCORM (Shareable Content Object Reference Model) Learning
Objects, which in themselves can be individual learning nodes on the network for students to learn from.
26
This should be within the limitations of copyright law and acknowledge the original authors.

125
 CHAPTER FOUR

4.7.5 VARK Learning Styles

Building on the review of VARK learning styles presented in section 2.4.7, this section further discusses
VARK principles and elaborates on how they are distilled into the e-learning heuristics outlined in Table
18.

ID Heuristic

16 Use multi-modal learning approaches.

16.1 Support visual modal preference.

16.2 Support aural modal preference.

16.3 Support read-write modal preference.

16.4 Support kinaesthetic modal preference.

Table 18: E-learning heuristics based on VARK learning styles

4.7.5.1 Heuristic 16: Use Multi-modal Learning Approaches

Students are not restricted to only one of the modal preferences (visual, aural, read-write, or
kinaesthetic). It is typical for students to exhibit a preference for one particular mode, and a relative
weakness or strength in some other modes. However, it must be recognised that even the relatively
weaker modes cannot be ignored; even if a student has a strong preference on a certain modality, they
should still be exposed to diverse learning experiences, and encouraged to develop into more versatile
learners. The e-learning software should accommodate the four modalities by providing a variety of
different learning options that consider the different learning styles. By combining a mixture of
approaches, the students are given the possibility to choose the instructional style that best fits their own
individual learning style(s).

The theoretical foundations and guidelines for this multi-modal learning heuristic are detailed in research
by (Dunn et al. 1981; Felder & Silverman 1988; Fleming & Mills 1992; Fleming 1995; Johnson & Johnson
1996; Jonassen 1999; Jones et al. 2003; Frank Coffield et al. 2004; Mitchell & Savill-Smith 2004; Fleming
& Baume 2006; Zapalska & Brozik 2006; Anderson 2008; Fadel & Lemke 2008; Gardner 2009; Pritchard
2009; Leite et al. 2010; Othman & Amiruddin 2010; Popescu 2010; Birch et al. 2011; Clark & Mayer 2011;
Fleming 2012). Distilling research in this area recommends that the e-learning software and collaborative
activities should consider a significant subset of the following guidelines:

1. The e-learning software and associated CLE activities incorporate learning material that
supports the four VARK modal preferences. The approaches and activities appropriate for each
modal preference are outlined in the following heuristics.
• 16.1: Support visual modal preference.
• 16.2: Support aural modal preference.
• 16.3: Support read-write modal preference.
• 16.4: Support kinaesthetic modal preference.

126
 CHAPTER FOUR

2. The e-learning software and associated CLE activities should provide an approximate balance
between the four modal preferences.
3. The choice and balance of modal channels must carefully consider and align with the learning
context and content. Not all modal channels are appropriate or optimal for conveying specific
learning material.
4. The e-learning software and associated CLE activities should support the four modal
preferences without causing cognitive overload to the students.

As discussed in section 2.4.7, the successful implementation of VARK learning styles and multi-modal
learning requires the VARK inventory to be used as a catalyst for reflection, self-awareness and dialogue.
It is therefore important to ensure the students have the metacognitive skills and strategies to effectively
utilise the multi-modal material.

Considering the previously mentioned authors, and specifically work from (Weiler 2005; Zapalska & Brozik
2006; Moreno & Mayer 2007; Fadel & Lemke 2008; Pritchard 2009; Othman & Amiruddin 2010; Popescu
2010; Birch et al. 2011; Katsioloudis & Fantz 2012), the use of multi-modal learning can potentially result
in the following educational benefits:

1. Knowledge of learning styles and adjusting instructional approaches and resources to

complement them is shown to increase learning performance.
2. The communication of learning material via multiple modalities provides increased support for
learners to understand and assimilate the learning into existing schemas in long-term memory
(deep learning).
3. Giving focus, or at least options, for students to learn via their preferred perceptual mode
increases their motivation.
4. Exposing students to learning material across the modalities and encouraging their learning
development can enable students to become more versatile learners.
5. The underlying nature of certain concepts, and even certain subjects, makes different modalities
more appropriate in their teaching.

It is important to note that multi-modal approaches must be mitigated by a consideration of cognitive

overload and the work of (Mayer et al. 2001; Moreno & Mayer 2002; Mayer et al. 2003; Mayer & Moreno
2003; Moreno & Mayer 2007; Fadel & Lemke 2008; Anderson 2008; Alonso et al. 2009; Birch et al. 2011)
on dual encoding; students do not have infinite working memory or cognitive capacity. It is important to
not overload the students with too many competing channels of learning communication. The
instructional design of the e-learning software must ensure that students do not feel pressured into trying
to take in information from all the modal channels, or worse, to take them in and try to compare them.

Additionally, from a more practical perspective, designing and producing multiple versions of
pedagogically appropriate learning material designed to work in concert is a skilled, time consuming, and
potentially expensive approach.

127
 CHAPTER FOUR

4.7.5.2 Heuristic 16.1: Support Visual Modal Preference

Visual learners prefer graphical and symbolic ways of representing information. They have good visual
recall and prefer information to be presented visually, in the form of diagrams, graphs, maps, posters,
displays, etc. In addition, where learning material is complex, includes invisible or difficult to see
phenomena, or has difficult concepts or process steps, then special attention must be given to
visualisation tools that help learners to construct appropriate mental images and visualise activities.

As a sub-heuristic of heuristic 16, this heuristic builds upon the research outlined in its parent, but gives
more specific focus to visual approaches and guidelines as detailed in research by (Fleming & Mills 1992;
Fleming 1995; Jonassen 1999; Jones et al. 2003; Hegarty 2004; Betrancourt 2005; Condie & Munro 2007;
McCartney et al. 2007; Nasr & Ramadan 2008; Jarmon et al. 2009; Pritchard 2009; Othman & Amiruddin
2010; Allen et al. 2011; Clark et al. 2006). Distilling research in this area recommends that the e-learning
software and collaborative activities should support visual learning via an appropriate subset of the
following.

1. Learning material that is rich in visual depictions such as:

Diagrams Maps Mind-maps

Posters Graphs Displays

Flowcharts Multimedia Symbolic representations

Graphical organisers Visual demonstrations Visual modelling

2. Represent thought processes as visual representations.

3. Focus on the big picture with holistic instead of reductionist approaches.
4. Use underlining, highlighters and different colours.
5. Link text with associated diagrams and pictures.
6. Use non-visual learning that is appealing to visual learners:
a. Provide past examples of finished products
b. Activities that allow freedom and emphasise creativity
c. Group learning
d. Roleplay
7. Avoid over focus on word usage, syntax and grammar.
8. Promote activities to convert notes into one-page pictures, and vice versa.
9. Provide the opportunity for students to use diagrams and visual elements in answering
questions and in assignments.
10. Support visualisation by including visual representations of learning material that is very
complex, includes invisible or difficult to see phenomena, or has difficult concepts or process
steps. The e-learning software should include the appropriate use of task or domain-specific
images, animations, simulations, static frames and video to enable students to visualise learning
material.

128
 CHAPTER FOUR

This heuristic builds upon the educational benefits discussed in heuristic 16. However, in addition, with
regard to guideline-10, embodying learning material according to visualisation approaches optimises
essential processing by reducing the complexity of the learning material and presenting it in a way that is
easier for the learner to assimilate. Furthermore, it helps in building accurate mental schemas that can be
integrated with existing knowledge, thereby maximising germane cognitive processing.

4.7.5.3 Heuristic 16.2: Support Aural Modal Preference

Auditory learners prefer to learn from listening. They have good auditory memory and benefit from
lectures, tutorials, discussions with other students and teachers, interviewing, hearing stories, audio
tapes, etc.

As a sub-heuristic of heuristic 16, this heuristic builds upon the research outlined in its parent, but gives
more specific focus to aural approaches and guidelines, as detailed in research by (Fleming & Mills 1992;
Fleming 1995; Pritchard 2009; Othman & Amiruddin 2010; Allen et al. 2011; Birch et al. 2011). Distilling
research in this area recommends that the e-learning software and collaborative activities should support
aural learning via an appropriate subset of the following:

1. Give additional focus to auditory learning material.

2. Promote lectures and tutorials that are primarily focused around hearing the teacher talk.
3. Emphasise oral presentation, instructions, questioning, answers and reward.
4. Promote discussion activities with other students and/or teacher.
5. Promote activities to orally describe overheads, pictures and other visuals to somebody else.
6. Promote activities to record interesting examples, stories and jokes as memory aids.
7. Promote activities to create audio versions of instructional texts and learner notes.
8. Promote activities to interview experts.
9. Promote activities to read written notes aloud.
10. Promote activities to supplement existing written notes by talking with others and collecting
notes from the textbook or other learning resources.
11. Promote activities to orally report your understanding of a topic or explain your notes to
another aural person.

Audio recording and playback has become quite an easy activity using an audio player/recorder or via the
vast majority of mobile phones and computers. Likewise, there is a large number of audio/video streaming
and chat software that can be used for lectures, tutorials and discussions. The aural approaches outlined
can be used in face-to-face activities, or implemented in the e-learning software or through pedagogically
appropriate activities in the CLE. However, it is important that the e-learning software gives support to
either enable / disable audio, and remains pedagogically effective even when audio is disabled.

Although, the recording of audio has become a relatively simple activity, it still may require additional
support from teachers to educate students and potentially to give the student access to audio recording
equipment. Furthermore, audio recording and playback activities are complicated by a shared classroom

129
 CHAPTER FOUR

environment and noise overlap between students. This can be partially mitigated by headphones, but this
in turn becomes an additional hardware resource necessary to support the e-learning software.

4.7.5.4 Heuristic 16.3: Support Read-write Modal Preference

Read-write learners prefer to learn through information displayed as a word; they benefit from lecture
notes, note-taking, journals, lists, definitions, textbooks, etc.

As a sub-heuristic of heuristic 16, this heuristic builds upon the research outlined in its parent, but gives
more specific focus to read-write approaches and guidelines, as detailed in research by (Fleming & Mills
1992; Fleming 1995; Pritchard 2009; Othman & Amiruddin 2010; Allen et al. 2011). Distilling research in
this area recommends that the e-learning software and collaborative activities should support read-write
learning via an appropriate subset of the following.

1. Give increased focus on individual learning.

2. Avoid vague, non-specific activities in favour of giving more concrete direction on expectations
and deliverables.
3. Provide written learning material (lecture notes, handouts, and references to textbooks and
manuals).
4. Use lists, headings, glossaries and definitions.
5. Promote the use of:
a. Written directions
b. Written questions
c. Well-structured open-ended questions with text response
d. Essay writing activities
e. Journaling activities
f. Word wall activities (wordle)
6. Allocate reading time.
7. Promote activities to write notes, then rewrite and reread repeatedly as a revision tactic.
8. Promote activities to rewrite subject ideas and principles using different words.
9. Promote activities to reconstruct any visual elements such as diagrams, graphs, charts etc. into
textual statements.
10. Promote activities to reconstruct actions, events or behaviours into textual statements.
11. Promote activities to arrange learning material and notes into titles, hierarchies and points.
12. Promote activities to represent list-based learning material into multiple-choice questions.

The above points can all be implemented via the appropriate design and implementation of the e-learning
software and CLE activities; specifically, the use of: note apps, e-portfolios, cloud–based word processors,
Wikis, blogging, journals and text chats.

It should be noted that read-write learners often prefer individual learning, so less focus should be given
to group and collaborative learning; however, the CLE can still be used for activities such as Wikis,
blogging, journals and text chats.

130
 CHAPTER FOUR

4.7.5.5 Heuristic 16.4: Support Kinaesthetic Modal Preference

Kinaesthetic learners prefer learning that connects to their experience and reality. They are more adept
at recalling events and associated feelings, or physical experiences from memory. This experience can be
derived from physical activity such as field trips, manipulating objects, and other practical first-hand
experience. However, it can also be derived through simulation and the presentation of information
strongly tied to experience and reality. Hence, kinaesthetic learning can be multi-modal since the
information describing experience and reality can be presented in a visual, aural or read-write form.

As a sub-heuristic of heuristic 16, this heuristic builds upon the research outlined in its parent, but gives
more specific focus to kinaesthetic approaches and guidelines, as detailed in research by (Fleming & Mills
1992; Fleming 1995; McCartney et al. 2007; Pritchard 2009; Othman & Amiruddin 2010; Allen et al. 2011;
Fleming 2012). Distilling research in this area recommends that the e-learning software and collaborative
activities should support kinaesthetic learning via an appropriate subset of the following:

1. Promote practical activities (experiments) either real or simulated that engage understanding
by doing, i.e. by manipulating objects, building or constructing, or hands-on projects.
2. Promote learning material directly connected to experience and reality.
3. Promote learning based on real-life examples.
4. Provide case studies and real-life applications to help with the understanding of principles and
abstract concepts.
5. Provide questions based on practical activities.
6. Provide learning based on live demonstrations.
7. Promote activities focused on finding solutions to real-life problems.
8. Promote activities that incorporate an element of trial and error.
9. Provide learning material that uses exhibits, samples, pictures and photographs that illustrate
an idea and tie back to real life.
10. Promote activities to recall experiments and physical experiences from memory.
11. Promote learning that uses multiple senses, such as sight and hearing, or tries to evoke the
senses of touch, taste and smell.
12. Promote activities that use previous exam papers and conditions.

It is important to reaffirm that although kinaesthetic learners learn from real-life and personal experience,
and find great value in physical activity, they still derive significant learning value from simulations and
other learning that is directly linked to reality or experience. The kinaesthetic learner cannot physically
experience everything; what is important is that the sense of experience and reality is conveyed to them.
The latter can be aptly provided by educational technology.

131
 CHAPTER FOUR

4.7.6 ARCS Theory of Motivation

Building on the review of ARCS Theory of Motivation presented in section 2.4.8, this section further
discusses ARCS principles, and elaborates on how they are distilled into the e-learning heuristics outlined
in Table 19.

ID Heuristic

14 Use the concepts of Attention, Relevance, Confidence and Satisfaction (ARCS) to attain and sustain learner
motivation.

14.1 Use “Attention” grabbing strategies to increase learner motivation.

14.2 Explain the “Relevance” of the learning material to increase motivation.

14.3 Build “Confidence” to increase learner motivation.

14.4 Build “Satisfaction” to increase learner motivation.

Table 19: E-learning heuristics based on ARCS theory of motivation

4.7.6.1 Heuristic 14: Use the Concepts of Attention, Relevance, Confidence and
Satisfaction (ARCS) to Attain and Sustain Learner Motivation

Irrespective of the effectiveness of the learning material, without motivation, students are hampered
from learning, and whilst motivation cannot be directly controlled, it can be positively influenced.
According to the ARCS model, to predictably improve motivation and performance, the instructional
material and environment should capture the learners’ attention, ensure relevance to the learner, build
learner confidence and ensure learner satisfaction.

As a parent heuristic, heuristic 14 distils research in the ARCS motivational model, and recommends that
the e-learning software and collaborative activities should consider a balanced subset of the following
sub-heuristics:

• 14.1: Use “Attention” grabbing strategies to increase learner motivation.

• 14.2: Explain the “Relevance” of the learning material to increase motivation.
• 14.3: Build “Confidence” to increase learner motivation.
• 14.4: Build “Satisfaction” to increase learner motivation.

Considering the research of (Johnson et al. 1984; Hidi & Harackiewicz 2000; Lepper et al. 2005; Nuutila et
al. 2005; Virvou et al. 2005; Schraw et al. 2006; Condie & Munro 2007; Keller 2008; Clark & Ernst 2009;
Papastergiou 2009; Shroff & Vogel 2009; Liu et al. 2011; Muntean 2011; de Freitas & de Freitas 2013; Kim
et al. 2014; Pirker et al. 2014) the successful implementation of motivational instructional techniques, and
specifically the ARCS model, can potentially result in the following educational benefits.

1. A positive impact on learners’ interest, engagement, and motivation;

2. Due to the increased learner motivation, a resultant improvement in learning and academic
success;
3. An increased likelihood that learners will persevere in difficult situations and approach
demanding activities more enthusiastically than less motivated peers;

132
 CHAPTER FOUR

4. A decrease in dropout rates in distance learning courses from motivated learners; and
5. The emphasis of ARCS heuristics on authentic learning, practice activities and increased learner
control can also lead to deeper learning.

For further information on the potential challenges of this parent heuristic, refer to the sub-heuristics
listed previously.

4.7.6.2 Heuristic 14.1: Use “Attention” Grabbing Strategies to Increase Learner

Motivation

The first step in increasing learner motivation is to capture their attention, and then employ strategies to
sustain their attention throughout the learning process. This involves an initial inquiry arousal, stimulating
a deeper level of curiosity and then sustaining attention via varied instructional techniques.

This heuristic builds upon the research outlined in its parent (Heuristic 14), but gives specific focus to
approaches and guidelines to capture and maintain learner attention. Distilling research in this area by (J.
M. Keller 1987a; J. M. Keller 1987b; Lebow 1993; Huffaker & Calvert 2003; Karagiorgi & Symeou 2005;
Nuutila et al. 2005; Palmer 2005; Connolly & Stanfield 2006; Schweitzer & Brown 2007; Anderson 2008;
Chang & Smith 2008; Keller 2008; Mayer & Griffith 2008; Hattie 2009; Jang 2009; Papastergiou 2009; Shen
et al. 2009; Shroff & Vogel 2009; Alsumait & Al-Osaimi 2010; Lee & Choi 2011; Liu et al. 2011; Stuchlikova
& Benkovska 2011; Stott & Neustaedter 2013; Pirker et al. 2014) recommends that the instructional
design and implementation of the e-learning software and associated collaborative activities should
consider an appropriate subset of the following guidelines:

1. Capture student attention at the start of the learning process. This can be achieved by
stimulating graphics, animations and/or instructional material that invoke:
a. A sense of wonderment, incongruity, conflict, or personal or emotional resonance with
the students.
b. An orienting reflex based on an unexpected change or novel stimulus that attracts
student attention.
c. A sensation-seeking reflex by stimulating the learners’ inherent inclination to pursue
sensory pleasure, excitement, and experiences that are varied, novel, complex or
intense.
2. Stimulate a deeper level of curiosity by fostering the learner’s inherent nature to explore,
discover and understand.
a. Instructional material that uses progressive disclosure to maintain suspense and
gradually build learning.
b. Question-based techniques that prompt thinking and inquiry.
c. Activities that create a paradox or that arouse a sense of mystery, problem-solving, or
experiential situations. These in turn encourage a sense of inquiry and knowledge-
seeking behaviour.

133
 CHAPTER FOUR

3. Maintain attention via variable instructional design: avoid using the same instructional
approaches repeatedly. Instead, include variations in instructional design, presentation style and
modality.

The successful implementation of heuristic 14.1, as part of a balanced implementation of all the sub-
heuristics of heuristic 14, can potentially result in the educational benefits listed in heuristic 14. However,
it is important that this heuristic is implemented with careful and measured instructional design that
avoids unnecessary learning material that could in turn increase cognitive load.

4.7.6.3 Heuristic 14.2: Explain the “Relevance” of the Learning Material to Increase
Motivation

To ensure that motivation is maintained the learner must perceive the learning material has a personal
relevance to them; there must be a “connection between the instructional environment, which includes
content, teaching strategies, and social organization, and the learner’s goals, learning styles, and past
experiences” (Keller 2008, p.177).

This heuristic builds upon the research outlined in its parent (Heuristic 14), but gives specific focus to
approaches and guidelines to explain the relevance of the learning material to the learners. Distilling
research in this area by (Johnson et al. 1984; J. M. Keller 1987a; J. M. Keller 1987b; Schunk 1991; Lebow
1993; Hidi & Harackiewicz 2000; Huffaker & Calvert 2003; Keller & Suzuki 2004; Nuutila et al. 2005; Palmer
2005; Connolly & Stanfield 2006; Nicol & Macfarlane-Dick 2006; Schraw et al. 2006; Condie & Munro 2007;
Long 2007; Schweitzer & Brown 2007; Anderson 2008; Chang & Smith 2008; Keller 2008; Zhang & Bonk
2008; Clark & Ernst 2009; Papastergiou 2009; Shroff & Vogel 2009; Alsumait & Al-Osaimi 2010; Hadjerrouit
2010; Anderman & Dawson 2011; Lee & Choi 2011; Nicholson 2012; Stott & Neustaedter 2013; Morrison
& DiSalvo 2014) recommends that the instructional design and implementation of the e-learning software
and associated collaborative activities should consider an appropriate subset of the following guidelines:

1. Aligning with Learner Goals - The learning material reflects an understanding of the learners’
needs and demonstrates how the new knowledge or skills will support them in achieving their
goals.
a. Communicate clear learning goals that answer: Why should I learn this? What is the
importance of this learning material? How can I use this new knowledge in real-life
situations?
b. Communicate how the learning material and learning goals are related to the learners’
goals.
c. Focus on intrinsic goal orientation in which the learners are engaged in learning that is
personally interesting and freely chosen.
d. Where intrinsic goal orientation is a challenge, then extrinsic goals can be used to tie
learning to an external factor such as earning a reward, passing an exam, college
acceptance, etc.
e. Ask learners to explain their own perceptions of relevance.

134
 CHAPTER FOUR

2. Aligning with Learning Styles - A secondary or supplementary approach in establishing relevance

is related to how something is taught rather than to the substance of what is being taught.
a. Use a variety of pedagogical strategies, such as individual activities and strategies that
focus on cooperative group work, thereby appealing to both learners that prefer
individual work and those that need affiliation.
b. Use a multi-modal approach to appeal to the full range of modal preferences (heuristic
16).
3. Aligning with what is Familiar – Engage the students on a personal level and relate the learning
material back to the learner’s real life.
a. Engage the learners on a personal level by using their names and asking them for their
experience, examples and ideas.
b. Use authentic learning approaches that connect the learning material to intrinsically
interesting topics relevant to the learner’s current situation or future aspirations.

The successful implementation of heuristic 14.2, as part of a balanced implementation of all the sub-
heuristics of heuristic 14, can potentially result in the educational benefits listed in heuristic 14. However,
it should be noted that with e-learning software, as opposed to a classroom environment where the
teacher is dynamically interacting with students, there is an increased challenge in building a personalised
learning profile of the learner’s individual goals and motives. As such, the learning profile is typically
generalised based on the target audience and the learner analysis undertaken at the start of the
motivational design process.

4.7.6.4 Heuristic 14.3: Build “Confidence” to Increase Learner Motivation

Building learners’ confidence in their ability to learn also increases their motivation to learn; any learned
helplessness or fear of the topic, skill or environment that hinders learning should be addressed and
replaced by an expectation of success. The positive expectancy for success should then be followed
promptly by actual success that the learners can clearly attribute to their own abilities and effort.

This heuristic builds upon the research outlined in its parent (Heuristic 14), but gives specific focus to
approaches and guidelines to build learner confidence. Distilling research in this area by (J. M. Keller
1987a; J. M. Keller 1987b; Schunk 1991; Lebow 1993; Black & Wiliam 1998; Sweller et al. 1998; Jonassen
1999; Hidi & Harackiewicz 2000; Keller & Suzuki 2004; Karagiorgi & Symeou 2005; Palmer 2005; Connolly
& Stanfield 2006; Nicol & Macfarlane-Dick 2006; Schraw et al. 2006; Condie & Munro 2007; Anderson
2008; Chang & Smith 2008; Keller 2008; Black & Wiliam 2009; Clark & Ernst 2009; Hattie 2009;
Papastergiou 2009; Shen et al. 2009; Shroff & Vogel 2009; Alsumait & Al-Osaimi 2010; Corbalan et al.
2010; Hadjerrouit 2010; Anderman & Dawson 2011; Liu et al. 2011; Nicholson 2012; de Freitas & de Freitas
2013; Stott & Neustaedter 2013; Kim et al. 2014; Morrison & DiSalvo 2014; Pirker et al. 2014) recommends
that the instructional design and implementation of the e-learning software and associated collaborative
activities should consider an appropriate subset of the following guidelines:

1. Establish trust and positive expectations for learning success:

135
 CHAPTER FOUR

a. Inform the learners of what is expected from them by explaining the requirements for
success and the evaluation criteria.
b. Address any fear or anxiety held by the learners.
c. If using a progressive disclosure strategy, the above points will need to be softened to
not remove suspense.
d. Provide ongoing encouragement (coaching) to complete the lesson.
2. Provide opportunities for meaningful success:
a. Opportunities for success should be provided as soon as possible in the lesson.
b. Meaningful success is contingent on there being enough challenge to require a degree
of effort to succeed, but not so much that it creates serious anxiety.
c. The success must be clearly attributable to the learner’s efforts and ability rather than
to luck or the task being too easy.
d. If the learning material is new, the level of challenge should be relatively modest and
combined with frequent feedback that confirms the learner’s success or helps the
learner progress towards success.
e. When the basics are mastered, progress to higher levels of challenge that help learners
exercise and sharpen their skills.
f. Pacing should be adjusted as competency levels change; typically, success activities
should move quickly from simple to complex, or known to unknown, to avoid boredom,
but not so quickly that learners become anxious.
3. Balance a stable e-learning environment with the learners’ need to feel in control and
responsible for their success:
a. Provide a stable learning environment that sets the standards expected and guides the
learning experience.
b. Within that stable learning environment, allow the learners as much personal control
(as possible) over their actual learning experience. Techniques that support such
personal control are:
i. Fostering an environment where it is all right to make mistakes and learn from
them.
ii. Using learning activities and problem-based methods that require the learner
to exercise personal control and judgement to solve.
iii. Providing corrective feedback that helps the learner see the cause of their
mistake and allows them to learn and retry.

The successful implementation of heuristic 14.3, as part of a balanced implementation of all the sub-
heuristics of heuristic 14, can potentially result in the educational benefits listed in heuristic 14. However,
e-Learning software used in isolation, without teacher involvement, cannot effectively address learners’
fear and anxieties. Furthermore, the instructional design of the e-learning software will require careful
balance in the following areas:

136
 CHAPTER FOUR

• The guideline to explain to learners the learning objectives, requirements for success and the
evaluation criteria may need to be mitigated when using progressive disclosure techniques and
problem-based learning (Heuristic 4).
• The criteria related to giving learners personal control in their learning needs to be balanced
against the provision of a stable learning environment (Heuristics 21, 21.1).
• The provision of success opportunities creates a challenge to move quickly enough to avoid
boredom, but not so quickly that learners become anxious.

4.7.6.5 Heuristic 14.4: Build “Satisfaction” to Increase Learner Motivation

Once motivation is inspired in the learner, it needs to be maintained by providing the learner with a sense
of satisfaction with the process and/or results of the learning experience. This is achieved by a
combination of intrinsic methods, extrinsic reinforcement, and a sense of fairness in the learning results.

This heuristic builds upon the research outlined in its parent (Heuristic 14), but gives specific focus to
approaches and guidelines to build learner satisfaction. Distilling research in this area by (Johnson et al.
1984; J. M. Keller 1987a; J. M. Keller 1987b; Schunk 1991; Lebow 1993; Jonassen 1999; Hidi & Harackiewicz
2000; Huffaker & Calvert 2003; Keller & Suzuki 2004; Karagiorgi & Symeou 2005; Nuutila et al. 2005;
Palmer 2005; Connolly & Stanfield 2006; Nicol & Macfarlane-Dick 2006; Schraw et al. 2006; Condie &
Munro 2007; Long 2007; Schweitzer & Brown 2007; Anderson 2008; Chang & Smith 2008; Keller 2008;
Clark & Ernst 2009; Hattie 2009; Jang 2009; Papastergiou 2009; Shen et al. 2009; Shroff & Vogel 2009;
Alsumait & Al-Osaimi 2010; Boyle et al. 2010; Hadjerrouit 2010; Anderman & Dawson 2011; Liu et al. 2011;
Muntean 2011; Bertacchini et al. 2012; Nicholson 2012; Yang & Chang 2012; de Freitas & de Freitas 2013;
Stott & Neustaedter 2013; Pirker et al. 2014; Aritajati et al. 2015) recommends that the instructional
design and implementation of the e-learning software and associated collaborative activities should
consider an appropriate subset of the following guidelines:

1. Use intrinsically motivational learning experiences.

a. Provide meaningful opportunities for learners to use their newly-acquired knowledge
and skills. These opportunities typically have the following characteristics:
i. They are authentic problems, simulations, or worked examples that allow
learners to apply what they have learnt and help to transfer from classroom to
real-life.
ii. They focus on providing the learner with the intrinsic satisfaction that they
have mastered the learning.
iii. They are typically scheduled towards the end of the lesson or the course.
b. Provide opportunities for learners to coordinate, collaborate, and interact with other
learners.
c. Provide opportunities for learners to have their views heard and respected.
d. Encourage in learners a sense of control over their learning experience.

137
 CHAPTER FOUR

e. Support learners in seeing how the various pieces of the learning experience fit together
holistically.
2. Provide positive extrinsic reinforcement to learners’ successes.
a. Extrinsic reinforcement can be used to supplement, but not replace, intrinsic
motivational approaches; both should be used in combination.
b. Extrinsic reinforcement includes the use of opportunities for advancement, verbal
praise, certificates, and any real or symbolic rewards and incentives that provide
external recognition of achievement.
3. Ensure learners perceive the learning process, assessment, and rewards as being fair.
a. Ensure that course outcomes are consistent with the initial presentation of course
purpose and expectations.
b. Assessment results and extrinsic reinforcements are not seen in isolation or in terms of
absolute value. They need to be consistent, commensurate with the efforts expended
in the learning, and comparatively fair in relation to learning peers.

The successful implementation of heuristic 14.4, as part of a balanced implementation of all the sub-
heuristics of heuristic 14, can potentially result in the educational benefits listed in heuristic 14. However,
with reference to guideline-1.b, as noted in heuristic 11, collaborative activities are not appropriate in all
circumstances, and therefore should not be considered to always lead to increased satisfaction.

4.7.7 Cognitive Load

Building on the review of cognitive load theory (CLT) presented in section 2.4.9, this section further
discusses CLT research and elaborates on how it is distilled into the e-learning heuristics outlined in Table
20.

ID Heuristic

17 Integrate words and graphics together, instead of words alone.

17.1 Apply contiguity by aligning words (audio or screen text) with corresponding graphics.

17.2 Representing words as audio, on-screen text or both

18 Avoid adding learning content that does not directly support your instructional goal.

19 Optimise essential processing by segmenting learning material and providing pre-training.

20 Use a conversational style in screen text and audio narration.

21 Provide restricted navigational control in the e-learning software.

21.1 Provide consistent navigational elements and signposts for learning.

Table 20: E-learning heuristics based on cognitive load theory

4.7.7.1 Heuristic 17: Integrate Words and Graphics Together, Instead of Words Alone

An important part of active processing is to mentally construct visual and text representations of learning
material, and to mentally connect them. The e-learning software should therefore include both words
(audio or screen text) and graphics (static illustrations, animations or videos) to support learners in

138
 CHAPTER FOUR

developing their mental models. The visual elements should not be treated as an afterthought after the
text has been written; instead, multimedia lessons should contain words and corresponding visuals that
work together to explain the learning content. These visual elements must represent the critical
information and relationships in the learning material, and be designed in consideration of the cognitive
processes necessary for deep learning.

The integration of words and graphics together to support learning is commonly known as the multimedia
principle, and is detailed in research by (Mayer 1989; Chandler & Sweller 1991; Alonso et al. 2005; Fletcher
& Tobias 2005; Butcher 2006; McCrudden & Schraw 2007; Clark & Mayer 2011; Clark et al. 2011). Distilling
research in this area recommends that the e-learning software and collaborative activities should consider
the following guidelines: visual elements27 should be integrated with accompanying aural or printed text;
a variety of graphical types28 can be used in accordance with the intended learning objectives (refer to Vol
2 Appendix E section B.7 for a brief description of these graphical types); and visual elements should be
used to provide navigational support and signposts for learning.

The implementation of the multimedia principle as compared to learning from words only leads to
education benefits such as: improved learning performance; a deeper understanding, in which knowledge
is encoded and remembered better; being particularly relevant for scientific and technical learning that
can be complex; and being especially important for novice learners (with low domain knowledge) who are
less experienced in creating their own mental models.

It should be noted that the implementation of this heuristic is further governed by two sub-heuristics:

• 17.1 Apply contiguity by aligning words (audio or screen text) with corresponding graphics.
• 17.2 Representing words as audio, on-screen text, or both

In addition, guideline-3 of this heuristic refers to visual elements providing navigational support and
signposts for learning, this is elaborated in sub-heuristic 21.1.

4.7.7.2 Heuristic 17.1: Apply Contiguity by Aligning Words (Audio or Screen Text) with
Corresponding Graphics

Building on heuristic 17, it is important to avoid learning material that requires learners to split their
attention between, and mentally integrate, multiple sources of information (Ayres & Sweller 2005). The
process of integrating distinct sources of information creates an extraneous cognitive load that can be
avoided by aligning and integrating words (audio or screen text) in close proximity (i.e. contiguous) to
corresponding graphics.

The theoretical foundations of this heuristic are based on the split-attention (contiguity) principle that is
detailed in research by (Mayer 1989; Chandler & Sweller 1991; Sweller et al. 1998; Moreno & Mayer 1999;

27
Illustrations, diagrams, photographs, maps, concept-maps, charts and graphs, animations, videos, etc.
28
Representative, organisational, relational, transformative, interpretative and decorative.

139
 CHAPTER FOUR

Kalyuga & Ayres 2003; Sweller 2004; Alonso et al. 2005; Ayres & Sweller 2005; Fletcher & Tobias 2005;
Mayer 2005; Kester et al. 2006; Clark et al. 2006; Clark & Mayer 2011). Distilling research in this area
recommends that the e-learning software and collaborative activities should support contiguity via the
following guidelines:

1. For contiguity to apply, the multiple sources of information are essential for understanding,
and difficult to understand in isolation. Careful consideration must be given to the logical
relation between the information: whether the information is complex, whether understanding
is possible without integration, whether integration is negated by the learners’ experience level,
and whether the same information is being presented redundantly.
2. Place printed words near corresponding graphics. Text should not be placed above or below the
corresponding graphic, but as close as possible to (even in) the graphic. When parts of an object
are being described, place the text close by and, where necessary, use a pointed line to connect.
When a process is being described, number and place the text close by and, where necessary,
use a pointed line to connect.
3. Avoid the overuse of text. Long passages and too much text can become discouraging for many
learners. This should be avoided by using multiple editing iterations to cut down the text to the
bare essence. Tips on how to reduce unnecessary text are outlined in Vol 2 Appendix E section
B.9.
4. Synchronise spoken words with corresponding graphics. A spoken word narration that
describes an animation or video should play at the same time and be temporally synchronised
with the corresponding animation or video. Providing separate controls for the audio and video
or providing the audio and video in succession should be avoided.

To reflect this heuristic an example of contiguity is presented by Mayer (1989, p.241) in Figure 19.

Figure 19: Example of contiguity - Illustration of hydraulic braking system

140
 CHAPTER FOUR

The successful implementation of this heuristic reduces extraneous cognitive load by removing the need
for learners to mentally integrate disparate learning sources. This in turn allows the learner to focus their
cognitive resources on understanding the learning material. Furthermore, by integrating learning material
in this manner, they are held together in working memory, therefore making a meaningful connection
between them. This in turn is fundamental to the sense-making process that leads to deeper learning.

A challenge with this heuristic is when the student is expected to simultaneously focus on written text
and graphics, this can in turn lead to cognitive overload; guidance for this scenario is detailed in heuristic
17.2.

In addition, it should be noted that this heuristic is most beneficial to low-knowledge learners needing
support in developing their mental models and is impacted by the expertise reversal effect, whereby
experienced learners cannot avoid the integrated format, which in turn can interfere with existing mental
models (Moreno & Mayer 1999; Kalyuga & Ayres 2003).

4.7.7.3 Heuristic 17.2: Representing Words as Audio, On-screen Text, or Both

When words accompany visual elements, and both require the learner’s simultaneous attention, it is
typically better to present the words as audio instead of on-screen text. This avoids cognitive overload
(refer to Figure 20) by balancing the learning material across two separate cognitive channels - words in
the auditory channel and graphics in the visual channel (the modality principle). Furthermore, it is typically
recommended to not duplicate words via audio and screen text (the redundancy principle). This avoids
situations where the learner focuses too much on-screen text to the detriment of the graphics, or
potentially focusing on the screen text and narration and comparing whether they are equivalent (refer
to Figure 21).

Figure 20: Overload of visual channel with presentation of screen text and graphics.

Source (Clark & Mayer 2011, p.122).

Figure 21: Overload of visual channel with graphics explained by words in audio and screen text.

141
 CHAPTER FOUR

Adapted from (Clark & Mayer 2011, p.123).

The modality and redundancy principles are detailed in research by (Sweller et al. 1998; Kalyuga 1999;
Moreno & Mayer 1999; Kalyuga et al. 2000; Mayer et al. 2001; Craig et al. 2002; Moreno & Mayer 2002;
Kalyuga & Ayres 2003; Mayer et al. 2003; Mayer & Moreno 2003; Mayer 2005; Ginns 2006; Harskamp et
al. 2007; Jamet & Le Bohec 2007; Lee et al. 2008; Mayer & Johnson 2008; Clark et al. 2006; Clark & Mayer
2011). Distilling research in this area recommends an increased focus on applying audio narration with
visual elements, but also requires more discerning decision making in identifying when on-screen text
remains the best learning option. The following guidelines apply:

1. When explaining graphical elements, it is better to avoid duplicating words in both audio and
screen text.
2. Words communicated in audio form should be preferred over on-screen text if the text needs
to be synchronised with more dynamic visual elements, such as animations, videos, or a series
of static frames.
3. The audio material must be clear and concise, and synchronised with the visual learning
material.
4. When you do not have a simultaneous graphical presentation then modality does not apply,
and screen text can be presented alone, or simultaneously with audio.

However, exceptions to these guidelines do apply (Kalyuga 1999; Moreno & Mayer 2002; Mayer &
Johnson 2008; Clark & Mayer 2011):

5. In some scenarios, keywords should still be highlighted on screen with visual elements to act
as a graphical organiser, and to direct the learner’s attention. This aids retention.
6. With certain learning contexts and learning material, screen text should be preferred even if
there is a simultaneous graphical presentation. For instance:
a. When words should remain available to the learner over time.
b. When the words are technical, unfamiliar or formulae.
c. When the words are not in the learners’ native language.
d. When lengthy text is being presented or is necessary for future reference.
e. When the text lists key steps in a procedure or gives directions in a practical exercise.
7. There remain some conditions when the use of redundant on-screen text in conjunction with
audio narration can give learning benefits:
a. When there is no graphical element, you may decide to have narration and some text,
therefore using dual channels and not overloading either.
b. The scenarios listed in point 6.
c. When the learning material is not complex, or there is ample time to process the visual
elements; for instance, when text and graphics are presented sequentially or when the
pace of the presentation is sufficiently slow.

142
 CHAPTER FOUR

The successful implementation of this heuristic efficiently utilises dual channels to integrate learning
material in a manner that reduces cognitive load and supports retention and deep learning. However, as
noted in the previous sub-heuristic, it is most beneficial for low knowledge learners, and is susceptible to
the expertise reversal effect (Moreno & Mayer 1999; Kalyuga et al. 2000; Kalyuga & Ayres 2003; Harskamp
et al. 2007).

Additionally, increased focus on audio material involves some additional practical challenges, such as:
introducing too much noise into the learning environment; adding cost and potential delay to the
development process; and is comparatively restrictive in supporting quick and easy changes when
learning material changes.

4.7.7.4 Heuristic 18: Avoid Adding Learning Content that does not Directly Support
your Instructional Goal

Considering humans limited cognitive capacity, it is seemingly intuitive and evident that learning content
should be focused on directly supporting the instructional goals. However, frequently there is a strong
implicit temptation to add extra material in e-learning that will either grab student attention and keep
them engaged, or add extra explanatory detail that is nonessential. This can lead to interesting but
unnecessary learning material, the use of overly dramatic stories and examples, and gratuitous use of
text, audio, visual and multimedia elements, which in turn can harm the learning process.

This heuristic is founded on research into the arousal effect 29, coherence principle and seductive details
effect as detailed in research by (Chandler & Sweller 1991; Harp & Mayer 1998; Moreno & Mayer 2000a;
Mayer et al. 2001; Mayer et al. 2003; Mayer & Moreno 2003; Alonso et al. 2005; Sanchez & Wiley 2006;
Lehman & Schraw 2007; Mayer & Griffith 2008; Clark & Mayer 2011). The inclusion of extraneous learning
material can sap limited cognitive capacity and interfere with the learner’s endeavour to make sense of
the learning material by a combination of distraction, disruption and diversion (Harp & Mayer 1998;
Mayer et al. 2001; Mayer & Griffith 2008; Clark & Mayer 2011):

1. Seductive details distract limited attention away from the main ideas of the learning material;
2. Seductive details disrupt and interfere with the construction of an organised mental model
(schema) by introducing noise (unnecessary learning material) into the learning process;
3. Seductive details divert correct schema integration by activating inappropriate prior knowledge,
which is then used to incorrectly organise the incoming learning material.

However, research findings from Mayer and Griffith (2008) give greater credence to diversion being the
underlying cause for negative learning impact.

Distilling research in this area (Harp & Mayer 1998; Kalyuga et al. 2000; Moreno & Mayer 2000a; Moreno
& Mayer 2000b; Knez & Hygge 2002; Sanchez & Wiley 2006; Lehman & Schraw 2007; Clark et al. 2011;

29
The arousal effect theorises that the inclusion of entertaining and interesting material causes learners to become
more emotionally aroused, and therefore work harder to learn the material (Mayer et al. 2001).

143
 CHAPTER FOUR

Clark & Mayer 2011) promotes an increased focus on learning material that is concise and focused on the
intended learning outcomes, via the following guidelines:

1. The learning material should not be embellished with unnecessary detail, or with graphics,
audio, text or multimedia that focus only on creating interest.
2. The interest and engagement should come from the core learning content and instructional
design.
3. Avoid the addition of environment sounds and / or background music since it can lead to a
reduction in learning retention and transfer.
4. Avoid pictures and graphics that are purely decorative since they will not lead to improved
learning.
5. Avoid visual elements that are somewhat (indirectly or tentatively) related to learning
objectives since they disrupt the learning process.
6. Use simpler visuals. Simpler visuals include visuals with fewer details presented at one time.
7. Avoid adding extraneous text with embellished textual or narrative descriptions in preference
to concise focused text or narrative.
8. Avoid lengthy audio-visual material which can cause learner frustration from having to view
and/or listen through all the material to extract the relevant learning.

The avoidance of unnecessary, embellished, overly detailed or decorative learning content enables a truer
focus on instructional goals that, in turn, positively influences both recall and deep cognitive processing
of learning material. However, due concern should be taken in instructional design to balance
constructivist principles, and the need to gain the learner’s attention, against the possibility of causing
cognitive overload.

4.7.7.5 Heuristic 19: Optimise Essential Processing by Segmenting Learning Material

and Providing Pre-training

As discussed in section 2.4.9, in cognitive learning theory, essential processing reflects the learning
processes used by the student to understand the core learning material. It is fundamental to the learning
process, but can be greatly impacted by the inherent complexity of the material. Complex learning
material can overwhelm the learner’s working memory to the extent that they do not have enough
remaining processing capacity to mentally organise the incoming material and integrate it with existing
schemas (deep learning and far transfer).

Learning complexity is discussed by (Pollock et al. 2002; Kalyuga & Ayres 2003; Sweller 2004; Jonassen &
Hung 2008; Clark & Mayer 2011) as being a product of the number of individual elements of information
to be processed, and the number of interrelationships between these elements that also need to be
processed. Therefore, rather than treating complex learning material as a whole; deep learning is more
effectively supported by managing complexity via two approaches: segmenting breaks a lesson into
manageable segments that do not overload the student’s cognitive processes; and pre-training provides

144
 CHAPTER FOUR

foundation information that gives names and characteristics of key concepts that can be built upon and
used in the main learning segments.

Concepts of segmenting and pre-training have been studied by (Mayer & Chandler 2001; Pollock et al.
2002; Mayer & Moreno 2003; Mayer et al. 2003; Sweller 2004; Kester et al. 2006; Moreno 2007; Moreno
& Mayer 2007; Anderson 2008; Clark & Mayer 2011; Clark et al. 2011) and have been distilled into the
following guidelines:

1. Provide a stable foundation for learning by giving pre-training which orients the learner and
explains terminology and pre-requisite concepts. Even when learning material is segmented, it
can sometimes still be complex or introduce a lot of unfamiliar terms or concepts. The pre-
training principle reduces this complexity by giving the learner an orientation which explains
terminology or pre-requisite concepts. In effect, this redistributes some of the student’s essential
processing from the main lesson to the pre-training, and provides a stable foundation for the
student to build their learning on.
2. Break learning material into smaller segments and present them sequentially. Segmenting
helps the learner manage complexity by breaking the lesson into smaller pieces that convey just
two or three steps in the process, or describe just two or three major relations between
elements. These segments are then presented sequentially, under the learner’s controlled
progress, which in turn allows the learner to digest the material at their own pace.

In addition, based on qualitative findings from this study (refer to section 6.5), segmented learning
material can be organised into metaphorical chapters. To help transition students to new e-learning
concepts it can be helpful to mimic approaches they know and feel comfortable with. The e-learning
software can be structured as chapters, complete with learning objectives, learning material, an end of
chapter summary, and end of chapter review questions.

The implementation of this heuristic does not necessarily lead to improvements in immediate learning
performance (retention) but improves deep learning (understanding) and supports improved learning as
measured in learning transfer tests. In addition, using metaphorical chapters gives structure to the
learning experience, and using the chapter summary and review questions reasserts to learners the
important learning material necessary for the examinable curriculum.

However, it should be noted that this heuristic is most effective with students who are beginners in the
subject area. More expert learners, able to call upon a robust set of schemas on the subject matter, are
more adept at managing the complexity and integrating the new learning into their schemas.

To some extent, segmenting learning material and structuring it into chapters goes against constructivist
and connectivist principles, and the discovery inherent in problem-based learning. However, this
pedagogy proposes a balance between extremes and seemingly incompatible approaches. Moderate
approaches should be used that can marry aspects of constructivism and problem-based learning within
a more structured e-learning environment. For example, guided discovery can be used in which problem-

145
 CHAPTER FOUR

based learning is supported by macro-scaffolding (pre-training) on how the problem should be

approached, and just-in-time delivery of segmented learning material to support problem resolution.

4.7.7.6 Heuristic 20: Use a Conversational Style in Screen Text and Audio Narration

Psychology research has shown that personalised messages can promote deep learning by actively
engaging students; learners work harder to understand material when they feel they are in a conversation,
rather than being presented with information. The conversational style works on two levels: it presents
the information; but also primes the appropriate cognitive processes to motivate the learner to try to
make sense of the learning material. It is therefore recommended that the e-learning software should use
a conversational style (using first- and second-person and active language) in both screen text and audio
narration, and should avoid the use of formal and passive voice.

Research by (Moreno & Mayer 2000b; Moreno 2004; Moreno & Mayer 2004; Moreno & Mayer 2007;
Clark & Mayer 2011) extend to multimedia learning a wide body of psychology research into the self-
referential effect. The self-referential effect theorises that people react differently to situations that
involve personal reference, and that information recall and understanding is improved when it is encoded
with respect to themselves rather than a different frame of reference. Building on this research, this
heuristic recommends that the copywriting of the screen text and audio narration in the e-learning
software and CLE activities should consider the following guidelines:

1. Use a more informal conversational style in narration and screen text to give learners a
sense that they are in a conversation with a partner; this motivates learners to work harder
to understand the material.
2. The conversational tone and style mean using words like “I”, “we” and “you”.
3. Words and phrasing should be less formal but without reducing the importance of the
learning material.
4. Do not overuse the conversational style to the point it becomes a distraction to learning;
it should remain polite, friendly and respectful, whilst not degenerating into slang and /or
colloquialisms.
5. In an audio narration, make sure the voice of the narration is human with a standard
accent instead of a computer-generated voice.

Adjusting text and narration to be more conversational, has been shown to increase engagement and
improve comprehension (deep learning) by enabling learners to interpret and interrelate information in
familiar conversation versus abstract instruction. Although results from multimedia research in this area
are mixed, conversational style has also been shown in some studies to improve retention.

4.7.7.7 Heuristic 21: Provide Restricted Navigational Control in the E-learning

Software

Learner control is implemented by navigational features that allow the learner to choose the path they
take through the e-learning software by selecting the topics and instructional elements they prefer, and

146
 CHAPTER FOUR

the pace at which they undertake learning. The tacit expectation is that more learner control will improve
learning and will satisfy the learner’s desire for more control. However, the challenge is that often learners
may not have enough self-awareness and meta-cognitive skills to accurately judge what they already
know, what they need to learn, and which instructional elements (with learning value) to select. In
consideration of the high-school age-group, this pedagogy recommends a restricted level of navigational
control that focuses more towards program control. However, within a structured e-learning
environment, the learner must still be given freedom in several key areas that are outlined in the
guidelines below.

Research by (Squires & Preece 1999; Mayer et al. 2003; Moreno & Mayer 2007; Clark & Mayer 2011; Clark
et al. 2011; Kalyuga et al. 2000; Mayer & Chandler 2001) give guidance in setting stable learner
interactions, within the e-learning environment and navigational control, for improved learning
performance. This is distilled into the following guidelines:

1. Restrict learners’ ability to control the order of lessons, topics and screens within the e-learning
software, but allow the previously covered material to be revisited.
2. Display all important educational material as default to avoid it being skipped by the learner.
3. Allow learners the flexibility to learn at their own pace.
4. Balance a stable e-learning environment with the learners’ need to feel in control and
responsible for their success. In consideration of heuristic 14.3 guideline-3, it is critical that
within a stable learning environment, learners must still be allowed as much personal control (as
possible) over their actual learning experience, such as: fostering an environment where it is
acceptable to make mistakes and learn from them; using learning activities and problem-based
methods that require the learner to exercise personal control and judgement to solve; providing
corrective feedback that helps the learner see the cause of their mistakes; and allowing them to
learn and retry.

It must be considered that this heuristic is subject to the expertise reversal effect in that experienced
learners who have prior knowledge and skills in the subject matter, or good metacognitive skills, or who
are taking an advanced lesson or course, can be given increased learner control to support effective
learning.

A further consideration in relation to program control is heuristic 10 and supporting the learners’ Zone of
Proximal Development via adaptive control. Adaptive Control entails the e-learning software dynamically
adjusting lesson difficulty and support, based on an evaluation of ongoing learner responses. This offers
personalised instruction to the learner and, to some extent, can be used to mitigate the aforementioned
expertise reversal effect.

The benefits from focusing the e-learning software towards program control rather than learner control
are that:

1. It reduces the extraneous cognitive load of the learner having to accurately understand their
learning needs and plan their path through the e-learning software,

147
 CHAPTER FOUR

2. With learner control, low prior knowledge learners may choose to view up to 50% fewer learning
screens.
3. Low prior knowledge learners typically perform worse under learner control; whilst high prior
knowledge learners typically perform equally well under both scenarios.

4.7.7.8 Heuristic 21.1: Provide Consistent Navigational Elements and Signposts for
Learning

Building on heuristic 21, sub-heuristic 21.1 further reinforces the stable and structured e-learning
environment, by providing consistent and meaningful navigational elements and signposts for learning.
The e-learning software should provide a clear and consistent Graphical User Interface (GUI) that places
minimal cognitive demand on the learner, and intuitively supports learning. One important part of this is
to provide clear navigational elements and visual cues (signposts) of the learning material that emphasises
recognition rather than recall.

A pre-existing body of research in this area has been supplemented in more recent times by research by
(Harp & Mayer 1998; Kalyuga 1999; Squires & Preece 1999; Mayer & Moreno 2003; Moreno 2007; Mayer
& Griffith 2008; Mayer & Johnson 2008; Clark & Mayer 2011); this has been distilled into the following e-
learning guidelines:

1. Provide a clear, consistent and meaningful navigational interface (navigational menus,

movement buttons, course maps, multimedia controls, etc.) that allow the learner to intuitively
progress through the e-learning software and control the educational material.
2. Provide clear, consistent and meaningful signposts for learning, such as a course map, learning
objectives, screen titles, embedded topic headers, labels, summaries, links (including summary
previews), etc.
3. Ensure pre-training and/or a guide is provided to learners that explains the navigational
interface and learning signposts.

The successful implementation of this heuristic enables a consistent and intuitive e-learning interface that
is innately meaningful and supports the learning experience without creating an additional and
unnecessary burden on cognitive load. Furthermore, meaningful signalling can be used to help the learner
to select and mentally organise the learning material, thereby improving retention and comprehension.
However, it should be noted that some studies have not demonstrated improved learning from signalling
approaches when used in conjunction with extraneous learning material (Harp & Mayer 1998; Moreno
2007; Mayer & Griffith 2008).

4.7.8 Gamification
Building on the review of gamification presented in section 2.4.10, this section further discusses
gamification research, and elaborates on how it is distilled into the e-learning heuristics outlined in Table
21.

148
 CHAPTER FOUR

ID Heuristic

15 Use gamification to increase motivation and learning performance.

15.1 Integrate gamification elements tightly within existing learning processes.

15.2 Build extrinsic gamification elements on top of existing learning processes.

Table 21: E-learning heuristics based on gamification

4.7.8.1 Heuristic 15: Use Gamification to Increase Motivation and Learning

Performance

In the context of e-learning, gamification is the use of game design elements within e-learning software
to increase the pleasure, fun, and motivation in the learning process, and to encourage positive learning
behaviour. Game design elements must be tightly integrated with existing intrinsically motivational
aspects of the software. Suggested game design elements include: points, leaderboards,
achievements/badges, levels, rewards, progression, challenge, storytelling, clear goals, rapid feedback,
explanatory feedback, freedom to fail, etc. The implementation of gamification can be broadly split into
two high-level guidelines that are discussed in detail as separate sub-heuristics:

• 15.1: Integrate gamification elements tightly with existing learning processes.

• 15.2: Build extrinsic gamification elements on top of existing learning processes.

The overriding benefit of gamification is to create a gameful and playful learning experience that
motivates the intended learning behaviour, and generally increases engagement and the joy of use.
Gamification attempts to build upon social psychological processes such as self-efficacy, group
identification, and social approval to provide learning rewards that motivate continued good work.

Various studies (Papastergiou 2009; Muntean 2011; Deterding 2012; de Freitas & de Freitas 2013; Decker
& Lawley 2013; Hamari et al. 2014; Iosup & Epema 2014; Morrison & DiSalvo 2014) cite that gamification
provides positive effects on motivation, and ultimately on learning performance. However, the findings
from previous studies have some variation in results and some shortcomings in research design that leave
some doubt. Current implementations of gamification often focus too heavily on the points and rewards
systems from games (guideline-2) and ignore intrinsically motivational aspects (guideline-1). This
approach can in fact be detrimental to the learning process by reducing the learner’s own internal
motivation (Nicholson 2012; Decker & Lawley 2013; Deterding et al. 2013; Stott & Neustaedter 2013;
Morrison & DiSalvo 2014).

It is also cautioned (de Freitas & de Freitas 2013; Decker & Lawley 2013; Iosup & Epema 2014; Morrison
& DiSalvo 2014; Pirker et al. 2014) that meaningful gamification is a difficult non-trivial task that increases
teacher effort, is time consuming, is likely to require computer-assisted management, and may increase
student focus on aesthetics.

Furthermore, it is postulated (Nicholson 2012) that once applied, the removal of extrinsic gamification
elements may have a detrimental effect due to the loss of earned badges and points, and the dependency

149
 CHAPTER FOUR

on external motivators. Finally, there is some caution that the positive results of gamification may not be
long term, and are in fact caused by a novelty effect.

4.7.8.2 Heuristic 15.1: Integrate Gamification Elements Tightly within Existing

Learning Processes

Many aspects of good game design correlate with existing pedagogical practices; therefore, they should
already exist within the instructional design of the e-learning software. These should not be reinvented
for gamification purposes; instead, gamification elements should simply integrate with the existing
pedagogical elements. These pedagogical elements include: storytelling, progressive challenge,
intrinsically motivational activities, rapid explanatory feedback, tutorials on how to use (play), and social
interaction.

Research by (Vygotsky 1978; Papastergiou 2009; Deterding et al. 2011; Kim 2011; Muntean 2011;
Deterding 2012; IVETIC & Petrović 2012; Nicholson 2012; Wu et al. 2012; de Freitas & de Freitas 2013;
Stott & Neustaedter 2013; Hamari et al. 2014; Iosup & Epema 2014; Morrison & DiSalvo 2014; Pirker et
al. 2014; TechnologyAdvice 2014) is synthesised into the following guidelines that are already present in
the pedagogical heuristics:

1. Gamified learning should offer progression and progressive challenge. As with all good games,
there needs to be progression and progressive challenge as the learner moves from novice to
expert, and eventually to master. The level of challenge needs to be enough to avoid boredom,
but not so much that its causes frustration or demotivation (Heuristics 8, 10, 14.3 guideline 2,
14.4 guideline 1).
2. Gamified learning should offer an engaging story. As with all good games, there needs to be a
surrounding story (context) and narrative that the player can engage in; likewise, people learn
better when the educational material is embedded in a story they can relate to. A unifying story
should be used throughout a curriculum to put learning into a realistic context in which actions
and tasks can be practised (Heuristics 1, 1.1, 4, 6.2, 14.1 guideline-2, 14.2 guideline-3).
3. Gamified learning should allow learners the freedom to fail and provide rapid feedback to
support learning. Games offer a fictional environment in which various game cues provide
players with continuous feedback on how they are doing, and when they make a mistake, they
are not excessively penalised, they simply lose a life or start at the last completed level, etc.
Feedback is already an important part of education, but specifically, the kind of continuous and
rapid feedback used in games has direct pedagogical links to formative assessment in which the
learner receives ongoing assessment and feedback that is separated from permanent marks. By
removing the fear of grades, learners are encouraged to explore and take risks in their learning
without fear of reproach (Heuristics 7, 7.1, 14.3 guideline 3).
4. Gamified learning should provide activities for learners to engage in. Gamers are not passive
consumers of a game, they are actively engaged in the game; each step and each decision make
a difference in the game. Likewise, constructivist principles promote active learning in which the

150
 CHAPTER FOUR

learner builds their knowledge through mindful activity (Heuristics 13 guidelines-6 and 4, 4.1,
6.2, 7, 7.1, 14.3 guideline-2, 14.4 guideline-1).
5. Game rules and tutorials need to be explained to players. All games have an associated set of
rules which need to be explained to the players; depending on the complexity of the game, these
are also supplemented by hints, tutorials, training missions and guides which help the players
advance in the game. In gaming parlance, this is called novice onboarding. Likewise in education,
students need to be advised what are the intended learning outcomes and, more specifically for
e-learning, to have some training or guidance on how to use the e-learning software (Heuristics
3 guideline-2, 11 guideline-4, 14.2 guideline-1, 14.3 guideline-1).
6. Gamified learning is a social activity. Gaming is typically a social activity that revolves around
competition or cooperation with other players. Competition often takes the form of challenging,
taunting, bragging, etc., and cooperation can take the form of greeting, sharing, helping and
gifting, etc. Kim (2011), subdivides gamers into 4 main groups: Killers, Achievers, Explorers and
Socializers, and then maps their social actions accordingly. Refer to Figure 22.

Figure 22: Social actions drive social engagement.

Source (Kim 2011, p.119)

A significant subset of the aforementioned social actions should be supported by the e-learning software
or the collaborative learning environment. However, considering the pedagogical focus on high-school
learning, the negative social actions in the top left quadrant of the graph should be heavily de-emphasised
or not supported. This pedagogy supports collaborative and social interaction through the following
existing heuristics 9, 11, 11.1, 12.

4.7.8.3 Heuristic 15.2: Build Extrinsic Gamification Elements on top of Existing

Learning Processes

Certain gamification elements are not part of established pedagogical approaches; they attempt to
leverage people’s love of competition and reward to encourage desired learning behaviour. They reflect
learner progress and attempt to motivate desired learning behaviour through extrinsic rewards, such as

151
 CHAPTER FOUR

Points, Leader-boards, Achievements/Badges and Levels. Since these gaming elements are not inherently
part of existing pedagogical practices, they need to be built in; however, they should only be used if there
is an existing foundation in heuristic 15.1.

Research by (Papastergiou 2009; Deterding et al. 2011; Kim 2011; Muntean 2011; IVETIC & Petrović 2012;
Haaranen et al. 2014; Hamari et al. 2014; Iosup & Epema 2014; Morrison & DiSalvo 2014;
TechnologyAdvice 2014) is synthesised into gamification guidelines that focus on extrinsic reward; the
instructional design of the e-learning software and CLE activities should consider a significant subset of
the following guidelines:

1. Encourage desired learning behaviour with instant reward. In gamification terms, this is called
feedback mechanics and is designed to give an immediate positive feedback (reward) that makes
the player feel good about completing something, and motivates them to continue with the
desired behaviour. This is achieved by the following mechanisms:
a. Points are a quantifiable metric that track and define progress; they are awarded for
undertaking some action or completing some activity. Typical examples include:
i. Experience points are earned directly via the player’s actions and are used to track
and reward progress or certain activities
ii. Redeemable points are similar to experience points, but can also be ‘cashed in’ to
purchase virtual or real goods or services.
iii. Skill points are earned by interacting with the game, and reflect mastery of an
activity.
iv. Social Points are earned by undertaking socially valuable contributions and actions,
and are used to track the social reputation or influence of a player.
b. Badges are usually awarded for actions a player has just completed. Badges offer a more
visual display of achievement than points, and can be shown on profile pages or user
accounts.
2. Communicate progress to the learner.
a. Learner progress can be reflected to the learner via a combination of gamification
mechanisms such as points, badges, a progress bar and levels.
b. Games are typically segmented into levels of increasing difficulty; likewise, levels can be built
into the e-learning software. Their completion marks progress, but also reflects achievement
and status.
3. Provide Social Recognition for desired learning behaviour. Social recognition can be given to
learners by integrating with social media platforms or, in this instance, giving social recognition
within the collaborative learning environment (learning community). This can be achieved by
showing points, levels and badges on user profiles, or by leaderboards that showcase the most
skilled and devoted learners.

152
 CHAPTER FOUR

4.7.9 Computational Thinking

Building on the review of computational thinking (CT) presented in section 2.4.11, this section further
discusses CT research and elaborates on how it is distilled into the e-learning heuristics outlined in Table
22.

ID Heuristic

6 Support problem-solving through computational thinking.

6.1 Build a foundation for computational thinking.

6.2 Exemplify computational thinking in problem-solving activities.

Table 22: E-learning heuristics based on computational thinking

4.7.9.1 Heuristic 6: Support Problem Solving Through Computational Thinking

Computational thinking is a way of thinking based on computer science concepts with which to
reformulate and solve problems. There is currently no authoritative definition of what these computer
science thought processes are, but one stable definition involves six concepts: a thought process,
abstraction, decomposition, algorithmic design, evaluation, and generalisation. Computational thinking is
both an important computer science topic that arguably deserves its own pedagogical heuristics, but also
a way of thinking that influences the heuristics for problem-solving. The successful implementation of
computational thinking in e-learning software and CLE activities should consider the following sub-
heuristics:

• Heuristic 6.1: Build a foundation for computational thinking.

• Heuristic 6.2: Exemplify computational thinking in problem-solving activities.

A wide body of research (Wing 2006b; Wing 2006a; Barr & Stephenson 2011; Wing 2011; Zhou 2011; The
Royal Society 2012; Grover & Pea 2013; Curzon et al. 2014; Van Dyne & Braun 2014) theorises that an
understanding of computational thinking provides a set of mental tools that can be used in higher-order
thinking and problem-solving that extends beyond the computer science domain. This is intuitively
understood by computer scientists, and supported by government initiatives. However, as previously
noted in section 2.4.11, the empirical evidence to link computational thinking to improved problem-
solving, or far transfer to other problem domains, is not conclusive.

4.7.9.2 Heuristic 6.1: Build a Foundation for Computational Thinking

Before students can employ computational thinking, they must first have a clear understanding of what
the elements of computational thinking are, be presented with real-world examples to broaden their
knowledge base, and become comfortable with the use of computational vocabulary to describe problems
and solutions. This acts as scaffolding or pre-training before students start using computational thinking
techniques.

153
 CHAPTER FOUR

Research by (Barr & Stephenson 2011; Lee et al. 2011; Wing 2011; Gouws et al. 2013; Grover & Pea 2013;
Selby & Woollard 2013; CAS Barefoot 2014; Curzon et al. 2014) is distilled to the following guidelines that
can be used to build a foundation for computational thinking:

1. Emphasise to students that the focus is not in creating tangible artefacts, but about fostering
specific thought processes.
2. Provide to students a clear definition of the following computational thinking concepts:
abstraction, decomposition, algorithmic design, evaluation, and generalisation.
3. Make use of computational vocabulary to describe problems and solutions to increase the
students’ comfort with concepts and terminology.
4. Bring computational thinking concepts to life with the use of real-world examples.
5. Make computational thinking more tangible to students by exemplifying it, using algorithms
either represented as flowcharts or pseudocode.

As a sub-heuristic of heuristic 6, heuristic 6.1 shares the same educational benefits and potential
challenges as its parent (refer to the previous section).

4.7.9.3 Heuristic 6.2: Exemplify Computational-Thinking in Problem-solving Activities

Once a stable foundation of computational thinking concepts and terminology is established, we must
exemplify the ethos, approaches and concepts used in computational thinking through worked examples
and problem-solving activities that learners can actively engage in.

Research by (Wing 2006b; Wing 2006a; Barr & Stephenson 2011; Lee et al. 2011; Zhou 2011; Wing 2011;
Rick et al. 2012; Gouws et al. 2013; Grover & Pea 2013; Selby & Woollard 2013; Curzon et al. 2014; Lye &
Koh 2014; Van Dyne & Braun 2014) is distilled to the following guidelines that can be used to exemplify
computational thinking in problem-solving activities:

1. Use problem-solving activities and worked examples as a vehicle to use and exemplify
computational thinking practice. (Heuristics 4, 4.1)
2. Make computational thinking processes explicit to learners whilst walking through worked
examples and problem-solving activities. (Heuristic 3)
3. Use a three-stage progression model: Use-Modify-Create. Use-Modify-Create is a pattern of
engagement that initially scaffolds the learning process and then progressively removes that
scaffolding (similar to heuristic 4.1). Initially, students are consumers of someone else’s creation,
then they modify the model, game or program etc. with increasing sophistication until eventually
they have learnt the skills to create entirely new models, games or programs.
4. Use a problem manipulation environment for students to engage in computational thinking. A
fundamental prerequisite for guideline-3 is the provision of a rich problem manipulation
environment in which the learner can be immersed in the Use-Modify-Create approach; the key
consideration is to provide learners with several mindful activities in which they can engage.
Several example activities are outlined in Vol 2 Appendix E section E.

154
 CHAPTER FOUR

5. Instil in learners the ethos behind computational thinking. As well as a set of concepts that
guide thinking processes, computational thinking also has an underlying ethos that is a guide to
how to approach a problem. The computational thinking ethos is discussed further in Vol 2
Appendix E section B.4.

As a sub-heuristic of heuristic 6, heuristic 6.2 shares the same educational benefits as its parent (refer to
the previous section). In addition, the focus in this heuristic on authentic learning, activities and problem-
solving means it offers a positive influence on intrinsic motivation.

4.8 Chapter Summary

This chapter focuses on the synthesis of the e-learning pedagogy; a pedagogy that supports high-school
computing with a comprehensive set of heuristics that can be used by teachers or other educators in
either the design or evaluation of e-learning software for computer science. The pedagogy considers the
following learning theories and other inputs: constructivism, social constructivism, collaborative learning,
connectivism, VARK learning styles, ARCS motivational design, cognitive load theory, gamification, and
computational thinking. These are then distilled into 21 heuristics and 19 sub-heuristics. The underlying
characteristics of the heuristics are that they are:

1. Not intended to instruct good pedagogy to teachers, but

2. Are offered as guidance to teachers and instructional designers in the design or selection of e-
learning software appropriate for high-school computer science education;
3. Are not intended to be prescriptive, reductionist or implemented as a mandatory checklist; but
4. Are intended to be used as a toolset in which the correct tools are selected by a teacher or
instructional designer based on the specific learning material, intended audience and intended
learning objectives; and
5. Although not explicitly considered in this pedagogy, usability is recognised to be critical to e-
learning software, and is a mandatory prerequisite.

Considering the substantial number of heuristics and sub-heuristics, and the level of detail associated with
each heuristic, the teacher audience is given further support by the provision of summary visual material
on how heuristics interrelate, and their potential benefits and dis-benefits. This is facilitated via an:

1. Educational Benefits Matrix,

2. Heuristics Interrelationship Map, and a
3. Heuristics Interrelationship Matrix.

This pedagogy is unique in that it sits at the intersection between pedagogy, e-learning, high-schools, and
computer science. It distils an enormous body of pedagogical knowledge into a format accessible to the
intended teacher audience, and gives comprehensive coverage to the learning process: to the two
processes of external interaction and internal acquisition; and the three dimensions of content, incentive
and environment.

155
 CHAPTER FIVE

5 SYNTHESIS OF THE E-LEARNING EVALUATION

PROTOCOL
5.1 Chapter Brief
This chapter focuses on the synthesis of the e-learning evaluation protocol; section 2.3.2 and 2.3.3
outlined the need to focus on pedagogy for e-learning software, and the limitations of existing research.
Those research gaps in turn guide the underlying characteristics of the proposed e-learning evaluation
protocol. This chapter outlines the origins of the evaluation protocol, and discusses its important
characteristics.

5.2 Versions of the E-learning Evaluation Protocol

As discussed in Chapter Three, the e-learning evaluation protocol was iteratively developed and refined
through the three phases of this research study; four versions of the e-learning evaluation protocol were
developed and released. These are outlined in Table 23.

Phase Date Pedagogy

Phase 1 11/05/2015 E-Learning Evaluation Rubric (Pedagogical Coverage.xlsx)

Phase 2-Cycle 2 11/08/2016 E-Learning Software Evaluation Protocol v0.1b

Phase 3 25/06/2017 E-Learning Software Evaluation Protocol (P3) v0.3

Final 09/10/2018 E-Learning Software Evaluation Protocol (Final) v0.4

Table 23: Four versions of the e-learning evaluation protocol

These evaluation protocol releases form the basis of the evaluation results presented in section 6.7.

5.3 Phase 1: The Origins of the E-learning Evaluation Protocol

As discussed in section 3.7.1, during the e-learning design activity (storyboard creation), it became
apparent that each e-learning software design / development has a specific pedagogical coverage that
needs to be assessed. In response to this need, an initial Phase 1 e-learning evaluation rubric was
constructed as follows:

1. Assess as a percentage the e-learning software coverage for each learning objective.
2. Considering that each learning objective is equally weighted, calculate the average learning
objective coverage. This becomes the weight factor for the pedagogical coverage.
3. For each educational screen, identify whether it:
a. Applies a heuristic = 1
b. Neither applies or violates a heuristic = 0
c. Violates a heuristic = -1
4. For each heuristic, calculate the percentage of screens that apply the heuristic.

156
 CHAPTER FIVE

5. Excluding heuristics that cannot be implemented in software, calculate the overall percentage of
pedagogical heuristic coverage.
6. The final pedagogical heuristic coverage is weighted based on learning objective coverage, so we
multiply learning objective coverage by pedagogical heuristic coverage.

The above rubric and its inherent limitations led to some important areas of consideration that are
reported in the results section 6.7.1, and were used to set the direction for the research into e-learning
evaluation protocols. Based on this relatively uninformed start, this study iteratively developed an e-
learning evaluation protocol designed with the characteristics discussed in the remainder of this chapter.

5.4 Characteristics of the E-learning Evaluation Protocol

Building on the academic foundations of section 2.3.3, the e-learning evaluation protocol presented in
this thesis, is characterised by:

1. Support for both formative and summative evaluations;

2. A focus on pedagogical quality rather than usability issues;
3. Support for the evaluation of learning objectives as well as pedagogical quality;
4. Consideration of the educational setting in which the e-learning software will be used;
5. The definition of a robust protocol for evaluation steps based on existing best practice in
evaluation (including guidelines on the number and experience of evaluators);
6. The definition of detailed design and evaluation criteria to give evaluators a clear basis for
evaluation;
7. The provision of quantifiable results that can be used for comparative evaluations; and
8. A focus on the reliability and validity of evaluation results.

5.4.1 Intended Usage

As is the norm, the pedagogical heuristics and criteria defined in this research are dual purpose: to guide
the pedagogical design of e-learning software, and to serve as the basis of a protocol to guide evaluation
activities (Ardito et al. 2006; Masip et al. 2011). When used for assessment purposes, the evaluation
protocol can be used both for formative and summative assessments. Considering the context of this
research and its focus on high-school computing, it is envisaged that school teachers will typically use it
in three scenarios (formative evaluation, summative evaluation, and comparative evaluation); these are
visually represented in Figure 23.

157
 CHAPTER FIVE

Figure 23: Intended usage of pedagogical heuristics and evaluation protocol

The formative evaluation of e-learning software occurs during the design and development stages to
improve the educational value of the software before it is released. During the design and development
process, after the storyboard design is stable, a different set of (impartial) teachers or instructional
designers can use the evaluation protocol to give a formative evaluation. The formative evaluation can
focus either on a detailed storyboard design or a pre-release version of the software. In this manner, the
evaluation group can iteratively feed into the design and development process to improve the educational
value of the software.

The evaluation of e-learning software prior to its use is employed so that teachers can make the right
software selection, and decide which e-learning resources best fit with the pedagogical strategies
intended for use in their classroom. Squires and Preece concur that: “Using educational software requires
teachers to decide which software to use with their students, for what purpose and in what situations, i.e.
to conduct predictive evaluations of the use of educational software. Informal predictive evaluations rely
on past personal experience to make value judgements about the quality and potential use of an
educational software application before using it in the classroom.” (1999, p.467)

The systematic comparative evaluation of an e-learning software with one or more different
implementations of equivalent software comes last. In this scenario, the evaluation protocol is used to
inform selection decisions for e-learning software that are intended to support long-term curriculum and
teaching needs (Squires & Preece 1996; Squires & Preece 1999; Çelik 2012). Reeves (1994) reinforces this
point by asserting that summative assessments are particularly important for the comparative evaluation
of different e-learning implementations that focus on the same learning objectives. He states that, “it is

158
 CHAPTER FIVE

imperative that criteria for evaluating various forms of CBE be developed that will result in more valid and
useful evaluations” (1994, p.222).

5.4.2 Focus on Overall Pedagogical Quality, not on Individual Usability

Issues
The history of heuristic evaluations is based on identifying usability problems. The focus is almost
exclusively on the negative process of finding and classifying the severity of usability issues. This is
reflected in the work of Nielsen and Molich (1990), Nielsen (1994), Dringus (1995), Squires and Preece
(1996), Parlangeli et al. (1999), Squires and Preece (1999), Reeves et al. (2002), Law and Hvannberg (2004),
Dringus & Cohen (2005), Ardito and Costabile (2006), Lanzilotti and Ardito (2006), Masip et al. (2011),
Inostroza and Rusu (2012) and Zaharias and Koutsabasis (2012).

This evaluation protocol has its basis in heuristic evaluations, but in contrast measures overall support, be
that negative (identifying pedagogical issues) or positive (identifying areas of pedagogical strength). This
has a more conceptual alignment with the work of Shee and Wang (2008), who studied e-learning
evaluations based on positive ratings and rank comparisons, and the work of Abderrahim et al. (2013),
whose evaluations were based on positive ratings.

Furthermore, building on the literature review in section 2.3.3, this e-learning evaluation protocol
proposes that a foundation of usability must already exist to resolve potential technical and graphical
issues. Meaning a separate usability evaluation is strongly recommended. This frees this evaluation
protocol to focus more specifically on educational content and instructional design (pedagogy).

5.4.3 Evaluate Content Quality as well as Pedagogical Quality

This evaluation protocol gives focus to the pedagogical quality of e-learning software. However, it is
important to note that even if a specific e-learning implementation is pedagogically excellent, it still may
not lead to the desired learning outcomes if the educational content is not appropriate. As such, in
alignment with the observations of Squires and Preece (1999) , this evaluation protocol is one of the few
that explicitly measures content quality and the e-learning software’s support of intended learning
objectives (curriculum). This evaluation protocol therefore focuses on educational value, which is the
combination of content quality and pedagogical quality. This relationship is reflected in Figure 24.

159
 CHAPTER FIVE

Figure 24: The composition of Educational Value

5.4.4 Not all Heuristics or Learning Objectives are Applicable or are of

Equal Importance
The evaluation of any e-learning software cannot be held in isolation; there must first be consideration of
the educational setting (learning context) in which the software will be used. An important characteristic
of this evaluation protocol is that it explicitly considers the intended learning context, such as the target
audience and their characteristics, and the typical context for using the e-learning software. Squires and
Preece emphasise that “the use of educational software is context dependent. It is not possible to evaluate
an educational software application predictively without reference to a perceived educational setting”
(1996, p.19).

If teachers are undertaking evaluations for themselves, they will typically already know this information.
If the evaluation is requested by an educational institution, then they may give this information to the
evaluation group, or potentially there may be some interchange between those who have requested the
evaluation and the evaluation group in order to finalise the educational setting.

The educational setting is particularly important since it guides evaluators in defining whether specific
learning objectives and heuristics are applicable within the given learning context. The adaptability of e-
learning heuristic evaluations is encouraged by Squires and Preece (1999), Dringus and Cohen (2005) and
Inostroza and Rusu (2012). Dringus and Cohen advise “the need for instructors to carefully select their own
heuristic items” (2005, p.3).

160
 CHAPTER FIVE

It follows that if a learning objective or heuristic is judged applicable, then the evaluator assesses its
relative importance in the given learning context. This increases the accuracy and validity of the evaluation
since it eliminates noise (non-applicable learning objectives and heuristics), and gives a weighted focus
on the remaining learning objectives and heuristics.

5.4.5 Number of Evaluators

One shortcoming of heuristic evaluations is the potential for subjectivity since the evaluation (although
guided) is ultimately based on the evaluator’s judgement, experience and disposition at the time of the
evaluation. As well as potential subjectivity, this can also lead to incomplete evaluation coverage, in which
an evaluator overlooks pedagogical issues or strengths. Based on six previous research studies, Nielsen
(1994) contends that a single evaluator typically identifies only 35% of usability problems in an interface.
Arguably, these concerns are further compounded by the ethereal nature of teaching and learning.

Based on the work of Nielsen and Molich (1990) and Nielsen (1994), the widely accepted approach to
mitigate this risk is to employ multiple evaluators who work individually, to avoid bias, and then aggregate
their findings. They offer evidence that there is considerable non-overlap between the sets of usability
problems found by different evaluators, and that significantly better performance is possible by
aggregating the evaluations from several evaluators. However, they conclude that beyond a certain point,
there are diminishing returns from adding evaluators. The final recommendation is to use between three
to five evaluators. This approach has become a de facto standard, and has been used in a large number
of studies (Dringus 1995; Parlangeli et al. 1999; Squires & Preece 1999; Reeves et al. 2002; Lanzilotti &
Ardito 2006; Alsumait & Al-Osaimi 2010; Inostroza & Rusu 2012; Zaharias & Koutsabasis 2012).

5.4.6 Experience of Evaluators

The evaluation protocol is intended for use by teachers or other education specialists in the ‘expert’
evaluation of e-learning software; however, the term expert does not necessarily mean an expert in the
evaluation process.

There is evidence to suggest that there can be a significant performance differential between evaluators,
even those with broadly the same background and qualifications. However, we cannot conclude that
supposed ‘good’ evaluators will consistently perform better. Nielsen and Molich (1990) identify that
‘good’ evaluators will not necessarily find all the easy problems found by ‘poor’ evaluators, and then go
on to identify hard problems. In fact, poor evaluators often find hard problems, and good evaluators can
overlook easy problems. Nielsen (1994) elaborates that although there is a better than random
consistency in the evaluators' ability to find usability problems across multiple evaluations, there remains
a significant unexplained performance variance from one evaluation to the next. It is postulated that it
might be possible to identify a group of good evaluators based on historical performance. However, it is
likely to be a slow process which would sacrifice performance in early evaluations. This analysis of
evaluator performance is the basis of why multiple evaluators are highly recommended for an e-learning
evaluation. In addition, instead of trying to find expert evaluators, it is more productive to focus on the

161
 CHAPTER FIVE

level and area of expertise of the evaluator. Citing his research from 1992, Nielsen (1994) discusses the
conceptual grouping of evaluators as novices, single experts and double experts.

In the context of this research, a novice has general knowledge and expertise in using computers, a single
expert has knowledge and expertise in teaching and pedagogy, and a double expert additionally has
knowledge and expertise in the domain of computer science. The evaluation protocol is prescribed to the
level of detail that a novice could undertake an e-learning evaluation. However, according to Nielsen
(1994), a single expert is 1.8 times as good as a novice, and a double expert is 2.7 times as good (identifying
60% of usability issues). These findings do not directly transfer to this research since our focus is not on
usability issues, but on educational value. However, it does give a good indicator of the potential value of
having double experts. It should also be noted that to conduct a full e-learning evaluation of educational
value (content quality and pedagogical quality) the evaluator must be a double expert since only someone
with computer science expertise can evaluate content quality. What exactly constitutes knowledge and
expertise in a given area is difficult to determine, but it is postulated to be: a relevant academic
qualification and a minimum of one year, or preferably two years, of experience. If the evaluator does not
have a relevant qualification, then their experience should be significantly longer. For improved clarity to
the intended teacher audience, in the final version of e-learning evaluation protocol the terminology used
to describe evaluator experience of novice, single expert and double expert was replaced with novice,
experienced and expert.

5.4.7 Protocol Steps for E-learning Evaluation

To ensure reliability and build towards the validity of the e-learning evaluation; individual evaluators and
different evaluation groups must approach the evaluation in a consistent manner. Hence, a set of detailed
guidance steps for e-learning evaluations is prescribed. This is documented in detail in Vol 3 Appendix N
section 4.

One area of importance for the e-learning evaluation protocol is its appropriateness for the intended
teacher audience; this is discussed in sections 6.7 and 7.9. However, it must be noted there is a tension
between specifying a rigorous set of guidance steps for e-learning evaluations and whether that
evaluation process will ultimately be accessible and usable for the teacher audience.

Innovation is pursued in the e-learning evaluation protocol through: the underlying heuristics, the
evaluation coverage of both pedagogy and subject content (Squires & Preece 1999), the focus on
quantifiable metrics, and a focus on heuristic support rather than only heuristic violations. However,
considering the dictionary definition of protocol as “a detailed plan of a scientific or medical experiment,
treatment, or procedure” (Merriam-Webster 2016), it is purposely decided that the protocol steps (i.e.
the evaluation procedure) should be as standard as possible. The protocol steps are based on the stability
of existing academic research, so as to avoid the possibility that they in some way corrupt the evaluation
results. The evaluators should follow a common, simple protocol for conducting their evaluation (Zaharias
& Koutsabasis 2012); this in turn supports research reliability and validity.

162
 CHAPTER FIVE

The protocol steps for carrying out the evaluation have their popular origins in the work of Nielsen (1994),
and have remained comparatively stable, as reflected in studies using very similar variations of the same
protocol. Such studies include, but are not limited to: Dringus (1995), Parlangeli et al. (1999), Reeves et
al. (2002), Ardito and Costabile (2006), Lanzilotti and Ardito (2006), Masip et al. (2011), Inostroza and Rusu
(2012) and Zaharias and Koutsabasis (2012). The protocol steps used in this study follow a similar vein
and, in particular, pay reference to Reeves et al. (2002). At a high-level, the e-learning evaluation protocol
follows a predefined process which is broken into three broad stages: Preparation, Individual Evaluations
and Building Consensus. Please refer to Figure 25 for a visual representation of the evaluation process.

Figure 25: High-level flow of e-learning evaluation

5.4.7.1 Preparation

The exact form and level of attention given to the preparatory stage are highly dependent on whether:

1. The evaluators are novices, single experts or double experts;

2. Whether the evaluators have previous knowledge of the heuristics;

163
 CHAPTER FIVE

3. Whether the evaluators have previously undertaken a similar evaluation, and therefore have
knowledge of the evaluation protocol; and
4. Whether the evaluators are documenting the educational setting or are being given the
educational setting by the evaluation requestor.

The preparatory phase is used to provide the evaluators with the structure and guidance needed to
undertake a reliable and valid evaluation. Typically, information (pre-reading) and/or training is provided
for: the evaluation process; the evaluation heuristics; domain-specific content and curriculum; and,
potentially, the educational context in which the e-learning software will be used. In some studies, the
evaluators are then given the opportunity to customise the evaluation heuristics according to this
information / training.

5.4.7.2 Individual Evaluations

Each evaluator conducts his or her evaluation individually and independently to form an unbiased opinion.
Typically, evaluators use a minimum of two iterations through the e-learning software; a free exploration
is used to get familiar with the flow of the software, its basic functions, and to form a general view of the
software’s design ethos.

Subsequent iterations are used to identify findings in relation to content and pedagogical quality. As
discussed previously, the proposed heuristics are not intended to be implemented as a mandatory
checklist, but as a toolset in which the correct tools (heuristics) are selected by a teacher or instructional
designer based on the educational setting. It is therefore critical that the evaluators consider the
educational setting in which the software will be used when deciding whether specific heuristics and
learning objectives are applicable, and if applicable, their importance and level of support within the
software. Each evaluator documents their findings in parallel in a structured evaluation report, which
includes descriptive comments and quantitative evaluation metrics.

5.4.7.3 Building Consensus

The final objective of the evaluation process is a single evaluation report that fairly represents the
aggregated findings of the set of evaluators. In support of this objective, a debrief session is run by an
unbiased facilitator whose role is to encourage effective discussion, align the individual evaluation
findings, and aggregate them into a consolidated response, which should be representative of the group.

The consolidated response is documented by the facilitator in a final report, which is then shared with the
evaluators, either for their feedback or confirmation that it is accurate.

5.4.8 The Need for Quantifiable Evaluation

It is typical during usability evaluations to assess the severity of each usability issue. According to Nielsen
(1994) this: ensures that appropriate resources are used to fix the most serious problems; helps to
ascertain whether additional usability efforts are necessary; and supports the decision-making process on
whether a software can be released to the intended audience. The assignment of such severity levels

164
 CHAPTER FIVE

requires the use of quantifiable metrics. In the context of this research, the assignment of quantifiable
metrics is necessary for the previous reasons, but it is all the more important since we are not evaluating
the severity of individual usability issues, but instead administering a ground-up evaluation of the
educational value of the entire e-learning software. Reeves (1994) cites Scriven (1993) in asserting that
there is an almost universal need to do comparative evaluations. He argues that criteria for evaluating
various forms of Computer-Based Education must be developed to ensure more valid and useful
evaluations. Reeves concludes that “there is certainly merit and utility in eventually grounding the ratings
in quantitative values” (1994, p.241). The underlying focus on quantifiable learning evaluation metrics
that enable selection and comparative evaluations is also proposed by the lesser known work of
Abderrahim et al. (2013).

This evaluation protocol proposes to ground the e-learning evaluation in quantitative values by evaluating
educational value in terms of content quality (the level of support for the given learning objectives) and
pedagogical quality (the level of support for the pedagogical heuristics relevant to the learning context).

As already discussed, the importance of each learning objective and each heuristic is specified by the
evaluator after consideration of the learning context (educational setting). The importance value is then
used as a weight factor in calculating the maximum support and the weighted support.

The support the e-learning software provides to each learning objective and each heuristic is also specified
by the evaluator. Both the importance and support values are then used to calculate the following:

• Weighted Support for Learning Objectives (refer to Equation 1)

• Maximum Support for Learning Objectives (refer to Equation 2)
• Percentage Support for Learning Objectives (refer to Equation 3)
• Weighted Support for Pedagogical Heuristics (refer to Equation 4)
• Maximum Support for Pedagogical Heuristics (refer to Equation 5)
• Percentage Support for Pedagogical Heuristics (refer to Equation 6)
• Percentage Educational Value (refer to Equation 7)

n = Number of Learning Objectives

m = Number of Pedagogical Heuristics

𝑛

∑ 𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒 𝑜𝑓 𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 Objective 𝑥 𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑓𝑜𝑟 𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 Objective

𝑖=1
Equation 1: Weighted support for learning objectives
𝑛

∑ 𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒 𝑜𝑓 𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 Objective 𝑥 𝑀𝑎𝑥𝑖𝑚𝑢𝑚 𝑆𝑢𝑝𝑝𝑜𝑟𝑡

𝑖=1
Equation 2: Maximum support for learning objectives
𝑊𝑒𝑖𝑔ℎ𝑡𝑒𝑑 𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑓𝑜𝑟 𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 Objectives
𝑥 100
𝑀𝑎𝑥𝑖𝑚𝑢𝑚 𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑓𝑜𝑟 𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 Objectives
Equation 3: Percentage support for learning objectives

165
 CHAPTER FIVE

∑ 𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒 𝑜𝑓 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙 𝐻𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐𝑠 𝑥 𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑓𝑜𝑟 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙 𝐻𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐𝑠

𝑖=1
Equation 4: Weighted support for pedagogical heuristics
𝑚

∑ 𝐼𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒 𝑜𝑓 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙 𝐻𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐𝑠 𝑥 𝑀𝑎𝑥𝑖𝑚𝑢𝑚 𝑆𝑢𝑝𝑝𝑜𝑟𝑡

𝑖=1
Equation 5: Maximum support for pedagogical heuristics
𝑊𝑒𝑖𝑔ℎ𝑡𝑒𝑑 𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑓𝑜𝑟 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙 𝐻𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐𝑠
𝑥 100
𝑀𝑎𝑥𝑖𝑚𝑢𝑚 𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑓𝑜𝑟 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙 𝐻𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐𝑠
Equation 6: Percentage support for pedagogical heuristics

%𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑓𝑜𝑟 𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 Objectives + %𝑆𝑢𝑝𝑝𝑜𝑟𝑡 𝑓𝑜𝑟 𝑃𝑒𝑑𝑎𝑔𝑜𝑔𝑖𝑐𝑎𝑙 𝐻𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐𝑠

2
Equation 7: Percentage educational value

5.4.8.1 Scale Format

As discussed in the previous section, a key objective of the e-learning evaluation protocol is to provide
quantitative metrics by which to assess the educational value of e-learning software. With this in mind,
the response scales and associated scale labels used in the evaluation protocol are of particular
importance in securing quantitative responses that are reliable and valid. Unfortunately, the definition of
appropriate response scales and scale labels are not simple tasks since the academic debate in these areas
span over a century, numerous academics, and many competing findings. Cox (1980) offers an indicative
literature review of research from 1915 onwards that spans over 40 authors. It touches upon areas such
as: scales of value methods vs unstructured responses, the number of response alternatives, reliability,
validity, response times, respondent preference, inclusion of an uncertain category, information on
theoretic measures, interpretability of descriptive statistics, and statistical efficacy of sample estimates.
However, the areas which are most vigorously debated are the optimal number of response alternatives,
whether that number is content-specific, and what impact, if any, it has on reliability and validity. Cox’s
contention that there is a lack of consensus in these areas is also shared by Matell and Jacoby (1971),
Preston and Colman (2000), Darbyshire and McDonald (2004), Weijters et al. (2010), and other
researchers.

The scope of this research is not to become embroiled in this debate but to focus on defining an equitable
set of response scales and scale labels to ensure reliability and validity. To achieve this objective, the
following has been considered.

5.4.8.1.1 There is No Optimal Number of Response Alternatives for all Circumstances

According to Cox (1980, p.407), “If the number of response alternatives were to be established
democratically, seven would probably be selected (Symonds 1924; Morrison 1972; Ramsey 1973); seven is
the modal number of response alternatives for the scales reviewed by Peter (1979).” Also, citing a number
of authors, Preston and Colman (2000) concur, that in current practice, most rating scales contain either
five or seven response alternatives. However, other researchers propose as few as two or three

166
 CHAPTER FIVE

alternatives, and others suggest over 20 can be used. Matell and Jacoby (1971), cite Champney and
Marshall (1939) in their assertion that, under favourable rating conditions, scales with five to seven points
may offer inexcusably inaccurate results, and that under the right conditions a scale of 18 to 24 points
may be appropriate.

Conklin (1923), Wittink and Bayer (1994), Myers (2000), and Preston and Colman (2000) all advocate
greater than seven response alternatives, with Preston and Colman concluding that “rating scales with 7,
9, or 10 response categories are generally to be preferred” (2000, p.10).

The key consideration is that the response scale should be refined enough for the respondent to be able
to transmit the necessary information, but not so refined that it encourages response error. Following this
rationale, Cox (1980), Garland (1991), Darbyshire and McDonald (2004), Preston and Colman (2000) , and
Weijters et al. (2010) concur that there is no optimal number of response alternatives, and that it is
dependent on the context and the purpose of the scale. However, Cox (1980) does offer some
recommendations: that two or three alternatives are inadequate since the alternatives cannot transmit
enough information; and that offering beyond nine alternatives will likely yield marginal returns in
improving the survey instrument. Finally, it is asserted that in a balanced scale an odd number of response
alternatives is preferable if the respondent can justifiably take a neutral stance. These guidelines are
echoed by Darbyshire and McDonald (2004, p.17) who conclude that “longer, balanced and unlabelled
scales offer the maximum flexibility and reliability in the majority of cases”.

5.4.8.1.2 Reliability and Validity

The impact of scale format on reliability is another area of contention amongst academics; Symonds
(1924) contends that deciding the number of response alternatives of a scale is primarily one of reliability,
and claims that optimal reliability is found in a 7-point scale. Other academics offer alternate suggestions
to maintain optimal reliability: Champney and Marshal (1939) suggest a 9-point scale; Wittink and Bayer
(1994) suggest a 10-point scale; whereas Jahoda et al. (1951) and Ferguson (1941) assert that scale
reliability increases, within limits, as the number of response alternatives increases. In stark contrast,
Bendig (1954) and Komorita (1963) advise that instrument reliability is independent of the number of
response alternatives. In addition, Cronbach (1950) cautions that to increase the reliability of an
instrument is meritless unless validity is increased proportionally.

In consideration of instrument validity, Preston and Colman (2000) report that there are comparatively
few studies in which validity is a criteria to evaluate the optimal numbers of response alternatives. Again,
the findings are mixed. Matell and Jacoby (1971) conclude that both reliability and validity are
independent of the number of response alternatives and that, in practice, longer scales can be collapsed
to as few as two or three response alternatives. In contrast, Loken et al. (1987) found that 11-point scales
offer superior validity compared to 3- or 4-point scales. Furthermore, Hancock and Klockars (1991)
surmise that the scores from 9-point scales correlate better than 5-point scales against objective
measures of the original stimuli.

167
 CHAPTER FIVE

In the face of such mixed findings, this research aligns with previous studies that assert that longer scales
support increased reliability and validity, and in particular draws upon the findings of Preston and Colman
(2000). Preston and Colman conclude that “The most reliable scores were those from scales with between
7 and 10 response categories, the most valid and discriminating were from those with six or more response
categories or – in the case of intertertile discriminating power – those with nine or more” (2000, p.10).
Furthermore, that respondents prefer scales with 10, 9, or 7 response alternatives.

5.4.8.1.3 Interval Response Scale and Scale Labels

A further consideration in defining a scale is what type of data that scale can collect, and therefore
whether the scale supports the analysis techniques required by the research. There is wide academic
debate on whether Likert scales are ordinal or interval in nature. However, it is widely accepted that
interval data offers a higher level of measurement precision, and supports a superior set of statistical
operations (Field & Hole 2003; Lazar et al. 2010; Pallant 2001; Field 2013). Darbyshire and McDonald
(2004) acknowledge this concern, and suggest that where a more sophisticated statistical analysis is
needed that a numeric scale is more suitable since a labelled scale is more prone to be interpreted as
ordinal data. Furthermore, labelled scales offer an increased challenge for larger scales in defining
meaningful labels for each point that can be perceived as equidistant (Myers 2000). This can in turn lead
to different interpretations of what the labels mean and therefore reduce reliability. However, it is also
argued that labelled scales reduce ambiguity regarding the meaning of each point on the scale and
therefore can increase the level of reliability in comparison to numeric scales (Bradlow & Fitzsimons 2001).
Considering all these points, Darbyshire and McDonald recommend anchoring “numeric scales with terms
which are specific enough to convey the meaning of the scale to the respondent” (2004, p.25).

5.4.8.1.4 Respondent Preference

According to Preston and Colman (2000), respondent preference on scale format has not been
investigated in depth in previous studies. In their study, Preston and Colman measure respondent
preference for scales with 2 to 11 response alternatives and 101 response alternatives. Respondent
preferences were measured according to the dimensions of ease of use, speed of use, and the
expressiveness of the scales. They offered statistically significant results indicating that short and very
long response scales have the lowest overall ratings, whilst the best overall preference rating was for the
ten-point scale, closely followed by the seven-point and nine-point scales. Building on these findings, and
in alignment with Jones (1968), a graphical scale is also considered to further improve the perceived
accuracy, reliability, interest and, most importantly, to reduce potential ambiguity.

5.4.8.1.5 Design of Response Scale

5.4.8.1.5.1 Support for Learning Objectives and Heuristics

Based on the guidelines discussed in section 5.4.8.1 and its sub-sections, and in conjunction with the
purpose of the scale to quantitatively measure an e-learning software’s support of each learning objective
and each heuristic, it is decided that:

168
 CHAPTER FIVE

1. A numeric scale is used since this most aligns with interval data.
2. A balance scale is used since an e-learning software may counteract or support each
heuristic.
3. An odd scale is used since an e-learning software may justifiably offer neutral support to a
heuristic.
4. Based on points 1 to 3, a nine-point scale is chosen, which is accordingly viewed to support
both reliability and validity.
5. A nine-point scale is chosen since it allows four response alternatives on each side, which
gives good granularity in measuring the level of support or counter-support an e-learning
software has for a heuristic.
6. To remove ambiguity, labels are used on the extreme and middle scale points.
7. To further remove ambiguity, two sliders are graphically represented under the numeric
response alternatives.

Considering the previous points, the resulting response scale is presented in Figure 26.

Figure 26: Response scale for adherence of e-learning software to this heuristic.

5.4.8.1.5.2 Importance of Learning Objectives and Heuristics

Similarly, the following decisions were made in the design of the response scale for measuring the
Importance of Learning Objectives and Heuristics.

1. A numeric scale is used since this most aligns with interval data.
2. A unidirectional scale is used since the importance of a learning objective or heuristic is a positive
attribute. However, it can be decided that a learning objective or heuristic is Not Applicable for
a specific learning context or e-learning software.
3. To remove ambiguity, labels are used as anchors on the extreme points.
4. A five-point scale is chosen since it allows a Not Applicable option as well as 4 support options,
which gives good granularity in measuring the level of support.

Considering the previous points, the resulting response scale is presented in Figure 27.

169
 CHAPTER FIVE

Figure 27: Response scale for level of importance.

When reviewing the response scale prior to the Phase 3 workshop, the researcher recognised that having
three anchor points on a five-point scale was counterintuitive and since the five-point scale is reasonably
short, decided to use the following labelled response scale to denote level of importance of both learning
objectives and heuristics (refer to Figure 28).

Figure 28: Adjusted response scale for level of importance.

5.4.9 Uniqueness of the E-learning Evaluation Protocol

As reflected in the work of Lanzilotti and Ardito (2006), Alsumait and Al-Osaimi (2010), Masip et al. (2011),
Inostroza and Rusu 2012 (2012) and Zaharias and Koutsabasis (2012), a standard approach to ascertain
the performance and validity of an evaluation protocol is to conduct a comparative evaluation of the same
software using an established evaluation protocol with the new protocol.

However, in consideration of the extensive literature review carried out in this study, and with reference
to characteristics of the e-learning evaluation protocol discussed in the previous sub-sections, there does
not seem to be an equivalent established e-learning evaluation protocol that is: specified at this level of
detail; that focuses on pedagogy instead of usability; that evaluates subject content to give an overall
focus on education value; and that measures adherence (positive and negative) instead of identifying
issues and severity levels.

This poses a challenge in establishing the performance and validity of the e-learning evaluation protocol
via a direct comparison with another evaluation protocol; however, ultimately, it is a positive indicator of
the uniqueness of this e-learning evaluation protocol.

170
 CHAPTER FIVE

5.5 Chapter Summary

This chapter focuses on the synthesis of the e-learning evaluation protocol; it outlines the rudimentary
beginnings of the original evaluation rubric in Phase 1. More importantly, it discusses in detail the
characteristics of the evaluation protocol that make the fourth version (refer to Vol 3 Appendix N) of the
protocol novel and arguably unique. The e-learning evaluation protocol is characterised by:

1. An intended audience of high-school teachers, or education experts typically in the field of

instructional design;
2. Support for formative, summative, and comparative evaluations;
3. A focus on pedagogical quality rather than usability issues;
4. Support for the holistic evaluation of educational value, which considers both learning objectives
as well as pedagogical quality;
5. Consideration of the educational setting in which the e-learning software will be used, and
explicit consideration that not all heuristics or learning objectives may be applicable in that
educational setting;
6. The definition of a robust protocol for evaluation steps, based on existing best practice in
evaluation (including guidelines on the number and experience of evaluators);
7. The definition of detailed design and evaluation criteria to give evaluators a clear basis for
evaluation.
8. The provision of quantifiable results that can be used for comparative evaluations; and
9. A focus on the reliability and validity of evaluation results.

The importance of points 8 and 9 in defining quantifiable results that are valid and reliable means the
scale format of the evaluation protocol is discussed at length, giving the rationale for the design of the
response scales for both adherence and importance of heuristics and learning objectives.

171
 CHAPTER SIX

6 IMPLEMENTATION AND RESULTS

6.1 Chapter Brief
This chapter presents the results from all phases and all research elements within the study. The results
are typically presented thematically, and where it adds value within a theme, according to the phases.
Specifically, this chapter presents:

1. The research instruments and materials, used as the basis to collect research results;
2. The Phase 1 pilot findings, and how they affected the remaining study and its direction;
3. The teacher and education expert feedback on the e-learning pedagogy;
4. Exploratory findings from student usage of the e-learning software prototype during Phase 1 and
2;
5. Phase 3 results;
6. The e-learning evaluation protocol results.

On consideration that this research is based on a mixed methods exploratory design, there is a qualitative
priority in the early exploratory phases. However, due to the significant volume of research findings and
results, the thick first-person literary description customary in qualitative research is avoided. With
respect to brevity, the results for points 2, 3 and 5 above are presented in an abridged form in this chapter.
However, they reference detailed appendices in which the exploratory findings are presented with a
richer text description and the voice of the participants is carried through in the use of direct quotes. The
results for Phase 3 (Point 4) are primarily quantitative, and are presented fully in this chapter. The results
from this chapter are then pulled together and interpreted in a cohesive discussion in Chapter Seven.

6.2 Research Instruments and Materials

Building on Chapter 3, this section outlines the instruments (measurement devices) and supporting
materials used in performing this research study.

6.2.1 Pedagogy Evaluation Feedback Document

A central component of this research was the involvement of teachers and education experts in the
evaluation and feedback of the e-learning pedagogy. In the Phase 1 Pilot, this was guided more informally
by several high-level questions (refer to section 3.4.4.2).

To ensure greater consistency between evaluations and evaluators, from Phase2-Cycle1 onwards, a
Pedagogy Evaluation Feedback document was introduced. This document incorporated similar questions
to those in Phase 1, via a combination of Likert and open questions. The Likert scale ranged across:
Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree and Strongly Agree. For reference, a
completed Pedagogy Evaluation Feedback document is included in Vol 3 Appendix G.

172
 CHAPTER SIX

Based on the findings from Phase2-Cycle1, this document was updated to include further questions and
was used in Phase2-Cycle2 and in Phase 3. For reference, the updated (and completed) Pedagogy
Evaluation Feedback document is included in Vol 3 Appendix F.

6.2.2 E-learning Software Prototype

Building on section 3.2.6, as a proof of concept, prototype e-learning software was developed according
to the pedagogical heuristics outlined in the e-learning pedagogy document. An incremental and iterative
development methodology was used to deliver updated versions of the e-learning software prototype for
each phase/cycle.

6.2.2.1 Phase 1

In Phase 1, a single e-learning software prototype was developed with a focus on algorithms (Algorithms
V04a). This was hosted on a SCORM cloud platform called SCORMCloud, which in turn linked to a
collaborative learning environment called TEAMIE.

6.2.2.2 Phase 2 - Cycle 1

The prototype e-learning software developed in Phase 1 was revised in accordance with the research
findings from the Phase 1 study and the updated pedagogical heuristics resulting from Phase 1. The new
version of the software (Algorithms V08a) was also expanded to include learning content on
computational thinking.

In Phase2–Cycle1, the TEAMIE collaborative learning environment was replaced by Google Apps for
Education. This decision was taken since Google Apps provides additional functionality, is free for Schools,
and is provided by a well-known brand with a large market presence and support infrastructure. These
points make the adoption of Google Apps for Education a much more realistic proposition for schools.

6.2.2.3 Phase 2-Cycle 2

The prototype e-learning software developed in Phase2-Cycle1 was then revised in accordance with the
research findings and the updated pedagogical heuristics resulting from Phase2-Cycle1. The software
prototype was now split into two levels (L1 White V03 and L2 Yellow V02). In Phase2–Cycle2, the
prototype also implemented gamification that required Learning Management System (LMS) support;
hence, the SCORMCloud platform was temporarily replaced by Adobe Captivate Prime.

6.2.2.4 Phase 3

The prototype e-learning software, started in Phase 1, refined in Phase 2, was now finalised in Phase 3 in
accordance with the research findings from the Phase2-Cycle2 study and the final set of pedagogical
heuristics. The software prototype was updated in accordance with 2016 GCSE Computer Science
specifications from OCR, AQA and Edexcel, and split into four levels. The splitting into levels was in
accordance with Heuristic 19: Optimise essential processing by segmenting learning material and

173
 CHAPTER SIX

providing pre-training, and considers the increasing difficulty of the material and the segmenting of
learning content into logically-related topics.

1. Level 1 White, Algorithms and Computational Thinking V0.5. This level was evaluated as having
Content Quality of 85%, a Pedagogical Quality of 74%, and overall Educational Value of 80%. For
a detailed breakdown of the evaluation results, please refer to Vol 3 Appendix S.
2. Level 2 Yellow, Algorithms and Computational Thinking V0.5. This level was evaluated as having
Content Quality of 78%, a Pedagogical Quality of 65%, and overall Educational Value of 71%. For
a detailed breakdown of the evaluation results, please refer to Vol 3 Appendix T.
3. Level 3 Orange, Flowcharts V0.3. This level was evaluated as having Content Quality of 88%, a
Pedagogical Quality of 70%, and overall Educational Value of 79%. For a detailed breakdown of
the evaluation results, please refer to Vol 3 Appendix U.
4. Level 4 Blue, Pseudo-Code V0.3. This level was evaluated as having Content Quality of 75%, a
Pedagogical Quality of 74%, and overall Educational Value of 74%. For a detailed breakdown of
the evaluation, results please refer to Vol 3 Appendix V.

The software provided learning material for: algorithms, computational thinking, flowcharts and
pseudocode. Level-1 and Level-2 provide foundational concepts on algorithms and computational
thinking, whereas Level-3 and Level-4 provide learning material on flowcharts and pseudocode, which are
of direct relevance to the pre-, post-test exam used in Phase 3.

In the final study, the prototype software was hosted on SCORMCloud, which in turn was linked to Google
Apps for Education.

6.2.3 VARK Learning Styles Instrument

Except for Phase 2-Cycle 230, prior to each observation study, an online VARK instrument was administered
to collect learning style preferences from the student participants. This instrument is based on the original
VARK research by Fleming and Mills (1992), but is tailored to younger people. With Neil Fleming’s
involvement, the various versions of the instrument are managed under VARK Learn Limited. The
instrument and its marking scheme are listed for reference in Vol 3 Appendix I.

6.2.3.1 Validity and Reliability

As a well-established learning styles measurement instrument, the VARK instrument is considered to have
high validity. Furthermore, the ready availability of the instrument, its relative simplicity, and the
availability of a clear marking scheme, ensure repeatability within this study and across other studies.
These underlying characteristics towards repeatability are also strong indicators of the instrument’s
reliability.

30
VARK responses were not collected in Phase2–Cycle2 since all participants had previously responded to the
instrument.

174
 CHAPTER SIX

6.2.4 Survey Instrument

After each observation study, an online survey was administered to collect student rating and opinion on
the e-learning software prototype; this was then used to extrapolate towards the underlying pedagogical
heuristics used in the design of the software. The survey instrument was developed by the researcher and
contained a combination of Likert, rank, and open questions. Prior to its pilot in Phase 1, the survey
instrument was reviewed by experienced researchers (members of the supervisory team), and as part of
the ethical approval process.

In preparation for each phase/cycle, the instrument was refined, based on previous findings and the
objectives of the phase/cycle. Prior to each use, it was again piloted internally, reviewed by experienced
researchers (members of the supervisory team) and approved by the ethics committee.

Also, included in the instrument was John Keller’s Instructional Materials Motivation Survey (IMMS)
(2006) which measures student motivation according to the ARCS motivation model (J. Keller 1987).

For reference, the final survey instrument used in Phase 3 is listed in Vol 3 Appendix H.

6.2.4.1 Validity and Reliability

To ensure validity and reliability, the decision was taken to reuse another well-established instrument.
Similar rationale as that discussed in section 6.2.3.1 (VARK Instrument Validity and Reliability), was
considered in using the IMMS. Additionally, Keller (2006) has documented his own reliability estimates
based on Cronbach’s alpha, which he conservatively judges to be satisfactory (refer to Table 24). Within
Phase 3, Cronbach’s alpha was also used to assess the internal reliability of the IMMS results.

Reliability Estimate
Scale
(Cronbach )

Attention .89

Relevance .81

Confidence .90

Satisfaction .92

Total scale .96

Table 24: Keller’s IMMS Reliability estimate.

The remainder of the survey instrument was new and developed by the researcher specifically for this
research. As such, several measures were taken to improve validity and reliability. These measures were
more significant in Phase 3 since in Phases 1 and 2 the results of the survey instrument were considered
within a qualitative context, and triangulated with the observation study, the LMS results and the focus
groups. This triangulation of results is one of the most important indicators of the validity of the Phase 3
survey instrument since it gives multiples measures that validate the survey instrument responses.

The validity and reliability of the survey instrument are further safeguarded in that it has undergone a
pilot study and was used (with some adjustments) in both Phase 2 (two cycles) and Phase 3. It should be

175
 CHAPTER SIX

noted that prior to Phase 3, a survey instrument phase analysis showed that 54% of questions were used
in all the previous three iterations, 69.5% of questions were used in two of the last three iterations, and
the questions that were changed involved primarily minor wording changes.

The Content validity of the survey instrument was safeguarded by the review of experienced researchers
in the subject area before each phase.

Alternate-form reliability was employed in the Phase 3 survey instrument by using differently worded
questions to measure the same attribute. In this case, the wording is adjusted, and the scale reversed
between question pair: 4 A) and 12 A).

Internal consistency reliability was employed in the Phase 3 survey instrument by assessing groups of
questions which were intended to measure different aspects of the same concept. In this case, the
following questions are measured for internal consistency:

• Questions 4A), 4C), 4D), 4E), 4H), 4I), 4J) and 4K) all measure usability of the prototype software.
• Questions 8A), 8B) and 8C all measure the difficulty of the prototype software.

6.2.5 Technical Specification Document

During each observation study, the students’ accessed the e-learning software and the collaborative
learning environment. Although both are cloud-based, there remain some technical pre-requisites and
checks that need to be carried out in the schools’ computer labs by the admin staff, and by the students
before home use.

In Phase 1, these pre-requisites and technical checks were communicated via email; however, from Phase
2 onwards, they were communicated in a Technical Specification document that was tailored towards the
objectives and technical requirements of each phase and cycle. For reference, the Technical Specification
document from Phase 3 is included in Vol 3 Appendix K.

6.2.6 Phase 3 - Pre-test / Post-Test

The pre-, post-test was developed to measure student learning performance in relation to usage of the e-
learning software prototype. The test is based on specimen exam papers from Paper 2: Application of
computational thinking, of the new Computer Science GCSEs. The exam questions were taken from the
following examination boards: EDEXCEL, OCR and AQA. Specimen papers were used since the new
Computer Science GCSEs started in September 2016 and were first examined in 2018.

The test was designed to recreate as closely as possible exams for the new specification. There were 31
marks in the test, instructions and information were provided in accordance with exam conditions, and
the test was taken in pencil/paper form under exam conditions. The test duration was reduced to 35
minutes in alignment with the reduced number of questions and marks. For reference, the pre-, post-test
and marking scheme are in Vol 3 Appendix J.

176
 CHAPTER SIX

6.2.6.1 Validity and Reliability

The test instrument is considered to have high validity and reliability since: it accurately reflects exam
questions on the subject matter; was confirmed as appropriate by the teachers involved in Phase 3; and
was confirmed by the teachers as being used under exam conditions. The marking of the tests was guided
by official marking schemes from the examination boards, and sample moderation was applied to ensure
the marking was fair.

It is important to note that none of the questions are memory based; in terms of Bloom’s taxonomy, they
all focus on higher-order thinking such as analysing and evaluating. This means that the students were
unlikely to memorise the answers in the 2-week period between the pre- and post-test. Additionally, the
students were not informed of the correct answers until after the post-test. Before the post-test, students
communicated whether they had accessed other learning material on the subject matter, and those that
had were removed from the analysis.

6.2.7 Phase 3 - Research Protocol Confirmation Document

Since the Phase 3 study was executed in two schools with three teachers, it was important to ensure each
teacher and group followed the same detailed procedures, thereby safeguarding reliability and validity. A
Phase 3 Research Protocol Confirmation document was distributed to the teachers ahead of the study,
which documented the steps and activities to be followed during the study. The document acted as a
checklist for the teachers to confirm they followed the procedures, and to document any potential events
or disruptions that could have affected study results. For reference, a completed Research Protocol
Confirmation document is included in Vol 3 Appendix L.

6.3 Phase 1 Pilot Findings

As a pilot study, Phase 1 was highly valuable in setting a stable foundation for the research, and identifying
the need for several changes in the research direction.

6.3.1 Feedback into Research Methods and Experiment Protocol

Phase 1 provided the following points of direction in relation to the research methods and experiment
protocol for later phases.

1. The overall framework of observation study, feedback survey and then focus group, although
effort-intensive, is considered successful.
2. As discussed in detail in section 3.4.2.1, the student participants in Phases 1 and 2 are not fully
representative of the GCSE Computer Science student population. However, their qualitative
student feedback on the e-learning software and the underlying pedagogy remain highly
valuable, and transferable to similar contexts.
3. The survey instrument from Phase 1 was carried forward into Phase 2 and 3; however, the survey
instrument was too lengthy, therefore was revised to reduce the number of questions.

177
 CHAPTER SIX

4. A greater focus was given to comprehensive and uniform participant instructions; this was
accomplished by a standard instruction handout.
5. A basic training on the e-learning software and the collaborative learning environment was given
before the observation studies in later phases; this was accomplished by a pre-recorded training
video.
6. The technical environment specification from Phase 1 was further enhanced before Phase 2. It
was updated to outline a detailed and specific target environment for use in the observation
study.
7. Before the observation studies in later phases, a more rigorous set of technical checks were
executed on the school’s computer lab to ensure alignment with the environment specification
necessary to run the e-learning software.

6.3.2 E-learning Pedagogy – Change of Focus

Initially, at the research approval stage, it was planned to deliver two separate pedagogical strategies:
one for computer science teaching, and one for computer science e-learning software development.
However, as the research progressed, this split became quite arbitrary due to the interdependent nature
of both pedagogies; hence, they were combined into a single document, which in Phase 1 outlined 36
pedagogical principles.

A more significant change of focus was that originally the research was focused purely on creating e-
learning software. Again, as the research matured, it became clear that e-learning software loses value as
an isolated entity, and that significant educational value can be gained in integrating the e-learning
software within a collaborative learning environment. Hence, the pedagogy was extended to consider
collaborative learning.

6.3.3 E-learning Software Development – Change of Focus

One of the original aims of this research was to create an e-learning pedagogy for the development of
GCSE Computing e-learning software. Additionally, the pedagogy was originally targeted towards
teachers. However, the Phase 1 development process called into question this objective and the intended
target audience. What is clear is that the development of quality e-learning software is not a trivial task.
Despite the use of one of a leading e-learning Integrated Development Environments (IDE), there were
some significant challenges in the development process. E-learning IDEs are very powerful, and offer a
wide variety of templates, widgets, integration and e-learning functionality within a relatively moderate
learning curve. However, this standard functionality may not always be pedagogically appropriate, offers
limited customisation, and can be quite brittle in where it works and under what publication mechanism.
This means that an initial prototype may be relatively quick to develop, but professional e-learning
software that fully adheres to the pedagogical heuristics requires significant effort, skill, and the
experience of an educational technology developer. A more appropriate focus for the pedagogy was

178
 CHAPTER SIX

therefore the definition of a set of heuristics that guide teachers (and instructional designers) to design
e-learning software, or evaluate existing software, for inclusion in their teaching.

6.4 Teacher and Educational Expert Feedback on the E-learning

Pedagogy
The teacher and education expert evaluation and input into the e-learning pedagogy spanned across all
three phases and included supplementary teacher feedback from the e-learning workshop in Phase 3. This
is visually represented in Figure 29.

Supplementary Teacher
Phase 1 Phase 2 - Phase 2 - Phase 3 Feedback
Cycle 1 Cycle 2
1 teacher & 9 teachers in 2 focus
2 education groups
1 teacher & 1 teacher & 1 teacher &
experts
5 education 4 education 3 education
experts experts experts

Figure 29: Phase summary of teacher and education expert feedback on e-learning pedagogy

6.4.1 Phase 1: Teacher and Expert Feedback on E-learning Pedagogy

The review of the Phase 1 e-learning pedagogy by five education experts and one teacher is based on the
pedagogy document GCSE Computer Science E-Learning Pedagogy v0 7. The evaluation feedback is
summarised in Table 25, which is elaborated with richer qualitative description in Vol 2 Appendix A section
A.1.

Phase 1

1 teacher and 5 education experts

[+] 1 teacher & 1 expert confirm heuristics appropriate for 15 & 16-year olds.

[+] 1 teacher & 1 expert confirm heuristics appropriate for computer science.

[+] No disagreement with any of the heuristics.

[+] No report of gaps in the heuristic coverage.

[+] Pedagogy commented as comprehensive.

[±] Teacher recommends heuristics for collaborative and social learning are more appropriate for home use.

[±] 1 teacher & 1 expert express some concern over CL1.1 Static illustrations can be better than animations.

[±] Recommendation to consider gamification, reward and fun.

[±] Recommendation to update and rephrase heuristic titles for clarity and uniformity.

179
 CHAPTER SIX

[±] Action to remove Educational Change and Technology Implementation sections in favour of a Design and
Evaluation section.

[±] Not all heuristics are based on a Learning Theory – Action to add Pedagogical Area field to further categorise
each heuristic.

[±] Recommendation from education experts 2, 3, and 5 to describe how some heuristics may contradict each
other.

[±] IDEA: Map the heuristics according to the different elements of the CS curriculum – Declined, pedagogy is
proposed as a toolkit for the teacher to select in context.

[-] 1 expert commented that pedagogy does not explicitly address realistic school context – Action to add Potential
Challenges section.

[-] Concern over the non-standard approach to referencing - Clarified that standard academic referencing will be
used in the thesis literature review.

[-] Feedback on grammar, spelling, phrasing, clarity of expression and moderating strong wording.

[-] 1 expert commented to avoid political rhetoric.

[-] Concern over appropriateness of some terminology to the teacher audience – Decision to include glossary.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 25: Phase 1 teacher and expert feedback on e-learning pedagogy

6.4.2 Phase2-Cycle1: Teacher and Expert Feedback on E-learning

Pedagogy
The review of the Phase2-Cycle1 e-learning pedagogy by four education experts and one teacher is based
on the pedagogy document GCSE Computer Science E-Learning Pedagogy v1.2. The evaluation feedback
is summarised in Table 26, which is elaborated with richer qualitative description in Vol 2 Appendix A
section A.2.

Phase2-Cycle1

1 teacher and 4 education experts

[+] Heuristics included for motivation, computational thinking and gamification.

[+] 4 of 5 participants confirm heuristics are appropriate for 15 to 18-year olds.

• Request to correlate heuristics according to their suitability to DfE Key Stages 4 and 5 learning objectives.

[+] 3 of 5 participants confirm heuristics are appropriate for computer science education.

• Commented that heuristics also focus on underlying skills important to computer science.

[±] 2 of 5 participants confirm feasibility of the heuristics to be implemented in a high school environment.

• Commented that potential challenges to implementation are documented and addressed in the pedagogy.

[+] 4 of 5 participants confirmed there is balanced pedagogical coverage in the heuristics.

180
 CHAPTER SIX

• No additional heuristics identified.

[+] Teacher feedback supports the recommended moderate connectivist approach.

[+] Teacher feedback supports pedagogy heuristics on engagement and motivation, and visual learning.

[±] Pedagogy is proposed as a toolkit of which specific tools (heuristics) are selected based on teacher or designer
judgement.

[±] Computational thinking queried as to whether and exactly how it should fit within the pedagogy.

[±] Recommendation to further categorise heuristics, and where they become important.

• Decision to include a benefits matrix that visually represents the intended benefits of each heuristic.

[+] Overall feedback is positive regarding the structure and readability of the pedagogy document. However, still
room for improvement:

[±] Additional comments to update and rephrase heuristic titles for clarity and uniformity;

[±] Recommendation on how to sequence heuristics differently, and how some heuristics should become
sub-heuristics;

[-] Document increased in size to 130 pages - document to be shortened;

[-] Reduce number of heuristics and evaluation criteria, too much information for teacher audience;

[-] Usability of the document for the intended teacher audience is questioned;

[-] Commented that phrasing and consistency of coverage in the pedagogy needs further review;

[-] Recommendation that related heuristics sections need further work to effectively reflect the heuristic
interrelationships; and

▪ Decision to include interrelationship matrix that shows heuristics support or conflict with
each other.

[-] Concern over non-standard approach to referencing – reaffirmed that standard academic referencing
will be used in the thesis literature review – Reference section removed from pedagogy.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 26: Phase2-Cycle1 teacher and expert feedback on e-learning pedagogy

6.4.3 Phase2-Cycle2: Teacher and Expert Feedback on E-learning

Pedagogy
The review of the Phase2-Cycle2 e-learning pedagogy from three education experts and one teacher is
based on the pedagogy document GCSE Computer Science E-Learning Pedagogy v1.5. The evaluation
feedback is summarised in Table 27, which is elaborated with richer qualitative description in Vol 2
Appendix A section A.3.

181
 CHAPTER SIX

Phase2-Cycle2

1 teacher and 3 education experts

[+] All participants confirm heuristics are appropriate for 15 to 18-year olds.

• Pedagogical heuristics provide support for DfE Key Stage 4 objective 1 and strong support for objective 2.
• Pedagogical heuristics provide strong support for DfE Key Stage 5 learning objectives 1 through 4.

[+] All participants confirm heuristics are appropriate for computer science education.

[±] All education experts confirm feasibility of the heuristics to be implemented in a high school environment.

• Teacher neither agrees nor disagrees. “Very little time provided for student-content which deepens learning”.

[+] All participants confirmed there is balanced pedagogical coverage in the heuristics.

• Teacher recommends more emphasis on learning from mistakes.

[+] Shortening and restructuring of pedagogy document is well received by participants.

[+] All participants confirmed that the educational benefits of each heuristic are clearly described.

[±] Educational Benefits Matrix confirmed as valuable, but context of use and limitations of the matrix
need to be communicated.

[+] All participants confirm the interrelationships between heuristics are clearly described.

[-] The size and intricacy of the interrelationship matrix is a challenge for legibility.

[+] Decision to supplement with an interrelationship map.

[+] Minor comments relating to spelling, grammar and phrasing.

[+] Teacher gives very positive comments on appropriateness of pedagogy.

[±] Expert comments that the pedagogy is a valuable and rich resource for teachers, but that would still require
non-trivial efforts for schools to fully utilise.

[-] Appendices document requires further work to be brought to the standard of the pedagogy.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 27: Phase2-Cycle2 teacher and expert feedback on e-learning pedagogy

6.4.4 Phase 3: Teacher and Expert Feedback on E-learning Pedagogy

The review of the Phase 3 e-learning pedagogy from two education experts and one teacher provided the
feedback outlined in this section; this feedback was based on the pedagogy document GCSE Computer
Science E-Learning Pedagogy v1.7b(CLEAR).

In Phase 3, the evaluation feedback on the e-learning pedagogy had plateaued and was nominal; hence it
is included in this section rather than referenced in an appendix.

Education Expert 2 provided feedback in the Pedagogy Evaluation Feedback document; however, this
mainly reinforced the same positive comments as Phase2-Cycle2. Education Expert 3 provided some small
feedback in a short-written response. Teacher 4, responded in the feedback document; however, the

182
 CHAPTER SIX

comments were universally positive. The remainder of this section will briefly outline the new comments
from the education experts and the new feedback from Teacher 4.

6.4.4.1 Education Expert Feedback

Education Expert 3 offered positive comments and confirmation on:

1. The e-learning pedagogy document overall,

2. The heuristic summary table at the beginning of the document,
3. The educational benefits matrix,
4. The heuristics interrelationship map, and
5. That the appendices document was now acceptable.

One constructive comment from Education Expert 3 was to query the red counteractive relationships in
the heuristics interrelationship map. After further discussion, it was agreed that these counteractive
relationships should be re-reviewed by the researcher, and a guidance note placed on the diagram.

6.4.4.2 Teacher Feedback

In addition to reviewing the e-learning pedagogy, Teacher 4 and his GCSE class also participated in the
Phase 3 experiment to use the e-learning prototype in a school context. He offered Strong Agreement to:

1. Appropriateness of heuristics for 15 to 18 years olds (Key Stages 4 & 5),

2. Appropriateness of heuristics for computer science education,
3. Feasibility of heuristics to be implemented in a high-school environment,
4. The balanced pedagogical coverage,
5. The education benefits of each heuristic are clearly described, and
6. The interrelationships between heuristics are clearly described

Teacher 4 commented that the pedagogy “reflects on a new generation of learners”, and in relation to its
appropriateness for computer science, stated that:

“Yes, it is as Computer Science can be boring. This study shows how it can be made more interactive and
interesting.” [Teacher 4]

Considering both the pedagogy and his school’s involvement in the Phase 3 study, Teacher 4 advised that
the “students have made massive progress and learnt a lot by following the e-learning software based on
pedagogy.”

As a final comment, Teacher 4 advised that the:

“Research is very clear and it has a strong connection with all the activities carried out during school
activities.”

183
 CHAPTER SIX

6.4.5 Supplementary Teacher Feedback on the E-learning Pedagogy from

the Phase 3 Workshop
The Phase 3 study for the e-learning evaluation protocol was encompassed within a teacher workshop
focused on e-learning and effective teaching practices for high-school computer science. A substantial
portion of this workshop was dedicated to educating the teachers on the e-learning heuristics. This
instructional session, and the teachers’ access to the evaluation protocol document, indicate a reasonable
grounding in the e-learning heuristics. However, at this point, these teachers did not have access to the
e-learning pedagogy document and the opportunity to evaluate it in detail. It is for this reason that the
strength of the following findings should not be overstated; the strength of these findings does not equate
with previous in-depth evaluations from teachers and education experts. Nevertheless, they do offer
supporting evidence of the appropriateness and validity of the e-learning heuristics.

This feedback is based on the pedagogy document GCSE Computer Science E-Learning Pedagogy
v1.7b(CLEAR), it is summarised in Table 28, which is elaborated with richer qualitative description in Vol
2 Appendix A section A.4.

Supplementary Teacher Feedback

9 teachers in 2 focus groups

[±] Strength of findings is weaker since feedback is based on pedagogy seminar, not detailed review of pedagogy
document.

[+] Both focus groups offered consensus of appropriateness of the heuristics for 15 to 18-years-olds.

[+] Both focus groups offered consensus of appropriateness of the heuristics for computer science.

[+] Overall, both groups offered consensus that the heuristics offered balanced pedagogical coverage.

[-] One teacher disagreed, commenting that there is no balance, and there are many gaps and weak
areas.

[±] Measured response regards the feasibility of the heuristics to be implemented in a high-school environment.

[+] One teacher offered positive support for approaches used in pedagogy to summarise heuristic
interrelationships and key benefits.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 28: Supplementary teacher feedback on the e-learning pedagogy from the Phase 3 workshop

As reflected in Figure 29, the e-learning pedagogy received detailed feedback and input from a total of
five education experts and four in service teachers, with further supplementary feedback from an
additional nine in service teachers in the Phase 3 workshop. This iterative and incremental feedback
spanned across three phases, including two cycles in Phase 2.

In summary, feedback in all phases was mainly positive and always constructive. Throughout there was
feedback on grammar, spelling, phrasing, and clarity of expression, this gradually tapered off to almost
nothing in Phase 3. In Phase 1 and Phase 2-Cycle 1 there was some advice to rephrase heuristic titles for

184
 CHAPTER SIX

clarity and uniformity, to re-sequence some heuristics and restructure some heuristics as sub-heuristics,
thereby facilitating a more natural flow and grouping. Throughout, the pedagogy was confirmed to be
appropriate for the target age group, and in Phase 2-Cycle 2 it was evaluated and confirmed to be
appropriate for DfE Key Stages 4 and 5 learning objectives (i.e. 15 to 18-year olds). Throughout, there was
confirmation from education experts and teachers that the heuristics are appropriate for computer
science education, this confirmation strengthened across the phases as the pedagogy matured.
Throughout the phases the pedagogy was commented as balanced and comprehensive; there was no
disagreement with any of the heuristic and no report of gaps in the heuristic coverage. Although in Phase
1 there was some minor comment on one sub-heuristic and a response from one education expert to
consider gamification, reward and fun. The latter comment gave further stimulus towards the inclusion
of heuristics on motivation, computational thinking and gamification in Phase 2-Cycle 1. In Phase 1, one
expert commented that the pedagogy did not explicitly address a realistic school context, this led to the
addition (in Phase 2-Cycle 1) of the potential challenges section for each heuristic. In follow on phases
there was increasing strength of confirmation of the feasibility of the heuristics being implemented in a
high school environment. However, there was always some residual concern that little time is allocated
in schools towards deep learning, also that change in a school context can be difficult thereby the
pedagogy would still require non-trivial efforts for schools to fully utilise.

By Phase 3, the evaluation of the e-learning pedagogy remained aligned with the positive feedback
received in Phase 2-Cycle 2, constructive comments had plateaued and were almost non-existent. The
detailed feedback from the Phase 3 teacher was overwhelmingly positive and was supplemented by
broadly positive feedback from the workshop teachers.

6.5 Phase 1 and 2 – Findings from Student Usage of E-learning

Software Prototype
As discussed in Chapter Three, Phase 1 and 2 were used as exploratory phases to iteratively develop the
e-learning pedagogy and test aspects of that pedagogy in depth within a school context. A small sample
of GCSE students was used within an observation study, which was later triangulated with survey and
focus group results.

The student participants have already been discussed in detail; however, for a brief recap of student
participants in Phase 1 and 2, please refer to Table 29.

Phase2 Phase2
Phase 1 Cycle 1 Cycle 2

Student Group 1 7 6 3

Student Group 2 3 2
Table 29: Phase 1 and 2 student participants

185
 CHAPTER SIX

The results discussed in this section are grouped thematically across the phases; however, not all themes
received research focus in each phase and cycle. Figure 30 gives a visual representation of how the themes
were addressed across the phases.

Phase 1 Phase2-Cycle1 Phase2-Cycle2

Usability and Navigation

Learning Styles and Multi-Modal Approach

Computational Thinking

Understanding and ZPD

Learning Object Ranking

Visual Elements

Video

Activities

Text

Audio

Collaboration

Authentic Learning vs Avoiding Overload

Virtual Manipulation Environment

Recreating Books

Connectivism and the Web

Motivation

Gamification

Figure 30: Thematic coverage of results in Phase 1 and 2

As discussed previously, in the interest of brevity, abridged findings are presented in this section, but are
further elaborated upon in Vol 2 Appendix B. These findings are summarised in Table 30 through Table
45.

Usability and Navigation

Phase 1 and both cycles of Phase 2

[+] Positive student feedback regarding the ease of use of the e-learning software.

[-] Negative student feedback regarding the reliability of the e-learning software.

[-] As the software became more sophisticated fewer of the IDE’s standard outputs and less standard
functionality could be used, leading to an increase in bugs and usability concerns.

[+] Positive student feedback on navigation and program control.

[+] Strong positive support for heuristics on restricted navigational control (Heuristics 21 and 21.1).

[+] Positive student feedback on restricted navigation tree.

[+] Positive student feedback on instructions and prompt messages.

[-] Observation that students were not reading screen instructions; hence missed within-screen navigation.

[-] Learning Object Icons, (signposting educational content) were being overlooked.

[±] Weakly positive feedback on mitigation steps to make instructions, within-screen navigation and
learning object icons more attention-grabbing.

[±] Weak response to pre-training video introducing the e-learning software and CLE.

[±] Pre-training video is broken into individual tutorial videos.

186
 CHAPTER SIX

[+] Students recommend tutorials not be placed at the start of the software, but spread as needed
across the software screens.

Heuristics Impacted: 17: Integrate words and graphics together, instead of words alone.

19: Optimise essential processing by segmenting learning material and providing pre-
training.

21: Provide restricted navigational control in the e-learning software.

21.1: Provide consistent navigational elements and signposts for learning.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 30: Phase 1 and 2 summary findings for usability and navigation

Learning Styles and Multi-Modal Learning

Phase 1 and Phase2-Cycle1

[+] VARK results show comparative balance between the modalities.

[+] Positive student feedback on the use of varying methods to represent the same educational concepts.

[+] Negative response to Phase 1 text bias, offers supporting evidence towards multi-modal approach.

Heuristics Impacted: 16: Use multi-modal learning approaches.

16.1: Support visual modal preference.

16.2: Support aural modal preference.

16.3: Support read-write modal preference.

16.4: Support kinaesthetic modal preference.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 31: Phase 1 and 2 summary findings for learning styles and multi-modal learning

Learning Object Ranking

Phase 1 and Phase2-Cycle1

[±] Learning object rankings between two phases are not consistent, but show some preference towards visual and
active learning object types.

[-] Negative ranking is consistent for audio and collaborative activities.

Heuristics Impacted: 4: Use problem-based learning (PBL) to facilitate learning.

4.1: Use worked examples to support problem-based learning.

187
 CHAPTER SIX

6.2: Exemplify computational thinking in problem-solving activities.

7: Distribute well-designed practice activities across the lesson to support learning.
9: Use social-interaction to increase learning and promote higher-order thinking.
11: Use collaborative learning activities.
16.1: Support visual modal preference.
16.2: Support aural modal preference.
17: Integrate words and graphics together, instead of words alone.
17.1: Apply contiguity by aligning words (audio or screen text) with corresponding
graphics.
17.2: Representing words as audio, on-screen text or both.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 32: Phase 1 and 2 summary findings for learning object ranking

Understanding and the Zone of Proximal Development

Phase 1 and both cycles of Phase 2

[+] Positive student feedback that educational material was represented in a clear and understandable way –
Supports many of the heuristics.

[+] Positive student feedback that there was no need to supplement the e-learning software with further textbook
reading.

[±] In Phase2-Cycle2, students reported that educational material was at the right level and assessment activities
were at the right level or a little difficult.

[-] In Phase2-Cycle2, students learning performance in quiz results was unfavourable:

[-] Poor question design;

[+] Identified areas for improvement of question design;

[±] Mixed student feedback on restricting number of attempts on a question;

[-] Students tired due to study being after exam period; and

[-] Students not committed to passing since acknowledged as a voluntary research study.

[±] Students avoided reviewing learning material even when guided to do so by the software.

[-] Students preferred to guess on last attempt.

Heuristics Impacted: 1: Use authentic educational material, examples and activities.

4.1: Use worked examples to support problem-based learning.

7: Distribute well-designed practice activities across the lesson to support learning.

7.1: Provide explanatory feedback to practice activities to promote learning.

8: Provide scaffolding to advance learning progress.
9: Use social-interaction to increase learning and promote higher-order thinking.

10: Engage learners in a challenge; target learning towards the zone of proximal
development (ZPD).

16: Use multi-modal learning approaches.

188
 CHAPTER SIX

16.1: Support visual modal preference.

16.2: Support aural modal preference.

16.3: Support read-write modal preference.

16.4: Support kinaesthetic modal preference.

17: Integrate words and graphics together, instead of words alone.

18: Avoid adding learning content that does not directly support your instructional goal.
19: Optimise essential processing by segmenting learning material and providing pre-
training.

20: Use a conversational style in screen text and audio narration.

21.1: Provide consistent navigational elements and signposts for learning.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 33: Phase 1 and 2 summary findings for understanding and the zone of proximal development

Visuals

Phase 1 and Phase2-Cycle1

[+] The importance to students of visual elements is reflected in Phase 1 and 2 feedback.

[+] Positive student feedback that the visual elements are meaningful.

[-] Comparatively weaker student response on whether the visual elements are aesthetically appealing.

[+] Positive student feedback that the visual elements are engaging.

[+] Positive student feedback that the visual elements supported their understanding of the subject matter.

[+] Findings are aligned with pedagogy which focuses on meaningful and engaging visual elements that support
learning, with lesser focus on the aesthetic appeal.

Heuristics Impacted: 16.1: Support visual modal preference.

17: Integrate words and graphics together, instead of words alone.

17.1: Apply contiguity by aligning words (audio or screen text) with corresponding
graphics.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 34: Phase 1 and 2 summary findings for visual learning material

189
 CHAPTER SIX

Text

Phase 1 and both cycles of Phase 2

[+] Throughout Phase 1 and 2, student feedback shows recognition that text material is important in supporting
their understanding of subject matter.

[-] Heavy text bias in Phase 1 prompted negative student response.

[+] After corrective action in Phase2-Cycle1, there were no further negative comments in relation to text content.

Heuristics Impacted: 16.3: Support read-write modal preference.

17: Integrate words and graphics together, instead of words alone.

17.1: Apply contiguity by aligning words (audio or screen text) with corresponding
graphics.

20: Use a conversational style in screen text and audio narration.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 35: Phase 1 and 2 summary findings for text learning material

Activities and Active Learning

Phase 1 and both cycles of Phase 2

Activities are considered to be Problem-Solving, Games, Simulations, Assessment and Quizzes.

[+] Strong positive support that activity-based (interactive) components support student understanding of subject
matter.

[+] Positive student feedback that the assessment activities in the e-learning software encouraged them to think,
and work through problems, instead of recounting from memory.

[±] Partially diverging feedback in Phase2-Cycle2, where students agreed that the e-learning software also
encouraged them to recall previous knowledge to answer the questions.

[+] Overall, students agreed the e-learning software prepared them for the assessment activities.

[+] Positive student feedback that activity-based (interactive) components are engaging.

[±] Choice of Virtual Manipulation Environment identified as critical for student engagement and usage.

Heuristics Impacted: 1: Use authentic educational material, examples and activities.

2: Prompt reflective practice to support learning.

4: Use problem-based learning (PBL) to facilitate learning.

4.1: Use worked examples to support problem-based learning.

5: Integrate learning into long-term memory by using authentic examples, and non-trivial
practice and problems.

6.2: Exemplify computational thinking in problem-solving activities.

190
 CHAPTER SIX

7: Distribute well-designed practice activities across the lesson to support learning.

7.1: Provide explanatory feedback to practice activities to promote learning.

14.3: Build “Confidence” to increase learner motivation.

14.4: Build “Satisfaction” to increase learner motivation.

15.1: Integrate gamification elements tightly within existing learning processes.

16.4: Support kinaesthetic modal preference.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 36: Phase 1 and 2 summary findings for activities and active learning

Video

Phase 1 and Phase2-Cycle1

[±] In spite of the drop in ranking between Phase 1 and Phase 2-Cycle1, videos remain an important part of the e-
learning software.

[±] Mixed feedback between survey results and focus group; focus group offered clarification and more
positive response on videos.

[+] Videos are a valuable tool in the support of visual modal preference (heuristic 16.1), and visualisation approaches
remain important within the pedagogy.

Heuristics Impacted: 16.1: Support visual modal preference.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 37: Phase 1 and 2 summary findings for video learning material

Computational Thinking

Phase2-Cycle1

[+] Positive student feedback that the e-learning software helped them to understand what computational thinking
is.

[±] Student response remains positive, but progressively weakens between recognising computational thinking
concepts and being able to use them.

[±] The e-learning prototype did not provide enough opportunities to exemplify and practice
computational thinking (weak support for heuristic 6.2).

Heuristics Impacted: 6: Support problem-solving through computational thinking.

191
 CHAPTER SIX

6.1: Build a foundation for computational thinking.

6.2: Exemplify computational thinking in problem-solving activities.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 38: Phase 2 summary findings for computational thinking

Collaborative Learning

Phase 1 and Phase2-Cycle1

[-] Collaborative learning was consistently ranked low.

[±] Student feedback on whether collaborative activities supported their understanding of the subject matter is
mixed.

[-] Collaborative learning activities were underutilised by the students in Phase 1 and Phase2-Cycle1.

[+] Students understand at a conceptual level the reasons and value of collaborative activities.

[+] Focus group responses are positive towards collaborative learning.

[±] Technology-enhanced collaborative learning not applicable in class and should be used for homework.

[±] There remains a learning curve for technology-enhanced collaborative learning since it is not the traditional way
of working in school.

[±] Research and experiment design was not conducive to evaluating collaborative learning since it was primarily in
class, for a short period of time, and voluntary.

[±] Research findings cannot be taken at immediate face value; they require a more complex and in-depth
analysis and interpretation.

[-] The e-learning software did not consider the technical affordances of mobile devices or the responsive design
necessary for mobile devices.

Heuristics Impacted: 9: Use social-interaction to increase learning and promote higher-order thinking.

11: Use collaborative learning activities.

11.1: Support collaborative and situated learning via mobile devices.

12: Develop and nurture networks to support learning.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 39: Phase 1 and 2 summary findings for collaborative learning

192
 CHAPTER SIX

Audio

Phase 1 and both cycles of Phase 2

[-] In both phases, audio material has consistently been ranked low, but requested to remain, based on the following
conditions:

[±] Audio quality must be good.

[±] Mute button must be available.

[+] E-learning software should give support to either enable / disable audio, and must remain pedagogically
effective even when audio is disabled.

Heuristics Impacted: 16.2: Support aural modal preference.

17: Integrate words and graphics together, instead of words alone.

17.1: Apply contiguity by aligning words (audio or screen text) with corresponding graphics.

17.2: Representing words as audio, on-screen text or both.

20: Use a conversational style in screen text and audio narration.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 40: Phase 1 and 2 summary findings for audio learning material

Authentic Learning vs Avoiding Overload

Phase 2 both cycles

[±] Tension between the following heuristics:

• 1. Use authentic educational material, examples and activities.

• 18. Avoid adding learning content that does not directly support your instructional goal.

[±] Student preference is towards Heuristic 1: Use authentic educational material, examples and activities.

[+] Support to metacognitive decision making by using a red “i” to indicate additional information that is non-
examinable.

Heuristics Impacted: 1: Use authentic educational material, examples and activities.

18: Avoid adding learning content that does not directly support your instructional goal.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 41: Phase 2 summary findings for authentic learning vs avoiding overload

193
 CHAPTER SIX

Connectivism and the Web

Phase 1 and Phase2-Cycle1

[±] Students understand that a critical appraisal of the information found on the web is necessary since the source
and quality of information may be unreliable.

[±] Students’ survey responses showed a weak preference towards the e-learning software compared to using the
web for their subject learning.

[+] Students perceive e-learning software to be comprehensive and prefer the structure of having one
place to learn from.

[±] Students wish to avoid the wasted time in searching the web and evaluating whether the information
they find is correct.

[+] E-Learning software links to other resources, mitigating the previous point.

[+] Findings support moderate connectivist approach recommended in the e-learning pedagogy.

Heuristics Impacted: 1.1: Ensure the currency of learning material.

11: Use collaborative learning activities.

12: Develop and nurture networks to support learning.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 42: Phase 1 and 2 summary findings for connectivism and the web

Motivation

Phase 1 and both cycles of Phase 2

[+] Positive student feedback that it is more interesting to use the e-learning software to learn computing than the
textbooks.

[+] Positive student feedback that the students could use the e-learning software for independent study.

[+] Positive student feedback that the e-learning software has increased the students’ overall enthusiasm and interest
in computing.

[±] Overall IMMS results are consistent across Phase 1 and 2, at approximately 3.67 from 5.

[±] Considering the iterative changes in the software, the consistency across phases may be viewed as
discouraging.

[±] Must be considered within the context that many aspects of the software remain unchanged between
versions, and the student participants also remain the same; hence, some of the novelty and motivation
value is lost.

Heuristics Impacted: 1: Use authentic educational material, examples and activities.

7: Distribute well-designed practice activities across the lesson to support learning.

13: Use constructivist approaches to increase intrinsic motivation in the learner.

14: Use the concepts of Attention, Relevance, Confidence and Satisfaction (ARCS) to attain
and sustain learner motivation.

194
 CHAPTER SIX

14.1: Use “Attention” grabbing strategies to increase learner motivation.

14.2: Explain the “Relevance” of the learning material to increase motivation.

14.3: Build “Confidence” to increase learner motivation.

14.4: Build “Satisfaction” to increase learner motivation.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 43: Phase 1 and 2 summary findings for motivation

Gamification

Phase2-Cycle2

[+] Intrinsic aspects of gamification already in e-learning software.

[+] Extrinsic gamification elements included in the software in Phase-2-Cycle2.

o Support for levels, points, badges and a leaderboard.

[±] Student survey response to extrinsic gamification elements is weakly positive.

[-] Focus group discussion shows students to be unenthusiastic about points and badges.

[-] Challenges:

[-] Implementation of extrinsic gamification elements directly tied to LMS support.

[-] Ad-hoc definition of triggers for achievements not supported by LMS.

[-] Limited set of award triggers supported by the LMS: specified at module level, coarse granularity and not
supporting the pedagogy.

[-] Inappropriate to challenge the student participants.

[-] Student feedback on leaderboard functionality is mixed.

[+] Student survey response to the leaderboard’s motivational value is positive in the scenario that a student
is underperforming.

[±] Student survey response to the leaderboard’s motivational value is mixed in the scenario that a student
is performing well.

[±] Focus group highlighted concerns that a leaderboard may further demotivate students who are not
performing well, and may signal high performers to relax if they are well ahead of their classmates.

Heuristics Impacted: 15: Use gamification to increase motivation and learning performance.

15.1: Integrate gamification elements tightly within existing learning processes.

15.2: Build extrinsic gamification elements on top of existing learning processes.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 44: Phase 2 summary findings for gamification

195
 CHAPTER SIX

Recreating Books

Phase 2 both cycles

[±] Students requested a more explicit structure in the e-Learning software.

[±] Mimic a traditional chapter format, with the chapter learning objectives, learning material, review
questions and finally a learning summary.

[±] Software updated by breaking the learning material into levels, with learning objectives, learning
material and review questions.

[+] Positive student feedback that levels helped in estimating progress.

[+] Positive feedback that levels helped motivate students to progress to the next level.

Heuristics Impacted: 19: Optimise essential processing by segmenting learning material and providing pre-
training.

21: Provide restricted navigational control in the e-learning software.

21.1: Provide consistent navigational elements and signposts for learning.

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 45: Phase 2 summary findings for recreating books

It should be noted that the preliminary findings from Phase 1 were presented at the EAPRIL 31 2014
Conference and published in the associated conference proceedings (Yiatrou et al. 2014). The preliminary
findings from Phase 2-Cycle1 were presented at the FedCSIS 32 2016 Conference and published in the
associated conference proceedings (Yiatrou et al. 2016).

31
European Association for Practitioner Research on Improving Learning
32
Federated Conference on Computer Science and Information Systems

196
 CHAPTER SIX

6.6 Phase 3 Results

As a recap, Phase 3 used a convergent parallel mixed method design with a quantitative priority, in which
students:

1. Undertook a pre-test;
2. Responded on a standard VARK measurement instrument;
3. Undertook learning (under controlled conditions) on computational thinking and algorithms,
using the four levels of e-learning software and the collaborative learning environment;
4. Undertook a post-test; and
5. Responded on a survey instrument.

With reference to the above, this section reports on the:

1. Student responses to the survey instrument, such as their perception of the e-learning software
and their level of motivation,
2. Student VARK profiles,
3. Student learning performance related to pre- post-test results.

As a guiding reference, the key results from Phase 3 are summarised in Table 46 and Table 47.

Student Perception of E-learning Software

The average student response to the survey instruments indicated:

[+] Broadly positive feedback in terms of the usability of the e-learning software.

[-] Usability concerns in relation to bugs in the prototype software and the speed/responsiveness of the
cloud-based delivery.

[+] Broadly positive feedback on the impact that the different educational components had on students’ perceived
understanding.

[-] Response on the audio content was non-committal on whether audio helped students’ understanding
of the subject matter.

[+] Agreement that use of different methods to represent the same learning content helped understanding (i.e.
multi-modal learning).

[+] Learning material, practice activities and quiz questions were at the right difficulty level.

[±] Neither agreement nor disagreement on whether the students needed to supplement the e-learning software
with further textbook reading, or whether they felt the need to ask their teacher for support in understanding the
learning material.

[±] Neither agreement nor disagreement on whether, after completing the 4-levels of the e-learning software,
they felt confident of passing a test on the subject matter.

[±] Neither agreement nor disagreement on whether the students’ felt it more interesting to use the e-learning
software to learn computing than the textbooks.

[+] Agreement that the students could use the e-learning software for independent study to learn computing.

[+] Agreement that the e-learning software had increased their overall enthusiasm and interest in computing.

[+] A weakly positive feedback of moderately true to the IMMS motivational sub-categories of attention,
relevance, confidence and satisfaction.

197
 CHAPTER SIX

[+] Overall IMMS results are weakly positive, with a moderately true response of 3.15 from 5.

[±] Except for attention vs relevance, the summation of student responses on the IMMS sub-categories for the e-
learning prototype are broadly balanced and aligned.

[+] Except for visual vs. kinaesthetic, the summation of student responses on the VARK sub-categories are
comparatively balanced and aligned.
Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 46: Phase 3 - Summary of student perception of e-learning software

Student Learning Performance

The average student learning performance, and engagement with the e-learning software and CLE indicated that:

[±] The median average student engagement with the e-learning software, excluding CLE, was 196 minutes.

[±] The median average student engagement with levels 3 and 4 of the e-learning software, excluding CLE, was 64
minutes.

[-] Student engagement with the CLE and the assignment activities was inadequate.

[+] The results between both schools in relation to Pre-Test, Post-Test and %Change are not statistically different.

[+] Parametric and non-parametric statistical analysis indicates an increase of over 19% points between the pre-
and post-test.

[+] Parametric tests indicate that we cannot reject the null hypothesis that the %Pre-Test mean is equal to the
hypothesised GCSE Grade F target value.

[+] Parametric tests indicate that we cannot reject the null hypothesis that the %Post-Test mean is equal to the
hypothesised GCSE Grade C target value.

Non-parametric Correlation Analysis was broken into two tiers:

1. Tier 1 correlations include:

a. A medium positive correlation between %Post-Test result and Level-2 time
b. A medium negative correlation between KS4 Prediction and Level-1 Time
c. A medium positive correlation between KS4 Prediction and Level-4 Time
d. A medium positive correlation between KS4 Prediction and Level-3 and -4 Combined Time
e. A medium positive correlation between IMMS Overall and Level-2 Time
f. A medium negative correlation between %Change and the students’ relative feelings of being
overloaded by multi-modal learning
2. Tier 2 correlations include:
a. A small positive correlation between %Post-Test result and All Levels Combined Time
b. A small negative correlation between %Post-Test result and VARK Aural result
c. A small positive correlation between %Post-Test result and VARK Kinaesthetic result
d. A small negative correlation between %Change and Level-3 time
e. A medium positive correlation between IMMS Relevance and Level-2 Time
f. A medium positive correlation between IMMS Relevance and Level-3 and -4 Combined Time
g. A medium positive correlation between IMMS Satisfaction and Level-3 Time
h. A medium positive correlation between IMMS Satisfaction and Level-3 and -4 Combined Time
i. A medium positive correlation between IMMS Overall and Level-3 and -4 Combined Time
j. A medium positive correlation between %Post-Test result and the students’ perception of the e-
learning software being easy to use

198
 CHAPTER SIX

k. A medium positive correlation between %Post-Test result and the students’ perception of the e-
learning software being reliable (i.e. does not contain bugs or errors)
l. A medium negative correlation between %Post-Test result and the students’ relative feelings of
being overloaded by multi-modal learning

Legend:
[+] Positive Feedback
[±] Mixed Feedback or Recommendation
[-] Adverse Feedback

Table 47: Phase 3 - Summary of student learning performance

6.6.1 Phase 3 Participants – Merging of Groups

The Phase 3 study was comprised of two UK public schools offering the Computer Science GCSE (2016
Specification). The Phase 3 student sample (66 students) is discussed in section 3.6.2.2, and is deemed
representative of the wider GCSE student population.

To strengthen the statistical analysis in Phase 3, the response data from both schools were merged into a
single group. This decision was taken based on the following:

1. Both teachers responded on the Phase 3 Research Protocol Confirmation document, showing
that both sets of students undertook the study in the same controlled conditions.
2. Independent sample T-Tests of Pre-Test, Post-Test and %Change results showed that the null
hypothesis could not be rejected.

School1 Pre-Test results (M=5.11, SD = 3.22) were similar to the Pre-Test results from School2 (M=4.71
SD=3.96) conditions; t(46)=0.38, p=.70.

School1 Post-Test results (M=10.96, SD = 4.67) were similar to the Post-Test results from School2
(M=10.81 SD=3.83) conditions; t(46)=-0.12, p=.90.

School1 %Change results (M=18.488, SD = 12.77) were similar to the %Change results from School2
(M=19.66 SD=11.71) conditions; t(46)=-0.22, p=.83.

The above t-test results do not prove the two groups are equivalent, but gives an indication that the
responses from both groups are not statistically different, and can be combined without negative impact
to statistical analysis.

6.6.1.1.1 Reporting of Outliers

In alignment with section 3.6.5.3, which discusses the treatment of outliers, this section will primarily
report descriptive statistics including outliers, but will also report as a reference, descriptive statistics
without outliers. This approach will indicate the potential impact of the outlier(s), and give input on
whether subsequent statistical tests should be carried out with or without outliers.

199
 CHAPTER SIX

6.6.2 Student Perception of the E-learning Software

6.6.2.1 Participants

From the 66 student participants in Phase 3, 47 participants responded to the survey instrument: 28
participants from school1 and 19 participants from school2. There was minimal non-response to the
survey questions, except for the IMMS question set and the 2 ranking questions; hence, those were
analysed separately.

6.6.2.2 Survey Response

6.6.2.2.1 Response on E-learning Usability

Based on the following standard Likert scale: 1 Strongly Disagree, 2 Disagree, 3 Neither Agree nor
Disagree, 4 Agree and 5 Strongly Agree, the student participants gave broadly positive feedback on the
usability of the prototype e-learning software.

With reference to Table 48, the median student response showed agreement (4) with the following
statements:

1. All things considered, the e-learning software is easy to use;

2. The e-learning software is reliable (i.e. does not contain bugs or errors);
3. The learning content in the e-learning software was represented in a clear and understandable
way;
4. The various instructions and prompt messages are understandable;
5. The graphical parts (symbols, logos, diagrams, pictures and illustrations etc.) of the e-learning
software are meaningful;
6. The e-learning software gave accurate feedback in response to my interactions;
7. The navigation and program controls of the e-learning software are logically arranged and
consistent and
8. It is easy to use the navigation and program controls of the e-learning software.

The frequency distributions for the above responses are represented in the histograms in Figure 31 to
Figure 38.

200
 CHAPTER SIX

N Min Max Median Mean Std. Deviation Frequency Distribution

4 A) All things considered, the e-

47 1 5 4.00 3.66 0.89 Refer to Figure 31
learning software is easy to use.

4A) Outliers removed 46 2 5 4.00 3.72 0.80

4 J) The e-learning software is

reliable (i.e. does not contain 46 1 5 4.00 3.35 1.18 Refer to Figure 32
bugs or errors).

6 A) The learning content in the

e-learning software was
44 1 5 4.00 3.57 1.02 Refer to Figure 33
represented in a clear and
understandable way.

6 A) Outliers Removed 42 2 5 4.00 3.69 0.87

4 K) The various instructions and

prompt messages are 47 1 5 4.00 3.66 0.92 Refer to Figure 34
understandable.

4 K) Outliers Removed 45 2 5 4.00 3.78 0.74

4 C) The graphical parts

(symbols, logos, diagrams,
pictures and illustrations etc.) of 47 1 5 4.00 3.53 1.04 Refer to Figure 35
the e-learning software are
meaningful.

4 C) Outliers Removed 45 2 5 4.00 3.64 0.91

4 I) The e-learning software gave

accurate feedback in response to 44 1 5 4.00 3.48 1.05 Refer to Figure 36
my interactions.

4 I) Outliers Removed 41 2 5 4.00 3.66 0.83

4 D) The navigation and program

controls of the e-learning
47 1 5 4.00 3.57 0.88 Refer to Figure 37
software are logically arranged
and consistent.

4 D) Outliers Removed 46 2 5 4.00 3.63 0.80

4 E) It is easy to use the

navigation and program controls 47 1 5 4.00 3.60 1.01 Refer to Figure 38
of the e-learning software.

4 E) Outliers Removed 45 2 5 4.00 3.71 0.87

Table 48: Phase 3 - Descriptive statistics for usability results

201
 CHAPTER SIX

As detailed in Table 48, Figure 31 reflects a distribution (N=47) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 4, M=3.66, SD=0.89) agrees (4) with the
statement “All things considered, the E-Learning software is easy to use.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 31: Phase 3 - All things considered, the e-learning software is easy to use.

As detailed in Table 48, Figure 32 reflects a distribution (N=46) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 4, M=3.35, SD=1.18) agrees (4) with the
statement “The E-Learning software is reliable (i.e. does not contain bugs or errors).”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 32: Phase 3 - The e-learning software is reliable (i.e. does not contain bugs or errors).

202
 CHAPTER SIX

As detailed in Table 48, Figure 33 reflects a distribution (N=44) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 4, M=3.57, SD=1.02) agrees (4) with the
statement “The learning content in the E-Learning software was represented in a clear and understandable
way.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 33: Phase 3- The learning content in the e-learning software was represented in a clear and
understandable way.

As detailed in Table 48, Figure 34 reflects a distribution (N=47) that is not normal, is negatively skewed
and is leptokurtic, where the average student response (Mdn = 4, M=3.66, SD=0.92) agrees (4) with the
statement “The various instructions and prompt messages are understandable.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 34: Phase 3- The various instructions and prompt messages are understandable.
As detailed in Table 48, Figure 35 reflects a distribution (N=47) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 4, M=3.53, SD=1.04) agrees (4) with the

203
 CHAPTER SIX

statement “The graphical parts (symbols, logos, diagrams, pictures and illustrations etc.) of the E-Learning
software are meaningful.”

Figure 35: Phase 3 - The graphical parts (symbols, logos, diagrams, pictures and illustrations etc.) of the e-learning
software are meaningful.

As detailed in Table 48, Figure 36 reflects a distribution (N=44) that is not normal, is negatively skewed
and is kurtosis static, where the average student response (Mdn = 4, M=3.48, SD=1.05) agrees (4) with
the statement “The E-Learning software gave accurate feedback in response to my interactions.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 36: Phase 3 - The e-learning software gave accurate feedback in response to my interactions.
As detailed in Table 48, Figure 37 reflects a distribution (N=47) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 4, M=3.57, SD=0.88) agrees (4) with the

204
 CHAPTER SIX

statement “The navigation and program controls of the E-Learning software are logically arranged and
consistent.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 37: Phase 3 - The navigation and program controls of the e-learning software are logically arranged and
consistent.

As detailed in Table 48, Figure 38 reflects a distribution (N=47) that is not normal, is negatively skewed
and is kurtosis static, where the average student response (Mdn = 4, M=3.6, SD=1.01) agrees (4) with the
statement “It is easy to use the navigation and program controls of the E-Learning software.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 38: Phase 3- It is easy to use the navigation and program controls of the e-learning software.

However, the student participants did raise some usability concerns; with reference to Table 49, the
median student response showed neither agreement nor disagreement (3) with the following
statements:

205
 CHAPTER SIX

1. Sometimes I felt that I didn’t quite understand what the e-learning software was doing.
2. I found errors (bugs) in the e-learning software that were difficult to recover from.
3. The e-learning software felt speedy and responsive to my interactions.

The frequency distributions for the above responses are represented in the histograms in Figure 39 to
Figure 41.

Frequency
N Min Max Median Mean Std. Deviation Distribution

4 F) Sometimes I felt that I didn’t

quite understand what the e- 47 1 5 3.00 3.11 1.15 Refer to Figure 39
learning software was doing.

4 G) I found errors (bugs) in the

e-learning software that were 47 1 5 3.00 2.74 1.24 Refer to Figure 40
difficult to recover from.

4 H) The e-learning software felt

speedy and responsive to my 44 1 5 3.00 3.07 1.17 Refer to Figure 41
interactions.
Table 49: Phase 3 - Descriptive Statistics for mixed usability responses

As detailed in Table 49, Figure 39 reflects a distribution (N=47) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 3, M=3.11, SD=1.15) neither agree nor
disagree (3) with the statement “Sometimes I felt that I didn’t quite understand what the E-Learning
software was doing.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 39: Phase 3- Sometimes I felt that I didn’t quite understand what the e-learning software was doing.

As detailed in Table 49, Figure 40 reflects a distribution (N=47) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 3, M=2.74, SD=1.24) neither agree nor

206
 CHAPTER SIX

disagree (3) with the statement “I found errors (bugs) in the E-Learning software that were difficult to
recover from.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 40: Phase 3 - I found errors (bugs) in the e-learning software that were difficult to recover from.

As detailed in Table 49, Figure 41 reflects a distribution (N=44) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 3, M=3.07, SD=1.17) neither agree nor
disagree (3) with the statement “The E-Learning software felt speedy and responsive to my interactions.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 41: Phase 3- The e-learning software felt speedy and responsive to my interactions.

From 47 student participants, 14 offered additional comments relating to their usability feedback; the
following are a representative subset:

“At some points entering and checking an answer made me unable to answer the next questions affecting
my score.” [P3Student63]

207
 CHAPTER SIX

“I can’t open the course on my mac or phones.” [P3Student12]

“the level 2 didn't work for me at first and it lags considerably when I try to advance through its stages.”
[P3Student2]

“errors and bugs” [P3Student35]

“Sometimes the program stutters and you have to restart” [P3Student10]

“When you get most of the questions right in the e-learning software and you still get a low mark.”
[P3Student8]

“It crashes and you get 0%.” [P3Student23]

One recurring theme, mentioned by five students, was the marking and a perceived inconsistency
between the marking on the quiz questions and the final quiz result for each level. This was also recounted
by the teacher from School1 who commented on this issue during the debrief session.

6.6.3 Response on the Educational Components used in the E-learning

The student participants gave broadly positive feedback on the impact the different educational
components had on their perceived understanding. With reference to Table 50, the median student
response showed agreement (4) with the following statements:

• The text material in the e-learning software helped me understand the subject matter.
• The videos in the e-learning software helped me understand the subject matter.
• The visual material in the e-learning software helped me understand the subject matter.
• The collaborative activities (forum discussions, group or pair work) helped me understand the
subject matter.
• The practice activities (problem-solving) in the e-learning software helped me understand the
subject matter.
• The quiz questions (assessment activities) in the e-learning software helped me understand the
subject matter.
• The “extend your knowledge” learning material in the e-learning software helped me understand
the subject matter.

The frequency distributions for the above responses are represented in the histograms in Figure 42 to
Figure 48.

208
 CHAPTER SIX

Frequency
N Min Max Median Mean Std. Deviation Distribution

6 H) The text material in the e-

learning software helped me 44 1 5 4.00 3.61 1.02 Refer to Figure 42
understand the subject matter.

6 H) Outliers Removed 43 2 5 4.00 3.67 0.94

6 K) The videos in the e-learning

software helped me understand 44 1 5 4.00 3.50 1.11 Refer to Figure 43
the subject matter.

6 K) Outliers Removed 41 2 5 4.00 3.68 0.91

6 F) The visual material in the e-

learning software helped me 45 1 5 4.00 3.64 1.09 Refer to Figure 44
understand the subject matter.

6 F) Outliers Removed 43 2 5 4.00 3.77 0.95

6 I) The collaborative activities

(forum discussions, group or pair
45 1 5 4.00 3.44 1.12 Refer to Figure 45
work) helped me understand the
subject matter.

6 I) Outliers Removed 41 2 5 4.00 3.68 0.85

6 D) The practice activities

(problem-solving) in the e-
45 1 5 4.00 3.58 0.99 Refer to Figure 46
learning software helped me
understand the subject matter.

6 D) Outliers Removed 43 2 5 4.00 3.70 0.83

6 E) The quiz questions

(assessment activities) in the e-
45 1 5 4.00 3.67 1.15 Refer to Figure 47
learning software helped me
understand the subject matter.

6 E) Outliers Removed 41 2 5 4.00 3.93 0.82

6 J) The “extend your

knowledge” learning material in
the e-learning software helped 45 1 5 4.00 3.33 1.13 Refer to Figure 48
me understand the subject
matter.

6 J) Outliers Removed 40 2 5 4.00 3.63 0.81

Table 50: Phase 3 - The impact of the different educational components on perceived understanding

209
 CHAPTER SIX

As detailed in Table 50, Figure 42 reflects a distribution (N=44) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 4, M=3.61, SD=1.02) agrees (4) with the
statement “The text material in the E-Learning software helped me understand the subject matter.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 42: Phase 3 - The text material in the e-learning software helped me understand the subject matter.

As detailed in Table 50, Figure 43 reflects a distribution (N=44) that is not normal, negatively skewed and
is kurtosis static, where the average student response (Mdn = 4, M=3.50, SD=1.11) agrees (4) with the
statement “The videos in the E-Learning software helped me understand the subject matter.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 43: Phase 3 - The videos in the e-learning software helped me understand the subject matter.

As detailed in Table 50, Figure 44 reflects a distribution (N=45) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 4, M=3.64, SD=1.09) agrees (4) with the
statement “The visual material in the E-Learning software helped me understand the subject matter.”

210
 CHAPTER SIX

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 44: Phase 3 - The visual material in the e-learning software helped me understand the subject matter.
As detailed in Table 50, Figure 45 reflects a distribution (N=45) that is not normal, negatively skewed and
is kurtosis static, where the average student response (Mdn = 4, M=3.44, SD=1.12) agrees (4) with the
statement “The collaborative activities (forum discussions, group or pair work) helped me understand the
subject matter.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 45: Phase 3 The collaborative activities (forum discussions, group or pair work) helped me understand the
subject matter.

As detailed in Table 50, Figure 46 reflects a distribution (N=45) that is not normal, is negatively skewed
and is kurtosis static, where the average student response (Mdn = 4, M=3.58, SD=0.99) agrees (4) with
the statement “The practice activities (problem solving) in the E-Learning software helped me understand
the subject matter.”

211
 CHAPTER SIX

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 46: Phase3 - The practice activities (problem-solving) in the e-learning software helped me understand the
subject matter.

As detailed in Table 50, Figure 47 reflects a distribution (N=45) that is not normal, negatively skewed and
is kurtosis static, where the average student response (Mdn = 4, M=3.67, SD=1.15) agrees (4) with the
statement “The quiz questions (assessment activities) in the E-Learning software helped me understand
the subject matter.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 47: Phase 3- The quiz questions (assessment activities) in the e-learning software helped me understand
the subject matter.

As detailed in Table 50, Figure 48 reflects a distribution (N=45) that is not normal, negatively skewed and
is kurtosis static, where the average student response (Mdn = 4, M=3.33, SD=1.13) agrees (4) with the
statement “The “extend your knowledge” learning material in the E-Learning software helped me
understand the subject matter.”

212
 CHAPTER SIX

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 48: Phase 3 - The ‘extend your knowledge’ learning material in the e-learning software helped me
understand the subject matter.

One exception to the positive feedback, on the impact of the educational components on perceived
understanding, is audio content. With N=43, on average (Mdn=3, M=3.09 SD=1.19) the student
participants neither agreed nor disagreed (3) with the statement “The audio material in the e-learning
software helped me understand the subject matter.” Refer to Figure 49 for the associated frequency
distribution, which is not normal, not skewed and is kurtosis static.

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 49: Phase 3 - The audio material in the e-learning software helped me understand the subject matter.
In addition, the student participants ranked the educational components in terms of their perceived
benefit to learning and their perceived influence on enthusiasm and interest, where 1 has most benefit /
influence and 10 has least benefit / influence. Table 51 shows the rankings for both benefit to learning
and influence on enthusiasm and interest. Additionally, it reflects the difference in the rankings between

213
 CHAPTER SIX

the two. There is consistency between the bottom three rankings; however, there is a mirror effect where
the top three rankings for learning benefit are ranked lower in influence on interest and enthusiasm, and
the middle four rankings are ranked higher in terms of influence on interest and enthusiasm.

Perceived Benefit Perceived Influence

to Learning on Interest and Enthusiasm

Educational Component pct. (%) Rank Rank pct. (%) Educational Component

Text 8.48% 1 1 7.44% Games

Video 8.48% 1 2 8.82% Quizzes

Practice Activities 8.91% 3 3 9.04% Animations / Simulations

Quizzes 9.58% 4 4 9.26% Pictures / Photos / Diagrams

Pictures / Photos / Diagrams 9.58% 4 5 9.31% Video

Animations / Simulations 9.76% 6 6 9.75% Text

Games 10.06% 7 7 10.96% Practice Activities

Audio 11.21% 8 8 11.46% Audio

Collaborative Activities 11.58% 9 9 11.90% Collaborative Activities

Extended Knowledge 12.36% 10 10 12.07% Extended Knowledge

Table 51: Phase 3- Educational components impact on perceived learning, and perceived interest and enthusiasm

6.6.4 Response on Perceived Difficulty

Based on the following Likert scale: 1 Too Easy, 2 A Little Easy, 3 At the Right Level, 4 A Little Difficult and
5 Too Difficult, the student participants gave feedback on the perceived difficulty of the educational
content in the prototype e-learning software.

With reference to Table 52, the median student response showed the difficulty level of the following
educational areas to be at the right level (3):

1. Learning material
2. Practice activities (problem-solving)
3. Quiz questions (assessment activities)

The frequency distributions for the above responses are represented in the histograms in Figure 50 to
Figure 52.

214
 CHAPTER SIX

Frequency
N Min Max Median Mean Std. Deviation Distribution

8 A) Overall, at what difficulty

level do you believe the learning
40 2 5 3.00 3.35 0.62 Refer to Figure 50
material represented in the e-
learning software was at?

8 B) Overall, at what difficulty

level do you believe the practice
40 1 5 3.00 3.05 0.90 Refer to Figure 51
activities (problem-solving) were
at?

8 C) Overall, at what difficulty

level do you believe the quiz
40 2 5 3.00 3.15 0.86 Refer to Figure 52
questions (assessment activities)
were at?
Table 52: Phase 3- Difficulty level of the educational content in the prototype e-learning software

As detailed in Table 52, Figure 50 reflects a distribution (N=40) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 3, M=3.35, SD=0.62) asses the learning
material represented in the e-learning software to be at the right level (3).

1. Too Easy
2. A Little Easy
3. At the Right Level
4. A Little Difficult
5. Too Difficult

Figure 50: Phase 3- Overall, at what difficulty level do you believe the learning material represented in the e-
learning software was at?

As detailed in Table 52, Figure 51 reflects a distribution (N=40) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 3, M=3.05, SD=0.90) asses the difficulty of
practice activities (problem solving) to be at the right level (3).

215
 CHAPTER SIX

1. Too Easy
2. A Little Easy
3. At the Right Level
4. A Little Difficult
5. Too Difficult

Figure 51: Phase 3- Overall, at what difficulty level do you believe the practice activities (problem-solving) were
at?

As detailed in Table 52, Figure 52 reflects a distribution (N=40) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 3, M=3.15, SD=0.86) asses the difficulty of
quiz questions (assessment activities) to be at the right level (3).

1. Too Easy
2. A Little Easy
3. At the Right Level
4. A Little Difficult
5. Too Difficult

Figure 52: Phase 3 - Overall, at what difficulty level do you believe the quiz questions (assessment activities) were
at?
Although the student average response indicates that the e-learning software is at the right difficulty level;
the students remain non-committal on whether they needed to supplement the e-learning software with
additional learning material, or whether they felt the need to ask their teacher for support. With reference

216
 CHAPTER SIX

to Table 53, the median student response showed that they neither agreed nor disagreed (3) with the
following statements:

1. I supplemented, or needed to supplement, the learning material in the e-learning software with
further textbook reading.
2. I asked, or wanted to ask, my teacher for support in understanding the learning material in the
e-learning software.

N Min Max Median Mean Std. Deviation Frequency Distribution

6 L) I supplemented, or needed
to supplement, the learning
material in the e-learning 45 1 5 3.00 3.13 1.06 Refer to Figure 53
software with further textbook
reading.

6 L) Outliers Removed 40 2 5 3.00 3.40 0.78

6 M) I asked, or wanted to ask,

my teacher for support in
understanding the learning 45 1 5 3.00 3.31 1.15 Refer to Figure 54
material in the e-learning
software.

6 M) Outliers Removed 41 2 5 4.00 3.54 0.93

Table 53: Phase 3 - Student response on whether they needed to supplement the e-learning software.

As detailed in Table 53, Figure 53 reflects a distribution (N=45) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 3, M=3.13, SD=1.06) ) neither agree nor
disagree with the statement “I supplemented, or needed to supplement, the learning material in the E-
Learning software with further textbook reading.”

217
 CHAPTER SIX

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 53: Phase 3 - I supplemented, or needed to supplement, the learning material in the e-learning software
with further textbook reading.

As detailed in Table 53, Figure 54 reflects a distribution (N=45) that is not normal, not skewed and is
kurtosis static, where the average student response (Mdn = 3, M=3.31, SD=1.15) neither agrees nor
disagrees (3) with the statement “I asked, or wanted to ask, my teacher for support in understanding the
learning material in the E-Learning software. However, It should be noted that the removal of 4 outliers
from question 6 M) meant N=41 and the median student response now showed agreement (4) with the
previous statement.

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 54: Phase 3 - I asked, or wanted to ask, my teacher for support in understanding the learning material in
the e-learning software.
There is potential for concern that 19 of the 45 participants either agreed or strongly agreed with the
statement that they supplemented or needed to supplement the learning material in the e-learning

218
 CHAPTER SIX

software with further textbook reading. This sentiment is reinforced by the feedback that 22 of the 45
respondents asked or wanted to ask their teacher for support in understanding the learning material in
the e-learning software.

It should be noted that 13 students confirmed that they did use other educational material between the
pre-test and post-test.

The previous two responses give a potential explanation for why, with N=45, on average (Mdn=3, M=3.22
SD=1.04) the student participants neither agreed nor disagreed (3) with the statement “After completing
the 4 levels of the e-learning software, I was confident that I would be able to pass a test on it.” Refer to
Figure 55 for the frequency distribution of responses, which is not normal, not skewed and kurtosis static.

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 55: Phase 3 - After completing the four levels of the e-learning software, I was confident that I would be
able to pass a test on it.

It should be noted though that with the removal of three outliers, N=42 and the mean average (Mdn=3,
M=3.38 SD=0.88) increased.

6.6.4.1 Survey Reliability Results

In the Phase 3 survey instrument, alternate form reliability is established by adjusting the wording and
reversing the scale for the following question pair:

• 4 A) All things considered, the e-learning software is easy to use, and

• 12 A) Overall, the e-learning software was difficult to use.

With reference to Table 54, a medium effect negative correlation is reflected.

12 A) Overall, the E-Learning software

was difficult to use.

Spearman's rho Correlation Coefficient -.35

Sig. (2-tailed) .03

219
 CHAPTER SIX

4 A) All things considered, N

the e-learning software is 41
easy to use.
Table 54: Phase 3 - Alternate form reliability for survey instrument

Internal consistency is established by assessing two groups of questions using Cronbach’s alpha, to
identify whether they measure the same underlying concept

• Questions 4A), 4C), 4D), 4E), 4H), 4I) 4J) and 4K) all measure usability of the prototype software,
• Questions 8A), 8B) and 8C all measure the difficulty of the prototype software.

With reference to Table 55, acceptable internal consistency is reflected in both scales (refer to section
6.6.5.2 for a brief discussion on threshold values for Cronbach’s alpha).

Reliability Estimate
Scale Items in Scale
(Cronbach )

E-learning Usability .74 8

E-learning Difficulty .66 3

Table 55: Phase 3 - Internal consistency for survey instrument

6.6.5 Motivation and Preferences

The participant response on the impact of the e-learning prototype on motivation is mixed; with N=41,
on average (Mdn=3, M=3.15 SD=1.06) the student participants neither agreed nor disagreed (3) with the
statement “I would prefer using the Internet and the Web to support my learning of the computing subject
rather than this E-Learning software.” Refer to Figure 56 for the frequency distribution, which is not
normal, not skewed and kurtosis static.

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 56: Phase 3- I would prefer using the internet and the web to support my learning of the computing
subject rather than this e-learning software.

220
 CHAPTER SIX

With N=41, on average (Mdn=3, M=3.37 SD=1.18) the student participants neither agreed nor disagreed
(3) with the statement “It is more interesting to use the E-Learning software to learn Computing than the
textbooks.” Refer to Figure 57 for the frequency distribution, which is not normal, not skewed and kurtosis
static. However, has multiple modes in that an equal number of students neither agreed nor disagreed
(3) and agreed (4) with the aforementioned statement.

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 57: Phase 3- It is more interesting to use the e-learning software to learn computing than the textbooks.

However, with the removal of four outliers, N=37 and now, on average (Mdn=4, M=3.62 SD=0.92), the
student participants agreed (4) with the statement “It is more interesting to use the E-Learning software
to learn Computing than the textbooks.”

Furthermore, with reference to Table 56, the average student response showed agreement (4) with the
following statements:

1. I could use the e-learning software for independent study to learn computing.
2. The e-learning software has increased my overall enthusiasm and interest in computing.

Frequency
N Min Max Median Mean Std. Deviation Distribution

10 B) I could use the e-learning

software for independent study 41 1 5 4.00 3.54 0.98 Refer to Figure 58
to learn computing.

10 B) Outliers Removed 39 2 5 4.00 3.67 0.81

10 C) The e-learning software has

increased my overall enthusiasm 41 1 5 4.00 3.39 1.05 Refer to Figure 59
and interest in computing.

10 C) Outliers Removed 37 2 5 4.00 3.65 0.72

Table 56: Phase 3 - Participant responses on the prototype’s effect on overall enthusiasm and interest in
computing

221
 CHAPTER SIX

As detailed in Table 56, Figure 58 reflects a negatively skewed distribution (N=41) where the average
student response (Mdn=4, M=3.54, SD=0.98) agrees (4) with the statement “I could use the E-Learning
software for independent study to learn Computing.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 58: Phase 3 - I could use the e-learning software for independent study to learn computing.

As detailed in Table 56, Figure 59 reflects a negatively skewed distribution (N=41) where the average
student response (Mdn=4, M=3.39, SD=1.05) agrees (4) with the statement “The E-Learning software has
increased my overall enthusiasm and interest in Computing.”

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 59: Phase 3- The e-learning software has increased my overall enthusiasm and interest in computing.

6.6.5.1 Instructional Materials Motivation Survey

A more comprehensive view of the participants’ level of motivation is provided by Keller’s (2006)
Instructional Materials Motivation Survey (IMMS), which groups questions according to the motivational

222
 CHAPTER SIX

areas of attention, relevance, confidence and satisfaction, and measures responses on the following Likert
scale: 1 Not True, 2 Slightly True, 3 Moderately True, 4 Mostly True and 5 Very True.

With reference to Table 57, the median student response for the ARCS subcategories ranged between
3.00 and 3.33, which is between Moderately True (3) and Mostly True (4). These overall weakly positive
results for the ARCS subcategories are represented in Figure 60.

Phase 3 IMMS Results

5

4.5

3.5 3.24 3.14 3.24 3.15

3.03
3

2.5

1.5

0.5

0
Phase 3 ARCS Response
Attention Relevance Confidence Satisfaction Overall

Figure 60: Phase3 - IMMS ARCS summary results (mean)

The frequency distributions for ARCS (IMMS) overall and its subcategories are represented in the
histograms in Figure 61 to Figure 65.

Frequency
N Min Max Median Mean Std. Deviation Distribution

Student Attention 42 1.17 4.36 3.00 3.03 0.54 Refer to Figure 61

Outliers Removed 39 2.25 3.67 3.00 3.01 0.36

Student Relevance 42 1.44 4.75 3.29 3.24 0.71 Refer to Figure 62

Outliers Removed 40 2.00 4.75 3.33 3.33 0.60

Student Confidence 42 1.67 4.78 3.11 3.14 0.51 Refer to Figure 63

Outliers Removed 40 2.56 4.11 3.11 3.14 0.39

Student Satisfaction 42 1.00 4.83 3.33 3.24 0.97 Refer to Figure 64

Outliers Removed 39 2.00 4.83 3.33 3.41 0.77

IMMS Overall 42 1.56 4.56 3.11 3.15 0.56 Refer to Figure 65

Outliers Removed 40 2.00 4.34 3.11 3.15 0.46

Table 57: Phase 3 - IMMS summary results for Attention, Relevance, Confidence and Satisfaction

223
 CHAPTER SIX

As detailed in Table 57, Figure 61 reflects a normal distribution (N=42) where the average student
response (Mdn = 3, M=3.03, SD=0.54) is between moderately true (3) and mostly true (4) for attention.

1. Not True
2. Slightly True
3. Moderately True
4. Mostly True
5. Very True

Figure 61: Phase 3 - IMMS Attention frequency distribution

As detailed in Table 57, Figure 62 reflects a normal distribution (N=42) where the average student
response (Mdn = 3.29, M=3.24, SD=0.71) is between moderately true (3) and mostly true (4) for
relevance.

1. Not True
2. Slightly True
3. Moderately True
4. Mostly True
5. Very True

Figure 62: Phase 3 - IMMS Relevance frequency distribution

As detailed in Table 57, Figure 63 reflects a normal distribution (N=42) where the average student
response (Mdn = 3.11, M=3.14, SD=0.51) is between moderately true (3) and mostly true (4) for
confidence.

224
 CHAPTER SIX

1. Not True
2. Slightly True
3. Moderately True
4. Mostly True
5. Very True

Figure 63: Phase 3 - IMMS Confidence frequency distribution

As detailed in Table 57, Figure 64 reflects a normal distribution (N=42) where the average student
response (Mdn = 3.33, M=3.24, SD=0.97) is between moderately true (3) and mostly true (4) for
satisfaction.

1. Not True
2. Slightly True
3. Moderately True
4. Mostly True
5. Very True

Figure 64: Phase 3 - IMMS Satisfaction frequency distribution

As detailed in Table 57, Figure 65 reflects a normal distribution (N=42) where the average student
response (Mdn = 3.11, M=3.15, SD=0.56) is between moderately true (3) and mostly true (4) for IMMS
overall.

225
 CHAPTER SIX

1. Not True
2. Slightly True
3. Moderately True
4. Mostly True
5. Very True

Figure 65: Phase 3 - IMMS overall motivation frequency distribution

The mean participant result for Attention and Confidence is comparatively less than both Relevance and
Satisfaction. Using the Kolmogorov-Smirnov test, the distributions for Attention, Relevance, Confidence
and Satisfaction are confirmed to be normal. With reference to Table 58, paired sample T-Tests were used
to compare the ARCS subcategories.

Paired Samples Test

Paired Differences

95% Confidence
Interval of the
Difference
Std. Std. Error Sig. (2-
Mean Deviation Mean Lower Upper t df tailed)

Pair 1 StudentAttention -
-.21 0.58 0.09 -0.39 -0.03 -2.35 41 .02
StudentRelevance

Pair 2 StudentAttention -
-.11 0.35 0.05 -0.22 0.00 -1.99 41 .05
StudentConfidence

Pair 3 StudentAttention -
-.20 0.80 0.12 -0.45 0.04 -1.66 41 .11
StudentSatisfaction

Pair 4 StudentConfidence -
-.10 0.57 0.09 -0.28 0.08 -1.14 41 .26
StudentRelevance

Pair 5 StudentConfidence -
-.09 0.84 0.13 -0.36 0.17 -0.74 41 .47
StudentSatisfaction

Pair 6 StudentRelevance -
.01 0.57 0.09 -0.17 0.18 0.06 41 .95
StudentSatisfaction

Table 58: Phase 3 - Paired Sample T-Tests for ARCS sub-categories.

226
 CHAPTER SIX

The results from Table 58, indicate that only Attention vs Relevance results have a statistically significant
difference. Meaning overall, the average student response on the ARCS subcategories for the e-learning
prototype are broadly balanced and aligned.

6.6.5.2 IMMS Internal Reliability

As discussed in section 6.2.4.1, the internal reliability of the IMMS instrument was originally established
by Keller (2006); however, it is good practice to re-establish this reliability in the study itself. Refer to Table
59 for the Cronbach alpha results for Phase 3. Much is written on the threshold values for Cronbach’s
alpha; Nunnally (2017) advises that the level of reliability depends on how the measure will be used, and
recommends .70 and above for basic research. Others, such as Churchill et al. (Churchill & Peter 1984),
and Loewenthal and Eysenck (2001), advise that .60 can be acceptable. Based on this, we ascertain that
the internal consistency for the Attention and Confidence sub-scales are questionable, but that the
remaining sub-scales and the overall IMMS scale reliability have good to excellent internal consistency.

Reliability Estimate
Scale Items in Scale
(Cronbach )

Attention .59 12

Relevance .80 9

Confidence .50 9

Satisfaction .90 6

Total Scale .89 36

Table 59: Phase 3 IMMS Cronbach's alpha results

6.6.6 Learning Styles and Multi-modal Approach (VARK)

6.6.6.1 Participants

From the 66 student participants in Phase 3, 65 participants responded to the VARK instrument. There
was negligible non-response to the VARK questions; in any case, even a non-response is acceptable since
for each question there can be zero, one, or multiple responses from each participant.

227
 CHAPTER SIX

6.6.6.2 Survey Response

The student participants offered 1860 individual responses to the VARK instrument, which were grouped
according to Visual, Aural, Read-Write and Kinaesthetic modalities. With reference to Figure 66 the modal
preference across all students are comparatively balanced.

Phase 3 VARK Modal Preferences

30.00%

25.00% 26.51%
24.95% 25.11%
23.44%
20.00%

15.00%

10.00%

5.00%

0.00%

Visual Aural Read/Write Kineasthetic

Figure 66: Phase 3 - VARK modal preferences for students

With reference to Table 60, the VARK sub-category response distributions were tested for normality using
the Kolmogorov-Smirnov test, with only the Aural sub-category considered as normal. In accordance with
section 3.6.5.1, instead of moving to non-parametric tests, the results from both parametric and non-
parametric tests are presented.

Kolmogorov-Smirnov a

Statistic df Sig.

Visual .13 65 .01

Aural .11 65 .06

Read/Write .12 65 .02

Kinaesthetic .15 65 .00

a. Lilliefors Significance Correction

Table 60: Phase 3 - Kolmogorov-Smirnov results for VARC sub-categories

With reference to Table 61 and Table 62, both parametric and non-parametric tests reflect that only Visual
and Kinaesthetic sub-categories have a statistically significant difference. The differences between the
other VARK subcategories are not statistically significant, which in turn offers support to the multi-modal
approach proposed in the e-learning pedagogy.

228
 CHAPTER SIX

Paired Samples Test

Paired Differences

95% Confidence
Interval of the
Difference
Std. Std. Error Sig. (2-
Mean Deviation Mean Lower Upper t df tailed)

Pair 1 Visual - Aural -.43 3.05 .39 -1.19 .33 -1.14 64 .26

Pair 2 Visual – Read/Write -.48 2.88 .36 -1.19 .24 -1.33 64 .19

Pair 3 Visual - Kinaesthetic -.88 3.50 .43 -1.75 -.01 -2.02 64 .05

Pair 4 Aural – Read/Write -.05 3.25 .40 -.85 .76 -.12 64 .91

Pair 5 Aural - Kinaesthetic -.45 3.03 .38 -1.20 .31 -1.19 64 .24

Pair 6 Read/Write -
-.40 3.48 .43 -1.26 .46 -.93 64 .36
Kinaesthetic

Table 61: Phase 3 - Paired Sample T-Test for VARK sub-categories.

Sig.

Pair 1 Visual - Aural .22

Pair 2 Visual – Read/Write .19

Pair 3 Visual - Kinaesthetic .02

Pair 4 Aural – Read/Write .80

Pair 5 Aural - Kinaesthetic .29

Pair 6 Read/Write - Kinaesthetic .22

Table 62: Phase 3 - Related Samples Wilcoxon Signed Rank test for VARK sub-categories.

A positive result in relation to multi-modal learning is that with N=45, on average (Mdn=4, M=3.60
SD=0.94) the student participants agreed (4) with the statement “The use of different methods to
represent the same learning content helped my understanding.” Refer to Figure 67 for the associated
frequency distribution, which is not normal, is negatively skewed and kurtosis static. With the removal of
two outliers, N=43 and the mean average (Mdn=4, M=3.72 SD=0.77) increased.

229
 CHAPTER SIX

1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree

Figure 67: Phase 3- The use of different methods to represent the same learning content helped my
understanding.
A further weakly positive result for multi-modal learning is that with N=39, on average (Mdn=3, M=3.00
SD=0.95) the student participants neither agreed nor disagreed (3) with the statement “The use of
different methods to represent the same learning content made me feel overloaded.”

6.6.7 Student Learning Performance

6.6.7.1 Participants

From the 66 student participants in Phase 3, 56 students attempted the pre- and post-tests. After
removing all students: who did not get a result for both tests; who acknowledged they had used other
learning materials; or who spent zero time on both Level-3 and Level-4 of the e-learning prototype, the
number of viable pre- and post-test participants remaining was 48.

6.6.7.2 Student Engagement with the E-learning Prototype

Prior to taking the post-test, the students used the four levels of the e-learning software prototype. With
reference to Table 63, the average duration of student engagement with e-learning software is reported
with and without outliers. The frequency distributions for the student engagement durations are
represented in the histograms in Figure 68 to Figure 71.

Frequency
N Min Max Median Mean Std. Deviation Distribution

Level-1 Duration 48 21 300 99.50 103.81 53.30 Refer to Figure 68

Outliers Removed 45 21 198 96.00 93.89 36.68

Level-2 Duration 48 0 57 14.50 16.79 13.24 Refer to Figure 69

Outliers Removed 46 0 45 14.00 15.08 10.58

230
 CHAPTER SIX

Frequency
N Min Max Median Mean Std. Deviation Distribution

Level-3 Duration 48 0 167 46.00 49.19 34.04 Refer to Figure 70

Outliers Removed 45 0 79 45.00 41.89 19.05

Level-4 Duration 48 0 107 23.50 33.65 26.58 Refer to Figure 71

Table 63: Phase 3 - Average duration of engagement with the four levels of e-learning software

As detailed in Table 63, Figure 68 reflects a positively skewed and leptokurtic distribution (N=48) where
the average student learning duration for Level-1 was (Mdn =99.50, M=103.81, SD=53.30).

Figure 68: Phase 3 - Duration of student engagement with Level-1

As detailed in Table 63, Figure 69 reflects a positively skewed and leptokurtic distribution (N=48) where
the average student learning duration for Level-2 was (Mdn =14.50, M=16.79, SD=13.24).

231
 CHAPTER SIX

Figure 69: Phase 3 - Duration of student engagement with Level-2

As detailed in Table 63, Figure 70 reflects a positively skewed and leptokurtic distribution (N=48) where
the average student learning duration for Level-3 was (Mdn =46.00, M=49.19, SD=34.03).

Figure 70: Phase 3 - Duration of student engagement with Level-3

As detailed in Table 63, Figure 71 reflects a positively skewed and kurtosis static distribution (N=48) where
the average student learning duration for Level-4 was (Mdn =23.50, M=33.65, SD=26.58).

Figure 71: Phase 3 - Duration of student engagement with Level-4

It is important to recall that Level-1 and Level-2 of the e-learning software provide foundational concepts
on algorithms and computational thinking, whereas Level-3 and Level-4 provide learning material on
flowcharts and pseudocode, which are of direct relevance to the pre- post-test exam. These durations
have therefore been analysed in relation to all four levels (L1L2L3L4) of the e-learning software, and
separately for Level-3 and -4 (L3L4) of the e-learning software. Using the Kolmogorov-Smirnov test, the

232
 CHAPTER SIX

distributions for L1L2L3L4 and L3L4 durations, with and without outliers, are not all normal. Therefore,
Table 64 reflects the 95% Confidence Interval with and without outliers, based on median learning
duration; these are constructed without any distribution assumption.

Coefficient of
95% Confidence Interval for Median Variation

Lower Upper Actual Price- Related Coefficient of

Variable N Mdn Bound Bound Coverage Differential Dispersion Median Centred

L1L2L3L4 48 196.50 172.00 219.00 97.1% 1.00 .28 44.0%

No Outliers 44 188.00 167.10 205.00 95.1% 1.00 .21 27.2%

L3L4 48 64.50 56.00 100.00 97.1% 1.00 .55 80.0%

No Outliers 46 63.00 52.00 99.00 97.4% 1.00 .47 63.3%

Table 64: Phase 3 - Confidence Intervals for the median learning duration for the e-learning software

6.6.7.3 Collaborative Learning Activities and Individual Assignments

The student participants showed a high level of engagement with the practice and quiz activities in the e-
learning software; however, for the most part, this did not extend to collaborative learning and
assignment activities hosted on the collaborative learning environment. Refer to Table 65 for a summary
of student responses to collaborative learning and assignment activities.

Response
Software Title and Description Comment
Count

Level-1 Decomposition Activity 0 responses A moderately challenging activity to

White analyse a problem and break it
Adapted from a GCSE Non-Examinable
down into smaller more
Assessment (NEA).
manageable sub-problems.
The students are asked to consider two
problem areas and decompose them
into smaller sub-problems

Level-1 The Characteristics of Algorithms 13 posts by 12 Most of the responses incorrectly

White authors. define the characteristics of
Forum question for students to discuss
computational thinking instead of
the characteristics of algorithms, 31 Views
algorithms.
whether they are important and why,
and if algorithms are only used in All students identify the importance
computers? of Algorithms.

Except for one student, all students

identify that Algorithms are not
only used with computers.

233
 CHAPTER SIX

Response
Software Title and Description Comment
Count

Level-1 Think about the various criteria that 31 responses 10 responses are incorrect, three of
White can be used to evaluate the going to from 25 which have misunderstood the
school algorithms. participants. question.

Individual Assignment to outline Most of the remaining responses

different evaluation criteria and then were copy / paste from the hint
Students did
evaluate the four algorithms to go to dialogue in the e-learning software.
not use the
school.
assignment Three responses show a level of
mechanism, critical thinking in evaluating the
instead of algorithms and offering a
answering with recommendation.
comments,
similar to a
forum
question.

Level-2 Understanding the example Algorithm 15 posts by 14 Most of the responses are
Yellow authors. underdeveloped and four responses
Forum questions to discuss what a
were copy / paste from the hint
simple algorithm does and to discuss 47 Views
dialogue in the e-learning software.
whether the students' found the
pseudocode or flowchart version easier
to understand.

Level-3 Direction Indicators Algorithm 0 responses A moderately challenging

Orange assignment which is scaffolded by
An individual assignment to create a
multiple hint files that progressively
flowchart algorithm for the direction
complete the algorithm.
indicator in a car.

Level-3 Trainers Algorithm Flowchart 0 responses A moderately challenging

Orange assignment which is scaffolded by
An individual assignment to create a
multiple hint files that progressively
flowchart to convert US trainer sizes to
complete the algorithm.
the equivalent EU size.

Level-4 Practice in writing Pseudocode – Dice 0 responses A moderately challenging

Blue Game assignment which is scaffolded by
multiple hint dialogues that
An individual assignment to create a
progressively complete the
pseudocode algorithm of a given
pseudocode.
flowchart representing a dice game.

Level-4 Partner Work to Create a Dog Age 5 responses A moderately challenging

Blue Algorithm uploaded, but assignment which is scaffolded by
multiple hint dialogues that

234
 CHAPTER SIX

Response
Software Title and Description Comment
Count
all were blank progressively complete the
Pair work to create a pseudocode
documents. pseudocode.
algorithm converts a dog’s age to the
human equivalent.
Table 65: Phase 3 - Summary of student responses to collaborative learning and assignment activities

An important result in relation to learning performance is that none of the students attempted the four
assignment activities in Level-3 and -4, which would have given them an opportunity to practice skills
transferable to the pre-, post-test.

6.6.7.4 Learning Performance

6.6.7.4.1 Descriptive Statistics

For reporting purposes, the pre-, post-test and change variables have been transformed to percentage
values. Table 66, reflects descriptive statistics for %Pre-Test, %Post-Test and %Change, and the associated
frequency distributions are reflected in Figure 72 to Figure 74. In summary, with N=48, an improvement
in learning performance between pre-test and post-test is indicated (subject to further statistical
analysis):

• %Pre-Test had a mean average of 15.93% (SD=11.38), 95% CI [12.62%, 19.23%].

• %Post-Test had a mean average of 35.15% (SD=13.80), 95% CI [31.14%, 39.16%].
• %Change had a mean average of 19.22% points increase (SD=12.20), 95% CI [15.68%, 22.76%].

N=48 Statistic Std. Error

%Pre-Test Mean 15.93 1.64

95% Confidence Interval for Lower Bound 12.62

Mean Upper Bound 19.23

5% Trimmed Mean 14.98

Median 12.90

Variance 129.48

Std. Deviation 11.38

Minimum .00

Maximum 54.84

Range 54.84

Interquartile Range 12.90

Skewness 1.21 .34

Kurtosis 2.67 .67

235
 CHAPTER SIX

N=48 Statistic Std. Error

%Post-Test Mean 35.15 1.99

95% Confidence Interval for Lower Bound 31.14

Mean Upper Bound 39.16

5% Trimmed Mean 34.38

Median 32.26

Variance 190.52

Std. Deviation 13.80

Minimum 12.90

Maximum 83.87

Range 70.97

Interquartile Range 19.35

Skewness 1.08 .34

Kurtosis 2.02 .67

%Change Mean 19.22 1.76

95% Confidence Interval for Lower Bound 15.68

Mean Upper Bound 22.76

5% Trimmed Mean 19.06

Median 19.35

Variance 148.75

Std. Deviation 12.20

Minimum -6.45

Maximum 51.61

Range 58.06

Interquartile Range 16.13

Skewness .20 .34

Kurtosis .10 .67

Table 66: Phase 3 - Descriptive statistics for %Pre-Test, %Post-Test and %Change.

As detailed in Table 66, Figure 72 shows (N=48) a positively skewed and leptokurtic frequency distribution
for %Pre-Test with average (Mdn=12.90%, M=15.93%, SD=11.38). It should be noted that removal of
outliers removes the positive skew and leptokurtic kurtosis.

236
 CHAPTER SIX

Figure 72: Phase 3 - Frequency Distribution for %Pre-Test.

As detailed in Table 66, Figure 73 shows (N=48) a positively skewed and leptokurtic frequency distribution
for %Post-Test with average (Mdn=32.26%, M=35.15%, SD=13.80). It should be noted that removal of
outliers removes the positive skew and leptokurtic kurtosis.

Figure 73: Phase 3 - Frequency Distribution for %Post-Test

As detailed in Table 66, Figure 74 shows (N=48) a normal frequency distribution for %Change with average
(Mdn=19.35%, M=19.22%, SD=12.20).

237
 CHAPTER SIX

Figure 74: Phase 3 - Frequency Distribution for %Change.

Outliers are identified for %Pre-Test, %Post-Test and %Change; hence, the associated descriptive statistics
without outliers are reflected in Table 67. In summary, with outliers removed, an improvement in learning
performance between pre-test and post-test remains evident (subject to further statistical analysis):

• N= 46, %Pre-Test had a mean average of 14.38% (SD=8.70), 95% CI [11.79%, 16.96%].
• N=47, %Post-Test had a mean average of 34.11% (SD=11.91), 95% CI [30.61%, 37.61%].
• N=47, %Change had a mean average of 18.53% points increase (SD=11.34), 95% CI [15.20%,
21.86%].

Outliers Removed Statistic Std. Error

%Pre-Test Mean 14.38 1.28

N=46 95% Confidence Interval for Lower Bound 11.79

Mean
Upper Bound 16.96

5% Trimmed Mean 14.20

Median 12.90

Variance 75.70

Std. Deviation 8.70

Minimum .00

Maximum 35.48

Range 35.48

Interquartile Range 10.48

Skewness .12 .35

238
 CHAPTER SIX

Outliers Removed Statistic Std. Error

Kurtosis -.37 .69

%Post-Test Mean 34.11 1.74

N=47 95% Confidence Interval for Lower Bound 30.61

Mean
Upper Bound 37.61

5% Trimmed Mean 33.83

Median 32.26

Variance 141.95

Std. Deviation 11.91

Minimum 12.90

Maximum 61.29

Range 48.39

Interquartile Range 19.35

Skewness .43 .35

Kurtosis -.39 .68

%Change Mean 18.53 1.65

N=47 95% Confidence Interval for Lower Bound 15.20

Mean
Upper Bound 21.86

5% Trimmed Mean 18.59

Median 19.35

Variance 128.70

Std. Deviation 11.34

Minimum -6.45

Maximum 41.94

Range 48.39

Interquartile Range 16.13

Skewness -.09 .35

Kurtosis -.39 .68

Table 67: Phase 3 - Descriptive statistics for %Pre-Test, %Post-Test and %Change (outliers removed)

6.6.7.4.2 Paired Sample T-Test

To test whether there is a change in learning performance between %Pre-Test and %Post-Test, a Paired
Sample T-Test was carried out. With reference to Table 68, the Paired Sample T-Test was carried out, with

239
 CHAPTER SIX

and without outliers. Both showed a statistically significant learning performance improvement of
approximately 19% points.

Paired Differences

95% Confidence
Interval of the
Std.
Difference
Std. Error Sig. (2-
Mean Deviation Mean Lower Upper t df tailed)

%Post-Test - %Pre-Test
19.22 12.12 1.76 15.68 22.76 10.92 47 .00
(N=48)

%Post-Test - %Pre-Test
19.36 12.15 1.79 15.75 22.96 10.81 45 .00
(No Outlier, N=46)
Table 68: Phase 3 - Paired Sample T-Test for %Post-Test and %Pre-Test

Using the Kolmogorov-Smirnov test, the distribution for %Post-Test is assessed as not normal, hence
failing one of the assumptions for the use of a T-Test. To compensate for this and validate the above
results, the T-Test was re-run using the bootstrapping option, and additionally further validated using non-
parametric testing (Wilcoxon Signed-Ranks Test).

6.6.7.4.3 Paired Sample T-Test using Bootstrapping

With reference to Table 69, the Paired Sample T-Test was carried out using the Bootstrap option for 1000
samples, with and without outliers. Both showed a statistically significant learning performance
improvement of approximately 19% points.

Bootstrap for Paired Samples Test (1000 samples)

Bootstrapa

95% Confidence Interval

Std. Sig. (2-
Mean Bias Error tailed) Lower Upper

%Post-Test - %Pre-Test (N=48) 19.22 .06 1.75 .00 16.06 22.85

%Post-Test - %Pre-Test
19.36 .02 1.79 .00 15.64 22.72
No Outlier (N=46)
Table 69: Phase 3 - Paired Sample T-Test for %Post-Test and %Pre-Test using Bootstrap

6.6.7.4.4 Wilcoxon Signed-Ranks Test

A Wilcoxon Signed-Ranks Test (with outliers N=48) indicated that the median %Post-Test scores were
statistically significantly higher than the median %Pre-Test Scores Z= -5.86, p < .00.

A Wilcoxon Signed-Ranks Test (without outliers N=46) indicated that the median %Post-Test scores were
statistically significantly higher than the median %Pre-Test Scores Z=-5.74, p < .00.

Following on from the above results, Table 70 outlines the 95% Median Confidence Interval for %Change,
with and without outliers. In summary:

240
 CHAPTER SIX

• N=46, %Change had a median average of 20.97% points increase, 95% CI [12.90%, 25.81%].
• N=45, %Change had a median average of 19.35% points increase, 95% CI [12.90%, 22.58%].

Coefficient
95% Confidence Interval for Median of Variation

Lower Upper Actual Price Related Coefficient of Median

Variable N Mdn Bound Bound Coverage Differential Dispersion Centred

%Change 46 20.97 12.90 25.81 97.4% 1.00 .43 54.2%

%Change
45 19.35 12.90 22.58 96.4% 1.00 .43 53.8%
No Outliers
Table 70: Phase 3 - Confidence Interval for Median %Change between %Post-Test - %Pre-Test

6.6.7.5 Learning Performance vs GCSE Grade Boundaries

The pre-, post-test is based on sample exam questions from paper 1 of the new Computer Science GCSEs
from the following examination boards: EDEXCEL, OCR and AQA. The new computer science specification
started in 2016 and will be first examined in 2018, hence the grade boundaries have not been released.
Regarding the new GCSE grading scale, factsheets from the Department for Education (2017) advise that
the old letter and new numeric grading are not directly comparable, but offer three points of alignment:

1. The bottom of new grade 7 is aligned with the bottom of current grade A;
2. The bottom of new grade 4 is aligned with the bottom of current grade C; and
3. The bottom of new grade 1 is aligned with the bottom of current grade G.

With reference to recent GCSE Computer Science grade boundaries from EDEXCEL (Pearson 2016), OCR
(2017) and AQA (2016), Table 71 provides a realistic theoretical model of the grade boundaries for the
new GCSE specification.

Numeric
Grade Letter
Equivalent Grade AQA Edexcel OCR Mean
Total 84 100.00% 200 100.00% 80 100.00% 100.00%
9
A* 73 86.90% 157 78.50% 56 70.00% 78.47%
8
7 A 62 73.81% 129 64.50% 47 58.75% 65.69%
6
B 47 55.95% 101 50.50% 38 47.50% 51.32%
5
4 C 33 39.29% 73 36.50% 29 36.25% 37.35%
D 26 30.95% 61 30.50% 23 28.75% 30.07%
3
E 19 22.62% 49 24.50% 17 21.25% 22.79%
2
F 13 15.48% 37 18.50% 12 15.00% 16.33%
1 G 7 8.33% 25 12.50% 7 8.75% 9.86%
U U 0 0.00% 0 0.00% 0 0.00% 0.00%

Table 71: Phase 3 - Model for GCSE Computer Science (2016 specification) grade boundaries
The aforementioned model theorises that in the GCSE Computer Science population, a Grade F (Low
Grade 2), is approximately 16.33%.

241
 CHAPTER SIX

To test whether the average %Pre-Test result would attain Grade F, a one sample T-Test was carried out.
With Reference to Table 72, the one sample T-Test was carried out with and without outliers.

One-Sample Test (%Pre-Test)

Test Value = 16.33

95% Confidence Interval of

the Difference
Sig. (2- Mean
t df tailed) Difference Lower Upper

%Pre-Test
-0.25 47 .81 -0.40 -3.71 2.90
N=48

%Pre-Test

(No Outlier, -1.52 45 .14 -1.95 -4.54 0.63

N=46)
Table 72: Phase 3 - One sample T-Test of %Pre-Test towards Grade F threshold

Furthermore, with Reference to Table 73, the one sample T-Test was also carried out using the Bootstrap
option for 1000 samples, with and without outliers.

Bootstrap for One-Sample Test (%Pre-Test)

Bootstrapa

95% Confidence Interval

Mean Sig. (2-
Difference Bias Std. Error tailed) Lower Upper

%Pre-Test
-0.40 0.12 1.65 .81 -3.36 3.16
N=48

%Pre-Test

(No Outlier, -1.95 0.01 1.28 .15 -4.46 0.52

N=46)

a. Bootstrap results are based on 1000 bootstrap samples

Table 73: Phase 3 - One sample T-Test of %Pre-Test towards Grade F threshold using Bootstrap
Since all p > .05, we cannot reject the null hypothesis that the %Pre-Test mean is equal to the hypothesised
Grade F target value.

The aforementioned Grade Boundary model also theorises that in the GCSE Computer Science population
a Grade C (Grade 4 – Standard Pass) is approximately 37.35%.

To test whether the average %Post-Test result would attain Grade C, a one sample T-Test was carried out.
With reference to Table 74, the one sample T-Test was carried out with and without outliers.

242
 CHAPTER SIX

One-Sample Test (%Post-Test)

Test Value = 37.35

95% Confidence Interval of the

Difference
Sig. (2- Mean
t df tailed) Difference Lower Upper

%Post-Test
-1.11 47 .28 -2.20 -6.21 1.81
N=48

%Post-Test

(No Outlier, -1.86 46 .07 -3.24 -6.74 0.26

N=47)
Table 74: Phase 3 - One sample T-Test of %Post-Test towards Grade C threshold

Furthermore, with Reference to Table 75, the one sample T-Test was also carried out using the Bootstrap
option for 1000 samples, with and without outliers.

Bootstrap for One-Sample Test (%Post-Test)

Bootstrapa

95% Confidence Interval

Mean Sig. (2-
Difference Bias Std. Error tailed) Lower Upper

%Post-Test -2.20 0.05 2.01 .28 -5.83 1.96

%Post-Test

(No Outlier, -3.24 0.06 1.77 .08 -6.70 0.42

N=47)

a. Bootstrap results are based on 1000 bootstrap samples

Table 75: Phase 3 - One sample T-Test of %Post-Test towards Grade C threshold using Bootstrap

Since all p > .05, we cannot reject the null hypothesis that the %Post-Test mean is equal to the
hypothesised Grade C target value.

6.6.7.6 Correlation Analysis

In accordance with the correlation analysis approach outlined in section 3.6.5.2, correlation results are
reported in two tiers that reflect the strength of the result:

1. Tier 1: A positive or negative correlation effect of .1 or more, and p < .05.

2. Tier 2: A positive or negative correlation effect of .1 or more, and Bootstrapping Confidence
Interval (CI) does not cross over the 0 threshold. (i.e. CI is between -.09 and 1 or .09 and -1)

6.6.7.6.1 Time Engaged with E-learning Software vs. %Post-Test, %Change and KS4 Predictions

Table 76, reflects the tiered correlation results for %Post-Test vs. Level-2 Time and All Levels combined.

243
 CHAPTER SIX

Tier 1 Tier 2

Level2 All Levels

Spearman's rho Time Time

%Post-Test Correlation Coefficient .32 .21

Sig. (2-tailed) .03 .14

N 48 48

Bootstrapc Bias -.00 -.00

Std. Error .13 .14

95% Confidence Lower .07 -.05

Interval
Upper .55 .49
Table 76: Phase 3 - Tiered correlation results for %Post-Test vs. learning times

Table 77, reflects the correlation result for %Change vs. Level-3 Time.

Tier 2

Spearman's rho Level3 Time

%Change Correlation Coefficient -.23

Sig. (2-tailed) .12

N 48

Bootstrapc Bias .01

Std. Error .15

95% Confidence Lower -.49

Interval
Upper .09
Table 77: Phase 3 - Tiered correlation results for %Change vs. learning times

Table 78, reflects the tiered correlation results for Key Stage 4 Prediction (KS4Prediction) vs. Level-1 Time,
Level-4 Time, and Level-3 and -4 combined time.

Tier 1

Level1 Level4 Level 3 and 4

Spearman's rho Time Time Time

KS4Prediction Correlation Coefficient -.32 .47 .38

Sig. (2-tailed) .03 .00 .01

N 48 48 48

Bootstrapc Bias .00 -.01 -.01

Std. Error .15 .13 .13

244
 CHAPTER SIX

Tier 1

Level1 Level4 Level 3 and 4

Spearman's rho Time Time Time

95% Confidence Lower -.59 .19 .10

Interval
Upper .00 .70 .61
Table 78: Phase 3 - Tiered correlation results for KS4Prediction vs. learning times

6.6.7.6.2 VARK Modalities vs. Time Engaged with e-learning software, %Post-Test and %Change

Table 79, reflects the correlation result for Aural modality vs. %Post-Test.

Tier 2

Spearman's rho %Post-Test

Aural Correlation Coefficient -.25

Sig. (2-tailed) .09

N 47

Bootstrap Bias .01

Std. Error .14

95% Confidence Interval Lower -.51

Upper .05
Table 79: Phase 3 - Tiered correlation results for aural modality vs. %Post-Test, Level-3 Time, and Level-4 Time

Table 80 reflects the correlation result for Kinaesthetic modality vs. Level-4 time.

Tier 2

Level4
Spearman's rho Time

Kinaesthetic Correlation Coefficient .22

Sig. (2-tailed) .14

N 47

Bootstrapc Bias -.00

Std. Error .15

95% Confidence Interval Lower -.07

Upper .50
Table 80: Phase 3 - Tiered correlation results for kinaesthetic modality vs. Learning Times

245
 CHAPTER SIX

6.6.7.6.3 IMMS Results vs. Time engaged with E-learning software, %Post-Test and %Change

Table 81 reflects the tiered correlation results for IMMS Relevance vs. Learning Times.

Tier 2

Level 3 and 4
Level2 Combined
Spearman's rho Time Time

Student Correlation Coefficient .32 .32

Relevance
Sig. (2-tailed) .08 .09

N 30 30

Bootstrapc Bias -.01 -.01

Std. Error .17 .19

95% Confidence Lower -.04 -.09

Interval
Upper .61 .67
Table 81: Phase 3 - Tiered correlation results for IMMS Relevance vs. Learning Times

Table 82 reflects the tiered correlation results for IMMS Satisfaction vs. Learning Times.

Tier 2

Level 3 and 4
Level3 Combined
Spearman's rho Time Time

Student Correlation Coefficient .32 .32

Satisfaction
Sig. (2-tailed) .08 .08

N 30 30

Bootstrapc Bias -.01 -.01

Std. Error .20 .20

95% Confidence Lower -.07 -.08

Interval
Upper .68 .69
Table 82: Phase 3 - Tiered correlation results for IMMS Satisfaction vs. Learning Times

246
 CHAPTER SIX

Table 83 reflects the tiered correlation results for IMMS Overall vs. Learning Times.

Tier 1 Tier 2

Level 3 and 4
Level2 Combined
Spearman's rho Time Time

Student IMMS Correlation Coefficient .37 .33

Overall
Sig. (2-tailed) .04 .08

N 30 30

Bootstrapc Bias -.01 -.01

Std. Error .17 .19

95% Lower -.01 -.07

Confidence
Upper .67 .66
Interval
Table 83: Phase 3 - Tiered correlation results for IMMS Overall vs. Learning Times

6.6.7.6.4 Survey Instrument Questions vs. %Post-Test and %Change

Table 84 reflects the correlation results for %Post-Test vs. the following survey instrument questions.

1. 4 A) All things considered, the e-learning software is easy to use.

2. 4 J) The e-learning software is reliable (i.e. does not contain bugs or errors).
3. 12 B) The use of different methods to represent the same learning content made me feel
overloaded.

Tier 2

Spearman's rho 4 A) 4 J) 12 B)

%Post-Test Correlation Coefficient .32 .32 -.33

Sig. (2-tailed) .08 .08 .11

N 30 30 25

Bootstrapc Bias -.01 -.01 .01

Std. Error .17 .19 .16

95% Confidence Lower -.04 -.09 -.60

Interval
Upper .61 .62 .05
Table 84: Phase 3 - Tier 2 correlation results for %Post-Test and the survey instrument

247
 CHAPTER SIX

Table 85 reflects the correlation result for %Change vs. the following survey instrument questions.

• 12 B) The use of different methods to represent the same learning content made me feel
overloaded.

Tier 1

Spearman's rho 12 B)

%Change Correlation Coefficient -.41

Sig. (2-tailed) .04

N 25

Bootstrapc Bias .01

Std. Error .18

95% Confidence Interval Lower -.69

Upper -.01
Table 85: Phase 3- Tier 1 correlation results for %Change and the survey instrument

As previously noted, the research findings from Phase 1 and Phase2-Cycle1 have already been published;
additionally, it is planned that by the middle of 2019, the full set of research findings will be published to
a wider academic audience.

248
 CHAPTER SIX

6.7 E-learning Evaluation Protocol Results

This section reports how Phase 1 identified the need for an e-learning evaluation protocol, the feedback
in Phase 2 from education experts and pilot usage, and a summary of the Phase 3 findings from the e-
learning evaluation workshop. The Phase 3 findings focus on:

1. School characteristics in relation to e-learning,

2. Teacher characteristics in relation to e-learning,
3. Teacher feedback on the e-learning evaluation protocol,
4. Guidelines on the typical duration for evaluation activities,
5. The reliability and validity of evaluation results, and
6. Suggested refinements to the detailed evaluation procedures.

6.7.1 Phase 1: Identifying the Need for an E-learning Evaluation Protocol

As discussed in section 5.3, in Phase 1 an initial E-Learning Evaluation rubric was constructed; this initial
rubric acted as a strawman to be deconstructed, leading to some important areas of consideration:

1. The evaluation of Learning Objective Coverage is open to subjective interpretation and needs to
be more rigorously defined.
2. This rubric considers that all learning objectives are equal, whereas in a particular educational
setting, different learning objectives may have varying levels of importance.
3. This rubric counts the application of each heuristic, but does not measure the quality of
implementation.
4. This approach implies greater heuristic coverage is better, but this may not be the case for all
heuristics. For example, should we expect to “Use Social Interaction as a basis for learning” in all
screens. Logically, it is more realistic that each heuristic must be evaluated within the context of
the specific educational setting.
5. The process of analysing each screens for heuristic coverage is a very valuable tool during e-
learning software design or evaluation in identifying weak areas and ways to improve the e-
learning software.

Despite these areas of consideration, the Phase 1 e-learning prototype was evaluated as follows33:

• Based on the seven learning objectives defined, the Learning Objective Coverage is 75.71%.
• Pedagogical Heuristics Coverage is 35.60%
• The weighted pedagogical coverage (75.71% * 35.60%) is 26.96%.

A weighted pedagogical coverage of 27% (rounded) can potentially be perceived as low; however, this
relates to two points:

33
For further details, please refer to Vol 3 Appendix Q for the calculation breakdown

249
 CHAPTER SIX

1. Some heuristics were genuinely underutilised in the Phase 1 e-learning software prototype; this
was remedied in Phase 2.
2. Consideration of point 4 above has a significant impact on the pedagogical coverage; when the
educational setting is considered in an evaluation it has a significant impact on whether a learning
objective or heuristic is relevant within the setting, and if relevant, what its relative importance
is.

6.7.2 Phase2-Cycle2: Piloting the E-learning Evaluation Protocol

Feedback on the Phase2-Cycle2 e-learning evaluation protocol was provided by education experts, and
by the researcher piloting the protocol for the evaluation of Level-1 and -2 of the prototype software.

6.7.2.1 Education Experts

The review of the Phase2-Cycle2 e-learning software evaluation protocol was given by three education
experts; this feedback was based on the document: E-Learning Software Evaluation Protocol v0.1b. In
Phase2-Cycle2, the feedback did not structurally affect the evaluation procedure, but was significant in
defending decisions and approaches used in the protocol, and in documenting the protocol with enough
detail and clarity for effective usage. The main findings from the education expert review were:

1. Greater clarity and detail need to be given on how the e-learning evaluation protocol works in
practice.
• Use diagrams to support the discussion and help the audience to visualise the evaluation
steps, and assess the composition of educational value in terms of content quality and
pedagogical quality.
2. The evaluation protocol seems to document an “onerous activity”. Is there a way to make this
more efficient and feasible for the busy teacher audience?
3. Greater emphasis needs to be placed on whether all heuristics are equal; specifically, how
evaluators judge whether a heuristic is applicable to an educational setting and its level of
importance within that setting.
4. Greater clarity and detail need to be given on what the quantitative results really represent.
5. Review and document in more detail the process of how the evaluators might secure knowledge
of the heuristics, and the evaluation protocol, before the evaluation activity is undertaken.
6. Discuss in more detail the roles and responsibilities of evaluations, and the experience required
of teachers or instructional designers to qualify as expert evaluators.
7. Explain and defend the use of such a wide scale for the level of the support metric.
8. The explanatory text and title need to give greater clarity on what Section 4 is discussing.
9. Technical terms are used within the evaluation protocol; these should be defined.
10. Various grammar and phrasing changes are suggested within the evaluation protocol document.

250
 CHAPTER SIX

6.7.2.2 Pilot feedback

Piloting of the e-learning evaluation process confirmed that the overall structure of the protocol steps is
functional; additionally, it helped to explain and support the feedback from the education experts by
placing it in a more practical context. The main findings from the Phase2-Cycle2 evaluation pilot were:

1. Level-1 and -2 of the prototype software (total learning time of approximately 2.5 hours) were
evaluated and individual results recorded in 3 hours and 15 minutes. This is presumed to be an
underestimate for a typical evaluator since the researcher has an in-depth knowledge of the
software.
2. Sections 3.1, 3.2 and 3.3 of the evaluation protocol are unfocused; they need to be reviewed and
rewritten with more detail, clarity, and focus on the teacher audience. Recommendations from
education experts give some areas to address.
3. The composition of educational value in terms of content quality and pedagogical quality is not
discussed; what each measure, and how they interact, needs to be detailed in the evaluation
protocol.
4. On a theoretical level, Section 4 – Step 4 (to remove from consideration learning objectives and
heuristics that are not applicable to the particular learning context) is feasible before Step 5 (to
spend time to explore the e-learning software). However, in practical terms, it does not work
well and remains a speculative exercise. It is more effective to switch the order of these steps.
5. Section 4 – Point 3 does not currently discuss in what scenario the Educational Setting (Learning
Context) is given to the evaluators, and in what scenario they define and document it themselves.
6. A glossary is important to explain key terms to the teacher audience.
7. The importance scale is incorrectly reflected as ordinal; it does not reflect the interval number
scheme in parallel to scale titles.
8. The scale labels for the bi-directional level of support scale are inconsistent between
‘counteracts’ and ‘supports’ portions of the scale.
9. The excel object embedded in section 6 had an incorrect sequence of menu items for the bi-
directional level of support scale. This was causing incorrect calculations for the level of support.
10. The process of transferring quantitative results into the section 6 excel object is cumbersome,
requiring multiple iterations of scrolling backwards and forwards within the document; as a
workaround, the researcher transferred results into an intermediate text file to consolidate, then
used a dual screen setup to transfer the results into the excel object.
11. Reference to a doctorate degree on the title page is not relevant to the teacher audience and
can be removed.

251
 CHAPTER SIX

6.7.3 Phase 3: Usage of the E-learning Evaluation Protocol

Feedback on the Phase 3 e-learning evaluation protocol was provided by education experts and a
workshop used as a vehicle to study the usage of the evaluation protocol. The findings from the workshop
are quite lengthy and hence are summarised in this section.

6.7.3.1 Education Expert Feedback

The review of the Phase 3 E-Learning Software Evaluation Protocol was given by two education experts;
this feedback was based on the document: E-Learning Software Evaluation Protocol (P3_Group1) v0.3. In
Phase 3, the feedback on the e-learning evaluation protocol plateaued and was nominal; it is outlined
below:

1. Elaborate in more detail the description of novice, single expert and double expert classification.
2. Elaborate in more detail what pre-training would be given before an e-learning evaluation.
3. Briefly describe what actions should be taken in case the evaluation group cannot reach
consensus.
4. Describe the role of facilitator in terms of what is expected from him/her, their knowledge, and
who can act as a facilitator.
5. Make minor phrasing and grammar changes.
6. Add figure titles under four inline diagrams.

6.7.3.2 Phase 3: Workshop Findings on the E-learning Evaluation Protocol

The workshop research findings are based on nine computer science / ICT teachers from five private
schools and educational institutions. The findings are summarised in Table 86, but are then elaborated
with richer qualitative description and quantitative results in Vol 2 Appendix C.

Phase 3 – Workshop Findings on E-learning Evaluation Protocol

Usage of the E-learning Evaluation Protocol

9 teachers in 2 groups

[±] Both focus groups indicate their schools’ limited experience, limited use of e-learning software, and lack of
school-wide policy.

[±] Majority of teachers in both focus groups have limited or no personal experience of integrating e-learning
software into their teaching.

[±] Majority of teachers expressed that they ‘always’ or ‘very often’ use the internet and the web to support their
teaching.

[+] Both focus groups recognise the benefits of integrating e-learning software into their teaching.

[+] Both focus groups expressed that e-learning software can have positive educational and motivational impact.

[-] Participating teachers acknowledged that there are barriers that discourage them from using e-learning
software in their teaching, such as:

[-] Perceived quality, usability, and syllabus coverage of the software.

252
 CHAPTER SIX

[+] It is these perceived issues that the proposed evaluation protocol addresses.

[-] Socio-political and technology barriers to e-learning adoption.

[±] Participating teachers have limited or no experience in selecting or evaluating e-learning software.

[+] If called upon by their schools, teachers would get involved in an e-learning evaluation.

[+] Both focus groups expressed that they could use the heuristics and the evaluation protocol to evaluate e-
learning software.

[+] Minimal teacher feedback on how evaluation protocol could be improved.

[-] More time allocated to the hands-on evaluation activity.

[±] The heuristic evaluation criteria must be available throughout the evaluation process.

[±] Timeline analysis of the evaluation activities for a 1-hour e-learning software with six learning objectives
estimates that an end-to-end evaluation would take between 5 hr 45 min and 9 hr 30 min.

[+] Validity of the heuristic evaluation results was demonstrated.

[+] Accuracy of the feedback from both groups relating to the known characteristics of the Level-3 prototype
software.

[+] Validity and reliability further supported by the comparison of Group-1 and Group-2 heuristic evaluation
results.

[-] Validity and reliability of the learning objective evaluation results is open to doubt.

[-] The response by Group-2 on three of the six learning objectives was significantly different to Group-
1. These responses from Group-2 show indicators of being atypical or outlier responses.

[±] Postulated that atypical responses were brought about by:

a. Group-2 having an incorrect understanding of the educational setting for the evaluation.
b. Dominant members of Group-2 biasing overall team response.
c. Lack of facilitator mediation to correct the above points.

[±] Evaluation protocol judged to be broadly effective, with some refinements:

[+] Shorter group evaluation report.

[+] Improved facilitator guidelines.

[+] Walkthrough of the education setting should be repeated at the beginning of the debrief session.

[+] Heuristic and evaluation criteria should be accessible at all times and visible during the debrief
session.

Table 86: Workshop findings on e-learning evaluation protocol

6.8 Chapter Summary

This chapter has focused on the results from all three phases and all research elements within the study.
The mixed methods exploratory design, and specifically the qualitative priority in the first two phases, led
to a large volume of research findings. Due to this large volume, many results are presented in an abridged
form in this chapter, but reference appendices where the exploratory findings are presented in richer
detail and description. This chapter and this summary are structured accordingly:

1. The Phase 1 pilot findings,

2. The teacher and education expert feedback on the e-learning pedagogy,
3. Findings from student usage of the e-learning software prototype during Phase 1 and 2,

253
 CHAPTER SIX

4. Phase 3 results,
5. E-learning evaluation protocol results.

The Phase 1 Pilot Findings - Phase 1 acted as a pilot study to set a stable foundation for the research, and
identified the need for several changes in the research direction, including:

1. Preliminary positive feedback and direction on the research methods and experiment protocol
for uptake in the subsequent phases.
2. The refocusing of the research away from two separate pedagogical strategies (one for computer
science teaching and one for computer science e-learning software development) towards a
single e-learning pedagogy for design.
3. Inclusion within the pedagogy of collaborative and social learning and hence the inclusion of
heuristics related to collaborative learning environments.
4. The realisation that the development of quality e-learning software is a complex task and,
considering the teacher audience, a more appropriate focus for the pedagogy is the definition of
a set of heuristics that guide teachers (and instructional designers) to design e-learning software
or evaluate existing software for inclusion in their teaching.

Teacher and education expert feedback on the e-learning pedagogy - The teacher and education expert
evaluation and input into the e-learning pedagogy spanned all three phases, and included supplementary
teacher feedback from the e-learning workshop in Phase 3. Ultimately, in Phase 3 the e-learning pedagogy
was judged to be:

1. Appropriate for school pupils aged between 15 and 18-years old (Key Stage 4 and 5);
2. Appropriate to computer science and potentially other STEM subjects;
3. Offering comprehensive and balanced pedagogical coverage;
4. Appropriate for the teacher audience34; and
5. Feasible to be implemented in a high-school environment35.

Findings from student usage of the e-learning software prototype during Phase 1 and 2 - The findings in
this section are exploratory in nature, having been based on student usage of an iteratively evolving
version of the e-learning software prototype; these findings were later triangulated with survey and focus
group results, and grouped thematically:

1. Usability and Navigation – Broadly positive student response on usability and navigation, which
gave feedback, direction and support to heuristics: 17, 19, 21 and 21.1.

34
Although there remained some concern from one education expert that the relative time demands placed on
teachers means that although the pedagogy is a valuable and rich resource, its relative length and potential
complexity to implement may hinder it being put into practice.
35
The pedagogy provides guidance on heuristics that may prove more challenging to be implemented in schools.
However, there was some residual concern that the pedagogy does not explicitly consider the “alien environment”
that can be encountered in some schools.

254
 CHAPTER SIX

2. Learning Styles and Multi-modal Learning - Positive student response, which gave feedback,
direction and support to heuristics 16, 16.1, 16.2, 16.3 and 16.4.
3. Learning Object Ranking - The learning object ranking between the two phases are not
consistent. However, show some preference towards visual and active learning object types and
consistent negative ranking for audio and collaborative activities. These findings gave feedback,
direction and support to heuristics 4, 4.1, 6.2, 7, 9, 11, 16.1, 16.2, 17, 17.1 and 17.2.
4. Understanding and the Zone of Proximal Development - Positive student feedback that
educational material was represented in a clear and understandable way and that there was no
need to supplement the e-learning software with further textbook reading. In Phase2-Cycle2,
students reported that educational material was at the right level, and assessment activities were
at the right level or a little difficult. These findings gave feedback, direction and support to
heuristics 1, 4.1, 7, 7.1, 8, 9, 10, 16, 16.1, 16.2, 16.3, 16.4, 17, 18, 19, 20 and 21.1.
5. Visuals - Positive student response that visual elements are important, and judged to be
meaningful, engaging and support their understanding of the subject matter. However, there is
a comparatively weaker student response on the aesthetic appeal of the visual elements. These
findings gave feedback, direction and support to heuristics 16.1, 17, and 17.1.
6. Text – In spite of initial issues in Phase 1, overall, student response shows recognition that text
material is important in supporting their understanding of subject matter. These findings gave
feedback, direction and support to heuristics 16.3, 17, 17.1, 20.
7. Activities and Active Learning – Positive student response that the activity-based (interactive)
components are engaging and support student understanding of subject matter. Overall, the
students responded that the e-learning software prepared them for the assessment activities,
and that the assessment activities encouraged them to think rather than recall from memory.
These findings gave feedback, direction and support to heuristics 1,2, 4, 4.1, 5, 6.2, 7, 7.1, 14.3,
14.4, 15.1, and 16.4.
8. Video - In spite of the drop in ranking between Phase 1 and Phase 2-Cycle1, videos remain an
important part of the e-learning software, and are a valuable tool in the support of visual modal
preference (heuristic 16.1) and visualisation approaches.
9. Computational Thinking - Positive student response that the e-learning software helped them
to understand what computational thinking is, but positive response progressively weakens
between recognising computational thinking concepts and being able to use them. These
findings gave feedback, direction and support to heuristics 6, 6.1, and 6.2.
10. Collaborative Learning - Collaborative learning was consistently ranked lowly by students,
collaborative activities were underutilised by students, and received a mixed response on
whether it supported their understanding of the subject matter. However, students understand
at a conceptual level the reasons and value of collaborative activities, and the focus group
responses were positive towards collaborative learning. It is theorised that the research and
experiment design was not conducive to evaluating collaborative learning since it was primarily

255
 CHAPTER SIX

in class, for a short period of time, and voluntary. These findings gave feedback and direction to
heuristics 9, 11, 11.1, and 12.
11. Audio - Audio material was consistently ranked low by students, but its retention was requested
on condition that the audio quality is good, a mute button is available, and that the e-learning
software remains pedagogically effective even when audio is disabled. These findings gave
feedback, direction and support to heuristics 16.2, 17, 17.1, 17.2, and 20.
12. Authentic Learning vs Avoiding Overload - There is pedagogical tension between the following
- heuristics: 1: Use authentic educational material, examples and activities, and 18: Avoid adding
learning content that does not directly support your instructional goal. The student preference
was towards heuristic 1, and metacognitive decision-making was supported by the inclusion of a
red “i” to indicate additional information that is non-examinable.
13. Connectivism and the Web - Findings support the moderate connectivist approach
recommended in the pedagogy. Student showed understanding that a critical appraisal of the
information found on the web is necessary; accordingly, they showed a weak preference towards
the e-learning software since it is perceived to be accurate, comprehensive and reduces wasted
time searching the web. By design, the e-learning software acts as a hub that links to other
learning resources. These findings gave feedback, direction and support to heuristics 1.1, 11, and
12.
14. Motivation – Students responded positively regarding whether the e-learning software is more
interesting than textbooks, that it can be used for independent study, and it has increased their
overall enthusiasm and interest in computing. Overall IMMS results are consistent across Phase
1 and 2, at approximately 3.67 from 5. These findings gave feedback, direction and support to
heuristics 1, 7, 13, 14, 14.1, 14.2, 14.3, and 14.4.
15. Gamification - The intrinsic aspects of gamification garnered a positive student response. The
student response towards the extrinsic aspects of gamification is mixed with some trends
towards the negative. However, it is theorised that limitations in the LMS implementation of
gamification is a partial root cause. These findings gave feedback, direction and support to
heuristics 15, 15.1, and 15.2.
16. Recreating Books - Students requested a more explicit structure in the e-learning software that
mimics a traditional chapter format. In response, the e-learning software was divided into levels
with learning objectives, learning material and review questions. These findings gave feedback,
direction and support to heuristics 19, 21, and 21.1.

Phase 3 results - Phase 3 used a convergent parallel mixed methods design with a quantitative priority,
and reported primarily on student perception and learning performance, based on hands-on usage of the
e-learning software in class and for homework.

• Student Perception of E-learning Software - Student responses from the survey instrument
(including VARK and IMMS instruments) are reported in summarised format in Table 46. Please
refer there for a recap.

256
 CHAPTER SIX

• Student Learning Performance - The average student learning performance and engagement
with the e-learning software and CLE are reported in summarised format in Table 47. Please refer
there for a recap.

E-learning Evaluation Protocol Results

1. Phase 1 Pilot - The e-learning design activity identified the need for an e-learning evaluation protocol
to effectively assess the specific pedagogical coverage of e-learning software. This was then
incorporated as a second research objective of this study.
2. Phase2-Cycle2 - During Phase2-Cycle2, the e-learning evaluation protocol was reviewed by three
education experts, who gave constructive feedback. This feedback did not structurally affect the
evaluation procedure but was significant in defending decisions and approaches used in the protocol,
and in documenting the protocol with enough detail and clarity for effective usage. Additionally, the
e-learning evaluation protocol was piloted by the researcher. Piloting of the e-learning evaluation
process confirmed that the overall structure of the protocol steps is functional; additionally, it helped
to explain and support the feedback from the education experts by placing it into a more practical
context.
3. Phase 3 - Feedback on the Phase 3 e-learning evaluation protocol was provided by two education
experts and a workshop, used as a vehicle to study the usage of the evaluation protocol. In Phase 3
the feedback from the education experts plateaued and was nominal. The workshop research findings
are broadly positive, with some direction for future work, and are already reported in summarised
format in Table 86. Please refer there for a recap

257
 CHAPTER SEVEN

7 DISCUSSION
7.1 Chapter Brief
This chapter discusses and interprets the results from the three phases and the underlying research
elements within this study. These elements are pulled together and interpreted in a holistic and cohesive
discussion that briefly recaps the underlying problem domain, and then focuses on:

1. Providing a Solution and Research Contribution

a. Proposing e-learning software as one part of the solution;
b. Framing the status of e-learning software and the limitations of existing e-learning software
in terms of their pedagogical foundations;
c. Identifying a lack of research into high-school computing pedagogy;
d. Identifying a lack of research into e-learning heuristics focused on pedagogy; and
e. Identifying a lack of research into e-learning evaluation protocols focused on pedagogy.
2. Discussing the Results of the Study
a. Discussing the theoretical evaluation of the e-learning pedagogy by education experts and
teachers;
b. Discussing the student usage of the e-learning prototype and how it informed the e-learning
pedagogy; and
c. Discussing the impact that student usage of the e-learning prototype had on learning
performance.

7.2 Globalisation, Digital Ubiquity and the Need for a Strong IT

Sector
This study reasserts the ubiquitous nature of digital technology that is driving what is predicted to be the
fourth industrial revolution, in which a holistic ICT ecosystem will be tied into every aspect of our lives.
Furthermore, digital technology fuelling globalisation has led many countries to take on national
imperatives to build strong IT sectors that can drive their economies within the new global order. In the
UK, the growing unmet demand for IT professionals focused a dissatisfied political spotlight on computing
education in schools, which in turn led to far reaching reform.

The issue of whether these assertions were entirely valid: the exact role and importance of high-school
computing education for the nation’s supply of IT professionals; the steps taken by the Government, the
Department for Education (DfE), and other influential stakeholders in bringing computing education out
of crisis; and whether those steps have been effective, are not judged in this research. But they do serve
as a backdrop for the research, and motivate this research’s support for computing education and
computer science teachers through the difficult transition from ICT to computer science. What is clear is

258
 CHAPTER SEVEN

that despite initiatives to redress the situation, in the past five years (up to the end of 2017), the strain
placed on the high-school computing teachers has become more pronounced.

7.3 E-learning Software is Part of the Solution

This research does not contend that e-learning software is a solution for all the perceived challenges in
high-school computing education in the UK. However, quality e-learning software can be used as an
important tool to support teachers (especially those less confident with the computer science portion of
the curriculum) with subject-specific material that is pedagogically sound, and can be used to inform their
teaching methods. Furthermore, whilst the e-learning pedagogy is not intended to instruct teachers in
good teaching practice, many of the teachers involved in this research have explicitly commented that it
has positively informed their teaching.

Investment in, and the growth of, e-learning usage has been widely reported as one of the most rapidly
expanding areas of education and training in industry, and to a good extent in schools. Several benefits
are reported in terms of learning performance and motivation, serving both academically strong and
weaker students. However, the basis of this research acknowledges that e-learning software is not a ‘silver
bullet’36; there remains caution on the strength of the evidence regarding e-learning benefits, and concern
still persists that what is delivered in terms of e-learning software can fall short, especially in terms of
pedagogical usability and alignment with educational needs. It is this concern that lies at the core of this
research; e-learning software can be a beneficial tool to teachers, but only if the content and pedagogical
quality of the software is safeguarded. This can be achieved by defining a comprehensive set of
pedagogical heuristics to guide teachers and instructional designers in designing and/or evaluating e-
learning software for use in teaching. Additionally, teachers and other education stakeholders can be
supported in selecting appropriate e-learning software via a rigorous e-learning evaluation protocol.

This research recognises that there are several factors that contribute towards successful e-learning, such
as: usability; underlying infrastructure in schools and student homes; teacher readiness; level of student
e-maturity; and level of school, vendor and parental support, etc. However, a targeted research focus is
purposely given to an area of notable concern that is under-researched and has high impact: the
underlying pedagogy of e-learning software.

There are a variety of texts on e-learning instructional design; in contrast, this research supplements
existing literature by providing an accessible set of heuristics for e-learning pedagogy that focuses on, but
is not restricted only to, high-school computer science. These heuristics are designed to communicate the
essence of a wide body of pedagogical knowledge in an understandable and usable form to a teacher
audience. The heuristics are proposed due to the persisting concerns that were previously outlined
regarding e-learning pedagogical quality, and are necessary due to the scarcity of e-learning heuristics
that give adequate focus to pedagogy. Additionally, where research has concentrated on e-learning

36
A simple and seemingly magical solution to a complicated problem.

259
 CHAPTER SEVEN

pedagogy it has focused on specific pedagogical aspects of e-learning, rather than a holistic pedagogy for
e-learning. In contrast, this pedagogy considers the three dimensions of content, incentive and
environment to ensure comprehensive coverage of how learners’: interact with their external
environment to stimulate learning; build meaning and understanding; and direct the requisite mental
energy to support learning. At the time of writing (January 2018), this study had not identified an
equivalent holistic set of heuristics focused on e-learning software for high-school computing.

As common practice, the defined pedagogical heuristics are dual purpose: to guide the pedagogical design
of e-learning software, and to serve as the basis of a protocol to guide the evaluation of such software.
Similar to e-learning heuristics, e-learning evaluation protocols have a historical bias that continues today,
predisposed towards usability to the neglect of pedagogy. Whilst numerous researchers expound the
need to assess the educational quality of e-learning systems, few have a concerted and detailed focus on
the evaluation of pedagogy. Hence, this study attempts to redress this gap by having a detailed focus on
pedagogical heuristics, and the evaluation of e-learning pedagogy and content quality. This focus is by
design, but does not ignore the fact that there are other notable factors that contribute towards
successful e-learning that must also be evaluated. Most notable amongst these factors is the evaluation
of usability. It is therefore envisaged that the evaluation of pedagogy and content quality may not be the
only evaluation carried out, but is a majority contributor towards the results of a wider set of evaluations.

This study has synthesised a robust protocol for evaluation steps based on existing best practice in
evaluation. In addition, it is novel (and arguably unique) in that it is characterised by: support for both
formative and summative evaluations; considers the educational setting for software usage; is defined at
a detailed level; focuses on pedagogical and content quality (learning objectives) rather than usability
issues; provides quantifiable results that can be used for comparative evaluations; and focuses on the
reliability and validity of evaluation results. Considering these characteristics, the proposed e-learning
evaluation protocol would have added significant value in recent evaluations of high-school computer
science e-learning resources, previously discussed in section 2.3.3 (i.e. New Zealand, Morocco and
Australia). These evaluations were coloured by limitations, such as: neglecting pedagogical evaluation
entirely; giving limited coverage of pedagogical principles; giving limited detail in the evaluation criteria,
thereby leaving them open to interpretation; coarsely defined quantitative measurement and results; and
a lack of an encompassing protocol to reliably guide the evaluation. These recent studies reflect a need
for the evaluation of digital education resources; but also, the inherent limitations of each of the
evaluations reflect the need for a rigorously defined evaluation protocol focused on pedagogy.
Considering, the UK’s (Waite 2017) recent call to evaluate the pedagogical foundation of their high-school
digital education resources, this need is likely to continue.

260
 CHAPTER SEVEN

7.4 Contribution to a Small Body of Research and Establishing

Research Rigour
This study offers a contribution to the small body of research into high-school computing pedagogy. The
literature classification from Sentance and Selby (2015) of computing education research between 2005
and 2014 identified 2,225 papers, of which only 66 were focused on school age students or teachers, and
pedagogy. Supporting findings are also identified by the recent pedagogy literature review (Waite 2017)
commissioned by the Royal Society , which concentrates on research in the past ten years and initially
identified over 700 papers, which were eventually reduced to 86 papers focused on computing pedagogy
relevant to schools.

Further analysis by this study of a version of Sentance and Selby’s literature classification, gives further
evidence of this under-researched area (reported in Table 2), examples include:

1. Only two papers had a major theme of pedagogy for computing education in schools and a minor
theme of e-learning.
2. Of the 58 papers related to pedagogy for computing education in schools, none gave a holistic
view of pedagogy. Instead, their focus was on narrow aspects of pedagogy.
3. Of the 89 papers with a major theme of e-learning, one paper focused on pedagogical patterns,
and one focused on pedagogy with e-learning; however, the latter focused specifically on
pedagogy for virtual classrooms.
4. Four papers had a major theme of e-learning and a minor theme of school age students or
teachers, all of which focused on specific components of e-learning (i.e. learning objectives, e-
portfolios, algorithms, and outreach).
5. Four papers had a major theme of e-learning and a minor theme of pedagogy, none of which
took a holistic view towards pedagogy for e-learning.

These findings are also confirmed by the literature review from this study, which did not identify
comparable research to define a holist set of pedagogical heuristics for e-learning. The closest conceptual
alignment with the objectives of this study is Dabbagh (2005) who offers a theory-based framework for
e-learning design, but this framework is not specified in detail and is limited to a theoretical perspective.

The literature review from this study and the analysis of recent well-reputed literature reviews give a
strong indication that research into a comprehensive and holistic set of pedagogical heuristics for
teaching, or the design of e-learning software for high-school computer science, is currently a severely
under-researched area. This is also echoed in the calls by the Royal Society for research in high-school
computing pedagogy.

This study also contributes a measure of research rigour to the small body of research into high-school
computing pedagogy. Condie and Munro (2007), Means et al. (2009), Livingstone (2012), and Kim et al.
(2014) take a somewhat critical view of the evidence of improved pupil attainment through e-
learning/ICT; they contend that much of the evidence is taken from small-scale studies lacking the rigour

261
 CHAPTER SEVEN

and scale by which generalisations can be drawn. More recently, Sentance and Selby (2015), Waite (2017),
and Crick (2017) reiterate the same concerns, that most of the existing research is not conducted in a
classroom context, and lacks methodological rigour relating to sampling, sample size, statistical
significance and claimed impacts.

In contrast, the mixed methods research design used in this study offers a highly rigorous research design
that has involved five education experts, six teachers (excluding the 13 teachers who joined the Phase 3
workshop), and 76 students in total.

The phased mixed methods design and the use of action research ensured that what was theorised was
assessed in a realistic school context. In addition, the research design inherently supported the iterative
development of the research and multiple points of triangulation to validate research findings. The
approaches towards trustworthiness (credibility, transferability, dependability, and confirmability), and
internal validity, external validity (generalisable), reliability and objectivity are discussed at length in
Chapter Three; as are the validity and reliability of the instruments used in the study. Findings are
presented regards the reliability of the IMMS results and the reliability (alternate form and internal
consistency) of the survey instrument. Quantitative results for Phase 3 are presented with parametric
tests and, where appropriate, re-confirmed using bootstrapping and non-parametric tests.

Most importantly, the qualitative findings for the exploratory phases (Phase 1 and 2) are confirmed and
generalised by Phase 3, using a larger student sample.

7.5 International Contribution

This study primarily focuses on UK high-school computing education. However, the challenges faced in
the UK are by no means unique, and the transition of high-school curriculums to a more computer science
focus has been undertaken by several countries. Section 2.2.6 reflects that the USA, Israel, Germany, UK,
Austria, Slovakia, Switzerland, Italy, Lithuania, and Poland are active in varying degrees in this research
area. Also, that the US, New Zealand, Australia, Israel, Germany, France, Italy, Sweden and India are all in
varying stage of similar initiatives to give prominence to their high-school computer science curriculums.
Arguably, countries such as the US, Israel, Russia, Australia and New Zealand have made significant
progress in establishing their high-school computer science curriculums. The computer science
curriculums in the aforementioned countries may have some underlying differences, but it follows that
the objectives of this research are broadly transferable to the countries listed in this section, and more.
This is directly exemplified by clear patterns of usage of technical teaching resources for the computing
curriculums in the US, Australia, New Zealand and Morocco.

7.6 Synthesising an E-learning Pedagogy

The pilot phase of this study served its purpose well in giving a more realistic and directed focus to the
research. The outset of this research focused on the delivery of two pedagogies: one for computer science
teaching and one for computer science e-learning software development. In retrospect, there was an

262
 CHAPTER SEVEN

element of naivety in these objectives that the pilot quickly remedied. Considering the idiosyncrasies and
challenges faced by teachers every day, there is a credibility gap for an academic with limited teaching
experience to propose a pedagogy for teachers. However, with that in mind, the teacher evaluations of
the e-learning pedagogy did not reflect this point; in fact, they were very encouraging and reported that
the pedagogy was used to inform their own teaching practice.

“Reading through the report provided, I find myself inadvertently using a number of the heuristics. All of
the heuristics seem to cover every single aspect of a teaching method.” [Teacher 2]

“Extremely clear and very well structured. As I have mentioned earlier very detailed description and I have
already started to evaluate my teaching… Truly exceptional research and since I am currently exploring
more about computational thinking I personally found this as a very useful addition to my other books and
documents on this subject.” [Teacher 3]

“Overall, this pedagogy document is great learning for me and I have learnt a lot by just reading it. I will
be sharing this with my team (if you agree) and will be using your research in my teaching practice.”
[Teacher 4]

In addition, the pilot identified that a pedagogy for e-learning development targeted to a teacher
audience was not a realistic proposition. A more realistic objective was to concentrate on teachers’
existing experience in teaching and strengths in pedagogy, to support them in designing or evaluating e-
learning software.

The original focus of the research was purely on creating e-learning software; however, it became clear
that e-learning software is constrained when treated as an isolated entity, and that significant educational
value can potentially be gained by integrating the e-learning software within a CLE. Hence, the pedagogy
was extended to consider collaborative and social learning.

Finally, the pilot identified the need for an e-learning evaluation protocol; hence a secondary research
objective was added to the study. Overall, the pilot was instrumental in focusing the study towards
pedagogical heuristics for the design or evaluation of e-learning software (including CLE), and an
associated evaluation protocol. Both artefacts remain targeted to a teacher or instructional designer
audience.

The resulting pedagogy aimed to be comprehensive and holistic in its coverage of learning; it rejects the
learning theory debate and focuses on commonality and areas of interaction between the chosen
theories. It synthesises an overwhelming body of knowledge in learning theories into a set of heuristics
that are intended to be succinct and accessible to teachers. The pedagogy is intended as a toolkit, which
is not reductionist; it does not prescribe when to use particular heuristics, this remains in the teacher’s
control and judgement. However, it does provide detailed information on design criteria, potential
benefits, how the heuristics interrelate, and potential challenges in implementation, all of which inform
the teacher’s judgement of how and when a heuristic should be used.

The iterative evaluation of the e-learning pedagogy by education experts and teachers supported the
development of a pedagogy that is characterised as:

263
 CHAPTER SEVEN

Appropriate for school pupils aged between 15 and 18 years old: Throughout the evaluation iterations,
the education experts and teachers confirmed the heuristics appropriateness for 15 to 18-year olds.
Furthermore, an analysis of the UK Department for Education’s curriculum for computing showed a strong
alignment between the heuristics and learning objectives for both Key Stage 4 and 5.

Appropriate to Computer Science: There is broad confirmation from education experts and teachers that
the heuristics are appropriate for computer science education, support underlying skills important to
computer science, and potentially can be used to support other STEM subjects.

“The heuristics are appropriate for Computer Science education (as well as other STEM subjects). Many of
the heuristics, especially the ones related to problem solving are important for fields such as computer
science where students are expected to develop skills in problem identification, analysis, solution design
etc. as well as evaluation skills drawing on supportive evidence. “[Education Expert 2]

“Having in mind my previous answer I would like to add that incorporating heuristics in Computer science
or more specifically any STEM type of subject could greatly benefit students, subjects that require or rely
on problem solving or an experience, and also being able to actually view something happening can
motivate and increase likelihood of a more memorable lesson.” [Teacher 2]

“I strongly agree Computer Science education focuses more on Problem based learning and prompt
reflective practice to support learning. It is very important to build foundation on Computational thinking
before using it.” [Teacher 3]

Balanced pedagogical coverage: There is broad confirmation from education experts and teachers that
the heuristics are comprehensive and offer a balance pedagogical coverage. In Phase 1, there was
cautionary comment on the sub-heuristic that static illustrations can be better than animations; this
heuristic was eventually removed. Overall, there was limited feedback to include additional heuristics; in
Phase 1 there was expert feedback to give more focus on engagement and gamification, which was
followed by the addition of relevant heuristics in Phase2-Cycle1. In Phase2-Cycle2 there was teacher
comment on the educational value of learning from mistakes. This was not covered in a discrete heuristic,
but already existed in the pedagogy as design criteria in other heuristics.

“There is a balanced pedagogical coverage with appropriate emphasis on overarching educational

theories and key concepts. These are well linked to strategies that guide teaching practice in computer
science.” [Education Expert 3]

Appropriate for teacher audience: A significant focus of the pedagogy development was in ensuring it is
appropriate and adds educational value to the primary audience: teachers. This involved: multiple
iterations of re-phrasing and re-sequencing of heuristic titles, descriptions and criteria; rationalising the
heuristics and underlying criteria to shorten the document to a manageable length; focusing on
appropriate terminology and including a glossary; including sections on educational benefits, potential
challenges and the interrelationship between heuristics; and adding visual representations of
interrelationships and potential educational benefits. By Phase2-Cycle2, this had led to a pedagogy

264
 CHAPTER SEVEN

document that was vastly improved and overall was well received; but the scale of the pedagogy and the
endeavour needed to fully utilise it still left some residual concern.

“This version of the pedagogy document is much better than the previous version, with good reading flow
and structured organisation. The way each pedagogy is presented (the structure of each section) is
beneficial to a novice reader, since they can get familiar with the concepts through the Description, look
at the Design Evaluation Criteria and also be informed of the Educational Benefits and Potential
Challenges.” [Education Expert2]

“Extremely clear and very well structured. As I have mentioned earlier very detailed description... Clear
references to the different heuristic technique made it easier for me to look at the different approaches…
I could clearly see education benefits of each heuristic approach has been clearly explained.” [Teacher3]

“Document is very well structured and has a clear structure and contents page. Research is very clear and
it has a strong connection with all the activities carried out during school activities.” [Teacher 4]

“This is certainly a valuable and rich resource for teachers, however the complexity and length might
prevent some to fully utilise and put into practice. E.g. a school would need a series of staff meetings to
discuss this document and many mentoring sessions will be required for teams of professionals to use.
Similarly, for designers this is a useful resource, yet it requires time and effort to be understood.”
[Education Expert 3]

A noteworthy point of triangulation occurred in the workshop; the evaluators (in-service teachers) were
asked to give high-level feedback on the heuristics, based on the training they had received in the
workshop (without access to the pedagogy document). One evaluator offered the following feedback on
the heuristics:

“There are a lot of them, and they sometimes compete or contradict each other. This is fine, but busy
teachers are not going to be able to remember the full set. Perhaps a condensed version, including the
most important/beneficial heuristics, or a flowchart that can be followed easily, would make them more
useful.” [Evaluator 1]

This feedback is particularly insightful since it was identified in previous research phases during the
teacher and expert evaluations of the e-learning pedagogy document, and was implemented in the
document by the addition of a heuristic summary table, an educational benefits matrix, a heuristic
relationship diagram and a detailed heuristic relationship matrix.

Feasible to be implemented in a high-school environment: An area of critical importance is the feasibility

of implementing the heuristics within a school environment. In Phase 1, Education Expert 1 introduced a
note of concern:

“The school and the classroom is a very "irrational" and alien place...but it is the environment we are
designing for.” [Education Expert 1]

The pedagogy is designed as a toolkit in which heuristics that can be selected based on the specific
learning material, intended audience and intended learning objectives. In this respect, the majority of the

265
 CHAPTER SEVEN

heuristics can be implemented in a high-school environment without difficulty. However, some of the
heuristics have a bit more challenge to be implemented in a high-school; these challenges are also
documented and addressed in the pedagogy. In this respect, there was broad confirmation from the
education experts of the pedagogy’s feasibility for high-schools.

“The heuristics do not take into consideration of the broader school curriculum, ethos, timetable and goals
because this is not part of this study. Overall, the heuristics as they stand could be implemented in a high
school setting.” [Education Expert 3]

A weaker, more neutral stance was taken by some teachers, which seems to be more reflective of the
challenges in schools rather than shortcomings in the heuristics. Teacher 2 gave a balanced response
reflecting that “any addition to the syllabus can be both good and bad”. However, that incorporating such
heuristics into high schools “can certainly influence students learning and teaching in a positive manner”.
The challenge is that despite the potential benefits, the transition may not necessarily be smooth. Teacher
3 offered some optimism, but again reverted to the reality in schools:
“I could evidence that many of the heuristics are already implemented in high school environment. All
schools focus on results oriented learning so even though they implement many of the suggested
heuristics. I think new learning happens only when students carry out projects on their own or practical
activities.

There is very little time provided for student content which deepens learning, and this is one of the key
areas being ignored when considering why progression of students taking computer science from KS4 to
KS5 is not 100%.” [Teacher3]

An interesting point is that only one teacher evaluated the pedagogy and also used the e-learning software
with his students. This teacher’s response on whether the pedagogy is feasible to be implemented in
schools is markedly different:

“Yes, as students have made massive progress and learnt a lot by following the e-learning software based
on pedagogy.” [Teacher 4]

The overriding conclusion is that when designing e-learning software or evaluating software for use in
schools, the heuristics that will be applied must also consider the reality of the school context in which
the e-learning software will be applied.

By Phase 3, there were indicators that the e-learning pedagogy had reached a mature state, this is
evidenced by the plateau in feedback, which was overwhelmingly positive. This was further supported by
the broad consensus in the teacher workshop that the heuristics were appropriate for: 15 to 18-year olds
(Key Stages 4 & 5); computer science education; and had comprehensive and balanced pedagogical
coverage37.

37
This is with the exception of one evaluator.

266
 CHAPTER SEVEN

7.7 Pedagogy Feedback as Evidenced by Usage of E-learning

Software
The theoretical synthesis and evaluation of the e-learning pedagogical heuristics was also informed by a
practical implementation in which students used e-learning software designed in adherence with the
pedagogy.

As previously recognised, usability is important in software, and a significant prerequisite in e-learning

software, since its absence can hinder the learning process. The e-learning pedagogy presupposes
usability in the software and does not provide usability heuristics; however, there are a number of
heuristics that touch upon usability. These are:

• 17: Integrate words and graphics together, instead of words alone.

• 19: Optimise essential processing by segmenting learning material and providing pre-training.
• 21: Provide restricted navigational control in the e-learning software.
• 21.1: Provide consistent navigational elements and signposts for learning.

Considering the students’ age, the pedagogy proposes restricted navigational control, segmenting of
learning, the provision of pre-training, and sign-posting of learning; this culminates in a relatively
structured and stable learning environment. However, this is balanced with the learners need to feel in
control of the actual learning experience. These aspects were well received in Phase 1 and 2, which
reported broadly positive findings on the ease of use, usability, and clarity of the e-learning software.
Although, there were some refinements to be made to the instructions, learning object sign-posts and the
structuring of the pre-training videos. Phase 3 offered confirmatory results in that almost all survey
instrument responses related to usability had a median student response of agree (4).

An important factor to consider is that Tier-2 medium effect correlations of students’ perception of
usability (Question 4A) and reliability (Question 4J) were linked to %Post-Test results. Causality is
theorised that students who perceive the e-learning software to be usable and reliable are free
(unhindered by usability issues) to focus on the learning material, and therefore have improved
performance.

One potential area of discontinuity between the exploratory phases and Phase 3 is that the reliability of
the e-learning software received comparatively weaker findings in Phase 1 and 2, but received a median
student response of agree (4) in Phase 3. However, a closer look at some of the underlying questions on
reliability in Phase 3 confirmed the findings in Phase 1 and 2 since the median student response showed
neither agreement nor disagreement (3) with three metrics focused on reliability. Additionally, in Phase
3 a small number of responses to open questions did comment on bugs in the software.

Heuristics related to learning styles and multi-modal learning are well-evidenced in all three phases of
the study:

• 16: Use multi-modal learning approaches

267
 CHAPTER SEVEN

• 16.1: Support visual modal preference.

• 16.2: Support aural modal preference.
• 16.3: Support read-write modal preference.
• 16.4: Support kinaesthetic modal preference.

Phase 1 and 2 show comparative balance between the students VARK modalities and positive student
feedback on the use of varying methods to represent the same educational concepts. Even the initial
misstep in Phase 1, of the researcher not managing his dominant read-write modality, and creating e-
learning software with a text overload, provided inverse confirmation for multi-modal learning. Phase 3
offers confirmatory results in that parametric and non-parametric tests show comparative balance
between each of the four modalities, with the exception of a statistically significant difference between
visual and kinaesthetic modalities. Phase 3 also offered positive student feedback with a median response
of agree (4) with the statement “The use of different methods to represent the same learning content
helped my understanding.” Finally, a further weakly positive result for multi-modal learning is provided
by the median student response of neither agree nor disagree (3) with the statement “The use of different
methods to represent the same learning content made me feel overloaded.”

The correlation analysis identified a fascinating VARK result that is challenging to interpret; a Tier-2 weak
effect negative correlation was identified between aural modality and %Post-Test. Causality could
potentially be that the audio implementation in the e-learning software was unsatisfactory and hence
negatively impacted the learning performance of those with higher aural modalities. A more far-reaching
potential cause is that those with a higher aural modality are not so well served by multi-modal learning,
suffering impact to their learning performance.

Managing cognitive load is a central tenet of the pedagogy; it is recognised that there is a tension between
the cognitive load placed on students and some of the heuristics with constructivist, multi-modal and
connectivist foundations. The importance of managing cognitive load is clearly shown in Phase 3, in the
correlation analysis which reflected a Tier-2 medium negative correlation between the students’ feeling
of being overloaded by multi-modal approaches and %Post-Test results. In addition, a Tier-1 (statistically
significant) medium negative correlation can also be seen with %Change. Broadly speaking, the students’
learning performance is negatively correlated with any perception of being overloaded by the e-learning
software (and vice versa). Although, it is reminded that the students’ median response reflected that they
did not feel overloaded.

The pedagogy cautions that the implementation of the following heuristics needs to be considered in light
of the additional cognitive load they place:

• 1: Use authentic educational material, examples and activities.

• 1.1: Ensure the currency of learning material.
• 3: Make expert and learner thinking processes explicit.
• 4: Use problem-based learning (PBL) to facilitate learning.

268
 CHAPTER SEVEN

• 5: Integrate learning into long-term memory by using authentic examples, and non-trivial
practice and problems.
• 6.2: Exemplify computational thinking in problem-solving activities.
• 12: Develop and nurture networks to support learning.
• 13: Use constructivist approaches to increase intrinsic motivation in the learner.
• 16: Use multi-modal learning approaches.

Furthermore, the following heuristics can be used to manage and reduce cognitive load:

• 4.1: Use worked examples to support problem-based learning.

• 6.1: Build a foundation for computational thinking.
• 8: Provide scaffolding to advance learning progress.
• 17: Integrate words and graphics together, instead of words alone.
• 17.1: Apply contiguity by aligning words (audio or screen text) with corresponding graphics.
• 18: Avoid adding learning content that does not directly support your instructional goal.
• 19: Optimise essential processing by segmenting learning material and providing pre-training.
• 20: Use a conversational style in screen text and audio narration.
• 21: Provide restricted navigational control in the e-learning software.
• 21.1: Provide consistent navigational elements and signposts for learning.

As discussed earlier in relation to usability, the relatively structured and stable learning environment
proposed in heuristics 17, 19, 21, and 21.1 was well received by students across all phases. In addition, it
also has a contributing benefit in helping to reduce cognitive load. Heuristics 19, 21, and 21.1 were also
studied and well received by students in Phase 2; with the students request for a more explicit structure
in the e-learning software, in particular in mimicking a traditional chapter format, with chapter learning
objectives, learning material, review questions, and finally a learning summary. Again, this approach has
contributed benefits towards reducing cognitive load. It should be noted though, this was not explicitly
measured in Phase 3, hence was not confirmed using statistical tests.

A key area in managing cognitive load is the tension between authentic learning (Heuristic 1) and avoiding
learning content that does not directly support the instructional goals (Heuristic 18). In Phase 2, students
gave positive feedback in preference of Heuristic 1: “Use authentic educational material, examples and
activities” and the inclusion of learning material that is not directly examinable within the curriculum,
thereby increasing the cognitive load placed on them. However, to support their meta-cognitive
processes, such content was flagged with a red “i” icon, to help in their decision on whether to review the
learning material. It should be noted that although well received by the students; there was some
question on whether the red “i” icon accurately communicated to students that the material was “extend
your knowledge” material.

Phase 3 offered confirmatory results in that the median student response was agree (4) that the extend
your knowledge material helped students to understand the subject matter. However, this result is
weakened to some extent in that “extend your knowledge” material was ranked last from the ten learning

269
 CHAPTER SEVEN

object types, both in terms of perceived benefit to learning and perceived influence on interest and
enthusiasm. Additional, confirmatory evidence is reflected in the comparatively high IMMS Relevance
metric, with a median response of 3.29, which is between moderately true and mostly true.

The pedagogy gives focus to the communication of learning material in a clear and understandable way
and support to the social constructivist principle of the Zone of Proximal Development, in which students
are challenged just beyond their individual unassisted abilities. Clarity and understanding are provided in
the following heuristics:

• 16: Use multi-modal learning approaches.

• 16.1: Support visual modal preference.
• 16.2: Support aural modal preference.
• 16.3: Support read-write modal preference.
• 16.4: Support kinaesthetic modal preference.
• 17: Integrate words and graphics together, instead of words alone.
• 18: Avoid adding learning content that does not directly support your instructional goal.
• 19: Optimise essential processing by segmenting learning material and providing pre-training.
• 20: Use a conversational style in screen text and audio narration.
• 21.1: Provide consistent navigational elements and signposts for learning.

In Phase 1 and 2, there was positive student feedback that the educational material was represented in a
clear and understandable way, and that overall there was no need to supplement the e-learning software
with further textbook reading. In Phase 3, this is confirmed with a median student response of agree (4)
that the e-learning software was represented in a clear and understandable way. But only weakly
corroborated with a median student response of neither agree nor disagree (3) on whether the students
supplemented, or needed to supplement, the learning material with further textbook reading or teacher
support. The Phase 3 results indicate that, despite the e-learning software being clear and
understandable, a significant number (almost half) of the students would require further support. This is
not interpreted as a negative result since the e-learning software is not proposed to replace teachers or
to be used as an isolated learning resource. The pedagogy in keeping with connectivist and social learning
principles proposes that e-learning software should link to other learning nodes, and a learning
community in which the students can gain further support and scaffolding.

Challenging students in accordance with the ZPD is provided in the following heuristics:

• 1: Use authentic educational material, examples and activities.

• 4.1: Use worked examples to support problem-based learning.
• 7: Distribute well-designed practice activities across the lesson to support learning.
• 7.1: Provide explanatory feedback to practice activities to promote learning.
• 8: Provide scaffolding to advance learning progress.
• 9: Use social-interaction to increase learning and promote higher-order thinking.

270
 CHAPTER SEVEN

• 10: Engage learners in a challenge; target learning towards the zone of proximal development
(ZPD).

Phase 1 and 2 gave indicators that the students remain cognisant that this was a voluntary research study,
hence the level of effort and focus on learning material and passing quizzes is curbed. In Phase 3, the
tighter integration with the students’ in-school lessons diminishes this factor, but it is likely that it remains
a factor to some extent. Phase-2-Cycle2 also gave practical evidence of the importance of question design
within the overall instructional design, this feedback was incorporated in the Phase 3 software. The
pedagogy itself does not go to the level of detail of discussing question design; although, several of the
heuristics for e-learning software can be used to inform question design.

In Phase 1 and 2, the students perceive the difficulty level of the e-learning content to be split between
being at the right level and a little difficult, although the trend in Phase 2 was towards being at the right
level. In addition, the students perceived the difficulty level of the assessment activities to be split
between being at the right level and a little difficult, although the trend in Phase 2 was towards being a
little difficult. In Phase 3, the median student response showed the difficulty level of the learning material,
practice activities and quiz questions to be at the right level (3). Additionally, the median student
response showed agreement (4) that the e-learning software had prepared them for the quiz questions.

Overall, the findings from Phases 1 to 3 indicate that the implementation of the e-learning software gave
weak support for the ZPD. This was explained in more detail in the teacher workshop; in evaluating the
Level 3 Orange software, multiple evaluators commented that the e-learning software supported
progressive challenge that was appropriate for the average student, but may not be so appropriate for
weaker or high performing students. This reflects that heuristic 10 guideline 5: “Provide learning content
and activities that adapt to the learner’s current abilities and progress”, is particularly important and may
require greater emphasis in the pedagogy.

A noteworthy factor in the instructional design of e-learning software, and in particular in the selection of
which learning objects to use in communicating learning material, is that there is no consistent preference
in what learning objects students perceive to benefit their learning. Phase 1, Phase2-Cycle1 and Phase 3
all had different rankings for the learning object types. However, one clear trend was that audio and
collaborative activities were ranked poorly (last or close to last) in all three rankings. In Phase 3, audio,
collaborative learning and extended knowledge were ranked in the last spots, both in perceived benefit
to learning and perceived influence on interest and enthusiasm. However, in Phase 3, the students
reported a median response of agree (4) that all learning objects, with the exception of audio, helped
them understand the subject matter. In this respect, the median response is quite a blunt analysis; in
contrast a ranking of the mean averages gave a closer alignment with the Phase 3 learning benefit ranking.
However, the overriding conclusion remains: there is inconclusive evidence to support a preference
towards particular learning object types; their use must be judged in the context of the learning material,
and a balanced multi-modal delivery.

271
 CHAPTER SEVEN

Although, this research does not offer any conclusion on students’ preference towards learning object
types, the pedagogy does advocate activities and active learning; in this respect we consider problem-
solving, practice activities, quizzes, games and simulations. Phase 1 and 2 offered strong positive support
that activity-based (interactive) components support student understanding of subject matter.

“I mostly like the interactivity of the software and the fact that I am able to solve examples closely linked
to real life situations.” [Student 5]

“It is much better as it is interactive, it has some bugs though.” [Student 16]

“Games/Interactive components are really engaging. Videos and animations are an effective way of
learning something. Sense of accomplishment when you find the correct answer is satisfying.” [Student
14]

“The assessments summarised the content and connected different parts to each other, therefore making
it more interesting as I needed to be more creative.” [Student18]

The pedagogy’s focus on activities and active learning, is provided in the following heuristics:

• 1: Use authentic educational material, examples and activities.

• 2: Prompt reflective practice to support learning.
• 4: Use problem-based learning (PBL) to facilitate learning.
• 4.1: Use worked examples to support problem-based learning.
• 5: Integrate learning into long-term memory by using authentic examples, and non-trivial
practice and problems.
• 6.2: Exemplify computational thinking in problem-solving activities.
• 7: Distribute well-designed practice activities across the lesson to support learning.
• 7.1: Provide explanatory feedback to practice activities to promote learning.
• 14.3: Build “Confidence” to increase learner motivation (guidelines-2 and -3).
• 14.4: Build “Satisfaction” to increase learner motivation (guideline-1).
• 15.1: Integrate gamification elements tightly within existing learning processes.
• 16.4: Support kinaesthetic modal preference.

Phase 1 offered support and Phase 2 offered strong positive support that the activity-based (interactive)
components of the e-learning software are engaging.

Student perception of activity-based (interactive) components was not measured explicitly in the Phase 3
survey instrument; however, it was partially confirmed, since the students’ median response on whether
practice activities and quiz questions helped them understand the subject matter was agree (4).

Phase 1 and 2 also offered strong positive feedback that the assessment activities in e-learning software
encouraged them to think, and work through problems, instead of recounting from memory. However,
there is partially divergent feedback in Phase2-Cycle2, where students agreed that the e-learning software
also encouraged them to recall previous knowledge to answer the questions. The students’ perception of
these aspects was not measured in Phase 3.

272
 CHAPTER SEVEN

The e-learning pedagogy integrates connectivist principles in the following heuristics:

• 1.1: Ensure the currency of learning material.

• 11: Use collaborative learning activities.
• 12: Develop and nurture networks to support learning.

The investigation of connectivism and the web was carried out in Phase 1 and Phase 2-Cycle1, and showed
students to be intelligent and articulate in their response. The students reported that a critical appraisal
of the information found on the web is necessary since the source and quality of information may be
unreliable. This factor was also reported by Teacher 2 in Phase2-Cycle1:

“Students today are bombarded with sources of information, but actually having the ability to distinguish
what is most helpful and to actually be able to choose what tool or resource to use may be a challenge by
itself.” [Teacher 2]

This concern is addressed in heuristic 12, which proposes a moderate connectivist approach in which the
e-learning software acts as a focal point that recommends other learning resources to the students.
Heuristic 12 also discusses in the Potential Challenges section that the teacher should now reduce their
role of presenting learning material and readdress this focus towards teaching students’ critical thinking
and metacognitive skills for them to see connections between fields, ideas and concepts, and to critically
evaluate the learning material that they find.

The findings in Phase 1 and Phase 2-Cycle1 showed that students have a weak preference towards the e-
learning software compared to using the web for their subject learning; although the findings in Phase 2-
Cycle1 were more polarised. Overall, the students perceived the e-learning software to be comprehensive
and preferred the structure of having one place to learn from. Students wished to avoid the wasted time
in searching the web and evaluating whether the information they found is correct, which is addressed by
the e-learning software linking to other pre-vetted resources. Phase 1 student responses included:

“I believe the E-learned software has everything I need, hence not requiring any other information that I
must know to be obtained by the internet.” [Student 4]

“Through the use of an E-Learning software, I can use the Web as well, where the links to videos and blogs
will be included in the software.” [Student 2]

“Often on the Internet, details are mostly vague and one does not fully understand what he/she was
looking to learn.” [Student 5]

“They are different things. on the internet you can get easily immediately the information you want and
learn what you want. and in the E-Learning software IF you find what you want there is detailed reliable
information.” [Student 6]

Phase 2-Cycle1 student responses included:

“because I feel more independent as a learner when I search the information, I want to know on my own
and I don't like the information to be brought by someone else” [Student 25]

273
 CHAPTER SEVEN

“The internet: contains more information is easier to search through.” [Student 13]

“you have all the information collected in one source and the it is all checked and reliable when on the
internet you might not find what you want.” [Student 17]

“The E-Learning software has activities and features that are much more engaging than looking up
information on the Web. The Web sometimes contains sites with wrong information, so the E-Learning
software would be a much more reliable and easy source of information.” [Student 14]

“The E-Learning software has everything in one place, so I don't need to search the whole internet for all
of the different subjects separately, which helps me save quite a lot of time.” [Student 18]

As per the research design, Phase 3 did not secure qualitative findings in this area; however, the
quantitative results are aligned with previous phases in that the median student response is neither agree
nor disagree (3), with the statement “I would prefer using the Internet and the Web to support my learning
of the computing subject rather than this E-Learning software.”

A critical area within learning and within the pedagogy is motivation; the implementation of the e-
learning software and associated student usage of the software showed modestly positive findings for the
following heuristics:

• 1: Use authentic educational material, examples and activities.

• 7: Distribute well-designed practice activities across the lesson to support learning.
• 13: Use constructivist approaches to increase intrinsic motivation in the learner.
• 14: Use the concepts of Attention, Relevance, Confidence and Satisfaction (ARCS) to attain and
sustain learner motivation.
• 14.1: Use “Attention” grabbing strategies to increase learner motivation.
• 14.2: Explain the “Relevance” of the learning material to increase motivation.
• 14.3: Build “Confidence” to increase learner motivation.
• 14.4: Build “Satisfaction” to increase learner motivation.

Phase 1 and 2 showed broadly positive student feedback that: it is more interesting to use the e-learning
software to learn computing than the textbooks; the students could use the e-learning software for
independent study; and the e-learning software had increased the students’ overall enthusiasm and
interest in computing. The first statement is weakly confirmed in Phase 3, since the median response was
neither agree nor disagree (3) with the statement “It is more interesting to use the E-Learning software
to learn Computing than the textbooks.” However, the removal of four outliers improved the median
student response to agree (4). Further, confirmatory evidence is provided in Phase 3, since the median
student response to the second and third statements was agree (4).

274
 CHAPTER SEVEN

The IMMS results in Phase 1 and 2 are also modestly positive since the overall IMMS mean results are
consistent across Phase 1 and 2, at approximately 3.6738 from 5; this is between Moderately True (3) and
Mostly True (4). In each version of the e-learning software, a concerted effort was made to use
motivational approaches; Phase 2-Cycle1 focused on ARCS Motivational approaches and Cycle 2 focused
on gamification. Therefore, the stability in overall IMMS values could potentially be viewed as
discouraging. However, this needs to be considered within the context that many of the aspects of the
software remain unchanged between versions, and the student participants also remain the same; hence,
some of the novelty and motivation value is lost. By Phase2-Cycle2, three students had used the software
three times, and two students had used it twice. It is therefore not a surprise that IMMS results remained
stable. The Phase 3 IMMS results are broadly aligned with Phases 1 and 2, although comparatively weaker
in that the IMMS Overall result is (Mdn=3.11, M=3.15 SD=0.57) as compared to mean of 3.67 in Phases 1
and 2. There are no published threshold values of what IMMS results motivational e-learning software
should secure; therefore, this research postulates that results between Moderately True (3) and Mostly
True (4) are satisfactory with room for improvement.

An interesting factor identified in Phase 3 is that the e-learning software’s treatment of the four areas of
motivation are perceived by students to be comparatively balanced, with the exception of Student
Attention (Mdn=3.00, M=3.03 SD=0.54) and Student Relevance (Mdn=3.29, M=3.24 SD=0.71), which
showed a statistically significant difference. In real terms, the difference is likely to be negligible and
simply indicates that the software was slightly less effective in grabbing the students’ attention in
comparison to attuning to the students’ personal relevance.

Noteworthy findings were identified by the correlation analysis in Phase 3; a correlation link between
IMMS results and learning performance was not established; however, correlation between IMMS results
and the time students devoted to the e-learning software was established.

A statistically significant medium effect correlation between student IMMS overall and Level-2 time was
identified, and multiple Tier-239 medium effect correlations were identified between:

• Student Relevance and Level-2 time

• Student Relevance and Level-3 and -4 combined time
• Student Satisfaction and Level-3 time
• Student Satisfaction and Level-3 and -4 combined time
• Student IMMS Overall and Level-3 and -4 combined time

38
Phase 1 IMMS was 3.65, in Phase2-Cycle1 it was 3.68, and in Phase2-Cycle2 it was 3.67.
39
Tier 2: A correlation effect of ±.1 or more, and Bootstrap Confidence Interval (CI) does not cross over 0 threshold
(i.e. CI is between -.09 and 1 or .09 and -1)

275
 CHAPTER SEVEN

Causality is theorised that higher IMMS results in relevance, satisfaction and IMMS overall reflect a higher
student motivation, and that motivation leads to students willing to devote more time to the e-learning
software.

Gamification is also employed within the pedagogy and the associated e-learning software as a
motivational aid. Gamification is incorporated into the e-learning pedagogy in the following heuristics:

• 15: Use gamification to increase motivation and learning performance.

• 15.1: Integrate gamification elements tightly within existing learning processes.
• 15.2: Build extrinsic gamification elements on top of existing learning processes.

Intrinsic aspects of gamification defined in heuristic 15.1 were already inherently part of the e-learning
software from Phase 1, and are positively received. In Phase 2-Cycle 2, the extrinsic motivational aspects
defined in heuristic 15.2 were developed in the e-learning software and configured in the supporting
Learning Management System; specifically, this included support for levels, points, badges and a
leaderboard.

The student response to levels was positive in terms of helping them to estimate their progress and in
motivating them to get to the next level. The student response to points and badges in motivating them
to work hard and progress was mixed, and remained unenthusiastic in the focus group:

“Maybe since we’re at an age that… maybe we won’t care enough about virtual badges and awards and
stuff, I think that’s what he assumed. It won’t actually increase motivation for many of us, but maybe for
some of people it will work. Maybe if it was altered in some way it could increase motivation maybe
increase competition between the class to better perform.” [Student 19]

One of the challenges is that the implementation of gamification elements is tied directly to the Learning
Management Systems (LMS) which hosts the learning content. Support for gamification is not widespread
or equal between LMS’ and would therefore be impacted by a school’s choice of LMS. In the context of
Phase2-Cycle2, the LMS that was chosen was purported to offer native support for gamification elements
but, unlike some other systems, it did not allow the ad-hoc definition of triggers for achievements. Instead,
offered a limited set of award triggers based on completing a module, completing a module on time,
completing a module early, and completing skills which were associated to a module, skills advancement
which was further linked to advanced modules.

Overall, the level of granularity (module level) supported by the LMS was too coarse, and the triggers for
the award of achievements were inappropriate, both pedagogically and in motivating the student
participants. One student reported this, aptly:

“The award of badges should be on an achievement. How they should work is that if you complete a quiz
in a different way than the others, you get rewarded for something that others didn’t do differently, or
maybe badges for additional challenges that students can select.” [Student25]

The award of points become the basis of the leader board; again, the same challenge in configuring
pedagogically appropriate triggers is apparent. Results from the survey instrument showed the overall

276
 CHAPTER SEVEN

student response to leaderboards is non-committal; in addition, the student response to leaderboards is

non-committal in the scenario that they are a student doing well. However, a positive student survey
response is received in the scenario that they are a student performing weakly. The discussion in the focus
group did not lend much support to leaderboards.

“It’s like failing a test; it tells you to work harder for the next test. It is telling you to give more attention.”
[Student14]

“There are some people who will be very disappointed that they are at the bottom so instead of trying
harder they will try less.” [Student18]

“Maybe you don’t want other people to see your score, or you don’t want to see other people’s score so
you don’t get disappointed.” [Student19]

The findings in Phase2-Cycle2 do not offer supporting evidence towards heuristic 15.2; however, it is
postulated that this was primarily due to limited LMS support relating to coarse granularity and
inappropriate achievement triggers towards points, badges and the leader board. Heuristic 15.2 was
therefore updated with further guidance notes for teachers and a warning that:

“Extrinsic gamification elements are typically implemented within the Learning Management System
(LMS); therefore, it is crucial that the school’s LMS is evaluated to identify whether it supports the planned
gamification strategy. If it does not support gamification, or offers limited support to key aspects of the
gamification strategy, then consider not implementing gamification or implementing it in an unplugged
manner, outside of the LMS.”

Other challenges and bugs within the LMS meant that for Phase 3, the study reverted back to the original
SCORM LMS, and hence extrinsic gamification was not tested in Phase 3.

Several other heuristics were either not exemplified in the e-learning software or only partially evidenced.
The e-learning software did not include the responsive design and consideration for the technical
affordances of mobile devices; hence, it did not implement heuristic 11.1: Support collaborative and
situated learning via mobile devices. In addition, although the e-learning software is strongly focused on
active learning and contains numerous activities, the inclusion of true guided discovery and problem-
based learning (PBL) within the timeframes offered by each study cycle, and the five hours of e-learning
content, was not pedagogically feasible. Hence, heuristic 4: Use problem-based learning (PBL) to facilitate
learning is not validated beyond the evaluation from education experts and teachers. Following on from
this, it was identified in Phase2-Cycle1 that heuristic 6.2: Exemplify computational thinking in problem-
solving activities did not have a significant coverage in the e-learning software; hence, it is only partially
evidenced.

The most important area that is only partially evidenced and has received mixed (often negative) feedback
is collaborative and social learning; this is covered in the following heuristics:

• 9: Use social-interaction to increase learning and promote higher-order thinking.

• 11: Use collaborative learning activities.

277
 CHAPTER SEVEN

• 11.1: Support collaborative and situated learning via mobile devices.

• 12: Develop and nurture networks to support learning.

Collaborative activities were consistently ranked poorly in Phase 1 and 2, which was re-confirmed in Phase
3 with a larger student sample. Phase 1 and 2 student feedback on whether collaborative activities
supported their understanding of the subject matter is mixed; although Phase 3 results on the same
question offered a more positive median student response of agree (4). The more concerning factor is
that throughout the study, the students’ engagement with collaborative activities is limited, and when
engaged was often superficial. This is also clearly represented in Phase 3 with the larger student sample.
The Phase 3 study did not provide deeper qualitative feedback in this respect, but these seemingly poor
results are considered in light of the qualitative feedback in Phase 1 and 2. Open responses from the
survey instruments and focus groups in Phase 1 and Phase2-Cycle1, show the students have a good
conceptual understanding of the value and benefits of collaborative learning.

“The collaborative activities helped me learn a lot as I heard about other people’s opinions on the subject.”
[Student 25]

“The collaborative activities gave me and my partner the opportunity to help each other understand the
questions we had. One could answer the questions of the other, which was really helpful in order to
complete our task.” [Student 14]

“The collaborative activities gave me motivation to work harder in order to not fall behind my partner.”
[Student 18]

“Encouraged me to better understand the material rather than simply glossing over me, because I had to
explain it to my partner.” [Student 13]

The students acknowledged that collaborative learning is equally useful in class, but technology-enhanced
collaborative activities are artificial in a classroom context and much more suited for homework, where
you are not co-located with your fellow students. This was also commented by Teacher 1 in the pilot phase
of the study. In the context of this study, the majority of time spent using the e-learning software and CLE
was in class and so it did not feel natural to the students. The reality is that an optional research study is
not a suitably attractive proposition for students to spend their free time on, much less to coordinate with
others to jointly spend their free time on. However, the consensus from both Phase2-Cycle1 focus groups
was that if collaborative learning and a collaborative learning environment were integrated into the
normal learning process (including homework) that spans the academic year, then they would have really
liked to have done it.

“yes it’s much better than what we currently have to submit work from home.” [Student 14]

What was tentatively postulated in Phase 1 was more clearly established in Phase2-Cycle1: an experiment
design to give feedback on collaborative learning would need to more naturally reflect the learning
process across a longer duration in which the students have weeks/months to review, respond and
interact. In this study, this was not possible; even in Phase 3, the student engagement was approximately

278
 CHAPTER SEVEN

two weeks, of which most time was spent in class. What is clear and hence reflected in the pedagogy, is
that collaborative and social learning require a long-term investment to prepare students, and to build a
culture and learning community to support it. This is reflected in heuristics 11, 12 and Vol 2 Appendix E
section B.6.

Whilst the findings from student usage of the e-learning software does not offer positive evidence of the
aforementioned heuristics, the adverse, and overall mixed, findings are not so strong as to discredit
collaborative and social learning, and hence the heuristics are not removed from the pedagogy.

7.8 Phase 3 Learning Performance

As discussed previously, Level-1 and Level-2 of the e-learning software provide foundational concepts on
algorithms and computational thinking, whereas Level-3 and Level-4 provide learning material on
flowcharts and pseudocode of direct relevance to the pre- post-test exam. The duration of the student
engagement with the e-learning software is therefore reported for all four levels (L1L2L3L4) combined,
and separately for Level-3 and 4 (L3L4) combined. The learning performance discussed in this section is
based on a student engagement with all four levels with median 196.5 minutes (95% central range 172
minutes to 219 minutes), and with Level-3 and 4 combined with median 64.5 minutes (95% central range
56 minutes to 100 minutes). The aforementioned durations do not include time spent on the collaborative
learning environment working on collaborative or individual assignments. As reported in section 6.6.7.3,
based on student submissions, the level of student engagement with the CLE was perceived to be nominal,
but the duration of that engagement cannot be estimated. Of importance is that none of the students
submitted the four assignment activities in Level-3 and -4; these would have given students an
opportunity to practice skills directly transferable to the post-test. It is therefore postulated that the lack
of engagement with these assignment activities had an adverse effect on post-test results. Although, the
scale and significance of that effect cannot be estimated based on current data.

Interesting correlations were identified between the teachers’ Key Stage 4 predictions (KS4Prediction) for
students and the time they devoted to the e-learning software. A statistically significant medium effect
negative correlation was identified between KS4Prediction and Level-1 learning time. Causality is
theorised that students with a higher KS4 prediction require less time to understand the foundational
material covered in Level-1; hence, they spend less time on the level. Conversely, statistically significant
medium effect correlations were identified between KS4Prediction and Level-4 time and Level-3 and -4
combined time. Causality is theorised that students with a higher KS4 prediction are more driven to do
well in the later more challenging levels, and therefore are willing to spend more time on them. However,
it should be noted that a correlation was not identified between KS4Prediction and learning performance,
as defined in %Post-Test or %Change.

After their engagement with the four levels of the e-learning prototype, the students did not express
confidence that they could pass a test on the subject matter. The median student response was neither
agree nor disagree (3) with the statement “After completing the 4 levels of the e-learning software, I was
confident that I would be able to pass a test on it.” However, this lack of confidence (Question 6N) and

279
 CHAPTER SEVEN

the students’ perception of their own learning (Question 12 C) did not offer a statistically significant
correlation with learning performance metrics of %Post-Test and %Change.

Considering %Post-Test was assessed by the Kolmogorov-Smirnov test to have a non-normal distribution,
it was decided to analyse pre- post-test learning performance using parametric tests (Paired Sample T-
Test), bootstrapping and non-parametric tests (Wilcoxon Signed-Ranks Test). All three tests, with and
without outliers reflected a statistically significant improvement in learning performance. The learning
performance improvement between pre-test and post-test had a mean of 19.22% points (95% CI 15.68%
to 22.76%). The learning performance improvement of %Change had a median of 20.97% points (95%
central range 12.90% to 25.81%). We therefore conclude that based on the level of engagement
previously outlined and the limitations therein, a statistically significant learning performance
improvement can be generalised to the GCSE student population from usage of e-learning software
designed in adherence to the e-learning pedagogy.

This is further supported by the correlation analysis which shows a statistically significant medium effect
correlation between the time students spent using the Level-2 software and %Post-Test results.
Furthermore, there is a Tier-2 small effect correlation between the time students spent using all levels of
the e-learning software and %Post-Test results. Causality is theorised that the more time students
devoted to using the e-learning software, the better their post-test learning performance was.

Considering the post-test was created using exam questions from EDEXCEL, OCR and AQA examination
boards, it is possible to theorise the examination results possible within the GCSE student population. It
should be noted that there are limitations within this analysis since the first examinations were in 2018,
so grade boundaries for the new specification had not been released at the time of this study. With
reference to recent GCSE Computer Science grade boundaries from EDEXCEL (Pearson 2016), OCR (2017)
and AQA (2016), a realistic theoretical model of the grade boundaries for the new GCSE specification was
developed. Within this theoretical model, a Grade F (Low Grade 2), is approximately 16.33% and a Grade
C (Grade 4 – Standard Pass), is approximately 37.35%.

Using parametric tests (One-Sample T-Test) with and without outliers and with bootstrapping, all indicate
that we cannot reject the null hypothesis that the %Pre-Test mean is equal to the hypothesised Grade F
target value.

Although, %Post-Test is not normally distributed, there is no equivalent non-parametric test. Using
parametric tests (One-Sample T-Test) with and without outliers and with bootstrapping, all indicate that
we cannot reject the null hypothesis that the %Post-Test mean is equal to the hypothesised Grade C target
value.

We therefore conclude that, based on the level of engagement previously outlined and the limitations
therein, the GCSE student population can move from a hypothesised Grade F target value to a
hypothesised Grade C target value following usage of the e-learning software designed in adherence to
the e-learning pedagogy.

280
 CHAPTER SEVEN

7.9 E-learning Evaluation Protocol

As discussed in section 2.3.3, there is a research deficit for e-learning evaluation protocols that focus on
pedagogy. Additionally, the need for a rigorously defined e-learning evaluation protocol is established by
several recent national evaluations and the inherent limitations of those evaluations. This study proposes
a rigorously defined evaluation protocol based on evaluation best practice, but which is unique in its
explicit consideration of the educational setting for software usage, its detailed evaluation criteria, its
focus on both pedagogical and content quality (learning objectives), and its provision of quantitative and
comparable results.

The e-learning evaluation protocol had matured to the point where Phase 3 feedback from education
experts and teachers was minimal, and the proposed refinements have been implemented in the final
version of the protocol.

The teacher feedback indicated their school’s limited experience, limited use of e-learning software and
a lack of school-wide policy. Furthermore, the majority of teachers report limited or no personal
experience of integrating e-learning software into their teaching. Whilst current usage of e-learning
software is limited, the teachers showed a clear understanding of the benefits of e-learning software and
expressed that it could have positive educational and motivational impact towards computing.

Although the teachers had limited or no experience in selecting or evaluating e-learning software, they
expressed that if they were called upon by their schools, they would get involved in an e-learning
evaluation and, most importantly, that they could use the heuristics and the evaluation procedure
described in the workshop to evaluate e-learning software.

Despite the teachers’ positive feedback towards the e-learning evaluation protocol and their limited
recommendations for improvement, there remains an implied concern that the e-learning evaluation
process may be too time consuming for teachers to fit into their busy schedule. This was not reported by
teachers, but was a concern raised by one education expert; this could, in the future, be validated by
further empirical evidence of teacher usage. Acting on this concern, in Phase 3 a timeline analysis was
carried out to analyse each of the tasks in an e-learning evaluation to identify any scope for further
efficiency, and to estimate a best and worst-case duration for each activity. The best and worst-case
evaluation scenarios considered:

1. The timeline analysis reported in Vol 2 Appendix C.4,

2. E-learning software of 1-hour typical learning duration and 6 Learning Objectives, and
3. Evaluators who are double experts,
a. The worst-case scenario additionally considers the evaluators to have zero or minimal
experience with the heuristics and evaluation protocol,
b. The best-case scenario considers the evaluators experienced with the heuristics and
evaluation protocol.

281
 CHAPTER SEVEN

The worst-case scenario estimates approximately 14 hours from the facilitator and 9 hours 30 minutes
from each evaluator. The best-case scenario estimates approximately 6 hours 15 minutes from the
facilitator and 5 hours 45 minutes from each evaluator. It follows that the best-case scenario can be
accommodated within one day, whereas the worst-case scenario is better planned across multiple days.

This study provides findings that partially validate the accuracy of the heuristic evaluation. Both teacher
groups provided feedback that was accurate in relation to the known characteristics of the e-learning
software under evaluation. As per the intended purpose, the e-learning evaluations and in particular the
debrief sessions added great value in identifying specific pedagogical shortcomings and recommendations
to improve the software. Validity and reliability are further supported by the comparison of Group-1 and
Group-2 heuristic evaluation results, which show a notable similarity on the Pedagogical Quality results
of 59% and 56%. As detailed in Vol 2 Appendix C.6.1.1, a more in-depth review of the Group-2 responses
also demonstrated the potential for even closer alignment.

In contrast, a similar comparative analysis of the Group-1 and Group-2 evaluation results for the six
learning objectives throws concern on the reliability and validity of the evaluation protocol; there is a
significant discrepancy on the Content Quality results of 88% and 27%.

A deeper analysis offers a partial explanation of this discrepancy. Group-2 responded that for three of the
six learning objectives, they felt that the e-learning software slightly counteracted the successful learning
of the learning objectives. A negative value for support of learning objectives is typically reserved for e-
learning software, whose learning content is factually incorrect or inaccurate in terms of a given syllabus,
or that in some manner explicitly obstructs the learning of the learning objective. It is doubtful that the
learning content was inaccurate or poorly designed to the extent that it would obstruct learning on these
topics. This assertion is made because the learning content was synthesised from official material from
the three main examination boards, and practice questions and activities were sourced and adapted from
past examination papers and education experts from the Computing at Schools Network. Furthermore,
with reference to Figure 75, it is evident that for learning objectives 1, 2 and 6; the responses from
individual evaluators, and the mean, mode and median values, are all radically different to the debrief
responses.

282
 CHAPTER SEVEN

Figure 75: Phase 3 - Group-2 evaluation of support for learning objectives

This is further supported by a correlation analysis; both groups demonstrate the trend that the mean,
mode and median typically offer a strong correlation with the group’s final debrief response. The most
notable exception to this is the Group-2 - Learning Objective Support correlation, which shows that the
correlations are not statistically significant, the lowest mean to debrief correlation, and zero correlation
between mode and debrief, and median and debrief. The previous analyses indicate that the debrief
responses for learning objectives 1, 2 and 6 do not reflect normal evaluator behaviour, and are atypical.
An investigation of the audio transcripts from the debrief session offers a partial explanation; Group-2
had a significant challenge in aligning on the given education setting (refer to Vol 2 Appendix C.6.1.2). A
significant portion of their debrief discussion reflected an incorrect perception that the Level-3 software
should include pseudocode, and that the coverage of data types, and variables and constants should be
at a level to prepare students for coding (programming). This incorrect perception would explain a low
response for the level of support for learning objectives 1 and 2, but would not explain the group’s
assertion that the software counteracted the learning of these areas. Furthermore, it does not explain
that for software dedicated to the instruction of flowcharts, that Group-2 responded that it counteracts
the learning of flowcharts in the context of inputs, processing and outputs (LO6).

These findings identify two critical areas in an e-learning evaluation:

1. The importance of the facilitator’s role within the evaluation, and

2. Alignment with the educational setting prescribed for the evaluation.

The primary purpose of the debrief session is to aggregate findings and build consensus, it is therefore
the primary objective of the facilitator to promote and enable this. In addition, the facilitator in a debrief

283
 CHAPTER SEVEN

session goes beyond being an enabler and the gatekeeper of group-process; the facilitator must be a
content expert (knowledgeable of the syllabus, the heuristics, and the e-learning software) and therefore
able and active in correcting misconceptions. The debrief session offers a final opportunity to correct
mistakes by the evaluators or any misconceptions they may have, before they influence the final group
response.

In both groups, there were instances were individual evaluators had missed particular functionality in the
e–learning software and therefore responded that particular heuristics were weakly or not supported.
Such misconceptions are often corrected by other evaluators in the group, but if not, then they must be
corrected by the facilitator to avoid corrupting the group result.

Ensuring the group’s evaluation is aligned with the education setting is crucial; this was a challenge in
Group-2 and to a smaller extent in Group-1. If a group is evaluating based on a misconception of the
learning context then they will evaluate unduly harshly or leniently; therefore, the facilitator must step in
to realign the group’s understanding of the education setting. This is to ensure the evaluation is based on
an agreed, realistic and documented educational setting. This in turn affects reliability since other groups
doing a similar evaluation should evaluate based on the same education setting. In addition, it is also
recommended to increase the emphasis of the education setting presentation (before the evaluation) and
also include a short reminder presentation at the start of the debrief session.

On multiple occasions evaluators commented that their individual response was a mistake because they
had misunderstood the heuristic (and its criteria). This is often self-correcting since it comes out in
discussion with the group. Much rarer will a significant portion of the group be evaluating under a
mistaken understanding of the heuristic. To pre-empt this, it is important that the evaluators have access
to the evaluation criteria as they discuss a heuristic. Also, that the facilitator is able to give clarification of
a heuristic and its criteria, and that a facilitator actively engages if the evaluators are not evaluating a
heuristic according to the stated criteria.

The above scenarios are particularly damaging when there are dominant figures within the evaluation
group. It is already a part of the facilitators role to balance such dominant figures, but it becomes critical
if a dominant evaluator is working under a misconception and driving the group in the wrong direction.

It is unrealistic to expect that each evaluation will have a professional or highly experienced facilitator
engaged; nevertheless, it is important that this role is actively assigned within the evaluation debrief
session. More importantly, that the person taking on the facilitation role is a dual expert with
comprehensive knowledge of the education setting, and additional experience of both the heuristics and
the e-learning software. To support this, additional facilitator guidelines are provided in the evaluation
protocol under Vol 3 Appendix N section A.

284
 CHAPTER EIGHT

8 CONCLUSION, CONTRIBUTION AND FUTURE WORK

To support the UK’s high-school teaching community (and other countries undergoing a curriculum
transition to a more computer science focus) this study proposes that quality e-learning software tailored
to the computing curriculum can be a beneficial to the teaching community. It considers that pedagogy is
one of the most critical factors in successful e-learning and that concerns still persist over the quality of
e-learning software. In response, this research has developed an e-learning pedagogy and associated
evaluation protocol to ensure the pedagogical quality of computing e-learning software used in schools.

The proposed e-learning pedagogy is comprehensive in nature, and has received input and review through
multiple iterations from education experts and teachers. It has reached the point where it is assessed to
be: appropriate for computer science education; appropriate for 15 to 18-year olds (Key Stage 4 and 5);
offers balanced and comprehensive pedagogical coverage; and can feasibly be implemented within a high-
school environment.

It is, though, acknowledged that the scale of the pedagogy and the endeavour needed to fully utilise it
still leaves some residual concern, with one education expert questioning how and whether it will be
practically used by teachers. The heuristics in themselves can feasibly be implemented in schools and
where heuristics have possible implementation challenges, they are documented for appropriate curative
action. However, the overriding conclusion is that when designing e-learning software or evaluating
software for use in schools, the heuristics that will be applied must consider the reality of the school
context in which the e-learning software will be used.

Moving beyond a theoretical synthesis and review, the pedagogy was developed with input from a
practice-based implementation of e-learning software which received feedback from GCSE students. This
feedback gave input in refining some of the heuristics, demonstrated the viability of developing e-learning
software based on the pedagogy, and most importantly reflected positive support for the majority of the
heuristics. It is, though, acknowledged that within the limitations of this study, several heuristics were not
practically evidenced via student usage of the prototype e-learning software; the most notable
pedagogical areas were problem-based learning, and collaborative and social learning.

Finally, Phase 3 generalised statistically significant results showing that usage of the e-learning software
can lead to a mean 19%-point increase between pre- and post-tests. This in turn is theorised to be the
equivalent of a GCSE Computing student moving from an F to a C grade through the use of only the e-
learning software. In this respect, we can reject the null hypothesis (H0) in favour of the hypothesis:

• H1: The use of e-learning software designed in adherence to the pedagogical heuristics presents
improved student learning performance.

Phase 3 also identified notable correlations between motivation and student time spent using the
software, and between time spent using the e-learning software and improved learning performance. The
direction of causality is difficult to attribute between time spent and motivation. However, causality is
apparent, increased time spent using the software leads to improved learning performance.

285
 CHAPTER EIGHT

Similar to the e-learning pedagogy, the e-learning evaluation protocol received input and review through
multiple iterations from education experts, and practice-based feedback from two groups of teachers
during the Phase 3 workshop. The evaluation protocol has matured to the point where feedback from
education experts and teachers was minimal, and had already been implemented in the final version of
the protocol. The teachers expressed the learning and motivational benefits of e-learning software as a
teaching resource, and the opinion that they could use the heuristics and the evaluation protocol to
evaluate e-learning software for school use.

Although not reported by the teachers, there remains an implied concern that the e-learning evaluation
process may require time not easily available in their busy schedule; this can be validated by further
empirical evidence of teacher usage.

This study provides findings that offer preliminary validation of the accuracy of the heuristic evaluation in
terms of: feedback that was accurate in relation to the known characteristics of the e-learning software;
valuable feedback in identifying specific pedagogical shortcomings and recommendations to improve the
software; and the validity and reliability of the inter-group comparison of pedagogical quality. However,
a similar inter-group comparison of content quality showed a sizeable discrepancy in evaluation results.
A hypothesised explanation for this desynchronization has been provided, and mitigating guidelines
added to the evaluation protocol; nevertheless, it throws some concern on the reliability and validity of
the evaluation protocol, which can only be fully dispelled by a follow-on study.

8.1 Research Contribution

Considering the Royal Society’s recent report (2017b), the three critical factors that prompted this
research (shortage of qualified specialist teachers, lack of subject specific pedagogy, and inadequate
technical teaching resources), have further intensified in the UK. This research addresses these factors by
supporting the increased use of e-learning software in high-school computer science, whilst
simultaneously safeguarding the pedagogical quality of the software.

Specifically, the research aims were fulfilled by the provision of an:

1. e-learning pedagogy that focuses on high-school computer science (as of January 2018, an
equivalent holistic set of heuristics could not be identified); and
2. an e-learning evaluation protocol for the evaluation and selection of e-learning software for use
high-school teaching (again the characteristics of the protocol, make it arguably unique).

Although the pedagogy and associated evaluation protocol were well received by the teachers involved
in this research, their practical contribution remains unproven since that is contingent on their significant
uptake by the high-school teaching community.

What is clear though, is that this research contributes to a severely under-researched area, in terms of
high-school computing pedagogy, and specifically e-learning pedagogy. Furthermore, this study’s

286
 CHAPTER EIGHT

contribution is executed with a high degree of research rigour and generalisable findings, which has been
reported as typically lacking in educational research.

Whilst there are a variety of texts on e-learning instructional design; this research differs by providing an
accessible set of heuristics for e-learning pedagogy that provides the essence of e-learning pedagogy to
the teacher audience. This research focus contrasts with previous research on e-learning heuristics, which
focused almost entirely on usability with little reflection of the complexities of the learning process.
Furthermore, whilst numerous researchers expound the need to assess the educational quality of e-
learning software, few have proposed a tangible and detailed focus on the evaluation of e-learning
pedagogy and content quality. Hence, this study also redresses that gap.

Whilst this study takes the UK as its backdrop, a significant number of developed countries are
undertaking similar initiatives to give an increased focus to computer science in their high-school
curriculums. Those in the earlier stages of these initiative have a common thread of challenges in
transitioning teachers to a new curriculum, ensuring teachers have both subject and pedagogical
knowledge to support their teaching, and the relevant teaching resources to support student instruction.
These factors thereby make this research broadly transferable to those countries.

8.2 Future Work for the E-learning Pedagogy

It is reaffirmed that the aim of this research is not to propose e-learning as a replacement for high-school
teachers, but as a support and as a tool for teacher use. Hence by design, this study does not consider a
control group by which students are taught by traditional didactic methods vs e-learning software
designed in adherence with the pedagogy.

Despite the significant work and findings of this research, it is considered a mature beginning, primarily
focused on the development of the pedagogy, its associated evaluation protocol, and the effect on
student learning and motivation. There remain several possibilities to further empirically establish this
research or develop it further into new research avenues.

An initial step in further establishing the e-learning pedagogy is to build an empirical body of evidence of
multiple e-learning software whose design is guided by the pedagogy. In this manner, further refining the
pedagogy, based on practice-based evidence in schools.

Building on the proposed body of empirical evidence would be a correlation between the e-learning
evaluation score and learning performance. In this manner establishing that increases in the “Educational
Value” (as defined in section 5.4.3) of the e-learning software correlates to an improvement in student
learning performance. As discussed in the next section (8.3), this could potentially lead to comparative
evaluations of several purportedly equivalent e-learning software, some designed in adherence to the
pedagogy and some without pedagogical design guidance. This would establish findings on whether the
evaluated “Educational Value” of an e-learning software is impacted by adherence to the pedagogy and
whether there is a statistically significant correlation with learning performance.

287
 CHAPTER EIGHT

As acknowledged in this study, certain pedagogical areas were weaker evidenced in the practice-based
intervention. Longitudinal studies can be carried out that give specific focus to these pedagogical areas
and provide more conclusive school-based evidence of their potential benefits.

The pedagogy’s hypothetical transferability can be established using the pedagogy in the design, or the
evaluation of e-learning software from other national curriculums, or in other STEM subjects.

8.3 Future Work for the E-learning Evaluation Protocol

As discussed in section 5.4.9, a standard approach to ascertain the performance and validity of an
evaluation protocol is to conduct a comparative evaluation of the same software using an established
evaluation protocol with the new protocol. This would be an important area of investigation. However,
based on the existing literature review, there does not seem to be an equivalent established protocol,
which suggests a positive indication of the uniqueness of this e-learning evaluation protocol.
Nevertheless, the preliminary findings from this study suggest that additional studies are needed to
further establish the reliability and validity of the evaluation protocol; follow-on studies can use multiple
evaluation groups to evaluate the same software and correlate their findings.

An interesting area of further investigation would be to use the evaluation protocol for a comparative
evaluation of several purportedly equivalent e-learning software. This would help establish whether the
protocol offers practical value in the selection of e-learning software.

Another interesting and potentially valuable follow-on study would be to correlate the findings of the e-
learning evaluation against an evaluation based on Kirkpatrick’s (1967) training evaluation model. This
research already goes some way in comparing the e-learning evaluation findings against Kirkpatrick’s 1st
and 2nd steps: reaction and learning. A long-term study could be devised to establish whether the e-
learning evaluation protocol offers any indicators towards Kirkpatrick’s 3 rd and 4th steps: changes in
behaviour and results.

With long term use of the evaluation protocol, empirical evidence can be used to define reference values
for each of the heuristics and for overall educational value. This would enhance the evaluation protocol
by setting reference thresholds for good, moderate and poor e-learning software; this has previously been
explored (at a cursory level) by Abderrahim et al. (2013).

288
 CHAPTER NINE

9 REFERENCES
Abderrahim, E.M., Mohamed, E. & Azeddine, N., 2013. An Evaluation Model of Digital Educational
Resources. International Journal of Emerging Technologies in Learning, 8(2), pp.29–35. Available at:
http://www.scopus.com/inward/record.url?eid=2-s2.0-84879340049&partnerID=tZOtx3y1.

ACM et al., 2016. K-12 Computer Science Framework, ACM. Available at:
https://dl.acm.org/citation.cfm?id=3079760 [Accessed December 15, 2017].

Aleven, V. & Koedinger, K., 2002. An effective metacognitive strategy: Learning by doing and explaining
with a computer-based Cognitive Tutor. Cognitive science, 26(2), pp.147–179. Available at:
http://www.sciencedirect.com/science/article/pii/S0364021302000617 [Accessed February 2,
2015].

Allen, K., Scheve, J. & Nieter, V., 2011. Understanding Learning Styles Making a Difference for Diverse
Learners., CA: Shell Education. Available at:
https://books.google.com.cy/books/about/Understanding_Learning_Styles.html?id=lfKkGRm_gPQ
C&redir_esc=y [Accessed November 13, 2014].

Alonso, F. et al., 2005. An instructional model for web-based e-learning education with a blended learning
process approach. British Journal of Educational Technology, 36(2), pp.217–235. Available at:
http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8535.2005.00454.x/full [Accessed February 4,
2015].

Alonso, F., Manrique, D. & Viñes, J.M., 2009. A moderate constructivist e-learning instructional model
evaluated on computer specialists. Computers & Education, 53(1), pp.57–65. Available at:
http://www.sciencedirect.com/science/article/pii/S0360131509000025 [Accessed October 2,
2014].

Alptekin, S.E. & Karsak, E.E., 2011. An integrated decision framework for evaluating and selecting e-
learning products. Applied Soft Computing, 11(3), pp.2990–2998. Available at:
http://www.sciencedirect.com/science/article/pii/S1568494610003005.

Alsumait, A.A. & Al-Osaimi, A., 2010. Usability Heuristics Evaluation for Child E-learning Applications.
Journal of Software, 5(6), pp.654–661. Available at: http://dl.acm.org/citation.cfm?id=1806417
[Accessed June 23, 2015].

Ambient Insight Research, 2015. The 2015-2020 Worldwide Self-paced eLearning Market. Available at:
http://www.ambientinsight.com/Reports/eLearning.aspx#section1 [Accessed October 25, 2015].

Anderman, E.M. & Dawson, H., 2011. Learning and Motivation. In R. E. Mayer & P. A. Alexander, eds.
Handbook of research on learning and instruction. Routledge - Taylor & Francis Group, pp. 219–241.
Available at: http://www.jafarianhamid.persiangig.com/Research on Learning and
Instruction.pdf#page=287 [Accessed February 2, 2015].

Anderson, T., 2008. Theory and Practice of Online Learning 1st ed. T. Anderson & F. Elloumi, eds.,

289
 CHAPTER NINE

Athabasca University Press. Available at:

http://books.google.com/books?hl=en&lr=&id=RifNwzU3HR4C&oi=fnd&pg=PA91&dq=The+Theor
y+and+Practice+of+Online+Learning+&ots=Sf9mJmQWow&sig=Wpcfe2wh3PYM3SCumT_FVQmwL
oY [Accessed October 9, 2015].

Anthony, G., 1996. Active Learning in a Constructivist Framework. Educational Studies in Mathematics,
31(4), pp.349–369. Available at: http://link.springer.com/10.1007/BF00369153 [Accessed
December 3, 2017].

AQA, 2013. GCSE Computer Science: Connecting students to today’s world.,

AQA, 2016. Grade boundaries – June 2016 exams GCSE, Available at:
http://filestore.aqa.org.uk/over/stat_pdf/AQA-GCSE-GRADE-BDYS-JUNE-2016.PDF [Accessed
October 23, 2017].

Ardito, C. et al., 2006. An approach to usability evaluation of e-learning applications. Universal Access in
the Information Society, 4(3), pp.270–283. Available at:
http://link.springer.com/article/10.1007/s10209-005-0008-6 [Accessed June 23, 2015].

Ardito, C. et al., 2004. Usability of e-learning tools. In Proceedings of the working conference on Advanced
visual interfaces. New York, NY: ACM, pp. 80–84. Available at:
http://dl.acm.org/citation.cfm?id=989873 [Accessed June 23, 2015].

Aritajati, C. et al., 2015. A Socio-Cognitive Analysis of Summer Camp Outcomes and Experiences. In
Proceedings of the 46th ACM Technical Symposium on Computer Science Education - SIGCSE ’15.
ACM, pp. 581–586. Available at: http://dl.acm.org/citation.cfm?id=2676723.2677292.

Armoni, M. & Gal-Ezer, J., 2014. High school computer science education paves the way for higher
education: the Israeli case. Computer Science Education, 24(2–3), pp.101–122. Available at:
http://www.tandfonline.com/doi/abs/10.1080/08993408.2014.936655 [Accessed December 15,
2017].

ATD Research, 2014. 2014 State of the Industry, Available at: https://www.td.org/Professional-
Resources/State-Of-The-Industry-Report?mktcops=c.learning-and-development~c.lt~c.sr-leader
[Accessed May 4, 2015].

ATD Research, 2017. 2017 State of the Industry, Available at: https://www.td.org/research-reports/2017-
state-of-the-industry [Accessed December 20, 2017].

Ayres, P. & Sweller, J., 2005. The Split-Attention Principle in Multimedia Learning. In The Cambridge
handbook of multimedia learning. Cambridge University Press, pp. 135–146. Available at:
http://books.google.com/books?hl=en&lr=&id=duWx8fxkkk0C&oi=fnd&pg=PA135&dq=ayres+swe
ller+2005&ots=x8Xcx1AlbC&sig=whYiwuY7xGAJw5m1l_EoFJ6BVc0 [Accessed January 8, 2015].

Balanskat, A., Blamire, R. & Kefala, S., 2006. The ICT Impact Report. A Review of Studies of ICT Impact on
Schools in Europe, Available at:
http://colccti.colfinder.org/sites/default/files/ict_impact_report_0.pdf [Accessed December 17,

290
 CHAPTER NINE

2017].

Baller, S., Dutta, S. & Lanvin, B., 2016. The Global Information Technology Report 2016 Innovating in the
Digital Economy, World Economic Forum. Available at:
http://www3.weforum.org/docs/GITR2016/WEF_GITR_Full_Report.pdf.

Barab, S. & Squire, K., 2004. Design-based research: Putting a stake in the ground. The journal of the
learning sciences, 13(1), pp.1–14. Available at:
http://www.tandfonline.com/doi/pdf/10.1207/s15327809jls1301_1 [Accessed July 13, 2015].

Barbour, M. & Rich, P., 2007. Social Constructivist E-Learning: A Case Study. In Proceedings of E-Learn
2004--World Conference on E-Learning in Corporate, Government, Healthcare, and Higher
Education. Association for the Advancement of Computing in Education, pp. 1754–1759. Available
at: http://digitalcommons.sacredheart.edu/ced_fac/157/ [Accessed October 23, 2014].

Barker, L., McDowell, C. & Kalahar, K., 2009. Exploring factors that influence Computer Science
Introductory Course Students to Persist in the Major. ACM SIGCSE Bulletin, 41(2), pp.282–286.
Available at: http://dl.acm.org/citation.cfm?id=1508923 [Accessed July 13, 2015].

Baron, G.-L. et al., 2014. Computer Science Education in French Secondary Schools: Historical and
didactical perspectives. ACM Transactions on Computing Education, 14(2), pp.1–27. Available at:
http://dl.acm.org/citation.cfm?doid=2642651.2602486 [Accessed December 15, 2017].

Barr, V. & Stephenson, C., 2011. Bringing Computational Thinking to K-12. ACM Inroads, 2(1), pp.48–54.
Available at: http://dl.acm.org/citation.cfm?id=1929905 [Accessed February 23, 2015].

Baskerville, R.L. & Wood-Harper, T., 1996. A critical perspective on action research as a method for
information systems research. Journal of Information Technology, 11(3), pp.235–246. Available at:
http://search.ebscohost.com/login.aspx?direct=true&db=a9h&AN=6270082&site=ehost-live
[Accessed August 19, 2014].

BCS, 2012. The case for Computer Science as an option in the English Baccalaureate, Available at:
www.bcs.org.

Beck, L.L. & Chizhik, A.W., 2008. An experimental study of cooperative learning in cs1. In Proceedings of
the 39th SIGCSE technical symposium on Computer science education - SIGCSE ’08. New York, New
York, USA: ACM Press, p. 205. Available at:
http://portal.acm.org/citation.cfm?doid=1352135.1352208.

Bell, F., 2011. Connectivism: Its place in theory-informed research and innovation in technology-enabled
learning. The International Review of Research in Open and Distributed Learning, 12(3), p.98.
Available at: http://www.irrodl.org/index.php/irrodl/article/view/902.

Bell, F., 2010. Network theories for technology-enabled learning and social change: Connectivism and
actor network theory. In Proceedings of the 7th International Conference on Networked Learning.
pp. 526–533. Available at:
http://usir.salford.ac.uk/9270/?utm_source=twitterfeed&utm_medium=twitter [Accessed March

291
 CHAPTER NINE

27, 2014].

Bell, T., 2014. Establishing a nationwide CS curriculum in New Zealand high schools. Communications of
the ACM, 57(2), pp.28–30. Available at: http://dl.acm.org/citation.cfm?doid=2556647.2556937
[Accessed December 15, 2017].

Bell, T., Andreae, P. & Lambert, L., 2010. Computer science in New Zealand high schools. In Proc. 12th
Australasian Computing Education Conference (ACE. pp. 15–22. Available at:
http://dl.acm.org/citation.cfm?id=1862223 [Accessed May 3, 2014].

Bell, T., Andreae, P. & Robins, A., 2012. Computer science in NZ high schools. In Proceedings of the 43rd
ACM technical symposium on Computer Science Education - SIGCSE ’12. New York, New York, USA:
ACM Press, p. 343. Available at: http://dl.acm.org/citation.cfm?doid=2157136.2157240 [Accessed
July 1, 2016].

Bellettini, C. et al., 2014. Informatics Education in Italian Secondary Schools. ACM Transactions on
Computing Education, 14(2), pp.1–6. Available at:
http://dl.acm.org/citation.cfm?doid=2642651.2602490.

Bendig, A.W., 1954. Reliability and the number of rating-scale categories. Journal of Applied Psychology,
38(1), pp.38–40. Available at: http://psycnet.apa.org/journals/apl/38/1/38/ [Accessed April 19,
2017].

Bertacchini, F. et al., 2012. Motivating the learning of science topics in secondary school: A constructivist
edutainment setting for studying Chaos. Computers & Education, 59(4), pp.1377–1386. Available at:
http://www.sciencedirect.com/science/article/pii/S0360131512001224 [Accessed September 19,
2014].

Bessenyei, I., 2008. Learning and teaching in the information society. Elearning 2.0 and connectivism.
Revista de Informatica Sociala, 2008(9), pp.1–14. Available at:
http://www.mmiweb.org.uk/egyptianteachers/site/downloads/Besseneyi_2008.pdf.

Betrancourt, M., 2005. The Animation and Interactivity Principles in Multimedia Learning. In The
Cambridge handbook of multimedia learning. Cambridge University Press, pp. 287–296. Available
at:
http://books.google.com/books?hl=en&lr=&id=duWx8fxkkk0C&oi=fnd&pg=PA287&dq=the+anima
tion+and+interactivity+principles&ots=x8Xcx1wi6u&sig=NJWu2yDQak9mFamTVi54QzEGt0o
[Accessed January 8, 2015].

Birch, D., Sankey, M.D. & Gardiner, M.W., 2011. The impact of multiple representations of content using
multimedia on learning outcomes. International Journal of Education and Development using
Information and Communication Technology, 7(3), pp.18–35. Available at:
http://ijedict.dec.uwi.edu//viewarticle.php?id=1255.

Black, P. & Wiliam, D., 1998. Assessment and Classroom Learning. Assessment in Education: Principles,
Policy & Practice, 5(1), pp.7–74. Available at:

292
 CHAPTER NINE

http://www.tandfonline.com/doi/full/10.1080/0969595980050102 [Accessed December 4, 2017].

Black, P. & Wiliam, D., 2009. Developing the theory of formative assessment. Educational Assessment,
Evaluation and Accountability, 21(1), pp.5–31. Available at:
http://link.springer.com/article/10.1007/s11092-008-9068-5 [Accessed December 4, 2017].

Di Blas, N. & Poggi, C., 2007. European virtual classrooms: building effective “virtual” educational
experiences. Virtual Reality, 11(2–3), pp.129–143. Available at:
http://link.springer.com/10.1007/s10055-006-0060-4 [Accessed October 4, 2015].

Borrego, M., Douglas, E.P. & Amelink, C.T., 2009. Quantitative, qualitative, and Mixed Research Methods
in Engineering Education. Journal of Engineering Education, 98(1), pp.53–66.

Boud, D., 2000. Sustainable Assessment: Rethinking assessment for the learning society. Studies in
continuing education, 22(2), pp.151–167. Available at:
http://www.tandfonline.com/doi/abs/10.1080/713695728 [Accessed July 13, 2015].

Boulos, M., Maramba, I. & Wheeler, S., 2006. Wikis, blogs and podcasts: a new generation of Web-based
tools for virtual collaborative clinical practice and education. BMC medical education, 6(41).
Available at: http://www.biomedcentral.com/1472-6920/6/41. [Accessed October 11, 2015].

Boulton, H., 2008. Managing e-Learning: What Are the Real Implications for Schools?. The Electronic
Journal of e-Learning, 6(1), pp.11–18. Available at: https://eric.ed.gov/?id=EJ1098713 [Accessed
December 17, 2017].

Box, G.E.P., 1953. Non-Normality and Tests on Variances. Biometrika, 40(3/4), pp.318–335. Available at:
www.jstor.org/stable/2333350.

Boyle, F. et al., 2010. Student–student mentoring for retention and engagement in distance education.
Open Learning: The Journal of Open, Distance and e-Learning, 25(2), pp.115–130. Available at:
http://www.tandfonline.com/doi/abs/10.1080/02680511003787370 [Accessed October 23, 2014].

Bradlow, E.T. & Fitzsimons, G.J., 2001. Subscale Distance and Item Clustering Effects in Self-Administered
Surveys: A New Metric. Journal of Marketing Research, 38(2), pp.254–261. Available at:
http://journals.ama.org/doi/abs/10.1509/jmkr.38.2.254.18837 [Accessed April 19, 2017].

Brinda, T., Puhlmann, H. & Schulte, C., 2009. Bridging ICT and CS: educational standards for computer
science in lower secondary education. In ITiCSE ’09 - 14th annual ACM SIGCSE conference on
Innovation and technology in computer science education. pp. 288–292. Available at:
http://dl.acm.org/citation.cfm?id=1562965 [Accessed May 3, 2014].

Brooks, J.G. & Brooks, M.G., 1999. In Search of Understanding: The Case for Constructivist Classrooms,
with a new introduction by the authors, Alexandria, VA, USA: Association for Supervision &
Curriculum Development (ASCD).

Brown, N.C.C. et al., 2014. Restart. ACM Transactions on Computing Education, 14(2), pp.1–22. Available
at: http://dl.acm.org/citation.cfm?id=2642651.2602484.

293
 CHAPTER NINE

Brown, R., 2001. The process of community-building in distance learning classes. Journal of asynchronous
learning networks, 5(2), pp.18–35. Available at:
http://spot.pcc.edu/~rsuarez/rbs/school/EPFA_511/articles/from Erica/community building.pdf
[Accessed November 11, 2015].

Brusilovsky, P. & Millán, E., 2007. User Models for Adaptive Hypermedia and Adaptive Educational
Systems. In The Adaptive Web. Springer-Verlag Berlin Heidelberg, pp. 3–53. Available at:
http://link.springer.com/10.1007/978-3-540-72079-9%7B_%7D1 [Accessed July 11, 2018].

Butcher, K.R., 2006. Learning from text with diagrams: Promoting mental model development and
inference generation. Journal of Educational Psychology, 98(1), pp.182–197. Available at:
http://psycnet.apa.org/journals/edu/98/1/182/ [Accessed January 8, 2015].

Buzzetto-More, N. a & Alade, J.A., 2006. Best Practices in e-Assessment. Journal of Information Technology
Education, 5, pp.251–269. Available at: http://jite.org/documents/Vol5/v5p251-
269Buzzetto152.pdf.

Caballé, S., Xhafa, F. & Barolli, L., 2010. Using mobile devices to support online collaborative learning.
Mobile Information Systems, 6(1), pp.27–47. Available at:
http://iospress.metapress.com/index/L166620835601JV2.pdf [Accessed October 23, 2014].

Campbell, C. & Rozsnyai, C., 2002. Quality Assurance and the Development of Course Programmes. Papers
on Higher Education. L. C. Barrows, ed., UNESCO. Available at: http://eric.ed.gov/?id=ED475532
[Accessed June 25, 2015].

Campbell, L., Campbell, B. & Dickinson, D., 1996. Teaching & Learning through Multiple Intelligences.,
Allyn & Bacon. Available at:
http://scholar.google.com/scholar_url?url=http%3A%2F%2Feric.ed.gov%2F%3Fid%3DED415009&
hl=en&sa=T&ct=res&cd=1&ei=wjHRVI6YI6e80gG2xIG4Cg&scisig=AAGBfm284MSwAFY153M5i44l2
fYBn4gmqA&nossl=1&ws=1342x536 [Accessed February 3, 2015].

Campbell, P.F. & McCabe, G.P., 1984. Predicting the success of freshmen in a computer science major.
Communications of the ACM, 27(11), pp.1108–1113. Available at:
http://dl.acm.org/citation.cfm?id=358288 [Accessed December 12, 2017].

CAS, 2011. Computing at School International Comparisons, Available at:

http://www.computingatschool.org.uk/data/uploads/internationalcomparisons-v5.pdf [Accessed
May 27, 2015].

CAS, 2013. The re-launch of the Computing curriculum. , (June), pp.1–4. Available at:
https://www.computingatschool.org.uk/.

CAS Barefoot, 2014. Computational Thinking Concepts. , pp.1–3. Available at:

https://barefootcas.org.uk/barefoot-primary-computing-resources/concepts/computational-
thinking/ [Accessed April 24, 2018].

CBI, 2011. Building for growth : business priorities for education and skills, Available at:

294
 CHAPTER NINE

http://www.cbi.org.uk/media/1051530/cbi__edi_education___skills_survey_2011.pdf [Accessed
May 27, 2015].

Çelik, S., 2012. Development of usability criteria for e-learning content development software. Turkish
Online Journal of Distance Education, 13(2), pp.336–345. Available at:
http://eric.ed.gov/?id=EJ983666 [Accessed July 14, 2016].

Chai, C.S. & Lim, C.P., 2011. The Internet and teacher education: Traversing between the digitized world
and schools. The Internet and Higher Education, 14(1), pp.3–9. Available at:
http://www.sciencedirect.com/science/article/pii/S1096751610000345 [Accessed August 4, 2014].

Champney, H. & Marshall, H., 1939. Optimal refinement of the rating scale. Journal of Applied Psychology,
23(3), pp.323–331. Available at: http://psycnet.apa.org/journals/apl/23/3/323/ [Accessed April 9,
2017].

Chan, C.H. & Robbins, L.I., 2006. E-Learning Systems: Promises and Pitfalls. Academic Psychiatry, 30(6),
pp.491–497. Available at: http://www.ncbi.nlm.nih.gov/pubmed/17139020.

Chandler, P. & Sweller, J., 1991. Cognitive load theory and the format of instruction. Cognition and
Instruction, 8(4), pp.293–332. Available at:
http://www.tandfonline.com/doi/abs/10.1207/s1532690xci0804_2 [Accessed January 9, 2015].

Chang, S.H. & Smith, R.A., 2008. Effectiveness of Personal Interaction in a Learner-Centered Paradigm
Distance Education Class Based on Student Satisfaction. Journal of Research on Technology in
Education, 40(4), pp.407–426. Available at:
http://www.tandfonline.com/doi/abs/10.1080/15391523.2008.10782514.

Chen, N.-S. et al., 2009. Effects of high level prompts and peer assessment on online learners’ reflection
levels. Computers & Education, 52(2), pp.283–291. Available at:
http://www.sciencedirect.com/science/article/pii/S0360131508001334 [Accessed October 11,
2014].

Chi, M.T.H. et al., 1989. Self-Explanations: How Students Study and Use Examples in Learning to Solve
Problems. Cognitive Science, 13(2), pp.145–182. Available at:
http://doi.wiley.com/10.1207/s15516709cog1302_1 [Accessed February 2, 2015].

Chiong, R. & Jovanovic, J., 2012. Collaborative Learning in Online Study Groups : An Evolutionary Game
Theory Perspective. Journal of Information Technology Education : Research, 11, pp.81–101.

Churchill, G.A. & Peter, J.P., 1984. Research Design Effects on the Reliability of Rating Scales: A Meta-
Analysis. Journal of Marketing Research, 21(4), p.360. Available at:
http://www.jstor.org/stable/3151463?origin=crossref [Accessed November 16, 2017].

Chyung, Y., Winiecki, D. & Fenner, J., 1999. Evaluation of effective interventions to solve the dropout
problem in adult distance education. In B. Collins & R. Oliver, eds. Proceedings of ED-MEDIA 1999--
World Conference on Educational Multimedia, Hypermedia & Telecommunications. Association for
the Advancement of Computing in Education (AACE), pp. 51–55. Available at:

295
 CHAPTER NINE

https://www.editlib.org/index.cfm/files/paper_17397.pdf?fuseaction=Reader.DownloadFullText&
paper_id=17397 [Accessed August 8, 2018].

Clark, A.C. & Ernst, J., 2009. Gaming Research for Technology Education. Journal of STEM Education:
Innovations & Research, 10(1), pp.25–30. Available at:
http://search.ebscohost.com/login.aspx?direct=true&db=a9h&AN=51650857&site=ehost-live.

Clark, R., Nguyen, F. & Sweller, J., 2011. Efficiency in learning: Evidence-based guidelines to manage
cognitive load, Pfeiffer. Available at: https://www.wiley.com/en-
cy/Efficiency+in+Learning%3A+Evidence+Based+Guidelines+to+Manage+Cognitive+Load-p-
9781118046746 [Accessed February 2, 2015].

Clark, R.C. et al., 2006. Efficiency in learning: Evidence-based guidelines to manage cognitive load.
Performance Improvement, 45(9), pp.46–47. Available at:
http://doi.wiley.com/10.1002/pfi.4930450920 [Accessed February 2, 2015].

Clark, R.C. & Mayer, R.E., 2011. e-Learning and the Science of Instruction: Proven Guidelines for Consumers
and Designers of Multimedia Learning 3rd ed., John Wiley & Sons. Available at:
http://books.google.com/books?hl=en&lr=&id=twoLz3jlkRgC&pgis=1 [Accessed January 8, 2015].

Cobb, P., 1994. Where is the mind? Constructivist and sociocultural perspectives on mathematical
development. Educational researcher, 23(7), pp.13–20. Available at:
http://edr.sagepub.com/content/23/7/13.short [Accessed October 23, 2014].

Cockburn, A., 2008. Using Both Incremental and Iterative Development. STSC CrossTalk, 21(5), pp.27–30.
Available at: www.stsc.hill.af.mil [Accessed April 27, 2019].

Coffield, F. et al., 2004. Learning styles and pedagogy in post 16 learning. A systematic and critical review,
Available at: http://www.leerbeleving.nl/wp-content/uploads/2011/09/learning-styles.pdf.

Coffield, F. et al., 2004. Should we be using learning styles? What research has to say to practice, Learning
and Skills Research Centre. Available at: http://www.voced.edu.au/content/ngv:12401 [Accessed
December 7, 2017].

Cohen, J., 1988. Statistical power analysis for the behavioral sciences 2nd ed., Lawrence Erlbaum
Associates. Available at: http://www.sciencedirect.com/science/book/9780121790608 [Accessed
November 7, 2017].

Cole, M., 2009. Using Wiki technology to support student engagement: Lessons from the trenches.
Computers & Education, 52(1), pp.141–146. Available at:
http://www.sciencedirect.com/science/article/pii/S0360131508001061 [Accessed July 9, 2014].

Computing at School Working Group, 2012. Computer Science as a school subject: Seizing the opportunity,
Available at: http://www.computingatschool.org.uk.

Computing At School Working Group, 2009. Computing at School : the state of the nation, Available at:
http://www.computingatschool.org.uk/.

296
 CHAPTER NINE

Condie, R. & Munro, B., 2007. The impact of ICT in schools - a landscape review, Available at:
http://dera.ioe.ac.uk/1627/7/becta_2007_landscapeimpactreview_report_Redacted.pdf
[Accessed December 17, 2017].

Conklin, E., 1923. The Scale of Values Method for Studies in Genetic Psychology. University of Oregon
Publications, II, pp.1–36. Available at: http://doi.apa.org/?uid=1926-07847-001 [Accessed April 9,
2017].

Connolly, T. & Stanfield, M., 2006. Using Games-Based eLearning Technologies in Overcoming Difficulties
in Teaching Information Systems. Journal of Information Technology Education, 5, pp.459–476.
Available at: https://www.learntechlib.org/p/111557/.

Cook, D.A. et al., 2009. Measuring Motivational Characteristics of Courses: Applying Kellerʼs Instructional
Materials Motivation Survey to a Web-Based Course. Academic Medicine, 84(11), pp.1505–1509.
Available at: https://insights.ovid.com/crossref?an=00001888-200911000-00022 [Accessed July 13,
2015].

Corbalan, G., Paas, F. & Cuypers, H., 2010. Computer-based feedback in linear algebra: Effects on transfer
performance and motivation. Computers & Education, 55(2), pp.692–703. Available at:
http://www.sciencedirect.com/science/article/pii/S0360131510000722 [Accessed February 4,
2015].

Couros, A., 2009. Open, connected, social–implications for educational design. In Readings in Education
and Technology: Proceedings of ICICTE 2008. pp. 61–70. Available at:
http://www.emeraldinsight.com/journals.htm?articleid=1796100&show=abstract [Accessed
November 11, 2014].

Cousineay, D. & Chartier, S., 2011. Outliers detection and treatment: a review. International Journal of
Psychological Research, 3(1), pp.58–67. Available at:
http://www.redalyc.org/html/2990/299023509004/ [Accessed November 7, 2017].

Cox, E.P., 1980. The Optimal Number of Response Alternatives for a Scale: A Review. Journal of Marketing
Research, 17(4), p.407. Available at: http://www.jstor.org/stable/3150495 [Accessed February 15,
2017].

CPHC, 2009. A Consideration of the Issues relating to the Revision of the GCSE ICT and Computing Curricula,
Available at: https://cphc.ac.uk/.

Craig, S.D., Gholson, B. & Driscoll, D.M., 2002. Animated pedagogical agents in multimedia educational
environments: Effects of agent properties, picture features and redundancy. Journal of Educational
Psychology, 94(2), pp.428–434. Available at: http://doi.apa.org/getdoi.cfm?doi=10.1037/0022-
0663.94.2.428 [Accessed January 11, 2015].

Creswell, J., 2007. Qualitative Inquiry and research design: Choosing among five approaches 2nd ed., Sage
Publications. Available at: http://dspace.utamu.ac.ug:8080/xmlui/handle/123456789/177
[Accessed July 24, 2015].

297
 CHAPTER NINE

Creswell, J. & Plano Clark, V., 2010. Designing and Conducting Mixed Methods Research 2nd ed., Sage
Publications. Available at: http://www.amazon.com/Designing-Conducting-Mixed-Methods-
Research/dp/1412975174 [Accessed September 1, 2015].

Crick, T., 2017. FINAL DRAFT : Computing Education : An Overview of Research in the Field, Available at:
https://royalsociety.org/topics-policy/projects/computing-education/.

Cronbach, L.J., 1950. Further Evidence on Response Sets and Test Design. Educational and Psychological
Measurement, 10(1), pp.3–31. Available at:
http://journals.sagepub.com/doi/10.1177/001316445001000101 [Accessed April 11, 2017].

CSTA, 2013. Bugs in the system: Computer Science Teacher Certification in the US. , p.64. Available at:
https://www.csteachers.org/page/ResearchandReports.

Cunningham, D., 1991. Assessing Constructions and Constructing Assessments: A Dialogue. Educational
Technology, 31(5), pp.13–17. Available at:
https://www.jstor.org/stable/44427515?seq=1#page_scan_tab_contents.

Curzon, P. et al., 2014. Developing computational thinking in the classroom: a framework, Available at:
http://eprints.soton.ac.uk/369594/ [Accessed July 10, 2015].

Dabbagh, N., 2005. Pedagogical Models for E-Learning : A Theory-Based Design Framework. International
Journal of Technology in Teaching and Learning, 1(1), pp.25–44. Available at:
https://pdfs.semanticscholar.org/1a29/4c19e0a8a72697abd6e979231feeec9c0179.pdf [Accessed
December 17, 2017].

Darbyshire, P. & McDonald, H., 2004. Choosing Response Scale Labels and Length: Guidance for
Researchers and Clients. Australasian Journal of Market Research, 12(2), pp.17–26. Available at:
http://dro.deakin.edu.au/view/DU:30006485 [Accessed April 3, 2017].

David, L., 2007. Design-Based Research Methods (DBR). Learning Theories, (January). Available at:
https://www.learning-theories.com/design-based-research-methods.html. [Accessed January 4,
2018].

DCMS & DBIS, 2009. Digital Britain Final Report, Available at:
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/228844/7650.p
df [Accessed May 27, 2015].

Decker, A. & Lawley, E.L., 2013. Life’s a Game and the Game of Life: How Making a Game Out of it Can
Change Student Behavior. In Proceeding of the 44th ACM technical symposium on Computer science
education - SIGCSE ’13. New York, New York, USA: ACM Press, pp. 233–237. Available at:
http://dl.acm.org/citation.cfm?doid=2445196.2445269.

Denton, D., 2012. Enhancing Instruction through Constructivism, Cooperative learning, and Cloud
Computing. TechTrends, 56(4), pp.34–41. Available at:
http://link.springer.com/article/10.1007/s11528-012-0585-1 [Accessed November 11, 2014].

298
 CHAPTER NINE

Deterding, S. et al., 2013. Designing Gamification: Creating Gameful and Playful Experiences. In CHI ’13
Extended Abstracts on Human Factors in Computing Systems on - CHI EA ’13. New York, New York,
USA: ACM Press, pp. 3263–3266. Available at: http://dl.acm.org/citation.cfm?id=2479662
[Accessed November 23, 2015].

Deterding, S. et al., 2011. From game design elements to gamefulness. In Proceedings of the 15th
International Academic MindTrek Conference on Envisioning Future Media Environments - MindTrek
’11. New York, New York, USA: ACM Press, pp. 9–15. Available at:
http://dl.acm.org/citation.cfm?id=2181037.2181040 [Accessed July 11, 2014].

Deterding, S., 2012. Gamification: Designing for Motivation. interactions, 19(4), p.14. Available at:
http://dl.acm.org/citation.cfm?doid=2212877.2212883 [Accessed July 10, 2015].

Dewey, J., 1948. Reconstruction in philosophy, Boston: Beacon Press. Available at:
http://www.gutenberg.org/ebooks/40089.

DfE, 2017. Revised GCSE and equivalent results in England, 2014 to 2015, Available at:
https://www.gov.uk/government/statistics/revised-gcse-and-equivalent-results-in-england-2015-
to-2016.

DfE, 2014. School Workforce in England: November 2013, Available at:

https://www.gov.uk/government/collections/statistics-school-workforce.

DfE, 2011. Subject and course choices at ages 14 and 16 amongst young people in England, Available at:
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/182677/DFE-
RR160.pdf.

DfE, 2013. The National Curriculum in England Framework document for consultation, Available at:
https://www.gov.uk/national-curriculum.

DFE, 2017. GCSE new grading scale: factsheets, Available at:

https://www.gov.uk/government/publications/gcse-new-grading-scale-factsheets [Accessed
October 23, 2017].

Dillenbourg, P. et al., 1995. The evolution of research on collaborative learning. In Learning in Humans
and Machine: Towards an interdisciplinary learning science. Elsevier, Oxford, pp. p189-211.
Available at: https://telearn.archives-ouvertes.fr/hal-00190626/ [Accessed July 13, 2015].

Downes, S., 2006. Learning Networks and Connective Knowledge. In Collective Intelligence and E-Learning
2.0. IGI Global, pp. 1–26. Available at: https://philpapers.org/archive/DOWLNA [Accessed
December 6, 2017].

Downes, S., 2005. Places to Go: Connectivism & Connective Knowledge. Available at:
http://bsili.3csn.org/files/2010/06/Places_to_Go-__Connectivism__Connective_Knowledge.pdf
[Accessed November 7, 2014].

Dringus, L., 1995. An Iterative Usability Evaluation Procedure for Interactive Online Courses. Journal of

299
 CHAPTER NINE

interactive instruction development, 7(4), pp.10–14. Available at:

http://salt.org/weblink/jpdf/92.pdf [Accessed June 29, 2015].

Dringus, L. & Cohen, M., 2005. An Adaptable Usability Heuristic Checklist for Online Courses. In
Proceedings Frontiers in Education 35th Annual Conference. IEEE, p. T2H–6–T2H–11. Available at:
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1611918 [Accessed July 24, 2015].

Dron, J., 2007. Control and Constraint in E-Learning: Choosing When to Choose, IGI Global. Available at:
http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-59904-390-6 [Accessed
February 3, 2015].

Dunn, R., DeBello, T. & Brennan, P., 1981. Learning style researchers define differences differently.
Educational Leadership, 38(5), pp.372–375. Available at:
http://ascd.com/ASCD/pdf/journals/ed_lead/el_198102_dunn.pdf [Accessed February 3, 2015].

Dutta, S. & Mia, I., 2011. The global information technology report 2010–2011, Available at:
http://www.ifap.ru/library/book494.pdf [Accessed June 6, 2014].

Van Dyne, M. & Braun, J., 2014. Effectiveness of a computational thinking (CS0) course on student
analytical skills. In Proceedings of the 45th ACM technical symposium on Computer science education
- SIGCSE ’14. New York, New York, USA: ACM Press, pp. 133–138. Available at:
http://dl.acm.org/citation.cfm?doid=2538862.2538956.

e-skills, 2008. Technology Counts IT & Telecoms Insights 2008, Available at:
https://webarchive.nationalarchives.gov.uk/20160105113017/http://www.e-
skills.com/research/archive/insights-2008/.

e-skills UK, 2009. Raising the bar for ICT : Securing KS4 curriculum and relevant ICT pathways A review for
QCDA, Available at: https://webarchive.nationalarchives.gov.uk/20100116165248/http://www.e-
skills.com/Research-and-policy/2661.

e-skills UK, 2012. Technology Insight 2012. Available at: http://www.e-skills.com/research/research-

publications/insights-reports-and-videos/technology-insights-2012/ [Accessed May 27, 2015].

EDEXCEL, 2013. Pearson Edexcel GCSE Computer Science frequently asked questions,

Egger, D.L., Elsenbaumer, S.M. & Hubwieser, P., 2012. Comparing CSTA K-12 computer science standards
with Austrian curricula. In Proceedings of the 7th Workshop in Primary and Secondary Computing
Education on - WiPSCE ’12. New York, New York, USA: ACM Press, p. 125. Available at:
http://doi.acm.org/10.1145/2481449.2481479.

Ejiwale, J.A., 2012. Facilitating Teaching and Learning Across STEM Fields. Journal of STEM Education:
Innovations and Research, 13(3), pp.87–94. Available at:
http://ojs.jstem.org/index.php?journal=JSTEM&page=article&op=view&path[]=1711.

Elaine, A. & Seaman, J., 2011. Going the Distance - Online Education in the united state 2011, Available at:
http://files.eric.ed.gov/fulltext/ED529948.pdf [Accessed October 25, 2015].

300
 CHAPTER NINE

Ericson, B. et al., 2016. State-Based Progress Towards Computer Science for All. ACM Inroads, 7(4), pp.57–
60. Available at: http://dl.acm.org/citation.cfm?doid=3018995.2994607.

Ericsson, K., 2006. The influence of experience and deliberate practice on the development of superior
expert performance. The Cambridge handbook of expertise and expert performance. Available at:
http://www.skillteam.se/wp-content/uploads/2011/12/Ericsson_delib_pract.pdf [Accessed
February 4, 2015].

Evergreen Education Group, 2017. Keeping Pace with K-12 Online Learning, Available at:
https://static1.squarespace.com/static/59381b9a17bffc68bf625df4/t/593efc779f745684e6ccf4d8
/1497300100709/EEG_KP2016-web.pdf [Accessed October 28, 2018].

Fadel, C. & Lemke, C., 2008. Multimodal learning through media: What the research says. San Jose, CA:
CISCO Systems. Retrieved October. Available at:
http://scholar.google.com/scholar?hl=en&q=multimodal+learning+through+media&btnG=&as_sd
t=1%2C5&as_sdtp=#1 [Accessed July 13, 2015].

Falkner, K. & Vivian, R., 2015. A review of Computer Science resources for learning and teaching with K-
12 computing curricula: an Australian case study. Computer Science Education, 25(4), pp.390–429.
Available at: http://www.tandfonline.com/doi/full/10.1080/08993408.2016.1140410 [Accessed
December 14, 2017].

Felder, R. & Silverman, L., 1988. Learning and teaching styles in Engineering Education. In Engineering
Education. pp. 674–681. Available at: http://www.academia.edu/download/31039406/LS-
1988.pdf.

Ferguson, L.W., 1941. a Study of the Likert Technique of Attitude Scale Construction. Journal of Social
Psychology, 13(1), pp.51–57. Available at:
http://www.tandfonline.com/doi/pdf/10.1080/00224545.1941.9714060 [Accessed April 19, 2017].

Field, A.P., 2013. Discovering Statistics Using IBM SPSS Statistics 4th ed., Sage Publications.

Field, A.P. & Hole, G., 2003. How to design and report experiments, Sage publications London. Available
at:
https://books.google.com.cy/books?id=IqeHAwAAQBAJ&printsec=frontcover&dq=How+to+design
+and+report+experiments&hl=en&sa=X&ved=0ahUKEwjC9Zum7ZrXAhWHfxoKHaqRASUQ6AEIKzA
B#v=onepage&q=How to design and report experiments&f=false [Accessed October 31, 2017].

Fleming, N.D., 2012. Facts, Fallacies and Myths: VARK and Learning Preferences Learning Style or Learning
Preferences ? Available at: http://vark-learn.com/wp-content/uploads/2014/08/Some-Facts-
About-VARK.pdf [Accessed June 28, 2018].

Fleming, N.D., 1995. I’m different; not dumb. Modes of presentation (VARK) in the tertiary classroom.
Research and Development in Higher Education, Proceedings of the Annual Conference of the Higher
Education and Research Development Society of Australasia, 18, pp.308–313. Available at:
http://www2.ccga.edu/TutorTrain/files/different_not_dumb.pdf.

301
 CHAPTER NINE

Fleming, N.D. & Baume, D., 2006. Learning Styles Again: VARKing up the right tree! Educational
Developments, 7(4), pp.4–7. Available at: http://vark-learn.com/wp-
content/uploads/2014/08/Educational-Developments.pdf [Accessed June 28, 2018].

Fleming, N.D. & Mills, C., 1992. Not another inventory, rather a catalyst for reflection. To Improve the
academy, 11, pp.137–155.

Fletcher, G., Schaffhauser, D. & Levin, D., 2012. Out of Print: Reimagining the K-12 Textbook in a Digital
Age., Available at: http://files.eric.ed.gov/fulltext/ED536747.pdf [Accessed March 27, 2014].

Fletcher, J.D. & Tobias, S., 2005. The Multimedia Principle. In R. Mayer, ed. The Cambridge Handbook of
Multimedia Learning. Cambridge: Cambridge University Press, pp. 117–134. Available at:
https://www.cambridge.org/core/product/identifier/CBO9780511816819A015/type/book_part
[Accessed January 8, 2015].

Fowler, R.L., 1987. Power and Robustness in Product-Moment Correlation. Applied Psychological
Measurement, 11(4), pp.419–428. Available at:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.958.9366&rep=rep1&type=pdf
[Accessed October 25, 2017].

Franklin, R.E., 1987. What academic impact are high school computing courses having on the entry-level
college computer science curriculum? In Proceedings of the eighteenth SIGCSE technical symposium
on Computer science education - SIGCSE ’87. New York, New York, USA: ACM Press, pp. 253–256.
Available at: http://portal.acm.org/citation.cfm?doid=31820.31768 [Accessed December 12, 2017].

Franklin, S., 2006. VAKing out learning styles—why the notion of ‘learning styles’ is unhelpful to teachers.
Education 3-13, 34(1), pp.81–87. Available at:
http://www.tandfonline.com/doi/abs/10.1080/03004270500507644 [Accessed July 4, 2018].

de Freitas, A.A. & de Freitas, M.M., 2013. Classroom Live: A software-assisted gamification tool. Computer
Science Education, 23(2), pp.186–206.

Gal-ezer, J. & Stephenson, C., 2014. A Tale of Two Countries: Successes and Challenges in K-12 Computer
Science Education in Israel and the United States. ACM Transactions on Computing Education, 14(2),
pp.169–170.

Gardner, H., 2009. Multiple approaches to understanding. In K. Illeris, ed. Contemporary Theories of
Learning: Learning theorists... in their own words. Routledge - Taylor & Francis Group, pp. 106–115.

Garland, R., 1991. The mid-point on a rating scale: Is it desirable. Marketing bulletin, (2), pp.66–70.
Available at: http://marketing-bulletin.massey.ac.nz/V2/MB_V2_N3_Garland.pdf [Accessed
November 18, 2016].

Ghahrai, A., 2018. Iterative Model in Software Development and Testing. Available at:
https://www.testingexcellence.com/iterative-model/ [Accessed April 27, 2019].

Gilbert, J., Morton, S. & Rowley, J., 2007. e‐Learning: The student experience. British Journal of

302
 CHAPTER NINE

Educational Technology, 38(4), pp.560–573. Available at:

http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8535.2007.00723.x/pdf [Accessed July 13,
2015].

Ginns, P., 2006. Integrating information: A meta-analysis of the spatial contiguity and temporal contiguity
effects. Learning and Instruction, (16), pp.511–525. Available at:
http://www.sciencedirect.com/science/article/pii/S0959475206000806 [Accessed January 11,
2015].

Golafshani, N., 2003. Understanding Reliability and Validity in Qualitative Research. The Qualitative
Report, 8(4), pp.597–606. Available at: http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf.

Gouws, L.A., Bradshaw, K. & Wentworth, P., 2013. Computational thinking in educational activities.
Proceedings of the 18th ACM conference on Innovation and technology in computer science
education - ITiCSE ’13, p.10. Available at: http://dl.acm.org/citation.cfm?doid=2462476.2466518.

Gove, M., 2012. Michael Gove gives a speech at the BETT Show 2012 on ICT in the National Curriculum.
Available at: https://www.gov.uk/government/speeches/michael-gove-speech-at-the-bett-show-
2012 [Accessed May 27, 2015].

Govindasamy, T., 2001. Successful implementation of e-Learning. The Internet and Higher Education, 4(3–
4), pp.287–299. Available at:
http://www.sciencedirect.com/science/article/pii/S1096751601000719.

Gray, S. et al., 2007. Suggestions for graduated exposure to programming concepts using fading worked
examples. In Proceedings of the third international workshop on Computing education research -
ICER ’07. New York, New York, USA: ACM Press, p. 99. Available at:
http://portal.acm.org/citation.cfm?doid=1288580.1288594.

Greenwood, D. & Levin, M., 2007. Introduction to action research: social research for social change, Sage
Publications. Available at: http://library.wur.nl/WebQuery/clc/1850731 [Accessed September 10,
2015].

Grover, S. & Pea, R., 2013. Computational Thinking in K–12 A Review of the State of the Field. Educational
Researcher, 42(1), pp.38–43. Available at: http://edr.sagepub.com/content/42/1/38.short
[Accessed July 10, 2015].

Grutzner, I., Weibelzahl, S. & Waterson, P., 2004. Improving courseware quality through life-cycle
encompassing quality assurance. In ACM Symposium on Applied Computing. pp. 946–951. Available
at: http://dl.acm.org/citation.cfm?id=968092 [Accessed May 11, 2017].

Guba, E. & Lincoln, Y., 1989. Fourth generation evaluation, Sage Publications. Available at:
http://scholar.google.com/scholar?q=fourth+generation+evaluation&btnG=&hl=en&as_sdt=0%2C
5#0 [Accessed August 19, 2015].

Guba, E. & Lincoln, Y., 2005. Paradigmatic Controversies, Constradictions and Emerging Confluences. In
The Sage Handbook of Qualitative Research. Sage Publications, pp. 191–215.

303
 CHAPTER NINE

Guzdial, M., 2008. Education Paving the way for computational thinking. Communications of the ACM.
Available at: http://dl.acm.org/citation.cfm?id=1378713 [Accessed July 10, 2015].

Guzdial, M. et al., 2014. Georgia Computes! An Intervention in a US State, with Formal and Informal
Education in a Policy Context. ACM Transactions on Computing Education, 14(2), pp.1–29. Available
at: http://dl.acm.org/citation.cfm?doid=2642651.2602488.

Haaranen, L. et al., 2014. How (not) to introduce badges to online exercises. In Proceedings of the 45th
ACM technical symposium on Computer science education - SIGCSE ’14. New York, New York, USA:
ACM Press, pp. 33–38. Available at: http://dl.acm.org/citation.cfm?doid=2538862.2538921.

Hadjerrouit, S., 2010. A conceptual framework for using and evaluating web-based learning resources in
school education. Journal of Information Technology Education: Research, 9(1), pp.53–79. Available
at: https://www.learntechlib.org/p/111356/.

Hadjerrouit, S., 2005. Constructivism as guiding philosophy for software engineering education. ACM
SIGCSE Bulletin, 37(4), p.45. Available at:
http://portal.acm.org/citation.cfm?doid=1113847.1113875.

Halpern, E., 1983. Auditing Naturalistic Inquiries: Some Preliminary Applications. Part 1: Development of
the Process. Part 2: Case Study Application. Available at: http://eric.ed.gov/?id=ED234084
[Accessed September 9, 2015].

Halse, M.. M., Mallinson, B. & Mallison, B.J., 2009. Investigating popular Internet applications as
supporting e-learning technologies for teaching and learning with Generation Y. International
Journal of Education and Development using ICT, 5(5), pp.58–71.

Hamari, J., Koivisto, J. & Sarsa, H., 2014. Does Gamification Work? -- A Literature Review of Empirical
Studies on Gamification. In 2014 47th Hawaii International Conference on System Sciences. IEEE, pp.
3025–3034. Available at: http://ieeexplore.ieee.org/document/6758978/ [Accessed July 10, 2015].

Hancock, G.R. & Klockars, A.J., 1991. The effect of scale manipulations on validity: targetting frequency
rating scales for anticipated performance levels. Applied ergonomics, 22(3), pp.147–54. Available
at: http://www.ncbi.nlm.nih.gov/pubmed/15676809 [Accessed April 19, 2017].

Harp, S. & Mayer, R., 1998. How seductive details do their damage: A theory of cognitive interest in science
learning. Journal of Educational Psychology, (90), pp.414–434. Available at:
http://psycnet.apa.org/journals/edu/90/3/414/ [Accessed January 15, 2015].

Harskamp, E., Mayer, R. & Suhre, C., 2007. Does the modality principle for multimedia learning apply to
science classrooms? Learning and Instruction, (17), pp.465–477. Available at:
http://www.sciencedirect.com/science/article/pii/S0959475207000916 [Accessed January 11,
2015].

Harvey, L. & Green, D., 1993. Defining quality. Assessment & evaluation in higher education, 18(1), pp.9–
34. Available at: http://www.tandfonline.com/doi/abs/10.1080/0260293930180102 [Accessed
June 25, 2015].

304
 CHAPTER NINE

Hattie, J., 2009. Visible learning: A synthesis of over 800 meta-analyses relating to achievement, Routledge
- Taylor & Francis Group. Available at:
https://books.google.com/books?hl=en&lr=&id=ZO8jmUjQbs0C&oi=fnd&pg=PR5&dq=visible+lear
ning:+a+synthesis&ots=1-LzhmX-rJ&sig=huQ3lO-2eoXMa--KWxLme2HmGIU [Accessed February 3,
2015].

Hazzan, O., Gal-ezer, J. & Blum, L., 2008. A Model for High School Computer Science Education : The Four
Key Elements that Make It ! In Proceedings of the 39th SIGCSE technical symposium on Computer
science education -. New York, New York, USA: ACM Press, p. p281. Available at:
http://dl.acm.org/citation.cfm?id=1352135.1352233 [Accessed April 16, 2014].

Hegarty, M., 2004. Dynamic visualizations and learning: Getting to the difficult questions. Learning and
Instruction, (14), pp.343–351. Available at:
http://scholar.google.com/scholar?q=dynamic+visualizations+and+learning&btnG=&hl=en&as_sdt
=0%2C5#0 [Accessed January 8, 2015].

Hemmendinger, D., 2010. A plea for modesty. ACM Inroads, 1(2), p.4. Available at:
http://dl.acm.org/citation.cfm?doid=1805724.1805725 [Accessed November 29, 2015].

Herman, G.L., 2012. Designing contributing student pedagogies to promote students’ intrinsic motivation
to learn. Computer Science Education, 22(4), pp.369–388.

Herzberg, F., 1987. One More Time: How Do You Motivate Employees? Harvard Business Review,
81(September-October), pp.5–16. Available at:
https://numerons.files.wordpress.com/2012/04/how-do-you-motivate-employees-frederick-
herzberg.pdf [Accessed May 9, 2017].

Hidi, S. & Harackiewicz, J., 2000. Motivating the academically unmotivated: A critical issue for the 21st
century. Review of educational research, 70(2), pp.151–179. Available at:
http://rer.sagepub.com/content/70/2/151.short [Accessed February 2, 2015].

Hillman, D.C.A., Willis, D.J. & Gunawardena, C.N., 1994. Learner‐interface interaction in distance
education: An extension of contemporary models and strategies for practitioners. American Journal
of Distance Education, 8(2), pp.30–42. Available at:
http://www.tandfonline.com/doi/abs/10.1080/08923649409526853 [Accessed February 3, 2015].

Hinchey, P., 2008. Action research primer, New York: Peter Lang. Available at:
http://books.google.com/books?hl=en&lr=&id=ARN8vNkRNhEC&oi=fnd&pg=PA1&dq=action+rese
arch:+primer+hinchey&ots=K75Jl3i-BE&sig=d-BakHyrwBvyMiMeTSCAEeYpoAM [Accessed
September 10, 2015].

Honey, P. & Mumford, A., 2000. The learning styles helper’s guide, Peter Honey Publications. Available at:
https://www.amazon.com/Learning-Styles-Helpers-Guide/dp/1902899105.

Hsu, C., Yeh, Y. & Yen, J., 2009. Development of design criteria and evaluation scale for web-based learning
platforms. International Journal of Industrial Ergonomics, 39(1), pp.90–95. Available at:

305
 CHAPTER NINE

http://www.sciencedirect.com/science/article/pii/S0169814108001388 [Accessed July 14, 2016].

Hubbs, D. & Brand, C., 2005. The paper mirror: Understanding reflective journaling. Journal of Experiential
Education, 28(1), pp.70–71. Available at: http://jee.sagepub.com/content/28/1/60.short [Accessed
February 2, 2015].

Hubwieser, P. et al., 2011. Computer science/informatics in secondary education. In Proceedings of the

16th annual conference reports on Innovation and technology in computer science education -
working group reports - ITiCSE-WGR ’11. New York, New York, USA: ACM Press, p. 19. Available at:
http://dl.acm.org/citation.cfm?doid=2078856.2078859 [Accessed December 15, 2017].

Huffaker, D.A. & Calvert, S.L., 2003. The New Science of Learning: Active Learning, Metacognition, and
Transfer of Knowledge in E-Learning Applications. Journal of Educational Computing Research,
29(3), pp.325–334. Available at:
http://baywood.metapress.com/openurl.asp?genre=article&id=doi:10.2190/4T89-30W2-DHTM-
RTQ2.

Humphreys, P., Greenan, K. & McIlveen, H., 1997. Developing work‐based transferable skills in a university
environment. Journal of European Industrial Training, 21(2), pp.63–69. Available at:
http://www.emeraldinsight.com/doi/abs/10.1108/03090599710161739 [Accessed February 2,
2015].

Illeris, K., 2009. Contemporary Theories of Learning: Learning Theorists... In Their Own Words K. Illeris, ed.,
New York: Routledge - Taylor & Francis Group.

Illeris, K., 2003. Towards a contemporary and comprehensive theory of learning. International Journal of
Lifelong Education, 22(4), pp.396–406.

Inostroza, R. & Rusu, C., 2012. Usability Heuristics Validation through Empirical Evidences: A Touchscreen-
Based Mobile Devices Proposal. 2012 31st International Conference of the. IEEE. Available at:
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6694074 [Accessed June 23, 2015].

Iosup, A. & Epema, D., 2014. An experience report on using gamification in technical higher education.
Proceedings of the 45th ACM technical symposium on Computer science education - SIGCSE ’14,
(2008), pp.27–32. Available at: http://dl.acm.org/citation.cfm?doid=2538862.2538899.

ITU, 2017a. ICT facts and figures 2017, Available at: http://www.itu.int/en/ITU-
D/Statistics/Documents/facts/ICTFactsFigures2017.pdf.

ITU, 2013. Measuring the Information Society, Available at: http://www.itu.int/en/ITU-

D/Statistics/Pages/publications/mis2013.aspx.

ITU, 2017b. Measuring the Information Society Report 2017 - Volume 1, Available at:
http://www.itu.int/en/ITU-D/Statistics/Pages/publications/mis2017.aspx.

ITU, 2014. The World in 2014 ICT Facts and figures, Available at: https://www.itu.int/en/ITU-
D/Statistics/Documents/facts/ICTFactsFigures2014-e.pdf [Accessed May 27, 2015].

306
 CHAPTER NINE

Ivankova, N., 2014. Applying Mixed Methods in Action Research. In Mixed Methods Applications in Action
Research. SAGE Publications Ltd, p. 472. Available at: https://uk.sagepub.com/en-gb/eur/mixed-
methods-applications-in-action-research/book237798#contents [Accessed August 5, 2015].

IVETIC, D. & Petrović, V., 2012. Gamifying education: A proposed taxonomy of satisfaction metrics. In
Conference proceedings of ‘eLearning and Software for Education’. pp. 345–350. Available at:
http://www.ceeol.com/aspx/getdocument.aspx?logid=5&id=558f18e72e3249aa983cf01dba4b4b9
5 [Accessed November 23, 2015].

Jahoda, M. et al., 1951. Research methods in social relations, New York Dryden Press. Available at:
http://www.sidalc.net/cgi-
bin/wxis.exe/?IsisScript=IICACR.xis&B1=Buscar&formato=1&cantidad=50&expresion=METODOS
DE INVESTIGACION [Accessed April 19, 2017].

James, E.A., Milenkiewicz, M.T. & Bucknam, A., 2007. Participatory Action Research for Educational
Leadership: Using Data-Driven Decision Making to Improve Schools. SAGE Publications (CA).
Available at: http://eric.ed.gov/?id=ED500337 [Accessed September 10, 2015].

Jamet, E. & Le Bohec, O., 2007. The effect of redundant text in multimedia instruction. Contemporary
Educational Psychology, 32(4), pp.588–598. Available at:
http://www.sciencedirect.com/science/article/pii/S0361476X06000294 [Accessed January 13,
2015].

Jang, S.-J., 2009. Exploration of secondary students’ creativity by integrating web-based technology into
an innovative science curriculum. Computers & Education, 52(1), pp.247–255. Available at:
http://www.sciencedirect.com/science/article/pii/S0360131508001309 [Accessed October 4,
2015].

Jarmon, L. et al., 2009. Virtual world teaching, experiential learning, and assessment: An interdisciplinary
communication course in Second Life. Computers & Education, 53(1), pp.169–182. Available at:
http://www.sciencedirect.com/science/article/pii/S0360131509000141 [Accessed July 15, 2014].

Jenkins, J. et al., 2012. Perspectives on active learning and collaboration. Proceedings of the 43rd ACM
technical symposium on Computer Science Education - SIGCSE ’12, p.185. Available at:
http://dl.acm.org/citation.cfm?doid=2157136.2157194.

Johnson, D. & Johnson, R., 1996. Cooperation and the Use of Technology. In Handbook of Research on
Educational Communications and Technology. Taylor and Francis, pp. 401–424. Available at:
http://books.google.com/books?hl=en&lr=&id=GgCPAgAAQBAJ&oi=fnd&pg=PA401&dq=David+Jo
hnson+cooperation&ots=A8X1JRS4r5&sig=dSIBa5B_i2JY8QVy03U1FpqupRk [Accessed July 13,
2015].

Johnson, D.W. et al., 1984. Circles of learning. Cooperation in the classroom., ASCD Publications. Available
at: http://eric.ed.gov/?id=ed241516 [Accessed July 13, 2015].

Johnson, R. & Onwuegbuzie, A., 2004. Mixed methods research: A research paradigm whose time has

307
 CHAPTER NINE

come. Educational Researcher, 33(7), pp.14–26. Available at:

http://edr.sagepub.com/content/33/7/14.short.

Jonassen, D. & Hung, W., 2008. All problems are not equal: Implications for problem-based learning.
Interdisciplinary Journal of Problem-based Learning, 2(2), pp.6–28. Available at:
http://books.google.com/books?hl=en&lr=&id=KhF-
BgAAQBAJ&oi=fnd&pg=PA17&dq=All+Problems+are+Not+Equal:+Implications+for+Problem-
Based+Learning&ots=avk9t0mBZu&sig=cL1KMgSqkO7bR2yllhlliuKidr0 [Accessed July 13, 2015].

Jonassen, D.H., 1999. Designing Constructivist Learning Environments. In C. M. Reigeluth, ed.

Instructional-Design Theories and Models. Routledge - Taylor & Francis Group, pp. 215–239.
Available at: http://www.davidlewisphd.com/courses/EDD8121/readings/1999-Jonassen.pdf
[Accessed October 3, 2014].

Jones, J., Mccowan, D. & Stephenson, C., 2003. Curriculum for K – 12 Computer Science : Final Report,

Jones, R.R., 1968. Differences in response consistency and subjects’ preferences for three personality
inventory response formats. In Proceedings of the 76th Annual Convention of the APA. pp. 247–248.

Jung, I., 2011. The dimensions of e-learning quality: from the learner’s perspective. Educational
Technology Research and Development, 59(4), pp.445–464. Available at:
http://link.springer.com/article/10.1007/s11423-010-9171-4 [Accessed June 28, 2015].

Kafai, Y.B. & Vasudevan, V., 2015. Constructionist gaming beyond the screen: Middle school students’
crafting and computing of touchpads, board games, and controllers. Proceedings of the 2015
Workshop in Primary and Secondary Computing Education, pp.49–54. Available at:
http://dl.acm.org/citation.cfm?id=2818334 [Accessed December 14, 2017].

Kalapesi, C., Willersdorf, S. & Zwillenberg, P., 2010. The Connected Kingdom How the Internet Is
Transforming the U.K. Economy,

Kalyuga, S., 1999. Managing split-attention and redundancy in multimedia instruction. Applied Cognitive
Psychology, (13), pp.351–372. Available at:
http://visuallearningresearch.wiki.educ.msu.edu/file/view/Kalyuga,+Chandler,+%26+Sweller+(199
9).pdf/47290509/Kalyuga,+Chandler,+%26+Sweller+(1999).pdf [Accessed January 13, 2015].

Kalyuga, S., 2005. Prior knowledge principle in multimedia learning. The Cambridge handbook of
multimedia learning, pp.325–337. Available at:
http://books.google.com/books?hl=en&lr=&id=duWx8fxkkk0C&oi=fnd&pg=PA325&dq=prior+kno
wledge+principle+in+multimedia&ots=x8Xcx1si9x&sig=37dNlkf6NOZr8uQrXKHmlohOzvg
[Accessed January 8, 2015].

Kalyuga, S. & Ayres, P., 2003. The expertise reversal effect. Educational Psychologist, (38), pp.23–31.
Available at: http://www.tandfonline.com/doi/abs/10.1207/S15326985EP3801_4 [Accessed
January 8, 2015].

Kalyuga, S., Chandler, P. & Sweller, J., 2000. Incorporating learner experience into the design of

308
 CHAPTER NINE

multimedia instruction. Journal of Educational Psychology, (92), pp.126–136. Available at:

http://psycnet.apa.org/journals/edu/92/1/126/ [Accessed January 13, 2015].

Kang, M. et al., 2010. Developing an Educational Performance Indicator for New Millennium Learners.
Journal of Research on Technology in Education, 43(2), pp.157–170. Available at:
http://www.tandfonline.com/doi/abs/10.1080/15391523.2010.10782567.

Karagiorgi, Y. & Symeou, L., 2005. Translating Constructivism into Instructional Design: Potential and
Limitations. Educational Technology & Society, 8(1), pp.17–27. Available at: http://www.e-
biblioteka.lt/resursai/Uzsienio leidiniai/IEEE/English/2006/Volume 8/Issue
1/Jets_v8i1.pdf#page=22 [Accessed October 2, 2014].

Katsioloudis, P. & Fantz, T.D., 2012. A Comparative Analysis of Preferred Learning and Teaching Styles for
Engineering, Industrial, and Technology Education Students and Faculty. Journal of Technology
Education, 23(2), pp.61–69. Available at:
http://scholar.lib.vt.edu/ejournals/JTE/v23n2/katsioloudis1.html.

Keller, J., 1987. Development and use of the ARCS model of instructional design. Journal of Instructional
Development, 10(3), pp.2–10. Available at: http://link.springer.com/10.1007/BF02905780
[Accessed March 18, 2015].

Keller, J., 2006. Development of two measures of learner motivation. Unpublished Manuscript in progress.
Florida State University. Available at:
http://scholar.google.com/scholar?q=Development+of+Two+Measures+of+Learner+Motivation&
btnG=&hl=en&as_sdt=0%2C5#0 [Accessed July 13, 2015].

Keller, J. & Suzuki, K., 2004. Learner motivation and E-learning design: A multinationally validated process.
Journal of Educational Media, 29(3), pp.229–239. Available at:
http://www.tandfonline.com/doi/abs/10.1080/1358165042000283084 [Accessed July 13, 2015].

Keller, J.M., 2008. First principles of motivation to learn and e 3 ‐learning. Distance Education, 29(2),
pp.175–185. Available at: http://www.tandfonline.com/doi/abs/10.1080/01587910802154970
[Accessed July 13, 2015].

Keller, J.M., 1987a. Strategies for stimulating the motivation to learn. Performance + Instruction, 26(8),
pp.1–7. Available at: http://doi.wiley.com/10.1002/pfi.4160260802.

Keller, J.M., 1987b. The systematic process of motivational design. Performance + Instruction, 26(9–10),
pp.1–8. Available at: http://onlinelibrary.wiley.com/doi/10.1002/pfi.4160260902/abstract
[Accessed July 13, 2015].

Kellogg, R. & Whiteford, A., 2009. Training advanced writing skills: The case for deliberate practice.
Educational Psychologist, 44(4), pp.250–266. Available at:
http://www.tandfonline.com/doi/abs/10.1080/00461520903213600 [Accessed February 3, 2015].

Kester, L., Kirschner, P.A. & Van Merriënboer, J.J.G., 2006. Just-in-time information presentation:
Improving learning a troubleshooting skill. Contemporary Educational Psychology, 31(2), pp.167–

309
 CHAPTER NINE

185. Available at: http://www.sciencedirect.com/science/article/pii/S0361476X05000299

[Accessed February 4, 2015].

Khenner, E. & Semakin, I., 2014. School Subject Informatics (Computer Science) in Russia. ACM
Transactions on Computing Education, 14(2), pp.1–10. Available at:
http://dl.acm.org/citation.cfm?doid=2642651.2602489 [Accessed December 15, 2017].

Kim, A.J., 2011. Gamification 101: Design the Player Journey. Available at:
http://www.slideshare.net/amyjokim/gamification-101-design-the-player-journey.

Kim, C., Park, S.W. & Cozart, J., 2014. Affective and motivational factors of learning in online mathematics
courses. British Journal of Educational Technology, 45(1), pp.171–185. Available at:
http://doi.wiley.com/10.1111/j.1467-8535.2012.01382.x [Accessed July 13, 2015].

Kirkpatrick, D., 1967. Evaluation of training R. L. Craig & L. R. Bittel, eds., New York: McGraw-Hill. Available
at: http://files.eric.ed.gov/fulltext/ED057208.pdf#page=41 [Accessed June 2, 2017].

Kirschner, F., Paas, F. & Kirschner, P.A., 2009. A Cognitive Load Approach to Collaborative Learning: United
Brains for Complex Tasks. Educational Psychology Review, 21(1), pp.31–42. Available at:
http://link.springer.com/article/10.1007/s10648-008-9095-2 [Accessed February 4, 2015].

Kirschner, P.A., Sweller, J. & Clark, R.E., 2006. Why Minimal Guidance During Instruction Does Not Work:
An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-
Based Teaching. Educational Psychologist, 41(2), pp.75–86. Available at:
http://www.tandfonline.com/doi/pdf/10.1207/s15326985ep4102_1?needAccess=true [Accessed
November 30, 2017].

Klein, P.D., 2003. Rethinking the multiplicity of cognitive resources and curricular representations:
Alternatives to ‘learning styles’ and ‘multiple intelligences’. Journal of Curriculum Studies, 35(1),
pp.45–81. Available at: http://www.tandfonline.com/doi/abs/10.1080/00220270210141891
[Accessed February 8, 2014].

Knez, I. & Hygge, S., 2002. Irrelevant speech and indoor lighting: effects on cognitive performance and
self‐reported affect. Applied Cognitive Psychology, (15), pp.709–718. Available at:
http://onlinelibrary.wiley.com/doi/10.1002/acp.829/abstract [Accessed January 15, 2015].

Knight, C., Knight, B. & Teghe, D., 2006. Releasing the pedagogical power of information and
communication technology for learners: A case study. International Journal of Education and
Development using Information and Communication Technology, 2(2), pp.27–34. Available at:
http://ijedict.dec.uwi.edu/viewarticle.php?id=167&layout=html [Accessed August 26, 2014].

Koh, G. et al., 2008. The effects of problem-based learning during medical school on physician
competency: a systematic review. Canadian Medical Association Journal, 178(1), pp.34–41.
Available at: http://www.cmaj.ca/content/178/1/34.short [Accessed February 3, 2015].

Kolb, D., 1984. The Process of Experiential Learning. In Experiential learning: Experience as the source of
learning and development. Available at:

310
 CHAPTER NINE

http://scholar.google.com/scholar?q=Experiential+learning%3A+experience+as+the+source+of+le
arning+and+development&btnG=&hl=en&as_sdt=0%2C5#0 [Accessed July 13, 2015].

Komorita, S.S., 1963. Attitude Content, Intensity, and the Neutral Point on a Likert Scale. The Journal of
Social Psychology, 61(2), pp.327–334. Available at:
http://www.tandfonline.com/doi/pdf/10.1080/00224545.1963.9919489 [Accessed April 19, 2017].

Kop, R. & Hill, A., 2008. Connectivism: Learning theory of the future or vestige of the past? The
International Review of Research in Open and Distance Learning, 9(3), pp.1–13. Available at:
http://www.irrodl.org/index.php/irrodl/article/view/523/1121 [Accessed November 7, 2014].

Koshy, E., Koshy, V. & Waterman, H., 2011. Action Research in Healthcare, SAGE Publications Ltd. Available
at: http://library.wur.nl/WebQuery/clc/1850731 [Accessed September 10, 2015].

Krathwohl, D., 2002. A revision of Bloom’s taxonomy: An overview. Theory into practice. Available at:
http://www.tandfonline.com/doi/pdf/10.1207/s15430421tip4104_2 [Accessed September 18,
2016].

Krosnick, J.A., 1999. Survey research. In Annual Review of Psychology. pp. 537–567. Available at:
https://www.annualreviews.org/doi/abs/10.1146/annurev.psych.50.1.537.

Lanzilotti, R. & Ardito, C., 2006. eLSE methodology: a systematic approach to the e-learning systems
evaluation. Journal of Educational Technology and Society, 9(4), pp.42–53. Available at:
http://www.jstor.org/stable/jeductechsoci.9.4.42 [Accessed July 14, 2015].

Larman, C. & Basili, V.R., 2003. Iterative and incremental developments. a brief history. Computer, 36(6),
pp.47–56. Available at: http://ieeexplore.ieee.org/document/1204375/ [Accessed April 27, 2019].

Law, E. & Hvannberg, E., 2004. Analysis of strategies for improving and estimating the effectiveness of
heuristic evaluation. Proceedings of the third Nordic conference on Human-computer interaction.
Available at: http://dl.acm.org/citation.cfm?id=1028051 [Accessed June 23, 2015].

Lazar, J., Feng, J. & Hochheiser, H., 2010. Research Methods in Human-Computer Interaction, John Wiley
& Sons. Available at: http://www.amazon.com/Research-Methods-Human-Computer-Interaction-
Jonathan/dp/0470723378/ref=sr_1_1?s=books&ie=UTF8&qid=1431248958&sr=1-
1&keywords=research+methods+in+human-computer+interaction [Accessed May 10, 2015].

Lebow, D., 1993. Constructivist values for instructional systems design: Five principles toward a new
mindset. Educational Technology Research and Development, 41(3), pp.4–16. Available at:
http://link.springer.com/10.1007/BF02297354 [Accessed October 3, 2014].

Lee, I. et al., 2011. Computational thinking for youth in practice. ACM Inroads, 2(1), p.32. Available at:
http://dl.acm.org/ft_gateway.cfm?id=1929902&type=html [Accessed May 8, 2015].

Lee, M.J.W., Pradhan, S. & Dalgarno, B., 2008. The effectiveness of screencasts and cognitive tools as
scaffolding for novice object-oriented programmers. Journal of Information Technology Education:
Research, 7(1), pp.61–80.

311
 CHAPTER NINE

Lee, Y. & Choi, J., 2011. A review of online course dropout research: implications for practice and future
research. Educational Technology Research and Development, 59(5), pp.593–618. Available at:
http://link.springer.com/article/10.1007/s11423-010-9177-y [Accessed October 25, 2015].

LeFevre, J.-A. & Dixon, P., 1986. Do Written Instructions Need Examples? Cognition and Instruction, 3(1),
pp.1–30. Available at: http://www.tandfonline.com/doi/abs/10.1207/s1532690xci0301_1
[Accessed February 2, 2015].

Lehman, S. & Schraw, G., 2007. Processing and recall of seductive details in scientific text. Contemporary
Educational Psychology, (32), pp.569–587. Available at:
http://www.sciencedirect.com/science/article/pii/S0361476X06000300 [Accessed January 15,
2015].

Leite, W.L., Svinicki, M. & Shi, Y., 2010. Attempted validation of the scores of the VARK: Learning styles
inventory with multitrait-multimethod confirmatory factor analysis models. Educational and
Psychological Measurement, 70(2), pp.323–339. Available at:
http://journals.sagepub.com/doi/pdf/10.1177/0013164409344507 [Accessed June 28, 2018].

Lepper, M.R., Corpus, J.H. & Iyengar, S.S., 2005. Intrinsic and Extrinsic Motivational Orientations in the
Classroom: Age Differences and Academic Correlates. Journal of Educational Psychology, 97(2),
pp.184–196. Available at: http://psycnet.apa.org/journals/edu/97/2/184/ [Accessed October 26,
2015].

Levy, P., 2003. A methodological framework for practice-based research in networked learning.
Instructional Science, 31(1–2), pp.87–109. Available at:
https://link.springer.com/article/10.1023/A:1022594030184.

Levy, Y., 2007. Comparing dropouts and persistence in e-learning courses. Computers & Education, 48(2),
pp.185–204. Available at: http://www.sciencedirect.com/science/article/pii/S0360131505000096
[Accessed October 15, 2015].

Lim, J., Reiser, R. & Olina, Z., 2009. The effects of part-task and whole-task instructional approaches on
acquisition and transfer of a complex cognitive skill. Educational Technology Research and
Developement, 57(1), pp.61–77. Available at: http://link.springer.com/article/10.1007/s11423-007-
9085-y [Accessed February 3, 2015].

Lincoln, Y. & Guba, E., 1985. Naturalist inquiry, Sage Publications. Available at:
http://scholar.google.com/scholar?hl=en&q=naturalist+inquiry&btnG=&as_sdt=1%2C5&as_sdtp=
#0 [Accessed August 19, 2015].

Litwin, M., 1995. How to Measure Survey Reliability and Validity, Sage Publications.

Liu, M. et al., 2011. A study of learning and motivation in a new media enriched environment for middle
school science. Educational Technology Research and Development, 59(2), pp.249–265. Available
at: http://link.springer.com/10.1007/s11423-011-9192-7 [Accessed October 6, 2014].

Livingstone, S., 2012. Critical reflections on the benefits of ICT in education. Oxford Review of Education,

312
 CHAPTER NINE

38(1), pp.9–24. Available at:

http://www.tandfonline.com/doi/abs/10.1080/03054985.2011.577938 [Accessed December 17,
2017].

Liyanagunawardena, T.R., Adams, A.A. & Williams, S.A., 2013. MOOCs: A systematic study of the published
literature 2008-2012. The International Review of Research in Open and Distributed Learning, 14(3),
pp.202–227. Available at: http://www.irrodl.org/index.php/irrodl/article/view/1455 [Accessed July
13, 2015].

Loewenthal, K. & Eysenck, M.W., 2001. An Introduction to Psychological Test and Scales, Psychology Press.
Available at:
https://books.google.com.cy/books?hl=en&lr=&id=3waN2zeIKHUC&oi=fnd&pg=PR8&ots=JbGv9P
qvis&sig=FdmPQ6U5DEBjp5CAkMW7W9dqXRE&redir_esc=y#v=onepage&q&f=false [Accessed
November 16, 2017].

Loken, B. et al., 1987. The use of 0-10 scales in telephone surveys. Journal of the Market Research Society,
29(3), pp.353–362. Available at:
https://scholar.google.com/scholar?q=Loken%2C+B.%2C+Pirie%2C+P.%2C+Virnig%2C+K.+A.%2C+
Hinkle%2C+R.+L.%2C+%26+Salmon%2C+C.+T.+%281987%29.+The+use+of+0-
10+scales+in+telephone+surveys&btnG=&hl=en&as_sdt=0%2C5 [Accessed April 19, 2017].

Long, J., 2007. Just For Fun: Using Programming Games in Software Programming Training and Education-
A Field Study of IBM Robocode Community. Journal of Information Technology Education, 6,
pp.279–290. Available at: http://jite.org/documents/Vol6/JITEv6p279-290Long169.pdf.

Lou, Y., Abrami, P. & D’Apollonia, S., 2001. Small group and individual learning with technology: A meta-
analysis. Review of educational research, 71(3), pp.449–521. Available at:
http://rer.sagepub.com/content/71/3/449.short [Accessed February 4, 2015].

Lye, S.Y. & Koh, J.H.L., 2014. Review on teaching and learning of computational thinking through
programming: What is next for K-12? Computers in Human Behavior, 41, pp.51–61. Available at:
http://dx.doi.org/10.1016/j.chb.2014.09.012.

MacKenzie, S., 2013. Human-Computer Interaction: An Empirical Research Perspective. 1st ed., Morgan
Kaufmann. Available at: http://www.amazon.com/Human-Computer-Interaction-Empirical-
Research-Perspective/dp/0124058655 [Accessed May 10, 2015].

Martens, B. & Hofkens, T., 2013. Positioning computer science in Flemish K-12 education. Proceedings of
the 8th Workshop in Primary and Secondary Computing Education on - WiPSE ’13, pp.35–36.
Available at: http://dl.acm.org/citation.cfm?doid=2532748.2532769.

Masip, L., Granollers, T. & Oliva, M., 2011. A Heuristic Evaluation Experiment to Validate the New Set of
Usability Heuristics. In 2011 Eighth International Conference on Information Technology: New
Generations. IEEE, pp. 429–434. Available at:
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5945274 [Accessed June 23, 2015].

313
 CHAPTER NINE

Matell, M.S. & Jacoby, J., 1971. Is There an Optimal Number of Alternatives for Likert Scale Items? Study
I: Reliability and Validity. Educational and Psychological Measurement, 31(3), pp.657–674. Available
at:
https://www.researchgate.net/profile/Jacob_Jacoby/publication/232587963_Is_There_an_Optim
al_Number_of_Alternatives_for_Likert_Scale_Items_Study_I_Reliability_and_Validity/links/5412e
c010cf2788c4b3587fc.pdf [Accessed November 18, 2016].

Maxwell, J., 1992. Understanding and validity in qualitative research. Harvard educational review, 62(3),
pp.279–301. Available at:
http://hepgjournals.org/doi/abs/10.17763/haer.62.3.8323320856251826 [Accessed August 18,
2015].

Mayer, R., 2005. Principles for reducing extraneous processing in multimedia learning: Coherence,
signaling, redundancy, spatial contiguity, and temporal contiguity principles. In R. Mayer, ed. The
Cambridge handbook of multimedia learning. Cambridge University Press, pp. 183–200. Available
at: https://www.amazon.com/Cambridge-Handbook-Multimedia-Handbooks-
Psychology/dp/0521838738/ref=tmm_hrd_swatch_0?_encoding=UTF8&qid=1543061493&sr=8-1
[Accessed January 8, 2015].

Mayer, R. & Chandler, P., 2001. When learning is just a click away: Does simple user interaction foster
deeper understanding of multimedia messages? Journal of educational psychology, 93(2), p.390.
Available at: http://psycnet.apa.org/journals/edu/93/2/390/ [Accessed February 4, 2015].

Mayer, R., Dow, G. & Mayer, S., 2003. Multimedia learning in an interactive self-explaining environment:
What works in the design of agent-based microworlds? Journal of Educational Psychology, (95),
pp.806–813. Available at: http://psycnet.apa.org/journals/edu/95/4/806/ [Accessed January 11,
2015].

Mayer, R. & Griffith, E., 2008. Increased interestingness of extraneous details in a multimedia science
presentation leads to decreased learning. Journal of Experimental Psychology, (14), pp.256–339.
Available at: http://psycnet.apa.org/journals/xap/14/4/329/ [Accessed January 15, 2015].

Mayer, R., Heiser, J. & Lonn, S., 2001. Cognitive constraints on multimedia learning: When presenting
more material results in less understanding. Journal of educational psychology, (93), pp.187–198.
Available at: http://psycnet.apa.org/journals/edu/93/1/187/ [Accessed January 15, 2015].

Mayer, R. & Johnson, C., 2008. Revising the redundancy principle in multimedia learning. Journal of
Educational Psychology, (100), pp.380–386. Available at:
http://psycnet.apa.org/journals/edu/100/2/380/ [Accessed January 13, 2015].

Mayer, R. & Moreno, R., 2003. Nine ways to reduce cognitive load in multimedia learning. Educational
psychologist, (38), pp.43–52. Available at:
http://www.tandfonline.com/doi/abs/10.1207/S15326985EP3801_6 [Accessed January 13, 2015].

Mayer, R.E., 1989. Systematic thinking fostered by illustrations in scientific text. Journal of Educational

314
 CHAPTER NINE

Psychology, 81(2), pp.240–246. Available at: http://doi.apa.org/getdoi.cfm?doi=10.1037/0022-

0663.81.2.240 [Accessed January 9, 2015].

Mayes, T. & de Freitas, S., 2004. Review of e-learning theories , frameworks and models, Available at:
http://researchrepository.murdoch.edu.au/id/eprint/32662/1/review-of-e-learning-theories.pdf
[Accessed December 19, 2017].

McCartney, R., Eckerdal, A. & Mostrom, J., 2007. Successful students’ strategies for getting unstuck. ACM
SIGCSE …, 39(3), p.156. Available at: http://dl.acm.org/citation.cfm?id=1268831 [Accessed April 27,
2016].

McCrudden, M. & Schraw, G., 2007. The effect of causal diagrams on text learning. Contemporary
Educational Psychology, (32), pp.367–388. Available at:
http://www.sciencedirect.com/science/article/pii/S0361476X05000652 [Accessed January 8,
2015].

McLoughlin, C. & Lee, M., 2008. Mapping the digital terrain: New media and social software as catalysts
for pedagogical change. Ascilite Melbourne. Available at:
http://cms.ascilite.org.au/conferences/melbourne08/procs/mcloughlin.pdf [Accessed July 13,
2015].

McNiff, J. & Whitehead, J., 2005. Teachers as educational theorists: transforming epistemological
hegemonies. In British Educational Research Association 2005 Annual Conference. Available at:
http://www.jeanmcniff.com/items.asp?id=25 [Accessed September 11, 2015].

Means, B. et al., 2009. Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and
Review of Online Learning Studies., US Department of Education. Available from: ED Pubs. P.O. Box
1398, Jessup, MD 20794-1398. Tel: 877-433-7827; Fax: 301-470-1244; Web site:
http://www.edpubs.org. Available at: https://eric.ed.gov/?id=ED505824 [Accessed December 17,
2017].

Means, T.B., Jonassen, D.H. & Dwyer, F.M., 1997. Enhancing relevance: Embedded ARCS strategies vs.
Purpose. Educational Technology Research and Development, 45(1), pp.5–17. Available at:
http://link.springer.com/10.1007/BF02299610 [Accessed August 8, 2018].

Mehlenbacher, B. et al., 2005. Usable E-Learning : A Conceptual Model for Evaluation and Design
Extending Usability. HCI International: 2005: 11th International Conference on Human-Computer
Interaction, 4, pp.1–10. Available at:
http://www4.ncsu.edu/unity/users/b/brad_m/web/research/elearning05.pdf [Accessed July 24,
2015].

Melero, J., Leo, D. & Blat, J., 2012. A Review of Constructivist Learning Methods with Supporting Tooling
in ICT Higher Education: Defining Different Types of Scaffolding. Journal of Universal Computer
Science, 18(16), pp.2334–2360. Available at:
http://www.jucs.org/jucs_18_16/a_review_of_constructivist/jucs_18_16_2334_2360_melero.pdf

315
 CHAPTER NINE

[Accessed October 22, 2014].

Mellors-Bourne, R., 2008. Do undergraduates want a career in IT ? A study of the attitudes of current UK
undergraduates in relation to careers and work in the IT sector, The Career Development
Organisation. Available at: https://www.crac.org.uk/home.

Merriam-Webster, 2018a. Definition of Extrinsic. Merriam-Webster Dictionary. Available at:

https://www.merriam-webster.com/dictionary/extrinsic [Accessed October 26, 2018].

Merriam-Webster, 2018b. Definition of instrumentation. Merriam-Webster Dictionary. Available at:

https://www.merriam-webster.com/dictionary/instrumentation [Accessed March 29, 2018].

Merriam-Webster, 2018c. Definition of Intrinsic. Merriam-Webster Dictionary. Available at:

https://www.merriam-webster.com/dictionary/intrinsic [Accessed October 26, 2018].

Merriam-Webster, 2016. Definition of Protocol. Available at: https://www.merriam-

webster.com/dictionary/protocol [Accessed June 16, 2017].

Merriam-Webster, 2017. Definition of Theory. Merriam-Webster Dictionary. Available at:

https://www.merriam-webster.com/dictionary/theory [Accessed December 7, 2017].

Mills, G.E., 2011. Action Research: A Guide for the Teacher Researcher, Pearson Education. Available at:
http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dstripbooks&field-
keywords=Action+Research%3A+A+Guide+for+the+Teacher+Researcher [Accessed September 10,
2015].

Mitchell, A. & Savill-Smith, C., 2004. The use of computer and video games for learning: A review of the
literature, Available at: www.lsda.org.uk.

Moore, M.G., 1989. Editorial: Three types of interaction. American Journal of Distance Education, 3(2),
pp.1–7. Available at: http://www.tandfonline.com/doi/pdf/10.1080/08923648909526659
[Accessed February 3, 2015].

Moreno, R., 2004. Decreasing cognitive load for novice students: Effects of explanatory versus corrective
feedback in discovery-based multimedia. Instructional science, 32(1–2), pp.99–113. Available at:
http://link.springer.com/article/10.1023/B:TRUC.0000021811.66966.1d [Accessed February 4,
2015].

Moreno, R., 2007. Optimising learning from animations by minimising cognitive load: Cognitive and
affective consequences of signalling and segmentation methods. Applied cognitive psychology,
21(6), pp.765–781. Available at:
http://scholar.google.com/scholar_url?url=http%3A%2F%2Fonlinelibrary.wiley.com%2Fdoi%2F10.
1002%2Facp.1348%2Ffull&hl=en&sa=T&ct=res&cd=0&ei=4T3SVKDOJ8eEqgGQ9YHYCA&scisig=AA
GBfm2PxNBXfpymi0dnwXgUnv0g65Vk4w&nossl=1&ws=1342x570 [Accessed February 4, 2015].

Moreno, R. & Mayer, R., 2000a. A coherence effect in multimedia learning: The case for minimizing
irrelevant sounds in the design of multimedia instructional messages. Journal of Educational

316
 CHAPTER NINE

Psychology, (92), pp.117–125. Available at: http://psycnet.apa.org/journals/edu/92/1/117/

[Accessed January 15, 2015].

Moreno, R. & Mayer, R., 1999. Cognitive principles of multimedia learning: The role of modality and
contiguity. Journal of educational psychology, (91), pp.358–368. Available at:
http://psycnet.apa.org/journals/edu/91/2/358/ [Accessed January 9, 2015].

Moreno, R. & Mayer, R., 2000b. Engaging students in active learning: The case for personalized multimedia
messages. Journal of Educational Psychology, 92(4), p.724. Available at:
http://psycnet.apa.org/journals/edu/92/4/724/ [Accessed February 4, 2015].

Moreno, R. & Mayer, R., 2007. Interactive multimodal learning environments. Educational Psychology
Review, 19(3), pp.309–326. Available at: http://link.springer.com/article/10.1007/s10648-007-
9047-2 [Accessed February 4, 2015].

Moreno, R. & Mayer, R., 2004. Personalized messages that promote science learning in virtual
environments. Journal of Educational Psychology, 96(1), p.165. Available at:
http://psycnet.apa.org/journals/edu/96/1/165/ [Accessed February 4, 2015].

Moreno, R. & Mayer, R., 2005. Role of Guidance, Reflection, and Interactivity in an Agent-Based
Multimedia Game. Journal of educational psychology, 97(1), p.117. Available at:
http://psycnet.apa.org/journals/edu/97/1/117/ [Accessed February 4, 2015].

Moreno, R. & Mayer, R., 2002. Verbal redundancy in multimedia learning: When reading helps listening.
Journal of Educational Psychology, (94), pp.156–163. Available at:
http://psycnet.apa.org/journals/edu/94/1/156/ [Accessed January 13, 2015].

Morrison, B.B. & DiSalvo, B., 2014. Khan academy gamifies computer science. In Proceedings of the 45th
ACM technical symposium on Computer science education - SIGCSE ’14. New York, New York, USA:
ACM Press, pp. 39–44. Available at: http://dl.acm.org/citation.cfm?doid=2538862.2538946.

Morrison, B.B., Dorn, B. & Guzdial, M., 2014. Measuring cognitive load in introductory CS. Proceedings of
the tenth annual conference on International computing education research - ICER ’14, pp.131–138.
Available at: http://dl.acm.org/citation.cfm?doid=2632320.2632348.

Morse, J. & Barrett, M., 2008. Verification strategies for establishing reliability and validity in qualitative
research. International Journal of Qualitative Methods, 1(2), pp.13–22. Available at:
https://ejournals.library.ualberta.ca/index.php/IJQM/article/viewArticle/4603 [Accessed August
18, 2015].

Muntean, C., 2011. Raising engagement in e-learning through gamification. In Proc. 6th International
Conference on Virtual Learning …. pp. 323–329. Available at:
http://icvl.eu/2011/disc/icvl/documente/pdf/met/ICVL_ModelsAndMethodologies_paper42.pdf
[Accessed July 10, 2015].

Munteanu, C. et al., 2006. The effect of speech recognition accuracy rates on the usefulness and usability
of webcast archives. Proc. CHI ’06, ACM Press, p.493. Available at:

317
 CHAPTER NINE

http://doi.acm.org/10.1145/1124772.1124848 [Accessed November 6, 2017].

Murugesh, S., Bell, T. & McGrath, A., 2010. A Review of Computer Science Resources to Support NCEA. In
First Annual Conference of Computing and Information Technology Research NZ (CITRENZ2010). pp.
173–181. Available at:
http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:A+Review+of+Computer+Scienc
e+Resources+to+Support+NCEA#1 [Accessed May 3, 2014].

Myers, J.H., 2000. Measuring customer satisfaction: hot buttons and other measurement issues 1st ed.,
South-Western Educational Pub.

Nasr, K.J. & Ramadan, B.H., 2008. Impact Assessment of Problem-Based Learning in an Engineering
Science Course. Journal of STEM Education: Innovations & Research, 9(3), pp.16–24. Available at:
http://search.ebscohost.com/login.aspx?direct=true&db=a9h&AN=36557315&site=ehost-live.

Newton, P.M., 2015. The learning styles myth is thriving in higher education. Frontiers in Psychology,
6(DEC), p.1908. Available at:
http://journal.frontiersin.org/Article/10.3389/fpsyg.2015.01908/abstract [Accessed July 4, 2018].

Nicholas, H. & Wan, N., 2009. Engaging Secondary School Students in Extended and Open Learning
supported by Online Technologies. Journal of Research on Technology in Education, 41(3), pp.305–
328. Available at: http://www.tandfonline.com/doi/abs/10.1080/15391523.2009.10782533
[Accessed August 26, 2014].

Nicholson, S., 2012. A user-centered theoretical framework for meaningful gamification. Games+
Learning+ Society. Available at: http://www.quilageo.com/wp-
content/uploads/2013/07/Framework-for-Meaningful-Gamifications.pdf [Accessed July 10, 2015].

Nicol, D.J. & Macfarlane-Dick, D., 2006. Formative assessment and self‐regulated learning: a model and
seven principles of good feedback practice. Studies in Higher Education, 31(2), pp.199–218.
Available at: http://www.tandfonline.com/doi/abs/10.1080/03075070600572090 [Accessed
January 20, 2014].

Nielsen, J., 1994. Heuristic Evaluation J. Nielsen & R. L. Mack, eds., Available at:
http://www.sccc.premiumdw.com/readings/heuristic-evaluation-nielson.pdf [Accessed July 14,
2016].

Nielsen, J. & Molich, R., 1990. Heuristic Evaluation of user interfaces. In CHI ’90 Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems. New York, New York, USA: ACM Press, pp.
249–256. Available at: http://portal.acm.org/citation.cfm?doid=97243.97281 [Accessed July 14,
2016].

Nokelainen, P., 2006. An empirical assessment of pedagogical usability criteria for digital learning material
with elementary school students. Journal of Educational Technology & Society, 9(2), pp.178–97.
Available at: http://www.jstor.org/stable/jeductechsoci.9.2.178 [Accessed July 24, 2015].

Norman, G., 2010. Likert scales, levels of measurement and the “laws” of statistics. Advances in Health

318
 CHAPTER NINE

Sciences Education, 15(5), pp.625–632. Available at: http://link.springer.com/10.1007/s10459-010-

9222-y.

Nunnally, J.C., 2017. Assessment of Reliability 3rd ed., New York: McGraw-Hill. Available at:
https://www.amazon.com/Psychometric-Theory-Jum-C-
Nunnally/dp/0071070885/ref=mt_paperback?_encoding=UTF8&me=&qid=1543075615.

Nuutila, E., Törmä, S. & Malmi, L., 2005. PBL and Computer Programming — The Seven Steps Method with
Adaptations. Computer Science Education, 15(2), pp.123–142. Available at:
http://www.tandfonline.com/doi/abs/10.1080/08993400500150788.

O’Brien, R., 2001. An overview of the methodological approach of action research. Theory and Practice of
Action Research, pp.1–15.

OCR, 2012. GCSE Computing 2012, Available at: www.ocr.org.uk/computing/gcse%0ACSE.

OCR, 2017. Unit level raw mark and UMS grade boundaries June 2017 series, Available at:
http://www.ocr.org.uk/Images/408006-legacy-gcse-grade-boundaries-june-2017.pdf [Accessed
October 23, 2017].

Office of National Statistics, 2013. Internet Access - Households and Individuals, 2013. Available at:
http://www.ons.gov.uk/ons/rel/rdit2/internet-access---households-and-
individuals/2013/index.html [Accessed May 27, 2015].

Ofsted, 2011. ICT in schools 2008–11,

Oliver Hogan, Colm Sheehy, S.U. and R.J., 2015. The economic impact of Basic Digital Skills and inclusion
in the UK - a report for Tinder Foundation and GO ON UK, Available at:
http://www.tinderfoundation.org/sites/default/files/research-
publications/the_economic_impact_of_digital_skills_and_inclusion_in_the_uk_final_v2.pdf.

Othman, N. & Amiruddin, M.H., 2010. Different perspectives of learning styles from VARK model. In
Procedia - Social and Behavioral Sciences. Elsevier, pp. 652–660. Available at:
https://www.sciencedirect.com/science/article/pii/S1877042810020926 [Accessed June 28, 2018].

Oxford Economics, 2011. Capturing the ICT Dividend: Using technology to drive productivity and growth
in the EU. white paper, Oxford Economics, September. Available at:
http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Capturing+the+ICT+Dividend+:+
Using+technology+to+drive+productivity+and+growth+in+the+EU#0 [Accessed June 6, 2014].

Ozkan, S. & Koseler, R., 2009. Multi-dimensional students’ evaluation of e-learning systems in the higher
education context: An empirical investigation. Computers and Education, 53(4), pp.1285–1296.
Available at: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5350590 [Accessed July 14,
2016].

Pachler, N. et al., 2010. Formative e-assessment: Practitioner cases. Computers & Education. Available at:
http://www.sciencedirect.com/science/article/pii/S0360131509002759 [Accessed July 9, 2015].

319
 CHAPTER NINE

Pachler, N., 2010. ICT CPD Landscape: Final Report, Available at:
http://www.academia.edu/2062079/ICT_CPD_Landscape_Final_Report [Accessed April 19, 2014].

Pallant, J., 2001. A step by step guide to data analysis using SPSS for Windows. A step by step guide to
data analysis using SPSS for Windows (Version 10), p.289. Available at:
https://books.google.com/books?hl=en&lr=&id=fZZTBgAAQBAJ&oi=fnd&pg=PR7&dq=spss+surviv
al+guide&ots=KUOumTSWEX&sig=tHEQLMZkBO74kvjsbn_sbfQEZP0 [Accessed April 5, 2017].

Palmer, D., 2005. A motivational view of constructivist‐informed teaching. International Journal of Science
Education, 27(15), pp.1853–1881. Available at:
http://www.tandfonline.com/doi/abs/10.1080/09500690500339654 [Accessed February 2, 2015].

Papanikolaou, K. & Boubouka, M., 2010. Promoting Collaboration in a Project-Based E-Learning Context.
Journal of Research on Technology in Education, 43(2), pp.135–155. Available at:
http://www.tandfonline.com/doi/abs/10.1080/15391523.2010.10782566.

Papastergiou, M., 2009. Digital Game-Based Learning in high school Computer Science education: Impact
on educational effectiveness and student motivation. Computers & Education, 52(1), pp.1–12.
Available at: http://www.sciencedirect.com/science/article/pii/S0360131508000845 [Accessed
July 18, 2014].

Pardamean, B. & Suparyanto, T., 2014. A Systematic Approach to Improving E-Learning Implementations
in higthschools. TOJET: The Turkish Online Journal of Educational Technology, 13(3), pp.19–26.
Available at:
http://www.researchgate.net/profile/Bens_Pardamean/publication/263506723_A_Systematic_Ap
proach_to_Improving_E-
Learning_Implementations_in_High_Schools/links/0046353b231df78259000000.pdf [Accessed
October 4, 2015].

Pardo-ballester, C. & Rodríguez, J.C., 2009. Using design-based research to guide the development of
online instructional materials. Developing and evaluating language learning materials, pp.86–102.
Available at: https://apling.engl.iastate.edu/tsll/2008/11 DBR Rodriguez Pardo.pdf.

Parlangeli, O., Marchigiani, E. & and Sebastiano Bagnara, 1999. Multimedia systems in distance education:
effects of usability on learning. Interacting with computers, 12(1), pp.37–49. Available at:
http://iwc.oxfordjournals.org/content/12/1/37.short [Accessed June 29, 2015].

Pearson, 2016. Grade Boundaries Edexcel GCSE June 2016, Available at:
http://qualifications.pearson.com/content/dam/pdf/Support/Grade-boundaries/GCSE/1706-gcse-
grade-boundaries-v3.pdf.

Perkins, D., 1999. The many faces of constructivism. Educational leadership, 57(3), pp.6–11. Available at:
http://eric.ed.gov/?id=EJ597073 [Accessed February 2, 2015].

Perkins, D., 1991. What constructivism demands of the learner. Educational technology, 31(9), pp.19–21.
Available at: http://eric.ed.gov/?id=EJ433312 [Accessed February 2, 2015].

320
 CHAPTER NINE

Perugini, S., Anderson, T.J. & Moroney, W.F., 2007. A study of out-of-turn interaction in menu-based, IVR,
voicemail systems. In Proceedings of the SIGCHI conference on Human factors in computing systems
- CHI ’07. p. 961. Available at: http://ecommons.udayton.edu/cps_fac_pub [Accessed November 6,
2017].

Phipps, R. & Merisotis, J., 2000. Quality on the line: Benchmarks for success in Internet-based distance
education, Available at:
http://www.ihep.org/sites/default/files/uploads/docs/pubs/qualityontheline.pdf [Accessed
December 17, 2017].

Piaget, J. & Inhelder, B., 1969. The Psychology Of The Child 2000th ed., Basic Books. Available at:
https://www.bookdepository.com/Psychology-Child-Jean-Piaget/9780465095001 [Accessed
November 30, 2017].

Pirker, J., Riffnaller-Schiefer, M. & Gütl, C., 2014. Motivational active learning. In Proceedings of the 2014
conference on Innovation & technology in computer science education - ITiCSE ’14. New York, New
York, USA: ACM Press, pp. 297–302. Available at:
http://dl.acm.org/citation.cfm?doid=2591708.2591750.

Pollock, E., Chandler, P. & Sweller, J., 2002. Assimilating complex information. Learning and Instruction,
12(1), pp.61–86. Available at: http://linkinghub.elsevier.com/retrieve/pii/S0959475201000160
[Accessed February 4, 2015].

Popescu, E., 2010. Adaptation provisioning with respect to learning styles in a Web-based educational
system: an experimental study. Journal of Computer Assisted Learning, 26(4), pp.243–257. Available
at: http://doi.wiley.com/10.1111/j.1365-2729.2010.00364.x [Accessed August 19, 2014].

Prescot, C., 2017. Internet Access - Households and Individuals: 2017, Available at:
https://www.ons.gov.uk/peoplepopulationandcommunity/householdcharacteristics/homeinterne
tandsocialmediausage/bulletins/internetaccesshouseholdsandindividuals/2015-08-06.

Preston, C.C. & Colman, A.M., 2000. Optimal number of response categories in rating scales: reliability,
validity, discriminating power, and respondent preferences. Acta Psychologica, 104(1), pp.1–15.
Available at: http://www.sciencedirect.com/science/article/pii/S0001691899000505 [Accessed
February 15, 2017].

Pritchard, A., 2009. Ways of learning learning theories and learning styles in the classroom 2nd ed.,
Abingdon, Oxon; New York, NY: Routledge.

PWC, 2009. Champion for Digital Inclusion: The Economic Case for Digital Inclusion, Available at:
http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Champion+for+Digital+Inclusion
:+The+Economic+Case+for+Digital+Inclusion.#0 [Accessed April 16, 2014].

Raman, R. et al., 2015. Computer Science (CS) Education in Indian Schools. ACM Transactions on
Computing Education, 15(2), pp.1–36. Available at:
http://dl.acm.org/citation.cfm?doid=2767124.2716325 [Accessed December 15, 2017].

321
 CHAPTER NINE

Reeves, T.C., 1994. Evaluating What Really Matters in Computer-Based Education. In M. Wild & D.
Kirkpatrick, eds. Computer education: New perspectives. MASTEC Edith Cowan University, pp. 219–
246. Available at: http://140.127.40.36/ftp/Speech/Evaluating What Really Matters in Computer-
Based Education.doc [Accessed June 23, 2015].

Reeves, T.C. et al., 2002. Usability and Instructional Design Heuristics for E-Learning Evaluation. In ED-
MEDIA 2002 World Conference on Educational Multimedia, Hypermedia & Telecommunications.
Association for the Advancement of Computing in Education (AACE), P.O. Box 3728, Norfolk, VA
23514. Tel: 757-623-7588; e-mail: info@aace.org; Web site: http://www.aace.org/DL/., p. 8.
Available at: http://eric.ed.gov/?id=ED477084 [Accessed May 25, 2015].

Rich, K., Strickland, C. & Franklin, D., 2017. A Literature Review through the Lens of Computer Science
Learning Goals Theorized and Explored in Research. In Proceedings of the 2017 ACM SIGCSE
Technical Symposium on Computer Science Education - SIGCSE ’17. New York, New York, USA: ACM
Press, pp. 495–500. Available at: http://dl.acm.org/citation.cfm?doid=3017680.3017772 [Accessed
December 14, 2017].

Rick, D., Morisse, M. & Schirmer, I., 2012. Bringing contexts into the classroom. Proceedings of the 7th
Workshop in Primary and Secondary Computing Education on - WiPSCE ’12, p.105. Available at:
http://dl.acm.org/citation.cfm?doid=2481449.2481476.

Rochefort, B. & Richmond, N., 2011. Connecting Instruction to Connected Technologies–Why Bother? An
Instructional Designer’s Perspective. RUSC, Universities and Knowledge Society Journal, 8(1),
pp.217–232. Available at: https://link.springer.com/article/10.7238/rusc.v8i1.942 [Accessed
November 13, 2014].

Roger, T. & Johnson, D., 1988. Cooperative learning two heads learn better than one. Transforming
Education, 18, p.34. Available at: http://ininfo.mggu-sh.ru/data/resources/collaborative-
learning/libr/02_cooperative learning_short.doc [Accessed July 13, 2015].

Rolandsson, L. & Skogh, I.-B., 2014. Programming in School. ACM Transactions on Computing Education,
14(2), pp.1–25. Available at: http://dl.acm.org/citation.cfm?doid=2642651.2602487 [Accessed
December 15, 2017].

Rountree, N. et al., 2004. Interacting factors that predict success and failure in a CS1 course. In SIGCSE
Bulletin. New York, New York, USA: ACM Press, p. 101. Available at:
http://portal.acm.org/citation.cfm?doid=1044550.1041669 [Accessed December 12, 2017].

Rourke, A. & Sweller, J., 2009. The worked-example effect using ill-defined problems: Learning to
recognise designers’ styles. Learning and Instruction, 19(2), pp.185–199. Available at:
http://www.sciencedirect.com/science/article/pii/S095947520800039X [Accessed February 2,
2015].

Rovai, A.P., 2002. Building Sense of Community at a Distance. The International Review of Research in
Open and Distributed Learning, 3(1), pp.1–16. Available at:

322
 CHAPTER NINE

http://www.irrodl.org/index.php/irrodl/article/viewArticle/79 [Accessed November 11, 2015].

Rowntree, D., 1992. Exploring open and distance learning, London: Kogan Page. Available at:
http://dialnet.unirioja.es/servlet/libro?codigo=556712 [Accessed October 25, 2015].

Royal Society, 2017a. After the reboot : computing education in UK schools, Available at:
https://royalsociety.org/topics-policy/projects/computing-education/.

Royal Society, 2017b. After the reboot : computing education in UK schools (Executive Summary), Available
at: https://royalsociety.org/topics-policy/projects/computing-education/.

Ruparelia, N.B., 2010. Software development lifecycle models. ACM SIGSOFT Software Engineering Notes,
35(3), p.8. Available at: http://doi.acm.org/10.1145/1764810.1764814 [Accessed April 27, 2019].

Ryoo, J.J. et al., 2013. Democratizing computer science knowledge: transforming the face of computer
science through public high school education. Learning, Media and Technology, 38(2), pp.161–181.
Available at:
http://www.tandfonline.com/doi/abs/10.1080/17439884.2013.756514?instName=University+of+
Central+Lancashire [Accessed September 18, 2015].

Sadler, D.R., 1989. Formative assessment and the design of instructional systems. Instructional Science,
18, pp.119–144. Available at: http://www.springerlink.com/index/x7l185036h762m45.pdf
[Accessed December 4, 2017].

Sambell, M., 2016. Education , Skills and the Economy Sub Committee Enquiry into Apprenticeships,
Available at: https://www.thetechpartnership.com/resources/full-library/.

Sanchez, C. & Wiley, J., 2006. An examination of the seductive details effect in terms of working memory
capacity. Memory & Cognition, (34), pp.344–355. Available at:
http://link.springer.com/article/10.3758/BF03193412 [Accessed January 15, 2015].

Sandoval, W.A. & Bell, P., 2004. Design-Based Research Methods for Studying Learning in Context:
Introduction. Educational Psychologist, 39(4), pp.199–201. Available at:
http://www.tandfonline.com/doi/abs/10.1207/s15326985ep3904_1 [Accessed August 12, 2015].

Saris, W.E. & Gallhofer, I.N., 2014. Design, evaluation, and analysis of questionnaires for survey research
2nd ed., Hoboken, NJ, USA: John Wiley & Sons, Inc. Available at:
http://doi.wiley.com/10.1002/9780470165195 [Accessed September 6, 2018].

Scardamalia, M. & Bereiter, C., 1996. Adaptation and Understanding: A Case for New Cultures of
Schooling. In S. Vosniadou et al., eds. International perspectives on the design of technology-
supported learning environments. Routledge - Taylor & Francis Group, pp. 149–165. Available at:
https://www.routledge.com/International-Perspectives-on-the-Design-of-Technology-supported-
Learning/Vosniadou-De-Corte-Glaser-Mandl/p/book/9780805818543 [Accessed February 3, 2015].

Schiaffino, S., Garcia, P. & Amandi, A., 2008. eTeacher: Providing personalized assistance to e-learning
students. Computers and Education, 51(4), pp.1744–1754. Available at:

323
 CHAPTER NINE

http://trafficlight.bitdefender.com/info?url=https%3A//www.sciencedirect.com/science/article/pi
i/S036013150800078X&language=en_US [Accessed July 11, 2018].

Schmidt, E., 2011. “MacTaggart lecture.” Edinburgh TV Festival. Available at:

http://www.theguardian.com/media/interactive/2011/aug/26/eric-schmidt-mactaggart-lecture-
full-text [Accessed May 27, 2015].

Schmidt, H. et al., 2009. Constructivist, problem-based learning does work: A meta-analysis of curricular
comparisons involving a single medical school. Educational …, 44(4), pp.227–249. Available at:
http://www.tandfonline.com/doi/abs/10.1080/00461520903213592 [Accessed February 3, 2015].

Schoenfeld, A., 1987. What’s All the Fuss About Metacognition? In A. Schoenfeld, ed. Cognitive science
and mathematics education. Routledge - Taylor & Francis Group, pp. 189–216. Available at:
https://www.taylorfrancis.com/books/9780203062685 [Accessed February 3, 2015].

Schraw, G., Crippen, K. & Hartley, K., 2006. Promoting self-regulation in science education: Metacognition
as part of a broader perspective on learning. Research in Science Education, 36, pp.111–139.
Available at: http://link.springer.com/article/10.1007/s11165-005-3917-8 [Accessed July 28, 2015].

Schunk, D., 1991. Self-efficacy and academic motivation. Educational psychologist, 26(3–4), pp.207–231.
Available at: http://www.tandfonline.com/doi/abs/10.1080/00461520.1991.9653133 [Accessed
October 26, 2015].

Schweitzer, D. & Brown, W., 2007. Interactive visualization for the active learning classroom. ACM SIGCSE
Bulletin, 39(1), p.208.

Schworm, S. & Renkl, A., 2007. Learning argumentation skills through the use of prompts for self-
explaining examples. Journal of Educational Psychology, 99(2), p.285. Available at:
http://psycnet.apa.org/journals/edu/99/2/285/ [Accessed February 2, 2015].

Scriven, M., 1993. Hard-Won Lessons in Program Evaluation. New directions for program evaluation.
Available at: http://eric.ed.gov/?id=EJ478673 [Accessed May 14, 2017].

Selby, C., Dorling, M. & Woollard, J., 2014. Evidence of assessing computational thinking. Author’s original.
Available at: http://eprints.soton.ac.uk/372409/ [Accessed September 22, 2015].

Selby, C. & Woollard, J., 2013. Computational thinking: the developing definition. Available at:
http://eprints.soton.ac.uk/356481/7/Selby_Woollard_bg_soton_eprints.pdf [Accessed November
30, 2015].

Sendall, P., Ceccucci, W. & Peslak, A., 2008. Web 2.0 matters: An analysis of implementing Web 2.0 in the
classroom. Information Systems Education Journal, 6(64), pp.1–17. Available at:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.455.5466&rep=rep1&type=pdf
[Accessed October 4, 2015].

Sentance, S. et al., 2012. Grand Challenges for the UK: Upskilling Teachers to Teach Computer Science
Within the Secondary Curriculum. In Proceedings of the 7th Workshop in Primary and Secondary

324
 CHAPTER NINE

Computing Education. WiPSCE ’12. New York, NY, USA: ACM, pp. 82–85. Available at:
http://doi.acm.org/10.1145/2481449.2481469.

Sentance, S. & Selby, C., 2015. A classification of research into computer science education in school from
2005-2014 : Initial report, Available at:
https://community.computingatschool.org.uk/resources/4119/single.

Shee, D. & Wang, Y., 2008. Multi-criteria evaluation of the web-based e-learning system: A methodology
based on learner satisfaction and its applications. Computers & Education, 50, pp.894–905.
Available at: http://www.sciencedirect.com/science/article/pii/S0360131506001424 [Accessed
July 13, 2015].

Shen, R., Wang, M. & Gao, W., 2009. Mobile learning in a large blended computer science classroom:
System function, pedagogies, and their impact on learning. IEEE TRANSACTIONS ON EDUCATION,
52(4), pp.538–546. Available at: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5191064
[Accessed July 10, 2015].

Shroff, R.H. & Vogel, D.R., 2009. Assessing the Factors Deemed to Support Individual Student Intrinsic
Motivation in Technology Supported Online and Face-to-Face Discussions. Journal of Information
Technology Education, 8, pp.59–85. Available at: http://jite.org/documents/Vol8/JITEv8p059-
085Shroff416.pdf.

Siemens, G., 2004. Connectivism: A learning theory for the digital age. International Journal of
Instructional Technology and Distance Learning, 2(1), pp.3–10.

Siemens, G., 2006. Connectivism: Learning theory or pastime of the self-amused. Manitoba, Canada:
Learning Technologies Centre. Available at:
http://www.elearnspace.org/Articles/connectivism_self-amused.htm [Accessed February 3, 2015].

Siemens, G., 2008. Learning and knowing in networks: Changing roles for educators and designers. In
University of Georgia IT Forum. p. 20. Available at:
https://www.calvin.edu/~dsc8/documents/GeorgeSiemensPresentation-Jan2008.pdf [Accessed
November 7, 2014].

Skudder, B. & Luxton-Reilly, A., 2014. Worked Examples in Computer Science. In Sixteenth Australasian
Computing Education Conference (ACE2014). pp. 59–64. Available at:
http://crpit.com/confpapers/CRPITV148Skudder.pdf [Accessed December 3, 2017].

Slack, F. et al., 2003. Assessment and Learning Outcomes: The Evaluation of Deep Learning in an On-line
Course. Journal Information Technology Education: Research, 2, pp.305–317. Available at:
https://www.learntechlib.org/p/111480/.

Slavin, R.E., 2011. Instruction based on cooperative learning. In R. E. Mayer & P. A. Alexander, eds.
Handbook of research on learning and instruction. Routledge - Taylor & Francis Group, pp. 344–360.
Available at: http://jafarianhamid.persiangig.com/Research on Learning and
Instruction.pdf#page=359 [Accessed February 4, 2015].

325
 CHAPTER NINE

Sloboda, J. et al., 1996. The role of practice in the development of performing musicians. British journal
of psychology, 87(2), pp.287–309. Available at: http://onlinelibrary.wiley.com/doi/10.1111/j.2044-
8295.1996.tb02591.x/full [Accessed February 4, 2015].

Small, R. V. & Gluck, M., 1994. The relationship of motivational conditions to effective instructional
attributes: A magnitude scaling approach. Educational Technology, 34(8), pp.33–40. Available at:
https://www.jstor.org/stable/44428228 [Accessed August 8, 2018].

Snyder, L., 2012. Status update. ACM Inroads, 3(2), pp.82–85. Available at:
http://dl.acm.org/citation.cfm?doid=2189835.2189861 [Accessed December 15, 2017].

So, W., 2002. Constructivist teaching in primary science. Asia-Pacific Forum on Science Learning and
Teaching, 3(1), pp.1–31. Available at:
http://www.ied.edu.hk/apfslt/download/v3_issue1_files/sowm/sowm.pdf [Accessed February 2,
2015].

Song, S.H. & Keller, J.M., 2001. Effectiveness of motivationally adaptive computer-assisted instruction on
the dynamic aspects of motivation. Educational Technology Research and Development, 49(2),
pp.5–22. Available at: http://link.springer.com/10.1007/BF02504925 [Accessed August 8, 2018].

Squires, D. & Preece, J., 1999. Predicting quality in educational software: Interacting with Computers,
11(5), pp.467–483. Available at: https://academic.oup.com/iwc/article-lookup/doi/10.1016/S0953-
5438(98)00063-0 [Accessed June 23, 2015].

Squires, D. & Preece, J., 1996. Usability and learning: evaluating the potential of educational software.
Computers & Education, 27(1), pp.15–22. Available at:
http://www.sciencedirect.com/science/article/pii/0360131596000103 [Accessed June 29, 2015].

Stephenson, C. et al., 2005. The new educational imperative: Improving high school computer science
education, Available at:
http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:The+New+Educational+Imperati
ve+:+Improving+High+School+Computer+Science+Education.#0 [Accessed May 3, 2014].

Stott, A. & Neustaedter, C., 2013. Analysis of gamification in education, Available at:
http://clab.iat.sfu.ca/pubs/Stott-Gamification.pdf [Accessed November 23, 2015].

Stuchlikova, L. & Benkovska, J., 2011. eLearn central - the journey to e-learning. In 2011 14th International
Conference on Interactive Collaborative Learning. IEEE, pp. 16–23. Available at:
http://www.scopus.com/inward/record.url?eid=2-s2.0-81355142836&partnerID=tZOtx3y1
[Accessed July 8, 2015].

Sturman, L. & Sizmur, J., 2011. International Comparison of Computing in Schools, Available at:
https://www.nfer.ac.uk/publications/cis101/cis101.pdf.

Sun, P.-C. et al., 2008. What drives a successful e-Learning? An empirical investigation of the critical factors
influencing learner satisfaction. Computers & Education, 50(4), pp.1183–1202. Available at:
http://linkinghub.elsevier.com/retrieve/pii/S0360131506001874 [Accessed January 21, 2014].

326
 CHAPTER NINE

Susman, G. & Evered, R., 1978. An assessment of the scientific merits of action research. Administrative
science quarterly, 23. Available at: http://www.jstor.org/stable/2392581 [Accessed September 11,
2015].

Sweller, J., 2004. Instructional design consequences of an analogy between evolution by natural selection
and human cognitive architecture. Instructional science, 32(1–2), pp.9–31. Available at:
http://link.springer.com/article/10.1023/B:TRUC.0000021808.72598.4d [Accessed February 2,
2015].

Sweller, J. & Cooper, G., 1985. The use of worked examples as a substitute for problem solving in learning
algebra. Cognition and Instruction, 2(1), pp.59–89. Available at:
http://www.tandfonline.com/doi/abs/10.1207/s1532690xci0201_3 [Accessed February 2, 2015].

Sweller, J., Van Merrienboer, J. & Paas, F., 1998. Cognitive architecture and instructional design.
Educational Psychology Review, 10(3), pp.251–296. Available at:
https://www.jstor.org/stable/23359412.

Symonds, P., 1924. On the Loss of Reliability in Ratings Due to Coarseness of the Scale. Journal of
Experimental Psychology. Available at: http://psycnet.apa.org/journals/xge/7/6/456/ [Accessed
April 11, 2017].

Taras, M., 2005. Formative and Summative Assessment - Summative and Formative - some theoretical
reflections. British Journal of Educational Studies, 21(1), pp.466–478. Available at:
http://www.tandfonline.com/doi/pdf/10.1080/03057269308560014 [Accessed July 9, 2015].

Taylor, C., 2006. Introducing Action Research. In Doing Action Research. London: SAGE Publications Ltd,
pp. 1–16. Available at: http://methods.sagepub.com/book/doing-action-research/d3.xml.

Tech Partnership, 2015. Datasheet : Tech graduates Datasheet : Tech graduates 2013 / 14, Available at:
https://www.thetechpartnership.com/resources/full-library/.

Tech Partnership, 2016. Datasheet : Tech specialists in the UK, Available at:
https://www.thetechpartnership.com/resources/full-library/.

TechnologyAdvice, 2014. Gamification 101: Learn the Basics of Gamification Strategy. Available at:
http://www.slideshare.net/technologyadvice/gamification-101-how-gami?related=1 [Accessed
November 26, 2015].

TechPartnership, 2016. Factsheet: Demand for Digital specialists, Available at:

https://www.thetechpartnership.com/resources/full-library/.

The Design-Based Research Collective, 2003. Design-based research: An emerging paradigm for
educational inquiry. Educational Researcher, 32(1), pp.5–8. Available at:
http://www.jstor.org/stable/3699927 [Accessed August 12, 2015].

The Royal Academy of Engineering, 2009. ICT for the UK ’ s Future : the implications of the changing nature
of Information and Communications Technology, Available at:

327
 CHAPTER NINE

http://www.raeng.org.uk/publications/reports/ict-for-the-uks-future [Accessed May 27, 2015].

The Royal Society, 2012. Shut down or restart? The way forward for computing in UK schools. Technology,
(January), pp.1–122. Available at: https://royalsociety.org/education/policy/computing-in-
schools/report/ [Accessed May 27, 2015].

Thomas, K., Britain, E.I.U.G. & Alliance, B.S., 2007. The Means to Compete: Benchmarking IT Industry
Competitiveness, Economist Intelligence Unit. Available at:
http://books.google.com.cy/books/about/The_Means_to_Compete.html?id=T3eutgAACAAJ&pgis
=1 [Accessed June 12, 2014].

Tseng, S. & Tsai, C., 2007. On-line peer assessment and the role of the peer feedback: A study of high
school computer course. Computers & Education. Available at:
http://www.sciencedirect.com/science/article/pii/S0360131506000297 [Accessed October 11,
2015].

TSIRIGA, V. & VIRVOU, M., 2004. EVALUATING THE INTELLIGENT FEATURES OF A WEB-BASED INTELLIGENT
COMPUTER ASSISTED LANGUAGE LEARNING SYSTEM. International Journal on Artificial Intelligence
Tools, 13(02), pp.411–425. Available at:
http://www.worldscientific.com/doi/abs/10.1142/S0218213004001600 [Accessed July 11, 2018].

Tuffiash, M., Roring, R. & Ericsson, K., 2007. Expert performance in SCRABBLE: implications for the study
of the structure and acquisition of complex skills. Journal of Experimental Psychology: Applied, 13(3),
p.124. Available at: http://psycnet.apa.org/journals/xap/13/3/124/ [Accessed February 4, 2015].

Tutty, J. & Klein, J., 2008. Computer-mediated instruction: A comparison of online and face-to-face
collaboration. Educational Technology Research and Development, 56(2), pp.101–124. Available at:
http://link.springer.com/article/10.1007/s11423-007-9050-9 [Accessed February 4, 2015].

Underwood, J., 2009. The impact of digital technology: a review of the evidence of the impact of digital
technologies on formal education, Available at:
http://dera.ioe.ac.uk/10491/3/A9RF934_Redacted.pdf [Accessed December 17, 2017].

Verhagen, P., 2006. Connectivism : a new learning theory ? University of Twente, 2, pp.3–10. Available at:
https://www.scribd.com/doc/88324962/Connectivism-a-New-Learning-Theory.

Virvou, M., Katsionis, G. & Manos, K., 2005. Combining Software Games with Education: Evaluation of its
Educational. Educational Technology & Society, 8(2), pp.54–65. Available at:
http://trafficlight.bitdefender.com/info?url=https%3A//www.jstor.org/stable/pdf/jeductechsoci.8
.2.54.pdf&language=en_US [Accessed July 11, 2018].

Visser, J. & Keller, J.M., 1990. The clinical use of motivational messages: an inquiry into the validity of the
ARCS model of motivational design. Instructional Science, 19(6), pp.467–500. Available at:
http://link.springer.com/10.1007/BF00119391 [Accessed August 8, 2018].

Visser, L., 1998. The development of motivational communication in distance education support, Available
at: http://dialnet.unirioja.es/servlet/libro?codigo=555501 [Accessed October 25, 2015].

328
 CHAPTER NINE

Vlăsceanu, L., Grünberg, L. & Pârlea, D., 2004. Quality assurance and accreditation: A glossary of basic
terms and definitions, Unesco-Cepes. Available at:
http://dev.observatoriorh.org/sites/default/files/webfiles/fulltext/unesco_cepes_qual_assur_acre
d.pdf [Accessed June 25, 2015].

Voelkel, S., 2013. Combining the formative with the summative: the development of a twostage online
test to encourage engagement and provide personal feedback in large classes. Research in Learning
Technology, 21. Available at:
http://www.researchinlearningtechnology.net/index.php/rlt/article/view/19153 [Accessed July 9,
2015].

Vogel-Walcutt, J.J. et al., 2011. Cognitive load theory vs. constructivist approaches: which best leads to
efficient, deep learning? Journal of Computer Assisted Learning, 27(2), pp.133–145. Available at:
http://doi.wiley.com/10.1111/j.1365-2729.2010.00381.x [Accessed August 4, 2014].

Vrasidas, C., 2000. Constructivism versus objectivism: Implications for interaction, course design, and
evaluation in distance education. International Journal of Educational Telecommunications, 6(4),
pp.339–362. Available at: http://vrasidas.intercol.edu/continuum.pdf [Accessed October 3, 2014].

Vygotsky, L.S., 1978. Mind in Society: The Development of Higher Psychological Processes M. Cole et al.,
eds., Harvard University Press. Available at:
http://www.hup.harvard.edu/catalog.php?isbn=9780674576292.

Vygotsky, L.S., 1986. Thought and Language 2nd ed. A. Kozulin, ed., MIT Press.

Waite, J., 2017. Pedagogy in teaching Computer Science in schools : A Literature Review, Available at:
https://royalsociety.org/~/media/policy/projects/computing-education/literature-review-
pedagogy-in-teaching.pdf.

Wang, F. & Hannafin, M., 2005. Design-based research and technology-enhanced learning environments.
Educational Technology Research and Developement, 53(4), pp.5–23. Available at:
http://link.springer.com/article/10.1007/BF02504682 [Accessed July 13, 2015].

Wang, Z., Chen, L. & Anderson, T., 2014. A framework for interaction and cognitive engagement in
connectivist learning contexts. The International Review of Research in Open and Distance Learning,
15(2). Available at: http://www.irrodl.org/index.php/irrodl/article/view/1709/2859 [Accessed
November 13, 2014].

Weijters, B., Cabooter, E. & Schillewaert, N., 2010. The effect of rating scale format on response styles:
The number of response categories and response category labels. International Journal of Research
in Marketing, 27(3), pp.236–247. Available at:
http://www.sciencedirect.com/science/article/pii/S0167811610000303 [Accessed April 3, 2017].

Weiler, A., 2005. Information-Seeking Behavior in Generation Y Students: Motivation, Critical Thinking,
and Learning Theory. The Journal of Academic Librarianship, 31(1), pp.46–53. Available at:
http://linkinghub.elsevier.com/retrieve/pii/S0099133304001521.

329
 CHAPTER NINE

Welsh, E.T. et al., 2003. E-learning: emerging uses, empirical results and future directions. International
Journal of Training and Development, 7(4), pp.245–258. Available at:
http://kanjiteacher.googlecode.com/svn/!svn/bc/76/Non-Code/Papers/Learning/welsh_E-
learning.pdf [Accessed June 25, 2015].

Wesson, J. & Cowley, L., 2003. The challenge of measuring e-learning quality: Some ideas from HCI. Quality
Education@ a Distance. Available at: http://link.springer.com/chapter/10.1007/978-0-387-35700-
3_25 [Accessed June 23, 2015].

Williams, P., Iglesias, J. & Barak, M., 2008. Problem based learning: application to technology education
in three countries. International Journal of Technology and Education, 18(3), pp.319–335. Available
at: http://link.springer.com/article/10.1007/s10798-007-9028-2 [Accessed July 13, 2015].

Wilson, C. et al., 2010. Running On Empty: The Failure to Teach K–12 Computer Science in the Digital Age,
Available at: http://www.ncbi.nlm.nih.gov/pubmed/24256681 [Accessed May 27, 2015].

Wing, J.M., 2011. Computational thinking., Available at:

http://csta.acm.org/Curriculum/sub/CurrFiles/WingCTPrez.pdf [Accessed July 10, 2015].

Wing, J.M., 2006a. Computational thinking. Communications of the ACM, 49(3), pp.33–35. Available at:
http://dl.acm.org/ft_gateway.cfm?id=1118215&type=html [Accessed September 22, 2015].

Wing, J.M., 2006b. Computational Thinking and CS@CMU. CT and CS@CMU, pp.1–30. Available at:
http://www-cgi.cs.cmu.edu/afs/cs/usr/wing/www/CT_at_CMU.pdf [Accessed April 19, 2014].

de Winter, J.C.F., Gosling, S.D. & Potter, J., 2016. Comparing the Pearson and Spearman correlation
coefficients across distributions and sample sizes: A tutorial using simulations and empirical data.
Psychological Methods, 21(3), pp.273–290. Available at:
http://doi.apa.org/getdoi.cfm?doi=10.1037/met0000079 [Accessed October 25, 2017].

Wittink, D.R. & Bayer, L.R., 1994. The measurement imperative. Marketing Research, 6(4), p.14. Available
at: https://search.proquest.com/openview/d850c79be163ce82835ba99bfd139b4f/1?pq-
origsite=gscholar&cbl=31079.

Wolf, A., 2011. Review of vocational education: the Wolf report, Available at:
http://dera.ioe.ac.uk/11621/1/DFE-00031-2011.pdf [Accessed April 16, 2014].

Wong, S.K. et al., 2003. Usability Metrics for E-learning. In On The Move to Meaningful Internet Systems
2003: OTM 2003 Workshops. pp. 235–252. Available at:
http://www.springerlink.com/index/P8UR5N0E0DT1U2BM.pdf [Accessed May 9, 2017].

Woodhouse, D., 2004. The quality of quality assurance agencies. Quality in Higher Education, 10(2),
pp.77–87. Available at: https://www.tandfonline.com/doi/full/10.1080/1353832042000230572
[Accessed June 25, 2015].

Wu, W.-H. et al., 2012. Investigating the learning-theory foundations of game-based learning: a meta-
analysis. Journal of Computer Assisted Learning, 28(3), pp.265–279. Available at:

330
 CHAPTER NINE

http://doi.wiley.com/10.1111/j.1365-2729.2011.00437.x [Accessed July 7, 2015].

Yacob, A.B., Bin Mohd Saman, M.Y. & Bin Yusoff, M.H., 2012. Constructivism learning theory for
programming through an e-learning. In 2012 6th International Conference on New Trends in
Information Science, Service Science and Data Mining (ISSDM2012). IEEE / Institute of Electrical and
Electronics Engineers Incorporated, pp. 639–643. Available at:
https://ieeexplore.ieee.org/abstract/document/6528711 [Accessed October 22, 2014].

Yang, C. & Chang, Y.-S., 2012. Assessing the effects of interactive blogging on student attitudes towards
peer interaction, learning motivation, and academic achievements. Journal of Computer Assisted
Learning, 28(2), pp.126–135. Available at: http://doi.wiley.com/10.1111/j.1365-2729.2011.00423.x
[Accessed February 8, 2014].

Yiatrou, P. et al., 2016. The synthesis of a unified pedagogy for the design and evaluation of e-learning
software for high-school computing. In Proceedings of the 2016 Federated Conference on Computer
Science and Information Systems. Institute of Electrical and Electronic Engineers, pp. 927–931.
Available at: https://ieeexplore.ieee.org/abstract/document/7733353.

Yiatrou, P., Polycarpou, I. & Read, J.C., 2014. The Synthesis of a Unified Pedagogy for the development of
E-Learning Software for the Computing GCSE. In EAPRIL Conference Proceedings 2014. pp. 1–14.
Available at: https://eapril.org/sites/default/files/2017-03/eapril-2014-proceedings.pdf#page=7.

Yvonne Feilzer, M., 2010. Doing Mixed Methods Research Pragmatically: Implications for the Rediscovery
of Pragmatism as a Research Paradigm. Journal of Mixed Methods Research, 4(1), pp.6–16. Available
at: http://journals.sagepub.com/doi/10.1177/1558689809349691 [Accessed November 1, 2017].

Zaharias, P. & Koutsabasis, P., 2012. Heuristic evaluation of e-learning courses: a comparative analysis of
two e-learning heuristic sets. Campus-Wide Information Systems, 29(1), pp.45–60.

Zaharias, P. & Poylymenakou, A., 2009. Developing a Usability Evaluation Method for e-Learning
Applications: Beyond Functional Usability. International Journal of Human-Computer Interaction,
25(1), pp.75–98. Available at: http://dx.doi.org/10.1080/10447310802546716 [Accessed February
8, 2014].

Zaharias, P., Vassilopoulou, K. & Poulymenakou, A., 2002. Designing on-line learning courses: Implications
for usability. JAPIT. Available at:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.98.5139&rep=rep1&type=pdf
[Accessed May 9, 2017].

Zapalska, A. & Brozik, D., 2006. Learning styles and online education. Campus-Wide Information Systems.
Available at: http://www.emeraldinsight.com/journals.htm?articleid=1581868&show=abstract
[Accessed November 13, 2014].

Zenios, M., 2011. Epistemic activities and collaborative learning: Towards an analytical model for studying
knowledge construction in networked learning settings. Journal of Computer Assisted Learning,
27(3), pp.259–268. Available at: http://onlinelibrary.wiley.com/doi/10.1111/j.1365-

331
 CHAPTER NINE

2729.2010.00394.x/full [Accessed July 13, 2015].

Zenios, M., Goodyear, P. & Jones, C., 2004. Researching the impact of the networked information
environment on learning and teaching. Computers and Education, 43(1–2 SPEC ISS.), pp.205–213.

Zhang, K. & Bonk, C., 2008. Addressing diverse learner preferences and intelligences with emerging
technologies: Matching models to online opportunities. Canadian Journal of Learning and
Technology, 34(2), pp.309–332. Available at:
http://cjlt.csj.ualberta.ca/index.php/cjlt/article/viewArticle/496 [Accessed August 18, 2014].

Zhou, X., 2011. The research on online independent learning model based on computational thinking. In
2011 International Conference on E-Business and E-Government (ICEE). IEEE, pp. 1–5. Available at:
http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5881653 [Accessed July 10,
2015].

Zuckerman, M., 1994. Behavioral expressions and biosocial bases of sensation seeking, Cambridge
University Press. Available at:
http://books.google.com/books?hl=en&lr=&id=ApiyY8LX5fAC&oi=fnd&pg=PR9&dq=Behavioral+ex
pressions+and+biosocial+bases+of+sensation+seeking&ots=MSyFQWI7cO&sig=_UxDwRJwT18mCc
50vrhenKOMAEs [Accessed November 27, 2015].

Zvacek, S., 1991. Effective affective design. TechTrends, 36(1), pp.40–43. Available at:
http://link.springer.com/article/10.1007/BF02761286 [Accessed October 25, 2015].

332