There is incredible potential for digital technology in and beyond the classroom, but it is vital to rethink how learning is organized if we are to reap the rewards.

- Geoff Mulgan quoted in Burns (2012)

A story in the USA Today newspaper (García Mathewson and Butrymowicz 2020) was titled “Online programs used for coronavirus-era school promise results. The claims are misleading.” The article highlighted the fact that many online education providers who market their products to K-12 educators claim that their online programs are “proven” to be effective by scientific research, but that the evidence for such claims is typically very weak or even nonexistent. The article went on to state that this problem has been exacerbated by a virtual stampede to provide online learning opportunities in the wake of the coronavirus pandemic. School district administrators and other decision-makers have had to rush to find resources to support teaching and learning activities that have suddenly been forced online, and in this scramble they are likely to have been susceptible to false advertising. The authors pointed out that whereas the Food and Drug Administration in the USA restricts what pharmaceutical manufacturers can claim about their drugs and supplements, merchandizers of digital educational technologies have no such restraints.

The authors of the USA Today article are staff writers from The Hechinger Report, an independent nonprofit, nonpartisan organization focused on educating the public about education and how it can be improved. Perhaps there would be little need for The Hechinger Report if the What Works Clearinghouse that was established nearly twenty years ago by the U.S. Department of Education actually functioned as a reliable source of information about the quality of research on education products and programs. However, García Mathewson and Butrymowicz (2020) noted that “out of 10,654 studies included in the What Works Clearinghouse in mid-April, only 188—less than 2 percent—concluded that a product had strong or moderate evidence of effectiveness.”

So where are educational administrators, practitioners, and others to go to find the information that they need to make informed decisions about which educational programs to adopt if commercial advertising is untrustworthy and the What Works Clearinghouse can’t tell them what actually works? Hopefully they can find guidance in the types of systematic reviews found in this special edition of Educational Technology Research and Development. In this concluding paper, we examine the reviews published in this special issue, with special attention to the advice for practitioners that might be found in them. A caveat is warranted given that the systematic reviews in this special issue were not conducted specifically to provide practical implications, but nonetheless they all do to some extent as delineated below.

Countering bogus claims with evidence-based guidance

Exaggerated claims about education technology innovations are hardly new nor are literature reviews. In describing programmed instruction more than 50 years ago, Slack (1968) wrote:

Thanks to the genius of a few men who have devoted their efforts to new ways to write self-instructional materials — men such as B. F. Skinner, Thomas F. Gilbert, Lloyd Homme, Donald Tosti, Frederick Keller and others — the lowly workbook has undergone marvelous improvement in the last decade. The workbook has become so good that it works all by itself without a teacher to help fill in the answers. The new workbooks are guaranteed to teach all by themselves, without a teacher. They must be passed out, however, filled in by the student and collected by someone capable of managing the class. But this person does not have to do the teaching. That is done by the workbook, which, by the way, works just as well with drop-outs and delinquents as it does with nice little boys and girls. All in all, a marvelous thing. (Bold in original.)

The over one-hundred-year history of educational technology is replete with such outlandish claims starting perhaps with Edison’s prediction in 1913 that films would soon replace textbooks in classrooms (Reiser 2001). Literature reviews have also been around for decades. For example, more than a half century ago, Chu and Schramm (1967) synthesized the research on instructional television. As comprehensive as their seminal literature review was, the distinguished authors provided at best ambiguous guidance to practitioners:

Assuming a degree of caution in applying results, what kinds of guidelines can we extract from this body of research? For one thing, it has become clear that there is no longer any reason to raise the question whether instructional television can serve as an efficient tool of learning. This is not to say that it always does. But the evidence is now overwhelming that it can, and, under favorable circumstances, does. (Underlined words were italicized in original.) p. 98

The editors of this special edition of ETRD point out that the papers herein are not literature reviews in the traditional sense, but systematic reviews that utilize “a specific methodology to locate, select, evaluate, analyze, synthesize, and report evidence that helps to draw conclusions.” Meta-analysis is utilized in five of these systematic reviews whereas the other eight employ different cutting-edge review approaches such as scoping reviews (Arksey and O'Malley 2005).

Meta-analysis was largely unknown in educational technology research until Glass (1976) promoted its value for the educational research community at large. Since then, it has flourished with such high-profile educational researchers as Robert E. Slavin (cf. Cheung and Slavin 2012), John Hattie (cf. Hattie 2009), and James A. Kulik (cf. Kulik et al. 1985) conducting these types of analyses over the years. Unfortunately, educational technologies per se generally have not fared well in these analyses. For example, Cheung and Slavin (2013) conducted a meta-analysis to answer the question: “Do education technology applications improve mathematics achievement in K-12 classrooms as compared to traditional teaching methods without education technology?” They reported “the findings suggest that educational technology applications generally produced a positive, though modest, effect (ES =  + 0.15)” (p. 88).

The modest effect size reported by Cheung and Slavin (2013) does not come close to the 0.4 effect size that Hattie (2009) argued is necessary for any educational treatment to be taken seriously. Hattie’s (2009) comprehensive review of meta-analyses examining the effectiveness of 135 educational variables showed that only instructional design processes (e.g. feedback to students and mastery learning) rather than technologies per se (e.g. computer-based instruction or games) have demonstrated worthwhile impact on learning.

What guidance do systematic reviews provide practitioners?

In this section of the paper, we have attempted to tease out kernels of practical advice for practitioners that can be found in the 13 systematic reviews of the learning technologies and environments. Of course, we recognize that these systematic reviews were not written for practitioners but were primarily oriented toward other researchers and/or developers. We also acknowledge that the evidence that educational decision makers and practitioners actually avail themselves of educational research findings of any kind is lacking (Ion and Iucu 2014; Vanderlinde and van Braak 2010). Table 1 (below) summarizes the recommendations for other researchers proffered by the authors in their assorted reviews as well as the implications for practitioners that we have gleamed from the reviews.

Table 1 Summary of advice for researchers and practitioners based on the 13 systematic reviews
Full size table

The reviews published in this special issue of Educational Technology Research and Development have all been conducted rigorously according to the guidelines of the specific review processes used and reported skillfully with the guidance of the editors. As such, the papers provide an up-to-date portrayal of educational technology research across a wide variety of contexts. These reviews are especially useful in providing departure points for other researchers who seek to advance educational technology research in these contexts. Although clearly not intended to do so, each review also offers useful implications for practitioners.

In the light of the research we have, what is research we need?

Despite the high quality of the systematic reviews found in this special issue, they are only as good as the research studies that were incorporated into them. A major contributor to the paucity of practical guidance in these and other types of systematic reviews is that they are primarily focused on the “things” of our field such as wearable technologies and mobile devices rather than on the “problems” faced by teachers and students such as the lack of engagement of students in online learning (Stott 2016) or the failure to develop higher order learning in STEM education (Sadler and Zeidler 2009). Bonk and Wiley (2020) also note this tendency to focus on things rather than problems in the Preface to this special issue.

Improving educational opportunities is essential to addressing the major problems that the world confronts today such as poverty, climate change, racism, and the current global pandemic (Desai et al. 2018). Unfortunately, educational technology research does not have a distinguished record in dealing with local educational problems, much less global ones (Reich, in press). Arguably, a major contributor to this poor record is the focus on the things of educational technology rather than the problems of practitioners (Reeves and Reeves 2015a).

Fortunately, there are other ways to conduct educational technology research than the quasi-experimental methods employed in most of the studies included in the systematic reviews found in this special issue. One viable option is educational design research (EDR) (McKenney and Reeves 2019). EDR a genre of educational research in which the iterative development of solutions to complex educational problems through empirical investigations are pursued in tandem with efforts to reveal and enhance theoretical understanding. Such efforts can serve to guide educational practitioners as well as other researchers. As a genre of educational inquiry rather than a discrete methodology, EDR encompasses a family of approaches that endeavor to accomplish the twofold goal of designing and implementing interventions that solve serious educational problems while at the same time developing enhanced theoretical understanding to inform further practice. This family of approaches includes design-based research (Design-Based Research Collective 2003), design-based implementation research (Fishman et al. 2013), development research (van den Akker 1999), design experiments (Middleton et al. 2008), and formative research (Newman 1990), among others.

For most of its decades long history as a distinct field (Reiser 2001), educational technology researchers have focused primarily on the question “What works?” with respect to teaching and learning with technology leading to findings that are often so weak as to be practically inconsequential. Educational design research fundamentally changes the focus of research from the often fruitless “what works?” question to the more socially responsible questions “what is the problem, how can we solve it, and what new knowledge can derived from the solution?”

It should be clear we can no longer afford to have research agendas exclusively focused on things (e.g. digital games) rather than problems (e.g. the under-representation of women and minorities in STEM fields). Fortunately, the situation is improving. Notably, the new 5th edition of the Handbook of Research on Educational Communications and Technology (Bishop et al. 2020) has a very different focus from the previous four Handbooks that included many chapters focused on specific technologies as things. Whereas the fourth edition of the Handbook (Spector et al. 2014) has a dozen chapters on ‘emerging technologies’ such as e-books and open educational resources (OER), most chapters in the 5th edition are focused on complex problems and review the extent that these problems can be addressed using innovative learning designs and appropriate applications of technology. In the new Handbook, researchers and practitioners will find an excellent set of contributions that target serious educational problems such as increasing the accessibility of online learning environments and motivating and engaging students using emerging technologies. As a bonus, the 2020 Handbook includes thirteen design cases that are uniquely indexed with the research chapters focused on specific problems.

What else can be done to move from the research we have to the research we need? Reeves and Reeves (2015b) considered this question and responded by stating that our field is at a critical fork in the road:

Educational technology researchers may continue as we have for decades, conducting isolated studies focused on new things rather than significant problems, publishing our research in refereed journals just enough to ensure that our careers are advanced, working as recognized “scholars” but having virtually no impact on practice, and being largely unresponsive to the enormous challenges the world confronts around issues related to teaching and learning. Or we could take a new direction whereby we develop robust, multi-year research agendas focused on important problems and innovative solutions, judge our worthiness for promotion and tenure on evidence of impact rather than simple article counts, closely collaborate with practitioners, and establish our field as preeminent in meeting global problems related to education.

Of course, we are not suggesting that research focused on the things of educational technology has no merit, and we are fully cognizant that such studies will continue to be conducted every time an educational technology innovation appears. However, we encourage readers of this noteworthy special issue of ETR&D to focus on serious problems related to teaching, learning, and performance, collaborate closely with teachers, administrators, and other practitioners, and seek to make a difference in the lives of learners around the world. Instead of expecting the next “killer app” or technical innovation to transform education, perhaps we can better transform education by fostering incremental changes through collaborative research and development with practitioners (Reich, in press).