Teacher Development

An international journal of teachers' professional development

ISSN: 1366-4530 (Print) 1747-5120 (Online) Journal homepage: http://www.tandfonline.com/loi/rtde20

Measuring twenty-first century skills: development

and validation of a scale for in-service and pre-
service teachers

Yueming Jia, Youn Joo Oh, Bernadette Sibuma, Frank LaBanca & Mhora
Lorentson

To cite this article: Yueming Jia, Youn Joo Oh, Bernadette Sibuma, Frank LaBanca & Mhora
Lorentson (2016) Measuring twenty-first century skills: development and validation of a
scale for in-service and pre-service teachers, Teacher Development, 20:2, 229-252, DOI:
10.1080/13664530.2016.1143870

To link to this article: http://dx.doi.org/10.1080/13664530.2016.1143870

Published online: 23 Mar 2016.

Submit your article to this journal

Article views: 41

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=rtde20

Download by: [Universite Laval] Date: 06 April 2016, At: 19:23

 Teacher Development, 2016
Vol. 20, No. 2, 229–252, http://dx.doi.org/10.1080/13664530.2016.1143870

Measuring twenty-first century skills: development and validation

of a scale for in-service and pre-service teachers
Yueming Jiaa*, Youn Joo Oha, Bernadette Sibumaa, Frank LaBancab and
Mhora Lorentsonc
a
Leadership for Learning Innovation, Learning and Teaching Division, Education
Development Center Inc., 43 Foundry Avenue, Waltham, MA, USA; bCenter for 21st Century
Skills, EDUCATION CONNECTION, 355 Goshen Road, PO Box 909, Litchfield, CT, USA;
c
Center for Collaborative Evaluation and Strategic Change, EDUCATION
CONNECTION, 355 Goshen Road, PO Box 909, Litchfield, CT, USA
Downloaded by [Universite Laval] at 19:23 06 April 2016

(Received 29 May 2013; final version received 23 February 2015)

A self-report scale that measures teachers’ confidence in teaching students about

twenty-first century skills was developed and validated with pre-service and
in-service teachers. First, 16 items were created to measure teaching confidence
in six areas: information literacy, collaboration, communication, innovation and
creativity, problem solving, and responsible citizenship. Then, the items were
tested in two groups of pre-service teachers and two groups of in-service teach-
ers respectively. Exploratory factor analyses and confirmatory factor analyses
were conducted to examine the scale’s validity and reliability. The analyses iden-
tified a three-construct scale including innovation and problem solving, collabo-
ration, and utility of technology for pre-service teachers and a one-dimension
scale, cross-functional skills, for in-service teachers. This study suggests that dif-
ferent scales should be used to measure in-service teachers’ and pre-service
teachers’ confidence in teaching twenty-first century skills.
Keywords: twenty-first century skills; instrument development; factor analysis;
survey research; teacher effectiveness

Introduction
Teachers have been called to better prepare students for work in the twenty-first cen-
tury workplace. Twenty-first century workforce readiness requires students to be
able to apply knowledge to complex and challenging tasks using skills such as prob-
lem solving, evaluation, reasoning, decision-making, and the ability to use digital
technology (Darling-Hammond 2007; Greenhill 2010; Trilling and Fadel 2009). In
response, curriculum and instruction strategies that incorporate twenty-first century
skills into teaching and learning are being implemented nationwide (Coutinho and
Mota 2011; Gibson 2005; Lowther et al. 2012; O’Sullivan and Dallas 2010;
Thomas, Ge, and Greene 2011). However, there is a shortage of instruments
available to measure teachers’ development of twenty-first century skills teaching.

*Corresponding author. Email: yueming.jia@gmail.com

© 2016 Teacher Development

 230 Y. Jia et al.

The assessment of twenty-first century skills learning or twenty-first century

skills teaching has proven to be challenging for researchers, teachers, program devel-
opers, and administrators due to the variety of interacting factors of complex skills –
such as expert thinking, complex communication, and creative problem solving
(Mayrath et al. 2012). The few existing empirical studies focused on measuring stu-
dents’ twenty-first century skill learning and examined only a subset of twenty-first
century skills (Hodge and Lear 2011; Sardone and Devlin-Scherer 2010), used an
exploratory approach to assess such skills (McCreery, Schrader, and Krach 2011;
Thomas, Ge, and Greene 2011), and failed to conduct a systematic assessment of
the entire twenty-first century skill set. The lack of rigorous empirical testing and
validation creates major barriers to the development of scales that assess twenty-first
century skill learning and teaching.
From a teacher development perspective, the lack of reliable scales makes it dif-
ficult to know when additional training is needed to better prepare teachers to teach
twenty-first century skills in their classroom instruction. This study focuses on
Downloaded by [Universite Laval] at 19:23 06 April 2016

development and validation of such a self-report scale for the teaching of twenty-
first century skills. Using such a scale can pinpoint specific areas in which pre-ser-
vice teacher education and teacher professional development are needed to better
instruct and enhance twenty-first century skills teaching.
To develop the scale, the authors first generated a pool of items based on a
thorough review of literature. Subsequently, two experts in measurement and cur-
riculum reviewed the items. After revision of the items based on expert reviews, the
items in the scale were tested by two groups of pre-service teachers and two groups
of in-service teachers for revision and validation.

The twenty-first century skills frameworks

In parallel with the increased interest in teaching and learning of twenty-first century
skills, educational researchers have devoted significant efforts to the development of
a framework for twenty-first century skills. These efforts have led to the emergence
of a number of frameworks. Ruiz-Primo (2009) conducted a thorough review of five
major frameworks created by the Partnership for 21st Century Skills (2002), AASL
(2007), ISTE (2000), North Central Regional Educational Laboratory (Lemke 2002)
and National Research Council (Hilton 2008).
The Partnership for 21st Century Skills (2002) developed a Framework for 21st
Century Learning that combined specific student outcomes of twenty-first century
skills with support systems to prepare students to work in a global economy and
become more internationally competitive. This framework delineated outcome areas
for students as life and career skills, learning and innovation skills, information,
media, and technology skills, core subjects and twenty-first century themes. Since its
creation, many have referenced this particular framework in the development of
twenty-first century programs.
The AASL (2007) Standards for the 21st Century Learner identified the impor-
tance of four skill components to increase information literacy in the twenty-first
century including the ability to: (1) inquire, think critically and gain knowledge; (2)
draw conclusions, make informed decisions, apply and create knowledge; (3) share
knowledge and participate ethically and productively as members of a society; and
(4) pursue personal and aesthetic growth. Within each component, the AASL
standards outlined necessary skills, dispositions in action, responsibilities, and
 Teacher Development 231

self-assessment strategies that students may exhibit as evidence of achieving these

standards.
The ISTE’s (2000) National Educational Technology Standards focused on the
development of the ability to use technology effectively as a tool both for learning
and leading in an increasingly digital world. In the NETS framework, teachers serve
as role models for students who guide them to the best ethical and appropriate uses
of technology in face-to-face and online environments. It can be expected that, for
teachers to appropriately guide students, they must be able to adopt and model the
use of technology tools. ISTE includes five categories of standards with each cate-
gory defined by the intended audience (i.e. students (NETS-S), teachers (NETS-T),
school administrators (NETS-A), coaches (NETS-C), and computer science educa-
tors (NETS-CSE)).
The enGauge framework (Lemke 2002) identified four major skill clusters neces-
sary for success in the twenty-first century: Digital age literacy, inventive thinking,
effective communication, and high productivity. In the enGauge framework, twenty-
Downloaded by [Universite Laval] at 19:23 06 April 2016

first century learning occurs when a learner gains skills in each of the four clusters.
Acquisition of these skills is considered to lead to academic achievement. Addition-
ally, enGauge stated that the use of these standards in practice would decrease
achievement gaps and provide equal opportunities for individuals from all socioeco-
nomic, racial, and cultural groups.
Similarly, the National Research Council (Hilton 2008, 2010) designated five
key twenty-first century skills: adaptability, complex communication/social skills,
non-routine problem solving, self-management/self-development, and systems think-
ing. These skills have been found to be critical to success in the current and future
workplace. A detailed summary of the five frameworks based on Ruiz-Promo’s
review is provided in Table 1.
While each framework contains similar components, there is a lack of consis-
tency as to how specific indicators of twenty-first century skills are categorized.
Ruiz-Primo (2009) compared the above frameworks and concluded that non-routine
problem solving and complex communication/social skills most closely indicated
science proficiency and proposed that the dimensions in these frameworks could be
loaded cross-functional skills – cognitive skills such as thinking, understanding,
learning, and remembering that are likely to be used in any domain. Cross-functional
skills, in combination with dispositions (general inclinations or attitudes of mind)
and science knowledge, were proposed as complete domains of twenty-first century
skills in the context of science education.
In the meantime, based on NCREL/enGauge, ISTE/NETS, and the Partnership
for 21st century Skills standards, Costa and Cogan-Drew (2009) further developed
the areas of twenty-first century skills and proposed the following six essentials:
Information Literacy, Collaboration, Communication, Innovation/Creativity, Problem
Solving, and Responsible Citizenship. In our study, we used Costa and Cogan-
Drew’s six skills as a framework to create our item pool of twenty-first century skill
teaching measure.

Empirical studies
The majority of articles on twenty-first century skills describe implementation of the
skills but with little evidence of corresponding data (Lowther et al. 2012; Moylan
2008; Thomas, Ge, and Greene 2011). Of the empirical studies that have been
 Downloaded by [Universite Laval] at 19:23 06 April 2016

Table 1. Summary of twenty-first century skills frameworks.

232

Framework Components and descriptions

Partnership for 21st Century Skills (2002) 1. Life and career skills: flexibility, initiative and self-direction, social and cross-culture skills, productivity
and accountability, leadership and responsibility
2. Learning and innovation skills: critical thinking, communication, collaboration and creativity
3. Information, media, and technology skills: practical and ethical literacy in the use of new information,
Y. Jia et al.

media, and technologies

4. Core subjects: the traditional subject categories in school curriculums, with the added twenty-first
century themes of global awareness, business/financial literacy, civic literacy, health literacy, and
environmental literacy
AASL Standards for the 21st Century 1. Inquire, think critically and gain knowledge: the importance of student acquisition of inquiry-based
Learner (2007) learning skills: knowing how to use different technologies to search for and evaluate information and
sources that answer questions
2. Draw conclusions, make informed decisions, apply and create knowledge: critical thinking skills as well
as the organization and creation of new knowledge based on what is known
3. Share knowledge and participate ethically and productively as members of a society: social skills,
dealing with communication and work skills with others in one’s community and as part of a pluralistic,
democratic society
4. Pursue personal and aesthetic growth: maintaining a curiosity for the world around oneself by engaging
in reading new literature, in conversation with those with different perspectives, and pursuing
independent learning about topics that are personally interesting
ISTE’s National Educational Technology Standards for students (NETS-S):
Standards (2000) 1. Creativity and innovation
2. Communication and collaboration
3. Research and information literacy
4. Critical thinking, problem solving and decision-making
5. Digital citizenship
6. Technology operations and concepts
Standards for teachers (NETS-T):
1. Facilitate and inspire student learning and creativity
2. Design and develop digital age learning experiences and assessments
3. Model digital age work and learning
4. Promote and model digital citizenship and responsibility
5. Engage in professional growth and leadership
 Downloaded by [Universite Laval] at 19:23 06 April 2016

NCREL’s enGauge framework (2002) 1. Digital age literacy: basic, scientific, economic, and technological literacy skills; visual, information,
multicultural literacies, and global awareness
2. Inventive thinking: an individual’s adaptability to change, management of complexity, curiosity,
creativity, risk-taking, and higher order thinking
3. Effective communication: the ability to collaborate and interact with others, and one’s sense of personal,
social, and civic responsibility
4. High productivity: being able to prioritize, plan, and manage for results. Use tools effectively and create
relevant, high-quality products
National Research Council (2008) 1. Adaptability: an individual’s ability to function well under new or different circumstances, such as
learning new tools and technologies on the job, handling crisis, new personalities, or cultures with ease
2. Complex communication/social skills: being able to effectively articulate ideas as well as incorporate
verbal and nonverbal information to generate favorable responses from others, including subordinates
and superiors
3. Non-routine problem solving: thinking creatively about solutions to problems or products
4. Self-management/self-development: the ability to work independently, including taking initiative to learn
skills needed for a job
5. Systems thinking: the ability to understand the interrelatedness/interconnections between different parts
of an organization
Teacher Development
233
 234 Y. Jia et al.

conducted, study designs rarely map directly on to a specific twenty-first century

framework. Instead, twenty-first century skills are discussed in broad terms, such as
communication, collaboration, or problem solving. In short, rigorous research
designs and empirical data to understand how twenty-first century skills can be
assessed effectively and accurately are lacking.
A number of articles identify the impact of a particular activity or curriculum
intervention on twenty-first century skill development in project-based learning
designs (Moylan 2008), video game design (Thomas, Ge, and Greene 2011), pod-
casts (Coutinho and Mota 2011), laptop programs (Lowther et al. 2012), global
learning environments (Gibson 2005), and research assignments (O’Sullivan and
Dallas 2010). Unfortunately, empirical data is not cited. For instance, Smith and
Dobson’s (2011) study used ISTE’s NET-T and NET-S to align the Voice thread pro-
ject for pre-service teachers. Using student surveys, lesson plans, online discussions,
and teacher reflections, the authors investigated teacher perceptions of integrating
twenty-first century skills using Web 2.0 tools, but did not specify precisely which
Downloaded by [Universite Laval] at 19:23 06 April 2016

skills were imparted by which tools. Likewise, in Lambert and Gong’s (2010) study,
ISTE’s NET-S standards were used to redesign an educational technology course for
pre-service teachers. They examined the effects of the redesign on pre-service teach-
ers’ self-efficacy, attitudes, and acquisition of technology skills. Although the con-
tent of the course was twenty-first century skills, the study specifically focused on
technology skill acquisition, not communication, collaboration, or problem solving.
A few empirical studies map directly on to the stated twenty-first century skills.
Hodge and Lear (2011) conducted a survey of 254 business students and 37 profes-
sors to determine what skills were perceived to be the most important for success in
the twenty-first century workforce. These skills were selected after a literature
review including the Partnership for 21st Century Learning. The researchers asked
the United States and international faculty and students to rank 17 twenty-first cen-
tury workforce skills according to their perceived importance to successful employ-
ment after graduation. Differences were found in the rankings between faculty and
the United States and international students, and between students and employers.
The United States students ranked managerial skills as the most important skills to
obtain a job after graduation while faculty members ranked them significantly lower
but, instead, considered interpersonal skills to be most valuable. Critical thinking
and problem-solving skills were considered more important to job procurement by
faculty than by the United States students. The researchers found that the United
States students perceived critical thinking, problem solving, communication, and
creativity to be lower in importance than business executives.
Sardone and Devlin-Scherer (2010) conducted a qualitative study on teacher can-
didates’ use of digital games with high school and middle school students to teach
twenty-first century skills: critical thinking, problem solving, creativity, innovation,
communication, and collaboration. They determined that peer modelling and posi-
tive student feedback made it more likely that teacher candidates would continue to
use digital games in the classroom.
A study conducted in Malaysia cited the enGauge twenty-first century skills and
the Partnership for 21st Century learning frameworks and focused on inventive
thinking as it was presented in both frameworks (Abdullah and Osman 2010). Using
a questionnaire framed by the enGauge twenty-first century skills, the study deter-
mined that 410 grade 5 students in Brunei scored higher on inventive thinking than
students in Malaysia while Malaysian students scored higher on risk taking and
 Teacher Development 235

curiosity. Both groups scored low on curiosity and problem-solving skills. Abdullah
and Osman’s research provides an interesting lens to examine cross-cultural differ-
ences in twenty-first century skill acquisition using the existing enGauge twenty-first
century skills and the Partnership for 21st century learning frameworks.
In summary, although a few published empirical studies were found, evidence
for the effectiveness of interventions which enhanced twenty-first century skills in
teaching and learning is noticeably lacking. Needless to say, the field is ripe for rig-
orous research that is grounded on a well-developed twenty-first century framework.
The present study develops a self-report scale to understand teachers’ perceptions of
their capability and readiness to teach twenty-first century skills to high school
students.

The current study

The development of an initial pool of items
Downloaded by [Universite Laval] at 19:23 06 April 2016

Based on Costa and Cogan-Drew’s (2009) framework of six foundational skills for
twenty-first century success, a pool of 32 items was created tapping six aspects of
twenty-first century skills teaching: Information Literacy, Collaboration, Communi-
cation, Innovation/Creativity, Problem Solving, and Responsible Citizenship. Infor-
mation literacy contained items about teaching students how to locate, select, and
use information. Collaboration contained items related to teaching students to work
in a team, be a team leader, and work independently. Communication assessed items
related to the use of technology to effectively communicate ideas and concepts.
Innovation and Creativity items examined the ability to create an innovative solution
to a scientific or business problem and strategies to determine if an idea for a service
or product is original, useful, and valuable. Problem solving included items on
designing research to solve real work problems, analyzing data, evaluating questions
and validity of data, and applying research findings. Responsible citizenship was
indicated by items related to respect for individuals from other cultures, responsible
use of technology, and creation of projects or ideas valuable to society.

Expert review
After the establishment of the initial pool of items, two experts in measurement, cur-
riculum, and literature on twenty-first century skills individually reviewed all items.
The experts agreed on the quality of 16 items, which were evaluated based on the
clarity of language, relevancy to the concept to be assessed, and uniqueness of idea
that the item conveyed relative to other items. The 16 items kept the six aspects of
teaching twenty-first century skills: Information Literacy, Collaboration, Communi-
cation, Innovation/Creativity, Problem Solving, and Responsible Citizenship.

Field testing
After the initial creation of the 16-item scale, we administered the scale to teachers
for further revision. Some researchers have argued that teachers’ perceptions were
formed based on both formal knowledge and classroom teaching experience (Novak
and Knowles 1992; Powell 1992). We speculated that the scale may work differently
for pre-service teachers whose teaching perceptions were shaped mainly through
formal knowledge, than for in-service teachers whose teaching perceptions may be
 236 Y. Jia et al.

substantially influenced by their classroom teaching experience. Therefore, we

piloted this scale with two groups of teachers, one was pre-service teachers and the
other group consisted of in-service high school teachers who taught science, technol-
ogy, engineering, or mathematics (STEM).
Exploratory factor analyses (EFA) were conducted with data collected from the
two groups of teachers to test item quality and dimensionality of the scale. Revisions
were made based on results of this pilot. After that, the revised scale(s) were admin-
istered to another two groups of pre-service teachers and in-service high school
teachers for confirmation of the scale construct.

The 21st Century Skills Teaching Scale

The examined scale contains a total of 16 items. These items tap into six aspects of
teaching twenty-first century skills including information literacy, collaboration,
communication, innovation and creativity, problem solving, and responsible citizen-
Downloaded by [Universite Laval] at 19:23 06 April 2016

ship. In this current study, the majority of the in-service teachers taught STEM.
Therefore, questions were framed in STEM settings to specifically address twenty-
first century skills when teaching in STEM, e.g. ‘Teaching students to identify
necessary information to accomplish a STEM task.’ Participants rated their level of
confidence in teaching activities defined by each item using a 7-point Likert scale
ranging from 1 (not at all confident) to 7 (completely confident).

Procedures
Two groups of teachers, one group was pre-service teachers and the other was in-
service teachers, were administered 16 items of the 21st Century Skills Teaching
Scale online using a web-enabled computer. Another two groups of pre-service and
in-service teacher participants were administered 10 items. Consent was collected
prior to survey administration.

Analytic strategies
The purpose of field testing is to help us understand the construct validity of the
scale, providing evidence for scale revision and validation. To achieve this goal, we
relied on two types of analyses, EFA and confirmatory factor analysis (CFA).
The EFA examined the dimensionality and the quality of items, forming the base
for removal or maintenance of factors or items. The number of dimensions was
determined based on scree plot, eigenvalues, and model fit indices. Model fit indices
such as standardized root mean square residual (SRMR), root mean square error of
approximation (RMSEA), comparative fit index (CFI), and the Tucker–Lewis index
(TLI) were used as the criteria of the goodness of fit.1 The value of Chi-square was
also provided, however, Chi-squares are often biased by sample size. Therefore, χ2/
df was calculated and a value close to 2 indicates a good fit of the model (Bentler
1995; Byrne 2001). Item quality was examined based on their factor loadings. A
high-quality item should have a factor loading larger than .45 and no cross-loading
with other factors.2 The quality of dimensions was evaluated according to the num-
ber (more than three items) and the quality of items in the dimension. In each round
of analyses, low-quality items and dimensions were removed and then new rounds
of analyses were conducted. After several rounds of EFAs, a factor model contained
 Teacher Development 237

high-quality items and dimensions were selected and examined in a CFA model
using a new group of participants.
The CFA was to verify the validity of the items and factors identified in EFA.
The goodness of fit for the CFA model was judged based on the same model fit
indices used in the EFA such as SRMR, RMSEA, CFI, and TLI. Cronbach’s alpha
of the final model was also calculated as an estimate of the reliability.
All analyses were conducted using Mplus Version 6.1 (Muthén and Muthén
1998–2010). Maximum Likelihood estimation was used to a correlation matrix. The
potential factors in the scale were assumed to correlate with each other, and oblique
rotation was applied to reach an interpretable solution. Although Likert scale vari-
ables are categorical in nature, the use of seven points allowed us to treat them as
continuous variables (Johnson and Creech 1983).

Pre-service teachers
Downloaded by [Universite Laval] at 19:23 06 April 2016

Participants
This sub-study involved two groups of pre-service teachers. Group 1 included 151
pre-service teachers studying at the school of education in one Canadian university.
Seventy-five percent of the participants were female. Fifty percent of this sample
focused on social science education, 17% on art education, 14% on science educa-
tion, 2% on music education, 1% on technology education, 4% on other areas, and
13% failed to reported their area of study. EFAs were conducted with data collected
from this group.
Group 2 involved 70 pre-service teachers. Sixty-one percent of the participants
were female. Sixteen percent of them focused on science education, 16% on math edu-
cation, 69% on other areas. CFAs were conducted with data collected from this group.

Exploratory factor analysis

Factorability of the correlation matrix
Before conducting EFA, we examined whether our current sample size is adequate
for this type of analysis. We examined the factorability of the study using three indi-
cators, the ratio of participants to items, Bartlett’s test of sphericity, and Kaiser–
Meyer–Oklin (KMO) measure of sampling adequacy. The ratio of participants to
item in our study was about 9:1, slightly lower than Gorsuch’s (1983) suggestion for
a minimum ratio of 10:1. Bartlett’s test of sphericity revealed that the correlation in
the matrix of the sample was significantly different from 0. The size of KMO mea-
sure was .89, which was larger than the cutoff point of .60 suggested by Tabachnick
and Fidell (2001) for good factor analysis. In sum, the above tests showed a good
factorability of the study.

EFA
Based on indicators such as eigenvalue, scree plot, and model fit indices,3 a promis-
ing 4-construct model of the scale was revealed after the first round of factor analy-
sis, which means that the items were clustered into groups that measured four
different constructs. A review of the items in the four clusters showed that the items
assessed constructs related to utility of technology, collaboration, innovation and
 Downloaded by [Universite Laval] at 19:23 06 April 2016

238
Y. Jia et al.

Table 2. Goodness of fit indicators for factor solutions.

Study 1 Study 2
2 2
Round Factor Solution x d χ /df RMSEA SRMR CFI x2 df χ2/df RMSEA SRMR CFI
1 1 361.41 104 3.47 .13 .08 .77 387.73 104 3.73 .13 .06 .85
2 264.77 89 2.97 .11 .06 .84 285.01 89 3.20 .12 .05 .89
3 204.34 75 2.72 .11 .05 .89 188.11 75 2.51 .10 .03 .94
4 131.81 62 2.12 .09 .04 .94
2 1 274.11 77 3.56 .13 .08 .79 283.19 77 3.68 .13 .06 .86
2 177.94 64 2.78 .11 .05 .88 198.78 64 3.11 .12 .05 .91
3 112.13 52 2.16 .09 .04 .94
3 1 230.54 54 4.27 .15 .09 .76 184.40 54 3.41 .12 .06 .90
2 140.29 43 3.26 .12 .06 .87 105.27 43 2.45 .10 .04 .95
3 75.24 33 2.28 .09 .04 .94
4 1 187.52 35 5.36 .17 .09 .75 84.92 35 2.43 .10 .04 .95
2 105.18 26 4.05 .14 .06 .87
3 44.15 18 2.45 .10 .04 .97
 Downloaded by [Universite Laval] at 19:23 06 April 2016

Table 3. Study 1: Factor loadings.

Round 1 Round 2 Round 3 Round 4 (Final)
Factors Factors Factors Factors
Item 1 2 3 4 1 2 3 1 2 3 1 2 3
1. Teaching students to use digital tools to locate information .85 .07 .02 .02 .95 .01 −.13 .89 .05 −.01 .89 .05 −.01
(digital tools: online books and articles, Internet searches,
online discussions, gathering information from websites,
or online databases)
2. Teaching students to identify necessary information to .32 .33 .31 −.07 .32 .36 .20 \\ \\ \\ \\ \\ \\
accomplish a task
3. Teaching students to take the lead on a group project .09 .63 .16 −.03 .08 .70 .01 .07 .56 .22 .06 .56 .23
4. Engaging students in collaborating with peers to achieve a .01 .85 −.02 .08 −.00 1.05 −.27 −.02 .96 −.02 −.01 .96 −.01
goal on a project
5. Teaching students to use technology tools (e.g. online .71 .02 .06 .00 .74 −.02 .01 .68 .00 .10 .69 .00 .08
forums/discussion, PowerPoint presentations, digital
media project) to clearly communicate concepts
6. Teaching students oral presentation skills to clearly .08 .40 .23 .06 .09 .48 .10 .07 .38 .28 \\ \\ \\
communicate concepts
7. Teaching students to use technical writing to clearly −.01 .30 .40 −.01 −.01 .42 .25 −.02 .32 .41 \\ \\ \\
communicate topics
8. Engaging students in identifying ‘real-world’ challenges −.08 .20 .59 .02 −.03 .29 .48 −.03 .21 .59 −.04 .21 .58
or problems
9. Teaching students to determine an innovative solution to a .15 −.07 .53 .15 .23 −.02 .50 .22 −.02 .53 .20 −.01 .56
challenge (e.g. digital animation, oil spill clean-up,
application of nanotechnology)
10. Teaching students to evaluate the quality of an idea for a .05 −.08 .91 .04 .13 .02 .80 .12 .00 .83 .09 .01 .85
product
11. Teaching students to evaluate the validity of data or −.03 .19 .68 −.10 −.00 .24 .60 .01 .16 .65 .00 .17 .63
evidence collected from a project
12. Encouraging students to apply concepts to solve problems .01 .29 .51 .08 .06 .36 .42 \\ \\ \\ \\ \\ \\
Teacher Development

in other areas

(Continued)
239
 Downloaded by [Universite Laval] at 19:23 06 April 2016

Table 3. (Continued).
240

Round 1 Round 2 Round 3 Round 4 (Final)

Factors Factors Factors Factors
Item 1 2 3 4 1 2 3 1 2 3 1 2 3
13. Teaching students to respectfully work with individuals −.05 .50 −.03 .50 −.01 .59 −.02 −.01 .49 .13 −.01 .49 .14
Y. Jia et al.

from different cultures

14. Engaging students in using reflective practices (e.g. blog .02 .07 .39 .42 \\ \\ \\ \\ \\ \\ \\ \\ \\
journal, discussions with mentor, teacher, or peer) to
foster a lifelong learning process
15. Teaching students to use technology in a responsible way .49 −.05 −.00 .52 .52 −.01 .08 .47 −.01 .17 .48 −.01 .15
(e.g. protecting digital identity, protecting the rights of
others, following licensing laws)
16. Engaging students in conducting a project that has a value .01 .36 .15 .49 \\ \\ \\ \\ \\ \\ \\ \\ \\
to society
 Teacher Development 241

problem solving, and responsible citizenship (see Table 3) respectively. However, all
items in the sub-construct of responsible citizenship simultaneously loaded on
multiple clusters, indicating that this construct is not uniquely defined by its items.
Therefore, we dropped this construct from the scale. Correspondingly, the two items
that primarily loaded on responsible citizenship, ‘Using reflective practices (e.g. blog
journal, discussions with mentor, teacher, or peer) to foster a lifelong learning pro-
cess’ and ‘Conducting a project that has a value to society’ were also removed from
the scale. Using the remaining 14 items, we conducted another round of factor
analysis and examined the items and factors again. A total of three rounds of factor
analyses were conducted before a final acceptable model was identified. During
these analyses, another four items were removed from the scale because they were
loaded into multiple clusters or had a weak bond with a cluster or didn’t load to any
cluster or they belonged to a low-quality factor.
In the fourth rounds of factor analyses, a three-construct solution containing 10
items (Table 3) was eventually identified. All items in this model clearly loaded to
Downloaded by [Universite Laval] at 19:23 06 April 2016

one or another factor and had a loading larger than .45. The model fit indices table
(Table 2) revealed that this three-factor solution had an acceptable model fit. The 10
items measured three constructs: Construct 1, utility of technology, contained three
items; Construct 2, collaboration, contained three items; Construct 3, innovation and
problem solving, contained four items (Table 3).The three constructs explained 68%
of the total variance. Correlations among the three constructs ranged from .32 to
.47, which indicated they were separate but related.4 The internal consistency
reliability of each factor, which indicated how much the items were correlated and
measured the same idea, was acceptable with .78 for Construct 1, .74 for Construct
2, and .81 for Construct 3.

Confirmatory factor analysis

The newly identified 10-item model was examined again with group 2 participants
to verify its construct validity using CFA. The results showed that the factor load-
ings for the three constructs were acceptable.5 However, the model fit indices
showed a less desirable model fit (Table 6). An examination of modification indexes
(MI) revealed three highly correlated pairs of items. One pair was ‘Teaching stu-
dents to take the lead on a group project’ and ‘Engaging students in collaborating
with peers to achieve a goal on a project’. Both items were related to skills of work-
ing in a group. Therefore, correlations between these pairs were added to the model.
Other pairs were not connected conceptually, and thus no correlation of these pairs
was included in the model.
The new model with correlation of one pair showed an improved and acceptable
fit (Table 6). The internal consistency for each construct also ranged from good to
acceptable, .83 for Innovation and Problem Solving, .81 for Utility of Technology,
and .75 for Collaboration. In sum, the CFA model proved that the three-factor 21st
Century Skills Teaching Scale had a good construct validity.

In-service teachers
Participants
This sub-study involved two groups of in-service teachers. Group 1 included 158 in-
service STEM teachers. Sixty-three percent of the participants were female. Forty-
 242 Y. Jia et al.

three percent were science teachers, 24% math teachers, 22% technology teachers,
and 11% taught other subjects .The majority (81%) had more than five years
teaching experience. EFAs were conducted with data collected from this group.
Group 2 involved 95 in-service STEM teachers. Sixty-three percent of the partic-
ipants were female. Forty percent were science teachers, 22% math teachers, 37%
technology teachers, 1% did not identify their teaching area. The majority (77%)
had more than five years teaching experience. CFAs were conducted with data
collected from this group.

Exploratory factor analysis

Factorability of the correlation matrix
The ratio of participants to items for the EFA was about 10:1, which met Gorsuch’s
(1983) suggestion for minimum ratio. Bartlett’s test of sphericity revealed that the
correlation in the matrix of the sample was significantly different from 0. Further-
Downloaded by [Universite Laval] at 19:23 06 April 2016

more, the KMO measure of sampling adequacy size for the sample was .93. The test
showed a good factorability of the current sample.

EFA
The first round of analysis revealed a promising three-construct model,6 in which
the 16 items clustered into three groups. However, two of the three constructs were
highly correlated (r = .75).Therefore the two constructs were merged into one, which
led to a two-construct solution. One group of items assessed things related to utility
of technology; the other included items assessing collaboration, communication,
innovation and creativity, problem solving as well as responsible citizenship, which
corresponded to Ruiz-Primo’s (2009) definition of cross-functional skills. Therefore,
we named this construct cross-functional skills (see Table 4). Two items, ‘Teaching
students to identify necessary information to accomplish a STEM task’ and ‘Teach-
ing students to take the lead on a group STEM project’ were cross-loaded on both
of the two factors (see Table 4). Thus, the items were removed from the scale and a
new round of analysis was conducted with the remaining 14 items.
After four rounds of analyses, a good-quality one-construct model with 10 items
was revealed, measuring cross-functional skills. The one-construct model showed an
acceptable fit (see Table 4), and explained 62% of the total variance. The internal
consistency of the scale was also high (.93).

Confirmatory factor analysis

The one-construct model explored in EFA was verified with Group 2 in-service
teachers using confirmatory factor analysis. The results showed that most of the
items’ factor loadings were quite high and ranged from .81 to .91. Only one item
had a relatively lower loading of .47. The model fit indices, however, indicated a
less desirable fit of the model (Table 7).
An examination of modification indexes (MI) revealed four highly correlated
pairs of items. Pair 1 was, ‘Engaging students in making oral presentation to clearly
communicate STEM topics’ and ‘Teaching students to use technical writing to
clearly communication STEM topics’, both of which were related to communication
 Downloaded by [Universite Laval] at 19:23 06 April 2016

Table 4. Study 2: Factor loadings.

Round 1 Round 2 Round 3 Round 4
Factors Factors Factors Factors
Item 1 2 1 2 1 2 1
1. Teaching students to use digital tools to locate information to understand STEM (Science, .89 −.00 .18 .62 .34 .45 \\
Technology, Engineering, and Math) concepts. (digital tools: online books and articles,
Internet searches, online discussions, gathering information from websites, or online
databases)
2. Teaching students to identify necessary information to accomplish a STEM task .39 .48 \\ \\ \\ \\ \\
3. Teaching students to take the lead on a group project in STEM .42 .52 \\ \\ \\ \\ \\
4. Engaging students in collaborating with peers to achieve a goal on a STEM project .29 .57 .58 .25 .67 .12 .73
5. Teaching students to use technology tools (e.g. online forums/discussion, PowerPoint .61 .19 −.00 .92 .00 1.09 \\
presentations, digital media project) to clearly communicate STEM concepts
6. Engaging students in making oral presentation to clearly communicate STEM topics .00 .71 .57 .20 .61 .16 .70
7. Teaching students to use technical writing to clearly communicate STEM topics .10 .59 .61 .07 .63 .05 .65
8. Engaging students in identifying ‘real-world’ challenges or problems in STEM areas −.08 .87 .82 −.00 .84 −.04 .82
9. Teaching students to determine an innovative solution to a STEM challenge (e.g. digital .21 .68 .66 .25 .73 .17 .81
animation, oil spill clean-up, application of nanotechnology)
10. Teaching students to evaluate the quality of an idea for a STEM product −.02 .86 .89 −.04 .90 −.07 .86
11. Teaching students to evaluate the validity of data or evidence collected from a STEM .01 .69 .80 −.13 .79 −.14 .71
product
12. Encouraging students to apply STEM concepts to solve problems in other areas −.06 .86 .84 −.01 .83 −.00 .83
13.Teaching students to respectfully work with individuals from different cultures −.12 .70 .57 .08 .57 .04 .60
14. Engaging students in using reflective practices (e.g. blog journal, discussions with mentor, .03 .68 .50 .30 \\ \\ \\
teacher, or peer) to foster a lifelong learning process
15. Teaching students to use technology in a responsible way (e.g. protecting digital identity, .27 .47 .46 .29 \\ \\ \\
protecting the rights of others, following licensing laws)
16. Engaging students in conducting a STEM project that has a value to society .06 .82 .77 .14 .82 .06 .85
Teacher Development
243
 244 Y. Jia et al.

skills. Pair 2 was items associated with skills of creating ideas, ‘Engaging students
in identifying real-world challenges or problems in STEM areas’ and ‘Teaching stu-
dents to evaluate the quality of an idea for a STEM product’. Pair 3 focused on
STEM application, ‘Teaching students to evaluate the validity of data or evidence
collected from a STEM product’ and ‘Encouraging students to apply STEM con-
cepts to solve problems in other areas’. Therefore, correlations between these pairs
were added to the model. However, Pair 4, ‘Engaging students in collaborating with
peers to achieve a goal on a STEM project’ and ‘Engaging students in making oral
presentations to clearly communicate STEM topics’ were not connected conceptu-
ally, and thus no correlation of this pair was included in the model.
The new model with correlation of three pairs showed an improved and
acceptable fit (Table 7). The internal consistency of this one-dimensional scale was
high (.96). In sum, the CFA showed that this one-dimensional 21st Century Skills
Teaching Scale had a good construct validity.
Downloaded by [Universite Laval] at 19:23 06 April 2016

Comparison between hypothesized and observed factors

In this study, we first created an original 16-item scale measuring teaching of
twenty-first century skills in six categories: (1) information literacy, (2) collabora-
tion, (3) communication, (4) innovation and creativity, (5) problem solving, and (6)
responsible citizenship, which was based on Costa and Cogan-Drew’s (2009)
framework of six twenty-first century skills. Then, we administered this scale to
several samples of in-service teachers and pre-service teachers respectively to
examine the constructs and item quality of the scale for further revision. The
results identified different constructs and items for in-service teachers and pre-
service teachers.
For pre-service teachers, a 10-item, three-construct scale was identified based on
the results of exploratory and confirmatory factor analyses. In this scale, items in the
original category of innovation and creativity and problem solving were found to
load into one single construct instead of two. We named this construct to be ‘innova-
tion and problem solving’; items in the original category of collaboration did load
together as hypothesized. However, one item, ‘Teaching students to respectfully
work with individuals from different cultures’, which was originally assigned in the
category of responsible citizenship, was also loaded to the collaboration category.
The item seems to be a good match with this category since it does address a skill
needed in a twenty-first century teamwork environment. The hypothesized category
of responsible citizenship, information literacy, and communication failed to be iden-
tified. Instead, three items addressing skills-related technology usage extracted from
each of the three hypothesized categories, loaded together, forming a new construct
which was named utility of technology.
For in-service teachers with a focus on STEM teaching, the analyses revealed
a 10-item (five overlapped with the pre-service teachers’ scale), one-construct
scale. In this scale, items in the original category of collaboration, communication,
innovation and creativity, and problem solving were clustered together, and named
‘cross-functional skills’. The hypothesized categories of responsible citizenship and
information literacy were also not identified. Table 5 provides a summary of these
results.
 Downloaded by [Universite Laval] at 19:23 06 April 2016

Table 5. Comparison between hypothesized factors and observed factors.

Hypothesized Observed factors in Observed factors
Items factors study 1 in study 2
1. Teaching students to use digital tools to locate information to understand (STEM) Information Utility of \\
concepts (digital tools: online books and articles, Internet searches, online discussions, literacy technology
gathering information from websites, or online databases)
2. Teaching students to identify necessary information to accomplish a (STEM) task Information \\ \\
literacy
3. Teaching students to take the lead on a group (STEM) project Collaboration Collaboration \\
4. Engaging students in collaborating with peers to achieve a goal on a (STEM) project Collaboration Collaboration Cross-functional
skills
5. Teaching students to use technology tools (e.g. online forums/discussion, PowerPoint Communication Utility of \\
presentations, digital media project) to clearly communicate (STEM) concepts technology
6. Engaging students in oral presentation skills to clearly communicate (STEM) concepts Communication \\ Cross-functional
skills
7. Teaching students to use technical writing to clearly communicate topics Communication \\ Cross-functional
skills
8. Engaging students in identifying ‘real-world’ challenges or problems Innovation and Innovation and Cross-functional
creativity problem solving skills
9. Teaching students to determine an innovative solution to a (STEM) challenge (e.g. digital Innovation and Innovation and Cross-functional
animation, oil spill clean-up, application of nanotechnology) creativity problem solving skills
10. Teaching students to evaluate the quality of an idea for a (STEM) product Problem Innovation and Cross-functional
solving problem solving skills
11. Teaching students to evaluate the validity of data or evidence collected from a (STEM) Problem Innovation and Cross-functional
project solving problem solving skills
12. Encouraging students to apply (STEM) concepts to solve problems in other areas Problem \\ Cross-functional
solving skills
13. Teaching students to respectfully work with individuals from different cultures Responsible Collaboration Cross-functional
citizenship skills
14. Engaging students in using reflective practices (e.g. blog journal, discussions with Responsible \\ \\
mentor, teacher, or peer) to foster a lifelong learning process citizenship
Teacher Development

(Continued)
245
 Downloaded by [Universite Laval] at 19:23 06 April 2016

Table 5. (Continued).
246

Hypothesized Observed factors in Observed factors

Items factors study 1 in study 2
15. Teaching students to use technology in a responsible way (e.g. protecting digital identity, Responsible Utility of \\
protecting the rights of others, following licensing laws) citizenship technology
16. Engaging students in conducting a (STEM) project that has a value to society Responsible \\ Cross-functional
Y. Jia et al.

citizenship skills
 Teacher Development 247

Table 6. Goodness of fit indicators for the CFA model.

x2 df χ2/df RMSEA SRMR CFI TLI
Model 1 51.42 32 1.61 .09 .07 0.92 0.89
Model 2 42.93 31 1.38 .07 .06 0.95 0.93

Table 7. Goodness of fit indicators for the CFA model.

x2 df χ2/df RMSEA SRMR CFI TLI
Model 1 119.40 35 3.41 0.16 .04 0.91 0.86
Model 2 73.24 32 2.29 0.12 .04 0.96 0.94

Discussion
Downloaded by [Universite Laval] at 19:23 06 April 2016

The goal of the study was to develop a self-report scale to understand teachers’ per-
ceptions of their capability and readiness of teaching to teach twenty-first century
skills. After a literature review, pilot test, and expert reviews, a pool of 16 items was
developed and examined to understand their dimensionalities and item quality in
two groups, one group was pre-service teachers and the other was in-service teach-
ers. The results identified two different scales for pre-service teachers and in-service
STEM teachers. The former was a 10-item three-dimensional scale including utility
of technology, collaboration, and innovation and problem solving; the latter was a
10-item uni-dimensional scale tapping into teaching of what Ruiz-Primo (2009)
called ‘cross-functional skills’. The factor structures and item quality of the two
identified scales were confirmed in further analyses with two new samples of
pre-service and in-service teachers.
Consistent with the concept of the ‘Four Cs’ twenty-first century skills (Beers
2001), both of the two scales contain items tapping into teaching of creative and
innovative thinking, problem-solving skills, effective communication, and working
in a group. However, the categorization of these items did not map well with the
hypothesized subcategories (information literacy, collaboration, communication,
innovation and creativity, problem solving, and responsible citizenship) that served
as the basis for this scale. In the two studies, the six categories either collapsed
together or fell out of the scale, resulting in one (in-service teachers) or three
(pre-service teachers) broader categories. For instance, items in the innovation and
creativity and problem solving categories were loaded together in both studies.
Moreover, for in-service teachers, these items were related to items from collabora-
tion, communication, and responsible citizenship, leading to one single category.
This supports Ruiz-Primo’s (2009) proposition that twenty-first century skills are
highly intertwined and could be framed in fewer broad categories.
A new dimension: utility of technology, was identified in pre-service teachers.
This dimension consisted of three items, ‘Teaching students to use digital tools to
locate information’, ‘Teaching students to use technology tools to clearly communi-
cate concepts’, and ‘Teaching students to use technology in a responsible way’. The
first item was designed to test information literacy teaching, the second to tap into the
issue of teaching communication, and the last item to examine teaching of responsible
citizenship. However, the three items did not load with the other items in respective
hypothesized categories; rather, they loaded together into an independent factor
 248 Y. Jia et al.

termed utility of technology. The utility of technology category was also formed once
in in-service teachers, but was eventually removed from the scale due to the poor
quality of the factor and the items (only two items in the factor and one of them was
double loaded). A revision of items may lead to the establishment of this dimension
for in-service teachers. The utility of technology dimension was supported by both
the enGauge framework (Lemke 2002) and NRC Technological Literacy (Gamire and
Pearson 2006). Future studies are needed to develop items tapping the issues in this
dimension using samples of in-service teachers.
For both scales, the category of responsible citizenship failed to be identified. In
pre-service teachers, the four items of responsible citizenship loaded together into
one factor in the first round of analysis. However, each of them also cross-loaded on
other factors, indicating that it is not a distinguishable category. After several rounds
of analysis, two items in this category could not be identified with any established
factors and eventually were removed from the scale. The other two items were sepa-
rately loaded on two different factors. In in-service teachers, the four items did not
Downloaded by [Universite Laval] at 19:23 06 April 2016

load on an independent factor during all rounds of analyses. Finally, only two of the
items remained in the scale. These results indicate that responsible citizenship is not
a well-recognized domain of twenty-first century skills teaching.
In this study, a pool of items tapping twenty-first century skills teaching were
tested in two groups: in-service STEM teachers and pre-service teachers. The analy-
ses revealed a different scale for each group. According to Richardson (1996),
personal experience, experience with school and instruction, and experience with
formal knowledge shape pre-service teachers’ pedagogical beliefs. Similarly, in-ser-
vice teachers’ pedagogical beliefs are formulated based on the same three sources,
but with an addition of knowledge gained through current teaching practice (Novak
and Knowles 1992; Powell 1992).Therefore, we speculate that the different percep-
tions of twenty-first century skills between in-service and pre-service teachers may
represent some gaps between theories of twenty-first century skills teaching in tea-
cher education programs and how this understanding is actually put into practice
within the classroom setting. Furthermore, in this study, the in-service teachers
taught mainly STEM; the term ‘STEM’ was added to most of the items to specify
the subject domain in the in-service teacher survey. Thus, the different perceptions
of pre-service teachers and in-service teachers revealed in the study may represent
domain-specific characteristics of twenty-first century skills. Future studies are
needed to examine these hypotheses by testing pre-service teachers in the STEM-
specific domain and in-service teachers in the general domain.

Limitations and future directions

This study presented the development of a 21st century Skills Teaching Scale. The
two small samples of teachers limited the power of the study and its generalizability.
Twenty-first century skills teaching may differ by subjects or grade levels potentially
leading to different constructs of the scale. As such, future studies are recommended
to examine the scale in large and diverse samples representing various teacher popu-
lations with regard to subjects, grade levels, and pre-service or in-service status. In
addition to the above limitations, this study involved only EFA and CFA. Future val-
idation studies can examine how this newly developed measure is correlated with or
differs from other relevant measures.
 Teacher Development 249

Overall, developing a scale to study the teaching of twenty-first century skills is

beneficial for teacher education or professional development programs to understand
the readiness, strengths, and weaknesses of teachers with regard to their twenty-first
century skill teaching capacity. These scales can also serve as a self-reflection or
evaluation tool for current and future teachers to recognize their own ability to teach
twenty-first century skills and identify areas which could be strengthened. Likewise,
it is hypothesized that better preparation of teachers can improve twenty-first century
skill student outcomes in STEM areas. Depending on the type of teacher (pre-ser-
vice or in-service), one of the two different versions of scale can be used to measure
teacher confidence in teaching twenty-first century skills. If used with pre-service
teachers, the scale can target skill areas in which further education is needed; if used
with in-service teachers, the scale can suggest skill areas for further professional
development. From a policy perspective, not many validated and reliable instru-
ments exist to measure twenty-first century skills. By using this brief, easy-to-use
scale with in-service and pre-service teachers, policymakers can quickly gather data
Downloaded by [Universite Laval] at 19:23 06 April 2016

to assess the condition of the national teacher workforce in regards to twenty-first

century skills. Such information can lead to more informed policy decisions regard-
ing the structure and evaluation of teacher preparation programs and professional
development opportunities as they address twenty-first century skill workforce
development.

Disclosure statement
No potential conflict of interest was reported by the authors.

Funding
This work was supported by the Research and Evaluation Program at the Nellie Mae Educa-
tional Foundation.

Notes
1. Conventionally, SRMR smaller than .05, RMSEA smaller than .06, CFI larger than .96,
and TLI larger than .95 are considered to be a good fit (Hu and Bentler 1999). Whereas
SRMR smaller than .08, RMSEA smaller than .1, CFI larger than .90, and TLI larger
than .90 are considered to be an acceptable fit (Brown and Cudeck 1993).
2. Loading size larger than .60 was considered to be high and larger than .45 was
considered acceptable. Loading size larger than .35 in over one factor was considered as
cross-loading (Pett, Lackey, and Sullivan 2003; Silvera, Martinussen, and Dahl 2001;
Tabachnick and Fidell 2001).
3. Four factors had an eigenvalue larger than 1, indicating a four-factor solution. The scree
plot also showed that from the fourth factor on, the fractions of the total variance
explained by each successive factor were minimal. Model fit indices for each of the four
factor models were shown in Table 2.
4. According to Cohen (1988), a correlation of .30 to .50 was moderate.
5. The factor loadings from .75 to .81 for Utility of Technology, .55 to .82 for Collabora-
tion, and .64 to .81 for Innovation and Problem Solving.
6. Three factors had an eigenvalue larger than 1, indicating a three-factor solution. The
scree plot showed that, after the third factor, the fraction of the total variance explained
by each successive factor dramatically dropped. Model fit indices for each of the four
factor models were shown in Table 4.
 250 Y. Jia et al.

Notes on contributors
Yueming Jia, PhD, conducted this work while a Senior Research Associate in the Learning
and Teaching Division at Education Development Center, Inc. She is now a primary research
scientist at ABA Research & Educational Consulting at Cambridge, MA.

Youn Joo Oh, EdD, formerly a Project Director in the Learning and Teaching Division at
Education Development Center, Inc., is now a primary research scientist at ABA Research &
Educational Consulting at Cambridge, MA.

Bernadette Sibuma, EdD, is a Research Associate in the Learning and Teaching Division at
Education Development Center, Inc. in Waltham, MA.

Frank LaBanca, EdD, is the Director of the Center for 21st Century Skills at EDUCATION
CONNECTION.

Mhora Lorentson, PhD, is the Director of the Center for Collaborative Evaluation and Strate-
gic Change at EDUCATION CONNECTION.
Downloaded by [Universite Laval] at 19:23 06 April 2016

References
AASL (American Association for School Librarians). 2007. The American Association for
School Librarians Standards for the 21st Century Learner. Chicago: The American Asso-
ciation for School Librarians.
Abdullah, M., and K. Osman. 2010. “Inventive Thinking Skills in Science: A Comparative
Study between Students in Malaysia and Brunei.” International Journal of Learning 17
(9): 227–236.
Beers, S. 2001. Teaching 21st Century Skills: An ASCD Action Tool. Alexandria: ASCD.
Bentler, P. M. 1995. EQS Structural Equations Program Manual. Encino: Multivariate Soft-
ware.
Brown, M.W., and R. Cudeck. 1993. “Alternative Ways of Assessing Model Fit.” In Testing
Structural Equation Models, edited by K. A Bollen and J. S. Long, 36–62. Thousand
Oaks: Sage.
Byrne, B. M. 2001. Structural Equation Modeling with AMOS: Basic Concepts, Applications
and Programming. Mahwah: Erlbaum.
Cohen, J., ed. 1988. Statistical Power Analysis for the Behavioural Sciences. 2nd ed.
Mahwah: Lawrence Erlbaum.
Costa, J. P., and D. Cogan-Drew. 2009. Six Skills That Form the Foundation for 21st Century
Success. Litchfield: EDUCATION CONNECTION.
Coutinho, C., and P. Mota. 2011. “Web 2.0 Technologies in Music Education in Portugal:
Using Podcasts for Learning.” Computers in the Schools 28 (1): 56–74.
Darling-Hammond, L. 2007. “Building a System for Powerful Teaching and Learning.” In
Building a 21st Century U.S. Education System, edited by B. Wehling and C. Schneider,
65–74. Washington, DC: National Commission on Teaching and America’s Future.
Gamire, E., and G. Pearson. 2006. Tech Tally: Approaches to Assessing Technological Liter-
acy. Washington, DC: The National Academies Press.
Gibson, I. W. 2005. “Constructing Meaning in a Technology-rich, Global Learning Environ-
ment.” Computers in the Schools 22 (1–2): 169–182.
Gorsuch, R. L. 1983. Factor Analysis. 2nd ed. Mahwah: Lawrence Erlbaum.
Greenhill, V. 2010. 21st Century Knowledge and Skills in Educator Preparation. Washington,
DC: Partnership for 21st Century Skills.
Hilton, M. 2008. Research on Future Skill Demands: A Workshop Summary. Washington,
DC: The National Academies Press.
Hilton, M. 2010. Exploring the Intersection of Science Education and 21st Century Skills: A
Workshop Summary. Washington, DC: The National Academies Press.
Hodge, K. A., and J. L. Lear. 2011. “Employment Skills for 21st Century Workplace: The
Gap between Faculty and Student Perceptions.” Journal of Career and Technical Educa-
tion 26 (2): 28–41.
 Teacher Development 251

Hu, L., and P. M. Bentler. 1999. “Cutoff Criteria for Fit Indexes in Covariance Structure
Analysis: Conventional Criteria Versus New Alternatives.” Structural Equation Model-
ing: A Multidisciplinary Journal 6: 1–55.
International Society for Technology in Education. 2000. “ISTE National Educational Tech-
nology Standards.” Accessed June 2012. http://www.iste.org/standards.
Johnson, D. R., and J. C. Creech. 1983. “Ordinal Measures in Multiple Indicator Models: A
Simulation Study of Categorization Error.” American Sociological Review 48: 398–407.
Lambert, J., and Y. Gong. 2010. “21st Century Paradigms for Pre-Service Teacher Technology
Preparation.” Computers in the Schools 27 (1): 54–70. doi:10.1080/07380560903536272.
Lemke, C. 2002. EnGauge 21st Century Skills: Digital Literacies for a Digital Age. Naper-
ville: North Central Regional Educational Laboratory.
Lowther, D. L., F. A. Inan, S. M. Ross, and J. Strahl. 2012. “Do One-to-One Initiatives
Bridge the Way to 21st Century Knowledge and Skills?” Journal of Educational Comput-
ing Research 46 (1): 1–30.
Mayrath, M., J. Clarke-Midura, D. H. Robinson, and G. Schraw. 2012. Technology-Based
Assessments for 21st Century Skills: Theoretical and Practical Implications from Modern
Research. Charlotte: Information Age Publishing.
McCreery, M. P., P. G. Schrader, and S. Krach. 2011. “Navigating Massively Multiplayer
Downloaded by [Universite Laval] at 19:23 06 April 2016

Online Games: Evaluating 21st Century Skills for Learning within Virtual Environ-
ments.” Journal of Educational Computing Research 44 (4): 473–493. doi:10.2190/
EC.44.4.f.
Moylan, W. 2008. “Learning by Project: Developing Essential 21st Century Skills Using
Student Team Projects.” International Journal of Learning 15 (9): 287–292.
Muthén, L. K., and B. O. Muthén. 1998–2010. Mplus Version 6.1 [Computer Software]. Los
Angeles: Muthén and Muthén.
Novak, D., and J. G. Knowles. 1992. “Life Histories and the Transition to Teaching as a Sec-
ond Career.” Paper presented at the annual meeting of the American Educational
Research Association, Chicago, IL, April.
O’Sullivan, M. K., and K. B. Dallas. 2010. “A Collaborative Approach to Implementing 21st
Century Skills in a High School Senior Research Class.” Education Libraries 33 (1):
3–9.
Partnership for 21st Century Skills. 2002. “Learning for the 21st Century: A Report and Mile
Guide for 21st Century Skills.” http://www.21stcenturyskills.org/images/stories/otherdocs/
p21up_Report.pdf
Pett, M. A., N. R. Lackey, and J. J. Sullivan. 2003. Making Sense of Factor Analysis: The
Use of Factor Analysis for Instrument Development in Health Care Research. Thousand
Oaks: Sage.
Powell, R. 1992. “The Influence of Prior Experiences on Pedagogical Constructs of Tradi-
tional and Nontraditional Preservice Teachers.” Teaching and Teacher Education 8 (3):
225–238.
Richardson, V. 1996. “The Role of Attitudes and Beliefs in Learning to Teach.” In Handbook
of Research on Teacher Education, 2nd ed., edited by J. Sikula, 102–119. New York:
Macmillan.
Ruiz-Primo, M. A. 2009. “Towards a Framework for Assessing 21st Century Science Skills.”
Paper prepared for the Workshop on Exploring the Intersection of Science Education and
the Development of 21st Century Skills, Washington, DC, February. http://www7.nation
alacademies.org/bose/RuizPrimo.pdf
Sardone, N. B., and R. Devlin-Scherer. 2010. “Teacher Candidate Responses to Digital
Games.” Journal of Research on Technology in Education 42 (4): 409–425.
Silvera, D. H., M. Martinussen, and T. I. Dahl. 2001. “The Tromso Social Intelligence Scale,
A Self-report Measure of Social Intelligence.” Scandinavian Journal of Psychology 42:
313–319.
Smith, J. J., and E. Dobson. 2011. “Beyond the Book: Using Web 2.0 Tools to Develop 21st
Century Literacies.” Computers in the Schools 28 (4): 316–327. doi:10.1080/
07380569.2011.620939.
Tabachnick, B. G., and L. S. Fidell. 2001. Using Multivariate Statistics. 4th ed. New York:
Harper and Row.
 252 Y. Jia et al.

Thomas, M. K., X. Ge, and B. A. Greene. 2011. “Fostering 21st Century Skill Development
by Engaging Students in Authentic Game Design Projects in a High School Computer
Programming Class.” Journal of Educational Computing Research 44 (4): 391–408.
Trilling, B., and C. Fadel. 2009. 21st Century Skills: Learning for Life in Our Times. San
Francisco: John Wiley & Sons.
Downloaded by [Universite Laval] at 19:23 06 April 2016