See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/253993992

Problem Solving Skill Evaluation Instrument — Validation Studies

Article · January 2007

DOI: 10.1063/1.2508681

CITATIONS READS
22 9,682

2 authors:

Wendy Adams C. E. Wieman

Colorado School of Mines Stanford University
69 PUBLICATIONS 5,307 CITATIONS 264 PUBLICATIONS 39,999 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Wendy Adams on 15 April 2015.

The user has requested enhancement of the downloaded file.

 Problem Solving Skill Evaluation Instrument – Validation
Studies
Wendy K. Adams and Carl E. Wieman

Department of Physics, University of Colorado, Boulder, Boulder, CO 80309, USA

Abstract. Researchers have created several tools for evaluating conceptual understanding as well as students’ attitudes
and beliefs about physics; however, the field of problem solving is sorely lacking a broad use evaluation tool. This
missing tool is an indication of the complexity of the field. The most obvious and largest hurdle to evaluating physics
problem solving skills is untangling the skills from the physics content knowledge necessary to solve problems. We are
tackling this problem by looking for the physics problem solving skills that are useful in other disciplines as well as
physics. We report on the results of a series of interviews comparing physics students’ skills when solving physics
problems with their anonymous completion of the problem solving instrument. There is an encouragingly good match.
Keywords: Problem Solving, Physics, Undergraduate Education
PACS: 01.40.Fk, 01.50.gf

INTRODUCTION the purposes of this paper is drawn from the literature:

“Problem solving is cognitive processing directed at
Physics Education Researchers over the past 20 achieving a goal when no solution method is obvious
years have developed many very useful assessment to the problem solver.” [6]. In addition we will break
tools including a range of conceptual evaluations such problem solving down into a set of skills, including
as the Force Concept Inventory [1], Force and Motion those motivational aspects relevant to successful
Conceptual Evaluation [2], and the Conceptual Survey completion of solution. Note that this definition of
of Electricity and Magnetism [3] among others. There problem solving is based on the solver. If a person is
has also been work in the area of attitude and belief an ‘expert’ in their field, then it is very likely that tasks
surveys [4]. However, one of the most highly valued that are problems for students will only be an
assets of physics courses by educators is problem ‘exercise’ where the solution or path to solution is
solving; but, very little has been published regarding obvious, for an expert.
ways to evaluate students problem solving skills. To The extensive problem solving research in
date, research on problem solving in physics has been psychology and cognitive science has included transfer
focused predominantly on curriculum improvements of problem solving ability as well as work on
and the identification of differences between expert characterizing specific skills needed to solve problems.
and novice problem solving characteristics [5]. There Many researchers have gone as far as to declare that
have been efforts to develop a problem solving problem solving abilities do not transfer between
evaluation tool, but none have been successful. disciplines [5]. Being an expert problem solver in a
In this paper we will discuss the problem solving particular area is not an indication of that person’s
instrument that is being created at Colorado. First we problem solving skills in any other area. However,
will describe the direction of our approach and how it other researchers present evidence that some problem
avoids problems that have hindered previous attempts. solving skills do transfer between disciplines [6].
Then we will outline our validation procedures to date, We argue that these contradictory results on
and present the specific set of measurable skills that problem solving transfer can be explained with the
we have identified. same reasons that have made developing a physics
APPROACH problem solving evaluation tool difficult. First, solving
physics problems requires content knowledge, both
First we define what we mean by “problem factual and procedural; second, that it is important to
solving”, because this term is used very broadly. The view problem solving as a combination of many
fairly specific definition of problem solving used for different specific skills, and the different skills have
varying degrees of importance in various types of 57 questions that must be answered as they go through
problems. the script. These questions are about planning,
The physics content requirement is a fairly straight procedures, calculations, reflection as well as analysis
forward yet seemingly unassailable complication. A of the two scripted solvers’ skills. This combination of
particular student may be unsuccessful at solving a questions evolved from multiple interviews that were
problem because they are a poor problem solver or used to evaluate students’ skills as they solved the
because they simply lack familiarity with the specific scripted problem. In this way, we have identified 34
bit of physics knowledge needed for the problem. On skills which show up as useful when solving this
the other hand, when a student is successful at solving problem.
a particular problem, it is very difficult to tell the
difference between a strong problem solver or a VALIDATION
student who is so familiar with that particular content We have performed a series of validation studies
area that the problem was actually an exercise for that that involved several iterations to revise and refine the
student. In both of these cases, one cannot clearly instrument. The validation process included: face
delineate the student’s problem solving ability because validity – interviews with a wide range of people to
the student’s content knowledge is inextricably confirm and clarify the meaning of questions and the
intertwined within these skills. story; anonymous written instrument results compared
A great deal of research in problem solving – to a series of physics problem solving interviews; and
physics, chess, etc. is result oriented. Researchers are concurrent validity – comparison of instrument results
interested only in who can successfully solve problems to professor and employer evaluations. In addition we
rather than each solvers’ strengths and weaknesses in a have two studies which are in process: interviewer
variety of specific problem solving skills. This focus observations of physics majors using the instrument
greatly limits both the information that can be gained will be compared to an independent interviewer’s
by these studies and the development of assessment observations of the same students solving quantum
tools. mechanics problems; and written instrument responses
Our approach comes from the hypothesis that a of undergraduate teaching assistants will be compared
person has a set of skills that vary in strength that they to independent instructor evaluation of the teaching
use to tackle problems. This includes problems in any assistants using the same rubric.
context, the physics classroom or the workplace. If The development of the survey was intertwined
this is the case, students’ skills can be analyzed with a with the first phase of face validation. Interviews were
problem that does not require physics content conducted with a variety of subjects including non-
knowledge. Although still preliminary, we have science majors, physics majors, professors, elementary
growing evidence that our hypothesis is correct. education majors, and adult professionals with a
variety of backgrounds including a high school drop
ASSESSMENT TOOL
out. The first few interviewees were asked to think out
We have developed an assessment tool which uses loud as they solved the Jasper problem. The two
the work of the Cognition and Technology Group at person script and the questions that are asked
Vanderbilt (CTGV)[7]. CTGV developed the Jasper throughout the instrument were created based on these
Woodbury series of problems for 6th – 7th grade math first few interviews. Further interviews were
students to solve in small groups. These are long, conducted with the scripted instrument. Periodically
involved problems that each have no less than 14 steps the script was refined and questions were added based
to solution. CTGV carefully designed and researched on interviews. A total of 23 interviews were
each Jasper problem on its effectiveness. Our problem completed, each lasting between one and two hours.
solving instrument was developed through a series of Eventually the instrument and analysis rubric were
interviews with a wide range of subjects using one of refined to the point that for the last nine interviewers,
the Jasper problems. The evaluation instrument the interviewer was able to obtain a complete analysis
instructs the student to analyze a script of two of the subjects skills without further interaction after
individuals working through the solution to the Jasper initial instructions on think-aloud style. The analysis
problem. This format provides several benefits: 1) rubric includes 34 separate problem solving skills that
motivation for students to work through the entire have been identified during the interviews (see Table
solution, 2) removes the stress of being analyzed 3) 1).
scaffolds the problem so that a solution will be reached The next stage of instrument construction and
even if the student has a specific weakness that would validation of the rubric involved having the students
have prevented further progress if they were take a written version (no interview) of the survey.
attempting to solve the problem in isolation. There are These were then graded using the same rubric. If
enough information was not provided in the written version of the instrument. These students brought the
responses to rate the student in all 34 categories, then completed instrument to their first interview. The
questions were added or adjusted and tested with new instrument was graded at a later date without knowing
students. Currently 16 written responses have been the identity of the student. Interviews consisted of
graded and 90% of the skills that are identifiable having the student sit down with the interviewer whom
during interviews can be consistently graded with the they had never met and immediately begin solving a
latest written version of the survey. slightly modified problem about the Great Pyramid of
Giza[7]. In this problem students must determine how
Anonymous Written Results Matched to Physics
many blocks are in the pyramid and how many men
Problem solving Interviews
were needed to build the pyramid if it took 20 years to
The next step in face validation was to test how build. A few facts about the block size and men’s
the set of problem solving skills that are identified capabilities were included without any other
while solving this 6th grade math trip-planning scaffolding. These interviews were mathematically
problem matched with the skills a student uses to solve intensive so had to be limited to an hour. Each student
a physics problem. For this portion of the validation required two to three interview sessions to complete
procedure we gave five science majors who were the problem. They were evaluated on the skills that
currently enrolled in introductory physics the written

TABLE 1. Problem Solving Skills

Skills Metaskill Will Uncatergorized
Skills that were measured by both the written instrument and the pyramid interview
ŠMath Skills (add/sub/mult/div) ŠPlanning What – Question ŠConfidence ŠReal Life vs.
Formation Classroom Approach
ŠSpatial – Mapping ŠPlanning How – Way to get ŠEnjoyed Solving the Problem ŠOverall Success
answer.
ŠEstimation ŠPlanning Big Picture – ŠWanted to Succeed on ‘Test’
Visualizes the Problem.
ŠNumber Sense ŠConnects Steps and Pieces ŠAttribution (responsible for
own mistakes)
ŠReal World (informal) ŠMonitors Own Progress ŠWanted to Find Best Solution
Experience for Themselves.
ŠKnowledge of Own Strengths ŠWanted to find Best Solution
and Weaknesses to Please Interviewer.
` ŠCreativity
ŠJudgment of Reasonable Issues
ŠTies in Personal Experience
Skills that were not measured by the pyramid problem interviews and why
ŠAcquires Information 1st Time ŠKeeps Problem Framework in ŠEnjoyed Analyzing Interns
Through (pyramid data of a Mind (Pyramid framework not (Specific to instrument
numerical nature) complicated) scenario)
ŠRemembers Previously Noted ŠAdaptability - Shift Direction ŠEnjoyed Complete Survey
Facts. (pyramid interviews (A forced change of plans was (Same as enjoyed solving the
done in 2 segments w/ a 2 week not inserted into Pyramid problem for pyramid)
break.) problem)
ŠOutside Factual Knowledge ŠSkepticism – Thinks about
(Not necessary for pyramid Information that is Supplied.
problem or in separate (Specific to instrument
category such as Geometry) scenario)
ŠMath – Equation Formation ŠChecks Scripted Solvers’
(Difficult to see w/ Pyramid Calculations (Specific to
response format) instrument scenario)
ŠReading Comprehension ŠAware of How Scripted
(Pyramid presentation makes Solvers Helped (Specific to
this difficult to judge) instrument scenario)
Skills that were not measured by the written problem solving instrument and why
ŠGeometry (Not necessary for ŠPhysics/Math They Think They
Survey) ‘Should’ Know Blocks Progress
(No outside formal knowledge
needed for survey)
 were exhibited while completing the problem. The identified in Table 1. A separate interviewer is
interview sessions were separated by two weeks which interviewing these same students as they solve the
made some of the grading of skills difficult, as noted problem solving evaluation instrument. The results of
in Table 1. these separate interviews will be compared.
The five graded anonymous written instruments
were then compared to each student’s set of skill CONCLUSION
ratings identified during the pyramid interviews. The We have created an evaluation instrument that
skills that could be identified with both the anonymous identifies 34 specific problem solving skills. We have
written instrument and the pyramid interviews are evidence to support our hypothesis that a particular
listed in Table 1. There were a few skills that only student has the same strengths and weaknesses when
worked in one situation or the other. These are listed solving a complicated trip planning problem as they do
in Table 1 with explanations. Many of the skills when solving a physics problem or performing in the
identified by the instrument that could not be graded work place. There are a few specific skills that may be
by the pyramid interviews have been recognized as necessary for one type of problem that are not required
important skills when solving physics problems, they to solve another; however, the strengths and
simply could not be rated with the structure of the weaknesses of a particular person are the same,
pyramid interviews. regardless of the environment. Consistent strengths
The five students’ written instruments were easily and weaknesses match intuition – certain people are
matched up with their interview results. This was done stronger problem solvers – however; on the surface
without any identifying information about the students this may seem to disagree with research on context and
other than the rating of problem solving skills from transfer. A closer look at problem solving shows that
each problem scenario. The same set of problem our results are in fact consistent with other research on
solving strengths and weaknesses were evident in how problem solving as well. The difference is that most
each student solved these two different problems, research is complicated with the need for specific
After the students written results were matched with content knowledge to solve a particular problem and
their interview results, the specific ratings in each of many researchers focus on the end result rather than
the skills matched up for 75% of the 22 skills listed in looking at the specific skills used by the solver during
Table 1. Of the 25% that did not match exactly, most the problem solving process.
were a neutral response versus (strong/weak). Only
two of the students had a single skill that appeared ACKNOWLEDGMENTS
strong for one problem and weak on the other.
Supported in part by the NFS Distinguished
Concurrent Validation Teaching Scholar.

Our concurrent validation studies include REFERENCES

comparison between the instrument results and 1. D. Hestenes, M. Wells and G. Swackhamer, The
evaluation of problem solving skills in various Physics Teacher 30, 141-158 (1992).
environments. We asked three different instructors 2. R. K. Thornton and D. R. Sokoloff, American Journal
(class sizes of 40 or less) to evaluate particular of Physics 66, 338-352 (1998).
students’ strengths and weaknesses over the course of 3. D. P. Maloney, T. L. O’Kuma, C. J. Hieggelke and A.
a semester. These same students were interviewed or Van Heuvelen, Am. Jrnl. of Phys. 69, S12-S23 (2001).
took the written version of the problem solving 4. W. K. Adams, K. K. Perkins, M. Dubson, N. D.
evaluation tool. The instructor’s assessment of the Finkelstein and C. E. Wieman, Phys. Rev. Special
Topics – PER 2 (2006).
students strengths and weaknesses in problem solving
5. D. P. Maloney, “Research on Problem solving:
have matched the skills identified by instrument results Physics,” in Handbook of Research on Science
in all 15 cases. In addition we had an employer to do Teaching and Learning edited by D. L. Gabel,
the same for five different employees. Again, all five Macmillan, Toronto 1993, pp 327-354.
instrument results matched the employer’s evaluations. 6. R.E. Meyer, Thinking, problem solving, cognition (2nd
We are in the process of conducting an additional ed) Freeman, New York, 1992.
validation study involving the comparison of 7. The Cognition & Technology Group at Vanderbilt, Ed.
instrument results with skills identified while students Tech. Rsrch. and Dev. 40, 65-80 (1992).
solve quantum mechanics problems. Six students 8. K. Vick, E. Redish, and P. Cooney, Retrieved 8/18/06
from Activity-Based Physics (ABP) Alternative
underwent a series of seven interviews where they
Homework Assignments (AHAs) Problem site:
solved quantum mechanics problems. The interviewer http://www.physics.umd.edu/perg/abp/aha/pyramid.htm
is in the process of scoring these students quantum 9. A copy of the problem solving instrument can be
mechanics problem solving skills on the 34 areas viewed at: http://cosmos.colorado.edu/CPPSS/

View publication stats