research-article

Evaluating Micro Parsons Problems as Exam Questions

Authors:

David H. Smith, IVAuthors Info & Claims

ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1

Pages 674 - 680

https://doi.org/10.1145/3649217.3653583

Published: 03 July 2024 Publication History

Abstract

Parsons problems are a type of programming activity that present learners with blocks of existing code and requiring them to arrange those blocks to form a program rather than write the code from scratch. Micro Parsons problems extend this concept by having students assemble segments of code to form a single line of code rather than an entire program. Recent investigations into micro Parsons problems have primarily focused on supporting learners leaving open the question of micro Parsons efficacy as an exam item and how students perceive it when preparing for exams.

To fill this gap, we included a variety of micro Parsons problems on four exams in an introductory programming course taught in Python. We use Item Response Theory to investigate the difficulty of the micro Parsons problems as well as the ability of the questions to differentiate between high and low ability students. We then compare these results to results for related questions where students are asked to write a single line of code from scratch. Finally, we conduct a thematic analysis of the survey responses to investigate how students' perceptions of micro Parsons both when practicing for exams and as they appear on exams.

References

[1]

Frank B Baker. 2001. The basics of item response theory. ERIC.

[2]

Mark A Barton and Frederic M Lord. 1981. An upper asymptote for the three-parameter logistic item-response model. ETS Research Report Series, Vol. 1981, 1 (1981), i--8.

[3]

Jeff Bender, Bingpu Zhao, Alex Dziena, and Gail Kaiser. 2023. Integrating Parsons puzzles within Scratch enables efficient computational thinking learning. Research and Practice in Technology Enhanced Learning, Vol. 18 (2023), 022--022.

[4]

Allan Birnbaum. 1968. Some latent trait models and their use in inferring an examinee's ability. Statistical theories of mental test scores (1968).

[5]

André F De Champlain. 2010. A primer on classical test theory and item response theory for assessments in medical education. Medical education, Vol. 44, 1 (2010), 109--117.

[6]

Paul Denny, Andrew Luxton-Reilly, and Beth Simon. 2008. Evaluating a new exam question: Parsons problems. In Proceedings of the Fourth International Workshop on Computing Education Research (Sydney, Australia) (ICER '08). Association for Computing Machinery, New York, NY, USA, 113--124. https://doi.org/10.1145/1404520.1404532

Digital Library

[7]

Robert L Ebel and David A Frisbie. 1972. Essentials of educational measurement.

[8]

Barbara J. Ericson, Paul Denny, James Prather, Rodrigo Duran, Arto Hellas, Juho Leinonen, Craig S. Miller, Briana B. Morrison, Janice L. Pearce, and Susan H. Rodger. 2022. Parsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs. In Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR '22). Association for Computing Machinery, New York, NY, USA, 191--234. https://doi.org/10.1145/3571785.3574127

Digital Library

[9]

Barbara J. Ericson, James D. Foley, and Jochen Rick. 2018. Evaluating the Efficiency and Effectiveness of Adaptive Parsons Problems. In Proceedings of the 2018 ACM Conference on International Computing Education Research (Espoo, Finland) (ICER '18). Association for Computing Machinery, New York, NY, USA, 60--68. https://doi.org/10.1145/3230977.3231000

Digital Library

[10]

Barbara J. Ericson, Mark J. Guzdial, and Briana B. Morrison. 2015. Analysis of Interactive Features Designed to Enhance Learning in an Ebook. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research (Omaha, Nebraska, USA) (ICER '15). Association for Computing Machinery, New York, NY, USA, 169--178. https://doi.org/10.1145/2787622.2787731

Digital Library

[11]

Barbara J. Ericson, Janice L. Pearce, Susan H. Rodger, Andrew Csizmadia, Rita Garcia, Francisco J. Gutierrez, Konstantinos Liaskos, Aadarsh Padiyath, Michael James Scott, David H. Smith, Jayakrishnan M. Warriem, and Angela Zavaleta Bernuy. 2023. Multi-Institutional Multi-National Studies of Parsons Problems. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education. Association for Computing Machinery, New York, NY, USA, 57--107. https://doi.org/10.1145/3623762.3633498

Digital Library

[12]

Geela Venise Firmalo Fabic, Antonija Mitrovic, and Kourosh Neshatian. 2019. Evaluation of Parsons problems with menu-based self-explanation prompts in a mobile python tutor. International Journal of Artificial Intelligence in Education, Vol. 29 (2019), 507--535.

[13]

Carl C. Haynes and Barbara J. Ericson. 2021. Problem-Solving Efficiency and Cognitive Load for Adaptive Parsons Problems vs. Writing the Equivalent Code. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 60, bibinfonumpages15 pages. https://doi.org/10.1145/3411764.3445292

Digital Library

[14]

Slava Kalyuga. 2009. The expertise reversal effect. In Managing cognitive load in adaptive multimedia learning. IGI Global, 58--80.

[15]

Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications.

[16]

Amruth N. Kumar. 2017. The Effect of Providing Motivational Support in Parsons Puzzle Tutors. In Artificial Intelligence in Education, Elisabeth André, Ryan Baker, Xiangen Hu, Ma. Mercedes T. Rodrigo, and Benedict du Boulay (Eds.). Springer International Publishing, Cham, 528--531.

[17]

Brooke Morin and Krista M Kecskemety. 2021. Collaborative Parsons Problems in a Remote-learning First-year Engineering Classroom. In 2021 ASEE Virtual Annual Conference Content Access.

[18]

Brooke C Morin, Krista M Kecskemety, Kathleen A Harper, and Paul Alan Clingan. 2020. Work in Progress: Parsons Problems as a Tool in the First-Year Engineering Classroom. In 2020 ASEE Virtual Annual Conference Content Access.

[19]

Briana B. Morrison, Lauren E. Margulieux, Barbara Ericson, and Mark Guzdial. 2016. Subgoals Help Students Solve Parsons Problems. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (Memphis, Tennessee, USA) (SIGCSE '16). Association for Computing Machinery, New York, NY, USA, 42--47. https://doi.org/10.1145/2839509.2844617

Digital Library

[20]

Solomon Sunday Oyelere, Friday Joseph Agbo, Ismaila Temitayo Sanusi, Abdullahi Abubakar Yunusa, and Kissinger Sunday. 2019. Impact of puzzle-based learning technique for programming education in Nigeria context. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), Vol. 2161. IEEE, 239--241.

[21]

Dale Parsons and Patricia Haden. 2006. Parson's programming puzzles: a fun and effective learning tool for first programming courses. In Proceedings of the 8th Australasian Conference on Computing Education-Volume 52. 157--163.

Digital Library

[22]

Seth Poulsen, Hongxuan Chen, Yael Gertner, Benjamin Cosman, Matthew West, and Geoffrey L Herman. 2023. Measuring the Impact of Distractors on Student Learning Gains while Using Proof Blocks. arXiv preprint arXiv:2311.00792 (2023).

[23]

Seth Poulsen, Mahesh Viswanathan, Geoffrey L Herman, and Matthew West. 2022. Evaluating proof blocks problems as exam questions. ACM Inroads, Vol. 13, 1 (2022), 41--51.

Digital Library

[24]

David H. Smith, Max Fowler, and Craig Zilles. 2023 a. Investigating the Role and Impact of Distractors on Parsons Problems in CS1 Assessments. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 417--423. https://doi.org/10.1145/3587102.3588819

Digital Library

[25]

David H. Smith and Craig Zilles. 2023. Discovering, Autogenerating, and Evaluating Distractors for Python Parsons Problems in CS1. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 924--930. https://doi.org/10.1145/3545945.3569801

Digital Library

[26]

David H. Smith, IV, Seth Poulsen, Max Fowler, and Craig Zilles. 2023 b. Comparing the Impacts of Visually Grouped and Jumbled Distractors on Parsons Problems in CS1 Assessments. In Proceedings of the ACM Conference on Global Computing Education Vol 1 (CompEd 2023). Association for Computing Machinery, New York, NY, USA, 154--160. https://doi.org/10.1145/3576882.3617927

Digital Library

[27]

David H. Smith IV. 2023. Useful Distractions? Investigating the Utility of Distractors in Parsons Problems. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2 (Chicago, IL, USA) (ICER '23). Association for Computing Machinery, New York, NY, USA, 62--63. https://doi.org/10.1145/3568812.3603463

Digital Library

[28]

Matthew West, Geoffrey L Herman, and Craig Zilles. 2015. Prairielearn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning. In 2015 ASEE Annual Conference & Exposition. 26--1238.

[29]

Zihan Wu. 2023. Investigating the Effectiveness of Variations of Micro Parsons Problems. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2 (Chicago, IL, USA) (ICER '23). Association for Computing Machinery, New York, NY, USA, 120--122. https://doi.org/10.1145/3568812.3603447

Digital Library

[30]

Zihan Wu, Barbara Ericson, and Christopher Brooks. 2021. Regex Parsons: Using Horizontal Parsons Problems to Scaffold Learning Regex. In Proceedings of the 21st Koli Calling International Conference on Computing Education Research (Joensuu, Finland) (Koli Calling '21). Association for Computing Machinery, New York, NY, USA, Article 31, bibinfonumpages3 pages. https://doi.org/10.1145/3488042.3489968

Digital Library

[31]

Zihan Wu, Barbara J. Ericson, and Christopher Brooks. 2023. Using Micro Parsons Problems to Scaffold the Learning of Regular Expressions. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 457--463. https://doi.org/10.1145/3587102.3588853

Digital Library

Cited By

Hassany MBrusilovsky PSavelka JLekshmi Narayanan AAkhuseyinoglu KAgarwal AHendrawan R(2025)Generating Effective Distractors for Introductory Programming Challenges: LLMs vs HumansProceedings of the 15th International Learning Analytics and Knowledge Conference10.1145/3706468.3706529(484-493)Online publication date: 3-Mar-2025
https://dl.acm.org/doi/10.1145/3706468.3706529

Index Terms

Evaluating Micro Parsons Problems as Exam Questions
1. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

Investigating the Effectiveness of Variations of Micro Parsons Problems
ICER '23: Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2

Parsons problems have been used to provide scaffolding for introductory learners. Instead of asking learners to write from scratch, Parsons problems provide blocks of mixed-up code, and ask learners to rearrange them into the correct order. In ...
Using Micro Parsons Problems to Scaffold the Learning of Regular Expressions
ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1

Regular expressions (regex) are a text processing method widely used in data analysis, web scraping, and input validation. However, students find regular expressions difficult to create since they use a terse language of characters. Parsons problems can ...
Evaluating a new exam question: Parsons problems
ICER '08: Proceedings of the Fourth international Workshop on Computing Education Research

Common exam practice centres around two question types: code tracing (reading) and code writing. It is commonly believed that code tracing is easier than code writing, but it seems obvious that different skills are needed for each. These problems also ...

Comments

comments powered by Disqus.

Information & Contributors

Information

Published In

ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1

July 2024

776 pages

ISBN:9798400706004

DOI:10.1145/3649217

General Chairs:
Mattia Monga
University of Milan, Italy
,
Violetta Lonati
University of Milan, Italy
,
Erik Barendsen
Radboud University, The Netherlands
,
Program Chairs:
Judithe Sheard
Monash University, Australia
,
James Paterson
Glasgow Caledonian University, Scotland

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 July 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ITiCSE 2024

Sponsor:

SIGCSE

ITiCSE 2024: Innovation and Technology in Computer Science Education

July 8 - 10, 2024

Milan, Italy

Acceptance Rates

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Upcoming Conference

ITiCSE '25

Sponsor:
sigcse

Innovation and Technology in Computer Science Education

June 27 - July 2, 2025

Nijmegen , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
67
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)6

Reflects downloads up to 23 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hassany MBrusilovsky PSavelka JLekshmi Narayanan AAkhuseyinoglu KAgarwal AHendrawan R(2025)Generating Effective Distractors for Introductory Programming Challenges: LLMs vs HumansProceedings of the 15th International Learning Analytics and Knowledge Conference10.1145/3706468.3706529(484-493)Online publication date: 3-Mar-2025
https://dl.acm.org/doi/10.1145/3706468.3706529

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.

Alternative Proxies:

Alternative Proxy