Computer Supported Collaborative Learning for Programming:

A Systematic Review

Ricardo Sol1 a
, Elci Alcione Santos2 b
, Manuel C. Reis3 c
and Lucas Pereira1 d
1
ITI, LARSyS, Polo Cientíco e Tecnológico da Madeira, floor-2, 9020-105 Madeira, Portugal
2Faculty of Exact Sciences and Engineering, University of Madeira, Madeira, Portugal
3Department of Engineering, University of Trás-os-Montes e Alto Douro, Vila Real, Portugal

Keywords: Computer Supported Collaborative Learning, Programming, Educational Technology, Reviews.

Abstract: The objective of this paper is to present the current evidence relative to the effectiveness of computer
supported collaborative learning as a pedagogical tool in teaching programming. A systematic literature
review in the IEEE Xplore, Web of Science, and ACM Digital Libraries was performed with studies that
investigated factors affecting the effectiveness of computer supported collaborative learning for students
learning programming and studies that measured the effectiveness of computer supported collaborative
learning for students learning programming. Twelve papers were used in the analysis. The results showed that
the object oriented programming languages are the ones that have been most frequently adopted by educators
who use computer-supported collaborative learning as tools to teach programming, that course critique
surveys and questionnaires are the most frequently reported methods used to assess the effectiveness of
computer-supported collaborative learning interventions, and that the amount of participants who have taken
part in research to evaluate the value of computer-supported collaborative learning in teaching programming
varies notably between studies. Finally, in total, 83.3% of the included papers report that computer supported
collaborative learning is an effective teaching tool and can help programmers in their studies.

1 INTRODUCTION outset. The daily analysis of matters is clearly planned

in relation to the technology, not the cognitive growth
Learning programming can be a difficult task. Many of the learner. These methods start from the fifth and
references can be found in literature, all over the sixth stages of Bloom’s Taxonomy of Educational
world, pertaining to the difficulties numerous Objective, when these last two stages are contingent
students have in understanding and learning upon proficiency in the previous four stages (Lister,
programming courses. A miscellaneous collection of 2000). The programming task crosses the Learning
reasons has been identified as the difficulties Style Inventory environments (Kolb, 1985),
demonstrated by these students. Some authors depending on whether a learner is trying to solve a
highlight that a preexisting mental model of problem (a symbolically complex environment),
knowledge can affect the acquisition and use of employing abilities (behaviorally complex),
programming concepts. The literature identified recognizing and understanding the association
numerous bugs that are made by learners who can between notions (perceptually complex). This may
show a sort of negative influence from natural imply that diverse learning styles come to the fore
language or rudimentary models of how a process throughout the whole programming procedure (Byrne
works (Gray et al., 1993). and Lyons, 2000). Constructivism applied to
Customarily, the teaching of initial programming programming in practice has certainly not obeyed
has highlighted the writing of programs from the theoretical foundations. Instead of an improvised

a https://orcid.org/0000-0003-4333-7140
b https://orcid.org/0000-0002-1189-4076
c
https://orcid.org/0000-0002-8872-5721
d
https://orcid.org/0000-0002-9110-8775

184
Sol, R., Santos, E., Reis, M. and Pereira, L.
Computer Supported Collaborative Learning for Programming: A Systematic Review.
DOI: 10.5220/0010407001840191
In Proceedings of the 13th International Conference on Computer Supported Education (CSEDU 2021) - Volume 2, pages 184-191
ISBN: 978-989-758-502-9
Copyright c 2021 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
 Computer Supported Collaborative Learning for Programming: A Systematic Review

attitude to planning and applying transformation, a solo programming, because the programmers inserted
strict research outline is needed in order to improve less code anomalies (Estácio et al., 2015).
transformations grounded on meaningful theoretical The results of several studies do not back the
understanding (Bruce and McMahon, 2002). Focused impression that cognitive styles are fixed traits and
upon the ability of learners to consistently execute give more support for the plasticity of cognitive
code are aspects of programming on which educators styles, in those occasions where learners are helped
should assess learners. Also, to emphasize by computer-supported systems (Angeli et al., 2016).
exclusively upon those parts of programming is to An intelligent tutoring is a computer system that
emphasize upon the three inferior stages of the SOLO targets to offer instruction or feedback to students and
taxonomy (Collis & Biggs, 1979): the pre-structural, is more successful than the customary classes,
unistructural, and multi-structural levels. From think- learning is faster and foments better performance on
aloud responses, the researchers found that teachers tests (Reiser et al., 1985).
tended to show a SOLO relational response on minor We hope that this article will make clear which
reading issues, while learners leaned towards claims of computer supported collaborative learning
showing a multi-structural response (Lister et al., are supported by scientific studies. We aim to present
2006). It was found that there are at least two the prevalence of these claims within a systematic
cognitive factors that show themselves as sample. Specifically, the objective of the review is to
opportunities that may make learning of answer five research questions stated in the
programming problematic, being those of learning methodology.
style and of motivation (Jenkins, 2002). The method that has been implemented in this
The results of a survey about programming Systematic Literature Review is described in depth in
concepts presented to 500 learners worldwide Section 2 where the research questions are presented,
confirm that the most difficult concepts to learn are while Section 3 is dedicated to the results of the
the ones that need understanding of greater entities of literature review search. In Section 4, a discussion
the program as opposed to just details. The results takes place in an attempt to answer the five research
also sustained the notions that abstract concepts like questions and in regard to different aspects of the
pointers and memory handling are hard to learn. The literature review. This is followed by a conclusion
results also exposed a set of topics (e.g., language from the literature review in Section 5.
libraries, input and output) that would perhaps need
additional attention, since understand them was not
related to understand the essentials of programming 2 METHODOLOGY
(Lahtinen et al., 2005). An empirical study was
motivated by the idea that diverse people create
diverse outlines of information in any new learning
2.1 Research Questions
process and proven that how each learner deals with
This literature review is influenced by the work of
problems in a singular way is grounded on their
Kitchenham and Charters (2010) that proposed
mental model. The preliminary study implies that
guidelines for performing Systematic Literature
accomplishment in the first phase of an introductory
Reviews in Software Engineering. An initial protocol
programming course is anticipated, by observing
was developed as part of this literature review. The
steadiness in use of mental models that learners apply
primary focus of this literature review is to
to a initial programming problem even earlier than
understand and identify computer-supported systems
they have had any interaction with programming
for collaboratively learning for programming. The
(Dehnadi, 2006).
following research questions were formulated in
Nevertheless, the biggest problem of beginner
order to achieve this goal:
programmers does not seem to be the understanding
1. What computer languages are being taught?
of basic concepts but instead learning to use them.
2. What are the characteristics of the learners being
However, research shows that learners of
taught?
programming operating collaboratively beat solo
3. What types of research studies are performed to
programmers (Nosek, 1998). Over a qualitative and
investigate the computer supported collaborative
quantitative analysis, collaborative practices showed
learning?
encouraging results on fundamental characteristics of
4. What is the number of participants in studies that
learning programming skills, i.e., learning and
are being performed by researchers?
motivation. Indeed, as far as learning is concerned,
the quantitative analysis showed that it outperformed

185
CSEDU 2021 - 13th International Conference on Computer Supported Education

5. Do studies suggest that using computer supported research questions and measure the quality of the
collaborative learning for programming is primary studies. When designing the quality checklist
effective? of the study, the eleven criteria discussed by Dyba
and Dingsøyr were used, (Dyba and Dingsøyr, 2008)
2.2 Search that were based in the Critical Appraisal Program
(Gilb, 2005). The checklist was comprised of eleven
Within the field there are several expressions that general questions to measure the quality of both
relate to programming education, collaborative quantitative and qualitative studies according to the
learning and tutoring systems. It was used a Boolean following ratio scale: Yes = 1 point, Partially = 0,5
search string that included synonyms: point, and No = 0 points. Ranging the resulting total
(“collaborative learning” OR “cooperative quality score for each study between 0 (very poor)
learning”) AND (“intelligent tutor*” OR “adaptive and 11 (very good).
tutor*" OR “cognitive tutor*" OR “smart tutor*”) The eleven criteria used to assess the quality of each
AND programming AND (novice OR beginner OR publication are quoted as follows:
introductory OR teaching OR learning OR CS1 OR 1. Is the paper based on research or is it a ‘lessons
“first time”). learned’ report based on expert opinion?
The systematic sample was retrieved from the 2. Is there a clear statement of the aims of the
IEEE Xplore, Web of Science, and ACM Digital research?
Libraries. Whenever a paper was found suitable, it 3. Is there an adequate description of the context in
was added to the list of papers qualified for the which the research was carried out?
synthesis. Web of Science was the last to be looked 4. Was the research design appropriate in order to
at, and thus it only returned duplicate studies. address the aims of the research?
5. Was the recruitment strategy appropriate with
2.3 Inclusion and Exclusion Criteria regards to the aims of the research?
6. Was there a control group with which to compare?
The inclusion and exclusion criteria were used to 7. Was the data collected in a way that addressed the
guaranty that only significant literature was added to research issue?
the literature review. In order to determine whether 8. Was the data analysis sufficiently rigorous?
articles met the inclusion or exclusion criteria 9. Has the relationship between researcher and
abstracts were read. participants been considered to an adequate
degree?
2.3.1 Inclusion Criteria 10. Is there a clear statement of findings?
11. Is the study of value for research or practice?
1. Publications that have tutoring systems used by
students learning programming collaboratively. The first two of these criteria represent the minimum
2. If papers reported the same study, only the latest quality threshold that was observed during this
was added. literature review. The following nine criteria are
3. Papers were added independently of their date of intended to determine the rigor and credibility of the
publications. research methods employed as well as the relevance
4. Relevant grey literature is accepted. of each paper in relation to the literature review.

2.3.2 Exclusion Criteria 2.5 Data Extraction

1. Publications that do not report a system. When publications were identified as meeting the
2. When only the Abstract and not the full text is criteria for inclusion, the full text was read. Then in
available. order to answer the research questions the following
3. Publications with Systems that are only partially data were extracted from each publication included in
prototyped. the literature review:
4. Position papers, editorials, and letters were all  Publication type;
excluded.  Publication aims and objectives;
 Methodology of the publication;
2.4 Study Quality Assessment  Number of participants in a study;
 How data was gathered and analyzed during the
To aid the data extraction process, a form was created, study;
which was used to collect evidence relating to the  Characteristics of the learners being tutored;

186
 Computer Supported Collaborative Learning for Programming: A Systematic Review

 Programming languages taught by the tutor; were not based on research or presented a “lessons
 Did the system(s) include task related to learners learned account “, but offered some description of the
generated program planning or visualizations; context in which the research was carried out. All
 Did the system(s) use visualizations or plans as articles clearly stated the aims of the research;
instructional resources? however only one had an adequate relationship
between participants and researchers.
One reviewer extracted all data during the first
More than half of the studies had an adequate
semester of 2020. In order to validate the extraction
recruitment strategy. None of the studies included in
process, a random sample comprising of 20% of the
the literature review was awarded the maximum score
total number of primary studies had their data
of 11, with the highest score awarded being 9.5. The
extracted by a second reviewer. These results were
average quality score of publications included in the
then compared. Whenever the data extracted differed,
literature review was 8.29 with standard deviation of
where differences never surpassed more than 7%,
1.2. The lowest score that articles were awarded was
such differences were discussed until consensus was
1.5. Because the average quality score of the included
reached. The data extraction strategy was deemed to
publications varied, it was decided to maintain all
be appropriate. All extracted data was stored in a
papers due to the small number of publications
spread sheet.
selected. In the following section, we present the
results for the literature review research questions.

3 RESULTS 3.3 Research Questions Results

In this section the synthesis of the literature review is Answers to the research questions outlined in Section
presented, beginning with the analysis from the 2.1 will now be discussed.
literature search results. During the selection process,
the IEEE Xplore, Web of Science, and ACM Digital 3.3.1 What Computer Languages Are Being
Libraries were chosen as the baseline databases due Taught?
to its reputation.
When analyzing the studies included in the literature
3.1 Search Results review, three different categories were established
regarding the programming languages used: Object
The initial phase of the search process identified two Oriented, Non Object Oriented, and Dedicated.
hundred and four publications matching the search Object oriented languages were the largest
string. Of these, only thirty-seven were potentially contributor to the literature review having been the
relevant based on the screening of titles and abstracts. main programming language used in seven papers.
Each of these thirty-seven studies was filtered Evidence was collected that stresses how efforts have
according to the exclusion and inclusion criteria been made to use designed programming languages
before being accepted in the literature review list. If in order to teach programming principles, as LeJOS
titles and abstracts were not sufficient to identify the [NOGUEZ07]. LeJOS is an open-source project
relevance of a paper, full articles were read. It was created to develop a technological infrastructure to
also checked if there were any very similar studies or develop software to robots using Java technology.
duplicate studies that were published in more than Evidence was also collected that stresses how efforts
one publication. have been made to use web-programming languages
Based on the search, 12 studies (32,4% of the 37 in order to teach programming principles
studies) were accepted in the literature review list [STARBIRD11, WANG09].
after a detailed assessment of the abstract, full text,
and exclusion of duplicates. In the following section, 3.3.2 What Are the Characteristics of the
it is presented the quality assessment results are Learners Being Taught?
presented (see the Appendix for the list of studies
used in this literature review). The diverse context of each study was scrutinized in
order to determine the characteristics of the students
3.2 Quality Assessment Results that have been taught programming. Two different
groups were established as a result of this and these
Each study had been assigned a quality score out of were ‘University’, and ‘various’. Out of the 12 papers
eleven. Only two of the articles included in the list 8 reported on the use of technology in a university
setting. Three discussed the implementation in

187
CSEDU 2021 - 13th International Conference on Computer Supported Education

multiple environments [GUO15, JENKINS12, One paper was identified to be “unclassifiable”

STARBIRD11]. because it did not provide a measure of the
effectiveness of CSCL when used in such context.
3.3.3 What Types of Research Studies Are
Performed to Investigate the 3.4 Limitations of the Review
Computer Supported Collaborative
Learning? The most important limitation of the validity of the
literature review is the fact that it was performed only
The use of surveys and questionnaires was equally in IEEE Xplore, Web of Science, and ACM Digital
commonly found method by which the reviewed Libraries.
studies evaluated their findings and proposals (six Other important limitations of the validity of the
papers reported the use of such methods). Log and literature review are in relation to bias in the selection
video record also have been described (each in one of papers and imprecise data extracted. Search strings
paper). Analysis of student grades has also been were devised as the literature review employed
reported in one paper that also examined the impact exclusively the electronic resources of IEEE Xplore,
that computer-supported tools had upon retention Web of Science, and ACM Digital Libraries. These
rates [DING17]. In addition, comparative analysis were established after applying trial searches.
has also taken place; this has included contrasting the Notwithstanding this, it is not possible to assure that
effect on the learners of learning with computer- all studies in the IEEE Xplore, Web of Science, and
supported tools to learning without (one paper). Two ACM Digital Libraries relevant to the topic under
papers included in the literature review were ‘lessons consideration were returned and there is a fare risk
learned’ [JOVANOVIC15, LEUNG07]. that some studies may have been omitted due to the
search terms used. Furthermore, the phenomena
3.3.4 What Is the Number of Participants of where ‘negative’ results are less likely to be
Studies That Are Being Performed by published, known as ‘publication bias’ may also have
Researchers? had a fair impact on the findings of the literature
review, though it is difficult to determine whether this
There are two papers included in the review that have was the case. The data extraction procedure can also
examples of ‘lessons learned’ or experience style have been undesirably impacted by bias when
reports, whereas six papers offer evidence that an choosing publications. This is due to the fact that data
empirical study took place. The scale of studies extraction procedure has been performed by only one
included in the review varied notably. These ranged reviewer. In addition, it is possible that the inclusion
from small-scale studies that contained 6 participants and exclusion criteria may have unintentionally
through to larger studies that reported sample sizes disqualified some relevant publications. This is
with more than 600 participants. Ten papers report the because the applied criteria stopped being of added
exact number of participants that took part in the papers that contained no ‘lessons learned’ element.
research performed. In contrast one paper discusses Finally, non-English language and abstract only
conducting experiments or collecting information papers were excluded from addition to the literature
from participants but did not state the precise number review. However, no papers were found that were
of participants involved [LEUNG07]. written in another language probably due to the
search string. Furthermore, by excluding the
3.3.5 Do Studies Suggest That using publications where only the abstract of the paper is
Computer Supported Collaborative available, one could have unintentionally avoided
Learning for Programming Is acceptable publications from being included in the
Effective? literature review.
After analyzing the papers included in the literature
review, it is possible to show an analysis on whether
the included publications report the use of computer- 4 DISCUSSION
supported collaborative learning (CSCL) to be an
effective intervention in the learning of programming. In this section aspects of additional analysis that has
Of the 12 papers included in the literature review, 10 been assumed to corroborate the results of the
papers report that the use of CSCL is effective when literature review are presented. Furthermore, a
learning introductory programming concepts. discussion regarding the findings of the literature
review is also portrayed.

188
 Computer Supported Collaborative Learning for Programming: A Systematic Review

It was noted throughout this process that of the 12 5 CONCLUSIONS

papers included in the literature review 9 were
published in Conference Proceedings, and 2 in This review has examined the effectiveness of using
Journals. Of the 9 papers published in conference CSCL to teach programming by using a systematic
proceedings 2 were short papers. literature review methodology. After employing a
Several interpretations can be made as a result of search strategy, 12 papers were initially included in
the literature review. When looking at the original the literature review. The findings of the literature
quality score, with the average being 8.29 out of 11, review show how the use of CSCL can be an effective
it is accepted that this number is satisfactory. A high learning tool when used in programming courses.
proportion of publications contained in the initial set Definitely, 83.3% of the publications included in the
of 12 required fundamental experimental features like review reported this.
a control group, whereas the relationship between Several findings and tendencies, with regards to
researcher and participants was often considered to be the teaching of programming using CSCL, have been
of a poor standard. This is due to 2 of the 12 papers noticed as a result of the literature review. These
included in the review being ‘lessons learned’ or comprise the detection firstly of object-oriented
experience workshop reports. Such papers do not programming languages as the ones that have been
score well in the quality assessment criteria that has most frequently adopted by educators who use CSCL
been used. as tools to teach programming. Secondly, that course
Two large-scale comparative studies were critique surveys and questionnaires are the most
included in the literature review. Only one paper frequently reported methods used to assess the
reported a semester long experiment that compared effectiveness of CSCL interventions. Thirdly, that the
the results of more than 600 participants. Usually, a amount of participants who have taken part in
large study may be considered to offer far more research to evaluate the value of CSCL in teaching
compelling evidence than the results of small non- programming varies notably between studies.
comparative studies. However, two small studies The most significant finding of the literature
describe the results of an experiment that compared review, which researchers should have in account,
the results from the participants on tests from both nevertheless, is that there is an obvious need for large-
computer-supported collaborative learning and non scale and high-quality research to be undertaken in
computer-supported collaborative learning order to discover the true effectiveness of CSCL as a
programming sessions. programming teaching tool.
Five research questions were created in order to Due to the fact that the included publications
determine the value of using CSCL when teaching utilize a broad variety of methods to collect data, in
programming. Several findings and tendencies, combination with the samples size, statistical analysis
regarding the learning of programming using CSCL, methods have not been used during this study and so
can be noticed as a result. these results are not statistically significant. As a
These comprise the observations that: consequence, additional research is needed in order to
• The object-oriented programming languages are establish the true effectiveness of CSCL that can be
the ones that have been most frequently adopted used to support the teaching of programming.
by educators. However, this work emphasizes that there is a lot of
• Course critique surveys and questionnaires are the potential for future work of researchers within the
most usually described methods used to assess the field to build upon the body of existing knowledge
effectiveness of CSCL sessions. documented in the literature review.
• The amount of participants who have taken part in Upon completion of this first study, the
research to assess the value of CSCL in learning identification of relevant literature will continue with
programming varies a lot between studies. the second search phase. During the second phase, all
of the references in the papers identified in the first
In general, the findings of the literature review imply phase will be reviewed.
that the use of CSCL can be an effective learning tool This systematic literature review shows that there
when used in a programming course. This is evident is a clear need for large-scale and high-quality
as 10 of the 12 papers included in the literature review research to be done in order to determine the true
clearly state that computer-supported collaborative effectiveness of computer supported collaborative
learning is valuable when used in such a way. learning as a programming teaching tool. From this
study, it is also possible to find numerous areas of
relevance that future research may pursue and

189
CSEDU 2021 - 13th International Conference on Computer Supported Education

investigate. A theme that future researcher could also acquiring programming skills: Findings of a controlled
follow is the study of the advantages of using diverse experiment. In Software Engineering (SBES), 2015
types of programming languages, in order to teach the 29th Brazilian Symposium on (pp. 150-159). IEEE.
participants. A research of this nature may uncover Gilb, T., 2005. Competitive engineering: a handbook for
systems engineering, requirements engineering, and
whether one computer language in particular is the software engineering using Planguage. Elsevier.
most appropriate for use with CSCL tools. Finally, an Gray, W. D., Goldberg, N. C., & Byrnes, S. A., 1993.
analysis of the wider hypothesis of using CSCL as Novices and programming: Merely a difficult subject
programming teaching tools is more effective than (why?) or a means to mastering metacognitive skills?
other non-computer-supported collaborative learning Review of the book Studying the Novice Programmer.
approaches would also be important and could help to Journal of Educational Research on Computers, 9(1),
both enlighten and enhanced future teaching. 131-140.
Kitchenham, B., Pretorius, R., Budgen, D., Brereton, O. P.,
Turner, M., Niazi, M., & Linkman, S., 2010. Systematic
literature reviews in software engineering–a tertiary
ACKNOWLEDGEMENTS study. Information and software technology, 52(8),
792-805.
This research received funding from the Portuguese Jenkins, T., 2002, August. On the difficulty of learning to
Foundation for Science and Technology (FCT) under program. In Proceedings of the 3rd Annual Conference
of the LTSN Centre for Information and Computer
grant LARSyS - UIDB/50009/2020.
Sciences (Vol. 4, No. 2002, pp. 53-58).
Kolb, D., 1985. Leaning Style Inventory. Hay/McBer,
Boston, Ma.
REFERENCES Lister, R., 2000, December. On blooming first year
programming, and its blooming assessment. In
Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruik- Proceedings of the Australasian conference on
shank, K. A., Mayer, R. E., Pintrich, P. R., ... & Computing education (pp. 158-162). ACM.
Wittrock, M. C., 2001. A taxonomy for learning, Lister, R., Simon, B., Thompson, E., Whalley, J. L., &
teaching, and assessing: A re-vision of Bloom’s Prasad, C., 2006. Not seeing the forest for the trees:
taxonomy of educational objectives, abridged edition. novice programmers and the SOLO taxonomy. ACM
White Plains, NY: Longman. SIGCSE Bulletin, 38(3), 118-122.
Angeli, C., Valanides, N., Polemitou, E., & Fraggoulidou, Lahtinen, E., Ala-Mutka, K., & Järvinen, H. M., 2005. A
E., 2016. An interaction effect between young study of the difficulties of novice programmers. ACM
children's field dependence-independence and order of SIGCSE Bulletin, 37(3), 14-18.
learning with glass-box and black-box simulations: Nosek, J. T., 1998. The case for collaborative
Evidence for the malleability of cognitive style in programming. Communications of the ACM, 41(3),
computer-supported learning. Computers in Human 105-108.Reiser, B.J., Anderson, J.R. and Farrell, R.G.,
Behavior, 61, 569-583. 1985, August. Dynamic Student Modelling in an
Bruce, C. S., & McMahon, C. A., 2002. Contemporary Intelligent Tutor for LISP Programming. IJCAI (V. 85,
developments in teaching and learning introductory p. 8-14).
programming: Towards a research proposal. Faculty of
Information Technology, QUT Teaching & Learning
Report, (2/2002). APPENDIX
Byrne, P., & Lyons, G., 2001, June. The effect of student
attributes on success in programming. In ACM
SIGCSE Bulletin (Vol. 33, No. 3, pp. 49-52). ACM. • [DING17] Ding, Qing, and Sitan Cao. "RECT: A cloud-
Collis, K. F., & Biggs, J. B., 1979. Classroom Examples of based learning tool for graduate software engineering
Cognitive Development Phenomena: the SOLO practice courses with remote tutor support." IEEE
Taxonomy. Access 5 (2017): 2262-2271.
Dehnadi, S., 2006, September. Testing programming • [DONG17] Dong, Zhijiang, Cen Li, and Roland H.
aptitude. In Proceedings of the 18th Annual Workshop Untch. "Build peer support network for CS2 students."
of the Psychology of Programming Interest Group, pp. In Proc. of the 49th Annual Southeast Regional
22-37). Conference, pp. 42-47. ACM, 2011.
Dybå, T., & Dingsøyr, T., 2008. Empirical studies of agile • [GOLDMAN11] Goldman, Max, Greg Little, and
software development: A systematic review. Robert C. Miller. "Real-time collaborative coding in a
Information and software technology, 50(9-10), 833- web IDE." In Proc. of the 24th annual ACM symp. on
859. User interface software and technology, pp. 155-164.
Estácio, B., Oliveira, R., Marczak, S., Kalinowski, M., ACM, 2011.
Garcia, A., Prikladnicki, R., & Lucena, C., 2015, • [GUO15] Guo, Philip J., Jeffery White, and Renan
September. Evaluating collaborative practices in Zanelatto. "Codechella: Multi-user program

190
 Computer Supported Collaborative Learning for Programming: A Systematic Review

visualizations for real-time tutoring and collaborative

learning." In Visual Languages and Human-Centric
Computing (VL/HCC), 2015 IEEE Symp. on, pp. 79-
87. IEEE, 2015.
• [HARSLEY17] Harsley, R., Fossati, D., Di Eugenio,
B., & Green, N. (2017, March). Interactions of
individual and pair programmers with an intelligent
tutoring system for computer science. In Proceedings of
the 2017 ACM SIGCSE Technical Symposium on
Computer Science Education (pp. 285-290).
• [JENKINS12] Jenkins, Jam, Evelyn Brannock, Thomas
Cooper, Sonal Dekhane, Mark Hall, and Michael
Nguyen. "Perspectives on active learning and
collaboration: JavaWIDE in the classroom." In
Proceedings of the 43rd ACM technical symposium on
Computer Science Education, pp. 185-190. ACM,
2012.
• [JOVANOVIC15] Jovanovic, Dusan, and Slobodan
Jovanovic. "An adaptive e‐learning system for Java
programming course, based on Dokeos LE." Computer
Applications in Eng. Education 23, no. 3 (2015): 337-
343.
• [LEUNG07] Leung, Chi-Hong, and Yuen-Yan Chan.
"Knowledge management system for electronic
learning of IT skills." In Proceedings of the 8th ACM
SIGITE conference on Info. technology education, pp.
53-58. ACM, 2007.
• [NOGUEZ07] Noguez, Julieta, Gilberto Huesca, and L.
Enrique Sucar. "Shared learning experiences in a
contest environment within a mobile robotics virtual
laboratory." In Frontiers In Education Conference-
Global Engineering: Knowledge Without Borders,
Opportunities Without Passports, 2007. FIE'07. 37th
Annual, pp. F3G-15. IEEE, 2007.
• [STARBIRD11] Starbird, Kate, and Leysia Palen.
"More than the usual suspects: the physical self and
other resources for learning to program using a 3D
avatar environment." In Proc. of the 2011 iConference,
pp. 614-621. ACM, 2011.
• [TSOMPANOUDI13] Tsompanoudi, Despina, Maya
Satratzemi, and Stelios Xinogalos. "Exploring the
effects of collaboration scripts embedded in a
distributed pair programming system." In Proc. of the
18th ACM conference on Innovation and technology in
computer science education, pp. 225-230. ACM, 2013.
• [WANG09] Wang, Shu-Ling, Gwo-Haur Hwang, Ju-
Chun Chu, and Pei-Shan Tsai. "The role of collective
efficacy and collaborative learning behavior in learning
computer science through CSCL." ACM SIGCSE
Bulletin 41, no. 3 (2009): 352-352.

191