<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="discussion">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Psychol.</journal-id>
<journal-title>Frontiers in Psychology</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Psychol.</abbrev-journal-title>
<issn pub-type="epub">1664-1078</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.3389/fpsyg.2016.01390</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Psychology</subject>
<subj-group>
<subject>Opinion</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>Learning from the Past&#x02013;The Need for Empirical Evidence on the Transfer Effects of Computer Programming Skills</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<name><surname>Scherer</surname> <given-names>Ronny</given-names></name>
<xref ref-type="author-notes" rid="fn001"><sup>&#x0002A;</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/213357/overview"/>
</contrib>
</contrib-group>
<aff><institution>Faculty of Educational Sciences, Centre for Educational Measurement, University of Oslo</institution> <country>Oslo, Norway</country></aff>
<author-notes>
<fn fn-type="edited-by"><p>Edited by: Layne Kalbfleisch, George Washington University, USA</p></fn>
<fn fn-type="edited-by"><p>Reviewed by: Jonathan Plucker, Johns Hopkins University, USA</p></fn>
<fn fn-type="corresp" id="fn001"><p>&#x0002A;Correspondence: Ronny Scherer <email>ronny.scherer&#x00040;cemo.uio.no</email></p></fn>
<fn fn-type="other" id="fn002"><p>This article was submitted to Educational Psychology, a section of the journal Frontiers in Psychology</p></fn>
</author-notes>
<pub-date pub-type="epub">
<day>14</day>
<month>09</month>
<year>2016</year>
</pub-date>
<pub-date pub-type="collection">
<year>2016</year>
</pub-date>
<volume>7</volume>
<elocation-id>1390</elocation-id>
<history>
<date date-type="received">
<day>31</day>
<month>07</month>
<year>2016</year>
</date>
<date date-type="accepted">
<day>30</day>
<month>08</month>
<year>2016</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#x000A9; 2016 Scherer.</copyright-statement>
<copyright-year>2016</copyright-year>
<copyright-holder>Scherer</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/"><p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.</p></license>
</permissions>
<kwd-group>
<kwd>computational thinking</kwd>
<kwd>computer programming</kwd>
<kwd>creativity</kwd>
<kwd>problem solving</kwd>
<kwd>transfer of learning</kwd>
</kwd-group>
<counts>
<fig-count count="0"/>
<table-count count="1"/>
<equation-count count="0"/>
<ref-count count="45"/>
<page-count count="4"/>
<word-count count="3406"/>
</counts>
</article-meta>
</front>
<body>
<sec id="s1">
<title>Introduction to the problem</title>
<p>In recent years, education has put considerable emphasis on the development of twenty-first century skills&#x02014;a set of skills that can almost universally be applied to a broad range of domains and problems, and that help students to deal with the challenges and demands of complex, real-world problem situations (Pellegrino and Hilton, <xref ref-type="bibr" rid="B33">2012</xref>). Among others, these skills comprise problem solving, creativity, critical thinking, collaboration, adaptability, digital literacy, and computational thinking, and are considered to be critical in our information- and knowledge-rich society (Binkley et al., <xref ref-type="bibr" rid="B2">2012</xref>; Wagner, <xref ref-type="bibr" rid="B41">2012</xref>; Scherer, <xref ref-type="bibr" rid="B36">2015</xref>; Care and Anderson, <xref ref-type="bibr" rid="B6">2016</xref>). Against this background, it has become the designated aim of educators to help students to develop these skills (Kay and Greenhill, <xref ref-type="bibr" rid="B19">2011</xref>). The question of <italic>how</italic> the development of these skills and the ability to transfer them to different contexts and knowledge domains can be fostered has therefore gained significance (Greiff et al., <xref ref-type="bibr" rid="B15">2014</xref>). Nonetheless, this question is by no means trivial, because the transfer of knowledge and skills does not automatically happen, as Tricot and Sweller (<xref ref-type="bibr" rid="B40">2013</xref>) argued.</p>
<p>In the pursuit of finding ways to foster twenty-first century skills and their transfer, voices have become loud inspiring education to incorporate computer programming into K-12 curricula (Lye and Koh, <xref ref-type="bibr" rid="B25">2014</xref>). The reactions on these voices have been tremendous; some countries developed an entire curriculum around computer programming (Sturman and Sizmur, <xref ref-type="bibr" rid="B38">2011</xref>; Webb et al., <xref ref-type="bibr" rid="B43">2016</xref>). Behind this development is the belief that fostering programming skills improves students&#x00027; performance on other critical skills such as creativity and problem solving (Liao and Bright, <xref ref-type="bibr" rid="B24">1991</xref>; Clements, <xref ref-type="bibr" rid="B9">1995</xref>). Mitchel Resnick, the director of MIT&#x00027;s Media Lab and facilitator of the Scratch&#x000AE; programming language, argued that &#x0201C;programming supports &#x0201C;computational thinking,&#x0201D; helping you learn important problem-solving and design strategies [&#x02026;] that carry over to nonprogramming domains&#x0201D; (Resnick et al., <xref ref-type="bibr" rid="B34">2009</xref>, p. 62). Along the same lines, Barr and Stephenson (<xref ref-type="bibr" rid="B1">2011</xref>) proposed that computer programming &#x0201C;is a problem solving methodology that can be automated and transferred and applied across subjects&#x0201D; (p. 51). Brown and K&#x000F6;lling (<xref ref-type="bibr" rid="B4">2012</xref>) took this argument even further and claimed that the &#x0201C;use of programming skills can allow for a deeper and more direct understanding of the subjects under investigation, using Computing to support learning in the same way that Mathematics supports the learning of subjects such as Physics.&#x0201D; (p. 1) Whereas there has been a great body of research supporting these claims in the 1980s and 1990s (for an overview, please refer to Liao and Bright, <xref ref-type="bibr" rid="B24">1991</xref>), it seems as if there is very little evidence on the transfer effects of computer programming skills in the context of twenty-first century education (Grover and Pea, <xref ref-type="bibr" rid="B16">2013</xref>; Lye and Koh, <xref ref-type="bibr" rid="B25">2014</xref>). Although computer programming and other skills share a number of cognitive and even metacognitive processes (Clements, <xref ref-type="bibr" rid="B7">1986</xref>, <xref ref-type="bibr" rid="B9">1995</xref>; Brown and K&#x000F6;lling, <xref ref-type="bibr" rid="B4">2012</xref>; Lye and Koh, <xref ref-type="bibr" rid="B25">2014</xref>; Rich et al., <xref ref-type="bibr" rid="B35">2014</xref>), therefore supporting potential transfer effects, I argue that educational research lags behind in sharing sufficient evidence for these claims.</p>
<p>Against this background, the main position this opinion paper conveys is that&#x02014;although the conceptual argumentation about the potential transfer effects of computer programming skills on other skills such as problem solving and creativity is reasonable&#x02014;there is a strong need for empirical evidence supporting this, particularly in the context of the recent advancements of digital technologies.</p>
</sec>
<sec id="s2">
<title>Current status of knowledge</title>
<sec>
<title>Computer programming skills and the concept of computational thinking</title>
<p>Computer programming skills are considered to be an integral part of what is called &#x0201C;computational thinking&#x0201D; (CT; Denning, <xref ref-type="bibr" rid="B11">2010</xref>; Lye and Koh, <xref ref-type="bibr" rid="B25">2014</xref>), and often find their way into frameworks of digital literacy (Siddiq et al., <xref ref-type="bibr" rid="B37">2016</xref>). CT has in fact gained importance in STEM education, and there is a growing interest in exploring how CT can be introduced in K-12 curricula (Lye and Koh, <xref ref-type="bibr" rid="B25">2014</xref>). Wing (<xref ref-type="bibr" rid="B45">2006</xref>) defined CT as a thought process that &#x0201C;involves solving problems, designing systems, and understanding human behavior, by drawing on the concepts fundamental to computer science&#x0201D; (p. 33). In their review, Lye and Koh (<xref ref-type="bibr" rid="B25">2014</xref>) argued that computer programming&#x02014;an activity that requires the abstraction and decomposition of problems&#x02014;exposes students to these thinking processes, and claimed that it may therefore foster the development of CT. This claim has largely been supported by studies using the first (e.g., Logo&#x000AE;; Dyck and Mayer, <xref ref-type="bibr" rid="B12">1989</xref>; Pardamean et al., <xref ref-type="bibr" rid="B31">2015</xref>) and more recently developed programming tools for education (e.g., Scratch&#x000AE;; Wilson et al., <xref ref-type="bibr" rid="B44">2013</xref>).</p>
</sec>
<sec>
<title>Transfer effects on problem solving</title>
<p>Having established that computer programming and CT are closely connected, the question arises to what extent transfer effects on problem solving exist. The starting point for addressing this question is to examine which particular thinking processes are involved in programming and problem solving. According to Brooks (<xref ref-type="bibr" rid="B3">1999</xref>), Clements and Merriman (<xref ref-type="bibr" rid="B10">1988</xref>), and Denning (<xref ref-type="bibr" rid="B11">2010</xref>), programming comprises a number of processes that range from information retrieval and processes of understanding the problem at hand to discovering methods and algorithms that solve the problem, and evaluating them (Table <xref ref-type="table" rid="T1">1</xref>). According to a framework proposed by the (OECD, <xref ref-type="bibr" rid="B29">2014</xref>), domain-general problem solving comprises similar processes of exploring and understanding, representing and formulating, planning, and executing, and monitoring and reflecting (Table <xref ref-type="table" rid="T1">1</xref>). These conceptualizations reveal a considerable overlap between the two constructs: The processes involved in programming are by and large of the same nature as those involved in problem solving, in that the problem space needs to be understood and explored first, hypotheses and methods are to be developed second, and the proposed solution&#x02014;be it an algorithm, product, or any other problem solution&#x02014;finally needs to be evaluated (Klahr and Dunbar, <xref ref-type="bibr" rid="B21">1988</xref>). This parallelism brings forth the question whether computer programming can actually be considered a form of problem solving (Barr and Stephenson, <xref ref-type="bibr" rid="B1">2011</xref>; Jonassen, <xref ref-type="bibr" rid="B18">2011</xref>), and suggests that transfer effects may exist. Investigating these ideas systematically, Liao and Bright (<xref ref-type="bibr" rid="B24">1991</xref>) conducted a meta-analysis of 65 studies that were conducted between 1969 and 1989 with the goal to synthesize empirical evidence on the effects of computer programming on problem solving abilities. The authors found an average effect size of 41 and concluded that students&#x02014;while learning to program&#x02014;acquire reasoning, logical thinking, and planning (i.e., problem solving) skills that go beyond computer programming. This meta-analysis was, however, followed by only a very limited number of experimental studies that continued examining these effects (Maloney et al., <xref ref-type="bibr" rid="B26">2004</xref>; Gibbon, <xref ref-type="bibr" rid="B14">2007</xref>; Pardamean et al., <xref ref-type="bibr" rid="B31">2015</xref>), some of which were insignificant (Lai and Yang, <xref ref-type="bibr" rid="B23">2011</xref>; G&#x000FC;lbahar and Kalelio&#x0011F;lo, <xref ref-type="bibr" rid="B17">2014</xref>; Korkmaz, <xref ref-type="bibr" rid="B22">2016</xref>). This is somehow surprising, because Liao and Bright (<xref ref-type="bibr" rid="B24">1991</xref>) clearly showed that the average effect was moderated by the type of programming environment used in the treatment group, thus suggesting that further advancements in these tools may affect the transfer effect. In fact, many publications that argued for the transfer effects later on only described programming tools or feasibility studies thereof on a conceptual level (Grover and Pea, <xref ref-type="bibr" rid="B16">2013</xref>). By contrast, a larger body of research exists on the transfer effects on mathematical thinking and conceptual understanding (e.g., Calder, <xref ref-type="bibr" rid="B5">2010</xref>; Kazakoff et al., <xref ref-type="bibr" rid="B20">2013</xref>; Rich et al., <xref ref-type="bibr" rid="B35">2014</xref>).</p>
<table-wrap position="float" id="T1">
<label>Table 1</label>
<caption><p><bold>Key processes involved in computer programming, problem solving, and creative thinking</bold>.</p></caption>
<table frame="hsides" rules="groups">
<thead><tr>
<th valign="top" align="left"><bold>Computer programming</bold></th>
<th valign="top" align="left"><bold>Problem solving</bold></th>
<th valign="top" align="left"><bold>Creative thinking</bold></th>
</tr>
</thead>
<tbody>
<tr>
<td valign="top" align="left">&#x025AA; Representing, storing, and retrieving information in order to understand the problem (i.e., knowledge acquisition of basic problem elements such as objects, relations, initial and final states of objects) <break/>&#x025AA; Discovering algorithms for information processes &#x02014;finding a method in order to represent the real-world problem, develop, and execute an action plan and code <break/>&#x025AA; Evaluating the performance of the designed complex systems on the basis of the written code; bridging the gap between the problem statement and the solution</td>
<td valign="top" align="left">&#x025AA; Exploring and understanding the problem (e.g., by decomposing the problem into sub-problems) <break/>&#x025AA; Representing and formulating the problem by creating representations of the problem situation and formulating hypotheses <break/>&#x025AA; Planning and executing the sequential steps to solve the problem <break/>&#x025AA; Monitoring progress and reflecting on the problem, the solution, and solution strategy</td>
<td valign="top" align="left">&#x025AA; Acquiring knowledge and skills relevant to the creative act, setting goals; encoding, recognizing, and formulating the problem (preparation) <break/>&#x025AA; Building a representation of the problem (e.g., by combining components of the problem) and unconscious processing (incubation) <break/>&#x025AA; Searching for and finding solutions (illumination) <break/>&#x025AA; Evaluating the creative product, monitoring the process of creative activities, and improving shortcomings (verification)</td>
</tr>
</tbody>
</table>
</table-wrap>
</sec>
<sec>
<title>Transfer effects on creativity</title>
<p>Similar to the reasoning on the transfer effects on problem solving, researchers have claimed that learning to program fosters students&#x00027; creative thinking. In the most recent systematic review that I could identify (Clements, <xref ref-type="bibr" rid="B9">1995</xref>), the author found that computer programming instruction fosters creativity and divergent thinking; significant effects on originality&#x02014;a facet of creative thinking that involves selective encoding and combining&#x02014;could be identified. Clements (<xref ref-type="bibr" rid="B9">1995</xref>) concluded his review by summarizing the key elements of computer programming that are also essential for creative thinking: decide on the nature of the problem, combine components of the problem, select a mental representation, monitor progress, acquire knowledge, and encode (see also Clements and Merriman, <xref ref-type="bibr" rid="B10">1988</xref>). In a slightly different framework, Wallas (<xref ref-type="bibr" rid="B42">1926</xref>) proposed four stages of creativity, namely preparation, incubation, illumination, and verification, which are still widely used today. Comparing these processes with those required for computer programming (Table <xref ref-type="table" rid="T1">1</xref>), substantive similarities exist, such that transfer effects between them could be expected. In fact, programming is considered to be a creative human activity (Denning, <xref ref-type="bibr" rid="B11">2010</xref>; Grover and Pea, <xref ref-type="bibr" rid="B16">2013</xref>). Yet, the existing body of empirical evidence supporting this expectation is rather meager: Except for a single study (Pardamean, <xref ref-type="bibr" rid="B30">2014</xref>), the bulk of published empirical research on the transfer effects on creativity dates back to the 1980s and 1990s (Clements, <xref ref-type="bibr" rid="B7">1986</xref>, <xref ref-type="bibr" rid="B8">1991</xref>, <xref ref-type="bibr" rid="B9">1995</xref>; McGrath, <xref ref-type="bibr" rid="B28">1988</xref>; Subhi, <xref ref-type="bibr" rid="B39">1999</xref>).</p>
</sec>
</sec>
<sec id="s3">
<title>What is needed</title>
<p>In light of this brief review of the current status of knowledge, I would like to follow Mayer (<xref ref-type="bibr" rid="B27">2015</xref>), who discussed the strong need for research evidence on game-based learning, and propose that research on the transfer effects of computer programming needs to move beyond untested claims and the mere description of programming tools, their feasibility, and students&#x00027; interest or enjoyment thereof Fessakis et al. (<xref ref-type="bibr" rid="B13">2013</xref>) raised similar concerns: &#x0201C;Instead of focussing on the cognitive effects of programming, more recent studies concern the development of new programming environments for children [&#x02026;]&#x0201D; (p. 89) Along these lines, educational research needs to focus more on the cognitive consequences of learning to program by designing experimental studies that systematically evaluate the transfer effects, making use of the well-developed programming tools such as Scratch&#x000AE; (Resnick et al., <xref ref-type="bibr" rid="B34">2009</xref>). I believe that the outstanding number of feasibility studies on these tools create a fruitful ground for experimental comparisons. Such comparisons, however, need an appropriate research design, which fulfills at least three criteria (Mayer, <xref ref-type="bibr" rid="B27">2015</xref>): (1) Appropriate <italic>outcome measures</italic> of academic learning or other skills that go beyond students&#x00027; self-reports, enjoyment, or interest; (2) <italic>Experimental control</italic>&#x02014;pretest-posttest designs with a treatment group (i.e., the group that learns to program) and a control group; (3) <italic>Random assignment</italic> of students to treatment and control groups. Despite the very few studies during the last 20 years, which adhered to these criteria (e.g., Pardamean et al., <xref ref-type="bibr" rid="B31">2015</xref>), my observation is that research on the transfer effects of computer programming is in need of methodologically sound, experimental studies.</p>
</sec>
<sec sec-type="conclusions" id="s4">
<title>Conclusion</title>
<p>From a conceptual perspective, the claims that learning computer programming may translate into the development of other, cognate skills such as problem solving or creative thinking, do have their standing, particularly because a considerable conceptual overlap between these skills exists. From an empirical perspective though, it seems as if the conceptual claims have been supplemented with evidence from experimental studies to a very limited extent. My observation is that most of the empirical research on the transfer effects dates back to the 1980s and 1990s; yet, too few studies have looked into these effects in the twenty-first century. This observation is somehow unexpected, particularly because Pea and Kurland (<xref ref-type="bibr" rid="B32">1984</xref>) pointed to the strong need for evidence on the transfer effects of computer programming that takes into account the development of digital technologies more than 20 years ago. Although this plea has been followed by a wave of experimental studies (Liao and Bright, <xref ref-type="bibr" rid="B24">1991</xref>; Clements, <xref ref-type="bibr" rid="B9">1995</xref>), educational research has not systematically followed up on examining the transfer effects. On the basis of the limited research on the transfer effects of computer programming skills on other cognitive skills on the one hand, and the conceptual claims that these transfer effects exist on the other hand, I would like to encourage researchers to fill this gap by learning from the past and reviving this research area in the twenty-first century.</p>
</sec>
<sec id="s5">
<title>Author contribution</title>
<p>RS has conducted the literature review, drafted the manuscript, and performed revisions.</p>
<sec>
<title>Conflict of interest statement</title>
<p>The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p>
</sec>
</sec>
</body>
<back>
<ref-list>
<title>References</title>
<ref id="B1">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Barr</surname> <given-names>V.</given-names></name> <name><surname>Stephenson</surname> <given-names>C.</given-names></name></person-group> (<year>2011</year>). <article-title>Bringing computational thinking to k-12: what is involved and what is the role of the computer science education community?</article-title> <source>ACM Inroads</source> <volume>2</volume>, <fpage>48</fpage>&#x02013;<lpage>54</lpage>. <pub-id pub-id-type="doi">10.1145/1929887.1929905</pub-id></citation>
</ref>
<ref id="B2">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Binkley</surname> <given-names>M.</given-names></name> <name><surname>Erstad</surname> <given-names>O.</given-names></name> <name><surname>Herman</surname> <given-names>J.</given-names></name> <name><surname>Raizen</surname> <given-names>S.</given-names></name> <name><surname>Ripley</surname> <given-names>M.</given-names></name> <name><surname>Miller-Ricci</surname> <given-names>M.</given-names></name> <etal/></person-group>. (<year>2012</year>). <article-title>Defining Twenty-First Century Skills</article-title>, in <source>Assessment and Teaching of 21st Century Skills</source>, eds <person-group person-group-type="editor"><name><surname>Griffin</surname> <given-names>P.</given-names></name> <name><surname>McGaw</surname> <given-names>B.</given-names></name> <name><surname>Care</surname> <given-names>E.</given-names></name></person-group> (<publisher-loc>Dordrecht</publisher-loc>: <publisher-name>Springer</publisher-name>), <fpage>17</fpage>&#x02013;<lpage>66</lpage>. <pub-id pub-id-type="pmid">18046747</pub-id></citation>
</ref>
<ref id="B3">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Brooks</surname> <given-names>R.</given-names></name></person-group> (<year>1999</year>). <article-title>Towards a theory of the cognitive processes in computer programming.</article-title> <source>Int. J. Hum. Comput. Stud.</source> <volume>51</volume>, <fpage>197</fpage>&#x02013;<lpage>211</lpage>. <pub-id pub-id-type="doi">10.1006/ijhc.1977.0306</pub-id></citation>
</ref>
<ref id="B4">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Brown</surname> <given-names>N.</given-names></name> <name><surname>K&#x000F6;lling</surname> <given-names>M.</given-names></name></person-group> (<year>2012</year>). <article-title>Position paper: programming can deepen understanding across disciplines [DRAFT]</article-title>, in <source>Paper Presented at the IFIP Working Conference&#x02013;Addressing Educational Challenges: the Role of ICT</source> (<publisher-loc>Manchester, UK</publisher-loc>: <publisher-name>Manchester Metropolitan University</publisher-name>).</citation>
</ref>
<ref id="B5">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Calder</surname> <given-names>N.</given-names></name></person-group> (<year>2010</year>). <article-title>Using scratch: an integrated problem-solving approach to mathematical thinking.</article-title> <source>Aust. Prim. Math. Classroom</source> <volume>15</volume>, <fpage>9</fpage>&#x02013;<lpage>14</lpage>.</citation>
</ref>
<ref id="B6">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Care</surname> <given-names>E.</given-names></name> <name><surname>Anderson</surname> <given-names>K.</given-names></name></person-group> (<year>2016</year>). <source>How Education Systems Approach Breadth of Skills</source>. <publisher-loc>Washington, DC</publisher-loc>: <publisher-name>Center for Universal Education at BROOKINGS</publisher-name>.</citation>
</ref>
<ref id="B7">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Clements</surname> <given-names>D. H.</given-names></name></person-group> (<year>1986</year>). <article-title>Effects of Logo and CAI environments on cognition and creativity.</article-title> <source>J. Educ. Psychol.</source> <volume>78</volume>, <fpage>309</fpage>&#x02013;<lpage>318</lpage>. <pub-id pub-id-type="doi">10.1037/0022-0663.78.4.309</pub-id></citation>
</ref>
<ref id="B8">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Clements</surname> <given-names>D. H.</given-names></name></person-group> (<year>1991</year>). <article-title>Enhancement of creativity in computer environments.</article-title> <source>Am. Educ. Res. J.</source> <volume>28</volume>, <fpage>173</fpage>&#x02013;<lpage>187</lpage>. <pub-id pub-id-type="doi">10.2307/1162883</pub-id></citation>
</ref>
<ref id="B9">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Clements</surname> <given-names>D. H.</given-names></name></person-group> (<year>1995</year>). <article-title>Teaching creativity with computers.</article-title> <source>Educ. Psychol. Rev.</source> <volume>7</volume>, <fpage>141</fpage>&#x02013;<lpage>161</lpage>. <pub-id pub-id-type="doi">10.1007/bf02212491</pub-id></citation>
</ref>
<ref id="B10">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Clements</surname> <given-names>D. H.</given-names></name> <name><surname>Merriman</surname> <given-names>S.</given-names></name></person-group> (<year>1988</year>). <article-title>Componential developments in LOGO programming environments</article-title>, in <source>Teaching and Learning Computer Programming: Multiple Research Perspectives</source>, ed <person-group person-group-type="editor"><name><surname>Mayer</surname> <given-names>R. E.</given-names></name></person-group> (<publisher-loc>Hillsdale</publisher-loc>: <publisher-name>Lawrence Erlbaum Associates, Inc.</publisher-name>), <fpage>13</fpage>&#x02013;<lpage>54</lpage>.</citation>
</ref>
<ref id="B11">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Denning</surname> <given-names>P. J.</given-names></name></person-group> (<year>2010</year>). <article-title>Great principles of computing.</article-title> <source>Am. Sci.</source> <volume>98</volume>, <fpage>369</fpage>&#x02013;<lpage>372</lpage>. <pub-id pub-id-type="doi">10.1511/2010.86.369</pub-id></citation>
</ref>
<ref id="B12">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Dyck</surname> <given-names>J. L.</given-names></name> <name><surname>Mayer</surname> <given-names>R. E.</given-names></name></person-group> (<year>1989</year>). <article-title>Teaching for transfer of computer program comprehension skill.</article-title> <source>J. Educ. Psychol.</source> <volume>81</volume>, <fpage>16</fpage>&#x02013;<lpage>24</lpage>. <pub-id pub-id-type="doi">10.1037/0022-0663.81.1.16</pub-id></citation>
</ref>
<ref id="B13">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Fessakis</surname> <given-names>G.</given-names></name> <name><surname>Gouli</surname> <given-names>E.</given-names></name> <name><surname>Mavroudi</surname> <given-names>E.</given-names></name></person-group> (<year>2013</year>). <article-title>Problem solving by 5&#x02013;6 years old kindergarten children in a computer programming environment: a case study.</article-title> <source>Comput. Educ.</source> <volume>63</volume>, <fpage>87</fpage>&#x02013;<lpage>97</lpage>. <pub-id pub-id-type="doi">10.1016/j.compedu.2012.11.016</pub-id></citation>
</ref>
<ref id="B14">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Gibbon</surname> <given-names>L. W.</given-names></name></person-group> (<year>2007</year>). <source>Effects of LEGO Mindstorms on Convergent and Divergent Problem-Solving and Spatial Abilities in Fifth and Sixth Grade Students (Doctor of Education)</source>. <publisher-loc>Seattle, WA</publisher-loc>; <publisher-name>Seattle Pacific University.</publisher-name></citation>
</ref>
<ref id="B15">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Greiff</surname> <given-names>S.</given-names></name> <name><surname>W&#x000FC;stenberg</surname> <given-names>S.</given-names></name> <name><surname>Csap&#x000F3;</surname> <given-names>B.</given-names></name> <name><surname>Demetriou</surname> <given-names>A.</given-names></name> <name><surname>Hautam&#x000E4;ki</surname> <given-names>J.</given-names></name> <name><surname>Graesser</surname> <given-names>A. C.</given-names></name> <etal/></person-group>. (<year>2014</year>). <article-title>Domain-general problem solving skills and education in the 21st century</article-title>. <source>Educ. Res. Rev.</source> <volume>13</volume>, <fpage>74</fpage>&#x02013;<lpage>83</lpage>. <pub-id pub-id-type="doi">10.1016/j.edurev.2014.10.002</pub-id></citation>
</ref>
<ref id="B16">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Grover</surname> <given-names>S.</given-names></name> <name><surname>Pea</surname> <given-names>R.</given-names></name></person-group> (<year>2013</year>). <article-title>Computational thinking in K-12: a review of the state of the field</article-title>. <source>Educ. Res.</source> <volume>42</volume>, <fpage>38</fpage>&#x02013;<lpage>43</lpage>. <pub-id pub-id-type="doi">10.3102/0013189x12463051</pub-id></citation>
</ref>
<ref id="B17">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>G&#x000FC;lbahar</surname> <given-names>Y.</given-names></name> <name><surname>Kalelio&#x0011F;lo</surname> <given-names>F.</given-names></name></person-group> (<year>2014</year>). <article-title>The effects of teaching programming via Scratch on problem solving skills: a discussion from learners&#x00027; perspective</article-title>. <source>Inform. Educ.</source> <volume>13</volume>, <fpage>33</fpage>&#x02013;<lpage>50</lpage>.</citation>
</ref>
<ref id="B18">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Jonassen</surname> <given-names>D. H.</given-names></name></person-group> (<year>2011</year>). <source>Learning to Solve Problems: A Handbook for Designing Problem-Solving Learning Environments</source>. <publisher-loc>New York, NY</publisher-loc>: <publisher-name>Routledge</publisher-name>.</citation>
</ref>
<ref id="B19">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Kay</surname> <given-names>K.</given-names></name> <name><surname>Greenhill</surname> <given-names>V.</given-names></name></person-group> (<year>2011</year>). <article-title>Twenty-First century students need 21st century skills</article-title>, in <source>Bringing Schools into the 21st Century</source>, eds <person-group person-group-type="editor"><name><surname>Wan</surname> <given-names>G.</given-names></name> <name><surname>Gut</surname> <given-names>M. D.</given-names></name></person-group> (<publisher-loc>Dordrecht</publisher-loc>: <publisher-name>Springer</publisher-name>), <fpage>41</fpage>&#x02013;<lpage>65</lpage>. <pub-id pub-id-type="pmid">17017265</pub-id></citation>
</ref>
<ref id="B20">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kazakoff</surname> <given-names>E. R.</given-names></name> <name><surname>Sullivan</surname> <given-names>A.</given-names></name> <name><surname>Bers</surname> <given-names>M. U.</given-names></name></person-group> (<year>2013</year>). <article-title>The effect of a classroom-based intensive robotics and programming workshop on sequencing ability in early childhood</article-title>. <source>Early Child. Educ. J.</source> <volume>41</volume>, <fpage>245</fpage>&#x02013;<lpage>255</lpage>. <pub-id pub-id-type="doi">10.1007/s10643-012-0554-5</pub-id></citation>
</ref>
<ref id="B21">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Klahr</surname> <given-names>D.</given-names></name> <name><surname>Dunbar</surname> <given-names>K.</given-names></name></person-group> (<year>1988</year>). <article-title>Dual space search during scientific reasoning.</article-title> <source>Cogn. Sci.</source> <volume>12</volume>, <fpage>1</fpage>&#x02013;<lpage>48</lpage>. <pub-id pub-id-type="doi">10.1207/s15516709cog1201_1</pub-id></citation>
</ref>
<ref id="B22">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Korkmaz</surname> <given-names>&#x000D6;.</given-names></name></person-group> (<year>2016</year>). <article-title>The effect of scratch- and lego mindstorms Ev3-Based programming activities on academic achievement, problem-solving skills and logical-mathematical thinking skills of students</article-title>. <source>Malays. Online J. Educ. Sci.</source> <volume>4</volume>, <fpage>73</fpage>&#x02013;<lpage>88</lpage>.</citation>
</ref>
<ref id="B23">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Lai</surname> <given-names>A.-F.</given-names></name> <name><surname>Yang</surname> <given-names>S.-M.</given-names></name></person-group> (<year>2011</year>). <article-title>The learning effect of visualized programming learning on 6 th graders&#x00027; problem solving and logical reasoning abilities</article-title>, in <source>Paper presented at the International Conference on Electrical and Control Engineering (ICECE)</source> (<publisher-loc>Yichang</publisher-loc>).</citation>
</ref>
<ref id="B24">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Liao</surname> <given-names>Y.-K. C.</given-names></name> <name><surname>Bright</surname> <given-names>G. W.</given-names></name></person-group> (<year>1991</year>). <article-title>Effects of computer programming on cognitive outcomes: a meta-analysis</article-title>. <source>J. Educ. Compu. Res.</source> <volume>7</volume>, <fpage>251</fpage>&#x02013;<lpage>268</lpage>. <pub-id pub-id-type="doi">10.2190/e53g-hh8k-ajrr-k69m</pub-id></citation>
</ref>
<ref id="B25">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lye</surname> <given-names>S. Y.</given-names></name> <name><surname>Koh</surname> <given-names>J. H. L.</given-names></name></person-group> (<year>2014</year>). <article-title>Review on teaching and learning of computational thinking through programming: what is next for K-12?</article-title> <source>Comput. Human Behav.</source> <volume>41</volume>, <fpage>51</fpage>&#x02013;<lpage>61</lpage>. <pub-id pub-id-type="doi">10.1016/j.chb.2014.09.012</pub-id></citation>
</ref>
<ref id="B26">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Maloney</surname> <given-names>J.</given-names></name> <name><surname>Burd</surname> <given-names>L.</given-names></name> <name><surname>Kafai</surname> <given-names>Y.</given-names></name> <name><surname>Rusk</surname> <given-names>N.</given-names></name> <name><surname>Silverman</surname> <given-names>B.</given-names></name> <name><surname>Resnick</surname> <given-names>M.</given-names></name></person-group> (<year>2004</year>). <article-title>Scratch: A sneak preview</article-title>, in <source>Paper presented at the Second International Conference on Creating, Connecting and Collaborating Through Computing</source> (<publisher-loc>Kyoto</publisher-loc>).</citation>
</ref>
<ref id="B27">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Mayer</surname> <given-names>R. E.</given-names></name></person-group> (<year>2015</year>). <article-title>On the need for research evidence to guide the design of computer games for learning.</article-title> <source>Educ. Psychol.</source> <volume>50</volume>, <fpage>349</fpage>&#x02013;<lpage>353</lpage>. <pub-id pub-id-type="doi">10.1080/00461520.2015.1133307</pub-id></citation>
</ref>
<ref id="B28">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>McGrath</surname> <given-names>D.</given-names></name></person-group> (<year>1988</year>). <article-title>Programming and problem solving: will two languages do it?</article-title> <source>J. Educ. Comput. Res.</source> <volume>4</volume>, <fpage>467</fpage>&#x02013;<lpage>484</lpage>.</citation>
</ref>
<ref id="B29">
<citation citation-type="book"><person-group person-group-type="author"><collab>OECD</collab></person-group> (<year>2014</year>). <source>PISA 2012 Results: Creative Problem Solving: Students&#x00027; Skills in Tackling Real-Life Problems</source>, <volume>Vol. 5</volume>. <publisher-loc>Paris</publisher-loc>: <publisher-name>OECD Publishing</publisher-name>.</citation>
</ref>
<ref id="B30">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Pardamean</surname> <given-names>B.</given-names></name></person-group> (<year>2014</year>). <article-title>Enhancement of creativity through logo programming</article-title>. <source>Am. J. Appl. Sci.</source> <volume>11</volume>, <fpage>528</fpage>&#x02013;<lpage>533</lpage>. <pub-id pub-id-type="doi">10.3844/ajassp.2014.528.533</pub-id></citation>
</ref>
<ref id="B31">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Pardamean</surname> <given-names>B.</given-names></name> <name><surname>Suparyanto</surname> <given-names>T.</given-names></name> <name><surname>Evelyn</surname></name></person-group>. (<year>2015</year>). <article-title>Improving problem-solving skills through Logo programming language</article-title>. <source>New Educ. Rev.</source> <volume>41</volume>, <fpage>52</fpage>&#x02013;<lpage>64</lpage>. <pub-id pub-id-type="doi">10.15804/tner.2015.41.3.04</pub-id></citation>
</ref>
<ref id="B32">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Pea</surname> <given-names>R. D.</given-names></name> <name><surname>Kurland</surname> <given-names>D. M.</given-names></name></person-group> (<year>1984</year>). <article-title>On the cognitive effects of learning computer programming.</article-title> <source>New Ideas Psychol.</source> <volume>2</volume>, <fpage>137</fpage>&#x02013;<lpage>168</lpage>.</citation>
</ref>
<ref id="B33">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Pellegrino</surname> <given-names>J. W.</given-names></name> <name><surname>Hilton</surname> <given-names>M.</given-names></name></person-group> (<year>2012</year>). <source>Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century</source>. <publisher-loc>Washington, DC</publisher-loc>: <publisher-name>The National Academies Press</publisher-name>.</citation>
</ref>
<ref id="B34">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Resnick</surname> <given-names>M.</given-names></name> <name><surname>Maloney</surname> <given-names>J.</given-names></name> <name><surname>Monroy-Hern&#x000E1;ndez</surname> <given-names>A.</given-names></name> <name><surname>Rusk</surname> <given-names>N.</given-names></name> <name><surname>Eastmond</surname> <given-names>E.</given-names></name> <name><surname>Brennan</surname> <given-names>K.</given-names></name> <etal/></person-group>. (<year>2009</year>). <article-title>Scratch: programming for all.</article-title> <source>Commun. ACM</source> <volume>52</volume>, <fpage>60</fpage>&#x02013;<lpage>67</lpage>.</citation>
</ref>
<ref id="B35">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rich</surname> <given-names>P. J.</given-names></name> <name><surname>Bly</surname> <given-names>N.</given-names></name> <name><surname>Leatham</surname> <given-names>K. R.</given-names></name></person-group> (<year>2014</year>). <article-title>Beyond cognitive increase: investigating the influence of computer programming on perception and application of mathematical skills</article-title>. <source>J. Comput. Math. Sci. Teach.</source> <volume>33</volume>, <fpage>103</fpage>&#x02013;<lpage>128</lpage>.</citation>
</ref>
<ref id="B36">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Scherer</surname> <given-names>R.</given-names></name></person-group> (<year>2015</year>). <article-title>Is it time for a new measurement approach? A closer look at the assessment of cognitive adaptability in complex problem solving</article-title>. <source>Front. Psychol.</source> <volume>6</volume>:<issue>1664</issue>. <pub-id pub-id-type="doi">10.3389/fpsyg.2015.01664</pub-id><pub-id pub-id-type="pmid">26579045</pub-id></citation>
</ref>
<ref id="B37">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Siddiq</surname> <given-names>F.</given-names></name> <name><surname>Hatlevik</surname> <given-names>O. E.</given-names></name> <name><surname>Olsen</surname> <given-names>R. V.</given-names></name> <name><surname>Throndsen</surname> <given-names>I.</given-names></name> <name><surname>Scherer</surname> <given-names>R.</given-names></name></person-group> (<year>2016</year>). <article-title>Taking a future perspective by learning from the past &#x02013; A systematic review of assessment instruments that aim to measure primary and secondary school students&#x00027; ICT literacy</article-title>. <source>Educ. Res. Rev.</source> <volume>19</volume>, <fpage>58</fpage>&#x02013;<lpage>84</lpage>. <pub-id pub-id-type="doi">10.1016/j.edurev.2016.05.002</pub-id></citation>
</ref>
<ref id="B38">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Sturman</surname> <given-names>L.</given-names></name> <name><surname>Sizmur</surname> <given-names>J.</given-names></name></person-group> (<year>2011</year>). <source>International Comparison of Computing in Schools</source>. <publisher-loc>Slough</publisher-loc>: <publisher-name>National Foundation for Educational Research</publisher-name>.</citation>
</ref>
<ref id="B39">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Subhi</surname> <given-names>T.</given-names></name></person-group> (<year>1999</year>). <article-title>The impact of LOGO on gifted children&#x00027;s achievement and creativity</article-title>. <source>J. Comput. Assist. Learn.</source> <volume>15</volume>, <fpage>98</fpage>&#x02013;<lpage>108</lpage>. <pub-id pub-id-type="doi">10.1046/j.1365-2729.1999.152082.x</pub-id></citation>
</ref>
<ref id="B40">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Tricot</surname> <given-names>A.</given-names></name> <name><surname>Sweller</surname> <given-names>J.</given-names></name></person-group> (<year>2013</year>). <article-title>Domain-specific knowledge and why teaching generic skills does not work.</article-title> <source>Educ. Psychol. Rev.</source> <volume>26</volume>, <fpage>265</fpage>&#x02013;<lpage>283</lpage>. <pub-id pub-id-type="doi">10.1007/s10648-013-9243-1</pub-id></citation>
</ref>
<ref id="B41">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Wagner</surname> <given-names>T.</given-names></name></person-group> (<year>2012</year>). <source>Creating Innovators - The Making of Young People Who Will Change the World</source>. <publisher-loc>New York, NY</publisher-loc>: <publisher-name>Scribner</publisher-name>.</citation>
</ref>
<ref id="B42">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Wallas</surname> <given-names>G.</given-names></name></person-group> (<year>1926</year>). <source>The Art of Thought</source>. <publisher-loc>New York, NY</publisher-loc>: <publisher-name>Harcourt, Brace and Company</publisher-name>.</citation>
</ref>
<ref id="B43">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Webb</surname> <given-names>M.</given-names></name> <name><surname>Davis</surname> <given-names>N.</given-names></name> <name><surname>Bell</surname> <given-names>T.</given-names></name> <name><surname>Katz</surname> <given-names>Y. J.</given-names></name> <name><surname>Reynolds</surname> <given-names>N.</given-names></name> <name><surname>Chambers</surname> <given-names>D. P.</given-names></name> <etal/></person-group>. (<year>2016</year>). <article-title>Computer science in K-12 school curricula of the 2lst century: why, what and when?</article-title> <source>Educ. Inf. Technol</source>. [Epub ahead of print]. <pub-id pub-id-type="doi">10.1007/s10639-016-9493-x</pub-id></citation>
</ref>
<ref id="B44">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wilson</surname> <given-names>A.</given-names></name> <name><surname>Hainey</surname> <given-names>T.</given-names></name> <name><surname>Connolly</surname> <given-names>T. M.</given-names></name></person-group> (<year>2013</year>). <article-title>Using Scratch with primary school children: an evaluation of games constructed to gauge understanding of programming concepts</article-title>. <source>Int. J. Game-Based Learn.</source> <volume>3</volume>, <fpage>93</fpage>&#x02013;<lpage>109</lpage>. <pub-id pub-id-type="doi">10.4018/ijgbl.2013010107</pub-id></citation>
</ref>
<ref id="B45">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wing</surname> <given-names>J. M.</given-names></name></person-group> (<year>2006</year>). <article-title>Computational thinking.</article-title> <source>Commun. ACM</source> <volume>49</volume>, <fpage>33</fpage>&#x02013;<lpage>35</lpage>. <pub-id pub-id-type="doi">10.1145/1118178.1118215</pub-id></citation>
</ref>
</ref-list>
</back>
</article>