Seven studies examined the effect of CS on interest in science or project-specific content. Most of these studies had a low internal validity because they used simple research designs (e.g., Hiller and Kitsantas, 2014; Kelemen-Finan et al., 2018). For example, Kelemen-Finan et al. (2018) implemented a one-time posttest with N = 428 students after participation in a biodiversity CS project and had no control group. Because one major weakness of this research design is that it could not measure a change in individual outcomes (Campbell and Stanley, 1963), causal assumptions about the effects of CS on interest are not valid. Causally attributing a positive effect on interest to participation in CS, this study serves as an example of overly interpreted posttest data. As shown in Table 2, only two studies investigating the effect of CS on students’ interest met the criteria of two measurement points (Seifert et al., 2016; Wallace, 2018). Seifert et al. (2016) examined the effects of a CS project on ticks and Lyme disease. After a one-hour introduction in the classroom, more than 200 high school students collected field data and ticks along 100-m transects in the surroundings of their schools, which where then committed and analyzed in university research. They implemented a pre- and posttest with a subsample of n = 23 students and were able to measure changes in the interest in science. Their results showed that students had an increased interest in pursuing science in college and graduate school after participation in CS. However, in the absence of a control group, the increase in interest cannot be causally attributed to participation in the CS project. Further, the absence of theoretical work leads to low construct validity. That is, these findings are of limited use for testing theoretical assumptions about CS and further developing basic scientific theories. Wallace (2018) implemented experimental and control groups as well as pre- and posttests, and was thereby able to control for group differences. In this study, N = 137 ninth-grade students (53.3% female) participated in the CS project BudBurst and were asked to monitor phenological changes of local tree species. To further investigate the effects of mobile devices, students were divided into three groups (participation in CS, participation in CS with mobile devices, no participation in CS). The different groups were surveyed before and after participation, that consisted of an introduction in the classroom and –depending on condition– two data collection events over a 5 week period. Results indicated an increase in STEM interest through participation in CS. Groups had comparable pretest scores and differed from each other statistically on the posttest score. Both experimental groups (participation in CS) had a significantly higher score than the control group on the STEM interest scale, but they did not differ from each other. To sum up, both studies (Seifert et al., 2016; Wallace, 2018) had an institutional context, whereas CS addresses the general population. This casts results in a critical light regarding their transferability to the overall population (external validity). External validity was limited further because both studies investigated the effect of a multi-day participation in CS projects in the field of biodiversity. These limitations to a specific sample and certain project forms do not yet allow any general conclusion about the effect of CS on interest in science. Given the importance of interest in learning processes, the results at least supported the role of CS in the school context and have an individual informative value.

In contrast to the studies on CS and interest, the 43 studies found on motivation rarely investigated a change in educational outcomes. There was often a different research interest, such as why people participated in CS (descriptive) or how motivation related to the extent of participation (correlative). Consequently, only four studies implemented two measurement points providing evidence for changes in motivation. However, a key problem with two of these studies (Domroese and Johnson, 2017; Dem et al., 2018) was that they did not use the same pre- and posttest. Due to these methodological limitations, any potential change in motivation through CS cannot be correctly confirmed empirically. The remaining studies surveyed participants of the CS projects Gardenroots (Sandhaus et al., 2019) and STEMhero (Condon and Wichowsky, 2018). In the study by Sandhaus et al. (2019), N = 94 participants were trained in data collection and were asked to collect plant, water and soil samples in their private gardens. After the samples have been analyzed by researchers, the results were presented at multiple data sharing events. The study revealed few significant changes in motivation for science learning and environmental action on the level of individual items. Due to the small number of completed posttests (n = 16) the external validity is rather low. The study by Condon and Wichowsky (2018) was based on a larger sample of N = 551 students from Catholic middle schools. In course of the CS project STEMhero, that was carried out by the respective teacher of a class, students were asked to track their water meters at home, to analyze their consumption, and to implement strategies to increase water efficiency. Results revealed that the experimental group, that was participating in STEMhero over a period of 2.5 weeks showed significantly greater gains in motivation to pursue further study in science or math. However, since this study has particularities regarding the sample and the extent of participation, the available data on an eventual change in participants’ motivation do not allow for generalized statements.

We found 15 studies focusing on attitudes toward science. Similar to the studies on interest and motivation, only a few studies (n = 7) investigated the effect of CS on attitudes toward science using questionnaires at multiple measurement points (Table 2). Six studies (Brossard et al., 2005; Jordan et al., 2011; Basham, 2012; Crall et al., 2012; Gottschalk-Druschke and Seltzer, 2012; Vitone et al., 2016) did not find any effect of CS on attitudes toward science. In the study by Basham (2012), for example, the effect of the Mastodon Matrix Project was studied with N = 11 participants of an English Language course for adults. They were asked to collect data on organic and inorganic materials by sorting through soil samples over a period of 3 weeks after having received a short introduction. The project also included a collaborative analysis and discussion of data. However, the results did not show any significant change in attitudes toward science.

Beyond these results, what these studies did have in common was their advanced research design: all had two measurement points, and two studies implemented a control group (Brossard et al., 2005; Crall et al., 2012). The CS Project The Birdhouse Network (Brossard et al., 2005) asked participants to submit data on the use of nest boxes over a certain time period. In this study, the use of an established measurement scale (MATOSS) was theory-driven and provided high construct validity. However, the comparison between pre- and posttest in the treatment group did not show any statistically significant change in attitudes toward science. N = 214 participants (75% female in experimental condition, 61% female in control condition) in the NISS project (Crall et al., 2012) were required to map invasive plant species after receiving 8 h of training. Participants in the experimental and control groups similarly received the MATOSS (Brossard et al., 2005) for pre- and posttests to validly capture attitudes toward science and ensure comparability of findings. Nonetheless, as in the study of Brossard et al. (2005), the comparison between pre- and posttest did not show any differences in attitudes toward science. Only one study investigating the effect of a CS project in the field of astronomy found a positive effect of CS on attitudes toward science (Price and Lee, 2013). The effect of CS on attitudes toward science was studied using the online CS project Citizen Sky. Since this project placed particular emphasis on communication between scientists and citizens and on empowering citizens to independently conduct their research, this study further investigated whether the level of participation was responsible for these effects. Overall, N = 333 citizen scientists of which the majority were men participated in monitoring the star epsilon Aurigae and responded to pre- and posttest. Because this study included established measurement instruments, a certain degree of construct validity in the results can be assumed. Besides a positive effect of participation on attitudes toward science, the results of the study revealed that those who participated more actively in the communication (e.g., online discussion forums) showed a significantly greater change from pre- to posttest. In contrast to the other studies, the CS project Citizen Sky provided for a high degree of participation, and the sample consisted of people with long-term participation in the project (posttest during the first login to the website 6 months after pretest). Thus, long-term participation in the project might have been a crucial condition for this positive effect. However, results were of lower internal validity due to the absence of a control group. All things considered, the methodological approaches used in these studies allow more general statements to be derived. Due to the implementation of control groups in some studies, their results were of high internal validity because they were able to test for causal assumptions on CS (Kaya, 2015). The main trend is that attitudes toward science tend not to be changed by brief participation in a CS project. Initial study results suggest that long-term and intensive participation in a CS project is likely to be required to achieve change in this relatively stable personal variable (Price and Lee, 2013). Participation in the entire scientific process and its reflection can provide enough occasions to initiate change. To support this assumption, however, further research is necessary in this field. In terms of design, research should be oriented toward the studies conducted to date and, if possible, include further measurement instruments.

A total of 173 studies addressed the question of data quality. This high number is probably due to the controversy over the ability of CS data to answer scientific research questions (Hunter, 2013; Callaghan et al., 2019). Recent studies have reported an alignment of data derived by citizen scientists with those gathered by experts—e.g., a classification accuracy of trail camera images of more than 93% (Clare et al., 2019). Callaghan et al. (2020) also mentioned a similarity of data collection performance (regarding frogs in Australia) between citizen scientists and field experts. However, they also emphasized a bias toward certain sampling areas (i.e., citizen scientists favored areas with high human populations when collecting data). To overcome this issue and other potential flaws of citizen-generated data, Fucillo et al. (2015) found that training citizen scientists on data collection skills proved to raise data quality close to the experts’ level. Twenty-eight citizen scientists received formal training for plant observation and the accuracy of nearly 11,000 observations was compared to those collected by a professional. An overall accuracy of 91% was found for CS data, which did not differ for different extent of participation (few vs. many observations). A less resource-intensive strategy was introduced by Torre et al. (2019), who showed that giving the option “I do not know” in classification activities enhanced accuracy and thus data quality. Ninety-four participants were asked to classify images of polluted water bodies whether this pollution was a threat to the environment or not. If the participants received the option “I do not know,” there was an improvement in the true negative rate, i.e., this group showed higher accuracy in the identification of “no threat” than the group without this option. The increased data quality goes along with a reduction in data quantity, but may be accepted to ensure the accuracy of CS data.

About 20% of the studies (n = 279) used CS data to answer scientific questions. CS data is not limited to a few scientific fields but is very diverse. One example is clarifying how coral reefs change as an ecosystem and might react to future disturbances (Gouraguine et al., 2019). In this study, a decline in coral was revealed with data from about 275 citizen scientists who monitored permanent transects over a period of 10 years. Another example is using CS data to provide answers on how Yellowhammers’ song dialects are distributed in Czechia (Diblíková et al., 2018). For this purpose, nearly 4,000 recordings were used, which were collected from citizens over a period of 6 years. Apart from questions on various organisms, data gathered by citizen scientists have also been beneficial in other disciplines: for instance, American astronomers identified a new planet with the help of CS data from the project Planet Hunters (Citizen Science Finds Planet, 2012). Data from citizen scientists have also aided in the medical field, e.g., by providing information on the prevalence of ticks carrying pathogens causing borreliosis in the US (Nieto et al., 2018). Citizens collected over 16,000 ticks and submitted them with information on where they were found (host, location) to Northern Arizona University where the ticks were tested for pathogens.

There were 240 project descriptions. A suitable example for studies in this category is Chiovitti et al.’ (2019) presentation of an educational barcoding project in which students (N = 406) extracted and analyzed DNA from reptile livers. Inter alia, the authors describe the project’s process, the (scientific and educational) materials used, and compliance with the school curriculum. The participating students collected and analyzed about 200 samples creating also new sequences for 8 reptile species. Most studies in this category aimed to describe a particular CS project and present results covering the number of samples and species that the participants examined. However, they did not report empirical results on either a project’s effects on its participants or on a project’s empirical findings and the quality of generated data.

There were 74 studies dealing with the clustering of numerous studies in one area such as biodiversity. For example, the study by Pocock et al. (2017) conducted a systematic search for environmental and ecological CS projects. Each of the 509 projects found was scored systematically for 32 attributes—inter alia, methodological approaches, types of data recording, and target groups. It was found, for example, that the number of biodiversity projects increased exponentially and over 90% of the analyzed CS projects limit citizen involvement to data collection. Hence, this study clustered and described the projects without either going into detail on possible influences on the participants or discussing the data quality.

There were 105 publications theoretically describing the advantages and disadvantages of CS. Pocock et al. (2018), for example, derived opportunities for, benefits of, and barriers to CS in East Africa from a collaborative prioritization among 22 experts because CS is insufficiently distributed in developing countries. From this systematic theoretical assessment, the authors concluded that biodiversity and environmental monitoring are opportunities, that increasing environmental awareness as well as useful data and approaches are benefits, and that institutional capacity and (lack of) ascribed value to such activities are barriers that are applicable across developing countries. As another example, Fritz et al. (2019) described how CS as a non-traditional data source could support the Sustainable Development Goals (SDGs). As traditional data sources are presented as insufficient for measuring the SDGs, the potential uses of CS data are outlined as well as a roadmap for integrating CS in SDG reporting.

Almost 20% of the articles (n = 221) dealt with developing technical devices that could be used in the implementation of CS projects. Ožana et al. (2019) presented a mobile application to map species occurrences of dragonflies. The technical structure of the app as well as the various functions, such as support in the classification of dragonflies or the storage of GPS data, are presented. Another example is the study by Dennis et al. (2017) presenting new approaches for the analysis of biodiversity data with logistic regression, as CS data usually have a large spatial coverage and provide an important basis for biodiversity monitoring.

The large number of project descriptions (b1, b2) show that there are already many different CS projects in which numerous people participate and can thus contribute to an extensive database. Additionally, studies on technical devices (c2) show that numerous tools further facilitate participation in CS projects; and, in some cases, allow it to take place at any time around the globe. The large number of studies using CS data (a3) shows that scientific publications based on CS data have already been published very frequently, and that they have contributed to relevant research questions. Finally, many studies on data quality (a2) provide evidence of high quality in CS data allowing confidence to be placed in the scientific publications. All these results together provide initial evidence for scientific outcomes of CS.

One key requirement for future studies is to be of high internal validity, because it “is the basic minimum without which any experiment is uninterpretable” (Campbell and Stanley, 1963, p. 5). Internal validity is significantly ensured by the implementation of control groups. To achieve a true experimental design, participants need to be randomly assigned to different groups (Campbell and Stanley, 1963). If random assignment cannot be realized (quasi-experiment), confounding variables must be controlled with at least one pretest allowing control of preexisting group differences. It is certainly challenging to conduct experimental designs in CS contexts, because numerous organizational and ethical aspects have to be taken into account. One organizational aspect that limits feasibility in the context of CS is the difficulty in recruiting comparable control groups. In addition, it is necessary to balance the contrast between the voluntary nature of participating in CS to actively conduct research and the fact that participants become the research object in the experiment itself. This is probably why only a few studies are available with quasi-experimental designs ensuring a moderate internal validity (e.g., Wallace, 2018). Because only studies implementing control groups provide robust results on the effect of CS on educational outcomes, the implementation of such designs should be a central goal for future research. Further, it is important to consider the evaluation of effects from the outset “largely because many impact measurement tools (particularly those offering high-quality data) require baseline measurements before participation” (Somerwill and Wehn, 2022, p. 9). Dickinson and Crain (2019) also emphasize that “controlled studies are needed to determine whether citizen science projects meet the specific learning objectives for which they are designed” (p. 1). This lack of experimental studies has already been noted in previous reviews focusing on changes in knowledge, behavior, and attitudes through participation in CS (Peter et al., 2019; Aristeidou and Herodotou, 2020) and we can generalize their demands to cover empirical studies.

Causal knowledge of individual experiments does not allow conclusions to be drawn on whether this causal relationship holds over variations in participants and settings (Shadish et al., 2002). To make generalized statements about the effect of CS on educational outcomes, numerous experiments would need to achieve similar results while differing in terms of sample, setting, treatment, and measure (Shadish et al., 2002). Many recent studies have been carried out in an institutional context granting continued access to the participants for recurring surveys. These limitations to a specific sample and setting restrict the transferability of previous results on the effect of CS. Future research should try to focus on CS projects outside of institutional boundaries, because the majority of projects have been developed for the general population and not for classroom implementation. Further, most of the studies on educational outcomes examine CS projects in the subject field of biodiversity (e.g., Aivelo and Huovelin, 2020). Other subjects that are also represented more frequently are astronomy and water quality (e.g., Price and Lee, 2013). To ensure the generalizability of results, future research needs to provide equal representation to the different scientific disciplines in which CS projects are offered. With regard to the treatment, the vast majority of studies investigate CS projects providing for a low level of citizen participation in the scientific process (e.g., Seifert et al., 2016; Tinati et al., 2017). To further clarify the effect of CS on educational outcomes and to confirm the findings with respect to attitudes toward science, future research should consider different project forms providing for different levels of participation (e.g., full participation in the scientific process). Finally, different (motivational) variables should become the object of investigation. This would allow comprehensive statements on educational outcomes. In this context, different and, ideally, already established measurement instruments should be used. This leads to the aspect of construct validity.

Construct validity is another fundamental concern in any empirical study on the effect of educational outcomes of CS. In the referenced studies, we observed a frequent absence of theory-based operationalizations of constructs and consequently the use of self-developed measurement instruments with no assessment of their quality. Our results also confirm previous reviews criticizing “the scarcity of established theoretical frameworks” (Peter et al., 2019, p. 14) and the “tendency to use self-reported methods” (Aristeidou and Herodotou, 2020, p. 9) in CS projects. Due to the lack of theoretical work, findings from individual studies can mostly be used only to a limited extent for testing theoretical assumptions about CS and further developing basic scientific theories. To validly draw inferences about the construct under consideration, it is important to define it theoretically before developing suitable items. Moreover, the use of established measurement instruments should be preferred. For instance, scales to measure attitudes have been developed and evaluated by, for example Kind et al. (2007), Lederman et al. (2014), and Hartman et al. (2017). Levontin et al. (2022) suggest a standardized, theory-based approach to measure participants’ motivation in a comparable manner. These measures ensure the validity of the intended interpretation of the results and enable further development of basic scientific theories. Because theoretical work and the use of instruments with high test quality are rare to date, this should be given greater consideration in future research. Interdisciplinary cooperation may be helpful in theoretically defining the constructs and in carrying out a well-founded selection of instruments here.