In this study, we aim to develop and evaluate a refined set of scales that distinguishes key components of disciplinary authenticity, drawing on the authenticity framework proposed by Betz et al. (2016). The resulting instrument comprises six theoretically derived dimensions: location, instructor, materials, methods, innovation, and the newly introduced dimension content. To address issues of conceptual overlap and insufficient discrimination, the definitions for the materials and methods scales were revised and a new definition was created for content, while the original definitions for location, instructor, and innovation were mostly retained. In addition, each dimension is explicitly differentiated from other related dimensions to ensure conceptual clarity and to avoid overlap in interpretation. As the original definitions were formulated in German, the following section provides English translations of all six dimensions, including the revised and newly developed ones.

Location refers to the physical context surrounding the learning setting. This dimension captures how authentically students perceive the buildings, rooms and places visited. It also includes locations beyond student laboratories, such as botanical gardens, excavation sites or research vessels (Braund and Reiss, 2006). A location is considered authentic if it is actually or potentially used for scientific work (Schüttler et al., 2021). This dimension differs from instructor, materials, and methods in that it focuses solely on the spatial characteristics of the learning setting rather than on the people, artifacts or procedures involved in scientific work.

Instructor refers to individuals who take on a teaching or guiding role within the learning activity. This dimension reflects how authentically students perceive these individuals. Instructors are considered authentic if they work professionally as scientists and offer direct access to real scientific practices within their discipline (Nicaise et al., 2000).

Innovation refers to the novelty of the research question or problem addressed. A question is considered authentic if it is embedded in ongoing, relevant research and allows students to contribute to its resolution. The more central the inquiry into something genuinely new is and the less existing knowledge is merely received or replicated, the more innovative the project is perceived to be (Sommer et al., 2020). This dimension differs from content in that it focuses on the epistemic novelty of the task rather than on the scientific relevance of the subject matter itself.

Methods refer to the specific scientific procedures used to investigate a research question. This dimension reflects the extent to which students perceive their working process as scientific. A method is considered authentic when it aligns with the standards of disciplinary inquiry and is embedded in a systematic process of knowledge generation. Typical examples include experimentation, modeling, simulation, data analysis or observation (Bolger et al., 2021). Although there is a pluralism of methods within scientific disciplines, in most cases, only one of these methods is focused on in educational contexts. This dimension differs from materials in that it concerns the scientific procedures themselves rather than the physical artifacts used to carry them out.

Materials refer to all objects that students interact with during the activity, excluding instructional materials. This dimension reflects how authentic students perceive these materials. Materials are considered authentic when they originate from scientific research or closely resemble such materials in form, function and use, with minimal didactic simplification (Sommer et al., 2018). This includes both objects of investigation (e.g., soil samples, texts, interview transcripts) and instruments or visualizations that support real research (e.g., microscopes, measuring devices, simulations or lab software; Smetana and Bell, 2012). This dimension differs from methods in that it focuses on the physical artifacts used in scientific work and not on the procedural steps through which knowledge is generated.

Content refers to the scientific topics, concepts or questions addressed during the learning activity. This dimension reflects how authentically students perceive the subject matter. Content is considered authentic when it relates to a real research topic in the discipline and has not been overly simplified or artificially constructed for instructional purposes (Sommer et al., 2018). This dimension differs from innovation in that it focuses on the relevance of the subject matter itself, not on the epistemic novelty or potential for generating new knowledge.

To clarify the conceptual refinements made to the original FEWAW instrument, it is necessary to specify how the present definitions differ from and expand upon those previously proposed by Finger et al. (2022).

In their version, materials were broadly defined as any non-instructional objects that students interact with, including objects of investigation (e.g., soil samples, texts) and research instruments (e.g., microscopes, indicator solutions). We mostly retained this two-part structure but made the criteria for authenticity more explicit by emphasizing the origin of the materials (i.e., from authentic scientific research) and their similarity in form, function and use to those typically found in scientific practice. Additionally, the definition was extended to include scientific visualizations and digital tools (e.g., simulations or laboratory software), which are widely used in science (Smetana and Bell, 2012).

Similarly, the original methods dimension referred to the scientific character of students' activities and their participation in systematic inquiry. While the core idea was preserved, concrete examples of disciplinary procedures (e.g., experimenting, modeling and data analysis) were added to reflect the methodological diversity of scientific practice and to help learners identify what constitutes authentic scientific work (Bolger et al., 2021).

The initial item pool was developed based on theoretically derived dimensions of perceived authenticity, using the FEWAW questionnaire by Finger et al. (2022) as the primary reference. Definitions for the dimensions location, instructor, and innovation, which showed empirical distinctiveness and acceptable mean values in the original study, were adopted with minor linguistic modifications for clarity and consistency. The original item sets for these three dimensions were retained to preserve continuity with the established scale structure. In contrast, new items were created for the methods, materials, and content dimensions according to the extended theoretical definitions. The content dimension, which was not included in the original instrument, was added to capture the perceived authenticity of scientific topics and disciplinary concepts (Betz et al., 2016).

To refine and validate the item pool, several iterative rounds of theoretical and linguistic review were conducted. A group of eight experts in science education, including professors, teachers, and doctoral students, as well as a group of five secondary school students, independently classified each item into one of the six dimensions of perceived authenticity. For this purpose, all participants were provided with the item pool and the corresponding dimensions. In addition to the classification task, students were asked to provide feedback on the linguistic comprehensibility of the items using an optional open-response format. Experts were additionally asked to evaluate item unidimensionality, that is, whether each item clearly reflected only one dimension or could plausibly be assigned to multiple dimensions. This qualitative feedback was incorporated into subsequent item revisions. Despite substantial inter-rater reliability (Fleiss' κ>0.85; Landis and Koch, 1977), several items required revision due to ambiguous or overlapping content.

Subsequent internal discussions focused on items that potentially reflected more than one dimension. For instance, several items in the materials and methods categories included implicit references to scientific instruments, also associated with the location dimension. To minimize anticipated side and cross-loadings, wording was refined to better represent the intended constructs. The conceptual distinction between content and innovation was also revisited, given the difficulty students may have in distinguishing novel scientific topics from those simply new to them.

Unlike the original FEWAW questionnaire, which used a response scale based on the perceived correctness of statements, the present version was designed to assess students' subjective perceptions, aligning with the interpretative nature of perceived authenticity. Most items were formulated in the first-person perspective and used active constructions to encourage participants to draw on personal experiences rather than evaluate externally verifiable facts. A five-point Likert scale was implemented, ranging from “strongly disagree (1)” to “strongly agree (5).”

To gather initial evidence for construct validity, selected external criteria theoretically related to specific aspects of perceived authenticity were included. Most validation scales were adapted from the original FEWAW study by Finger et al. (2022) with minor linguistic adjustments for stylistic consistency. These included measures of general authenticity (Damerau, 2012), authenticity of tasks (Nachtigall et al., 2018), perceived instructor expertise (Huber et al., 2009) and epistemological beliefs regarding the origin (Kremer, 2010) and changeability of knowledge (Moschner and Gruber, 2017).

Two additional short scales were included to reflect the extended model structure. To further validate the methods scale, an adapted version of a measure on participation in scientific practices based on the PISA framework and revised according to Chi et al. (2018) was used. It assesses engagement in core elements of scientific inquiry such as planning, conducting, and reflecting on experiments. For the materials scale, we included a scale adapted from Schüttler et al. (2021) capturing the perceived authenticity of laboratory equipment and scientific tools used during the program. A summary of all validation scales (N = 39 items) and descriptive statistics is presented in Table 1.

All statistical analyses were conducted in R (RStudio version 2025.05.0, Build 496) using psychometric packages including lavaan (Rosseel, 2012). As the data were collected using a pencil-and-paper format, responses were digitized independently by two researchers. To assess data entry accuracy, each researcher independently re-digitized a subset of 15 questionnaires originally entered by the other, resulting in a total of 30 double-coded questionnaires. This procedure yielded excellent inter-rater agreement (Cohen's κ = 0.998). Questionnaires with extensive missing data or implausible response patterns—such as uniformly selecting the same option across scales or other typical indicators of careless responding (Meade and Craig, 2012)—were excluded, resulting in a cleaned dataset of N = 130 valid cases. The full R script used for all data preprocessing steps and statistical analyses is provided as Supplementary material to this article.

Overall, 1.67% of item responses were missing and considered missing at random. Prior to model estimation, univariate and multivariate distributions were examined. Most items exhibited acceptable skewness and kurtosis, yet Mardia's test indicated significant multivariate skewness (b_1p = 851.72, p < 0.001) but no excess kurtosis (b_2p = 1838.57, p = 0.44), suggesting moderate deviations from normality (Finney and DiStefano, 2006). Therefore, all confirmatory factor analyses (CFA) were estimated using full information maximum likelihood (FIML) with a robust maximum likelihood estimator (MLR), which accounts for both non-normality and missing data (Little and Rubin, 2002).

The initial CFA model included all items from the six theoretically defined dimensions. Model evaluation was based on standard fit indices: Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), Root Mean Square Error of Approximation (RMSEA, 90% CI), Standardized Root Mean Square Residual (SRMR) and the chi-square to degrees of freedom ratio (χ²/df). Good model fit is typically indicated by CFI/TLI>0.95, SRMR < 0.08, RMSEA < 0.06 and χ²/df ≤ 2 (Hu and Bentler, 1999; Kline, 2016; Schermelleh-Engel et al., 2003).

To optimize model fit, an iterative item reduction procedure was applied, consistent with recommended and common practice in CFA. The three established dimensions—location, instructor, and innovation—were locked to preserve comparability with the original FEWAW instrument (Finger et al., 2022). Items of the remaining dimensions (methods, materials, content) were evaluated in multiple steps using established CFA thresholds (Brown, 2015; Kline, 2016): (1) strong cross-loadings as indicated by high modification indices (MI >10); (2) low standardized loadings (λ < 0.40); and (3) low explained variance (R² < 0.30). After each modification step, the model was re-estimated and fit indices recalculated until no further exclusions were warranted.

The resulting model was then conceptually inspected for content redundancy, theoretical representativeness and comparable scale lengths. Additional items were removed where appropriate to further improve model fit while maintaining adequate construct coverage.

Once satisfactory fit indices were achieved, internal consistency was evaluated using McDonald's ω as a measure of scale reliability under heterogeneous factor loadings. In addition, Cronbach's α was reported to ensure comparability with the original FEWAW instrument (Finger et al., 2022), where scale reliability was evaluated using this coefficient. Following common conventions, values of ω≥0.70 and α≥0.70 were considered indicative of acceptable reliability (Bortz and Döring, 2015; McDonald, 1999; Cheung et al., 2024).

Furthermore, discriminant and convergent validity were also assessed. Discriminant validity was examined by inspecting the latent inter-factor correlations among the FEWAW dimensions, with all correlations remaining below the recommended threshold of r = 0.80, indicating adequate separation of the constructs. Additionally, discriminant validity was assessed using the Heterotrait–Monotrait (HTMT) ratio of correlations, defined as the ratio of the average correlations between items of different constructs to the average correlations between items within the same construct. HTMT therefore reflects the extent to which two scales overlap in what they measure, with values at or below 0.85 indicating that the dimensions are empirically separable (Henseler et al., 2015; Cheung et al., 2024). Convergent validity was evaluated by correlating latent factor scores with external validation scales to confirm theoretically expected relationships (Brown, 2015). In addition, the Average Variance Extracted (AVE) was examined for each dimension, with values of at least 0.50 indicating that a latent construct explains more variance in its indicators than is due to measurement error (Fornell and Larcker, 1981).

Finally, to test group equivalence, measurement invariance was examined for gender and grade level using a stepwise comparison of configural, metric, and scalar models. Following Chen (2007), changes of ΔCFI ≤ 0.01 and ΔRMSEA ≤ 0.015 were used as invariance criteria.

To evaluate the robustness of the final six-factor structure, measurement invariance was tested across gender and grade level (see Section 2.3.5). A sequence of multiple-group CFA models was estimated using robust MLR estimation, comparing configural, metric, and scalar models following the recommendations by (Chen 2007). A post hoc power analysis was conducted using G*Power (Version 3.1.9.7) to assess the statistical sensitivity of the test (Faul et al., 2009). Based on the sample size of N = 130 and assuming a medium effect size (w = 0.30), the analysis indicated sufficient statistical power (1−β = 0.87) to detect deviations from measurement equivalence in two groups. Model comparisons were based on changes in CFI and RMSEA, complemented by the Satorra–Bentler scaled χ² difference test (Satorra and Bentler, 2001). The results are summarized in Table 7.

Across both gender and grade level, changes in fit indices were negligible (ΔCFI ≤ 0.01, ΔRMSEA ≤ 0.015) and all Satorra–Bentler χ² difference tests were non-significant (p>0.20). These results indicate that the six-factor model is scalar invariant across groups.

Building on these results, we conducted group comparisons of observed scale means using non-parametric Mann–Whitney U-tests (Bortz and Döring, 2015). Across all six dimensions, no statistically significant differences were found between male and female students. Holm-adjusted p-values ranged from p_Holm = 0.194 to p_Holm = 1.000, with effect sizes ranging from r = −0.102 to r = 0.114, indicating negligible group differences (Holm, 1979). Comparisons between students of different grades yielded a similar pattern. Although descriptive medians suggested slightly higher authenticity ratings for lower grade students in the methods, materials, and location dimensions, none of these differences reached statistical significance after Holm correction (p_Holm = 0.450–1.000). The corresponding effect sizes were consistently small, with values ranging from r = −0.130 to r = 0.171. These findings suggest that, beyond measurement equivalence, perceptions of authenticity were consistent across gender and grade level in this sample.

The final 21-item version achieved strong psychometric properties while maintaining broad construct coverage. The trimming strategy combined (a) statistical thresholds (factor loadings, item-level explained variance, and modification indices) with (b) conceptual trimming to remove redundancy and harmonize scale length. This two-step approach is consistent with recommendations in the measurement literature (Brown, 2015; Kline, 2016; Bortz and Döring, 2015) and meets demands in time-constrained environments such as out-of-school labs, where additional measurement instruments are often administered.

Internal consistencies of the six scales ranged from ω = 0.64 to 0.87 (see Table 4). Five of the six dimensions exceeded the commonly recommended threshold of ω≥0.70, indicating solid reliability even with the reduced number of items. Cronbach's α values showed a comparable pattern, ranging from α = 0.63 to 0.87, with five of the six dimensions exceeding the commonly recommended threshold of α≥0.70. The highest reliability was observed for materials (ω = 0.87, α = 0.87), followed by instructor, location, and content (all ω≥0.82, α≥0.82). In comparison to Finger et al. (2022), who reported consistently high reliability values across all scales (α≥0.81), the present results are largely comparable—with the exception of the innovation scale, which showed a noticeably lower internal consistency (ω = 0.64, α = 0.63).

This result may be partially explained by the fact that the innovation scale, along with location and instructor, was retained without revision. Since these scales were locked for comparability, they could not be optimized or improved in the same way as the newly extended dimensions. We return to the consequences of this decision, especially with regard to the innovation dimension, in Section 4.2.3.

Inter-factor correlations for all factors remained below the conventional threshold of r = 0.80, generally indicating that the six dimensions capture distinct aspects of perceived authenticity. However, the magnitude of the intercorrelations between several dimensions, notably content and materials (r = 0.67) and materials and methods (r = 0.68), should be reviewed critically. High inter-factor correlations may raise questions regarding the discriminant validity of these dimensions and signal the potential existence of an overlying, higher-order factor. The association between materials and methods, for example, is theoretically coherent as disciplinary authenticity involves both access to the tools and engagement with the practices of the field. This overlap suggests that learners can and do differentiate between what they used and how they used it, yet the resulting correlations indicate that these facets are perceived as closely connected. Similarly, the strong link between content and materials may reflect the fact that authentic scientific content is typically embedded within the artifacts through which it is explored.

These patterns are reflected in the HTMT ratio of correlations, which provides a stricter indication of conceptual distinctiveness. Most HTMT values remained below the recommended threshold of 0.85 (Henseler et al., 2015; Cheung et al., 2024), supporting the separability of the majority of dimensions. However, elevated values were observed for the pairs content–materials (HTMT = 0.79) and materials–methods (HTMT = 0.83), indicating particularly close empirical associations. This suggests that while the dimensions are conceptually defined as distinct, they may not be fully separated in learners' perceptions. One plausible explanation is the inherent structure of the learning activities in our sample. The tasks were designed such that the materials followed directly from the content and the methods naturally emerged from the available materials. For instance, activities on robotics (content) involved working with the robot itself (materials) and required the students to program (methods). This built-in sequencing may have made it difficult for learners to perceive these dimensions as clearly distinct from one another.

Future research should therefore investigate a hierarchical factor model to determine whether these dimensions are best represented as distinct yet strongly associated facets or as expressions of a broader, underlying construct. Despite these challenges, the intercorrelations and HTMT values remained below conventional thresholds, providing evidence for adequate empirical separability. The refined wording and the clear conceptual distinction between dimensions likely contributed to the overall acceptable statistical separation.

The analyses of the correlations between FEWAW scales and external validation measures support the convergent validity of most dimensions. In particular, the scales materials, content, and methods showed the strongest associations with constructs that are theoretically aligned with key components of perceived authenticity such as general and task-specific authenticity, participation in scientific practices, and perceptions of lab equipment (e.g. Damerau, 2012; Nachtigall et al., 2018; Schüttler et al., 2021; Chi et al., 2018). These findings suggest that learners' authenticity judgments are closely linked to material and procedural features of the learning environment, which appear to serve as salient cues for authenticity perception (Schüttler et al., 2021). This interpretation aligns with earlier research indicating that authenticity is often anchored in tangible characteristics of science education settings (Braund and Reiss, 2006; Roth, 1995).

Among these, materials and content stood out as the dimensions with the most pronounced and consistently high correlations across multiple external measures, suggesting that these two dimensions play a particularly central role in shaping students' authenticity perceptions. From a theoretical perspective, both dimensions reflect foundational elements of scientific practice, namely the physical artifacts and the substantive focus of scientific inquiry.

The location and instructor scales were also positively associated with several external constructs, though their relationships were less pronounced. Both dimensions correlated moderately with general authenticity and perceived authenticity of lab equipment, while location additionally related to participation in scientific practices. These results point to a secondary role of spatial and interpersonal aspects in shaping perceived authenticity, possibly reflecting that these features are more context-sensitive and subject to variation in learners' interpretations.

In contrast, the innovation scale displayed lower correlations with most external validation measures. Although it was modestly associated with task-related authenticity and epistemological beliefs about the origin of knowledge, it showed no meaningful correlation with other constructs we measured. This pattern is consistent with the relatively low AVE_AUIN = 0.386, indicating weaker convergence of its items and providing an internal validity-based explanation for the selective and generally weaker external associations. Importantly, the weaker empirical performance reflects a deeper conceptual issue that arises when operationalizing innovation in formal learning contexts. While the present data does not allow for definitive conclusions about the underlying causes, a few considerations may help interpret the observed pattern.

Authenticity in formal learning settings typically involves a simulation of scientific practice rather than direct engagement in authentic research (Betz et al., 2016). Students' perceptions of authenticity may therefore be bounded by their role expectations in school-based learning, as many see themselves as exploring existing ideas rather than generating new knowledge. Unlike the other dimensions, which are anchored in tangible cues such as physical materials, tools, spatial settings, or even instructors, innovation relies on learners' subjective assessment of epistemic novelty. Such assessments draw on beliefs about who is entitled to create new scientific knowledge, making the construct inherently more complex than dimensions triggered directly by environmental features. This asymmetry is also reflected in the current item formulations, which adopt an “objective” perspective on scientific discovery (e.g., “I tried to discover something new for science,” “I contributed to answering a current research question”). These formulations implicitly assume a professional scientific role and therefore may exceed what students consider realistic within a classroom or an outreach setting. This interpretation aligns with prior research emphasizing that authenticity or student perceptions on epistemological beliefs depend not only on environmental features but also on learners' epistemic role expectations (Sandoval, 2005; Stroupe, 2014; Nachtigall et al., 2024). In this view, innovation can be considered an outcome of how learners position themselves within these contexts.

The selective correlation with epistemological beliefs about the origin of knowledge further supports this explanation. Learners who have a broader view of who can generate scientific knowledge appear more likely to perceive their own activities as innovative. Conversely, students who believe that only professional scientists produce new knowledge may experience a mismatch between item wording and their perceived role, resulting in overall lower scores. This pattern is consistent with research showing that many students differentiate sharply between school science and professional science and attribute knowledge generation primarily to scientists rather than learners (Voitle et al., 2022). The absence of a corresponding relationship with beliefs about the changeability of knowledge suggests the importance of learners' assumptions about who is allowed to generate new ideas. Taken together, these findings indicate that the subjective experience of innovation is constrained by role-based plausibility boundaries. Students may feel they can explore or test ideas, but not “discover something new for science.”

From a psychometric perspective, this conceptual misalignment helps explain the low AVE for the innovation scale and the lack of meaningful correlations with most external validation measures. It also clarifies why this dimension has shown weak performance in Finger et al. (2022). To improve both theoretical coherence and empirical functioning, future iterations of the scale may benefit from shifting toward a more learner-centered conceptualization of epistemic novelty, emphasizing personal discovery (“I explored something new to me”) and idea generation (“I was able to generate and test my own ideas”) rather than contributions to professional scientific knowledge. Such formulations would align more closely with authenticity as a subjective experience and be more consistent with the role expectations typically held by students in structured learning contexts.

Furthermore, the unexpectedly weak association between instructor and external ratings of instructor expertise should be noted. A plausible explanation lies in the specific wording of the items. The FEWAW scale refers to instructors as “real scientists,” invoking authenticity markers rooted in scientific identity and professional status. By contrast, the external validation scale refers to “our tutors” and emphasizes perceived competence rather than professional affiliation. Particularly in outreach contexts such as student labs, where instruction is often provided by university students or early-career staff, learners may recognize expertise without perceiving the instructors as authentic representatives of science.

The analysis provided initial evidence for scalar invariance across gender and grade level, suggesting that the six-factor structure functions equivalently across these groups. This indicates that the instrument allows for unbiased mean comparisons between male and female students as well as across grade levels.

Nevertheless, these findings must be viewed as preliminary. Given the small sample size (N = 130), the absence of significant differences does not conclusively confirm full invariance. Minor variations in the pattern of loadings or intercepts might remain undetected in small samples. Furthermore, as often observed in small-to-moderate samples, a slight deterioration of overall model fit occurred during the stepwise invariance testing. This pattern is well-documented as a sample-size-sensitive artifact rather than a substantive refutation of invariance (Chen, 2007). Accordingly, future research should validate the model structure with larger and more diverse datasets encompassing different school types, age groups, and disciplinary contexts. This could help assess the robustness of the FEWAW measurement model and to determine whether the observed factorial stability generalizes across sub-populations.