Reliable and accurate measurement is the cornerstone of scientific progress. The ability to define and, with validity, quantify phenomena consistently underpins the development of theories, the testing of hypotheses (Michell, 1997), and the application of findings to real-world challenges (Pendrill, 2018). Although not widely discussed in the field of psychological research, there has been a long-standing critique of the dominant measurement practices (Michell, 1997) with increased attention in recent years (Elson et al., 2023; Flake and Fried, 2020; Johansson et al., 2023).

Current practices often ignore issues regarding measurement (Flake et al., 2022), as exemplified by Lilienfeld and Strother (2020), seldom motivate the validity of instruments, and often rely on psychometric evaluations based on small samples (Elson et al., 2023). Furthermore, psychometric evaluations often rely on sum scores (McNeish and Wolf, 2020b) using Cronbach’s alpha (McNeish, 2018) to assess scale properties, an approach that has faced substantial methodological critique (McNeish, 2022). There is no widespread consensus on how to assess the psychometric quality; however, four key psychometric properties have been suggested as a minimal framework for guiding psychometric analyses: unidimensionality, local independence of items, ordering of response categories, and invariance (Christensen et al., 2013; Johansson et al., 2023; Kreiner, 2007).

Although these properties can be evaluated through various methods, Rasch Measurement Theory (RMT) offers the distinct advantage of treating the ordinal sum score as a sufficient statistic for measurement (Andrich and Marais, 2019). Rasch analysis models the probability of a response to an item based on a person’s ability or trait (β) and an item’s difficulty ( δ ), and the formulation of the model suggests that these two parameters can be separated, known as “specific objectivity.” Figure 1 shows the modeled probability of a correct response to a dichotomous item (Equation 1) with difficulty δ = 0 , depending on the varying level of a person’s latent trait β :

Equation 1 is simplified in terms of notation and subscripts, as shown by Andrich and Marais (2019). Figure 1 illustrates that as a person’s trait increases, the probability of a correct response also increases. When the item difficulty (set to 0 in the figure) equals a person’s latent score β = δ = 0 , the probability of a correct answer is 50% (see dashed lines in Figure 1).

As a person’s latent trait increases further from the item difficulty, β > δ , the probability of a correct answer increases toward 100%, and vice versa. Latent traits and item difficulties are both expressed in logit units (log-odds) with an arbitrary center or reference point (such as 0). In clinical psychology, a typical situation exemplified by Figure 1 could be a dichotomous item that indicates the presence of suicidal ideation. This question could be answered as yes or no with a numerical representation of 1 or 0, respectively. Such an item could be included in a questionnaire to measure the trait of depression; as this inferred trait increases (a more depressed patient), the probability of endorsing “yes” to suicidal ideation also increases. In short, the difficulty or δ of such an item would indicate that those with more of the latent trait of depression would indicate the presence (answer yes) of suicidal ideation. Specific objectivity essentially means that differences between items’ difficulties ( δ ) can be assessed independently of the current sample of respondents providing answers. Similarly, differences between persons ( β ) can be assessed independently of the difficulties of the items. When a set of items fulfills the criteria previously listed, Rasch analysis allows the transformation of an ordinal raw sum score into an interval-level latent trait score (latent score) for each person, with specific measurement error at each level of the scale. This represents an individual’s latent trait regarding what is being measured. In this article, a trait or latent score is the inferred amount of what is being measured, not an inherent trait.

In prediction research using self-rated symptom measures, as in psychological research, limited attention has been paid to measurement practices. Clinical prediction models for psychological treatment have the potential to improve treatment outcomes (Barkham et al., 2023; Forsell et al., 2019; Hentati Isacsson et al., 2024a). Therefore, improving these measures could have a significant impact on patients. There are several experiments investigating the impact of measurement error in prediction research involving self-rated symptoms. Jacobucci and Grimm (2020) showed using simulations that a predictor’s reliability can heavily influence the prediction performance (Jacobucci and Grimm, 2020), and medical prediction models have been shown to be less valid as a function of increased measurement error in predictors (Luijken et al., 2019). McNamara et al. (2022) took this one step further and simulated both measurement error in predictors and varying degrees of outcome misspecification (McNamara et al., 2022). The result was that the underlying non-linear relationship was not identified by non-linear models (e.g., random forest) or regression models, and non-linear models had indistinguishable performance. Furthermore, an attempt to attenuate measurement problems by increasing the sample size from 4,000 to 100,000 yielded negligible improvements in predictive performance. Thus, when the measurement error is high, the predictive performance plateaus at a low level compared to models using more reliable variables (McNamara et al., 2022), indicating heavy influence of reliability for both predictors and outcomes. This does not consider the validity of measurement. As such, it is perhaps surprising that the recently updated TRIPOD+AI statement for research on clinical prediction models does not mention measurement practices in relation to predictors or outcomes, except that it should be noted “…how and when they (predictors/outcomes) were measured…” (Collins et al., 2024). This is especially true in clinical psychology, where measurement error is an issue. Although using psychometrically sound measures is important and measurement reliability significantly affects predictions, there is limited empirical evidence that psychometrically refined measures offer the predictive advantages over summed ordinal scores based on a set of items that have not been subject to an adequate psychometric assessment. However, a few studies indicate that a measure validated using RMT can outperform a traditional measure in ROC/AUC classification performance, although these were for pregnancy tests and intensive care unit admissions (Fisher and Burton, 2010; Pendrill et al., 2023).

Recently, there has been an encompassing debate about the use of sum scores in psychological research (McNeish, 2024; McNeish and Wolf, 2020b; Sijtsma et al., 2024; Widaman and Revelle, 2023). In a simulation setting, for certain conditions, the sum score can correlate to a stronger degree with the true underlying latent score than the estimated latent score itself (Sijtsma et al., 2024). Furthermore, Sijtsma et al. (2024) showed using a simulation that the sum score can adequately represent the score from each item (similar to an estimated latent score). Thus, while the sum score can perform adequately both in a predictive context and for inference (Sijtsma et al., 2024), its current use of the sum score is seldom motivated properly (McNeish, 2024) in relation to psychometric properties such as dimensionality, validity, and invariance assessments. Therefore, the naive use of the sum score has a high risk for bias. However, when the sum score shows stochastic ordering, the latent score and sum can have similar performances in predicting an external variable (Sijtsma et al., 2024). Stochastic ordering means that, as the sum score increases, the latent variable also increases when conditioned on the sum score. While sum scores have pragmatic uses, a more psychometrically sound latent score is expected to improve predictive performance based on previous simulation studies. At the same time, empirical research on measuring change in psychometrically evaluated standardized tests of preschool children shows that the latent and ordinal sum scores show marginal differences (Bezruczko et al., 2016). Thus, the overall advantage of using latent scores over sum scores in an empirical predictive framework remains largely unexplored.

This is a prospective prediction study using longitudinal observational data from a regular care clinic providing therapist-guided psychological treatment. Ethical approval was received from the regional ethical review board in Stockholm (Dnr: 2011/2091–31/3, amendment 2016/21–32, 2017/2320–32, and 2018/2550–32). The supplement contains the results data, code for analysis, and further details of the methods.

The differences between the Base and Rasch datasets were very small, with less than a 0.0081-point difference in the RMSE score for each model in favor of the Base dataset (Figure 3). Thus, the Rasch dataset models had only a marginally worse score in RMSE, 5% (0.1389/0.1318) higher compared to the Base dataset models. For LR, the Base dataset had an RMSE of 0.1318 (95% CI, 0.1284, 0.1353), and Rasch had an RMSE of 0.1389 (95% CI, 0.1359, 0.1418) with a mean difference = −0.0070, t(198.57) = −3.09, p = 0.0023. For BR, the Base dataset had an RMSE of 0.1318 (95% CI, 0.1284, 0.1352), and Rasch had an RMSE of 0.1388 (95% CI, 0.1358, 0.1418), with a mean difference = −0.0070, t(205.44) = −3.04, p = 0.0027. For RF, Base had an RMSE of 0.1322 (95% CI, 0.1287, 0.1357), and Rasch had 0.1403 (95% CI, 0.1374, 0.1432), with a mean difference = −0.0081, t(218.02) = −3.51, p = 0.0006. All models were significantly better than the null model of the dummy regression.

Using the standard errors as weights did not improve the predictive performance of the Rasch models, having an identical performance for each of the models (Figure 3).

While the original questionnaires were less psychometrically robust (Hentati Isacsson and Johansson, 2025) than the shortened versions based on Rasch Measurement Theory, they maintained strong stochastic ordering, with higher sum scores reliably indicating (correlated) higher latent scores, as evidenced by correlations of 0.90–0.95. Table 1 shows the superior fit of the one-factor CFA for the shortened version of each questionnaire based on the RMT. We note that while some minor misspecifications also exist for the shortened version, it is always superior to the original version of the questionnaire, and the fit metrics for the shorter questionnaires had a much larger margin than the cutoffs. Thus, this CFA supports the unidimensional quality of the reformulated questionnaires but not the original versions.

Latent scores calculated using the psychometrically optimized, substantially shorter version, comprising 23% of the full scale, showed similar predictive performance (although slightly and negligibly worse) compared to the sum score of the full scale. Overall, the original ordinal sum scores had a marginally better performance, and there were no differences in performance across models. Jacobucci and Grimm (2020) showed that varying only predictor reliability, and no other psychometric criteria, had a large impact on predictive performance, with underfitted models as a function of decreased reliability, and they did not simultaneously modify the outcome. McNamara et al. (2022), who modified both predictors and outcome at the same time, showed that there was no difference between predictive models, as was found in our results. At the same time, they showed that less reliable predictors and outcomes showed decreased predictive performance. Again, this result pertains only to modifying the noise or reliability of the predictors and outcome. Meanwhile, this study was reformulated according to four psychometric criteria using empirical data, and we do not have the same reliability estimates or control as in the simulation studies (Jacobucci and Grimm, 2020; McNamara et al., 2022).

As shown in Figure 2, the reformulated questionnaires’ latent scores have a largely simple linear relationship with the original ordinal sums. While the reformulated questionnaires are more psychometrically robust and retain essential information, the original sum scores could contain useful predictive signals despite a greater amount of noise. The results are in line with those of Sijtsma et al. (2024), which indicate similar performance between sum scores and latent scores in a predictive framework. This suggests that while the reformulated questionnaires provide a more streamlined, unidimensional measure of the trait, as shown by the CFA, some variability present in the original scores may contribute to predictive performance. This is supported by the fact that the original sum scores show stochastic ordering with largely high correlations to the (what we assume to be) underlying latent trait, as estimated using Rasch analysis. While it is not perfectly linear at the periphery of the latent continuum (Figure 2), which a non-linear model such as random forest could detect, it remains largely linear overall, reminiscent of the empirical findings of Bezruczko et al. (2016). Consequently, because the format of the outcome variable in each case aligns with the format of the predictors, the predicted relationships remain largely linear and unchanged. Thus, based on both the experimental setup in this study and previous studies, it is not entirely unexpected that the predictive performance is similar.

However, we did not interpret the higher predictive performance of the sum score as reflective of a more truthful way of handling self-rated data. As clearly argued for and shown in the study by Shmueli (2010), predictive results cannot be used to draw inferences about which models are more truthful, but rather about which could be more predictive. Another possibility is that the Rasch-improved models improved precision and reduced overfitting. This resulted in a lower predictive performance but could be more generalizable (e.g., in overpredictive or other psychiatric settings) compared to the original sum scores. This hypothesis would need to be tested with other datasets from different settings.

Furthermore, while there was a statistically significant difference in the predictive performance, the absolute difference in the RMSE score was negligible. In addition, the latent scores used only a fraction of the original questionnaire items. This was an unintended consequence of the psychometrically valid versions of the questionnaires. The latent scores’ performance was worse by 5% relative to the sum scores, corresponding to an RMSE difference of 0.70%, which is a considerable retention of information considering that only 23% of the items were retained across all questionnaires. Fewer items could be beneficial because it could significantly reduce the response burden for patients and reduce the risk of missing data, which in turn could facilitate repeated and more widespread measurements within routine care.

While we also trained models that weighted their predictions based on the inverse sum of the standard errors of the latent scores, the information from this weight did not improve the prediction compared with not using the weights. While not explicitly containing the weight variable in the non-weighted models, the non-weighted models did have access to the standard errors, which composed the weights, and these possibly had a larger influence than the composite. Future studies could explore other ways to utilize the standard errors, perhaps using another predictive framework that implements the standard error explicitly in the model (Kurz, 2023).

Since both the predictor and outcome changed simultaneously in the two conditions, a future study might investigate an independent set of predictors to predict the two different types of outcomes instead. This could possibly reveal whether a more psychometrically solid outcome variable could improve predictive performance. In addition, a future study could investigate if latent scores from the original questionnaires, without improving their psychometric qualities, have an impact on predictive performance. Furthermore, previous research has indicated improved predictive performance using items as predictors, as opposed to their summation (McNeish, 2024). While this is counter to our previous research in a similar setting (Hentati Isacsson et al., 2024a), it could be valuable in a setting that also investigates latent scores compared to a simple summation. The similar predictive performance of the latent score setup, despite using only 23% of the items, is beneficial. This study did not aim to investigate methods of shortening questionnaires and their subsequent impact on predictive performance. To investigate these aspects, another methodological setup would be needed, e.g., using the same subset of items. Such a study could also incorporate the possible impact of missing data on the analyses, such as comparisons of the impact of missing data, where one could simulate missing data and compare complete-case analyses and imputation setups. Additionally, this experimental setup could benefit from longitudinal models that take the repeated structure of the data into account, despite previous findings. Finally, there has been recent progress in predictive models with measurement errors, which indicates that it could be beneficial to predict intervals and use these instead of point predictions (Jiang and Ma, 2024).

In conclusion, using empirical data from psychological treatment, our findings indicate that using latent scores as predictors and outcomes from a psychometrically improved version of the questionnaire showed similar predictive performance to the original ordinal sum scores. While the psychometric properties were improved by the Rasch analyses, it is inconclusive whether this also improved precision and reduced overfitting or if the Base dataset retained useful variability. For the models using the latent score, their predictive performance was marginally reduced by 5% (a 0.70% RMSE increase) while using only 23% of the original items. This suggests that while reformulated questionnaires can streamline measurement and lower the burden on patients, their impact on improving predictive performance in this study was limited.