Method CA is based on the CA formula (Equation 1) from classical test theory (Nunnally, 1967):

where ρ x y is the true correlation between the constructs x and y, r x y is the observed correlation between the two measures, and r x x and r y y are the reliabilities of the two measures. This formula can be used to relate two measures of the same construct, rather than different constructs (Wanous et al., 1997). If two instruments measure the same construct, then the true correlation should be 1.0; replacing ρ x y with 1, using the observed correlation between the single-item assessment and a multi-item assessment of the same construct ( r x y ), and the reliability of the multi-item assessment, it is possible to solve for the reliability of the single-item assessment.

Method CA is the most commonly used method for estimating the reliability of single-item assessments (Zijlmans et al., 2018). However, in practice, the true correlation between a single-item assessment and a multi-item assessment of the same construct ( ρ x y ) is usually less than 1.0, which may lead to an underestimation of the reliability of the single-item assessment (Christmann and Aelst, 2006).

Method FA is based on the context of FA where the variance of an item is equal to its communality, specificity, and unreliability, and the reliability of an item is the sum of its communality and specificity (Bailey and Guertin, 1970; Harman, 1976). By conservatively assuming that specificity is zero, the minimum reliability of a single item can thus be estimated by its communality (Ginns and Barrie, 2004). Principal axis factoring is performed on the set of items including those from the multi-item assessment and the single-item assessment of the same construct. Using this technique to estimate reliability for single-item assessments results in underestimation (Arvey et al., 1992; Ginns and Barrie, 2004; Wanous and Hudy, 2001).

Method DMM is based on the DMM of Molenaar and Sijtsma (1988). Let us assume that a scale has N items (N > 1), where i and j denote items in the scale. Let us further assume that each of these N items has m + 1 item scores, e.g., if these N items are dichotomous, the item scores can be 0 and 1 (m = 1); if these N items use a 5-point Likert scale, the item scores can be 0, 1, 2, 3, and 4 (m = 4). The notation x denotes the item i’s score, and y denotes the item j’s score (x = 0, …, m; y = 0, …, m). π_x(i) = P(X_i ≥ x) denotes the marginal cumulative probability of getting at least x on item i, π_y(j) = P(X_j ≥ y) denotes the marginal cumulative probability of getting at least y on item j. Obviously, π_0(i) = 1 and π_0(j) = 1. π_x(i)y(j) = P(X_i ≥ x, X_j ≥ y) denotes the joint cumulative probability of obtaining at least x on item i and at least y on item j.

If we test item i twice independently in the same group of subjects, denoted by i and i’, then π_x(i)y(i’) represents the joint cumulative probability of getting at least x on the first test and at least y on the second test. In practice, however, we cannot test an item twice independently in the same group of subjects because of the practice effect, so we have to estimate π_x(i)y(i’) from a single test.

Reliability is defined as Equation 2

Molenaar and Sijtsma (1988) proved that the true score variance (σ²_T) can be expressed as the sum of the differences between the joint cumulative probability and the product of the marginal cumulative probabilities (Equation 3):

Then reliability can be expressed as

Equation 4 can be further decomposed into two parts, in one part, i ≠ j, in another part, i = j (i.e., j = i’):

Equation 5 can be adapted to estimate the reliability of a single item in a multi-item assessment (Zijlmans et al., 2018), where the first ratio and the first summation sign in the second ratio disappear, the reliability of an item is:

In Equation 6, in addition to π_x(i)y(i’), other terms can be observed or calculated from a single test.

Molenaar and Sijtsma (1988) developed method DMM for estimating π_x(i)y(i’) in Equation 6, which they described using an artificial 4-item assessment, where each item has three ordered categories (i.e., item scores can be 0, 1, 2). Table 1 shows the marginal cumulative probabilities for 4 items.

We rank all the marginal cumulative probabilities in Table 1 from small to large:

Then construct a matrix (Table 2) of joint cumulative probabilities in which the rows and columns are ordered by the size of the corresponding marginal cumulative probabilities, where NA indicates that a joint cumulative probability is unobservable and must be estimated. For convenience, π_x(1)y(i’) is in the cell or row r and column c; π_x(1)y(i’) is denoted P_r,c, and the corresponding marginal cumulative probabilities are P_r and P_c, respectively.

To calculate NA in Table 2, we define four types of joint cumulative probabilities: (1) the lower neighboring joint cumulative probability: P_lo = P_{r + 1, c}; (2) the right neighboring joint cumulative probability: P_ri = P_r,c+; (3) the upper neighboring joint cumulative probability: P_up = P_r-1,c; and (4) the left neighboring joint cumulative probability: P_le = P_{r, c-1}. Not all four neighboring joint cumulative probabilities exist for each NA, e.g., for P_1,5, P_up does not exist, P_lo = 0.3, P_le = 0.2, and P_ri = 0.2.

Molenaar and Sijtsma (1988) estimate π_x(i)y(i’) (i.e., P_r,c) eight times using the following eight different equations:

P_r,c is estimated as the mean of eight estimates in Equations 7–14. For example, using Equations 7–14 to estimate π_2(4)1(4′) (i.e., P_1,5) in Table 2, π_2(4)1(4′) = 0.21. However, Molenaar and Sijtsma (1988) pointed out that P_r,c should be in the interval (P_rP_c, min(P_r, P_c)); for π_2(4)1(4′), the lower bound is 0.2 × 0.6 = 0.12 and the upper bound is 0.2, so π_2(4)1(4′) = 0.2.

Now that π_x(i)y(i’) can be estimated, we can use Equation 6 to estimate the reliability of an item in a multi-item scale. To estimate the reliability of a single-item assessment, researchers combine a single-item assessment and a multi-item assessment of the same construct into one scale.

Method DMM is based on the double monotonicity model. This model has two assumptions, first, that the multi-item scale is unidimensional; and second, that there is no intersection of the response functions between different items (Sijtsma and Molenaar, 2002). Therefore, to use method DMM to estimate the reliability of a single-item assessment, the corresponding multi-item assessment should be unidimensional and have non-intersecting item response functions.

Method λ₆ is based on Guttman’s λ₆ (Guttman, 1945) to estimate π_x(i)y(i’) in Equation 6. Let us assume that one scale has N items (N > 1). We ran a regression to predict the item score X_i with the remaining N-1 item scores (i = 1, …, N), where ε²_i denotes the variance of the residual error of the regression. Guttman (1945) defined λ₆ as the lower limit of the reliability of a multi-item scale:

To calculate ε²_i, let us use Σ_ii to represent the (N-1) × (N-1) variance–covariance matrix for N-1 items other than item i. σ_i denotes the (N-1) × 1 vector containing the covariances of item i with the other N-1 items. Jackson and Agunwamba (1977) verified that ε²_i can be expressed as:

To estimate the reliability of an item in a multi-item scale, we insert Equation 16 into Equation 15:

Since λ₆ is the lower limit of reliability, Equation 17 can be approximated to Equation 18 by Equation 6 (Zijlmans et al., 2018):

π_x(i)y(i’) can be expressed as:

By inserting Equation 19 into Equation 6, we can calculate the reliability of an item in a multi-item scale. To estimate the reliability of a single-item assessment, we again combine a single-item assessment and a multi-item assessment of the same construct into one scale.

Since λ₆ is the lower limit of reliability, method λ₆ will underestimate the reliability of single-item assessments in most cases.

Method LCM uses the latent class model to estimate π_x(i)y(i’) in Equation 6 (McCutcheon, 1987; McCutcheon et al., 2002). Let us assume that a group of subjects take an N-item survey, each of these N items having m + 1 item scores. Suppose further that there is a latent categorical variable ξ that accounts for the relationship between the N items, the latent variable ξ has Q latent classes. McCutcheon et al. (2002) defined the latent class model as:

where P(X₁ = x₁, …, X_N = x_N) is the joint probability distribution of the N items, P(ξ = q) is the probability that a randomly selected subject is in latent class q of latent variable ξ, P(X_i = x_i| ξ = q) is the conditional probability of a particular item score given class q.

The latent class model in Equation 20 asserts that items are conditionally independent given a particular class in ξ (Goodman, 2002). Conditional independence means that when the latent variable ξ that influences subjects’ responses to items is held constant, subjects’ responses to any two items are independent.

Zijlmans et al. (2018) restricted Equation 20 to only one item in a multi-item scale to estimate π_x(i)y(i’):

Method LCM inserts Equation 21 into Equation 6 to estimate the reliability of an item in a multi-item scale. To estimate the reliability of a single-item assessment, we combine a single-item assessment and a multi-item assessment of the same construct into one scale.

Method LCM will accurately estimate the reliability of single-item assessments only if Q latent classes are accurately selected and P(ξ = q), P(X_i = u|ξ = q), P(X_i = v|ξ = q) in Equation 21 are equal to the population parameter (McCutcheon et al., 2002). These two assumptions are difficult to meet in reality, so method LCM may often misestimate the reliability of single-item assessments.

Of the five methods for estimating the reliability of single-item assessment, researchers do not know which is the best. The most commonly used method is method CA, which assumes that the true correlation between single-item assessments and their corresponding multi-item test is 1. In practice, the true correlation is unlikely to be 1, so method CA often underestimates the reliability of single-item assessments (Krieglstein et al., 2022). Method FA uses the communality of a single item as a conservative estimate of the reliability of single-item assessments, while many studies of single-item assessments have shown that the results via method FA are not much lower than those via method CA (Buchner et al., 2024; Dolan et al., 2015; Leung and Xu, 2013; McDonald et al., 2019). Given that λ₆ is the lower bound of reliability, method λ₆ will underestimate the reliability of single-item assessments in most cases. Method DMM and method LCM are based on assumptions that are often violated.

This study plans to conduct a simulation study to investigate which method most accurately estimates the reliability of single-item assessment. The simulation study will generate scores from single-item assessments and item scores from multi-item scales. As Likert scales are the most popular format used in scale design (Foddy, 1994), all simulated items will be polytomous. Given that single-item assessments are typically designed to estimate unidimensional constructs, the multi-item scale in this study will be unidimensional. To fully represent different types of multi-item scales in reality, the multi-item scales in this study will vary in two aspects, test length and discrimination parameters. For the simulated datasets in each condition, this study will compare the median bias, IQR, RMSE, and percentage of outliers (Zijlmans et al., 2018) produced by method CA, method FA, method DMM, method λ₆, and method LCM for the single-item assessments.

Simulation and statistical analyses were performed using R, version 4.3.2, graphs were generated using the package “ggplot2,” and scripts were uploaded as Supplementary material. Method CA, method FA, method DMM, method λ₆, and method LCM all use a single-item assessment and a multi-item assessment of the same construct to estimate the reliability of the single-item assessment, so in the simulation study, the last item of each simulation dataset is the single-item assessment, other items construct the multi-item assessment of the same construct. For example, a simulation dataset contains scores for 7 items, the first 6 items are items from the multi-item assessment, and the last item is a single-item assessment.

Because single-item assessments are designed to measure unidimensional or global constructs, researchers usually use multiple single-item assessments to measure multidimensional constructs (one item for one dimension), so unidimensional multi-item assessments were used in this simulation study. After reviewing approximately 100 articles on single-item assessments, it was found that all single-item assessments measuring psychological constructs were polytomous, most of which used a 5-point Likert scale. Joshi et al. (2015) found that the 5-point Likert scale was the most commonly used and that 7-point or 10-point Likert scales did not outperform the 5-point Likert scale in terms of psychometric properties (Colvin et al., 2020; Jebb et al., 2021). Therefore, all single-item assessments and multi-item assessments of the same construct in this simulation study would use a 5-point Likert scale.

All polytomous item scores were generated using the multidimensional graded response model (Penfield, 2014):

where θ = (θ₁, …, θ_Q) represents the Q-dimensional latent variable vector which has a Q-variate standard normal distribution, P(X_i ≥ x|θ) denotes the probability that the item score is greater than or equal to x for a given value of θ for item i, a_iq is the discrimination parameter of item i, relative to the latent variable q, b_ix is the difficulty parameter for categorical x (x = 1, 2, 3, 4) of item i.

To examine the precision of the five methods for estimating the reliability of single-item assessments, we first need to know the true reliability of single-item assessments (ρ_ii). For single-item assessments, the simulation study generated one million sets of item scores and then we used the variance based on the latent variable vectors (θs) of the 1 million sets of item scores to divide by the variance of the single-item score to obtain the true single-item assessment’s reliability (Feinberg and Rubright, 2016; Zijlmans et al., 2018). As shown in Table 3, some simulation conditions differ only in sample size (e.g., condition 1 and condition 19, condition 2 and condition 20); the true single-item assessments’ reliabilities in these conditions are the same given the calculation process. Second, we estimated the reliability of single-item assessments (r_ii) using method CA, method FA, method DMM, method λ₆, and method LCM. In each simulation condition, given that 1,000 datasets were drawn, 1,000 estimates of the reliability of single-item assessments were obtained using each method. Given that there are different conditions in the simulation study, a method may be the most precise in one condition, while it may not be the best choice in other conditions. Therefore, in each simulation condition, we used four criteria to examine the precision of the five methods. The first three criteria are based on bias, (r_ii–ρ_ii), and relative bias, ( r i i − ρ i i ρ i i ):

The final criterion is the root mean square error (RMSE). Root mean square error is the standard deviation of the prediction errors, where the prediction errors are a measure of the difference between the predicted value and the true value (Chai and Draxler, 2014). In this simulation study, prediction errors are the absolute value of (r_ii–ρ_ii)s, and RMSE is the standard deviation of |r_ii–ρ_ii|s.

For method CA, method FA, method DMM, method λ₆, and method LCM, the median and IQR of bias, the median and IQR of relative bias, the percentage of outliers of bias (i.e., the percentage of outliers of relative bias), and the RMSE (the standard deviation of|r_ii–ρ_ii|s) are presented in Tables 4–7.

Looking at the percentage of outliers (see Table 6), method λ₆ produced 6.4% of outliers in simulation condition 17 and 5.5% of outliers in simulation condition 35, and method LCM produced 5.7% of outliers in simulation condition 17. Apart from these exceptions, all other percentages of outliers were less than 5%. Method CA, method FA, method DMM, method λ₆, and method LCM generated a small percentage of outliers in most simulation conditions.

When considering bias, method LCM produced an IQR of 0.073 in simulation condition 10 and an IQR of 0.090 in simulation condition 7, apart from these two IQRs, other IQRs were not greater than 0.070 (see Table 4). All RMSEs were less than 0.044 (see Table 7). Based on the percentage of outliers of bias, the IQR of bias and the RMSE, method CA, method FA, method DMM, method λ₆, and method LCM produced acceptable deviation values for estimating the reliability of single-item assessments in most simulation conditions.

Looking at the median of bias, only 13.89% of the medians were positive (see Table 4). Method CA, method FA, method DMM, method λ₆, and method LCM underestimated the reliability of single-item assessments in most simulation conditions.

In each simulation condition, we first focused on the percentage of outliers, if the percentage of outliers for a method is greater than 5%, then that method is discarded, as we are not 95% confident that this method can precisely estimate the reliability of single-item assessments. Among the remaining methods for estimating the reliability of single-item assessments, if one method has the smallest absolute value of the median, IQR, RMSE, and the percentage of outliers compared to other methods, then that method is the most precise. If a method has the smallest absolute value of the median while having a comparable deviation (IQR, RMSE, percentage of outliers) to other methods, then this method will be the most precise method for estimating the reliability of single-item assessments. If two methods are indistinguished in the above four dimensions (e.g., two methods are almost identical in the above four dimensions; one method has a smaller absolute value of the median while another has a smaller deviation), we will choose the easier method to estimate the reliability of single-item assessments, e.g., if method CA and method LCM are almost identical in the above four dimensions, we will choose method CA to estimate the reliability of single-item assessments because method CA is much easier to perform than method LCM. Following the above criteria, the most precise method in each simulation condition is shown in Table 8.

The conclusions about precision (Table 8) were based on an initial review of the median and IQR of bias, the median and IQR of relative bias, the percentage of outliers of bias, and the RMSE, but in some simulation conditions it was difficult to determine the most precise method based on such an initial review. For example, in simulation condition 30, method FA and method DMM are comparable in terms of deviation (IQR, RMSE, and percentage of outliers), we chose method DMM because the absolute value of the median of bias for method DMM (0.10) was smaller than that for method FA (0.11), while the difference was very small (0.01); could we conclude that method DMM is the most precise method based on such a small difference between two absolute values of the median of bias? To address this issue, we used Kruskal–Wallis tests to examine the difference between group medians in each simulation condition. The Kruskal–Wallis test is a non-parametric equivalent of a one-way ANOVA test, testing whether there is a significant difference between three or more group medians. When a Kruskal–Wallis test shows that there is a significant difference between medians, researchers perform Kruskal–Wallis multiple comparisons to compare the medians of each pair of groups (Rayner and Best, 1997). After running Kruskal–Wallis tests and Kruskal–Wallis multiple comparisons in each simulation condition, the results in Table 8 remained almost the same except for two simulation conditions, condition 14 and condition 30. In simulation condition 14, the Kruskal–Wallis test was significant (H(4) = 12.06, p < 0.05), but there was no significant difference in the median of bias between method CA and method FA (p = 0.62); given that method CA and method FA had comparable deviation, method CA and method FA were both the most precise methods for estimating the reliability of single-item assessments in simulation condition 14. In simulation condition 30, the Kruskal–Wallis test was significant (H(4) = 8.73, p < 0.05), but there was no significant difference in the median of bias between method FA and method DMM (p = 0.17); given that method DMM and method FA were comparable in deviation, method DMM and method FA were both the most precise methods to estimate the reliability of single-item assessments in simulation condition 30. Based on the results in Table 8 and the results of the Kruskal–Wallis test in each simulation condition, the most precise method in each simulation condition is shown in Table 9.

As shown in Table 9, method DMM is the most precise method for estimating the reliability of single-item assessments in 30 simulation conditions, method LCM is the most precise method in simulation condition 8, method CA is the most precise method in simulation conditions 10 and 32, and method λ₆ is the most precise method in simulation condition 26. In simulation condition 14, both method CA and method FA are the most precise methods. In simulation condition 30, both method FA and method DMM are the most precise methods.

Out of 36 simulation conditions, method DMM is the most precise method for estimating the reliability of single-item assessments in 31 simulation conditions (in condition 30, both method DMM and method FA are the best methods), method CA is the most precise method in 3 simulation conditions (in condition 14, both method CA and method FA are the most precise). All simulation conditions may occur in real practice, it is a tedious task for researchers to examine the multi-item assessment and the correlation between single-item assessment and multi-item assessment and then decide which method should be adopted. Given that we are 94.44% confident in precisely estimating the reliability of single-item assessments using method DMM and method CA, if researchers want to estimate the reliability of single-item assessment, they should use both method DMM and method CA regardless of test length, discrimination, correlation and sample size; these two reliability estimates should be provided simultaneously to show the range of reliability of single-item assessments.