The phenomenon of attenuation in the estimates by PMC is well-known. Pearson (1903) and Spearman (1904) may be the first scholars discussing the mechanical errors in estimators of correlation, while Brown (1910) and Spearman (1910) may be the first to connect this to reliability. All of them tried to find a solution to the known challenge in the estimates of correlation known today as restriction of range (see the literature in Sackett and Yang, 2000; Sackett et al., 2007;, Meade 2010). It is known that when only a portion of the range of values of the variable is actualized in a sample it leads to inaccuracy in the estimates of PMC, that is, the values are attenuated. Schmidt and Hunter (1999), specifically, discusses the need of utilizing the knowledge from attenuation correction when estimating measurement error.

Even if there was no obvious restriction of range obtained due to a reduced variance in the score variable within the sample, PMC underestimates the true correlation always if the scales of the variables are not equal (see algebraic reasons in, e.g., Metsämuuronen, 2017). This kind of deflation in PMC caused by mechanical reasons is easy to illustrate by two identical continuous variables with an obvious perfect correlation, ρ_XX = 1. If we dichotomize one to be a binary variable (item g) and polytomize the other to include several ordinal or interval-scaled bins (score X), PCM between these variables cannot reach the obvious true (perfect latent) correlation. Instead, the value depends, among others, on the cut-off where the ordered continuous variable is dichotomized to 0s and 1s, that is, of the item difficulty. If the cut-off is extreme, PMC approximates 0 irrespective of the fact that the true correlation between the variables was perfect (see simulation e.g., in Metsämuuronen, 2021b). Even at the highest, PMC cannot reach the perfect ρ_XX = 1; if there are no ties in the score, the highest value approximates 0.866.² Then, because of deflation, the loss of information in PMC may vary 13–100% depending on the item difficulty and the sample size. This loss of information is illustrated in Figure 1.

To give a practical illustration of the magnitude of error caused by deflation of correlation by different estimators, let us consider the situation described above: two identical variables with (obvious) perfect correlation ρ_XX = 1. Let there be 1000 cases and a normal distribution in the original variables. One of the variables becomes an item g by categorizing it into three categories (0, 1, and 2; df(g) = 2) and the other is polytomized into 21 categories (score X, df(X) = 20). The cut points are arbitrary from the illustration viewpoint; let the average difficulty level of the item be p(g) = 0.90 (or, p(g) = 0.10) that is, we have a very easy (or difficult) item, and the test score be of a medium difficulty level, p(X) = 0.50. Figure 1 illustrates the differences between some known estimators of correlation; the estimators are discussed later with literature.

Knowing that the latent correlation is perfect, the magnitude of the correlation strictly indicates the amount of deflation. We note that, of the estimators in the example, tau-b, biserial correlation (Pearson, 1909), and PMC (Rit) cannot reach the (obvious) perfect correlation between the two versions of the same variable and, more, the magnitude of deflation is remarkable (0.43, 0.34, and 0.31 units of correlation, respectively). Of the estimators, D, D₂, and RREG give far better approximations of the latent correlation even if there still is some error in the estimates (0.010, 0.009, and 0.001 units of correlation, respectively). In contrast, RPC, G, and G₂ reach the perfect latent correlation, that is, there is no deflation in the estimates when it comes to difficulty level of the items. Notably though, there may be other factors causing deflation or underestimation of association. Some of these factors are discussed in what follows (see also Metsämuuronen, 2021d).

By modifying the above example of two identical variables with relevant traditional coefficients of correlations such as RPC, G, and D, Metsämuuronen (2021b) concluded that PMC is affected (at least) by six sources of MEC: (1) Discrepancy in scales of the variables in general: PMC cannot reach the true (perfect) correlation between the item and the score when the dimensions of the variables differ from each other; (2) Item difficulty and item variance: the more extreme the item difficulty, the less variance, and the more underestimation in PMC. The loss of information approximates 100% with extremely easy and difficult items; (3) The number of categories in the item: the fewer the categories, the more underestimation in PMC; (4) The number of categories in the score: the fewer the categories, the lesser predictable the underestimation is; (5) The number of tied cases in the score: more there are tied cases in the score, lesser predictable the underestimation is. This is related to the sample size and the number of categories in the score (point 4); (6) The distribution of the latent variable: PMC underestimates the true correlation more if the latent variable is normal or skewed than in the cases of even distribution. These sources of the MEC are not the only possible ones although they are characteristics to PMC (see Metsämuuronen, 2021b).

Although rigorous studies have been done on these elements (e.g., Martin, 1973, 1978; Olsson, 1980; Anselmi et al., 2019; Metsämuuronen, 2021b) these tend to be fragmentary; systematic studies of the several elements of MEC would enrich our knowledge of the phenomenon. Notably, in all the six conditions above related to the attenuation in PMC, such benchmarking coefficients as RPC and G appeared to be MEC-free in the simulation (see Metsämuuronen, 2021b); the estimates reach the perfect correlation either strictly (G = 1) or asymptotically (RPC≈ 1) irrespective of the condition. D appeared to be less affected by MEC than PMC but not to the extent as RPC and G (see also Figure 1). The reason for the latter is that while RPC and G are not affected by the tied cases, D is, specifically, with short tests (see the differences of D and G also in Metsämuuronen, 2021a).

PMC is deep-rooted to the practices within the test theory and measurement modeling settings. From the reliability viewpoint, on the one hand, PMC is strictly visible in such classic estimators as ρ_BS, ρ_FR, ρ_KR21, ρ_α, ρ_GLB, and λ₁−λ₆ discussed above. Common to these estimators is that the variance of the test score (σX2) inherited from the basic definition of reliability is visible in the formula³ and σX2, on its behalf, can be expressed by using the item–score correlation (Rit = ρ_iX = PMC): σX2=(∑i=1kσi×ρi⁢X)2(Lord et al., 1968) where k refers to number of items in the compilation and σ_ito the standard deviations of partitions or items. Then, as an example, coefficient alpha can be expressed as (Lord et al., 1968):

On the other hand, PMC is embedded in the estimators based on factor- and principal component analysis because the factor- and principal component loadings (λ_i) are, essentially, correlations between an item and the score variable (e.g., Cramer and Howitt, 2004; Yang, 2010). This concerns such estimators of reliability as coefficient theta (ρ_TH; Armor, 1973; see also Lord, 1958; Kaiser and Caffrey, 1965), known also as Armor’s theta:

where λ_i are principal component loadings of the (first or only) principal component, coefficient omega (ρ_ω; Heise and Bohrnstedt, 1970; McDonald, 1970), known also as McDonald’s omega total:

and coefficient rho, known also as maximal reliability (ρ_MAX) or Raykov’s rho (Raykov, 1997a,2004) based on the conceptualization suggested by Li et al. (1996) and Li (1997):

(e.g., Cheng et al., 2012) where λ_i are factor loadings.

From the traditional measurement modeling viewpoint (see, e.g., McDonald, 1999; Revelle and Condon, 2018) the forms in Eqs. (1) to (4) implicitly assume that ρ_iXand λ_i are deflation-free. However, on the one hand, ρ_iX is known to be severely deflated (see above). On the other hand, if we use the operationalization familiar in principal component analysis (PCA), exploratory factor analysis (EFA), and structural equation modeling (SEM) where λ_i is a principal component- or factor loading, assumption of deflation-free estimates is too optimistic assumption because λ_i is, essentially, a correlation between item and the factor (or principal component) score variable (Yang, 2010). That is, λ_i is (essentially) ρ_iX being deflated as discussed above.

The effect of MEC in deflation in the estimates of reliability may be remarkable. Two empirical examples are given. The first example comes from Gadermann et al. (2012) who report a dataset where, by using ordinal alpha (α_ORD; Zumbo et al., 2007), another kind of DCER based on replacing the inter-item matrix of PMCs by a matrix of RPCs, the estimate by ρ_α was deflated from 0.85 (α_ORD) to 0.46 (ρ_α), that is, 0.39 units of reliability which equals 46% (=0.85–0.46)/0.85).

Another example comes from a national level testing program of learning outcomes (n = 7,770; Metsämuuronen and Ukkola, 2019) where the preconditions of understanding the instruction language were assessed with a very easy 8-item, 11-point test. It was expected that only students with second language background in the instruction language would make mistakes in the test; of all test takers, 72% gave the full marks. The magnitude of the estimate of reliability by the traditional coefficient alpha was found to be ρ_α = 0.25 and by rho ρ_MAX = 0.48. By using a DCER based on Somers D where ρ_iXis replaced by D(i|X) = D_iX in the formula of alpha (see later Eq. 23), the magnitude of deflation-corrected alpha was ρ_{α_DiX} = 0.86. Then, the magnitude of the estimate by ρ_α was deflated around 0.60 units of reliability (71%) and the estimate by ρ_MAX around 0.38 units of reliability (44%). The obvious reason for the remarkably higher estimate by ρ_{α_DiX} is that, in the case of binary items with extreme difficulty level, PMC as well as the factor loadings are severely attenuated while, in the binary case, D is less deflated. In both examples, the deflation in the estimates by the traditional estimators is remarkable. The latter example will be re-analyzed in section “Practical Example of Calculating Deflation-Corrected Estimators of Correlations Discussed in This Article” in details.

Suggesting a radically new way of estimating reliability urges in-depth discussion of theoretical foundations of the new approach. However, here, the new concepts are built based on the traditional measurement models (see, e.g., McDonald, 1999; Cheng et al., 2012) which are, however, rethought and reconceptualized to also include the elements of deflation. Some further alternatives to consider for rethinking reliability are discussed in section “Options for Correcting the Deflation in Estimators of Reliability.” The effect of deflation is discussed here theoretically only to the extent that makes the notation in deflation-corrected estimators of reliability understandable.

Let w_i be a general weight factor that links the observed values (x_i) of an item g_i with the latent variable θ manifested as a score variable:

generalized from the traditional one-latent variable model (e.g., McDonald, 1999; Cheng et al., 2012). It is relevant to assume that the weight factor w_i is a coefficient of correlation (−1≤w_i≤ + 1) such as Rit, RPC, G, or D, or principal component- or factor loadings (λ_i). Also, the latent variable θ may be manifested as varying types of relevantly formed compilation of items such as a raw score (θ_X), factor score variable (θ_FA), principal component score variable (θ_PC), a theta score formed by the item response theory (IRT) or Rasch modeling (θ_IRT), or a possible non-linear compilation of the items (θ_NonL).

Eq. (5) generalizes to the compilation of items as

where k is the number of items in the compilation. Eq. (6) corresponds with the classic relation of the observed score (X), true score (T), and error (E) in the classical measurement model, that is, X = T + E discussed above. To visualize the differences between different models, this general (congeneric, one-latent variable) model without considering the elements of deflation is as in Figure 2A.

From the correlation viewpoint, knowing that all generally used estimators of correlation give identical estimates of the correlation for original variables and for the standardized versions of the variables, without loss of generality, we can assume that g_i and θ are standardized, x_i,θ∼N(0,1). Then, parallel to the traditional model (see e.g., Cheng et al., 2012), the error variance of the test score ψi2 can be estimated as

Eq. (7) can be strictly used in estimating the reliability of the score variable (REL = 1-σE2⁢/⁢σX2). If we use principal component loadings as the weight factor and principal component score as a manifestation of θ, the conceptualization of error variance in Eq. (7) is used strictly in ρ_TH (Eq. 2) and, when using factor loadings and factor score variable, it leads to such estimators as ρ_ω and ρ_MAX (Eqs. 3 and 4).

The traditional estimators of reliability assume that Rit and factor/principal component loadings are deflation-free. This is a too optimistic assumption as discussed and illustrated above (see Figure 1). If the observed value of w_i embeds deflation, as it typically does when using the traditional estimators of correlation and loadings, the magnitude of the observed correlation or loading by a deflated or MEC-defected (MECD) weight factor (w_{i_ MECD}) is, obviously, lower than MEC-free (MECF) weight factor (w_{i_MECF}), that is,

or

where the exact magnitude of the error element related to deflation in estimation (e_{wi_ MEC}) is largely unknown although it is positive (e_{wi_ MEC} > 0), and it depends on the characteristics of the item and the weight factor as discussed above. While knowing that a certain part of the measurement error is strictly technical or mechanical in nature, but its magnitude could be reduced, it makes sense to reconceptualize the classic relation of X = T + E into a form

where the element E_MEC related to deflation is something we can deal with. Notably, this kind of “systematic error” is not a kind we usually consider as “systematic” such as a typo in the test item or some technical problem in processes (see Gulliksen, 1950; Krippendorff, 1970). The latter type of error is usually considered harmless from the reliability viewpoint and its effect is added to the random part of the error. Consequently, we can reconceptualize the measurement model in Eq. (5) as

where the notation e_{wiθ _ MEC}refers to the fact that the magnitude of the deflation depends on the characteristics of the weighting factor w, item i, and the score variable θ. This model using a weight factor including radical deflation such as Rit or λ_i may be illustrated as in Figure 2B. Notably, the magnitude of the total error (e_{i_Random} + e_{wiθ_MEC}) is, factually, equal to the one seen in the model in Figure 2A. However, now the two components are just visual.

While knowing that some estimators of correlation are less deflated than some others, it makes sense to select such coefficient as the weighting factor where the quantity of technical or mechanical error would be as low as possible. However, it may be difficult to find an estimator of correlation without deflation, that is, that would be totally deflation- or MEC-free. In what follows, the concept of deflation-corrected and, specifically, MEC-corrected estimator (MECC) is used to refer such estimators where the deflation is known to be radically smaller than in PMC. If selecting wisely the weight factor, the magnitude of error component related to deflation may be near zero, that is, e_{wiθ_MEC}≈0. If we use options of w_i that would lead us to the condition of e_{wiθ_MEC}≈0, because of Eq. (10), this will lead us to a model where the measurement error would be as near the MEC-free condition as possible, that is,

This measurement model where MEC-corrected weight factors such as RPC, G, or D are used, could be illustrated as in Figure 2C.

As with Eq. (7), knowing that all generally used estimators of correlation give identical estimate of the correlation for original variables (g_i and θ) and for the standardized versions of the variables, we can assume that g_i and θ are standardized, x_i,θ∼N(0,1). Then, assuming that item-wise random errors do not depend on the true scores, the item-wise MEC-corrected error variance (ψi⁢_⁢M⁢E⁢C⁢C2) is

that is, ei⁢_⁢M⁢E⁢C⁢C∼N⁢(0,ψi⁢_⁢M⁢E⁢C⁢C2) where ψi⁢_⁢M⁢E⁢C⁢C2=1-wi⁢_⁢M⁢E⁢C⁢C2. Then, after the deflation-correction, the Eq. (9) could be written as

and Eq. (10) as

Consequently, the deflation-corrected error variance of the test score can be written as

where the form corresponds to the traditional error variance

used in the traditional estimators of omega and rho in Eqs. (3) and (4) (see, e.g., Cheng et al., 2012). In the deflation-corrected estimators or reliability, instead of using factor- or principal component loadings we use deflation-corrected estimators of correlation.

By being open for different manifestations of w_i and θ, some options for the base of the deflation-corrected estimators of reliability are theoretical deflation-corrected alpha based on Eq. (1):

theoretical deflation-corrected theta based on Eq. (2):

theoretical deflation-corrected omega based on Eq. (3):

and theoretical deflation-corrected rho based on Eq. (4):

where w_iθ refers to the general model where the manifestations of θ may vary as well as the linking coefficient w and, obviously, the estimate varies item-wise. Obviously, using the estimators (17) to (20) outside of their original context of raw scores or principal component- and factor analysis is debatable. Here, a stand-point is taken that the forms could be used as stand-alone estimators even without their original contexts. This is consistent with a more general measurement model discussed above. Alternatively, the estimators (18) to (20) may be taken as an output of renewed procedures in the principal component- and factor analysis where w_i is a less deflated estimator of correlation than the traditional principal component- and factor loading.

By combining the theoretical estimators in Eqs. (17) to (20) and different operationalizations of w_i, we get varying families of deflation-corrected estimator of reliability. Let us assume that we do not fix the manifestation of θ, and we use such MEC-corrected weight factors as RPC, G and D directed so that “item given score” or D = D(i|X) usually labeled as “score dependent” in the common software packages (of the correct direction of D, see Metsämuuronen, 2020b). This leads us to such practical family of deflation-corrected estimators of reliability as deflation-corrected alpha based on Eq. (17) as

and

Because of using totally different type of estimator than PMC, these could be called special types of DCERs, namely, MEC-corrected estimators of reliability. If using some relevant attenuation-corrected estimator of correlation (see some options in Metsämuuronen, 2021c), a family of attenuation-corrected alpha would be obtained.

The notation in names ρ_{α_RPCiθ}, ρ_{α_Giθ}, and ρ_{α_Diθ} refers to the facts that the base of the estimator is alpha (α), the weight factor is manifested as RPC, G, or D representing different types of correlations between item and the score variable, and the manifestation of the score variable (θ) could be a raw score (θ_X) or factor score variable (θ_FA), as examples. Some of these kinds of estimators are discussed by Metsämuuronen and Ukkola (2019) and Metsämuuronen (2020b,2021a,2021b). Another type of solution is discussed by Zumbo et al. (2007) and Gadermann et al. (2012) by replacing the matrix of PMCs by a matrix of RPCs in forming the factor loadings; this leads to a coefficient called ordinal alpha discussed above.

More effective estimators than above are expected if coefficient theta (Eq. 18) is used as a base for the estimators and

RPC, G, and D as w_i.⁴ We get a family of deflation-corrected theta based on Eq. (18):

and

or a family of deflation-corrected omega based on Eq. (19):

and

or a family of deflation-corrected rho based on Eq. (20):

and

These families could be called also MEC-corrected theta, omega, and rho. Notably, Zumbo et al. (2007) and Gadermann et al. (2012) also discuss the use of Armor’s theta as a basis for ordinal theta by replacing the matrix of PMCs by a matrix of RPCs in the estimation.

Many good or even better alternative could be found for RPC, G, and D considering that using RPC may lead us to challenges in interpreting the reliability as reflecting unobservable variables (see critique in Chalmers, 2017) and G tend to underestimate correlation when there are more than four categories in the item and D with three categories or more (see Metsämuuronen, 2021b). For the polytomous case, instead of G and D, the dimension-corrected G and D are suggested (Metsämuuronen, 2021b).

The characteristics of the estimators above are not discussed in-depth here; simulations would be beneficial in this matter. However, in the hypothetic extreme datasets with deterministic item discrimination in all items leading to RPC_i = RPC_j ≈ G_i = G_j = D_i = D_j = 1,⁵ DCERs based on theta and omega would lead to perfect reliability: ρ_{TH_RPCiθ}≈ρ_{TH_Giθ} = ρ_{TH_Diθ} = k/(k−1)(1−1/k)≡1 and ρ_{ω_RPCiθ}≈ρ_{ω_Giθ} = ρ_{ω_Diθ} = (k)²/((k)² + 0)≡1. In the case, estimators (21) to (23) based on alpha can reach the value ρ_{α_RPCiθ}≈ρ_{α_Giθ} = ρ_{α_Diθ} = 1 only when all item variances are equal (σ_i = σ_i = σ), that is, for instance, when the items are standardized. In the case, ρα⁢_=R⁢P⁢C⁢i⁢θρα⁢_=G⁢i⁢θρα⁢_=D⁢i⁢θk/(k−1)(1−kσ²/(k(σ×1))²) = k/(k−1)×(1−1/k)≡1. Otherwise, the maximum value is ρα⁢_⁢R⁢P⁢C⁢i⁢θM⁢a⁢x≈ρα⁢_⁢G⁢i⁢θM⁢a⁢x=ρα⁢_⁢D⁢i⁢θM⁢a⁢x=kk-1⁢(1-∑i=1kσi2⁢/⁢(∑i=1kσi)2). Notably, in the deterministic case, estimators based on rho (Eqs. 30 to 32) could not be used because this would require division by zero which is not defined. Aquirre-Urreta et al. (2019) also noted that rho may produce overestimates of the true reliability with finite samples familiar in real-world testing settings. A practical reason for this is that the formula is sensitive to very high values of loadings. In small sample sizes familiar in the real-world datasets, the possibility to obtain deterministic or near-deterministic situation in one or several items increases. In deterministic patterns, ρ_MAX cannot be estimated at all and in the near-deterministic patterns the factor loading may be artificially high leading to obvious overestimation in reliability. In what follows in a numerical example, the outcomes based on the DCERs in Eqs. (21) to (23), (30) and (31) are illustrated and the traditional estimators (1) to (4) are used as benchmarks.

As a simple numerical example, the dataset consisting of a set of 30 multiple choice questions forming 30 binary items and n = 49 randomly selected test-takers from a national level datasets of mathematics test (N = 4,023; FINEEC, 2018) representing small-scale tests with finite samples is used in illustrating the difference between the traditional estimators and deflation-corrected estimators of reliability. The dataset with estimates of different score variables and weight factors are in Supplementary Appendix 1.⁶

The general measurement model discussed in section “Conceptual Base of the Deflation-Corrected Estimators of Reliability” is applied. By using the general one-factor model and by varying w and the operationalization of θ, examples of traditional and deflation-corrected estimates of reliability of the score are given by modifying mainly the form of rho (Eq. 20) with some benchmarking estimates by the form of alpha (Eq. 17).

In the empirical section, five operationalizations for θ are used: an unweighted raw score (θ_X), a principal component score variable (θ_PC), a factor score variable by maximum likelihood estimation (θ_FA), a theta score by one-parameter IRT model or Rasch model (θ_IRT), and a nonlinear weighted score by a simple weighting factor 1/p_i (θN⁢o⁢n⁢L=θP⁢I=∑i=1kgi⁢/⁢pi) where the test-takers are weighted by the proportion of correct answers p_i; the more demanding item, the higher the weight.

Seven options as the weighting factor between θ and g_i are used. First, traditional estimators used in the traditional estimators of reliability: Rit with θ_X, principal component loading with θ_PC, and ML-estimate of the factor loading with θ_FA; second, alternative coefficients RPC, G, and D for deflation-corrected estimators of reliability; and, third, the traditional PMC (later, R or R_iθ) as a benchmarking coefficient for the DCERs when not using the traditional alpha. The statistics for and calculations of the estimates are collected in Supplementary Appendix 1.

Combining the operationalizations above, we get estimators of reliability related to five different scores and seven linking factors; only selected combinations are used (see condensed in Table 1).

First, traditional estimators (alpha, theta, omega, and rho; Eqs. 1–4) of which rho is re-notated here to match with the other estimators:

where the notation ρ_{MAX_ λiθ _FA}refers to facts that coefficient rho is the base of the coefficient (MAX), the manifestation of the score variable is the factor score variable (θF⁢A), and the manifestation of the weight factor is the ML-estimate of the factor loading (w_{i = λ_i =λ_iθFA}).

Second, five estimators based on the form of rho and item–score correlation (ρ_iθ = R_iθ) as the linking factor:

where the score is θ_X and w_i = R_iθX, (34)

where the score is θ_PC and w_i = R_iθPC,

where the score is θ_FA and w_i = R_iθFA,

where the score is θ_IRT and w_i = R_iθIRT, and

where the score is θ_PI and w_i = R_iθPI.

Third, the parallel estimators using RPC = RPC_iθ as the linking factor:

where the score is θ_X and w_i = RPC_iθX,

where the score is θ_PC and w_i = RPC_iθPC,

where the score is θ_FA and w_i = RPC_iθFA,

where the score is θ_IRT and w_i = RPC_iθIRT, and

where the score is θ_PI and w_i = RPC_iθPI.

Fourth, the parallel estimators using G = G_iθ as the linking factor:

where the score is θX and w_i = G_iθX,

where the score is θ_PC and w_i = G_iθPC,

where the score is θ_FA and w_i = G_iθFA,

where the score is θ_IRT and w_i = G_iθIRT,

and

where the score is θ_PI and w_i = G_{iθ _PI}.

Additionally, DCERs based on coefficient alpha (Eqs. 21–23) are used as benchmarks to the traditional estimators (see Table 1). Of the calculation of the estimates, see Supplementary Appendix 1.

The root challenge related to deflation in the traditional estimators of reliability seems to be the classical definition of reliability based on variances (σX2, σT2, and σE2) leading to use PMC in the practical solutions of estimating reliability. If we would start to create a theory concerning reliability by knowing all the deficiencies of PMC we know today, we may be trying to avoid PMC and, consequently, the variances in the process. To rectify this root challenge, it may be beneficial to rethink the definition of reliability from this perspective. Alternative bases to consider for rethinking reliability may be related to, among other, “sufficiency of information” by Smith (2005), or several options within IRT modeling such as “person separation” by Andrich and Douglas (1977), Andrich (1982), and Wright and Masters (1982), or “information function” discussed by, e.g., McDonald (1999), Cheng et al. (2012), and Milanzi et al. (2015). One alternative for defining reliability is discussed briefly here based on Metsämuuronen (2020b) related to the definition of “ultimately discriminating test score.”

Metsämuuronen (2020b) proposes an operational definition of the ultimate item discrimination as a condition where the score can predict response pattern of the test-takers in a single item in a deterministic manner. This could be generalized as a theoretical condition for ultimate reliability as being a condition where the score can predict the order (or item response pattern) of the test takers in a deterministic manner in all items. This operational definition alone is not very practical when it comes to estimation of the reliability because the deterministic patterns cannot be estimated by using maximum likelihood method, for example. However, this could be a starting point to develop estimators where different types of estimators of item discrimination as well as a-parameter in IRT-modeling could be a visible part of the estimator as in Eqs. (21) to (32). Theoretical and empirical work in this area would be beneficial.

While waiting for development of a sound basis for a new way of thinking, defining, and estimating reliability, practical options lead to a kind of new paradigm in the settings related to measurement modeling: the extended families of deflation-corrected estimators of reliability. One set of family, attenuation-corrected estimators of reliability, not discussed in this article, would be obtained if attenuation-corrected estimators of PMC were used instead of PMC in the estimators. Another set of family, MEC-corrected estimators of reliability focused in this article, is obtained if PMC is replaced by a totally different estimator of correlation that would not be deflated at all or where the magnitude of deflation is remarkably smaller than that in PMC. Several new estimators of deflation-corrected estimators were proposed based on using RPC, G and D as examples instead of PMC in some known estimators of reliability.

In the empirical part, it was demonstrated that if RPC, G, or D would be used instead of PMC in some known estimators of reliability, the deflation in reliability would be corrected to a notable extent. Further simulations with different types of datasets, different item types, different weighting factors, and different base of the estimators (e.g., alpha, theta, omega, or rho) would be beneficial in this regard. The estimates by deflation-corrected estimators are not, factually, “real” reliabilities as such. However, they are closer to the deflation-free reliability than the traditional estimates. Empirical examples show that, in specific forms of datasets as in very easy or very difficult tests, the estimates by traditional estimators such as coefficient alpha and rho may be deflated 40–70% because of technical reasons. The DCERs discussed in this article are strong with these kinds of datasets and could be used as a benchmark to the traditional estimators.

To give a practical example of the DCERs discussed in this article, let us re-analyze the reliability of the extremely easy dataset (n = 7,770) by Metsämuuronen and Ukkola (2019) discussed in section “Practical consequences of Mechanical Error in the Estimates of Correlation in reliability.” The advance of DCERs may be notable in these kinds of datasets where the item difficulties are extreme leading to an ultimately non-normal score (see Table 3). Because of ultimately easy items with mainly binary scales combined with a non-normal score variable, the non-parametric coefficients of correlation may be better options than PMC.

The article did not tackle the issue of differences between the estimators of correlation. Notably, PMC, RPC, and G (as well as D) discussed in the article indicate different aspects of the correlation: PMC estimates the observed correlation between two variables, and this is radically deflated in the measurement modeling settings. RPC estimates the inferred correlation of two unobservable continuous variables by their ordinal manifestations. G and D estimate the probability that the test takers are in the same order both in an item and a score. The outcome of different estimators of reliability may, then, indicate different viewpoints of reliability.

Chalmers (2017) is skeptical of the usefulness of coefficients using RPC in practical settings because RPC refers to correlation between unobservable and unreachable variables and, therefore, the outcome may be useless in the factual interpretation of the observed score. He proposes that using RPC leads to infer something about theoretical reliability. However, some estimators of reliability such as ordinal alpha and theta by Zumbo et al. (2007; see also Gadermann et al., 2012), factually, use RPC in the estimation. Comparing the estimators related to RPC in Eqs. (21), (24), and (27) and (39) to (43) with ordinal alpha or ordinal theta based on the matrix of inter-item RPCs instead of matrix of PMCs may be worth studying.

Estimators based on G and D refer to observed variables and, therefore, the outcome may be more useful than those by RPC in the factual analysis of the observed score. Knowing the interpretation of G and D in the measurement settings (see Metsämuuronen, 2021a,b), estimators (22) and (23), (25) and (26), (31) and (32), and (44) to (48) reflect the average proportion of logically ordered test takers in all items as a whole. In this, the estimators based on D are more conservative than the ones based on G.

A relevant question is, how different is the interpretation of the estimates by G (or D) in comparison to those by PMC or RPC? Knowing that G estimates the probability that the test takers are in the same order in the item and in the score, the ultimate magnitude of reliability by the estimators based on G would indicate that all items discriminate the higher-performing test takers from the lower-performing test takers in a deterministic manner after the test takers are ordered by the score. The same interpretation would be obtained when using RPC except that RPC can reach the value RPC = 1 only approximatively. From this viewpoint, the deflation-corrected estimators in Eqs. (24) to (32) related to RPC, G, and D seems to refer strictly to the discrimination power of the score. This makes sense from the standard error of measurement viewpoint. Notably, under the condition of deterministic item discrimination, the estimators using PMC cannot reach the perfect reliability because the estimates by PMC cannot detect the deterministic correlation unless the number of categories is equal in the variables. More studies and theoretical work in the interpretation of the estimators would enrich us.

Some typological characteristics of different estimators of the estimators described in the article are summarized in Table 8. Notably, again, RPC, G, and D are not the only options for DCERs; further studies related to such estimators as r-bireg- and r-polyreg correlations, G₂, D₂, as well as attenuation-corrected Rit and eta, as examples, would be beneficial (see footnote 6).