Mild cognitive impairment (MCI) is estimated to affect about 7% of adults aged 60–65, increasing to 25% by age 80–85 years (Petersen et al., 2018). MCI is defined by the American Psychiatric Association as subtle decline from baseline cognition accompanied by decreased performance of cognitively-complex daily activities (American Psychiatric Association [APA], 2013; Petersen et al., 2018). Evidence guiding clinical screening of cognition during routine health care for healthy older adults is evolving along with recent advances in the understanding of both normal aging and neurocognitive disease (Livingston et al., 2020). Recognition has grown that healthy older adults may benefit from cognitive screening as part of preventative health care (Anderson, 2019) because cognitive impairment contributes to disability (Jekel et al., 2015; Lindbergh et al., 2016) and has been found to progress into dementia in about 15% of cases (Petersen et al., 2018). In clinical practice, accurate early identification of mildly-impaired cognition enables assessment and treatment of causes of reversible cognitive decline (e.g., sleep or mood disorders), pursuit of non-pharmacological interventions to support cognitive aging (Tsolaki et al., 2011), and management of comorbidities that contribute to dementia risk (Livingston et al., 2020). Thus there is a critical need for rapid, reliable, and sensitive objective screening tools that detect subtle cognitive change from baseline (Anderson, 2019).

However, to meet these evolving clinical needs, more evidence is needed on the value of cognitive screening in healthy older adults. The U.S. Preventive Services Task Force rates the evidence for benefit from screening asymptomatic older adults for cognitive decline as insufficient (Lin et al., 2013), and many cognitive screening instruments may not be reliable for this purpose as they were designed to detect dementia, not subtle impairment. The clock drawing test (CDT) is one such classic cognitive screening instrument. The CDT is a paper-and-pencil tool developed for dementia screening, administered by asking the patient to draw an analog clock face with a specific time setting which is then scored by the assessor using a scale of criteria for success. Although developed over one hundred years ago (Hazan et al., 2018), it has recently gained interest for its potential to detect mild impairment (Duro et al., 2019), its neurofunctional correlations with the executive functions required to complete complex daily activities (Royall et al., 1999; Dion et al., 2022), and an adaptation to digital format (Müller et al., 2017; Davoudi et al., 2020; Dion et al., 2020; Yuan et al., 2021).

Clock drawing is of interest for screening healthy older adults based on its plausible neurocognitive relevance to milder impairment (Strauss et al., 2006) such as its reliance on constructional praxis and executive function and their neuroanatomical correlates (Talwar et al., 2019) which are often affected in early dementia and MCI (Mendez et al., 1992; Rouleau et al., 1992, 1996; Chiu et al., 2008; Talwar et al., 2019). A free-drawn electronic CDT using an instrumented pen and electronic tablet-style writing surface has been found to distinguish people with MCI from healthy controls (Yuan et al., 2021), supporting the assumption that the cognitive skills required to draw a clock are relevant to the diagnostic purpose of screening for mild impairment. Compared to this electronic version, paper versions are not as accurate in detecting subtle change (Müller et al., 2017); however, conventional versions are more practical, brief, easy to obtain and administer, familiar [a CDT task is incorporated into several test batteries such as the Mini-Cog (Lam et al., 2011) and the Montreal Cognitive Assessment (Nasreddine et al., 2005)], and still have potential to yield rich clinical information. This makes conventional CDT appealing as a screening instrument for mild impairment in community-dwelling older adults.

Despite its benefits and potential, further examination of conventional CDT’s appropriateness for detection of subtle impairment is needed. First, some of the index measures used in initial validation studies have since been shown to be insufficiently sensitive to MCI in a range of patient populations (Arevalo-Rodriguez et al., 2021), so these psychometric references do not relate to current diagnostic practice or evidence. Second, it is unknown whether the scoring ranges for many versions, scaled to detect potential dementia, can detect small but meaningful decreases from an unimpaired cognitive baseline. Third, although paper-and-pencil CDT’s have been shown to have low sensitivity to MCI (Pinto and Peters, 2009; Duro et al., 2019), only a few recent studies have focused on community-dwelling older adults (one of the most important populations to screen for MCI). Particularly in older studies establishing cut scores, study design precluded conclusions about mild impairment. Finally, we were not able to find previous work that used methods such as item response theory to determine whether these scales’ levels of difficulty are suitable for screening this population. Although item response theory has been employed to evaluate tests that include a clock drawing task [e.g., the Montreal Cognitive Assessment (Luo et al., 2020), the Texas Functional Living Scale (Lowe and Linck, 2021)] its application to stand-alone clock scales has not been reported. Because (a) the CDT was developed primarily to detect dementia, (b) the diagnostic scope of neurocognitive disorder has since expanded to include MCI, (c) gold standard screening tools are now more sensitive to MCI, and (d) more evidence is needed for community-dwelling older adults who are the target population for screening, further study is needed to confirm conventional clock drawing’s psychometric properties for detecting small differences between unimpaired and mildly impaired cognition in community-dwelling older adults.

This study examined the reliability, sensitivity and specificity, and construct validity of two CDT scoring systems, the Rouleau (Rouleau et al., 1992, 1996) and the Clock Drawing Interpretation Scale (CDIS) (Mendez et al., 1992), for identifying cognitive performance consistent with MCI in community-dwelling older adults. We hypothesized that (a) the CDIS would exhibit higher reliability and validity than the Rouleau, and (b) both scales would distinguish people with subtle cognitive deficits from cognitively unimpaired individuals at a score close to the maximum. Additionally, we will examine the construct validity of both scales’ measurement of cognitive ability.

In this community-dwelling sample of older adults, we found that the CDIS had moderate reliability (though superior to the Rouleau scale), and that both scoring systems were moderately sensitive to MoCA scores consistent with mild cognitive impairment. Our item response theory models showed that both scales provided very limited information about small shifts toward impaired cognition, and that the “hands” items appeared to be the most discriminating. These findings are consistent with several prior findings on the Rouleau scale in particular (Chiu et al., 2008; Pinto and Peters, 2009; Talwar et al., 2019). Thus, our hypotheses were partially supported; the CDIS scale was more reliable (but not more sensitive) than the Rouleau scale, and a score of only one unsatisfied criterion had the highest sensitivity to mild impairment. Based on these results, we found insufficient evidence to recommend either scale be used as a stand-alone screening instrument in this population.

Of the two scales, the CDIS showed higher internal consistency, which we expected since reliability tends to increase as the number of items in a scale increases. One possible contribution to our low reliability is item composition. Quantitative ratings of a free-drawn object require some amount of subjective interpretation, so each item must be written for narrow interpretation with unmixed criteria for success (Bandalos, 2018b). Best practices include giving one criterion for success per item, avoiding qualifiers, and using single clauses (Bandalos, 2018b). Several of these recommendations are violated in these two scales. For instance, the Rouleau method uses qualifiers - such as “most,” “minor,” and “slight”–in 8 out of 10 items, and the CDIS in 3 out of 20. These qualifiers are adjectives expressing relative degree, but neither scale provides metrics to guide clinicians in placing performance between degrees. The influence of wording on reliability could be explored in the future by comparing item-total correlations and item statistics to identify the items with the lowest correlation to the total score, then dropping the low-correlated items and re-analyzing the scales.

Another consideration is our reliability estimator’s underlying assumptions and limitations. We chose Cronbach’s alpha to allow comparisons with prior work (Duro et al., 2019), however, there is debate as to whether this statistic is an appropriate estimator of internal consistency for this type of scale (Cho, 2021; Sijtsma and Pfadt, 2021). First, use of the alpha statistic is based on the assumption that the items of the scale are unidimensional, that is, they measure the same construct. Although theses scales are screening measures for dementia (one construct), they require many overlapping abilities such as visuospatial, executive function, motor and perceptual integration, and a cultural context for chronological measurement of time (multiple constructs)–thus they may not measure dementia per se, but rather multiple skills as proxies of dementia. Second, the error variance of all items should be uncorrelated–that is, items should vary independently of one another (Trizano-Hermosilla and Alvarado, 2016). In the assumptions of classical true score theory that underlies alpha, error originates unpredictably from a random process, so item variances should not influence one another. This is unlikely to apply to these scales because elements of a drawing rely on previous elements, for instance the clock’s hand placement relies on the clock face being symmetrical and the numbers evenly spaced. Third and finally, the scale’s items should be essentially tau-equivalent, meaning they may differ in mean and variances but must have similar covariances; if this assumption is violated then alpha is likely to underestimate the scale’s true reliability (Bandalos, 2018a). On the other hand, Cronbach’s alpha has also been found to mask multidimensionality of test items, and slightly overestimate ordinal scales such as the Rouleau scale (Vaske et al., 2017). We retain Cronbach’s alpha for this study as a lower-bound estimate of reliability, and acknowledge that other methods, such as a stratified alpha (estimations for face, numbers, and hands singly), the greatest lower bound, or expanded factor analytic methods could provide more accurate estimates for this type of scale in a larger sample.

Our criterion validity results were generally consistent with other recent work. At the MoCA cut score of 26, the sensitivity and specificity of these clock scales were moderate, and the AUC point estimates were nearly identical to that of another recent ROC analysis of the Rouleau scale (Duro et al., 2019). Lowering the criterion score to 23 had a negligible effect on AUC, but shifted both clock scales’ sensitivity 5–6 points higher, specificity 8–10 points lower, and confidence intervals several points wider, suggesting lower precision. At either cut, the highest sensitivities were lower than the 90% (Nunnally, 1978) to 95% (Kaplan and Sarcuzzo, 2001) recommended for a clinical screening tool, found at a score of only one incorrect item (9/10 on the Rouleau, 19/20 on the CDIS).

There is evidence that the MoCA score of 26 is overly sensitive, resulting in too many false positives (Carson et al., 2018; Townley et al., 2019). With deference to the inconvenience and cost this can cause, we argue that in high-functioning community-dwelling adults it is preferable for a non-invasive and brief screening tool to err on the side of greater sensitivity with the potential for false positives, rather than greater specificity with the potential to overlook cases. The MoCA is a screening measure to detect potential impairment, not to diagnose (Dautzenberg et al., 2020), and administering clinicians will also consider trends, subjective report, and patient concerns in balance with scores. Compared to the studies recommending lower cut scores (Carson et al., 2018; Milani et al., 2018; Townley et al., 2019), our sample had higher mean education in both impaired and unimpaired groups regardless of classification score, and very low racial and ethnic heterogeneity. Our ROC results at a cut score of 23 may be more valid for community samples with low mean education (Carson et al., 2018) and people from underrepresented racial groups (Milani et al., 2018; Rossetti et al., 2019); however, we could not verify this with our sample.

Our application of item response theory models is the first use of these methods with these two scales. Our results help clarify the relationship between ability and test performance, a key aspect of the test’s appropriateness for subtle cognitive decline. In our sample, these scales had levels of difficulty that were more appropriate to rating performance by a person with a lower level of cognition than the early or subtle decline we hope to capture in community-dwelling older adults; in other words, the scoring criteria on these scales appear to be too easy for this population. We found a ceiling effect that begins to detect change well below an unimpaired level, in turn driving the scales downward in cognitive ability and resulting in restriction of range that masks performances with subtle impairments. We note that both scales’ test information curves are bimodal with a leftward shift (right skew), suggesting that they are very informative for a narrow range of ability at a low cognitive level, and not informative for very small decreases from baseline. This is consistent with prior study on their psychometric properties for detecting dementia (Mendez et al., 1992; Chiu et al., 2008) and was also consistent with our and others’ ROC analyses (Duro et al., 2019). Hence, many of the items in these scales contributed little to no information toward characterizing subtle change in our sample. A caveat is that the GRM and 2PL models had poor indicators of fit–the model “expects” a response pattern that differed from that of our sample. We suspect two sources of this high error approximation; the scales may contain poorly-performing items that introduce noise into the model, and, the task’s step-by-step nature may constitute multidimensional latent constructs contributing to each item. Both of these hypotheses could be explored using exploratory factor analysis or other latent trait modeling in a future study.

It is also notable to consider the change in diagnostic definition and practice since these scales were developed. These scales’ purpose was to detect impairment within a binomial “dementia vs. no dementia” construct of cognitive function, which contrasts with our current understanding of neurocognitive decline as a continuum. For instance, the original validation paper for the CDIS compared healthy older adults to persons with dementia but excluded those with “questionable” cognitive impairment, establishing just one cut score for dementia rather than a range of scores across the cognitive spectrum. Similarly, the scoring criteria for the Rouleau scale (Rouleau et al., 1992, 1996) were developed according to errors made by people with dementia, and did not characterize a range of unimpaired performance as a reference. At that time, a range of cognitive function was not necessarily considered clinically useful, but this is no longer the case.

Based on our findings, we suggest that the task of drawing a clock is not “too easy” for a person with subtle cognitive deficits to execute without errors, but rather that dementia-validated paper scales used to score their execution may be too lenient. If the scales’ sensitivity and specificity to MCI could be improved, they would meet established criteria for good clinical screening tools (Edwards et al., 2019). Future studies could attempt to improve these scales by repeating testing and analysis after removing items with low item-total correlations and RMSEA, conducting expanded factor analyses, and using alternative reliability estimators.

Although the task of free-drawing a clock has been found to represent cognitive abilities affected by MCI (Royall et al., 1999; Yuan et al., 2021) our analysis of the CDIS and Rouleau clock drawing scales found that they did not reliably identify persons scoring in the mildly impaired range on the MoCA. We recommend use of these scales in combination with other screening tools primarily as an opportunity to observe visuospatial and executive abilities, with awareness of a ceiling effect in the interpretation of their numeric score.

Cognitive changes are often subtle and granular; even if diagnosis is categorical, small changes are both clinically meaningful for care-planning and functionally meaningful for patients. Assessment tools must reflect small differences in function to be useful for screening community-dwelling older adults. In our community-dwelling sample, the Rouleau and CDIS clock drawing scales did not reflect these small differences.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

The studies involving human participants were reviewed and approved by the University of Wisconsin-Madison Health Sciences Institutional Review Board. The patients/participants provided their written informed consent to participate in this study.

KK-F, TM, GG, and DE: concept, data collection, and design. KK-F: statistical analysis and writing. TM, GG, and DE: editing. All authors contributed to the manuscript revision, and approved the final submission.