Multiple sclerosis (MS) is a chronic immune-mediated disease of the central nervous system and is characterized by inflammation, demyelination, gliosis, and axonal destruction, which lead to the accrual of neurological disability (1). The most common measure of disability in MS is the Expanded Disability Status Scale (EDSS). Originally developed by Kurtzke (2, 3), it is a clinician-rated instrument based on the standard neurological examination of seven functional systems (FS; visual, brainstem, pyramidal, cerebellar, sensory, bowel/bladder and cerebral), and an evaluation of the maximal walking distance without rest (ambulation score). The combined scoring of functional systems and ambulation produces an ordinal scale from 0.0 (normal neurological exam) to 10.0 (death due to MS) with 0.5 increments interval after 1.0 (4). Although well-known limitations such as suboptimal intra- and interrater reliability, non-linearity, marginal sensitivity to change and bias to locomotor function have been described, the EDSS remains the gold standard to classify disability level and worsening in clinical trials (5).

While the EDSS is widely used in clinical trials, it is typically not available in most electronic health records (EHR) or administrative claims databases. This is a major challenge to real-world studies in MS relying on EHR or claims data, especially for the analysis of treatment patterns or related clinical and health-economic outcomes, as information on MS severity and disability level is essential to account for potential confounding and other biases. The estimation of disease severity using claims data is challenging due to missing severity measures (6–8). Administrative claims data provide a detailed comorbidity record and full capture of health care resource use and costs, however clinical information of disease severity is best captured in patient medical records or disease-specific registries. Quantifying the level of disability and disease progression among patients with MS observed in claims databases may improve real-world evidence research in MS, including the investigation of long-term benefits and risks of therapeutic options, optimal treatment utilization, disease behavior, as well as economic and cost-benefit evaluations (9).

In recent years, several studies have proposed approaches to estimate disability levels using claims or EHR data, consisting of algorithms that ranged from expert-led code mapping (6–8, 10–12), regression and machine learning models (13, 14), to more sophisticated deep learning-based natural language processing methods (15). However, only two studies used clinician-recorded EDSS scores as the reference standard for validation of the algorithms (12, 13), with model features derived from unstructured clinical notes, or indicators based on use of particular health care services, specific diagnostic codes, and codes based on employment or social security allowances. Unfortunately, these approaches may not be generalizable to all claims databases depending on data availability.

This study aimed to (1) develop an administrative claims-based proxy EDSS (pEDSS) using a comprehensive list of MS symptoms, treatments, as well as aids and remedies, (2) validate the algorithm against clinician-recorded EDSS scores obtained from a tertiary MS center in Germany, and (3) implement the algorithm to determine pEDSS among patients with MS in a large German sickness fund.

The EDSS is the gold standard for measuring disability and disease severity in MS and plays an important role in monitoring disease progression and informing clinical decisions (4, 26). However, in real-world data sources, EDSS scores are documented infrequently. In this study, we developed a rule-based algorithm to predict EDSS, using symptoms, medications, and aids recorded in administrative healthcare claims data from a large sickness fund (AOK PLUS) in Germany. The algorithm was then validated against clinician-derived EDSS data obtained from a large, specialized MS care center. While a number of groups have previously attempted to derive EDSS or disability in MS from real-world data, especially claims or electronic medical records (6–8, 10–15), only two previous studies followed a similar validation strategy using a clinical reference standard (i.e., true EDSS) (12, 13).

We built three different models, one that would allow an estimation of single EDSS steps (eight-fold classifier of EDSS scores 1.0–7.0, ≥8.0), and two predicting disability levels according to well-established EDSS categories (three-fold categorical classifier of low, moderate, or severe disability and binary classifier of severe or non-severe disability). When trying to estimate eight different EDSS steps, the pEDSS exhibited overall low precision and low sensitivity, particularly for pEDSS scores of 4.0 and 5.0, but high accuracy. An overestimation of the step 1.0 was also observed, with pEDSS 1.0 calculated for 42% of patients compared to 18% of patients with a true rescaled EDSS 1.0 (MSDS^3D). This is partly due to the imputation of pEDSS 0.0 values as pEDSS 1.0, given that EDSS 0.0 was not recorded for any of the patients in the MSDS^3D dataset. Overall, these observations suggest that mild to moderate symptoms and respective treatments are poorly captured in claims data, potentially reflecting low relevance for reimbursement purposes. In contrast, such symptoms are likely to be recorded in the medical records. The pEDSS 7.0 had the highest F1-score (0.50) reflecting the more complete recording of severe indicators in claims data, such as symptoms requiring immediate medical attention and ambulatory aids relevant for reimbursement (6 of 7 true patients with a wheelchair were captured). However, the pEDSS 6.0 had a very low F1-score (0.18), which resulted from use of walking sticks not being captured for all patients requiring unilateral or bilateral assistance, potentially due to relative accessibility of such aids outside of the insurance reimbursement system.

Similar to our work, two previous studies attempted to derive multiple EDSS steps from real-world sources. One study in Canada linked administrative data to a large clinical dataset to develop a regression-based algorithm that predicted the EDSS (including 0.5 increments) as a continuous measure. The best performing model explained 40% of EDSS variation (pseudo-R² 0.40) with a MSE of 2.09, which is consistent with the MSE of 2.6 observed for our algorithm (12). Our work affirms the challenges with deriving a granular EDSS proxy, largely attributed to claims data coding practices of relevant signs and symptoms of lower severity. Another study used a natural language processing model that combined a rule-based approach with a deep learning model for extracting and/or deriving EDSS scores from the records of patients with MS. In almost two thirds of cases, the model worked by extracting the exact EDSS score annotation and, not surprisingly, this resulted in a macro F1-score of 0.90. However, when the same model was applied to the clinical notes without an explicit EDSS score, the performance was much lower with a macro F1-score of 0.39, which is similarly low to the macro F1-score of 0.25 observed for our eight-fold model (15).

When using the algorithm to estimate broader EDSS categories, we observed that EDSS scores 4.0 and 5.0 were poorly classified and underestimated (precision/PPV 0.38, F1-score 0.44), whereas EDSS scores 1.0–3.0 and EDSS ≥6.0 showed higher precision, with more than 85% of cases correctly predicted. The lower precision observed for pEDSS 4.0–5.0, can be partly explained by the non-linear properties and the bimodal distribution of EDSS, which is reflected by patients staying for the shortest time in the middle scores (4.0–5.0) and peaks at 1.0–3.0 and 6.0–7.0 (27). As PPV is a metric that depends on the pre-test probability (i.e., probability of presence of disease state before the measurement) (28), the lower the prevalence of certain EDSS levels, the lower the PPV (and higher the NPV) will be.

The best performing classifier was a binary assessment of EDSS ≥6.0 vs. EDSS <6.0, resulting in a sensitivity 0.65 and a PPV of 0.85 for EDSS ≥6.0 (NPV = 0.93 and F1-score = 0.73), compared to a sensitivity of 0.98 and a PPV of 0.93 for EDSS <6.0 (NPV = 0.85 and F1-score = 0.95). Notably, the EDSS combines two distinct scales, whereby EDSS <6.0 is reflective of sign and symptoms based on the FS and EDSS ≥6.0 reflecting ambulation status. With challenges in the coding of relevant signs/symptoms in claims data, the binary classifier may be most useful in describing the overall ambulation status at population level. Our model shows a better performance compared to other models previously reported. Alves et al. (13) developed a machine learning model that estimated a numeric EDSS score at a specific encounter date based on clinician notes from the medical records. The model was able to estimate EDSS ≥6.0 with a PPV of 0.85 and NPV of 0.85 (13). In the Canadian study already discussed above, Marrie et al. (12) reported a sensitivity of 0.49, a PPV of 0.72, and a maximum Kappa coefficient of 0.55 (our binary model achieved a kappa of 0.69) for predicting an EDSS ≥6.0 (12).

It should also be noted that the application of our algorithm to the wider AOK PLUS cohort, resulted in a typically bimodal distribution of EDSS (27). Moreover, older patients, patients with progressive forms of MS and those with higher comorbidity burden showed higher pEDSS values which is consistent with the MS epidemiology (29, 30). This further reinforces the validity of our algorithm.

Our study has multiple strengths. While the eight-fold classifier performed poorly, our model using three and two category groupings of disability showed good to high predictive performance, and their practical utility was demonstrated in a large MS population. The development of our model was an interdisciplinary effort, involving clinicians, epidemiologists and data scientists with vast experience in MS and real-word research. This allowed to create a rule-based algorithm with comprehensive information on symptoms, medications, and aids. Most importantly, we followed a validation strategy using the EDSS derived by clinicians as the reference standard. Several studies have previously developed algorithms to assign MS disability levels based on observable claims data or EHR data sources, however a formal validation was not possible due to lack of true EDSS measures (6–8, 10, 11, 31–33).

We acknowledge some limitations. The validation cohort included only 100 patients, with each patient able to contribute multiple EDSS measurements across the follow-up period (620 measures in total). Patients were insured by AOK PLUS in the regions of Saxony/Thuringia and receiving care at a single MS center. While the results are likely generalizable to Germany, given that uniform healthcare regulations and standard clinical practices are imposed nationally, they may not be generalizable to other countries. There was also an imbalance in the true EDSS distribution, with a bias toward lower EDSS levels (1.5 to 4.0) and fewer patients with EDSS ≥7.0. A number of factors contributed to this, namely slow recruitment rates (patients were recruited based on regularly scheduled visits at ZKN), mismatch of data coverage timelines between the two datasets, and data linkage issues. It is possible that the performance of the algorithm would decrease if the population had different characteristics. However, the binary model that separated the cohort into patients with EDSS <6.0 and EDSS ≥6.0 had excellent precision and sensitivity which suggests that the algorithm is robust. It should also be noted that, as data used in this study come from standard clinical practice, there may be a small degree of miscoding, missing, or incorrect entries. Despite this, claims data are a valid source of real-world evidence and systems are in place to ensure quality of MSDS^3D data entries. Finally, we followed a rule-based approach informed by clinical input to develop our algorithm, an approach that could be biased. Machine learning models, and other more sophisticated deep learning-based natural language processing methods are promising alternatives (13, 15). However, as discussed above, the performance of these models was overall inferior to our algorithm, which indicates that further work is required. Insurance claims are also probably unsuitable to estimate low EDSS scores as the relevant information will likely be recorded in the clinical notes in the EHR. A combination of rule-based and machine-learning models using data from both insurance claims and the medical notes is likely to yield the best results, and we recommend that this should be an area of active research.

In summary, we developed and validated a rule-based proxy EDSS algorithm for estimating disability status using claims data, with a model for two and three EDSS categories showing good-to-high performance. We highlight the need for creating and maintaining linked databases such as the one used for this validation study to leverage the strengths of different real-world sources. Our study is another step forward in quantifying the level of disability and disease progression among real-world patients with MS observed in claims databases, and in turn improving real-world evidence research in MS.

The datasets presented in this article are not readily available because of ethical and privacy restrictions. Requests to access the datasets should be directed to the corresponding author.

The studies involving humans were approved by the Ethics Committee of the University of Dresden and the Ministry of Saxonia (SGB § 75). Patient written informed consent was obtained from all patients whose retrospective data was used for validation prior to linking patient data from MSDS^3D to claims records from AOK PLUS. The studies were conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation among patients described in the broader AOK PLUS cohort was not required from the participants or the participants' legal guardians/next of kin in accordance with the national legislation and institutional requirements.

EM-L: Conceptualization, Methodology, Supervision, Writing—original draft, Writing—review and editing. MG: Data curation, Formal analysis, Methodology, Project administration, Supervision, Writing—original draft, Writing—review and editing. EZ: Formal analysis, Methodology, Project administration, Writing—original draft, Writing—review and editing. AD: Methodology, Resources, Writing—review and editing. UM: Methodology, Resources, Writing—review and editing. TW: Methodology, Supervision, Writing—review and editing. TZ: Methodology, Resources, Supervision, Writing—review and editing. LC: Conceptualization, Methodology, Supervision, Writing—original draft, Writing—review and editing.