We conducted this study in the central Canadian province of Manitoba, which has a population of ~1.33 million persons. In Manitoba, health care is universal and publicly funded for medically necessary services. Health system contacts are captured in administrative databases. Specialty MS care, including prescriptions for disease-modifying therapies, is provided through one provincial MS Clinic and is captured in the MS Clinic Registry. Ethics approval was provided by the Health Research Ethics Board of the University of Manitoba. Approvals for data access were granted by the Health Information Privacy Committee and the Winnipeg Regional Health Authority.

We used clinical and administrative data sources. The Winnipeg MS Clinic was established in 1998 and serves the entire province of Manitoba, delivering care to ~2,700 people with MS annually. All Manitobans who wish to receive provincially-funded disease-modifying therapies are re-evaluated at least annually. In May 2011, the MS Clinic established a registry; >2,100 persons are participating (89% of those approached) and have agreed to linkage of their clinical and administrative data for research purposes. The registry can be linked to the administrative dataset using the personal health identification number. The registry contains data on disability measured using the Expanded Disability Status Scale score (EDSS) (10) and the corresponding dates of assessment, recorded during routine MS clinic appointments by EDSS-certified neurologists. The EDSS is an ordinal measure of disability based on the neurological examination. Total EDSS scores range from 0 (no disability) to 10 (death due to MS), and are derived from scores on visual, brainstem, pyramidal, sensory, cerebellar, sphincter, and cerebral functional systems, as well as an observed walk of up to 500 meters. An EDSS score of 6.0 indicates the need for unilateral assistance to walk, while a score of 6.5 indicates the need for bilateral assistance, and 7.0 indicates the need for a wheelchair. EDSS scores of >5 are associated with impaired activities of daily living (11).

We used administrative datasets held in the Population Health Data Repository at the Manitoba Centre for Health Policy. When selecting the datasets to be used, we considered their utility for identifying individuals with disability and whether they were also available in other Canadian provinces to allow widespread application of our case definition. The datasets used included the population registry, Discharge Abstract Database (DAD), medical services (physician) claims, Drug Program Information Network (DPIN) database, National Rehabilitation Reporting system (NRS), long-term care database, home care database, and employment and income assistance (EIA)/Social Allowances Management Information (EIA/SAMIN). The information used from each dataset is described further below.

The population registry provided sex, dates of birth, death and health care coverage, and postal code. The DAD captures hospitalizations including dates of admission and separation. Before 2004, diagnoses were recorded using International Classification of Disease 9^th edition, clinical modification (ICD-9-CM) codes. From 2004 onward, they were recorded using ICD-10-Canadian version (CA). Medical services provided the date and type of service, and one physician-assigned diagnosis recording using a 3-digit ICD-9-CM code. DPIN records all community-dispensed prescriptions including the date, drug name and drug identification number (DIN). The NRS provided admissions for physical rehabilitation including admission dates and discharge dates. The long-term care database indicated the date of all personal care (nursing) home admissions. In Canada, such admissions are generally permanent rather than for rehabilitation purposes. Over 70% of persons with MS in Canadian long-term care settings use wheelchairs, two-thirds have bowel incontinence, and 43.5% have difficulty swallowing (12), making this setting a potentially specific indicator of disability. The home care database provided the dates for opening and closing of home care services which allowed the determination of the number of days with an open home care file. To be eligible for home care individuals must have limitations in their activities of daily living; 48% of persons with MS in this setting use a wheelchair and one-third have urinary incontinence (12). This database is population-based for individuals living in the Winnipeg Regional Health Authority, where 70% of the Manitoba population resides. EIA is available to Manitobans with no other means of financial support. Individuals with a mental or physical disability likely to last >90 days and which prevents the individual from earning enough money to money for basic financial needs are eligible to apply. An indicator of physical disability can be recorded with values of hearing, illness, mobility/coordination-no wheelchair, mobility/coordination-wheelchair, other, speech and vision. These datasets were linked using an encrypted unique identifier to crate the administrative dataset. Using postal code, this dataset was linked to census data from Statistics Canada to determine household income of each individual's enumeration area (13), as a measure of socioeconomic status (SES).

Using hospital, physician and prescription claims for the period April 1, 1984 to March 31, 2017, we identified Manitobans with MS using a validated case definition (14, 15). This case definition required an individual to have at least three health care encounters for MS, and has a sensitivity of 99.5%, specificity of 98.5% and positive predictive value of 99.5%. The date of the first demyelinating disease claim (e.g., optic neuritis) was designated as the index date (Supplementary Table 1) (14). This constituted the prevalent MS cohort. For the development of the disability case definition we limited this prevalent cohort to adults aged ≥18 years, alive and living in Manitoba during the period April 1, 2014 to March 31, 2017. This reflected the most recent data available at the time the study was approved. We then linked this cohort to data from the MS Clinic Registry for consenting participants. This constituted the “disability” cohort.

Health care use outcomes included: annual ambulatory physician visit rates and annual hospitalization rates due to any cause; and due to infection specifically. Infections were identified using ICD-9/ICD-10 codes listed in Supplementary Table 2, consistent with prior research in British Columbia, Canada (9).

Covariates included sex (female as reference group), current age (continuous), age at the index date (continuous), year of diagnosis (1984–1989 [reference], 1990–1994, 1995–1999, 2000–2004, 2005–2009, 2010 onward), SES, region, comorbidity, and use of disease-modifying therapy. We categorized SES into quintiles (lowest quintile of SES as reference group) at the index date. Regions were classified as urban (Winnipeg, population >600,000 and Brandon, population >47,000) or rural. Comorbidity was captured using a modified version of the Charlson score (categorized as 0, 1, ≥2) to avoid misclassifying symptoms or signs of MS as comorbidities (16). MS disease-modifying therapy (DMT) use was identified using drug identification numbers (DINs) for therapies available during the study period (Supplementary Table 3 and was classified as any vs. none, updated annually.

We randomly split the “disability” cohort into discovery and validation cohorts. For each of the indicators we identified the closest EDSS score in time. Due to small numbers of individuals with some of the indicators we allowed the interval between the measurement of the EDSS score and subsequent assessment of the indicator to be up to 365 days. In a complementary analysis we allowed EDSS measurements to occur 365 days before or after the measurement of the indicator. For individuals who did not have a particular indicator and therefore no time point of reference we used the EDSS score closest to the study midpoint. Then we reported the minimum, maximum and median (IQR) EDSS scores at that time. Based on these findings, and the proportion of scores which could be classified as mild (0–3.0), moderate (3.5–5.5) and severe (6.0–9.5) (18), we subsequently categorized disability as severe (EDSS 6.0–9.5) or not severe (EDSS <6.0) and used this as the reference standard for assessing performance of the case definitions. For each indicator, we determined the sensitivity, specificity, positive predictive value (PPV) and negative predictive values (NPV), and agreement (as measured using a kappa statistic) (19) as compared to the severe disability variable. We conducted this analysis in the discovery cohort, then repeated it in the validation cohort, as a means of internal validation; we report the findings in the validation cohort. Our goal was a PPV ≥ 70% (20).

Since algorithms designed in this fashion may not capture interactions between component indicators and non-linear effects, we also applied statistical approaches to see if superior case definitions could be identified. After considering several approaches including classification trees, random forests and support vector machines we opted to use logistic regression because it has the greatest clinical interpretability. Logistic regression models the linear association between the variables of interest and the log-odds of the outcome, in this case severe disability. Initially, we considered all possible indicators. We also included age (continuous variable) since it is strongly associated with disability severity in MS (21). As models are more data intensive for this analysis we used the entire sample and used bootstrapping for internal validation and to avoid overfitting related to testing multiple models in a small sample (22). Specifically, we trained the logistic regression on the entire sample and then recorded the area under the receiver operating characteristic curve (AUC, referred to as the naïve C-statistic). We then drew 500 bootstrap samples with replacement from the original sample and trained the same logistic model on each sample to obtain 500 AUCs (C-statistics). The mean of the differences between the original AUC (naïve C-statistic) and those computed from each of the bootstrap samples is known as optimism. The difference between optimism and the naïve C-statistic is called the optimism-corrected C-statistic (23). This statistic gives an idea of how the model would perform in a dataset that it was not trained on. Given the unbalanced distribution with respect to severe disability, we also repeated these models in which we randomly under-sampled the larger class. This can improve performance of classification algorithms (24), and may produce superior performance to other methods. We assessed model discrimination using the AUC, classification accuracy, sensitivity, specificity, positive and negative predictive value.

Given the bounded distribution of the EDSS (0–10), and to ensure that predicted values did not fall outside the range of possible values we used truncated regression models to predict the EDSS. Because an EDSS of 10 indicates death, and we were interested in disability status among living individuals we set the upper bound as 9.5. The variables used in this approach were based on those selected using the bootstrap-validated logistic regression of the Disability-Sev outcome, given the high values of the naïve C-statistic and the optimism-corrected C-statistic. We assessed assumptions of normality using quantile-quantile plots. We assessed overall model fit using a pseudo-R² estimated as the squared correlation of the observed and predicted values of the EDSS (25), and mean square error. We also report the mean difference between the observed and expected values, and used Bland-Altman plots to further evaluate agreement between the observed and expected values (26).

Relapses may be associated with transient increases in EDSS. Our clinical dataset lacked information regarding relapses but we applied a previously validated case definition for severe (treated) relapses, and repeated our regression analyses for algorithm development after excluding affected individuals as a sensitivity analysis (4).

After we selected our preferred algorithms for disability, we tested their association with health care use, using the entire linked clinical-administrative dataset. First, we report annualized hospitalization rates and physician visits and overall, and due to infection.

Second, we tested the association between severe disability, based on the clinically recorded EDSS (hereinafter c-EDSS, updated annually), and hospitalization rates using zero-inflated Poisson regression models using generalized estimating equations (GEE) to account for dependence of observations within individuals. We modeled physician visits using negative binomial regression model with GEE. Both models included as an offset the number of person-days the patient was resident in MB, to account for differing follow-up time between individuals. Covariates included sex, age, SES (updated annually at the beginning of the fiscal year), year of diagnosis, disease duration, comorbidity (updated annually at the beginning of the fiscal year), and use of disease-modifying therapy as previously defined. Then, we repeated the analysis substituting the preferred algorithm for severe disability (Disability-Sev) for the c-EDSS dichotomized at 6.0, expecting to find similar associations between the c-EDSS and health care use as between the disability algorithms and health care use. We used a similar approach to testing the second algorithm for disability (Disability-Cont) in which we included the c-EDSS as continuous then substituted Disability-Cont for c-EDSS. Finally, we categorized the Disability-Cont as mild (0–3.0), moderate (>3.0 and ≤5.5) and severe (>5.5–9.5). We report rate ratios (RR) and 95%CI for the observed associations.

Statistical analyses used SAS V9.4 (SAS Institute Inc., Cary, NC).

We identified 3,703 individuals with MS who were alive and living in Manitoba during the study period. Of these, 1767 had linkable clinical data. The characteristics of the linked and unlinked cohorts are shown in Table 1. Most individuals with MS were women and two-thirds lived in urban areas. The median number of EDSS scores available during the study period was three. Among the 1,419 participants with more than one EDSS score during the study period, 747 (52.6%) had the same score at all time points, and 298 (21.0%) had a subsequent EDSS score within ±0.5 of the prior EDSS score. At the beginning of the study period 392 (22.2%) of participants had an EDSS score ≥6.0. Overall, 143 (8.1%) individuals were treated for 204 relapses during the study period.

During the study period nearly 15% used home care services, fewer than 10% used EIA, and fewer than 2% were admitted for rehabilitation or were admitted to long-term care (Table 1). The proportion of individuals using these services was higher when we considered whether they were ever used, that is, if they were used before or during the study period. The median (IQR) EDSS for individuals using home care services was 6.5 (6, 7.5), admitted to long-term care was 7 (6.5, 8), and discharged after a rehabilitation admission was 6.5 (4, 7.5). Although an indicator of physical disability can be recorded in the EIA dataset this information was missing for nearly half of individuals (49.3%), and the mobility codes which would map well to the c-EDSS were only reported for 5.6% of individuals ever using EIA. Therefore, we did not move forward with using this indicator. Most of the diagnostic codes for symptoms commonly associated with MS and captured in some way in the c-EDSS were used infrequently or not all. Only diagnoses of visual disturbance and other paralytic syndromes were used in over 5% of the members of the linked and unlinked cohorts during the study period. These findings did not change when we altered the intervals between diagnosis codes to ≥60 days or ≥90 days (Supplementary Table 4). The median (IQR) for individuals with visual disturbance codes was 2 (1.5, 4), and for other paralytic syndromes was 6.25 (5.5, 7.5).

All of the individual indicators tested had specificities exceeding 90% for severe disability (c-EDSS ≥6.0), and all but one (visual disturbance) had PPV exceeding the 70% threshold (Table 2). Negative predictive values exceeded 70% but sensitivities were generally low; the highest sensitivity was for home care use (58.2%). Combining indicators improved sensitivity at the expense of lower specificity and lower PPV. For example, the combination of home care or long-term care use or rehabilitation admission had a sensitivity of 61.9%, specificity of 90.76%, PPV of 70.06% and NPV of 87.21%. Findings were similar when we used indicators requiring diagnostic codes to be ≥60 days apart (Supplementary Table 5). Findings were similar when we allowed c-EDSS to be measured within 365 days before or after the indicator (data not shown).

When we applied the logistic regression approach several indicators were relevant for prediction, including home care use, long-term care admission, rehabilitation admission, visual disturbance and spasticity. As the findings from logistic model most readily translate into an interpretable algorithm that can be applied in future studies we present those findings. Based on the preferred logistic model (see Table 3, Model A), at a predicted probability ≥0.25 the sensitivity for c-EDSS ≥6.0 was 49%, specificity 93.8%, PPV 72%, and NPV 85%. The naïve C-statistic for this model was 0.844 with an optimism value of 0.0024 and optimism-corrected C-statistic of 0.842. After excluding individuals with a treated relapse, the regression equations were similar (Supplementary Table 6).

Based on our truncated regression models, the best-performing model for predicting the EDSS as a continuous variable included age, home care use (ever), long-term care admission (ever), hospitalization for rehabilitation (ever), visual disturbance, other paralytic syndromes and spasticity (Table 4). This model explained 40% of the variation in EDSS scores, and the mean difference between observed and predicted values of the EDSS was −0.0644 (95%CI −0.1632, 0.0304). The model tended to overestimate EDSS scores at the lower end of the range as evidenced by the model intercept which exceeded 0. The Bland-Altman plot is shown in Supplementary Figure 1, and shows some outliers. After excluding individuals with a treated relapse, the regression equations were similar (Supplementary Table 7).

Using the linked clinical dataset, annual rates of physician visits and hospitalizations for any cause, and specifically due to infection over the study period are shown in Supplementary Table 8. Overall, 376 (21.3%; 95%CI: 19.4–23.2%) of individuals was hospitalized at least once, and 117 (6.6%; 95%CI: 5.5–7.8%) were hospitalized for infection at least once. The mean (SD) number of physician visits over the study period was 24.2 (18.9) for any reason, and 1.4 (2.3) for infection; 878 (49.7%) of individuals had at least one physician visit for infection.

Without adjusting for covariates, severe disability (clinical EDSS ≥6.0) was associated with increase rates of physician visits (RR 1.12; 95%CI: 1.02, 1.22). After adjusting for age at diagnosis, diagnosis year, sex, SES, Charlson comorbidity score, disease duration and use of DMT, the association between severe disability and physician visits was attenuated for any reason or for infection, but was associated with an increased rate of hospitalizations due to any cause and due to infection (Supplementary Table 9, Figure 1). The associations of hospitalizations due to infection with severe disability are extreme with broad confidence intervals due to the small number of events. After adjusting for covariates, as a continuous variable, the c-EDSS was also associated with an increased rate of hospitalizations and physician visits due to any cause and due to infection.

When we substituted the Disability-SEV or Disability-Cont into the model, the associations between disability and health care use were similar. Similarly, when we substituted the categorized version of Disability-Cont into the model the findings remained similar to those when mild, moderate and severe c-EDSS categories were included.