Natural language processing (NLP) is a growing area of research that aims to develop algorithms and models capable of processing and decoding unstructured, human-generated text. In recent years, there has been interest in using NLP to analyze unstructured data within Electronic Health Record (EHR) systems (1). Unlike structured data, which consists of data organized into predefined data fields, typically categorical or numerical values, unstructured data consists of free text or fields without a predefined format. Although structured EHR sections contain most health information for individuals, often defined by standardized data elements such as ICD-10-CM diagnosis and procedure codes, a significant proportion of this information is contained within unstructured sections, such as physician notes (2). This poses a challenge to researchers who rely solely on structured EHR sections for data analysis. We developed NLP methods to extract critical data from text-based narrative sections of the patient's record, using immunization¹ administration as a specific use case.

NLP methods take various forms and can employ a rule-based approach, machine learning (ML) models, or other artificial intelligence (AI) techniques, depending on the research needs of a specific project (2). In a rule-based model, experts develop a set of keywords and a series of rules that contextually detect terms in a sentence. In contrast, an ML model uses a preexisting training dataset to predict criteria or labels in a test dataset (3). Rule-based systems have been widely implemented for NLP tasks, such as computational phenotyping and clinical decision support; rule-based systems have been successfully developed for information extraction (3–9). Notably, Deady et al. (10) employed rule-based NLP methods to extract influenza vaccination data from clinical notes, resulting in a 16.8% increase in captured vaccinations.

Researchers have applied rules and model-based NLP methods across a range of therapeutic areas, including identification of bleeding events (11–13, 37), chronic cough (14), surgical site infections (15), prediction of falls (16), back pain (17), Hepatitis C (18), and a range of mental health issues (19, 20). The application of NLP methods has been valuable in situations where traditional diagnosis codes do not capture the disease of interest, such as the identification of diseases in undiagnosed populations and in situations where ICD 10-CM diagnosis and procedure codes are not specific enough to accurately identify the disease population (14, 18, 38). NLP methods have also been effective in identifying early predictors and longitudinal progression of disease and provide an alternative to labor-intensive chart review (16, 18). Rule-based methods are popular and effective because they do not require labeled training data (11, 14) and are able to algorithmically detect modifiers, such as negation, self-diagnosis, and affirmation (14, 16, 19). Methods range from relatively simple regular expressions to algorithms that incorporate semantic mappings, grammatical structure, and clinical terminologies (15). Rule-based methods have also been used in combination with ML approaches, including as a preprocessing step to generate labeled training data (17) and for identification of modifiers (12).

The success of previous studies using a rule-based NLP model to extract immunizations from text-based EHR sections guided our research efforts. Moreover, an analysis of NLP research on unstructured EHR data revealed that rule-based NLP models significantly outperform ML approaches while also providing increased simplicity, transparency, and interpretability (8). During an exploratory phase, preceding the extraction of EHR study data, we developed preliminary models based on publicly available EHR data using the Medical Information Mart for Intensive Care, version four dataset (MIMIC-IV) (9). Compared to a less malleable ML-based model, a rule-based approach provided increased flexibility in further developing our approach in this vaccine administration setting.

Respiratory viruses, such as SARS-CoV-2, influenza, and respiratory syncytial virus (RSV), remain persistent contributors of morbidity and mortality within the United States (21–23). As such, tracking data related to immunization coverage, effectiveness, and severe disease outcomes by immunization status are crucial in supporting public health efforts targeting these viruses (24, 25, 27). The need for a rapid public health response increases the reliance on timely and accurate EHR data. Vaccine effectiveness (VE) estimates support public health programs and policies by routinely collecting and synthesizing data on respiratory illnesses and associated outcomes in vaccinated and unvaccinated subjects, and publishing timely statistical analyses (26, 28–30). As pandemic-era regulations requiring providers to report immunization data to state and local immunization information systems have expired, there may be an impact on the reliability and completeness of immunization history data. Furthermore, structured EHR data may not reflect immunizations received in locations outside of healthcare networks, including unaffiliated clinical practices, pharmacies, and workplaces (10). Due to the importance of consistent and reliable immunization data for estimating VE and assessing immunization coverage, we investigated and developed methods to extract additional immunization data from relevant text-based EHR sections for study participants.

This study specifically examines the use and optimization of a rule-based NLP algorithm to identify and characterize immunization events in unstructured EHR sections, and broadly considers the application of a rule-based NLP algorithm to capture other measures and characteristics that are not reliant on ICD-10-CM diagnosis and procedure codes.

We developed rule-based NLP methods to enhance the identification of COVID-19, influenza, and RSV immunizations from clinical notes, with the goal of supplementing structured EHR data and thereby improving immunization estimates. Based on manual review results, the algorithm identified 97.2% of COVID-19 vaccinations, 92.3% of influenza vaccinations, and 95.6% of RSV immunizations among immunizations recorded in progress notes, with few false positives and slightly more false negatives. Precision varied (70% for COVID-19, 91% for influenza, and 53% for RSV). For COVID-19 and influenza specifically, high precision indicated that when we identified vaccinations in the record, there is a higher probability of being correct. Similarly, high recall suggested that we can identify a high proportion of immunizations in the notes (97% for COVID-19, 92% for influenza, and 96% for RSV). Differences in detection rates across subgroups suggest that our algorithm can identify disparities in how immunization history is documented within structured EHR fields. Conversely, the comparison between immunizations detected by the algorithm and immunizations that are already present in structured data suggests the unstructured text contributed little additional vaccine information beyond what is already captured in the structured data, with potential increases in vaccination coverage limited to 1.7% for COVID-19, 2.7%, for influenza, and 0.5% for RSV. This reinforces our assumptions around the clinical workflow and our understanding of how immunization data are added to unstructured notes. These results are explained by the low recall when treating the structured data as the gold standard, which suggests that there are many more vaccinations in the structured data than are represented in the notes. Taken together, these results suggest that while the algorithm performs well in identifying instances of immunizations in the text, there were relatively few to be found in this application.

As with prior studies, our algorithm performed well and was able to identify relevant clinical terms (i.e., vaccine administrations) when they were present in the data; however, the utility of the algorithm was dependent on the prevalence of those terms in the clinical notes. In contrast to our approach, Deady et al. (10) applied a simpler algorithm limited to the detection of COVID-19, which resulted in a higher number of additional vaccinations being identified (16.8%) despite overall lower performance. We hypothesize that this was due to more robust recording of vaccination in the structured data in the intervening time since that work was done, but this assumption requires more detailed investigation. Similar levels of performance have been achieved in other therapeutic areas. Bucher et al. (15) achieved 95% sensitivity and 86% specificity when using a rules-based approach to identify post-operative surgical site infections. Comparable performance was achieved by Taggart et al. (11), with sensitivity of 85% and positive predictive value of 63%, while Murff et al. (35) in an earlier study achieved slightly lower performance with 77% sensitivity and 63% specificity. Paterson et al. (16) achieved a high level of performance in identifying falls without chart review with precision of 92% and recall of 95%. Rule-based methods have also been shown to be effective in identifying broader mental health and social conditions. Gray et al. (20) identified patients with residential stability issues with 92% precision and 84% recall, while Chandran et al. (19) were successful in disentangling obsessive compulsive disorder from related comorbidities, with 77% precision and 67% recall. These results are consistent with our finding that rule-based NLP methods are effective in identifying clinical terms when they are present in the EHR record.

The performance of our method can be attributed to several key innovations. Our method used grammatical dependencies to understand the relationships between and functions of words in complex sentences, where traditional keyword-based approaches might fail. This allowed us to identify immunization administration in sentences of arbitrary length where the indicator that an immunization was given could be relatively distant from the name of the virus. This advancement proved valuable in accurately interpreting negations and distinguishing between recommended vs. administered immunizations. The effectiveness of our method was further validated by comparing immunizations detected by the algorithm against immunizations recorded in structured data in the EHR. Across the full 20,000 patient sample, the NLP algorithm identified 315 COVID-19 vaccinations, 503 influenza vaccinations, and 47 RSV immunizations that were not captured in the structured data. Due to anticipated differences in immunization uptake across demographic groups, we stratified by immunization status, presence of test result, and immunosuppressed status when calculating immunization detection rates across the full sample. This process enabled us to generate a sample for each immunization that accurately reflected the underlying population, helping us assess the algorithm's performance in a real-world setting. For example, among patients with no evidence of immunization in the structured EHR data, the algorithm detected a higher proportion of influenza vaccinations in immunocompromised patients and older adults, suggesting that it was effective in identifying differences in immunization administration under-capture when they were represented in the clinical notes.

While we have demonstrated our method's performance in identifying immunizations in clinical notes that were not included in structured EHR data, we have not formally evaluated how this performance varies across key analytic variables, including demographics. We hope to perform that evaluation and combine estimates with detection rates across the same variables to estimate the level of completeness in immunization records. One limitation is that most immunized individuals do not have immunization references in clinical notes; however, in addition to supplementing structured EHR data, results on the proportion of patients with immunization references in both notes and structured records can be used to adjust estimates of missingness among patients without immunization references in structured notes. Further research is needed to investigate the impact on VE estimates and explore ways to mitigate biases in these estimates. Failure to capture all vaccine administrations means that some vaccinated individuals would be wrongly included in unvaccinated reference groups during analyses, with the expected effect of underestimating vaccine effectiveness. We also performed preliminary work on identifying the date of immunization; however, due to the appearance of multiple dates in a highly unstructured format, the performance was variable. The accuracy of extracted dates ranged from 20.8% for influenza to 66.7% for RSV. The date of immunization can be a crucial variable in analyses; however, further research is needed to identify administration dates accurately.

In this research, we have developed a generic, keyword-based algorithm that can be applied to multiple clinical data extraction tasks. While our approach has been effective in identifying vaccine administrations when present in clinical text and has addressed some of the limitations of prior methods, more work is needed to extract more complex entities, such as the date of immunization. We have only evaluated it at a single participating institution and applied it solely to identifying immunization characteristics; therefore, its performance remains unclear in more general settings. Future work could apply the algorithm across multiple sites and expand to include other patient characteristics. Other emerging methods that utilize generative AI may capture more complex grammatical structures and semantic similarities necessary to isolate nuanced factors associated with immunization (e.g., date of administration, manufacturer) (36). Therefore, additional work is also needed to quantify and validate the results across different demographics to assess potential bias. If this can be accomplished, our method can be used to estimate missingness in the structured EHR data, which would enhance broader EHR-based analyses.

Due to our specific focus on vaccine administration, we limited our approach to analysis of clinical notes. For more general applications, a potential enhancement would be the incorporation of a broader range of data, such as lab values, imaging studies, and prescription records, which would have the potential to support more complex, multi-modal modelling approaches. Our methods would have more utility if the quality and consistency of clinical documentation was improved, especially for clinical conditions and events that are not well represented by ICD-10-CM or CPT codes, thus suggesting an opportunity to engage with clinicians to improve documentation protocols. A key advantage of our approach is that implementation within the clinical setting is relatively straightforward, requiring only the identification of terms that are typically well-represented in the notes (e.g., disease symptoms), defining of key words through clinical consultation, and the application of the algorithm to the clinical data warehouse. In conclusion, we have shown that rule-based NLP methods are effective in identifying vaccine administrations from clinical text. Due to the ease of implementation they could have utility in general clinical data extraction tasks, especially those that are not reliant on ICD-10-CM diagnosis and procedure codes, which are easily captured in the structured EHR, such as in the identification of adverse events, transfers or referrals, use of treatments and medications, and more nuanced descriptions of disease symptoms.