Large pre-trained language models are increasingly used in healthcare care, showing promise in performing a variety of complex natural language processing (NLP) tasks, such as text summarization, concept recognition, and answer questions (1–3). Large language models can identify medical concepts in the text and normalize them to an ontology (4–7). However, when large language models normalize medical terms to a standard ontology such as the human phenotype ontology (HPO), the retrieved code is not always accurate.

Shlyk et al. (8) demonstrated that the accuracy of large language models in term normalization tasks can be improved with Retrieval-Augmented Entity Linking (REAL). This method generates definitions of HPO terms and target terms needing normalization. The definitions are converted into word embeddings, and cosine similarity is used to identify the three closest candidate terms. The large language model then selects the best normalization from these candidates.

In this work, we introduce a simplified but effective retriever that bypasses the need for definition generation. Instead, it matches HPO terms to target terms using BioBERT contextual word embeddings, identifying the closest matches by semantic similarity. By prompting GPT-4o with the 20 closest candidate terms, we enable the model to take advantage of its implicit knowledge of HPO terms and select a semantically equivalent normalization. This approach achieves accuracy comparable to more complex methods without using explicit definitions.

Phenotyping, which involves recognizing the signs and symptoms of disease in patients and mapping them to an appropriate ontology such as HPO, is critical for precision medicine (9–13). Manual phenotyping is labor intensive, which makes high-throughput automated methods essential (11, 14–18). Phenotyping can be seen as part of the broader task of term normalization in different vocabularies, such as drugs to RXNORM (19), diseases to ICD-11 (20), or laboratory tests to LOINC (21).

A distinction can be made between surface phenotyping, which assigns a diagnosis from a disease ontology (22), and deep phenotyping, which assigns an HPO concept and ID to each symptom (11). Our work focuses on deep phenotyping (17).

Although often performed together, concept extraction (identification) and concept normalization are distinct NLP tasks. In concept extraction, the goal is to find relevant medical concepts within free text. Concept normalization involves matching irregular medical terms to standardized terms in an ontology and their corresponding machine-codes. If the irregular term to normalize has an exact match in the ontology, the task is straightforward such a matching “hyporeflexia” to its standard form in the HPO which is “Hyporeflexia.” If the irregular term has no exact match (e.g., “diminished reflexes”), the model must select the most semantically similar concept (e.g., “decreased reflexes”). Additionally, concept normalization involves mapping irregular terms to their standard forms in an ontology and their corresponding machine codes (“hyporeflexia,” HP:0001265).

Advances in deep learning, transformer architectures, and dynamic word embeddings have facilitated the development of tools that perform concept identification and normalization such as Doc2Hpo, ClinPheno, BERN2, PhenoBERT, and FastHPOCR (23–30). Evaluations of concept recognition tools show F1 values ranging from 0.50 to 0.76, with FastHPOCR performing the best on manually annotated corpora (29, 31).

Although deep learning and transformer-based methods have shown promise, they require extensive training data, which can be time consuming to acquire. Pre-trained large language models offer an alternative approach by eliminating the need for new training data. Despite their impressive performance, large language models may still make errors in retrieving the correct HPO ID (32).

Retrieval-augmented generation (RAG) (33) addresses this problem by using a retriever to provide relevant information, improving the likelihood of generating accurate output (34). In this paper, we extend the work of Shlyk et al. (8) by demonstrating that a simplified retriever, which relies on embedded terms, can improve normalization accuracy without the need to generate embedded term definitions. By prompting a large language model with candidate terms that have similarity to the target term, we achieve high accuracy with a more efficient approach.

Table 1 shows the model accuracies for the phenotype normalization of the 1,820 terms to normalize. To be rated as “accurate,” the term to normalize had to be semantically equivalent to the HPO entry term and and the model had to retrieve the term’s correct HPO ID.

The spaCy and BioBERT methods used word embeddings and NLP algorithms to find the best match in a complete table of HPO terms. The spaCy embeddings were general-purpose word embeddings, whereas the BioBERT embeddings were optimized for biomedical terminologies. spaCy averaged the vectors of the component tokens to get a global term vector, whereas BioBERT utilized a transformer architecture and hidden states to generate a global term vector. Both methods used cosine similarities to find the best match for each term to normalize and candidate terms in the HPO. The BioBERT method, at 69% accuracy, outperformed the spaCy method at 46%.

The GPT-4o-mini, GPT-3.5-Turbo, and GPT-4o models without a retriever had no access to external data sources and relied entirely on pre-training to find HPO IDs. Errors made by these language models typically involved the retrieval of incorrect HPO IDs rather than errors in the HPO entry terms. In most cases, when the models made an error, the HPO entry term was correct or nearly correct, but the HPO ID was inaccurate and matched an incorrect concept in the HPO. Among large language models without a retriever, GPT-4o, the largest and most advanced model, performed best with a accuracy of 62%, while GPT-4o-mini, the smallest model, performed worse with an accuracy of 12%.

The best-performing methods combined a language model with a retriever. Each model had 20 candidate normalizations for a term to normalize based on the closest embedding similarities. Each language model was prompted to choose the “best match” from the twenty closest candidates. In this scenario, the GPT-4o and GPT-3.5-Turbo models outperformed the BioBERT method with 85% to 88% accuracies (Table 1).

We investigated the optimal number of potential matches to submit to GPT-4o or GPT-3.5-Turbo (Figure 1). Accuracy improved as the number of candidate normalization increased to 20 and then reached a plateau with further increases in candidates not improving accuracy of normalization.

The results indicate that the most accurate method for phenotype term normalization combines a language model with a retriever. GPT-4o and GPT-3.5-Turbo, when paired with a retriever, achieved the highest accuracies of 85% to 88%, demonstrating the benefits of augmenting language models with a retrieval mechanism.

The standalone BioBERT, specifically optimized for biomedical text, performed better than spaCy or GPT-4o without retrieval, with an accuracy of 69%. This highlights the limitations of a standalone large language model relying solely on pre-training when no external retrieval is available. BioBERT’s ability to generate specialized biomedical embeddings allowed it to outperform GPT-3.5-Turbo and GPT-4o without a retriever.

GPT-4o-mini, smaller and less pre-trained than GPT-4o and GPT-3.5-Turbo, showed weak performance on term normalization with 12% accuracy, highlighting its lack of exposure to HPO terms and their HPO IDs during training.

Examining the cases where large language model combined with a retriever outperformed BioBERT reveals that the “best match” selected by large language model can deviate from the candidate term with the highest cosine similarity (Table 2). This demonstrates the strength of large language models in interpreting semantic equivalence beyond simple cosine metrics. For example, “foot drop” was selected as a better semantic match for “bilateral foot drop” than “bilateral clubfoot,” despite having a lower cosine similarity score. Similarly, “depigmented fundus” was preferred for “pale fundi” over “pale eyelashes.” The ability of a large language model to override cosine similarity based on contextual understanding explains the superior accuracy of the retriever-augmented method. Part of this superiority likely resides on focusing attention on the semantically most important words in compound terms such as “depigmented fundu” where the focus is appropriately directed to “fundus” in favor of the less important word “depigmented.”

Our results suggest that, while large language models possess significant capabilities, their performance on phenotype normalization tasks can be enhanced by retrieval augmentation. The accuracy of normalization improves when the large language model is presented with candidate terms selected via cosine similarity. The plateau observed for 20 candidate terms in the prompt suggests that presenting more candidates neither degrades nor improves accuracy (Figure 1).

Shlyk et al. (8) demonstrated the efficacy of retrieval-augmented entity linking using embedded term definitions. However, our approach, which bypasses the need for explicit definitions, achieves comparable results. This suggests that advanced models like GPT-4o and GPT-3.5-Turbo can draw on pre-trained knowledge to assess semantic equivalence without relying on external definitions, streamlining the normalization process.

Regarding limitations, our study focused solely on term normalization and did not evaluate term identification. Additionally, our definition of “semantic equivalence” remains qualitative rather than exact. Determining whether “impaired sensation” is semantically equivalent to “decreased sensation” can be approached in three ways: (1) by measuring cosine similarity between appropriate word embeddings, (2) by the judgment of large language models trained for semantic reasoning, or (3) by gold-standard review from human domain experts. However, as datasets grow beyond 2,000 terms, human review becomes increasingly impractical, particularly for large ontologies such as SNOMED CT with over 400,000 terms. Another limitation is the relatively small and specialized list of terns derived from the OMIM summaries of neurogenetic diseases, which may not capture the full diversity of phenotype terms that require normalization. Expanding the dataset to cover a broader range of phenotypic terms from other domains could provide further insight.

Looking ahead, our retriever-based method is potentially generalizable to other terminologies, such as Gene Ontology (GO), UniProt, and SNOMED CT. By avoiding term definitions and relying solely on word embeddings, our approach could simplify normalization tasks in domains where definitions are ambiguous or difficult to generate. Further research into alternative retrieval strategies could improve the accuracy of the model and expand its applicability.

In conclusion, retrieval-augmented prompts based on BioBERT word embeddings improve the accuracy of phenotype normalization tasks. This simplified retriever performs as well as methods based on term definitions, offering an alternative solution for biomedical term normalization.