Within the interest group category, we establish distinctions among sentences originating from a caregiver, a patient, a patient’s association, the scientific or medical community, or other groups. These categories are mutually exclusive. Defining patients and caregivers is a crucial task, and their definitions can vary significantly in medical literature. For instance, Delestre-Levai et al. (2021) describe patients’ and caregivers’ posts as follows: “First-person mentions of the disease or treatment were automatically classified as patient conversations, whereas third person references to the disease experiences of a child, parent or family member, or their care were identified as caregiver conversations.” Cook et al. (2019a) follow a similar scheme. During the data analysis, it was discovered that many instances of first-person mentions of a disease or treatment do not fit the criteria for patients’ posts. As a result, we developed more restrictive definitions:

Caregiver (CG): The author of the sentence either explicitly or implicitly indicates a close relationship with a sick person or a patient. A specific indication that the person is taking care of that sick person or patient is not required. Caregivers typically include family members, friends, or acquaintances. Caregiver posts are written in the first person, thus conveying personal information or experiences. Examples are: “My loved one has been diagnosed with idiopathic pulmonary fibrosis, which is considered terminal, making long-distance travel increasingly difficult for us.”; “Since June 10th, my mother has been extremely fatigued, but unfortunately, we have been unable to access suitable medical care within our country.”

Patient (PA): The author of the sentence either explicitly or implicitly identifies themselves as a patient or a sick person. To classify a post as originating from a patient, it must contain explicit mentions of a medical condition. Patient posts are written in the first person, thus conveying personal information or experiences. Examples are: “As someone who lives with disabilities, our community has recently faced a devastating natural disaster that left us struggling to cope - unfortunately, we received little support from local organizations, which only highlights the pressing need for accessible home modifications.”; “After battling two episodes of pneumonia, I’m left with irreparable lung damage from my compromised immune system, which has led to the development of both emphysema and pulmonary fibrosis.”

Besides that, we added two more interest groups, namely:

Patient’s association (PAA): The sentence pertains to events, organizations, workshops, etc., designed for patients and their caregivers. The exception to this rule is for scientific conferences or scientific workshops. Examples are: “I recently contributed to a tribute fund for {ASSOCIATION}. This effort was initiated by our community living with Pulmonary Fibrosis, who came together to create a user-friendly online platform (#{PROJECT}, accessible at {URL}) for easily navigating relevant data.”

Scientific or medical community (SC): The sentence pertains to scientific or medical papers, studies, clinical trials, findings, as well as events like conferences or workshops. Examples are: “A leading research institution advocates for pulmonary fibrosis treatment by enhancing the recycling process of TGF-beta receptor type I [1]. This organization has made significant contributions to tissue repair and wound healing across various organs.”; “I am pleased to share that my abstract: {TITLE} has been selected for an oral presentation at our organization’s annual research conference later this year.”

Other (O): Any sentence that cannot be categorized into the four previously listed categories falls into this miscellaneous group. Examples are: “I’m afraid this is not the situation we are dealing with.”; “We’d love to relocate, but it’s just not feasible for us right now.”; “But not this.”

We annotated 1323 sentences (cf. Table 1 for more detailed information) which were used for both training and evaluating a text classification model.

We distinguish between primary and secondary biomedical named entities. The category of primary biomedical named entities includes:

Disease (DI): Names and abbreviations of diseases, such as pulmonary fibrosis, IPF etc.

Drug (DR): Names of chemicals and drugs, including brand names, such as pirfenidone, nintedanib, etc.

Symptom (SY): Names of symptoms associated with a disease.

Treatment (TR): Names of medical methods, techniques or devices that are used to treat diseases or symptoms. The key difference to drug is that treatments are not chemical.

These categories are mutually exclusive, meaning they do not overlap. Distinguishing between these primary biomedical named entities is generally clear, except for the distinction between disease and symptom, as suggested by the overlapping entries in ontologies (see Section 4.2).

Under the term secondary biomedical named entity, we cover those biomedical named entities whose identification relies on the previous primary biomedical named entities. Often, the identification of secondary biomedical named entities requires capturing the interaction between the primary biomedical named entities:

Diagnosis: Any disease or symptom that is stated to be diagnosed or detected is an instance of diagnosis. Therefore, to identify a diagnosis, you need to be able to identify diseases or symptoms first.

Misdiagnosis: Any disease or symptom that is stated to be misdiagnosed or was missed during examination is an instance of misdiagnosis. Therefore, to identify a misdiagnosis, you need to be able to identify diseases or symptoms first.

While not directly tied to the primary task of extracting biomedical named entities, this category also encompasses the extraction of time-related mentions. Analyzing time-related mentions and their correlation with the previously mentioned biomedical named entities yields additional insights into the challenges and support experienced by patients and caregivers. For instance, the burden of living with a disease for an extended period seems to be greater than dealing with a recently acquired condition. The labels for time-related mentions are taken directly from the dataset of Almasian et al. (2022) and are:

Duration (DU): Expressions of any durations, such as 3 months, 2 weeks, days etc.

Date (DT): Expressions of a date, such as yesterday, tomorrow, November 3rd, 2023.

Time (T): Expression of a time of day, such as 3 o’ clock, 7 pm etc.

Set (S): Expressions that describe some sort of frequency, such as daily, weekly etc.

As outlined in Section 4.2, for named entity recognition we use external ontologies as well as an already fine-tuned model. Hence, the dataset was used exclusively for testing purposes (cf. Table 2).

The relationships between extracted medical concepts and time mentions must be identified. In line with the research topics at hand, we have established the following relations:

Ameliorates (AM): This relation exists between a treatment (including drugs) and a disease/symptom when the use of the treatment leads to the improvement of symptoms caused by the disease. It implies a therapeutic interaction where the treatment positively impacts the disease’s progression or alleviates the severity of its symptoms, contributing to the patient’s relief or recovery. Typical trigger words are improve, ameliorates, amelioration, decrease, reduce etc.

Creates (CR): This relation is established when one biomedical named entities (such as a disease, treatment, or drug) directly leads to the occurrence of another medical condition or symptom. For example, this relation holds if a disease causes a specific symptom, or conversely, if a symptom is indicative of a particular disease. Similarly, it applies when a treatment or drug results in the emergence of a new disease or symptom. The creates relation signifies a causal or contributory link, where the presence or administration of one entity is responsible for the genesis of another condition or symptom. While we drew inspiration from the existing definition of causality by Dunietz et al. (2017), we opted for a more restrictive definition. For example, according to our annotation guidelines, temporal causation, as indicated by conjunctions such as after, does not qualify for the relation creates. Also, expressions such as IPF cough do not qualify as instances of the relation IPF #causes# cough, since an explicit cue word indicating a causation is missing.

Diagnoses (DG): This relation specifically exists between a treatment or diagnostic method and a disease or symptom. It is applicable when a particular treatment or diagnostic technique is explicitly used to detect or diagnose a disease or symptom. The diagnoses-relation implies that the method or treatment in question is instrumental in identifying the presence, nature, or severity of a particular medical condition or symptom. This relation is central to clinical practice, as it connects diagnostic procedures directly with the medical conditions they are intended to identify or confirm.

Exacerbates (EX): This relation is used to describe a scenario where one entity X, a disease, symptom, treatment, or drug, causes a worsening or aggravation of another entity, typically a medical condition or symptom. This relation is specifically applied when there is an explicit description of the worsening effect. It encompasses cases where X leads to an increase or intensification of Y. The exacerbates-relation also subsumes instances where one entity X inadvertently promotes the progression or severity of a disease, like the growth of a tumor. The key aspect of this relation is the explicit and discernible intensification of a medical condition or symptom due to the influence of another factor. Typical trigger words are exacerbate, worsen, worse etc.

Is associated with (IAW): There are cases where it is very likely that one entity X causes another entity Y, nevertheless the linguistic information provided is too weak or ambiguous to draw this conclusion. These cases, where there is a lack of an explicit connective or a connective that is too general (expressions such as is associated with, is connected to, is linked to etc.), will be subsumed under the relation is associated with.

Is used for (IUF): This term encompasses instances where an entity X is identified as a treatment for entity Y. In such cases, no explicit information about the efficacy of the treatment is provided; that means, it remains unspecified whether the treatment ameliorates or exacerbates a disease or symptom. Common indicators include phrases like is used as a treatment for or is a treatment for. Sometimes, direct connectives may be absent, necessitating reliance on context. Here, if a treatment is recognized and the sentence contextually links the treatment with a disease or symptom, it is inferred that the treatment pertains to the disease/symptom. This inference approach is deemed appropriate as it does not speculate on the treatment’s effectiveness or outcome.

Is time of (ITO): This designation encompasses instances where a specific point in time marks the onset of a disease, symptom, or similar event. While recognizing that a single time-relation might oversimplify the complexity inherent in time-relation extraction, as indicated by the extensive range of possible time-relations identified in Gumiel et al. (2022), this approach nonetheless proves adequate for initial results.

Almost all relations are unidirectional, meaning that swapping the positions of the nodes in a triple change the meaning of the relation: cough #causes# IPF ≠ IPF #causes# cough. The only exception to this rule is the relation is associated with, where swapping the positions of the nodes does not impact the semantics of the resulting triple: cough #is associated with# IPF = IPF #is associated with# cough.

The semantic distinction between some relations is subtle. For example, the relations creates and exacerbates can be considered subclasses of the broader relation is associated with. A similar relationship holds between ameliorates and is used for, with the former being a subclass of the latter. In the subsequent section we will refer to the is associated with and is used for relations as macro-relations and to their subclasses as micro-relations.

As detailed in Section 3.2.3, relation extraction is deployed after named entity recognition, since the listed relations in the pipeline can only exist between pre-extracted concepts. Therefore, the relation extraction dataset developed for this study follows the structure of the SemEval-2010 Task 8 (Hendrickx et al., 2009): For each sentence, we provide information about two concepts present in the sentence. The dataset’s task is to extract the relation and determine the direction of the relation between these two concepts. To simulate real-world scenarios, the dataset also contains sentences with concepts that do not exhibit any relations.

It is essential to emphasize that the annotations of relations and, consequently, relation extraction aims to identify relations as they are stated in the text and not as they exist in reality. This becomes particularly significant when considering the diversity of social media post authors, the majority of whom lack a medical background and may provide scientifically untenable information.

As outlined in Section 4.3, for relation extraction we use natural language inference models in combination with heuristics and ontologies. Hence, the dataset was used exclusively for testing purposes (cf. Table 3).

Named entity linking is the process of identifying and linking entities mentioned in text to corresponding entities in a knowledge base, thereby providing context and unambiguous identification of these entities. This technique finds application in various fields, including the biomedical domain (French and McInnes, 2023). The specific named entities we aim to extract using this method, which we refer to as primary biomedical named entities, include disease, symptom, drug, treatment.

As the set of concepts, we aimed to detect is extensive, annotating a large dataset for training a named entity recognition model would have been a laborious task. Consequently, we chose to leverage ontologies, which have proven to be a promising approach (Yoon et al., 2022). This was achieved by utilizing spaCy’s matcher pipeline⁶ to process thousands of entries from various ontologies (cf. Supplementary Table 2) and identify their presence in documents. Most of these ontologies are medical in nature, except for the Wikidata which, in absence of dedicated ontologies, we used to compile a list of terms for the concept treatment: We retrieved all terms classified under the superclasses therapy and medical procedure. Additionally, we excluded entries classified as health assessment, invasive test, medical diagnosis, and medical test. This exclusion was necessary as these categories are more likely to be associated with diagnostic methods rather than treatments. Upon manual review, the compiled list for treatments was found to produce favorable outcomes. Employing ontologies was especially effective, primarily because medical ontologies tend to consist of unambiguous terms. Nevertheless, to minimize the occurrence of false positives, we implemented several filtering mechanisms, including:

Ignoring all matches that were part of spans belonging to the entity class ORGANIZATION. This step was considered crucial, as many foundations and organizations include disease names in their titles, as seen with the Canadian Pulmonary Fibrosis Foundation.

Table 5 (Panel A) showcases the results of the named entity recognition pipeline applied to the named entity recognition dataset. As anticipated, the precision score outperforms the recall score, a pattern most noticeable in the extraction of symptoms and treatments. Future improvements in named entity recognition, such as thorough ontology curation or advanced training of a named entity recognition model, are crucial. This is especially important since the effectiveness of relation extraction (see Section 4.3) hinges on the named entity recognition performance. Currently, the results are adequate to operationalize the pipeline and to gain initial insights.

This approach facilitates preliminary conclusions concerning RT2 and RT3. For example, by analyzing symptoms that co-occur with IPF or its synonyms within the same sentence, we can derive initial insights into the potential health-related challenges faced by patients (cf. Supplementary Figure 1). In a similar manner, a co-occurrence analysis of treatments allows us to make preliminary assumptions regarding the support provided to patients and their caregivers (cf. Supplementary Figure 2). For comparable overviews pertaining to the other category disease, please see Supplementary Figure 3.

However, mere co-occurrence is not sufficient to draw meaningful conclusions, which is why we implemented relation extraction (see Section 4.3).

For the extraction of time-related mentions, we utilized the BERT-based named entity recognition model for time mention extraction described by Almasian et al. (2022)⁷ the results of which are depicted in Table 5 (Panel B). A preliminary error analysis revealed discrepancies between our definitions of certain time concepts and those used to annotate the dataset in the study by Almasian et al. (2022). For instance, the adverb now was consistently tagged as a date instance in our test dataset: We deemed this significant for identifying the onset of diseases or symptoms. The utility of time-related mentions becomes particularly evident when integrated with biomedical named entities for relation extraction, as discussed in Section 4.3.

We aim to extract new information dynamically, beyond the predefined sets of named entities used in previous methods. This objective includes understanding the general needs expressed by different interest groups and the support they receive, which extends past mere medical concept extraction. This poses a challenge due to the abstract nature of these concepts, as they are not limited to a specific domain. For instance, needs may be social or psychological, and the same variability applies to support types. To address this complexity, we utilize zero-shot text classification and question-answering models, leveraging their capabilities to decipher abstract and domain-agnostic information.

Figure 6 illustrates the deployment of zero-shot text classification and question-answering models. Initially, a post is divided into sentences. Then, based on the target named entity, each sentence (i.e., the context) along with several predefined hypotheses is processed by a zero-shot text classification model to determine which sentences might contain relevant information. For instance, the hypothesis “Someone needs something.” is used to classify sentences according to whether they express a need. The model generates a confidence score ranging from 0 to 1 for each hypothesis. If the confidence score for a particular sentence exceeds a predefined threshold—determined through trial and error—it is inferred that the sentence entails the hypothesis; in other words, the hypothesis is validated. For example, the first sentence in Figure 6 indicates a need, the second sentence does not express a need. In cases of entailment, the first sentence then serves as context for a question-answering model. Corresponding to each hypothesis, one or more questions are formulated to extract more specific information within the sentence. In the provided example, by inputting the first sentence and the question “What is needed?” Into the question-answering model, a lung transplant is extracted as the explicit need expressed in that sentence. In a similar vein, information regarding support can also be extracted.

While natural language inference models are effective in zero-shot classification tasks, their performance varies across domains. For instance, domain-specific hypotheses like “Someone talks about a diagnosis.” may not be well-handled by models fine-tuned on non-medical datasets. However, incorporating domain knowledge to some extent is feasible by means of ontologies, as depicted in Figure 7. According to Section 3.2.2, any disease or symptom identified or detected qualifies as an instance of diagnosis. To pinpoint instances of diagnosis, an effective strategy involves first identifying any symptom or disease and then evaluating, within the given context, if it is a diagnosis. Ontologies facilitate the extraction of diseases and symptoms, which, along with the full context (i.e., the sentence), enables the crafting of hypothesis-context pairs for truth value assessment. For example, upon identifying idiopathic pulmonary fibrosis as a disease instance, we can formulate the hypothesis “idiopathic pulmonary fibrosis was diagnosed.” or “idiopathic pulmonary fibrosis was detected.” Given the context, zero-shot classification allows us to validate this hypothesis, thus categorizing idiopathic pulmonary fibrosis as an instance of diagnosis. In a similar vein, information regarding misdiagnosis can also be extracted. An overview of the hypotheses used, i.e., templates, along with the pertinent questions, is provided in Supplementary Table 3.

For zero-shot-classification we used a DeBERTa-based model,⁸ and for question-answering a RoBERTa-based model.⁹

We conducted a qualitative analysis of randomly selected samples from each category, namely need, diagnosis, misdiagnosis, and support. This analysis revealed that the results are adequate for obtaining initial insights. Table 6 showcases example sentences for extracted instances of need, diagnosis, and misdiagnosis, serving as representatives of potential challenges in RT2. It also includes instances of support pertinent to addressing RT3.

The potential of zero-shot classification for extracting information about diverse interest groups within the PFDD context has been previously recognized (Karmalkar et al., 2023). Building upon this, our work integrates question-answering techniques to pinpoint precise information within extended posts. This methodology parallels traditional information extraction pipelines, which often employ text embeddings like Sentence Transformers (Reimers and Gurevych, 2019). Such pipelines typically utilize these embeddings to identify a set of relevant documents which is semantically most similar to a search query (akin to our hypothesis), for example, by calculating the cosine similarity. Subsequently, they apply question-answering to the most relevant documents to extract specific information. A parallel can also be drawn with retrieval-augmented generation frameworks (Lewis et al., 2020), which have recently gained prominence alongside the rise of Large Language Models (LLM). Both these approaches and ours share the goal of deriving answers from extensive document collections. However, the key difference lies in our use of an extractive approach, aiming to extract answers from each individual document or even sentence, as opposed to abstractive approaches that generate a single response from a large corpus. While extractive results could be obtained solely through question-answering, prefacing this with zero-shot classification effectively reduces the incidence of false positives.