Results were synthesized using descriptive and thematic synthesis methods. No statistical synthesis was performed. Thematic analysis was performed to derive an initial basis for a systematic categorization of PICO-elements as well as the classification of the likelihood to derive conclusive evidence from the full texts. For the thematic analysis Braun and Clarke's six phase framework (19) was applied. Phase 1 (familiarizing) was conducted by organizing abstracts according to the level of PICO-specification and extracting results and conclusions. In phase 2 (initial coding), relevant information related to each PICO-element and to the reporting of results, quantification and direction of effects, and conclusions was isolated. Phase 3 (searching for themes) involved initial secondary research on existing approaches to the categorization of each PICO-element as well as criteria for the conclusiveness of evidence. Themes describing the PICO-elements and evidence were generated by allocating the coded information to the initially identified themes for categories and criteria and searching for additional themes. In phase 4 (reviewing themes), themes were iteratively refined until consensus was reached (UB, OK, JL). In phase 5 (defining themes), themes were named and defined (UB, JL). The final categories and classification criteria included in the framework were discussed and discrepancies were resolved with JL serving as thematic auditor. Phase 6 (writing) was completed with this article. A complete documentation of the thematic analyses is available in Supplementary Material 3, while Supplementary Material 4 provides Tables 2–5 with the thematic categories of PICO-elements including references.

For the internal validation, 138 open access articles providing full texts were identified from the sample of 250 studies for rapid full text review to test the accuracy of the proposed abstract-based categorization and classification. The use of this availability-based subsample was a pragmatic decision. Details are available in Supplementary Material 3—sheets PICO Validation and Evidence conclusiveness.

“P” was found to be mostly described in terms of a disease, disorder or condition.

The basis for the categorization of the problem were the six most common chronic or noncommunicable diseases (NCDs), as described for example by the WHO (20) or Robert Koch Institute (21), i.e., cardiovascular diseases (CVDs), cancers, diabetes, mental health issues, diseases of the musculoskeletal system and chronic pulmonary diseases, especially asthma and COPD. The WHO identifies tobacco use, physical inactivity, the harmful use of alcohol, and unhealthy diets as major factors increasing the risk of dying from an NCD (20). These factors were categorized as behavior. Other sources including the Centers for Disease Control and Prevention (CDC) and National Council on Aging (NCOA) (22, 23) additionally list stroke, Alzheimer's disease and dementia, which were categorized as neurological disorders (24), although stroke may also be categorized as a cardiovascular disease (CVD) (25).

In the thematic analysis, these eight categories occurred in about two thirds of all studies (193/250, 63.7%). For the remaining third, we defined three additional thematic categories covering diverse specific conditions that were investigated in less than three studies, diverse groups of conditions that encompass more than three of the other categories, and diverse, unspecified for studies without focus on any medical condition or specialty.

Table 2 provides an overview of the thematic categorization focusing on the various problems (P) described in the sample abstracts.

“I” refers to the digital health interventions and their underlying technologies.

Secondary research did not identify a suitable basis for the categorization of DHIs, nor deliver consistent definitions. Consequently, the intervention technologies were inductively identified from the abstracts and, through thematic analysis, systematically categorized according to consistencies in existing definitions, while aiming to ensure clear differentiation among categories.

In secondary sources, we found a plethora of definitions for digital health, which most commonly encompassed mHealth and eHealth, but were often related to various other concepts such as telehealth (167). There appeared to be a particular lack of consensus on the concept and definition of eHealth (168–170). The term mHealth was defined more precisely overall (171, 172), yet there was some overlap with other concepts, especially eHealth, telehealth, telemedicine and telerehabilitation. However, telehealth, telemedicine (173) and telerehabilitation (174) were relatively well and consistently defined and could be distinctly differentiated from mHealth by their intervention focus and apparent health provider interaction.

In the abstracts, we found the terms digital health, eHealth, mHealth, remote, web-based or online interventions, but also telehealth, telemedicine and telerehabilitation were sometimes interchangeably or misleadingly used by researchers and study authors. As a result, we categorized some “mHealth” studies as telehealth, telemedicine and/or telerehabilitation (29, 106, 122, 130, 175, 176). Similarly, some “telehealth” studies were categorized as telemedicine and/or telerehabilitation (32, 120, 135, 150, 177–179).

Thematic analysis of the random sample identified 11 categories to which at least three studies could be allocated, and two additional themes that may represent preliminary categories.

Table 3 provides an overview of the thematic categorization focusing on the various interventions (I) described in the sample abstracts.

“C” is here intended to reflect the healthcare settings to which the DHIs primarily compare or apply.

The basis for the categorization was identified and pre-defined from listings of healthcare settings at various levels of detail and with varying overlaps (232–234). Most common across secondary sources were hospital (emergency and inpatient care), outpatient (primary and specialist care practices and clinics), rehabilitation centers, and long-term care facilities (e.g., nursing homes). Residential long-term care and many of the diverse remaining settings, e.g., birth centers, mental health facilities, hospices, pharmacies, were categorized as community-based care. Community care also covers a wide range of additional health services including health promotion, occupational therapy, palliative care, physiotherapy, speech and language therapy, etc. (235). Other healthcare facilities were allocated to one of the before-mentioned categories, e.g., urgent care centers (236) and outpatient surgery centers to outpatient care. In line with some sources, we considered home-based self-care as a separate setting (232, 233).

Imaging and radiology centers, laboratories and pathology services represent healthcare facilities that do not directly provide care to patients but facilitate diagnoses by other clinicians. Thus, we added “intersectoral” to reflect that some studies investigate interventions or conditions requiring or focusing on health information exchange or a multidisciplinary approach between care providers.

Using the pre-defined categories and applying implicit rules when no setting was explicitly specified, we were able to allocate a category to each abstract.

Table 4 provides an overview of the thematic categorization focusing on the various comparative settings (C) described in the sample abstracts.

“O” refers to the anticipated results, impacts or effects of DHIs.

Outcomes were initially categorized based on the Six Domains of Healthcare Quality recommended by the IOM (now the National Academy of Medicine): Patient safety, Effectiveness, Patient-centeredness, Timeliness, Efficiency, Equity/Access (271).

We added “Satisfaction”, in accordance with Donabedian (272), and acknowledging that patient satisfaction is becoming more important in evaluating care quality (273), as confirmed by our thematic analysis.

Table 5 provides an overview of the thematic categorization focusing on the various outcome dimensions (O) described in the sample abstracts.

A rapid full text review was conducted on 138 open access articles for which the full texts were available to test the accuracy of the abstract-based PICO-categorization.

We considered our categorization as confirmed, when the full text detailed the respective PICO-elements in the methodology and/or in results, e.g., in overview tables of included studies. Partial confirmation refers to either additional categories being identified or not all categories (i.e., fewer) being confirmed in the full text. Cases in which each category had to be changed were considered as not confirmed.

Table 6 provides an overview of the validation test results. Details are available in Supplementary Material 3.

The variability in the accuracy reflects that the abstracts provided a clearer and more comprehensive description of the problems and interventions than they did of the comparator and outcomes.

Most partial confirmations related to additions of one or more categories that were identified at least for some included primary studies.

Rarely, categories were allocated wrongly—in only five cases for the comparative setting and outcomes, respectively. While four implicitly determined settings were not confirmed (e.g., categorized as self-care, but primary studies were conducted in outpatient settings), some outcome categories had to be removed since no concrete results were present in the full texts.

To address research question 2, we defined an abstract-based scheme that classifies reviews as high, medium, low, or inconclusive with respect to the likelihood that the full text contains conclusive evidence. While evidence conclusiveness indicates the degree to which evidence can support a specific conclusion or outcome (285), it must be distinguished from evidence quality, which refers to the methodological rigor and reliability of the data (286), and evidence strength, which pertains to the magnitude and consistency of the observed effects (287). In line with Babić et al. (2022) (2), we found no standardized definition of evidence conclusiveness. We therefore relied on transparent, reproducible signals that are routinely reported in titles and abstracts and prioritize designs at the top of evidence hierarchies (288, 289), which increase power and precision (290).

The following signals were identified:

Meta-analysis: explicit mention of a meta-analysis in title or abstract.

Following Babić et al. (2), we conducted a thematic analysis of abstracts capturing quantitative and qualitative signals of effect direction, statistical significance, and conclusions. Details are provided in Supplementary Material 3.

Across abstracts, 32.8% (82/250) explicitly referenced significance, typically without distinguishing statistical from clinical significance. Quantitative reporting appeared in 46.0% (115/250), including 37 abstracts with details on significant effects. In 31 abstracts, quantification was limited to numbers or percentages of studies or outcomes by effect direction.

The qualitative assessment yielded the following themes: a) explicitly significant positive effects (66/250, 26.4%), b) clearly positive (140/250, 56.0%), c) vaguely positive (146/250, 58.4%), d) neutral with comparable effects or no differences (36/250, 14.4%), e) explicitly no significant differences or benefits (34/250, 13.6%), f) negative effects (13/250, 5.2%), and g) insufficient, inconclusive, or conflicting evidence (57/250, 22.8%). Multiple allocations occurred in 61.2% (153/250), mostly within the same overall direction (104/250, 41.6%) and or across different interventions or outcomes (71/250, 28.4%).

Table 7 provides an overview of the final classification rules and associated signals.

We conducted a rapid full-text review of 138 open-access articles to test the accuracy of the abstract-based classification. The sub-sample comprised 16 of 22 studies predicted as high (72.7%), 46 of 74 as medium (62.2%), 63 of 133 as low (47.4%), and 13 of 21 as inconclusive (61.9%).

Table 8 reports confirmation rates by class as well as re-classifications. Details are available in Supplementary Material 3.

High and low likelihood of conclusive evidence were predicted very well (100.0% and 96.8%) from abstracts, whereas inconclusiveness was least accurately predicted (61.5%).

All 16 studies predicted as high reported quantitative aggregated meta-analytic results with multiple significant effects.

Among 46 predicted as medium, 31 included meta-analyses, of which six showed more than one significant effect and were re-classified as high. Ten of the remaining 15 full texts quantified at least one significant effect in over 25% of relevant studies, while five were re-classified as low because they lacked quantitative substantiation.

Of 63 predicted as low, none reported meta-analyses or aggregated significant quantitative effects; eight noted at least one significant effect in more than 25% of relevant studies, yet only one met the quantitative substantiation required for re-classification to medium. While four full texts stated limited evidence, three concluded with a positive direction, and only one had to be re-classified as inconclusive.

Eight of 13 predicted as inconclusive met at least one criterion, and five were re-classified to low because the full texts provided a directional conclusion that was absent from the abstracts.

The practical implication is limited attrition of potentially relevant studies. The re-classification from medium to low represents about eleven percent (5/46) of discarded full-text reviews, while the shift from low to medium implies fewer than two percent (1/63) missed opportunities for conclusive evidence. Medium classifications would in practice still enter full-text screening if there were insufficient high-likelihood studies, and low or inconclusive classifications would typically be excluded.

A limitation of this study is the sample size of 250 abstracts, which represents about ten percent of eligible studies and provides a 95% confidence level that the actual value is within ±6% of the measured/surveyed value. This is considered sufficient, since our analysis aims for thematic saturation rather than statistical representativeness. The validation sample was limited to 138 open-access, full-text articles. However, with an error margin of ±5.6% and a 95% confidence level, this sample adequately confirmed that the categories and criteria are suitable for avoiding false positive or negative categorization and classification.

The search period was confined to publications from 2011 to October 2023. An updated search strategy indicated a significant increase in publications, showing a 60.6% rise (from 12,608 to 20,246) when comparing search results from October 2023 to October 2025. Although excluding the most recent publications may alter the relative distribution of studies across existing categories (e.g., increasing focus on AI), it is not anticipated to result in the omission of entirely new emerging categories. However, the observed surge in the number of publications underscores the necessity for more efficient methods of evidence screening and synthesis.

The framework was derived from an analysis of these types of review, while other common types, such as scoping reviews, were excluded. However, the framework's application is not limited to systematic reviews, and it can be used for any form of evidence synthesis or thematic evidence collection. During development, the framework's suitability for EGM was tested by manually creating a visual overview representing the intervention and outcome categories combined with the classification of evidence conclusiveness.

Our definition of the comparator, focusing on the healthcare settings where DHIs are applied, diverges from the original PICO concept, which typically compares DHIs to alternative care practices or interventions. Our approach addresses limitations of the original concept within our framework's objectives and applicability to abstracts. Many systematic reviews lack a clear comparator and abstracts often do not specify one. By considering the healthcare settings where DHIs are used, we offer a broader “comparison” that helps decision-makers by identifying where DHIs are most beneficial.

While the proposed PICO categories effectively provided a framework for structuring problems (P), comparative settings (C), and outcomes (O) across abstracts from systematic reviews and meta-analyses on DHI impacts, the interventions (I) may require further expansion to accommodate additional digital health interventions as they emerge.

Our methodology emphasizes systematic reviews and meta-analyses to mitigate bias associated with varying evidence levels. The disparate ranking, grading, and rating systems currently in use complicate the consistent assessment of evidence quality and strength and are not suited for efficient screening, since they are inconsistently reported in the abstracts.

Our validation analysis represents an internal validation based on an availability-based subsample of 138 open-access full texts. We acknowledge the lack of an independent validation sample as a potential source of selection bias. However, this risk appears neglectable, with the validation sample representing 55.2% of the development sample. While the internal validation confirmed suitability and informed immediate application, we encourage external validation on an independent sample by peer researchers wishing to implement the PACE4DHI framework. Risk of future manipulation in abstracts: There is a potential risk for bias if authors deliberately craft abstracts to suggest a higher likelihood of conclusive evidence, should the framework become widely adopted. Despite this, we consider the risk manageable, as the use of the framework can simultaneously enhance research question formulation and the articulation of findings, thereby potentially improving the overall quality of abstracts and research.

The study successfully developed and validated the PACE4DHI framework to systematically categorize and classify existing evidence for digital health interventions (DHIs) using the PICO approach. It offers an efficient structured means for abstract screening and is intended to facilitate continuous evidence and gap mapping, providing a practical foundation for identifying conclusive evidence of achievable impacts and research gaps in digital health.

The research successfully identified categories aligned with current research as well as established schemes and definitions and demonstrated that the framework could effectively organize research evidence from systematic reviews and meta-analyses along PICO. This organization aids in elucidating the diverse impacts of DHIs, given the wide range of populations, interventions, comparisons, and outcomes.

The study identified and validated four key criteria for determining the likelihood of conclusive evidence being present in the full texts based on information that can be retrieved from abstracts: presence of a meta-analysis, the mention of significance related to results, result quantification, and indication of inconclusiveness. This approach enhances the probability to identify and retrieve all relevant evidence and can thus improve the evaluation of achievable DHIs' impacts.

The framework is intended to be used by researchers in a) formulating research questions aligned with current research and established definitions, b) informing comprehensive search strategies by providing associated search terms, c) driving efficiency in abstract screening of an ever-increasing body of studies, d) training artificial intelligence tools that support screening, extraction and synthesis from systematic reviews and meta-analyses, e) enabling subsequent continuous EGM guiding future research priorities and aiding decision-makers in evaluating the potential benefits of DHIs.

The PACE4DHI framework strengthens the strategic approach to building a comprehensive evidence base in the digital health sector. The application of the framework can help decision-makers at different levels (organization, politics, research) a) to inform digitization strategies more easily, b) to design implementation plans in an evidence-based manner, c) to identify research gaps more quickly and systematically. Further and ongoing refinement of terms associated with the categories and criteria is warranted to adapt the framework for use with advanced artificial intelligence tools, which can automate and enhance the research process in this rapidly evolving field.