We performed a systematic scoping review and followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist (15) (Supplementary File S5). The review protocol is available in the online Supplementary Material (File S1).

A literature search was conducted in September 2016, updated in November 2019, and included PubMed Medline, EMBASE, Scopus, and Web of Science. The search strategy was developed including an experienced pharmacoepidemiologist and counseling by an information specialist. The PubMed Medline search strategy is available in the online Supplementary Material (File S2).

We included studies if they: (1) were primary research studies that involved web data including social media or search engine data such as Google Trends, Google Correlate, Google Insights for Search, Google search engine, Facebook, Twitter, and Instagram; (2) involved any kind of comparison data such as drug sales or drug prescription volumes acquired from surveys, registry data, physician databases, and others. Not all of these data originated from validated sources; and (3) included any kind of drug utilization data such as utilization frequencies of vaccines, vitamins, supplements, nicotine alternatives, prescription drugs, and over-the-counter drugs for both data sources.

Articles were excluded if they: (1) focused on E-cigarettes; (2) involved incidence rates of diseases instead of drug utilization volumes; or (3) involved only web data sources but no other kind of comparison data source.

In addition, we excluded non-English study documents, literature reviews, posters, PowerPoint presentations, articles presented at doctoral colloquia, or if the article's full text was not accessible to the study authors (e.g., conference abstracts). Only peer-reviewed proceedings were included in this review.

All identified references were downloaded into Endnote, where duplicates were removed. Two independent reviewers conducted the screening with the free online tool Cadima (15). First, titles and abstracts were screened, followed by screening of the articles' full texts. The reference lists of the included articles were checked for additional studies. Any remaining disagreements about study inclusion or exclusion were resolved by a third investigator.

One reviewer independently extracted the prespecified information of the articles into a Microsoft Excel sheet with 22 columns containing information on the following aspects: (1) General information on the included studies (e.g., study objective), (2) characteristics of the involved data sources (e.g., web data source), and (3) additional study items (e.g., conflict of interest). The full list can be accessed in the online Supplementary Material (File S4).

Additionally, the reporting quality of the included studies was assessed using the STROBE checklist (16) (Strengthening the Reporting of Observational Studies in Epidemiology) as well as the statement's extensions RECORD (Reporting of studies conducted using observational routinely collected data) (17) and RECORD-PE (Reporting of studies conducted using observational routinely collected data for pharmacoepidemiological research) (18). Items were excluded if they were considered out of scope for the investigated population of research studies. One reviewer subsequently reviewed the adherence of the articles to the checklists' items. The checklist items were marked “yes” if the item was described satisfactorily well, “partly” if described partially, and “no” if it was not described at all. If an item was not applicable due to a study's nature or design, the item was marked “n/a”.

One reviewer additionally reviewed the study authors' perceptions of the challenges of using web data for drug utilization estimation reported in the discussion sections of the papers. The abstracted data items were verified by a second reviewer, and any disagreements were resolved in consensus. The full list can be accessed in the online Supplementary Material (File S3). The extracted data were synthesized narratively. Descriptive statistics were performed using Microsoft Excel (e.g., frequencies, and measures of central tendency).

Risk of bias assessment was not conducted, which is consistent with the scoping review methods manual by the Joanna Briggs Institute (19).

A total of 6,563 deduplicated citations from electronic databases were screened (Figure 1). Of these, 6,427 (98%) papers were excluded during the title- and abstract-screening process, leaving 137 (2%) articles eligible for full-text screening. A total of 123 (90%) full texts were found to be ineligible for study inclusion, the most common reason being wrong study design as they did not include relevant datasources or any comparison with drug utilization data [see exclusion criteria 2, n = 70 (57%)]. Ultimately, 14 (10%) papers were considered eligible for inclusion. A first search was conducted in September 2016, identifying eight eligible articles, and the updated search in November 2019 yielded six additional papers. The full list of included documents can be found in the online Supplementary Material (File S4).

The articles' publication dates ranged from 2010 to 2019, with 93% (13/14) of papers published from 2014 onwards (Table 1). The document types comprised journal articles (79%) and (full) conference papers (21%) (see Supplementary File S4).

Of all reviewed articles, the most employed web data source was Google Trends' search volumes assessed in eight (57%) studies (21–24, 26, 27, 32, 33). Two (14%) studies used Twitter posts (22, 34), and two (14%) other studies utilized search volumes from former Google services similar to Google Trends: specifically, the Google Health Trends API (30) and Google Insights for Search (25). One (7%) study utilized both Google Insights for Search' and Google Trends' search volume (20), and another (7%) study assessed the frequency of website hits where a certain keyword is found using the Google search engine (28).

Datasources used for comparison with Web data included: Elven (79%) studies used data from public/government organizations drug utilization estimates as comparator to the web data (21, 24, 25, 27, 29, 31, 32). U.S. databases [Medical Expenditure Panel Survey (MEPS) (21, 24), Database from Centers for Disease Control and Prevention (CDC) (25, 31), Center for Disease Dynamics Economics & Policy (32), the flu vaccination rate surveillance system used by the U.S. Department of Health and Human Services (DHHS) (29), Medicaid (26), State Serum Institute (27), Register of Medicinal Product Statistics (30), Drug prescription report, Germany (23), European Drug Report 2014: Trends and Developments (28), UNODC World Drug Report 2011 (28)], and three studies (21%) used privately owned databases [the 2004 to 2008 Pfizer Annual Shareholder Reports (20), IMS Health (22) and the administrative claims database provided by JMDC Inc. (33)].

Twelve (86%) out of fourteen studies provided the time of data collection for both the web and the comparison data source. In these studies, the web data were gathered for a median duration of 5.3 years (interquartile range of 3.9 to 8.6 years), while the comparative data were collected for a median duration of 5.0 years (interquartile range of 3.7 to 9.6 years). One (7%) study only reported the time of data collection for the comparison data source (21), while in another (7%) study, the time of data collection could not conclusively be identified (28).

Nine (64%) of the fourteen studies quantitatively compared web-mined and comparison data using different types of correlation analyses (Pearson -, Spearman - and Cross-correlation) (20, 21, 23, 25, 26, 29, 31–33). Two studies (14%) quantitatively compared the performance of different prediction models (27, 30) using web and comparison data in terms of root mean squared and mean absolute error. One study qualitatively compared different popularity ranking lists (28). Furthermore, two (14%) studies did not directly compare drug utilization volumes but reported the results of both data sources as part of an ecological analysis without statistical comparison (22, 24).

With a total of four (28%) studies, vaccines were the most frequently investigated drug class (25, 27, 29, 31). Two (14%) studies examined antibiotics (26, 30), and one (7%) study focused on both antibiotics and probiotics (32). The remaining studies included: Psychoactive drugs (28), statins (20), drugs for benign prostatic hyperplasia (22), antidepressants (23), medications with seasonal patterns (21), moisturizer (heparinoid) (33) and oral bisphosphonates (24).

Overall, positive associations between drug utilization estimates reported in web data sources and comparison data sources were found in all studies, with significant results reported in eight of the nine studies that used correlation analyses (20, 21, 23, 25, 26, 29, 31, 33). Kamiński et al. found antibiotic consumption to be significantly associated with internet search data of probiotics but not antibiotics (32). Kalichman et al. found that the internet search term H1N1 independently predicted H1N1 vaccine coverage, while the search term vaccine independently predicted HPV vaccination coverage as results of ordinal regression analyses (25). Two studies built and evaluated models to predict future drug utilization and reported the best predictions when combining web and comparison data (27, 30). Jankowski et al. developed a drug popularity ranking list using internet search data and found the list to be similar to those reported by two international drug data sources (28). Two studies conducted ecological analyses (22, 24), of which Skeldon et al. study reported both increased web search interest and drug prescription rates, separately after two sequential advertising campaigns (22). The study of Jha et al. found a series of temporally correlated spikes in internet search activity and a decline in drug utilization estimates following media reports of medication safety concerns (24).

Three studies found similar seasonal patterns across the web and comparison data sources (21, 26, 31). Moreover, one study found correlations between internet search volumes and drug prescription volumes not only at the same time but also following a one-month time lag for the population aged 20 to 59 years, suggesting that people obtain health-related information from the internet, which may subsequently affect their behavior and medication requests (33).

The adherence of the articles to the individual items of the STROBE, RECORD, and RECORD-PE statements is presented in Table 2. In over 80% of the studies, the following items were reported: title and abstract (1.1), background rationale (2), objectives (3), variables (7.1.b), statistical methods (12-a), and outcome data (15). The following (sub-)items were considered in more than 20 to 50% of the studies: title and abstract (1-a, 1-b, 1.2), study design (4), setting (5), data access (12.1), key results (18), limitations (19.1), interpretation (20), generalisability (21), and funding aspects. Less than 20% of the studies described the following items: variables (7.1, 7.1-a), bias (9), statistical methods (12-e), participants (13-c), other analyses (17), and accessibility of protocol, raw data, and programming code (22.1).

Several limitations and biases of using web-mined data for drug utilization estimation were discussed by the study authors. A total of five studies stated that there might be a selection bias as the web data source might not sufficiently represent the whole population and that important vulnerable populations such as the elderly might be underrepresented (21, 23, 29, 31, 33). Furthermore, unmeasured factors, such as users' search intents and attitudes as well as the potential impact of media attention might influence web-mined drug utilization volumes (20, 25, 32). Additional challenges were identified resulting from low search volumes when web data is narrowed down to specific regions or populations (31, 32). In two studies web data was considered to be inadequate to draw causal relationships (20, 25) and it was also stated that web-mined data might generally be unreliable as it is based on self-reported experiences (29).

Four studies specifically addressed limitations of using web-mined data from Google Trends (21, 26, 32, 33). Of these, three studies highlighted that Google Trends only reported a normalized share of the number of searches in the form of “relative search volume” rather than an absolute number of total searches (21, 26, 32). Furthermore, Google Trends provided no details about how research words were recognized or aggregated (33).

Twelve (86%) out of 14 included studies employed search engine data retrieved from various Google services such as Google Trends, Google Insights for Search, Google Health Trends, and the Google search engine. Connected to this, the total duration of access was very similar with a median duration of 5.3 years for the web and 5.0 years for the comparison data source. This is notably more than has previously been reported by a review focusing on the utility of social media for pharmacovigilance, where social media posts were followed for a median duration of 1.1 years (8). In addition, the predominance of search engine web data sources might be explained by the greater ease of accessing search engine data through services such as Google Trends compared to retrieving unstructured social media data, which typically involves a labor-intense processing pipeline containing multiple steps (8) to extract datasets suitable for analysis and comparison to other sources. We recommend that research in this field would use a wide range of web data rather than only focussing on one type of research engine (e.g. Facebook, Twitter, specific health forums).

Seven out of 14 (50%) studies focused with both antibiotics (n = 3) or vaccines (n = 4), respectively, on drug classes that belong to the field of infectious diseases. The remaining studies focused on drug classes of diverse other fields, such as diabetes, depression, and the misuse of psychoactive drugs. Studies included medications used either as short treatment (e.g., antibiotics or vaccines) or chronic use (e.g., statins for lipid lower, or antidepressants). However, as most studies used web search engines, they could only evaluate the prevalence of drug use as it is not possible to differentiate former and new users only from these data sources. Using specific analyses of posts content from Facebook, Twitter or specific health forums would allow more information to be retrieved on drug utilization. For instance, one could screen for information on the time patient are on medications or on the concomitant use of other medications. Analysing the content of social media posts has already been used in the past for pharmacovigilance (41). Considering that the investigated studies found consistent positive results of using web data for estimating drug utilization across the vast majority of the investigated drug classes, we advise future studies to extend research to include drug classes from other fields additionally and use a wider diversity of web data sources such as those including specific users posts.

The mentioned limitations of the included primary research studies highlighted potential challenges of using web data for estimating drug utilization, such as the potential lack of representativeness between web data-creating users and the general population, difficulties identifying the populations who created the web data, difficulties interpreting relationships between web data and comparison drug utilization data (e.g., due to the presence of potentially unmeasured confounding factors such as users' search intent or effects of media attention), and problems dealing with low search volume if data is narrowed down to specific regions or populations. These critical aspects should be systematically targeted in further studies using web data to assess drug utilization.

The overall reporting of the studies' quality according to the STROBE, RECORD, and RECORD-PE checklists was mediocre and strongly varied between the different items. The most commonly reported items (>80%) were background/rationale, objectives, and outcome data. Items with low reporting (<20%) were other analyses, bias, and the accessibility of protocol, raw data, and programming code. Of particular relevance is the poor reporting of the two latter items, since both items were rated to be applicable for all reviewed studies and since these points are increasingly recommended as they target research transparency and reproducibility. The finding that articles tend to underreport biases has also been observed in two other studies that assessed the compliance of the articles with the STROBE checklist in different fields (42, 43). One of the issues may be that these guidelines are not specific to internet user content research.

Moreover, many items were rated to be out of scope for the type and design of the studies we included in our review. In many cases, this was due to the fact that the users who created the web data could not directly be regarded as study participants as, for example, eligibility criteria cannot be controlled and important information such as descriptive user characteristics can hardly be retrieved from web data.

In conclusion, the three checklists include all important items necessary to assess the reporting quality of the included studies. However, a variety of items were not applicable as they were out of scope for these types of studies. Therefore, we recommend utilizing a shortened and adapted version of the current STROBE, RECORD, and RECORD-PE checklists for future studies. For example, as web data was usually sourced through social media platforms and open-access websites for search analysis, no actual participant recruitment procedures took place in those studies. Therefore, all items relating to the recruitment and assessment of real-world participants could be omitted in a future version of this checklist (i.e., items: 6(a), 6(b), 6.1, 6.2, 6.3, 6.1.a, 13(a), 13(b), 13.1, 14(a), 14(b), 14(c)) and replaced by more suited item such as: the type of web data (e.g. search terms volumes, number of tweets/posts of interest…).

This systematic scoping review was conducted and reported according to the standardized PRISMA guidelines (15). We conducted an extensive literature search, defined the study eligibility criteria, rigorously assessed studies that contained drug utilization information from web data sources, and compared it to other sources with drug utilization information.

One limitation of this review was the heterogeneity of methodologies in terms of study objectives and analysis methods in the included studies, which made it impossible to draw more general conclusions. This, together with the relatively small number of identified studies, underlines the complexity and novelty of the field and justifies the selection of a scoping review approach.

Finally, in our assessment of the studies' reporting quality employing the STROBE, RECORD, and RECORD-PE checklist, a substantial number of items had to be considered out of scope for these types of studies. This requests for an adapted (standard) checklist.