Recently the question has been asked whether altmetrics can measure research impact on policy (Bornmann et al., 2016) (Haunschild and Bornmann, 2017). Waltman and Costas (2014) reflected: “altmetrics opens the door to a broader interpretation of the concept of impact and to more diverse forms of impact analysis.” Some of these forms of impact, such as public engagement and changes to practice in society, might benefit from the development of altmetrics (Khazragui and Hudson, 2015). With a growing pool of research in this area, attention is moving toward the evolution of altmetric data relating to policy documents, and what policy makers could or should do with those data (Didegah et al., 2014). Funders should have a natural interest in such impact: if research they have supported is referenced as part of the evidence supporting a national and/or international clinical guideline, for example, then it is an indication that this research is likely to be influencing policy (Kryl et al., 2012). Until recently it was almost impossible to discover whether a piece of research had been cited in a policy document without either manually sifting through potentially relevant documents, by finding out through direct communication between the policy makers, or by pure chance. However, in 2014, Altmetric.com announced that they had added the ability to track policy documents that cited a research output (Lui, 2014). Funders saw the importance of this development, with The Wellcome Trust taking an immediate keen interest in the development of altmetrics relating to the policy sphere (Haustein et al., 2016).

Altmetric.com now has a growing list of policy documents that it trawls for citations. This includes, but is not limited to, the UK National Institute for Care and Health Excellence (NICE), the World Health Organization (WHO), and the European Food Safety Authority. Altmetric.com is able to do this by processing each policy document and extracting text to search for possible citations. The Altmetric.com “scraper” evaluates the text in the policy document and determines if it is appropriate data to make a positive match with a research output. Any references that are detected are checked in the PubMed and CrossRef databases to determine whether or not they are actual scholarly citations tied to actual research articles. If a match is made between a policy document and a research paper in the form of a citation, then it is added to the Altmetric.com details badge, which contributes to the paper’s Altmetric.com Score. Altmetric.com scrapes data within policy documents going back to 1928 and as a result is able to provide a long tail of research influence in newly published policy. For example, a paper could be published in a journal in 2005 and then get cited in a 2017 policy document; using Altmetric.com data we can now see this policy impact. It is important to note that the results of this research are influenced by the coverage limitations of Altmetric.com. At present, only data on the policy documents and subsequent citations are presented in this database. The full extent of research cited in global policy documents is not yet known.

Given the massive scale of the available data, it was necessary to create a reasonably sized and coherent sample for the analysis. Previously, other authors have explored the usefulness and viability of Altmetric.com policy documents data by identifying the proportion of publications within a single database [Web of Science (WoS)] that were cited in policy-related documents (0.5%) (Haunschild and Bornmann, 2017). For this study, the authors chose to limit the sample to a single center, their own institution, the University of Sheffield. This provided a sizeable but manageable, cross-disciplinary sample, the findings from which could be considered to be generalizable to very many other, similar institutions. No previous work has been published on the altmetrics for policy impact of research publications from a single institution.

As with other UK higher education institutions, The University of Sheffield is increasingly interested in the impact of its research, especially given the power of the periodical REF assessment to reward institutions based on the quality and impact of their research outputs. Citations in policy documents are, therefore, of interest to universities as they are a potential indicator of societal impact (Haunschild and Bornmann, 2017). Altmetric.com provides the facility to explore data back to 1928 and does not only help identify research that has recently been published and is having an influence on policy, but also permits and exploration of this influence in the past. We, therefore, also wanted to look at research published by The University of Sheffield from previous years that was still having some influence in new policy documents which the Altmetric.com data was able to highlight. The authors also had access to The University of Sheffield’s Altmetric.com institutional account, which permitted an exploration of the degree to which this organization’s research was being cited in policy.

The interest in altmetrics and policy impact is, therefore, growing. This paper seeks to contribute to the research in this area by addressing the following questions:

First, how useful are Altmetric.com data for identifying and categorizing policy documents that cite the research of a particular institution?

We wanted to see whether research conducted at The University of Sheffield was limited in its citation lifespan within policy documents. Access to the Altmetric.com data allowed us to look back to 1928 to see how far back a piece of research could exist but still be cited in policy. The earliest piece of research in our sample of 1,463 records was published in 1976. We wanted to explore the time lag between a piece of research being published and being cited in policy. We found that one paper was recorded in the data as being published in 1965, but that was incorrect on closer inspection of the Altmetric.com record. We also found this to be the case for several other papers that were recorded as being published pre-1979. All of these papers were actually published post-2000 after cross-checking the records. The earliest paper to be cited in policy was from 1979 and had a time lag of 25 years to its first citation in a policy document, and the shortest time between research publication and policy citation was about 3 months (see Table 3). Time lapses between research publication and policy vary and are complicated with one estimate being an average of 17 years (Morris et al., 2011). There have been calls to accelerate policy impact of relevant research (Hanney et al., 2015). As seen from our sample and other published research, the vast majority of published research will never feature in policy documents and despatches. It is important to remember that not all research is published in a policy context. Altmetric.com’s continual trawling for fresh policy sources and documents can aid the process for discovering the minority that are.

The research articles focused on a narrow field of disciplines: health economics (four articles); economics and public health (both three articles); medicine and environmental science (two articles each) (see Table 4). We were very interested to explore the global impact of University of Sheffield research in policy documents and it was relatively easy to assess the national and international policy impact of research articles using Altmetric.com. This sample of 14 research articles with authors from the University of Sheffield in the UK was associated with a total of 175 “policy documents.” Including duplicates, 107 (61%) of these documents were “national,” i.e., from the UK, and 68 (39%) were “international,” i.e., from countries other than the UK or from international bodies, such as the United Nations or the WHO. It should be noted, however, that these “Sheffield” authors shared authorship with non-UK authors in 5/14 research articles (Rushton, 2015; Coleman, 2013; Wyn Morgan, 2010; Lomas/Woodward, 2009; Storey, 2009) and that the proportion of international policy documents is much higher in these cases: 31/34 policy documents (91%) were international for these five research articles (excluding duplicates and journal articles erroneously designated as “policy documents”). This compares with a 46% (28/61) of international policy documents for the 9/14 research articles with only UK authorship. This is consistent with other findings on international co-authorship and impact (Narin et al., 1991). In the field of health economics, which is less generalizable to non-UK contexts, the four research articles all had UK only authorship, and 25/34 (74%) of non-duplicate citing “policy documents” were also from the UK (i.e., national). In the three more general economics research articles, by contrast, even with almost exclusive UK authorship (only the Wyn Morgan, 2010 paper had an “international author,” from Italy), only 3/24 (13%) were “national” policy documents, so 25/28 (87%) were “international.” The topic clearly influences the national or international nature of policy impact more than other factors. However, it is important to remember that this is only a small sample.

From our sample we looked at the total policy document citation count per research paper. Twenty-three out of the 175 policy documents (13%) were not accessible via the Altmetric.com pages either because the link was broken, the document was currently unavailable or because the full document itself was behind a paywall. Where non-duplicate documents could be accessed, an assessment was made of the number of citations of the relevant research article within each policy document (see Table 5). This was possible for 75 out of 117 documents from this sample (64%). The majority of these research articles were cited only a single time or not at all (only appearing in a reference list) in the various “policy documents”: 53/75 (71%). In this sample, the research articles are cited twice in 8/75 (11%) and three or more times in 14/75 (17%). This suggests that impact generally is very limited, but is perhaps consistent with known figures for academic research impact (Carroll, 2016).

We also explored supplementary Altmetric.com data (See Table 6) to investigate whether there were any relationship between the attention that research receives across the web and citation in policy documents. We looked at individual Altmetric.com data counts rather than the total Altmetric.com Score. The best Altmetric.com performers were Mendeley saves and Tweets, while some of the research outputs received some media attention. It is important to note that not all Altmetric.com outputs receive the same weighting with news having the highest importance and Altmetric.com score, for example, news scores 8 points, blogs 5, Twitter 1, and policy documents 3 (per source).⁴ Mendeley had by far the most attention with the 14 most frequently cited papers saved 4,963 times in Mendeley users’ reference management databases, although just three papers (Lomas/Woodward, 2009; Peasgood, 2008; Webb, 2006) accounted for about 40% of those Mendeley saves. Twitter accounts for 458 Tweets but again the majority of these Tweets were as the result of just two papers (Rushton, 2015; Coleman, 2013). Coleman, 2013, Lomas/Woodward, 2009, Storey, 2009, and Webb, 2006 received coverage in 20 of the 24 news articles that were captured in the data, while Coleman, 2013, Lomas/Woodward, 2009, and Storey, 2009 also account for over two-thirds of the blog posts. Of the two most-cited pieces of research in policy, Peasgood, 2008 received a great deal of attention in Mendeley, but very little elsewhere. Brazier and Ara’s work was picked-up by Mendeley users but received no other coverage. Coleman, 2013 was the only research article to receive a score in each of the columns, while Brazier/Ara, 2008, Freckleton/Gaston, 2007, Weich, 2007, Brazier, 2005, and McIntosh, 2002 received no Altmetric.com attention other than policy document citations. Work by Cadwallader (2016) looked at selected Altmetric.com data from some departments at The University of Cambridge and found that “sustained news and blog attention could be a key indicator for identifying those papers likely to make it into policy.”

The Altmetric.com data did permit the quick and easy identification of research publications from an institution and the identification of policy documents that cited those publications. However, the system and data are not without errors or limitations.

Altmetric.com had problems in the accurate identification of some papers: one-third (7/21) of the sample of publications that appeared to be cited five or more times in policy documents were not by current or previous Sheffield authors at all. It is not entirely clear how research articles were identified as having an author who was currently at The University of Sheffield, when none of the listed authors was actually employed at the institution; or how citations of one research article came to be attributed to another (another paper with the same or a very similar title, from the same year, but with different authorship: e.g., Geddes, 2011 and Prescott, 1988, see Table 2). We, therefore, looked at the published research and the authors and their affiliations at the time of publishing, with most being based in the United States. In the case of Prescott, where we have a notable academic of the same surname at Sheffield presently, we looked at the publication date and discipline and compared it to research by the Sheffield-based author. We found that there was not only a change in discipline but also a notable gap between the publications of about 20 years. Exploring the issue further we contacted our research department to find out why the data were incorrect. We found that data such as academics’ names, departments, and faculties are all held within institutional systems and fed to Altmetric.com as a way of connecting the Altmetric.com data with academics’ profiles. These data, which are hosted by Sympletic Elements, are updated by institutions and any changes within the institution are reflected by the Altmetric.com database. In the case of incorrect authors, we found that three of them had been linked to Sheffield academics after three researchers based at our institution had incorrectly claimed the publications as their own, before rejecting them later. One was rejected by a Sheffield author but remained in the system, while another had pulled in the wrong paper through in Altmetric.com from the PubMed ID provided by Sympletic Elements. Other incorrect attributions to Sheffield authors were more understandable: research articles identified as being authored by Holdsworth were actually authored by others, although the name of this academic does appear elsewhere in the article. However, this still represents an issue with the software as the Sheffield author did not actually appear in the author list of the paper.

The result was that one-third of the sample of 21 research articles subjected to in-depth analysis was incorrectly identified (false positives). This has implications for the utility of data from Altmetric.com, as it suggests that perhaps as much as one-third of such a data set might consist of erroneous attributions (e.g., 487 of this sample of 1,463 research articles with policy citations), which can only be verified by checking each individual research article and its Altmetric.com data. As noted above, the research publications, authors, departments, and institutions that Altmetric.com trawls for information have also been found to be inaccurate, for example, an author at an institution accepting publications that are not their own via in-house research databases, such as Sympletic Elements. The origin of these inaccuracies is not always easy to identify. We can only assume that such examples are genuine mistakes by authors, although we cannot rule-out gaming or foul play in some instances.

The policy document citation and supplementary data are only as good as what goes into the system at the institution and Altmetric.com databases. Policy documents are being added to the Altmetric.com database all of the time, so it should follow that policy citations will increase, while other Altmetric.com data such as news coverage and social media can only be accurate if there is some kind of unique identifier tied to the research output. If a paper is discussed across the media, but there is no linking to any of its unique identifiers within that communication, it does not get picked-up by Altmetric.com. The Altmetric.com score for our sample is quite likely to be higher than we have reported.

The two papers that received the highest number of policy document citations Peasgood, 2008, Brazier/Ara, 2008 received almost no media attention according to Altmetric.com. Peasgood, 2008 received notable attention via the 780 Mendeley users who had saved a copy of the paper to their database but very little other attention. Brazier/Ara, 2008 received less Mendeley attention with 61 saves and no other social or traditional media attention. This may suggest three things, first, that these are niche pieces of research that are not news or social media-friendly. Second, that it may be a function of date, that these were published at a time when social media was still quite embryonic and sharing this research would have happened more if it was published today. Third, that the research has been shared but without any corresponding unique IDs attached which would mean Altmetric.com would fail to pick it up. The latter is much less likely due to the number of Mendeley Saves that have been recorded in the system.

While acknowledging that this case study is based on a small sample, the in-depth analysis indicates that the number of policy documents identified by Altmetric.com is also likely to be an over-estimation of the true number of “policy documents” citing a research article. It is apparent from Table 4 that, based on this sample, up to 14% of research articles might have between 20 and 50% of duplicate “policy documents” in their assigned numbers. It is also noteworthy that not all documents identified from “policy sources” should be considered “policy documents,” under any conventional definition. Four citing “policy documents” in this sample were clearly standard peer-reviewed journal articles, but were categorized as policy documents citing the research articles by Moreno-Serra, 2012 (Hendriks et al., 2014) Lomas, Woodward, 2009 (MacDicken, 2015), and Brazier, 2005 (Armstrong et al., 2013; Holmes et al., 2014). The quality of ascribed policy sources might potentially be an issue of concern, very much like that of predatory journals where argument can ensue as to the authenticity and credibility of a journal (Cartwright, 2016). Altmetric.com will need to ensure that they maintain a quality system that only uses legitimate policy sources. Users of Altmetric.com can suggest policy sources, so it is essential that a peer review process of such sources is employed.

The earliest policy document in the sample of 14 research articles was from 2006, but only 3/175 policy documents identified by Altmetric.com in this sample were from before 2010 even though 5of the 14 research articles were published between 2002 and 2007 [McIntosh, 2002 (economics), Brazier, 2005, Weich, 2007 (both health economics), Webb, 2006 (public health), and Freckleton/Gaston, 2007 (environmental science)]. It is not clear if this reflects a natural time lag to policy impact for research of this type, or the current limitations of the Altmetric.com search. The “policy impact” of some research articles was incredibly rapid: the six papers by Wyn Morgan, 2010 and Moreno-Serra, 2012 (both economics); Storey, 2009 and Coleman, 2013 (both medicine); Lomas/Woodward, 2009 (environmental science), and Rushton, 2015 (public health) all appeared in policy documents the year after they were published. Interestingly, all of these “early” policy documents were international and five of the six papers were international collaborations (Narin et al., 1991). This suggests that, in some disciplines, international authorship might facilitate and speed-up international policy impact.

It is perhaps not surprising that the majority of the research with greatest policy impact had been undertaken within the fields of economics and health (see Tables 1 and 4): alongside education, these are arguably the two major areas of policy development nationally and internationally (Haunschild and Bornmann, 2017), with the environment being another focus of policy work (Bornmann et al., 2016). It is, therefore, no surprise to see the relative numbers of research articles, categorized by Faculty, with medicine and health far outstripping any other discipline.