The development of the Internet, the Web, and, lastly, social networks has gradually widened the spectrum of tools that researchers can use to generate content (not only peer-reviewed content, but all sorts of materials related to research activities as well) and to communicate and disseminate their results. In addition, the nature of these tools has made it possible to register new pieces of evidence (metrics) about the activities and actions of researchers, some of which are related to activities that researchers had already done in the pre-web era, whereas some others are completely new. These metrics have also broadened the possibilities to assess the impact of authors themselves [author-level metrics (AutLMs)] as well as the impact of the documents they produce, disseminate, and use (article-level metrics).

The first pieces of evidence of academic impact in the Web that were quantified were the ones related to visits (the pioneering log analyses) and hyperlinks. Those were the first of an extensive corpus of scientific literature on link analysis (Thelwall, 2004; Thelwall and Kousha, 2015), which can be further separated into the studies that focus on web impact (links received), on the one hand, and the ones focused on connectivity (graphs), on the other hand. Therefore, the beginning of Altmetrics (understood as alternative metrics for the study of academic impact) dates back to the last decade of the twentieth century, before even the term Altmetrics was coined.

Obviously, these studies only analyzed few metrics, which were severely limited by the tools available at the time to collect the data (mainly search engines), and their orders of magnitude—although far superior to those of other metrics like citations received—were still low at the author-level and document-level, only reaching high levels in certain aggregation levels, such as journals or universities.

It was not until the widespread adoption of the so-called Web 2.0 or Social Web that the number of metrics available and their orders of magnitude skyrocketed. In addition, the Social Web brought a change in the reference system, changing the focus from the website (that belongs to a person or an organization) to user profiles and the resources with which the user interacts (users can mention, download, tag, share, vote, link, or disseminate a resource, as well as interact directly with other users).

As a response to the transformations that scientific communication (and the way it is measured) is undergoing, the Altmetric Manifesto¹ came to light in October 2010. This document offers incontrovertible pieces of evidence of the need to observe, quantify, and understand the academic activities that are already taking place in social network platforms (Priem and Hemminger, 2010). Since then, scientific literature on altmetrics has boomed, becoming one of the hottest topics in Bibliometrics (Weller, 2015; Sugimoto et al., 2017). The number of studies that include the term Altmetrics in the title, abstract, or keywords (published between 2012 and 2016) is 283 according to Scopus (Figure 1). Moreover, this list does not include studies that, under the umbrella of “link analysis” or “webometrics,” take an “altmetrics” approach, although they do not use that term necessarily.

Within Altmetrics, the study of the so-called AutLMs can be considered a specific line of work (Fenner, 2014). Obviously not all these metrics fall necessarily within Altmetrics. For example, the h-index (one AutLM among others) can be computed using citation databases such as Scopus or WoS (Bar-Ilan, 2008; Wildgaard et al., 2014; Wildgaard, 2015), but this indicator has thrived in the web environment. In this study, we consider AutLM as metrics that originate from social network platforms, web academic profiles, repositories, or any other online application or environment that provides performance and impact data (not necessarily scientific but academic in the broad sense of the term) focused on authors. These metrics can be categorized in the following dimensions: bibliometrics, usage, participation, rating, social connectivity, and composite indicators (Orduna-Malea et al., 2016b).

Among the studies that discuss AutLM, there are those that address the issue of understanding the underlying relationships among the various metrics. For example, inter-platform correlations (correlations between the same metric according to different platforms; for example, the number of followers of an author in Linkedin, ResearchGate, Twitter, and Academia.edu) or intra-platform (correlations between different metrics in the same platform; for example, citations received, Reads, and Followers, all available from ResearchGate). In addition, these studies can be approached from an institutional (Mikki et al., 2015; Ortega, 2015b) or disciplinary point of view (Martín-Martín et al., 2017).

A disciplinary study adds special context to the analysis of correlations among AutLMs. By focusing in a population that belongs to the same scientific community (in which a priori we may assume certain homogeneity as regards the use of academic profiles services as well as productivity and citation patterns), we may unveil certain relationships that perhaps do not occur (or arise to a lower degree) in other disciplines. Studies at the institutional level (generally multidisciplinary) might mask these behaviors.

Considering apart other external variables (e.g., the popularity of the author and/or his/her online strategies to disseminate works), other factors may affect correlations between AutLMs in disciplinary studies, in particular, the authors’ behavior patterns (the authors’ products) and the type of platform used to disseminate their work (the authors’ display window), which shape the objectives of this study.

Regarding the first factor, there are authors who conform to a formal pattern, based on publishing in journals indexed in the JCR (Journal Citation Reports), and authors with more informal publication patterns, who publish other typologies of documents such as reports, presentations, working papers, etc. There are even authors who conform to a hybrid pattern, who publish a wide variety of document typologies.

As for the second factor, both the degree of exposure of an author in a platform (quantity of available production) and the features and characteristics of an academic profile platform will determine the performance of an author. In addition, the demographic aspects of the platform (number of authors from a scientific community with a profile) also affect the orders of magnitude of the metrics available.

Given that each “display window” offers its own particular side of the work of an author, failure to measure all these different sides may result in a biased measurement because it was done using incomplete data. Moreover, it is not enough to look at all these display windows, it is also necessary to analyze the products that “are sold” in each display. Some will be sold (receiving citations, reads, downloads, etc.) more than others depending on the typology of the product (articles, presentations, reports, etc.).

In this sense, the main goal of this study is to verify the existence of diverse behavior patterns in academic production and impact, both among members of the same scientific community (inter-author variability) and for a single author (intra-author variability), as well as to find out whether this fact affects the correlation among AutLMs in disciplinary studies.

In order to carry out this study from a disciplinary perspective (thus assuming certain homogeneity among the researchers in the sample), we decided to select the field of Bibliometrics as a case study. Because of its own nature, it is assumed that authors in this discipline are familiar with metrics and academic profiles.

To date, there have been various studies addressing AutLMs by analyzing the field of Bibliometrics. Specifically, Bar-Ilan et al. (2012) and Haustein et al. (2014) carry out mainly demographic studies (most used academic profile services by a sample of authors in the discipline). Martín-Martín et al. (2016) carry out the most exhaustive study to date (811 authors), generating a correlation matrix of 31 different AutLMs.

However, the study of the existence of different academic behavior patterns inside a same discipline (in this case Bibliometrics) and its potential influence in the correlations between AutLMs has not been addressed before. Therefore, in order to answer the main goal mentioned above, we pose the following research questions:

RQ1: Is it possible to detect different academic behavior patterns among authors working in the same discipline (inter-author variability) using the AutLMs available in online academic profile services?

In order to answer the research questions, we have selected a sample of authors from a discipline (in this case, Bibliometrics) and extracted a number of metrics about these authors available in various platforms (see Table 1). This sample was used to detect general patterns of behavior among the authors (inter-author variability). In addition, in order to find out the causes of that behavior, an in-depth analysis of an author is also carried out as a case study (intra-author variability).

The sample of authors was extracted from the database Scholar Mirrors,² a web application that compiles an exhaustive list of authors that have published either primarily (specialists) or sporadically in the field of Bibliometrics, and related disciplines (Informetrics, Scientometrics, Altmetrics, Webometrics). This web application provides a number of AutLMs extracted from the main academic profile platforms (Google Scholar Citations, ResearcherID, ResearchGate, Mendeley, and Twitter). The data available in this web application were collected in July 2015.

Once the main sample was obtained (made up of 811 authors), we proceeded to check whether those authors currently (September 2017) have a public profile in a series of academic profile services (Google Scholar Citations, ImpactStory, and ResearchGate). The selection of these three academic profile platforms was motivated by the need to analyze three of the main dimensions of online academic impact and their correlations: citations received (Google Scholar Citations), reads (ResearchGate), and, lastly, Online mentions in social network platforms (ImpactStory). Each platform has its own peculiarities:

Citations: Google Scholar Citations automatically computes the total number of citations received by a document. In this study, we will use the sum of all the citations of all the documents included by the authors in their profiles. There is no restriction on the origin of the citing documents nor in the documents available in the profile (cited documents) beyond the technical requirements imposed by Google Scholar.³

In the case of Google Scholar Citations and ResearchGate, the profiles of the authors in the sample were extracted directly from Scholar Mirrors, although it was checked that they were still functional. All ResearchGate accounts were active and claimed profiles.⁵ As regards ImpactStory, first we checked whether the author had an ORCID profile,⁶ because the ImpactStory profiles currently include data only from ORCID profiles. To check the existence of the profile, the search feature available in ORCID was used. In order to avoid incorrect identifications, the affiliation data in the documents recorded in the ORCID records were matched against the affiliation information available in Google Scholar Citations and/or ResearchGate. Using these ORCIDs, it was very simple to find out whether an ImpactStory profile were available, because the URL to the ImpactStory profile always contains the ORCID. For example, if the ORCID of an author is “0000-0002-1989-8477,” the ImpactStory profile will be accessible from https://profiles.impactstory.org/u/0000-0002-1989-8477.

In the cases where it was not possible to disambiguate the profile, it was determined that the author did not have an ORCID (nor, therefore, an ImpactStory profile). For this reason, the number of authors with an ORCID might be slightly higher than reported in this study. Nevertheless, this problem did not arise frequently (in less than 5% of the studied sample), and so the effect of any potential mistakes on the results is expected to be low.

511 authors with an ORCID profile were found, of which 393 had an ImpactStory profile as well (although 2.7% of these profiles were empty or without any mentions). Regarding ResearchGate, 315 authors in the sample were found to have a profile in this platform (Figure 2). The number of authors with a profile in the three platforms is, therefore, 315. This is the sample that was used in the analysis for this study.

Once all the profiles were identified, we proceeded to automatically extract the following AutLMs (Table 1).

Since the number of Online mentions reported by ImpactStory is mainly from Twitter (see Discussion), it was deemed appropriate to count the number of Tweets published by the authors in their personal Twitter accounts, in case this factor might affect the number of mentions received. The existence of personal Twitter accounts for the authors in the sample was checked in Scholar Mirrors and through direct searches in Twitter.

Once all the data had been obtained, a statistical analysis was carried out to answer the research questions previously proposed. Regarding RQ1, all the correlation coefficients between the various metrics were computed using Spearman’s method (α < 0.1) to account for the known non-linear distribution of web data. In addition, the same correlations were computed for the set of authors in the 10th percentile (P₁₀) for each of the academic impact metrics (Citations, Reads, and Online mentions), in order to check the potential existence of differences when compared to the entire sample (in this case, the highest ranked researchers according to each of the three metrics).

After computing these correlations, a cluster analysis (k-means) was carried out with the goal of identifying the various patterns of online academic behavior according to the performance of authors in the various academic profiles (inter-author variability). That is to say, the potential existence of a group of authors who stand out in the three measured impact dimensions, of other group of authors who only stand out in one of the metrics and have a lower performance in the remaining dimensions, or any other possible combination. For this, up to eight classes were defined to identify all the possible combinations on the basis of the three dimensions analyzed (Citations, Reads, and Online mentions). In addition, a principal component analysis (PCA, with varimax rotation) was carried out in order to visually represent in a bidimensional space the relationships between each of the three studied dimensions.

Lastly, an ego-analysis of one of the authors (the first author of this article) was also carried out to answer RQ2 (intra-author variability). For this, all the publications available in his Google Scholar Citations profile were exported (n = 119), and document types were assigned to each publication (journal article, conference paper, book, book chapter, presentation, report). For each of the publications, the number of Reads in ResearchGate as well as the number of mentions according to ImpactStory was gathered. This procedure can be justified on the basis of the high amount and wide variety of documents published by the author, who has both a formal profile (journal articles published in journals indexed in Web of Science and/or Scopus) as well as an alternative profile (reports, conference presentations, book chapters). It is also justified because of the certainty that the profile is a faithful representation of all the author’s publications, across the three platforms studied, and, of course, because of the in-depth knowledge of his work, a necessary aspect to be able to contextualize the results obtained.

All the data were collected and analyzed during September 2017.

As regards AutLMs, a non-linear distribution in the data extracted from the three platforms (Google Scholar Citations, ResearchGate, and ImpactStory) has been found. There are few authors with a high performance, and a long tail with moderate, low, or null performance. Moreover, the high performance authors are not the same across the three studied dimensions of impact (Citations, Reads, and Online mentions). The lack of correlation might be explained by the fact that each platform offers different documents, targeted to different audiences.

This fact has facilitated the identification of different patterns of online academic behavior in the studied platforms (RQ1). Some authors present a markedly formal performance (Citations, mainly to journal articles) while other authors stand out in Reads (both to articles and to other document typologies), or in Online mentions (mainly Articles). Combined patterns have also been found (high performance in Citations and Reads, and low in Online mentions). This issue evidences that the analysis of a single platform, not even considering of the demographic aspects related to the population of a discipline that is reflected, can mask the performance of an author who has particularly high or low values in any given platform.

Lastly, the ego-analysis has allowed us to confirm the existence of authors with different patterns of online academic behavior depending on the types of documents that they publish (RQ2). In this case, we could observe the existence of working papers with a high amount of Reads and Online mentions, as well as the existence of a large group of articles with a lower number of citations. That is, certain typologies are generating an impact (Reads in ResearchGate) that cannot be observed in other platforms. Again, the different nature of the research activity (article: generating knowledge; report: application of knowledge to solve a problem; educational materials: knowledge transfer, etc.) determines everything. The people who cite are scientists, the same ones that produce scientific knowledge, whereas practitioners read but do not cite as much, so it is less likely that they would cite other studies.

This fact again brings us to the need not only of considering different online academic profile platforms (in order to capture different impact profiles) but also to categorize the type of impact according to the document typologies, because a general analysis of authors might mask their actual impact.

All this makes us question the usefulness and precision of the correlation analyses of AutLMs within a discipline that have not taken into account inter-author or intra-author variability to model the multidimensional impact of authors. This is one of the aspects in which Altmetrics studies should focus their attention from now on.

EO-M: co-designed the original research, gathered data, performed statistical analysis, and co-written the manuscript. EDL-C: co-designed the original research and co-written the manuscript.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.