Michell (2003) argued the relevance and appropriateness of psychological measurement is almost universally assumed by psychological researchers. Although this does not constitute an obvious attempt to persuade, that very few psychological researchers question the feasibility of psychological measurement could be seen as a form of implicit persuasion that pervades both theoretical and empirical psychological research discourses. Of course, there are more explicit forms of rhetoric surrounding psychological measurement validation. The very objective of validation research is to provide compelling evidence that a measure or measurement data are valid in one or more of the many senses that exist of psychometric validity. Such research clearly plays an important role in persuading readers and consumers of research that a given measurement tool meaningfully quantifies the putative trait it was designed to measure or assess. In fact, it is now very common in empirical research reports to include evidence for justifying the use of the measures used in the study at hand.

The importance of providing persuasive evidence for measurement tools is also reflected in methodological standards and guidelines of the discipline. For example, the American Psychological Association (APA) Publication Manual (American Psychological Association, 2020) specifies an array of journal article reporting standards (JARS),⁵ including for reporting psychometric information concerning measurement data, the instruments used to generate these, and all other relevant psychometric information. Although clear reporting standards are essential within any scientific discipline, it is important to acknowledge the potential drawbacks Bazerman (1987) and others have identified that accompany overly rigid codification of research reports. Perfunctory reporting of psychometric information is a poor replacement for clear demonstration that the measures used and measurements generated in research studies are appropriate for study objectives.

In our article sample,⁶ we found examples of explicitly persuasive references to “important findings,” “substantial links,” “strong indicators,” and “robust” measures (e.g., models, effects, etc.), and “rich and informative” theoretical models. Some of these claims were not supported directly with empirical evidence and in some cases even accompanied weak empirical evidence, counter to the descriptions of “strong” or “robust” findings. We also found less direct appeals to the importance of study findings, such as references to the production of “useful” knowledge, “novel findings,” “advancing” knowledge in face of paucity of research or “gaps in the literature” and references to “confirming,” “reaffirming,” “reinforcing” expectations or findings from previously published research. Not surprisingly, most articles in our sample made as least one reference to “reliable” or “valid” measures or to the “reliability” or “validity” of the measures used in the study, over half of which (29 articles for “reliable”/“reliability” and 24 articles for “valid”/“validity”) either reported no direct evidence or vaguely gestured to previously published psychometric evidence. Examples of each of these kinds of explicitly persuasive forms of rhetoric are given in Table 2.

As with methodology discourse practices generally, there are some expressions and turns of phrase that have become prevalent in psychological researchers’ reporting of psychometric properties. As first illuminated by Weigert (1970), it is extremely common for psychological researchers to merely state that the measures used are “reliable and valid” or have “good,” “acceptable” or “sufficient” reliability and validity, often with no definitions of or distinction made between these concepts or evidence provided for the putative reliability or validity of the measurements or measurement instruments in question. The use of such rote expressions presents numerous problems, including that reliability and validity are quite different psychometric properties and, in the case of validity, bear on multiple different aspects of measures and measurements and uses thereof; that both may be assessed with different metrics (depending on the nature of the scale of measurement); and that reliability is required for validity but not vice versa. Another problem is that ordinary and technical senses of reliability become conflated when references are made to reliable and valid measures as opposed to of measurements (i.e., data): To state that a measure (i.e., the measurement instrument itself) is reliable (i.e., dependable, suitable) is quite a different claim than to state that measurements (i.e., scores or data from administering the measure) have strong psychometric reliability (i.e., a low ratio of error variance to observed variance of scores on a random variable). Another example of rote-like reporting on psychological measurement is the common practice of cursorily reporting only traditional aspects of validity (i.e., content, criterion-oriented [predictive and concurrent] and construct), which fails to reflect the seven decades of validity theory and methodology since Cronbach and Meehl’s seminal 1955 article (Cronbach and Meehl, 1955). In which validity was narrowly conceptualized in terms of these three broad types.

In our sample, phrases combining reliability and validity into a seemingly single psychometric property (i.e., “reliability and validity,” “reliable and valid”) did appear in the main body of some of the articles in our sample (see Table 2). The descriptor “good” was used often and to qualify everything from general reliability and validity or “psychometric/measurement properties” to specific kinds of validity (e.g., “model fit,” “convergent”) or reliability (e.g., “test–retest,” “internal reliability,” “stability,” “agreement”). There appears to be at least some degree of rhetorical motivation for these appeals to “goodness,” given that typically little elaboration was provided. Such underspecified claims appear to rhetorically stand in for any direct evidence of the psychometric properties of the measure being used to generate data for the study.

The use of metaphors in scientific discourse is hardly rare and there have been many celebrated cases in the physical and life sciences (e.g., Bohr’s “planetary” model of the hydrogen atom; evolutionary “tree” of life; DNA as a “twisted ladder”). Psychological measurement discourse also contains some commonly used metaphors, such a “tapping” “probing,” and “emerging” in reference to putative fundamental factors or “constructs” said to “underlie” an observed correlation matrix of a set of item or subscale scores. Item-level scores are framed as “indicators” of “latent” factors, the latter of which are sometimes described as “driving” observed relations among item-level or subscale scores. Other common literary styles include the use of passive voice (e.g., “the measure was administered to…”; “…was assessed by…”) and nominals in place of verb clauses (e.g., “…measure the construct of extraversion”) of the kind Billig (2011) has identified. Both the uses of passive voice and nominalization of actions and activities of persons into traits presumed be “tapped” or “probed” by psychological measures constitute examples of depopulating texts, whereby the specific researchers making and acting upon decisions about the measurement tools used in their research become obscured. Such discourse styles serve a “rhetoric of scientificity” (Bourdieu, 1975) which is intended to give the impression that the research was conducted rigorously and objectively and, therefore, the findings can be trusted.

In our sample, each of the articles contained metaphors of one kind or another. The most common terms were “tap” (or “tapping”) in relation to the phenomenon putatively measured or assessed and “reveal” (or “revealing”) in reference to data or findings. We found that the terms “tap” and “reveal” were used to convey that measurement data had unveiled an underlying or latent realm. Across the sample, other common metaphors were “emerge/emerging” and “detect/detectable/detection.” More unique metaphor use was exemplified by “metaphorical storytelling” (Carlston, 1987), in which a concept or phenomenon is elaborated through a narrative style that relies on the use of metaphors. Examples of the use of these terms and discourse styles in our article sample are listed in Table 2.

We also found that the use of passive voice was ubiquitous in our article sample, appearing multiple times in every article (e.g., “was evaluated,” “was assessed,” “were measured,” “were observed,” “were obtained,” etc.). It was also common, for example, to see such references to the administration of tests such as: “The Structured Clinical Interview for the DSM–IV, nonpatient edition … was administered to assess for Axis I DSM–IV disorders” (Article 15; emphasis added). This example is particularly noteworthy as the assessment tool in question is not a survey or trait measure, but a clinical interview, something that is inherently grounded in human interaction. To remove the interviewer from the “administration” of this test is indicative of the rhetoric of scientificity mentioned above.

In our sample, authors’ use of nominals in place of verbs, as with the use of passive voice, was encountered in every article. This is not surprising, as it is virtually impossible to write efficiently without simplifying at least some verbal clauses with nominals (e.g., “perception” instead of “X perceived Y”), as Billig and discourse scholars have acknowledged. It has become so commonplace in social science writing that it is almost unnatural to describe human actions and capacities in verbal clauses.

Although not a literary device per se, it has become common in psychological discourse for writers to inappropriately ascribe to the subject of a sentence an action or capacity which could not, on logical grounds, be attributed to that subject (see examples in Table 2). Although such misattributions have become more common in contemporary discourse and often do not create too much confusion about what is being stated, they do contribute to the textual depopulating that Billig has identified as having a rhetorical aim.

All forms of discourse at times contain unclear or confusing expressions; psychological scientific discourse is no exception. Although encountering the occasional ambiguous claim does not always create problems, science does not thrive in the face of pervasive ambiguity, and certainly not in unjustifiable statements. The discourse surrounding psychological “constructs” is one area where confusion, ambiguity and, in some cases, unjustifiable claims are commonly encountered.

Discussion of constructs pervades psychological research across theoretical, methodological and empirical domains. Yet, nowhere is there more ambiguity in the psychological measurement and validity discourse then with the “ever-evasive” construct concept (Slaney, 2017). Not only is the ontology of psychological constructs fuzzy, it is often difficult to discern what relationship constructs have to putative psychological “traits” and “mechanisms” (“qualities,” “properties,” “inferred entities,” “processes,” etc.); factors or “latent variables”; or with theoretical concepts, operational definitions, theories, theoretical statements, models or hypotheses (Maraun and Gabriel, 2013; Slaney, 2017). That is, constructs have been variously and confusingly characterized as concepts (e.g., theoretical constructs, hypotheses, models, theories), objects of inquiry (i.e., real but unobservable or only indirectly measurable theoretical entities, or features thereof) and, more generally, as the particular domain under study (e.g., “executive functioning,” “prosociality,” “attachment”). In fact, that psychological characteristics of persons are referred to as “traits,” “mechanisms” and “processes” (and other such objectivist terminology) could be viewed as a form of rhetoric in presuming psychological attributes are just like physical traits, except that they are psychological in nature.

Although ambiguity is not itself an explicit form of rhetoric, if let unexamined it can carry rhetorical weight. For instance, in allowing constructs to be ontologically “fluid,” some claims by researchers might appear stronger on the face of it than they really are. For example, Colman (2006, p. 359) defines a (hypothetical) construct as “a conjectured entity, process, or event that is not observed directly but is assumed to explain an observable phenomenon.” While this all sounds fine on the surface, it is unclear what it means for an “entity, process or event” to “explain” observable phenomenon. Although it has the ring of a precise scientific statement concerning the causal origins of the phenomenon under study, how the presence of causal structures and mechanisms could possibly be picked up by aggregate measurements is left unclear, at best. Similar ambiguities concerning the relationship between psychological constructs, observability and knowledge are prevalent in the discourse, as well as with other measurement-related concepts (e.g., “factor,” “variable,” “latent,” “uni/multidimensional”; see, e.g., Green et al., 1977; Maraun and Gabriel, 2013; Slaney, 2017). As noted by Flake and Fried (2020), such “unjustified measurement flexibility” compromises the extent to which sound evidence about the measures used in a study can be provided which, in turn, casts doubt on the study findings overall.

In our sample, approximately half the articles referred to either of the terms “construct” or “construct validity.” Construct validity was often claimed without direct appeal to psychometric evidence. For example, in some instances construct validity was presumed to be established through the common practice of simply invoking a previous single study. In one article, it was stated that “[s]uch improvements in ADHD knowledge, use of behavioral strategies, and adaptive thinking skills, as measured by our study-specific measures, speak to their potential role as clinical change mechanisms, lending support to the construct validity of our design” (Article 3; emphasis added). The references to both “clinical change mechanisms” and construct validity are vague, leaving unclear what is meant by the terms themselves, what the “construct” that has been validated is and how the results evidence the putative validity of said construct.

In terms of constructs themselves, authors from our sample referred to these without providing much if any indication of the specific natures of the constructs at hand. Several examples are listed in Table 2. Taking these examples together, it is difficult to determine the nature of psychological constructs such that they can be “driven by underlying” emotional states and considered to be “living” and “evolving,” but also to “represent” putative traits (attributes, etc.) and “display” relationships with other constructs.

We found other confusing or ambiguous forms of writing in our sample. These include reference to missing information and hedging. Additional examples include conflating ordinary and technical meanings of psychological concepts as well as conflating aggregate statistical findings with individual-level causal claims. We also found a small number of completely unclear or confusing statements. Examples of each of these kinds of confusing and/or ambiguous claims can be found in Table 2.

It is important to note that rhetoric of science scholars are not united in how they frame rhetoric in science discourse or whether they view it as useful and essential, harmful and misleading, or inevitable or avoidable. Haack (2007, pp. 217–223) draws an important distinction between “reasonable” and “radical” rhetoric of science and between “modes of communication that promote the epistemologically desirable correlation, and those that impede it.” She contrasts between two very different scenarios, one in which a scientific claim is accepted because clear and strong evidence is clearly communicated and the other in which a scientific claim comes to be accepted in the absence of good evidence because it is promoted by means of “emotive language, snazzy metaphors,… glossy photographs, melodramatic press conferences, etc.” (p. 223). Whereas Haack describes the first scenario as legitimately persuasive, she views the second as “strictly rhetorical.” Simons (1993) echoes something similar, noting that rhetorical argumentation does not necessarily make for bad argumentation; however, the slope from rhetoric to fraud may be slippery (Simons, 1993). More optimistically, Carlston (1987) characterizes an intertwining relationship between rhetoric and empirical science, wherein “empirical efforts complement but do not replace rhetorical practices, and rhetorical analysis illuminates but does not invalidate empirical pursuits,” and both are legitimate tools for accumulating “useful understandings and knowledge” (p. 156).

For example, on the use of scientific metaphors as one potential rhetorical strategy, Haack (2007) concedes that although they “oil the wheels of communication” and can be a source of new and important avenues of inquiry, “their worth…depends on the fruitfulness of the intellectual territory to which these avenues lead” (p. 227). Further, Haack notes, a given scientific metaphor may lead scientists in different directions, some better, some worse. As Nagel (1961; as cited in Carlston, 1987) warned over six decades ago, the use of scientific metaphors can be detrimental if the limits of their uses are not properly acknowledged and attended to.

It is fair to ask why scientists would not genuinely wish to persuade readers and consumers to accept research findings they believe are based on strong scientific practice. We agree with Haack that it would be quite counter-intuitive for psychological or any other researchers to avoid making persuasive claims that their research findings are both valid and important. At the same time, it is not always fully clear or agreed upon as to what constitutes “strong” or “good” evidence. Simply claiming strong or good evidence is questionable rhetoric. Moreover, there is no necessary connection between radical (poor) rhetoric and bad (weak) evidence: One can use radical rhetoric in reference to valid and strong evidence and reasonable rhetoric in reference to poor evidence.⁷ On the basis of the current sample of psychological research reports, we see that although some uses of rhetorical writing are relatively harmless (e.g., some nominalization, especially when its use is explicitly justified as descriptive efficiency) or even useful (e.g., metaphorical “story-telling” to clarify a concept), others create ambiguity, at least, and outright confusion, at worst. For example, sometimes using “variable,” “factor,” “construct,” etc. interchangeably is harmless, as the intended meanings of these terms in some contexts need not be precise (e.g., in highly general references to the phenomenon under study); however, in other instances, conflating these terms can be truly confusing, such as when constructs are portrayed as theoretical (explanatory) models and at the same time the putative trait measured by a given instrument. Clearly a construct cannot both be a theory and that which is the subject of the theory. Moreover, reifying aspects of psychological functioning through nominalization and other styles of discourse (e.g., “trait” terminology) can also affirm naïve naturalist and realist views on the nature of psychological reality, thus obscuring important conceptual connections between ordinary and scientific senses of psychological concepts (Danziger, 1990; Brock, 2015; Slaney, 2017; Tafreshi, 2022; Tafreshi and Slaney, in press).

Of course, the answer to this question depends on who you ask, as even rhetoricians are divided on the question of where rhetorical analysis fits within the grand scheme of science (Simons, 1993). As noted at the beginning of the paper, we view examining rhetorical and other discourse practices as an important part of metascience, a primary aim of which is to improve science through better understanding of science (Ceccarelli, 2001), or of a given discipline or area of study (Overington, 1977) as it evolves within current social contexts. As such, it constitutes a part of recent movements within the discipline to acknowledge and address fundamental problems with psychological research (e.g., replication crisis; fraud; identification of QRPs, QMPs, etc.) and, in so doing, improve psychological science (e.g., Society for the Improvement of Psychological Science [SIPS]).⁸ We emphasize psychological measurement discourse not because it is unique in involving rhetorical features but because psychological measurement – even if not always explicitly acknowledged – provides the foundation for psychological research methods, more broadly. That is, a prevalence of questionable measurement practices “pose a serious threat to cumulative psychological science” and, yet, have received much less scrutiny and attention than failures of replication and other QRPs (Flake and Fried, 2020, p. 457), neither of which can be fully understood in the face of potentially widespread invalidity of the psychological measurement tools that generate the data which are the inputs for other psychological research methods.

It is also important to acknowledge that rhetoric and other discourse practices that might misrepresent the phenomena under study or otherwise create ambiguity or confusion occur neither in isolation nor in a vacuum. Most psychological research reports, including those in our sample, have been subject to peer and editorial review prior to publication.⁹ Yet, problematic discourse practices, such as those we have identified, manage to make it past the peer-review and editorial filters. This signals that the use of confusing or unclear language (rhetorical or otherwise) in psychological research discourse is a systemic problem, not to be blamed just on individual researchers. As with other QRPs that threaten the integrity of psychological research, a response is needed to address the questionable discourse practices in psychology that have been illuminated here and elsewhere. How researchers frame their theoretical positions, methods choices, the data that arises from their implementation, and the interpretations they make of findings should be, we argue, an essential part of the discussion about QMPs and QRPs. The upside is that illuminating the detrimental effects of such practices can, if taken seriously, be rectified by broad implementation of training in such areas as philosophy of science, metatheory, and scientific writing for psychology (Billig, 2013; Slaney, 2017; Kail, 2019; Uher, 2023). We believe that exposing pervasive hidden assumptions researchers take into their research can influence how reflective researchers (and, by extension, the discipline) will be regarding the relevant subject matters they are concerned with. We see the current work, and that of other critical methods scholars, as making important contributions to current discussions about methodological crisis and reform.