Meta-analyses document small to medium-sized associations between personality measures and job performance (Barrick and Mount, 1991; Tett et al., 1991; Salgado, 1997; He et al., 2019; Watrin et al., 2023), and the use of personality tests in applicant screening has been shown to improve firm performance and selection outcomes (Ichniowski et al., 1997; Autor and Scarborough, 2008; Hoffman et al., 2018). Despite their widespread use in selection, evidence concerning the effects on protected groups of personality testing in hiring is limited. Autor and Scarborough (2008) found that minority hiring rates at a firm's establishments were no different when screening was based on personality testing than when based on informal interviews, and meta-analyses in psychology have found little evidence of potential adverse impacts from personality testing on women (Hough et al., 2001) and minorities (Foldes et al., 2008). The studies in these meta-analyses, however, primarily examine group differences on unincentivized measures even though tests in selection are necessarily incentivized. Furthermore, the U.S. Equal Employment Opportunity Commission (EEOC) in 2018 found it probable that Best Buy adversely impacted applicants based on race through its use of personality tests (U. S. Equal Employment Opportunity Commission, 2018).

This study investigates how incentives influence gender and race differences on personality tests in a within-subject, laboratory experiment. Subjects first completed Big Five personality measures without incentives.¹ About a week later, the subjects were given a job ad and took personality and IQ tests. Subjects were informed that a bonus would be paid to subjects who best met the hiring criteria. In the Extroversion (Introversion) treatment, the ad indicated that an extrovert (introvert) was desired, while in the No Priming treatment the ad contained no information about desired personality.

Using the unincentivized measures, women in our sample were more agreeable but less open and emotionally stable than men. Incentives, however, proved a great equalizer as we fail to reject the gender equality of the mean scores for all traits in all treatments using the incentivized measures.

Turning to racial differences, East Asian subjects were less extroverted, open, and emotionally stable than White subjects using the unincentivized measures. These differences, however, are not attenuated in many cases when measured with incentives. Given a clear indication of the desired personality trait in the Extroversion treatment, the East Asian-White gaps in extroversion and openness attenuate. Given no information in the No Priming treatment or counterintuitive priming in the Introversion treatment, the East Asian-White gaps fail to attenuate and in some cases gaps emerge that did not exist using the unincentivized measures.

Our “other race” category consists of non-White, non-East Asian subjects. These subjects were more conscientious than White subjects using the unincentivized measures, but there were no significant differences for other traits. By contrast, we find significant differences using the incentivized extroversion, agreeableness and emotional stability measures in the Introversion treatment. For conscientiousness, the gap favoring subjects of other races using the unincentivized measure reverses signs and is insignificant using the incentivized measure. Similar patterns exist in the No Priming treatment, suggesting that unusual personality requirements or the absence of cues result in incentivized measures that disadvantage subjects of other races relative to White subjects.

To assess whether hypothetical hiring based on incentivized measures would adversely impact women or minorities, we apply the EEOC's “four-fifths rule” under which a protected group is deemed adversely impacted by a hiring practice if the fraction of the protected group selected is < 80% of the fraction selected from the reference group. Using this rule, we find little evidence that women would be adversely impacted by selection based on any incentivized measure in any treatment. By contrast, East Asians and individuals of other races would be adversely impacted by selection on incentivized personality scores in half to two-thirds of the trait-treatment combinations in the experiment.

The data used in our study were originally examined in McGee and McGee (2024), which tested hypotheses about the correlations between incentivized and unincentivized personality measures, the influence of subject characteristics on incentivized measures, and the mediating effects of information about desired traits. That study documented that incentivized personality scores were weakly to moderately correlated with unincentivized personality scores, but the information provided in the job ads had little effect on these correlations. Most relevant for our study, McGee and McGee (2024) found little evidence that the subject characteristics they hypothesized would influence incentivized personality measures (e.g., IQ, Machiavellianism, impression management, willingness to lie) could explain the heterogeneity in these measures.² By contrast, our secondary analysis of the same data suggests that racial differences—characteristics that were not hypothesized to influence incentivized personality measures when the experiment was conceived—might underpin some of the heterogeneity in incentivized scores.

Our study makes two primary contributions. First, we contribute to the literature on non-cognitive traits by highlighting that incentives and the information available to test-takers can influence gender and race differences on personality measures. In their review of personality psychology, Almlund et al. (2011) stress accounting for the role of incentives and situations when measuring personality. Our findings suggest that these factors may also influence measured group differences. In this respect, the study complements (Gneezy et al., 2019), who find that incentives influence group differences on cognitive tests between students in the United States and China.

Second, we demonstrate that the racial differences on incentivized Big Five measures in our sample are large enough to adversely impact minorities in hypothetical selection. Understanding the effects of personality testing on protected groups is particularly important as algorithmic screening using personality tests becomes more widespread in recruitment. While several studies evaluate the biases that machine learning may introduce in hiring (e.g., Li et al., 2020; Cowgill, 2020), there is little evidence on the biases that personality tests may introduce.³ Especially noteworthy from our study is that where racial differences emerge on incentivized tests where no similar differences exist on the unincentivized tests, in every instance these racial differences would disadvantage minorities relative to their white peers. Personalities are assessed in many contexts, and selection on incentivized personality measures has been shown to influence the racial distribution of selected individuals.⁴ Were our findings to generalize to other settings, racial differences in personality misrepresentation could disadvantage minorities in outcomes ranging from university admissions to promotion decisions.

Whether personality tests adversely impact protected groups in hiring has been examined largely by extrapolating from group differences on unincentivized tests (e.g., Feingold, 1994; Goldberg et al., 1998; Weisberg et al., 2011). The standardized mean gender and race differences in these studies tend to be small. Hough et al. (2001) report in a meta-analysis that women were less extroverted (d = −0.09), open to experience (d = −0.07), and emotionally stable (d = −0.24) than men but more conscientious (d = 0.08) and agreeable (d = 0.39). Also in a meta-analysis, Foldes et al. (2008) find that Blacks and Hispanics are more conscientious than Whites (d = 0.17 and d = 0.20, respectively), while Asians were more agreeable (d = 0.63) and open (d = 0.11), but less extroverted (d = −0.07) and emotionally stable (d = −0.16) than Whites. While the meta-analyses in Hough et al. (2001) and Foldes et al. (2008) include a few incentivized applicant samples, the incentivized samples in Foldes et al. (2008), for instance, made up only 14% of their database.

Incentives for misrepresentation may matter for group differences for at least two reasons. First, groups may have different notions about optimal personalities (Hofstede and McRae, 2004; Ryan et al., 2021) that result in differences in faking (Fell and König, 2016). Second, group differences in traits result in differences in “opportunity-to-fake” (McFarland and Ryan, 2000; Tett and Simonet, 2011). That is, members of a group with lower scores can increase their scores by more on average than members of the higher scoring group given that the scores are bounded above. This dynamic should attenuate group differences when personalities are measured with incentives for misrepresentation. Leveraging its within-subject design, ours is the first study to document how opportunity-to-fake dynamics influence gender and race gaps on incentivized personality measures.

A within-subject experiment also identifies the effects of incentives without the sampling variation present when comparing applicant and incumbent samples (e.g., Hough et al., 1990; Rosse et al., 1998; Weekley et al., 2004). Hu and Connelly (2021) identified only ten published and seven unpublished within-subject studies comparing incentivized responses in selection settings to unincentivized responses in low-stakes settings—what might be considered the ideal experiment. Studies such as Griffith et al. (2007), Ellingson et al. (2007), and Isaacson et al. (2009) highlight the difficulties of this research: small samples, high non-response rates, and idiosyncratic settings. Ellingson et al. (2007), for example, identify 218 individuals who took tests in a development setting and when applying for actual jobs, but the tests were for different jobs and taken as much as seven years apart.

In the only within-subject study we are aware to examine racial differences on incentivized measures, Isaacson et al. (2009) find no Black–White difference in faking on a conscientiousness measure among individuals completing the measure both in a job application and for research purposes.⁵ They report that men faked more frequently than women, but they do not report means under the honest-reporting conditions making it impossible to assess the role of opportunity-to-fake. By contrast, our study assesses whether faking differs by gender and race on all Big Five traits while accounting for differences in opportunity-to-fake.

We are aware of only two studies that adopt the EEOC's “four-fifths” rule to assess the potential for adverse impacts from personality tests on women and racial minorities. Powell et al. (2011) assess whether women applying for positions at an oil refinery would be adversely impacted by selection on extroversion and conscientiousness scores or the related facet scores. While they find the potential for adverse gender impacts from selection on facet scores, they find little such evidence using the factor scores—suggesting that gender differences at the facet level cancel each other out when aggregated to the factor-level. We likewise find little evidence of the potential for adverse gender impacts based on the incentivized factor measures. In our case, however, the absence of adverse impacts appears to stem from opportunity-to-fake dynamics that attenuate gender differences in the unincentivized measures.

Finally, Foldes et al. (2008) evaluate whether adverse impacts on racial minorities are likely to arise when selection is based on each of the Big Five factors using the standardized mean differences from their meta-analysis and assuming standard normal distributions. They find evidence of advantages for Blacks and Hispanics when selection is based on conscientiousness measures—similar to our findings for non-White, non-East Asian subjects—but conclude that there is some potential for adverse impacts on Blacks when selection is based on emotional stability or extroversion scores. For Asians, they find some potential for adverse impact using emotional stability, but again their inferences are based largely on studies using unincentivized measures.

The study conducts an exploratory analysis using data from the experiment in McGee and McGee (2024). Subjects participated in two sessions. In the Baseline session lasting ~15 min, subjects responded to a Big Five assessment (DeYoung et al., 2007), other psychometric instruments detailed in the Appendix, and demographic questions.^6,7 The demographic questions including race and gender were part of a standard code module we used when initiating experiments, but we did not ex ante have hypotheses regarding the relationships between race and gender and incentivized personality measures. Subjects earned $7 for participating. Afterwards, subjects were invited to sign up for the Evaluation session a week later.

In the Evaluation session lasting approximately an hour, subjects received a job ad and completed the same personality test and Raven's Progressive Matrices (RPM) (Raven, 1998), an intelligence test. The instructions informed subjects that after these tests a $25 bonus would be awarded to one subject in each group of four subjects. After the bonus was awarded, subjects completed other psychometric instruments, a risk preferences measure (Holt and Laury, 2002), and questions about work experiences.

The treatments varied the job description in the Evaluation session given that applicants are often routed to tests from ads, and the information in ads likely influences applicants' understanding of personality demands.⁸ In the Extroversion treatment, the instructions indicated that “All members of a group will receive the same job description. We will then administer an intelligence test and a personality test to determine who to “hire” for the job. We will weigh these two tests according to some criteria and one subject within each group who best meets these criteria will receive a bonus of $25; the remaining subjects in each group will not receive any bonus.” Subjects received an ad for a “client services representative” based on an actual ad. To indicate that an extrovert was desired, the job description incorporated words and phrases associated with extroversion such as “outgoing” and “able to take initiative, be assertive” while attempting to avoid indicating that the firm sought other personality traits.

Subjects in the Introversion and No Priming treatments received the same instructions. In the Introversion treatment, the ad indicated that an introvert was desired through the use of words and phrases associated with introversion such as “contemplative” and “reserved.” In the No Priming treatment, the ad contained little detail beyond listing tasks and requirements common to all of the ads. In contrast to much of the psychology literature on faking, at no point did we direct subjects to fake or suggest strategies for responding.⁹ For each trait, the incentivized measure from the Evaluation session minus the unincentivized measure from the Baseline session serves as our measure of faking.¹⁰

Concerning the differences in incentives between subjects and job applicants, we argue that these incentives are reasonably similar. Most job applicants take personality tests at the same time that they submit resumés and other application materials. Job boards hosting job ads in some cases even provide employers with personality tests for applicants, who presumably take these tests not long after reading the job ad. Similar to our instructions, employers do not explicitly reveal how this information (i.e., resumés, references, test scores) will be used, but applicants understand that a decision will be reached using this information: they will either hear back from the employer or they will not. Personality tests at this stage are unlikely to be used to select applicants to hire so much as to eliminate applicants from consideration.¹¹ Applicants likely understand that in order to be hired, they must avoid being in that fraction of the applicant pool eliminated from consideration based on these early signals to employers in order to advance to the next stage of hiring (typically job interviews). Similarly, subjects in our experiment understand that they must be in the top quartile within their group to win the group. Both applicants and subjects likely view personality testing as a hurdle to be passed, and their test-taking strategies presumably reflect this.¹²

The client services representative occupation was selected as the sort of job to which young graduates might apply. In the 2016 American Community Survey, 25% of respondents in the services sales representative occupation were age 30 or less and 91% had a bachelor's degree or less (authors' calculations). Brenčič and McGee (2023) find in a sample of job ads that 42% of ads in this occupation signaled a demand for extroverts through terms associated with extroversion (nearly twice the proportion expressing demands for any other trait), while less than 1% of such ads indicated a demand for introversion. At the same time, 36% of ads for this occupation used no terms associated with desired personality traits. As such, we view the Extroversion and No Priming treatments as representative of the ads job seekers in this occupation encounter.¹³

In addition to a $7 participation fee and (if applicable) the $25 bonus, subjects were paid $0.20 for each correct answer on the RPM and earnings from one randomly selected realization of a paired lottery of the risk preference instrument.

Between 2014 and 2016, 474 undergraduates at Simon Fraser University participated in both a Baseline and Evaluation session: 167 in the Extroversion treatment, 160 in the Introversion treatment, and 147 in the No Priming treatment. Subjects were recruited via email from the registered subject pool at Simon Fraser's experimental lab consisting of students from all majors. Subjects self-reported their gender and race with 48% of the sample identifying as men. Subjects who identified as White were coded as such, while subjects who identified as Chinese, Filipino, Japanese, Korean, or Asian were categorized as East Asian. All remaining subjects identifying as Indian, Pakistani, Black, Latino, or an unspecified race were placed in the “other race” category.¹⁴ In the sample, 23% of subjects reported being White, 49% East Asian, and 28% another race.^{15, 16}

The experiment did not attempt to balance the treatments by gender and race. In the Extroversion treatment, 57% of subjects indicated that they were women, 19% White, and 59% East Asian. In the No Priming (Introversion) treatment, 49% (51%) indicated that they were women, 25% (29%) White, and 50% (36%) East Asian. Using Pearson chi-squared tests, we fail to reject the hypothesis of independence between gender and treatment (p = 0.67), but we reject the independence hypothesis between the race categories and treatment (p = 0.001). This lack of balance primarily affects our statistical power to detect small group differences, an issue we elaborate on in the Discussion section. See Appendix A Tables 1–4 for summary statistics for all measures by treatment and group.

We investigate how incentives influence gender and race differences on personality measures. Gender gaps on unincentivized measures essentially disappear when subjects have incentives for misrepresentation. The attenuating effect of incentives highlights the importance of “opportunity-to-fake” dynamics: if two groups respond to incentives similarly (as we show that men and women do), then the lower scoring group on unincentivized measures will tend to close the gap when incentives are provided because they have more scope to manipulate their responses.

Incentives have very different effects on racial gaps. For East Asian subjects and subjects of other races, gaps relative to white subjects on unincentivized measures are in many cases not attenuated by incentives, or gaps appear where no similar gaps existed in the unincentivized scores. It should give researchers pause that in every instance where racial differences appear on incentivized personality measures where none existed on unincentivized measures, racial minorities would be adversely affected relative to Whites were hiring based on these measures.

In the experiment, most racial differences on incentivized measures occur where cues are absent or counterintuitive, which might reflect differences in norms and expectations concerning ideal personalities. Differences in cultural norms have been shown to contribute to personality differences in cross-country studies (Hofstede and McRae, 2004). Likewise, Ryan et al. (2021) find evidence of cultural differences in the social desirability of personality traits for Confucian Asian and South Asian contexts relative to Germanic and Eastern Europe. Future research should investigate why racial differences on incentivized personality measures emerge. A potential implication for employers wishing to avoid adverse impacts is that they should be explicit about what they are looking for when screening applicants using personality tests—a suggestion that echoes Ziegler's (2011) observation that providing job information before personality tests may be necessary to “level the playing field” given differences in applicants' prior knowledge of jobs.

We note two important limitations of our study. First, the experiment was not designed to study the effects of incentives on gender and race differences. As a consequence, our exploratory analysis is underpowered for identifying small group differences (i.e., d ≤ 0.2), but we have power >0.5 to detect medium-sized group differences as indicated by the power calculations reported in Panel D of Table 1. In our view, the lack of power to detect plausible but small group differences is a second-order concern given that such differences are less likely to influence hiring. More important is the fact that the racial differences that emerge with incentives in many cases are as large as half of a standard deviation and would almost certainly influence selection.

Second, differences between the laboratory and hiring limit the external validity of our findings. Subjects' incentives were not the same as those of actual job applicants, and our subjects may not resemble the applicants for any given job. We attempted to minimize these concerns by using ads based on an actual ad for a job to which young graduates might apply without directing subjects to fake.

In view of these limitations, we conclude by stressing the need for additional evidence regarding whether personality testing in hiring adversely impacts minorities. Evidence from a single firm as in Autor and Scarborough (2008) cannot settle the issue given the importance of information, applicant populations, and other factors.²⁶ Likewise the evidence from this study is not definitive. Research using larger, more diverse samples ideally in a hiring setting is necessary. Larger samples would offer greater power to identify small racial differences while also permitting the investigation of differences among racial groups under-represented in our study.

The project was approved by Simon Fraser University's Research Ethics Board (2012s0590). The experimental method is described in the Section 3. Further materials can be found in the Appendices. Appendix A reports additional summary statistics. Appendix B provides the instructions, job ads, and the instruments used in the experiment. Appendix C discusses the potential for adverse impacts from hypothetical selection using the realized distributions of subjects' scores rather than assuming that the scores are drawn from standard normal distributions as in the main text.