In 1998, Feldman Barrett and Swim published a theoretical treatment linking the concepts of prejudice detection with the signal detection framework. They applied SDT concepts to propose theoretical rationales for why some individuals may be more likely than others to perceive prejudice in ambiguous situations. Since that time, Feldman Barrett and Swim's (1998) work has received further conceptual attention, having been cited by almost 300 works to date. However, examination of those works suggests that few studies have employed SDT methods in empirical research on prejudice detection. Rather—consistent with Feldman Barrett and Swim's (1998) focus on theory—citations of their chapter most often concern theoretical points, such as arguments that many factors influence prejudice detection and that some individuals/groups may be more vigilant for prejudice than others (e.g., Ashburn-Nardo et al., 2008; Deitch et al., 2003; Frost et al., 2015; Kaiser and Major, 2006; Major et al., 2002a; Major and O'Brien, 2005; McCord et al., 2018; Mendoza-Denton et al., 2002; Purdie-Vaughns et al., 2008; Ragins et al., 2007; Sellers and Shelton, 2003; Swim and Hyers, 1999). Burke (2015) and Padgett and Morris (2023) employed SDT to analyze responses in an officer's dilemma task, but to my knowledge, no studies have yet used SDT methodologies to understand perceivers' interpretations of ambiguous situations as either prejudiced or unprejudiced. Therefore, despite frequent citations of Feldman Barrett and Swim's (1998) paper, SDT methods are still under-utilized in the prejudice detection domain, resulting in continued missed opportunities to advance our understanding of prejudice detection.

Non-SDT-based quantitative methodologies used for examining prejudice detection are unable to disentangle, or unconfound, two aspects of bias detection: (a) individuals' accuracy, or their skill in correctly distinguishing prejudiced from non-prejudiced events, vs. (b) their general tendencies toward labeling ambiguous events as prejudiced or non-prejudiced. Each of these two elements can be interesting and important, as I will discuss further later. Confounding them not only decreases the validity of prejudice detection assessments but also creates missed opportunities for examining interesting research questions. Reiterating and extending the call made by Feldman Barrett and Swim (1998), I will argue that if we incorporate the theory and methods of Signal Detection Theory (SDT) into our empirical research, we can substantially advance our understanding of prejudice detection. SDT's methods provide researchers with the ability to pose and test distinct hypotheses about (a) detection accuracy and (b) general tendencies, thereby opening doors that had previously been closed (Green and Swets, 1966). These techniques can be applied to the study of any form of bias detection, whether it is related to race, gender, sexual orientation, religion, or other social characteristics.

I propose four plausible reasons for the dearth of empirical adoption of SDT in the study of prejudice detection. First, SDT is better-known in some disciplines—such as cognitive psychology and human factors—than in others, such as social psychology or community psychology. As such, many prejudice researchers may be less familiar with SDT Feldman Barrett and Swim, (1998, notwithstanding).

Second, SDT is often considered challenging to learn, regardless of one's discipline (e.g., Anderson, 2015; Fisher, 2014). Introductions to SDT frequently present an array of technical terms within the first few pages, such as “signal,” “noise,” “false alarms,” “hits,” “sensitivity,” “criterion,” “response bias,” “base rate,” “likelihood ratio,” “receiver operating characteristics,” “signal distribution,” and “noise distribution” (e.g., Abdi, 2007; DeCarlo, 1998; Macmillan, 2002; Pastore and Scheirer, 1974). The density of terminology, symbols, and statistical concepts in many introductory treatments may contribute to a perception that SDT is difficult to learn (although, as Anderson (2015) noted, that sense is “deceptive,” as SDT is easier to comprehend than it may appear). To address these first two challenges, I will re-introduce the foundations of SDT to the reader with a minimum of jargon and equations—relying on storytelling and examples to illustrate the major concepts, and to highlight the ways in which SDT methodology can benefit prejudice detection researchers.

Third, SDT is applied across a broad variety of contexts, including physiological responses, memory tasks, medical diagnoses, and more (see Stanislaw and Todorov, 1999). While SDT's broad applicability is one of its strengths, it can also make SDT more difficult to learn. That is, introductions to SDT will typically use generic terms designed to apply in any context, rather than using the terms of a specific context such as prejudice detection. For example, they will typically use the term “signal” instead of “prejudice” and “noise” instead of “non-prejudiced.” Further compounding the potential for confusion, several SDT terms appear very similar to concepts from the prejudice literature, such as “discrimination,” “bias,” and “conservative vs. liberal.” However, their meanings in the SDT context are quite different from their meanings in the prejudice context. Pedagogical scholars argue that it may be difficult for learners who are new to a paradigm such as SDT to abstract its principles to their specific contexts of interest (Anderson, 2015). Therefore, to address this third challenge, I will guide the reader by using terminology from the prejudice context in place of more generic terminology. For example, instead of saying “signal,” I will say “prejudice;” instead of “noise,” I will say “non-prejudiced;” and instead of saying “sensitivity,” I will say “accuracy in prejudice detection.” Although I strive to keep equations to a minimum in this introduction, SDT does require familiarity with the standard normal distribution, so a basic understanding of the normal distribution is assumed.

Fourth, in the context of prejudice detection, there are some practical challenges that need to be addressed, such as the creation of large and valid stimulus sets. Feldman Barrett and Swim's (1998) intent was to present a theoretical treatment of SDT and prejudice detection, and the logistical matters of applying SDT's methodology to the study of prejudice detection received less attention. In this paper, I expand upon those practical methodological issues, providing more guidance to readers on how these methods could be implemented. It is important to recognize that the methods are flexible enough to apply to a variety of different contexts and theoretical approaches to prejudice. I do not wish to impose a single perspective upon the reader; therefore, I present a variety of different options for pursuing this type of research depending on the specific types of -isms, contexts, and theoretical approaches of interest.

In summary, this paper reiterates and extends Feldman Barrett and Swim's (1998) call for SDT to be incorporated into prejudice detection research, with a greater focus on empirical methods. I seek to address the four challenges I listed above, which I believe have contributed to the lack of adoption of SDT methodologies in empirical studies of prejudice detection. To instill readers with confidence that they can employ these techniques, I will intentionally limit the presentation to the basic foundations of SDT. To increase accessibility, I will provide references to R packages and access to syntax that can assist with the procedures. Once readers have become familiar with the basic SDT concepts and metrics, they will be poised to consult additional sources for learning more advanced SDT techniques, if desired.

The primary benefit of incorporating SDT methods into our studies of prejudice detection is that they provide separate metrics for two key aspects of decision-making. As I will argue later, many of the more traditional approaches to prejudice detection that we have been using to this point can be problematic in that they confound these two aspects. Confounding them creates both potential confusion and missed opportunities. First, I will explain that SDT methods provide separate metrics for each of these two aspects.

The first is detection accuracy. This metric reflects the individual's ability to correctly distinguish between prejudice and non-prejudice¹. The second is response tendency. Response tendency tells us how each person tended to respond when they were unsure. For example, one individual may have a tendency toward responding “yes” when unsure if the event was prejudiced, whereas another individual may have a tendency toward responding “no” when unsure. The response tendency metric gives us a number that tells us whether each person had an overall learning toward either saying yes (the situation was prejudiced) or no (the situation was not prejudiced) across situations. A person's response tendency may be associated with individual and situational differences, with the base rates at which events occur, and/or with the perceived consequences of errors Feldman Barrett and Swim, (1998). For example, if someone believes that prejudice is pervasive, their response tendency score may indicate a leaning toward broadly viewing situations as prejudiced. Conversely, if someone believes that their coworkers will derogate them for making claims of prejudice, their response tendency score may indicate a leaning toward broadly viewing situations as non-prejudiced.

When we don't intentionally separate accuracy and response tendency, they can get confounded. To illustrate the problem that may occur, consider Jason, a hypothetical airport security screener whose task is to look for dangerous items in X-ray scans of luggage. In Jason's hypothetical airport, 98% of luggage is benign. Further, Jason's response tendency is to rarely, if ever, flag any bags for searching. If we wanted to assess Jason's performance with a conventional measure, we might calculate the percentage of the time Jason is correct. By that metric, Jason would appear to be a highly proficient detector because he would be correct 98% of the time. However, when we know Jason's response tendency, we understand that his apparently-high success rate only occurs because the overwhelming majority of suitcases do not contain any dangerous items.

Next, consider Camilla, who is also an airport security screener and who works in the same airport alongside Jason. Unlike Jason, Camilla flags almost every suitcase that comes through. Based on the percentage correct, Camilla would appear to be a very poor performer. However, the same problem occurs; Camilla's success rate reflects her response tendency, not her ability. If one were unaware of the screeners' respective response tendencies, one would conclude, incorrectly, that Jason is an excellent screener while Camilla is a terrible screener. Actually, neither Jason nor Camilla has demonstrated any detection ability; they simply respond the same way every time.

Furthermore, we might miss out on interesting opportunities to understand why Jason and Camilla have such different response tendencies. Perhaps Camilla is aware of a recent terrorist threat at another airport, and the increased salience of that threat affected her own response tendency. Perhaps she believes that the world is generally a dangerous place, and that people cannot be trusted. Or perhaps she believes that most people are innocent but is so terrified of the negative outcomes associated with letting a weapon onto a plane that she is willing to endure the complaints of passengers subjected to long lines. Once we can accurately measure people's response tendencies, we can start to ask many interesting and illuminating questions about them.

Now, imagine that instead of being airport security screeners, Jason and Camilla are people who are vigilant for subtle prejudice in various interactions. First, we might be interested in understanding detection accuracy: how skilled Jason and Camilla are in identifying prejudiced vs. non-prejudiced interactions. We might seek to predict why one of them is better at prejudice detection than the other, or we might implement training programs to improve detection accuracy. Secondly, we might be curious about factors influencing their response tendencies. Do they have different cost/benefit perceptions about what will happen if they make a mistake? Perhaps Camilla, but not Jason, believes that failing to perceive prejudice could create dangers or risks (i.e., trusting the wrong person or entering a physically dangerous setting). Perhaps Camilla believes prejudice occurs frequently, whereas Jason believes it is rare. Their respective response tendencies may be influenced by their worldviews or by individual differences such as social dominance orientation. These types of questions are clearly of interest to scholars studying subtle prejudice detection (e.g., Ako-Brew, 2020; Banks and Landau, 2019; Barreto and Ellemers, 2015; Jenkins et al., 2023; Midgette and Mulvey, 2024; Mirick and Davis, 2021; Parker, 2017; Perry et al., 2015; Stanke et al., 2024).

Thus far in the prejudice detection literature, detection accuracy and response tendency may have been confounded. In a common experimental approach, either a single scenario or a relatively small subset of scenarios is administered to participants, who are tasked with determining whether each scenario reflected prejudice (e.g., Basford et al., 2014; Hughey et al., 2017; Jenkins et al., 2023; Kim et al., 2019; Tao et al., 2017). Typically, most of the scenarios are indeed prejudiced, although one or more non-prejudiced scenarios may also be included as controls or distractors. Most often, detection accuracy is operationalized as the mean ratings given to the prejudiced scenarios on a continuous, Likert-type response scale. The problem is that in this type of design, accuracy cannot be separated from general response tendencies. Each participant's responses will reflect some mix of their general response tendency and their ability to distinguish prejudice from non-prejudice.

In one example, Owen et al. (2018) examined therapists' ability to correctly identify racial-ethnic microaggressions in video vignettes. In this case, each therapist was randomly assigned to view a single video scenario—either a scenario that contained three microaggressions or a scenario that contained no microaggressions. One key outcome of interest was the extent to which therapists could correctly identify racial microaggressions. However, because each therapist only viewed one scenario, detection ability was completely confounded with response tendency. That is, those who had the tendency to respond “yes” when unsure may have been assigned high accuracy scores, whereas those who saw the same evidence but had a tendency to say “no” when unsure may have been assigned low accuracy scores.

Even when multiple scenarios are used, researchers often miss the opportunity to separate accuracy and response tendency. In a recent study, Midgette and Mulvey (2024) presented each participant with five vignettes, four of which were microaggressions (80%). Participants then rated how biased they perceived each vignette to be. Our hypothetical participant Camilla would likely have rated all five scenarios as biased, while our hypothetical Jason would have rated all five as unbiased. Given this design, the researchers would incorrectly conclude that Camilla's detection ability was far superior to Jason's. Actually, neither Camilla nor Jason had demonstrated any ability to tell the difference between prejudiced scenarios vs. non-prejudiced scenarios. In the next section, I will provide more detail about how the methods developed from the SDT paradigm can help solve this problem.

In any SDT design, it is necessary to present each participant with a series of scenarios. The series should be comprised of both signal and noise scenarios—that is, prejudiced and unprejudiced scenarios, respectively. Further, to use SDT, each scenario needs to have a pre-determined “correct” answer. Some options for addressing these challenges will be presented later. For now, I continue with describing how the stimulus set is used once it is created.

For each scenario, participants indicate whether they believe the situation was prejudiced (yes) or (no)². Each decision has four possible outcomes, which are shown in Figure 1. The four possible outcomes include: a hit (correctly identifying prejudice when it is present), miss (failing to recognize prejudice when it is present), false alarm (mistaking a non-prejudiced situation for a prejudiced one), and correct rejection (correctly identifying a non-prejudiced situation).

For each participant, we calculate the rates of hits and false alarms across the series of stimuli. The hit rate is the number of hits divided by the total number of prejudiced scenarios. So, if there are 20 prejudiced scenarios and the participant correctly identified 15 of them, their hit rate would be 0.75. Likewise, the false alarm rate is the number of false alarms divided by the total number of non-prejudiced scenarios. If there were 20 non-prejudiced scenarios but the participant thought seven of them were prejudiced, their false alarm rate would be 0.35. We do not need to separately calculate the miss and correct rejection rates because they are the inverses of the hit rate and false alarm rate. Hit and false alarm rates can be easily calculated using software such as Excel, as shown in Figure 2. The hit and false alarm rates then become the main inputs into the calculations for accuracy and response tendency.

The accuracy and response tendency metrics can be easily obtained using software such as the dprime function from the {psycho} package in R (Makowski, 2018). The user simply inputs counts of each participant's hits, false alarms, misses, and correct rejections, and the dprime function calculates metrics for accuracy and response tendency. Despite the availability of such tools, it is nevertheless helpful to understand conceptually how the metrics are produced. Therefore, I will explain the theory underlying the computation of these metrics.

Theoretically, SDT assumes that in each scenario, the decision-maker implicitly or explicitly perceives a certain amount of evidence supporting the presence of prejudice. For example, a scenario in which an individual with a swastika tattooed on their face uses a racial slur would provide very strong evidence for prejudice. However, the amount of evidence perceived varies from scenario to scenario, because each scenario offers a different degree of evidence about whether prejudice occurred. Some scenarios will include stronger evidence than others. A scenario in which a sales clerk occasionally checks on a shopper while they browse the store may be less obvious—many people may perceive it as strongly indicating prejudice, but many others may perceive it as offering very little evidence suggesting prejudice.

Furthermore, the variation in the degree to which each person perceives the evidence is expected to occur both between-persons and within-persons. In other words, some people may perceive the salesperson scenario as providing stronger evidence of prejudice than others. Further, if the same person saw the same scenario repeatedly, we would still expect some variance in their reactions based on fluctuations in attention, mood, recent events, or other factors. The key point is that there will be variation in the degree to which evidence suggesting prejudice is perceived, and in traditional forms of SDT, we make the assumption that the variance is normally distributed. Making this assumption allows us to use the known properties of the standard normal distribution to calculate areas under the curve, etc.

It is very helpful to create figures showing participants' response patterns. This helps visualize the links between participants' hit/false alarm rates and their metrics for accuracy and response tendency. While it is possible to obtain the metrics without producing these figures, the visualizations help us better understand the metrics' conceptual meanings. As such, any primer on SDT will likely feature at least one of these diagrams, such as the one shown in Figure 3. Although these figures may appear complex at first glance, they can be understood quickly through a series of examples. As such, I will present several examples showing different people's response patterns.

Figure 3 shows the response pattern of one single participant. (Each participant's pair of distributions will look different because they reflect each participant's responses to the stimuli.) The x-axis represents perceived evidence suggesting prejudice, with the higher end of the axis representing stronger perceived evidence for prejudice. Along the x-axis, we see two normal distributions. I have color-coded them orange and blue, but the specific choices of colors are not important; they are simply used here as visual aids. The orange distribution represents the participant's response pattern in the set of scenarios that did contain prejudice (as pre-determined by the researcher). The blue distribution represents the same participant's response pattern across the set of scenarios that were non-prejudiced. As a reminder, there will be variation in the ways participants respond to different scenarios within the set, and we are assuming that the variance is normally-distributed. That is why the figure displays normal distributions.

In this particular example, the mean of the orange distribution is .67, while the mean of the blue distribution is −0.13. The higher mean for the orange distribution indicates that the participant (correctly) found more evidence supporting prejudice in the set of prejudiced scenarios than in the set of non-prejudiced scenarios. That is, they were overall more likely to say “yes, the situation was prejudiced” for prejudiced scenarios than for non-prejudiced scenarios.

However, we can also observe that the two response distributions overlap substantially. The area of overlap indicates that this participant was not always able to correctly distinguish between prejudiced and non-prejudiced scenarios. The size of the overlap relates to the frequency of incorrect decisions. The better a participant does at distinguishing the prejudiced from non-prejudiced scenarios, the more separated their two distributions will be.

Now that we understand that the placement and overlap of the distributions reflects participants' performance, I will explain how placement and overlap are determined. The placement and overlap of the two distributions are determined using the participant's hit rate and false alarm rate. The mean/center of the prejudiced distribution (orange) is determined by calculating the z score associated with the participant's hit rate. Likewise, the mean/center of the non-prejudiced distribution (blue) is determined by calculating the z score associated with the participant's false alarm rate. The conversion between probabilities and z scores is likely familiar to many readers from classes such as introduction to statistics. Conversions can be done using a z table, online calculators, or AI tools. A visualization showing the standard normal distribution along with probabilities and associated z scores is provided in Figure 4.

For example, hypothetical participant Gordon correctly identified 50% of the prejudiced trials. As such, his hit rate is 50%. The z score associated with 50% in the standard normal distribution is 0. So, Gordon's prejudice-present distribution would be centered at zero along the x-axis. If, instead, Gordon's hit rate was 80%, his corresponding z score would be 0.84, and his distribution for the prejudice-present scenarios would be centered at 0.84 on the x-axis.

Given our theoretical understanding of the decision process and given our metrics for accuracy and response tendencies, we can better compare detection performance across people. The hypothetical participant whose responses are displayed in Figure 6a has achieved essentially-perfect performance. The two response distributions are widely-spaced, reflecting a large d' score. Essentially all the responses for the no-prejudice trials fall below the criterion line (indicating that the participant correctly responded “no” to all of the non-prejudiced stimuli). Conversely, essentially all the responses for the prejudiced scenarios all fall above the criterion line (indicating that the participant correctly identified all of the scenarios containing prejudice). This is an idealized goal.

Unless the prejudice is extremely obvious, perfect performance is unlikely. Subtle prejudice—as its name suggests—is not necessarily obvious and can frequently be missed or misperceived. Thus, the two distributions will usually overlap. The hypothetical participant whose response pattern is displayed in Figure 6b has made some errors. They had some false alarms—they mislabeled some of the non-prejudiced scenarios as prejudiced (the area of the curve shaded in black). Likewise, they missed/failed to perceive some of the prejudiced scenarios (the area of the curve shaded in red). Likewise, the hypothetical participant whose responses are displayed in Figure 6c has also made some errors, but their pattern of responses is different. Compared to hypothetical Participant B, this person has a lower criterion level for labeling scenarios as prejudiced. For this hypothetical participant, few instances of prejudice are missed (the hit rate is high), but correspondingly, the false alarm rate is also relatively high.

These SDT metrics allow us to understand detection performance at a more nuanced and accurate level than we could by simply looking at overall responses. To illustrate these gains, let us return to our TSA screening agents. Let us imagine that the airport manager wishes to recognize the “best security screener” with a prize. Further, the manager wishes to base the award on performance data. Recall that Jason rarely, if ever, flags a bag for inspection. Using a conventional approach of simply counting how many times each screener was correct, Jason would receive the best screener award. This is because most suitcases do not contain dangerous items, so his strategy of clearing every bag will produce a large number of correct decisions. However, recognizing Jason as the best screener would be a mistake, because he has no true ability to detect dangerous items in the luggage, and his response tendency risks allowing dangerous items onto the planes.

Next, we turn to Camilla, who flags nearly every bag for inspection. If we decided to base the award on the percentage of dangerous suitcases that were correctly flagged, then Camilla's strategy will prevail, as she flags every bag and therefore catches all of the dangerous ones. However, those “hits” come at a cost of many false alarms, which create unreasonably long security lines and cause customers to miss their flights. Furthermore, over time, her coworkers will perceive the high number of false alarms and start to disregard her warnings (a phenomenon known as the “cry wolf effect”; e.g., Bliss et al., 1995; Dixon and Wickens, 2006). Only by using SDT can the best screener be identified.

Table 2 shows the SDT response distributions for Jason, Camilla, and two additional candidates—Romeo and Tonya. Both Romeo and Tonya have response tendencies that lean somewhat toward avoiding misses. This is desirable in safety-related contexts, where the consequences of one miss far outweigh the consequences of one false alarm. Yet, their response tendencies are not as extreme as Camilla's, so false alarms are less likely to accumulate to unacceptable levels. Also, both Romeo and Tonya have larger d′ scores than Jason and Camilla, indicating better screening performance. Because Romeo has the higher of the two d' scores (d′ = 3.50 vs. Tonya's d′=1.86), it seems reasonable to present the screening performance award to Romeo.

It is interesting to contemplate what the ideal response tendency for prejudice detection might be. Is it akin to safety contexts such as luggage screening, where the priority should be avoiding misses? Or is it more desirable to develop a neutral response tendency? Further, are there group-based or philosophical differences that predict these preferences? This could be an interesting and useful avenue for future research. I believe that jointly incorporating both accuracy and response tendency metrics opens a variety of new avenues. In addition, I argue that it can help scholars address questions already under discussion in the research literature.

One possible approach to creating prejudiced and non-prejudiced stimuli could be called the reasonable person approach. Under this definition, an act would be defined as prejudice if a reasonable, unbiased person would view the behavior as either severe or pervasive and due to the target's social group membership (race, age, gender, disability status, etc.). This approach is consistent with the United States' standard for legally identifying harassment or negligence (constructs that, like prejudice, can be somewhat subtle and ambiguous at times; https://www.eeoc.gov/laws/guidance/enforcement-guidance-harassment-workplace#_Toc164807998). Note that in this approach, the “person” in question is a rather “generic” individual, who is not specifically affiliated with the target's demographic group. The reasonable person standard essentially says that an instance will be legally considered harassment if a person who is essentially representative of the general population would usually consider it to be harassment.

If the reasonable person approach is the researcher's chosen definition of prejudice, then the researcher would then norm (i.e., pilot test) a set of candidate scenarios with a sample that is representative of the general population. That is, the “Dave” scenario, along with the other candidate scenarios, would be rated by a sample of raters who could be considered “reasonable people” (e.g., they are not considered outliers on any potentially confounding traits, such as prejudice levels or authoritarianism). The raters would indicate, perhaps on a Likert-type scale, how prejudiced they believed the woman's actions toward Dave to be. The response distribution, including its mean and variance, would be examined to determine whether it met the “reasonable person” standard for being classified as prejudiced. For example, if the scenarios were rated on a 5-point Likert-type scale ranging from 1 (definitely not prejudiced) to 5 (definitely prejudiced), then the researcher might decide to retain items with means of either 1–2 (for unprejudiced stimuli) or 4–5 (for prejudiced stimuli). Ideally, these items would also have relatively small variances—reflecting agreement among the raters.

Many researchers might prefer an adaptation of this approach that we could call the reasonable target approach. This approach, like the reasonable person approach presented above, defines prejudiced scenarios as those in which a reasonable person would perceive prejudice. However, this approach prioritizes the perspectives of the members of the social group receiving prejudice. That is, if the researcher is studying prejudice against Muslim individuals, then the standard could be labeled the “reasonable Muslim” standard. Similarly, if the researcher is studying prejudice against women, the standard could be labeled the “reasonable woman” standard, etc. In this approach, the researcher would still collect item ratings from a sample of raters. However, in this case, the group of raters would be composed only of Muslims, or women, etc. Compared to the reasonable person approach, the reasonable target approach centers the perspectives of the members of those groups who would be the targets in the scenarios. This approach is consistent with the arguments of Sue (2017) that the experiential accounts of targets should be prioritized over non-targets.

A third possible approach could define prejudiced behaviors as actions that are more likely to be performed by people with higher levels of prejudice than by people with lower levels of prejudice. This could be called the perpetrator mindset approach. Theoretically, prejudiced behaviors are manifestations of latent stereotypes and biases, so individuals with greater biases should be more likely to engage in prejudiced behaviors (Dovidio et al., 2018; Sue et al., 2007). Kanter et al. (2017) used this logic to develop a procedure for identifying prejudiced behaviors, and their method could be adapted for creating SDT stimuli. In their study, which focused on racial prejudice against African-Americans, a set of candidate scenarios was presented to a sample composed of White students. The White students were asked to rate the extent to which they would think, say, or do each behavior (e.g., “Did you get into school through a minority scholarship?”). They also completed several self-report measures of racial prejudice. Then, the extent to which participants' racial prejudice scores predicted their likelihood ratings was examined. Those scenarios with stronger positive correlations were classified as prejudiced.

To use this process for selecting SDT scenarios, the researcher would similarly collect prejudice scores for each participant and likelihood ratings for each candidate scenario, and examine the relationships between them. For the “Dave” scenario, a sample of White participants would be asked to rate how likely they would be to think/behave the same way that the woman in the scenario behaved toward Dave. Those same White participants would also complete one or more measures assessing their level of racial prejudice toward Black individuals. The researcher would then examine the magnitude of the correlation between the scores for the likelihood of thinking/acting the same way as the woman in the scenario and the scores for respondents' racial prejudice. The scenarios that showed stronger positive correlations with prejudice levels would then be classified as prejudiced stimuli. That is, those scenarios in which more prejudiced individuals were more likely to think, do, or perform the behavior would be retained in the prejudiced scenario set. Those items that are not significantly correlated with the measures of racial prejudice would be classified as non-prejudiced stimuli.

As a cautionary note, researchers must also carefully consider the noise stimuli (i.e., the non-prejudiced scenarios). It is crucial to avoid confounds. For example, if all of the prejudiced scenarios are negative in tone but all the non-prejudiced scenarios are positive in tone, a confound would be created. This would compromise our ability to assess accuracy because we would be unable to tell whether respondents were truly discriminating prejudice from non-prejudice or whether they were discriminating negative from positive interactions. A similar confound would occur if the social identity of the perpetrator varied systematically along with the definition of prejudiced/non-prejudiced behavior (e.g., all sexist behaviors are performed by a man, but all non-sexist behaviors are performed by a woman). As such, it is important to ensure that only the presence of prejudice varies systematically between the prejudiced and non-prejudiced stimuli.

For SDT analyses to be effective, the stimulus sets should be large enough to obtain variance in hits and false alarms. A rule of thumb might be 10–20 prejudiced stimuli and 10–20 non-prejudiced stimuli. This recommended set size is larger than what has been seen in most past studies on prejudice [for example, Basford et al. (2014) study included eight total scenarios]. Thus, building a sufficiently large pool of valid prejudiced and non-prejudiced stimuli is one of the potential practical challenges to the use of SDT methods.

To address this challenge, I would suggest that researchers start by pooling existing resources. Smaller stimulus sets have already been built by several different researchers, including published scholars and doctoral students [e.g., Ako-Brew, 2020 (4 vignettes); Conover et al., 2021 (20 vignettes), Parker, 2017 (6 vignettes); Hughey et al., 2017 (3 vignettes); Jenkins et al., 2023 (16 vignettes), Kim et al., 2019 (8 vignettes), Mellor et al., 2001 (6 vignettes), and Tao et al., 2017 (4 vignettes)]. These are often short videos or text-based vignettes a few sentences long. For example, this vignette from Kim et al. (2019):

Given that several stimulus sets have already been created, researchers could pool their existing stimuli into a communal bank. These pooled stimulus sets could be published in journals that focus on publishing data and materials sets—incentivizing researchers to collaborate in return for authorship on the resulting publication and citations when the stimulus set is used in future research.

If additional stimuli must be created, they should be generated using standard psychometric procedures; the reader may be familiar with these from having studied the development of self-report items. Item development and norming procedures are discussed in detail elsewhere (e.g., Hinkin, 1995; Tharenou et al., 2007), so I will limit this discussion to providing a general overview of the process. In this case, the “items” being developed and normed are the scenarios/vignettes to be presented.

Importantly, the first step is for the researcher to clearly specify the way they conceptually distinguish between prejudiced and non-prejudiced scenarios (e.g., an agreement-based approach, a perpetrator mindset approach, or another approach). Several options are available; the key point being that the researcher should choose and justify their definition (e.g., whether it is based on agreement, the perspective of an expert, an examination of stereotype endorsement, or something else). The subsequent steps must follow from, and be consistent with, that theoretical clarity.

Secondly, a large pool of possible vignettes reflecting both prejudiced and non-prejudiced scenarios should be created. This initial pool should be larger than the ultimately desired number of scenarios so that poorly-performing scenarios can be eliminated. To create this initial item pool, I recommend soliciting suggestions for vignettes from members of the target group(s) to ensure that their perspectives are represented (e.g., Kanter et al., 2017; Kim et al., 2019). This may be accomplished via focus groups, interviews, or survey procedures. One may ask respondents to provide examples of situations they thought at least might have been prejudiced. Then, to create non-prejudiced items, one might—for example—ask the sample to describe “non-prejudiced interactions that happen day-to-day” (the exact phrasing will differ depending on the researcher's unique context). Additional scenarios could also be drawn from the research literature, from expert input, or other relevant sources.

Thirdly, a separate sample should provide norming data (i.e., pilot data) by responding to the items. The questions asked of the norming sample should be consistent with the researcher's conceptualization of prejudice. For an agreement-based approach, the sample would respond to the scenarios in the same way that a focal research sample would. For example, they might indicate whether they judge each scenario to be prejudiced on a yes/no scale or on a continuous scale (depending on which form of SDT analyses the researcher intends to use). The researcher should examine the means, variances, and distributions of the scenario judgments to select those that best distinguish among individuals (for example, items on which everyone agrees may not be useful).

If the researcher has adopted a perpetrator-mindset definition of prejudice, then the researcher must use a norming sample composed of those individuals most likely to perpetrate the prejudice (e.g., if studying prejudice against women, then the norming sample should be composed of men; if studying disability-related prejudice, then the norming sample should be composed of individuals without disabilities). Following Kanter et al. (2017), these individuals should be asked to rate the extent to which they would be likely to think, say, or do the perpetrator's behavior described in each scenario. At least one measure of the individual's level of prejudice should also be completed by the norming sample. The empirical data should be used to identify the scenarios for which the respondents' likelihood ratings are significantly (positively) correlated with their prejudice scores. Those items should be retained as prejudice-present stimuli. Those with no significant association may be classified as non-prejudiced stimuli.