To provide some context, we will first examine a well-known task designed to elicit the use of a heuristic that leads to a biased judgment: the bat and ball problem from Frederick’s (2005) Cognitive Reflection Test:

Most people respond that the answer is 10 cents (Frederick, 2005). Some mental shortcut (heuristic) is thought to construct this response quickly and seemingly effortlessly, without the need for mental calculations. A simple verification will show that the answer is five cents, however. Thus, items in the Cognitive Reflection Test elicit heuristically constructed but incorrect responses that must be checked and corrected. Indeed, many of the tasks used to establish that, and further investigate how we use heuristics when making decisions are similar to this. That is, most H&B tasks present a question that allows for a normatively correct response, but which people often avoid.

Not long after their emergence, it became apparent that individuals varied in their susceptibility to committing errors and biases elicited by H&B tasks (e.g., Stanovich and West, 1998a,b,c). For example, while many people respond “10 cents” to the problem above, some respond correctly. Individuals’ performance on a range of H&B tasks has now been found to correlate in a weak to moderate fashion (e.g., Stanovich and West, 2000; Parker and Fischhoff, 2005; Bruine de Bruin et al., 2007; West et al., 2008; Stanovich et al., 2012; Teovanović et al., 2015). An ongoing challenge being addressed by this literature has been to explain the nature and source of this covariance.

There have been two prominent approaches for explaining these individual differences in H&B tasks. Researchers originally hypothesized that a single rationality-like construct might be the source. The first approach was to therefore model covariance among H&B tasks via factor analysis to extract common sources underlying performance. Factor analyses of H&B tasks tend to yield solutions with two or more factors that are difficult to interpret within and between studies, however (Parker and Fischhoff, 2005; Bruine de Bruin et al., 2007; Weaver and Stewart, 2012; Teovanović et al., 2015). Furthermore, these sorts of psychometric studies demonstrate that H&B tasks tend to have relatively low reliability estimates. It is clear from this work that H&B tasks are not collectively and consistently tapping a single, general, and desirable construct.

The second approach has involved seeking correlates/predictors of individual’s H&B task performance on the basis of dual-process theories. According to these theories, two broad categories of cognitive processes construct judgments and actions (Evans and Stanovich, 2013 for a review). Type 1 processes are automatic and tend to rely on knowledge structures acquired via learning. They include processes like associative and constructive intuition (Glöckner and Witteman, 2010; Evans and Stanovich, 2013). Type 1 processes are therefore the typical source of heuristic responses that lead to errors on H&B tasks. Type 2 processes are deliberative and effortful mental operations. A classic example is fluid reasoning ability (Gf; Carroll, 1993; Stankov, 2000; McGrew, 2005, 2009). Gf is defined as “deliberate and controlled mental operations to solve novel problems that cannot be performed automatically” (McGrew, 2009, p. 5). Their reliance on working memory and controlled attention impose limits of their processing capacity (Evans and Stanovich, 2013; Shipstead et al., 2014). Such Type 2 abilities are the source of accurate responses on many H&B tasks. Investigated predictors of H&B tasks, therefore, tend to relate to Type 2 abilities or constructs that help shift decision makers from erroneous Type 1 to more accurate Type 2 thinking.

Stanovich and West (2008) proposed that Type 1 heuristic errors on tasks like the Cognitive Reflection Test must be detected so that Type 2 processes like Gf can become engaged. By this account, individuals with stronger Type 2 cognitive abilities and who are better able to detect errors of judgment will perform better on H&B tasks. Researchers have therefore tested models such as that shown in Figure 1A.

Adopting this approach, researchers measure individuals’ performance on a H&B task and predict the scores with cognitive abilities (e.g., Gf) and variables related to error detection (e.g., West et al., 2008; Toplak et al., 2011; Del Missier et al., 2012; Thompson and Johnson, 2014). In these studies, individuals’ ability to detect errors tends to be measured via executive control tasks in which they must suppress proponent and inaccurate responses in response time tasks. Alternatively, individuals are asked to self-report their tendency to engage in or enjoy Type 2 cognitive processing via self-report measures. Such studies typically find that Gf and error detection-like constructs are significant positive predictors of performance on H&B tasks in the manner proposed above.

Given these findings, we will assess here two ways to obtain greater information about individual differences in H&B tasks like the Cognitive Reflection Test. The first will be to model changes in covariance among H&B tasks. Rather than factor analyzing all tasks together or regressing each task independently on predictors, we will adopt a new approach. Specifically, we will regress individuals’ H&B task scores on a set of predictors in a single Structural Equation Model, and allow their residuals to correlate freely. If the predictors are generally underlying performance on the H&B tasks, then we should observe more than just the significant regression coefficients that are found when each H&B task is used in separate regression models. More specifically, the residual covariance among the H&B tasks in a single SEM (i.e., the correlated error terms after being regressed on the predictors) should be considerably lower than their zero-order correlations, and possibly null. If the predictors are accounting for task-specific variance only, then we should observe significant regression coefficients, but little change from the zero-order to the residual correlations among the H&B tasks.

Our second addition will be to disentangle the notion of error detection into distinct aspects of monitoring and control. That is, model monitoring and control variables separately, and use these as predictors of H&B performance instead of a single error detection construct. Overall, our goal here is to build on previous work by extending theory and measurement surrounding error detection as a predictor of individuals’ H&B task performance. Figure 1B shows our proposed model. The following section will describe the monitoring and control aspects, and their inclusion as independent predictors of individuals’ H&B performance is a novel approach in the literature.

Monitoring processes accumulate evidence in favor of judgments and actions during their construction (via Type 1 or Type 2 processes; Flavell, 1979; Koriat, 2012; Ratcliff and Starns, 2013). Confidence refers to this feeling or experience of evidence. For example, a clinician with excellent monitoring would be very confident for correct diagnoses but not at all confident for incorrect diagnoses. Confidence is known to be a stable individual differences trait and a predictor of action initiation. For example, Jackson et al. (2016a,b) had participants answer test items, indicate answer confidence, and decide whether to submit the answer for marking. Figure 2 shows an example item.

Individuals’ trait Confidence is computed as the average of their confidence ratings for all test items. Jackson et al. (2016a,b) found that individuals with higher trait Confidence consistently submitted more answers for marking (correct and incorrect) than individuals with lower Confidence.

Control processes determine the threshold point of confidence at which an action that is being considered will be initiated (Koriat and Goldsmith, 1996; Ratcliff and McKoon, 2008; Ackerman, 2014; White et al., 2015)¹. For example, a clinician might treat a patient only once s/he is more than 90% confident in the diagnosis (e.g., Pauker and Kassirer, 1980; Djulbegovic et al., 2014). Jackson et al. (2016a,b) also assessed this by computing the threshold of confidence at which individuals would submit their answers (referred to in their work as the Point Of Sufficient cerTainty, or POST). Controlling for their confidence, individuals with a higher control threshold submitted fewer answers (correct and incorrect) for marking. Setting a high threshold, therefore, represents a risk-averse strategy, as the decision maker has more time to process potential actions while confidence is being accumulated. A higher threshold increases the likelihood of detecting errors, but the cost is that it may also increase the number of missed opportunities to submit correct answers, though it can never decrease that number (e.g., Koriat and Goldsmith, 1996). Setting a lower threshold will decrease (or maintain) the proportion of missed opportunities, but at the cost of also decreasing the likelihood of detecting errors. Including monitoring confidence and control thresholds in the model provides a more detailed account of how we manage the initiation of actions than the error detection theory proposed earlier. Shown in Figure 3, we can conceive the process of answering an H&B question as Type 1 and Type 2 processes occurring over time. Type 1 heuristics quickly generate an incorrect response, and, given the fluency with which this answer is generated, monitoring confidence will quickly rise in support of it. If this level of confidence reaches the control threshold, this initial response will be given. If, however, this does not happen, Type 2 processes will start to check the original response and, possibly, generate the accurate response. This will likely cause confidence in the initial response to decline, and confidence in the Type 2 response to increase until it reaches the control threshold. Note that this does not describe the general state of affairs. Type 1 responses are not always incorrect or faster than Type 2 responses. This just tends to apply for typical H&B tasks. What should generally apply, however, is that a response is chosen when confidence reaches a threshold.

From this perspective, it is likely that an individual who tends to be confident will often be confident enough in their initial and incorrect response to give it without further consideration. The same logic applies to an individual who tends to set a low control threshold. However, an individual who tends to be less confident, and/or hold a higher control criterion, will be more likely to engage in Type 2 processing. Provided that they also possess and utilize the necessary Type 2 cognitive abilities, individuals who tend to experience low monitoring confidence and/or hold high control thresholds will more probably reach the correct answer. Thus, from an individual differences perspective, individuals with relatively low monitoring confidence, a high control threshold, and strong cognitive abilities (like Gf) should do best on H&B tasks.

Although similar, the monitoring and control view differs from the error detection perspective in an important way. By the latter, the ability to detect errors should differ between individuals. Such a skill should be evidenced by monitoring confidence discriminating between good and bad judgments/actions. However, the literature on these sorts of monitoring skills has revealed that they tend to be unreliable as individual differences variables (Stankov and Crawford, 1996; Jackson and Kleitman, 2014; Stankov et al., 2014). That is while people tend to discriminate between accurate or inaccurate judgments, in general, individuals tend not to be better or worse than each other at doing this. Such results speak against the theory that individuals are better or worse than each other at “detecting” errors, independent of intelligence, skills, etc.

Individuals do differ consistently in their overall level of monitoring confidence and control threshold, however. Individuals vary in their overall average level of confidence (Pallier et al., 2002; Kleitman and Stankov, 2007; Stankov et al., 2014). Similarly, individuals tend to vary in the control threshold they set within a given context (Jackson et al., 2016a,b). These results suggest that people adopt different strategies for checking errors of judgments. As described above, low-confidence-high-threshold individuals would be more likely to check for errors than high-confidence-low-threshold individuals, regardless of errors being made or not. The low-confidence-high-threshold combination should, therefore, lead to the highest performance level on H&B tasks, which do elicit automatic errors.

We will examine one implication of this theoretical difference: that monitoring confidence and, in particular, control thresholds link a broader range of decision constructs than the ability to detect error. For example, a wide class of Bias tasks are gambling measures that assess risk and ambiguity preference. These differ from tasks like the Cognitive Reflection Test as the final score indicates a difference in preference rather than performance. The notion of error detection cannot account for how or why these sorts of tasks should relate to the broader range of H&B tasks. Such gambling biases should relate, however, to control thresholds. As mentioned above, individuals who set a higher threshold are adopting a more risk-averse strategy than individuals setting lower thresholds. This is because adopting a high control threshold means that you will not make a decision until you are sufficiently certain that the decision is a good one. For example, holding a threshold of say 90–100% means that you must be very certain of a decision before initiating it. Therefore, adopting a high control threshold is conceptually similar to having a low preference for risky, uncertain, or ambiguous outcomes. Adopting a low threshold such as 30–40%, however, means that you are willing to make a decision despite great uncertainty. This is akin to having a high preference for risk, uncertainty, or ambiguity. As such, control thresholds provide a theoretical basis for expecting a negative correlation between individuals’ preference for risk on gambling tasks and performance on H&B tasks. Thus, in addition to employing typical H&B tasks, we will expand our dependent variables to include measures assessing biases for uncertain and ambiguous outcomes (Risky Gambles and Ellsberg Paradox). Individuals holding higher control thresholds are expected to select more certain options (meaning fewer uncertain and ambiguous options) on the gambling tasks than individuals with lower thresholds.

To summarize, our goal is to investigate the combined predictive relations of individuals’ fluid reasoning abilities (Gf), monitoring confidence, and control thresholds with a range of H&B tasks. We will be extending previous work in two ways. First, by modeling the residual covariance among the H&B tasks to determine the extent to which cognitive abilities, monitoring confidence, and control thresholds account for common variance. The degree to which the zero-order correlations among the H&B tasks are reduced will indicate the degree to which their covariance can be explained by cognitive abilities, monitoring confidence, and control thresholds. Second, by independently accounting for monitoring and control variables, as an extension of previous theories proposing that the ability to detect errors differs between individuals. We hypothesize that:

These hypotheses will be tested via structural equation modeling to account for shared covariance among the variables. Before these final tests, however, inference will be made, where possible, on the basis of the zero-order correlation matrix.

In return for partial course credit, 250 first year psychology students at the University of Sydney participated in the study (N = 250, 158 female, 92 male, M_age = 20.46 years, age range: 17.41–60.65 years). Participants completed all tests below within a larger battery, which included measures of executive function².

Participants signed up to participate over the Internet and received a URL to access these tests (within a larger battery) and complete them online. A notification appeared before each test reminding participants to take a short break. However, they were required to progress through the entire protocol in a single session. That is, they could have a short break after completing each test, but not exit the Internet browser and return at a later time or different place. Most participants completed the entire battery within a single 2-h period. Participants received partial course credit upon completion of the protocol but were not otherwise incentivized. Although this is a potential concern, previous research in our lab has revealed negligible differences between online participation (as done here) and in-person lab participation (Jackson, 2016).

Table 1 shows the descriptive statistics for all variables. Internal consistency estimates were computed as Cronbach’s alpha (Cronbach, 1951) for all variables except the control threshold. For the control threshold, internal consistency was computed on the basis of an Odd/Even item split correlation corrected using the Spearman-Brown Formula (Stankov and Crawford, 1996; Jackson et al., 2016a,b).

Descriptive statistics for all measures were typical for an Australian undergraduate sample. Mean Accuracy in the Gf tasks indicated a suitable level of difficulty (range = 57.30–76.66%). Confidence levels for the Gf tasks (range = 65.84–81.14%) and control thresholds from the other decision tasks (52.85 and 44.20%) were also within a typical range comparing to previous work on similar cohorts (Jackson et al., 2016a,b). For these variables, internal consistency estimates were poor for two variables: Accuracy in the Number Series Test (0.58), and the control thresholds for the novel Financial Decision-Making Test (0.56). The likely reason for this is that both of these tests were relatively short and newly developed. However, internal consistency estimates for the remaining variables above were good (range = 0.68–0.94).

Mean scores for the performance-based H&B tasks were in the typical range (e.g., Bruine de Bruin et al., 2007). However, internal consistency estimates for two of the H&B tasks were unacceptably low: Resistance to Sunk Costs task (0.39) and the Ellsberg Paradox (0.43). These low estimates indicated that we could not interpret the scores for these tasks in a meaningful fashion (Cronbach, 1951). We, therefore, removed these tasks from further analyses. In line with previous research (Bruine de Bruin et al., 2007; Del Missier et al., 2012; Weller et al., 2012), internal consistency estimates for the remaining H&B tasks were low but acceptable for research purposes (range = 0.50–0.70).

Table 2 shows the correlations between all retained variables. The Gf Accuracy variables all correlated in a moderate fashion with each other. Confidence scores from the cognitive ability tests also correlated with each other and Accuracy scores. In the line with recent work using the same tests (Jackson et al., 2016a,b), Confidence and Accuracy scores from the same test had somewhat higher correlations than those observed with achievement tests in the past (Stankov et al., 2012). Following this recent work, test-specific correlations were managed in the forthcoming structural equation modeling by allowing residuals of the Accuracy and Confidence variables from the same test to correlate. The two control threshold scores correlated positively with each other and weakly with the Accuracy and Confidence variables.

In line with previous work (Parker and Fischhoff, 2005; Bruine de Bruin et al., 2007; Del Missier et al., 2012), the H&B tasks correlated with each other in a weak to moderate fashion. Applying Decision Rules and the Cognitive Reflection Test related most strongly to the other variables, with these tasks correlating significantly with almost all other tasks. Consistent with Hypothesis 1a, Gf Accuracy scores correlated, by-and-large, positively with the H&B tasks. Consistent with Hypothesis 1b, confidence scores also correlated positively and significantly with some H&B tasks. Though, these were not as strong as the correlations of the H&B tasks with the Accuracy scores, and some of these correlations were negative (but weak). Although we had not hypothesized these positive correlations, the strong positive correlations of Confidence with Accuracy scores was likely the reason for this. Hypothesis 1c, however, was not supported: The control threshold variables did not correlate significantly with the performance-based H&B tasks. It did, however, correlate significantly and negatively with the number of risky alternatives selected on the gambling task, consistent with Hypothesis 2. This matrix was submitted to structural equation modeling to more clearly investigate the hypotheses.

A structural equation model was constructed in two steps to investigate the hypotheses.

Nonetheless, the results align with a theoretical stance that extends existing explanations of individuals’ H&B task performance in an important way. Existing explanations propose the involvement of error detection ability (e.g., Stanovich and West, 2008). Here, we proposed that error detection involves monitoring and control processes. Including monitoring confidence and control thresholds into the model suggests that, accounting for intellectual capacities and skills, individuals adopt a particular decision-making strategy that indirectly results in them being more or less likely to detect errors. The importance of this becomes evident considering tasks outside of the H&B literature. By existing theories, individuals should make better or worse decisions in general (e.g., Fischhoff, 2010), as they are capable of detecting whether they are going to make an error. Thus, an individual who performs well on H&B tasks will perform well in other contexts. Extending this to include monitoring confidence and control thresholds suggests that individuals will make better or worse decisions depending on the context. For example, H&B tasks deliberately elicit the use of heuristics that lead to an incorrect response. Individuals do best on these tasks if they hold low confidence and a high threshold because it gives them time to engage Type 2 processes. However, it is possible to develop highly effective heuristics in other contexts like firefighting (Kahneman and Klein, 2009). In such contexts, the performance of individuals with low confidence and a high threshold will suffer because they will be wasting unnecessary time. The present results, therefore, contribute to the mounting evidence that individuals do not detect errors due to some ability. Rather, individuals who detect more errors have likely adopted a slower, risk-averse strategy, making them best suited for contexts like those faced in H&B tasks.

The one major unknown for this perspective surrounds individual differences in control processes. Compared to cognitive abilities and monitoring confidence, little individual differences research has been conducted on control thresholds. It is possible, for example, that the capacity to set one’s threshold at an optimal point depending on the environment differs between individuals. There is some evidence that this is possible. For example, individuals’ are known to adjust their thresholds differently across contexts (Aminoff et al., 2012; Jackson et al., 2016b). Furthermore, the ability to discriminate accurate from inaccurate judgments via binary decisions differs reliably between individuals (Jackson and Kleitman, 2014; Jackson et al., 2016a,b). Given that monitoring discrimination does not differ between individuals (Stankov and Crawford, 1996; Jackson and Kleitman, 2014), it is possible that differences in thresholds are the cause of this. Overall, control processes are clearly more flexible than monitoring processes. Indeed, it is likely that the relatively weak contribution of control thresholds observed in this study was a result of these domain-specific attributes. It is possible, therefore, that the ability to optimize one’s threshold for a decision task differs among individuals. This notion differs from the traditional concept of error “detection” (which relates more to monitoring) but provides an alternative means by which certain individuals could be better decision makers than others.

Replicating previous results, personality variables shared no meaningful relationships with any of the variables under investigations, and were, therefore, omitted from any discussions in this manuscript. We recommend that if timing is an issue and on-task measures of decision-making and confidence are employed, future studies omit personality measures from their research protocol.

Additional to these theoretical considerations, considering decision-making in terms of cognitive abilities, monitoring confidence, and control thresholds carries important applied implications. Consider, for example, that individuals of comparable cognitive abilities might be better or worse decision makers depending on the context. Organizations should, therefore, select individuals for very specific decision-making tasks based on their monitoring confidence and control thresholds. Further implications for decision training exist. Certain training programs have adopted H&B tasks as a benchmark for competent decision-making (e.g., Jacobson et al., 2012). Given the results here, it is possible that such training might be encouraging individuals to adopt a particular decision strategy that will only suit a narrow range of contexts. Using H&B tasks for selection or a benchmark of training efficacy should, therefore, be conducted with caution. Good performance on H&B tasks might indicate poorer decision-making in other contexts.

The topic of training is particularly interesting for these reasons. On one hand, the notion of having to train people for particular decision environments is alarming, as it complicates and limits the efficacy of training programs. On the other hand, the possibility that individuals are capable of optimizing their control thresholds opens new avenues. Cognitive processes are notoriously resistant to intervention (Harrison et al., 2013). Monitoring has proven to be slightly more amenable to change (de Bruin and van Gog, 2012; Kleitman and Costa, 2014). Control thresholds are already known to be more flexible and set by our perceptions of payoffs (e.g., Koriat and Goldsmith, 1996; Aminoff et al., 2012; Ackerman, 2014; Jackson et al., 2016b). It is thus likely to be a prime target for intervention. For this reason, we see the potential for training decision control to adapt optimally to different environments as a novel and exciting prospect.

Our results and these implications are not without their limitations, however. As mentioned, we did not explicitly test the error detection ability versus strategy hypothesis. Our findings do contribute to accumulating evidence in favor of the strategy perspective, but these contrasting hypotheses will need to be examined concurrently to make firm conclusions. Should such work be undertaken, we’d advise that researchers take care to dissociate the constructs of importance. For example, thinking-style measures tapping individuals’ tendency to think rationally is often seen as valuable, but this tendency could simply be a product of high control thresholds. It will be useful in future research to concurrently capture a range of constructs related to error detection and compare how they relate to tasks that are representative and non-representative of the environment.

Another general limitation of our results is the use of a student population. On one hand, this has been a consistent element of our previous research (Jackson et al., 2016a,b). We have thus been able to formulate robust expectations about undergraduate students and feel confident that we are tapping into reliable constructs. Still, students differ from the general population on some important attributes that will require investigation. For example, Type 1 knowledge structures and heuristics tend to be acquired over time via extensive learning (Stanovich et al., 2008; Weller et al., 2012). Older populations might, therefore, generate Type 1 responses more frequently and with greater confidence than younger populations. Similarly, university students might generate Type 1 answers more frequently and confidently than young children. It is also important to bear in mind that university students differ from the general population on attributes like cognitive abilities, and our sample were not incentivized to complete the tasks appropriately. Thus, although replicable within a university population, researchers will need to examine the results found here in other age groups or for individuals with different cognitive abilities.

Focal to the results at hand, the domain specificity of the control threshold might be a cause for its relatively weak contribution compared to cognitive abilities and monitoring confidence. Strong domain specificity in control thresholds has been shown to exist (Jackson et al., 2016b). The tests used to derive control thresholds imposed some loss for decision errors. For example, incorrect treatment of a patient in the MDMT results in patient death. There are no penalties, however, for incorrect answers on the H&B tasks. It is thus possible that individuals’ control thresholds in the H&B tasks differed from their thresholds in the other tasks, making the total contribution of control thresholds relatively weak. We suspect that capturing control thresholds in tasks with similar outcomes to the H&B tasks will yield stronger results.

Finally, although we endeavored to collect a broad range of H&B tasks, we were of course limited from a logistical point of view. Our selection of tests did not represent a great many H&B tasks. It will likely be important to test our hypotheses using a different range of H&B tasks. To add to this, we would also recommend that researchers consider the role of cognitive abilities, monitoring confidence, and control thresholds in other decision tasks. Our position is that these three constructs are involved quite generally in the decision-making process. Over 170 tasks are listed in the Decision-Making Individual Differences Inventory (Appelt et al., 2011). It will be important to discern which of these are valid, general and useful for studying decision-making and applying the research findings. We suspect that measures of cognitive abilities, monitoring confidence and control thresholds will yield strong results in such investigations.