A first quantitative model of risk was developed by Daniel Bernoulli in the eighteenth century in response to the famous St. Petersburg paradox (Bernoulli, 1954/1738). In this paradox a fair coin is tossed repeatedly and every time a head comes up the value of the jackpot is doubled, but the game ends as soon as a tail appears. Therefore, if the jackpot is initialized with $1 and tail appears in the first toss you win $1, if it appears in the second toss you win $2, in the third toss it is $4, and so forth. The question was how much a gambler should be prepared to pay to enter such a game of chance. Since the expected payoff E[$]=(1/2)$1+(1/4)$2+(1/8)$4+…=∑k=0∞(1/2k+1)$2kis infinity, the apparent answer seemed to be that one should be prepared to pay any arbitrarily high amount of money to enter this game – a rather questionable result. Bernoulli therefore introduced the distinction between the objectively given nominal value of a gamble (e.g., $1000) and the subjective utility assigned to it by a decision-maker [e.g., U($1000)]. In particular, he noted that $1000 has a higher utility for a pauper compared to a rich man that has already $1000, even though both would gain the same amount. The hypothesis that the second $1000 has less utility, is known as the diminishing marginal utility of money. Bernoulli postulated that the perceived utility of money follows a logarithmic law U($x) ∝ log($x) such that increments in payoff have a diminishing utility. This hypothesis can also explain why a risk-averse person would prefer a sure-bet with a risk-premium in the example described above, because for a risk-averse decision-maker the perceived utility U($45) is higher than the utility of 1/2U($100).

Since subjective utilities are not directly observable, this raises the question of how such utilities could be measured. Von Neumann and Morgenstern (1944) devised a mathematical framework to address this question, based on the notion of preference between “lotteries.” A lottery can be imagined as a roulette wheel where differently sized segments correspond to the probabilities p₁, p₂,…, p_N of the N different outcomes X₁, X₂,…, X_N. If we now create different lotteries by varying the size of the segments then we can ask a decision-maker to indicate preferences between the lotteries. Von Neumann and Morgenstern showed mathematically that if these preferences follow the four basic axioms of completeness, transitivity, continuity, and independence, then the decision-maker's choice between lotteries can be predicted by the maximum expected utility principle with a utility function U(X₁), U(X₂), …, U(X_N) over the outcomes. A typical shorthand for such lotteries is to write them as [p₁, U₁; p₂, U₂; …; p_N, U_N]. As utilities are only cardinal, they can only be determined up to an affine transform – that is the utility U(X) and the utility U˜(X)=aU(X)+b represent the same preference patterns. Importantly, the probabilities p₁, p₂,…, p_N are assumed to be known objectively in this framework – an assumption that was later dropped by Savage (1954) who introduced subjective probabilities into decision theory, that is probabilities that can be inferred from observed preference patterns just like subjective utilities.

The hypothesis of subjective utilities that are marginally diminishing seems to suggest that risk-sensitivity could be defined in terms of the curvature of the utility function. This definition is, however, problematic, since utility functions are only determined up to an affine transform, which makes the second derivative dependent on the arbitrarily chosen scaling parameter of the affine transform. Arrow and Pratt (Pratt, 1964; Arrow, 1965) therefore developed invariant measures of risk-sensitivity that are defined as the absolute risk-aversion A(x) = U′(x)/U′(x) and the relative risk-aversion RA(x) = −xA(x). For Bernoulli's log-utility, for example, the absolute risk-aversion would be decreasing according to RA(x) = 1/x, and the relative risk-aversion would be constant RA(x) = 1. A decreasing absolute risk-aversion means that a wealthier decision-maker is willing to pay a smaller risk-premium to avoid uncertainty, while a constant relative risk-aversion implies that the decision-maker is prepared to put the same percentage of wealth at risk at all levels of wealth. Importantly, this notion of risk requires a continuously differentiable utility function and essentially equates risk with the concept of diminishing marginal utility. Other models of risk do not require these two concepts to be the same.

One of the most popular risk models in finance is the risk–return model proposed by Markowitz (1952). In this model an investor has to decide on a portfolio consisting of diverse financial products such as bank deposits, government bonds, shares, gold, etc. Some of these products may have very predictable payoffs (e.g., a savings account), whereas others might be more volatile (e.g., shares). An investor who considers both the return and the risk (variability of the return) of a portfolio is risk-sensitive. Such an investor bases his decisions on a trade-off f(x) between expected return r(x) and risk R(x) of a portfolio x such that f(x) = r(x) − θR(x), where θ expresses the investor's individual risk-attitude. A risk-neutral investor (θ = 0) only cares about the return, whereas a risk-averse investor (θ > 0) considers risk as a discount in utility and a risk-seeking investor (θ < 0) considers it as a bonus. A special case of the risk–return model is the mean–variance model, where return is formalized as the expected value r(x) = E[x] and risk is formalized as the variability in payoff, i.e., R(x) = VAR[x].

While the risk–return model provides an alternative approach to risk that is essentially different from the conception of risk in expected utility theory, under certain circumstances the two formalizations can be shown to be equivalent. For example, a decision-maker with a quadratic utility function will make decisions based on expected utilities that only depend on mean and variance of the payoff. Similarly, if payoffs have a Gaussian distribution and the utility function is monotonic and concave, then a decision-maker who follows expected utility theory maximizes a trade-off between mean and variance. In general, one can use Taylor series expansion to locally approximate smooth concave utility functions with a finite number of moments, such as mean and variance (Levy and Markowitz, 1979). Thus, the mean–variance formulation can always be considered as locally equivalent to the expected utility framework for such general utility functions. Some approaches have also suggested risk-sensitive models that consider higher order moments, for example, by using exponential risk functions that generate all moments (Whittle, 1981; Sarin, 1984).

Risk–return models, however, generally do not need to be consistent with expected utility theory. In fact, generalized risk–return models can account for preference patterns that cannot be captured by expected utility theory. If we assume a value function V(x), for example, that measures riskless preference (strength of preference), then we can define the return r(x) = E[V(x)], the risk R(x) = VAR[V(x)] and the trade-off f(x) = E[V(x)] − θVAR[V(x)] to account for preference patterns that violate the independence axiom of expected utility theory (Allais, 1953; Allais and Hagen, 1979; Sarin and Weber, 1993; Bar-Shira and Finkelshtain, 1999) – compare Figure A1 in Appendix for details. This approach also allows modeling perceived returns and perceived risks, where risk is treated as a fundamental quantity measured through direct judgments very much like psychophysical quantities such as brightness or loudness (Sarin and Weber, 1993).

Prospect theory was developed as a descriptive theory of decision-making in response to a host of experimental studies on human choice behavior that had reported violations of the normative axioms underlying expected utility theory (Allais, 1953; Attneave, 1953; Ellsberg, 1961; Lichtenstein et al., 1978). One of the most famous violations is the Allais paradox (Allais, 1953), where a decision-maker is faced with two different decisions that essentially present the same choice, but reveal an inconsistent preference reversal. In an adapted version reported by Kahneman and Tversky (1979), the first decision is between lottery A [0.33, U($2500); 0.66, U($2400); 0.01, U($0)] and lottery B [1.0, U($2400)], and the second decision is between lottery C [0.33, U($2500); 0.67, U($0)] and lottery D [0.34, U($2400); 0.66, U($0)]. If we assume that U($0) = 0 then both decisions only differ in their “common consequence” in that lotteries A and B simply add 0.66U($2400) to lotteries C and D. However, most subjects faced with these decisions choose B over A and C over D. This is apparently inconsistent and violates the independence axiom of expected utility theory, as the first choice would imply 0.33U($2500) < 0.34U($2400) and the second choice would imply 0.33U($2500) > 0.34U($2400), thus, reversing the first preference. This reversal can be considered as a special case of the fourfold pattern of risk suggested by Kahneman and Tversky (Tversky and Kahneman, 1992; Glimcher, 2008) and confirmed by several other studies (Fishburn and Kochenberger, 1979; Hershey and Schoemaker, 1980; Payne et al., 1981). These studies found that for high-probability gains subjects are typically risk-averse (as in the first decision of the Allais paradox), whereas for low-probability gains they are risk-seeking (for example, when playing in a casino). However, if lotteries are “framed” in terms of losses rather than gains then the risk pattern is reversed. Subjects are risk-seeking for high-probability losses (as when desperately trying to avert a sure loss) and risk-averse for low-probability losses (for example, when purchasing insurance).

Prospect theory accounts for this fourfold pattern of risk through a two-stage decision process. In the first stage, outcomes are “framed” as losses or gains relative to a reference point that depends on how prospects are presented and how they are mentally “edited” by the decision-maker. This is in contrast to expected utility theory where utilities are defined for absolute states of wealth. In the second stage, the prospect with the highest expected subjective value is selected. The expected subjective value of a prospect with outcomes X₁ and X₂ occurring with probabilities p₁ and p₂ is defined as V = ∑_i w(p_i)v(X_i). The subjective value function v(X_i) measures the subjective gain or loss of the outcome X_i, and the probability weighting function w(p_i) measures the subjective distortion of the probability p_i as a decision weight.

The subjective value function (Figure 1A) for human subjects is concave for gains (implying marginally diminishing value and contributing to risk-aversion for pure gains) and convex for losses (contributing to risk-seeking for pure losses). The subjective value function is also steeper for losses than for gains, a property known as loss aversion, leading to risk-averse behavior for mixed gain–loss gambles. For example, subjects typically only accept a 50:50 gamble when the potential gain is higher than the potential loss. The probability weighting function (Figure 1B) for human subjects is inverse-S shaped, thus, overweighting low probabilities and underweighting high probabilities. This helps explaining the fourfold pattern of risk, since underweighting of high probabilities reinforces risk-aversion for gains, and risk-seeking for losses as already implied by the shape of the value function, whereas overweighting of low probabilities counteracts the effects of the value function and permits risk-seeking for gains and risk-aversion for losses in the case of low probabilities. Both the subjective value function and the probability weighting function can be measured either by assuming particular parametric forms for v(X_i) and w(p_i) (Kahneman and Tversky, 1979; Prelec, 1998) or by non-parametric methods that allow determining v(X_i) and w(p_i) for particular X_i and p_i without assuming a specific parametric form (Wacker and Deneffe, 1996; Gonzalez and Wu, 1999; Abdellaoui, 2000; Bleichrodt and Pinto, 2000; Abdellaoui et al., 2007). In cumulative prospect theory the probability weighting function transforms cumulative probabilities rather than single event probabilities, which allows for a consistent generalization to arbitrary numbers of outcomes (Tversky and Kahneman, 1992). However, in both variants of the theory the fourfold pattern of risk depends on both the shape of the subjective value function and the distortion through the probability weighting function.

Throwing a dart at a dart board is a paradigmatic example of a motor task that involves explicitly given point rewards. The points define a payoff landscape that determines where on the board it is best to aim given the sensorimotor variability of the thrower. Trommershauser et al. (2003a,b, 2008) have exposed human subjects to pointing tasks similar to dart throwing and investigated whether subjects’ aiming behavior could be explained by statistical decision theory. In their experiments subjects could point to different target regions R_i each of which was labeled with a monetary reward G_i. The pointing movements had to be performed under time constraints. As subjects’ movements were inherently noisy, movement endpoints could be represented with a Gaussian probability distribution P(x′, y′/x, y) around the aim point (x, y). Thus, the probability of hitting target region R_i is given by P(Ri|x,y)=∫RiP(x′,y′|x,y)dx′ dy′. The hypothesis of maximum expected gain then states that subjects should choose their aim point (x, y) so as to maximize Γ(x, y) = ∑_i G_iP(R_i|x, y). Importantly, given a measure of the variability in pointing, specifying the payoffs G_i and the locations of the target regions R_i allows one to predict the optimal aiming point (x, y), which was tested experimentally. Trommershauser et al. (2003a,b, 2008) found that in contrast to many economic decision-making tasks, subjects’ motor behavior (i.e., their aim points) could be well described by the expected gain hypothesis. Since this model implies a linear utility function U(G_i) = G_i, this also implies risk-neutrality – compare for example both the absolute and the relative Arrow–Pratt measures of risk which are zero for linear utility functions.

In many motor tasks there is no explicit numerical reward, for example, when walking, cycling, or lifting a cup of coffee. Nevertheless, such motor tasks are amenable to theoretical investigation by optimality principles (Todorov, 2004). Optimality models for motor tasks can be classified into open-loop and closed-loop control schemes. Open-loop models predict an optimal desired trajectory from biomechanical constraints and a performance criterion like energy consumption or trajectory smoothness (Hatze and Buys, 1977; Flash and Hogan, 1985; Uno et al., 1989; Alexander, 1997; Nakano et al., 1999; Smeets and Brenner, 1999; Fagg et al., 2002). Most of these models deal with deterministic dynamics and are therefore devoid of risk, although some open-loop models take motor variability into account for planning optimal movements (Harris and Wolpert, 1998). In contrast, optimal feedback control (Todorov and Jordan, 2002) is a closed-loop modeling scheme in which sensory and motor noise are considered for finding the optimal feedback control law (sometimes also called “optimal policy”), which is a contingent observation-action plan for all possible states, rather than a pre-determined sequence of actions. Such optimal control laws have been found to successfully explain diverse phenomena, such as variability patterns and flexibility of arm movement trajectories (Todorov and Jordan, 2002; Liu and Todorov, 2007; Guigon et al., 2008), coordination in bimanual movements (Diedrichsen, 2007; Braun et al., 2009a; Diedrichsen and Dowling, 2009), adaptation to force-fields and visuomotor transforms (Izawa et al., 2008; Braun et al., 2009b), preservation of movement stability under uncertainty (Crevecoeur et al., 2010), adaptive control of saccades (Chen-Harris et al., 2008), object manipulation (Nagengast et al., 2009) and snowboard-like full-body movements (Stevenson et al., 2009).

Optimal feedback control models typically assume a biomechanical system (e.g., the arm) with state x_t and dynamics x_t+1 = f(x_t, u_t, ε_t), where u_t is the control command and ε_t is the motor noise. The controller receives feedback y_t = g(x_t, η_t) that is contaminated by sensory noise η_t (e.g., visual or haptic feedback). At each time-step this system incurs a cost c_t(x_t, u_t) that can depend on effort, task error, speed, and possibly other states of the biomechanical system. The optimal control problem is to find the control law that minimizes the total expected cost E⌊∑_t c_t(x_t, u_t)⌋, where the expectation is taken with respect to the probability distribution over trajectories induced by the control law. Thus, the optimal feedback control problem can be considered as a temporally extended motor “lottery” where the probabilities are given implicitly by the uncertainty over trajectories and the choices correspond to different policies that map past observations y₁, y₂, …, y_t to a motor command u_t. Since the expectation E⌊∑_t c_t(x_t, u_t) is linear in the cost, optimal feedback control models that minimize this expectation value are risk-neutral with respect to the cost.

Given the apparent discrepancy between economic studies on decision-making and the studies in motor control described above, the question arises whether the same subjects that exhibit risk-sensitivity in an economic decision task would act differently if the same decision-problem was presented as a motor task. Recently, this question was addressed experimentally (Wu et al., 2009). Wu et al. trained subjects on a motor task that required accurate pointing movements under time constraints, so that after training they could establish subjects’ probability p_i of hitting target region i with payoff $V_i. By manipulating the payoffs and probabilities, by adjusting the size of the target regions and the associated monetary rewards, they could present subjects with binary choices between varying motor lotteries of the form [p₁, V₁; p₂, V₂; p₃, V₃]. In particular, they were able to induce lotteries that only differed in “common consequences,” as in the Allais paradox, to study whether violations of expected utility theory also occur in motor tasks.

In their experiments Wu et al. used this paradigm of “common consequences” both in the motor task and in the equivalent economic decision task. In both cases they observed preference reversals that were inconsistent with expected utility theory. Importantly, “common consequence” lotteries also differ in their riskiness, that is in the variance of their payoffs. Wu et al. found that in the motor task subjects chose riskier lotteries with significantly higher frequencies than in the economic decision task. To explain this phenomenon they fit a prospect theory model to their subjects’ choice data. While there was no significant difference in the inferred value functions of the economic and the motor task, Wu et al. found a characteristic difference in the probability weighting function. In the motor task the inverse-S shaped weighting function of the economic decision task appeared mirrored on the diagonal, such that in the motor task low probabilities were underweighted and high probabilities overweighted. Since the same subjects overweighted low probabilities and underweighted high probabilities in the economic decision-making task, subjects exhibited opposite patterns of probability distortion in the motor and the economic decision-making task (compare Figure 1B). This difference in the probability weighting function also accounts for the increased risk-seeking observed during the motor task, since low probabilities of not winning are systematically underweighted.

In its simplest form the risk–return model formalizes risk-sensitivity as a trade-off between mean payoff and the variance of the payoff. Recently, this mean–variance model of risk-sensitivity has been tested by Nagengast et al. (2010b) in a motor task that required effort as an implicit payoff. In their task, subjects had a choice between a sure motor action associated with a fixed and certain effort and a risky action that was probabilistically associated with an effort that was either lower or higher than the certain effort. Similar to the studies by Trommershauser et al. (2003a,b, 2008), in the case of the risky action the probabilistic outcome was determined by subjects’ probability of hitting a designated target region within a short time limit. Crucially, the two possible outcomes of the risky action not only entailed a mean effort that could be compared to the certain effort, but also a variance. By manipulating the two outcomes of the risky choice appropriately, Nagengast et al. could fix the level of effort variance, while exposing subjects to different mean levels of effort. In this way they measured the indifference points where subjects chose equiprobably between the certain effort and the mean effort for a given variance level. Thus, subjects could be classified as risk-seeking, risk-averse, or risk-neutral depending on whether they accepted risky actions that had a higher, lower, or equal mean effort compared to the certain effort. In fact, the null-hypothesis of risk-neutrality could be rejected for most subjects in this task, with the majority being risk-seeking (Figure 2).

The risk-seeking behavior observed in this motor task with implicit effort payoffs is similar to the risk-seeking behavior reported by Wu et al. (2009) in their motor task with monetary payoffs. Accordingly, a fit of the trial-by-trial choice data with a prospect theory model reconfirmed the finding of Wu et al. about the probability weighting function underweighting small probabilities in motor tasks (compare Figure 1B). To this end, we assumed the commonplace parametric forms v(x) = −x^α and w(p) = exp[−(−lnp)^γ] for the subjective value function and the probability weighting function respectively (Wu et al., 2009), and we conducted a maximum likelihood fit for the parameters α and γ. However, this prospect theory model fit did not provide a better explanation of subjects’ choices than the mean–variance model. Unlike the study by Wu et al., the experimental setup allowed the mean and variance of the payoff to be manipulated separately, which in turn allowed the mean–variance trade-off to be directly measured. However, whether the brain represents risk in agreement with the mean–variance approach or with the prospect theory account is still subject to an ongoing debate (Boorman and Sallet, 2009). Recent evidence from electrophysiological and functional imaging studies has provided support for both theories. In support of the mean–variance approach, separate encoding of reward magnitude and risk has been reported in humans (Preuschoff et al., 2006; Tobler et al., 2007, 2009) as well as in non-human primates (Tobler et al., 2005). However, recent studies have also found neural evidence in favor of prospect theory, such as neural correlates of framing processes (Martino et al., 2006) and neural responses that depended on probabilities in a non-linear fashion during a risky task (Hsu et al., 2009). Both effects are cornerstones of prospect theory. However, further studies are needed to elucidate how the brain represents value and how the brain's different valuation and action selection systems interact and vie for control to arrive at an overt behavioral decision (Rangel et al., 2008).

In contrast to risk-neutral optimal feedback controllers that have been widely used to model motor behavior (Todorov and Jordan, 2002), a risk-sensitive optimal feedback controller depends not only on the mean expectation value of the cost but also on higher order moments, such as the variance of the cost (Whittle, 1981). An optimal controller with risk-sensitivity θ optimizes the criterion γ(θ)=−(2/θ)log⁡E[e(−1/2)θ∑tct(xt,ut)], with θ < 0 for a risk-averse controller, θ > 0 for a risk-seeking controller, and θ = 0 for the risk-neutral controller where the criterion function coincides with the standard expected cost function E ⌊∑_t c_t(x_t, u_t)⌋. Interestingly, this criterion is compatible with a mean–variance notion of risk, since the first two terms of a Taylor series approximation of γ(θ) correspond to mean and variance, i.e., γ(θ) ≈ E[∑_t c_t(x_t, u_t)] − θ/4 VAR[∑_t c_t(x_t, u_t)]. In the case of a system with linear dynamics, quadratic cost function, and additive noise, such a risk-sensitive control scheme predicts that the control gain should be dependent on the magnitude of the process noise, whereas a risk-neutral control scheme predicts that the control gain should be unaffected by this noise level.

Recently, Nagengast et al. (2010a) tested this prediction of the risk-sensitive control framework in a human movement task – compare Figure 3. Subjects had to control a virtual ball that underwent Brownian motion resulting from an additive noise process with a given magnitude. The ball moved forward with constant velocity toward a target line, but the ball's trajectory fluctuated randomly to the left and to the right according to the process of Brownian motion. Subjects were required to minimize an explicit cost displayed in points that was a combination of the final positional error measured as the distance of the ball from the center of the target line and the integrated control cost. Accordingly, the Brownian motion introduced a task-relevant variability directly affecting the task cost (Franklin and Wolpert, 2008). Since the control costs were given explicitly in this task and did not have to be fitted to subjects’ behavior, the only free parameters of the risk-sensitive model were the magnitude of the noise and the risk-parameter. By testing subjects on different levels of Brownian motion noise, it was possible to test whether subjects changed their control gains in accordance with a risk-sensitive account of control, or were indifferent to such changes in accordance with a risk-neutral account of control. Nagengast et al. found that for the same error experienced in a particular trial, most subjects intervened more in a task with large error statistics than in a task with small error statistics (Figure 4). Thus, the statistics of preceding trials affects the reaction to the same error.

This behavior is consistent with risk-aversion, but inconsistent with a risk-neutral account of motor control. Therefore, subjects acted pessimistically in the presence of noise, as they were prepared to accept higher control costs in order to avoid losses.