Computational Modeling of Behavior

Computational models of behavior offer a large and powerful explanatory repertoire with the potential of integrating information derived from a variety of different sources (e.g., electrophysiology, neuroimaging, behavior) into a coherent theoretical structure. Applying such models to psychiatric disorder symptomology offers a number of advantages:

• 1.   Models requirehypotheses to be explicitly quantified. That is to say, experimental design and analysis require an explicit proposal of what is driving behavior or neural activity. Rather than specifying that major depression is associated with, say, “negative affective bias” (Roiser, Elliott, & Sahakian, 2012), the component parts driving that bias must be specified. For instance, one must specify if a single parameter drives sensitivity to reward and punishment processing (i.e., whether an individual dislikes punishments to the exact same degree that he or she likes rewards) or whether these should be considered separate processes (allowing particularly extreme dislike of punishments with ambivalence toward rewards, for example).
• 2.   Although “all models are wrong” (Box, 1976), some are more wrong than others, and evaluation of the components that improve model fit in a model comparison procedure can formally assess the relative strengths of competing hypotheses to explain a given dataset. For example, the relative explanatory validity of single versus separate reward and punishment sensitivity parameters can be directly compared. Simulations and comparison across datasets can then provide further support for or against a given model (Palminteri, Wyart, & Koechlin, 2017).
• 3.   Rather than relying on summary mean or variance statistics, models can be used to explore trial-by-trial variance. This is particularly important when studying cognitive and learning processes (which are strongly implicated in many psychiatric disorders, including major depression; Rock, Roiser, Riedel, & Blackwell, 2014) in which subjects’ behav ior can change dynamically through the task. This temporally rich approach explores variance over time and thus obtains more information from a given dataset. Indeed, subtle effects that are often overwhelmed in collapsed mean accuracy or response times can only be revealed using the modeling approach (White, Ratcliff, Vasey, & McKoon, 2010).
• 4.   Models can be constrained by our understanding of what is biophysically plausible given our understanding of neuronal and pharmacological interactions. Combined with neuroimaging, winning model parameters can provide us with a means of mechanistically linking observable behavior and underlying neural substrates.
• 5.   Models can show emergent properties, and explanations that might otherwise be considered excessively complex can be specified and tested. A good example of this is where meta parameters—or variance of model parameters—are implicated in pathologies. For instance, trait anxiety is associated with reduced learning rate adaptability (Browning, Behrens, Jocham, O’Reilly, & Bishop, 2015). Specifying and identifying this alteration without a model would be more difficult from both practical and conceptual standpoints.

In this review, we focus on the application of computational modeling to anhedonia—diminished reward processing—in major depression. We review findings from two primary classes of models (Figure 1) that have been implicated in anhedonia (and, to a lesser extent, in anxiety, which is often comorbid with depression): (a) reinforcement learning models and (b) reaction time models. We also briefly refer to (c) models of economic choice under uncertainty, as they allow us to illustrate a point about the importance of temperature (see section “Simulation Showing the Importance of Temperature in Decision-Making Models”). Notably, although we consider them separately here, these models can also be combined to generate more complicated models (Pedersen, Frank, & Biele, 2016).

Anhedonia and Impairments of Reward-Directed Behavior

A recurrent feature of a variety of psychiatric disorders is a “loss of interest or pleasure” in previously enjoyable activities. This phenotype is referred to as anhedonia and is particularly prevalent in disorders such major depression, schizophrenia (SZ), and addiction (Franken, Rassin, & Muris, 2007). Here we focus on major depression, in which anhedonia is a key symptom used for diagnosis. Although anhedonia might be straightforwardly characterized in terms of a reduction in response (across cognitive domains) to rewarding events, many features of the phenotype remain puzzling (Pizzagalli, 2010). For instance, it has been argued that anhedonia should be broken down into anticipatory, decisional, and consummatory components (Argyropoulos & Nutt, 2013; Treadway & Zald, 2011) as well as further distinct disturbances in the socialdomain (Christianson et al., 2008). Within these categories, responses to certain kinds of reinforcers might be selectively altered (e.g., affective responses to music; Martinez-Molina, Mas-Herrero, Rodriguez-Fornells, Zatorre, & Marco-Pallares, 2016).

Anhedonia is often measured, for both experimental and clinical purposes, using self-report questionnaires and interviews (Rizvi, Pizzagalli, Sproule, & Kennedy, 2016). Although these tools are frequently psychometrically reliable, they suffer from prominent limitations, namely, demand characteristics (i.e., individuals give you the answer they think you expect) and anchoring effects (i.e., individuals give answers consistent with the first answer they gave). Indeed, as highlighted by the memory literature, an individual’s pattern of behavioral responses on cognitive tasks can be a more reliable proxy of the degree of memory encoding than self-report (Shanks & St. John, 1994; Vadillo, Konstantinidis, & Shanks, 2016). Consistent with this is evidence that behavioral measures of anhedonia can vary independently from self-reported anhedonia (Pechtel, Dutra, Goetz, & Pizzagalli, 2013). Moreover, despite clear differences in self-reported anhedonia, prior reviews have yielded little evidence of substantial differences between depressed and healthy individuals in terms of their hedonic responses on consummatory tests (Treadway & Zald, 2011). For instance, the pleasure-evoking properties of sweet tastes are generally similarly rated by depressed and healthy individuals (Treadway & Zald, 2011). This discrepancy is somewhat perplexing—if anhedonic individuals describe a selective lack of interest in rewarding experiences, why is this not borne out in their behavioral responses? The computational approach might help resolve this question. For instance, it may be that, while the end points (e.g., response to sweet tastes) appear identical in anhe donic individuals, the individual components of the mechanisms (e.g., detection of, or learning about, the tastes) that lead to this end point differ. The anhedonic individual may take longer, or require more evidence, to reach the same end point. These effects will be hidden if one measures the end point alone: Computational models enable the specification and exploration of these hidden component parts.

Sensitivity to Value

One way to consider a deficit in reward-oriented behavior is to argue that individuals are able to learn associations between actions and reinforcers but fail to use this information to guide behavior. A typical choice rule (Equation 1), also referred to as an observation model, describes the probability of eliciting a response (i.e., action probability) given an action’s value using what is known as a softmax function:

In two alternative forced-choice (2AFC) cognitive tasks frequently used to study RL, a participant will have no option but to accept one option, so two equally (un)attractive options will lead to essentially random responding, because neither is favored. In go/no-go tasks—where the alternative choice is to do nothing (no-go)—this would lead to random decisions to wait (or, in the case of animal models, engage in grooming behavior). Broadly, however, the empirical evidence has not strongly supported a clear role of reduced sensitivity to value in anhedonia. There are numerous examples of intact or at least adequate instrumental control by reinforcers: The majority of studies in Table 1 report broadly compatible task acquisition across healthy and depressed cohorts.

Learning Rate

While patients with depression can show moderate deficits in memory (Rock et al., 2014), they typically show broadly intact acquisition of the basic reward contingencies employed in RL tasks, indicating intact learning processes. Computational methods provide some support for this view, with tasks that are sensitive to anhedonia showing intact task acquisition learning (Huys, Pizzagalli, Bogdan, & Dayan, 2013). Moreover, tasks which have designed to isolate specific learning rates for rewards and punishments (i.e., where the learning rate is different for rewarding and aversive outcomes, respectively; see Equation 2) have provided mixed data but have frequently failed to support diminished reward learning or heightened punishment learning (see Table 1). For example, in the case of the Probabilistic Selection Task (PST), one study showed similar rates of positive and negative learning between controls and patients (Chase, Frank et al., 2010) and a second showed relatively enhanced positive learning in low- but not high-dysphoric individuals (Kunisato et al., 2012), while two further studies showed complex findings depending on trial type (Cavanagh et al., 2011) or rumination induction (Whitmer et al., 2012). Finally, a theoretically compatible but somewhat distinct procedure revealed asymmetrical learning rates as well as an influence of medication (Herzallah et al., 2013). Thus, when taken together, the complexity of these findings argues against a simple alteration of learning rates.

In some cases, differences in performance may emerge because individuals adopt a different learning model from that outlined in Equation 2. Individuals may vary in their tendency to change behavior on the trial immediately after unexpected feedback. Within the basic RL framework, this might be driven by a heightened (Murphy et al., 2003) or by a diminished (Chase, Frank et al., 2010; Steele, Kumar, & Ebmeier, 2007) impact of the previous trial. The broader interpretation of this type of feedback sensitivity is a topic of active debate: Recent computational approaches sometimes posit a win–stay, lose–shift parameter (den Ouden et al., 2013; Myers et al., 2016), which is, in essence, a simplistic 1-back learning model and may reflect the function of working memory (Collins & Frank, 2012). This representational system does not keep track of reinforcement history beyond a very restricted number of recent outcomes (e.g., one). This approach can work well in some environments but will perform poorly on tasks (such as the commonly used Iowa Gambling Task; IGT) in which the most rewarding choice is also associated with large, if occasional, losses and in which incremental learning is necessary.

It is possible that these sorts of decision-making strategies (as well as different exploratory strategies; Knox, Otto, Stone, & Love, 2011), rather than a more fundamentally impaired learning rate mechanism, are responsible for the relatively subtle deficits that are observed in patient groups. Critically, the adoption of such strategies does not preclude intact learning: They may simply reflect selection of a strategy that will perform favorably in deterministic designs but suboptimally in others (Collins, Brown, Gold, Waltz, & Frank, 2014). Further support for this position is the presence of deficits on tasks with shifting contingencies, such as probabilistic reversal learning (Murphy et al., 2003) and the IGT (Must et al., 2006), performance on which may be particularly dependent on such strategies (e.g., den Ouden et al., 2013). Some modeling approaches (Collins & Frank, 2012; Myers et al., 2016) have therefore sought to combine these two approaches, with both RL and win–stay, lose–shift components in the same model, combined with a free parameter to determine the likelihood of one or another when the two systems come into conflict (e.g., straight after negative feedback). Overall, however, clear evidence for altered learning rates in anhedonia across a variety of paradigms is lacking.

Pavlovian Bias

Learned or prepotent Pavlovian biases can represent powerful decision-making shortcuts that can influence RL. Specifically, approaching rewards and avoiding punishments, or stimuli which may predict them, consitutes a simple behavioral heuristic that will serve well in a wide range of scenarios. These adaptive behavioral heuristics are often referred to as Pavlovian biases. Pavlovian instrumental transfer (PIT) describes the influence that these biases can have over instrumental control (e.g., the simple goal-directed behaviors described by Equations 1 and 2). Experimentally, this manifests as, for example, making it extremely difficult for both humans and animal models to “approach” (e.g., make a go response to) a cue that has previously been associated with punishment (Guitart-Masip et al., 2011). Cavanagh, Eisenberg, Guitart-Masip, Huys, and Frank (2013) have operationalized Pavlovian influence over RL as a multiplier on stimulus value (and is often represented by an independent, additive parameter to that representing action or instrumental value: cf. Holland, 2004):

If the value (see Equation 2) is positive (i.e., rewarding), then this bias parameter increases value, whereas if the value is negative (punishing), this parameter serves to decrease value. Examining PIT, Huys, Golzer, et al. (2016) observed a reduced influence of Pavlovian bias on approach and avoidance behavior in depressed individuals compared to healthy controls (HC). Similarly, work has demonstrated that an approach bias for positive stimuli and an avoidance bias for negative stimuli in HC—that is, Pavlovian bias—is absent in depressed individuals (Radke et al., 2014). In other words, although overall instrumental behavior is generally intact in depression, the influence of incidentalPavlovian cues on behavior may be reduced. These findings from PIT or PIT-like paradigms may also be relevant for interpreting data from emotional cues more generally: A meta-analysis of responses to emotional cues, both positive and negative, suggested a general deficit in response to affective cues in depression (Bylsma et al., 2008).

Of note, however, the impact of Pavlovian bias is dependent on individuals having obtained the learned association. If they do not know that a cue leads to reward, then they will not be driven to approach it. As such, the impact of these sorts of biases may change over time. They will have minimal impact at the start of the process before the cue–outcome contingencies have been aquired. Reduced Pavlovian influence in depressed patients could therefore plausibly reflect delayed instrumental learning in some cases (despite eventual intact learning, given enough trials). Notably, reduced Pavlovian influence in depressed individuals may differ from what is found in those with increased clinical anxiety symptoms who demonstrate increasedreliance on Pavlovian avoidance (and not approach) biases (Mkrtchian et al., 2017). Existing work is on this topic is limited, so further exploration of the interaction between Pavlovian bias and instrumental learning in major depressive disorder (MDD) would be worthwhile.

Choice Variability (Temperature)

Returning to the softmax choice function (Equation 1), we can add another parameter, which provides a modulatory impact on choice. This parameter is referred to in different ways across the literature, but one common description is temperature, with greater temperature referring to greater response variability:

Increased reliance on the temperature parameter has been used to explain performance on tasks in which depression is linked to increased exploration (Huys et al., 2012; Kunisato et al., 2012). Reinforcement learning–based analyses have argued that increased exploration drives response switching on RL tasks (Blanco et al., 2013), probabilistic reversal learning (Dombrovski, Szanto, Clark, Reynolds, & Siegle, 2013; Murphy et al., 2003; Taylor Tavares et al., 2008), and IGTs (Must et al., 2006). However, in many of these designs, it is also possible to explain behavior in terms of increased temperature. This, in turn, may be due to impaired task acquisition but could also be due to unmodeled decision-making strategies (e.g., the win–stay, lose–shift process described in section “Learning Rate”).

Perhaps the clearest evidence for the value–temperature trade-off comes from the probabilistic reward task (Pizzagalli, Jahn, & O’Shea, 2005), in which individuals are asked to discriminate between two lengths of lines presented on a screen. One of the responses has a greater probability of reinforcement following an accurate response, so that, for example, the accurate selection of the long line is three times more likely to be rewarded than the accurate selection of the short line. By providing asymmetric reward for one option, a reward-related bias is introduced in healthy individuals who normatively select the rewarded option if unsure (Huys et al., 2013). Individuals with high levels of anhedonia generally fail to show such a bias (Huys et al., 2013). Critically, a meta-analysis of this task using a computational analysis of performance identified increased temperature as the key parameter driving these effects in anhedonia (Huys et al., 2013). In other words, anhedonia led to noisier, more variable choices. As far as we are aware, the only existing study in which temperature was shown to be reduced by major depression/anhedonia (i.e., the opposite effect) was a study of the probabilistic selection task referred to in Table 1 (Chase, Frank et al., 2010). However, a reduced overall learning rate associated with anhedonia was also observed in this study, and it is generally difficult to dissociate estimation of temperature from overall performance or learning rate estimation (Daw, 2011). In summary, therefore, it seems plausible that anhedonia may be associated with increased temperature.

Simulation showing the importance of temperature in decision-making models

Looking at the softmax choice equation (Equation 4), increasing the value of the chosen option can be directly counteracted by increasing the temperature parameter, such that these parameters are now necessarily “underdetermined” or, more informally, “two sides of the same coin.” This point is emphasized by Huys and colleagues (2013) in their analysis of the probabilistic reward task and is valuable for two reasons. First, it increases focus on aspects of paradigm design and model fitting that might obscure identification of choice variability (or be obscured by it). Second, it has implications for sample size and statistical power: Specifically, choice variability, whether well specified or controlled by a particular task, or not, adds noise and provides an impediment to accurately identifying other modeled parameters. This could potentially affect the estimation of model parameters for many choice-oriented tasks across a variety of cognitive domains.

We can illustrate this interaction between value and temperature through simulation. Here we considered economic decision making under uncertainty (Figure 1C) rather than RL, because temperature is more transparently estimated in the absence of trial-by-trial learning (Daw, 2011). Critically, this simulation assumes that values and associated probabilities of outcomes are explicitly stated so that changes in decisions are not confounded with differences in learning the values:

((5))

$Utility=Probability\cdot (E{V}^{EiskAversion}).$

Consider the following: A decision-making paradigm of 100 trials assesses risk preference by asking the participant to choose between a risky 50% chance of winning $1 (vs. $0) or a sure 100% chance of winning $0.50 (i.e., on average, the value of both options is the same). Preference is determined by the relative valuation of $0.50 relative to $1, for example, a curved utility function (Equation 5). If risk aversion is greater than 1, the risky option is preferred, and if it is less than 1, the safe option is preferred. However, we might also assume that choice is stochastic, controlled by a softmax function and an accompanying temperature parameter (Equation 4).

We conducted 10,000 simulations of this scenario, allowing temperature to vary between 0 and 2 and the curvature (risk aversion) to vary between 0.5 and 1.5.¹ A multiple regression model was then fit to the resulting data, with risk preference (the proportion of risky options selected) being predicted by temperature, risk aversion, and their interaction. As expected, the curvature/risk aversion parameter (t = 67.96) was a highly significant predictor of risky choice, and temperature on its own had little effect (t = −1.87). Critically, however, their interaction was highly significant (t = −62.99). This interaction reflects the crucial influence of the temperature parameter, controlling the effect of the mapping between the key construct of interest (the valuation function) and the dependent measure (risk preference; see Figure 1C). This simulation demonstrates how temperature can play a crucial moderating role, with increases in temperature diminishing the influence of a given manipulation (in this case, risk aversion). Specifically, as can be seen by the red dots in Figure 2, at high temperatures, the probability of a risky choice is essentially random (50%) regardless of the level of risk aversion, and the curvature evident in the low-temperature blue dots is absent.

Figure 2.  

Moderation of relationship between the risk aversion parameter (see Equation 5) and risk preference by temperature. High temperatures are red; low temperatures are blue. A low score on the risk aversion parameter amplifies the utility of small wins, leading to risk aversion, but this is only clearly manifest in behavior if the temperature is low. Likewise, a high score reduces the utility of small wins, leading to risk seeking, but again, only if the temperature is low.

Ultimately, choice variability can have a major impact on observed behavior, and failing to account for it may lead to erroneous inferences regarding the influence of an experimental manipulation. For instance, reduced temperature might lead to a pattern of risky choice in healthy individuals, while high temperature would lead to (relatively) more random responding in a patient group. A pattern of noisier decisions could therefore be incorrectly taken as evidence of reduced risk taking in the patient group. In fact, the consequences are much broader than the present example: High temperature would make any other parameter within the same model harder to estimate, regardless of the paradigm or direction of between-group effects.

Anhedonia Reduces Drift Rate but No Other Parameters in Drift Diffusion Models?

A smaller number of studies have explored the effect of the drift diffusion model in depression. In a study using the flanker task—where subjects have to identify the orientation of an arrow in the face of congruent and incongruent proceeding distractors—depressed individuals were slower but more accurate on incongruent trials. Modeling showed that this was driven by reduced prepotent and executive drift rates across both congruent error and correct incongruent trials (Dillon et al., 2015) in depressed individuals compared to controls. This replicated an effect of reduced drift rate in depressed patients on a signal (color ratio) detection task (Vallesi, Canalaz, Balestrieri, & Brambilla, 2015) and prior work with the flanker task (Pe, Vandekerckhove, & Kuppens, 2013). Critically, drift rate correlated negatively with overall depressive symptoms (Pe et al., 2013; Vallesi et al., 2015) and with anhedonia specifically (Dillon et al., 2015). In none of the studies was depression associated with increased nondecision time, arguing against a generic psychomotor slowing effect; decision threshold (boundaries), arguing against speed–accuracy trade-offs (Dillon et al., 2015); or starting point, arguing against asymmetrical differences in boundaries.

However, it should be noted that there is some concern that the diffusion model can be overspecified, especially when trial numbers are limited and the study is not designed to detect more subtle effects (van Ravenzwaaij, Donkin, & Vandekerckhove, 2017). Indeed, very basic versions of the model can outperform more complex versions in simulations (van Ravenzwaaij et al., 2017). This is because more complex models can lead to overfitting, especially when the number of trials recorded is limited, which in turn reduces the power to detect group differences in core parameters (e.g., decision boundaries; van Ravenzwaaij et al., 2017). Comprehensive model comparison is needed in future research to be confident that effects of depression are in fact restricted to drift rate.

Altered Reward-Related Striatal Reactivity in Major Depressive Disorder

The learning and decision-making processes described can be integrated with neuroimaging in an attempt to map model parameters onto the underlying neuronal hardware and bridge the symptomatic, computational, and neurobiological levels. Release of dopamine in the striatum has long been linked to RL models (Kishida et al., 2016; Lohrenz, Kishida, & Montague, 2016; Pessiglione, Seymour, Flandin, Dolan, & Frith, 2006), so a large body of work has focused on the role of the striatum in anhedonia and depression. Before discussing explicitly computational work, we first highlight noncomputational work implicating the ventral striatum (VS) in anhedonia and depression.

A widely used paradigm for assessing reward and motivational processes in psychiatric populations is the monetary incentive delay paradigm (MID; Knutson, Fong, Adams, Varner, & Hommer, 2001; Knutson & Heinz, 2015). Although there are a variety of task versions, the general structure involves an expectancy cue that informs the participant about the amount of a reward he or she will receive if he or she makes a fast response. In general, the task is designed such that it is possible to separately model neural responses to (a) anticipation of reward and (b) receipt of rewarded outcomes.

We identified 13 studies in the literature using this task that are of relevance to the present discussion (Table 2). The task elicits quite reliable activation within the striatum in healthy individuals (Wu, Samanez-Larkin, Katovich, & Knutson, 2014), but there are inconsistencies as to when this activation occurs. While most studies reveal reduced activity in depression and anhedonia, they are coupled variably to anticipation or outcome and to different regions within the striatum. It is often less explicitly articulated (but see Knutson, Bhanji, Cooney, Atlas, & Gotlib, 2008) that the presence of null results—no difference between patients and controls—opens the potential for robust VS activations in the MDD group. Thus there is an obvious parallel with behavioral studies reviewed earlier: While deficits are regularly observed, the overall picture is not one of global hyporesponsivity to rewards. In addition, although there is a theme that less severely depressed individuals are less likely to show deficits (Hagele et al., 2015), Table 2 includes some studies with relatively severe patients that report null findings (Knutson et al., 2008).

It should be also noted that some studies, one in healthy adults (Schlagenhauf et al., 2013) and another in a geriatric depressed population (Dombrovski et al., 2015), have demonstrated correlations between VS activation and generic measures of cognitive function (IQ and self-reported executive function, respectively). These findings provide a complication in that cognitive dysfunction and anhedonic symptoms may be correlated and confounded. It is not always possible to correct for these associations within a given analysis. Moreover, the limitations of reverse inference are relevant in interpreting striatal activations coupled to reward events (Poldrack, 2011). Specifically, a reduction in striatal response, frequently observed in the absence of behavioral or symptomatic effects, is often interpreted as reduced reward sensitivity. This is clearly an oversimplification, as many studies have shown that striatal activation can be elicited by a wide range of behaviors and stimuli, including, for instance, punishment (Robinson, Overstreet, Charney, Vytal, & Grillon, 2013). Nevertheless, we interpret the evidence in Table 2 as broadly supporting a role of attenuated VS response to reward in anhedonia.

Integrating Computational Approaches and Brain Imaging

The work reviewed in Table 2 nevertheless shows considerable heterogeneity. As argued earlier, computational models can help isolate sources of this variability. The integration of computational and fMRI methods is sometimes referred to as model-based analysis and represents a departure from the typical approach to modeling stimuli in neuroimaging that assumes that the eliciting stimulus is essentially equivalent on every trial. Specifically, computational models enable the exploration of parameters that vary on a trial-by-trial basis within subjects. Parameters such as temperature, explored in the section “Simulation Showing the Importance of Temperature in Decision-Making Models,” are generally fixed for individual subjects (i.e., they differ between subjects but do not change on a trial-by-trial basis), but these fixed values can be combined with the pattern of rewards and punishments received by the individual to calculate trial-by-trial regressors like PEs or value. These regressors can then be used as param etric modulators of functional magnetic resonance imaging acquisition outputs over time to determine their neural correlates. Specifically, PEs at a given time represent deviations from expected values (see also Equation 2).

What Is the Role of Temperature?

As discussed, it may well be that some decision-making differences in anhedonia are down to noisier decisions captured by the temperature parameter, but a major empirical challenge for studying the temperature parameter is that it is difficult to separate from value (see also Huys et al., 2013). Given that these two, supposedly distinct, constructs are inextricably linked within the choice rule (e.g., the softmax in Equation 4), it may not be possible to disambiguate them cleanly within one single paradigm. Moreover, although other choice rules for 2AFC paradigms might be considered, they often show very similar properties and a similar underlying mathematical form (Bogacz, Brown, Moehlis, Holmes, & Cohen, 2006; Yuille & Gaiger, 2003).

We therefore identify four potential strategies for resolving the role of temperature. First, if noisy, temperature-driven choices are a critical component of anhedonia, then they should be seen across different paradigms and task contingencies. If high choice variability were not also seen on other, conceptually distinct paradigms in the same individuals, then it might be possible to build a case that the effects on an effort task (for instance) are not driven by temperature differences. Second, it may be possible to manipulate outcome uncertainty to promote or reduce noisy decisions (Le Pelley, Suret, & Beesley, 2009), while keeping value consistent (and vice versa). Alternatively, forms of directed exploration have been proposed (Frank, Doll, Oas-Terpstra, & Moreno, 2009; Wilson, Geana, White, Ludvig, & Cohen, 2014), which might exist independently of a more passive, disinterested choice variability and be revealed on certain kinds of paradigms. The temperature parameter as realized within the softmax algorithm (Equation 4) is theoretically silent on this difference, and it may be that isolating these two components will provide more specificity into the underlying alteration in individuals with MDD. Together, these experimental approaches might enable the identification of exploration-related phenomena, which can be modeled independently of option value.

Third, it is worth considering whether a participant whose data provide evidence of high choice variability is just poorly fit by the model employed. Temperature could simply reflect the “residual” behavior that is not explained by the model itself. Thus other potential models of behavior might be explored that would better account for observed performance to clarify whether increased temperature, rather than just a poorly fitting model, is indeed the driving factor. Full comparison of a wide range of models, followed by clear simulations that reiterate nonmodeled behavioral patterns, can help determine the extent of this issue (Palminteri et al., 2017).

Finally, following from the preceding, large sample sizes can also provide benefits to model comparison and fitting. They can provide a clearer picture of parameter distributions and assist in their estimation, while nonnormal or discontinuous distribution of data (e.g., Chase et al., 2017; Chung et al., 2017) may be more readily identified. In addition, deriving more precise predictions from previous studies and/or via direct replication may facilitate Bayesian parameter estimation (Gershman, 2016).

Can We Disentangle Responses to Outcome Versus Cue?

Adopting a reinforcement learning–based approach to modeling and interpreting ventral striatal activation may help to explain differences between findings arising from reward paradigms of different design or contingencies (e.g., see Table 2), without needing to suggest separate mechanisms for cue- and outcome-locked activation. For instance, some paradigms may employ relatively sparse reinforcement, that is, a weak relationship between predictive cues and rewards (e.g., Segarra et al., 2015), and thus show mostly low cue-locked reward anticipation. In this case, outcome-locked activation would be expected to carry a mostly PE-related signal, but it may also vary between subjects or groups in terms of outcome sensitivity, but not in terms of learning rate. In other words, a patient versus control group difference in a sparse reinforcement contingency is more likely to be driven by differences in outcome value or PE than by differences in learning rate. By contrast, in paradigms in which rewards are more frequent, and preceding cues can be effective predictors of the value of the outcome, both outcome- and anticipation-locked activation might show variability related to individual differences in both learning rate and outcome value. Thus, in this case, a group difference related to learning rate might be revealed in a way that would not be apparent in a sparse reinforcement design. Direct exploration of the impact of sparse versus frequent rewards in anhedonic individuals may help shed some light on this issue.

However, there are areas where the explanatory capability of a simple RL model may become limited. Importantly, for example, RL in neuroimaging has generally been applied to describe phasic, spiking responses of midbrain dopamine (DA) neurons (Schultz, Dayan, & Montague, 1997), as opposed to tonic DA release. While modification of the RL framework has been suggested to account for tonic DA fluctuations (Daw, Kakade, & Dayan, 2002; Niv, Daw, Joel, & Dayan, 2007), other evidence suggests that DA release at different timescales can be integrated into a single signal (Hamid et al., 2016). The operation of motivational systems at different timescales may be relevant for MDD, with respect to both behavioral evidence (Dillon et al., 2015) and neural responses to the cue and to the outcome. In particular, several studies have suggested that sustained rather than phasic responses to reward can be associated with MDD (Admon & Pizzagalli, 2015) and response to treatment (Heller et al., 2013). Thus building models that distinguish between the phasic and sustained responses to reward is likely to be an important future direction and may contribute to resolving some of the existing discrepancies with respect to cue- and outcome-locked activations.

What Is the Role of Patient Heterogeneity?

Much of the logic behind the present work is based on the idea that patients with MDD will show high levels of anhedonia, which might drive the reward-related deficits observed. However, although it is true that anhedonia is a core clinical feature of MDD, a diagnosis of MDD may be given on the basis of heightened negative mood, despite normal hedonic tone. A radical consequence of this heterogeneity may be the inability to reproduce patterns of neuroimaging findings across cohorts (Muller et al., 2017). Although reward-based neuroimaging paradigms have been successful in identifying differences between patients with MDD and controls (Zhang et al., 2013), analyzing data as a function of continuous constructs may provide more rigorous tests by accounting for the heterogeneity within a given cohort. For example, using continuous measures, such as anhedonia questionnaire scores—as suggested by the research domain criteria framework (Kozak & Cuthbert, 2016)—may help resolve aspects of the conflicting data presented in Table 2. However, this endeavor will also benefit from more effective and validated measures of anhedonia (Rizvi et al., 2016). In this regard, computational methods may provide some assistance: A metric of approach motivation or reward sensitivity derived from a computational model of behavior could, for instance, be used as a between-subject continuous variable in such analyses.

Medication remains another important source of variability, one that can be difficult to control in patient populations (e.g., Hafeman, Chang, Garrett, Sanders, & Phillips, 2012). Potential solutions to this problem are to (a) examine the effect of medication in a control population on a comparable task (Kumar et al., 2008) or (b) examine unmedicated patients to compare with findings from medicated patients (Greenberg et al., 2015; Robinson et al., 2012). All of these possibilities are associated with inferential blind spots, but there should be no reason why the influence of medication could not be isolated eventually, as has been possible with Parkinson’s disease, for example (Cools, Barker, Sahakian, & Robbins, 2001).

How Does Anxiety Interact With Anhedonia?

Related to the question of patient heterogeneity is comorbidity. Anhedonia is highly comorbid with anxiety. If anhedonia can be operationally characterized as reduced processing of rewards, the affective state of anxiety can be operationally characterized as increased processing of threats (Davis, Walker, Miles, & Grillon, 2010; Grillon, 2008; Robinson, Vytal, Cornwell, & Grillon, 2013). These are complementary and competing drives, and a full understanding of mood disorders will likely require integrating understandings of both.

Prior work using computational approaches to understand elevated aversive processing in anxiety is, however, extremely limited (but for a review, see Raymond, Steele, & Series, 2017). From a RL perspective, anxiety induced by threat of shock has been shown to increase a positively signed aversive PE signal to unexpected fearful face stimuli in the VS (Robinson, Overstreet, et al., 2013). This replicates animal work (Oleson, Gentry, Chioma, & Cheer, 2012) and is broadly consistent with a wide range of studies implicating the VS in aversive PEs under normal conditions (Delgado, Li, Schiller, & Phelps, 2008; Robinson, Frank, Sahakian, & Cools, 2010; Seymour, Daw, Dayan, Singer, & Dolan, 2007) but contrasts with the putative attenuated striatal response to reward in anhedonia reviewed earlier in this article. Similarly, while the earlier reviewed evidence did not suggest a role of learning rate differences in the manifestation of anhedonia, trait anxiety has been shown to be negatively associated with the ability to modulate learning rates (Browning et al., 2015) in response to environmental volatility. This could potentially result in aversive outcomes being experienced as less predictable and controllable than they are and thus help maintain the anxious state. Moreover, pathological anxiety has also been shown to increase rather than decrease reliance on Pavlovian avoidance parameters (Mkrtchian et al., 2017). Notably, in this particular study, the winning model had two separate Pavlovian parameters: one for approaching rewards and one for avoiding punishments. Anxiety disorders were associated with increased reliance on the avoidance parameter but not on the approach parameter. An interesting question for future work in anhedonia, therefore, is whether a similar, albeit attenuated bias exists for the approach parameter alone if it is modeled separately. Another study looking at models of economic decision making under uncertainty (Figure 1C) suggested that anxiety was associated with risk but not with loss aversion (Charpentier et al., 2016). Finally, there is evidence that high trait anxiety is associated with increased boundary separation and nondecision time on the drift diffusion model (White et al., 2010) rather than the reduced drift rate seen in anhedonia (although see Aylward, Hales, Robinson, & Robinson, 2017, for potential evidence of drift rate changes in anxiety disorders).

In sum, research into the computational parameters of anxiety, while in their infancy, reveals effects that might differ in important ways from those in anhedonia, and understanding how these interact in the clinical manifestation of mixed anxiety and depression is a key unresolved question.

What Are the Clinical Implications?

The onset and causes of mood disorders remain puzzling (Kendler & Halberstadt, 2013). The efficacy of our dominant treatment strategies—pharmacotherapy and psychological therapy (NICE, 2014)—are unpredictable. The increasing empirical focus in the mood disorders literature on reinforcement processes has been accompanied by increasing acknowledgment of the antidepressant potential of dopaminergic agents (e.g., Fawcett et al., 2016; Racagni, Canonico, Ravizza, Pani, & Amore, 2004), which may act by influencing RL processes. As such, understanding individual differences in reward pathways, for example, using RL paradigms, may offer an effective way to stratify patients and predict treatment response. For example, SSRIs may show reduced efficacy for individuals with high levels of anhedonia and, hence, deficits in reward-related behavior (Vrieze et al., 2012), while there may be learning signatures that indicate that an individual will respond well to psychological intervention (see, e.g., Culver, Vervliet, & Craske, 2015). Future work integrating computational approaches with clinical trials may ultimately improve our ability to target treatments.