Using Drift Diffusion and RL Models to Disentangle Effects of Depression On Decision-Making vs. Learning in the Probabilistic Reward Task

Daniel G. Dillon; Emily L. Belleau; Julianne Origlio; Madison McKee; Aava Jahan; Ashley Meyer; Min Kang Souther; Devon Brunner; Manuel Kuhn; Yuen Siang Ang; Cristina Cusin; Maurizio Fava; Diego A. Pizzagalli

Research on the impact of depression on reward processing and reinforcement learning (RL) has flourished (; ; ) with the development of reward tasks that are sensitive to depressive illness (), improved understanding of brain reward systems (), and increased use of computational models (; ; ). The current study addresses three topics important to the continued success of this work: the complexity of behavior, psychometrics, and the predictive power of reward paradigms. We investigated these topics using the probabilistic reward task (PRT; ). The PRT is commonly used to assess RL in depression, but recent data show that it can also provide insight into decision-making (; ; ).

Behavioral Complexity

Behavior in RL tasks can be surprisingly complex. In the PRT (Figure 1A), which is rooted in the signal detection framework (), participants must distinguish between two stimuli (schematic mouths) of similar length, and correct identifications of one (rich) stimulus are rewarded three times more often than correct identifications of the other (lean) stimulus. Consequently, participants typically develop a response bias: they respond “rich” more than “lean” (note: we use quotes when referring to “rich” and “lean” responses, but omit them when referring to rich and lean stimuli). Weak response bias may be a marker of anhedonia () and many data are consistent with this proposal (; ,; ). Moreover, research has probed neural mechanisms that support response biases (; ; ). This body of work is sufficiently extensive that the PRT appears in the NIH RDoC Matrix as a validated test of reward learning. Nevertheless, in a recent study comparing unmedicated adults with Major Depressive Disorder (MDD) to healthy controls, we identified two aspects of PRT performance that were previously unrecognized ().

Figure 1

The (A) Probabilistic Reward Task (PRT) and (B) Drift Diffusion Model.

Note. (A) PRT trial stucture. (B) Blue and red lines show the drift process reaching the rich and lean boundaries, respectively. The HDDM, as setup in this study, returns one average drift rate per participant.

First, we found that the response bias effect was not uniform: it was strong when response times (RTs) were fast (≤ 0.3 quantile), but weaker when RTs were slower. This indicates that the mechanism(s) underlying response bias likely involve preparation of fast motor actions rather than, for example, sustained stimulus evaluation (). Second, reward totals were better predicted by discriminability than by response bias. In other words, the participants who earned the most rewards were those who responded most accurately, not those who developed the strongest tendency to respond “rich”. Given the emphasis on response bias in the PRT literature, this finding was striking. Moreover, it had a consequence: although no group difference in response bias emerged, controls earned significantly more rewards than depressed adults due to better discriminability. This indicates that weak response bias may not necessarily indicate anhedonia: by responding accurately, a participant with excellent discriminability could form no response bias and yet harvest more rewards, more quickly, than a participant with a stronger response bias but poorer discriminability. We replicated these results in a study of social anxiety disorder (SAD; ), although here the socially anxious adults showed better discriminability (and earned more rewards) than healthy controls, especially after gaze training () for their anxiety. In the current study, we attempted to replicate our findings in a new depressed sample.

Models and Psychometrics

The results just described suggest that there are multiple ways to model the PRT, with implications for how to conceptualize task performance and MDD pathophysiology. Past research () successfully accounted for PRT data using an RL model, called the “belief” model, that envisions participants updating the estimated value of the rich and lean stimuli by computing trial-level prediction errors (PEs). This is sensible given the emphasis on rewards in the PRT, strong relationships between PEs and dopamine (), and the long-standing hypothesis that depression involves dopamine dysfunction (; ; ). The belief model has parameters for reward sensitivity and learning rate, two important contributors to RL, and a meta-analysis found relationships between MDD, anhedonia, and reward responsivity (). In short, focusing on RL has been productive.

In the PRT, however, the reward probabilities for correctly identifying the rich and lean stimuli are fixed and reward magnitude does not change. Consequently, trial-level value updates may not be critical. Furthermore, the reward probability gap across the response options is so large that precise value estimates may not be necessary—a simple policy (press “rich”) based on rough approximations may suffice (). Finally, participants are asked to make a difficult perceptual decision on each trial. This is unusual in the RL space, but tasks with similar attributes have been successfully analyzed with sequential sampling models, including the drift diffusion model (DDM; ; ), for decades (; ; ). Inspired by these examples, in our prior study of MDD () we analyzed the PRT data with the Hierarchical Drift Diffusion Model (HDDM; ).

The DDM conceptualizes decision-making as a process of evidence accumulation to thresholds (Figure 1B). When fit to the data in Lawlor et al. (), it captured the novel results mentioned earlier. Specifically, the DDM modeled the dependency between response bias and RT by moving the starting point of the diffusion process closer to the rich boundary, such that relatively little evidence needed to accumulate to elicit a “rich” response. Poorer discriminability in the MDD group was accounted for by reducing the drift rate, which captures the speed of evidence accumulation. Drift rate strongly predicted discriminability, supporting the conclusion that slow drift rate in MDD led to poor discriminability, which led to low reward totals. In short, the DDM—although not designed to address RL—provided an excellent account of the data. The relationships among these parameters replicated in our study of SAD (). Attempting to replicate these relationships again, and to confirm that drift rate is slow in adults with MDD, were aims of the current analysis.

Therefore, we fit our data with the HDDM and the RL models in Huys et al (), including the belief model. We used simulations to assess each model’s ability to capture behavior, and we investigated two psychometric properties: internal consistency and retest reliability. Internal consistency indexes a variable’s within-session stability and serves as a floor for the strength of all other relationships—a measure cannot be more strongly related to another measure than it is to itself (; ). We also assessed retest reliability. In this study, data were collected twice about three weeks apart (but see below for the limitation of an intervening manipulation), allowing us to examine the reproducibility of parameter estimates. In our prior work (; ), HDDM parameters estimated from PRT data showed good to excellent internal consistency. Our data in SAD (), and larger datasets from different tasks (; ; ), also indicate that the retest reliability of DDM parameters is adequate to good. By contrast, the psychometrics of the belief model have not been studied. Examining psychometrics is important for precision medicine: measures with high internal consistency are needed to predict outcomes for individuals. As described next, predicting treatment responses was the goal of the larger study for which the current data were collected.

Predicting Placebo Responses

The data described here come from a project examining whether reward tasks can predict placebo responses. Such predictions could facilitate placebo responding in clinical settings (to reap therapeutic benefits) while removing placebo responders from drug trials (to more easily detect drug effects). The primary hypothesis was that individuals with better reward system function would be more likely to show placebo responses (; ).

To test this account, adults with MDD completed a positron emission tomography (PET)/magnetic resonance imaging (MRI) scan to assess mesolimbic dopaminergic reward system activity before beginning a clinical trial in which they were initially randomized to placebo or to the atypical antidepressant bupropion, which is a dopamine/norepinephrine reuptake inhibitor (); most participants who received placebo and did not respond were switched to bupropion (see Methods for details; see Figure S1 for a summary). About three weeks later, participants completed the tasks a second time. Details of the trial and the PET/MRI data will be presented elsewhere. Here we focus on PRT data, collected from a subset of participants after the PET/MRI scans. We used the data to address two questions. First, do pre-treatment model parameters or standard PRT metrics (response bias, discriminability) predict placebo responses? Second, do pre-treatment scores on these variables predict responses to bupropion in placebo non-responders? As detailed below, low statistical power limited our ability to answer these questions. Nonetheless, the data may provide leads for future investigations.

Summary

This PRT analysis had four aims: (a) replicate the dependency between response bias and RT, and the finding that reward totals are better predicted by discriminability vs. response bias; (b) fit the data with the HDDM and the belief model, with the expectation that these models will provide insight into mechanisms underlying discriminability and response bias, and will uncover slow drift rates in this depressed sample; (c) examine internal consistency and retest reliability of model parameters; and (d) determine whether response bias, discriminability, or any model parameter can predict responses to placebo or responses to bupropion in placebo non-responders.

Methods

Data and code are available at the Open Science Framework (OSF) at https://osf.io/347rm. The study was not pre-registered.

Participants and Protocol

This research was performed following a protocol approved by the Mass General Brigham Institutional Review Board (“Neurobiological underpinnings of placebo response in depression”, #2014P000889), and written informed consent was obtained. Prospective participants were recruited by the Depression and Clinical Research Program at Massachusetts General Hospital (MGH). Inclusion criteria included: (a) meeting Diagnostic and Statistical Manual IV (DSM-IV; ) criteria for MDD; (b) 18–45 years old; (c) score of 17 or greater on the Hamilton Depression Rating Scale-31 (HAMD-31; ); (d) continuing to meet criteria for current MDD and Clinical Global Impression (CGI; ) improvement scores ≤ 3 in the interval between the screen and Session 1 visit; and (e) no failed antidepressant trials of adequate dose and duration, as defined by the MGH Antidepressant Treatment History Questionnaire (ATHQ; ). Exclusion criteria minimized comorbidity and eliminated risks associated with neuroimaging; see the Supplement for details.

Eligible individuals were enrolled in a clinical trial. The trial and clinical data will be detailed elsewhere; because the current manuscript is focused on the PRT, a summary is given. Briefly, the trial used a sequential parallel comparison design () to increase the number of placebo responders. Participants were randomized to placebo or 300 mg bupropion in a 7:1 ratio (87.5% placebo, 12.5% bupropion) and were told that bupropion is a “fast-acting antidepressant” because expectations of success can drive placebo responses () and affect reward circuitry (). After four weeks, participants randomized to bupropion stayed on bupropion, whereas placebo responders and non-responders were re-randomized to placebo or bupropion in a 1:7 ratio (12.5% placebo vs. 87.5% bupropion, computed separately for placebo responders and non-responders). Depression was assessed throughout with the HAMD-31. Responder status was assessed four weeks post-baseline and defined as a 50% or greater reduction in HAMD-31 score; drop outs were considered non-responders.

PRT data were collected before randomization and again three weeks later; these timepoints are referred to as Sessions 1 and 2. Because placebo responses often develop in about two weeks (; ), scheduling Session 2 three weeks after Session 1 should assess those responses. Note that while treatment continued after Session 2, we call this session “post-treatment” because it occurred after randomization.

PRT

PRT methods followed prior reports (). Participants sat at a PC with their index fingers on the “v” and “m” keys of a keyboard and completed three blocks of 100 trials. As shown in Figure 1A, trials began with presentation of a schematic face (500 ms) onto which a “short” (11.5 mm) or “long” (13.0 mm) mouth was flashed (100 ms). The task was to indicate, as quickly as possible, which length was shown. Most trials yielded no feedback, but 20 cent rewards were delivered three times more often for correct identifications of one mouth (the rich stimulus, up to 30 rewards) vs. the other (the lean stimulus, up to 10 rewards). Assignment of lengths to rich/lean conditions was counterbalanced across participants, and the keys used for “rich” vs. “lean” responses were counterbalanced across sessions. Participants were paid $15.80 or $16.20.

Quality Control

A prior quality control assessments were used to exclude poor quality datasets; these and all subsequent analyses were conducted using Python in Jupyter notebooks (). Outliers were defined as trials with raw RTs faster than 150 ms or slower than 2,500 ms, or where the ln(RT) exceeded the participant’s mean ln(RT) ± 3S.D., computed separately for rich vs. lean trials. Blocks were QC failures if there were: fewer than 80 non-outlier trials; a rich/lean reward ratio < 2.0; fewer than 20 rewarded rich trials; or fewer than 6 rewarded lean trials. If any blocks was marked as a QC failure, the dataset was excluded.

Signal Detection Analyses

Response bias and discriminability were computed by counting the number of correct/incorrect rich and lean responses per block and entering them in these formulas ():

r e s p o n s e b i a s = 0.5 * l o g 10 (r i c h c o r r e c t * l e a n i n c o r r e c t r i c h i n c o r r e c t * l e a n c o r r e c t)

M1 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ response\;bias = 0.5*lo{g_{10}}(\frac{{ric{h_{correct}}*lea{n_{incorrect}}}}{{ric{h_{incorrect}}*lea{n_{correct}}}}) \] \end{document}

d i s c r i m i n a b i l i t y = 0.5 * l o g 10 (r i c h c o r r e c t * l e a n c o r r e c t r i c h i n c o r r e c t * l e a n i n c o r r e c t)

M2 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ discriminability = 0.5*lo{g_{10}}(\frac{{ric{h_{correct}}*lea{n_{correct}}}}{{ric{h_{incorrect}}*lea{n_{incorrect}}}}) \] \end{document}

Counts were initialized at 0.5 to avoid division by zero ().

Relationships Among Behavioral Variables

We studied relationships among PRT variables to better understand how the task is performed, to identify regularities that might be captured by the models, and to replicate prior findings (; ). First, quantile-probability plots were used to examine the RT distribution. Based on earlier work, we expected a strong right skew in our depressed sample. In other words, we anticipated that many trials would be characterized by slow RTs, leading to a long right tail when the RT distribution is plotted horizontally. This is important because the shape of the RT distribution yields predictions for the HDDM: a strongly right-skewed distribution is often well-fit by a slow drift-rate (). Second, the quantile-probability plots were used to determine whether the rich > lean accuracy effect, which captures response bias, varied by RT. We expected the rich > lean accuracy difference to be larger for faster vs. slower RTs. Finally, we again expected reward totals to be more strongly related to discriminability than response bias, because accurate responding allows participants to efficiently collect rewards on rich and lean trials.

HDDM

We fit the HDDM (version 0.7.5) to trial-level choice and accuracy data, separately by session. We retained the default priors () and modeled the data using the HDDM’s StimulusCoding tool. The model was the same as in prior studies (; ):

m = hddm.StimCoding(data, include = ‘z’, stim_col = ‘stim’, split_param = ‘v’)

This approach estimates the starting point bias (z) and returns an absolute value for drift rate (v), which is coded –v and +v for trials in which the lower (0 = “lean” response) vs. upper (1 = “rich” response) boundary is reached.

Models were estimated by drawing three chains of 2,500 samples from the posterior (burn-in: 500 samples). Chains were concatenated and convergence was assessed by inspecting posterior distributions and by checking that the maximum $R^$ M4 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \hat R \] \end{document} did not exceed 1.1 () (max $R^$ M5 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \hat R \] \end{document} : Session 1 = 1.003; Session 2 = 1.016). Next, the HDDM’s post_pred_gen tool was used to generate 500 simulated datasets per participant, and post_pred_stats was used to check if simulated accuracy and RT matched the observed data.

Relationship with standard metrics

To facilitate interpretation, we regressed each participant’s (across-block) mean response bias and discriminability against the HDDM parameters. We expected response bias to be most strongly related to starting point bias and discriminability to be most strongly related to drift rate and threshold (because response accuracy is facilitated by rapid evidence accumulation and wide boundaries, which maximize the likelihood of the drift process reaching the correct boundary).

RLDDM

The data were also fit with the Reinforcement Learning Drift Diffusion Model (RLDDM; ). Briefly, the RLDDM uses trial-level PEs to assign values to (stimulus, action) pairs—e.g., “rich” responses to rich stimuli—in a Q-learning framework, but it uses the DDM (rather than the softmax function) as the choice rule: here, drift rate is scaled on each trial by the difference in Q-values for the two response options. It was unclear if the combination of RL plus the DDM would improve on the HDDM fits. To investigate this, we used Deviance Information Criterion (DIC) values and posterior predictive checks to compare RLDDM vs. HDDM fits.

RL Models

The data were next fit with RL models successfully used in prior PRT research (; code for all models at https://github.com/mpc-ucl/emfit). Group priors were computed via expectation-maximization () and Laplace approximation of posterior distributions was used to compute subject-specific parameters. To compare model fit, integrated group-level Bayesian Information Criterion (iBIC) was used. Consistent with prior findings (), the most parsimonious account of the data was provided by the belief model, so for brevity we do not discuss the other RL models. The belief model postulates that rewards serve to inform two uncertainty-weighted stimulus-action associations (since participants may be unsure about which stimulus was presented in a given trial). The model has five parameters: reward sensitivity, learning rate, belief, instruction sensitivity, and initial action bias.

A detailed treatment of the belief model is in Huys et al. (), but here we provide an overview. Briefly, this is a Q-learning model in which prediction errors (PEs)—the discrepancy between expected and actual rewards (0 = no reward, 1 = reward)—serve to update Q-value estimates for each action/stimulus pair (e.g., responding “rich” to the rich stimulus). Reward sensitivity scales reward impact: when it is larger, rewards have greater impact, and when it is smaller they are devalued. This allows for the possibility that rewards are “liked” more or less by different participants. Learning rate is also a scaling factor—it controls the speed with which PEs affect Q-values. When learning rate is higher, Q-values change more rapidly from trial-to-trial, whereas when it is lower, the Q-values change more gradually. In the belief model, participants estimate four Q-values, one each for the 2 StimulusType × 2 ResponseType design. The similarity of stimuli, however, makes it hard for participants to know which stimulus was shown on any trial. The belief parameter captures this uncertainty. Instruction sensitivity tracks accuracy: it covaries with participants’ ability to correctly identify the rich vs. lean stimulus. Finally, action bias corresponds to a preference for one of the two responses. To avoid non-Gaussianity, parameters were transformed. To conduct posterior predictive checks, we used the best-fitting parameters to simulate 500 datasets per particpant, as done with the HDDM.

Relationship with standard metrics

We regressed response bias and discriminability against the belief model’s five parameters.

Psychometrics

To examine internal consistency, the data were split into odd and even trials and submitted to signal detection, HDDM, and belief model analyses. The belief model involves value updates driven by PEs on sequential trials, which might make this approach suboptimal. Therefore, we also split each participant’s data in half and examined the consistency of analyses conducted on each half. The Spearman-Brown (SB) formula (; ) was calculated to measure internal consistency:

S B = 2 * r 1 + r

M3 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ SB = \frac{{2*r}}{{1 + r}} \] \end{document}

Here, r is the Pearson correlation coefficient for odd vs. even trials, or for the first vs. second half of trials.

To examine test-retest reliability, Pearson correlations comparing Sessions 1 and 2 were computed for mean (across-block) response bias and discriminability values, and for all model parameters; the mean ± SD days between sessions was 24 ± 5. Recall that all participants received either placebo or bupropion between Sessions 1 and 2. Consequently, estimates of test-retest reliability should be interpreted cautiously as they may be affected by treatment.

Predicting Responses to Placebo and Bupropion

To examine whether pre-treatment PRT data could predict placebo responses, we ran a 2 (PlaceboResponder: yes, no) × 3 (Block) ANOVA on Session 1 response bias and discriminability. These ANOVAs were repeated using Session 2 PRT data; here the expectation was that Session 2 response bias would be higher in placebo responders vs. non-responders. Next, the HDDM Session 1 and 2 analyses were re-run, this time allowing all parameters to vary by placebo response status. For the belief model, all five parameters were submitted to between-group t-tests (placebo responder vs. non-responder) for Sessions 1 and 2.

Finally, we identified placebo non-responders re-randomized to bupropion. Session 1 and 2 data from these participants were analyzed in 2 (BupropionResponder: yes, no) × 3 (Block) ANOVAs on response bias and discriminability. The HDDM was re-estimated allowing all parameters to vary as a function of bupropion response status, and between-groups t-tests (bupropion responders vs. non-responders) were computed for all belief model parameters.

Statistics

Statistical tests include linear regressions, ANOVAs, pairwise comparisons, and t-tests conducted in R, version 4.0.5 “Shake and Throw” (). Packages included afex for ANOVAs (), emmeans for pairwise comparisons (), and cocor for comparing correlations ().

Results

Sample Characteristics and Responder Status

Session 1 data were collected from 59 participants;10 QC failures were discarded, leaving 49 datasets. Session 2 data were collected from 57 participants; 13 QC failures were discarded, leaving 44 datasets. The two most common problems were losing too many trials to outlier RTs (30% of failures in Session 1, 22% in Session 2) and having too few usable rich trials in which a reward was delivered (33% of failures in Session 1, 41% in Session 2). The elevated rate of QC failures may reflect fatigue as the PRT was completed after the PET/fMRI session.

The samples were predominantly White and not Hispanic or Latino, with a roughly equal mix of males and females and a mean education level equivalent to some years of college (see Table S1). Due to scheduling issues, 34 participants provided usable data from Sessions 1 and 2. Consequently, we provide pre- and post-treatment HAMD scores separately for Session 1 and 2, since these two samples were not entirely overlapping.

Of 49 Session 1 participants, 43 were randomized to placebo: 11 responded, 32 did not. There was no difference in baseline HAMD (mean ± S.D., responders: 29.82 ± 6.08; non-responders: 32.41 ± 6.84), but four weeks later scores were lower in responders (10.45 ± 4.99) vs. non-responders (26.30 ± 7.03), t(39) = 6.84, p < 0.001. The remaining 6 participants were randomized to bupropion: 3 responded, 3 did not. HAMD scores were similar at baseline (responders: 26.67 ± 10.02; non-responders: 40.67 ± 6.66), but differed four weeks later (responders: 7.00 ± 7.94; non-responders: 37.33 ± 10.21); no test conducted given small samples.

Of the 44 Session 2 participants, 40 had been randomized to placebo: there were 14 responders vs. 26 non-responders (note: there were more placebo responders at Session 2 vs. 1 because not all responders completed the Session 1 PRT, and due to Session 1 QC failures). There was no difference in baseline HAMD scores for placebo responders (29.07 ± 7.60) vs. non-responders (31.19 ± 6.71), but after four weeks scores were lower in responders (10.29 ± 5.43) vs. non-responders (26.35 ± 6.89), t(38) = 7.54, p < 0.001. The remaining four participants were randomized to bupropion; three responded. In this small group, baseline HAMD scores were similar (responders: 26.00 ± 9.00; non-responder: 33) but four weeks later scores were lower in responders (6.00 ± 6.24); the non-responder’s score (33) did not change.

Finally, Session 1 data were collected from 26 placebo non-responders who were re-randomized to bupropion and subsequently classified as responders (n = 12) or non-responders (n = 14). HAMD scores did not differ at baseline (responders: 30.25 ± 6.37; non-responders: 34.14 ± 6.59), but they differed at the final clinical session (4 weeks after Session 2: responders: 9.67 ± 4.58; non-responders: 25.82 ± 7.64; t(21) = 6.21, p < 0.001). Session 2 data were collected from 23 placebo non-responders who were re-randomized to bupropion and subsequently classified as responders (n = 8) or non-responders (n = 15). Baseline HAMD scores did not differ (responders: 30.25 ± 6.50; non-responders: 31.53 ± 7.38), but HAMD scores were lower in responders (9.88 ± 3.18) vs. non-responders (25.54 ± 7.42) at the final sesssion t(19) = 5.62, p < 0.001.