Inclusion and exclusion criteria

All participants were between the ages of 18 and 55, fluent in English with no history of seizure, no neurological impairment or learning disorder, no current use of illicit substances, no history of head trauma resulting in loss of consciousness for over 5 minutes, and were not seeking or receiving treatment for AUD. Hazardous drinking participants had an Alcohol Use Disorders Identification Test (AUDIT) (Bush et al., 1998; Saunders et al., 1993) score >7 for men and >6 for women. Control participants had to have an AUDIT score <4 prior to participation during a phone screening and at the time of the experiment. Hazardous drinkers were recruited from ABQDrinQ (R01AA023665), a separate study conducted by investigators from the UNM Center on Alcohol, Substance use, and Addictions (CASAA) and the Mind Research Network (MRN) to assess neurocognitive patterns associated with changes in the drinking behavior of non-treatment seeking individuals with AUD (Al-Khalil et al., 2021).

Participants

30 participants (17 female) were recruited from ABQDrinQ for the hazardous drinking sample. These participants were invited back to the present study 18–32 months later and they were re-assessed with the AUDIT. 28 controls (16 female) were recruited from other studies and the community. Five participants were excluded after data collection (one control participant had a high AUDIT score on test day, two had poor EEG data quality, and two were outliers (SD ≥ 3) on our outcome measures). The final sample comprised n = 27 (15 female) in the hazardous drinking group and n = 26 (15 female) controls. Table S1 includes descriptive statistics for demographic and questionnaire data. There were no significant differences between groups for sex (p = 0.874), tobacco/nicotine use (p = 0.638), ethnicity (p = 0.933), race (p = 0.231), age (p = 0.741), or years of education (p = 0.151). The groups did significantly differ for AUDIT (p < .001) and Beck Depression Inventory (BDI) scores (Beck et al., 1988) (p = 0.023), with BDI differences being expected given the greater severity of depressive symptoms reported in hazardous drinkers (Lannoy et al., 2021). Additionally, within the hazardous drinking group, BDI and AUDIT scores were correlated (rho(25) = 0.445, p = 0.018).

Procedure

Participants completed paper questionnaires including the AUDIT and BDI followed by a series of computerized tasks including the three-armed bandit task with alcohol and non-alcohol beverage stimuli (Figure 1a). The task consisted of 350 trials in which participants chose between three images that were probabilistically associated with a reward. Each trial comprised i) a jittered duration fixation cross (600–800 ms), ii) a choice event (3 targets presented in the upper, left, or right area pseudo-randomly assigned; participants had 1500 ms to select one of the images by pressing buttons corresponding to their locations), iii) a jittered duration blank screen (100–300 ms), and iv) a feedback event (+1 or ~ ; 1000 ms; Figure 1a). If the participant failed to respond in time, a null signal (“No Response Detected”) was displayed (≈2.97% of trials per participant). During the experiment, 50 images (split evenly between alcohol and non-alcohol stimuli) were introduced which randomly replaced one of the existing options with a minimum of 5 trials and a maximum of 9 trials between novel insertions (# trials between novel insertions: M = 6.86, SD = 1.63). At the start of the experiment, the three initial choices were randomly assigned a reward probability of 0.2, 0.5, or 0.8. Novel choice options were also randomly assigned one of these reward probabilities when introduced. No more than two of the three options could be assigned the same reward probability at a time. Participants were instructed to win as many points as possible by choosing the image that rewarded them most often. They were also told about the probabilistic nature of the rewards and that the images’ positions did not affect the probability of receiving a reward.

Beverage Image Set and Participant Ratings

An image rating task was administered to obtain ratings of liking of various image categories. Participants were presented with an image from one of five categories: puppies, scenes of nature, babies, alcohol, neutral (e.g., a chair), and negative images (e.g., garbage). Pictures were selected from web image searches (e.g., “high-definition puppy images”). Six images from each category were presented (a total of 36 images). This task was previously reported (Brown et al., 2022), but here alcohol images substituted cow images. Participants rated how pleasant they felt the image was on a scale from 1 (very unpleasant) to 9 (very pleasant). Both controls and hazardous drinkers rated all classes of affective imagery similarly except for the alcohol images, which were rated more pleasant in hazardous drinkers (p < .001). Within hazardous drinkers, AUDIT score correlated significantly with pleasantness ratings for alcohol images (rho(25) = 0.34, p = 0.041). Therefore, the chosen image set demonstrates face validity for the salience of alcohol images to hazardous drinkers.

Computational Modeling

Optimal explore-exploit decision-making was modeled based on estimates of state and action values derived from a partially observable Markov decision process (POMDP) (Averbeck, 2015). This allows estimation of the utility of choosing particular options (i.e., actions) at particular timepoints (i.e., states). The utility of each information state changes trial-by-trial and varies as a function of the action that maximizes total action value. The action values in each state are the sum of the immediate (IEV) and future (FEV) expected values of choosing a particular option. IEV is an estimate of the likelihood that a given option will be rewarded based on prior outcomes, while FEV reflects the discounted future gains or losses that can be expected given what is learned after choosing a particular option. The IEV is easy to compute, based on explicit positive or negative feedback about prior decisions. In contrast, the FEV is more difficult to compute and involves simulating the number of future gains or losses that might result after choosing a particular option. Notably, because subjects could choose freely among the three-arms in the bandit task, the FEV of a given option is highly tied to the overall richness of the current choice set. FEV is high if the best available option(s) presented have a high versus a low IEV. The state space and formalism of the POMDP has been described in depth elsewhere (Averbeck, 2015; Costa et al., 2019; Hogeveen et al., 2022), with the only difference here being that the weighted utility functions were computed separately for alcohol and non-alcohol stimuli.

The FEV of one option compared to the average FEV across all options is crucial to determining the relative increase or decrease in future value associated with exploration. We refer to this relative FEV quantity as the exploration BONUS. Immediately after a novel option is added to the choice set, uncertainty about its IEV is high, and the BONUS associated with choosing that option relative to familiar alternatives is at its highest. This BONUS value then decreases over trials as the subject samples the novel option and learns its IEV. In contrast, the BONUS values associated with familiar alternative options are negative when a novel option is introduced. This occurs because the subject has already sampled those options, lowering their FEV relative to the novel stimulus. Notably, as the subject forgoes these familiar alternatives to explore the novel stimulus, their FEV and BONUS values steadily increase over trials.

In sum, within this POMDP framework BONUS values above 0 are associated with exploratory choices (i.e., driven by small relative differences in FEV of a given option), whereas BONUS values below 0 are associated with exploitative choices. To be clear, we do not think participants mentally simulate all possible choice sequences to estimate exploration bonuses prior to each choice. But given the fact that the POMDP model-derived estimates fit well with observed choice behavior on the current task across nonhuman primates and human subjects (Costa & Averbeck, 2020; Costa et al., 2019; Hogeveen et al., 2022)—this modeling approach does capture meaningful variance in the way individuals utilize uncertainty about the value of new information to motivate the decision to explore novel choice options.

EEG Data Preprocessing and Analysis

EEG was recorded on a 64-channel Brain Vision system (Brain Products GmbH, Munich, Germany) between .01–100 Hz at a sampling rate of 500 Hz. CPz was used as the reference electrode and FPz as the ground electrode. Electrocardiogram (EKG) was recorded as was vertical electrooculogram (VEOG) via two auxiliary electrodes placed above and below the left pupil. All EEG preprocessing was executed using custom Matlab scripts and EEGLAB (Delorme & Makeig, 2004) functions. Data were band-pass filtered between 0.1–20 Hz and independent components analysis (ICA) was used to remove eyeblinks. Data were average referenced and poor-quality electrodes and epochs were removed after visually inspecting data. EEG data were epoched at –500 to 1000 ms relative to stimulus presentation, baseline corrected, and then averaged to generate ERPs.

ERP regions of interest were selected from electrodes which were closest to maximal activation along the midline during and one trial after novel insertions. The P3a is elicited by novel distracting stimuli (Donchin & Coles, 1988; Pfefferbaum et al., 1979) and was expected to be enhanced due to novel insertions. The P3b is elicited by cues with enhanced motivational significance (Cavanagh, 2015; Nieuwenhuis et al., 2005) such as cues with higher reward probabilities. P3a was quantified at electrode FCz and P3b was quantified at electrode POz. These distinct ERP components were calculated by taking the average across time windows of peak activation between 350–530 ms for P3a and 230–430 ms for P3b (Figure 1b; Polich, 2007).

Modeling Approach and Outlier Removal

To test hypotheses about the neural substrates of explore-exploit behavior in hazardous drinkers, data were modeled as a function of both (1) choices following novel options, and (2) computational parameters reflecting decision-making behavior. Specifically, individual differences in subjects’ explore-exploit decision-making tendencies based on latent value parameters, IEV and exploration BONUS, from the POMDP model associated with alcohol and non-alcohol choice options. Linear mixed effects models were computed with group and cue type as fixed effects and participant as a random effect. ERP trial outliers were removed when they exceeded a Mahalanobis’ distance of 13.82 (for 2 predictors at alpha = 0.001). Estimated marginal means with two test false discovery rate correction were used for paired comparisons of conditions between groups.

Evaluating Assumptions of the General Linear Model

Across both the alcohol and non-alcohol novel stimulus conditions, our EEG components of interest (P3a, P3b) and key computational model-derived parameters of interest (BONUS, IEV) violated the assumption of normality (ps ≤ 0.023). Additionally, the probability of exploring a novel stimulus, EEG components of interest, and BONUS parameters all violated the assumption of homoscedastic variances across both groups (all ps ≤ 0.044). Due to these violations of the assumptions of the general linear model, we used robust variants of all inferential tests where stimulus condition or participant group were independent predictors (Field & Wilcox, 2017). Specifically, we computed robust linear mixed effects models that down-weighted observations with excessively high residual values to reduce their impact on model estimates (Koller, 2016). Discrete explore-exploit behaviors were measured via participant’s selection of the novel option (i.e., exploration), the best alternative (i.e., exploiting the familiar option with highest reward probability), or the worst alternative (i.e., familiar option with lowest reward probability) on the first trial after a novel insertion (Hogeveen et al., 2022). In addition to discrete explore-exploit behaviors on the early post novel insertion trial, subject-specific POMDP coefficients enabled us to model continuous variation in the value of exploration (i.e., BONUS) and exploitation (i.e., IEV) for alcohol or non-alcohol stimuli across all trials.

Increased Exploration of Novel Alcohol Cues in Hazardous Drinkers

Periodic novel stimulus insertions explicitly forced participants to make explore-exploit choices. Participants did not tend to explore on the insertion trial and instead waited until the next trial (post-insertion) to sample new stimuli (Figure 2a). To directly quantify explore-exploit behavior, we computed the choice probability of selecting the novel stimulus (exploration) versus the best available alternative (exploitation) on the first post-insertion trial. Participants tended to exploit more often than they explored (Figure 2a; MD = 0.063; t(52) = 2.73, p = 0.008), but in turn they also explored more often than they chose the worst stimulus (Figure 2a; MD = 0.064; t(52) = 3.78, p < .001).

To evaluate how participants responded to the presentation of each novel cue type, we subtracted exploit choice probability from explore choice probability. To test the effect of cue type on explore-exploit behavior between groups, we ran a model predicting choice behavior from group and cue type (b = –0.416, SE = 0.083, p < .001) and found that hazardous drinkers explored alcohol stimuli more often than controls (Figure 2b; b = –0.208, SE = 0.079, p = 0.009). Controls also explored non-alcohol stimuli more often than hazardous drinkers (Figure 2b; b = 0.208, SE = 0.079, p = 0.009), demonstrating alcohol cue-specific exploration in hazardous drinkers relative to controls.

Increased Exploration BONUS for Alcohol Cues in Hazardous Drinkers

Within the POMDP framework, immediate expected value (IEV) is an estimate of the likelihood that a given option will be rewarded based on prior outcomes. When a novel option is introduced, the uncertainty about its IEV is high, and the exploration BONUS associated with choosing the novel option vs. familiar alternatives is at its highest. POMDP model-derived estimates for all choices correlated with choice behavior for each participant (M = 0.737, SD = 0.267, 95% CI = 0.6–0.84) and this correlation did not differ between controls and hazardous drinkers (t = 0.308, p = 0.758). Subject-level POMDP estimated coefficients reflected each individual’s degree of reliance on the exploration BONUS and IEV signals in shaping their decision-making separately for alcohol (rho(51) = –0.192, p = 0.167) and non-alcohol cues (rho(51) = –0.298, p = 0.031). To test whether POMDP parameters related to group and cue type we ran two models, one predicting BONUS and one predicting IEV from group and cue type. We found that BONUS values were predicted by a two-way interaction between group and cue type (b = –0.126, SE = 0.059, p = 0.038). Hazardous drinkers had higher BONUS values for alcohol cues than did controls (Figure 2c; b = –0.15, SE = 0.07, p = 0.031). No other main effects were significant, and IEV coefficients did not differ as a function of group, cue type, or their interaction (Figure 2d). Building off the post-insertion trial behavioral results, these differences in the BONUS parameter demonstrate that hazardous drinkers continued alcohol-specific exploration across trials.

Increased P3a with Exploration of Alcohol Cues in Hazardous Drinkers

P3a and P3b amplitudes were calculated as the difference in the amplitude between the ERP elicited by the presentation of the three choice stimuli on the trial after a novel option was introduced (i.e. post-insertion trial) minus the ERP elicited by the immediately preceding trial (i.e. pre-insertion trial) (Figure 1b). Prior research indicates that novel minus non-novel difference waves can be useful for distinguishing clinical populations (Kappenman & Luck, 2016) and have been used to do so for assessing novelty-sensitive neural responses (Bertram et al., 2020). To test whether each P3 component related to cue-specific explore-exploit behavior between groups, we ran two models predicting choice behavior from group and cue type, one with P3a amplitude and one with P3b amplitude as additional predictors. We found that probability of selecting the novel stimulus was predicted by a three-way interaction between group, chosen cue type, and P3a (b = 0.155, SE = 0.068, p = 0.025). In hazardous drinkers, the decision to explore novel alcohol cues was associated with an increase in the P3a relative to controls (Figure 3; b = 0.133, SE = 0.05, p = 0.007). P3b amplitude was not influenced by explore-exploit tendencies toward alcohol or non-alcohol cues in controls or hazardous drinkers (b = –0.045, SE = 0.08, p = 0.575). In sum, hazardous drinkers were specifically characterized by a P3a marker of alcohol-specific exploratory behavior.

Increased P3a with Exploration BONUS for Alcohol Cues in Hazardous Drinkers

To test whether the P3a component related to cue-specific POMDP parameters between groups, we ran two models with group, cue type, and P3a amplitude as predictors for BONUS in one model and IEV in the other. We found that BONUS coefficients were predicted by a two-way interaction between chosen cue type and P3a amplitude (b = –0.116, SE = 0.039, p = 0.004). The BONUS and P3a had a stronger relationship for alcohol relative to non-alcohol stimuli across groups (b = 0.078, SE = 0.027, p = 0.004). Separated by group, this difference was only apparent in hazardous drinkers (Figure 4; b = 0.04, SE = 0.037, p = 0.003). This demonstrated that as with exploration, the P3a increases as the BONUS values for alcohol cues increase. IEV coefficients were predicted by a three-way interaction between group, chosen cue type, and P3a (b = –0.227, SE = 0.082, p = 0.007). Paired comparisons showed that this effect was driven primarily by greater attenuation of the P3a in response to familiar alcohol cues in hazardous drinkers (Figure 4; b = –0.179, SE = 0.06, p = 0.003). To test whether the P3b component related to cue-specific POMDP parameters between groups, we ran two models with group, cue type, and P3b amplitude as predictors for BONUS in one model and IEV in the other. IEV coefficients were predicted by a three-way interaction between group, chosen cue type, and P3b amplitude (b = 0.198, SE = 0.097, p = 0.045). P3b amplitude was associated with increased IEV of alcohol cues for hazardous drinkers rather than for controls (Figure 5; b = 0.171, SE = 0.074, p = 0.02). In sum, hazardous drinkers were specifically characterized by a P3b marker of alcohol-relevant exploitation and a P3a marker of alcohol-relevant exploration, identifying discrete neural systems underlying alcohol-related decision-making.