Experimental Observations Database

In patients suffering from schizophrenia, oscillatory deficits in general and ASSR deficits in particular have been extensively studied using electroencephalography (EEG) and magnetoencephalography (MEG; e.g., Brenner, Sporns, Lysaker, & O’Donnell, 2003; Hamm et al., 2015; Krishnan et al., 2009; Kwon et al., 1999; Light et al., 2006; Mulert, Kirsch, Pascual-Marqui, McCarley, & Spencer, 2011; O’Connell et al., 2015; Spencer, 2012; Spencer, Niznikiewicz, Nestor, Shenton, & McCarley, 2009; Spencer, Salisbury, Shenton, & McCarley, 2008; Vierling-Claassen et al., 2008; Zhang, Ma, Li, Yang, & Qin, 2016).

Neural oscillations have been hypothesized to subserve important functions in the brain and are critically involved in cognitive processes (see, e.g., the review of Başar, Başar-Eroglu, Karakaş, & Schürmann, 2001). Gamma oscillations, for example, have been demonstrated to underlie the formation of coherent percepts in different sensory modalities (e.g., Engel, Kreiter, König, & Singer, 1991; Jokeit & Makeig, 1994). Interestingly, abnormal power and synchrony in the gamma band have been found in schizophrenic patients in a number of different tasks and paradigms (e.g., Cho, Konecky, & Carter, 2006; Kwon et al., 1999; Spencer et al., 2003) and linked to the schizophrenic symptom profile (e.g., Gordon, Williams, Haig, Wright, & Meares, 2001; K.-H. Lee, Williams, Haig, & Gordon, 2003). Although deficits in the generation and maintenance of gamma oscillations are not a classical symptom of schizophrenia, given the importance of gamma oscillations in sensory processing and cognition and the link between deficits and symptoms, these biomarkers might reflect a characteristic trait of the disorder.

Here we focus on three of these studies, looking at entrainment deficits in the gamma and beta ranges. Kwon et al. (1999) used a click train paradigm to study ASSRs at 20, 30, and 40 Hz in schizophrenic patients using EEG and found a prominent reduction of power at the driving frequency for 40 Hz drive but no changes of power at the driving frequency for 30 Hz and 20 Hz. Although Figure 3 in Kwon et al. (1999) seems to show an increase of the subharmonic 20 Hz component for 40 Hz drive, no statistical comparison is presented in the article. Vierling-Claassen et al. (2008) reproduced this reduction of power at the driving frequency for 40 Hz drive using the same paradigm with MEG. Additionally, they found an increase in power at the driving frequency during 20 Hz drive and changes of power at certain harmonic/subharmonic frequencies, namely, an increase of power at 20 Hz for 40 Hz drive and a decrease of power at 40 Hz for 20 Hz drive. Krishnan et al. (2009) used a slightly different paradigm, which employed amplitude-modulated tones instead of click trains, and tested a wide range of driving frequencies from 5 to 50 Hz. They found reduction of power at the driving frequency in the gamma range (i.e., at 40, 45, and 50 Hz) and no changes at other frequencies. Furthermore, they did not find any changes of power at harmonic or subharmonic frequencies.

The experimental database is realized as a nested Python dictionary, with an entry for each study included (a dictionary is a special data structure in which you can access data or values by a key[word]; in a nested dictionary, the values themselves can be dictionaries). Each study entry consists of two entries (i.e., two value-key pairs), which describe the study observations, one in a quantitative way and the other in a qualitative way. We have included the qualitative description because often either computational models do not allow for a strict quantitative comparison with experimental data or publications of experimental studies do not provide enough detail on the results, and in these cases, only a qualitative comparison is possible.

Together with the database, ASSRUnit provides basic methods to query and visualize the content of the database. These methods include commands to retrieve all studies or observations in the database and a method to display an overview of the results for the whole database or for certain studies or observations. Finally, the metadata associated with each study (e.g., the number of participants, the modality, the patient group) can also be retrieved and displayed.

Prediction Database

The prediction database is also implemented as a nested Python dictionary. Similar to the experimental observation database, methods that retrieve and visualize the content of the database are included in ASSRUnit.

Models, Capabilities, Tests, and More

Use Case 1: Overview of the Experimental Literature

The first use case demonstrates how the experimental database can be used to get a comprehensive overview of the current experimental literature related to a neurophysiological or neurocognitive biomarker, in our case, ASSR deficits in patients suffering from schizophrenia. Figure 3 shows that with two simple commands, one can retrieve the names of all studies and all observations present in the database. These names will have to be used for all further queries of the database.

Figure 4 then shows how to get a complete overview of all observations of all studies in the database. As we can see in Figure 5, simply adding the parameter meta=true, to the command will additionally output the metadata associated with each study. This contains information on the subjects, modality, and so on. The overview command presents the data in a simple table and can be used to see which studies provided which observation and what the results were. However, as we can already see for our small demonstration database containing only three studies, a full overview is likely to become very large and therefore hard to grasp fully. By explicitly stating the studies and/or the observations in which one is interested, one can reduce the complexity of the table and get a clear and simple overview, as depicted in Figure 6. Note that in the examples, we have only used the qualitative description of the observations; the same functionality also applies to the quantitative descriptions. The functionality described here, along with more examples, can be explored in the accompanying Jupyter notebooks (https://github.com/ChristophMetzner/ASSRUnit/blob/CompPsychArticle/assrunit/Notebooks/Example_Experimental_Database.ipynb).

This simple querying functionality allows the user to get a quick, clean, and comprehensive overview of the experimental literature, to identify observations that are supported by many studies (see, in our case, the reduction of gamma power for stimulation at gamma frequency) but also to detect controversial findings. Furthermore, the display of the associated metadata allows for checking, for example, whether identified common observations extend over different modalities and postprocessing techniques and also whether controversial findings might be explained by differences in the experimental setup or other related aspects. In the future, it will also be possible to look at more than one database and compare the same observations across different patient groups to highlight commonalities and differences between disorders.

Use Case 2: Model Comparisons

While our first use case only exploited the experimental database, we now show the additional benefits of joining experimental and modeling data.

Simple model comparison

By creating tests based on the model capabilities and grouping them into test suites, we can easily compare models against experimental data and against each other. Figure 7 demonstrates how we can use the module to create two different models along with several tests, run the models to produce the data relevant for the tests, and then judge the model outputs against experimental data and display the results together. Note that in this context, we use the term model as the in silico instantiation of a theoretical/conceptual model. Two different models may share the same code but differ only in parameter values. Again, the functionality described here, along with more examples, can be explored in an accompanying Jupyter notebook (https://github.com/ChristophMetzner/ASSRUnit/blob/CompPsychArticle/assrunit/Notebooks/Example_Model_Comparison.ipynb).

Figure 7.  

Contrasting the results of comparing two models against experimental observations. First, the model instances are created and the parameters for the control network and the schizophrenia-like network are passed on together with a name. Then, appropriate tests are created and experimental observations are passed on. In this particular example, the observation is passed on as a “ratio,” which means that the value of the output of the schizophrenia-like simulation is divided by the value of the output of the control simulation. Afterward, the tests are grouped together to form a test suite, and the two example models are run against the test suite. The results of this run are stored in the matrix score_matrix, and by evoking the view method of the SciUnit score matrix, a comparison table is shown displaying the performance of each model against each test. Note that in this example, the two models and their resulting performance are purely hypothetical and do not reflect any actual model, and furthermore, the experimental observations do not reflect any actual findings.

Advanced modeling data and visualization

As already described in the Models subsection of Models, Capabilities, Tests, and More above, there is a second version of each model class that contains not only the standard methods that implement the necessary capabilities but also so-called …_plus methods, which can generate model data for different trials and/or subjects, depending on the type of model. Together with the methods from the visualization class, this additional model data can be used to better understand the model behavior, to judge the robustness of findings, and to statistically analyze model output. Figure 8 shows a simple example demonstrating the use of these classes/methods. When creating an instance of this class for the simple model from Vierling-Claassen et al. (2008), an additional parameter containing a number of RNG seeds is passed on. When the model is then run, a simulation is executed for each RNG seed, and the model output is a list containing the result for each simulation.

Figure 8.  

Generating additional data. First, model instances are created and the produce_XY_plus method is used to run the simulation. The additional seed parameter contains a list of 20 RNG seeds, and a simulation is executed for each seed in that list. Thus, each simulation differs in background noise. The produce_XY_plus methods return the mean values of the outputs for the simulation runs (mcontrol4040 and mschiz4040 above), which can be used analoguously to the output of the standard produce_XY methods. However, the values of the output of each single simulation run are returned for each run (control4040 and schiz4040 above) and can then be visualized or further analyzed statistically. Note that the model parameters used in this example are not based on any actual experimental findings in schizophrenic patients and that they do not aim to reproduce any experimental observations; they are only used for demonstration purposes (for model parameters of this model that reproduce experimental observations, see the original article by Vierling-Claassen et al. [2008]).

Use Case 3: Overview of Model Predictions

Finally, we show how predictions can be generated from existing models (see Figure 9). To generate the predictions, a set of prediction tests along with prediction capabilities, that is, capabilities the models must have for the models to generate the relevant data, needs to be created. For demonstration purposes, we have chosen to implement a single, simple prediction test. Because in ASSRUnit so far, we have only looked at experimental observations and computational models that cover gamma- and beta-range entrainment, the first test simply generates a prediction about how, in a given model, power in the alpha band (here at 10 Hz) differs between the control network and the schizophrenia-like network at 10 Hz drive. Note that this prediction test has been studied in the experimental literature, which means that it could have already been included in the experimental database and therefore does not represent a true prediction. However, we have chosen to include it for the purpose of demonstration. As before, more detailed information can be found in the accompanying Jupyter notebooks (https://github.com/ChristophMetzner/ASSRUnit/blob/CompPsychArticle/assrunit/Notebooks/Example_Prediction.ipynb).

The Potential Role of the Framework Within Computational Psychiatry

The use of computational approaches has seen a significant increase over the last decades in almost all areas of medicine and life sciences. Especially in psychiatry, it has become clear that the complex and often polygenic nature of psychiatric disorders might only be understood with the help of computational models (Adams et al., 2016; Corlett & Fletcher, 2014; Friston et al., 2014; Montague et al., 2012; Siekmeier, 2015; Stephan & Mathys, 2014; Wang & Krystal, 2014). Naturally, the number of computational models in the field of psychiatry has also increased significantly over the last years, and it has been argued that in silico instantiations of biomarkers are a crucial step toward understanding underlying disease mechanisms (Siekmeier, 2015). While this large increase in the number of modeling studies shows the importance of computational methods in the field, it also raises several issues that impede the community in exploiting these approaches to their full potential. For a computational model to be a substantial contribution to knowledge, it has to adequately instantiate experimental observations, correctly implement the mathematical equations of the model, and generate experimentally testable predictions. The approach presented here addresses two of these three requirements, namely, the instantiation of experimental observations and the generation of testable predictions. While correctness of the code is an equally important requirement, it was out of the scope of the current work, because it very strongly depends on the type of computational model and on the programming language used to implement the model. Nevertheless, the approach presented here offers significant benefits for not only the computational psychiatry community but the psychiatry community as a whole, while imposing little additional effort on users and contributors. It gives modelers a tool to query experimental observations on neurophysiological and neurocognitive biomarkers and therefore helps them include current relevant experimental data in their modeling efforts. It further enables them to validate their modeling output against experimental observations during model construction and to demonstrate the performance of their models, both with respect to the experimental literature and with respect to other competing models. In addition to the benefits it offers modelers, it also enables experimentalists to quickly gain insight into the current state of modeling and to extract experimentally testable predictions from the models. Last, but not least, it offers a tool to reviewers that allows them to judge a newly proposed model by making explicit its performance against experimental data and competing models.

The concept of automated code testing and validation has been successfully applied in computer science for many years now; however, it is only slowly finding its way into the computational branches of scientific fields. SciUnit attempts to satisfy this demand by providing a simple, flexible, yet powerful framework to address the earlier mentioned issues. The computational neuroscience community has started to adopt this framework for the automatic validation of single neuron models (NeuronUnit; Gerkin & Omar, 2013). We are not aware of any similar efforts in the field of psychiatry.

Because schizophrenia is a polygenic, multifactorial, and very heterogeneous disorder, it has been argued that the usefulness of biomarkers lies in their potential to dissect the disorder into subtypes, which might even be linked more closely to findings on the genetic level (Markou et al., 2009; Meyer-Lindenberg & Weinberger, 2006; Perlis, 2011). The proposed ASSRUnit module together with computational models of biomarkers and specifically designed test suites could strongly facilitate this process by providing mechanistic links between neurophysiological or neurocognitive biomarkers and changes at the synaptic, cellular, and/or network level.

Future Directions for ASSRUnit

The presented ASSRUnit module can be easily extended and modified by others to fit their needs (e.g., to include more specialized visualization tools). Our efforts for establishing ASSRUnit as a widely used tool will focus on three main areas. (a) We aim to cover the majority of existing experimental studies with our experimental database in the future. Furthermore, we hope to convince experimentalists to provide more detailed experimental data or to ideally create database entries themselves. (b) We also aim to cover the majority of current computational models that describe the cortical circuitry responsible for the ASSR. Again, we hope to encourage modelers to contribute actively to ASSRUnit. (c) We aim to extend our set of prediction tests and thus our prediction database.

The most straightforward extension, in our view, is to include information on phase locking in addition to pure power in certain frequency bands. Several studies have reported, additionally to a reduction in gamma power, a reduction in the phase-locking factor for patients suffering from schizophrenia (e.g., Brenner et al., 2003; Krishnan et al., 2009; Kwon et al., 1999; Light et al., 2006; Vierling-Claassen et al., 2008). These observations can very easily be incorporated into the existing module by including the experimental observations in the database, adding the necessary capabilities to the model classes, and adding the appropriate tests that link the experimental observations to the model capabilities.

Furthermore, the changes in oscillatory activity upon auditory stimulation are not limited to the gamma and the beta ranges for schizophrenic patients but also extend to lower-frequency bands, such as alpha, theta, and delta. For example, Brockhaus-Dumke, Mueller, Faigle, and Klosterkoetter (2008) found reduced phase locking in the alpha and theta bands for schizophrenic patients in an auditory paired-click paradigm, and Ford, Roach, Hoffman, & Mathalon (2008) found a reduction of phase locking in the delta and theta ranges for schizophrenic patients in an auditory oddball task. Abnormalities in these frequency bands have also been found in many other paradigms outside of the auditory system (see Basar & Guntekin [2013]). To the best of our knowledge, ASSRs to entrainment stimuli in the theta and delta ranges have not been looked at in schizophrenia. Therefore ASSRUnit could be used to generate predictions in these frequency ranges, as demonstrated in use case 3.

However, an inclusion of the earlier mentioned observations together with computational models explaining these deficits is not straightforward, because either the paradigms are different from the ones used to elicit ASSRs and/or the mechanisms underlying the effect are different, and therefore the computational models are substantially different to models of ASSRs. Therefore these deficits are better explored in separate modules solely focusing on each paradigm/deficit. However, it would be very interesting to co-explore computational models that have the capabilities to explain both ASSR gamma/beta band and delta/theta/alpha phase-locking deficits. Such an analysis could highlight interactions between different mechanisms underlying different symptoms or biomarkers.

Another very interesting and promising extension of the current module would be to include data and models from different psychiatric disorders, because schizophrenia is not the only disorder where patients show entrainment deficits. Wilson et al. (2007) explored gamma power in adolescents with psychosis and found reductions compared to normally developing controls. Their patient group consisted of patients suffering from schizophrenia and also from schizoaffective disorder and bipolar disorder. Interestingly, these disorders show overlapping symptoms, neurobiological substrates, and predisposing gene loci. Other studies have found reduced power and phase locking in the gamma range in patients with bipolar disorder (O’Donnell et al., 2004; Rass et al., 2010; Spencer et al., 2008). The presented module is perfectly suited to highlight commonalities and differences across disorders and to link those to mechanistic explanations via different theoretical and computational models.

Other Modules Beyond ASSRUnit

The approach presented here, combining an experimental database with a collection of models, tests, prediction tests, and a resulting predictions database, can be readily applied to a number of other neurophysiological biomarkers of schizophrenia as well as other psychiatric disorders.

In patients suffering from schizophrenia, a dysfunction of the auditory system has long been suspected. In fact, a large number of biomarkers for schizophrenia, other than ASSR deficits, involve auditory processing. Several alterations of event-related potentials (ERPs), such as mismatch negativity (MMN), N100, and P50, have been described in the literature (see Shi, 2007; Siekmeier, 2015, for reviews of potential biomarkers and computational models thereof).

Naturally, our approach is well adaptable to brain circuits outside of the auditory system. Working memory deficits are probably one of the most robust and best described cognitive deficits in schizophrenic patients (reviewed in J. Lee & Park, 2005; Piskulic, Olver, Norman, & Maruff, 2007). Patients show a decrease in working memory capacity, that is, the capacity to maintain, manipulate, and use information online for a relatively short period of time, across a broad range of paradigms. Again, several theoretical and computational models have been proposed, aiming to provide mechanistic descriptions of the underlying mechanisms (e.g., Cano-Colino & Compte, 2012; Compte, Brunel, Goldman-Rakic, & Wang, 2000; Durstewitz, Seamans, & Sejnowski, 2000; Singh & Eliasmith, 2006; Wang, 2001; Wang, Tegnér, Constantinidis, & Goldman-Rakic, 2004).

All these deficits and alterations, along with relevant computational models, could be integrated into packages similar to the proposed ASSRUnit package. Such a unified framework would be of great benefit for the study of schizophrenia pathology due to the diversity of symptoms, biomarkers, and experimental observations linked to the mental disease.