To facilitate standardization across laboratories, we have developed software applications for oddball-type tasks (Sutton et al., 1965) and an Eriksen flankers task (Eriksen and Eriksen, 1974). These applications are coded in the Java programming language and work on Windows, but can also be run on other platforms through a Windows emulator. Each application allows the user to input a range of configurations to comprehensively cover parameter variations (inter-trial interval, visual or auditory modality, error feedback, instructions, etc.). A user can save a configuration file with these parameters, as well as any type of response requirement. This latter feature facilitates the creation of a variety of passive and active tasks, including go/no-go and vigilance tasks. By providing these programs, we hope to encourage smaller site-specific patient studies to include an additional short assessment to their protocols for the purpose of open-source data sharing.

If popular, we hope to include a multitude of task types in the future, such as reward gambling, stop-signal, basic language, and motor tasks, etc. Some existing task batteries for EEG assessment capitalize on simultaneous acquisition of a large number of EEG events/ERP components (Kappenman and Luck, 2012; Kieffaber et al., 2016; Nair et al., 2016); this may be a promising direction for future expansion.

An upload tab facilitates user requests for contributing data to PRED+CT. A download tab contains study information (Table 2) and will be fully open (no log in or request required). All data will be hosted in Matlab readable format (.mat or .set files, which are interchangeable) for a few reasons. Matlab is the current most common platform for academic EEG research, and the popular EEGLab suite (Delorme and Makeig, 2004) will be utilized as a common data structure. Many native file types contain information that could be a threat to confidentiality, and EEGLab import tools strip many of these markers. Matlab files are easily imported into other (open-source) programs like Octave, Python, and R, so this common structure should not be limiting in any way.

The most likely threats to personal health information in candidate PRED+CT database entries are names or initials, locations, and dates. The user is required to ensure that none of these remain in the subject identifier or in the EEG metadata: for example, BrainVision.vhdr files contain times and .vmrk files contain a date stamp. EEGLab import to Matlab strips the data of such possible hidden threats to confidentiality. PRED+CT administrators will perform a double check on potential threats to confidentiality and can assist with the translation from native formats into .mat files. Most institutional review boards (IRBs) do not consider data sharing itself to fall under the definition of “human subjects research,” but interested users should request a determination from their IRB. Including such language in the informed consent is the best way to ensure that ethical issues are proactively well-managed.

It is highly recommended that users include data in its raw form prior to any pre-processing; however, this may be infeasible in some cases. In many instances, important recording information is embedded in the native data format (sampling rate, reference, electrode labels) and is translated directly into the EEG data structure. If this information is missing (i.e., data are from a clinical system) then the user should include a readme file with this information.

For data other than rest, is important that users can understand what trigger types (TTL pulses) are used to represent each type of event. At minimum, a comprehensive list of trigger types needs to be included, and we highly recommend that users include the stimulus presentation script (i.e., Eprime run file, Matlab Psychtoolbox files, etc.). Ideally, the task was designed so that behavioral responses can be recoverable from the triggers; if not then upload of separate behavioral logs is encouraged. While there are common neuroinformatics structures that can facilitate sophisticated data organization schemes across studies (Teeters et al., 2008; Landis et al., 2016; Plis et al., 2016; Wiener et al., 2016), we opted for a more simplified approach in PRED+CT based on simple and well-documented descriptions of idiosyncratic TTL triggers.

Pattern classifiers use cross-validation or bootstrap approaches where the whole dataset is partitioned into non-overlapping training and test sets. The classifiers involve optimizing information theoretic and multidimensional metrics to generate models based on signal shapes and do not over-fit the data as traditional predictive models do (Parra et al., 2005; Pereira et al., 2009; Lemm et al., 2011). Such pattern-based classification is thus necessary to generalize predictive models for diagnostic subtyping and recovery trajectory to other groups (aka biomarkers).

There are a number of tutorials for general brain science classification (Pereira et al., 2009; Lemm et al., 2011), EEG-specific tutorials (Parra et al., 2005; Dyrholm and Parra, 2006; Schirrmeister et al., 2017), and open-source sets of analytic tools (Detre et al., 2006; Hanke et al., 2009). Our goal is not to replicate these resources, but to provide a repository for computational approaches that can bolster feature selection or patient classification, particularly on existing datasets in the archive. While we envision hosting a multitude of scripts, there is no reason that entries on this page couldn’t link to external resources (i.e., github or NITRC).

Unless otherwise noted, this database and its contents are made available under the Public Domain Dedication and License v1.0 whose full text can be found at: https://opendatacommons.org/licenses/pddl/1.0/. We hope that all users will follow the ODC Attribution/Share-Alike Community Norms, including the expectation that there will be no attempt to de-anonymize any data.

Figure 2 shows an example of the type of outcome we hope to cultivate with PRED+CT. This figure shows a receiver operating characteristic plot detailing classification of Parkinson’s patients on and off medication vs. well-matched controls based on three-auditory oddball task conditions (Cavanagh et al., under review). That report details the reasons why aberrant orienting to novelty is mechanistically interesting in Parkinson’s disease, and why EEG is uniquely well-suited to assess to biomarker potential of the associated neural response. These raw data and scripts are available on the PRED+CT website (accession nos.: d001 and t001). Since this task is very brief and very easy for patients to perform, we encourage other groups to contribute similar datasets to examine the generalizability of this phenomenon.

Interested users can follow @PREDiCT_Admin or #PREDiCT + #UNM on Twitter for updates, including new dataset and tool contributions.

While data sharing is “good,” a prevalent challenge is to share high-quality usable data (Kennedy, 2004). By archiving EEG and metadata in a common EEGLab structure, we can fulfill these criteria. Substantive hardware and software advancements over time are unlikely to change basic aspects of EEG data. Numerous algorithms exist to assist in pre-processing EEG data (Delorme and Makeig, 2004; Nolan et al., 2010; Oostenveld et al., 2011; Bigdely-Shamlo et al., 2015; Chaumon et al., 2015). In the Computational Tools section, we have provided our Algorithmic Pre-Processing Line for EEG (APPLE.m; accession no.: t002), which leverages a combination of FASTER (Nolan et al., 2010), ADJUST (Mognon et al., 2011), EEGLab, and custom algorithms for automatically interpolating bad channels, removing bad epochs, and identifying the most likely independent component associated with eye blinks.

Electroencephalography datasets come in a variety of reference schemes, topographical layouts, and sampling rates, complicating integration. However, these are addressable problems. Use of the average reference and relative measurements (decibel, percent change, relative power) facilitate common analytic space. EEG data tends to be oversampled, so down-sampling to a common lowest denominator is a viable option. As pattern classifiers leverage any difference between training sets, it will be critical to ensure that spurious differences between combined datasets do not interfere with the aim of classifying patients from controls. Having an equal number of patients and well-matched controls in each dataset is a good first step for experimental control over this issue, but additional steps like controlled randomization of training and testing sets may be required to control for dataset-specific biases.

Once pre-processed, EEG data offer a rather simple data structure that is accessible by non-experts. A two-dimensional matrix of channels ^∗ time can be easily restructured to include a third dimension based on discrete events (i.e., the EEG.data field of EEGlab), and no special software, opaque statistical constraints, advanced processing, or other complicated considerations are strictly necessary for interpretation. We hope this increases the appeal to computer and data scientists, who should be able to manage EEG data as an input variable with very minimal special training.

A well-known adage in machine learning is that achieving 80% classification accuracy is easy, and closing the gap toward 100% accuracy will take between a few years and eternity. We think that PRED+CT can assist with strategies for (partially) closing this gap, which we detail in order of their intuitiveness. The most straightforward solution to boost generalizability is to utilize larger training sets (i.e., more EEG data), which is the primary purpose of the site. Another immediately apparent solution is to leverage algorithmic advancements. In isolation it is hard to know how revolutionary different classification procedures will be, but open communication may help set standards and constraints on parameter selection which otherwise act as a hidden threat to generalizability. It is important to note that theoretical validation requires the ability to interpret which of the input features led to successful classification (cf. Steele et al., 2014; Cavanagh and Castellanos, 2016; Doshi-Velez and Kim, 2017).

We submit that the best way to achieve these overarching goals will involve taking advantage of the EEG feature(s) reflecting the neural computations that maximally discriminate groups. While resting activities may be the optimal solution for some patient groups, specifically designed active tasks are likely necessary to elicit the requisite brain responses that characterize the nature of the departure from statistical normality. Error signaling in anxiety has already been described above as a defining neural computation related to the etiology of the disorder, but other candidate responses have been advanced for other disorders, including diminished reward signals for major depression (Proudfit, 2015), a broken target-updating P3b in schizophrenia (Ford, 1999), a reduced novelty orienting P3a in Parkinson’s (Solís-Vivanco et al., 2015), and reduced brainstem evoked responses in acute traumatic brain injury (Kraus et al., 2016) to name just a few.

Finally, a less intuitive recipe for success may be to simply ask more specific questions. More constrained hypotheses can help to collapse insurmountable prior probabilities into the realm of plausibility. Consider that the base rate for any specific neurological or psychiatric disease is low enough to dismiss the plausibility of a new EEG-based diagnostic test with viable sensitivity and specificity. Yet if a patient is already being treated for a disease, this obviates some base rate problems. For example, instead of trying to develop a fast and easy brain scan to identify if someone has major depressive disorder, it is more plausible to ask if a diminished brain response to reward can help guide treatment options to address melancholic vs. atypical features of depression.

To be medically useful, a test must have positive prognostic value above and beyond current status quo, or reduce time, cost, or uncertainty (Nuwer et al., 2005). While brain-based diagnostics may achieve impressive sensitivity, they are often associated with high false positives (low specificity), which is a particular deterrent to clinical use for differential diagnosis (Nuwer et al., 2005). These challenges are addressable, especially since direct clinical application is not a necessary outcome of brain-based patient classification.

High sensitivity in the context of low specificity may nevertheless have important clinical utility for rapidly assessing the potential presence of diagnostic complications (Hanley et al., 2013; Ayaz et al., 2015), quantifying the degree of injury severity (Thatcher et al., 2001), or for tracking differences in disease progression in treatment studies. Identification of the maximally discriminable neural computation that defines a patient group has additional translational utility: for instance it could be used as concurrent validation of a novel imaging or biomarker measure. In sum, reliable novel findings are important successes even if they do not lead to direct clinical translation.