The total sample consisted of 246 individuals with an ARMS, recruited at first presentation from consecutive referrals to the Outreach and Support in South London (OASIS) high-risk service, South London and Maudsley NHS Foundation Trust (29). The presence of ARMS was assessed using the CAARMS, a detailed clinical assessment (30). When the subjects were first diagnosed as having an ARMS (i.e., baseline) a set of data were acquired: (a) a sMRI scan; (b) genome-wide genotypes; and (c) assessment of environmental risk exposures. Subjects were labeled as transitioned to psychosis (ARMS-T) if they later presented a FEP or as not-transitioned to psychosis (ARMS-NT) if they did not present a FEP within a period of at least 2 years. For a detailed description of the recruitment, inclusion and exclusion criteria please refer to the Supplementary material. Additional socio-demographic and clinical measures were also assessed at baseline, including: age; sex; handedness; self-reported ethnicity; full scale intelligence quotient measured by the National Adult Reading Test (31); years of education; and global assessment of function using the GAF instrument tool at baseline and at follow-up (32), and CAARMS (at baseline and follow-up) (30). Regarding the sMRI, genetic and environmental sub-samples: 99, 135 and all the 246 individuals with an ARMS had a baseline sMRI scan (Table 1), genome-wide genotyped data (Table 2), and environmental risk factors assessment data (Table 3), respectively (more details in the Supplementary material). Over the 2-years follow-up period, 23, 41, and 60 individuals at an ARMS from each of the previous sub-samples developed psychosis (AMRS-T) and the remaining 15, 94, and 186 did not (ARMS-NT), respectively. Moreover, part of the study’s data collection occurred under the Genetic and Psychosis (GAP) umbrella project (33). Ethics approval was obtained by the NHS South East London Research Ethics Committee (Project GAP; Ref. 047/04), consistent with the Helsinki Declaration of 1975 (as revised in 2008) and all subjects gave written informed consent.

Socio-demographic and clinical variables were analyzed using a two-tailed independent t-test or a Univariate Analysis of Variance (ANOVA) for continuous data and a chi-square test or Fisher’s exact test (if there were less than 5 subjects in one group) for ordinal data (Tables 1–3). These statistical analyses were performed using the Statistical Package for the Social Sciences 26 (SPSS 26 for Windows, Chicago, IL, USA).

Genotyping procedures have been previously described (26, 38). In summary, samples were genotyped at two different sites with two distinct chips (Illumina HumanCore Exome BeadChip and Genome-wide Human SNP Array 6.0). A standard quality control screening (exclusion of SNPs with low minor allele frequency, high genotypic failure and not in Hardy Weinberg equilibrium) followed by imputation procedures were conducted. Then, samples from both sites were merged by keeping only the overlapped imputed SNPs followed by a second quality control screening. Finally, a population stratification analysis was conducted with principal component analysis (PCA) to select only subjects with a European ancestry (the number of subjects per self-reported ethnicity is in the Supplementary Table 3). For a detailed description see the Supplementary material.

Each subject was assessed on at least one of eight environmental risk factors: (1) tobacco and (2) cannabis consumption; (3) being migrant; (4) belonging to an ethnic minority; (5) the upbringing urbanicity level; (6) the parental age at birth; (7) the presence of childhood trauma; and (8) the season of birth (detailed description of how the risk for psychosis was assessed in each factor is in Supplementary material).

Several ML strategies to generate prediction models for transition to psychosis from sMRI data using our ARMS sample were investigated (Figures 1, 2). These include: (a) sample balancing and bootstrapping; and testing several: (b) feature types; (c) feature manipulation approaches; and (d) CV approaches. The analyses were conducted using the neuroimaging ML tool NeuroMiner v1.0 ELESSAR³ for sMRI data, chosen given that it was used in the previous above-mentioned ARMS prognosis studies and provided therein high accuracy results (12, 39, 40), and R software 4.0.5 (41) for genetic (16) and environmental data. As detailed below, we have used SVM on the neuroimaging data since that is the approach which not only is more often employed with sMRI data but also that which has shown higher accuracies in psychiatric diagnostic classifications using sMRI data including in the ARMS population (10–14) which we herein attempt to replicate. We have used elastic-net algorithm for the genetic data (SNPs and eQTL scores) and environmental risk factors as it a well-suited method for dealing with high-dimensional data and possibly correlated data; and it performs an embedded feature selection and model fitting at once. The PRS and the environmental risk score were analyzed with logistic regression, given that only one feature was used.

In this study we applied ML to structural neuroimaging data using a relatively larger sample and an ML approach, improved to the best of our ability, to detect transition to psychosis from an ARMS and to replicate previous positive findings of accuracies 74 to 84% of six studies, which together used 3 independent samples (10–15). For this, we decided: to: (1) use only the most recent versions of the image processing tools (i.e., CAT12) and ML tools (i.e., NeuroMiner); (2) replicate as accurately as possible the methods that were described in the abovementioned MRI papers since it was not possible to access their processing and ML pipelines; (3) add a layer of ML generalizability by bootstrapping and fitting a model to each subsample; and (4) overcome previous studies’ limitations (e.g., sample unbalancing for demographics). Furthermore, we explored, for the first time, the use of whole brain white matter volume and regional white matter volume, cortical thickness, and surface-based brain gyrification, sulci depth, and complexity indexes with ML to predict transition to psychosis.

Unexpectedly, we did not replicate previous findings. After balancing the samples for binary classification of transition to psychosis accounting for age, sex, and the three different scan acquisition protocols to avoid overoptimistic results, the performance of all tested combinations (i.e., of feature type–ROIGM, ROIWM, ROISurface, VBGM, or VBWM; feature manipulation–feature dimensionality reduction through PCA, no feature selection, or forward feature selection; and CV scheme–LSO CV, LPO CV, or 5-fold CV) were not significantly better than chance level.

Compared to the previous studies reporting high balanced accuracies (74 to 84%) in predicting transition to psychosis from sMRI maps (10–15), the current study has some advantages. First, this study’s sample is drawn from a more naturalistic ARMS population as it includes subjects whose sMRI images were acquired using three different scan acquisition protocols. Training a classification model with data from different centers potentially increases its generalizability. Only one of the previous transition to psychosis prediction studies used a two-site group balanced sample (12), combining the samples reported in two previous studies by the same authors (10, 11). The main differences between this report and our study are the following: (a) Their sample was larger than our balanced bootstrapped samples (i.e., 36% larger than ours, measured as the absolute value of the change in sample size, divided by the average of the size of the two samples). However, we tested our ML models on five balanced subsamples (i.e., through bootstrapping), allowing us to obtain a measure of generalizability of these models’ performance. Moreover, they do not present a measure of the statistical significance of the model’s BAC, which we do herein. (b) They controlled the effect of site on the classification using partial correlations during the training phase of the CV cycle, whereas we controlled it by keeping the same proportion of subjects at an ARMS that transitioned to psychosis and those who did not in each scan protocol during the training phase of the CV cycle (i.e., when using the LPO CV scheme as the previous study did). Additionally, we also guaranteed that the pair of subjects left out for testing/validation were from the same site. This potentially increases the generalizability of the classification model by training it with a more heterogeneous sample (and, as explained above, more naturalistic) and diminishing the effect of site on the testing/validation classification accuracy, which is not taken into account in the previous report (12).

Second, we trained our classification models with samples balanced for group (subjects at an ARMS who later transitioned to psychosis and who did not), age at scan and sex. Balancing for group is important to avoid biasing the classification model to the most represented group and it was not taken into account by three out of six previous reports (10, 11, 14). Moreover, the effects of age (49) and sex (50) on brain structure, rate of transition to psychosis from ARMS (2), and prevalence of psychosis (3, 51), have been consistently reported and, therefore, should be taken into account in these studies. All previous reports (and the current study) matched transition proportion for age and sex (10–14), except for one (15). Das and colleagues reported a statistically significant and better than chance level BAC in predicting transition to psychosis using a sample unbalanced for both group and sex. Although they used a ML algorithm with class (i.e., group) weighing–which in summary increases the influence of the minority class when training the model by assigning higher weights to rare cases, the authors performed an unspecified correction for sex effect (as well as for age and TIV effects) to the data during the training CV cycle. This approach may not be the most appropriate given the known effect of sex on brain structure (50) and the, abovementioned, empirically tested association between sex and group (i.e., transition to psychosis from an ARMS vs. no transition) (15), which makes sex a potential confounder in this analysis. Furthermore, in three of the six previous reports, the effects of age and sex were corrected before entering the ML analysis (10), and during the training CV cycle (11, 15) using partial correlations (10, 11) or an unspecified method (15)–which we did not perform. Correction for age effects in ML analysis has been previously shown to increase classification accuracy in Alzheimer’s disease, when it is estimated from healthy subjects (52). Correction for effects of no interest in ML analyses should be done with extreme caution as it can easily remove relevant subject-specific information (53). This is especially important when the correction is being performed in a non-healthy (i.e., non-standard) population, because the effect of external variables such as age and sex might be modulated by the presence of the disease (e.g., being at ARMS or having schizophrenia).

Third, this study’s sample is composed of subjects whose clinical diagnosis of an ARMS was based on having a schizotypal personality disorder or on the subject’s familial-high risk coupled with functioning decline and on the CAARMS (54), which mainly evaluates positive symptoms. These were not the same criteria as those used in the previous studies predicting transition to psychosis from an ARMS. These previous studies all used samples of subjects clinically assessed with tools that evaluate not only positive symptoms, but also basic and negative symptoms (10–12, 14, 15), except one (13), which included only familial-high risk subjects in its sample. This potentially increases the inclusion of subjects in the early phase of the psychosis prodrome (characterized by the presence of basic and negative symptoms), whereas our sample includes mainly subjects in the late prodromal phase of psychosis (characterized mainly by the presence of positive symptoms) (2). Therefore, our results suggest that previously reported accuracies in predicting transition to psychosis may be population-specific, poorly generalizable to differently clinically characterized populations (as ours herein).

In this study we applied ML to genetic data and used three types of genetic features to detect transition to psychosis from an ARMS: (a) a schizophrenia PRS that we have previously shown to distinguish FEP patients from healthy controls (26) and ARMS-T from ARMS-NT (16), (b) a set of psychosis-associated SNPs previously associated with schizophrenia in a recent GWAS meta-analysis (27), and (c) a brain-specific expression Quantitative Trait Loci (eQTL) score including the latter genes.

Genetic data showed a poor performance in predicting transition to psychosis from an ARMS. SNPs-based classification models have been previously shown to classify schizophrenia (18, 19, 21), and FEP patients (23) (vs. healthy controls) better than chance level, but not subjects at an ARMS vs. healthy controls or FEP patients (23). Furthermore, one of these studies has selected a list of SNPs from the Psychiatric Genomics Consortium 2 (PGC2) (21, 42), which potentially overlaps with the ones selected in this study (27).

Despite the (scarce) evidence of the potential of PRS for schizophrenia (20–22) to classify schizophrenia patients (vs. healthy controls) and the one report showing the schizophrenia PRS’s ability to predict transition to psychosis (16) we were not able to predict transition to psychosis from an ARMS using this type of genetic feature. Although the latter study (16) used a larger sample (i.e., 106% higher than ours, measured as the absolute value of the change in sample size, divided by the average of the size of the two samples) to train the PRS-based model, sample balancing in terms of group and age or sex were not taken into account or that was unclear, respectively. Furthermore, herein we applied a bootstrapped sample approach to estimate generalizability of the PRS-based model by assuring that each bootstrapped sample met the balancing conditions for group, age, and sex–which does not seem to be the case in that study (16). Furthermore, another possible explanation for the PRS negative results is that although the genetic architecture, conveyed through a PRS, has been shown to differ between patients with schizophrenia and healthy controls, one cannot exclude the possibility that it is specific to schizophrenia (a fully developed psychotic disorder), and might even be present in all subjects at an ARMS, i.e., those who later transition to psychosis and those who do not. The constellation of genetic variations (i.e., SNPs) that might confer susceptibility to transition to psychosis already from a prodromal stage is not necessarily the same as the one for schizophrenia (when drawn in comparison to healthy controls). This may justify the advantage of using a less hypothesis-based approach for the selection of genetic features (as we did by pre-selecting a large list of SNPs and performing an embedded feature selection using elastic net regression). Lastly, using a PRS formula made specifically for transition to psychosis from an ARMS would require a larger and independent sample to estimate SNP effect sizes, which might be better provided by multicenter projects, such as NAPLS 2 (55) and PRONIA⁴ over the next years.

Expression Quantitative Trait Loci (eQTL) scores for psychosis associated genes expressed in the brain were also not able to predict transition to psychosis from an ARMS. Only one previous study has shown the predictive value of gene expression profiling in the frontal brain region in classifying schizophrenia patients (vs. healthy controls) (17). In the present study, instead of actual gene expression measures we used a proxy for a-genetically regulated component of the expression of genes, the eQTL scores. Although we have computed eQTL scores only for the genes having a validated eQTL score model (43), this does not guarantee that the estimated gene expression represents (or correlates perfectly with) the real levels of the expression. Furthermore, although we have selected the initial list of genes as the ones most associated with schizophrenia (vs. healthy controls), this selection did not take into account the expression profile of these genes in the brain, and we have computed an eQTL score for several brain tissues. A future improvement of this step would be to test an eQTL scores-based model with a selection of genes that: (a) are highly expressed in the brain in healthy subjects, and (b) their expression is associated to a schizophrenia diagnosis, or even better with the transition to psychosis from an ARMS.

In this study we applied, for the first time, ML to environmental data using two types of features to detect transition to psychosis from an ARMS: (a) a schizophrenia ERS which we have previously reported (28), and (b) a set of environmental risk factors as predictors. Overall, neither environmental risk assessment, could predict transition to psychosis from an ARMS with an averaged accuracy, i.e., across bootstrapped samples, better than chance level. Although we know of no similar longitudinal ARMS transition study, the closest other report using ML and environmental data to diagnose schizophrenia (vs. healthy controls) (22) also found a BAC not statistically better than chance level, even having included features such as the presence of obstetric complications and of developmental anomalies, the parental socio-economic status; and –without feature selection–trained and tested the model in a 13 times larger, albeit age, sex, and group -unbalanced, sample (103 patients and 337 controls) than ours (22). However, due to the still poorly understood environmental risk mechanisms one cannot exclude the lack of statistical power as a potential explanation for these negative findings including ours.

The ML model trained with the ERS for schizophrenia, which we have tested as an (admittedly limited) exploratory predictor of the transition to psychosis from an ARMS, showed a poor performance, i.e., a BAC similar to chance level. Indeed, ERS is a composite score of individual risk factors computed under the assumption that the risk factors are completely independent (28), which has been shown not to be the case (56)–i.e., intercorrelated risk factors may inflate the ERS estimation. This crude approach may limit the ability of the ERS to capture the detailed environmental architecture underlying psychosis. Moreover, just as for a PRS, an ERS for schizophrenia may not be a good substitute of a potential ERS for transition to psychosis from an ARMS (57).

Lastly, our criterion for training and testing a fully multimodal ML model with modalities that would show an ML model performance statistically better than chance (i.e., 50%) predicting transition to psychosis from an ARMS in at least 3 of 5 bootstrapped samples was not fulfilled given that none of the modality-based ML models survived that threshold. This conservative criterion was chosen given the already small sample size available for the training of the multimodal ML model, i.e., only 6 ARMS-T and 23 ARMS-NT (only this subset of subjects had data for the three data modalities, simultaneously). The decrease in sample size, remarkably impairs the prediction power of the model, i.e., its accuracy. Without previous evidence of the ability to predict transition to psychosis from an ARMS by modality supporting its integration in a multimodal ML model, negative results from this multimodal model would be highly difficult to explain, as they could theoretically be explained by the increase of noise in the model due to the inclusion of features that did show previous predictive ability or by the lack of predictive power due to the very small sample size. Moreover, the parallel-to-ours, multi-site study, albeit very group-unbalanced (only 26 ARMS-T patients vs. 308 ARMS-NT), from the PRONIA project, showed that a stacked model combining similar data to our study’s plus human prognostic ratings could predict transition to psychosis with a balanced accuracy of 86% and a good geographical generalizability (25). This multimodal approach was showed to improve biological-based unimodal models by 15% (VBGM volume maps-based model) and 20% (PRS for schizophrenia-based model). As such, the replication of this promising finding, following the same multimodal approach as that study, using in our study’s sample and data features co-existing in both samples, would be interesting as an additional method to ascertain whether our negative findings are due to lack of power or to no discriminability with our feature sets.

This study was limited by several factors. First, and foremost, the small sample size may have limited the performance of classification models, even though our sample size was informed by previous ML studies showing 74–84% accuracies in predicting transition to psychosis from an ARMS (10–15). Indeed, this is a critical limitation when dealing with high dimensional data, such as neuroimaging and genetics–which we have used herein. Although we have taken measures to avoid overfitting and an overestimation of the classification models’ performance such as artificially increasing the sampling through bootstrapping and employing CV strategies, this might not be enough to overcome this limitation. Indeed, our complementary analysis comparing the models’ training and testing performance (results in the Supplementary material) is indicative that some of the tested classification models (mainly trained with neuroimaging or with SNPs) might suffer from some degree of overfitting. Ultimately, we cannot determine whether our negative findings were due to lack of power to obtain a good performance or due to a true lack of association between the predictors and the transition to psychosis from an ARMS (and hence inflated findings from previous studies). This is one of the reasons why replication studies in independent datasets are essential in ML literature. As a final note, a power analysis for this study design would have been the most informative way to define the sample size needed to achieve an accuracy in predicting transition to psychosis from an ARMS better than chance level. However, this is not a trivial task in ML analysis and there is no established method to perform this analysis as there is for univariate analysis [for examples of studies exploring innovative ways of computing sample size for classification problems see Refs. (58, 59)] and, therefore, it was not performed.

Second, in order to dilute possible confounding effects in the developed classification models we have restricted the samples used to train the models to: (a) be class-balanced, i.e., with the same number of ARMS-T and ARMS-NT subjects; (b) be matched for age, sex, scanning acquisition protocols for neuroimaging data; (c) include subjects with European ancestry only for genetic data; and (d) limit the proportion of missing data for the environment data. Although this has artificially homogenized the study sample thus avoiding the presence of statistical confounders, it has deemed the sample to be less representative of the ARMS population. Third, overall, the findings of this study are only valid to young help-seeking individuals, i.e., that are clinically screened for ARMS criteria, and whose ARMS diagnosis was based on having a schizotypal personality disorder or on the subject’s familial-high risk coupled with functioning decline and on the CAARMS (54), which mainly evaluates positive symptoms.