Seventy-eight participants took part, including 28 participants (age range 21–54 years) diagnosed with schizophrenia (Sz) using the Structured Clinical Interview for DSM-IV (First et al., 1994), 20 adults with autism spectrum disorder (ASD) (age range 19–43 years), confirmed by the Autism Diagnostic Observation Schedule, Second Edition, and 30 neurotypical controls (age range 19–54 years) (Table 1). All Sz participants were on a stable dose of antipsychotic medication. All participants had at least 20/22 corrected visual acuity on a Logarithmic Visual Acuity Chart. On average, Sz participants were older [F(1, 56) = 7.24, p = 0.009] and had lower IQ scores [F(1, 56) = 6.54, p = 0.013] than controls. All ASD participants and a subset of 19 Sz and 17 controls participated in our previous EEG/fMRI study of visual sensory dysfunction as reported in Martinez et al. (2019), which did not include data from the present paradigm. Participants were recruited from the central research database and volunteer recruitment pool at the Nathan Kline Institute for Psychiatric Research (NKI). The investigation was approved by the Nathan Kline Institute (NKI) institutional review board. Informed consent was obtained from all participants.

Unique video clips of five actors (three male) dynamically expressing each of four emotions (happy, sad, angry, fearful) were selected from the University of Cambridge Mind Reading Emotions Library (adult level 6) (Golan et al., 2006). Five additional actors (two male) from the NKI community acted a neutral expression consisting of non-emotionally salient head/eye movements (left/right, up/down). Neutral videos were matched in size, resolution and luminance to emotion videos. For each video, representative single frames were extracted and used as corresponding static stimuli. Both dynamic and static stimuli were presented for 2 s each followed by a 400 ms interstimulus interval (ISI). In each of two ∼7.5-min fMRI scans, dynamic and static stimuli of a single emotion or neutral were delivered in 12-s blocks (5 stimuli per block), interleaved with 10-s of fixation-only. A total of ten blocks of static and 10 blocks of dynamic faces were presented in random order per scan (Figure 1A). Across both scans a total of 200 stimuli were delivered, 20 of each of four emotions plus neutral, either dynamic or static. To ascertain that participants were attending the stimuli, participants responded by button press to a single predesignated actor chosen randomly for each participant, irrespective of emotion or motion. The target actor appeared in ∼10% of all stimuli.

A forced-choice behavioral task was administered following the fMRI scan using the same static and dynamic emotional face stimuli (80 stimuli total, 20 of each type; neutral faces were not included). As in the fMRI scan, each static/dynamic stimulus was presented for 2 s. After each presentation, subjects were prompted to press one of five buttons to indicate if the actor’s expression was (1) happy, (2) sad, (3) angry, (4) fearful, or (5) none of the above. Accuracy, as opposed to response time was emphasized. The trial ended when subjects responded. To compare findings from our FER paradigm with those from a validated and reliable measure of FER, the Penn Emotion Recognition test (ER-40) (Taylor and MacDonald, 2012) was also administered to all participants and its results compared to those of the present FER paradigm. The ER-40 uses forty color photographs of faces expressing four basic emotions—happiness, sadness, anger, or fear—plus neutral—with eight photographs for each category, presented in random order. Participants were instructed to choose the correct emotion from among the five listed choices (forced choice) by clicking a computer mouse as quickly as possible without sacrificing accuracy. Each image was displayed until a choice was made.

Imaging took place on a Siemens 3T TiM Trio scanner. Two-hundred-twenty T2*-weighted echo-planar images (EPIs) (TR = 2,000 ms; TE = 30 ms; FA = 90°; FOV = 240 mm; slice thickness = 2.8 mm) were acquired on each of 36 contiguous slices in the axial plane. At least one high-resolution structural image of the entire brain was acquired from each participant using an MPRAGE sequence (TR = 2500 ms; TE = 3.5 ms; FOV = 256 mm, slice thickness = 1.0 mm, 192 slices).

Individual cortical surfaces were rendered from the high-resolution anatomical images using Freesurfer and registered to the std 0.141 fsaverage mesh (Fischl et al., 1999) with SUMA.¹ The pulvinar and amygdala were derived individually using a Bayesian atlas-based automated segmentation methods (Saygin et al., 2017; Iglesias et al., 2018; Bocchetta et al., 2020) incorporated in Freesurfer. Functional data were preprocessed and analyzed using the Analysis of Functional NeuroImages (AFNI) software (Cox, 1996; Saad and Reynolds, 2012). Preprocessing consisted of concatenating data from two runs, removal of signal deviation >2.5 SDs from the mean (AFNI’s 3dDespike), temporal alignment, identification of motion outliers per run and scaling of blood-oxygen-level-dependent (BOLD) values to mean percent signal change (Taylor et al., 2018). For surface-based analyses, the data was spatially smoothed with a 6 mm full width at half maximum Gaussian kernel. Single-participant statistical analyses were conducted within the framework of the general linear model (GLM). The GLM model included regressors for each stimulus type (emotional dynamic, emotional static, neutral dynamic, neutral static) as well as regressors for the six motion parameters (three rotations, three translations) and their first derivatives, per run. Time points with large head motion between successive time points were censored. Surface-based analyses were carried out on the gray-matter ordinates of each individual cortical surface aligned to the Freesurfer 141-standard mesh. Cortical data was sampled to the Human Connectome Project multimodal cortical parcellation (HCP-MMP1.0) (Glasser et al., 2016), resampled to fsaverage, which delineates 180 brain parcels per hemisphere based on functional and structural properties. To assess activation of pulvinar and amygdala, analyses were conducted in the individual native-space volumes. Primary analyses involved the entire pulvinar. In secondary analyses, beta parameters were extracted from pulvinar subdivisions and tested separately.

To avoid issues related to circularity in data analysis (Kriegeskorte et al., 2009), activated parcels were first identified by an unpaired t-test of mean activation (vs. 0), collapsed over all stimuli, across all seventy-eight participants thresholded at an (uncorrected) p-value of 0.001 (Figure 2A). This analysis defined a mask consisting of 35 bilateral parcels (70 parcels total) (Figure 2B) in the HCP MMP1.0 parcellation atlas which was used in subsequent analyses of functional data. Table 2 lists each parcel.

Mean beta parameter estimates were extracted from individual parcels and entered into an omnibus repeated measures analysis of variance (ANOVA) collapsing over all parcels identified as having significant across-group activation (n = 35 parcels per hemisphere, Table 2). Diagnostic group (control, Sz, ASD) was included as a between-subject factor. Face-motion type (dynamic, static) and face-emotion type (emotional, neutral) were included as within-subject factors. To minimize concerns regarding multiple comparisons, follow-up tests on individual parcels were conducted only if the initial group X parcel interaction was significant. Pre-planned subcortical regions (PulN, amygdala) were evaluated in separate ANOVAs with factors group, face-motion and face-emotion.

Effect sizes of between-group differences were calculated using Cohen’s d (mean/std dev).

The interrelationship between behavioral measures of FER and cortical/subcortical activation patterns was assessed by analysis of covariance (ANCOVA) with group membership as the categorical factor. Follow up ANCOVAs were conducted in a sequential fashion, for each significant covariate effect obtained in the prior analysis. In each case, only a single ANCOVA was performed for the region being tested as the dependent variable with all remaining covariates. The covariate x group interaction was used to evaluate group differences in the relationship between covariates.

All statistics were two-tailed with preset α level for significance of p < 0.05.

Based on our prior findings (Martinez et al., 2019), we used exploratory linear mediation analyses to explore the role of PulN subdivisions and cortical activation patterns in Sz patients. Analyses were conducted within SPSS26² using the PROCESS macro (Hayes, 2013). A three-variable path model (model 4) was used to examine the predictor-outcome relationship between interrelated regions with impaired activation in Sz (relative to controls) and the potential mediating role of each PulN subdivision (PL, PI, PM). As per standard conventions, the link between the predictor and mediator variable is referred to as path a, and that between the mediator and the outcome (controlling for the predictor), is path b. The overall predictor-outcome relationship is effect c, and the direct effect, after controlling for the mediator is, c′. The indirect (mediation) effect is the product of a*b and tests the significance of c – c′. Statistical significance of indirect pathways, reflecting the impact of mediation, was evaluated using a non−parametric bootstrap approach with 10,000 replication samples to obtain a 95% confidence interval (CI) (Preacher and Hayes, 2008). The mediation effects were considered statistically significant if the bootstrapped 95% CI did not include zero.

Overall, FER accuracy was significantly lower in both Sz [F(1, 56) = 15.02, p < 0.001] and ASD participants [F(1, 48) = 7.67, p = 0.009] compared to controls but did not differ between Sz and ASD [F(1, 46) = 0.03, p = 0.863] participants (Figure 1B).

Across groups, FER accuracy was equivalent for dynamic and static faces [F(1, 75) = 0.314, p = 0.576], however, the group x face-motion interaction was significant [F(2, 75) = 8.26, p < 0.001], reflecting relatively greater deficits for processing dynamic faces in Sz patients compared to controls [F(1, 56) = 11.70, p = 0.001; d = 1.4 dynamic faces; d = 0.82 static faces]. Consequently, mean FER accuracy for dynamic vs. static faces was significantly lower in the Sz group [t(54) = −2.65, p = 0.010] but did not differ within either the ASD [t(38) = 1.01, p = 0.317] nor control [t(58) = 0.55, p = 0.584].

FER accuracy did not differ overall as a function of specific face-emotion type [F(6, 146) = 0.84, p = 0.541] (Figure 1C).

As expected, ER-40 scores were also lower in both Sz [F(1, 56) = 36.58, p < 0.001] and ASD [F(1, 48) = 24.57, p < 0.001] subjects compared to control participants (Figure 1D). Further, across subjects, ER-40 scores significantly predicted mean accuracy on the FER task [F(1, 74) = 8.22, p = 0.005; R² = 0.407] (Figure 1E).

Lastly, behavioral performance on the identity recognition task during fMRI was marginally worse in Sz [F(1, 56) = 3.76, p = 0.058] and significantly worse in ASD [F(1, 48) = 4.52, p = 0.039] compared to controls.

The interrelationship between FER performance and cortical/subcortical activation patterns was assessed by ANCOVA. The results are summarized schematically in Figure 5A (see also Supplementary Table 2).

An initial omnibus analysis tested FER simultaneously against mean activation in all nine fMRI regions identified in the between-group fMRI analyses (seven cortical parcels, amygdala, PulN) (Supplementary Table 2A). This model tested the null hypothesis that no regions significantly predicted FER performance beyond the effect of group membership. The model incorporating these covariates (Adj. R² = 0.49) was statistically superior to a model incorporating group membership alone [Adj. R² = 0.35; F(9, 68) = 3.42, p = 0.0002] (Supplementary Table 2B) indicating that incorporation of these covariates significantly improved model fit.

The statistical contribution of the independent covariates was therefore considered in order to evaluate which regions contributed most to the overall model improvement. As expected, activation of the STSdp parcel most significantly predicted FER scores across participants [F(1, 66) = 16.61, p < 0.001]. Moreover, a model incorporating only STSdp as a covariate showed a model fit (Adj. R² = 0.51; Supplementary Table 2C) similar to that of the more complex model incorporating all covariates. In this simpler model, the relationship between STSdp and FER was highly significant [F(1, 74) = 26.24, p < 0.001]. In all groups, greater activation of STSdp correlated with improved behavioral performance (covaried by age and IQ) (Sz: r_p = 0.46, p = 0.018; ASD: r_p = 0.53, p = 0.020; control: r_p = 0.41, p = 0.029) (Figure 5B). By contrast, no significant correlation was observed between FER and the other covariates in the analysis.

The relationship between the nine fMRI covariates was assessed in follow-up ANCOVAs run in stepwise fashion and including interactions with group membership in the model. As these were not independent tests of the overall null hypothesis, they were not considered to increase family-wise error regarding potential predictors of FER impairments across groups. Rather, the goal was to determine stepwise contributions to impaired STSdp activation, which was shown in the omnibus test to significantly predict FER across groups.

Mean activation of STSdp was predicted across groups by both V1 [F(1, 51) = 6.19, p = 0.016] and TPOJ1 [F(1, 51) = 6.77, p = 0.012] (Figure 5C). In turn, V1 was predicted by PulN activity [F(1, 54) = 8.80, p = 0.004], and TPOJ1 was predicted by MST [F(1, 57) = 18.29, p < 0.001], which also predicted FEF activation [F(1, 61) = 6.63, p = 0.012]. There were no further significant interrelationships across groups.

An interaction involving group membership, indicative of different slopes, was obtained between STSdp and V2 [F(2, 51) = 5.95, p = 0.005] such that in ASD participants enhanced activation of V2 was associated with diminished STSdp activity (r_p = −0.71, p = 0.001) whereas in controls (r_p = 0.42, p = 0.023) and Sz (r_p = 0.44, p = 0.020) there was an inverse relationship (Figure 5D).

Lastly, group-specific interactions involving subcortical (PulN) activity were also obtained. Specifically, in Sz patients only, activation of STSdp [F(2, 51) = 4.51, p = 0.016], TPOJ1 [F(2, 54) = 4.48, p = 0.016], and MST [F(2, 63) = 3.01, p = 0.047] were all significantly and positively associated with PulN activation (Figure 5E).

A more detailed analysis of subcortical activation patterns in Sz was carried out with exploratory mediation analyses involving specific PulN subdivisions and interrelated cortical regions. Based on known anatomical interrelationships within the early visual system (Bridge et al., 2016) as well as the regression analyses described above, three specific predictor-outcome paths were evaluated (V1-STSdp, MST-TPJO1 and TPOJ1-STSdp) with PL, PM, or PI as potential mediators. The results are detailed in Table 3.

Consistent with known anatomical projections of the lateral subdivision, activation of V1 significantly predicted mean PL activity (p = 0.001), which in turn predicted STSdp activation (p = 0.005) (Figure 6A). After controlling for PL activity, V1 was no longer associated with STSdp (p = 0.694), however, using a bootstrapping approach, the (unstandardized) coefficient for the indirect pathway from V1 to STSdp (V1 → PL → STSdp) was significant (CI: [0.08, 0.84]), consistent with full mediation.

In addition, a significant mediating effect of PM was observed between TPOJ1 and STSdp (TPOJ1 → PM → STSdp; CI: [0.002, 1.13]) (Figure 6B) and of MST between PI and TPOJ1 (PI → MST → TPOJ1; CI: [0.13, 0.60]) (Figure 6C). In both cases, after controlling for the proposed mediator the direct paths between were not significant, suggesting full mediation.

PulN is divided into discrete anatomical subdivisions which mirror the dorsal/ventral stream distinction of visual cortex (Kaas and Baldwin, 2019) such that the more lateral regions (PL) project predominantly to primary visual cortex and ventral visual stream, whereas a subset of nuclei in the inferior subdivision (PI) project mainly to dorsal stream regions including motion-sensitive cortex (e.g., MST) (Arcaro et al., 2015; Tamietto and Morrone, 2016; Kaas and Baldwin, 2019). The medial subdivision (PM) is primarily connected with multimodal sensory association areas as well as prefrontal and cingulate cortices and has been tied to emotion processing (Homman-Ludiye and Bourne, 2019).

In the present study, in addition to intercorrelated cortical activation deficits, we observed correlations between cortical regions and PulN subnuclei. We therefore conducted a series of mediation analyses to evaluate underlying pathways.

An initial analysis evaluated the potential pathways underlying intercorrelated deficits between V1 and STS in the schizophrenia group. A significant indirect pathway from V1→PL→STSdp was observed, in support of indirect mediation by PL. Consistent with mediation by PM, the intercorrelation between the STS parcels (TPOJ1 and STSdp) was not significant once an indirect path via PM was modeled.

In contrast, PI appeared to mediate its effects via MST. Of note, unlike PL and PM, activation of PI was relatively intact in patients. Given that PI receives much of its driving inputs from neurons in the superior colliculus (Kaas and Baldwin, 2019), this finding is suggestive of unimpaired input via the retinotectal system. By contrast, the observed deficits in V1 are consistent with impaired input via the geniculostriate visual pathway (Martinez et al., 2008, 2019).

Although anatomical abnormalities in PulN are well-documented in schizophrenia (Dorph-Petersen and Lewis, 2017; Huang et al., 2020), the functional consequences of these abnormalities have, to date, remained relatively unexplored. Here, we provide evidence that impairments in visual PulN function significantly undermines visual processing required for effective face processing. In specific, deficits in PL function may mediate effects of impaired V1 activation, which, in turn likely reflects impaired magnocellular input to cortex. In addition, impaired PM activation mediated impaired input from more posterior (TPOJ1) to mid-STS (STSdp) regions, suggesting that accumulating deficits across successive cortico-pulvinar loops may lead to the large effect-sized deficits in FER-related reduced STS activation in Sz.

While deficits in social cognition are a prominent component of Sz, they are not unique to the disorder. In particular, we (Martinez et al., 2019) and others (Couture et al., 2010; Sasson et al., 2016) have reported FER deficits in adult ASD subjects that are as severe as those observed in Sz, despite the much higher level of overall function. Consistent with these prior results, adult ASD participants in the present study showed FER and STS activation deficits that were similar to those of Sz, supporting a role for STS as a common mediator of FER dysfunction across disorders.

Despite the similar STS impairments, ASD participants showed markedly different patterns of disturbance within early visual regions. In contrast to Sz patients, markedly increased responses and an opposite slope of the relationship between V2 and STSdp activation were observed in the ASD group. Activation within other task-activated visual regions, including V1 and MST, was unaffected in ASD, echoing our recent study in which response amplitudes were also normal within V1, but increased in early visual and dorsal visual regions (Martinez et al., 2019). Similar visual hypo/hyper activation patterns in Sz vs. ASD have been observed in both fMRI (reviewed in Samson et al., 2012; Taylor et al., 2012) and electrophysiological (Martinez et al., 2018, 2019; Shah et al., 2018; Kovarski et al., 2019) studies, supporting the concept that dysregulation of the early visual system may undermine later stages of visual processing.

Patterns of amygdalar and subcortical activation also distinguished between ASD and Sz participants. In the amygdala, activation was elevated overall in Sz (but not ASD), as reported previously (reviewed in Dugre et al., 2019), possibly reflecting abnormal salience attribution to neutral stimuli (Holt et al., 2006; Gur et al., 2007; Premkumar et al., 2008), heightened anxiety (Du and Grace, 2016) and/or paranoid ideation (Pinkham et al., 2015).

In contrast, whereas PulN activity was markedly reduced in Sz, activation of the inferior PulN subdivision was significantly elevated in ASD participants, in line with findings from our previous studies (Martinez et al., 2018, 2019) and those of others (Zurcher et al., 2013; Hadjikhani et al., 2017; Martinez et al., 2019). Although the source of the increased activation in ASD is not known, a parsimonious explanation would be hyperactivity of the subcortical retino-collicular pathway, which provides preferential input to PI (Kaas and Baldwin, 2019) and which, in turn, acts like a driver to V2 (de Souza et al., 2020). In humans, this system typically weakens with age as the retinogeniculate system increases in functionality (reviewed in Bourne and Morrone, 2017). Abnormal persistence of this system into adulthood could thus underlie the activation disturbance pattern observed in ASD.

Regardless of underlying pathophysiology, these findings support the concept that dysregulation of the early visual system, whether in the direction of increased or decreased activation, may undermine later stages of visual processing and further highlight the importance of sensory processing abnormalities to the pathophysiology of social cognitive impairment across neuropsychiatric disorders.

Despite our differential findings, some limitations must be considered. First, Sz participants were receiving antipsychotic medication which may have impacted measures on brain activity. We did not observe any correlations with medication dose, however, this issue could be best addressed in future studies involving medication-naïve patients. Moreover, we note that the overall sample size remains small, and the findings need to be replicated in an independent sample. Additionally, we did not track fixation locations either in the scanner or during behavior. Thus, we do not know if activation failures relate to inability to process information, or simply from differential facial scanning approaches. Lastly, Sz participants had lower IQ and were older than controls or ASD participants, although correcting for IQ and age did not diminish the findings.