Fifteen paid volunteers were initially recruited for three experiments. The sample size was selected to match that of previous studies with similar methods (Roberts et al., 2019). One participant declined to participate in one of the three experiments and was excluded from the study. Analysis of the data from the remaining fourteen participants showed that three participants did not achieve above-chance orientation decoding accuracy at any time point, making a cross-decoding analysis (see below for more details) infeasible for these participants. Consequently, these four participants were excluded from the study and the results for the remaining eleven participants is presented (age: mean ± SD = 25.0 ± 3.84 years) [Similar individual differences in EEG decoding accuracy have also been reported in previous studies (Simanova et al., 2010)]. All participants had normal or corrected-to-normal vision. All experiments were approved by the Ethics Committee of the University of Tokyo and conducted in accordance with the guidelines of the Declaration of Helsinki. Written informed consent was obtained from all participants.

Six Gabor patterns with a diameter of 4.8 deg. (spatial frequency: 2.0 c/deg.; Michelson contrast: 0.99) were presented against a gray background with an average luminance of 109.1 cd/m² (Figure 1). The orientation of the Gabor patterns was determined according to the aim of each experiment, as explained in the following section, and each Gabor had a random phase on each trial. The center of the six Gabor patterns was located at a circle 5.8 deg away from the fixation point. The stimulus was presented on a 24-inch LCD monitor (BenQ GW2480B) at a viewing distance of 100 cm.

Two types of experiments (Experiments 1 and 2) were conducted. Experiment 1 comprised of an EEG experiment with Gabor patterns of a single uniform orientation. Participants were presented with six Gabor patterns for 500 ms (Figure 1, left panel, Experiment 1). After a blank period of 500–800 ms, the participants were asked to judge whether the patterns were tilted clockwise or counterclockwise relative to the vertical by pressing one of two buttons. The next trial was initiated 1,500 ms after the response. All six patterns had the same orientation, which was randomly determined in each trial from six orientations ranging between −75° to 75° in 30° increments, with 0° corresponding to the vertical. The response keys were randomly alternated between blocks. Participants completed 8 blocks of 120 trials, apart from one participant who only completed 6 blocks due to time constraints.

Experiment 2 consisted of EEG and behavioral experiments (Experiments 2A and 2B, respectively) with Gabor patterns with multiple orientations. In both Experiments 2A and 2B, participants were presented with a stimulus set of six Gabor patterns with different orientations (Figure 1, top center and right panels, Table 1: four stimulus sets used in the experiment). Two stimulus sets had an average orientation of 45° while the others possessed an orientation of −45°. As we were interested in how the decoding of the average orientation (45° or −45°) is affected by the presence of 45° or −45° patterns in the set, we created stimulus sets with and without elements corresponding to the average (w and w/o, respectively) for each average orientation. In each trial, participants were randomly presented with one of the sets for 500 ms. The position of each element in the set varied randomly between trials.

In Experiment 2A, the participants’ task was to indicate with a button whether the average orientation of the set was tilted clockwise or counterclockwise relative to the vertical (Figure 1, center panel). Note that in both Experiments 1 and 2, we instructed participants not to attend to only a subset of the six patterns but to the entire display to ensure that participants did not explicitly change their strategy depending on the task. Moreover, the two EEG experiments (Experiments 1 and 2A) were conducted on the same day. These facts ensure the validity of our cross-decoding analysis (see below).

In Experiment 2B, participants were asked to report perceived average orientation by adjusting a white rotating bar presented at the center of the monitor (Figure 1, right panel) instead of a binary response. Note that participants were instructed to accurately report the average orientation they initially perceived and not to change it while rotating the bar. There was no response time limit; however, the mean response time was 3,627 ± 813 (mean ± SD) ms across participants, indicating that participants responded quickly based on our instruction, given the time needed to align the bar with their perceived average orientation. Also, we used four additional stimulus sets that served as dummy stimuli as well as the ones listed in Table 1: stimulus set to prevent participants from being aware that they were always presented with only a few stimulus sets and reporting almost the same orientation on all trials. The components of the dummy stimuli are as follows: (−5°, −5°, −35°, −35°, −65°, −65°), (5°, 5°, 35°, 35°, 65°, 65°), (30°, 30°, 45°, 45°, 60°, 60°), and (−30°, −30°, −45°, −45°, −60°, −60°). The trials in which the dummy stimuli were presented, or the sign of the response was opposite to that of the true average (e.g., a trial with a response of −20° when the set had an average of 45°) were excluded from the following analyses. Experiment 2B was conducted on an average of 21 days after the EEG experiments. Participants completed 4 blocks of 120 trials in Experiments 2A and 2B, except for one participant who only completed 2 blocks in Experiment 2A due to time constraints.

In Experiments 1 and 2A, the EEG signals were recorded at a sampling rate of 1,000 Hz from 32 electrodes (BrainVision Recorder, BrainAmp Amplifier, EasyCap, BrainProducts) located at FP1, FP2, F3, F4, F7, F8, Fz, T7, T8, C3, C4, Cz, FC1, FC2, FC5, FC6, P3, P4, P7, P8, Pz, TP9, TP10, CP1, CP2, CP5, CP6, PO3, PO4, O1, O2 and Oz. An electrode located between Fz and Cz served as a reference. The impedance of all electrodes was kept below 5 kΩ throughout the experiments.

We preprocessed the raw EEG signals with the EEGLAB toolbox for MATLAB. The raw EEG signals were downsampled to 500 Hz, band-pass filtered between 1 and 80 Hz, and epoched from 1,000 ms before to 1,500 ms after stimulus onset. The epochs were visually inspected to remove trials containing transient muscular activity and electrodes with persistent noise. Epochs with incorrect responses were also removed. Next, independent component analysis (ICA) was performed and the ICLabel plugin (Pion-Tonachini et al., 2019) was used to automatically obtain estimated labels for each component. Components with an estimated probability of more than 50% being artifacts (i.e., a label of eye, muscle, heart, line noise, or channel noise) were removed, resulting in an average of 21 ± 2.5 (mean ± SD) components across all participants and experiments. Epochs were baseline corrected with respect to the pre-stimulus period from −300 to 0 ms. This procedure resulted in 922 ± 136 (mean ± SD, Experiment 1) and 444 ± 150 (mean ± SD, Experiment 2A) trials for further analyses.

To test if orientation could be decoded from EEG signals for our stimulus setting, orientation decoders were constructed in a time-resolved manner for each participant using potentials within a 100 ms time window and from ten occipital and parietal electrodes (P3, P4, P7, P8, Pz, PO3, PO4, O1, O2, Oz) recorded in Experiment 1 (Figure 2). These ten electrodes were selected in accordance with a previous study (Roberts et al., 2019). We trained two linear SVM classifiers (with a regularization parameter C of 1) for each participant to discriminate between three orientations included in each stimulus set of Experiment 2 (i.e., 75°/45°/15° classifier or −75°/−45°/−15° classifier). Specifically, the number of trials were first matched for each orientation in Experiment 1 through undersampling. The spatiotemporal EEG patterns for each orientation were then divided into 5 sets and the patterns within each set were averaged for each orientation to increase the signal-to-noise ratio, thus decoding accuracy (Grootswagers et al., 2017). Four sets were used as training data while the remaining set was used as test data. The averaged signal within each set was z-scored using the mean and standard deviation of the training data over −300 ms to 900 ms relative to stimulus onset, in accordance with the procedure used in a previous study (Hermann et al., 2022). Subsequently, a linear multiclass support vector machine (SVM) classifier was trained on the training data (12 samples = 4 sets × 3 orientations) and tested on the remaining 3 samples (1 set × 3 orientations). This procedure was repeated 5 times until all sets had served as test data (5-fold cross-validation). The entire procedure was repeated 20 times with samples randomly assigned to each set. This resulted in 100 decoding accuracies, averaged together to obtain a single decoding accuracy for a single time window. The 100 ms time window was shifted by a time step of 2 ms and the same training and testing procedure was performed, resulting in time courses of 3-class orientation decoding accuracy over −300 ms to 900 ms relative to stimulus onset (Figures 2, 3A).

We also computed three posterior probabilities of the three orientations (75°/45°/15° or −75°/−45°/−15°) from each decoder to confirm whether all six orientations were decodable. We defined SVM decision function values (obtained from sklearn.svm.LinearSVC.decision) passed through a softmax function as posterior probabilities for the test data. This procedure was again performed 100 times (5-fold cross-validation × 20 times) to generate 300 (=3 orientations × 100 repetitions) probabilities, which were averaged over 100 repetitions to obtain three posterior probabilities for each orientation. Time courses of posterior probabilities were obtained for three orientations by shifting the time window and repeating the same procedure. For further analyses, all 32 electrodes and different types of classifiers were used to construct decoders, and similar results were obtained (Supplementary Figures S1, S2).

After we confirmed that EEG signals contained orientation information, we adopted a reasonable computational model to estimate when ensemble representations emerge during an ensemble judgment task based on previous studies (Oh et al., 2019). Specifically, we constructed inverted encoding models (Brouwer and Heeger, 2009, 2011; Sprague et al., 2015) to decode orientation representations at each time point.

This model is based on a biologically plausible assumption that EEG signals can be modeled as a linear weighted sum of population responses of multiple orientation-selective neurons in the brain. Following previous studies (Oh et al., 2019; Tark et al., 2021), we assumed an equal number of orientation channels as orientations used in the experiments, each tuned to a specific orientation (i.e., 75°, 45°, 15°, −75°, −45°, −15°). The response of each channel is described as: (1) R = cos 7 θ − θ max where θ is an orientation presented to the channel and θ max is an optimal orientation that maximizes the response of each channel. Next, we modeled EEG signals at each electrode as a linear weighted sum of the six hypothetical orientation channel responses. Let m be the number of electrodes, n be the number of experimental conditions (stimulus sets), and k be the number of hypothetical orientation channels (six, in our case). The matrix of EEG signals to each stimulus set B ∈ R m × n is equal to the multiplication of a weight matrix W ∈ R m × k and the matrix of hypothetical orientation channel responses C ∈ 0 1 k × n (Figure 4A, top): (2) B = W C

The response matrix C has response values (defined in Eq. 1) corresponding to the presented orientation in each column (Figure 4B). First, we estimated the weight matrix W using EEG data from Experiment 1 and the channel response matrix C. We then applied the estimated weight matrix to the EEG data from Experiment 2A to compute the channel response matrix for stimulus set w and w/o (Figure 4A bottom). This enabled us to investigate whether and when representations corresponding to the average orientation (i.e., 45°/−45°) emerge in the brain during orientation ensemble perception.

We performed 5-fold cross-validation as in the first decoding analysis to estimate the weight matrix W for our EEG data (see Decoding orientations with linear classification). The training data B 1 from Experiment 1 (EEG signals averaging within the 100 ms time window) and the response matrix C 1 were used to compute the least-squares estimate of a weight matrix: (3) W ̂ = B 1 C 1 T C 1 C 1 T − 1

The estimated weight matrix Ŵ was then applied to the test EEG data B 2 from Experiment 2A to estimate six orientation channel responses. (4) C 2 ̂ = W ̂ T W ̂ − 1 W ̂ T B 2

This procedure was repeated 100 times with random data partition for each participant, and the estimated response matrices were averaged across all folds and repetitions. As the estimated response matrix reveals the response strength for each hypothetical channel to each stimulus set, we can use this matrix to quantify the representational strength of each orientation, especially the average orientation. Repeating this procedure for all time points yields the time courses of the representational strength of each orientation for each stimulus set.

To quantify the strength of representations corresponding to the average orientation, we extracted a representative value, referred to as orientation sensitivity, from the estimated responses of each channel, following the approach adopted in previous studies (Oh et al., 2019; Sutterer et al., 2019). Specifically, we first circularly shifted the responses of orientation channels along the rows such that the responses of the channel tuned to the average orientation aligned at the center (fourth row of the matrix). Subsequently, we averaged the response of orientation channels along the columns (within the stimulus set), which yielded two vectors of six elements for each stimulus set type (w and w/o). Each vector had the central (fourth) value representing the strength of the representation corresponding to the average orientation of the stimulus set and the adjacent values representing the strength of representations corresponding to orientations ±30°, ±60°, and −90° away from the average orientation. Furthermore, we performed linear regression on these six channel response values, disregarding the sign of orientation (fitting a line to the response values for −90°, −60°, −60°, −30°, −30°, 0°). We defined the slope of the fitted line as orientation sensitivity. Positive orientation sensitivity indicates the presence of a peak in the channel responses, suggesting strong representations corresponding to the average orientation for Experiment 2; in contrast, zero sensitivity indicates no representations corresponding to the average orientation.

To gain further insights into when orientation representations corresponding to the participants’ perceived average orientation emerge in the brain, we used the estimated channel responses to decode orientation continuously at each time point using the same approach as a previous study (Brouwer and Heeger, 2009). This analysis included computing six channel responses ( R in Eq. 1) to 180 orientations ranging from −89° to 90°, correlating these responses with the estimated responses in the cross-decoding analysis, and choosing the orientation that showed the highest correlation among those with the same sign as the average orientation of the stimulus set. Subsequently, these decoded orientations were tested to assess the correlation with participants’ perceived average orientations in Experiment 2B. A higher correlation at one time point makes it more likely that the ensemble perception was fully formed in the brain at that time point. Note that the correlations for stimulus sets w and w/o were calculated by collapsing the data over the true average orientation (45° or −45°). Specifically, the true average orientation was subtracted from both the estimated orientation and the perceived average orientation. Then, the correlation between these two values was computed for each time point, resulting in two-time courses of correlation for stimulus sets w and w/o.

We assessed the statistical significance of orientation decoding accuracy using a cluster-based permutation test. The orientation labels for the EEG data were permuted and the orientation decoding accuracy was recalculated according to the procedure described above. After repeating this procedure 1,000 times, null distributions of decoding accuracy were generated for each time window, from which the p-values for the actual decoding accuracy were calculated. The original decoding accuracy curve was thus transformed into a time course of p-values. Clusters were defined as neighboring time points at which all p-values were below a significance level, and deemed significant if their size exceeded the calculated threshold from a null distribution of cluster size. The same procedure was used to assess the statistical significance of sensitivity in the estimated orientation channel responses.

Regarding the correlation coefficients between the perceived average orientation and the estimated orientation representation, we again permuted the orientation labels for the EEG data and constructed inverted encoding models to estimate the weight matrices for each participant. These matrices allowed us to obtain orientation channel responses and estimated orientation at each time window as described above. We computed the correlation coefficient between the estimated orientations and perceived average orientations that participants reported. Repeating this procedure 1,000 times resulted in null distributions of the correlation coefficients for each time window. We again performed a cluster-based permutation test to assess the statistical significance of the observed correlation coefficients for each stimulus set.

We first tested whether the uniform orientation of six Gabor patterns could be reliably decoded from EEG signals. Specifically, orientation decoders were trained and tested using EEG signals (from ten occipital and parietal electrodes) elicited by six Gabor patterns with identical orientations (Figure 1, left panel). Participants successfully discriminated the orientation of the patterns (clockwise or counterclockwise inclined relative to the vertical) in most trials with a mean accuracy of 98.4% (SD = 0.02%). An orientation decoder was constructed using EEG signals within a 100 ms time window that was shifted in 2 ms steps, resulting in a time course of orientation decoding accuracy (Figure 2). Note that two types of decoders were constructed, each discriminating orientations contained in the stimulus set used in Experiment 2 (i.e., 75°/45°/15° decoders or −75°/−45°/−15° decoders). Figure 3A shows that decoding accuracy reached significance at 174 ms and remained above chance until 828 ms after stimulus onset (cluster-based permutation test; p < 0.05 cluster-defining threshold; p < 0.05 cluster-threshold).

To test whether all six orientations were reliably decodable, the posterior probabilities of each orientation were calculated for each stimulus set by using the output of the decoder (see Methods). As can be seen in Figure 3B, for each stimulus set, we obtained the highest probability for the presented orientation throughout the stimulus presentation (e.g., the probability of 75° was higher than that of 45° and 15° when 75° was presented), ensuring that all six orientations were reliably decoded.

The results of Experiment 1 showed that the orientation information was reliably decoded from the EEG signals. Next, we constructed an encoding model that predicts EEG signals by assuming biologically plausible orientation channels to examine when ensemble representation is formed in the brain during ensemble perception. This model was then inverted to decode the temporal evolution of the representational strength of multiple orientations (inverted encoding models) (Brouwer and Heeger, 2009, 2011; Sprague et al., 2015; Oh et al., 2019; Sutterer et al., 2019) from EEG signals in Experiment 2. In this experiment, we utilized six Gabor patterns with two or three different orientations and an average of 45° or −45° (Table 1). For each average orientation, two stimulus sets, with and without elements corresponding to the average (w and w/o), were created to see whether the temporal dynamics of the representational strength of each orientation were affected by the presence of the average orientation pattern in the set. Participants were asked to indicate whether the average orientation was tilted clockwise or counterclockwise relative to the vertical with a binary response.

The critical idea of inverted encoding models is to assume multiple orientation channels (derived from populations of multiple orientation-selective neurons in the brain) and model EEG signals at each electrode and time point as a linear weighted sum of the channel responses (Figure 4B). This analysis is explained in detail in the Methods and briefly summarized as follows: we first divided the EEG data into training and test datasets, consistent with our initial decoding analysis. We then estimated a weight matrix for six hypothetical orientation channels corresponding to the six orientations used in Experiment 1 (Figure 4A, top). This weight matrix was subsequently applied to the EEG data from Experiment 2A to estimate the responses of the hypothetical orientation channels during ensemble perception (Figure 4A, bottom). Importantly, these responses can be considered as the representational strength of each orientation at a given time point. We repeated this procedure for all time points and participants to investigate how orientation representation evolves over time during orientation ensemble perception. We hypothesized that if ensemble representation is explicitly formed in the brain at a specific time point, a significantly high response of the hypothetical orientation channel tuned to the average orientation (45° or −45°) would be observed at that time, allowing us to estimate the exact time the ensemble perception was formed.

We estimated the hypothetical orientation channel responses for the EEG data from Experiment 2A, wherein EEG signals were recorded while participants discriminated the average orientation of the set (2AFC discrimination task, Figure 1, right panel; behavioral performance averaged 92.6%, SD = 0.10%). Note that participants were instructed to attend to the entire display during the task to minimize the possibility that they merely attended to only one element to achieve such high accuracy. The color maps in Figure 5A show the estimated responses of hypothetical orientation channels for each time point. The central row of the map represents the responses of the channel tuned to the true average orientation (45°, −45°), indicating the representational strength of the true average orientation in EEG signals. We found that for both sets w and w/o, the representation of the true average orientation was high between 400 and 700 ms after stimulus onset. To quantitatively evaluate the strength of the representation of the true average orientation (the degree of the peak in the channel response profile), we collapsed the response values across the signs of the orientations and fitted a line to them to estimate its slope, which we refer to as orientation sensitivity (Oh et al., 2019; Sutterer et al., 2019). Figure 5B shows that significantly high orientation sensitivity was observed in 376–474 ms and 592–732 ms for set w, and 374–442 ms and 512–672 ms for set w/o, indicating the emergence of ensemble representation at later time points. These findings reveal that ensemble representation emerges at 400–500 ms during the perception of orientation ensembles and is sustained till approximately 700 ms after stimulus onset, regardless of the presence of the average orientation in the stimulus set.

To more precisely estimate the time at which ensemble representation is fully formed in the brain, we computed the neural correlate of the participants’ perceived average orientation. This is based on our idea that a high correlation between the decoded orientation representation and the perceived average orientation at a specific time point indicates the complete formation of ensemble representation corresponding to an individual’s perception. Therefore, to precisely measure the average orientation perceived by each participant, we conducted another psychophysical experiment using adjustment methods in which participants indicated the perceived average orientation on a continuous scale by rotating a bar stimulus (Experiment 2B; Figure 1, right panel). Figure 6 shows the perceived average orientations for each participant and stimulus set. The perceived orientations varied from participant to participant; however, they were mostly close to the true average, indicating that participants perceived the average orientation quite well.

Furthermore, we computed the correlation between the participants’ average orientations in Experiment 2B and orientation decoded from the estimated hypothetical channel responses at each time point (see Methods for further details). Figure 7 shows the time courses of the correlations. Despite unstable correlation coefficients observed in the earlier period, which are likely due to the relatively small number of participants, a cluster-based permutation test revealed significant correlations at 568–668 ms for set w and slightly later time points (640–698 and 722–766 ms) for set w/o. These results suggest that ensemble representation corresponding to the individual perception is fully formed at 600–700 ms after stimulus onset.