Classical models of online gradient descent learning in high dimensions can be often be summarized with simple summary statistics (Watkin et al., 1993; Engel and van den Broeck, 2001; Ben Arous et al., 2022; Arnaboldi et al., 2023; Goldt et al., 2019, 2020; Biehl and Schwarze, 1995; Saad and Solla, 1995). In this section, we discuss how the generalization performance of perceptrons and shallow (two-layer) neural networks trained on large quantities of high dimensional data can be summarized by simple weight alignment measures. Most simply, the perceptron model f(x)=σ(1Dw·x) seeks to learn a weight vector w∈ℝ^D which correctly classifies a finite set of randomly sampled training input-output pairs (x_μ, y_μ). If the inputs are random, xμ~N(0,ID), and the targets y_μ = y(x_μ) are generated by a teacher network y(x)=σ(1Dw⋆·x), then the generalization performance (performance of the model on new unseen data, Ex[(f(x)-y(x))2]) is completely determined by the overlap of w with itself and with the target direction w_⋆

If the learning rate is scaled appropriately with the dimension D, the high-dimensional (large-D) limit of online stochastic gradient descent is given by a deterministic set of equations for Q and R:

where the continuous training “time” τ is the ratio of the number of samples seen to the dimension and F:ℝ² → ℝ² is a nonlinear function that depends on the learning rate, the loss function, and the link function σ(·) (Engel and van den Broeck, 2001; Ben Arous et al., 2022; Arnaboldi et al., 2023; Goldt et al., 2019; Saad and Solla, 1995). Integrating this update equation allows one to predict the evolution of the generalization error as more training data are provided to the algorithm. Despite the infinite dimensionality of the original optimization problem, only two dimensions are necessary to capture the dynamics of generalization error.

The analysis of online perceptron learning can be extended to two layer neural networks with a small number of hidden neurons N,

In this setting with isotropic random data, the relevant summary statistics are the readout weights a∈ℝ^N, along with overlap matrices Q∈ℝ^N×N and R∈ℝ^N×K with entries

For this system, we can track the gradient descent dynamics for a, Q, and R through a generalization of Equation 2 (Goldt et al., 2019; Saad and Solla, 1995; Biehl and Schwarze, 1995; Goldt et al., 2020). This reduces the dimensionality of the dynamics from the N+DN trainable parameters {a_i}, {w_j} to N+N²+NK summary statistics, which is significant when D≫N+K. This reduction enables the application of analyses that cannot scale to high dimensions, for instance control-theoretic methods to study optimal learning hyperparameters and curricula (Mori et al., 2025; Mignacco and Mori, 2025). Recent works have also begun to study approximations to these summary statistics when the network width N is also large, as further dimensionality reduction if possible when Q and R have stereotyped structures (Montanari and Urbani, 2025; Arnaboldi et al., 2023).

Under what conditions is this reduction possible? Fundamentally, the summary statistics a, Q, and R are sufficient to determine the network's performance so long as the preactivations h_i and hk⋆ are approximately Gaussian. Thus, one can relax the assumption that the inputs x are exactly Gaussian so long as a central limit theorem applies to h_i and hk⋆ (Goldt et al., 2019, 2020). Moreover, one can allow for correlations between the different input dimensions so long as h_i and hk⋆ remain Gaussian. If E[xx^⊤] = Σ, with a modification of the definition of the overlaps to Qij=1Dwi·Σwj and Rik=1Dwi·Σwk⋆ a similar reduction applies (Arnaboldi et al., 2023). One can even consider extensions to plasticity rules other than stochastic gradient descent. For example, online node perturbation leads to a different effective dynamics for the same set of summary statistics (Hara et al., 2011, 2013).

How could the overlaps Q and R be accessed from measurements of neural activity? And, in the absence of detailed knowledge of a teacher network, how could one identify the relevant overlaps? Under the simple structural assumptions of these models, one could estimate the overlaps from covariances of network activity across stimuli, i.e., with isotropic inputs one has Ex[hihk⋆]=Rik and E_x[h_ih_j] = Q_ij. Moreover, one can in some cases detect this underlying low-dimensional structure by examining the principal components of the learning trajectory (Ben Arous et al., 2023). However, more theoretical work is required in this vein.

Another strategy to reduce the complexity of multilayer deep neural networks is to analyze the dynamics of learning in terms of representational similarity matrices (kernels) for each hidden layer of the network. Consider, for example, a deep fully-connected network with input x∈ℝ^D,

where t denotes training time. Instead of using online stochastic gradient descent to train the weights as we did in the preceding section, suppose we use gradient flow to minimize the average error on a fixed set of training examples. Moreover, instead of considering a regime where the hidden layer width N is small relative to the input dimension D, let us now consider very wide networks with N≫D (Figure 1a).

What are the relevant summary statistics in this case? Applying the chain rule to the dynamics of the network outputs, one finds the differential equation

where L is the loss function and Ex′ denotes expectation over the training dataset (Jacot et al., 2018; Lee et al., 2019; Bordelon and Pehlevan, 2023b). Here,

are representational similarity matrices, and

are gradient similarity matrices, which respectively compare the hidden states ϕ(hi(ℓ)(x,t)) and the gradient signals gi(ℓ)(x,t) at each hidden layer ℓ for each pair of data points (x, x′) and each pair of training times (t, t′). Thus, as Φ^(ℓ) and G^(ℓ) determine the dynamics of f, these matrices are suitable summary statistics of learning if they are low-dimensional relative to the set of synaptic weights, and if we can write down a closed set of equations for their dynamics.

First, it is easy to see that the criterion of dimensionality reduction requires that the number of training examples P is much less than the network width N, as the number of similarity matrix elements and the number of synaptic weights are of order P² and N², respectively. Second, it turns out that one can close the equations for Φ^(ℓ) and G^(ℓ) provided that the width is large and that the synaptic weights start from an uninformed initial condition (i.e., Gaussian random matrices) (Jacot et al., 2018; Lee et al., 2019; Yang and Hu, 2021; Bordelon and Pehlevan, 2023b). Depending on how weights and learning rates are scaled, one can obtain different types of large-width (N → ∞) limits (Figure 1b). In the lazy/kernel limit where γ is constant, these representational similarity matrices are static over the course of learning (Jacot et al., 2018; Lee et al., 2019). However, an alternative scaling (γ∝N) can be adopted where these objects evolve in a task-dependent manner even as N → ∞ (Figure 1c) (Yang and Hu, 2021; Bordelon and Pehlevan, 2023b).

While this provides a description of the training dynamics of a model under gradient flow, one can extend this description in terms of similarity matrices to other learning rules which use approximations of the backward pass variables g~i(ℓ)(x,t), which we called pseudo-gradients in Bordelon and Pehlevan (2023a). Such rules include Hebbian learning, feedback alignment, and direct feedback alignment (Hebb, 2005; Lillicrap et al., 2016; Nøkland, 2016). In this case, the relevant summary statistics to characterize the prediction dynamics of the network include the gradient-pseudogradient correlation, which measures the alignment between the gradients used by the learning rule and the gradients that one would have used with gradient flow,

as G~(ℓ) governs the evolution of the function output:

The summary statistics encountered in Section 3 are robust to sub-sampling thanks to their basic nature as averages over the population of neurons. These statistical theories in fact post a far stronger notion of benign sub-sampling: they result in neurons that are statistically exchangeable. This is highly advantageous from the perspective of long-term recordings of neural activity, as reliable measurement of summary statistics does not require one to track the exact same neurons over time. Instead, it suffices to measure a sufficiently large subpopulation on any given day. This obviates many of the challenges presented by tracking neurons over multiple recording sessions (Masset et al., 2022). Moreover, the variability and bias introduced by estimating summary statistics from a limited subset of relevant neurons can be characterized systematically (Kang et al., 2025; Bordelon and Pehlevan, 2024; Zavatone-Veth et al., 2022a). Taken together, these properties mean that summary statistics are relatively easy to estimate given limited neural measurements, provided that exchangability is not too strongly violated (Gao et al., 2017). We will return to this question in the Discussion, as a detailed analysis of the effects of non-identical neurons will be an important topic for future theoretical work. There are limits, however, to how far one can sub-sample. For instance, representational similarity kernels are more affected by small, coordinated changes in the tuning of many neurons than large changes in single-neuron tuning (Figure 2) (Kriegeskorte and Wei, 2021). Determining the minimum number of neurons one must record in order to predict generalization dynamics across learning will be an important subject for future theoretical work (Gao et al., 2017; Kriegeskorte and Wei, 2021).

Though by our definition the summary statistics mentioned in Section 3 are sufficient to predict the network's performance, they are not sufficient statistics for all properties of the neural code. In particular, in part because they arise from theories in which neurons become exchangable, they have many invariances. These invariances mean that individual tuning curves can change substantially without altering the population-level computation (Kriegeskorte and Wei, 2021). For instance, the representational similarity kernels are invariant under rotation of the neural code at each layer, enabling complete reorganization of the single-neuron code without any effect on behavior. Similarly, overlaps with task-relevant directions are invariant to changes in the null space of those low-dimensional projections. These invariances mean that focusing on summary statistics of learning sets a particular aperture on what aspects of representations one can assay.

At the same time, the invariances of summary statistics have important consequences for functional robustness. In particular, they are closely related to theories of representational drift, the seemingly puzzling phenomenon of continuing changes in neural representations of task-relevant variables despite stable behavioral performance (Rule et al., 2019; Masset et al., 2022). Many models of drift explicitly propose that representational changes are structured in such a way that certain summary statistics are preserved (Figure 2a) (Masset et al., 2022; Pashakhanloo and Koulakov, 2023; Qin et al., 2023). Identifying the invariances of the summary statistics sufficient to determine task performance can allow for a more systematic characterization of what forms of drift can be accommodated by a given network. Conversely, identifying the invariances of a representation once task performance stabilizes might suggest which summary statistics are relevant for the learning problem at hand.

An important lesson from the theory of high-dimensional statistics is that of universality: certain coarse-grained statistics are asymptotically insensitive to the details of the distribution. The most prominent example of statistical universality is the familiar central limit theorem: the distribution of the sample mean of independent random variables tends to a Gaussian as the number of samples becomes large. A broader class of universality principles arise in random matrix theory: the distribution of eigenvalues and eigenvectors of a random matrix often become insensitive to details of the distribution of the elements as the matrix becomes large. Most famously, the Marčenko-Pastur theorem specifies that the singular values of a matrix with independent elements have a distribution that depends only on the mean and variance of the elements (Marchenko and Pastur, 1967). In the context of learning problems, universality manifests through insensitivity of the model performance to details of the distributions of parameters or of features (Hu and Lu, 2022; Misiakiewicz and Saeed, 2024).

From the perspective of summary statistics, statistical universality can allow simple theories to make informative macroscopic predictions even if they do not capture detailed properties of single neurons. For instance, the mean-field description of the learning dynamics of wide neural networks introduced in Section 3 are universal in that they depend on the initial distribution of hidden layer weights only through its mean and variance, even though the details of that distribution will affect the distribution of weights throughout training (Figures 2b–d) (Golikov and Yang, 2022; Williams, 1996). Like the invariances to transformations of the neural population code mentioned before, this is nonetheless a double-edged sword: these universality properties mean that focusing on predicting performance commits one to coarse-graining away certain microscopic aspects of neural activity. Though these features are not required to predict macroscopic behavior, they may be important for understanding biological mechanisms.