We tested the robustness of the network by creating a series of testing datasets that were based on the contour stimulus (Figure 3). We tested network tolerance under fragmented border conditions, as objects in natural scenes are often displayed as discontinuous boundaries due to occlusion or fragmentation caused by shadow or texture. Nevertheless, BOS-selective neurons continue to respond when dealing with disrupted borders (Zhang and von der Heydt, 2010), which indicates the ability of biological vision to merge separate visual elements into single border representations. To probe whether CNNs exhibit the ability to preserve BOS assignment under fragmented conditions, we systematically designed two classes of fragmented figures to examine network performance.

For the first class of condition, we produced 10 groups of figures with discontinuous borders by dividing the contours of rectangles from the contour stimuli group into evenly spaced segments with increasing intervals (Figure 3A). We did not fragment the solid stimuli group since fragmenting filled shapes would introduce ambiguity about whether removed regions should be interpreted as holes or as changes in surface appearance rather than as missing contour information. The degree of interval was controlled by a border-to-gap ratio, ranging from 5 border pixels with 1 gap pixel ( n = 1 5 ) to 1 border pixel with 5 gap pixels ( n = 5 ). These figures were constructed using the same procedure as in the training set, but with their contours modified into dashed outlines.

For the second class, we used methods similar to Zhang and Von Der Heydt (2010), where the rectangle was divided into 8 fragments with each fragment either absent or present during testing. This factorial design created a total of 256 combinations. Fragments were paired diagonally between left and right rectangles since our stimuli generative model always placed the right rectangle at the bottom right quadrant relative to the left. Each fragment was subsequently assigned an index (Figure 3B). By doing this, we generated a unique 8-digit binary code for every combination. Specifically, the order of digits was represented by the numbering of fragments from Figure 3B with 1 being present and 0 being absent. For instance, 00101111 means that fragments 1, 2, and 4 were absent in the stimuli (Figure 3C).

AlexNet is an early and relatively simple CNN structure that processes images through a sequence of convolutional and pooling layers arranged in a strictly feedforward manner. Information flows feedforwardly through the network, with each layer operating on the output of the previous one. Consequently, AlexNet is a comparatively shallow and structurally simple architecture by modern standards, which makes it a standard benchmark for examining how well purely one-to-one feedforward connections could account for BOS selectivity.

Compared to the traditional CNN structure that consists of combinations of convolution-nonlinearity-pooling, Inception networks extend the basic CNN framework by allowing multiple types of visual features to be extracted in parallel at each processing stage. It incorporates multiple parallel convolutional layers with different filter sizes and concatenates the resulting feature map at the end as a single joint representation (Szegedy et al., 2015). Therefore, instead of using a single filter size at each layer, Inception modules apply several convolutional filters of different spatial scales simultaneously and combine their outputs, enabling the network to capture both fine local details and broader spatial patterns within the same layer.

The entire module consists of four individual pathways, a pathway of single 1×1 convolution that performs channel recombination, two pathways of 1×1 convolution to reduce dimensions following by either a 3×3 or 5×5 convolution to integrate medium to large spatial pattern, and finally a 3×3 maxpooling to introduce translational invariance which is potentially significant as our stimuli have varying position in the image. With appropriate padding, the four pathways result in the same spatial dimension but different number of channels, making the final merge achievable.

ResNet provides a solution to the degradation problem in deep networks by introducing residual (skip) connections that allow information to propagate from shallower layer to deeper layer without going through the intermediate layers. This design enables feature reuse and preserves low-level visual information, such as edges and corners, which are particularly critical for accurately processing our geometric stimuli and supporting reliable BOS decisions.

Considering the multiplicity of networks and training sets, we used the pairwise McNemar’s test (Spoerer et al., 2017) to compare classification accuracy between different networks trained on the same dataset. The non-parametric attribute makes it suitable for the evaluation of significant differences in their prediction outcomes when tested on the same datasets. To account for the multiple comparisons, p-values were adjusted using the Benjamini–Hochberg false discovery rate (FDR), with the FDR set to 0.05, to control the expected proportion of false positives.

For comparison across versions of an individual network, we implemented the two-proportion z-test to determine whether the performance achieved by a network trained on contour when tested on solid differs significantly from the performance achieved in the reverse direction. This thereby determined the generalization and robustness of the networks to other types of stimuli.

In order to probe the internal decision-making process of the network and identify the features driving its predictions, we examined the saliency of the learned representations through Gradient-weighted Class Activation Mapping (Grad-CAM) that highlighted the most informative image regions for classification in a heatmap (Selvaraju et al., 2020). Through saliency mapping, we analyzed what image cues contribute to BOS representation and how important features evolved at different stack bundles of convolution-nonlinearity-pooling in the network hierarchy. Grad-CAM functions by calculating the target class score gradient with respect to the feature maps of the selected layer and averaging it to determine the importance of each channel. The weighted combination of feature maps is then rectified and upsampled to produce a class-discriminative heatmap that highlights the image regions most influential to the model’s prediction. This allowed visualization of how each network represents BOS-related cues and how these cues affect ownership decisions in different architectures and training scenarios.

The generalization was shown by networks trained under one luminance condition and tested under opposite conditions. All networks showed significant differences ( p < 0.001 ; z I = 6.03 , z R = 22.26 , z A = 11.23 ) where networks trained on solid stimuli outperformed networks trained on contour stimuli (Figure 5). This implies that solid stimuli supplied more transferable information through combined contour and surface cues, whereas contour stimuli did not sufficiently prepare the network to interpret ownership when luminance-defined regions were introduced. Overall, the results demonstrated a strong directional dependence in generalization, with solid-trained networks transferring more effectively to contour conditions than contour-trained networks transfer to solid conditions.

In addition to the luminance conditions, we also tested whether networks generalized beyond specific shapes by performing a cross-shape evaluation (Circle vs. Rectangle) in which all networks were trained on contour rectangles and tested on contour circles with similar occlusion relationships. Despite the networks never having been exposed to other geometric stimuli, all networks were able to retain above-chance (50%) accuracies (Figure 6). This in part indicates that the learned representations did not depend on the general geometry of the object, but rather on more shape-invariant cues that signal BOS, such as junction configuration and relative contour arrangement. The accuracy of the three networks also followed a trend in line with their accuracies observed in the solid/contour cross-condition.

The networks were trained using contour rectangles with continuous borders to recognize discontinuous borders with increment gaps to test the robustness and generalization of different feedforward structures (Figure 3A). Within the small gap ( n ≤ 1 ) conditions, Inception performed the best and exhibited strong robustness with consistent high accuracies above 95% and significant differences with both ResNet and AlexNet (Figure 7). ResNet performed moderately well in this regime, with accuracies near 90%, but still significantly below Inception for nearly every gap size. AlexNet showed an irregular fluctuation where the steepest degradation happened as even modest discontinuities were introduced but rose again at larger gaps, though recovery was unstable. As gap levels increased ( n ≥ 1 ), all networks showed progressive decline in performance. The lack of significance at extreme gap level ( n ≥ 4 ) indicates that all networks failed under highly discontinuous contours, consistent with the absence of sufficient edge information for reliable BOS assignment.

The contrast between the performance of Inception under fragmentation (Figure 7) and in the solid/contour cross-condition (Figure 5) demonstrates that the two generalizations, i.e., across fragmentation and across solid/contour, were different. Fragmentation maintains contour-based representation and simply removes portions of the stimulus, whereas solid/contour cross-condition generalization requires the network to bridge between qualitatively different cue domains. Inception’s multiscale structure appeared to be well suited for integrating incomplete contour information, but less effective at generalizing across contour vs. solid-based representations.

To evaluate the significance of individual fragments and their interaction, we used a second-order regression model similar to Zhang and Von Der Heydt (2010) but without the contrast polarity term as follows:

Where R is the accuracy of the network, β 0 is the baseline accuracy of the network and a fitted intercept of all conditions, F i and F j are fragment variables (0 when absent; 1 when present). d i and f i j are regression coefficients that estimate the significance of individual fragment i and the interaction between two fragments in the BOS decision. g ( N ) is a normalization function that eliminates the global effect of fragment availability N in a given stimulus.

The model revealed a highly structural pattern in terms of how networks perform BOS where the individual fragment coefficient displayed a sharply unequal distribution of significance. Among eight of the fragments (Figure 8A), only fragment 8 exhibits a strong positive effect on BOS decision. In contrast, the remaining seven fragments showed only small negative or near-zero effects, suggesting that they contribute little individually and even present local ambiguity when present alone. This asymmetric pattern implies that the networks rely selectively on specific, spatially critical fragments rather than treating all boundary segments equivalently. The fragment interaction coefficient further clarified this pattern (Figure 8B). Several fragment pairs exhibited strong positive interaction (1–8, 3–7, 3–6, 5–6), demonstrating a synergy effect in which BOS information became more complete and obvious to the network when these fragments existed together than independently. This interaction likely reflects the grouping mechanism in biological vision that relies on a certain combination of elements for recognition. Conversely, there were some fragment pairs that interfered with the network’s decision (1–3, 5–8) by introducing confounding or opposite structural cues about BOS.

To account for the influence of global fragment counts, we introduced a normalization function that adjusted for the general improvement in the model’s accuracy as more fragment information was presented in the stimulus.

Where the linear coefficient γ 1 represents the improvement from adding fragments regardless of their specific location. The nonlinear coefficient γ 2 captures the curvature of this relationship, expressing the degree of saturation or acceleration as more fragments are added. From the simulation, the two coefficients indicated a consistently positive effect of fragment availability (Figure 9). The large linear term indicated that each additional fragment increased accuracy on average, and the small quadratic term suggested that this improvement did not saturate within the tested range. Instead, the network continued to benefit from incremental fragment cues even at high fragment availability. The observed bimodal distribution arose primarily from the strong contribution of fragment 8, whose presence or absence largely determined whether sufficient occlusion-specific structure was available to support a reliable BOS decision.

Across all three networks that trained on a mixed dataset, Grad-CAM visualizations showed a clear hierarchical progression in how BOS information was represented and evolved (Figure 10). Shallow layers were more responsive to low-level visual elements such as edges, corners (L-junctions), and isolated contour fragments, generating saliency patterns that were spatially scattered and not yet aligned with the ownership-defining border. Intermediate layers began to integrate these local features into larger spatial structures, showing more continuous activation along contour segments and increased sensitivity to the relative arrangement of the two rectangles. By the deep layers, the saliency converged into a compact, highly localized region centered on the occluding area, indicating that ownership decisions were made at later stages of processing, and the activation patterns consistently corresponded to the correct ownership boundary regarding the stimulus type. Overall, these results demonstrate that BOS selectivity is a hierarchical computation that gradually evolves from distributed local feature detection to a coherent, border-specific representation in deeper stages of the network.

For fragmentation, Grad-CAM visualizations showed a gradual loss of and reorganization of BOS representations with increasing gap level (Figure 11). For small gaps, saliency strongly aligned with the fragmented contour segments that together defined the owned border, suggesting that the networks captured discontinuous local evidence and assembled this into a coherent ownership signal. As gap size grew, saliency became less continuous and distributed more unevenly among the remaining contour fragments. At very large gap levels, however, saliency became spatially non-specific, especially at deep layers where saliency was more spread out and shifted toward regions other than those occluded. Under these conditions, the deep layer activation patterns ceased to be coherently associated with the true ownership boundary, and correspondingly the classification accuracy dropped sharply even when early layer saliency remained consistent with that observed for intact contours. In general, the representation of BOS in CNNs degrades with increasing fragmentation, as we see a transition from continuous contour-based representations to the sparse and unstable cues lacking in power to assign ownership reliably.