Our neural networks have been trained and evaluated using as input cryo-EM micrographs of RNPs of the influenza A virus. In the following, we provide details about how this data was produced.

For segmenting RNP filaments and RNP filament ends, we implemented a 2D U-net-like convolutional neural network, adapted from Ronneberger et al. (2015). Our neural network architecture comprises three downsampling and three upsampling blocks, each connected with skip connections for feature preservation. Every block includes two convolutional layers activated by RELU functions. The convolutional layers in these blocks use filters of sizes 128, 256, and 512, respectively, each with a kernel size of 7 × 7 to enhance noise robustness. Downsampling in our network is achieved through strided convolutions, while upsampling utilizes transposed convolutions. The final output layer classifies each pixel into two categories, employing a generalized Dice loss function to counteract class imbalance issues. Our network processes images of size 640 × 448 pixels. An essential aspect of our method is the normalization of the input images, where we calculate and apply the 98th and 2nd quantiles for contrast adjustment, clipping values outside the 0–1 range. We divided the labeled dataset into training and validation sets with an 80:20 split. To enhance model robustness, we included random translation transformations (within [−10, 10] pixels range) in the training phase. The model was trained using the Adam optimizer, with batch sizes of 30 images over 50 epochs.

After training the Full-RNP and RNP-E networks, they are applied to segment RNP filaments and their ends across all micrographs. These segmented images are then analyzed to pinpoint their coordinates. Our pipeline for this analysis is as follows:

1. Preprocessing: Each input micrograph is first Fourier band pass filtered, downsampled, contrast inverted and normalized as previously described.

For processing RNP ends, the centroid coordinates of each labeled region are determined and adjusted by the previously applied downsampling factor. On the other hand, for the processing of RNP filaments, after step 4 the next steps are followed:

5. Skeletonization: For the RNP filaments, the identified regions are skeletonized or thinned using the homotopic thinning algorithm. (Lee et al., 1994), transforming the filament’s thickness into a 1D curve that represents its skeletal structure.

This pipeline ensures precise and efficient localization of both RNP filaments and their ends in the micrographs.

In our study, we trained the Full-RNP model utilizing a dataset of 150 micrographs, each meticulously annotated by a human expert. The evaluation of the model’s semantic segmentation predictions, when benchmarked against the ground truth data, provided the outcomes presented in Table 1. To further scrutinize the reliability of manual annotations, a subset of 50 micrographs from the training set underwent dual rounds of manual labeling by the same person, facilitating a comparative analysis of human annotation consistency using identical evaluate metrics. These results, aimed at appraising the precision of human annotations, are shown in Table 1 (a) at row “H-H” and in Table 1 (c).

Table 1 (a) employs a suite of conventional metrics for assessing semantic segmentation, encompassing global accuracy, mean accuracy, mean intersection over union (IoU), weighted IoU, and the boundary F1 (BF) score. Global accuracy (GAccuracy) quantifies the overall proportion of pixels correctly classified across all categories. This metric provides a rapid and computationally efficient assessment of the fraction of pixels correctly classified. Mean accuracy (MAccuracy) calculates the average rate of accurately identified pixels for each category across the dataset. The IoU metric, or Jaccard similarity coefficient, gauges the overlap between the predicted and actual pixels for each class, with MeanIoU averaging this score across all categories. WeightedIoU adjusts the IoU score for each class based on its pixel prevalence, mitigating the influence of minor class discrepancies on the collective metric. The BF score assesses the alignment of predicted class boundaries with their actual counterparts, with MeanBFScore averaging this alignment for each class across all images.

Additionally, Table 1 sections (b) and (c) shows normalized confusion matrices for the background and RNP categories, contrasting the network’s segmentation predictions with the ground truth obtained from the validations set (Table 1 (b)) and juxtaposing the two sets of manual annotations by the same expert (Table 1 (c)). The consistency between the model’s performance on both validation and training sets underscores its robust generalization capability. Finally, Table 1 (d) and (e) shows the intersection over union (IoU) for each class and the average of the BFScore for each class across all images in the validation set and for the subset of 50 micrographs labelled twice for the same person. As can be seen from these results, the congruence of the model’s metrics with those derived from human annotations underscores the model’s potential to achieve near human-level accuracy in RNP semantic segmentation tasks.

In Figure 1A, we show examples of preprocessed micrographs (Fourier band pass filtered, downsampled, contrast inverted and normalized), obtained ground-truth labels (labelled) and the predictions made by the RNP-FULL network (predicted). As can be seen from this figure, there is a good visual agreement between the ground truth and the predictions made by the Full-RNP model. In Figure 1B, we show three examples of the processing pipeline followed by our proposed approach. First the preprocessed micrograph is automatically labelled by the trained Full-RNP model, and the segmented images are cleaned, labelled, and thinned. Then, the coordinates of each of these labelled skeletal structures are extracted.

For the training of the RNP-E model, we used 500 manually labelled preprocessed micrographs. In Table 2, we use the same metrics used in Table 1 for assessing semantic segmentation done by the network. In Table 2 (a) row H-H and Table 2 (c) and (e), we show again as reference the results obtained by assessing the accuracy of human annotations for the manual labelling of full RNPs. According to these results, we can concur again the good performance of the RNP-E model and the good similarity between the model metrics and the ones obtained by the same person when labelling the full RNPs showing again the model’s potential to achieve near human-level accuracy in RNP semantic segmentation tasks. In Figure 2, we show examples of preprocessed micrographs (micrographs) and corresponding micrographs with superimposed labelled RNP ends segmented manually (labelled) and predicted by the RNP-E network (predicted). As can be seen from this figure, there is a good agreement between the ground-truth and the automatically segmented RNP ends. Finally, in Figure 3 we show the workflow followed by the proposed method to localize RNP ends. The preprocessed micrographs are segmented automatically by the RNP-E network. Then these images are labelled with unique integer labels based on the connectivity of pixels to their neighbors and too small and too big regions are filtered out. For the remaining regions their centroids are computed localizing or picking the ends of the RNPs.

To compare with our deep learning-based picker, we tested the performance of other particle selectors on the same RNP data set. We choose four of the most used pickers: the template matching picker Gautomatch (https://sbgrid.org/software/titles/gautomatch) and the picker included in the Relion software suite (Kimanius et al., 2021) (https://github.com/3dem/relion), Topaz (Bepler et al., 2020) (https://github.com/tbepler/topaz) and CrYOLO (Wagner et al., 2019) (https://pypi.org/project/cryolo/).

The template matching algorithm implemented in Gautomatch software requires as main input one or more 2D averages of the particles to be selected, an estimated size of the box that will contain the entire particle, and an estimate of the average minimum distance between two particles in the image. Additionally, there is a tunable threshold value, ranging from 0 to 1, which indicates the level of cross-correlation between the templates and a feature in the micrograph to be considered a positive match. Figure 4 shows the results of Gautomatch picking using 2D averages of the central part of the molecule as templates. The two averages used (inset in panel 4a) were obtained by manually picking and aligning approximately 2000 particles from a random selection of 100 micrographs from the total set of 29,493 images. The panels display the results obtained at different thresholds. Higher values indicate a more restrictive search, where the selected particles are more similar to the 2D averages used as templates. When the threshold is low (0.15), the number of regions selected as positives is very large, including the actual particles and a substantial number of false positives, distributed in the background, in the contaminants, and along the carbon edges of the support. As the threshold value increases (0.2), the RNPs are marked correctly, and the number of false positives selected in the background decreases drastically, although those corresponding to contaminants and carbon edges (red arrows) persist. If the threshold value is increased further in an attempt to reduce the latter false positives (0.3), it is observed that unexpectedly the number of correctly selected particles decreases (blue arrows), while the false positives found in areas of higher contrast remain (red arrows). In summary, there is an optimal threshold (0.2) at which most of the particles are correctly selected, however some contaminations and the edge of the carbon support are also marked as false positives.

The ability of Gautomatch to pick the ends of the particles was also tested and the results are shown in Figure 5. Similar to the previous case, the ends of RNPs from 100 micrographs were manually selected and aligned to produce 2D averages, which were used as templates for Gautomatch (inset in Figure 5A). The field covered by the templates in this case was deliberately chosen to be larger than in the previous case to ensure that the image clearly showed the end of the particle, preventing misidentification as an intermediate part of the helix. In this context, multiple tests were performed with different template sizes, and the ones shown here produced the best results. In this case, and very similar to the previous test, using a low threshold (0.2) caused the program to select a large number of matches, including real particles along their entire length (not just the ends) and many false positives. This result is almost indistinguishable from when the 2D averages of the central part of the molecule were used as a template. Increasing the threshold value to 0.4 caused most of the false positives in the background to disappear, and the number of correct positives increased proportionally to the total number of labeled particles (green arrows), although some real ends were no longer selected (blue arrows). However, the number of false positives in the center of the particles and in high-contrast regions (carbon edges and contaminants) remained high (red arrows). Finally, increasing the threshold to 0.6 resulted in the loss of correct positives, with the program selecting only false positives in the high-contrast regions.

We also compared with our particle selector the picking algorithm implemented in Relion (Kimanius et al., 2021). Similar to the previous case, the software requires as main input data the 2D averages to be used as templates, the minimum distance between particles, and two parameters called “minimum mean noise” and “maximum standard deviation noise” designed to prevent the picker from selecting regions of high contrast. The values of these parameters should be determined empirically. Moreover, in Relion there is an adjustable threshold that indicates the level of similarity between the template and the selected feature in the micrograph. Figure 6 shows the results obtained using the same 2D averages from the Gautomatch tests (insets in Figures 4A, 5A) as templates. After empirically optimizing the “minimum mean noise” and “maximum standard deviation noise” parameters to minimize as much as possible the picking of incorrect high-contrast regions, several tests were performed at different thresholds. In the case of the central regions selection (Figure 6A), the results are shown at two thresholds around the optimal value. Using these thresholds, the software correctly identified most of the particles (green arrows), and the edges of the supporting carbon were not marked as positive matches. However, contaminations corresponding to ice crystals were mistakenly selected as particles (red arrows). As with Gautomatch, an increase in the particle selection threshold value resulted in fewer real particles being selected, but the false positives produced by ice crystals remained. Finally, in the particle end-picking test performed using the 2D averages of the insets of Figure 5A as templates, the results were very similar to those obtained using the centers as templates (Figure 6B). Additionally, the selection of regions containing contaminants as false positives (red arrows) also occurred, and increasing the threshold value did not resolve the issue.

In summary, the template-matching-based particle selectors analyzed here produce very similar results whether the central or terminal regions are used as templates, indicating that they can barely discern between these two regions in the images. Although particle selection results are slightly better when using the 2D averages of the central region, the programs tend to select high-contrast regions as positives to some extent, which cannot be resolved by varying the particle selection threshold. However, the results obtained with our software satisfactorily solve these problems and give results similar to those produced by a human expert.

We also compared the performance of our particle picker against other neural network-based software, specifically Topaz and CrYOLO. The results of these comparisons are presented in Figure 7. To maintain consistency in the evaluation, we used the same set of micrographs and coordinates for training as we did with our own program.

Topaz employs a convolutional neural network based on positive unlabeled learning (Bepler et al., 2020), with multiple adjustable parameters in its learning protocol, including the particle size in its longest dimension. However, for the dataset we used, which includes filamentary and highly flexible structures that often bend, determining an appropriate value for this parameter proved challenging. After extensive trial and error, we found that the renet8 model architecture yielded the best results. Figure 7A shows the particle selection from the central regions of the RNPs at the threshold that produced the optimal outcome. While the particle detection was accurate, with no false positives, the selected coordinates were positioned near the boundary between the particle and the background, rather than at the center of the filaments. This led to the appearance of a “zig-zag” pattern in the coordinates. Figure 7B illustrates the particle selection by Topaz when the network was trained using the ends of the RNPs at two different thresholds. The results were similarly accurate, although there was a slightly higher tendency to select false positives, particularly in areas with ice contamination or sharp bends in the RNPs.

CrYOLO employs a convolutional neural network based on supervised learning, requiring labeled data for training (Wagner et al., 2019). It offers multiple adjustable parameters, such as particle diameter, box size, and detection threshold, to adapt to different datasets. Figures 7C, D display the results of particle picking using CrYOLO, trained on the central regions and the ends of the RNPs, respectively, at the threshold that yielded the best results. Surprisingly, the results are quite similar in both cases, showing little difference between training on centers versus ends. Moreover, when trained on the ends, there was a greater tendency to select false positives.

In summary, neural network-based particle pickers, such as Topaz and CrYOLO, outperform traditional template-matching methods in terms of detection accuracy. However, fine-tuning their parameters is essential and often labor-intensive. Without careful optimization, this can lead to the selection of false positives.

The particle-picking system presented in this study offers not only accurate detection of regions of interest through semantic segmentation, but also introduces a novel method for generating coordinates used for particle extraction. In the case of localizing RNP filaments and unlike other software, which typically calculates the centroid of the detected region to determine coordinate placement, our approach utilizes a skeletonization process. This process identifies the geometric center of the filament, enabling the program to determine the position of the helical axis of the particle, regardless of its curvature. As a result, our software generates a line of coordinates that facilitates the extraction of a continuous series of images along the entire axis of the particle.

This coordinate generation method for the case of localizing RNP filaments has two significant advantages. First, the extracted images are centered on or near the particle axis, minimizing the shifts required for image alignment. This leads to a reduction in the computational resources needed for image processing. Second, this approach increases the total number of images to be processed in an efficient manner, extending the concept of equispaced and uniform extraction—commonly applied to straight helical particles—to more flexible, curved filaments. Figure 8 shows 2D averages computed with CryoSPARC software (Punjani et al., 2017) of RNP filaments (RNP helical central regions) obtained from particles picked using the software presented in this work, where it is recognizable secondary structure.