The acoustic recording of bee buzzes was conducted in five highbush blueberry orchards (V. corymbosum) located in southern Chile (Maule and Los Ríos Regions) between the months of September and November in 2020 and 2021 . The total area of cultivated blueberry, both organic and conventional farming, per orchard ranged 3.2 − 141 hectares. The most common growing cultivars were Legacy, Brigitta, Duke, and Elliot. Four of the five orchards were supplemented with colonies of managed exotic bees of Bombus terrestris and/or Apis mellifera ( Table 1 ).

Visual searches were made for foraging bees beginning at 10 : 00 h and ending at 18 : 30 h as bee activity declined. To record buzzing sounds, 3 − 4 researchers constantly walked through the rows of blueberries hand-holding a recorder while searching for flower-visiting-bees. When a bee was observed approaching a flower, it was followed, holding a digital acoustic recorder (Zoom H4n Pro Handy Recorder) such that it was within 1 − 5 cm of the bee when it landed on the flower. The microphone head was pointed at the dorsal surface of the bee thorax. All bee individuals that could not be immediately identified were captured just after leaving the flower with an entomological net and placed in glass vials with ethyl acetate for taxonomic identification in the laboratory. As a consequence of that, we can assume the number of audio samples corresponds to the number of bee individuals. All sampled bee individuals were taxonomically identified at the lowest possible level by experts.

We performed some data pre-processing before the training step in order to improve the performance of the ML models. The original sound recordings (.wav files) were manually classified and segments with bee buzzing sounds were selected. We categorized as sonication all the segments of buzzing sounds produced by bees vibrating blueberry flowers, and as flight the sounds produced by the flying displacement of the bees between flowers. Flight sounds and sonication buzzing could be easily distinguished on the recordings afterward by an experienced user since they present pronounced differences in acoustic characteristics; Ribeiro et al. (2021) showed that both sonication and flight sounds contribute equally to the training of a bee species classifier. Thus, we used both sound types together in all trials, since flight and sonication together sum a higher number of audio samples and include bee species not capable of sonicating. Recording segments with no bee sounds were not selected but were kept for subsequent steps. The set of recordings contained 518 audio samples (corresponding to 518 bees individuals) lasting on average 2 seconds, with 1 , 867 flight segments and 1 , 728 floral sonication segments (see Table 2 ). We performed these analyses using Raven Lite software (Cornell Laboratory of Ornithology, Ithaca, New York).

Audio feature extraction techniques transform raw audio data generated by acoustic pre-processing into features that explicitly represent properties of the data that may be relevant for ML classification. We compared two audio feature extraction techniques separately, Log Mel-Spectrogram and MFCC. The Mel Spectrogram is a way to process audio such that various DL and ML algorithms can learn from the recorded sounds. The Mel-scale is a logarithmic transformation of the signal frequency. The Mel-Spectrogram demonstrates a compressed form of sound in the time-frequency domain. This nonlinear transformation constitutes the outcome of the Short Time Fourier Transform (STFT) after the application of Mel-filters (a bank of bandpass filters with bandwidths modeled after the Mel-scale). The conversion of the frequency in hertz ( f ) to the Mel-scale is illustrated in Eq. 1.

In order to relate the performance of different ML classification techniques, we evaluated our bee buzzing sounds dataset using classical ML and DL classifiers.

We used the following metrics to evaluate the performance of classifications generated by classifiers: Accuracy (Acc), Macro-Precision (MacPrec), Macro-Recall (MacRec) and Macro-F1 (MacF1). These metrics are determined by the classification output that comes from the confusion matrix. In this matrix, diagonal elements show the object similar to the actual label whereas off diagonals tell the misclassification information of the model.

Let i be a class from the set of classes C . Let T be test set and let c be a classifier, such that c ( t ) = l , where t is an element of the test set T and l ∈ C is a label corresponding to a class in C assigned to t by c . Let g ( t ) be the ground truth class label of t . In regard to the c classifier, we define:

The above numbers are used to define traditional effectiveness measures of classifiers. These measures are: Precision, Recall and F1-score [for more detail see Ribeiro et al. (2021)].

We based performance mostly on the F1-score since classes were unbalanced and Accuracy tends to underestimate classes with a smaller number of samples in relation to those with a larger number (Steiniger et al., 2020). The F 1 measure is a combination of the precision and recall measures and is defined by Eq. 2.

When comparing the performance of classifiers generated from distinct learning methods, it is common to use a global measure. A global measure aims at resuming the performance of the classifier over all classes in the test set. In this work we use the following global measures to compare the results of the classifiers we used: Accuracy ( A c c ) (which is equivalent to Micro-F1), Macro-Precision ( M a c P r e c ), Macro-Recall ( M a c R e c ) and Macro-F1 ( M a c F 1 ) [for more detail see Ribeiro et al. (2021)]. The Macro measures are basically the average of the corresponding metric.

The majority baseline was used to compare the performance metrics of CNNs recognizers. This baseline consists in assigning all audio samples to the majority class, that is, the bee species with more audio samples: Cadeguala occidentalis and Bombus terestris. Additionally, we assigned the best ML algorithm (based on the highest Macro F1-score) as an ML baseline to compare its performance with those of CNNs. We compared the performance metrics of each CNN classifier with those from the two baselines (majority baseline and best ML classifier) using the combined 5 × 2 cross-validated F-test (detailed in “Data splitting” section). We assumed a significance level of α = 0.05 . If the p -value was smaller than α , we rejected the null hypothesis and accepted that there is a significant difference between a pair of models.

During 990 hours of sampling effort distributed among 554 non-consecutive days of 2020 and 2021 , we recorded 518 audio samples of 16 bee species visiting flowers of highbush blueberry cultivars in five orchards of southern Chile (see Tables 1 , 2 ); most, 13 species, were native Chilean bees and three were exotics. In the set of 518 audio samples, we identified 3 , 595 buzzing-sound segments, 1 , 728 were of sonication and 1 , 867 of flight (see Table 2 ). The distribution of samples per bee species was highly unbalanced and varied from five (Cadeguala albopilosa to 103 (Cadeguala occidentalis). The length of recordings ranged from 5 seconds to over one minute.

The performances (based on macro-F1 score) of the classical ML algorithms at recognizing flower-visiting bees of blueberry crops were low, ranging between 17.24 and 34.97 % . However, the performances of the classical ML classifiers at recognizing bee species visiting blueberry crops depended on the algorithm employed. Support-Vector Machines (SVM) reached the highest Macro-F1 among the classical ML classifiers ( Table 3 ), correctly predicting most audio samples of the majority classes (above 50 % ): 61.3 % of Apis mellifera, 67.3 % of Bombus dahlbomii, 84.6 % of Cadeguala occidentalis and 69.8 % of Bombus terrestris (see Figure 1 ). However, the SVM failed to recognize most audio samples of minority classes (below 50 % ): Manuelia postica 28.6 % , Ruizantheda mutabilis 19.6 % , and Ruizantheda proxima 34.2 % (see Figure 1 ).

On the other hand, the audio feature extraction technique had little effect on the performances of ML algorithms, ranging from 18.88 to 34.97 % with MFCC and from 17.24 to 33.11 % with Log Mel-Spectrogram. The ML algorithms presented a slightly higher performance (based on Macro-F1 score) when fed by MFCC than when fed by Log Mel-Spectrogram ( Table 3 ).

Both of the tested CNNs (EfficientNet V2 Small and PANNs) reached higher performance in recognizing buzzing bee sounds than the majority baseline (assigning all the audio samples to the majority class), regardless of whether they were tested unaccompanied or combined with pre-training and/or audio data augmentation or of sampling technique (see Table 4 ). However, without data pre-processing (audio data augmentation, sampling technique, or pre-training) the CNNs did not present an evident higher performance (based on Macro-F1 score; p ≤ 0.05 , combined 5 × 2 c v F-test) in relation to the best classical ML classifier (SVM) ( Figure 2 ). EfficientNet V2 Small overperformed SVM only when it was combined with some audio data augmentation and/or pre-training ( Figure 2 ). However, PANNs without pre-training was capable of overperforming SVM, though data pre-processing also boosted its Macro-F1 score (see Table 4 ; Figure 2 ).

Accordingly, pre-training increased the performance of CNNs at acoustic recognition of bee taxa ( Table 4 ) by reducing the variability of F1-scores reached per model run (see Figure 2 ). The average performances of EfficientNet V2 Small and PANNs models were higher with pre-training (see Table 4 ): The Macro-F1 score of EfficientNet V2 Small ranged from 14.74 % ( ± 4.14 ) to 47.55 % ( ± 9.27 ) without pre-training and from 22.70 % ( ± 6.04 ) to 58.04 % ( ± 2.47 ) with pre-training; for PANNs they ranged from 35.95 % ( ± 3.40 ) to 55.00 % ( ± 3.81 ) without pre-training and from 35.25 % ( ± 4.14 ) to 56.96 % ( ± 2.30 ) with pre-training.

Despite the better recognition of audio samples of the majority classes by PANNs, EfficientNet V2 Small was better at hitting the samples of minority classes. Also, EfficientNet V2 Small with Mixup RT with pre-training correctly predicted the most audio samples of the majority classes (above 50 % ): 61.3 % of Apis mellifera, 67.3 % of Bombus dahlbomii, 84.6 % of Cadeguala occidentalis and 69.8 % of Bombus terrestris (see Figure 3 ). On the other hand, EfficientNetV2 Small failed to recognize most audio samples of lower represented classes (below 50 % ): Manuelia postica 19 % , Ruizantheda mutabilis 14.1 % , Ruizantheda proxima 39.9 % (see Figure 3 ).

Regardless of the differences above, the overall performance for acoustic recognition of bee species did not vary significantly among the CNNs architectures employed (EfficientNet V2 Small and PANNs). Despite that, EfficientNet V2 Small combined with Mixup, audio Randomly Truncated (RT), and pre-training overreached PANNs and achieved the highest Macro F1-score of 58.04 % ( ± 2.47 ) among all CNNs models and baselines tested (see Table 4 ).

Although the Mel-frequency cepstral coefficient (MFCC) can often lead to worse performance than the raw Mel spectral data (Stowell and Plumbley, 2014; Valletta et al., 2017), our results indicated no difference between employing MFCC and Log Mel-Spectrogram on the performance of classical ML algorithms at assigning bee buzzing sounds to the species to which they belong. MFCC is a more time-consuming method than Log Mel-Spectrogram since MFCC is a manually-designed summary of spectral information whereas Log Mel-Spectrogram involves a much simpler representation of a raw spectrogram. Despite that, MFCC has some advantages, including providing a substantially dimension-reduced summary of spectral data, which is positive for use in classical ML systems since they cannot cope with high-dimensional data (Stowell and Plumbley, 2014). However, dimension reduction necessarily implies a loss of information that could be made available for later processing and the consequent risk of discarding information that a classifier could have used. Despite MFCC being originally designed to represent human speech [24,40], which differs perceptually and from the production of bee buzzing, it can be applied to the acoustic bee recognition task with classical ML algorithms. However, our results indicated that the Mel-spectrogram, not only MFCC as previously speech (Fayek, 2016; Logan, 2000), can be a suitable sound feature extraction method for the recognition of buzzing bee species.

To our knowledge, this is the first application of CNNs to the task of acoustical classification of bee species. Despite Support-Vector Machine (SVM) being the best classical ML algorithm for bee sound recognition (Ribeiro et al., 2021), our results indicated that convolutional neural networks (CNNs) can outperform them. In fact, SVM and other classical classifiers are designed to model small variations which result in the lack of time and frequency invariance (Van Noord and Postma, 2017) which is often insufficient to cover the high-dimensional audio data of bee buzzing sounds. Therefore, CNNs become a primary choice in other applications of DL, not only for bee sound recognition recognition (Takahashi et al., 2016). In contrast to classical ML, CNNs were designed to process high-dimensional data well, which is the direct representation of raw audio data, like Log Mel-Spectrogram (Stowell and Plumbley, 2014; LeCun et al., 2015).

However, our results did not indicate that CNNs models alone overperformed classical ML, it only become evident when CNNs were combined with Log Mel-spectrogram and data augmentation techniques. In fact, CNNs can address the former limitations by learning filters that are shifted in both time and frequency (Zhang et al., 2015). However, it also generated very fast pre-training overfitting, resulting in models excessively adapted to the training set and with reduced capacity to transfer learning to validation and testing sets (Chicco, 2017). To mitigate overfitting and improve the generalization of models, we used the spectrogram augmentation technique and cross-validation to counterbalance it by generating additional pre-training audio samples and acoustic noise by applying random time-frequency masks to Log Mel spectrograms. Cross-validation is a well-known technique to deal with overfitting and was implemented in all ML classifiers here. The trained model does not overfit to a specific training subset, but rather is able to learn from each data fold, in turn (Chicco, 2017). Yet, data augmentation techniques can lead to a significant improvement in the performance of DL classifiers, but not for classical ML. Deep Learning models can take advantage of the iterative characteristic of this type of optimization, by epochs, in which the same data set can be represented in different ways for the classifiers. In practical terms, it would be like the model being exposed to different data. On the other hand, the augmented data for classical ML algorithms would be static. Therefore, we suppose that CNNs models best overperformed ML in acoustic recognition of bee species when using Mel-spectrogram information and Mixup data augmentation. Even with this improvement, however, our results indicated that the performance of CNNs is still unsatisfactory at recognizing buzzing bees in relation to ML standards (maximum F1-score 58.04 % ( ± 2.47 ) ). Hence, the CNNs tested here would not be the ultimate model and still have room for improvement, especially from novel Neural Networks architectures based on attention like the “transformers/perceivers” are likely to achieve higher performance for the task of bee species identification (Elliott et al., 2021; Wolters et al., 2021).

However, it is important to highlight that the two DL classifiers tested here, employ the mixup data augmentation technique slightly differently. The technique was used in the PANNs model as described in the original work, directly on the log Mel spectrogram representation. However, in the EfficientNet V2 Small model, the mixup was applied to the waveform. Based on previous experiments, we conclude that for this specific model, the application of mixup on the waveform provides better overall results.

In general, Machine-Learning can review large volumes of data and discover specific trends and patterns that would not be apparent to humans. To generate suitable classifications, ML models need massive resources with a considerable amount of accuracy and relevancy. However, our data set as well as other bioacoustic data sets are usually imbalanced, and with background noise (Rodrigues, 2019). Consequently, the imbalance was the main challenge to handle our data set using ML models. In-field bee audio data collection and acoustic pre-processing require domain knowledge and were exhaustive and very time-consuming: we spent 990 hours of fieldwork to record 518 audio samples, which corresponds to an average of 1.9 hours to record one sound file. Moreover, audio data collection was susceptible to bee species richness and abundance differences, thus limiting the number of samples for rare species (see also Ribeiro et al. (2021)). This not only impacts the applicability domain of the implemented ML but also influences the utility of the models for prospective use (Rodrigues, 2019). A data set is imbalanced when one class is over-represented with respect to the others, causing the model to return sub-optimal solutions due to bias in the majority class (Chicco, 2017). As a result, these classifiers tend to ignore small classes while concentrating on classifying the large ones accurately. Here, we dealt with data imbalance by employing pre-training and data augmentation (as discussed in the previous section) and measuring the performance of the classifiers based on Macro-F1 score instead of Accuracy. We employed macro-F1 since Accuracy underestimates classes with a smaller number of samples in relation to the larger ones. Macro-F1 score is considered a suited metric for an unbalanced test set because it better describes performance by class and not by sample number (Steiniger et al., 2020).

However, relating bee species performance with its respective pollination efficiency (Cortés‐Rivas et al., 2023), we found that the bees most efficient at pollinating were also the majority classes here (e.g. B. terrestris, C. occidentalis, B. dahlbomii). In practice, this reduces the imbalanced data bias since the majority and most hit classes are also the most efficient pollinators. Thus, we suppose that the ML algorithms are capable of recognizing well the most efficient pollinators of highbush blueberry crops in Chile.

Besides imbalanced data bias, background noise corruption was another frequent problem in our data set. However, we decided to input the original audios without noise removal or attenuation. Since noise corruption must be unavoidable in practical situations, audios without noise removal/attenuation bring more realistic model projections. In addition, by not removing noise from the input data, we also gain two functionalities: ( 1 ) we get more data for our deep neural network to train; and ( 2 ) we can train our neural network on noisy data which means that it will generalize well on noisy test data as well.

Automating the taxonomic recognition of flower-visiting bees would be especially relevant for blueberry fruit set and size, since the quality of the pollination provided is dependent, among other factors, on the taxonomic identity of a flower visitor (Brewer and Dobson, 1969; Dogterom et al., 2000; Benjamin and Winfree, 2014; Nicholson and Ricketts, 2019). A parallel study focusing on pollinator performance and covering most of the species analyzed here revealed that only a subset of the flower-visiting bee species achieved high performance at pollinate blueberry cultivars, while others were poor pollinators or even considered flower resource thieves (Cortés-Rivas et al., 2023). Therefore, automating acoustic recognition of bee species, especially distinguishing pollinators from nectar/pollen thieves, could result in a comprehensive and powerful tool for agriculture decision-making processes. Farmers could recognize the best pollinators without needing an expert in insect taxonomy. Aware of the value of bees to crop income, farmers could be encouraged to consider the pollination perspective in their crop management, which results in the conservation of local wild bee species, thereby contributing to advances toward more sustainable and higher-yield agriculture.

In summary, we compared the performance of CNNs models at recognizing blueberry-pollinating bees with the current state-of-the-art models for bee automatic recognition. We found advantages for CNN classifiers in recognizing bee species based on their buzzing sounds over the classical ML algorithms used (Ribeiro et al., 2021). CNNs algorithms powered by a combination of transforming sound events into Mel-spectrogram images and strong data augmentation overperformed classical ML algorithms and could lead to automating the taxonomic recognition of flower-visiting bees of blueberry crops. As far as we know, the use of DL classifiers for bee taxa identification based on respective buzzing sounds has not been reported previously. However, there is still room to improve the performance of DL models. Further studies, focusing on recording samples for poorly represented classes, and/or applying algorithms that can perform more complex processing tasks like unsupervised learning systems, could help to achieve better classification results.