Turkey is one of the top five tomato-producing countries in the world. About 10% of Turkey’s tomato production occurs in Bursa, where tomatoes were the most produced vegetable in 2020, with 13.2 million tons (Kumbasaroğlu et al., 2021). This study was conducted from March 2022 to September 2023 in Bakirköy village, located in the Karacabey district of Bursa province in the northwest of Turkey, lying between latitudes 40°7’17.53”N and 40°10’40.36”N and longitudes 28°21’14.12”E and 28°26’2.37”E. Field campaigns were conducted from June to July 2023. The site covers an area of 47.16 km², and a total of 96 tomato fields were investigated. The identification of pests observed in the field photographs was carried out according to the morphological diagnostic keys available in the literature (Blackman and Eastop, 2000; Hoebeke and Carter, 2003; Desneux et al., 2010; Ashbrook et al., 2022; Li et al., 2023).

Detection models excel in identifying the presence and location of pests quickly and efficiently (Barbedo, 2016). However, segmentation models are more suitable when detailed morphological features or comprehensive damage maps are necessary (Arockia et al., 2023). The YOLOv8 model integrates both detection and segmentation capabilities. In this study, the yolov8s.pt model was employed for detection tasks, while the yolov8n-seg.pt model was utilized for segmentation tasks. The images used for detection and segmentation in this study encompass various pests and damage types affecting tomato crops. These include Dolycoris baccarum (Hemiptera: Pentatomidae), Phyllotreta spp. (Coleoptera: Chrysomelidae), Nezara viridula (Hemiptera: Pentatomidae), Myzus persicae (Hemiptera: Aphididae), Bemisia tabaci (Hemiptera: Aleyrodidae), Leptinotarsa decemlineata (Coleoptera: Chrysomelidae), Tuta absoluta (Lepidoptera: Gelechiidae), Helicoverpa armigera (Lepidoptera: Noctuidae), Liriomyza bryoniae (Diptera: Agromyzidae) damage, Frankliniella occidentalis (Thysanoptera: Thripidae) damage, and Tetranychus urticae (Trombidiformes: Tetranychidae) damage. These pests and damage types were systematically photographed and used to train the YOLOv8 model for accurate detection and segmentation tasks, aiming to enhance the model’s ability to identify and manage multiple pest species effectively.

From March 2023 to September 2024, high-resolution images of tomato plant diseases and pests were captured using a Canon EOS 700D camera with a resolution of 768 × 1024 pixels. To ensure consistency in image quality, all photographs were taken using cameras set to identical resolution settings. Images were taken at distances of 1 meter and 0.2 meters from the leaves, from various angles (Yong et al., 2020). A comprehensive dataset of over 1,000 images was compiled for each pest, documenting different angles and features. These images were then divided into three subsets: 80% for training, 18% for validation, and 2% for testing, as outlined in Table 1 .

Data preprocessing refers to a series of steps undertaken to prepare raw data for analysis or modeling. It is commonly used in data mining, machine learning, statistics, and data analysis to address data deficiencies, noise, and inconsistencies, thereby enabling more effective analysis (Şahin and Topal, 2016; Atalan et al., 2022; Baştürk and Şahin, 2022). For image processing models, preprocessing the dataset involves three steps: image labeling, resizing, and augmentation. Augmentation provides a large amount of training data to learn features and achieve accurate classification on unseen data, preventing issues like overfitting and poor generalization (Redmon and Farhadi, 2018; Rubanga et al., 2020; Güven and Şahin, 2022).

In this study, Python (version 3.11.8) was used for data preprocessing due to its extensive library ecosystem. To enhance the quality of model training, the original images taken under field conditions were augmented using the OpenCV library. During augmentation, transformations were applied to each image, including rotation, cropping, flipping, adding noise, adjusting lighting, and zooming out (Shorten and Khoshgoftaar, 2019). All images were resized to 600 × 600 pixels as required for model training (He et al., 2017). The analyses were conducted in the Spyder IDE, part of the Anaconda distribution, which offers various libraries for scientific computing and data science (Şahin et al., 2020; Yılmaz et al., 2021). Prior to any augmentation, the dataset was divided into training, validation, and test sets (80%, 18%, and 2%, respectively) to ensure that no data leakage occurred during the augmentation process. Augmentation was only applied to the training set to avoid introducing artificial examples into the validation and test sets, which could result in overly optimistic performance estimates (Shorten & Khoshgoftaar, 2019). Specifically, transformations such as rotation, cropping, flipping, adding noise, adjusting lighting, and zooming out were applied only to the training data after the initial dataset split. Labeling was performed using the LabelMe tool (https://github.com/wkentaro/labelme), with two main approaches: pixel-based segmentation for precise boundary definitions and rectangular bounding for approximate location and size.

YOLOv8 was selected for this study due to its superior speed and efficiency compared to slower yet more accurate models like Faster R-CNN, making it particularly well-suited for real-time agricultural pest detection, where timely decisions are essential for effective pest management (Tang et al., 2021). The YOLOv8 models were trained using the ultralytics library for model loading and training, and the google.colab library for accessing the dataset via Google Drive. Training parameters included over 100,000 epochs (with patience set to 50 to prevent overfitting), a batch size of 16, and an image size of 640 ( Table 2 ). The model’s hyperparameters, including the number of epochs, batch size, and learning rate, were optimized through an iterative process. Early stopping (patience) was employed to prevent overfitting, while cross-validation was used to fine-tune the learning rate and batch size. The optimal values for these parameters were selected based on the model’s performance on the validation set, ensuring robustness and preventing overfitting. Training was conducted on Google Colaboratory, utilizing an Intel Xeon CPU, 12.68 GB RAM, and a Tesla K80 GPU. Both detection and segmentation models were trained in Python on a custom dataset. Instance segmentation models were chosen to precisely identify damage caused by multiple pest species on tomato plants, which is crucial for accurately identifying specific damages on leaves and fruits (Mirhaji et al., 2021; Zhang et al., 2023a, 2023). The YOLO framework used in this study is illustrated in Figure 1 .

The model’s generalization capability was assessed using a pre-allocated dataset: 80% for training, 18% for validation, and 2% for testing. Key performance metrics, including precision, recall, mAP50, and mAP50-95, were calculated during training on both training and validation datasets. Detection and segmentation performances were evaluated at various IOU thresholds using precision (P), recall (R), and mean average precision (mAP). The mAP50 metric refers to the mean average precision at a 50% IOU threshold, indicating the accuracy of the model in identifying objects with at least 50% overlap with ground truth labels. On the other hand, mAP50-95 averages precision over IOU thresholds from 50% to 95%, offering a more comprehensive view of the model’s accuracy across different overlap scenarios, which is particularly important in agricultural contexts where pests may be occluded or vary in size (Li et al., 2023). These metrics provide insights into the model’s ability to handle various sizes and overlaps in real-world agricultural environments. Loss metrics—box_loss, cls_loss, and dfl_loss—were analyzed to identify areas for improvement. The confusion matrix summarized predictions across classes, highlighting correct and incorrect classifications. This comprehensive analysis provided a clear understanding of the model’s strengths and weaknesses. Metrics P, R, mAP50, and mAP50-95 are defined by Table 3 .

The study used Python and open-source libraries to integrate detection models with OpenAI’s GPT-4 via an API key. Initially, models identified trained objects, which were then linked to GPT-4. A good prompt should be clear, specific, and provide context to guide the AI’s response. Labels were defined as ‘det_labels_str’ and ‘seg_labels_str’. The prompt used in the study was: prompt_str = f”Could you provide a detailed explanation, in academic English, on the methods for controlling {det_labels_str} or {seg_labels_str} and the potential damage they can inflict on related plants, including preventative measures and integrated pest management strategies?”. Text outputs, limited to 250-400 tokens, were visualized with detection results. A ten-step coding sequence enabled the simultaneous operation of segmentation and detection models ( Table 4 ). The workflow, from image capture to ChatGPT-4 output, is depicted in Figure 2 .

In this study, significant improvements were observed during YOLOv8 model training for pest detection. For training metrics, the box_loss decreased from 1.84 to 0.54, cls_loss from 3.48 to 0.37, and dfl_loss from 1.54 to 0.86. Similarly, validation metrics showed a decline: val/box_loss reduced from 1.38 to 0.53, val/cls_loss from 3.45 to 0.31, and val/dfl_loss from 1.30 to 0.90. Training was halted at 749 epochs to prevent overfitting, demonstrating effective learning and performance. For segmentation, the train/box_loss decreased from 1.97 to 0.57, train/cls_loss from 4.04 to 0.37, and train/dfl_loss from 1.56 to 0.86, while validation metrics also improved, with val/box_loss reducing from 1.99 to 0.74, val/cls_loss from 2.65 to 0.37, and val/dfl_loss from 1.45 to 0.92. Training stopped at 372 epochs to avoid overfitting, indicating robust model performance ( Figure 3 ).

During YOLOv8 model training, significant improvements were noted across key metrics: precision increased from 0% to 98.91%, recall from 0% to 98.98%, mAP50 from 0% to 98.75%, and mAP50-95 from 0% to 97.72%. Training was halted at 749 epochs to prevent overfitting, demonstrating enhanced accuracy and reliability in object detection ( Figure 4A ). For segmentation, precision improved from 0% to 97.47%, recall from 0% to 98.81%, mAP50 from 0% to 99.38%, and mAP50-95 from 0% to 95.99%, with training stopping at 372 epochs to avoid overfitting ( Figure 4B ).

A confusion matrix, essential for evaluating a model’s performance, pinpoints misclassifications and highlights areas for potential improvement. The test set, comprising images of adult insects, nymphs, and larvae across 11 classes, facilitated the computation of the confusion matrix. The YOLOv8 model exhibited high accuracy in detection tasks. Specifically, D. baccarum adults were correctly classified 890 times with 15 misclassifications, N. viridula adults were accurately identified 695 times with 15 errors, and M. persicae adults were correctly classified 998 times. Additionally, B. tabaci adults achieved 1025 correct identifications, and L. decemlineata adults were correctly identified 666 times with no errors ( Figure 5 ). The model’s performance for segmentation on the test set revealed notable outcomes across 8 classes: 550 correct detections of L. bryoniae damage, 345 accurate detections of T. absoluta damage on fruit, 450 precise detections of T. urticae damage on leaves, and 565 correct identifications of healthy tomato leaves ( Figure 6 ).

The integration of YOLOv8 and ChatGPT-4 showcases the powerful combination of computer vision and natural language processing, enabling expert feedback on visual data. This integration was tested on five pictures from the test set, which were not included during the training phase. The responses to the crafted prompt, [prompt_str = f”Could you provide a detailed explanation, in academic English, on the methods for controlling {det_labels_str} or {seg_labels_str} and the potential damage they can inflict on related plants, including preventative measures and integrated pest management strategies?”] are presented in Figure 7 . The trained detection and segmentation models processed the test images in approximately 0.10 seconds, while the integration with ChatGPT-4 provided textual responses within 3.5 seconds via the API. Despite being limited to 250-400 tokens, the ChatGPT-4 responses, while not always fully comprehensive, demonstrated the potential to offer key information.