This study was conducted at the Fruit Vegetable Research Institute in Buyeo, South Korea, using a collected dataset of high-resolution RGB images of three tomato plant varieties: Amos Coli, Nonari-Cherry Tomato, and Dafnis-Hybrid. These varieties were cultivated under standard grower-managed conditions in semi-open greenhouses over two consecutive periods: January–July and August–December 2022. Table 1 details the plant varieties and cultivation periods.

During the first 25-week period, 100 Amos Coli plants were monitored in Crop Lines 2 and 3 of the greenhouse. In the second 18-week period (August–December), 100 plants, 50 of Nonari-Cherry and 50 of Dafnis-Hybrid were observed. In total, 200 plants were monitored over one year. Weekly site visits were conducted throughout the cultivation periods to capture high-resolution RGB images. Three images were taken per plant from different viewpoints—left, right, and top—focusing on the upper plant region, a key indicator of growth (Cho et al., 2023). This multi-view strategy enabled a comprehensive assessment of plant health.

Figure 2 presents sample images from various viewpoints collected for several weeks. Images were captured using smartphone cameras, with a color checker included for future color-based leaf analysis. Each plant was tagged with a QR code containing its plant number, slab number, and cultivation period for identification.

Following data collection, a domain expert in plant physiology from the Fruit Vegetable Research Institute in Buyeo, South Korea, meticulously annotated each image in the dataset, documenting various growth stages and conditions of tomato plants. Each image was assigned a health status level on a scale from 1 to 5. Table 2 provides detailed descriptions and specific indicators used for classification, while Figure 3 presents representative images for each health level, ranging from Class 1 (severe health issues) to Class 5 (optimal health), serving as visual references for the annotation process.

During labeling, we identified cases of plant health deterioration and their potential causes. While most plants remained in optimal condition, occasional issues arose, such as valve malfunctions that disrupted nutrient supply. In some instances, this led to sudden plant distress and, in severe cases, plant death. However, when problems were detected early, growers were able to intervene and correct the issue, preventing further damage. Further analysis of plant growth and health progression is presented in the Discussion section, supported by representation maps that illustrate the plants’ development over time.

Table 3 presents the distribution of image samples across datasets corresponding to different tomato plant varieties cultivated in Line 1, Line 2, Line 3, and Line 4 within the greenhouse. The number of samples varies significantly across health status ratings. Since the grower’s objective was to maintain optimal plant growth, data acquisition prioritized class levels 4 and 5, which represent plants in near-optimal or optimal conditions. This pattern was consistent across tomato plant varieties, cultivation lines, and seasons.

In contrast, although less frequently, data was also collected for class levels 3 to 1, representing cases where plant health deteriorated during the monitored period. These lower health levels provide insights into plant stress factors and potential causes of deterioration. After annotation by domain experts, the dataset was structured based on the assigned health status for each image, ensuring a well-balanced representation of plant conditions throughout the study. We will refer to each dataset as the corresponding line and number to facilitate the description.

Our deep learning-based plant health monitoring framework ( Figure 4 ) consists of multiple stages, beginning with the collection of high-resolution RGB images from semi-open greenhouse environments throughout the cultivation period. These images were annotated by domain experts, assigning health status labels ranging from 1 (very poor health) to 5 (optimal health). These labels serve as the ground truth for training deep learning models. A comparative evaluation was performed using a separate validation set to identify the most accurate and reliable model for plant health monitoring. Various techniques were applied to enhance dataset diversity and robustness, as detailed in the implementation section.

The proposed framework was implemented using Python 3.8 and the PyTorch deep learning library (version 1.10.1) with CUDA 11.3 for GPU acceleration. All experiments were conducted on a server equipped with an Nvidia GeForce RTX 3090 GPU, featuring 24,268 MB of memory to efficiently handle deep learning computations.

This section presents a detailed analysis of our experimental findings across four datasets: Line 1, Line 2, Line 3, and Line 4. We evaluate the performance of various CNN-based and transformer-based deep learning models (Section 2.4) using key performance metrics: accuracy, precision, recall, and F1-score.

Since average accuracy can be misleading in cases of imbalanced datasets (Thölke et al., 2023), we consider F1-score as the primary evaluation metric. The model achieving the highest F1-score is deemed the most effective for plant health assessment. To mitigate overfitting and ensure stable training, early stopping was implemented.

To further assess model performance, confusion matrices in Figure 5 illustrate classification outcomes for each dataset: (A) Line 1, (B) Line 2, (C) Line 3, and (D) Line 4. These matrices highlight correct classifications and misclassifications, offering insights into model strengths and weaknesses. Based on Table 6 , Swin Transformer was the best model for Lines 1 and 2, while ConvNeXt performed best in Lines 3 and 4.

In Line 2, the model shows strong classification accuracy for Classes 4 (80%) and 5 (84%), but misclassification is frequent between adjacent classes, particularly Classes 3 and 4 (62%). This highlights the challenge of distinguishing gradual health variations, where subtle differences lead to overlaps. A similar trend appears in Line 1, where Class 5 is misclassified as Class 4 in 39% of cases, suggesting the model struggles to differentiate the healthiest plants.

In Line 3 and Line 4, classification follows a similar pattern, with strong diagonal performance but lower accuracy in underrepresented classes. Class 3 in Line 4 and Class 2 in Line 3 exhibit the weakest performance, reinforcing the impact of class imbalance on model reliability. These results emphasize the need for improved data-balancing strategies to enhance the classification of minority health states.

The t-SNE plots in Figure 6 visualize class separability in a reduced two-dimensional space, providing insight into feature distributions. Each point represents a sample, with colors denoting different health states. Overlapping clusters indicate classification challenges, while well-separated clusters suggest effective distinction.

In Line 2 ( Figure 6B ), Class 3 and Class 4 exhibit significant overlap, aligning with confusion matrix findings. Class 3 and Class 5, however, show better separation, reflecting the greater visual difference between moderate and severe health deterioration. Similarly, Line 3 ( Figure 6C ) shows strong overlap between Classes 2 and 3, reinforcing the difficulty in distinguishing early-stage plant stress. In contrast, Class 5 is well separated, confirming distinct features in severely affected plants.

Line 1 and Line 4 follow similar patterns ( Figures 6A, D ), where adjacent health states overlap while distant ones are clearly separated. Notably, better separation is observed in classes with larger training samples, highlighting the role of dataset size in feature learning. These findings stress the importance of augmentation techniques and loss adjustments to improve class differentiation.

To interpret model decisions, Grad-CAM heatmaps ( Figure 7 ) highlight the most influential image regions during classification (Selvaraju et al., 2017). For CNNs, activations were extracted from final convolutional layers, while for transformers, they were taken before the last attention block.

Heatmaps show strong activation in the upper plant regions, suggesting that leaf structure, color, and texture play key roles in classification. However, Figure 8 reveals some challenging cases, especially when models misfocus on background plants, leading to misclassification. This issue arises when target plants are partially occluded or closely positioned to others with different health states. Overall, the trained models capture relevant plant features, yet background interference remains a limitation.

Across all lines, Figure 9 showcases diverse examples of correctly classified plants, emphasizing the adaptability of each model to its respective dataset. The high-confidence predictions suggest that our models effectively capture key features such as leaf color, shape, and structural integrity, which are indicative of plant health. Moreover, these qualitative results highlight the importance of dataset-specific optimization, as each model exhibits peak performance when applied to the dataset it was best suited for.

Despite these successes, challenges remain in cases where plant health conditions exhibit gradual transitions between classes, which may still contribute to occasional misclassification. This observation aligns with our confusion matrix and t-SNE findings, underscoring the need for further refinements, such as enhanced attention mechanisms and multi-modal inputs integrating environmental factors.

To evaluate the model’s ability to generalize, we revised the data partitioning strategy by dividing datasets based on individual plants rather than random splits. This setup simulates real-world deployment, where models must classify unseen plants instead of recognizing familiar ones from the training phase.

A major concern with random partitioning is that multiple images of the same plant may appear in both training and validation sets, leading to artificially inflated performance metrics due to memorization rather than generalization. To mitigate this, we allocated 70% of plants for training, 20% for validation, and 10% for testing across all datasets. The best-performing models—Swin Transformer for Line 1 and Line 2, and ConvNeXt for Line 3 and Line 4—were used with the same training strategies described in Section 3.1.

Table 8 presents the results of this experiment, showing performance comparable to the previous 80:20 random split. These findings confirm that the models retain high classification accuracy even when evaluated on entirely new plants, validating their potential for long-term monitoring and real-world deployment.

To comprehensibly understand the generalization of the trained models, we generated spatio-temporal health diagrams using new plants not included in training. These diagrams illustrate the evolution of plant health over time, comparing model predictions with expert annotations.

For each cultivation line, the data of unseen plants was selected, and its health trajectory was predicted using the trained models. These predictions were compared with ground-truth labels provided by domain experts. The models correctly classified health status in 83% of cases, demonstrating strong predictive accuracy for time-series plant health tracking.

Figure 10 visualizes health status changes over time, where each subplot represents the progression of an individual plant’s condition. The dashed red line indicates ground truth health status, while the model’s predictions are plotted over time. These results suggest that the framework is effective for continuous health monitoring and could be integrated into precision agriculture decision-support systems.

To analyze broader trends, Figure 11 visualizes the evolution of plant health across all cultivation lines and plants using expert annotations. The horizontal axis represents time (weeks), while the vertical axis shows health states (1-5), including state 0 for plant mortality.

Each plotted point represents the health status of an individual plant in a given week, capturing fluctuations in health conditions. The red and blue dashed lines indicate moving averages, providing insight into overall trends. This visualization helps track:

The successful deployment of the proposed plant health monitoring framework in real-world agricultural settings requires careful consideration of scalability, interoperability, and regulatory compliance. These factors determine the framework’s applicability to large-scale commercial farming, integration with existing agricultural technologies, and adherence to industry regulations, plant health monitoring systems must address three key criteria: