Since tip-burn manifests on the leaves tip, it is mandatory to acquire images with a top view of the whole table. We do so by taking images with an HR digital camera fixed above the rolling table shown in Figure 2. A table is a base on which plants are grown. Each table assembles into multiple trays, which in turn are further divided into multiple cells where plant seeds are placed.

We collected images of size 4.64E + 3×6.96E + 3×3 of the tables, using a camera Canon32.5 APS-C of 32.5 megapixels located above the rolling tables (shown in Figure 2). The whole set is made of 43 images, 30 for training and 13 for validation and testing. Images were collected in a period of tip-burn spread. As a tip burn is about 5×5 pixels in the camera image, we have devised a splitting process that allows to zoom into the table image. We split the 43 images of size 4, 640×6, 960×3 into smaller images of size 64×64×3, with an interface we have prepared for the task, and collected 2,127 images of tip-burn. We have automatically selected the same number of images of healthy plants, ensuring to be healthy by correlated with the stressed images from the same table. The images collected by splitting the original table images have been then augmented to finally obtain a training set of 16,323 images of tip-burn stressed and healthy plants, a validation set of 5,596 images and a test set of 1,399 images.

For the purpose of illustrating our method on other datasets, we used the PlantVillage, PlantLeaves, and CitrusLeaves datasets available on Tensorflow.org.

Preliminaries. The main practicality of weakly-supervised semantic segmentation methods (WSSS) is to avoid the resource-demanding manual labeling of each pixel of the categories of interest in an image, which is an impossible task for large canopies. Indeed, WSSS transforms the semantic segmentation task into the much less demanding effort of image-level class annotations. The problem is ill-conditioned and difficult, and a large literature is dedicated to the solution of it starting from Zeiler and Fergus (2014) and Zhou et al. (2016), up to most recent contributions (Chang et al., 2020; Sun et al., 2020, 2022; Wang et al., 2020; Wu et al., 2021; Zhang et al., 2021). Semantic segmentation is critical for detecting tip-burn on large canopies due to the difficulty of both identifying it on a dense set of plants and to individually localizing each tip-burned plant within the canopy, as shown in Figure 3. To ensure both identification and localization, we develop a new method for weakly-supervised semantic segmentation for the tip-burn stress (and for the visible disease in plants disease datasets) by defining a network pipeline using attention-based splitting, classification, and graph convolution.

A crucial aspect of our model is that we adopt the same classifier for both the image and the patches, suitably resized. For this idea to work, it is required that feature properties are shared between the image of the object as a whole and the image of sub-parts of the object. For example, any subset of the image of a canopy shares similar features with the image as a whole. See Figure 4, last image of the upper strip captioned as ‘input image'. Similarly, a leaf and part of a leaf have the same feature properties. This often occurs in natural images, though it is not true, for example, for a tree, which has different features for the trunk and the crown. We define this characteristic of an object feature property as the principle of decomposition. In this work, we show that this principle is valid for both stress detection and segmentation on large canopies and for disease detection and segmentation for leaves (from the cited datasets), which is the domain of interest in this work.

We consider a classification model f_ℓ(C|X, Y, θ), where C indicates the class a sample image X belongs to, Y = {1, …, c} is the vector of training labels, ℓ indicates the size of the images accepted by the network, and θ are the network parameters. The classification model maps each sample X to pX[C], which is the probability vector for the class C given X, as estimated by the softmax activation.

Let X be an N×M×d tensor specifying an image, and X^⋆ be any connected sub-tensor of it of size n×m×d, with m ≤ M and n ≤ N, where connected means that chosen row and column elements n and m from X are consecutive. We say that X enjoys the principle of decomposition if, given a deep classifier f_M×N(C|X, Y, θ) with probability p of correctly classifying X, with respect to classes C, we expect that it correctly classifies S(X^⋆) with approximately the same p. Here, S is a suitable scaling transformation, including appropriate filtering, transforming X^* to X^⋆′ having the same size as X.

Pre-processing and classification model for tip-burn on large canopies. Tip-burn pre-processing exposes three components. The first component is splitting the canopy image into two images of size (4.64E+ 3×6.96E + 3×3) representing half-table, then into all the sub-trays, and further each tray into 16 input images of size 352×576×3. The second component is the augmentation of the training images of size 64×64×3, by random rotation between 0 up to 90 degrees, flipping up and down and left to right, color quantization to 8 colors, zooming in by scaling and cropping, zooming out by padding, and finally by Gaussian blurring with random variance σ ∈ (0.5, 2). For the classification model, we have used as backbone Resnet50V2, trained on ImageNet 1000 and fine tuned with Global Average Pooling, drop-out (to introduce stochasticity in the training) and dense layers.

Pre-processing and classification model for the single leaf image datasets. For classifying the single leaf images of datasets, like PlantVillage, CitrusLeaves, and PlantLeaves, we used the same backbone as for the tip-burn. On the other hand, for weakly supervised semantic segmentation, we have also used a multi layer perceptron (MLP) to separate the background from the foreground. Profiting from the simple arrangement of a single leaf on a background of these datasets, we have automatically sampled from each image a patch of size 8×8×3 from each corner, labeling them background, and 6 patches of the same size from the image center as foreground and gave these data to the MLP to learn to separate the background from the foreground.

Local attention by splitting with hard strides. The main interest of splitting an image into patches with hard strides is to obtain the attention map, in a way similar to how the human gaze glimpses a scene focusing on interesting regions. Here, by hard strides, we intend strides that allow for a significant overlapping of the patches or, more specifically, strides that have a dimension much lower than the patch size.

Most of the work for attention estimation is done by the overlapping induced by the strides like when the gaze goes back to an interesting region of the scene several times. Yet, this kind of attention is local, as it does not capture the whole context. To obtain the context, we shall refine this spitting-based attention with spectral graph convolution, described in the next paragraph.

The splitting process and patch classification. Breaking an image into patches is a well-known technique (see Nowak et al., 2006; Zhou et al., 2009; Dong et al., 2011), requiring only algebraic manipulations of tensors. Consider the image X of size M×N×d. The splitting operation, along the spatial dimension, extracts from X patches of dimension (p_x, p_y, d). Here, the splitting combined with strides allows for overlapping the neighbors' patches according to the stride values (s_x, s_y). In some sense, it is like taking the inner product of X with the lower and upper shift matrices A, A′, and their transposed A^⊤ and A′^⊤ with suitable shifts, and then cropping the non-zero values. Or, similarly, convolving X with a shifting kernel and cropping the non zero elements. The number of obtained patches and their configuration depends on the sizes M, N of the image, the number of channels d, the patch sizes (p_x, p_y, d), and the spatial stride (s_x, s_y). k₁ and k₂ are obtained like in the convolution output, though here we do not consider padding:

We denote CO as a configuration of k₁×k₂ patches, each of size p_x×p_y×d. Namely, it is the shaping of k₁ patches on a row and k₂ patches on a column, shown in Figure 5.

Given X∈ℝ^M×N×d, the configuration CO, and a patch X_j in CO, with 0≤j≤k₁·k₂, the patch X_j is resized as S_ℓ(X_j) (including required filtering modes), where ℓ indicates the size of input images accepted by the network f_ℓ(C|X, Y, θ). The value of ℓ changes according to the considered dataset. For the plants, disease datasets, the classification entry corresponds to the size of the image in the dataset, may be reduced as for PlantLeaves, while a patch is proportional to the image size. This shows the extreme flexibility of the splitting process followed by classification which is adaptable to several kinds of backbones.

For each patch in the configuration CO, obtained by splitting the original image, the probability that it belongs to the class of interest (e.g., tip-burn) is estimated by the network f_ℓ resizing the patch to the input size ℓ accepted by the classification network f_ℓ.

The estimation amounts to the softmax applied to the logits of the classifier f_ℓ(C|X, Y, θ), here we used Resnet50V2 as a backbone. After each patch, probability to belong to C is estimated, a configuration of probabilities (CoP) is obtained, as shown in Figure 5. CoP has the same configuration as CO, though each patch π is defined by repeating at each pixel, the probability p computed by the softmax on classifying the patch. When we indicate the probability p_r,c of the patch located at indexes (r, c), we mean the probability p.

A mapping h from CoP to the reconstructed attention map (RAM) is defined by collapsing the patches in CoP into the corresponding pixels of the matrix RAM. Note that while the whole size of CoP is k₁·p_x×k₂·p_y, namely (M·⌈p_x/s_x⌉)×(N·⌈p_y/s_y⌉), with s_x ≪ p_x and s_y ≪ p_y, RAM has the same spatial dimension as the original image, namely M×N. Given a patch π_r,c in row r and column c in CoP, and a pixel at location (i, j) in π_r,c, the tuple ((r, i), (c, j)) is mapped by h to the pixel (x, y) in RAM, as follows:

Given Equation (2), we also obtain a matrix A_overlap by counting all times a pixel from CoP hits the corresponding pixel of RAM. Indeed, this matrix specifies how many overlapping patches contribute to a pixel in RAM:

The matrix A_overlap is used to count the accuracy of the classification at the pixel level, and it allows to suitably average RAM. The averaged RAM is obtained as follows:

Figure 5 shows an example of CO representation, of CoP, of the matrix A_overlap, of RAM given a random image X with tip-burn highlighted. Where, here and from now on, we denote RAM^⋆ RAM.

We can see that the accuracy of RAM is determined by the strides. For example, if the stride is s_x = s_y = 10, we have an accuracy at the level of a region of size 10×10, and if the stride is s_x = s_y = 1, the accuracy is at the pixel level allowing to effectively label each pixel. Clearly reducing the stride increases the number of patches of the same image. The average increase of the number of pixel is by a factor of 9.

Refining by graph convolution. The RAM results in pseudo segmentation masks for the tip-burn stressed leaves found in the RGB image, following the pipeline splitting-classification-reconstruction. Differently from CAM (Zeiler and Fergus, 2014; Zhou et al., 2016), RAM highlights in the same map all objects of interest quite accurately. Moreover, while in CAM the result is obtained by the gradient of the softmax outcome, with respect to the last feature map, which has very low resolution, thus requiring significant resizing inducing blurring, here we do not need any resizing, as we can obtain the original image by a single step merging, according to Equation 2. Despite classification accuracy for tip-burn is 98.3%, and for the other datasets is no less than 96%, there is still noise on the attention map because classification is done on S_ℓ(X), namely on the resized patch, given the decomposition principle.

Comparing the size of a patch in CoP and the size of the probabilities highlighted in RAM in Figure 5, we can note they have different sizes. This is due to overlapping and projection, which augment the resolution of the probability from uniform in a patch of size 64×64 to uniform on a patch of size 8×8. Indeed, the RAM probability resolution is 8×8. Having in RAM a higher probability resolution, we re-propose the splitting into sub-patches with size (s_x, s_y, d), namely of size 8×8×d, d ∈ {1, 3}, for both the RGB image of size 352×576×3 (see the paragraph above, on Preprocessing and Classification, and Figure 6) and the RAM of size 352×576×1. A schema of this further splitting follows:

The goal is to use CoP_new as supervision for training a convolutional graph network (GCN) improving the semantic segmentation accuracy obtained by classifying the patches. This further splitting step obtains a CO_new and a CoP_new, as specified in Equation (5), from which we obtain the features and the labels for the GCN. The softmax of the GCN classifies the nodes of the GCN, inducing an effective semantic segmentation for tip-burn for each image in the dataset, and similarly for the other datasets.

Following Kipf and Welling (2016), a number of approaches have experimented with graph convolution (GCN), especially on non grid structures. Though, recently, an increasing interest is devolved to apply graph convolution on images, for segmentation and attention purposes, such as Li and Gupta (2018) and Hu et al. (2020). Here, we apply an unsupervised node classification, conditioning the graph model both on the data and on the adjacency matrix via graph convolution. As a matter of fact, we are going to generate the adjacency matrix for the graph G = (V, E), fully unsupervised. We construct a graph for each RGB input image X of size 352×576×3, fixing the size of the nodes, so as to put the graphs in a batch.

We take the patches as node features, labeling them with a one hot encoding vector obtained by thresholding the score p_c,r in CoP_new of patch π_c,r ∈ CO_new. More precisely, we flatten each mini-patch of size 8×8×3 into a vector x ∈ ℝ^k, k = 192 and stack all the flattened patches into a matrix Xφ∈ℝn×k, n = 3168. To obtain a corresponding ordering, we use an index function idx:(r, c) → i, idx(r, c)=w(c−1)+r=i with w the number of rows, r and c the row and column indexes in CO_new and in CoP_new, respectively, and i the corresponding index in X_φ. X_φ is the input matrix to the network. At each layer of the network, a feature matrix is generated, starting with X_φ.

To connect subsets of nodes, based on their feature similarity, we generate the adjacency matrix Adj, which is symmetric and of size n×n, as follows. We keep the indexing idx to maintain the correspondence between CO_new and X_φ and between CoP_new and the labels. For each sub-patch, we estimate a non-parametric probability by computing the histogram using both the RGB and the HSV color transformation of the sub-patch and collapsing the 64·3·2 vector into a histogram with 64 bin-edges. For each pair of histograms q_i, q_j, we compute the Shannon-Jensen divergence:

The choice of JSD is required by the need of the adjacency matrix Adj to be symmetric. Then, two nodes v_i, v_j feature vectors x_i and x_j are similar, hence connected by an edge e_i,j∈E if JSD<β, we have chosen β=0.005 for the tip-burn dataset. Given the n nodes in V, the diagonal degree matrix D adds for each node the number of its connected ones. The normalized graph Laplacian matrix is Lnorm=In-D-12Adj D-12=UΛU⊤. Here, Λ is the matrix of the eigenvalues and U is the orthogonal eigenvectors. The graph convolution g_θ(L_norm) ⋆ X_j using L_norm is the spectral convolution, based on obtaining parameter filters from the eigenvectors of L_norm in the Fourier domain. Several simplifications have been proposed, we refer the reader to Defferrard et al. (2016) for spectral convolution in the Fourier domain and the approximation of the L eigenvectors by Chebyshev polynomial up to the K-th order. Kipf and Welling (2016) obtain a GCN by a first-order approximation spectral graph convolution. They define K = 1 and reduce the Chebyshev coefficients to a matrix of filter parameters.

The feed forward propagation of the GCN is recursively defined as:

Here σ is an activation function, H(t)={h1(t),…,hn(t)} are the hidden vectors of the t-th layer, with hi(t) the hidden feature vector of the node v_i.  is defined as follows. A = Adj + I_n, to include self loops, since nodes are self similar, D~ii=∑jAij and Â=D~-12AD~-12, so as to be normalized, I_n is the identity matrix of size n×n. The role of  is to aggregate information from connected nodes. W^(t) is the weight matrix to be learned. The dimensions are as follows:

Here, u_t and u_t+1 are the sizes of the hidden layers. A 3 layer GCN has the form:

Where the softmax is applied row-wise and

The optimization of the GCN uses cross-entropy loss on all labeled nodes (Kipf and Welling, 2016), where here the labels are the one hot encoded values obtained from RAM_new. Let us denote by I the indexes of the nodes and by Y_i,l an indicator which has value 1 if node v_i has label l:

According to the number K of layers, a GCN convolves the K-hop neighbors of a node, essentially clustering similar nodes, according to their probability labels and features. We use simple 3 layers GCN, since in the end tip-burn stresses on leaves are very small and rare. An overview of the whole learning process is given in Figure 6.

The GCN adjusts the RAM by looking at the context, going beyond the localized estimation of splitting plus classification. GCN estimates the probability that a node, corresponding to features of a patch 8×8×3, belongs to tip-burn or not, by updating the belief that two patches are similar. At the end of the training, CoP_new is updated with the new distribution. In Table 2, in Section 5, we show the advantages of the GCN by ablation.

Reconstruction. Given the initial image of the dense canopy, the question to be explored is “which plant suffers tip-burn stress and where it is?” including counting would not be so useful. Consider that when tables are unrolled, from the position of the plant on the table, it is possible to go back to the cell the plant comes from, and possibly revise its growing conditions, or make useful statistics. It is therefore pivotal to localize the stress segment on the table image. It turns out that by the proposed model it is extremely easy.

In fact, as noted in the paragraph on splitting, reconstruction is automatically done by projecting back a pixel in CoP into a pixel in RAM by Equation (2). Obviously, it can be done for any image, not only for the maps but also for RGB images.

Reconstruction is done both when the stride s_x>1 and s_y>1, that is when the splitting generates sub-images that overlap and, obviously, when they do not overlap. This, in fact, can be done for all the steps of splitting, from the table canopy up to Ram_new and for its dual RGB image, and again back to the large table canopy.

The back process requires preserving just the patches size for the maps and the scores, at each layer of the splitting. Then, the process is simply recursively applied to go from the patch up to the image of the whole canopy. Note that for the semantic segmentation, we need only to preserve the score vectors estimated by GCN. An image of a partial reconstruction of the half table is given in Figures 4, 6.

The whole model is implemented in Tensorflow 2.5, on a GeForce RTX 3080, 300 HZ. For the ResNet50V2, we use the Keras API in Tensorflow. We used the Keras functional API for fine tuning the model, with all the provided advances, such as early stopping, and learning rate decay. For early stopping we used patience 4, with delta 0.001. For reducing the initial learning rate when on a plateau, we used a factor of 0.2 and a minimum learning rate of 0.001 starting with an initial learning rate of 0.1. For the loss, we used categorical cross entropy with Adam as the optimizer (Kingma and Ba, 2014).

For the splitting and reconstruction, we use Tensorflow GradientTape, as the gradient computes both A_overlap and the mapping between CoP and RAM for both Equations (2) and (3).

We have implemented a good part of the GCN including the adjacency matrix, the features vectors and the joining step, to transfer probabilities, in Tensorflow. We used much intuition from DGL, an open-source graph library introduced by Zheng et al. (2021), though DGL is implemented in PyTorch. We also get inspired by Spektral of Grattarola and Alippi (2021), an open-source Python library, for building graph neural networks with TensorFlow and Keras interface. As specified in the Method Section, we have been using cross entropy loss and Adam optimizer like in the classifier together with early stopping.

The main contribution of our work is weakly-supervised semantic segmentation with the only supervision being the image class labels, whether there is tip-burn or not. For classification, we have been using Resnet50V2, because it is quite flexible, and fine-tuned it. As we have already mentioned, we expect that if f is a classifier that classifies correctly X with probability p, if the classifier generalizes well, it would classify the resized X, namely S(X), approximately with the same probability p. This is shown to be correct for tip-burn and plant disease datasets CitrusLeaves, PlantLeaves, and PlantVillage, according to the principle of decomposition.

Stress and disease detection. For training tip-burn CNN classification, we used 30 out of 43 images and compared our work with Gozzovelli et al. (2021), where DarkNet was used. For classification of the plant disease datasets, we considered the following recent works: Sujatha et al. (2021), Khattak et al. (2021), and Syed-Ab-Rahman et al. (2022) for CitrusLeaves; Mohameth et al. (2020), Mohanty et al. (2016), Chen et al. (2020), Agarwal et al. (2020), and Abbas et al. (2021) for PlantVillage; for PlantLeaves, we expose only our approach as there are no recent contributions.

Results are shown in Table 1, considering validation accuracy, as usual. The best accuracy in class is highlighted in bold. We note that for a number of species in PlantVillage, we obtained a validation accuracy of 1.0, in few epochs. Since we use early dropping, this was not caused by overfitting, as shown in Figure 8, where it can be observed that for all datasets we have a small number of epochs. Note that we have also removed the background because, according to Barbedo (2019b), accuracy drops without the background. It seems possible that the background induces overfitting. In Figure 8, we show some results motivated by changing the patience value in early stopping.

We split the leaves datasets 70/30% between train and validation plus test as in Mohanty et al. (2016). As shown by Mohanty et al. (2016), GoogleLeNet is the best backbone for plant disease detection, and we recall that the assessed performance of Resnet50V2 on the Imagenet validation set is 0.760 on Top-1 accuracy and 0.930 on Top-5 accuracy.

Testing accuracy by manually labeling ground-truth. Typically, accuracy metrics for segmentation are F₁, in the context of segmentation referred to as Dice similarity coefficient, and Intersection over Union (IoU), both required to compute the corresponding pixels (true positive), the exceeding pixels (false positives) or lacking pixels (false negative) between the ground truth mask and the estimated masks. Because, in none of the available datasets, we have ground truth masks available, we introduce the patch-based method that is not too demanding to obtain an approximate Dice coefficient. Here, approximate means that instead of computing the pixels we compute the super-pixels and also it means that we use a reduced number of test samples.

We consider 1 test image from the table canopy images and 20 test images for each of the plant disease datasets. Note that 1 test image is the half image of a table canopy, and it amounts to 346,464 images of size 64×64×3, that is 2, 406·16·9.

Now, assuming that we have segmented the test images, by the automatic decomposition and recomposition process, by definition of the model, we have made available all the patches that contribute to the final estimated segmentation. These patches are actually vectors Z holding the probability that the corresponding RGB vector is of class tip-burn or not, as estimated by the GCN and similarly, for the other datasets. At the same time, according to the described model, there is a one to one correspondence between the patches in CoP_new and CO_new and there is a correspondence, by Equation (2), between the patches and the attention map RAM, hence the image, and the final segmentation map estimated. So, it is enough to choose the patches in CO_new. The manually chosen patches are immediately aligned with CoP_new and the final segmentation. That is, suppose we have chosen a patch X_j which will be of size 8×8×3, by definition of the model, then we have automatically selected 64 pixels, and the process is significantly sped up. Once the patches are selected, we know the corresponding value in the segmentation map and can compute both the Dice similarity coefficient at sub-patch or super-pixel level, instead of pixel level, and the IoU. Let X = {X_j|X_j ∈ selected}, where each selected patch has value 1, and Y the corresponding patches in COPnew′ with value Z computed by the GCN, the DSC_patch= 2(X∩Y)/(|X| + |Y|), with | · | the cardinality.

Table 2 gives the results for the approximate meanIoU and F1 (Dice similarity coefficient), computed according to sub-patches (super-pixels) in place of pixels, and according to a limited subset of the test images. We give some ablation too.

Ablation. Consider first the segmentation using just thresholding of the attention map RAM. Introducing the CGN with two layers, we observe an improvement for all models. Extending the GCN to three layers, we observe that accuracy improves for all models but for PlantVillage. It is interesting to note also that doubling the number of nodes in the GCN lowers the accuracy for all models, which is reasonable because we have to choose patches with lower probability of being tip-burn.

In Figure 9, we provide some qualitative results facilitating an understanding of the extremely good results of the model.