The train set and validation set used in our experiments are based on the Plant Village dataset (Hughes and Salathé, 2015). It contains 82,161 pictures of plant leaves of varying sizes from 24 plants in 55 classes. The data set contains images with clean background and congested background, as shown in Figure 1. Clean background images consist of isolated leaves with uniform backgrounds, while cluttered background images comprise partial or full images of plants taken in a natural background. The number of images in each class ranges from 43 to 6,359. This data set is divided into three different sets. PlantLeaf1 contains 18 classes which contain pictures with a cluttered background. None of the images in this dataset contains laboratory-conditioned images. PlantLeaf2 contains 11 classes, which constitute both clean and cluttered images. Clean background images were used in the training, while cluttered background images were used in the testing of this dataset. PlantLeaf3 consists of 16 classes of 11 plants. These classes contain both clean and cluttered images, whereas the number of images per class varied from 892 to 5,507. This dataset consists of 10 classes of 10 different crop species and 6 classes of tomato plants infected by different diseases. The number of classes and frames for each PlantLeaf data set is detailed in Table 1.

To generalize the model and ensure a robust model, these image datasets were augmented using different data augmentation processes, such as flipping, random crops, rotations, shifts, and a combination of these techniques. Data augmentation aims to prevent overfitting by training the model to large data created artificially model.

In this section, a structured disease identification method and a lightweight neural network reduction method are proposed. The overall design of this study is shown in Figure 2.

Knowledge distillation is first proposed in (Weng and Preneel, 2011) for model compression. The key idea of knowledge distillation is that the soft probability of trained teachers’ network output contains not only class labels, but also more information about data points. For example, if multiple categories of high probability areas signed to an image, it may mean that the image must be located near the decision boundary between these categories. Therefore, forcing students to imitate these probabilities should enable students’ network to absorb some knowledge that teachers has found in the information outside the training label itself.

In the learning process of knowledge distillation (Shorten and Khoshgoftaar), the student model is trained by imitating the output of the teacher model in the same sample. In the traditional Softmax classifier, given any input image, the model generates a vector S t ( x ) = S 1 t ( x ) , S 2 t ( x ) , … , S K t ( x ) where S t ( x ) represents the score corresponding to the kth disease. We use Softmax as the classifier at the end of the neural network to convert the output S t ( x ) of the neural network into probability Distribution p t ( x ) , as shown in Formula 1. p k t x = e s k t x ∑ j e s j t x (1)

Hinton et al. (2015) proposed that the output of a well-trained teacher model would be infinitely close to the real output of One-Hot coding, which causes useful inter class information to be ignored in the training process, and directly lead to the unsatisfactory training effect of the student model. Therefore, it is necessary to use the temperature scale to “soften” these probabilities, as shown in Formula 2. p ̃ k t x = e s k t x / T ∑ j e s j t x / T (2) where T > 1 is an adjustable super parameter. By adding the parameter T to classifier, students wound similarly produce a softer classification probability distribution p ̃ s ( x ) , thus preserving the probability relationship between different categories of samples. Compared with the traditional One-Hot coding hard tag as the training target, because the soft target output by the teacher model after the Soft-Softmax classifier well retains the probability relationship between different categories of samples, it usually bring better performance. The loss function of students is a linear combination of the typical cross entropy loss function L cls and the loss function L K D in the process of knowledge distillation (Kim and Rush, 2016), as shown in Formula 3–5. L = α L cls + 1 − α L K D (3) L K D = − T 2 ∑ k p ̃ k t x log ⁡ p ̃ k t x (4) L c l s = − ∑ k q k ⁡ log ⁡ p ̃ k t x (5) where T and α are adjustable super parameters. The common choices are T ∈ 3,4,5 and α = ∈ [0.5,0.9], q k ( x ) is the real label of the sample.

As shown in Figure 3, for multiple DNN layers of the teacher model, an matrix is constructed, where is the number of DNN layers selected, and each element in the matrix represents the Euclidean distance between two characteristic graphs with corresponding subscripts. The matrix provides sufficient and general information about the characteristic distribution, so that the extracted knowledge can guide student networks with different structures. At present, most teacher-student frameworks based on knowledge distillation rely on strong constraints at the instance level.

At the same time, the correlation among multiple samples is also valuable for knowledge distillation. Using the sample correlation, the student model can better learn the relationship between different samples. Therefore, in the process of knowledge distillation, this paper takes n samples as the input of neural network at the same time. Because each sample will generate a m × m matrix, we will finally obtain a n × m × m three-dimensional matrix, which is IRM.

Let the input data set of the network be X = x 1 , x 2 , … , x n , n is the sample number of Mini-batch, C i T x i and C i S x i are the output characteristic graphs of the ith sample in the j layer of teacher and student models respectively, f a . h T x i an mapping f a . h S x i represents the Euclidean distances of the output characteristic graphs of the gth and hth layers in the teacher student model. Let F i T and F i S represents the set of distances between the jth layer and each layer in the mth layer in the teacher and student models, respectively via Formulas 6.1, 6.2: F j T x i = matrix  f j , 1 T x i , f j , 2 T x i , … , f j , m T x i (6.1) F j S x i = matrix  f j , 1 S x i , f j , 2 S x i , … , f j , m S x i (6.2)

At the same time, the mapping functions are introduced as follows (Formulas 7.1, 7.2): θ : F → G ∈ W m x m (7.1) φ : G → H ∈ E n x m x m (7.2) where G is the distance matrix of m × m and H is the three-dimensional matrix of n × m × m , each element in H represents the distance between the characteristic graph of layer C g and layer C h in the input ith sample.

The IRM formula can be expressed as follows (Formula 8): L I R M = Ψ φ T χ , φ S χ (8) where Ψ is the loss function as Formula 9: Ψ H T , H S = ∑ i ∈ 1 , n j , k ∈ 1 , m h i , j , k T − h i , j , k T 2 2 (9)

In order to avoid too strict constraints, cross layer feature space transformation is introduced as the third type of knowledge, and an IRM transformation is proposed to model the knowledge. The feature space transformation is a more relaxed description than the dense fitting of teacher’s case features in the middle layer. By combining IRM with IRM transformation, this method has more general, moderate and sufficient knowledge than the existing methods. Finally, two loss functions for IRM and IRM transformation are designed and optimized to improve the performance of the student model. Firstly, the mapping function is defined as follows (Formula 10): Φ : χ → D ∈ R n × n (10) where D is a two-dimensional matrix of n × n , let each element of D i T and D i S represents the distance of the output characteristic graph of the x i sample and the x i ′ sample input in the teacher and student models at the jth layer respectively. L I R M − t extracted the transformation knowledge of feature space by calculating the difference change of the feature graph between two layers in the network model, D g T ( χ ) − D h T ( χ ) is the amount of knowledge flow information from layer g to layer h in the teacher network, then IRM-t formula can be defined as follows (Formula 11): L I R M − t = ‖ D g T χ − D h T χ − D g S χ − D h S χ ‖ 2 2 (11)

L logits formula represents the Softmax loss of teachers’ network output and students’ network output as Formula 12: L logits = Y T − Y S 2 2 (12)

Finally, we define an L M T K loss function is used to train student network, which is based on IRM-t transform loss L I R M − t , IRM loss L I R M and Softmax loss L logits  , L G T as follows Formula 13. L M T K = α L I R M − t + 1 − α L I R M + β L logits  + γ L G T (13) where L G T is the loss function between the real tag and the student network output, and α , β , γ are the super parameters. Using MTK loss can optimize the student network and obtains three types of knowledge from the teacher network.

The hardware environment of this experiment includes Intel i9-10900x (3.20ghz) 10 core 20 thread CPU, NVIDIA geforce RTX2080ti 11 GB * 2 server. The software environment is Windows10 64 bit system, CUDA 9.0, cudnn 7.0, PyCharm 2018.2. The front end and back end of the experimental framework for training model are keras and tensorflow, respectively.

In this paper, 128 × 128 three channel RGB images are used to train 200 epochs. The size of batchis 512, the initial learning rate is 0.1, the momentum is 0.9, the weight depth is 1 0 − 4 , and the number of classes is,the random seed is 2. The experimental results of different knowledge distillation methods under different network structures are shown in Table 2.

In our experiment, VGG16, AlexNet, GoogleNet and ResNetis used as the teacher’s network structure, and MobileNet is used as the student’s network structure. The teacher column represents the accuracy of the teacher model. First, a teacher model is trained on plant village with 4 different neural network. After 120 iterations, the heighest accuracy of 4 teacher model reaches 95.85%. The column “baseline” indicates the accuracy of the basic student model. The same sample is used to train the student model on MobileNet and the model parameteris only 28.0 MB. Under the same conditions, the accuracy of the model is 91.57%. Figure 4 shows the accuracy change curve of the verification set during the training of four different teacher models.

The KD column represents the accuracy of the training results after the distillation of basic knowledge. From the table, we can see that the accuracy of the model after the distillation of knowledge has improved compared with the baseline column. AT and SP are the model accuracy of other knowledge distillation methods, L IRM and L IRM−t are the model accuracy of the proposed method. It can be seen that using the knowledge distillation method based on the instance relation matrix to transfer the knowledge from the pretrained teacher model VGG16 to the untrained student model MobileNet, the accuracy of the model had been significantly improved, with the highest accuracy of 93.60%. The network structures of teachers and students used are VGG16 (304.0 MB) and MobileNet (28.0M), and the accuracy is only 2.25% different from that of the teacher model. Moreover, the accuracy of the model with multiple loss functions is higher than that with only IRM loss, which proves that the IRM transformation loss ( L IRM−t ) is useful. Figure 5 shows the visualizations of the activation effects of different convolutional layers after IRMKD distillation. As can be seen from the figure, the model has effectively learned to extract and activate the disease spots on the sample leaves.

To evaluate the performance of the IRMKD method in real-world scenarios, this paper tested the MobileNet model trained using this method on the mango powdery mildew dataset provided by the Plant Protection Research Institute of the Guangxi Academy of Agricultural Sciences. The changes in accuracy and loss function during the model training process are illustrated in Figure 6. From the figure, it can be observed that during the training process, the loss function decreases, and simultaneously, the prediction accuracy on the test set shows an overall increasing trend. Moreover, the model converges rapidly, achieving a good convergence state after 50 iterations, the highest accuracy reached is 95.54%.

This experiment compares VGG16, AlexNet, GoogleNet and Resnet four main neural network structures, and L MTK model’s verification set accuracy, parameter size in three-channel RGB image of 128 × 128 sizes. The network with higher verification set accuracy and model parameter quantity is taken as teacher model. Otherwise, it is taken as a student model. It can be seen from the table that MobileNet had the least network parameters, and its accuracy is slightly lower than VGG16 and Resnet. Therefore, this paper chose MobileNet as the student model in knowledge distillation. The results show that VGG16 can make the model the highest accuracy in the search space. At the same time, the average accuracy of L MTK model optimized by knowledge distillation method is 97.62%, which is slightly improved compared with the other four network structures. The distributed MobileNet model has better performance in memory and average recognition time than other networks. The average recognition time is shortened to 0.218 s, and the model size compression is only 19.83MB, in general, the model after knowledge distillation has higher recognition performance, and can meet the requirements of different application scenarios in recognition time and disk occupation. This proves the effectiveness and feasibility of this method.

In order to compare the performance between our proposed method and other plant disease recognition models, we compared the four methods: generalized regression networks (GRNNs) (Wang et al., 2012a), probabilistic neural networks (PNNs), radial basis function (RBF) (Wang et al., 2012b), BP network with PAC (Wang et al., 2012c). The experimental results are shown in Table 3. The test results show that GRNN and PNN have the highest driving accuracy, 97.27%, 98.06% respectively. It can be seen that these four methods have higher recognition accuracy. The accuracy of RBF neural network, PCA and BP network are 96.06% and 95.44%, respectively, which is slightly lower than the previous four methods. Furthermore, although our method is not as accurate as the 6 methods above, it is clearly ahead of the other methods in terms of model parameters and training speed. In addition, the experimental results also show that the accuracy of the model after pre-processing is improved compared to the model without pre-processing, with an average increase of 0.54%.

Finally, the accuracy of verification set and the change of IRM loss value are visualized. It can be seen that within 100 iterations, with the increase of the number of iterations, the accuracy of verification set increases from about 65% to about 92%, and the loss value of IRM decreases from about 0.6 to about 0.2. The model effect is significantly improved.