Machine Learning-Based Classification of Powdery Mildew Severity on Melon Leaves

During the last decades, precision agriculture benefited from advances in robotics [1, 2], computer vision [3] and artificial intelligence [4, 5] to automate the monitoring of crops [6] and harvesting [7]. However, some activities of major importance for agriculture are still to take benefit from these advances. One such activity is plant variety testing. To register and protect a new variety in a country, a plant breeding company has to follow a process managed by a national examination office within an official framework. The national examination offices run tests to register new varieties in the official catalogue, protect them with « plant variety rights » and post control of certified seed lots. Currently, most of these tests are based on manual measurements performed with visual inspection. This is an issue for the sake of efficiency due to the time consuming nature of these tests. In this context, we focus on one of these plant variety tests. We propose an automated algorithm to detect and quantify the presence of powdery mildew on melon leaves via in vitro imaging to assess the resistance capability of the tested varieties. Powdery mildew is a fungal disease infecting melon leaves and causing a major reduction of yield. The typical symptoms of powdery mildew are white colonies on the leaf surface consisting of mycelium and spores of the fungal pathogen. We first describe the current manual method and then explain the computer vision procedure that we propose, based on machine learning and fusion of front and back light images. The performance of this automated procedure is compared with the manual method and previous automated methods before conclusion.

Plant disease detection using deep learning has attracted many attention in the recent past [8,9,10] due to the large variety of conditions in which diseases can be studied including conditions of plant-pathogen interactions (virus, bacteria, fungi), environmental conditions (field, controlled environment, in vitro, ...), imaging modalities (RGB, thermal, fluorescent, ...) and observation scales (tissue, leaf, canopy, ...). As closely related work, for powdery mildew detection observed on foliar disk in vitro, a spatio-spectral analysis based on hyperspectral images of wine grapes was developed to classify powdery mildew infection levels [11]. An accuracy of 87% was reported to classify “healthy”, “infected” and “severely” diseased bunches. In another work, a machine vision based phenotyping system was developed to assess the severity of grapevine powdery mildew [12]. The system is based on a high-resolution camera and a long working distance macro-focusing lens. The system acquires an image of each foliar disk inside a Petri dish and requires an XY motorised stage to move above one foliar disk to another. A GoogleNet neural network architecture was trained on 9920 images of two classes “infected” and “not infected”. The training lasted 3.4 h. The resulting CNN had a classification accuracy of 94.3%. By contrast with these methods, our computer vision system requires only a simple RGB camera with standard resolution and a lighting device. This simplicity and low-cost is important for dissemination of the method as the system is dedicated to pathology tests performed by biologists. Also, acquisition time is important as the global objective is to implement a high-throughput phenotyping system. Unlike microscope-based images of [12] that catches only a single foliar disk at a time, we acquire 9 foliar disks in the same snapshot. The previous works have improved disease detection accuracy by investing in the imaging system (hyper-spectral camera and microscope). On the side of optics, we propose to fuse front and back light images to improve classification accuracy and thus implement a high-throughput phenotyping system at a relatively low cost.

The current manual procedure to assess melon leaves resistance to powdery mildew is as follows. First, biologists extract foliar disks from melon leaves (as illustrated in Fig. 1(a)) and position them inside a Petri dish. A control foliar disk (very sensitive to powdery mildew), positioned at the center of the Petri dish, serves to validate the presence of powdery mildew. In the next step, foliar disks are inoculated with powdery mildew powder inside Petri dishes and left for an incubation period of 10 days. After that, biologists use a binocular loop to visualize leaves and assign an ordinal score according to powdery mildew density on the leaf surface as shown in Fig. 1(b). The encoding of the scoring is provided in Table 1.

Schematic visualisation of the current procedure. (a): Petri dish containing melon foliar disks inoculated with powdery mildew. (b): a biologist visualizes and annotates powdery mildew propagation using a binocular loop. (c): data generated after assessment following the annotation scale of Table 1. Each foliar disk is saved in a CSV file with its corresponding class.

Full size image

To automate the manual procedure described in the previous section, we propose to follow the pipeline given in Fig. 2. 70 Petri dish images are acquired with a digital color camera with resolution of 2448 by 2050 pixels. The size of each foliar disk is approximately 120 000 pixels. The camera is positioned vertically above the Petri dish as shown in Fig. 2(a). As illustrated in Fig. 2(b), Petri dish images are acquired under two lighting techniques, front and back light. After RGB to HSB conversion, the brightness channel of both images (front and back light) are fused in a linear blending to enhance the contrast between the lymb and the fungi.

Visual scheme of the proposed computer vision procedure. (a): imaging device. (b): fusing images following a linear blending. (c): generated fused image. (d): front, back and fused data sets. (e): The foliar disks are segmented from Petri dishes using the convolutional neural network architecture for semantic segmentation UNET [13] and cropped individually. (f): foliar disks are assigned to their corresponding ground truth, produced by an expert in the current manual procedure. (g): feeding training images to a supervised machine learning classification algorithm.

Full size image

The partition of foliar disk images per class in front, back and fused data sets is as follows: 180 images for “Resistant” class, 62 images for “Moderate” class and 131 images for “Sensitive” class. This data set is rather small in this work (compared to standard large data sets in machine learning). This constitutes a possible limit to the use of an end-to-end deep learning method due to the tendency of overfitting. Instead, a shallow supervised learning scheme based on the concatenation of a deep-feature extraction [14] stage followed by a linear support vector machine classifier was used for the comparison of the performance with different images (fused, front and back light). Deep features trained on ImageNet from well-known architectures were tested in this study including VGG16 and Resnet50. In addition, a small end-to-end CNN model with the architecture shown in Fig. 3 was fine-tuned on a validation data set of 20% of training images. The accuracy of the classification of all tested models was computed per class from the confusion matrix. Due to the lack of enough data and the imbalance classes in our data set, a data augmentation was used to improve the classification accuracy and to be able to compare fairly with a deep learning architecture. Data were augmented to force invariance to rotation since the leaves are randomly positioned in the Petri dish invariance to shearing and zoom to allow for robustness to some plasticity of the leaf tissue. The mix parameter \(\alpha \) in the linear blending to fuse front and back light images was chosen to maximize the contrast between powdery mildew and healthy lymb computed with the Fisher ratio which is defined as

where \(\mu \) is the mean pixel value in the selected area and \(\sigma \) is the standard deviation of the pixel values in the selected area. The Fisher ratio was computed on fused images generated by varying \(\alpha \) from 0 to 1. The optimal value of \(\alpha \) for which Fisher ratio is maximum, equals to 0.1. This value was applied in the linear blending to generate fused images for classification.

CNN architecture proposed.

Full size image

Classification performances of the supervised machine learning algorithm described in Sect. 4, are given in Table 2 and Table 3. Best results, highlighted in blue were systematically obtained with the fused images for the three tested classifiers with or without data augmentation. Highest scores are obtained with association of deep features from Resnet50 coupled with a linear SVM with data augmentation. Other classical classifiers such as random forest or non linear SVM were also tested (not shown) but results were not significantly improved. The confusion matrix for this best classifier is illustrated in Fig. 4 which shows that most errors come from the confusion between moderate and sensitive classes. Finally, these two classes are merged by biologists when varieties are registered officially. The classification accuracy achieved in classification of resistant and sensitive levels are provided in Table 3 with best performances culminating at 95% accuracy.

Confusion Matrix of Resnet50 algorithm to classify powdery mildew on 3 infection levels: resistant & moderate & sensitive.

Full size image

The previous section presented successful results for the classification of the presence of powdery mildew in foliar disks containing melon leaves. The obtained performances are similar to the recently published work on the classification of powdery mildew in two [11] or three classes [12] as presented in the related work section. It is to be noticed that the closest related method of [12] is applied to another crop but in a similar in vitro imaging conditions protocol. While neural networks are also used as the main element of the image processing pipeline, [12] notably differs from our approach. The work of [12] focuses on a metrological measurement of the powdery mildew performed with a high resolution imaging system enabling to detect individual mycelium. By contrast, we propose an ordinal classification of the foliar disk corresponding to the final annotation of an expert. We investigated the possibility of addressing this less demanding task by considering foliar disks as a texture with a much lower spatial resolution. Working at such degraded resolution could constitute a risk of loosing accuracy specially at the low grade of the development of the powdery mildew. However, we demonstrated that this was not the case when considering the final score recorded in variety testing which only keeps two classes (resistant, sensitive). Our method is especially suitable for high-throughput application of variety testing to avoid an overwhelming increase of data while keeping the accuracy of the tests at the current level. The performance of the classical CNN architecture is promising and should exceed the 95% accuracy of Resnet50 in case more training images were provided. A comparison on the same samples of our classification approach with the metrological quantification of [12] would be an interesting perspective.

In this paper, we presented a computer vision-based approach to automate a plant variety test performed to quantify the severity of powdery mildew infection levels on melon leaves. We demonstrated that fusing front light and back light images improved powdery mildew contrast. This fusion resulted an improvement of 10% accuracy with a very low-cost imaging system. Also, we highlighted the achievement of this performance level with a standard spatial resolution, while the state of the art on this problem reported the use of microscopic resolution to track individual mycellium. The use of deep features Resnet50 coupled with a standard SVM achieved an accuracy of 95%. This automated approach is expected to improve the speed and accuracy of disease detection and could be extended to other in vitro pathology tests. The fusion of front and back light was limited in this communication to a simple linear blending. In the future, we plan to explore various approaches of image fusion [15] to optimize the combination of front and back light images.