Multi-Modal Deep Learning for Weeds Detection in Wheat Field Based on RGB-D Images - Xu, Ke; Zhu, Yan; Cao, Weixing; Jiang, Xiaoping; Jiang, Zhijian; Li, Shuailong; Ni, Jun (Frontiers in Plant Science, 2021)
Single-modal images carry limited information for object representation, and RGB images fail to detect gramineous weeds in wheat fields because of their similarity to wheat in shape. We propose a framework based on multi-modal information fusion for accurate detection of weeds in wheat fields in a natural environment, overcoming the limitation of single modality in weeds species recognition. A three-channel network combining modality independence and complementarity is designed based on the concept of multi-scale object detection, and ensemble learning is performed by weight assignment at the decision level. An experiment shows a mean average precision (mAP) of 36.1% for gramineous weeds and 42.9% for broadleaf weeds, and the overall detection precision, as indicated by intersection over ground truth ( IoG), is 89.3%, with weights of RGB and depth images at  =0.4 and =0.3. The results suggest that this model can accurately detect the dominant species of weeds in wheat fields, and that multi-modal fusion can effectively improve model detection performance.