A Real-time Citrus Segmentation and Detection System using Mask R-CNN

Ⅰ. Introduction

Low birth and aging phenomenon is accelerating the decrease in agriculture population and rural aging. [1] The demand for the provision of alternative labor force or the development of a more convenient and advanced agricultural system is on the rise. As the global competition intensifies due to the proliferation of FTAs and opportunities for expanding access to export markets coexist simultaneously, researches are being actively conducted in advanced countries to innovate agricultural products and to harness future growth engines through the convergence of agriculture and robotic technology.

Recently, artificial intelligence, sensors, and big data technologies have been applied to various industrial and agricultural fields in line with the advent of the fourth industrial revolution. In particularly the powerful performance of deep-learning technology [2] in the field of image processing is making the world come alive. Examples of utilizing image processing techniques using deep running in the field of agricultural automation include yield prediction [3], pest awareness [4], and quality evaluation systems [5].

The development of an accurate fruit detection system is of strenuous work for fully automated harvesting robots, since vision with perception system needs to be exist before subsequent manipulation and grasping systems. Robots are unable to detect the differences between similar objects and fruit, hence the need for segmentation between the fruits and the background.

This paper is about technology for detecting fruits and mapping mask through object recognition technique. Our goal is to build a system that accurately detects fruits and makes mask from citrus photographs, measures the number of fruits and provides a better environmental friendly alternative for auto harvesting.

This is not only effective in terms of labor savings through unattended and automated systems, but also contributes to quality improvement challenge overseas and market competition with low prices due to the proliferation of FTA.

The composition of this paper is as follows. We will discuss some of the studies related to this problem in Chapter 2. Deep learning algorithms and methods for detecting fruits are described in Chapter 3. Chapter 4 is the experiment based on the photographs obtained from the citrus farm in Jeju Island, and the results will be discussed in Chapters 5.

Ⅱ. Related Works and Background

Various works related to fruit detection shows a significant interest across a variety of orchard for the yield mapping or autonomous harvesting [14]. Detection system is typically the task of finding the different fruit or background object in an image and classifying them.

Past few year, many researchers have mentioned the main problem of fruit detection, like the works issued in [8 – 13], the main problem of building a fast and accurate fruit detection system persists, as found by Kapach, K et al. [14] The high variation in the appearance of the fruits in field settings, including shape, size, color, texture and reflectance properties causes the problem that make hard to detect.

Most of the works presented in the past, address the problem of fruit detection as segmenting fruits from background. Ulzii et al. [15] examined the issue of citrus detection for yield prediction. They developed an image processing system to detect citrus and use the data to predict the yield estimation. the system is based on conversion of RGB image to HSV, noise removal and watershed segmentation[25]. Wang et al. [11] proposed a color and distinctive specular reflection pattern for detecting apples. By using two-camera stereo rig for image acquisition, the system was able to work at nighttime with controlled artificial lighting to reduce the variance of natural illumination. Hung et al. [13] presented a multi-class image segmentation for automate fruit segmentation. Relevant features from data were capture by unsupervised feature learning which is called Sparse Autoencoder(SAE). The classifier output which is used to learn the label association to the multiscale responses is passed into the CRF. They achieved outstanding segmentation performance, but did not present object detection.

More recently, Faster R-CNN[7] were used in the study for sweet pepper and rock melon detection in a greenhouse and showed expanded object detector which is collected from Google Image search.

Our work differs substantially from that study, because our system not only detects citrus but also segment each fruit which is essential for auto harvesting system.

Ⅲ.Materials and methods

This section presents the Mask R-CNN framework for fruit detection in orchards as well as introduces details about transfer learning and dataset of our work and data augmentation techniques, which are used for the studies experimented in section experimental results and discussions.

Fig. 1.

The Structure of Mask R-CNN

Ⅳ. Experimental results and discussions

The orchard data evaluated in this paper are collected during the day in orchards in Jeju Island, Korea. Citrus data was captured by citrus researchers. The data contains multiple rows of orchestral image data. The longer distance between the citrus trees was compensated with a higher resolution picture. In addition to the citrus image, we also collected fruit images to reduce various error between other fruits. Therefore, high resolution images were needed to express fruit well, and this image could be obtained using a handheld high pixel digital camera. The images captured at each orchard span the entire tree and were driven by key experimental objectives of effective output estimation and mapping. The fruit detection work presented in this document is therefore an important component of automatic harvesting.

This section describes the abstinence study for the detection network and evaluates the fruit detection performance related to the number of training images, the transfer learning between the orchard and data enhancement techniques. These studies were conducted using ResNet-101 because its accuracy is very high, but performance evaluation for FPN network is also presented. Finally, there is a simple technique for the proposed source network. Mask R-CNN framework allows for multi-class detection, binary problems are considered for orchard data, and new models must be trained for each fruit type. Limiting the number of classes can generally lead to better classification accuracy [18]. Orchard blocks are generally allowed in orchard applications because one fruit per block is homogeneous without mixing.

In the ResNet-101 and VGG16 networks, the sub-sampling factor is 16 in the final transformation layer, and the smallest possible object size is 16 pixels. To ensure this, all training sub-images were scaled to a shorter side of 800 pixels, which meant enlarging the citrus sub-image. The sub-image dimensions specified in the previous section are selected to rescale fruit of sufficient size later. All networks were initialized with a coco filter and trained until there is convergence of detection performance across the validation set. This was about 4k repetitions for citrus. Further investigation of the initialization and learning rate for the citrus data set will enable faster training.

Figure 2 shows a grid of ground truth objects and their predictions its prediction varies as the images of the citrus is hidden by a leaf. But as you can see in the Table 1 its precision-recall curve scored 0.971. Detection performance is reported using the average precision response to fruit, which is the area of the precision recall curve. As in [6], the result is reported using an F1 score, where the class threshold is evaluated against the set of head-out validation.

Fig. 2.

Grid of ground truth objects and their predictions

Table 1.

Fruit Detection Results (As F1-scores)

ResNet-101 scored the very best between two networks, 0.971 for citrus. This can be attributed to auto-harvesting platform. End to end learning for detection and segmentation. Fig. 3 and 4 show the results of citrus detection.

Fig. 3.

Example of detections and segmentations for each citrus (close shot)

Fig. 4.

Example of detections and segmentations for each citrus (full shot)The Mask R-CNN

detection framework yields state-of-the-art performance on orchard images. It creates a mask for citrus by its shape. The study of transferring learning turnout to be successful as we can see with fig 4. 400 of citrus image data was enough to perform a real-life use for auto-harvesting platform. Data augmentation was very helpful in training a few numbers of data which boosted detection performance. It efficiently reduced a number of images of training by almost 50% which eventually leads to reduce in necessity of labelled data.

The methods of detection presented in this paper can easily be extended to other orchards. The supplied labeling toolbox can be used in the harvest process, and a parallel training test process can alter the detection performance according to the increasing number of educational images. The labeling process can then be terminated according to the labeling budget and/or performance requirements. For small fruits, the image needs to be resized.

Although the minimum fruit size is less than 16 pixels, the fruit resolution can be harmful to labeling and sensing performance. The results are thought to use simple data growth techniques, such as image frying and relocation, and transfer learning between other fruits is not necessary. Transfer learning may still be important when the baseline task is very similar to the destination job. For example, a trained model in a given apple data set can still be useful for initializing the sense network for another captured apple data set using different lighting conditions and/or different sensors. However, due to these variations in the data set, further testing of how the model is applied is required [5] shows reasonable qualitative performance for a new set of data without re-learning.

Ⅴ. Conclusion

This paper introduces a fruit detection system for image data captured in orchards using the Mask R-CNN detection framework. In order to better understand the practical development of such a system, we conducted ablation studies on one of famous fruits in Korea. The study of detection performance for the number of training images showed the amount of training data required to reach convergence. Transfer learning analysis showed that the transfer weight between orchards did not result in significant performance improvements on a network that was directly initialized from the features of highly generalized coco dataset. Data expansion techniques such as flip-and-scale augmentation have improved performance on a variety of training images and achieved comparable performance in less than half the number of training images. In this study, the author's previous study obtained the best detection performance, and the citrus F1 score was 0.9. In advanced applications such as auto-harvesting, masks are extremely helpful in controlling robots. Future research will integrate the detection output from the mask R-CNN with yield mapping to provide object association between adjacent frames. Additional analysis for fruit detection is performed to understand a dataset representing the same fruit, a dataset captured under different illumination conditions, a sensor configuration, and transmission learning over a period of a year.

For future work, I plan to develop a platform that can take photographs, label them with a mask first, which will help people making mask easily and for those object that was not masked user can mask by themselves. And also by using GUI structure people will be able to train their own train set that is custom to agriculture easily.

References

• Roh, Jae-Sun, Jung, Jin Hwa, Jeon, Ji Yeon*, Returning Farmers and the Aging of Farm Households: Prospects of Changes in Rural Population by Their Influx, Journal of Korean Society of Rural Planning, 19(4), p203-212, (2013). [https://doi.org/10.7851/ksrp.2013.19.4.203]
• LECUN, Yann, BENGIO, Yoshua, HINTON, Geoffrey, Deep learning, nature, (2015), 521, 7553 p436.
• Aggelopoulou, A. D., Bochtis, D., Fountas, S., Swain, K. C., Gemtos, T. A., & Nanos, G. D., Yield prediction in apple orchards based on image processing, Precision Agriculture, 12(3), p448-456, (2011). [https://doi.org/10.1007/s11119-010-9187-0]
• Weizheng, S., Yachun, W., Zhanliang, C., & Hongda, W., Grading method of leaf spot disease based on image processing, In 2008 international conference on computer science and software engineering, Wuhan p491-494, (2008, December). [https://doi.org/10.1109/csse.2008.1649]
• Cubero, S., Aleixos, N., Moltó, E., Gómez-Sanchis, J., & Blasco, J., Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables, Food and Bioprocess Technology, 4(4), p487-504, (2011). [https://doi.org/10.1007/s11947-010-0411-8]
• BARGOTI, Suchet, UNDERWOOD, James P., Image segmentation for fruit detection and yield estimation in apple orchards, Journal of Field Robotics, 34(6), p1039-1060, (2017). [https://doi.org/10.1002/rob.21699]
• Girshick, R., Donahue, J., Darrell, T., & Malik, J., Rich feature hierarchies for accurate object detection and semantic segmentation, In Proceedings of the IEEE conference on computer vision and pattern recognition, p580-587, (2014). [https://doi.org/10.1109/cvpr.2014.81]
• Nuske, S., Achar, S., Bates, T., Narasimhan, S., & Singh, S., Yield estimation in vineyards by visual grape detection, In Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on, p2352-2358, IEEE, (2011, September). [https://doi.org/10.1109/iros.2011.6095069]
• Nuske, S., Wilshusen, K., Achar, S., Yoder, L., Narasimhan, S., & Singh, S., Automated visual yield estimation in vineyards, Journal of Field Robotics, 31(5), p837-860, (2014). [https://doi.org/10.1002/rob.21541]
• Yamamoto, K., Guo, W., Yoshioka, Y., & Ninomiya, S., On plant detection of intact tomato fruits using image analysis and machine learning methods, Sensors, 14(7), p12191-12206, (2014). [https://doi.org/10.3390/s140712191]
• Wang, Q., Nuske, S., Bergerman, M., & Singh, S., Automated crop yield estimation for apple orchards, In Experimental robotics p745-758, Springer, Heidelberg, (2013). [https://doi.org/10.1007/978-3-319-00065-7_50]
• Bac, C. W., Hemming, J., & Van Henten, E. J., Robust pixel-based classification of obstacles for robotic harvesting of sweet-pepper, Computers and electronics in agriculture, 96, p148-162, (2013). [https://doi.org/10.1016/j.compag.2013.05.004]
• Hung, C., Nieto, J., Taylor, Z., Underwood, J., & Sukkarieh, S., Orchard fruit segmentation using multi-spectral feature learning, In Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on, p5314-5320, IEEE, November), (2013. [https://doi.org/10.1109/iros.2013.6697125]
• Kapach, K., Barnea, E., Mairon, R., Edan, Y., & Ben-Shahar, O., Computer vision for fruit harvesting robots–state of the art and challenges ahead, International Journal of Computational Vision and Robotics, 3(1-2), p4-34, (2012). [https://doi.org/10.1504/ijcvr.2012.046419]
• I. Sa, Z. Ge, F. Dayoub, B. Upcroft, T. Perez, and C. McCool, “DeepFruits: A Fruit Detection System Using Deep Neural Networks”, Sensors, 16(8), p1222, (2016). [https://doi.org/10.3390/s16081222]
• Kaiming He, Georgia Gkioxari Piotr Dollar Ross Girshick, “Mask R-CNN” Computer Society, p2980-2988, (2017).
• J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?”, in Advances in neural information processing systems, p3320-3328, (2014).
• Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., ... & Zitnick, C. L., Microsoft coco: Common objects in context, In European conference on computer vision, p740-755, Springer, Cham, September), (2014. [https://doi.org/10.1007/978-3-319-10602-1_48]
• KRIZHEVSKY, Alex, UTSKEVER, Ilya, HINTON, Geoffrey E., Imagenet classification with deep convolutional neural networks, In: Advances in neural information processing systems, p1097-110, (2012).
• HE, Kaiming, et al., Deep residual learning for image recognition, In: Proceedings of the IEEE conference on computer vision and pattern recognition, p770-778, (2016).
• M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, The PASCAL Visual Object Classes (VOC) Challenge, IJCV, (2010). [https://doi.org/10.1007/s11263-009-0275-4]
• DENG, Jia, et al., Imagenet: A large-scale hierarchical image database, In: Computer Vision and Pattern Recognition, CVPR 2009. IEEE Conference on. IEEE, p248-255, (2009).
• Hyuk Jin Yang, Dong Hyun Kim, Yeong Geon Seo, Noise-robust Hand Region Segmentation In RGB Color-based Real-time Image, Journal of Digital Contents Society, 18(8), p1603-1613, Dec.), (2017.
• Ki-Hong Park, Hui-seong Noh, Effective Acne Detection using Component Image a* of CIEL*a*b* Color Space, Journal of Digital Contents Society, 19(7), p1397-1403, July), (2018. [https://doi.org/10.9728/dcs.2018.19.7.1397]
• Jae-Young Lee, Jun-Sik Kwon, Image Annotation System for mobile Augmented Reality Environment, Journal of Digital Contents Society, 16(3), p437-444, Jun.), (2015. [https://doi.org/10.9728/dcs.2015.16.3.437]