Skip to main content
Springer Nature Link
Log in

APD-229: a textual-visual database for agricultural pests and diseases

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The damage caused by agricultural pests and diseases has brought huge losses to the economy. Rapid recognition and timely treatment can minimize economic losses. Most of the existing image databases are produced in laboratories, where the shooting costs are expensive, and the background of these images are very different from the real farmland environment. Moreover, although the existing recognition systems can locate entities, they cannot provide discriminative evidence which is semantically interpretable, which makes it difficult for them to distinguish entities with very similar appearances. Fortunately, there are text descriptions in professional agricultural control documents that can clearly distinguish similar entities. In this paper, a textual-visual database for agricultural pests and diseases named APD-229 is constructed. The goal of APD-229 is to learn prior knowledge that can distinguish similar entities from the control documents, and to guide the image recognition system to complete the task of fine-grained classification. The database contains two sub-databases: pest set and disease set. A total of 121,213 images and 8,209 text descriptions belong to 229 categories. Furthermore, extensive experiments were carried out on APD-229, results show that in the single-modal image classification task, the accuracy of pest database is 75.15% and the accuracy of disease database is 61.23%. While in the multi-modal image classification task, the accuracy is 78.74% and 71.67% respectively. Compare with the single-model experiment, the accuracy of multi-model is improved by 4.78% and 17% respectively. APD-229 is publicly available at https://github.com/SDUST-MMML/APD-229.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Algorithm 1
Algorithm 2
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Algorithm 3
Fig. 13
Fig. 14

Similar content being viewed by others

Availability of Data and Material

The database used during the current study are available from the corresponding author on reasonable request.

Code Availability

Not applicable

References

  1. Abbas I, Liu J, Amin M, Tariq A, Tunio M H (2021) Strawberry fungal leaf scorch disease identification in real-time strawberry field using deep learning architectures. Plants 10(12):2643

    Article  PubMed  PubMed Central  Google Scholar 

  2. Al Hiary H, Bani Ahmad S, Reyalat M, Braik M, ALRahamneh Z (2011) Fast and accurate detection and classification of plant diseases. Int J Comput Appl 17:31–38. https://doi.org/10.5120/2183-2754

    Google Scholar 

  3. Ali A A, Chramcov B, Jasek R, Katta R, Krayem S (2021) Classification of plant diseases using convolutional neural networks. In: Computer science on-line conference. Springer, pp 268–275

  4. Alfarisy A A, Chen Q, Guo M (2018) Deep learning based classification for paddy pests & diseases recognition. In: ACM International conference proceeding series, pp 21–25. https://doi.org/10.1145/3208788.3208795

  5. Alom M Z, Taha T M, Yakopcic C, Westberg S, Sidike P, Nasrin M S, Hasan M, Van Essen B C, Awwal A A, Asari V K (2019) A state-of-the-art survey on deep learning theory and architectures. Electronics (Switzerland) 8:292. https://doi.org/10.3390/electronics8030292

    Google Scholar 

  6. Amakdouf H, Zouhri A, El Mallahi M, Tahiri A, Chenouni D, Qjidaa H (2021) Artificial intelligent classification of biomedical color image using quaternion discrete radial tchebichef moments. Multimed Tools Appl 80(2):3173–3192

    Article  Google Scholar 

  7. Ashraf Patankar A, Moon H (2020) Automatic radish wilt detection using image processing based techniques and machine learning algorithm. arXiv:200900173

  8. Center SFAI (1994) Classification and codes for forestry resources—tree diseases, vol GB/T 15161-1994. State Bureau of Technical Supervision

  9. Cheng X, Zhang Y, Chen Y, Wu Y, Yue Y (2017) Pest identification via deep residual learning in complex background. Comput Electron Agric 141:351–356. https://doi.org/10.1016/j.compag.2017.08.005. https://www.sciencedirect.com/science/article/pii/S0168169917304854

    Article  Google Scholar 

  10. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings—2005 IEEE computer society conference on computer vision and pattern recognition, vol 1, pp 886–893. https://doi.org/10.1109/CVPR.2005.177

  11. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255. https://doi.org/10.1109/cvprw.2009.5206848

  12. Dubey S R (2021) A decade survey of content based image retrieval using deep learning. IEEE Trans Circ Syst Video Technol

  13. Durmuş H, Güneş E O, Kırcı M (2017) Disease detection on the leaves of the tomato plants by using deep learning. In: 2017 6th International conference on agro-geoinformatics. IEEE, pp 1–5

  14. Espíndola RP, Ebecken N F (2005) On extending f-measure and g-mean metrics to multi-class problems. WIT Trans Inf Commun Technol 35:25–34. https://doi.org/10.2495/DATA050031

    Google Scholar 

  15. Fernández A, García S, del Jesus M J, Herrera F (2008) A study of the behaviour of linguistic fuzzy rule based classification systems in the framework of imbalanced data-sets. Fuzzy Sets Syst 159:2378–2398. https://doi.org/10.1016/j.fss.2007.12.023

    Article  MathSciNet  Google Scholar 

  16. Fina F, Birch P, Young R, Obu J, Faithpraise B, Chatwin C (2013) Automatic plant pest detection & recognition using k-means clustering algorithm & correspondence filters. Int J Adv Biotechnol Res 4:1052–1062

    Google Scholar 

  17. Gaonkar A, Chukkapalli Y, Raman P J, Srikanth S, Gurugopinath S (2021) A comprehensive survey on multimodal data representation and information fusion algorithms. In: 2021 International conference on intelligent technologies (CONIT). IEEE, pp 1–8

  18. Goldberg Y, Levy O (2014) word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arXiv:14023722

  19. Guo X, Li S, Yu J, Zhang J, Ma J, Ma L, Liu W, Ling H (2019) PFLD: a practical facial landmark detector. arXiv:190210859

  20. Gur S, Neverova N, Stauffer C, Lim S N, Kiela D, Reiter A (2021) Cross-modal retrieval augmentation for multi-modal classification. arXiv:210408108

  21. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Computer Society conference on computer vision and pattern recognition, vol 2016, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  22. Hebei Plant Protection PIS, National Agricultural Technology Extension Service Center ZCPP, Station PI (2009a) Safety technical specification of pest control for vegetables—part 2: solanaceous furuits vegetables vol GB/t 23416.2-2009. General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China; Standardization Administration

  23. Hebei Plant Protection PIS, National Agricultural Technology Extension Service Center ZCPP, Station PI (2009b) Safety technical specification of pest control for vegetables—part 3: gourd vegetables vol GB/t 23416.3-2009. General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China; Standardization Administration

  24. Hebei Plant Protection PIS, National Agricultural Technology Extension Service Center ZCPP, Station PI (2009c) Safety technical specification of pest control for vegetables—part 4: cole crops vegetables vol GB/t 23416.4-2009. General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China; Standardization Administration

  25. Hebei Plant Protection PIS, National Agricultural Technology Extension Service Center ZCPP, Station PI (2009d) Safety technical specification of pest control for vegetables—part 5: Chinese cabbage group vol GB/t 23416.5-2009. General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China; Standardization Administration

  26. Hebei Plant Protection PIS, National Agricultural Technology Extension Service Center ZCPP, Station PI (2009e) Safety technical specification of pest control for vegetables—part 6: green vegetables vol GB/t 23416.6-2009. General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China; Standardization Administration

  27. Hebei Plant Protection PIS, National Agricultural Technology Extension Service Center ZCPP, Station PI (2009f) Safety technical specification of pest control for vegetables—part 7: vegetable legumes vol GB/t 23416.7-2009. General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China; Standardization Administration

  28. Hebei Plant Protection PIS, National Agricultural Technology Extension Service Center ZCPP, Station PI (2009g) Safety technical specification of pest control for vegetables—part 8: root vegetables vol GB/t 23416.8-2009. General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China; Standardization Administration

  29. Hebei Plant Protection PIS, National Agricultural Technology Extension Service Center ZCPP, Station PI (2009h) Safety technical specification of pest control for vegetables—part 9: bulb crups vol GB/t 23416.9-2009. General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China; Standardization Administration

  30. Henan University of Technology CNGOIC (2018) Classification and codes of grain information—classification and codes of pest and disease in stored-grain, vol LS/t 1709-2018. State Administration of Grain

  31. Iandola F, Moskewicz M, Karayev S, Girshick R, Darrell T, Keutzer K (2014) Densenet: implementing efficient convnet descriptor pyramids. arXiv:14041869

  32. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. In: MM 2014—proceedings of the 2014 ACM conference on multimedia, pp 675–678. https://doi.org/10.1145/2647868.2654889

  33. Kaur H, Koundal D, Kadyan V (2021) Image fusion techniques: a survey. Archives of Computational Methods in Engineering

  34. Kheirkhah F M, Asghari H (2018) Plant leaf classification using gist texture features. IET Comput Vis 13:369–375. https://doi.org/10.1049/iet-cvi.2018.5028

    Article  Google Scholar 

  35. Krizhevsky A, Sutskever I, Hinton G E (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60:84–90. https://doi.org/10.1145/3065386

    Article  Google Scholar 

  36. Kulkarni O (2018) Crop disease detection using deep learning. In: 2018 Fourth international conference on computing communication control and automation (ICCUBEA). https://doi.org/10.1109/ICCUBEA.2018.8697390

  37. Li J, Wang Q (2022) Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: overview, challenges, and novel orientation. Inf Fusion 79:229–247. https://doi.org/10.1016/j.inffus.2021.10.018. https://www.sciencedirect.com/science/article/pii/S1566253521002219

    Article  Google Scholar 

  38. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft COCO: common objects in context. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 8693 LNCS, pp 740–755. https://doi.org/10.1007/978-3-319-10602-1_48

  39. Liu Z, Gao J, Yang G, Zhang H, He Y (2016) Localization and classification of paddy field pests using a saliency map and deep convolutional neural network. Sci Rep 6:20410. https://doi.org/10.1038/srep20410

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  40. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60:91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  41. Lu J, Hu J, Zhao G, Mei F, Zhang C (2017) An in-field automatic wheat disease diagnosis system. Comput Electron Agric 142:369–379. https://doi.org/10.1016/j.compag.2017.09.012

    Article  Google Scholar 

  42. Lu J, Goswami V, Rohrbach M, Parikh D, Lee S (2020) 12-in-1: multi-task vision and language representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10437–10446

  43. Lu T, Han B, Chen L, Yu F, Xue C (2021) A generic intelligent tomato classification system for practical applications using densenet-201 with transfer learning. Sci Rep 11(1):1–8

    Article  Google Scholar 

  44. Mahalakshmi S D, Vijayalakshmi K (2020) Agro suraksha: pest and disease detection for corn field using image analysis. J Ambient Intell Humaniz Comput 1–15. https://doi.org/10.1007/s12652-020-02413-0

  45. Nanni L, Maguolo G, Pancino F (2019) Research on insect pest image detection and recognition based on bio-inspired methods. arXiv:191000296. 169:139–148. https://doi.org/10.1016/j.ecoinf.2020.101089

  46. Nuyts J (2008) Modality: overview and linguistic issues. The expression of modality, pp 1–26

  47. Park H, Jeesook E, Kim S H (2018) Crops disease diagnosing using image-based deep learning mechanism. In: Proceedings of the 2nd international conference on computing and network communications, CoCoNet 2018, pp 23–26. https://doi.org/10.1109/CoCoNet.2018.8476914

  48. Prabhakar M, Purushothaman R, Awasthi D P (2020) Deep learning based assessment of disease severity for early blight in tomato crop. Multimed Tools Appl 79(39):28773–28784

    Article  Google Scholar 

  49. Rahman C R, Arko P S, Ali M E, Khan M A I, Apon S H, Nowrin F, Wasif A (2020) Identification and recognition of rice diseases and pests using convolutional neural networks. Biosyst Eng 194:112–120

    Article  Google Scholar 

  50. Ramcharan A, Baranowski K, McCloskey P, Ahmed B, Legg J, Hughes D P (2017) Deep learning for image-based cassava disease detection. Front Plant Sci 8:1852

    Article  PubMed  PubMed Central  Google Scholar 

  51. Rangarajan Aravind K, Raja P (2020) Automated disease classification in (selected) agricultural crops using transfer learning. Automatika: Časopis za Automatiku, Mjerenje, Elektroniku, Računarstvo i Komunikacije 61(2):260–272

    Article  Google Scholar 

  52. Redmon J, Farhadi A (2018) YOLOV3: an incremental improvement. arXiv:180402767

  53. Research Institute of Forest Ecology E, Protection NFAFPCS Chinese Academy of Forestry (2011) Classification and codes of forest pests vol GB/t 15775-2011. General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China; Standardization Administration

  54. Samanta R K, Ghosh I (2012) Tea insect pests classification based on artificial neural networks. Int J Comput Eng Sci 2:1–13

    Google Scholar 

  55. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L C (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  56. Selvaraj M G, Vergara A, Ruiz H, Safari N, Elayabalan S, Ocimati W, Blomme G (2019) AI-powered banana diseases and pest detection. Plant Methods 15:92. https://doi.org/10.1186/s13007-019-0475-z

    Article  Google Scholar 

  57. Sharif M, Khan M A, Iqbal Z, Azam M F, Lali M I U, Javed M Y (2018) Detection and classification of citrus diseases in agriculture based on optimized weighted segmentation and feature selection. Comput Electron Agric 150:220–234. https://doi.org/10.1016/j.compag.2018.04.023

    Article  Google Scholar 

  58. Shen J, Robertson N (2020) Bbas: towards large scale effective ensemble adversarial attacks against deep neural network learning. Inf Sci 569(7)

  59. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. arXiv:14091556

  60. Singh V, Misra A K (2017) Detection of plant leaf diseases using image segmentation and soft computing techniques. Inf Process Agric 4:41–49. https://doi.org/10.1016/j.inpa.2016.10.005

    Google Scholar 

  61. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S E, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2014) Going deeper with convolutions. CoRR arXiv:1409.4842

  62. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2015) Rethinking the inception architecture for computer vision. CoRR arXiv:1512.00567

  63. Tan M, Le Q (2019) Efficientnet: rethinking model scaling for convolutional neural networks. In: International conference on machine learning, PMLR, pp 6105–6114

  64. Thenmozhi K, Srinivasulu Reddy U (2019) Crop pest classification based on deep convolutional neural network and transfer learning. Comput Electron Agric 164:104906. https://doi.org/10.1016/j.compag.2019.104906. https://www.sciencedirect.com/science/article/pii/S0168169919310695

    Article  Google Scholar 

  65. TRI of Chinese Academy of Agricultural Sciences (2008) Grade and investigation method of tobacco diseases and insect pests vol GB/t 23222-2008. General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China; Standardization Administration

  66. Türkoǧlu M, Hanbay D (2019) Plant disease and pest detection using deep learning-based features. Turk J Electr Eng Comput Sci 27:1636–1651. https://doi.org/10.3906/elk-1809-181

    Article  Google Scholar 

  67. Venugoban K, Ramanan A (2014) Image classification of paddy field insect pests using gradient-based features. Int J Mach Learn Comput 4:1–5. https://doi.org/10.7763/ijmlc.2014.v4.376

    Google Scholar 

  68. Vo A T, Tran H S, Le T H (2017) Advertisement image classification using convolutional neural network. In: 2017 9th International conference on knowledge and systems engineering, pp 197–202. https://doi.org/10.1109/KSE.2017.8119458

  69. Wang Y (2021) Survey on deep multi-modal data analytics: collaboration, rivalry, and fusion. ACM Trans Multimed Comput Commun Appl 17(1s). https://doi.org/10.1145/3408317

  70. Wang Z, Lu B, Chi Z, Feng D (2011) Leaf image classification with shape context and sift descriptors. In: 2011 International conference on digital image computing: techniques and applications, pp 650–654. https://doi.org/10.1109/DICTA.2011.115

  71. Wang J, Lin C, Ji L, Liang A (2012) A new automatic identification system of insect images at the order level. Knowl-Based Syst 33:102–110. https://doi.org/10.1016/j.knosys.2012.03.014

    Article  CAS  Google Scholar 

  72. Wang L, Qian X, Zhang Y, Shen J, Cao X (2020a) Enhancing sketch-based image retrieval by cnn semantic re-ranking. IEEE Trans Cybern 50 (7):3330–3342

  73. Wang S, Wang R, Yao Z, Shan S, Chen X (2020b) Cross-modal scene graph matching for relationship-aware image-text retrieval. In: Proceedings of the IEEE/CVF Winter conference on applications of computer vision, pp 1508–1517. https://doi.org/10.1109/WACV45572.2020.9093614

  74. Wickramanayake S, Hsu W, Lee M L (2021) Learning semantically meaningful features for interpretable classifications. arXiv:210103919

  75. Wu X, Zhan C, Lai Y K, Cheng M M, Yang J (2019) IP102: a large-scale benchmark dataset for insect pest recognition. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, vol 2019, pp 8779–8788. https://doi.org/10.1109/CVPR.2019.00899

  76. Xiao XY, Hu R, Zhang SW, Wang XF (2010) Hog-based approach for leaf classification. In: International conference on intelligent computing, pp 149–155. https://doi.org/10.1007/978-3-642-14932-0_19

  77. Xie C, Wang R, Zhang J, Chen P, Dong W, Li R, Chen T, Chen H (2018) Multi-level learning features for automatic classification of field crop pests. Comput Electron Agric 152:233–241. https://doi.org/10.1016/j.compag.2018.07.014

    Article  Google Scholar 

  78. Zhang Y, Liu Y P (2021) Identification of navel orange diseases and pests based on the fusion of densenet and self-attention mechanism. Comput Intell Neurosci

  79. Zhou C, Sun C, Liu Z, Lau F C M (2015) A c-LSTM neural network for text classification. arXiv:151108630

Download references

Funding

This work is supported by National Key R&D Program of China [2022ZD0119500 and 2022ZD0119501]; NSFC [U1931207 and 61702306]; Sci. & Tech. Development Fund of Shandong Province of China [ZR2022MF288, ZR2023MF097]; the Taishan Scholar Program of Shandong Province[ts20190936], and SDUST Research Fund [2015TDJH102 and 2019KJN024]; Shandong Chongqing Science and technology cooperation project [cstc2020jscx-lyjsAX0008].

Author information

Authors and Affiliations

  1. College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao, 266590, China

    Shan-Song Wang, Wei-Jian Ni, Qing-Tian Zeng & Chao Li

  2. Institute of Agricultural Information, The Chinese Academy of Agricultural Sciences, Beijing, 100081, China

    Neng-Fu Xie

Authors
  1. Shan-Song Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei-Jian Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qing-Tian Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Neng-Fu Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Not applicable

Corresponding authors

Correspondence to Wei-Jian Ni or Qing-Tian Zeng.

Ethics declarations

Copyright

All the images contained in this dataset are from the Internet. Images from Baidu Images without strict copyrights are selected, and images from Google Images with ‘Creative Commons License’ or images without strict copyright are selected. The text data is collected from the websites mentioned in Table 3. The images in this dataset and the text data can only be used for academic research and not the business. The data involved in this paper has not been published before in the form of a textual-visual combination.

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, SS., Ni, WJ., Zeng, QT. et al. APD-229: a textual-visual database for agricultural pests and diseases. Multimed Tools Appl 83, 22189–22220 (2024). https://doi.org/10.1007/s11042-023-15393-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15393-y

Keywords

Search

Navigation

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy