Abstract
Acquiring fine-grained object detection annotations in unconstrained images is time-consuming, expensive, and prone to noise, especially in crowdsourcing scenarios. Most prior object detection methods assume accurate annotations; A few recent works have studied object detection with noisy crowdsourced annotations, with evaluation on distinct synthetic crowdsourced datasets of varying setups under artificial assumptions. To address these algorithmic limitations and evaluation inconsistency, we first propose a novel Bayesian Detector Combination (BDC) framework to more effectively train object detectors with noisy crowdsourced annotations, with the unique ability of automatically inferring the annotators’ label qualities. Unlike previous approaches, BDC is model-agnostic, requires no prior knowledge of the annotators’ skill level, and seamlessly integrates with existing object detection models. Due to the scarcity of real-world crowdsourced datasets, we introduce large synthetic datasets by simulating varying crowdsourcing scenarios. This allows consistent evaluation of different models at scale. Extensive experiments on both real and synthetic crowdsourced datasets show that BDC outperforms existing state-of-the-art methods, demonstrating its superiority in leveraging crowdsourced data for object detection. Our code and data are available at: https://github.com/zhiqin1998/bdc.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: AggNet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imag. 35(5), 1313–1321 (2016)
Baumgartner, C.F., et al.: PHiSeg: capturing uncertainty in medical image segmentation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 119–127. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_14
Bernard, J.M., Smith, A.F.M.: Bayesian Theory, chap. 3, pp. 136–138. Wiley, West Sussex (1993)
Bernhardt, M., et al.: Active label cleaning for improved dataset quality under resource constraints. Nature Commun. 13(1), 1161 (2022)
Budd, S., et al.: Can non-specialists provide high quality gold standard labels in challenging modalities? In: Albarqouni, S., et al. (eds.) DART/FAIR -2021. LNCS, vol. 12968, pp. 251–262. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87722-4_23
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-End object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Chen, Z., et al.: Structured probabilistic end-to-end learning from crowds. In: International Joint Conference on Artificial Intelligence, pp. 1512–1518 (2020)
Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. J. Roy. Statist. Soc. Ser. C (Appl. Statist.) 28(1), 20–28 (1979)
Dung, N.B., Nguyen, H.Q., Elliott, J., Nhan, N.T., Culliton, P.: KeepLearning: VinBigData chest x-ray abnormalities detection (2020). https://kaggle.com/competitions/vinbigdata-chest-xray-abnormalities-detection
Everingham, M., Luc Van Gool, C.K.I.W., Winn, J., Zisserman, A.: The PASCAL visual object classes challenge 2007 (VOC 2007). Int. J. Comput. Vis. 88, 303–338 (2010)
Fang, Y., et al.: EVA: exploring the limits of masked visual representation learning at scale. In: IEEE Conference on Computer Vision Pattern Recognition, Vancouver, Canada, pp. 19358–19369 (2023)
Goh, H.W., Tkachenko, U., Mueller, J.: CROWDLAB: supervised learning to infer consensus labels and quality scores for data with multiple annotators (2022). arXiv:2210.06812
Guan, M.Y., Gulshan, V., Dai, A.M., Hinton, G.E.: Who said what: modeling individual labelers improves classification. In: AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, pp. 3109–3118 (2017)
Hu, Y., Meina, S.: Crowd R-CNN: an object detection model utilizing crowdsourced labels. In: International Conference on Vision Image Signal Processing, Bangkok, Thailand, pp. 1–7 (2020)
Isupova, O., Li, Y., Kuzin, D., Roberts, S.J., Willis, K.J., Reece, S.: BCCNet: bayesian classifier combination neural network. In: Neural Information Processing System Workshop Machine Learning and Development, Montréal, Canada (2018)
Jensen, M.H., Jørgensen, D.R., Jalaboi, R., Hansen, M.E., Olsen, M.A.: Improving uncertainty estimation in convolutional neural networks using inter-rater agreement. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 540–548. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_59
Ji, W., et al.: Learning calibrated medical image segmentation via multi-rater agreement modeling. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 12336–12346 (2021)
Jungo, A., et al.: On the effect of inter-observer variability for a reliable estimation of uncertainty of medical image segmentation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 682–690. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_77
Kim, H., Ghahramani, Z.: Bayesian classifier combination. In: International Conference on Artificial Intelligence Statistics, La Palma, Canary Islands, pp. 619–627 (2012)
Kohl, S.A.A., et al.: A probabilistic u-net for segmentation of ambiguous images. In: Advances on Neural Information Processing System, Montréal, Canada, pp. 6965–6975 (2018)
Lazarus, E., Mainiero, M.B., Schepps, B., Koelliker, S.L., Livingston, L.S.: BI-RADS lexicon for US and mammography: interobserver variability and positive predictive value. Radiology 239(2), 385–391 (2006)
Le, K., Tran, T., Pham, H., Nguyen Trung, H., Le, T., Nguyen, H.Q.: Learning from multiple expert annotators for enhancing anomaly detection in medical image analysis. IEEE Access 11, 14105–14114 (2023)
LeCun, Y., Cortes, C., Burges, C.: MNIST handwritten digit database (2010). http://yann.lecun.com/exdb/mnist
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Maier-Hein, L., et al.: Crowdsourcing for reference correspondence generation in endoscopic images. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8674, pp. 349–356. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10470-6_44
Müller, R., Kornblith, S., Hinton, G.: When does label smoothing help? In: Advances on Neural Information Processing Systems, Vancouver, Canada, pp. 4694–4703 (2019)
Murphy, K.: Conjugate bayesian analysis of the gaussian distribution (2007). https://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf
Nguyen, H.Q., et al.: VinDr-CXR: an open dataset of chest x-rays with radiologist’s annotations. Sci. Data 9(1), 429 (2022)
Nowruzi, F.E., Kapoor, P., Kolhatkar, D., Hassanat, F., Laganiere, R., Rebut, J.: How much real data do we actually need: Analyzing object detection performance using synthetic and real data. In: International Conference on Machine Learning and Workshop AI Autonomous Driving, Long Beach, CA, USA (2019)
Raykar, V.C., et al.: Learning from crowds. J. Mach. Learn. Res. 11(43), 1297–1322 (2010)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances on Neural Information Processing Systems, Montréal, Canada, pp. 91–99 (2015)
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union: a metric and a loss for bounding box regression. In: IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, pp. 658–666 (2019)
Rodrigues, F., Pereira, F.C.: Deep learning from crowds. In: AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, pp. 1611–1618 (2018)
Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? improving data quality and data mining using multiple, noisy labelers. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, NV, USA, pp. 614–622 (2008)
Solovyev, R., Wang, W., Gabruseva, T.: Weighted boxes fusion: ensembling boxes from different object detection models. Image Vis. Comput. 107, 104–117 (2021)
Sudre, C.H., et al.: Let’s agree to disagree: learning highly debatable multirater labelling. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 665–673. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_73
Venanzi, M., Guiver, J., Kazai, G., Kohli, P., Shokouhi, M.: Community-based bayesian aggregation models for crowdsourcing. In: International Conference on World Wide Web, Seoul, Korea, pp. 155–164 (2014)
Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: IEEE Conference on Computer Vision Pattern Recognition, Vancouver, Canada, pp. 7464–7475 (2023)
Watadani, T., et al.: Interobserver variability in the CT assessment of honeycombing in the lungs. Radiology 266(3), 936–944 (2012)
Wei, H., Xie, R., Feng, L., Han, B., An, B.: Deep learning from multiple noisy annotators as a union. IEEE Trans. Neural Netw. Learn. Syst. 34, 10552–10562 (2022)
Wei, J., Zhu, Z., Luo, T., Amid, E., Kumar, A., Liu, Y.: To aggregate or not? learning with separate noisy labels. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Long Beach, CA, USA, pp. 2523–2535 (2023)
Whitehill, J., Wu, T.f., Bergsma, J., Movellan, J., Ruvolo, P.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In: Advances on Neural Information Processing Systems, Vancouver, Canada, pp. 2035–2043 (2009)
Wu, J., et al.: Multi-rater prism: learning self-calibrated medical image segmentation from multiple raters (2022). arXiv:2212.00601
Yu, S., et al.: Difficulty-aware glaucoma classification with multi-rater consensus modeling. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12261, pp. 741–750. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59710-8_72
Zhang, L., et al.: Disentangling human error from the ground truth in segmentation of medical images. In: Advances on Neural Information Processing Systems, pp. 15750–15762 (2020)
Zhu, X., Vondrick, C., Ramanan, D., Fowlkes, C.: Do we need more training data or better models for object detection? In: British Machine on Vision Conference, Surrey, UK (2012)
Acknowledgements
Z.Q. T. acknowledges the support of an Industrial Cooperative Awards in Science and Engineering (ICASE) studentship from the Engineering and Physical Sciences Research Council (EPSRC) for this work. G. C. acknowledges the support of the Engineering and Physical Sciences Research Council (EPSRC) through grant EP/Y018036/1 and the Australian Research Council (ARC) through grant FT190100525. The authors would like to thank the Satellite Applications Catapult for the provision of the disaster response dataset.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tan, Z.Q., Isupova, O., Carneiro, G., Zhu, X., Li, Y. (2025). Bayesian Detector Combination for Object Detection with Crowdsourced Annotations. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15121. Springer, Cham. https://doi.org/10.1007/978-3-031-73036-8_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-73036-8_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-73035-1
Online ISBN: 978-3-031-73036-8
eBook Packages: Computer ScienceComputer Science (R0)