Abstract
Image feature matching, which aims to establish correspondence between two images, is an important task in computer vision. Among image feature matching, the removal of mismatches is crucial to ensure the correctness of the matches. In recent years, machine learning has become a new perspective for mismatch removal. However, existing learning-based methods require a large amount of image data for training, which shows a lack of generalizability and is hard to deal with cases with high mismatch ratio. In this paper, we induce the triangular topology constraint into machine learning, where topology constraints around the matching points are summarized; combining with the idea of sampling, we achieve the task of removing mismatches. Topology constraints are studied in spite of the image input; our LTM (learning topology for matching) just needs fewer than 20 parameters as input, so that only ten training image pairs from four image sets involving about 3,000 matches are employed to train; it still achieves promising results on various datasets with different machine learning approaches. The experimental results of this study also demonstrate the superior performance of our LTM over existing methods.











Similar content being viewed by others
Data availability
All datasets used in the work are publicly available and can be accessed as described in each of the referenced presentation papers.
References
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004). https://doi.org/10.1023/B:VISI.0000029664.99615.94
Bay, H., Tuytelaars, T., Gool, L.V.: SURF: Speeded Up Robust Features. Springer-Verlag, Cham (2006)
C. Minchael, L. Vincent, S. Christoph, V. F. Pascal, BRIEF: binary robust independent elementary features. In: Proceedings of the 11th European Conference on Computer vision: Part IV Sept 2010 Pp. 778–792. https://doi.org/10.1007/978-3-642-15561-1_56.
Yi, K.M., Trulls, E., Lepetit, V., et al.: LIFT: learned invariant feature transform. Eur. Conf. Comput. Vis. (2016). https://doi.org/10.1007/978-3-319-46466-4_28
D. Detone, Malisiewicz T , Rabinovich A. SuperPoint: self-supervised interest point detection and description[C]// In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). https://doi.org/10.1109/CVPRW.2018.00060.
Richard, H., Andrew, Z.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)
Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981). https://doi.org/10.1145/358669.358692
Chum O, Matas J. Matching with PROSAC-progressive sample consensus. In: Proceeding of IEEE Computer Society Conference on Computer Society, 2005: 220–226. https://doi.org/10.1109/CVPR.2005.221
Ni Kai, Jin Hailing, Dellaert F. GroupSAC: efficient consensus in the presence of groupings. In: Proceeding of the 12th IEEE International Conference on Computer Vision. 2009:2193–2200. https://doi.org/10.1109/ICCV.2009.5459241
Chun O, Matas J. Randomized RANSAC with Td, d test. In: Proceeding of the 13th British Machine Vision Conference. Berlin: Springer, 2002: 448–457
Matas J, Chun O. Randomized RANSAC with sequential probability ratio test [C]. In: Proc of the 10th IEEE International Conference on Computer Vision (ICCV): IEEE Press, 2005: 1727–1732. https://doi.org/10.1109/ICCV.2005.198
M. Rahman, X. Li and X. Yin. DL-RANSAC: An improved RANSAC with modified sampling strategy based on the likelihood [C]. In: 2019 IEEE 4th International Conference on Image, Vision and Computing (ICIVC). 2019: 463–468. https://doi.org/10.1109/ICIVC47709.2019.8981025
Shi C, Wang Y, Li H, Feature point matching using sequential evaluation on sample consensus. In: International Conference on Security. IEEE, 2017: 302–306. https://doi.org/10.1109/SPAC.2017.8304294
Aguilar, W., Fraud, Y., Escolano, F., et al.: A robust Graph Transformation Matching for non-rigid registration. Image Vis. Comput. 27(7), 897–910 (2009)
Bian, J., Lin, W., Matsushita, Y., Yeung, S., Nguyen, T., Cheng, M.: GMS: grid-based motion statistics for fast, ultra-robust feature correspondence. IEEE Conf. Comput. Vis. Pattern Recogn. (CVPR). 2017, 2828–2837 (2017). https://doi.org/10.1109/CVPR.2017.302
Ma, J., Zhao, J., Jiang, J., et al.: Locality preserving matching. Int. J. Comput. Vision 127(5), 512–531 (2019). https://doi.org/10.1007/s11263-018-1117-z
Jiang, X., Xia, Y., Zhang, X.-P., Ma, J.: Robust image matching via local graph structure consensus. Pattern Recogn. 126, 108588 (2022). https://doi.org/10.1016/j.patcog.2022.108588
Zhao, X., He, Z., et al.: Improved keypoint descriptors based on Delaunay triangulation for image matching. Optik-Int. J. Light Electron. 125(13), 3121–3123 (2014). https://doi.org/10.1016/j.ijleo.2013.12.022
Luo Y, Li R, Zhang J, et al. Research on Correction Method of Local Feature Descriptor Mismatch. In: 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). https://doi.org/10.1109/IAEAC47372.2019.8997949
Zhao M, Chen H, Song T, et al. Research on image matching based on improved RANSAC-SIFT algorithm. International Conference on Optical Communications and Networks, pp 1–3 (2017). https://doi.org/10.1109/ICOCN.2017.8121270
Zhu, W., Sun, W., Wang, Y., Liu, S., Xu, K.: An improved RANSAC algorithm based on similar structure constraints. Int. Conf. Robot Intell. Syst. (ICRIS) 2016, 94–98 (2016). https://doi.org/10.1109/ICRIS.2016.19
X. Lan, B. Guo, Z. Huang and S. Zhang, "An improved UAV aerial image mosaic algorithm based on GMS-RANSAC". In: 2020 IEEE 5th International Conference on Signal and Image Processing (ICSIP), 2020, pp. 148–152. https://doi.org/10.1109/ICSIP49896.2020.9339283
He, Z., Shen, C., Wang, Q., Zhao, X., Jiang, H.: Mismatching removal for feature-point matching based on triangular topology probability sampling consensus. Remote Sens. 14, 706 (2022). https://doi.org/10.3390/rs14030706
Mikolajczyk, K., Tuytelaars, T., Schmid, C., et al.: A comparison of affine region detectors. Int. J. Comput. Vision 65(1), 43–72 (2005). https://doi.org/10.1007/s11263-005-3848-x
Balntas, V., Lenc, K., Vedaldi, A., Mikolajczyk, K.: HPatches: a benchmark and evaluation of handcrafted and learned local descriptors. IEEE Conf. Comput. Vision Pattern Recog. (CVPR) 2017, 3852–3861 (2017). https://doi.org/10.1109/ICSIP49896.2020.9339283
K. Cordes, B. Rosenhahn, and J. Ostermann. High-resolution feature evaluation benchmark. In: Proc. CAIP, pp. 327– 334, 2013
Jiang, X., Ma, J., Jiang, J., Guo, X.: Robust feature matching using spatial clustering with heavy outliers. IEEE Trans. Image Process. 29, 736–746 (2019). https://doi.org/10.1109/TIP.2019.2934572
Shao, F., Liu, Z., An, J.: Feature matching based on minimum relative motion entropy for image registration. IEEE Trans. Geosci. Remote Sens. 60, 1–12 (2022). https://doi.org/10.1109/TGRS.2021.3068185
L. Cavalli, V. Larsson, M. R. Oswald, T. Sattler and M. Pollefeys, "Handcrafted outlier detection revisited", In: Computer Vision–ECCV 2020: 16th European Conference, pp. 770–787, 2020
Yi KM, Trulls E, Ono Y, Lepetit V, Salzmann M, Fua P. Learning to find good correspondences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 (pp. 2666-2674) https://doi.org/10.1109/CVPR.2018.00282
Sarlin P.-E., DeTone D., Malisiewicz T. and Rabinovich A., "SuperGlue: learning feature matching with graph neural networks". In: Proceeding of IEEE/CVF Conference of Computer Vision and Pattern Recognition (CVPR), pp. 4938–4947, Jun. 2020. https://doi.org/10.1109/CVPR42600.2020.00499
J. Zhang et al., "OANet: Learning Two-View Correspondences and Geometry Using Order-Aware Network". In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 6, pp. 3110–3122, 1 June 2022. https://doi.org/10.1109/TPAMI.2020.3048013
Xia, Y., Ma, J.: Locality-guided global-preserving optimization for robust feature matching. IEEE Trans. Image Process. 31, 5093–5108 (2022). https://doi.org/10.1109/TIP.2022.3192993
Xia, Y., Jiang, J., Yifan, Lu., Liu, W., Ma, J.: Robust feature matching via progressive smoothness consensus. ISPRS J. Photogr. Remote. Sens. 196, 502–513 (2023)
Liu, X., Xiao, G., Chen, R., Ma, J.: PGFNet: preference-guided filtering network for two-view correspondence learning. IEEE Trans. Image Process. 32, 1367–1378 (2023). https://doi.org/10.1109/TIP.2023.3242598
Ma, J., Jiang, X., Jiang, J., Zhao, J., Guo, X.: LMR: learning a two-class classifier for mismatch removal. IEEE Trans. Image Process. 28(8), 4045–4059 (2019). https://doi.org/10.1109/TIP.2019.2906490
N. Mayer et al., "A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation". In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 4040–4048. https://doi.org/10.1109/CVPR.2016.438
Liu, Z., An, J., Jing, Y.: A simple and robust feature point matching algorithm based on restricted spatial order constraints for aerial image registration. IEEE Trans. Geosci. Remote Sens. 50(2), 514–527 (2012). https://doi.org/10.1109/TGRS.2011.2160645
Zhang, K., Li, X., Zhang, J.: A robust point-matching algorithm for remote sensing image registration. IEEE Geosci. Remote Sens. Lett. 11(2), 469–473 (2014). https://doi.org/10.1109/LGRS.2013.2267771
Wu, Y., Ma, W., Gong, M., Su, L., Jiao, L.: A novel point-matching algorithm based on fast sample consensus for image registration. IEEE Geosci. Remote Sens. Lett. 12(1), 43–47 (2015). https://doi.org/10.1109/LGRS.2014.2325970
Li, B., Ye, H.: RSCJ: robust sample consensus judging algorithm for remote sensing image registration. IEEE Geosci. Remote Sens. Lett. 9(4), 574–578 (2012). https://doi.org/10.1109/LGRS.2011.2175434
H. Zhang et al., "Remote sensing image registration based on local affine constraint with circle descriptor." In: IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022, Art no. 8002205, https://doi.org/10.1109/LGRS.2020.3027096
Feng, R., Shen, H., Bai, J., Li, X.: Advances and opportunities in remote sensing image geometric registration: a systematic review of state-of-the-art approaches and future research directions. IEEE Geosci Remote Sens Magaz 9(4), 120–142 (2021). https://doi.org/10.1109/MGRS.2021.3081763
Funding
This work was supported by the National Natural Science Foundation of China (52275547 and 52275514) and the Zhejiang Provincial Natural Science Foundation of China (LY21E050021).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Shen, C., He, Z., Zhao, X. et al. LTM: efficient learning with triangular topology constraint for feature matching with heavy outliers. Machine Vision and Applications 34, 130 (2023). https://doi.org/10.1007/s00138-023-01482-3
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00138-023-01482-3