Abstract
The evaluation of the performance of interactive multimedia retrieval systems is a methodologically non-trivial endeavour and requires specialized infrastructure. Current evaluation campaigns have so far relied on a local setting, where all retrieval systems needed to be evaluated at the same physical location at the same time. This constraint does not only complicate the organization and coordination but also limits the number of systems which can reasonably be evaluated within a set time frame. Travel restrictions might further limit the possibility for such evaluations. To address these problems, evaluations need to be conducted in a (geographically) distributed setting, which was so far not possible due to the lack of supporting infrastructure. In this paper, we present the Distributed Retrieval Evaluation Server (DRES), an open-source evaluation system to facilitate evaluation campaigns for interactive multimedia retrieval systems in both traditional on-site as well as fully distributed settings which has already proven effective in a competitive evaluation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Awad, G., et al.: Trecvid 2019: an evaluation campaign to benchmark video activity detection, video captioning and matching, and video search & retrieval. In: Proceedings of TRECVID 2019. NIST, USA (2019)
Clough, P., Sanderson, M.: The CLEF 2003 cross language image retrieval track. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 581–593. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30222-3_56
Gasser, R., Rossetto, L., Heller, S., Schuldt, H.: Cottontail DB: an open source database system for multimedia retrieval and analysis. In: Proceedings of the 28th ACM International Conference on Multimedia (MM 2020). ACM, Seattle, October 2020
Giangreco, I., Schuldt, H.: Adam pro: database support for big multimedia retrieval. Datenbank-Spektrum 16(1), 17–26 (2016)
Gurrin, C., et al.: Introduction to the third annual lifelog search challenge (LSC’20). In: Proceedings of the 2020 International Conference on Multimedia Retrieval, pp. 584–585 (2020)
Gurrin, C., et al.: [Invited papers] comparing approaches to interactive lifelog search at the lifelog search challenge (LSC 2018). ITE Trans. Media Technol. Appl. 7(2), 46–59 (2019)
Lokoč, J., et al.: Interactive search or sequential browsing? A detailed analysis of the video browser showdown 2018. ACM Trans. Multimed. Comput. Commun. Appl. 15(1), 1–18 (2019)
Pouyanfar, S., Yang, Y., Chen, S.C., Shyu, M.L., Iyengar, S.S.: Multimedia big data analytics: a survey. ACM Comput. Surv. 51(1) (2018)
Rossetto, L., et al.: Interactive video retrieval in the age of deep learning – detailed evaluation of VBS 2019. IEEE Trans. Multimed. 23, 243–256 (2021). https://doi.org/10.1109/tmm.2020.2980944
Rossetto, L., Schuldt, H., Awad, G., Butt, A.A.: V3C – a research video collection. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11295, pp. 349–360. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05710-7_29
Schoeffmann, K.: A user-centric media retrieval competition: the video browser showdown 2012–2014. IEEE Multimed. 21(4), 8–13 (2014)
Schoeffmann, K.: Video browser showdown 2012–2019: a review. In: 2019 International Conference on Content-Based Multimedia Indexing (CBMI), pp. 1–4. IEEE (2019)
Smeaton, A.F., Over, P., Taban, R.: The TREC-2001 video track report. In: TREC (2001)
Snoek, C.G., Worring, M., de Rooij, O., van de Sande, K.E., Yan, R., Hauptmann, A.G.: VideOlympics: real-time evaluation of multimedia retrieval systems. IEEE Multimed. 15(1), 86–91 (2008)
Acknowledgements
This work was partly supported by the Hasler Foundation in the context of the project City-Stories (contract no. 17055).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Rossetto, L., Gasser, R., Sauter, L., Bernstein, A., Schuldt, H. (2021). A System for Interactive Multimedia Retrieval Evaluations. In: Lokoč, J., et al. MultiMedia Modeling. MMM 2021. Lecture Notes in Computer Science(), vol 12573. Springer, Cham. https://doi.org/10.1007/978-3-030-67835-7_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-67835-7_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67834-0
Online ISBN: 978-3-030-67835-7
eBook Packages: Computer ScienceComputer Science (R0)