The 2014 Search and Hyperlinking Task
The underlying use scenario is a popular information seeking strategy that users on the internet execute millions of times every day: they search for relevant information using a query and then follow hyperlinks to further explore the collection. For the textual web, this use scenario is already supported by search engines and by the embedded hyperlinks in web pages. Furthermore, there is research in searching for multimedia content and in, mainly manually, linking such content. However, there is little research in supporting searching and hyperlinking videos automatically. In this task we model specific instances of this use scenario to encourage researchers to develop technology that supports home users in their daily routine.
The search and hyperlinking task involves the following three sub-tasks:
1. search: answering a query with a ranked list of documents,
2. anchor detection: finding potential anchors in the given segments of the videos,
3. linking: generating a ranked list of video segments in the same collection that provide information about these anchors.
The envisioned use-case is visualized in this Figure.
The Search and Hyperlinking Task is a continuation of its instances in MediaEval 2012-2013 Task. This year will emphasize on queries with multiple relevant results, anchor detection, and linking.
Target group
The task is of interest to researchers in the area of multimedia information retrieval, speech and video processing, and information seeking.
Data
We will use a set of roughly 2000h of archival data from a television broadcaster (BBC). The data will be in English and we are likely to be able to provide metadata, subtitles and output of visual analysis. The format of the data will be often similar to the on used during MediaEval 2013. The participants will be required to sign a licensing agreement with BBC in order to access the data that will be available via University of Twente servers.
Ground truth and evaluation
We ask users from a wide range of backgrounds to generate queries for which relevant videos exist in the collection. For a subset of the relevant segments, the users will define anchors within the relevant segment and provide a description of the expected target segments. The workshop participants are then asked to submit their runs.
Based on the top ranked documents in a (prioritized) sub-set of those runs, we will ask Amazon’s mechanical turks to judge whether the returned video segments are relevant to the query or the target description. For the both sub-tasks we will use derivations of popular retrieval measures, e.g. MAP, MASP, which we adapted to segment retrieval.
Recommended reading
Collection description
[1] Eskevich, M., Aly, R., Ordelman, R., Chen, S., Jones, G. J.F. The Search and Hyperlinking Task at MediaEval 2013. In Proceedings of the MediaEval 2013 Multimedia Benchmark Workshop, CEUR-WS.org, 1043, ISSN: 1613-0073. Barcelona, Spain, 2013.
Evaluation
[2] Aly, R., Eskevich, M., Ordelman, R., Jones, G. J.F. Adapting Binary Information Retrieval Evaluation Metrics for Segment-based Retrieval Tasks. Technical Report.
[3] Eskevich, E., Magdy, W., Jones, G. J.F. New Metrics for Meaningful Evaluation of Informally Structured Speech Retrieval. In Proceedings of ACM ECIR European Conference on Advances in Information Retrieval. ACM, Springer-Verlag Berlin, Heidelberg, 2012, 170-181.
[4] Aly, R., Ordelman, R. J.F., Eskevich, M., Jones, G. J.F., Chen, S. Linking Inside a Video Collection - What and How to Measure? In Proceedings of ACM WWW International Conference on World Wide Web Companion. ACM, Rio de Janeiro, Brazil, 457-460.
Search and Hyperlinking Task history
[5] Eskevich, M., Jones, G. J.F., Aly, R., Ordelman, R. J.F., Chen, S., Nadeem, D., Guinaudeau, C., Gravier, G., Sébillot, P., de Nies, T., Debevere, P., de Walle, R. V., Galuscakova, P., Pecina, P., Larson, M. Multimedia Information Seeking through Search and Hyperlinking. In Proceedings of ACM ICMR International Conference on Multimedia Retrieval. ACM, Dallas, USA, 2013.
Papers arising from previous years
[6] Eskevich, M., Jones, G. J.F., Chen, S., Aly, R., Ordelman, R., Larson, M. Search and Hyperlinking Task at MediaEval 2012. In Working Notes Proceedings of the MediaEval 2012 Workshop. Pisa, Italy, 2012.
[7] Eskevich, M., Jones, G. J.F., Wartena, C., Larson, M., Aly, R., Verschoor, T., Ordelman, R. Comparing Retrieval Effectiveness for Alternative Content Segmentation Methods for Internet Video. In Proceedings of IEEE CBMI Workshop on Content-Based Multimedia Indexing, Annecy, France, 2012, 1-6.
[8] Galušcáková, P., Pecina, P. Segmentation Strategies for Passage Retrieval in Audio-Visual Documents. In Proceedings of ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, Dublin, Ireland, 1143-1143.
Previous participants’ papers at ICMR'14 special session User-centric video search and hyperlinking:
[9] Bailer, W., Lokaj, M., Stiegler, H. Context in Video Search: Is Close-by Good Enough When Using Linking? In Proceedings of ACM ICMR International Conference on Multimedia Retrieval. ACM, Glasgow, Scotland, 2014.
[10] Bhatt, C., Pappas, N., Habibi M., Popescu-Belis, A. Multimodal Reranking of Content-based Recommendations for Hyperlinking Video Snippets. In Proceedings of ACM ICMR International Conference on Multimedia Retrieval. ACM, Glasgow, Scotland, 2014.
[11] P. Galušcáková and P. Pecina Experiments with Segmentation Strategies for Passage Retrieval in Audio-Visual Documents. In Proceedings of ACM ICMR International Conference on Multimedia Retrieval. ACM, Glasgow, Scotland, 2014.
Previous work in the field
[12] Eskevich, M., Jones, G. J.F. Exploring Speech Retrieval from Meetings using the AMI Corpus. In Proceeings of ISCA International Speech Communication Association, Computer Speech and Language. 2014 to appear.
[13] Gyarmati, A., Jones, G J.F. When to Cross Over? Cross-Language Linking Using Wikipedia for VideoCLEF 2009. In Proceedings of CLEF Workshop of the Cross-Language Evaluation Forum, Multilingual Information Access Evaluation II. Multimedia Experiments. Corfu, Greece, 2009, 409-412.
[14] Larson, M., Newman, E., Jones, G. J.F. Overview of VideoCLEF 2009: New Perspectives on Speech-based Multimedia Content Enrichment. In Proceedings of ACM CLEF International Conference on Cross-Language Evaluation Forum: Multimedia experiments. ACM, Springer-Verlag Berlin, Heidelberg, 2009, 354-368.
[15] Mihalcea, R., Csomai, A. Wikify! Linking Documents to Encyclopedic Knowledge. In Proceedings of ACM CIKM Conference on Information and Knowledge Management. ACM, Lisbon, Portugal, 2007, 233-242.
[16] Peters, C. What Happened in CLEF 2009. In Proceedings of CLEF Workshop of the Cross-Language Evaluation Forum, Multilingual Information Access Evaluation I. Text Retrieval Experiments. Corfu, Greece, 2009, 1-12.
[17] Raaijmakers, S., Versloot, C., de Wit, J. A Cocktail Approach to the VideoCLEF’09 Linking Task. In Proceedings of CLEF Workshop of the Cross-Language Evaluation Forum, Multilingual Information Access Evaluation II. Multimedia Experiments. Corfu, Greece, 2009, 401-408.
Task organizers
Robin Aly, University of Twente, Netherlands
Maria Eskevich, Dublin City University, Ireland
Shu Chen, Dublin City University, Ireland
Gareth Jones, Dublin City University, Ireland
Roeland Ordelman, University of Twente and Netherlands Institute for Sound & Vision, a.k.a. Beeld & Geluid, Netherlands
David Racca, Dublin City University, Ireland
Task schedule
30 May: Development data release
1-7 July: Test data release (new)
7 July: release of the test queries/anchors (new)
11 August: Run submission due (new)
12 September: Results returned to the participants
19 September: Working notes paper deadline
