Participation in this task requires requires signing not only the general MediaEval usage agreement, but also the data usage licensing agreement with the BBC. The agreement and signing instructions can be downloaded here: MediaEval 2013 BBC Agreement. Further, in order to use the automatic speech recognition (ASR) transcripts, a third licensing agreement is required, which can be downloaded here: MediaEval 2013 ASR Agreement
The 2013 Search and Hyperlinking Task
The underlying use scenario is a popular information seeking strategy that is executed millions of times every day: users search for relevant information using a query and then follow hyperlinks to to further explore the collection. For the textual web, this use scenario is already supported by search engines and embedded hyperlinks in web pages. Furthermore, there is research in searching for multimedia content and in, mainly manually, linking such content. However, there is virtually no research in supporting searching and hyperlinking videos. In this task we model specific instances of this use scenario to encourage researchers to develop technology that supports home users in their daily routine.
The search and hyperlinking task involves the following two sub-tasks:
1. Search: answering a query with a ranked list of documents,
2. Linking: generating a ranked list of video segments in the same collection that provide information about these anchors.
Target group
The task is of interest to researchers in the area of multimedia information retrieval, speech and video processing, information seeking.
Data
We will use a set of roughly 200h of archival data from a television broadcaster (BBC). The data will be in English and we are likely to be able to provide metadata, subtitles and output of visual analysis. The participants will be required to sign a licensing agreement with BBC in order to access the data that will be available via University of Twente servers.
Ground truth and evaluation
We are currently investigating the possibility of real users generating the known item queries and anchors based on a item they found in the collection. We might combine this assessment with the use of Amazon Mechanical Turk at the hyperlinking submissions evaluation. For the Search and Linking sub-tasks, we will use the same metric as in 2012 (MAP, MASP, mGAP for Search and MAP for Hyperlinking). For the the anchor selection sub-task (2) we plan to use the F-measure to balance precision and recall within the best identified tasks.
Recommended reading
- Multimedia information seeking through search and hyperlinking, ICMR 2013: pdf
- Linking inside a video collection – what and how to measure? pdf, ppt (Presentation contains more details about queries, known-items, and anchors creation)
- Comparing Retrieval Effectiveness for Alternative Content Segmentation Methods for Internet Video (CBMI 2012): pdf
- MediaEval 12 Search and Hyperlinking Task Overview: pdf
- MediaEval 11 Rich Speech Retrieval Task Overview: pdf
- CLEF 09 Linking Task Overview: pdf
- CLEF 09 Linking Task papers: pdf1, pdf2, pdf3, pdf4
- New Metrics for Meaningful Evaluation of Informally Structured Speech Retrieval: pdf
- Wikify!: linking documents to encyclopedic knowledge: - pdf
Task organizers
Robin Aly, University of Twente, Netherlands (r.aly at ewi.utwente dot nl)
Maria Eskevich, Dublin City University, Ireland (meskevich at computing.dcu dot ie)
Gareth Jones, Dublin City University, Ireland (Gareth.Jones at computing.dcu dot ie)
Roeland Ordelman, University of Twente and Netherlands Institute for Sound & Vision, a.k.a. Beeld & Geluid, Netherlands (ordelman at ewi.utwente dot nl)
Task auxiliaries
Shu Chen, Dublin City University, Ireland
Task schedule
13 May: Development data release
3 June: Test data release
9 September: Run submission due
23 September: Results returned
28 September: Working notes paper deadline
