MediaEval is a benchmarking initiative dedicated to developing and evaluating new algorithms and technologies for multimedia retrieval, access and exploration. It offers tasks to the research community that are related to human and social aspects of multimedia. MediaEval emphasizes the 'multi' in multimedia and seeks tasks involving multiple modalities, e.g., audio, visual, textual, and/or contextual.
MediaEval 2017 Tasks
Please contribute to the selection and design of the MediaEval 2017 Tasks by filling out the MediaEval 2017 survey:
Registration for task participation will open in February. The preliminary list of 2017 tasks is:
Retrieving Diverse Social Images Task
This task requires participants to refine a ranked list of Flickr photos retrieved using general purpose, multi-topic queries. Results are evaluated with respect to their relevance to the query and with respect to the degree of visual diversification of the final image set.
Emotional Impact of Movies Task
In this task, participating teams are expected to elaborate systems designed to predict the emotional impact of movie clips according to two use cases: (1) induced valence and arousal scores, and (2) induced fear. The training data consists of Creative Commons-licensed movies (professional and amateur) together with human annotations valence-arousal ratings. The results on a test set will be evaluated using standard evaluation metrics.
Predicting Media Interestingness Task
This task requires participants to automatically select frames or portions of movies/videos which are the most interesting for a common viewer. To solve the task, participants can make use of the provided visual, audio and text content. System performance is to be evaluated using standard Mean Average Precision.
Multimedia Satellite Task
This task requires participants to retrieve and link multimedia content from social media streams (Flickr, Twitter, Wikipedia) of events that can be remotely sensed such as flooding, fires, land clearing, etc. to satellite imagery. The purpose of this task is to augment events captured by satellite images with social media reports in order to provide a more comprehensive view. This year will focus on flooding. The multimedia satellite task is a combination of satellite image processing, social media retrieval and fusion of both modalities. The subtasks will be evaluated with Precision and Recall metrics.
Medico: Medical Multimedia Task
The goal of the task is to analyse content of medical multimedia data in an efficient way to identify and classify diseases. The data consists of video frames for at least five different disease of the human gastrointestinal tract. The evaluation is based on Precision, Recall and Weighted F1 score, the creation of a useful text report of the findings, and we also will evaluate the amount of training data used. The test set labels are created by medical
Content-based music genre recognition from multiple sources
This task invites participants to submit predictions of the music genre(s) of a given music recording using a set of precomputed state-of-the-art features extracted from audio (music features) as an input. The ground truth follows two-level hierarchy containing genres and sub-genres. In particular, this task is aimed at exploring how to combine multiple sources of ground truth into a single system, and how this affects the evaluation.
C@MERATA: Querying Musical Scores with English Noun Phrases Task
The input is a natural language phrase referring to a musical feature (e.g., ‘consecutive fifths’) together with a classical music score and the required output is a list of passages in the score which contain that feature. Scores are in the MusicXML format which can capture most aspects of Western music notation. Evaluation is via versions of Precision and Recall relative to a Gold Standard produced by the organisers.
MediaEval 2017 Timeline
3 December 2016: Task proposals due
End January 2017: MediaEval survey
Mid-February 2017: Tasks are announced
April-May: Training data release
May-June: Test data release
June-Mid-August: Work on algorithms
Mid-August: Submit runs
Early September: Working notes paper due
13-15 September MediaEval 2017 Workshop in Dublin
MediaEval 2017 Workshop
The MediaEval 2017 Workshop will be held 13-15 September 2017 at Trinity College Dublin in Dublin, Ireland, co-located with CLEF 2017.
See http://clef2017.clef-initiative.eu for information about the paper deadlines and lab deadlines for CLEF 2017.
Did you know?
Over its lifetime, MediaEval teamwork and collaboration has given rise to over 500 papers in the MediaEval workshop proceedings, but also at conferences and in journals. Check out the MediaEval bibliography.
General Information about MediaEval
MediaEval was founded in 2008 as a track called "VideoCLEF" within the CLEF benchmark campaign. In 2010, it became an independent benchmark and in 2012 it ran for the first time as a fully "bottom-up benchmark", meaning that it is organized for the community, by the community, independently of a "parent" project or organization. The MediaEval benchmarking season culminates with the MediaEval workshop. Participants come together at the workshop to present and discuss their results, build collaborations, and develop future task editions or entirely new tasks. MediaEval co-located itself with ACM Multimedia conferences in 2010, 2013, and 2016, and with the European Conference on Computer Vision in 2012. It was an official satellite event of Interspeech conferences in 2011 and 2015. Past working notes proceedings of the workshop include:
MediaEval 2012: http://ceur-ws.org/Vol-807
MediaEval 2013: http://ceur-ws.org/Vol-1043
MediaEval 2014: http://ceur-ws.org/Vol-1263
MediaEval 2015: http://ceur-ws.org/Vol-1436
Sponsors and Supporters:
Technical Committee TC12 "Multimedia and Visual Information Systems"
of the International Association of Pattern Recognition
For information on how to become a sponsor or supporter of MediaEval 2017, please contact Martha Larson m (dot) a (dot) larson (at) tudelft.nl