MediaEval 2017 Call for Task Proposals

MediaEval 2017 Multimedia Evaluation Benchmark
Call for Task Proposals
Proposal Deadline: 3 December 2016

MediaEval is a benchmarking initiative dedicated to developing and evaluating new algorithms and technologies for multimedia retrieval, access, and exploration. It offers tasks to the research community that are related to human and social aspects of multimedia. MediaEval emphasizes the 'multi' in multimedia and seeks tasks involving multiple modalities, including audio, visual, textual, and contextual.

MediaEval is now calling for proposals for tasks to run in the 2017 benchmarking season. Details on the content of the proposal can be found below.

If you are interested in submitting a proposal, you are welcome to start the process by writing an email to Martha Larson m (dot) a (dot) larson (at) tudelft (dot) nl and Gareth Jones gareth (dot) jones (at) computing (dot) dcu (dot) ie to express your intention. They can also answer any questions that you may have, or connect you with other members of the community with similar interests to form a team of task organizers.

Content of the Task Proposal
A task proposal is a .pdf containing the following four elements. Note that there is no specified length for the proposal, but in general proposals do not exceed three pages.

1. Task Description: This is an initial version of how your task would be described on the MediaEval website, should your task proposal is accepted. It consists of the following parts:
Task Title: Give your task an informative title.
Introduction: Describe the motivating use scenario, i.e., which application(s) motivate the task. State what the task requires of participants.
Target Group: Describe the type of researchers who would be interested in participating in the task.
Data: Describe the data set, including how the data will be collected and licensed.
Evaluation Methodology: Describe the evaluation methodology, including how the ground truth will be created.
References and recommended reading: list 3-4 references related to the task that teams should have read before attempting the task.
List of task organizers.
For example task descriptions, please see the website, e.g., Task links in sidebar of

2. Task Blurb: Write 2-3 sentences that summarizes key information on the task. It should be informative and well-crafted to attract potential participants. A standard pattern is to have each sentence answer in turn the major questions about the task: First sentence: What is the input and the output of the algorithm that participants need to design for the task? Second sentence: What is the data? Third sentence: How is the task evaluated?

3. Task Organization Team: Write a short paragraph outlining the relevant interests and experience of your organizing team. Your team should be large enough to handle the organization and management of the task. Ideally teams should consist of members from multiple research sites and multiple projects. A mix of experienced and early-career researchers is preferred. MediaEval has a strong tradition of encouraging and supporting early stage researchers in gaining experience in organisation of benchmark tasks, Note that your task team can add members after the proposal has been accepted.

4. Survey Questions: Write a list of questions (3-5) that you would like to include on the survey. These questions help you to ascertain the preferences of the research community for the aspects of the design of the task formulation, the data set design, and the evaluation methodology. For examples of the types of questions asked by tasks, please have a look at this .pdf of the MediaEval 2016 survey.

Proposal deadline: 3 December 2016

Please email your proposal (as a .pdf) to Martha Larson m (dot) a (dot) larson (at) tudelft (dot) nl and Gareth Jones gareth (dot) jones (at) computing (dot) dcu (dot) ie

Additional information on proposing a MediaEval Task

MediaEval offers two types of tasks: Brave New Tasks, which are tasks that are opening up a new multimedia challenge, and General Tasks, in already-established research topics. Task proposals are accepted on the basis of the existence of a community of task supporters (i.e., researchers who are interested and would plan to participate in the task). Support is determined using a survey, which is circulated widely to the multimedia research community. Tasks must also be viable given the design of the task and the resources available to the task organization team.

All participants must sign a MediaEval usage agreement, see last year’s agreement for example of task-specific licensing. MediaEval prefers to use Creative Commons data wherever possible.

In MediaEval tasks are run autonomously by the task organization team. However, they must respect the overall schedule:
  • April 1: First date for development data release
  • June 1: Latest date for test data release
  • Mid-August: Run submission
  • Early September: Deadline for two-page working notes papers
  • 13-15 September MediaEval 2017 Workshop (in Dublin, Ireland, co-located with CLEF 2017

We encourage task proposers to join forces with colleagues from other institutions and other projects to create an organizing team large enough to bear the burden of data set generation, results evaluation, and working notes paper review.

MediaEval was founded in 2008 as a track called "VideoCLEF" within the CLEF benchmark campaign. In 2010, it became an independent benchmark and in 2012 it ran for the first time as a fully "bottom-up benchmark", meaning that it is organized for the community, by the community, independently of a "parent" project. The MediaEval benchmarking season culminates with the MediaEval workshop. Participants come together at the workshop to present and discuss their results, build collaborations, and develop future task editions or entirely new tasks. Past working notes proceedings of the workshop include:

MediaEval 2012:
MediaEval 2013:
MediaEval 2014:
MediaEval 2015: