MediaEval 2019 Multimedia Evaluation Benchmark
Call for Task Proposals

Starting now: Please send us your indication of intent as soon as possible.
Friday 1 March 2019: Full proposal deadline

The Multimedia Evaluation Benchmark, MediaEval, offers challenges in the form of shared tasks. The goal of MediaEval is to develop and evaluate new algorithms and technologies for multimedia retrieval, access and exploration. MediaEval tasks are innovative, involving multiple modalities, (e.g., audio, visual, textual, and/or contextual) and focusing on the human and social aspects of multimedia. Our larger aim is to promote reproducible research that makes multimedia a positive force for society

MediaEval is now calling for proposals for tasks to run in the 2019 benchmarking season. The proposal should describe the motivation of the task and define the specific problem that task participants are required to solve. It should provide information on the data (including source and licensing), and on how the solutions developed by task participants will be evaluated. The proposal must also include a statement of how the task is related to MediaEval (i.e., the human or social component), and how it extends the state of the art in an area related to multimedia indexing, search or other technologies that support users in accessing multimedia content and collections.

Indication of Intent
If you plan to submit a task proposal, we strongly suggest that as soon as possible you submit, by email, an “Indication of Intent” in the form of a short task summary (a blurb of 50-100 words). The description should include a clear statement of what participants are expected to do, which data is used, and how participant submissions are evaluated. The summary should finish with a statement of the motivation for the task. (See the examples on past years' webpages.)

Full Task Proposal
A task proposal contains the following elements. Note that there is no specified length for the proposal, but in general proposals do not exceed three pages.

Part I: Task Description
This is a version of your task description that can be posted to the MediaEval website. Its goal is to inform and attract the interest of potential participants. It consists of the following parts:

• Task Title: Give your task an informative title.
• Introduction: Describe the motivating use scenario, i.e., which application(s) motivate the task. State what the task requires of participants.
• Target Group: Describe the type of researchers who would be interested in participating in the task.
• Data: Describe the data, including how the data will be collected and licensed.
• Evaluation Methodology: Describe the evaluation methodology, including how the ground truth will be created.
• References and recommended reading: list 3-4 references related to the task that teams should have read before attempting the task.
• List of task organizers. (Designate a lead task organizer whose contact details will appear on the website for the task.)

For example task descriptions, please see the website. Follow the task links in sidebar of past year's webpages, e.g., http://multimediaeval.org/mediaeval2017

Part II: Big Picture of the Task

• Innovation: MediaEval strives to offer innovative tasks. New tasks open up new terrain for multimedia researchers, continuing tasks introduce novel aspects every year that drive forward the state of the art. Write a couple sentences on the novelty of your task.
• Focus: MediaEval focuses on tasks that have a human or social aspect. This means that they serve new users groups, work with multimedia content produced by users, and/or address issues of affect and subjectivity. MediaEval strives to promote reproducible research that makes multimedia a positive force for society. Please comment on the focus of the task.
• Risk management: Please give a short summary of risks that you foresee for the task, and how you plan to address them (i.e., what challenges will you face in organizing the task and how do you expect to overcome them).

Part III: Task Organization Team
Write a very brief paragraph outlining the relevant interests and experience of your organizing team. Your team should be large enough to handle the organization and management of the task. This includes evaluating participant runs, and carrying out failure analysis on the results. Ideally teams should consist of members from multiple research sites and multiple projects. A mix of experienced and early-career researchers is preferred. MediaEval has a strong tradition of encouraging and supporting early-stage researchers in gaining experience in organization of benchmark tasks. Note that your task team can add members after the proposal has been accepted.

The MediaEval 2019 Workshop will be held in near Nice, France at the end of October 2019 (coordinated in time and place with ACM Multimedia).

Please email your proposal (as a .pdf or link to an editable Google doc file) to Martha Larson m (dot) a (dot) larson (at) tudelft (dot) nl and Gareth Jones gareth (dot) jones (at) computing (dot) dcu (dot) ie