The task has concluded at the data has been released. Please see MediaEval Datasets.
The 2013 Crowdsourcing in Multimedia Task
The goal of this task is to allow participants to explore the potential of crowdsourcing for enhancing the potential of visual content analysis for creating descriptions of social images or for improving descriptions of social images that have been contributed by users (i.e., tags).
We provide a dataset of Creative Common social images with the focus on fashion images. The development dataset is annotated by two group of annotators: one group are the AMT annotators (which can possibly contain noisy annotations) and the other are trusted annotators known to authors (which create the correct annotations). The participant can use the annotations of AMT workers as well as any methods for analyzing visual content or socially-contributed metadata to generate an enhanced set of labels for the images. The task targets two binary labels: whether or not an image is fashion-related and whether or not an image is correctly tagged with a particular fashion item.
Target group
This task is for all researchers working on the areas of Crowdsourcing, Multimedia Information Retrieval and Multimedia Content Analysis.
Data
The dataset contains actual images, annotation of images by both group of annotators and metadata including contextual information (such as title, description, geo-tags) and social information (such as uploader, comments).
Ground truth and evaluation
We used the annotations by trusted annotators as ground truth. For each images we created two clean binary labels. The results of participants are evaluated against this ground truth. We use F1-score as the evaluation metric. Please note that the results in the 2013 working notes papers are (due to human error) correlated with but not identical to the correct results. Please contact the organizers for more information.
Recommended reading
Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10). ACM, New York, NY, USA, 64-67.
S. Nowak, S. Ruger. How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation, MIR 2010 - Proceedings of the 2010 ACM SIGMM International Conference on Multimedia Information Retrieval, 557-566, 2010.
Task organizers
Babak Loni, Delft University of Technology, Netherlands
Alessandro Bozzon, Delft University of Technology, Netherlands
Martha Larson, Delft University of Technology, Netherlands
Luke Gottlieb, ICSI, USA
Task schedule (tentative)
15 July: Dev and Test set release (please contact task organizers concerning the postponed date)
10 September: Run submission deadline
17 September: Result release
28 September: Working notes paper deadline
Note that this task is a "Brave New Task" and 2013 is the first year that it is running in MediaEval. If you sign up for this task, you will be asked to keep in particularly close touch with the task organizers concerning the task goals and the task timeline.
