The Social Event Detection (SED) task of MediaEval 2014 requires participants to discover and describe social events in a collection of social media content gathered from multiple media-sharing services on the internet (for example from Flickr, Picasa, Instagram, Twitpic). By social events we mean that the events are planned by people, attended by people and that the social media content is captured by people. A single common dataset will be provided for this task, comprising a large set of images and videos together with their associated metadata (including, for instance, time-stamps, tags, geotags, and possibly uploader’s ID). Participants will be asked to i) perform an event-based clustering of the entire media collection, ii) discover specific media clusters corresponding to events of interest, as defined by a number of event queries that will be provided to them, and iii) provide a short textual description of the event corresponding to each retrieved media cluster. For completing the task, the participants can exploit the actual image/video files, the metadata provided to them as a file with predefined structure, as well as information coming from any external resources of the participant’s choice (e.g. Wordnet, Wikipedia, or even visual concept detectors trained on external media collections).
Target group
The task is of interest to researchers in the areas of information retrieval, multimedia content analysis, social media analysis, event detection and event-based multimedia indexing.
Data
The data set comprises the URLs of Web images (in the order of hundreds of thousands of images) and possibly also some (some hundreds to few thousands) videos. These media items are accompanied by their metadata. These metadata include time-stamps, geographic information, tags, title, description, etc. (in JSON format). As it is a real world dataset, there are some features like time-stamps and uploader information which are available for every picture, but there are also features (like geographic information) which are available for only a subset of the images.
Ground truth and evaluation
Ground truth information will record the true media-event associations and will be generated by the organizers. The ground truth is single label, meaning that no image can belong to more than one event. The results of event-related media item detection will be evaluated using adapted versions of the NMI (Normalized Mutual Information), Divergence from Random Baseline, Precision, Recall and F-Score measures, building on the evaluation methodologies that were used in previous editions of the SED task. For the short textual descriptions of the events, subjective evaluation of the submissions by an appropriate panel (comprising the task organizers and possibly also additional volunteers) is envisaged for complementing the objective evaluation.
Recommended reading
[1| Petkos, G., Papadopoulos, S., Kompatsiaris, Y. Social Event Detection using Multimodal Clustering and Integrating Supervisory Signals. In Proceedings of ACM ICMR International Conference on Multimedia Retrieval. ACM, Hong Kong, 2012.
[2] Reuter, T., Cimiano, P. Event-based Classification of Social Media Streams. In Proceedings of ICMR ACM International Conference on Multimedia Retrieval. ACM, Hong Kong, 2012.
[3] Reuter, T., Papadopoulos, S., Petkos, G., Mezaris, V., Kompatsiaris, Y., Cimiano, P., de Vries, C., Geva, S. Social Event Detection at MediaEval 2013: Challenges, Datasets, and Evaluation. In Proceedings of the MediaEval 2013 Multimedia Benchmark Workshop, CEUR-WS.org, 1043, ISSN: 1613-0073. Barcelona, Spain, 2013.
Task organizers
Georgios Petkos, CERTH/ITI, Greece
Symeon Papadopoulos, CERTH/ITI, Greece
Giuseppe Rizzo, Univ. of Torino, Italy
Vasileios Mezaris, CERTH/ITI, Greece
Raphael Troncy, EURECOM, France
Task schedule
15 May: Development data release
16 June: Test data release
12 September: Run submission due
19 September: Results returned
28 September: Working notes paper deadline
This task is made possible by a collaboration of projects including:
