The task has concluded at the data has been released. Please see MediaEval Datasets.
The 2013 Placing Task
The Placing Task requires participants to estimate the geographical coordinates (latitude and longitude) of photos, as well as to indicate how "placeable" they actually are by estimating the error of the predicted locations. Pinpointing the exact or approximate coordinates of a photo is not merely an intellectual challenge. It has clear utility and potential for far-reaching impact in multimedia and beyond. As an example, rescue teams may be able to narrow down where exactly a family disappeared in a remote area by discovering the locations shown in photos uploaded to a social network before they lost contact. The placing task integrates various aspects of multimedia: textual metadata, image content, location, time, users and context.
Task description
Main Task: The task requires participants to assign geographical coordinates (latitude and longitude) to each provided test image. Participants can make use of metadata, user social information, and visual features as well as external resources, depending on the run. Note that a minimum of one run that uses only visual features will be required (if participants envisage problems fulfilling this criteria, please contact the organisers).
Placeability (Sub-Task): This sub-task requires participants to estimate the error of the locations predicted in the main task. The question for the participants is if they can estimate the accuracy of their method(s) to predict the location. For example, for a given image, if the method returns a list of locations where the top-n are very far among them, the top location (i.e., the one that will be returned as estimated location) could have a low accuracy. Instead if the top-n locations are all very close among them (e.g., standard deviation of few kilometers), then the accuracy could be very high. The participants that will apply also for this sub-task, they have to specify for each test image, not only the location in terms of coordinates (main task), but also the error estimation in kilometers.
Target group
The task is of interest to researchers in the area of geographic multimedia information retrieval, social media and media analysis.
Data
The dataset for the Placing Task 2013 is a completely new one this year and it contains only images.
In total, it contains 7.8 Million Flickr images released by the owners with Creative Commons Licence. All the images are geo-tagged with the highest Flickr precision (16/16).
The test set is available in various sizes, ranging between 5,000 and 250,000 photos, in order to allow participation in the task even without very powerful resources.
The data is provided in two forms: textual meta-data and image features. Note, that in both cases, the test and training data is included in the same files. These files do NOT contain any latitude/longitude information. This information is provided in a separate file for the training data. The test data is released in five files (pick one to process) and lists only the photoIDs for which to estimate the location and the error.
Ground truth and evaluation
The geo-coordinates associated with the Flickr will be used as the ground truth. Since these do not always serve to precisely pinpoint the location of the image, we evaluate at each of a series of widening circles: 1km, 10km, 100km, 1000km, 10000km.
The estimated error (provided by the participants) is compared with the true error and the average distance between the two values will be recorded as well as the overall correlation between the estimated and true error.
Recommended reading
- Serdyukov, P., Murdock, V., and van Zwol, R. "Placing flickr photos on a map", SIGIR 2009.
- Hays, J. and Efros, A. A. "IM2GPS: Estimating geographic information from a single image", CVPR 2008.
- Trevisiol M., Jégou H., Delhumeau J., and Gravier G. "Retrieving geo-location of videos with a divide & conquer hierarchical multimodal approach, ICMR 2013.
- Hauff C., and Houben, G. “Geo-Location estimation of flickr images: social web based enrichment”, ECIR 2012.
- O’Hare, N. and Murdock, V. "Modeling locations with social media", Information Retrieval 2012.
- Crandall,D. and Backstrom, L. and Huttenlocher, D. and Kleinberg, J. “Mapping the World’ s Photos”, WWW 2009.
- Kelm, P. and S. Schmiedeke, S. and Sikora, T. “A Hierarchical , Multi-modal Approach for Placing Videos on the Map using Millions of Flickr Photographs”, SBNMA 2011.
- Penatti, O. A. B. and Li, L. T. and Almeida, J. and R. da S. Torres, “A Visual Approach for Video Geocoding using Bag-of-Scenes”, ICMR 2012.
Task organizers
Bart Thomee (Yahoo! Research Barcelona, Spain)
Michele Trevisiol (Universitat Pompeu Fabra, Spain)
Claudia Hauff (Delft University of Technology, Netherlands)
Task auxiliary
Xinchao Li (Delft University of Technology, Netherlands)
Task schedule (updated)
24 May: Development and test set released
15 September: Run submission deadline (updated)
21 September: Results of the task released (updated)
28 Sept: Working notes paper due (updated)
The MediaEval 2013 workshop will take place in Barcelona, Catalunya, Spain, on Friday-Saturday 18-19 October 2013.