C@merata: Querying Musical Scores with English Noun Phrases Task
Register to participate in this challenge on the MediaEval 2016 registration site.
Texts about classical music frequently make reference to specific features of works under discussion. These references can be specific (e.g., C sharp in the treble clef) or general (fugal passage, Thor’s Hammer). Two kinds of user might be interested in finding such passages in music. Firstly, musicologists or musicians who often wish to find detailed features of a work in order to carry out further analysis or to prepare for a performance. These users know exactly what they are looking for and can express it in sophisticated natural language but may need help finding it. Secondly, people learning about music who would like to see examples of concepts they are not sure about (e.g., a perfect cadence) so that they can learn more about music itself.
The C@merata task is aimed at both classes of user. Task participants are asked to develop a system capable of answering a series of short questions. The two inputs for each question are: (1) a short noun phrase in English and (2) a short classical music score in MusicXML. The required response is a list of one or more passages in the score which contain the feature.
For detailed information about C@merata including all the previous papers, please refer to the C@merata Website. The official Task Description and Training Data will also be available from there.
Target group
This task will suit computer scientists who have a good knowledge of Western classical music or musicians or musicologists who have a good knowledge of programming. In particular, if you are a Python programmer we have tools which can help you get started.
Data
The C@merata 2016 task builds on the first two editions of this task in 2014 and 2015. In those, there were 200 questions to be matched against twenty scores in a carefully defined distribution of question types. Some questions were simple (e.g., quarter note rest) while others were slightly harder (e.g., D4 in the right hand). A few were more complex musically (e.g., tonic triad). Half the questions used English terminology (e.g., quaver) and half American (e.g., eighth note). The 2015 task used a wider range of more challenging questions. However, there was still be a distribution of easier and more difficult types. This year, the basic format will remain the same (200 questions) but some questions will be more difficult. The 2014 and 2015 test questions will be available for training.
Ground truth and evaluation
To evaluate systems we define an answer to be Beat Correct if it starts and ends in the correct bars and at the correct beat in each bar. On the other hand an answer is Measure Correct if it starts and ends in the correct bars, not necessarily at the correct beats. We then define Beat Precision and Beat Recall in terms of Beat Correct passages, and similarly Measure Precision and Measure Recall. Evaluation is therefore automatic, based on a Gold Standard produced by the organisers. The evaluation method is expected to remain the same as in 2014 and 2015.
Recommended reading
[1] Sutcliffe, R., Peñas, A., Hovy, E., Forner, P., Rodrigo, A., Forascu, C., Benajiba, Y., Osenova, P. (2013). Overview of QA4MRE Main Task at CLEF 2013. Proceedings of QA4MRE-2013.
[2] Sutcliffe, R. F. E., Crawford, T., Fox, C., Root, D. L., & Hovy, E. (2014). The C@merata Task at MediaEval 2014: Natural language queries on classical music scores. In Proceedings of the MediaEval 2014 Workshop, Barcelona, Spain, October 16-17 2014.
[3] Sutcliffe, R. F. E., Fox, C., Root, D. L., Hovy, E., & Lewis, R. (2015). The C@merata Task at MediaEval 2015: Natural language queries on classical music scores. In Proceedings of the MediaEval 2015 Workshop, Wurzen, Germany, September 14-15 2015.
Task organizers
Richard Sutcliffe, University of Essex, UK
Chris Fox, University of Essex, UK
Richard Lewis, Goldsmiths College, University of London, UK
Tom Collins, De Montfort University, UK
Eduard Hovy, Carnegie-Mellon University, USA
Deane L. Root, University of Pittsburgh, USA
Task schedule
3rd May: Release of training data
13th-17th June: Download of questions, upload of answers
24th June: Results to participants
8th July: Submission of draft papers for review
30th September: Submission of final papers to MediaEval organisers
20th-21st October: MediaEval Workshop in Amsterdam, right after ACM MM 2016