AASP Challenges

Technical Committee Challenges

- Click here for Call for participation for active challenges

The AASP Technical Committee runs a series of ‘Challenges’ in order to encourage research and development with comparable and repeatable results, and to stimulate new ground-breaking approaches to specific problems in the AASP technical scope. This activity is coordinated by the Challenges Subcommittee listed below.

Call for Challenges

Proposals to organize an AASP Challenge are invited. This is an open call with no deadline. Please email a Stage 1 - Statement of Interest (see below) to the Challenges subcommittee chair Daniele Giacobello (giacobello@ieee.org).

To propose to organise a Challenge, please send a statement of interest in about 2 pages outlining the aim of the challenge and its value to the community, giving also a preliminary perspective of the practical elements including the planned test data and evaluation methodology.

If supported by the Subcommittee, a full proposal would be invited. The full proposal should include the following items:

Please send the Statement of Interest and the Full Proposal to the AASP Challenges Subcommittee chair at the address below. All proposals will be considered by the Challenges Subcommittee. The Subcommittee may request modifications to the challenge as a condition of acceptance.

1. Stage 1 - Statement of Interest
2. Stage 2 - Full Proposal
   
   • a textual description of the challenge and its context (1 to 2 pages);
   • a clear formulation of the problem to be addressed;
   • a specification of the evaluation methodology leading to an objective figure of merit (FoM) and, where appropriate, a software tool to compute the FoM;
   • a development dataset which represents the challenge and which will be made public (a public training dataset may also be needed in some challenges);
   • a test dataset which also represents the challenge but which will remain private during the challenge;
   • a commitment to provide a website to disseminate the challenge itself and, eventually, the results;
   • a commitment to evaluate the submitted results and publish the comparison on the website and elsewhere as appropriate;
   • a proposed schedule for the challenge (date of publication of the challenge, deadline of results submission, deadline of comparative results publication).

Participation

Researchers entering the challenge are invited to sign up at the challenge website. Participants will address the challenge specification and employ the evaluation methodology and the development dataset to develop their algorithm. Participation is open to all.

Evaluation

At the end of the challenge, the organizers will coordinate a comparative evaluation employing the defined FoM. Evaluation may be done for example by releasing the test dataset and asking the participants to return their results on the test data within a short period of time, typically two weeks. Participants would be honour-bound not to use the test dataset for tuning.

Dissemination

The evaluation results will then be published by the organizers. The Challenges Subcommittee will work with the challenge organizers towards publication of the challenge and its outcome in the IEEE Transactions and appropriate conferences, ICASSP in particular. In addition, the challenge organizers’ website will be linked from the TC webpage. Participants can choose to remain anonymous in publications.

The 'AASP Challenges' Subcommittee

The current membership of the subcommittee can be found on the subcommittee overview page: https://signalprocessingsociety.org/community-involvement/audio-and-acoustic-signal-processing/aasp-subcommittees

---

IEEE AASP CHALLENGES

DCASE 2018

Our everyday environments are rich in acoustic information that allows us to understand and recognize the acoustic scene we are in (street, park, library, etc), and to recognize individual sound sources in the scene (cars passing by, birds singing, etc). Automatic audio signal processing methods that have the capability to extract this information have wide applicability, including context-aware devices, multimedia information retrieval, and intelligent acoustic monitoring.

DCASE 2018 Challenge continues to support the development of computational scene and event analysis methods, by providing five tasks and benchmark datasets that allow comparison of different approaches:

 - Task 1: Acoustic scene classification
 - Task 2: General-purpose audio tagging of Freesound content with AudioSet labels
 - Task 3: Bird audio detection
 - Task 4: Large-scale weakly labeled semi-supervised sound event detection in domestic environment
 - Task 5: Monitoring of domestic activities based on multi-channel acoustics

The website can be found at: http://dcase.community/challenge2018/. It contains full description of the tasks, development and evaluation datasets download links, baseline system implementation, participation rules and contact information of coordinators for each task. All technical reports submitted for the challenge will also be publicly available on the challenge website.

The challenge outcomes will be presented at the DCASE 2018 Workshop (19-20 Nov 2018, Surrey, UK, http://dcase.community/workshop2018/) where a long oral presentation and a special poster session will be dedicated to presentation and analysis of challenge results.

LOCATA Challenge

The development dataset for the IEEE-AASP Challenge on Acoustic Source Localization and Tracking (LOCATA) has been published.  

The aim of this challenge is to provide researchers in the field of acoustic source localization and tracking the opportunity to benchmark their algorithms against competing approaches using a common data corpus that encompasses real multichannel recordings for different scenarios and microphone configurations.  The dataset can be obtained from the LOCATA website http://www.locata-challenge.org. 

The development dataset is intended to allow participants to familiarize themselves with and adapt their algorithms to the data corpus and evaluation framework. For this purpose, the development dataset contains the ground-truth positional data of all sources. For participation in the LOCATA challenge, participants submit the performance results of their algorithms applied to the evaluation dataset.

The LOCATA challenge outcome will be announced during a satellite workshop to be held at IWAENC 2018 (www.iwaenc2018.org). Challenge participants submit to the LOCATA workshop a 4-page paper detailing the algorithmic framework of their submission(s). Papers submitted to the LOCATA challenge will be published on arXiv in the proceedings of the IEEE-AASP LOCATA Challenge. The proceedings aim to provide an overview of the practical aspects of the algorithmic frameworks submitted to LOCATA.  Challenge participants are strongly encouraged to submit papers describing novel contributions, such as the proposal of new algorithms, in the form of regular papers to IWAENC in addition to participating in the LOCATA satellite workshop during IWAENC.

The current timetable is as follows:
• February 16,  2018: Release of the development (Dev) data
• April 16, 2018: Release of the evaluation (Eval) data
• April 20, 2018: IWAENC regular paper deadline
• June 1, 2018: LOCATA deadline for evaluation results
• August 1, 2018: LOCATA paper deadline
• September 20, 2018: LOCATA satellite workshop at IWAENC 2018 in Tokyo

The organizing committee of the LOCATA Challenge is composed by Heinrich Löllmann, Christine Evers, Patrick A. Naylor and Walter Kellermann

DCASE 2017

Sounds carry a large amount of information about our everyday environment and physical events that take place in it. We can perceive the sound scene we are within (busy street, office, etc.), and recognize individual sound sources (car passing by, footsteps, etc.). Developing signal processing methods to automatically extract this information has huge potential in several applications, for example searching for multimedia based on its audio content, making context-aware mobile devices, robots, cars etc., and intelligent monitoring systems to recognize activities in their environments using acoustic information. However, a significant amount of research is still needed to reliably recognize sound scenes and individual sound sources in realistic soundscapes, where multiple sounds are present, often simultaneously, and distorted by the environment.
Building up on the success of the previous editions (DCASE 2013 and DCASE 2016, both supported by IEEE AASP TC), this year's evaluation continues supporting the development of computational scene and event analysis methods by comparing different approaches using a common publicly available dataset. The continuous effort in this direction sets the milestones of development and anchors the current performance for further reference. 

Complete information is available at http://www.cs.tut.fi/sgn/arg/dcase2017/.

The website contains all necessary details, including tasks and datasets descriptions, baseline system implementation and performance on development dataset, submission procedure, challenge rules, and contact information of task coordinators. 

SPCup 2017

The IEEE Signal Processing Cup (SP Cup) is aimed at providing undergraduate students with an opportunity to form teams and work together to solve a challenging and interesting real-world problem using signal-processing techniques and methods. The SP Cup was first launched in 2013-2014, and SP Cup 2017 is now the fourth edition.

The 2017 SP Cup competition topic is related to real-time beat tracking of musical signals. The beat is a salient periodicity in a music signal. It provides a fundamental unit of time and foundation for the temporal structure of the music. The significance of beat tracking is that it underlies music information retrieval research and provides for beat synchronous analysis of music. It has applications in segmentation of audio, interactive musical accompaniment, cover-song detection, music similarity, chord estimation, and music transcription. It is a fundamental signal processing task of interest to any company providing information services related to music (adapted from Degara et al., Reliability-informed beat tracking of musical signals, IEEE Trans. on Audio, Speech, and Language Processing, 20:1, 290-301, 2011).

Participants are expected to utilize a number of fundamental signal processing theories and techniques at the undergraduate level, and the project involves hardware that is affordable and easy-to-build as well as algorithm design and software implementation.

DCASE 2016

Sounds carry a large amount of information about our everyday environment and physical events that take place in it. Humans can perceive the sound scene we are within (busy street, office, etc.), and recognize individual sound sources (car passing by, footsteps, etc.). The scope of this evaluation is to advance the development of computational scene and event analysis methods by comparing different approaches using a common publicly available dataset and similar metrics, and anchor the current performance for further reference. This challenge follows the success of first edition of the challenge, DCASE 2013.

Everyone is welcome to participate to the challenge, whether it is for only one task or multiple tasks.

On the website of DCASE 2016 you will find everything you need, including:

• Overview of the tasks providing details of the motivation
• Detailed descriptions of the tasks, including description of the training and development datasets, metrics and baseline systems for each
• Detailed description of the challenge rules
• Instructions on how to submit your results
• Download page with links to all tools and data packages

DCASE 2016 is an official IEEE Audio and Acoustic Signal Processing (AASP) challenge. Participants will be invited to present their work during a one day workshop that will be organized as a satellite workhop (to be confirmed) to the 2016 European Signal Processing Conference (EUSIPCO), to be held in Budapest, Hungary.

If you need any additional information or have questions about the challenge, do not hesitate to contact us at dcasechallenge@gmail.com.

If you are considering participating or just want to learn more then please join the Google group DCASE discussions to be up to date with all related information.

 

DCASE 2016 is a challenge organized by IEEE Signal Processing Society, Tampere University of Technology, Queen Mary University of London, Institut de Recherche et Communication et Cibernétique de Nantes and University of Surrey.

IEEE AASP CHALLENGES 2014

ACE

www.ace-challenge.org

Call for Participation:

Several established parameters and metrics have been used to characterize the acoustics of a room. The most important are the Direct-To-Reverberant Ratio (DRR), the Reverberation Time (T60) and the reflection coefficient. The acoustic characteristics of a room based on such parameters can be used to predict the quality and intelligibility of speech signals in that room. Recently, several important methods in speech enhancement and speech recognition have been developed that show an increase in performance compared to the predecessors but do require knowledge of one or more fundamental acoustical parameters such as the T60. Traditionally, these parameters have been estimated using carefully measured Acoustic Impulse Responses (AIRs). However, in most applications it is not practical or even possible to measure the acoustic impulse response. Consequently, there has been a growing research activity in the estimation of such parameters directly from speech and audio signals.

ACE Challenge Overview

The ACE Challenge is part of the programme of Challenges organised by the IEEE Audio and Acoustic Signal Processing Technical Committee http://www.signalprocessingsociety.org/technical-committees/list/audio-tc/.

The aim of this challenge is to evaluate state-of-the-art algorithms for blind acoustic parameter estimation from speech and to promote the emerging area of research in this field. Participants will evaluate their algorithms for T60 and DRR estimation against the ‘ground truth’ values provided with the data-sets. Furthermore, they are expected to present the results in a paper describing the method used.

Full details can be found at www.ace-challenge.org

• Data: A data-set specifically designed for the challenge tasks will be provided. This will include speech from male and female talkers in different sized rooms and different noise conditions for a single microphone and for microphone arrays with three (mobile), five (cruciform), eight (linear), and thirty-two (spherical) microphones
• Task 1: Single-microphone T60 and DRR estimation
• Task 2: Multi-microphone T60 and DRR estimation
• Evaluation: The evaluation metrics will be based on the ground truth values. This will consist of full-band and 1/3-octave band values for T60 and full-band values for the DRR

ACE Challenge Schedule

Please see www.ace-challenge.org for the key dates of the ACE Challenge.

ACE Challenge Workshop

The challenge participants will be invited to present their results at the ACE Challenge workshop, which is planned to be held as a satellite event in conjunction with WASPAA 2015 in New Paltz, NY, USA

Organizing Committee
Patrick A. Naylor and James Eaton (Imperial College London), Nikolay D. Gaubitch (Delft University of Technology)

IEEE AASP CHALLENGES 2013

REVERB

Call for Participation:

Recently, substantial progress has been made in the field of reverberant speech signal processing, including both single- and multi-channel de-reverberation techniques, and automatic speech recognition (ASR) techniques robust to reverberation. To evaluate state-of-the-art algorithms and draw new insights regarding potential future research directions, we call for participation in the REVERB (REverberant Voice Enhancement and Recognition Benchmark) challenge that will provide an opportunity to the researchers in the field to carry out comprehensive evaluation of their methods based on a common database and evaluation metrics. The challenge aims at bringing together researchers from a broad range of disciplines to discuss novel and established approaches to handle reverberant speech. This challenge is part of the IEEE SPS AASP challenge series. The following is an outline description of the challenge.

REVERB Challenge Overview

• Data: Real and simulated 1-, 2-, and 8-channel recordings in reverberant meeting rooms based on the Wall Street Journal Corpus. This data is common for the following 2 tasks*.
• Task 1: Enhancement of reverberant speech with single-/multi-channel de-reverberation techniques
  (Evaluation metrics: objective and subjective measures)
• Task 2: Robust recognition of reverberant speech
  (Evaluation metric: word error rate)
  * Participants are invited to take part in either or both of the above tasks.

REVERB Challenge Workshop

The results will be presented by the participants at the REVERB challenge workshop, which will be held in conjunction with ICASSP2014 and HSCMA2014.

Important Dates

July	1st,	2013	Release of development dataset and scripts for evaluation
Nov	5,	2013	Release of evaluation dataset
Dec	1,	2013	Deadline for submission of results
Jan	10,	2014	Deadline for submission of papers
Feb	28,	2014	Notification of acceptance
May	10,	2014	Workshop in conjunction with ICASSP2014 (Florence, Italy)

Further details at reverb2014.dereverberation.com

Organizing Committee
Marc Delcroix (NTT), Sharon Gannot (Bar-Ilan Univ.), Emannuel Habets (International Audio Labs Erlangen), Reinhold Haeb-Umbach (Paderborn Univ.), Walter Kellermann (Univ. of Erlangen-Nuremberg), Keisuke Kinoshita (NTT), Volker Leutnant (Paderborn Univ.), Roland Maas (Univ. of Erlangen-Nuremberg), Tomohiro Nakatani (NTT), Bhiksha Raj (Carnegie Mellon Univ.), Armin Sehr (Beuth Univ. of Applied Sciences Berlin), Takuya Yoshioka (NTT)

IEEE AASP CHALLENGES 2012

Detection and Classification of Acoustic Scenes and Events

Call for Participation:

On behalf of the IEEE AASP Technical Committee, I am happy to announce a new challenge entitled: "Detection and Classification of Acoustic Scenes and Events". The challenge has the form of a public contest for the evaluation of the performance of systems for the detection and classification of acoustic events and audio scenes.

The challenge includes a set of tasks for the detection and classification of acoustic scenes and events and its goal is to provide a focus of attention for the scientific community in developing systems for computational auditory scene analysis (CASA) that will encourage sharing of ideas and improve the state of the art, potentially leading to the development of systems that achieve performance closer to that of humans.

This challenge will help the research community move forward by providing a focus for better defining the specific tasks, and will also provide incentive for researchers to actively pursue research on this field. Finally, it will offer a reference point for future systems developed to perform similar tasks and it will provide the community with a high quality database for future research.

There will be a discussion phase where potential participants are invited to contribute their ideas, ending on 30th September 2012. The deadline for code submission is 31st March 2013. Results will be presented at a special session in WASPAA 2013 (http://www.waspaa.com/); participants are invited to present a poster at a special session. Also, authors of novel work are encouraged to submit their work as a regular paper at WASPAA 2013.

For more details as well as a copy of the full proposal of the challenge please visit:
http://www.elec.qmul.ac.uk/digitalmusic/sceneseventschallenge/

The challenge organisers,
Dimitrios Giannoulis (QMUL), Emmanouil Benetos (QMUL), Dan Stowell (QMUL), Mathieu Lagrange (IRCAM) and Mark Plumbley (QMUL)

---

2nd CHiME Speech Separation and Recognition Challenge

----------------------------------------------

Deadline: January 15, 2013
Workshop: June 1, 2013, Vancouver, Canada

http://spandh.dcs.shef.ac.uk/chime_challenge/

----------------------------------------------

Following the success of the 1st PASCAL CHiME Speech Separation and
Recognition Challenge, we are happy to announce a new challenge
dedicated to speech recognition in real-world reverberant, noisy conditions,
that will culminate in a dedicated satellite workshop of ICASSP 2013.

The challenge is supported by several IEEE Technical Committees and by
an Industrial Board.

FEATURED TASKS

The challenge consists of recognising distant-microphone speech mixed in
two-channel nonstationary noise recorded over a period of several weeks
in a real family house. Entrants may address either one or both of the
following tracks:

Medium vocabulary track: WSJ 5k sentences uttered by a static speaker

Small vocabulary track: simpler commands but small head movements

TO ENTER

You will find everything you need to get started (and even more) on the
challenge website:
- a full description of the challenge,
- clean, reverberated and multi-condition training and development data,
- baseline training, decoding and scoring software tools based on HTK.

Submission consists of a 2- to 8-page paper describing your system and
reporting its performance on the development and the test set. In
addition, you are welcome to submit an earlier paper to ICASSP 2013,
which will tentatively be grouped with other papers into a dedicated
session.

Any approach is welcome, whether emerging or established.

If you are interested in participating, please email us so we can
monitor interest and send you further updates about the challenge.

BEST CHALLENGE PAPER AWARD

The best challenge paper will distinguished by an award from the
Industrial Board.

IMPORTANT DATES

July 2012 Launch
October 2012 Test set release
January 15, 2013 Challenge & workshop submission deadline
February 18, 2013 Paper notification & release of the challenge results
June 1, 2013 ICASSP satellite workshop

INDUSTRIAL BOARD

Masami Akamine, Toshiba
Carlos Avendano, Audience
Li Deng, Microsoft
Erik McDermott, Google
Gautham Mysore, Adobe
Atsushi Nakamura, NTT
Peder A. Olsen, IBM
Trausti Thormundsson, Conexant
Daniel Willett, Nuance

WORKSHOP SPONSORS

Conexant Systems Inc.
Audience Inc.
Mitsubishi Electric Research Laboratories

ORGANISERS

Emmanuel Vincent, INRIA
Jon Barker, University of Sheffield
Shinji Watanabe & Jonathan Le Roux, MERL
Francesco Nesta & Marco Matassoni, FBK-IRST