Submission
We are operating a two-stage submission process with the following key dates.
-
1st June: Evaluation data release for MBSTOI evaluation for all entrants.
-
11th June: Registration deadline.
-
15th June: All entrants submit (i) audio for MBSTOI evaluation and (ii) a draft of their technical report (details below).
-
22nd June: Deadline by which all entrants must submit two page technical reports to Clarity-2021 workshop (details below).
-
22nd June: Entrants informed which systems are going forward to the listening test evaluation stage. Evaluation data for listening tests released to those entrants.
-
29th June: Entrants submit audio for listening tests.
Registration
Teams are required to register using the form below. Please submit one form per team, i.e., providing a single contact email address. Once you have registered, you will receive an email confirmation with a team ID and an individualised link to a Google Drive for submitting materials.
It is important that all teams who are intending to submit an entry complete the registration form no later than 11th June.
What evaluation data is provided?
The evaluation data consists of 1500 scenes. For each scene you are provided with the signals received at each of the three microphones on the left and right hearing aid device. You will also be provided with JSON formatted metadata consisting of
- the audiograms of a set of listeners and
- a mapping of which listeners will listen to which scenes.
For the MBSTOI evaluation, there will be one listener per scene and the scene-listener mapping will be the same for all teams. For the listening test evaluation, there will be five listeners per scene and each team will have a separate scene-listener mapping. The file formats will be the same as used for the development data; for details see the CEC1 Data page.
What audio do I need to submit?
You must submit the audio signals produced at the output of your simulated hearing aid for the evaluation datasets. You will be asked to provide two sets of signals: the first for the MBSTOI evaluation (due 15th June) and the second for the listening tests (due 29th June).
-
MBSTOI evaluation. Signals should be submitted in floating point WAV format with a 44.1 kHz sampling rate. For levels, we will follow the convention in the baseline hearing aid (at the output) and hearing loss models. That is, a +/-1 square wave has RMS = 0 dB FS and corresponds to 120 dB.
-
Listening tests. Signals should be submitted as 16-bit WAV files with a 32 kHz sampling rate (due to hardware limitations). You should ensure that any samples that are greater than +1 or less than -1 have been hard-clipped at +/-1 before submission. Here, 0 dB FS corresponds to approximately 100 dB, given the capabilities of the reproduction equipment. These signals will be played as is to the listener panel.
We also encourage you to submit your simulated hearing aid code.
See the page on listening tests for more information about the levels that can be reproduced by the listening test equipment. When playing signals to listeners we will then play them as is. The responsibility for the final signal level is therefore yours. It’s worth bearing in mind that should your signals overall seem too loud to be comfortable to a participant, they may well turn down the volume themselves.
Naming and packaging signals
Your processed signals should be named using the conventions used by the baseline system, i.e., <Scene ID>_<Listener ID>_HA-output.wav and explained on the CEC1 data page.
These should be placed in a directory whose name is the unique team ID that you will be sent, e.g., E001 and then packaged using zip or tar or any standard packaging tool.
The resulting file should be about 2 GB for the first round.
Technical report
The two page technical report must be submitted as a paper to the Clarity-2021 Workshop. Deadline 22nd June. An author kit and submission instructions are available at the workshop website.
A draft of the report needs to be uploaded to the Google Drive along with your MBSTOI signals by 15th June. The draft needs to be sufficiently complete for us to judge whether your system is compliant with the challenge rules.
Your report should include an abstract and introduction and sections on experimental setup/methodology including system information and model/network architecture, evaluation/results, discussion, conclusion and references. Please provide an estimation of the computational resources needed. You must describe any external data and pre-existing tools, software and models used.
Your report should cite the following document, which provides an overview of the challenge and the baseline system:
S. Graetzer, J. Barker, T. J. Cox, M. Akeroyd, J. F. Culling, G. Naylor, E. Porter, and R. Viveros Muñoz, “Clarity-2021 challenges: Machine learning challenges for advancing hearing aid processing,” in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH 2021, Brno, Czech Republic, 2021.
The document can be accessed here.
How will intellectual property be handled?
See here under Intellectual Property.
Where do I submit the signals?
When you have registered you will receive a link to a Google Drive to which you will be able to securely upload your signals. You will be able to use the same link to upload materials for both the 1st submission, and the 2nd submission if you are selected for the 2nd round. We also encourage you to submit your simulated hearing aid code via this link.
Materials uploaded will be visible to the Clarity Team but not to other entrants.
Note, in order to use the Google Drive you will need to have a Google account. If you anticipate problems using Google then please make arrangements to send us the materials by other means, e.g., via a service such as WeTransfer or similar.