Skip to main content

CEC3 Submission

info

All teams are required to register prior to submission and before the registration deadline Septeber 5th.

We will be operating a two-stage submission process:

  • Stage 1: Evaluation using HASPI
  • Stage 2: Evaluation using listening panel

Key dates.

  • July 25th: Evaluation data release for HASPI evaluation for all entrants.
  • Aug 28th: Registration deadline.
  • Sept 2nd Sept 9th: All entrants submit (i) audio for HASPI evaluation and (ii) a draft of their technical report.
  • Sept 9th Sept 16th: Entrants informed which systems are going forward to the listening test evaluation stage. Evaluation data for listening tests released to those entrants.
  • Sept 16th Sept 23rd: Entrants submit audio for listening tests.

What evaluation data is provided?​

For each task the evaluation data consists of 1500 scenes. For each scene, you are provided with the signals received at each of the three microphones on the left and right hearing aid device and a head rotation signal. All signals are in the same format as in the training and development data. You will also be provided with JSON or csv formatted metadata consisting of

  1. the audiograms for a set of listeners and
  2. a mapping of which listeners will listen to which scenes.
  3. the identity of the target talker in each scene.

There will also be some clean audio examples for the target talker, that are not the same as the target utterance.

For the stage 1 HASPI evaluation, there will be one listener per scene and the scene-listener mapping will be the same for all teams.

For the stage 2 listening test evaluation, there will be five listeners per scene and each team will have a separate scene-listener mapping. The file formats will be the same as used for the development data.

What audio do I need to submit?​

For each task that you participate in, you must submit the audio signals produced at the output of your simulated hearing aid for the evaluation datasets. You will be asked to provide two sets of signals: the first for the HASPI evaluation and the second for the listening tests (see submission dates above).

Signals should be submitted as 16-bit WAV files where 0 dB FS corresponds to 100 dB SPL, given the capabilities of the listening test reproduction equipment. The format of signals submitted for HASPI evaluation and for the listening tests is the same.

We also encourage you to submit your simulated hearing aid code but this is not mandatory.

See the page on listening tests for more information about the levels that can be reproduced by the listening test equipment. When playing signals to listeners we will then play them as is. The responsibility for the final signal level is therefore yours. It’s worth bearing in mind that should your signals overall seem too loud to be comfortable to a participant, they may well turn down the volume themselves.

Naming and packaging signals​

Read the following carefully as inconsistent naming and packaging can cause problems for our evaluation pipeline.

  • Your processed signals should be named using the conventions used by the baseline system, i.e., <Scene ID>_<Listener ID>_HA-output.wav.

  • For each task, these should be placed in a directory whose name is the unique team ID that you will be sent, e.g., E001 and then packaged using zip or tar or any standard packaging tool. These should be packaged such that the directory is at the root of the archive.

  • There should be a separate package file for each task that you are participating in. Please call these files <TEADM_ID>_task1, <TEADM_ID>_task2, <TEADM_ID>_task3 with whatever file extension is appropriate for the packaging tool you are using.

We anticipate that these file will be 1-2 GB for the first round.

Technical report​

  • The two page technical report must be submitted as a paper to the Clarity-2024 Workshop.
  • Please use the author kit here and submit the report by copying the pdf to the Google Folder link that you will be sent by email when you register.
  • A draft of the report needs to be uploaded to the Google Drive along with your HASPI signals - see above for deadline. The draft needs to be sufficiently complete for us to judge whether your system is compliant with the challenge rules.
  • Your report should include an abstract and introduction and sections on experimental setup/methodology including system information and model/network architecture, evaluation/results, discussion, conclusion and references. Please provide an estimation of the computational resources needed. You must describe any external data and pre-existing tools, software and models used.

How will intellectual property be handled?​

See here under Intellectual Property.

Where do I submit the signals?​

If you have registered, during August you will receive an individual email with a link to a personal Google Drive to which you will be able to securely upload your signals. You will be able to use the same link to upload materials for all tasks and for both the 1st submission and the 2nd submission if you are selected for the 2nd round. We also encourage you to submit your simulated hearing aid code via this link.

Materials uploaded will be visible to the Clarity Team but not to other entrants.

warning

Note, in order to use the Google Drive you will need to have a Google account. If you anticipate problems using Google then please make arrangements to send us the materials by other means, e.g., via a service such as WeTransfer or similar.