E032 - Exploiting Hidden Representations from a DNN-based Speech Recogniser for Speech Intelligibility Prediction in Hearing-impaired Listeners

The implementation of “Exploiting Hidden Representations from a DNN-based Speech Recogniser for Speech Intelligibility Prediction in Hearing-impaired Listeners”, accepted to InterSpeech 2022. For the 1st Clarity Prediction Challenge (CPC1) programme and details, please see here.

Please note: This code provides the implementation of the LS(LibriSpeech) + CPC1 model, as shown in the third row of Table 2 in the paper. Since the generation of CLS data (LS train-clean-100 set added with noises from the training set in CEC1) and the training with CLS data are over-complicated, the scripts are not provided here. Anyway, the improvement with CLS is limited…

As only the training data is provided in CPC1, we split the training data into a train set and dev set. The train set is used for ASR training, and the dev set is used for optimizing the logistic fitting function.

Run the scripts

Requirements

torch==1.10.0
torchaudio==0.10.0
speechbrain==0.5.9
fastdtw==0.3.4

You probably need a GPU as well…

Prepare ASR data

To train the ASR model and generate the hidden representations from it, the CPC1 data needs to be processed first. This part of code will (1) randomly split train and dev set; (2) run MSBG hearing loss simulation to all signals in clarity_data/HA_output; (3) resample signals to 16kHz and generate csv files for SpeechBrain ASR model training.

  1. Download clarity_CPC1_data.v1_1.tgz and clarity_CPC1_data.test.v1.tgz, untar them into clarity_CPC1_data_train and clarity_CPC1_data_test, respectively. See recipes/cpc1/baseline/README.

  2. Specify root in config.yaml. Both clarity_CPC1_data_train and clarity_CPC1_data_test should be in your root folder. You could also specify your own exp_folder (e032 by default).

  3. Specify cpc1_track as ‘open’ or ‘closed’.

  4. Run python prepare_data.py (Note, same as data preparation for the e029_sheffield recipe).

  5. cpc1_asr_data which contains the processed train & test CPC1 data and their csv files, and data_split which contains the train set and dev set scenes, will appear in your exp_folder (e032 by default).

Train ASR

  1. Create your transformer_cpc1 folder to save ASR models & results

  2. Specify output_folder & data_folder in transformer_cpc1.yaml: output_folder: !ref your_path/transformer_cpc1 & data_folder: !ref your_path/e032/cpc1_asr_data  # for closed-set OR data_folder: !ref your_path/e032/cpc1_asr_data_indep  # for open-set

  3. Download the save folder (i.e. ASR transformer checkpoint) from: https://drive.google.com/drive/folders/1ZudxqMWb8VNCJKvY2Ws5oNY3WI1To0I7, and place it under your transformer_cpc1 folder

  4. Run python train_asr.py transformer_cpc1.yaml

  5. The trained ASR model checkpoint will appear in your_path/transformer_cpc1/save

Infer hidden representation similarity

This part generates the similarities of encoder representations and decoder representations between the MSBG processed signals and reference signals.

  1. Run python infer.py. Four json files dev_dec_similarity.json, dev_enc_similarity.json, test_dec_similarity.json, test_enc_similarity.json will be generated in you exp_folder.

Evaluation

This part optimize a logistic fitting function with the dev set similarities and dev set labels, and applies the fitting function to the test set similarities for scaled predicted test set prediction. And the evaluation results of RMSE, Std, NCC and KT will be computed and stored in the results.json.

  1. Run python evaluate.py

Citation

If you use this code for your research, please cite:

@inproceedings{tu2022exploiting,
  title={Exploiting Hidden Representations from a DNN-based Speech Recogniser for Speech Intelligibility Prediction in Hearing-impaired Listeners},
  author={Tu, Zehai and Ma, Ning and Barker, Jon},
  booktitle={INTERSPEECH 2022},
  year={2022}
}