Multi-resolution approach to Identification of spoken languages and to improve overall Language Diarization System using Whisper Model

This research paper investigates the effectiveness of the Whisper decoder for Language Identification (LI) and Language Diarization (LD) tasks. An audio accent detection system was used as an attention mechanism to narrow down the Whisper LI output classes. The LI system was tested on different audio resolutions ranging from 1.0 to 11.0 seconds, and the segments obtained were combined to generate RTTM per audio resolution. Lastly, we ensemble different multi-resolution diarization systems using DOVER-Lap algorithm. This work was part of DISPLACE challenge organized in INTERSPEECH 2023 and hence the challenge dataset was utilized for all the experiments. It shows that 5-second of audio resolution (i.e.,S-1) yield optimum result of 38.12% and 42.45% DER on development and evaluation data respectively. Furthermore, combining multi-resolution diarization systems (i.e.,S-2) produced an absolute improvement of 3.22% over S-1 and 11.66% over the challenge baseline, with a total DER of 34.9% on the Development set.