Sequence-to-Sequence Neural Diarization With Automatic Speaker Detection and Representation

This paper proposes a Sequence-to-Sequence Neural Diarization (S2SND) framework for both online and offline speaker diarization. Built upon a sequence-to-sequence architecture, S2SND integrates two novel components: (1) a masked speaker prediction mechanism that enables the model to detect unknown speakers without preextracted embeddings and (2) a target-voice speaker embedding extraction module that infers speaker representations using predicted voice activities as reference. This joint modeling approach eliminates the need for unsupervised clustering or permutation-invariant training. During inference, S2SND processes long-form audio in continuous blocks, leveraging a speaker-embedding buffer to maintain speaker consistency and enable low-latency prediction. It supports real-time detection of new speakers and efficient rescoring for improved offline performance. The entire diarization network is trained end-to-end, with binary cross-entropy and ArcFace loss guiding the detection and representation branches, respectively. Experimental resultsdemonstrate that S2SND achieves state-of-the-art diarization error rates (DERs) across multiple conditions. Specifically, it achieves DERs of 24.41% (online) and 21.95% (offline) on DIHARD-II, and 17.12% (online) and 15.13% (offline) on DIHARDIII, without using oracle voice activity detection. latex version:This paper proposes a Sequence-to-Sequence Neural Diarization (S2SND) framework for both online and offline speaker diarization. Built upon a sequence-to-sequence architecture, S2SND integrates two novel components: (1) a masked speaker prediction mechanism that enables the model to detect unknown speakers without pre-extracted embeddings and (2) a target-voice speaker embedding extraction module that infers speaker representations using predicted voice activities as reference. This joint modeling approach eliminates the need for unsupervised lustering or permutation-invariant training. During inference, S2SND processes longform audio in continuous blocks, leveraging a speaker-embedding buffer to maintain speaker consistency and enable low-latency prediction. It supports real-time detection of new speakers and efficient rescoring for improved offline performance. The entire diarization network is trained end-to-end, with binary cross-entropy and ArcFace loss guiding the detection and representation branches, respectively. Experimental results demonstrate that S2SND achieves state-of-the-art diarization error rates (DERs) across multiple conditions. Specifically, it achieves DERs of 24.41\% (online) and 21.95\% (offline) on DIHARD-II, and 17.12\% (online) and 15.13\% (offline) on DIHARD-III, without using oracle voice activity detection.