Lightweight speaker verification with integrated VAD and speech enhancement

Reducing noise and non-speech segments that degrade speaker verification (SV) performance requires voice activity detection (VAD) and speech enhancement (SE) processing. However, this signal preprocessing introduces additional latency, especially on low-resource mobile devices with limited computational capacity. To address this limitation, we propose a novel, lightweight, unified VAD and masking-based SE module (VAD-SE module) that enhances resistance to acoustic distortions with minimal computational overhead. By integrating the VAD-SE module into a MobileNetV2-based SV model through a feature fusion module, we achieve an end-to-end model trained comprehensively. Experimental results on the ZaloAI and VinBigData datasets indicate that our model consistently outperforms conventional SV systems across various adverse conditions, achieving approximately 10% better accuracy even in noisy and silent environments, with only a 3.5% increase in parameters. Notably, when deployed on mobile devices, the model achieves superior SV accuracy with a minimal latency increase of approximately 13 milliseconds, making it highly suitable for real-time applications.