Estimating rainfall intensity from surveillance audio: A hybrid model-data-driven framework

Rainfall produces one of the most recognizable and variable sounds in nature. Audio data collected by widespread surveillance cameras provide a continuous record of rainfall events, which offers a potential opportunity for high spatiotemporal resolution rainfall estimation. However, surveillance audio (SA)1 often contains complicated environmental noise that challenges the characterisation of rainfall and makes it difficult to obtain rainfall information from SA data. This study proposes a hybrid model-data-driven framework for the numerical estimation of rain intensity based on SA. The framework is implemented in two steps: 1) a convolutional neural network (CNN) and long short-term memory (LSTM) were used to learn the frequency and temporal characteristics of rain sound, respectively, and a novel parallel neural network (PNN) was constructed to determine rain categories (e.g., light, moderate, and heavy) or the categories of rain intensities, which enabled a coarse-grained rain intensity estimation. 2) Subsequently, the Root-Mean-Square Energy (RMS-Energy) of the audio clip was employed as the indicator, and a fine-grained rainfall intensity numerical calculation model based on SA data was built. Experimental results reveal that the PNN achieves optimal performance compared to some existing relevant models, indicating that the proposed PNN can effectively determine the rain category from urban SA data. Moreover, observation from real-world surveillance scenarios demonstrates that our method achieves an average relative error of 8.01%-25.68% in the cumulative rainfall estimation. This research sheds light on building a new low-cost and high-resolution rainfall observation network based on the existing surveillance camera recourses and providing valuable support to the current rainfall observation networks.