Fast and accurate novelty detection for large surveillance video

Nowadays, fast and accurate novelty detection is crucial for public safety and security in surveillance videos. Given the high accuracy of deep learning technique, deep learning based novel detection is a trend. With the huge amount of surveillance videos being generated by surveillance cameras at any time, it is challenging to make novelty detection in surveillance videos efficiently while guaranteeing the accuracy. To address it, we propose a dynamic frame sampling method called ORLNet with both the frame similarity and the intensity of the object movement considered. It is based on the two observations as follows: firstly, there is a high similarity between adjacent frames in a video data. Secondly, in practice, since novel behaviors are always generated by moving targets, we only need to focus on a small number of frames that contain key information which we call key frames. Specifically, ORLNet speeds up surveillance video by setting a reinforcement learning agent to dynamically determine the indexes of key frames at run-time and replace end-to-end inference at non-key frame positions by reusing the last key frame's calculation. Typically, it defines frame similarity as novelty energy, which is the combination of novel semantic and motion features. On the premise of calculating the distance of novel energy between frames, the calculation of key frames can be reused for other frames corresponding to similar novelty energies, which can thus accelerate novelty detection while maintain accuracy. Finally, we evaluate ORLNet experimentally with two surveillance video datasets by comparing with existing methods. Experimental results show that ORLNet reduces processing time by 42% while guaranteeing the accuracy.