Unsupervised behaviour anomaly detection from fixed camera full motion video

Full motion video electro-optical/infrared (EO/IR) sensors are now ubiquitous in civil and defence applications. Modern intelligence, surveillance, and reconnaissance (ISR) platforms provide live video feeds containing critical surveillance/observations for applications such as search & rescue, peace support, disaster relief, and port security. A crucial form of intelligence extracted from video is patterns of life, which vary drastically over space and time. Detecting anomalous behaviour within normal patterns of life in near-real time is an important task because behaviour anomalies correlate with significant, actionable information. Many anomalous behaviours cannot be specified in advance, a problem that can be addressed using unsupervised, deep-learning algorithms to identify behaviour anomalies in the video stream. We propose an operator-in-the-loop algorithmic approach that uses the latest advances in deep learning to learn patterns of life and inform operators of behaviour anomalies. Our approach uses a pre-trained object detector (RetinaNet) to identify objects within each frame paired with an object tracker based on a discriminative correlation filter. After building up a dataset of tracks, we use a combination of clustering techniques and a convolutional autoencoder to build a baseline of patterns of life for different object types. We demonstrate the ability of the autoencoder to reliably reconstruct raw object tracks from a latent space and show that at inference time tracks with larger than average loss correlate with anomalous behaviour. We demonstrate the capabilities of our approach by staging an anomalous event in front of security system cameras and compare the extracted track behaviours to normal patterns of life for that area.