MREIFlow: Unsupervised dense and time-continuous optical flow estimation from and event data

Objectives: We focus on exploring an unsupervised learning-based model which can take advantage of a single image and events to estimate dense and time-continuous optical flow. Methods: We propose a multi-scale optical flow recurrent estimation network, called MREIFlow, which mainly consists of a triplet feature encoder, a feature fusion subnetwork, and a flow iterative decoder. The triplet feature encoder is capable of extracting multi-scale features from a single image and events. The feature fusion subnetwork is designed to integrate spatial-temporal information in features and generate pseudo-features. The flow iterative decoder can perform feature correlation and estimate full-resolution optical flow iteratively in a coarse-to-fine way. We train the network in an unsupervised learning way to avoid using labeled data. In order to provide plentiful and reliable supervisory signals, we design a hierarchical unsupervised learning scheme. In the scheme, the multi-level optimization objectives are designed to supervise network training in different aspects. We also introduce a teacher model to provide auxiliary supervision. Besides, we apply a loss selection strategy to mine promising supervisory signals. Results: Experiments on the MVSEC dataset show that the proposed method achieves remarkable performance under indoor and outdoor circumstances. Compared with previous methods, the proposed method can provide dense and time-continuous optical flow estimation. Conclusion: The proposed network can take advantage of a single image and events to produce dense and time-continuous optical flow estimation, and can be trained through unsupervised learning techniques. However, it requires high computation and memory to guarantee the precision of optical flow. To address this problem, we will explore an efficient and lightweight network architecture in the future.