Multispectral Visual-Inertial Navigation Using a Dual-Layer Estimator and Targeted Histogram Equalization

The fusion of a camera and an inertial measurement unit (IMU) is a rapidly-growing approach to GPS-denied navigation due to its minimal size, weight, and power requirements. In order to become a standard approach to for robust GPS-denied navigation, visual-inertial estimation must be able to perform in all situations, especially those present in challenging environments. Despite the popularity and wide-reaching applicability of the field, most visual-inertial research has focused on a standard platform and problem of a small aerial vehicle equipped with a machine-vision camera and inexpensive IMU operating in a well-lit indoor environment. As a result, the literature on extending visual-inertial estimation to illumination-challenged environments such as poorly-lit indoor scenarios and nighttime outdoor scenarios is shallow. These situations present significant challenges to visual navigation systems as their lack of contrast and illumination results in a reduced ability to extract and match features, therefore reducing the availability of valid visual measurements. To enable operation in such challenging situations, this paper presents a visual-inertial estimator which successfully operates with a visual-spectrum camera in poorly-lit environments and a thermal-infrared camera in illumination-denied environments. This success is achieved by a modification of the standard visual feature extraction and matching process which improves the robustness of features to contrast and illumination changes. After matching and processing such features, further robustness and accuracy improvements are achieved by the implementation of a dual-layer estimator which employs a robust and efficient EKF-based frontend to preprocess and validate incoming measurements before passing them into a batch least-squares optimizing backend. Solutions from the backend are periodically fed back into the frontend to correct accumulated error and maintain accurate real-time state estimates.