A robust and real-time DNN-based multi-baseline stereo accelerator in FPGAs

Stereo vision is essential to many robotic applications such as obstacle avoidance and 3D mapping in autonomous-driving cars, requiring substantial design efforts to achieve an appropriate balance between robustness and speed. Currently, most existing solutions for stereo vision accelerators are designed under twocamera stereo setups using handcrafted features for stereo estimation. This setup generally suffers from several drawbacks, including limited sensing range and high error rates. To resolve the above issues, we present a deep neural network based multi-baseline stereo accelerator for resource-constrained FPGAs, called Ultra-Stereo, that provides a robust and real-time solution for depth sensing. In Ultra-Stereo, binary neural network (BNN) is leveraged to obtain discriminative feature descriptors while a dynamic weighted cost fusion strategy helps to suppress the local minima in the matching cost function which effectively avoids incorrect stereo estimates. In addition, we provide a set of optimized hardware modules for Ultra-Stereo to estimate the depth map from multi-cameras in real-time. To process the data streams from multi-cameras efficiently, we first propose specific multi-baseline fusion hardware architectures against shorter and larger baseline stereo pairs. Then a time-sharing BNN is used to bridge the gap between processing speed and resource efficiency. Evaluation results demonstrate that Ultra-Stereo achieves higher matching accuracy while guaranteeing real-time responsiveness when compared with existing solutions.