An End-to-End Deep-Learning-Based Indirect Time-of-Flight Image Signal Processor

Indirect time-of-flight (iToF) is one of the most straightforward approaches to capture 3D images. However, due to the nature of the iToF camera, it is still challenging to get reliable and accurate depth images from the raw data in the image signal processor (ISP) pipeline due to environmental issues. Previous iToF ISP works mainly focus on the traditional pipeline, such as filters and depth calculation. In this work, we present an end-to-end deep-learning-based iToF ISP. The proposed iToF ISP system can generate real-time depth images with deep-learning-based noise reduction and multipath interference (MPI) reduction. With the mixed-bit convolutional neural network (CNN) with 96.5 % sparsity and the mixed-bit sparse accelerator, the CNN is accelerated by 2.78x and negligible mean average error (MAE) loss has been achieved on the FLAT dataset using the proposed ISP pipeline.