A Multimode Vision Sensor With Temporal Contrast Pixel and Column-Parallel Local Binary Pattern Extraction for Dynamic Depth Sensing Using Stereo Vision

This article presents a 0.56/0.8 V multimode vision sensor for temporal and spatial information extraction with processing in sensor (PIS) technique. The proposed six transistors and one capacitor (6T1C) temporal contrast pixel (TCP) structure using the exposure compensation scheme (ECS) can realize in-pixel temporal contrast calculation and motion event reporting (ER) with global shutter exposure and seamless frame difference (FD) based on the pulsewidth modulation (PWM) operation. In addition, the column-parallel local binary pattern (LBP) extraction provides further spatial feature information without the power-consuming analog-to-digital conversion to meet the power-efficient requirement for always-on-edge devices. According to the temporal and spatial information extracted from FD and LBP modes, this work realizes the PIS operations for dynamic depth calculation of a stereo vision system, which filters out the static scene and provides the depth information of the dynamic objects only. To further alleviate the data transfer bandwidth, the region of interest (ROI) capability is also embedded in the sensor for data windowing and object locating. The prototyped vision sensor was fabricated in TSMC's standard 0.18 mu m CMOS process and verified. The vision sensor provides five operating modes that can adapt to different tasks, which include the raw image mode (IM), FD mode, ER mode, LBP mode, and ROI mode. The measurement results show a maximum frame rate of 540/819/540 frames/s at a power consumption of 390/162.6/151.9 mu W with an achieved iFoM of 45.5/12.5/17.7 pJ/pixel center dot frame in IM/ER/LBP mode, respectively.