Single-Sensor RGB-NIR Imaging: High-Quality System Design and Prototype Implementation

In recent years, many applications using a set of red-green-blue (RGB) and near-infrared (NIR) images, also called an RGB-NIR image, have been proposed. However, RGB-NIR imaging, i.e., simultaneous acquisition of RGB and NIR images, is still a laborious task because existing acquisition systems typically require two sensors or shots. In contrast, single-sensor RGB-NIR imaging using an RGB-NIR sensor, which is composed of a mosaic of RGB and NIR pixels, provides a practical and lowcost way of one-shot RGB-NIR image acquisition. In this paper, we investigate high-quality system designs for single-sensor RGB-NIR imaging. We first present a system evaluation framework using a new hyperspectral image data set we constructed. Different from existing work, our framework takes both the RGB-NIR sensor characteristics and the RGB-NIR imaging pipeline into account. Based on the evaluation framework, we then design each imaging factor that affects the RGB-NIR imaging quality and propose the best-performed system design. We finally present the configuration of our developed prototype RGB-NIR camera, which was implemented based on the best system design, and demonstrate several potential applications using the prototype.