Optimal Noise-aware Imaging with Switchable Prefilters

Abstract

Most consumer digital cameras employ a single-chip image sensor with a color filter array (CFA), where the purpose of an in-camera imaging pipeline is to generate a noise-free and color-corrected standard RGB image from mosaic CFA RAW data. The joint design of camera spectral sensitivity (CSS) and the imaging pipeline has great potential to derive better imaging quality. However, since there is a trade-off between the robustness to noise and the accuracy of color reproduction, one fixed CSS cannot realize optimal imaging in terms of both aspects under various noise levels. Thus, in this paper, we propose noise-aware imaging using camera prefilter for each noise level, where we jointly design the spectral sensitivity of the prefilters, that of CFA, and imaging networks to realize optimal imaging in all noise levels. Experimental results under various noise levels demonstrate that our imaging method using the prefilters outperforms existing methods based on a fixed CSS.

Proposed Imaging System

In this paper, we propose a novel method for optimal noise-aware imaging using camera prefilters, which are easily switchable in front of the camera lens according to each noise-level condition, as illustrated in Fig.1.

To derive optimal imaging based on a deep learning framework, we have proposed a method for jointly designing the spectral transmittance of the prefilters, the CSS of the CFA, and the imaging networks associated with each prefilter.

Fig. 1. Our imaging setup with switchable prefilters, where we jointly design the spectral transmittance of the prefilters, the camera spectral sensitivity of the CFA (Base CSS), and imaging networks associated with each prefilter.

Proposed Method Overview

Fig. 2. Overall pipeline of optimization for the prefilters, the base CSS, and the imaging networks, where the parameters in the red boxes are optimized by the deep learning framework.

Experimental Results and Visualization Examples

Fig. 3. Examples of the reference CSSs, the prefilters, the base CSS, and the equivalent CSSs obtained by our method for the CAVE dataset, where red, green, and blue lines represent the color CSSs and the black line represents the prefilters.

Fig. 4. Visual comparisons and RMSE maps of CAVE dataset for noise level sigma=30, where Noisy, SP, and Net represent signal processing imaging without denoising, signal processing imaging, and network-based imaging, respectively.

Fig. 5. Visual comparisons and RMSE maps of TokyoTech dataset for noise level sigma=10, where Noisy, SP, and Net represent signal processing imaging without denoising, signal processing imaging, and network-based imaging, respectively.

Download Materials

Paper [PDF]

Supplementary Material [Link]

Python code [Link]

Trained model [Link]

Publication

Optimal Noise-aware Imaging with Switchable Prefilters [PDF]: Zilai Gong, Masayuki Tanaka, Yusuke Monno, and Masatoshi Okutomi; IEEE International Conference on Image Processing (ICIP), October, 2022.