CN110235441B

CN110235441B - Multi-camera system for low illumination

Info

Publication number: CN110235441B
Application number: CN201780085025.8A
Authority: CN
Inventors: 沼田肇
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2017-03-24
Filing date: 2017-03-24
Publication date: 2021-01-15
Anticipated expiration: 2037-03-24
Also published as: WO2018170918A1; CN110235441A

Abstract

A multi-camera system is provided. The multi-camera system includes a color image sensor for generating a color image and a monochrome filter image sensor for generating a monochrome image, wherein the monochrome filter image sensor has a monochrome filter image sensor comprising a zig-zag pixel array having a square root of a pixel size x1.4 or 2, the zig-zag pixel array being a pixel array rotated 45 degrees relative to a square pixel array having a pixel size x 1; an output color image is generated by fusing the color image and the monochrome image.

Description

Multi-camera system for low illumination

Technical Field

Embodiments of the present invention relate to multi-camera systems for low illumination.

Background

The performance and technology of smart phone cameras is constantly improving. The quality of a smartphone camera is now almost the same as that of a Digital Still Camera (DSC)/Digital single-lens reflex camera (DSLR). Since further improvement in convenience is required, higher sensitivity is often highly required.

Due to size limitations, smartphones require the use of small image sensors and resolutions higher than 8MP or 12 MP. To meet these two requirements, the size of the individual pixels is getting smaller, e.g. 1.4um, 1.12um, 1um or smaller. Due to the small pixel size, the sensor/pixel sensitivity is low, and the image quality of a smartphone camera under low light conditions is worse than that of a DSLR or higher quality still camera. With the development of smartphone camera image sensors, their efficiency (i.e., QE: quantum efficiency) has improved. For an image sensor used by a smartphone camera (e.g., a so-called second camera using a WRGB (or RGBW) array or a W sensor), a W (white) colored filter sensor may be used. FIG. 4 is a diagram of an exemplary prior art WRGB (or RGBW) sensor.

Patent document 1(US9147230B2) discloses an image sensor having an RGBW array including full wavelength type white (W) pixels in addition to RGB colors. When the pixel density of the image sensor increases and the amount of light incident on each pixel decreases, resulting in an increase in relative noise, we propose the RGBW array as a method of solving such a problem. In fig. 4, a portion having a diagonal line from left to right represents blue (B), a portion having a diagonal line from left to right represents red (R), and a dotted line portion represents green (G).

Patent document 2(US9253459B2) also discloses that the RGBW arrangement exhibits a reduction in the density of color pixels relative to the RGB (or bayer) pixel arrangement. Fig. 5 is a set of diagrams of an example of a related-art both monochrome and color camera. Fig. 5 shows that the monochrome (or B/W (black/white)) camera image sensor has the same pixel size as the color camera image sensor. In fig. 5, a portion having a diagonal line from left to right represents blue (B), a portion having a diagonal line from left to right represents red (R), and a dotted line portion represents green (G).

Disclosure of Invention

Problems to be solved by the invention

The technical problems of the related art can be summarized as follows:

(1) the improvement rate of QE is not very large. The QE of the prior art sensors is already sufficiently high. Therefore, even if QE improves, the rate is within a few percent, so the sensitivity does not increase much.

(2-1) although the white pixels of the WRGB array may be 1.3 to 2 times as sensitive as the color pixels, the WRGB array requires a complicated interpolation algorithm and false colors may occur due to aliasing due to low color resolution. Moreover, the 1.5 to 2 fold improvement is not so great for practical use.

(2-2) although the second camera using the W sensor may reduce interpolation and color resolution problems, the improvement rate of sensitivity is the same as that of the second camera using WRGB.

It is an object of embodiments of the present invention to use relatively large white pixels on a second camera with a different pixel array while maintaining the same resolution as a conventional color camera, thereby improving low-illumination performance.

Means for solving the problems

According to an embodiment of the present invention, there is provided a multi-camera system including: a color image sensor for generating a color image; a monochrome filter image sensor for generating a monochrome image, wherein the monochrome filter image sensor has a sensor including a zigzag pixel array having a pixel size of x1.4 (or the square root of 2), the zigzag pixel array being a pixel array rotated by 45 degrees with respect to a square pixel array having a pixel size of x 1; an output color image is generated by fusing the color image and the monochrome image.

Effects of the invention

The multi-camera system according to the embodiment of the present invention may employ a second camera having a sensitivity 3 to 4 times that of the color image sensor, and may also provide a fusion output realizing excellent low-illumination performance.

Drawings

FIG. 1 is a diagram of horizontal/vertical spatial sampling comparison;

FIG. 2 is a set of diagrams of a zig-zag pixel array process;

FIG. 3 is a diagram of an image processing pipeline;

FIG. 4 is a diagram of an exemplary prior art WRGB sensor;

FIG. 5 is a set of diagrams of an example of a prior art camera head for both monochrome and color;

FIG. 6 is a diagram of a comparison of a square/zig-zag pixel array;

FIG. 7 is a diagram of the use of a zig-zag pixel array having a pixel size of x 1.4;

in fig. 2 to 5, a portion having a diagonal line from left to right represents blue (B), a portion having a diagonal line from left to right represents red (R), and a dotted line portion represents green (G).

Detailed Description

Inventive example 1

According to an embodiment of the present invention, there is provided a multi-camera system including: a color image sensor for generating a color image; a monochrome filter image sensor for generating a monochrome image, wherein the monochrome filter image sensor has a sensor including a zigzag pixel array having a pixel size of x1.4 (or the square root of 2), the zigzag pixel array being a pixel array rotated by 45 degrees with respect to a square pixel array having a pixel size of x 1; an output color image is generated by fusing the color image and the monochrome image. Fig. 1 shows the spatial sampling time interval/frequency of a bayer colored filter sensor and a proposed monochrome filter image sensor (here an example of BW is used (we refer to the monochrome filter sensor as BW (black/white) filter sensor)).

The bayer colored filter sensor has a checkered pattern with a double density of green pixels and fewer red and blue pixels because the green information helps to enhance the luminance signal (Y signal). The output of the colored filter sensor is processed by an Image Signal Processor (ISP) and a red/green/blue (RGB) tristimulus plane is generated by interpolation (demosaicing) processing. Please refer to fig. 2.

Image Signal Processors (ISPs) for current camera products have good interpolation algorithms and can produce the same resolution in the horizontal and vertical line directions. Fig. 6 is a diagram of a comparison of a square/zigzag pixel array. The BW filter sensor has a zigzag pixel array as shown in fig. 6, which is a pixel array rotated 45 degrees with respect to a square pixel array or a standard array. If an appropriate interpolation is applied (if the same interpolation as the G (green) interpolation is applied), the horizontal and vertical (H/V) resolutions will be the same as those of the colored filter sensor even if 1.4 times-sized pixels are used (fig. 7). Fig. 7 is a diagram of the use of a zigzag pixel array with a pixel size of x 1.4. The symbol "x 1.4" is relative to the color sensor pixel. When the zigzag pixel array (45-degree rotation pixel array) is applied and the horizontal/vertical resolution of the color sensor having the pixel size of "x 1" is to be obtained, a pixel of 1.4 times the size may be used, as shown in fig. 7. When pixels of 1.4 times the size are used, the pixel area is twice as large, which means that the sensitivity is also twice. Furthermore, when the BW pixel is used, the sensitivity is 1.5 to 2 times.

The overall sensitivity is 3 to 4 times. This is a much greater improvement than other solutions.

An example of an overall image processing pipeline is shown in fig. 3.

Embodiment 2 of the present invention provides a multi-camera system according to an embodiment of the present invention. The multi-camera system includes a set of multiple cameras including multiple color image sensors, wherein the color image sensors are used to generate color images; the monochrome filter image sensor is used to generate a monochrome image. A monochrome filter image sensor has a sensor comprising a zigzag pixel array with a pixel size of x1.4 (or the square root of 2). The zigzag pixel array is a pixel array rotated 45 degrees with respect to a square pixel array having a pixel size of x1. An output color image is generated by fusing the plurality of color images and the monochrome image. The same interpolation is applied to the green pixels on the color image and the monochrome image. The sensitivity of the sensor may be 3 to 4 times.

Claims

1. A multi-camera system, comprising:

a color image sensor for generating a color image;

a monochrome filter image sensor for generating a monochrome image, wherein the monochrome filter image sensor has a sensor including a zigzag pixel array having a pixel size of a square root of x1.4 or 2, the zigzag pixel array being a pixel array rotated by 45 degrees with respect to a square pixel array having a pixel size of x 1;

an output color image is generated by fusing the color image and the monochrome image.

2. A multi-camera system, comprising:

a plurality of color image sensors for generating color images;

generating an output color image by fusing the color image and the monochrome image;

the same interpolation is applied to the green pixels on the color image and the monochrome image.

3. A camera module, comprising:

a multi-camera system according to claim 1 or 2.

4. A smart phone, comprising:

a camera module according to claim 3.

5. A mobile device, comprising:

a camera module according to claim 3.

6. An integrated circuit, comprising:

a multi-camera system according to claim 1 or 2.

7. A method of using a multi-camera system and a monochrome filter image sensor, wherein the multi-camera system comprises a color image sensor, wherein the monochrome filter image sensor has a sensor comprising a zigzag pixel array having a square root of a pixel size of x1.4 or 2, the zigzag pixel array being a pixel array rotated 45 degrees with respect to a square pixel array having a pixel size of x 1; the method comprises the following steps:

the color image sensor generates a color image;

the monochromatic filter image sensor generates a monochromatic image;

8. A method of using a multi-camera system and a monochrome filter image sensor, wherein the multi-camera system comprises a plurality of cameras including a plurality of color image sensors; the monochrome filter image sensor has a sensor comprising a zigzag pixel array having a square root of pixel size x1.4 or 2, the zigzag pixel array being a pixel array rotated by 45 degrees with respect to a square pixel array having a pixel size x 1; the method comprises the following steps:

the plurality of color image sensors respectively generate color images;

the monochromatic filter image sensor generates a monochromatic image;