CN105578006B - Imaging method, imaging device and electronic installation - Google Patents

Abstract

The invention discloses an imaging method, which includes: providing an image sensor, which includes a photosensitive pixel array, a filter and a filter control device, the filter includes an array of filter units, and the filter unit includes a plurality of filter pixels, The filter includes multiple filters, and the filter control device is used to switch the filter above the photosensitive pixel array so that the filter pixels covering the photosensitive pixels can be switched between color filter pixels and white filter pixels, and each filter unit The overlaid light-sensitive pixels constitute binning pixels. Next, control the filter control device according to the ambient brightness to switch the filter above the photosensitive pixel array to switch at least part of the filter pixels into color filter pixels, and process the photosensitive pixel output of the same combined pixel to obtain the pixel value of the combined pixel to generate Merge images. Using this imaging method, an appropriate number of white filter pixels can be switched out according to the ambient brightness to obtain an image with a high signal-to-noise ratio. The invention also discloses an imaging device and an electronic device.

Description

Imaging method, imaging device and electronic device

technical field

The invention relates to imaging technology, in particular to an imaging method, an imaging device and an electronic device.

Background technique

Images generated by existing imaging devices in high-ISO or low-illuminance environments may have shortcomings such as high noise and unclear. The imaging method of multi-frame synthesis is used to improve the image quality, and the photo taking time is too long. Embedding more white-filtered pixels improves image quality, but results in color distortion.

Contents of the invention

The present invention aims to solve at least one of the technical problems existing in the prior art. Therefore, the present invention needs to provide an imaging method, an imaging device and an electronic device.

The imaging method of the embodiment of the present invention comprises the following steps:

An image sensor is provided, the image sensor includes a photosensitive pixel array, a filter and a filter control device, the filter includes an array of filter units, the filter unit includes a plurality of filter pixels, and the filter Including multiple, the filter control device is used to switch the filter above the photosensitive pixel array so that the filter pixel covering the photosensitive pixel is between the color filter pixel and the white filter pixel Switching, the photosensitive pixels covered by each filter unit constitute binning pixels;

controlling the filter control device to switch the filter above the photosensitive pixel array according to ambient brightness to switch at least some of the filter pixels into color filter pixels; and

Reading the output of the light-sensitive pixel array, and processing the output of the light-sensitive pixel of the same combined pixel to obtain the pixel value of the combined pixel to generate a combined image.

By using this imaging method, the ratio of white filter pixels in the filter unit can be adjusted according to the brightness of the environment, so as to obtain images with higher signal-to-noise ratio, higher brightness and definition, and less noise under different illuminances. Certain disadvantages of existing imaging methods are overcome.

The present invention also provides an imaging device that can be used to implement the imaging method according to the embodiment of the present invention, which includes:

An image sensor, the image sensor includes:

photosensitive pixel array;

optical filters; and

A filter control device connected to the filter;

The filter includes a filter unit array, the filter unit includes a plurality of filter pixels, the filter includes a plurality of filters, and the filter control device is used to switch the photosensitive pixel array above the A filter allows the filter pixels covering the photosensitive pixels to switch between color filter pixels and white filter pixels, and the photosensitive pixels covered by each filter unit form a combined pixel;

The filter control device is used to control and switch the filter above the photosensitive pixel array according to the ambient brightness, so as to switch at least some of the filter pixels into color filter pixels;

The imaging device further includes an image processing module, the image processing module is used to read the output of the photosensitive pixel array, and process the output of the photosensitive pixel of the same binning pixel to obtain the pixel value of the binning pixel A merged image is thereby generated.

The present invention also provides an electronic device including the imaging device of the embodiment of the present invention.

Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and comprehensible from the description of the embodiments in conjunction with the following drawings, wherein:

FIG. 1 is a schematic flowchart of an imaging method according to an embodiment of the present invention.

FIG. 2 is a schematic flowchart of the control steps of the imaging method according to the embodiment of the present invention.

FIG. 3 is a schematic flowchart of the control steps of the imaging method according to the embodiment of the present invention.

FIG. 4 is a schematic flowchart of the control steps of the imaging method according to the embodiment of the present invention.

FIG. 5 is a schematic flowchart of the control steps of the imaging method according to the embodiment of the present invention.

FIG. 6 is a schematic flowchart of the control steps of the imaging method according to the embodiment of the present invention.

FIG. 7 is a schematic flowchart of the control steps of the imaging method according to the embodiment of the present invention.

FIG. 8 is a schematic flowchart of the reading steps of the imaging method according to the embodiment of the present invention.

FIG. 9 is a schematic flowchart of the reading steps of the imaging method according to the embodiment of the present invention.

FIG. 10 is a schematic flowchart of the reading steps of the imaging method according to the embodiment of the present invention.

FIG. 11 is a schematic structural diagram of an imaging device according to an embodiment of the present invention.

FIG. 12 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

Fig. 13 is a schematic diagram of a Bayer array.

FIG. 14 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

FIG. 15 is a schematic diagram of functional modules of an imaging device according to an embodiment of the present invention.

FIG. 16 is a schematic diagram of photosensitive pixels and related circuits of an imaging device according to an embodiment of the present invention.

FIG. 17 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

FIG. 18 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

FIG. 19 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

FIG. 20 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

FIG. 21 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

FIG. 22 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

FIG. 23 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

FIG. 24 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

FIG. 25 is a schematic diagram of an array of filter units of an imaging device according to an embodiment of the present invention.

FIG. 26 is a schematic diagram of functional modules of an imaging device according to an embodiment of the present invention.

FIG. 27 is a schematic three-dimensional structure diagram of an image sensor of an imaging device according to an embodiment of the present invention.

FIG. 28 is a schematic structural diagram of an image sensor of an imaging device according to an embodiment of the present invention.

FIG. 29 is a superimposed schematic diagram of a multi-layer optical filter according to an embodiment of the present invention.

FIG. 30 is a schematic diagram of functional modules of an electronic device according to an embodiment of the present invention.

FIG. 31 is a schematic diagram of functional modules of an electronic device according to an embodiment of the present invention.

Detailed ways

Embodiments of embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary, are only for explaining the embodiments of the present invention, and should not be construed as limiting the embodiments of the present invention.

The imaging method, imaging device and electronic device proposed by the embodiments of the present invention are described below with reference to the accompanying drawings.

Please refer to Fig. 1, the imaging method of the embodiment of the present invention comprises the following steps:

S1, the image sensor includes a photosensitive pixel array, a filter and a filter control device, the filter includes an array of filter units, the filter unit includes a plurality of filter pixels, the filter includes a plurality of filters, and the filter control device is used for The light filter above the light-sensitive pixel array is switched to switch the light-sensitive pixel covering the light-sensitive pixel between the color filter pixel and the white light-filter pixel, and the light-sensitive pixel covered by each light-sensing pixel constitutes a binning pixel.

S2. Control the filter control device to switch the filter above the photosensitive pixel array according to the ambient brightness to switch at least some of the filter pixels into color filter pixels.

S3, reading the output of the light-sensitive pixel array, and processing the output of the light-sensitive pixel of the same combined pixel to obtain the pixel value of the combined pixel to generate a combined image.

The color filter area of the filter unit in the imaging method according to the embodiment of the present invention is used to obtain the color information of the merged pixel, and the white filter pixel is used to obtain the brightness information of the merged pixel under low illumination, and the brightness information has less noise. In this way, the pixel values of the merged image include both color information and low-noise brightness information, and the merged image has complete color, better brightness and definition, and less noise. White filter pixels and color filter pixels can be switched to each other according to the needs of the environment, such as switching under low light to generate more white filter pixels to improve image quality.

In some embodiments, the array of filter elements includes a Bayer array. Using the Bayer array can get a full and lifelike image.

In some embodiments, each filter unit covers 2*2 photosensitive pixels.

Please refer to FIG. 2, in the imaging method of this embodiment, step S2 further includes:

S201. Sensing ambient brightness.

S203, judging whether the ambient brightness is smaller than a first predetermined threshold.

S205. When the ambient brightness is less than the first predetermined threshold, control the filter control device to switch the filter above the photosensitive pixel array to switch three or two filter pixels of the same filter unit into white filter pixels.

In this embodiment, the output of the photosensitive pixels can be used as an index for judging the brightness of the environment. Specifically, the average value of the outputs of all or part of the photosensitive pixels can be calculated, and then the average value can be correspondingly switched to illuminance. Thresholds can be set to implement corresponding controls. Set a first predetermined threshold, such as 20 Lux, and control to switch three or two of each filter unit to white filter pixels when the light is less than this value.

Please refer to FIG. 3, in the imaging method of this embodiment, step S2 further includes:

S207, judging whether the ambient brightness is greater than a first predetermined threshold but less than a second predetermined threshold.

S209, when the ambient brightness is greater than the first predetermined threshold and less than the second predetermined threshold, control the filter control device to switch the filter above the photosensitive pixel array to filter all the filters of the first green filter unit and the second green filter unit Light pixels are switched to white filter pixels.

Please refer to FIG. 4, in the imaging method of this embodiment, the filter unit includes a first green filter unit and a second green filter unit, and step S2 further includes:

S211. Determine whether the ambient brightness is greater than a second predetermined threshold and less than a third predetermined threshold.

S213, when the ambient brightness is greater than the second predetermined threshold and less than the third predetermined threshold, control the filter control device to switch the filter above the photosensitive pixel array to switch all the filter pixels of the first green filter unit to white filter pixels , or switch two or three filter pixels of the first green filter unit and the second green filter unit to white filter pixels.

In this embodiment, the third predetermined threshold may be set to 100 Lux. Therefore, when the ambient brightness is 50-100 Lux, some of the filter pixels replacing the green filter unit are white filter pixels.

Please refer to FIG. 5, in the imaging method of this embodiment, step S2 further includes:

S215. Determine whether the ambient brightness is greater than a third predetermined threshold and less than a fourth predetermined threshold.

S217, when the ambient brightness is greater than the third predetermined threshold and less than the fourth predetermined threshold, control the filter control device to switch the optical filter above the photosensitive pixel array so that the first green filter unit and/or the second green filter unit One filter pixel is switched to a white filter pixel.

The fourth predetermined threshold may be 200 Lux.

Please refer to FIG. 6, in the imaging method of this embodiment, step S2 further includes:

S219. Determine whether the ambient brightness is greater than a fourth predetermined threshold.

S221, when the ambient brightness is greater than the fourth predetermined threshold, control the filter control device to switch the filter above the photosensitive pixel array to switch all filter pixels into color filter pixels.

Please refer to FIG. 7, in another embodiment of the imaging method, step S2 further includes:

S223. Determine whether the ambient brightness is greater than a first predetermined threshold and less than a fifth predetermined threshold.

S225, when the ambient brightness is greater than the first predetermined threshold and less than the fifth predetermined threshold, control the filter control device to switch the filter above the photosensitive pixel array to switch one filter pixel of the same filter unit to a white filter pixel. In the imaging method of another embodiment, switching is not only performed in the green filter unit, but is switched simultaneously in all photosensitive units. For example, the fifth predetermined threshold may be 100 Lux. Therefore, when the ambient brightness is 20-100 Lux, one of the filter pixels in the filter unit is switched to a white filter pixel to increase image brightness and reduce noise.

Please refer to FIG. 8, in some embodiments of the imaging method, step S3 further includes:

S301. Calculate an average value or a sum of outputs of photosensitive pixels of the same binning pixel to obtain a pixel value of the binning pixel.

In the imaging method of the embodiment of the present invention, it is assumed that the output of each original photosensitive pixel is S, the noise is N, and the merged pixel includes photosensitive pixels, then the pixel value of the merged pixel is n*m*S, and the noise of the merged pixel is In the case of n=2, m=2, the noise of the synthesized pixel is about N/2. Therefore, the brightness of the merged image is improved in a low-light environment, and the signal-to-noise ratio is improved.

Please refer to FIG. 9, in the imaging method of this embodiment, step S3 further includes:

S303. Perform color restoration processing on the merged image.

Because the white filter pixels are mixed into the filter unit, the color of the generated merged image will be distorted to some extent, and the color restoration process can be performed on it to restore its true color as much as possible.

Referring to FIG. 10 , in some embodiments, the binned pixels include color pixel values and white pixel values. Step S3 further includes:

S305. Calculate the average value or sum of outputs of the photosensitive pixels covered by the color filter pixels and use it as a color pixel value.

S307, replace the luminance component of the color pixel value with the luminance component of the white pixel value as the pixel value of the merged pixel.

In this way, after obtaining the color pixel value, it can be converted into YUV format, and its brightness component can be obtained, and the brightness component of the color pixel value of the YUV color image can be replaced by the brightness component of the white pixel value, so as to generate complete color information and brightness at the same time and images with a high signal-to-noise ratio.

The imaging method of the embodiment of the present invention can be realized by the imaging device of the embodiment of the present invention.

Referring to FIG. 11 and FIG. 12 , an imaging device 100 according to an embodiment of the present invention includes an image sensor 10 and an image processing module 50 . The image sensor 10 includes a photosensitive pixel array 11 , a filter 13 and a filter control device 15 . The filter 13 includes a filter unit array 131, each filter unit 1311 includes a plurality of filter pixels 1312, each filter pixel 1312 covers a photosensitive pixel 111, and the filter pixels 1312 include a color filter pixel 1315 and a white filter pixel. The light pixel 1313, the filter control device 15 is used to control and switch the filter 13 above the photosensitive pixel array 11 so that the filter pixel 1312 covering the photosensitive pixel 111 is switched between the color filter pixel 1315 and the white filter pixel 1313, The photosensitive pixels 111 covered by each filter unit 1311 form binning pixels. The external light irradiates the photosensitive part 1111 of the photosensitive pixel 111 through the filter 13 to generate an electrical signal, that is, the output of the photosensitive pixel 111 .

The filter control device 15 is used to control and switch the filter 13 above the photosensitive pixel array 11 according to the ambient brightness to switch at least part of the filter pixels 1312 into color filter pixels 1315 . The imaging device 100 further includes an image processing module 50 for reading the output of the photosensitive pixel array 11 and processing the output of the photosensitive pixel 111 of the same binning pixel to obtain the pixel value of the binning pixel to generate a binning image.

An illumination sensor may be provided in the imaging device to sense ambient brightness. The illumination sensor includes a photoelectric conversion module capable of converting illumination intensity into an electrical signal, thereby measuring the ambient brightness. Alternatively, the output of the photosensitive pixels can be used as an indicator for judging the ambient brightness. Specifically, the average value of the outputs of all or part of the photosensitive pixels can be calculated as data for measuring the ambient brightness. Dividing the ambient brightness into different ranges corresponds to different control operations of the light filtering control device 15 .

The operation of the filter control device 15 to switch filters under different ambient brightness can be automatically controlled by the illumination sensor or the image processing module 50 according to the sensed ambient brightness, or manually controlled.

The filter control device 15 may include a filter switcher and a control circuit. A plurality of optical filters 13 are built in the optical filter switcher, and the control circuit is used to control the optical filter switcher to switch the optical filters 13 .

The color filter pixel 1315 of the filter unit 1311 in the imaging device according to the embodiment of the present invention is used to obtain the color information of the merged pixel, and the white filter pixel 1313 is used to obtain the brightness information of the merged pixel under low illumination and the brightness information is noisy. less. In this way, the pixel values of the merged image include both color information and low-noise brightness information, and the merged image has complete color, better brightness and definition, and less noise. The white filter pixels 1313 and the color filter pixels 1315 can be switched to each other according to the needs of the environment, such as switching under low light conditions to generate more white filter pixels to improve image quality.

Please refer to FIG. 13 , in this implementation manner, the filter element array includes a Bayer pattern.

Since the human eye is most sensitive to green light, the scene that the eyes usually see is equivalent to the result that the green scene is enhanced. The imaging device has to over-sensitize the green light so that the resulting image can be perceived as normal by the human eye. Therefore, the Bayer array or other common filter unit arrays generally have more green filter units, and the number ratio of green (G), red (R), and blue (B) filter units in the Bayer array is 2: 1:1, that is, every two green filter units, one red filter unit and one blue filter unit constitute the filter structure 1317 and are used to form image pixels with complete colors. Therefore, the Bayer array can be used to obtain images with full and vivid colors.

Moreover, the Bayer structure can be used to process the image signal using the traditional algorithm for the Bayer structure, so there is no need for major adjustments in the hardware structure.

Please refer to FIG. 14 and FIG. 27 together. In this embodiment, each filter unit 1311 covers 2*2 photosensitive pixels. They are green, red, blue, and green filter units respectively.

In a traditional filter unit array structure, each filter unit corresponds to a photosensitive pixel and an image pixel. Please refer to FIG. 16. In this embodiment, the filter unit array 131 adopts a Bayer structure, including filter structures 1317, and each filter structure 1317 includes green, red, blue, and green filter units 1311. Unlike traditional Bayer The structure of the filter unit array is different in that each filter unit 1311 corresponds to a plurality of photosensitive pixels 111 .

In addition to the 2*2 structure, there are also 3*3, 4*4, and even any n*m structures (n, m are natural numbers). It can be understood that the number of photosensitive pixels 111 that can be arranged on the photosensitive pixel array 11 is Limited, if there are too many photosensitive pixels 111 included in each merged pixel, the resolution of the image will be limited. For example, if the pixel value of the photosensitive pixel array 11 is 16M, a 2*2 merged pixel structure will be used to obtain resolution A merged image with a rate of 4M, and a merged image with a resolution of 1M can only be obtained with a 4*4 structure. Therefore, the 2*2 binning pixel structure is a better arrangement to improve image brightness and clarity while sacrificing resolution as little as possible. At the same time, the 2*2 structure is adopted to facilitate the reading and merging of the output of the photosensitive pixels on the hardware. Please refer to FIG. 15 , in some embodiments, the image sensor further includes a control module 17 for controlling the row-by-row exposure of the photosensitive pixel array 11 .

The control module 17 is connected to a row selection logic unit 171 and a column selection logic unit 173 to control the output of the photosensitive pixels 111 to be processed row by row. The method of progressive exposure and output is easier to implement on hardware.

Please refer to FIG. 15 together. In this embodiment, the image sensor 10 may further include a register 19, and the control module 17 is used to sequentially collect the outputs of the photosensitive pixels 111 of the kth row and the k+1th row that are currently exposed. into the register 19, wherein k=2n-1, n is a natural number, and k+1 is less than or equal to the total number of rows of the photosensitive pixel array 11.

Specifically, referring to FIG. 16 and FIG. 15 , the image sensor 10 includes a control module 17 connected to a row selection logic unit 171 and a column selection logic unit 173 . The row selection logic unit 171 and the column selection logic unit 173 are connected to the switch transistor 1115 corresponding to each photosensitive pixel 111, and the control module 17 is used to control the row selection logic unit 171 and the column selection logic unit 173 to gate the photosensitive pixel 111 at a specific position The switching tube 1115.

Please refer to FIG. 17 and FIG. 18 , in some embodiments, the light filtering control device 15 is used for sensing and judging the ambient brightness, and for switching the light filtering above the photosensitive pixel array 11 when the ambient brightness is less than the first predetermined threshold. slice 13 to switch three or two of the filter pixels 1312 into white filter pixels 1313 .

For the measured ambient brightness, a threshold can be set to perform corresponding control. Set a first predetermined threshold, such as 20Lux, and control the filter unit to switch three or two of each filter unit to white filter pixels when the light is less than this value.

Since the first predetermined threshold is a very low ambient brightness value, more white filter pixels need to be switched out to allow more light to enter, thereby generating an image with a higher brightness value and a higher signal-to-noise ratio. However, in order to retain certain information of each color, all pixels cannot be replaced with white filter pixels, so that the filter units of each color only switch part of the filter pixels.

Further, please refer to FIG. 19 , in this embodiment, the filter unit 1311 includes a first green filter unit 1314 and a second green filter unit 1316 . The light filter control device 15 is also used to switch the light filter 13 above the photosensitive pixel array 11 to switch the first green filter unit 1314 and the second green filter unit 1316 when the ambient brightness is greater than the first predetermined threshold and less than the second predetermined threshold. All filter pixels of 1313 are switched to white filter pixels.

If the second predetermined threshold is set, for example, 50 Lux, in this embodiment, when the ambient brightness is 20-50 Lux, all the filter pixels of the green filter unit are switched to white filter pixels. Since the human eye is most sensitive to green light, the imaging device itself must be overexposed to produce an image that looks normal to the human eye. The overexposure of the green filter unit results in less intense color distortion to the human eye than the red or blue filter unit. Therefore, in the embodiments of the present invention, it is often used to switch the filter pixels of the green filter unit to white filter pixels, so as to increase the amount of incoming light and reduce imaging noise.

Please refer to FIG. 20 and FIG. 21. In this embodiment, the filter control device 15 is also used to switch the filter 13 above the photosensitive pixel array 11 when the ambient brightness is greater than the second predetermined threshold and less than the third predetermined threshold to All the filter pixels of the first green filter unit 1314 are switched to white filter pixels 1313, or the filter 13 above the photosensitive pixel array 11 is switched to convert the first green filter unit 1314 and the second green filter unit 1316 Two or three filter pixels are switched to white filter pixels 1313 .

In this embodiment, the third predetermined threshold may be set to 100 Lux. Therefore, when the ambient brightness is 50-100 Lux, some of the filter pixels replacing the green filter unit are white filter pixels.

Please refer to FIG. 22 and FIG. 23. In this embodiment, the filter control device 15 is also used to switch the filter 13 above the photosensitive pixel array 11 when the ambient brightness is greater than the third predetermined threshold and less than the fourth predetermined threshold to One filter pixel of the first green filter unit 1314 and/or the second green filter unit 1316 is switched to a white filter pixel 1313 .

In this embodiment, the fourth predetermined threshold may be set to 200 Lux. Therefore, when the ambient brightness is 100-200 Lux, each green filter unit switches out a white filter pixel 1313, or one of the first green filter unit 1314 and the second green filter unit 1316 switches out a white filter pixel. Light pixels 1313 . The image obtained in this way retains relatively complete color information.

Please refer to FIG. 24 , in other embodiments, the filter control device 15 is also used to switch the filter 13 above the photosensitive pixel array 11 when the ambient brightness is greater than the first predetermined threshold and less than the fifth predetermined threshold, so that all filter units A filter pixel at 1311 is switched to a white filter pixel 1313 .

In other embodiments, switching is not only performed in the green filter unit, but in all photosensitive units at the same time. For example, the fifth predetermined threshold may be 100 Lux. Therefore, when the ambient brightness is 20-100 Lux, one of the filter pixels in the filter unit is switched to a white filter pixel to increase image brightness and reduce noise.

Further, please refer to FIG. 25. In other embodiments, the image processing module is also used to switch the filter 13 above the photosensitive pixel array 11 to switch all filter pixels to color filters when the ambient brightness is greater than the fourth predetermined threshold. Light pixels 1315 .

This results in no color loss in the resulting merged image since there are no white filter pixels.

In some embodiments, the image processing module is used to calculate the average value or the sum of the outputs of the photosensitive pixels of the same binning pixel to obtain the pixel value of the binning pixel.

In the imaging method of the embodiment of the present invention, it is assumed that the output of each original photosensitive pixel is S, the noise is N, and the merged pixel includes photosensitive pixels, then the pixel value of the merged pixel is n*m*S, and the noise of the merged pixel is In the case of n=2, m=2, the noise of the synthesized pixel is about N/2. Therefore, the brightness of the merged image is improved in a low-light environment, and the signal-to-noise ratio is improved.

In this embodiment, the image processing module is used to perform color restoration processing on the merged image.

Because the white filter pixels are mixed into the filter unit, the color of the generated merged image will be distorted to some extent, and the color restoration process can be performed on it to restore its true color as much as possible. According to the number of white filter pixels switched by the filter unit, the corresponding software algorithm can be set to restore the color.

In addition to summing or averaging the photosensitive pixel outputs of the merged pixels, in other embodiments, the merged pixel includes color pixel values and white pixel values, and the image processing module is used to calculate the output of the photosensitive pixels covered by the color filter pixels The average value or the sum is used as the color pixel value, and the brightness component used to replace the color pixel value with the brightness component of the white pixel value is used as the pixel value of the merged pixel.

In this way, after obtaining the color pixel value, it can be converted to YUV format and its brightness component can be obtained, and the brightness component of the color pixel value can be replaced by the brightness component of the white pixel value to generate complete color information, and at the same time brightness and signal-to-noise comparison High merged image. Compared with the implementation of calculating the sum or average value of the outputs of the photosensitive pixels of the merging pixels, the merging image obtained by these other implementations has less color loss, and generally does not need color restoration processing. YUV is an image format that describes color according to the principle of brightness and color difference. The YUV format includes many specific formats, such as YUV422, YUV420, etc. In some embodiments, the color combined pixels include green, red, and blue combined pixels, and the image processing module first obtains a color combined image in RGB format according to the pixel values of the color combined pixels, that is, obtains the red brightness value of each image pixel, The green luminance value and the blue luminance value are represented by R, G, and B respectively, and in some embodiments, the corresponding luminance value of the image pixel in YUV format is Y=0.299*R+0.587*G+0.114*B.

Referring to FIG. 26 and FIG. 16 , in some embodiments, the image sensor 10 includes an array of analog-to-digital converters 21 , and each photosensitive pixel 111 is connected to one analog-to-digital converter 211 . The analog-to-digital converter 211 is used to convert the analog signal output of the photosensitive pixel 111 into a digital signal output.

Please also refer to FIG. 16 , the photosensitive pixel 111 in this embodiment includes a photodiode 1113 . The photodiode 1113 is used to convert light into electric charges, and the electric charges generated are proportional to the light intensity. The switch tube 1115 is used to control the turn-on and turn-off of the circuit according to the control signals of the row selection logic unit 171 and the column selection logic unit 173. When the circuit is turned on, the source follower 1117 (source follower) is used to connect the photodiode 1113 The charge signal generated by light is converted into a voltage signal. The analog-to-digital converter 211 (Analog-to-digital converter) is used to switch the voltage signal into a digital signal for transmission to subsequent circuits for processing.

This output processing method converts the output of the photosensitive pixel into a digital signal, which is processed by software in the subsequent digital circuit or in the chip. Therefore, the output information of each photosensitive pixel can be retained. For example, for an image sensor with 16M pixels, the imaging method in the embodiment of the present invention can retain the information of 16M pixels (that is, the image before merging), and on this basis, the obtained Merged images of 4M pixels or images of other resolutions. The final generated image has a lower probability of bad pixels. In addition, this output processing method is less noisy and has a higher signal-to-noise ratio.

Referring to FIG. 27 and FIG. 28 , in some embodiments, the image sensor 10 includes a micromirror array 23 disposed on the filter 13 , and each micromirror 231 corresponds to one photosensitive pixel 111 .

Specifically, each micromirror 231 corresponds to one photosensitive pixel 111 , including size and position correspondence. In some embodiments, each filter unit 1311 corresponds to 2*2 photosensitive pixels 111 and 2*2 micromirrors 191 . With the development of technology, in order to obtain images with higher resolution, there are more and more photosensitive pixels 111 on the photosensitive sheet, and the arrangement becomes denser, and the single photosensitive pixel 111 is also smaller and smaller, which is affected by light, and the photosensitive pixels 111 The area of the photosensitive part 1111 is limited, and the micromirror 191 can gather light to the photosensitive part 1111, thereby increasing the intensity of light received by the photosensitive pixel 111 to improve image quality.

The filter control device 15 switches the filter 13 to switch the white filter pixels 1313 and the color filter pixels 1315 mainly in the following two ways.

Please refer to FIG. 29 , in some embodiments, the filter control device 15 is used to switch the color filter pixels 1315 to white filter pixels 1313 by increasing the filter 13 above the photosensitive pixel array 11, and by reducing the coverage The filter 13 of the photosensitive pixel array 11 is used to switch the white filter pixels 1313 into color filter pixels 1315 .

Specifically, a plurality of filters 13 are stacked, and the first green filter unit 1314 and the second green filter unit 1316 on the bottommost first layer are both white filter pixels 1313 . The second layer, the third layer, the fourth layer, etc. respectively include green filter pixels located in different positions of the green filter unit, and other positions are transparent filter pixels (that is, pixels that do not block light of any color).

It can be understood that by adding the filters 13 layer by layer, the number of green filter pixels increases and the number of white filter pixels 1313 decreases; the number of filters decreases layer by layer, the number of green filter pixels decreases and the number of white filter pixels increases. For example, by adding the second layer of filters 13 and the third layer of filters 13 on the basis of the first layer of filters 13, the filter unit array 131 in the embodiment corresponding to FIG. 21 can be obtained. Then, the setting of the filter 13 is not limited thereto, and other types of filters 13 can also be set, such as the filter 13 that each red filter unit includes two red filter pixels and two transparent pixels, To achieve more combinations of filters 13.

In other embodiments, the filter control device 15 is used to switch the filter pixels between the color filter pixels 1315 and the white filter pixels 1313 by replacing the filter 13 above the photosensitive pixel array 11 .

Specifically, there are multiple filters 13, but they are not overlapped. Each filter 13 includes filter units 1311 of various colors, which can filter light independently and generate images with relatively complete colors. For example, the first to fourth layers of filters 13 respectively include filter unit arrays 131 as shown in FIG. 19 , FIG. 20 , FIG. 22 , and FIG. 25 . By sequentially switching from the first layer to the fourth layer, the amount of incoming light can be reduced, corresponding to different situations of ambient brightness from low to high.

To sum up, the imaging device in the embodiment of the present invention can switch out a corresponding number of white filter pixels according to the ambient brightness, so that the pixel value of the merged pixel contains both color information and low-noise brightness information, so that subsequent circuits can generate color integrity. Combined images with high signal-to-noise ratio.

The invention also provides an electronic device using the imaging device. In some embodiments, the electronic device includes an imaging device. Therefore, the electronic device has a camera function and can generate a combined image with complete color, high signal-to-noise ratio, and high definition under low illumination.

The electronic device may be a mobile phone.

In some embodiments, the imaging device may be a front-facing camera of a mobile phone. Since the front-facing camera is mostly used for self-portraits, and self-portraits generally require image clarity but not high image resolution, the electronic device adopting this embodiment can meet this requirement.

Please refer to FIG. 30 , in some embodiments, the electronic device 200 includes a central processing unit 81 connected to the imaging device 100 and an external memory 83 , and the central processing unit 81 is used to control the external memory 83 to store the merged image.

In this way, the resulting merged image can be stored for later viewing, use or transfer. The external memory 83 includes an SM (Smart Media) card, a CF (Compact Flash) card, and the like.

Referring to FIG. 31 , in some embodiments, the electronic device 200 further includes a central processing unit 81 connected to the imaging device 100 and a display device 85 , and the central processing unit 81 is used to control the display device 85 to display the merged image. In this way, the image captured by the electronic device 200 can be displayed on the display device for the user to view. The display device includes an LED display and the like.

To sum up, the electronic device adopting the embodiment of the present invention has a camera function and can generate a combined image with complete color, high signal-to-noise ratio, and high definition under low illumination. In particular, when the electronic device is a front camera of a mobile phone, it can improve the brightness and definition of the selfie image under low illumination and reduce the noise.

For the unexpanded parts of the imaging method and the electronic device in the embodiments of the present invention, reference may be made to the corresponding parts of the image sensor and the imaging device in the above embodiments, and will not be elaborated here.

For the unexpanded parts of the imaging method and the electronic device in the embodiments of the present invention, reference may be made to the corresponding parts of the image sensor and the imaging device in the above embodiments, and will not be elaborated here.

In the description of this specification, descriptions with reference to the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific examples" or "some examples" mean that a combination of the embodiments or An example describes a particular feature, structure, material, or characteristic that is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including substantially concurrently or in reverse order depending on the functions involved, which shall It is understood by those skilled in the art to which the embodiments of the present invention pertain.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium, For use with instruction execution systems, devices, or devices (such as computer-based systems, systems including processors, or other systems that can fetch instructions from instruction execution systems, devices, or devices and execute instructions), or in conjunction with these instruction execution systems, devices or equipment used. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, since the program can be read, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable processing if necessary. The program is processed electronically and stored in computer memory.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, Programmable Gate Arrays (PGAs), Field Programmable Gate Arrays (FPGAs), etc.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. During execution, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (29)

1. an imaging method, is characterized in that, comprises the following steps: The image sensor providing step: provide an image sensor, the image sensor includes a photosensitive pixel array, a filter and a filter control device, the filter includes an array of filter units, and the filter unit includes a plurality of filter pixels, The filter includes a plurality of filters, and the filter control device is used to switch the filter above the photosensitive pixel array so that the filter pixels covering the photosensitive pixels are between the color filter pixel and the white filter pixel. Switching between filter pixels, each of the filter units covers 2*2 of the photosensitive pixels, and the photosensitive pixels covered by each of the filter units form a merged pixel; Control step: controlling the filter control device to switch the filter above the photosensitive pixel array according to the ambient brightness to switch at least some of the filter pixels into color filter pixels, the control step is by increasing the the filter above the photosensitive pixel array to switch the color filter pixel to the white filter pixel, and reduce the color filter to cover the photosensitive pixel array to switch the white filter pixel pixel switching to the color filter pixel, or the control step is by replacing the filter above the photosensitive pixel array so that the filter pixel is between the color filter pixel and the white filter pixel switch between pixels; and Reading step: reading the output of the light-sensitive pixel array, and processing the output of the light-sensitive pixel of the same combined pixel to obtain the pixel value of the combined pixel to generate a combined image. 2. The imaging method according to claim 1, wherein the filter unit array comprises a Bayer (Bayer) array. 3. The imaging method according to claim 1, wherein the controlling step further comprises: Sensing ambient brightness; judging whether the ambient brightness is less than a first predetermined threshold; and When the ambient brightness is less than the first predetermined threshold, the filter control device is controlled to switch the filter above the photosensitive pixel array so that three or two of the same filter unit The filter pixels are switched to the white filter pixels. 4. The imaging method according to claim 3, wherein the filter unit comprises a first green filter unit and a second green filter unit, and the controlling step further comprises: judging whether the ambient brightness is greater than the first predetermined threshold and less than a second predetermined threshold; and When the ambient brightness is greater than the first predetermined threshold and less than the second predetermined threshold, the filter control device is controlled to switch the filter above the photosensitive pixel array so that the first green filter All the filter pixels of the light unit and the second green filter unit are switched to the white filter pixels. 5. The imaging method according to claim 4, wherein the controlling step further comprises: judging whether the ambient brightness is greater than the second predetermined threshold and less than a third predetermined threshold; and When the ambient brightness is greater than the second predetermined threshold and less than the third predetermined threshold, control the filter control device to switch the filter above the photosensitive pixel array to filter the first green color All the filter pixels of the unit are switched to the white filter pixels, or two or three of the filter pixels of the first green filter unit and the second green filter unit are switched to The white filter pixels. 6. The imaging method according to claim 5, wherein the controlling step further comprises: judging whether the ambient brightness is greater than the third predetermined threshold and less than a fourth predetermined threshold; and When the ambient brightness is greater than the third predetermined threshold and less than the fourth predetermined threshold, the filter control device is controlled to switch the filter above the photosensitive pixel array so that the first green filter unit and the first green filter unit and /or one of the filter pixels of the second green filter unit is switched to the white filter pixel. 7. The imaging method according to claim 6, wherein the controlling step further comprises: judging whether the ambient brightness is greater than the fourth predetermined threshold; and When the ambient brightness is greater than the fourth predetermined threshold, the filter control device is controlled to switch the filter above the photosensitive pixel array to switch all the filter pixels into the color filter pixels. 8. The imaging method according to claim 3, wherein the controlling step further comprises: judging whether the ambient brightness is greater than the first predetermined threshold and less than a fifth predetermined threshold; and When the ambient brightness is greater than the first predetermined threshold but less than the fifth predetermined threshold, the filter control device is controlled to switch the optical filter above the photosensitive pixel array to use one of the same optical filter units The filter pixels are switched to the white filter pixels. 9. The imaging method according to any one of claims 1-8, wherein the reading step further comprises: calculating the average value or the sum of the outputs of the photosensitive pixels of the same binning pixel to obtain the pixel value of the binning pixel. 10. The imaging method according to claim 9, said reading step further comprising: Perform color restoration processing on the merged image. 11. The imaging method according to any one of claims 1-8, wherein the combined pixels comprise color pixel values and white pixel values; the reading step comprises: calculating the average or sum of the outputs of the photosensitive pixels covered by the color filter pixels as the color pixel value; and replacing the luminance component of the color pixel value with the luminance component of the white pixel value as the pixel value of the merged pixel. 12. An imaging device, characterized in that, comprising: An image sensor, the image sensor includes: photosensitive pixel array; optical filters; and A filter control device connected to the filter; The filter includes a filter unit array, the filter unit includes a plurality of filter pixels, the filter includes a plurality of filters, and the filter control device is used to switch the photosensitive pixel array above the The filter allows the filter pixels covering the photosensitive pixels to switch between color filter pixels and white filter pixels, each of the filter units covers 2*2 of the photosensitive pixels, and each of the photosensitive pixels The photosensitive pixels covered by the filter unit form a combined pixel; The filter control device is used to control and switch the filter above the photosensitive pixel array according to the ambient brightness to switch at least some of the filter pixels into color filter pixels, and the filter control device is used to pass adding the filter above the photosensitive pixel array to switch the color filter pixel to the white filter pixel, and reducing the filter covering the photosensitive pixel array to switch the white The filter pixels are switched to the color filter pixels, or the filter control device is used to replace the filter above the photosensitive pixel array to make the filter pixels between the color filter pixels and the color filter pixels switching between the white filter pixels; The imaging device further includes an image processing module, the image processing module is used to read the output of the photosensitive pixel array, and process the output of the photosensitive pixel of the same binning pixel to obtain the pixel value of the binning pixel A merged image is thereby generated. 13. The imaging device according to claim 12, wherein the filter element array comprises a Bayer array. 14. The imaging device according to claim 12, wherein the image processing module filter control device is used to switch the filter above the photosensitive pixel array when the ambient brightness is less than a first predetermined threshold slice to switch three or two of the filter pixels in the same filter unit to the white filter pixels. 15. The imaging device according to claim 14, wherein the filter unit comprises a first green filter unit and a second green filter unit; The filter control device is also used to switch the filter above the photosensitive pixel array to switch the first green filter unit and the All the filter pixels of the second green filter unit are switched to the white filter pixels. 16. The imaging device according to claim 15, wherein the filter control device is further configured to switch the photosensitive pixel array when the ambient brightness is greater than the second predetermined threshold and less than a third predetermined threshold The upper filter is used to switch all the filter pixels of the first green filter unit to the white filter pixels, or to switch the first green filter unit and the second Two or three of the filter pixels of the green filter unit are switched to the white filter pixels. 17. The imaging device according to claim 16, wherein the filter control device is further configured to switch the photosensitive pixel array when the ambient brightness is greater than the third predetermined threshold and less than a fourth predetermined threshold The upper filter is used to switch one filter pixel of the first green filter unit and/or the second green filter unit to the white filter pixel. 18. The imaging device according to claim 17, wherein the filter control device is further configured to switch the filter above the photosensitive pixel array when the ambient brightness is greater than the fourth predetermined threshold slice to switch all the filter pixels to the color filter pixels. 19. The imaging device according to claim 14, wherein the filter control device is further configured to switch the light above the photosensitive pixel array when the ambient brightness is greater than the first predetermined threshold and less than the fifth predetermined threshold. the filter to switch all the filter pixels to the color filter pixels. 20. The imaging device according to claim 12, wherein the image processing module is used to calculate the average value or the sum of the outputs of the photosensitive pixels of the same merged pixel to obtain the pixel value of the merged pixel . 21. The imaging device according to claim 20, wherein the image processing module is configured to perform color restoration processing on the merged image. 22. The imaging device of claim 12, wherein the combined pixel comprises a color pixel value and a white pixel value; The image processing module is used to calculate the average or sum of the outputs of the photosensitive pixels covered by the color filter pixels as the color pixel value, and the luminance component used to replace the color pixel value is the The luminance component of the white pixel value is then used as the pixel value of the merged pixel. 23. The imaging device according to claim 12, wherein the image sensor comprises an analog-to-digital converter, and each of the photosensitive pixels is respectively connected to one of the analog-to-digital converters. 24. The imaging device according to claim 12, wherein the image sensor comprises a micromirror array disposed on the filter, each of the micromirrors corresponds to one of the photosensitive pixels. 25. An electronic device, comprising the imaging device according to any one of claims 12-24. 26. The electronic device according to claim 25, wherein the electronic device comprises a mobile phone. 27. The electronic device according to claim 26, wherein the imaging device comprises a front camera of the mobile phone. 28. The electronic device according to claim 25, wherein the electronic device comprises a central processing unit and an external memory connected to the imaging device, and the central processing unit is used to control the external memory to store the combined image. 29. The electronic device according to claim 25, further comprising a central processing unit and a display device connected to the imaging device, the central processing unit is used to control the display device to display the Merge images.