CN110099230A - Image processing method and apparatus, and storage medium - Google Patents

Abstract

An embodiment of the present application discloses an image processing method and device, and a storage medium. The image processing method is applied to an image processing device, the image processing device includes a CIS, the CIS includes a sub-wavelength pixel unit having at least three PD columns with three different size parameters, the method includes: acquiring original data corresponding to incident light; wherein the original data includes RGB data corresponding to the sub-wavelength pixel unit; one sub-wavelength pixel unit corresponds to a group of RGB data; determining color data corresponding to the incident light according to the RGB data; and acquiring a color image corresponding to the incident light according to the color data.

Description

Image processing method and device, and storage medium

technical field

The embodiments of the present application relate to the field of image processing, and in particular, to an image processing method and device, and a storage medium.

Background technique

Traditional complementary metal oxide image sensors (CMOS Image Sensor, CIS) can include two different structures, Front Side Illumination (FSI) and Back Side Illumination (BSI), regardless of whether it is FSI or BSI, Both need to set the filter to control the absorption of one color in RGB by the same pixel. Therefore, when a photographing device uses a CIS provided with filters to acquire images, it is necessary to reconstruct the data output by the CIS into a generally browsable image format through demosaicing. Specifically, demosaicing is a digital image processing algorithm whose purpose is to reconstruct a full-color image from incomplete color samples output by a photosensitive element covered with a color filter array (CFA).

However, when the data output by traditional CIS is reconstructed by demosaicing, serious demosaicing noise is easily generated in the details of the image, which reduces the image quality, and the reconstruction process is cumbersome and the image processing efficiency is low.

SUMMARY OF THE INVENTION

The embodiments of the present application provide an image processing method and device, and a storage medium, which can effectively simplify the image reconstruction process and method, improve the efficiency of image processing, avoid noise problems caused by demosaicing, and improve image quality.

The technical solutions of the embodiments of the present application are implemented as follows:

An embodiment of the present application provides an image processing method, and the image processing method is applied to an image processing device, the image processing device includes a complementary metal oxide image sensor CIS, and the CIS includes at least three different size parameters set. A subwavelength pixel unit of three photodiode PD columns, the method comprising:

Acquiring raw data corresponding to incident light; wherein, the raw data includes RGB data corresponding to the sub-wavelength pixel unit; one sub-wavelength pixel unit corresponds to a group of RGB data;

Determine color data corresponding to the incident light according to the RGB data;

A color image corresponding to the incident light is acquired according to the color data.

An embodiment of the present application provides a computer-readable storage medium, on which a program is stored and applied to an image processing apparatus, where the program is executed by a processor to implement the above-mentioned image processing method.

Embodiments of the present application provide an image processing method and device, and a storage medium. The image processing method is applied to the image processing device. The image processing device includes a CIS, and the CIS includes at least three PD columns with three different size parameters. Sub-wavelength pixel unit, the method includes: acquiring original data corresponding to incident light; wherein, the original data includes RGB data corresponding to sub-wavelength pixel unit; one sub-wavelength pixel unit corresponds to a group of RGB data; Color data; obtain the color image corresponding to the incident light according to the color data. It can be seen that, in the embodiments of the present application, at least three PD columns with three different size parameters are set in the sub-wavelength pixel unit of the CIS. The distance between at least three PD columns in the pixel unit is much smaller than the wavelength of the incident light. In the category of geometric optics, it can be considered that multiple PD columns are coincident, so the incident light can be directly determined by the RGB data corresponding to the sub-wavelength pixel unit. The corresponding color data can be obtained to obtain the color image corresponding to the incident light, which can not only effectively simplify the image reconstruction process and method, improve the efficiency of image processing, but also avoid the noise problem caused by demosaicing and improve the image quality.

Description of drawings

Fig. 1 is the schematic diagram of CIS of FSI type;

Fig. 2 is the schematic diagram of CIS of BSI type;

FIG. 3 is an image processing method 1 proposed by an embodiment of the present application;

FIG. 4 is a second image processing method proposed by an embodiment of the application;

FIG. 5 is a schematic diagram 1 of the composition structure of an image processing apparatus proposed by an embodiment of the present application;

FIG. 6 is a second schematic diagram of the composition and structure of an image processing apparatus according to an embodiment of the present application;

7 is a schematic structural diagram 1 of a CIS proposed by an embodiment of the present application;

FIG. 8 is a second structural schematic diagram of a CIS proposed by an embodiment of the present application;

FIG. 9 is a schematic structural diagram three of a CIS proposed in an embodiment of the present application;

FIG. 10 is a top view 1 of a sub-wavelength pixel unit in an embodiment of the present application;

11 is a second top view of the sub-wavelength pixel unit in the embodiment of the present application;

12 is a third top view of the sub-wavelength pixel unit in the embodiment of the present application;

FIG. 13 is a fourth top view of the sub-wavelength pixel unit in the embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It should be understood that the specific embodiments described herein are only used to explain the related application, but not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the relevant application are shown in the drawings.

For traditional CIS, whether it is FSI or BSI, the photodiode (PD) in it all absorbs the light of 400nm-1100nm. Therefore, it is necessary to add filters of different colors on it, so as to make it a single A pixel only absorbs one color in RGB. At the same time, deep trench isolation (DTI) is required for isolation between each pixel, thereby preventing incident light of different colors from entering adjacent pixels, thereby avoiding crosstalk between adjacent pixels.

Figure 1 is a schematic diagram of an FSI type CIS, and Figure 2 is a schematic diagram of a BSI type CIS. As shown in Figures 1 and 2, the CIS includes a semiconductor substrate, a PD, a red filter, a green filter, and a blue filter. slices, pixel spacers, and metal wiring layers. Wherein, a lens is also provided before each filter.

Demosaic is a digital image processing algorithm designed to reconstruct full-color images from incomplete color samples output by CFA-coated sensors. This method is also called CFA interpolation or Color reconstruction. Among them, the simple interpolation method is a multi-variable interpolation method on a uniform grid, and the algorithm performs relatively direct mathematical operations on the same color elements of adjacent squares. The simplest method is nearest neighbor interpolation, which directly copies adjacent pixels of the same color channel. This method is not suitable for image quality, but it is an effective method for generating image previews with limited computing resources. Another method is bilinear interpolation, which uses the average of two or four adjacent red pixels to calculate the red value of non-red pixels, and blue and green are calculated in a similar way. Independent interpolation of each color plane is a more complex method, including bicubic interpolation, spline interpolation, and Lanxoss resampling.

Although the above image processing methods can achieve good results in the uniform area of the image, when using a pure color filter array, the edges and details of the image are prone to serious demosaicing noise, and due to the need for demosaic processing, the image reconstruction process It is cumbersome and the image processing efficiency is low.

The CIS in the image processing device proposed in the present application can be based on the nano-PD column structure, and can achieve selective absorption of different wavelengths through PD column structures with different diameters. Further, in this application, multiple PD column structures of three different diameters can be arranged in one pixel. In the sub-wavelength pixel array, the PD structure spacing in each sub-wavelength pixel unit is much smaller than the wavelength. It can be considered that multiple PD columns are coincident, so there is no need to perform demosaic processing on the original data obtained by the CIS, and the color information output by the CIS can be determined by the different components of R, G, and B corresponding to the sub-wavelength pixel unit. That is to say, in the present application, based on at least three PD columns with three different size parameters set by the sub-wavelength pixel unit, the image processing device can obtain the RGB data corresponding to all the sub-wavelength pixel units after the CIS absorbs the incident light, and then The color data corresponding to the incident light can be directly obtained by linearly superimposing the RGB data including the three channel data according to different weight values.

It should be noted that the CIS in the image processing apparatus proposed in this application may be either FSI or BSI, which is not specifically limited in this application, and the following embodiments take BSI as an example for description.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

An embodiment of the present application provides an image processing method, and FIG. 3 is an image processing method 1 proposed in an embodiment of the present application. The graphics processing method is applied to an image processing apparatus. As shown in FIG. 3 , the image processing apparatus The method of processing may include the following steps:

Step 101: Acquire raw data corresponding to the incident light; wherein, the raw data includes RGB data corresponding to sub-wavelength pixel units; one sub-wavelength pixel unit corresponds to a group of RGB data.

In the embodiment of the present application, the image processing apparatus may first acquire the original data corresponding to the incident light. The original data includes RGB data corresponding to the sub-wavelength pixel unit. Specifically, one sub-wavelength pixel unit corresponds to a group of RGB data.

It should be noted that, in the embodiments of the present application, a set of RGB data corresponding to one sub-wavelength pixel unit may include three channel data corresponding to at least three PD columns in the one sub-wavelength pixel unit. The three channel data may be red channel data, green channel data and blue channel data.

It should be noted that, in the embodiments of the present application, the CIS may be composed of sub-wavelength pixel units. The sub-wavelength pixel unit may include at least three PD pillars with three different size parameters, wherein, among the at least three PD pillars with three different size parameters, the number of PD pillars with one size parameter may be at least one. There is no specific limitation.

Further, in the implementation of the present application, at least three PD columns with three different size parameters can absorb the incident light according to the preset wavelength through optical resonance, so that the original data corresponding to the incident light can be obtained.

It should be noted that, in the embodiments of the present application, the preset wavelengths may include a first wavelength corresponding to red light, a second wavelength corresponding to green light, and a third wavelength corresponding to blue light. The first wavelength corresponding to red light may be 625 nm to 740 nm; the second wavelength corresponding to green light may be 492 nm to 577 nm; and the third wavelength corresponding to blue light may be 440 nm to 475 nm.

Further, in the embodiment of the present application, at least three PD columns are respectively used to absorb red light, green light and blue light in the incident light.

It should be noted that, in the implementation of the present application, the pixel size corresponding to the sub-wavelength pixel unit is smaller than any one of the first wavelength, the second wavelength and the third wavelength. For example, when the first wavelength, the second wavelength and the third wavelength are respectively 625 nm, 492 nm and 440 nm, the pixel size corresponding to the sub-wavelength pixel unit may be determined within a range of less than or equal to 400 nm. That is to say, in the embodiment of the present application, the pixel size corresponding to the sub-wavelength pixel unit is in the order of hundreds of nanometers.

Further, in the embodiment of the present application, it is precisely because the sub-wavelength pixel unit can include at least three PD columns with three different size parameters, and the at least three PD columns can respectively absorb red light and green light in the incident light. and blue light, so that the sub-wavelength pixel unit can simultaneously absorb the light of the three colors of RGB of the incident light through optical resonance, and because the pixel size corresponding to the sub-wavelength pixel unit is a hundred nanometers, at least three PD in the sub-wavelength pixel unit The distance between the columns is much smaller than the wavelength of the incident light. In the category of geometric optics, multiple PD columns can be considered to be coincident. Therefore, the image processing device does not need to perform the demosaic process for the raw data output by the CIS, but directly passes the subwavelength pixels. The RGB data corresponding to the unit determines the color data corresponding to the incident light, so as to obtain a color image corresponding to the incident light. Compared with the prior art, the image reconstruction process and method can be effectively simplified, the efficiency of image processing can be improved, the noise problem caused by demosaicing can be avoided, and the image quality can be improved.

Further, in the embodiment of the present application, three different size parameters corresponding to the at least three PD columns may be determined by the first wavelength, the second wavelength and the third wavelength, respectively. Specifically, the diameter corresponding to one PD column can be determined by the wavelength of the light absorbed by it. For example, if the PD column is used to absorb red light, the diameter of the PD column can be determined by the first wavelength to be 120 nm; if the PD column is used to absorb green light, the diameter of the PD column can be determined by the second wavelength to be 90 nm; The column is used to absorb blue light, then the diameter of the PD column can be determined to be 60 nm by the third wavelength.

Further, in the embodiment of the present application, the shape corresponding to the at least three PD columns may include one of a cuboid, a cylinder, or a parallelogram, and the specific shape may be selected according to the actual situation, which is not specified in the embodiment of the present application. specific restrictions.

Step 102: Determine color data corresponding to the incident light according to the RGB data.

In the embodiment of the present application, after acquiring the original data of the incident light including the RGB data corresponding to the sub-wavelength pixel unit, the image processing apparatus may further determine the color data corresponding to the incident light according to the RGB data.

It should be noted that, in the embodiments of the present application, after acquiring a set of RGB data corresponding to a sub-wavelength pixel unit, the image processing device can determine a color corresponding to the sub-wavelength pixel unit according to the set of RGB data. Further, after acquiring all the color information corresponding to all the sub-wavelength pixel units, the image processing device can determine the color data corresponding to the incident light according to all the color information.

Further, in the implementation of the present application, the image processing device can calculate and obtain color information corresponding to a sub-wavelength pixel unit according to the red channel data, green channel data and blue channel data corresponding to a sub-wavelength pixel unit, thereby facilitating Color data corresponding to the incident light may be further determined according to all color information corresponding to all sub-wavelength pixel units.

It can be seen that in the embodiments of the present application, since the pixel size corresponding to the sub-wavelength pixel unit is in the order of 100 nanometers, the distance between at least three PD columns in the sub-wavelength pixel unit is much smaller than the wavelength of the incident light. In the category of geometric optics, it can be considered that multiple PD columns are coincident. Therefore, the image processing device does not need to perform demosaic processing on the original data obtained by the CIS, but directly determines the color corresponding to the incident light through the RGB data corresponding to the sub-wavelength pixel unit. data, so as to obtain the color image corresponding to the incident light. Compared with the prior art, the image reconstruction process and method can be effectively simplified, the efficiency of image processing can be improved, the noise problem caused by demosaicing can be avoided, and the image quality can be improved.

Step 103: Acquire a color image corresponding to the incident light according to the color data.

In the embodiment of the present application, after determining the color data corresponding to the incident light according to the RGB data, the image processing apparatus may further acquire a color image corresponding to the incident light according to the color data.

It should be noted that, in the embodiments of the present application, after determining the color data corresponding to the incident light, the image processing device may further perform image algorithm processing on the color data, thereby obtaining a color image corresponding to the incident light.

Further, in the embodiments of the present application, an image algorithm refers to an algorithm used to process an image, and may specifically include image denoising, image transformation, image analysis, image compression, image enhancement, image blurring, and the like. For example, after obtaining the color data, the image processing apparatus may perform High-DynamicRange (HDR) processing on the color data, so as to obtain a color image with more dynamic range and image details. Accordingly, the obtained The color image is an HDR image; after obtaining the color data, the CIS can perform noise reduction processing on the color data, so as to reduce the noise generated during the digitization and transmission of the image.

An embodiment of the present application provides an image processing method, the image processing method is applied to an image processing device, the image processing device includes a CIS, and the CIS includes a sub-wavelength pixel unit provided with at least three PD columns with three different size parameters, the The method includes: acquiring original data corresponding to incident light; wherein, the original data includes RGB data corresponding to a sub-wavelength pixel unit; one sub-wavelength pixel unit corresponds to a group of RGB data; determining color data corresponding to the incident light according to the RGB data; Acquire a color image corresponding to the incident light. It can be seen that, in the embodiments of the present application, at least three PD columns with three different size parameters are set in the sub-wavelength pixel unit of the CIS. The distance between at least three PD columns in the pixel unit is much smaller than the wavelength of the incident light. In the category of geometric optics, it can be considered that multiple PD columns are coincident, so the incident light can be directly determined by the RGB data corresponding to the sub-wavelength pixel unit. The corresponding color data can be obtained to obtain the color image corresponding to the incident light, which can not only effectively simplify the image reconstruction process and method, improve the efficiency of image processing, but also avoid the noise problem caused by demosaicing and improve the image quality.

Based on the above embodiment, in another embodiment of the present application, a set of RGB data corresponding to one sub-wavelength pixel unit in the CIS may specifically include three channel data corresponding to the one sub-wavelength pixel unit, that is, the data corresponding to the one sub-wavelength pixel unit. The red channel data, green channel data and blue channel data corresponding to the sub-wavelength pixel unit, FIG. 4 is an image processing method 2 proposed by the embodiment of the application, as shown in FIG. 4 , for the above step 102, CIS according to RGB data The method for determining color data corresponding to incident light may include the following steps:

Step 102a, input the three channel data corresponding to any sub-wavelength pixel unit into the preset color calculation model, and output to obtain the color information corresponding to any sub-wavelength pixel unit; wherein, the preset color calculation model is used according to preset color information. The weight values are linearly superimposed on the three channel data.

In the embodiment of the present application, after acquiring the original data corresponding to the incident light, the image processing device may input the three-channel data corresponding to any sub-wavelength pixel unit into the preset color calculation model, and then output the sub-wavelength data to obtain the Color information corresponding to the pixel unit.

It should be noted that, in the embodiment of the present application, the preset color calculation model may be used to perform linear superposition calculation on the three channel data according to the preset weight value. Specifically, the image processing device performs the calculation according to the preset color calculation model. When calculating the color information, different channel data can be assigned different weight values, so as to calculate the original data corresponding to the sub-wavelength pixel unit according to different weights, and obtain the color information corresponding to the sub-wavelength pixel unit.

Further, in the embodiment of the present application, for any sub-wavelength pixel unit, when the CIS obtains the color information corresponding to the sub-wavelength pixel unit according to the preset color calculation model, it can be based on a group of RGB corresponding to the sub-wavelength pixel unit. The data is calculated according to formula (1):

P=a×R+b×G+c×B(1)

Among them, P is the color information corresponding to a sub-wavelength pixel unit, R is the red channel data in the RGB data, G is the green channel data in the RGB data, B is the blue channel data in the RGB data, a, b, c For the weight values corresponding to the three channels, the values of a, b, and c need to be corrected by color calibration.

Step 102b: Determine color data according to all color information corresponding to all subwavelength pixel units.

In the embodiment of the present application, after inputting the three channel data corresponding to any sub-wavelength pixel unit into the preset color calculation model, and outputting the color information corresponding to any sub-wavelength pixel unit, the image processing device can obtain the color information corresponding to any sub-wavelength pixel unit according to All color information corresponding to all subwavelength pixel units in the CIS is further determined for color data corresponding to incident light.

It should be noted that, in the embodiment of the present application, the image processing device may calculate and obtain the color information corresponding to each sub-wavelength pixel unit according to the preset color calculation model, and then calculate all the color information corresponding to all the sub-wavelength pixel units. Integrate to obtain the color data corresponding to the incident light received by the CIS.

It can be seen that in the embodiments of the present application, since the pixel size corresponding to the sub-wavelength pixel unit is in the order of 100 nanometers, the distance between at least three PD columns in the sub-wavelength pixel unit is much smaller than the wavelength of the incident light. In the category of geometric optics, it can be considered that multiple PD columns are coincident. Therefore, the image processing device does not need to perform demosaic processing on the original data obtained by the CIS, but directly determines the color corresponding to the incident light through the RGB data corresponding to the sub-wavelength pixel unit. data, thereby effectively simplifying the image reconstruction process and method, and can also avoid the noise problem caused by demosaicing.

An embodiment of the present application provides an image processing method, the image processing method is applied to an image processing device, the image processing device includes a CIS, and the CIS includes a sub-wavelength pixel unit provided with at least three PD columns with three different size parameters, the The method includes: acquiring original data corresponding to incident light; wherein, the original data includes RGB data corresponding to a sub-wavelength pixel unit; one sub-wavelength pixel unit corresponds to a group of RGB data; determining color data corresponding to the incident light according to the RGB data; Acquire a color image corresponding to the incident light. It can be seen that, in the embodiments of the present application, at least three PD columns with three different size parameters are set in the sub-wavelength pixel unit of the CIS. The distance between at least three PD columns in the pixel unit is much smaller than the wavelength of the incident light. In the category of geometric optics, it can be considered that multiple PD columns are coincident, so the incident light can be directly determined by the RGB data corresponding to the sub-wavelength pixel unit. The corresponding color data can be obtained to obtain the color image corresponding to the incident light, which can not only effectively simplify the image reconstruction process and method, improve the efficiency of image processing, but also avoid the noise problem caused by demosaicing and improve the image quality.

Based on the above embodiments, another embodiment of the present application provides an image processing apparatus including a CIS. FIG. 5 is a schematic diagram 1 of the composition and structure of an image processing apparatus proposed in an embodiment of the present application. As shown in FIG. 5 , in In the embodiment of the present invention, the image processing apparatus 1 includes: an acquisition unit 11 and a determination unit 12 .

The acquisition unit 11 is configured to acquire raw data corresponding to incident light; wherein, the raw data includes RGB data corresponding to the sub-wavelength pixel unit; one sub-wavelength pixel unit corresponds to a group of RGB data.

The determining unit 12 is configured to determine color data corresponding to the incident light according to the RGB data.

The acquiring unit 11 is further configured to acquire a color image corresponding to the incident light according to the color data.

Further, in the embodiment of the present application, the acquisition unit 11 is specifically used for the at least three PD columns in the sub-wavelength pixel unit to absorb the incident light according to a preset wavelength through optical resonance. , to obtain the RGB data.

Further, in the embodiment of the present application, a set of RGB data corresponding to the one sub-wavelength pixel unit includes three channel data corresponding to the one sub-wavelength pixel unit.

Further, in the embodiment of the present application, the determining unit 12 is specifically configured to input the three channel data corresponding to any sub-wavelength pixel unit into the preset color calculation model, and output to obtain any one of the color information corresponding to the sub-wavelength pixel units; wherein, the preset color calculation model is used to linearly superimpose the three channel data according to the preset weight value; and according to all the color information corresponding to all the sub-wavelength pixel units, determine the color data.

Further, in the embodiment of the present application, the obtaining unit 11 is specifically configured to perform high dynamic range image HDR processing on the color data to obtain the color image.

Further, in the embodiment of the present application, the obtaining unit 11 is further specifically configured to perform noise reduction processing on the color data to obtain the color image.

Further, in the embodiment of the present application, the preset wavelength includes a first wavelength corresponding to red light, a second wavelength corresponding to green light, and a third wavelength corresponding to blue light.

Further, in the embodiment of the present application, the three different size parameters corresponding to the at least three PD columns are respectively determined by the first wavelength, the second wavelength and the third wavelength.

Further, in the embodiment of the present application, the at least three PD columns are respectively used for absorbing red light, green light and blue light in the incident light.

Further, in the embodiment of the present application, the shape corresponding to the at least three PD columns includes one of a cuboid, a cylinder, or a parallelogram.

Further, in the embodiment of the present application, the pixel size corresponding to the sub-wavelength pixel unit is smaller than any one of the first wavelength, the second wavelength and the third wavelength.

FIG. 6 is a second schematic diagram of the composition and structure of an image processing apparatus proposed by an embodiment of the present application. As shown in FIG. 6 , the image processing apparatus 1 proposed by an embodiment of the present application may further include a processor 13 and store the processor 13 executable. The memory 14 and the CIS 15 for instructions, further, the image processing apparatus 1 may further include a communication interface 16 , and a bus 17 for connecting the processor 13 , the memory 14 and the communication interface 16 .

In the embodiment of the present application, the processor 13 may be an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), At least one of a programmable logic device (ProgRAMmable Logic Device, PLD), a field programmable gate array (Field ProgRAMmable GateArray, FPGA), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, and a microprocessor kind. It can be understood that, for different devices, the electronic device used to implement the function of the processor may also be other, which is not specifically limited in the embodiment of the present application. The display 1 may also include a memory 14, which may be connected to the processor 13, wherein the memory 14 is used to store executable program codes, which include computer operating instructions, the memory 14 may include high-speed RAM memory, and may also include Non-volatile memory, for example, at least two disk drives.

In the embodiment of the present application, the bus 17 is used to connect the communication interface 16 , the processor 13 and the memory 14 and the mutual communication among these devices.

In the embodiment of the present application, the memory 14 is used to store instructions and data.

Further, in the embodiment of the present application, the processor 13 is used to acquire raw data corresponding to incident light; wherein, the raw data includes RGB data corresponding to the sub-wavelength pixel units; one sub-wavelength pixel unit corresponds to a set RGB data; determine color data corresponding to the incident light according to the RGB data; acquire a color image corresponding to the incident light according to the color data.

In practical applications, the memory 14 may be a volatile memory (volatile memory), such as a random-access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only first Storage device (Read-Only Memory, ROM), flash memory (flash memory), hard disk (Hard DiskDrive, HDD) or solid-state drive (Solid-State Drive, SSD); Controller 13 provides instructions and data.

In addition, each functional module in this embodiment may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software function modules.

If the integrated unit is implemented in the form of software function modules and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or correct. Part of the contribution made by the prior art or all or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal A computer, a server, or a network device, etc.) or a processor (processor) executes all or part of the steps of the method in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.

FIG. 7 is a schematic structural diagram 1 of a CIS proposed by an embodiment of the present application. As shown in FIG. 7 , in the embodiment of the present application, for a BSI type CIS, the CIS 15 may include: a semiconductor substrate 151 and a sub-wavelength pixel unit 152 and readout circuit 153 .

It should be noted that, in the implementation of the present application, the sub-wavelength pixel unit 152 is disposed in the semiconductor substrate 11 . Among them, the sub-wavelength pixel unit 152 can be used for sensing incident light.

Further, in the implementation of the present application, the subwavelength pixel unit 152 is connected to the readout circuit 153 , so that the electrical signal obtained by converting the incident light can be transmitted from the subwavelength pixel unit 152 to the readout circuit 153 .

It should be noted that, in the embodiment of the present application, the sub-wavelength pixel unit 152 may include at least three PD columns with three different size parameters, which are 1521 , 1522 and 1523 respectively.

Further, in the implementation of the present application, among the at least three PD pillars with three different size parameters, the number of PD pillars with one size parameter may be at least one. For example, one sub-wavelength pixel unit 152 may include two 1521, one 1522 and one 1523; one sub-wavelength pixel unit 152 may also include one 1521, one 1522 and one 1523; one sub-wavelength pixel unit 152 also includes It can include one 1521, one 1522 and two 1523, which are not specifically limited in this application.

Further, in the implementation of the present application, at least three PD columns 1521 , 1522 and 1523 with three different size parameters can absorb incident light according to preset wavelengths through optical resonance to obtain RGB data.

It should be noted that, in the embodiments of the present application, the preset wavelengths include a first wavelength corresponding to red light, a second wavelength corresponding to green light, and a third wavelength corresponding to blue light. The first wavelength corresponding to red light may be 625 nm to 740 nm; the second wavelength corresponding to green light may be 492 nm to 577 nm; and the third wavelength corresponding to blue light may be 440 nm to 475 nm.

Further, in the embodiment of the present application, at least three PD columns 1521 , 1522 and 1523 are respectively used for absorbing red light, green light and blue light in the incident light. For example, the PD column 1521 is used for absorbing the red light in the incident light; the PD column 1522 is used for absorbing the green light in the incident light; the PD column 1523 is used for absorbing the blue light in the incident light.

It should be noted that, in the embodiments of the present application, it is precisely because the sub-wavelength pixel unit 152 may include at least three PD pillars 1521 , 1522 and 1523 with three different size parameters, and the at least three PD pillars 1521 , 1522 and 1523 can respectively absorb red light, green light and blue light in the incident light, so that the sub-wavelength pixel unit 152 can simultaneously absorb the light of the three colors of RGB of the incident light through optical resonance. The original data is processed demosaic, but the RGB data including the three-channel data are linearly superimposed according to different weight values, so that the color data corresponding to the incident light can be directly obtained, and the color image corresponding to the incident light can be obtained.

Further, in the embodiment of the present application, three different size parameters corresponding to the at least three PD columns may be determined by the first wavelength, the second wavelength and the third wavelength, respectively.

It should be noted that, in the embodiments of the present application, the diameter corresponding to one PD column may be determined by the wavelength of the light absorbed by it. For example, if the PD column 1521 is used to absorb red light, the diameter of the PD column 1521 can be determined by the first wavelength to be 120 nm; the PD column 1522 is used to absorb green light, then the second wavelength can be used to determine the diameter of the PD column 1522 The diameter of the PD column 1523 is 90 nm; the PD column 1523 is used to absorb blue light, so the diameter of the PD column 1523 can be determined to be 60 nm through the third wavelength.

Further, in the embodiment of the present application, the shape corresponding to the at least three PD columns may include one of a cuboid, a cylinder, or a parallelogram, and the specific shape may be selected according to the actual situation, which is not specified in the embodiment of the present application. specific restrictions.

It should be noted that, in the implementation of the present application, the pixel size corresponding to the sub-wavelength pixel unit is smaller than any one of the first wavelength, the second wavelength and the third wavelength. For example, when the first wavelength, the second wavelength and the third wavelength are respectively 625 nm, 492 nm and 440 nm, the pixel size corresponding to the sub-wavelength pixel unit may be determined within a range of less than or equal to 400 nm.

Further, in the implementation of the present application, subwavelength refers to a periodic (or aperiodic) structure whose characteristic size is equal to or smaller than the working wavelength. The feature size of the subwavelength structure is smaller than the wavelength, and its reflectivity, transmittance, polarization characteristics and spectral characteristics all show completely different characteristics from conventional diffractive optical elements, so it has greater application potential. So far, it is mainly used as anti-reflection surface, polarizing device, narrow-band filter and phase plate, etc. A typical subwavelength antireflection microstructure is a relief-structured subwavelength grating. By adjusting the grating material, structural parameters such as groove depth, duty cycle and period, the grating can have near-zero reflectivity.

In the implementation of the present application, further, based on the above-mentioned FIG. 7 , FIG. 8 is a second schematic structural diagram of a CIS proposed by an embodiment of the present application. As shown in FIG. 8 , the CIS 15 may further include an image processor 154 , wherein the reading The output circuit 153 is connected to the image processor 154 .

In the implementation of the present application, further, based on the above-mentioned FIG. 7 , FIG. 9 is a third structural schematic diagram of a CIS proposed by the embodiment of the present application. As shown in FIG. 9 , the CIS 15 may further include a lens 155 , wherein the lens 155 and the The subwavelength pixel unit 152 is connected.

It should be noted that, in the implementation of the present application, the lens 155 is used to focus the incident light. Since the CIS in this application can achieve selective absorption of different wavelengths of incident light through sub-wavelength pixel units composed of PD columns with three different size parameters, which can enhance the local optical density of states, therefore, the lens 155 in CIS15 It can also be omitted, that is, the lens 155 is not an essential part of the CIS 15 in the embodiment of the present application, and whether to provide a lens can be selected according to the actual situation, which is not specifically limited in the embodiment of the present application.

It should be noted that the CIS 15 proposed in this application may be either FSI or BSI. The embodiment of this application uses BSI as an example for description, but does not make any specific limitation.

Further, the CIS 10 is a structure that can realize the sub-wavelength pixel unit 12 . 10 is a top view of a sub-wavelength pixel unit in an embodiment of the present application, FIG. 11 is a top view of a sub-wavelength pixel unit in an embodiment of the present application, and FIG. 12 is a top view of the sub-wavelength pixel unit in an embodiment of the present application. As shown in 10, 11, and 12, the sub-wavelength pixel unit 152 may include at least three PD columns with three different size parameters, which are 1521, 1522, and 1523, respectively. Specifically, since the pixel size corresponding to the sub-wavelength pixel unit is less than or equal to 400 nm, the PD column 1521 , the PD column 1522 and the PD column 1523 are all PDs of hundreds of nanometers. For example, the diameter of the PD column 1521 may be 120 nm; The diameter of 1522 may be 90 nm; the diameter of PD pillar 1523 may be 60 nm.

It should be noted that, in the implementation of the present application, the PD column 1521 can be used to absorb red light, the PD column 1522 can be used to absorb green light, and the PD column 1523 can be used to absorb blue light. That is to say, in the implementation of the present application, at least three PD columns 1521 , 1522 and 1523 with three different size parameters can respectively absorb red light, green light and blue light in the incident light through optical resonance, so that the sub-wavelength The pixel unit 152 can simultaneously absorb the three colors of RGB of the incident light, that is, the sub-wavelength pixel unit 152 in this embodiment of the present application is configured with at least three PD columns 1521 , 1522 and 1523 with three different size parameters, which is different from the existing one. Compared with the technology, it is not necessary to perform demosaic processing on the original data obtained by CIS, but linearly superimpose the RGB data including three channel data according to different weight values, so that the color data corresponding to the incident light can be directly obtained to obtain the incident light. Color image corresponding to light.

13 is a fourth top view of the sub-wavelength pixel unit in the embodiment of the present application. As shown in FIG. 13 , the sub-wavelength pixel unit 152 may include three PD columns with three different size parameters, namely 1521 , 1522 and 1523 . Specifically, since the pixel size corresponding to the sub-wavelength pixel unit is less than or equal to 400 nm, the PD column 1521 , the PD column 1522 and the PD column 1523 are all PDs of hundreds of nanometers. For example, the diameter of the PD column 1521 may be 120 nm; The diameter of 1522 may be 90 nm; the diameter of PD pillar 1523 may be 60 nm.

It should be noted that, in the implementation of the present application, the PD column 1521 can be used to absorb red light, the PD column 1522 can be used to absorb green light, and the PD column 1523 can be used to absorb blue light. That is to say, in the implementation of the present application, the three PD columns 1521, 1522 and 1523 with three different size parameters can absorb the red light, green light and blue light in the incident light respectively through optical resonance, so that the sub-wavelength pixel The unit 152 can simultaneously absorb the three colors of RGB of the incident light, that is, the sub-wavelength pixel unit 152 of the embodiment of the present application is different from the prior art through the setting of three PD columns 1521, 1522 and 1523 with three different size parameters. It is not necessary to perform demosaic processing on the original data obtained by the CIS, but linearly superimpose the RGB data including the three-channel data according to different weight values, so that the color data corresponding to the incident light can be directly obtained to obtain the corresponding color data of the incident light. color image.

The present application proposes an image processing device, the image processing device includes a CIS, the CIS includes a sub-wavelength pixel unit provided with at least three PD columns with three different size parameters, and the image processing device acquires raw data corresponding to incident light; wherein , the original data includes RGB data corresponding to the sub-wavelength pixel unit; one sub-wavelength pixel unit corresponds to a group of RGB data; the color data corresponding to the incident light is determined according to the RGB data; the color image corresponding to the incident light is obtained according to the color data. It can be seen that, in the embodiments of the present application, at least three PD columns with three different size parameters are set in the sub-wavelength pixel unit of the CIS. The distance between at least three PD columns in the pixel unit is much smaller than the wavelength of the incident light. In the category of geometric optics, it can be considered that multiple PD columns are coincident, so the incident light can be directly determined by the RGB data corresponding to the sub-wavelength pixel unit. The corresponding color data can be obtained to obtain the color image corresponding to the incident light, which can not only effectively simplify the image reconstruction process and method, improve the efficiency of image processing, but also avoid the noise problem caused by demosaicing and improve the image quality.

Based on the above embodiments, in another embodiment of the present application, an embodiment of the present application provides a computer-readable storage medium on which a program is stored, and when the program is executed by a processor, the above-mentioned image processing method is implemented.

Specifically, a program instruction corresponding to an image processing method in this embodiment may be stored on a storage medium such as an optical disk, a hard disk, a U disk, etc. When the program instruction corresponding to an image processing method in the storage medium is stored in a storage medium When the electronic device reads or is executed, it includes the following steps:

Acquiring raw data corresponding to incident light; wherein, the raw data includes RGB data corresponding to the sub-wavelength pixel unit; one sub-wavelength pixel unit corresponds to a group of RGB data;

Determine color data corresponding to the incident light according to the RGB data;

A color image corresponding to the incident light is acquired according to the color data.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, a display, or a computer program product. Accordingly, the application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, optical storage, and the like.

The present application is described with reference to schematic flowcharts and/or block diagrams of implementations of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It will be understood that each process and/or block in the schematic flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the schematic flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a process or processes and/or a block or blocks in the block diagrams.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions An apparatus implements the functions specified in a flow or flows of the implementation flow diagram and/or a block or blocks of the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the implementing flow diagram and/or the block or blocks of the block diagram.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the protection scope of the present application.

Claims (19)

1. An image processing method, characterized in that, the image processing method is applied in an image processing device, and the image processing device comprises a complementary metal oxide image sensor CIS, and the CIS comprises at least three different size parameters. A subwavelength pixel unit of three photodiode PD columns, the method comprising: Acquiring raw data corresponding to incident light; wherein, the raw data includes RGB data corresponding to the sub-wavelength pixel unit; one sub-wavelength pixel unit corresponds to a group of RGB data; Determine color data corresponding to the incident light according to the RGB data; A color image corresponding to the incident light is acquired according to the color data. 2. The method according to claim 1, wherein the acquiring raw data corresponding to the incident light comprises: The at least three PD columns in the sub-wavelength pixel unit absorb the incident light according to a preset wavelength through optical resonance to obtain the RGB data. 3 . The method according to claim 1 , wherein the set of RGB data corresponding to the one sub-wavelength pixel unit includes three channel data corresponding to the one sub-wavelength pixel unit. 4 . 4. The method according to claim 3, wherein the determining the color data corresponding to the incident light according to the RGB data comprises: Inputting the three channel data corresponding to any sub-wavelength pixel unit into a preset color calculation model, and outputting the color information corresponding to any one of the sub-wavelength pixel units; wherein, the preset color calculation model is used for Linearly superimpose the three channel data according to the preset weight value; The color data is determined according to all color information corresponding to all subwavelength pixel units. 5. The method according to claim 1, wherein the acquiring a color image corresponding to the incident light according to the color data comprises: Performing high dynamic range image HDR processing on the color data to obtain the color image. 6. The method according to claim 1, wherein the acquiring a color image corresponding to the incident light according to the color data comprises: Noise reduction processing is performed on the color data to obtain the color image. 7 . The method according to claim 1 , wherein the preset wavelength comprises a first wavelength corresponding to red light, a second wavelength corresponding to green light, and a third wavelength corresponding to blue light. 8 . 8. The method of claim 7, wherein The three different size parameters corresponding to the at least three PD columns are respectively determined by the first wavelength, the second wavelength and the third wavelength. 9. The method of claim 2, wherein The at least three PD columns are respectively used for absorbing red light, green light and blue light in the incident light. 10. The method of claim 1, wherein: The shape corresponding to the at least three PD pillars includes one of a cuboid, a cylinder or a parallelogram. 11 . The method according to claim 1 , wherein a pixel size corresponding to the sub-wavelength pixel unit is smaller than any one of the first wavelength, the second wavelength and the third wavelength. 12 . 12. An image processing device, characterized in that the image processing device comprises: an acquisition unit and a determination unit, The acquisition unit is used to acquire original data corresponding to incident light; wherein, the original data includes RGB data corresponding to the sub-wavelength pixel unit; one sub-wavelength pixel unit corresponds to a group of RGB data; the determining unit, configured to determine color data corresponding to the incident light according to the RGB data; The acquiring unit is further configured to acquire a color image corresponding to the incident light according to the color data. 13. The image processing apparatus according to claim 12, wherein: The acquisition unit is specifically used for the at least three PD columns in the sub-wavelength pixel unit to absorb the incident light according to a preset wavelength through optical resonance to obtain the RGB data. 14 . The image processing apparatus according to claim 12 , wherein the set of RGB data corresponding to the one sub-wavelength pixel unit includes three channel data corresponding to the one sub-wavelength pixel unit. 15 . 15. The image processing apparatus according to claim 14, wherein: The determining unit is specifically configured to input the three channel data corresponding to any sub-wavelength pixel unit into the preset color calculation model, and output to obtain the color information corresponding to the any sub-wavelength pixel unit; The preset color calculation model is used to linearly superimpose the three channel data according to the preset weight value; and determine the color data according to all the color information corresponding to all the sub-wavelength pixel units. 16. The image processing apparatus according to claim 12, wherein: The obtaining unit is specifically configured to perform high dynamic range image HDR processing on the color data to obtain the color image; The obtaining unit is further specifically configured to perform noise reduction processing on the color data to obtain the color image. 17. The image processing apparatus according to claim 13, wherein: The preset wavelength includes a first wavelength corresponding to red light, a second wavelength corresponding to green light, and a third wavelength corresponding to blue light; The three different size parameters corresponding to the at least three PD columns are respectively determined by the first wavelength, the second wavelength and the third wavelength; the at least three PD columns are respectively used for absorbing red light, green light and blue light in the incident light; The shape corresponding to the at least three PD pillars includes one of a cuboid, a cylinder or a parallelogram; The pixel size corresponding to the sub-wavelength pixel unit is smaller than any one of the first wavelength, the second wavelength and the third wavelength. 18. An image processing device, characterized in that the image processing device comprises a processor, a memory storing executable instructions of the processor, a CIS, and the CIS comprises at least three optoelectronic devices provided with three different size parameters. The subwavelength pixel unit of the diode PD column, when the instructions are executed by the processor, implements the method of any one of claims 1-11. 19. A computer-readable storage medium having a program stored thereon and applied in an image processing apparatus, wherein the program implements the method according to any one of claims 1-11 when the program is executed by a processor.