CN113382185B

CN113382185B - Data generation method, data training method, data imaging method, storage medium, and photographing apparatus

Info

Publication number: CN113382185B
Application number: CN202110643631.5A
Authority: CN
Inventors: 张欣; 陈杭; 杨晓冬
Original assignee: Haowei Technology Wuhan Co ltd
Current assignee: Haowei Technology Wuhan Co ltd
Priority date: 2021-06-09
Filing date: 2021-06-09
Publication date: 2022-06-28
Anticipated expiration: 2041-06-09
Also published as: CN113382185A

Abstract

The invention provides a data generation method, a training method, an imaging method, a storage medium and a shooting device. Wherein the data generation method obtains the training image based on an output image of an image sensor of the 4Cell array. By the configuration, the similarity of the 4Cell array and the QPD array on the structure and the output data can be fully utilized, the accuracy of the trained model is ensured according to the quality of the training data, and further, the algorithm based on the training model can output better images, and the problem that the image quality output by the algorithm matched with the image sensor of the QPD array is poor in the prior art is solved.

Description

Data generation method, data training method, data imaging method, storage medium, and photographing apparatus

Technical Field

The present invention relates to the field of image signal processing, and in particular, to a data generation method, a training method, an imaging method, a storage medium, and a photographing apparatus.

Background

The photosensitive cells in the image sensor have different arrangement arrays, and one arrangement is a QPD (Quad Phase Detection four-Phase Detection) array, which is arranged by splitting each photosensitive cell in a conventional bayer array into four photosensitive cells arranged at 2 × 2, and the four photosensitive cells share one microlens (micro lens), so that four pixels having Phase information can be generated. So configured, can have great advantage in the aspect of the auto focus of lens, depth map calculation, low light, etc. However, due to the influence of its own structure, when the original output Image of the Image sensor of the QPD array is directly imaged using only a general-purpose ISP (Image Signal Processing) flow, the final Image quality may be poor due to spectral unevenness. There is a method for imaging an original output image after preprocessing, in which a parallax-based method is used in the preprocessing process, and the obtained image has the problems of low resolution and excessive smoothness, and the image quality is still poor although the image quality is improved.

In summary, in the prior art, there is a problem that an image quality output by an algorithm matched with an image sensor of a QPD array is poor.

Disclosure of Invention

The invention aims to provide a data generation method, a training method, an imaging method, a storage medium and a shooting device, and aims to solve the problem that in the prior art, the quality of an image output by an algorithm matched with an image sensor of a QPD array is poor.

In order to solve the above technical problem, according to a first aspect of the present invention, there is provided a data generation method for generating a training image of an imaging model of an image sensor of a QPD array, the data generation method comprising: shooting by an image sensor of the 4Cell array under a preset working condition; the training image is derived based on output images of the image sensors of the 4Cell array.

Optionally, the training image includes an input image and an output contrast image, and the data generating method includes: the original output image of the image sensor of the 4Cell array is subjected to preset operation to obtain an intermediate image; obtaining the output contrast image based on the intermediate image; and replacing four pixels of the same pixel group in the output contrast image with a four-in-one pixel to obtain the input image, wherein during replacement, the color channel of the four-in-one pixel is the same as the color channel of the corresponding pixel group, and the brightness value of the four-in-one pixel is the average value of the brightness values of the four pixels of the corresponding pixel group.

Optionally, the step of obtaining the output contrast image based on the intermediate image includes: cutting the intermediate image to obtain the output contrast image; and cutting the second image to obtain the input image.

Optionally, the data generating method includes: performing mirror image and/or rotation operation on the output contrast image to obtain an additional output contrast image; and carrying out corresponding operation on the input image to obtain an additional input image.

Optionally, the size of the output contrast image is 2N × 2N, and the size of the input image is N × N, where N is an odd number greater than or equal to 13.

Optionally, the value range of N is 13-63.

In order to solve the above technical problem, according to a second aspect of the present invention, there is provided a training method for training an imaging model of an image sensor of a QPD array, the training method including: and obtaining the training image based on the data generation method, and training the imaging method based on the training image.

In order to solve the above technical problem, according to a third aspect of the present invention, there is provided an imaging method, wherein a processing result is obtained by processing a raw output image of an image sensor of a QPD array based on an imaging model obtained based on the above training method;

And obtaining an image based on the processing result.

In order to solve the above technical problem, according to a fourth aspect of the present invention, there is provided a storage medium having a program stored thereon, the program executing the above data generating method, the above training method, or the above imaging method when executed.

In order to solve the technical problem, according to a fifth aspect of the present invention, there is provided a photographing device, the photographing device comprising an image sensor and an imaging unit, wherein an arrangement array of the image sensor is a QPD array, and the imaging unit is configured to process a raw output image of the image sensor based on the above-mentioned imaging method to obtain an output image.

Compared with the prior art, the data generation method, the training method, the imaging method, the storage medium and the shooting device provided by the invention have the advantages that the training image is obtained based on the output image of the image sensor of the 4Cell array by the data generation method. By the configuration, the similarity of the 4Cell array and the QPD array on the structure and output data can be fully utilized, the accuracy of the trained model is ensured from the quality of the training data, and then the algorithm based on the training model can output better images, so that the problem of poor image quality output by the algorithm matched with the image sensor of the QPD array in the prior art is solved.

Drawings

It will be appreciated by those skilled in the art that the drawings are provided for a better understanding of the invention and do not constitute any limitation to the scope of the invention. Wherein:

FIG. 1 is a flow diagram illustrating a data generation method according to an embodiment of the invention;

FIG. 2a is a schematic diagram of a QPD array pixel according to one embodiment of the present invention;

FIG. 2b is a schematic diagram of a 4Cell array according to an embodiment of the invention;

FIG. 3 is a schematic diagram of a four-in-one alternative process according to an embodiment of the invention;

FIG. 4a is a schematic diagram of a training image according to an embodiment of the present invention;

FIG. 4b is a schematic illustration of a training image obtained after horizontal mirroring of the training image shown in FIG. 4 a;

FIG. 4c is a schematic illustration of the training image shown in FIG. 4a after vertical mirroring;

FIG. 4d is a schematic illustration of the training image shown in FIG. 4a after horizontal mirroring and 90 counterclockwise rotation;

FIG. 4e is a schematic illustration of the training image shown in FIG. 4a after being vertically mirrored and rotated 90 counterclockwise;

FIG. 5a is the original output image of the QPD array under one condition according to one embodiment of the present invention;

FIG. 5b is an output image resulting from processing the raw output image shown in FIG. 5a with a comparison algorithm;

FIG. 5c is an output image obtained after the original output image shown in FIG. 5a is processed by the imaging method according to an embodiment of the present invention;

FIG. 6a is the original output image of the QPD array under another condition of the present invention;

FIG. 6b is an output image resulting from processing the raw output image shown in FIG. 6a with a comparison algorithm;

fig. 6c is an output image obtained by processing the original output image shown in fig. 6a by the imaging method according to an embodiment of the present invention.

Detailed Description

To further clarify the objects, advantages and features of the present invention, a more particular description of the invention will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. It is to be noted that the drawings are in greatly simplified form and are not to scale, but are merely intended to facilitate and clarify the explanation of the embodiments of the present invention. Further, the structures illustrated in the drawings are often part of actual structures. In particular, the drawings may have different emphasis points and may sometimes be scaled differently.

As used in this application, the singular forms "a", "an" and "the" include plural referents, the term "or" is generally employed in a sense including "and/or," the terms "a" and "an" are generally employed in a sense including "at least one," the terms "at least two" are generally employed in a sense including "two or more," and the terms "first", "second" and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicit to the number of technical features indicated. Thus, features defined as "first", "second" and "third" may explicitly or implicitly include one or at least two of the features, "one end" and "the other end" and "proximal end" and "distal end" generally refer to the corresponding two parts, which include not only the end points, but also the terms "mounted", "connected" and "connected" should be understood broadly, e.g., as a fixed connection, as a detachable connection, or as an integral part; can be mechanically or electrically connected; either directly or indirectly through intervening media, either internally or in any other relationship. Furthermore, as used in the present invention, the disposition of an element with another element generally only means that there is a connection, coupling, fit or driving relationship between the two elements, and the connection, coupling, fit or driving relationship between the two elements may be direct or indirect through intermediate elements, and cannot be understood as indicating or implying any spatial positional relationship between the two elements, i.e., an element may be in any orientation inside, outside, above, below or to one side of another element, unless the content clearly indicates otherwise. The specific meanings of the above terms in the present invention can be understood according to specific situations by those of ordinary skill in the art.

The core idea of the invention is to provide a data generation method, a training method, an imaging method, a storage medium and a shooting device, so as to solve the problem that the quality of an image output by an algorithm matched with an image sensor of a QPD array is poor in the prior art.

The following description refers to the accompanying drawings.

Referring to fig. 1 to fig. 6c, fig. 1 is a schematic flow chart illustrating a data generating method according to an embodiment of the invention; FIG. 2a is a schematic diagram of a QPD array pixel according to one embodiment of the present invention; FIG. 2b is a schematic diagram of a pixel of a 4Cell array according to an embodiment of the invention; FIG. 3 is a schematic diagram of a four-in-one alternative process of an embodiment of the present invention; FIG. 4a is a schematic diagram of a training image according to an embodiment of the present invention; FIG. 4b is a schematic illustration of a training image obtained after horizontal mirroring of the training image shown in FIG. 4 a; FIG. 4c is a schematic illustration of a training image obtained by vertically mirroring the training image shown in FIG. 4 a; FIG. 4d is a schematic illustration of the training image shown in FIG. 4a after horizontal mirroring and 90 counterclockwise rotation; FIG. 4e is a schematic illustration of the training image shown in FIG. 4a after being vertically mirrored and rotated 90 counterclockwise; FIG. 5a is the original output image of the QPD array under one condition according to one embodiment of the present invention; FIG. 5b is an output image resulting from processing the raw output image shown in FIG. 5a with a comparison algorithm; FIG. 5c is an output image obtained by processing the original output image shown in FIG. 5a according to an embodiment of the present invention; FIG. 6a is the original output image of the QPD array under another condition according to one embodiment of the present invention; FIG. 6b is an output image resulting from processing the raw output image shown in FIG. 6a with a comparison algorithm; fig. 6c is an output image obtained by processing the original output image shown in fig. 6a by the imaging method according to an embodiment of the present invention.

As shown in fig. 1, the present embodiment provides a data generation method for generating a training image of an imaging model of an image sensor of a QPD array. It will be appreciated that the imaging model is a neural network based model.

The data generation method comprises the following steps: the training image is derived based on output images of the image sensors of the 4Cell array.

Specifically, the data generation method includes:

s104, shooting by an image sensor of the Cell array under a preset working condition;

s204, obtaining an intermediate image by the original output image of the image sensor of the Cell array through preset operation;

s30 cutting the intermediate image to obtain the output contrast image;

s40, replacing four pixels of the same pixel group in the output contrast image with a four-in-one pixel to obtain the input image;

s50, carrying out mirror image and/or rotation operation on the output contrast image to obtain an additional output contrast image;

s60 performs a mirroring and/or rotation operation on the output contrast image to obtain additional output contrast images.

In step S10, the preset conditions include a preset shooting environment, a preset shooting angle, a preset focusing degree, a preset shooting object, and the like.

The pixel schematic of the QPD array is shown in fig. 2a, wherein a circle represents a micro lens (micro lens). For a QPD array, there are four photosites under one microlens, each represented by a square, one photosite for each pixel. The letters Bq, Gq and Rq in each square represent the color channel of this light-sensing unit, Bq representing blue, Gq representing green and Rq representing red. The pixel diagram of the 4Cell array is shown in fig. 2b, and the circles, squares and text have the same meaning as in fig. 2 a. The 4Cell array differs from the QPD array in that four photosensitive cells of the QPD array share one microlens, which can produce pixels with phase information; the four light sensing units of 4Cell correspond to four micro-lenses, which cannot produce pixels with phase information, but the light sensing performance is better and more balanced.

As can be seen from fig. 2a and 2b, if 4 pixels under one microlens of the QPD array correspond to 4 pixels in the 4Cell array, the data formats (including the color channel of each dot and the brightness value of each dot) of the two are completely the same, but since each photosensitive Cell of the 4Cell array is independent, more accurate data can be obtained, and therefore, the output data of the 4Cell array can be used as a training image of the imaging model.

It is to be understood that the training images include input images and output contrast images, and the training goal of the imaging model is that, when an input image is input to the imaging model, the output image needs to be the same as or close to the output contrast image corresponding to the input image. The output contrast image may be obtained based on a raw output image of a 4Cell array of image sensors, and the input image may be obtained by a four-in-one replacement of the raw output image.

In step S20, the purpose of the preset operation is to perform specific calculation to eliminate interference of ambient light, signal noise, etc., and in one embodiment, the preset operation includes white balance. In some embodiments, the preset operation may also be a no operation. The output contrast image may be obtained based on the intermediate image obtained in step S20, and in order to ensure the number of training samples and ensure the training speed of a single picture, the intermediate image may be cut to obtain a sufficient number of output contrast images. In other embodiments, the output contrast image may be derived based on other logic. The replacing step in step S40 can be understood with reference to fig. 3, where in the replacing step, the color channel of the four-in-one pixel is the same as the color channel of the corresponding pixel group, and the brightness value of the four-in-one pixel is the average value of the brightness values of the four pixels of the corresponding pixel group. The adjacent pixels with the same color channel belong to the same pixel group. For example, in fig. 2b, four pixels labeled Bq are considered as the same pixel group. In order to further increase the number of samples and improve the generalization ability of the imaging model, performing mirroring and/or rotation operations on at least a portion of the output contrast image to obtain an additional output contrast image; and simultaneously, carrying out mirror image and/or rotation operation on the corresponding output contrast image to obtain an additional output contrast image. In one embodiment, each of the output contrast images and each of the output contrast images are subjected to four operations, namely horizontal mirroring, vertical mirroring, horizontal mirroring followed by 90 ° counterclockwise rotation, vertical mirroring followed by 90 ° counterclockwise rotation, and finally the number of samples obtained is 5 times that before the operation. The above operation can be understood with reference to fig. 4a to 4e, wherein each figure can be understood as the output contrast image or the input image. That is, in step S30, a part of the output contrast image is obtained by cutting, and in step S50, another part of the output contrast image is obtained. In step S40, a part of the input image is obtained, and in step S60, another part of the input image is obtained.

In a preferred embodiment, the output contrast image has a size of 2N × 2N and the input image has a size of N × N. In some other embodiments, the output contrast image and the input image may also be non-equilateral rectangles.

Considering that there may be a certain correlation between each pixel and surrounding pixels, and that an excessively large range is not favorable for a single learning process, N is an odd number greater than or equal to 13, and a specific value may be set according to an actual situation, and since the larger the value of N is, the larger the calculation amount is also increased, in a preferred embodiment, the value range of N is 13 to 63, and is preferably 17.

The present embodiment also provides a training method for training an imaging model of an image sensor of a QPD array, the training method including: and obtaining the training image based on the data generation method, and training the imaging method based on the training image.

Specifically, in an embodiment, the training method specifically includes:

step 1, collecting original output images of 4Cell array image sensors with multiple angles, multiple distances and different focusing degrees, wherein a shooting scene comprises an indoor resolution card, a color card, outdoor data and the like, and the data size is 4672 multiplied by 3504.

And step 2, preprocessing 4Cell original data, enhancing the data and generating a large number of training samples.

White balance processing is carried out on the acquired 4Cell original data, the processed image is cut into a large number of small images (marked by P) with the size of 34 multiplied by 34, and a four-in-one image (marked by P-q) corresponding to each small image is calculated with the size of 17 multiplied by 17. The color channel of the four-in-one pixel is the same as the color channel of the corresponding pixel group, and the brightness value of the four-in-one pixel is the average value of the brightness values of the four pixels of the corresponding pixel group

Every (P-q, P) is taken as a sample, and 8,364,269 samples constitute a data set.

In order to expand the data volume and improve the generalization capability of the training model, data enhancement processing such as mirror image inversion is performed on each sample, the number of the expanded data set samples is 5 × 8,364,269, and 5 × (8,364,269 × 20%) samples are randomly selected for training each time. The mirror-image flip diagram can be understood with reference to fig. 4a to 4 e.

And 3, designing a convolutional neural network model, training the convolutional neural network model, and optimizing the network to obtain the optimal network model.

Training input: p-q plot, size 17X 17;

training output comparison graph: p map, size 34 × 34;

The model parameter quantity is 1.98MB, the multiplication operand is 150million, and the addition operand is 150 million;

the learning ability and precision of the network are improved by using the residual block and the long and short jump link;

and obtaining an optimal network model by adopting an adaptive moment estimation (Adam) optimization method.

The above-mentioned residual block, long and short skip chaining and Adam optimization method are all common knowledge in the art, and are not described in detail herein.

And 4, after the imaging model is obtained through the training method, placing the original output image of the image sensor of the QPD array into an optimal network model for testing to obtain a convolution neural network result.

Test input: a four-in-one map of the image sensor of the QPD array, with a size of m × m;

and outputting a result: the convolutional neural network results, size 2m × 2 m.

The raw output image (size 16mp, size 4672 × 3504) of a complete QPD array of image sensors was input into the imaging model to test its response speed, and the test results are shown in table 1.

TABLE 1 test time

As can be seen from Table 1, the computation time of the trained model is acceptable, and meets the operation requirements of normal working conditions.

The embodiment also provides an imaging method, wherein a processing result is obtained by processing an original output image of an image sensor of a QPD array based on the imaging model, and the imaging model is obtained based on the training method; and obtaining an image based on the processing result.

The imaging method specifically comprises the following steps: and inputting an original output image of an image sensor of the QPD array into the imaging model, and outputting a bmp (bit map) map from a result output by the imaging model through a general ISP flow.

In order to demonstrate the effect of the imaging method, the present embodiment further provides a comparison algorithm, where the comparison algorithm includes the steps of inputting a raw output image of the image sensor of the QPD array to a model obtained based on parallax, and outputting a bmp map from a result output by the model through a general ISP process.

The difference in the effect of the two algorithms can be observed by comparing fig. 5a to 5c, and fig. 6a to 6 c.

The present embodiment also provides a storage medium, where the storage medium stores a program, and when the program runs, the program executes the above data generation method, the above training method, or the above imaging method.

The present embodiment further provides a shooting device, where the shooting device includes an image sensor and an imaging unit, where the arrangement array of the image sensor is a QPD array, and the imaging unit is configured to process an original output image of the image sensor based on the above-mentioned imaging method to obtain an output image. Other components, connection modes and operation principles of the photographing device can be set by those skilled in the art according to actual needs and common knowledge, and are not described in detail herein.

In summary, in the data generating method, the training method, the imaging method, the storage medium, and the capturing apparatus provided in this embodiment, the data generating method obtains the training image based on the output image of the image sensor of the 4Cell array. By the configuration, the similarity of the 4Cell array and the QPD array on the structure and the output data can be fully utilized, the accuracy of the trained model is ensured according to the quality of the training data, and further, the algorithm based on the training model can output better images, and the problem that the image quality output by the algorithm matched with the image sensor of the QPD array is poor in the prior art is solved.

The above description is only for the purpose of describing the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention, and any variations and modifications made by those skilled in the art according to the above disclosure are within the scope of the present invention.

Claims

1. A data generation method for generating a training image of an imaging model of an image sensor of a QPD array, the data generation method comprising:

4, shooting by an image sensor of the Cell array under a preset working condition;

obtaining the training image based on an output image of an image sensor of the 4Cell array;

Wherein the training image comprises an input image and an output contrast image, the data generation method further comprising:

obtaining an intermediate image from an original output image of the image sensor of the 4Cell array through a preset operation;

obtaining the output contrast image based on the intermediate image, specifically including: cutting the intermediate image to obtain the output contrast image;

replacing four pixels of the same pixel group in the output contrast image with a four-in-one pixel to obtain the input image, wherein during replacement, the color channel of the four-in-one pixel is the same as the color channel of the corresponding pixel group, and the brightness value of the four-in-one pixel is the average value of the brightness values of the four pixels of the corresponding pixel group;

the size of the output contrast image is 2N × 2N, and the size of the input image is N × N.

2. The data generation method of claim 1, wherein the data generation method comprises:

performing mirror image and/or rotation operation on the output contrast image to obtain an additional output contrast image; and (c) a second step of,

and carrying out corresponding operation on the input image to obtain an additional input image.

3. The data generation method according to claim 1 or 2, wherein N is an odd number greater than or equal to 13.

4. The data generation method of claim 3, wherein N is between 13 and 63.

5. A training method for training an imaging model of an image sensor of a QPD array, the training method comprising:

obtaining the training image based on the data generating method according to any one of claims 1 to 4,

training the imaging model based on the training images.

6. An imaging method, characterized in that the imaging method comprises:

processing raw output images of an image sensor of a QPD array based on the imaging model obtained based on the training method of claim 5 to obtain a processing result;

and obtaining an image based on the processing result.

7. A storage medium having stored thereon a program which, when executed, performs the data generation method of any one of claims 1 to 4, the training method of claim 5, or the imaging method of claim 6.

8. A camera device, comprising an image sensor and an imaging unit, wherein the arrangement array of the image sensor is a QPD array, and the imaging unit is configured to process a raw output image of the image sensor to obtain an output image based on the imaging method according to claim 6.