CN111260580B

CN111260580B - Image denoising method, computer device and computer readable storage medium

Info

Publication number: CN111260580B
Application number: CN202010051403.4A
Authority: CN
Inventors: 杨帆; 彭刚
Original assignee: Allwinner Technology Co Ltd
Current assignee: Allwinner Technology Co Ltd
Priority date: 2020-01-17
Filing date: 2020-01-17
Publication date: 2023-05-12
Anticipated expiration: 2040-01-17
Also published as: CN111260580A

Abstract

The invention provides an image pyramid-based image denoising method, a computer device and a computer readable storage medium, wherein the method comprises the steps of obtaining an initial image and calculating an output chromaticity value of each pixel: acquiring a layered search area of a pixel to be denoised, performing downsampling on the layered search area, and acquiring a layer of downsampled image after each downsampling, wherein the layers of downsampled images form a first image pyramid; up-sampling each down-sampling image, and subtracting the up-sampled image from the next layer image of the down-sampling image to obtain a layer of subtracted image, wherein the multiple layers of subtracted images form a second image pyramid; carrying out mean denoising on each layer of subtracted image of the second image pyramid and the highest layer of downsampled image of the first image pyramid to obtain a denoising colorimetric value; an output chroma value is calculated using the initial chroma value and the de-noised chroma value. The invention also provides a computer device for realizing the method and a computer readable storage medium. The invention can reduce the calculated amount of image denoising and the hardware realization difficulty, and improve the denoising quality.

Description

Image denoising method, computer device and computer readable storage medium

Technical Field

The invention relates to the technical field of image processing, in particular to an image denoising method based on an image pyramid, a computer device for realizing the method and a computer readable storage medium.

Background

Many existing intelligent electronic devices have an image capturing function, for example, a smart phone, a tablet computer, a vehicle recorder and the like are all provided with an image capturing device, and the image capturing device is usually provided with a CMOS sensor to obtain an image. Typically, an image includes a large number of pixels, and color information of each pixel may be represented by an RGB value or a YUV value.

For example, a CMOS image sensor currently in common use generally adopts a BAYER arrangement format, and color information of each pixel is usually RGB, but the RGB is not three primary colors, and color distortion occurs in an image, so that a "demosaicing" process is required for each pixel to obtain RGB three primary colors information, so as to restore the original colors of the image.

As image resolution increases, the amount of light sensed by individual pixels decreases, and low-light scenes are increasingly used, image noise output by CMOS image sensors increases greatly. In the process of converting the RAW image into the RGB image, the demosaicing operation needs to refer to all image pixels in a certain area to obtain RGB three primary colors of one pixel, noise of each color component is diffused mutually in a large range, so that a large block (from a few pixels to hundreds of pixels) of color spots appear in a final image, and the visual perception of human eyes is seriously influenced. Therefore, it is generally necessary to perform denoising processing on an image output from the CMOS image sensor.

The information in the image has a high degree of correlation, and each pixel is not isolated, and has not only similarity of brightness, but also similarity of image texture structures. The Non-Local mean (NLM) method is a denoising method based on image Local similarity, and on the basis of the neighborhood average denoising method, the similarity weight coefficient of the NLM method is determined by the similarity between the pixel point to be denoised currently and the image slices with other pixel points in the neighborhood as the center, the calculation of the weight is not substantially related to the spatial positions of two pixels, and the similarity of the two image slices is only related, so that false information can be well avoided. Since the noise of the image can be equivalent to additive Gaussian white noise, the weighted average similar pixels can remove the noise of the image better. The NLM method has the characteristics of simple algorithm, excellent performance and easiness in improvement and expansion, and becomes one of the main methods for denoising the current image.

The basic principle of the NLM method is as follows: taking a neighborhood window with a pixel point to be denoised as a center, taking an image area with a certain surrounding size as a search area, namely M x M (the whole image is optional, but the calculated amount is too large and is generally smaller than 41), and searching similar image blocks in the search area, wherein the common method is to calculate Gaussian weighted Euclidean distance between the image blocks, and the calculation is carried out by using the following formula:

wherein Ga is Gaussian kernel with standard deviation a, and u (Ni) and u (Nj) are corresponding pixels in the image block of the central window and the image block of the search area respectively. Then, the denoised pixel estimation value is calculated using the following formula:

wherein w (i, j) =exp (-d (i, j)/h) ² ) As the weight coefficient of the light-emitting diode,

for normalization factor, Ω (i) represents the search area of the center pixel i, h is a similar weight parameter that determines the degree of balance after denoising the image.

Currently, the denoising method based on the non-local mean value mainly comprises the following three modes:

the first way is to transform the image, and denoise the image in the transform domain instead of the space domain by using the NLM method, which specifically comprises the following steps: transforming the initial image, such as wavelet transform, contourlet transform, laplace transform, etc., and decomposing into a low-frequency image and a high-frequency image; and then, denoising or correcting the low-frequency image and the high-frequency image respectively by using an NLM method, and finally, reconstructing and inversely transforming the obtained low-frequency image and high-frequency image to obtain a denoised image.

The second mode is to adaptively select the neighborhood block size and the weight coefficient of the NLM method by using image information, and the specific steps are as follows: firstly, performing pixel coarse classification on a noise image by using an evaluation operator; then, for each pixel in the noise image, classifying the class of the pixel into low-noise high-texture, medium-texture, high-noise secondary texture, smooth area and the like according to the rough classification result of the neighborhood pixels around the pixel; and finally, adaptively selecting filtering parameters and the size of a neighborhood block for each classified class, and carrying out pixel denoising by using a non-local mean denoising algorithm.

The third mode is a BM3D denoising method, which is improved to implement real-time video denoising, and specifically includes the steps of: dividing an image into a plurality of N multiplied by N pixel blocks, selecting N pixel blocks which are most matched with a reference block in a certain area of a current frame and a plurality of frames around the reference block to form an N multiplied by N three-dimensional pixel array, performing three-dimensional discrete cosine transform, hard threshold filtering and three-dimensional discrete cosine inverse transform on the three-dimensional pixel array, and performing reconstruction processing on an inverse transform result to obtain a denoising result.

However, the non-local mean-based denoising method has several problems:

first, the denoising result of the algorithm is closely related to the size of a denoising local window, when the image noise is large, a large local window is needed to obtain a good denoising effect, but the complexity of the algorithm and the cost of hardware realization are increased obviously. For scenes with low signal-to-noise ratio such as low illumination, the search area needs at least 33×33, which means that for each pixel to be denoised, more than 1088 similar image block matching operations and other related operations are needed, so that the calculation amount is very large, and the real-time denoising and hardware implementation are not facilitated.

Secondly, when the image noise is large, the NLM algorithm can effectively remove low-frequency noise, but high-frequency details of the image can be lost at the same time, so that the image definition is reduced. This is because the search area is larger, and similar image blocks are more, so that some pseudo-similar pixels participate in the denoising process, and the image texture is weakened.

Third, the current NLM denoising method has no complete technical scheme for RAW image denoising. The current photographing device is generally a CMOS camera, the output format is a RAW format, a Bayer color filter array is generally adopted, each pixel point only has one color information of R, G, B, and the number of pixels of G is twice as large as R, B.

Disclosure of Invention

The invention mainly aims to provide an image denoising method based on an image pyramid, which reduces the calculated amount of intelligent electronic equipment and has better denoising effect.

Another object of the present invention is to provide a computer apparatus for implementing the image denoising method based on image pyramid.

It is still another object of the present invention to provide a computer readable storage medium implementing the above image pyramid-based image denoising method.

In order to achieve the main purpose of the invention, the image denoising method based on the image pyramid provided by the invention comprises the steps of obtaining an initial image and calculating the output chromaticity value of each pixel of the initial image; wherein calculating the output chromaticity value for each pixel comprises: acquiring a layered search area of pixels to be denoised, acquiring initial chromaticity values of each pixel in the layered search area, performing at least one downsampling on the layered search area, acquiring a layer of downsampled image after each downsampling, and forming a first image pyramid by the multiple layers of downsampled images; up-sampling each down-sampling image, and subtracting the up-sampled image from the next layer image of the down-sampling image to obtain a layer of subtracted image, wherein the multiple layers of subtracted images form a second image pyramid; carrying out mean denoising on each layer of subtracted image of the second image pyramid and the highest layer of downsampled image of the first image pyramid to obtain a denoising colorimetric value; an output chroma value is calculated using the initial chroma value and the de-noised chroma value.

According to the scheme, the number of pixels of each layer of image in the image pyramid is smaller than that of pixels of the initial image, so that the image pyramid is used for denoising calculation, the calculated amount of denoising calculation can be greatly reduced, and the hardware implementation cost of the intelligent electronic equipment is reduced. In addition, each layer of image of the second image pyramid is obtained by calculating two adjacent layers of images of the first image pyramid, so that a high layer image of the second image pyramid can reflect small-scale image texture information, a high layer image of the second image pyramid can reflect large-scale image edge information, and the image is spatially divided into different frequency bands by utilizing a pyramid layering mode, thereby being beneficial to follow-up fine denoising treatment and enabling denoising effect to be more ideal.

In a preferred embodiment, obtaining the denoising color value comprises: and setting a denoising search area of each subtracted image and the highest-layer downsampled image, calculating a preset distance value in a preset range, and calculating the average value of the chromaticity values of all pixels with the preset distance value smaller than a noise threshold value.

Therefore, the distance value of the denoising search area in the preset range can be conveniently calculated by setting the noise threshold value, which pixels can be used for the chromaticity value can be conveniently determined, which pixels can not be used for the chromaticity value, namely, the image blocks with relatively close colors are determined to be used as denoising references by searching.

Further, the noise threshold is directly related to the characteristic value of the image sensor and/or the brightness value of the pixel.

In this way, the set noise threshold can reflect the characteristics of the image sensor, and the image can be subjected to targeted denoising according to the characteristics of the image sensor. Moreover, since the color noise has a certain relation with the brightness of the pixels, for example, the color noise may be generated due to insufficient illumination, and thus the noise threshold is related to the brightness value of the pixels, the color authenticity of the denoised image can be improved

In a further aspect, the denoising search area of the at least one layer of subtracted image is larger than the denoising search area of the highest layer of downsampled image.

It can be seen that, according to the size of each layer of image, an adaptive denoising search area is determined, for example, if the pixels of a certain layer of image are more, a larger search area is set, and if the pixels are less, a smaller search area is set, so that the denoising quality of the image is improved.

In a preferred embodiment, calculating the preset distance value within the preset range, and calculating the average value of the chrominance values of all pixels whose preset distance value is smaller than the noise threshold value includes: and calculating the preset distance value between the denoising search area of each layer of the subtracted image and the preset range, calculating the preset distance value between the denoising search area of the highest layer of the downsampled image and the preset range, and calculating the average value of the chromaticity values of all pixels of each layer, wherein the preset distance value of each layer is smaller than the noise threshold value.

It can be seen that the calculation of the preset distance value is performed for each layer of image, and the average value of the chromaticity values of the pixels of each layer is calculated, and more parameters can be used in calculating the output chromaticity values of the pixels, thereby improving the accuracy of the denoising calculation.

Still further, calculating the output chroma value using the initial chroma value and the de-noised chroma value includes: and calculating an output chromaticity value by using the initial chromaticity value, the denoising chromaticity value of each layer and a preset weighting coefficient.

Therefore, the output chroma value is calculated through the preset weighting coefficient, so that the proper weighting value can be determined according to the specific gravity of the denoising chroma value of different layers, the accuracy of calculating the output chroma value is improved, the denoised chroma value is more similar to the true color, and the denoising quality is improved.

Further, the preset distance value is one of a simplified euclidean distance, a manhattan distance, a Min Shi distance, a chebyshev distance and a cosine distance.

It can be seen that the simplified euclidean distance, manhattan distance, min Shi distance, chebyshev distance, and cosine distance are all common distance values used for image filtering calculation, and the workload of image denoising calculation can be simplified by using the distance values.

Further, when the hierarchical search area is downsampled, gaussian filtering calculation is performed; and/or performing a gaussian filter calculation when upsampling the downsampled image.

Therefore, the Gaussian filtering is carried out during up-sampling and down-sampling, the smoothness of the layered search area image can be improved, and the denoising calculation effect is more ideal.

To achieve the above another object, the present invention provides a computer apparatus including a processor and a memory, wherein the memory stores a computer program, and the computer program when executed by the processor implements the steps of the image pyramid-based image denoising method.

To achieve still another object of the present invention, there is provided a computer program stored on a computer readable storage medium, which when executed by a processor, implements the steps of the image pyramid-based image denoising method described above.

Drawings

FIG. 1 is a flow chart of an embodiment of an image pyramid-based image denoising method of the present invention.

Fig. 2 is a schematic diagram of a conventional RAW image arrangement in four BAYER formats.

Fig. 3 is a flowchart of image pyramid computation in an embodiment of an image pyramid-based image denoising method according to the present invention.

FIG. 4 is a flow chart of non-local mean denoising calculation in an embodiment of an image pyramid-based image denoising method of the present invention.

The invention is further described below with reference to the drawings and examples.

Detailed Description

The image pyramid-based image denoising method is applied to intelligent electronic equipment, and preferably, the intelligent electronic equipment is provided with an image pickup device, such as a camera, and the like, wherein the image pickup device is provided with an image sensor, such as a CMOS (complementary metal oxide semiconductor), and the intelligent electronic equipment acquires an initial image by using the image pickup device. Preferably, the intelligent electronic device is provided with a processor and a memory, wherein the memory stores a computer program, and the processor implements an image denoising method based on an image pyramid by executing the computer program.

Image pyramid-based image denoising method embodiment:

the embodiment mainly aims at an initial image acquired by an image sensor to carry out a denoising method, specifically, the embodiment takes an image area with an N multiplied by N range of a pixel center to be denoised as a processing area on an original RAW image, carries out repeated downsampling on pixels of the area to form a Gaussian pyramid image, can obtain 2-M layers (M is determined according to the area size) of Gaussian pyramid images according to the area size, then processes the Gaussian pyramid images to obtain 1-M-1 layers of Laplace pyramid images, carries out denoising calculation on the central pixels on each layer of Laplace pyramid images by using an NLM method, and finally synthesizes to obtain a final denoised image. The texture of the image is reserved by the method, the high-quality denoising image can be obtained, the denoising calculation amount is less, and the requirement on realizing hardware is lower.

The present embodiment will be described in detail with reference to fig. 1. First, step S1 is performed to acquire an initial image, and the initial image is preprocessed. In the present embodiment, the initial image is an image output by the CMOS image sensor, and the color information of the initial image is RGB information. Since the present embodiment is directed to processing an image having YCbCr information, if an initial image output by a CMOS image sensor is an RGB image, the initial image needs to be preprocessed to obtain YCbCr information for each pixel. Where Y is the luminance value of the pixel, cb is the blue chrominance value of the pixel, and Cr is the red chrominance value of the pixel. The present embodiment processes color noise of an image based on Cb and Cr of each pixel.

In step S1, a RAW image pixel preprocessing is required to convert the original RAW image into an approximate luminance and chrominance image. Referring to fig. 2, fig. 2 is a four-bar format arrangement of the most common RAW image, and for each pixel in the figure, approximate luminance Y and chrominance Cr/Cb information is obtained using pixels in a 3×3 region around it.

Specifically, for the format arrangement of fig. 2 (a), the following formula may be used for calculation:

Y＝(4×G11+B01+R10+R12+B21)/16

Cr＝(R10+R12)/2

cb= (B01+B21)/2 (formula 3)

For the format arrangement of fig. 2 (b), the following formula can be used for calculation:

Y＝(4×G11+B10+R01+R21+B12)/16

Cr＝(R01+R21)/2

cb= (B10+B12)/2 (formula 4)

For the format arrangement of fig. 2 (c), the following formula can be used for calculation:

Y＝(4×R11+B00+B02+B20+B22+2(G01+G10+G12+G21))/16

Cr＝R11

cb= (B00+B02+B20+B22)/4 (formula 5)

For the format arrangement of fig. 2 (d), the following formula can be used for calculation:

Y＝(4×B11+R00+R02+R20+R22+2(G01+G10+G12+G21))/16

Cr＝(R00+R02+R20+R22)/4

cb=b11 (6)

After preprocessing the image, step S2 is executed to obtain a hierarchical search area of the pixels to be denoised. In this embodiment, denoising operation is required for each pixel of the image, so that a pixel in the image is acquired first, and the pixel is the current pixel to be denoised. In this embodiment, an area where m×m pixels centered on a pixel to be denoised are located is used as a hierarchical search area, and this embodiment is described with a value of M being 33.

Then, step S3 is performed to generate a gaussian image pyramid, that is, a first image pyramid, based on the pixels of the hierarchical search region. Specifically, image pyramid processing is performed on color information of pixels in the hierarchical search area, that is, pyramid processing is performed on Y/Cr/Cb information of the pixels. Referring to fig. 3, the following describes in detail the steps of pyramid processing of luminance value information of pixels, that is, Y information, in the same manner as the processing of luminance value information, red color density information Cr and blue color density information Cb.

First, step S21 is performed to acquire chromaticity value information, i.e., Y information, of each pixel in the hierarchical search region, and step S22 is performed to use the entire layer of pixels in the hierarchical search region as the first layer image of the gaussian pyramid, and since the range of the hierarchical search region is 33×33, the range of the first layer image of the gaussian pyramid is also 33×33.

Then, step S23 is performed to perform gaussian filtering for each pixel in the hierarchical search area, where the gaussian filtering uses a 5×5 gaussian kernel with sigma=1.4, and the filtering template is:

then, step S24 is performed to downsample the hierarchical search area after the gaussian filtering, and 1/2 downsampling is specifically performed. Specifically, the 1/2 downsampling method uses the pixel to be denoised as a center point, and retains pixels of even rows and even columns, namely pixels of odd rows and odd columns are removed. The downsampling is a sampling mode that the number of the sampled pixels is smaller than that of the original image, and the 1/2 downsampling is a sampling mode that the number of the pixel rows and the number of the columns of the sampled image are 1/2 of that of the original image.

After the first 1/2 downsampling, a gaussian pyramid second layer image is obtained, that is, step S25 is performed, and it can be seen that the number of rows and columns of pixels of the gaussian pyramid second layer image are 17, so that the range of the gaussian pyramid second layer image is 17×17.

And so on, continuing to perform Gaussian filtering on the second-layer image of the Gaussian pyramid, namely performing the step S29, filtering by using the filtering template adopted in the step S23, performing the step S32, and performing 1/2 downsampling on the second-layer image of the Gaussian pyramid to obtain the third-layer image of the Gaussian pyramid, namely performing the step S33, wherein the number of rows and the number of columns of pixels of the second-layer image of the Gaussian pyramid are 9, so that the range of the third-layer image of the Gaussian pyramid is 9×9.

And continuing to perform Gaussian filtering on the image of the third layer of the Gaussian pyramid, namely performing step S35, filtering by using a filtering template adopted in step S23, performing step S38, and performing 1/2 downsampling on the image of the third layer of the Gaussian pyramid to obtain the image of the fourth layer of the Gaussian pyramid, namely performing step S39, wherein the number of rows and columns of pixels of the image of the fourth layer of the Gaussian pyramid are 5, so that the range of the image of the fourth layer of the Gaussian pyramid is 5×5.

In this embodiment, each layer of the gaussian image pyramid is an image obtained by downsampling, so that each layer of the gaussian image pyramid is a downsampled image.

After obtaining the gaussian pyramid image, step S4 is performed, and the laplacian image pyramid is obtained through the gaussian image pyramid. Specifically, up-sampling is performed on each downsampled image of the gaussian image pyramid, a layer of subtraction image is obtained after the up-sampled image is subtracted from the next layer of image of the downsampled image, and a second image pyramid, namely a laplacian pyramid image, is formed by the multiple layers of subtraction images.

For example, step S26 is performed for a gaussian pyramid second-layer image, which is up-sampled 2 times. The up-sampling mode is adopted, wherein the number of the sampled pixels is larger than that of the original image, and the up-sampling mode is adopted, wherein the number of the pixel rows and the number of the columns of the sampled image are 2 times that of the original image. Typically, upsampling requires interpolation operations.

In this embodiment, when up-sampling an image of a gaussian pyramid, gaussian filtering is required, for example, filtering is performed using a 5×5 gaussian kernel with sigma=1.4, and the filtering templates are:

after up-sampling the image of the second layer of the gaussian pyramid by 2 times, step S27 is executed, and the subtraction operation is performed on the image obtained by up-sampling the image of the first layer of the gaussian pyramid and the image of the second layer of the gaussian pyramid, so as to obtain the image of the first layer of the laplacian pyramid, that is, step S28 is executed. Since the range of the gaussian pyramid first layer image is 33×33, the range of the gaussian pyramid second layer image after upsampling is also 33×33, and thus the range of the laplacian pyramid first layer image is also 33×33.

And so on, up-sampling is also performed on the image of the third layer of the Gaussian pyramid by 2 times, and the subtraction operation is performed on the image obtained by up-sampling the image of the second layer of the Gaussian pyramid and the image of the third layer of the Gaussian pyramid, that is, the step S30 is performed, so that the image of the second layer of the Laplacian pyramid is obtained, that is, the step S31 is performed. Since the range of the gaussian pyramid second layer image is 17×17, the range of the gaussian pyramid third layer image after upsampling is also 17×17, and thus the range of the laplacian pyramid second layer image is also 17×17.

Finally, up-sampling is also performed on the image of the fourth layer of the gaussian pyramid by 2 times, and the subtraction operation is performed on the image of the third layer of the gaussian pyramid and the image obtained by up-sampling the image of the fourth layer of the gaussian pyramid, that is, step S36 is performed, so that the image of the third layer of the laplacian pyramid is obtained, that is, step S37 is performed. Since the range of the image of the third layer of the gaussian pyramid is 9×9, and the up-sampled range of the image of the fourth layer of the gaussian pyramid is 9×9, the range of the image of the third layer of the laplacian pyramid is 9×9. Therefore, each layer of image of the Laplacian image pyramid is obtained by subtracting images of two layers of Gaussian image pyramids, each layer of image of the Laplacian image pyramid is a layer of subtraction image, and the multiple layers of subtraction images form the Laplacian image pyramid serving as the second image pyramid. It should be noted that, in the downsampling and upsampling, the use of gaussian filtering is not necessary, i.e., gaussian filtering is not performed, or similar single peak functions can be used to achieve similar effects.

After the processing, the Laplace image pyramid with a three-layer structure is obtained, the high-level image of the Laplace image pyramid can reflect small-scale image texture information, and the low-level image can reflect large-scale image edge information, so that the pyramid layering mode is used for dividing the image into different frequency bands in space, and the subsequent fine denoising processing is facilitated.

Then, step S5 is executed to perform non-local mean denoising calculation on each layer image of the laplacian image pyramid and the highest layer image (fourth layer) of the gaussian image pyramid. Specifically, referring to fig. 4, step S51 is performed first to obtain each layer of image of the laplacian image pyramid and the highest layer of image of the gaussian image pyramid, and then step S52 is performed to obtain a denoising search area corresponding to each layer of image.

Taking the luminance value Y information as an example, the denoising search areas corresponding to the first, second and third layer images of the laplacian image pyramid and the highest layer image of the gaussian image pyramid are set to be 9×9, 5×5 and 5×5, respectively. It can be seen that the first denoising search region of the laplacian image pyramid is larger than the denoising search region of the highest-level image of the gaussian image pyramid.

Next, step S53 is performed to calculate a preset distance value in each denoising search region within a preset range. The preset range set in this embodiment is a 3×3 range, that is, a 3×3 range in which the pixel to be denoised is centered as the preset range, and the preset distance is a simplified gaussian euclidean distance, so the preset distance value d (i, j) is calculated using the following formula:

wherein Y (Ni) and Y (Nj) are corresponding pixels in a 3×3 region image block centered on a pixel to be denoised and a 3×3 region image block centered on any pixel of a search region, respectively, ga is a 3×3 gaussian template, for example, using the following template:

next, step S54 is executed to determine whether the preset distance value is smaller than the preset noise threshold NP (Yi, K), i.e. the reduced gaussian euclidean distance is compared with the preset noise threshold NP (Yi, K).

In this embodiment, variables sum_ Y, SUM _cr, sum_cb and variable CNT are defined, where sum_ Y, SUM _cr and sum_cb are accumulated values of chromaticity values of pixels meeting requirements, that is, accumulated values of luminance value Y, red color density information Cr and blue color density information Cb of pixels meeting requirements, CNT are the number of pixels meeting requirements, and initial values of sum_ Y, SUM _cr, sum_cb and variable CNT are all 0. Step S54 is to compare the calculated distance value d (i, j) with the preset noise threshold NP (Yi, K), determine whether the preset distance value d (i, j) is smaller than the noise threshold NP (Yi, K), if so, indicate that the 3×3 range in the search area is a range meeting the requirement, execute step S55, accumulate the chromaticity values of the pixels in the 3×3 range of the denoising search area to the variables sum_ Y, SUM _cr, sum_cb, and the variable CNT is increased once. After traversing all 3×3 ranges in the target denoising region, the values of the variables sum_ Y, SUM _cr and sum_cb are the SUM of the chromaticity values of all the pixels of the satisfactory 3×3 range, and the value of the variable CNT is the number of all the pixels of the satisfactory 3×3 range.

Step S55 may be implemented using the following function:

If(d(i,j)<NP(Yi,K))

{SUM_Y+＝Y(j)；SUM_Cr+＝Cr(j)；SUM_Cb+＝Cb(j)；CNT++；}

preferably, the noise threshold NP (Yi, K) is obtained by calibration in advance according to the characteristics of the CMOS sensor, and the noise threshold NP (Yi, K) is related to the brightness value of the pixel. More preferably, the noise threshold NP (Yi, K) is positively related to the characteristic value of the image sensor and the luminance value of the pixel. The noise threshold NP (Yi, K) has K values of 0, 1, 2, and 3, which represent the first layer image, the second layer image, the third layer image, and the highest layer image of the laplacian pyramid, respectively, so that the smoothness of each frequency band of the image can be controlled independently.

Of course, the simplified gaussian euclidean distance is adopted in the formula 7, and other distances can be used as preset distance values in practical application, for example, manhattan distance, min Shi distance, chebyshev distance, cosine distance and the like, and the same effect can be obtained.

If the determination result in step S54 is no, it indicates that the preset distance value does not meet the set requirement, and the chrominance value of the pixel value within the range is not added to the value variables sum_ Y, SUM _cr and sum_cb.

Then, step S56 is executed to determine whether all the areas of 3×3 size in the denoising area are traversed, if not, step S58 is executed to acquire the area of the next 3×3 size in the denoising search area, step S53 is executed again, and the preset distance value between the new image block of the 3×3 range and the image block of the 3×3 area centered on the pixel to be denoised is calculated.

If the traversal completes all the 3×3 area image blocks within the entire denoising area, step S57 is performed to calculate the average value of the chromaticity values, for example, using the following formula:

Y(i)_NR＝SUM_Y/CNT

Cr(i)_NR＝SUM_Cr/CNT

cb (i) _nr=sum_cb/CNT (formula 8)

The chroma values Y (i, K) _NR, cr (i, K) _NR and Cb (i, K) _NR after denoising the images of the first layer, the second layer and the third layer of the Laplacian pyramid and the image of the highest layer of the Gaussian pyramid are obtained through the steps, wherein the K values are 0, 1, 2 and 3, and represent the images of the first layer, the second layer and the third layer of the Laplacian pyramid and the image of the highest layer of the Gaussian pyramid respectively.

It can be seen that, in this embodiment, the preset distance value between the denoising search area and the preset range of each layer of the laplacian pyramid image is calculated, the preset distance value between the denoising search area and the preset range of the highest layer of the gaussian pyramid image is calculated, and the average value of the chromaticity values of the pixels with all the preset distance values smaller than the noise threshold value of each layer is calculated.

The above-mentioned range of 3×3 is used as a preset range, and in practical application, the preset range is not limited to the 3×3 image block as the preset range, and may be enlarged or reduced according to practical situations.

Thus, the denoising calculation of step S5 is completed to obtain denoising chrominance values, i.e., Y (i, K) _nr, cr (i, K) _nr, and Cb (i, K) _nr, and then step S6 is performed to perform image synthesis of pixels. The present embodiment performs weighted synthesis using the chromaticity value of each layer of pixels before denoising and the chromaticity value of each layer of pixels after denoising. Specifically, the following formula may be used to calculate the chromaticity value after each pixel is synthesized:

Y(i)_final＝W0×Y(i,0)+(1-W0)×Y(i,0)_NR+

W1×Y(i,1)+(1-W1)×Y(i,1)_NR+

W2×Y(i,2)+(1-W2)×Y(i,2)_NR+

W3XY (i, 3) + (1-W3) ×Y (i, 3) _NR (formula 9)

Cr(i)_final＝W0×Cr(i,0)+(1-W0)×Cr(i,0)_NR+

W1×Cr(i,1)+(1-W1)×Cr(i,1)_NR+

W2×Cr(i,2)+(1-W2)×Cr(i,2)_NR+

W3XCr (i, 3) + (1-W3) ×Cr (i, 3) _NR (formula 10)

Cb(i)_final＝W0×Cb(i,0)+(1-W0)×Cb(i,0)_NR+

W1×Cb(i,1)+(1-W1)×Cb(i,1)_NR+

W2×Cb(i,2)+(1-W2)×Cb(i,2)_NR+

W3XCb (i, 3) + (1-W3) ×Cb (i, 3) _NR (formula 11)

Wherein W0, W1, W2 and W3 are preset weighting coefficients, and the denoising degree of each frequency band of the image and the subjective feeling of the final image can be independently controlled.

Since the initial image is a RAW image, that is, an image whose color information is RGB, it is necessary to perform step S7 to perform conversion calculation of color information on the output chromaticity value, and to obtain RAW pixel information after denoising by linear transformation. Since RAW images have a plurality of BAYER format arrangements, for example, the 4 most common BAYER format arrangements shown in fig. 2, conversion for different BAYER format arrangements is required. For example, for the case of fig. 2 (a) and 2 (b), i.e., if the center pixel is G, the following formula can be used for calculation:

gfinal=4×y (i) _final- (Cr (i) _final+cb (i) _final)/2 (formula 12)

For the case of fig. 2 (c), i.e. if the center pixel is R, the following formula can be used for calculation:

r_final=cr (i) _final (formula 13)

For the case of fig. 2 (d), i.e. if the center pixel is B, the following formula can be used for calculation:

b_final=cb (i) _final (formula 14)

Thus, the denoising calculation of a pixel to be denoised is completed, and the output chromaticity value of the pixel to be denoised is obtained.

And finally, executing step S8, judging whether the denoising calculation of all pixels of the initial image is finished, if so, outputting the data of the image, if not, executing step S9, acquiring the next pixel, returning to step S2, acquiring a layered search area corresponding to the pixel, and denoising the pixel until all the pixels are finished, namely, denoising the whole image by a sliding window method.

Therefore, in the embodiment, the denoising calculation is performed by adopting the image pyramid mode, so that the overall calculation amount can be reduced by more than 80%, the storage space of an occupied memory is reduced, and the calculation amount of mean filtering is also greatly reduced. In addition, the original image is divided into different frequency bands by utilizing the image pyramid, similar image blocks are respectively searched on each frequency band, the accuracy is greatly improved, the denoising intensity and the final synthesis coefficient of each frequency band can be independently controlled, and the detail texture and the definition of the image can be effectively maintained while the noise is removed. In addition, the embodiment can be suitable for denoising of RAW images, and meets the use requirements of different sensors.

Computer apparatus embodiment:

the computer device of the present embodiment may be an intelligent electronic device, and the computer device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, where the processor executes the computer program to implement the steps of the image denoising method based on the image pyramid. Of course, the intelligent electronic device further comprises an image capturing device for acquiring the initial image.

For example, a computer program may be split into one or more modules, which are stored in memory and executed by a processor to perform the various modules of the invention. One or more of the modules may be a series of computer program instruction segments capable of performing specific functions for describing the execution of the computer program in the terminal device.

The processor referred to in the present invention may be a central processing unit (Central Processing Unit, CPU), or other general purpose processor, digital signal processor (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, the processor being a control center of the terminal device, and the various interfaces and lines being used to connect the various parts of the overall terminal device.

The memory may be used to store computer programs and/or modules, and the processor may implement various functions of the terminal device by running or executing the computer programs and/or modules stored in the memory, and invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other volatile solid-state storage device.

Computer-readable storage medium:

the computer program stored in the above-mentioned computer means may be stored in a computer readable storage medium if it is implemented in the form of software functional units and sold or used as a separate product. Based on such understanding, the present invention may implement all or part of the above-described embodiment method, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and the computer program may implement the steps of the above-described image pyramid-based image denoising method when executed by a processor.

Wherein the computer program comprises computer program code, which may be in the form of source code, object code, executable files or in some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the content of the computer readable medium can be appropriately increased or decreased according to the requirements of the jurisdiction's jurisdiction and the patent practice, for example, in some jurisdictions, the computer readable medium does not include electrical carrier signals and telecommunication signals according to the jurisdiction and the patent practice.

Finally, it should be emphasized that the invention is not limited to the above-described embodiments, such as variations of the filtering template, or variations of the specific algorithm for performing the mean filtering, etc., which are also intended to be included in the scope of the claims.

Claims

1. An image denoising method based on an image pyramid, comprising:

acquiring an initial image, and calculating an output chromaticity value of each pixel of the initial image;

wherein calculating the output chromaticity value for each pixel comprises:

acquiring a layered search area of pixels to be denoised, acquiring initial chromaticity values of each pixel in the layered search area, performing at least one downsampling on the layered search area, acquiring a layer of downsampled image after each downsampling, and forming a first image pyramid by multiple layers of downsampled images;

upsampling each downsampled image, subtracting the upsampled image from the next layer of image of the downsampled image to obtain a layer of subtracted image, and forming a second image pyramid by multiple layers of subtracted images;

carrying out mean denoising on each layer of the subtracted image of the second image pyramid and the highest layer downsampled image of the first image pyramid to obtain a denoising colorimetric value;

calculating the output chromaticity value using the initial chromaticity value and the denoising chromaticity value;

wherein obtaining the denoising color value comprises: and setting a denoising search area of each subtracted image and the highest-layer downsampled image, calculating a preset distance value in a preset range, and calculating an average value of chromaticity values of all pixels with the preset distance value smaller than a noise threshold value.

2. The image pyramid-based image denoising method according to claim 1, wherein:

the noise threshold is directly related to the characteristic value of the image sensor and/or the luminance value of the pixel.

3. The image pyramid-based image denoising method according to claim 1, wherein:

the denoising search area of at least one layer of the subtraction image is larger than the denoising search area of the highest layer of the downsampled image.

4. The image pyramid-based image denoising method according to claim 3, wherein:

calculating a preset distance value within a preset range, wherein calculating an average value of chromaticity values of all pixels with the preset distance value smaller than a noise threshold value comprises:

and calculating the preset distance value between the denoising search area of the subtracted image of each layer and the preset range, calculating the preset distance value between the denoising search area of the downsampled image of the highest layer and the preset range, and calculating the average value of the chromaticity values of all pixels of each layer, wherein the preset distance value is smaller than the noise threshold value.

5. A method of image pyramid-based image denoising according to any one of claims 1 to 3, wherein:

calculating the output chroma value using the initial chroma value and the de-noised chroma value includes: and calculating the output chromaticity value by using the initial chromaticity value, the denoising chromaticity value of each layer and a preset weighting coefficient.

6. A method of image pyramid-based image denoising according to any one of claims 1 to 3, wherein:

the preset distance value is one of a simplified Gaussian Euclidean distance, a Manhattan distance, a Min Shi distance, a Chebyshev distance and a cosine distance.

7. A method of image pyramid-based image denoising according to any one of claims 1 to 3, wherein:

performing Gaussian filtering calculation when downsampling the hierarchical search region; and/or

And when up-sampling is carried out on the down-sampled image, gaussian filtering calculation is carried out.

8. Computer arrangement, characterized in that it comprises a processor and a memory, said memory storing a computer program which, when executed by the processor, implements the steps of the image pyramid based image denoising method according to any of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program characterized by: the computer program, when executed by a processor, implements the steps of the image pyramid based image denoising method according to any one of claims 1 to 7.