CN103973963A - Image acquisition device and image processing method thereof - Google Patents

Abstract

The invention provides an image acquisition device and an image processing method thereof. This image processing method includes the following steps. A first image is acquired at a first focal length and a second image is acquired at a second focal length. Perform a geometric correction procedure on the second image to generate a displacement-corrected second image. A gradient operation is performed on each pixel of the first image to generate a plurality of first gradient values, and a gradient operation is performed on each pixel of the displacement-corrected second image to generate a plurality of second gradient values. Each first gradient value is compared with each corresponding second gradient value to generate a plurality of first pixel comparison results, and a first parameter map is generated according to the first pixel comparison results. A composite image is generated based on the first parameter map and the first image, and an output image is generated based on at least the composite image.

Description

Image acquisition device and image processing method thereof

technical field

The present invention relates to an image acquisition device and its image processing method, and in particular to an image acquisition device and its image processing method for blending images by calculating pixel gradient values.

Background technique

With the advancement of optical technology, digital cameras with adjustable aperture, shutter and even interchangeable lenses are gradually popularized, and the functions of digital cameras are also tending to be diversified. In addition to providing good imaging quality for digital cameras, the accuracy and speed of focusing technology are factors that consumers will refer to when purchasing products. However, with the existing optical system, since multiple objects have different distances in the stereoscopic scene, it is impossible to obtain a completely clear panoramic depth image in a single image capture process. That is to say, limited by the optical characteristics of the lens, only one of the depths can be selected for focusing when using a digital camera to capture images, so scenes at other depths will be blurred in the imaging.

Most of the existing methods for generating a depth-of-view image use a combination of multiple images obtained by shooting under different shooting conditions. By changing one or more parameters in the shooting conditions, multiple different images of the same scene are taken, and then these images are combined into a clear image through a sharpness discrimination method. The technique of using the above-mentioned various shooting conditions to synthesize a full-depth image must rely on a fixed image acquisition device for shooting. Generally speaking, users often use a stable tripod to fix the image acquisition device to ensure that there is no obvious geometric distortion among the acquired images. In addition, during the shooting process, it is necessary to avoid any movement of the target in the scene.

On the other hand, when using a camera to capture an image, in order to highlight the subject in the captured image, a shooting technique called bokeh is generally used. Bokeh means that in photographic imaging with a shallow depth of field, images falling outside the depth of field will gradually produce a loose and blurred effect. In general, camera lenses are limited in the amount of bokeh they can produce. To obtain a better bokeh effect, the following important conditions usually need to be met at the same time: large aperture and long focal length. In other words, in order to achieve the bokeh effect, it is necessary to rely on a large-aperture lens to enhance the blurring of distant objects, so that the clearly imaged subject can stand out from the background. However, large-aperture lenses are bulky and expensive, and cannot be equipped with general consumer cameras.

All in all, the existing methods for generating the depth of field or bokeh images tend to cause discontinuous or unnatural depth of field in the processed image. In addition, the restrictions on the operation of capturing images make users feel even more inconvenient, such as the average total shooting time is rather long or complicated, and even the final result images are not satisfactory.

Contents of the invention

In view of this, the present invention provides an image acquisition device and an image processing method thereof, which can determine the subject in the image through images captured with different focal length values, and then generate an image with a clear subject and a natural bokeh effect. On the other hand, the image processing method of the present invention can also avoid the ghosting problem when generating the full depth image by using the images captured with different focal length values.

The invention proposes an image processing method suitable for an image acquisition device, and the image processing method includes the following steps. A first image is acquired with a first focal length, and a second image is acquired with a second focal length, wherein the first focal length focuses on at least one subject. A geometric correction procedure is performed on the second image to generate a displacement-corrected second image. A gradient operation is performed on each pixel of the first image to generate a plurality of first gradient values, and a gradient operation is performed on each pixel of the displacement-corrected second image to generate a plurality of second gradient values. Each first gradient value is compared with each corresponding second gradient value to generate a plurality of first pixel comparison results, and a first parameter map is generated according to the first pixel comparison results. A composite image is generated according to the first parameter map and the first image, and an output image is generated at least according to the composite image.

In an embodiment of the present invention, in the above image processing method, at least the step of generating an output image according to the synthesized image includes: acquiring a third image with a third focal length. A geometric correction procedure is performed on the third image to generate a displacement-corrected third image. A gradient operation is performed on each pixel of the synthesized image to generate a plurality of third gradient values, and a gradient operation is performed on each pixel of the displacement-corrected third image to generate a plurality of fourth gradient values. Each third gradient value is compared with each corresponding fourth gradient value to generate a plurality of second pixel comparison results, and a second parameter map is generated according to the second pixel comparison results. According to the second parameter map, the displacement-corrected third image is mixed with the composite image to generate an output image.

In an embodiment of the present invention, in the above-mentioned image processing method, the step of performing a geometric correction procedure on the second image, and generating the displacement-corrected second image includes: performing movement estimation on the first image and the second image , to calculate the homography matrix. A geometric affine transformation is performed on the second image according to the homography matrix to obtain a displacement-corrected second image.

In an embodiment of the present invention, in the above-mentioned image processing method, each first gradient value is compared with each corresponding second gradient value to generate a plurality of first pixel comparison results, and a plurality of first pixel comparison results are generated according to these first pixel comparison results The step of parameter mapping includes dividing the second gradient values by the corresponding first gradient values to generate a plurality of gradient comparison values. A plurality of parameter values are generated from the gradient comparison values and recorded as a parameter map.

In an embodiment of the present invention, in the above image processing method, the step of generating a plurality of parameter values according to a plurality of gradient comparison values includes: determining whether the gradient comparison values are greater than a first gradient critical value. If the gradient comparison value is greater than the first gradient critical value, set the parameter value corresponding to the gradient comparison value to the first value.

In an embodiment of the present invention, in the above image processing method, the step of generating a plurality of parameter values according to these gradient comparison values includes: if the gradient comparison value is not greater than the first gradient critical value, judging whether the gradient comparison value is greater than the first gradient critical value Two gradient critical values. If the gradient comparison value is greater than the second gradient critical value, set the parameter value corresponding to the gradient comparison value to the second value. If the gradient comparison value is not greater than the second gradient threshold value, the parameter value corresponding to the gradient comparison value is set to a third value, wherein the first gradient threshold value is greater than the second gradient threshold value.

In an embodiment of the present invention, in the above image processing method, at least the step of generating a composite image according to the first parameter map and the first image includes: performing a blurring procedure on the first image to generate a blurred image. Blending the first image and the blurred image according to the first reference map to generate a sharp image of the subject.

In an embodiment of the present invention, in the above image processing method, the step of mixing the first image and the blurred image according to the first reference map to generate a clear image of the subject includes: determining whether the parameter value is greater than a first mixing threshold. If the parameter value is greater than the first mixing critical value, the pixel of the blurred image corresponding to the parameter value is taken as the pixel of the subject clear image. If the parameter value is not greater than the first mixing critical value, it is determined whether the parameter value is greater than the second mixing critical value. If the parameter value is greater than the second mixing critical value, the pixel points of the corresponding subject clear image are calculated according to the parameter value. If the parameter value is not greater than the second mixing critical value, the pixel of the first image corresponding to the parameter value is taken as the pixel of the subject clear image, wherein the first mixing critical value is greater than the second mixing critical value.

In an embodiment of the present invention, in the above image processing method, at least the step of generating a composite image based on the first parameter map and the first image includes: calculating Multiple sums of absolute differences (Sum of AbsoluteDifferences) corresponding to each pixel point, and adjust the parameter values in the first parameter map according to these sums of absolute differences. Based on the adjusted first reference map, the first image is blended with the displacement-corrected second image to generate a full depth image.

In an embodiment of the present invention, in the above-mentioned image processing method, the absolute difference sum corresponding to each pixel is calculated according to the pixel values of each pixel in the first image and the second image, and the absolute difference sum is calculated according to the absolute difference sum The step of adjusting the parameter values in the first parameter map includes: when the absolute difference sum is greater than the mobile critical value, determining the weight factor of each parameter value according to the absolute difference sum, and using the weight factor to adjust the parameter value, wherein each parameter value follows the The corresponding absolute difference sum increases and decreases.

In an embodiment of the present invention, in the above image processing method, the step of mixing the first image and the displacement-corrected second image to generate the full depth image according to the first reference map adjusted by the weight factor includes: judging Whether the parameter value is greater than the first blending threshold. If the parameter value is greater than the first mixing critical value, the pixel point of the displacement-corrected second image corresponding to the parameter value is taken as the pixel point of the full depth image. If the parameter value is not greater than the first mixing critical value, it is determined whether the parameter value is greater than the second mixing critical value. If the parameter value is greater than the second mixing threshold, the corresponding pixel points of the full depth image are calculated according to the parameter value. If the parameter value is not greater than the second blending threshold, the pixel of the first image corresponding to the parameter value is taken as the pixel of the full depth image, wherein the first blending threshold is greater than the second blending threshold.

From another point of view, the present invention provides an image acquisition device, which includes an image acquisition module, a displacement correction module, a gradient calculation module, a map generation module, and an image synthesis module. The image acquisition module acquires a first image with a first focal length, and acquires a second image with a second focal length, wherein the first focal length focuses on at least one subject. The displacement correction module performs a geometric correction procedure on the second image to generate a displacement-corrected second image. The gradient calculation module performs a gradient operation on each pixel of the first image to generate a plurality of first gradient values, and performs a degree operation on each pixel of the displacement-corrected second image to generate a plurality of second gradient values. The map generation module compares each first gradient value with each corresponding second gradient value to generate a plurality of first pixel comparison results, and generates a first parameter map according to the first pixel comparison results. The image synthesis module generates a composite image according to the first parameter map and the first image, and at least generates an output image according to the composite image.

Based on the above, the present invention uses the feature that different focal lengths will cause different images, shoots the same scene with different focal lengths, and compares the gradient differences of each pixel point between images to generate a parameter map. Through the information of the parameter map, a clear panoramic depth image or a bokeh image with a clear subject and a blurred background can be generated to achieve a good panoramic depth effect or bokeh effect.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail with reference to the accompanying drawings.

Description of drawings

Fig. 1 is a schematic functional block diagram of an image acquisition device shown in an embodiment of the present invention;

Fig. 2 is a flowchart of an image processing method shown in an embodiment of the present invention;

FIG. 3 is a schematic diagram of an image processing method shown in another embodiment of the present invention;

Fig. 4 is a block diagram of an image acquisition device shown in another embodiment of the present invention;

Fig. 5 is a flowchart of an image processing method shown in another embodiment of the present invention;

FIG. 6 is a detailed flowchart of step S550 in FIG. 5 shown in another embodiment of the present invention;

FIG. 7 is a detailed flowchart of step S560 in FIG. 5 shown in another embodiment of the present invention;

Fig. 8 is a block diagram of an image acquisition device shown in another embodiment of the present invention;

FIG. 9A is a schematic diagram of a pixel block shown in yet another embodiment of the present invention;

FIG. 9B is a schematic diagram of the relationship between the absolute difference sum and the weighting factor shown in another embodiment of the present invention.

Explanation of reference signs:

100, 400, 800: image acquisition device;

110, 410, 810: image acquisition module;

120, 420, 820: image correction module;

130, 430, 830: gradient calculation module;

140, 440, 840: map generation module;

150, 450, 850: image synthesis module;

460: image blur module;

860: map adjustment module;

Img1, Img2, Img3, Img_b, Img_F, Img1_blur, Img2_cal: image

G1, G2: Gradient value;

bokeh_map: bokeh map;

map, allin_map: parameter map;

S210～S250, S510～S560, S610～S625, S710～S750: steps.

Detailed ways

The present invention proposes a method for generating a bokeh image and a depth-of-view image by using multiple images taken with different focal length values. First focus on at least one subject to be photographed and shoot, and then use another focal length to shoot the same scene. By comparing the pixel gradients of the two images to generate a parameter map, the main part of the image can be judged, and then an image with a bokeh effect can be generated. On the other hand, by comparing pixel gradients of at least two images, a parameter map serving as a basis for blending images is generated, thereby generating a depth-of-view image. In order to make the content of the present invention clearer, the following examples are listed as examples in which the present invention can actually be implemented.

FIG. 1 is a schematic functional block diagram of an image acquisition device according to an embodiment of the present invention. Please refer to FIG. 1 , the image acquisition device 100 of this embodiment is, for example, a digital camera, a SLR camera, a digital video camera, or other smart phones, tablet computers, head-mounted displays, etc. with image acquisition functions, and is not limited to the above. The image acquisition device 100 includes an image acquisition module 110 , an image correction module 120 , a gradient calculation module 130 , a map generation module 140 and an image synthesis module 150 .

The image acquisition module 110 includes a zoom lens and a photosensitive element. The photosensitive element is, for example, a charge coupled device (Charge Coupled Device, CCD), a complementary metal oxide semiconductor (Complementary Metal-Oxide Semiconductor, CMOS) element or other elements, and the image acquisition module 110 can also include an aperture, etc., which are not limited here . The image acquisition module 110 can acquire different images according to different focal length values.

On the other hand, the image correction module 120 , the gradient calculation module 130 , the map generation module 140 and the image synthesis module 150 can be implemented by software, hardware or a combination thereof, which is not limited here. The software is, for example, source code, an operating system, application software, or a driver. The hardware is, for example, a central processing unit (Central Processing Unit, CPU), or other programmable general-purpose or special-purpose microprocessors (Microprocessor).

FIG. 2 is a flowchart of an image processing method according to an embodiment of the present invention. The method of this embodiment is applicable to the image acquisition device 100 in FIG. 1 , and the detailed steps of this embodiment are described below with each module in the image acquisition device 100:

First, in step S210 , the image acquisition module 110 acquires a first image with a first focal length, and acquires a second image with a second focal length, wherein the first focal length focuses on at least one subject. That is to say, the image acquisition module 110 captures two images with two different focal lengths. Among them, under the same conditions, the results of pictures taken with different focal lengths will be different. Specifically, as far as the first image focusing on the subject is concerned, the subject part in the image is the clearest.

In step S220, the image correction module 120 performs a geometric correction procedure on the second image to generate a displacement-corrected second image. Since the first image and the second image are continuously captured by the user on the same scene, images from different angles may be captured due to camera shaking or movement during the period, that is, displacement of the first image and the second image may occur. Therefore, the image correction module 120 performs a geometric correction procedure on the second image. In other words, the geometric correction procedure can make the position of the starting pixel of the displacement-corrected second image the same as that of the first image.

In step S230, the gradient calculation module 130 performs a gradient operation on each pixel of the first image to generate a plurality of first gradient values, and performs a gradient operation on each pixel of the displacement-corrected second image to generate A plurality of second gradient values. That is to say, each pixel point in the first image has its first gradient value, and each pixel point in the displacement-corrected second image has its second gradient value.

In step S240 , the map generation module 140 compares each first gradient value with each corresponding second gradient value to generate a plurality of first pixel comparison results, and generates a first parameter map according to the first pixel comparison results. In simple terms, the map generation module 140 will compare the gradient values of the pixels at the same position, and there will be a pixel comparison result for each pixel position.

In step S250 , the image synthesis module 150 generates a composite image according to the first parameter map and the first image, and at least generates an output image according to the composite image. In detail, after obtaining the parameter map, the image acquisition device 100 may mix the first image and images processed by other images according to the parameter map, so as to generate a composite image. In addition, the image acquisition device 100 may also mix the first image and the second image according to the parameter map, so as to generate a composite image.

It is worth mentioning that although the above-mentioned embodiment is an example of two images taken with two focal lengths, the present invention is not limited thereto. Depending on the actual application situation, the present invention can be extended to obtain the final output image by using multiple images captured by multiple focal lengths. For example, since images with different focal lengths have different clear image parts, a clear full depth image can be obtained through multiple images with different focal lengths. In addition, the image processing method of the present invention can generate an output image in which only the subject is clear by focusing on the three images of the subject, the background and the foreground. Another embodiment will be described in detail below.

FIG. 3 is a schematic diagram of an image processing method according to another embodiment of the present invention. In this embodiment, the image acquisition module 110 acquires the first image Img1 and the second image Img2 by using the first focal length and the second focal length. Afterwards, as described in the above-mentioned embodiment, through the processing of the image correction module 120 , the gradient calculation module 130 , the map generation module, and the image synthesis module 150 , the synthesized image Img_b can be generated accordingly, which will not be repeated here. It should be noted that in the above embodiments, the image synthesis module 150 can use the synthesized image Img_b as the final output image, but in this embodiment, the synthesized image Img_b will be further synthesized with another image to generate the final output image Img_F. In detail, as shown in FIG. 3 , the image acquisition module 110 will acquire a third image Img3 at a third focal length. The image correction module 120 performs a geometric correction procedure on the third image Img3 to generate a displacement-corrected third image Img3.

Afterwards, the gradient calculation module performs a gradient operation on each pixel of the synthesized image Img_b to generate a plurality of third gradient values, and performs the gradient operation on each pixel of the displacement-corrected third image Img3 to generate a plurality of The fourth gradient value. The map generation module 140 compares each third gradient value with each corresponding fourth gradient value to generate a plurality of second pixel comparison results, and generates a second parameter map according to the second pixel comparison results. The second parameter map here is obtained by calculating the gradient value of the synthesized image Img_b and the third image Img3, and its internal parameter values will be different from the aforementioned parameter map calculated by using the first image Img1 and the second image Img2 . The image synthesis module 150 mixes the displacement-corrected third image Img3 and the synthesized image Img_b according to the second parameter map to generate an output image Img_F. Based on the above, it can be seen that the present invention does not limit the number of images used to mix the final output image, which can be determined according to actual application requirements.

However, the implementation manner of the present invention is not limited to the above description, and the content of the above embodiments may be modified according to actual requirements. For example, in yet another embodiment of the present invention, the image acquisition device may further include an image blur module to produce a clear image of the subject with a bokeh effect. In addition, in yet another embodiment of the present invention, the image acquisition device may further include a map adjustment module to produce a full depth image with a good full depth effect. In order to further illustrate how the gradient calculation module, the map generation module, and the image synthesis module of the present invention synthesize bokeh images and depth-of-view images based on images with different focal lengths, examples are given below in detail.

FIG. 4 is a block diagram of an image acquisition device according to another embodiment of the present invention. The image acquisition device 400 includes an image acquisition module 410 , an image correction module 420 , a gradient calculation module 430 , a map yield module 440 , an image synthesis module 450 and an image blur module 460 . Wherein, the image acquisition module 410, the image correction module 420, the gradient calculation module 430, the map yield module 440 and the image synthesis module 450 are similar or similar to the image acquisition module 110, the image correction module 120, and the gradient calculation module 130 shown in FIG. , the map production module 140 and the image synthesis module 150 will not be repeated here. The embodiment shown in FIG. 4 can be deduced by referring to the relevant descriptions in FIG. 1 to FIG. 3 .

It should be noted that, unlike the image acquisition device 100 shown in FIG. 1 , the image acquisition device 400 further includes an image blur module 460 . Wherein, the image blurring module 460, for example, adopts a Gaussian filter (Gaussian filter), a bidirectional filter (Bilateral filter) or an average filter (Average filter), etc., to perform a blurring process on the first image Img1, which is not discussed in the present invention. limit. In addition, in this embodiment, it is assumed that the second focal length is the focal length focusing on the background.

FIG. 5 is a flowchart of an image processing method according to an embodiment of the present invention. The method of this embodiment is applicable to the image acquisition device 400 in FIG. 4 , and the detailed steps of this embodiment are described below with each module in the image acquisition device 400:

First, in step S510 , the image acquisition module 410 acquires the first image Img1 with the first focal length, and acquires the second image Img2 with the second focal length, wherein the first focal length is focused on at least one subject, and the second focal length is focused on the background. In the first image Img1 captured by focusing on the subject, the subject is relatively clear and the background is relatively blurred. Compared with the first image Img1, in the second image Img2 captured by focusing on the background, the background is clearer. Next, as described in step S520 , the image blur module 460 performs a blur procedure on the first image Img1 to generate a blurred image Img1_blur.

In step S530 , the image correction module 420 performs a geometric correction procedure on the second image Img2 to generate a displacement-corrected second image Img2_cal. In detail, the image correction module 420 may perform motion estimation on the first image Img1 and the second image Img2, so as to calculate a homography matrix. Next, the image correction module 420 performs geometric affine transformation on the second image Img2 according to the homography matrix to obtain a transformed second image Img2_cal after displacement correction. Accordingly, the starting pixel position of the main body area in the first image Img1 is the same as the starting pixel position of the main body area in the displacement-corrected second image Img2_cal.

Then, in step S540, the gradient calculation module 430 performs gradient calculation on each pixel of the first image Img1 to generate a plurality of first gradient values G1, and performs gradient calculation on each pixel of the displacement-corrected second image Img2_cal Gradient operation to generate a plurality of second gradient values G2. Wherein, the gradient operation may be a gradient value operation in a horizontal direction, a gradient value operation in a vertical direction, or a gradient value operation in a two-diagonal direction, which is not limited in the present invention. That is to say, the first gradient value and the second gradient value correspond to the gradient calculation method may be a gradient value in the horizontal direction, a gradient value in the vertical direction, or a gradient value in the two diagonal directions. Wherein, the gradient value in the horizontal direction is the sum of the absolute values of grayscale differences between this pixel point and two adjacent horizontal direction pixel points. The gradient value in the vertical direction is the sum of the absolute values of gray scale differences between this pixel and two adjacent pixels in the vertical direction. The gradient value in the diagonal direction includes the sum of the absolute value of the gray scale difference between this pixel point and the pixel point in the diagonal direction.

It should be noted that, in this embodiment, since the first image Img1 is an image focused on the subject, the subject in the first image Img1 will be clearer than the displacement-corrected image Img2_cal. That is to say, the gradient value of the pixel point in the in-focus area of the first image Img1 is greater than the gradient value of the pixel point at the same position in the displacement-corrected second image Img2_cal. Conversely, since the displacement-corrected second image Img2_cal is an image produced by focusing on the background, the gradient values of the pixels in the background area of the first image Img1 will be smaller than the gradient values of the pixels at the same position in the displacement-corrected image Img2_cal.

Based on this, in step S550 , the map generation module 440 compares each first gradient value G1 with each corresponding second gradient value G2 to generate a plurality of comparison results, and generates a parameter map according to the comparison results. It should be noted that, in this embodiment, the parameter map is called a bokeh map bokeh_map. In detail, the map generating module 440 will compare the gradient values of the pixels at the same positions in the first image Img1 and the displacement-corrected second image Img2_cal. Moreover, based on the relationship between the gradient value of each pixel in the first image Img1 and the displacement-corrected second image Img2_cal, it can be determined whether each pixel in the first image Img1 is located in the subject area or the background area through the comparison result. The map generating module 440 can generate a bokeh map bokeh_map by comparing the gradient values of each pixel in the first image Img1 and the displacement-corrected second image Img2_cal. In other words, the bokeh map bokeh_map carries the comparison result information of the gradient values of the pixels at the same positions in the first image Img1 and the displacement-corrected second image Img2_cal.

Finally, in step S560 , the image synthesis module 450 mixes the first image Img1 and the blurred image Img1_blur according to the bokeh map bokeh_map to generate the subject sharp image Img1_bokeh. It can be seen that the second image Img2 is used to generate the bokeh map bokeh_map, and the image synthesis module 450 mixes the first image Img1 and the blurred image Img1_blur according to the bokeh map bokeh_map to generate the subject clear image Img1_bokeh with a bokeh effect. This creates a bokeh image that maintains the sharpness of the subject area while blurring other background areas.

In addition, how the map generating module 440 generates the bokeh map bokeh_map according to the result of comparing each first gradient value G1 with each corresponding second gradient value G2 will be further described in detail below. FIG. 6 is a detailed flowchart of step S550 in FIG. 5 according to an embodiment of the present invention. Please refer to FIG. 4 and FIG. 6 at the same time. In step S610 , the map generation module 440 divides the second gradient value G2 by the corresponding first gradient value G1 to generate a gradient comparison value. In step S620, the map generation module 440 generates a plurality of parameter values according to the gradient comparison value, and records the parameter values as a bokeh map bokeh_map. For example, if the first image Img1 and the displacement-corrected image Img2_cal respectively have 1024*768 pixels, after being calculated by the image processing module 140, 1024*768 gradient comparison values will be generated, and the bokeh map bokeh_map will contain 1024* 768 parameter values. Here, step S620 can be divided into steps S621 to S625 for implementation.

The map generating module 440 determines whether the gradient comparison value of each position is greater than the first gradient critical value (step S621 ). If the gradient comparison value is greater than the first gradient critical value, the map generation module 440 sets the parameter value corresponding to the gradient comparison value as a first value (step S622 ), which is called the bokeh background value here. In other words, if the gradient comparison value is greater than the first gradient critical value, it means that the pixel at this position is located in the background area. If the gradient comparison value is not greater than the first gradient critical value, the map generation module 440 determines whether the gradient comparison value is greater than the second gradient critical value (step S623 ). If the gradient comparison value is greater than the second gradient critical value, the map generation module 440 sets the parameter value corresponding to the gradient comparison value as a second value (step S324 ), which is called a bokeh edge value here. In simple terms, if the gradient comparison value is between the second gradient critical value and the first gradient critical value, the pixel representing this position is located in the edge area between the subject and the background. If the gradient comparison value is not greater than the second gradient critical value, the map generation module 440 sets the parameter value corresponding to the gradient comparison value as a third value (step S625), and the third value is called the bokeh subject value here, That is, the pixel at this position is located in the area of the subject. Note that the Bokeh Edge value will be between the Bokeh Background value and the Bokeh Body value, and that the first gradient threshold is greater than the second gradient threshold, and the first gradient threshold and the second gradient threshold It is properly set according to the actual situation, and the present invention is not limited thereto.

For example, assuming that the map generation module 440 sets the parameter value between 0 and 255, the image processing module 140 can use the following code (1) to generate the bokeh map bokeh_map:

$\mathrm{<span\; class="notranslate">\; if</span>}<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; Gra</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; Gra</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; TH</span>}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Map=255

$\mathrm{<span\; class="notranslate">\; else\; if</span>}<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; Gra</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; Gra</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; TH</span>}\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

$\mathrm{<span\; class="notranslate">\; map</span>}<span\; class="notranslate">\; =</span>\frac{\frac{\mathrm{<span\; class="notranslate">\; Gra</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; Gra</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; TH</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; TH</span>}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; TH</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; 255</span>}$

else

Map=0

(1)

Wherein, in this exemplary embodiment, the bokeh background value is 255, the bokeh main value is 0, and the bokeh edge value can pass through the first gradient threshold, the second gradient threshold, and the second gradient value and the first gradient Calculated as a ratio between values. Gra2 is the second gradient value, Gra1 is the first gradient value, TH1 is the first gradient critical value, TH2 is the second gradient critical value, and Map is multiple parameter values in the bokeh map bokeh_map.

In addition, in order to describe in detail how the image synthesis module 450 utilizes the bokeh map bokeh_map to generate the sharp subject image Img1_bokeh, it will be described in detail below. FIG. 7 is a detailed flowchart of step S560 in FIG. 5 according to an exemplary embodiment of the present invention. Please refer to FIG. 4 and FIG. 7 at the same time. It should be noted that the pixel points at each position in the first image Img1 may respectively correspond to each parameter value in the bokeh map bokeh_map. In step S710, the image composition module 450 determines whether each parameter value is greater than a first mixing threshold. If the parameter value is greater than the first blending threshold, in step S720 , the image composition module 450 takes the pixels of the blurred image Img1_blur corresponding to these parameter values as the pixels at the same position in the subject clear image Img1_bokeh. That is, the pixels at these positions are identified as the background area, so the pixels of the blurred image Img1_blur are used to generate a blurred background image.

If the parameter value is not greater than the first blending threshold, in step S730, the image blending module 150 determines whether the parameter value is greater than the second blending threshold. If the parameter value is greater than the second blending threshold, in step S740 , the image composition module 450 calculates the pixel points of the clear subject image Img1_bokeh corresponding to the parameter value according to the parameter value. Specifically, the pixel positions corresponding to the parameter values between the first blending threshold and the second blending threshold are determined to be edge regions between the background region and the subject region. Therefore, the pixels in the edge area between the background area and the main body area in the main body clear image Img1_bokeh can be obtained by synthesizing the first image Img1 and the blurred image Img1_blur.

If the parameter value is not greater than the second blending threshold, in step S750 , the image composition module 450 takes the pixel of the first image Img_1 corresponding to the parameter value as the pixel of the subject clear image Img1_bokeh. That is to say, the positions corresponding to these parameter values are judged to be in the subject area, so the pixels of the subject area in the clear first image Img_1 are taken as the subject area pixels in the clear subject image Img1_bokeh. Wherein, the first mixing critical value is greater than the second mixing critical value.

For example, assuming that the image synthesis module 450 sets the parameter value between 0 and 255, the image synthesis module 450 can use the following program code (2) to generate the subject clear image Img1_bokeh:

if(Map≥Blend_TH1)//Background area

Img1_Bokeh=Img1_Blur

else if(Map≥Blend_TH2)//Transition area

w _Bokeh =LUT[Map](LUT is table and value range is0～255)

$\mathrm{<span\; class="notranslate">\; Img</span>}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; Bokeh</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; Bokeh</span>}}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; Img</span>}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 256</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; Bokeh</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; Img</span>}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; Blurred</span>}}{\mathrm{<span\; class="notranslate">\; 256</span>}}$

else//Subject

Img1_Bokeh=Img1

(2)

Wherein, in this exemplary embodiment, Blend_TH1 is the first blending threshold, Blend_TH2 is the second blending threshold, Map is a plurality of parameter values in the bokeh map bokeh_map, and LUT[] is a look-up table function. It is worth mentioning that the pixels in the edge area can be calculated by the concept of weight. As shown in the formula in the above exemplary program code, the parameter value is used as the synthesis weight w _bokeh , and the pixels in the edge area are synthesized by the synthesis weight w _bokeh . That is to say, for the pixels in the edge area, the degree of blur will be determined depending on its position closer to the main area or the blurred area, so that a clear main image Img1_bokeh with a natural connection between the main area and the background area can be generated. Makes the edges between the subject and the background soft and natural in bokeh images.

In the above embodiments, the second focal length value is used as an example to focus on the background, so a blurred background image with a blurred background and a clear subject can be generated accordingly. It can be seen from the description of FIG. 3 that the image processing method of the present invention can obtain a final output image through multiple images. Based on this, in other embodiments, it is assumed that the image acquisition device acquires another image with a third focal length focusing on the foreground. The image acquisition device can use the previously generated image with blurred background and another image focused on the foreground, and perform further calculations to generate an image with blurred foreground and background and clear subject through the same process as the above-mentioned generation of the blurred background image.

FIG. 8 is a block diagram of an image acquisition device according to yet another embodiment of the present invention. Please refer to FIG. 8 , in this embodiment, an image acquisition device 800 is used to generate a full-depth image. The image acquisition device 800 includes an image acquisition module 810 , an image correction module 820 , a gradient calculation module 830 , a map generation module 840 , an image synthesis module 850 and a map adjustment module 860 . Wherein, the image acquisition module 810, the image correction module 820, the gradient calculation module 830, the map generation module 840 and the image synthesis module 850 are similar or similar to the image acquisition module 410 shown in FIG. 4, the image correction module 420, the gradient calculation module 430, The map generation module 440 and the image synthesis module 450 will not be repeated here.

It should be noted that, unlike the image acquisition device 400 shown in FIG. 4 , the image acquisition device 800 in this embodiment does not have an image blur module but further includes a map adjustment module 860 . Wherein, the map adjustment module 860 is used for adjusting the parameter map generated by the map generation module 840 . In this embodiment, the image acquisition module 810 acquires the first image Img1 with the first focal length, and acquires the second image Img2 with the second focal length, wherein the first focal length is focused on at least one subject, and the second focal length is focused on an area other than the subject .

Next, the image correction module 830 performs a geometric correction procedure on the second image Img2 to generate a displacement-corrected second image Img2_cal. Then the gradient calculation module 840 performs a gradient operation on each pixel of the first image Img1 to generate a plurality of first gradient values G1, and performs a gradient operation on each pixel of the displacement-corrected second image Img2_cal to generate a plurality of The second gradient value G2. Next, the map generating module 840 compares each first gradient value G1 with each corresponding second gradient value G2 to generate a plurality of comparison results, and generates a parameter map according to the comparison results. The steps of the image correction module 830 generating the displacement-corrected second image Img2_cal, the steps of the gradient calculation module 830 performing the gradient calculation, and the steps of the map generation module 840 generating the parameter map map are similar to the image acquisition module 400 shown in FIG. 4 , The description of FIG. 4 and FIG. 5 can be referred to and analogized.

Generally speaking, the gradient values of the pixel at the same position on the two images are different, that is, the first gradient value G1 and the second gradient value G2 in this embodiment. On the other hand, for the pixels at the same position, if the gradient value of the pixel at the position in the first image is relatively high (that is, G1 is greater than G2), it usually means that the pixel at the position is located in the first image more easily. The clear area (i.e. the area within the first focal length). If the pixel at this position has a higher gradient value in the second image (ie, G2 is greater than G1), it usually means that the pixel at this position is located in a clearer area in the second image (ie, the area within the second focal length). That is to say, the map generating module 840 can also obtain the parameter map through the code (1), but the present invention is not limited thereto.

Therefore, in this embodiment, the map generation module 440 can generate a parameter map by comparing the gradient values of each pixel in the first image Img1 and the displacement-corrected second image Img2_cal. In other words, the parameter map carries the comparison result information of the gradient values of the pixels at the same positions in the first image Img1 and the displacement-corrected second image Img2_cal. In this way, the image acquisition device 800 can know according to the parameter map, whether the pixel at a certain position is located in the clear part within the first focal length in the first image Img1 or within the second focal length in the second image Img2 clear part of . Accordingly, the image synthesis module 850 can select clearer parts from the two images to synthesize an output image with more clear parts.

It is worth mentioning that during the process of continuous shooting of the same scene by the user and obtaining the first image and the second image, some objects may be moving due to the time difference in shooting. The image correction module 820 corrects the overall displacement (or camera displacement) of the image, and does not correct individual objects in the scene. Therefore, if there are individual moving objects in the image, the mixed depth-of-view image will appear Ghosting phenomenon. The map adjustment module 860 of this embodiment is used to improve the above-mentioned ghost phenomenon.

Here, the map adjustment module 860 calculates a plurality of sums of absolute differences (Sum of Absolute Differences) corresponding to each pixel according to the pixel values of each pixel in the first image Img1 and the second image Img2, and based on these absolute differences And adjust multiple parameter values in the parameter map map. The map adjustment module 860 mixes the first image Img1 and the displacement-corrected second image Img2_cal according to the adjusted reference map to generate a full depth image.

In detail, firstly, an n×n pixel block (n is a positive integer) is obtained from the first image Img1. Assuming that n is 5, the 5×5 pixel block obtained in this embodiment is as shown in FIG. 9A , which includes 25 pixel positions P ₀₀ ˜P ₄₄ . Similarly, an n×n pixel block centered on the pixel position is obtained in the displacement-corrected second image Img2_cal. Next, calculate the sum of absolute differences of specific color space components of each pixel in each n×n pixel block of the first image Img1 and the displacement-corrected second image Img2_cal and find its maximum value as a representative. The absolute difference can reflect whether the characteristics of the Img1 and the displacement-corrected second image Img2_cal in the local area of n×n pixel blocks are close. In the YCbCr color space, specific color space components include luma components, blue chrominance components, and red chrominance components, but the present invention does not limit the color space. Based on the YCbCr color space, this embodiment assumes that n=5 and uses the following formula to calculate the absolute difference and SAD between the pixel positions of the first image Img1 and the displacement-corrected second image Img2_cal:

$\mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; \&CenterDot;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 4</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; Y</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; Y</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; |</span>$

$\mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; Cb</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 4</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; Cb</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; Cb</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; |</span>$

$\mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; Cr</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 4</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; Cr</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; Cr</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; |</span>$

SAD=max(max(SAD_Y,SAD_Cb),SAD_Cr)

Among them, i and j represent the position of the pixel. As shown in the example of FIG. 9A , each pixel block includes 25 pixel positions P ₀₀ ˜P ₄₄ . Y1 _ij is the luminance component of the pixel P _ij in the first image, and Y2 _ij is the luminance component of the pixel P _ij in the second image. Cb1 _ij is the blue chroma component of the pixel P _ij in the first image, and Cb2 _ij is the blue chroma component of the pixel P _ij in the second image. Cr1 _ij is the red chroma component of the pixel P _ij in the first image, and Cr2 _ij is the red chroma component of the pixel P _ij in the second image. SAD_Y, SAD_Cb, and SAD_Cr are sums of absolute differences on each specific color space component, respectively.

Based on this, the map adjustment module 860 of the present invention, for example, obtains the absolute difference and SAD by using the above calculation formula. Afterwards, the map adjustment module 860 will determine whether the sum of absolute differences SAD is greater than the movement threshold TH_SAD. If the absolute difference and SAD are not greater than the movement threshold TH_SAD, it means that the pixel block does not have the subject moving, and there is no need to adjust the parameter value corresponding to the pixel block in the parameter map. If the absolute difference and SAD are greater than the movement threshold TH_SAD, it means that the object has moved in this pixel block. Therefore, the map adjustment module 860 will adjust the corresponding pixel block in the parameter map according to the absolute difference and SAD. parameter value. For example, the map adjustment module 860 can use the following code (3) to generate the adjusted parameter map allin_map:

if(SAD>TH_SAD)

Fac=LUT[SAD];

allin_map=map×Fac

else

allin_map=map

(3)

Wherein, Fac represents the weight factor used by the map adjustment module 8620 to adjust the parameter map. It can be seen that when the absolute difference and SAD are greater than the movement threshold TH_SAD, the map adjustment module 860 determines the weighting factor Fac of each parameter value according to the absolute difference and SAD, and adjusts the parameter value in the parameter map by using the weighting factor Fac. Among them, the weight factor Fac decreases with the increase of absolute difference and SAD.

FIG. 9B is a schematic diagram illustrating the relationship between the absolute difference sum and the weighting factor according to yet another embodiment of the present invention. As shown in FIG. 9B , when the absolute difference and SAD are greater than the movement threshold TH_SAD, the map adjustment module 860 determines the weighting factor of each parameter value according to the absolute difference and SAD, and adjusts the parameter value by using the weighting factor. The weighting factor decreases as the absolute difference and SAD increase, that is, each parameter value decreases as the corresponding absolute difference and SAD increase.

Afterwards, the image synthesis module 850 can mix the first image Img1 and the displacement-corrected second image Img2_cal according to the adjusted parameter map allin_map, so as to generate the full depth image Img_AIF without ghosting. Wherein, the step of the image synthesis module 860 to generate the panoramic depth image according to the adjusted parameter map allin_map is similar to the step of the image synthesis module 460 to generate the bokeh image according to the bokeh map bokeh_map, please refer to the relevant description in FIG. 7 and so on. No longer. For example, the image synthesis module 860 can also obtain the final full depth image Img_AIF through the program code (4).

if(Map≥Blend_TH1)//In-of-focus area of image2

Img1_AIF=Img2

else if(Map≥Blend_TH2)//Transition area

w _AIF =LUT[Map](LUT is table and value range is0～255)

$\mathrm{<span\; class="notranslate">\; Img</span>}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; AIF</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; AIF</span>}}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; Img</span>}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 256</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; AIF</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; Img</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; 256</span>}}$

else//In-of-focus area of image1

Img1_AIF=Img1

(4)

Wherein, in this exemplary program code (4), it is assumed that the parameter value is between 0 and 255, Blend_TH1 is the first blending threshold value, Blend_TH2 is the second blending threshold value, and Map is the multiple in the adjusted parameter map allin_map parameter value, LUT[] is the look-up table function. It is worth mentioning that the pixels in the edge area can be calculated by the concept of weight. As shown in the formula in the above exemplary code, the parameter value is used as the synthesis weight w _AIF , and the pixels in the edge area are synthesized by the synthesis weight w _AIF .

Similarly, it can be seen from the description of FIG. 3 that the image processing method of the present invention can obtain a final output image through multiple images. Based on this, in this embodiment, the image acquisition device 800 can acquire multiple images with different focal lengths, and use the multiple images with different focal lengths to synthesize a clear full depth image. As far as the actual application situation is concerned, the scene can be analyzed first to further determine how many images with different focal lengths are needed to synthesize a full depth of view image with clear images.

To sum up, the image acquisition device and image processing method provided by the present invention calculates a synthesized parameter map by using at least two images with different focal lengths, and synthesizes a clear subject image or panoramic depth based on the parameter map. image. Wherein, the image processing method provided by the present invention can make more than one main object clear and the background blurred, so as to highlight more than one main object in the image. In addition, through the present invention, the connection edge between the subject and the background in the image can be made soft and natural, and an image with good and natural bokeh effect can be achieved. On the other hand, the present invention can also use multiple images obtained from different focusing distances to create a full depth image in which every place in the image is clearly in focus. In addition, when the full depth image is created, the noise in the image can also be eliminated to ensure that the created full depth image will not lose details in the image.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (16)

1. an image processing method, is applicable to image acquiring device, it is characterized in that, this image processing method comprises:

Obtain the first image with the first focal length, and obtain the second image with the second focal length, wherein this first focal length is focused at least one main body;

This second image is carried out to geometric correction program, produce this second image after displacement correction;

Each pixel of this image is carried out to gradient computing to produce multiple the first Grad, and each pixel of this second image after displacement correction is carried out to this gradient computing to produce multiple the second Grad;

Respectively this first Grad, with corresponding respectively this second Grad to produce multiple the first pixel comparative results, and produces the first parameter map according to those the first pixel comparative results; And

Produce composograph according to this first parameter map and this first image, and at least produce output image according to this composograph.

2. image processing method according to claim 1, is characterized in that, the step that at least produces this output image according to this composograph comprises:

Obtain the 3rd image with the 3rd focal length;

The 3rd image is carried out to this geometric correction program, produce the 3rd image after displacement correction;

Each pixel of this composograph is carried out to this gradient computing to produce multiple the 3rd Grad, and each pixel of the 3rd image after displacement correction is carried out to this gradient computing to produce multiple the 4th Grad;

Respectively the 3rd Grad, with corresponding respectively the 4th Grad to produce multiple the second pixel comparative results, and produces the second parameter map according to those the second pixel comparative results; And

According to this second parameter map, the 3rd image and this composograph that mix after displacement correction produce this output image.

3. image processing method according to claim 1, is characterized in that, this second image is carried out to this geometric correction program, and the step that produces this second image after displacement correction comprises:

This first image and this second image are carried out to amount of movement estimation, use calculating homography matrix; And

According to this homography matrix, this second image is carried out to the affine conversion of geometry, to obtain this second image after displacement correction.

4. image processing method according to claim 1, it is characterized in that, respectively this first Grad is with corresponding respectively this second Grad to produce those the first pixel comparative results, and the step that produces this parameter map according to those the first pixel comparative results comprises:

Those second Grad, divided by those corresponding first Grad, are produced to multiple gradient comparison values; And

Produce multiple parameter values according to those gradient comparison values, and those parameter values are recorded as to this parameter map.

5. image processing method according to claim 4, is characterized in that, the step that produces those parameter values according to those gradient comparison values comprises:

Judge whether those gradient comparison values are greater than the first gradient critical value; And

If those gradient comparison values are greater than this first gradient critical value, setting corresponding those parameter values of those gradient comparison values is the first numerical value.

6. image processing method according to claim 5, is characterized in that, the step that produces those parameter values according to those gradient comparison values comprises:

Be greater than this first gradient critical value if those gradient comparison values there is no, judge whether those gradient comparison values are greater than the second gradient critical value;

If those gradient comparison values are greater than this second gradient critical value, setting corresponding those parameter values of those gradient comparison values is second value; And

Be greater than this second gradient critical value if those gradient comparison values there is no, set corresponding those parameter values of those gradient comparison values and be set as third value,

Wherein, this first gradient critical value is greater than this second gradient critical value.

7. image processing method according to claim 4, is characterized in that, the step that at least produces this composograph according to this first parameter map and this first image comprises:

This first image is carried out to obfuscation program, produce blurred picture; And

Mix this first image and this blurred picture to produce main body picture rich in detail according to this first Reference Map.

8. image processing method according to claim 7, is characterized in that, mixes this first image and this blurred picture comprises with the step that produces this main body picture rich in detail according to this first Reference Map:

Judge whether those parameter values are greater than the first mixing critical value;

If those parameter values are greater than this first mixing critical value, get the pixel of corresponding this blurred picture of those parameter values as the pixel of this main body picture rich in detail;

Be greater than this first mixing critical value if those parameter values there is no, judge whether those parameter values are greater than the second mixing critical value;

If those parameter values are greater than this second mixing critical value, go out the pixel of this corresponding main body picture rich in detail according to those parameter value calculation; And

Be greater than this second mixing critical value if those parameter values there is no, get the pixel of corresponding this first image of those parameter values as the pixel of this main body picture rich in detail, wherein, this first mixing critical value is greater than this second mixing critical value.

9. image processing method according to claim 4, is characterized in that, the step that at least produces this composograph according to this first parameter map and this first image comprises:

According to the calculated for pixel values of each pixel in this first image and this second image go out the corresponding multiple absolute differences of each pixel and, and according to those absolute differences with adjust those parameter values in this first parameter map; And

According to this first Reference Map after adjusting, mix this second image after this first image and displacement correction to produce full depth image.

10. image processing method according to claim 9, it is characterized in that, according to the calculated for pixel values of each pixel in this first image and this second image go out corresponding those absolute differences of each pixel and, and comprise according to those absolute differences and the step of adjusting those parameter values in this first parameter map:

When those absolute differences be greater than mobile critical value, according to those absolute differences with determine the respectively weight factor of this parameter value, and utilize this weight factor to adjust those parameter values, wherein respectively this parameter value along with this absolute difference of correspondence and rising and decline.

11. image processing methods according to claim 9, is characterized in that, according to this first Reference Map after adjusting via those weight factors, this second image mixing after this first image and displacement correction comprises with the step that produces this full depth image:

Judge whether those parameter values are greater than the first mixing critical value;

If those parameter values are greater than this first mixing critical value, get the pixel of this second image after the corresponding displacement correction of those parameter values as the pixel of this full depth image;

Be greater than this first mixing critical value if those parameter values there is no, judge whether those parameter values are greater than the second mixing critical value;

If those parameter values are greater than this second mixing critical value, go out the pixel of this corresponding full depth image according to those parameter value calculation; And

Be greater than this second mixing critical value if those parameter values there is no, get the pixel of corresponding this first image of those parameter values as the pixel of this full depth image, wherein, this first mixing critical value is greater than this second mixing critical value.

12. 1 kinds of image acquiring devices, is characterized in that, comprising:

Image collection module, obtains the first image with the first focal length, and obtains the second image with the second focal length, and wherein this first focal length is focused at least one main body;

Image correction module, carries out geometric correction program to this second image, produces this second image after displacement correction;

Gradient calculation module, carries out a gradient computing to produce multiple the first Grad to each pixel of this first image, and each pixel of this second image after displacement correction is carried out to this gradient computing to produce multiple the second Grad;

Map generation module, respectively this first Grad, with corresponding respectively this second Grad to produce multiple the first pixel comparative results, and produces the first parameter map according to those the first pixel comparative results; And

Image synthesis unit, produces composograph according to this first parameter map and this first image, and at least produces output image according to this composograph.

13. image acquiring devices according to claim 11, it is characterized in that, this image collection module is obtained the 3rd image with the 3rd focal length, this image correction module is carried out this geometric correction program and produces the 3rd image after displacement correction the 3rd image, this gradient calculation module is carried out this gradient computing to produce multiple the 3rd Grad to each pixel of this composograph, each pixel of three image of this gradient calculation module after to displacement correction is carried out this gradient computing to produce multiple the 4th Grad, this map generation module respectively the 3rd Grad with corresponding respectively the 4th Grad to produce multiple the second pixel comparative results, this map generation module also produces the second parameter map according to those the second pixel comparative results, three image of this image synthesis unit after according to this second parameter map mixing displacement correction and this composograph and produce this output image.

14. image acquiring devices according to claim 11, wherein this map generation module by those second Grad divided by those corresponding first Grad, to produce multiple gradient comparison values, and produce multiple parameter values according to those gradient comparison values, and those parameter values are recorded as to this parameter map.

15. image acquiring devices according to claim 11, it is characterized in that, also comprise image blurring module, module that this is image blurring is carried out an obfuscation program and is produced blurred picture this first image, and this image synthesis unit is mixed this first image and this blurred picture to produce main body picture rich in detail according to this first Reference Map.

16. image acquiring devices according to claim 11, it is characterized in that, also comprise map adjusting module, this map adjusting module according to the calculated for pixel values of each pixel in this first image and this second image go out the corresponding multiple absolute differences of each pixel and, this map adjusting module according to those absolute differences with adjust those parameter values in this first parameter map, this image synthesis unit is mixed this second image after this first image and displacement correction to produce full depth image according to this first Reference Map after adjusting.