CN104811680B - Image acquisition device and image deformation correction method thereof - Google Patents

Abstract

The invention provides an image acquisition device and an image deformation correction method thereof. The image acquisition device is provided with a first image sensor and a second image sensor. The image deformation correction method includes the following steps. A plurality of image groups are acquired through a first image sensor and a second image sensor. Each image group comprises a first image and a second image respectively, and the image groups comprise a reference image group. Whether the first reference image and the second reference image in the reference image group have image deformation or not is detected. And when the reference image group is detected to generate image deformation, updating the current correction parameters according to the comparison values of the plurality of characteristic points corresponding to the image groups. The current correction parameters are used for carrying out image correction on each first image and each corresponding second image.

Description

Image acquisition device and image deformation correction method thereof

technical field

The present invention relates to an image acquisition device, and in particular to an image acquisition device and an image deformation correction method thereof.

Background technique

For the current image depth sensing technology, it is a common method to use an image acquisition device with dual lenses to acquire images corresponding to different viewing angles, and the 3D depth information of the target can be calculated through the images corresponding to different viewing angles . Therefore, in order to accurately obtain the three-dimensional depth information of the target object from the two-dimensional image, the spatial arrangement relationship between the two lenses needs to be specially designed, and precise parameter correction is a necessary step. Furthermore, when the factory manufactures an image acquisition device with dual lenses, the respective spatial positions and directions of the dual lenses cannot be set extremely accurately at preset settings. Therefore, in the process of manufacturing the image acquisition device, the factory will calibrate the set dual-lens module in advance, so as to obtain a set of factory-preset calibration parameters. In the future, when the user operates the image acquisition device, the image acquisition device can use the factory preset calibration parameters to correct the image acquired through the dual lenses, so as to overcome the lack of precision in the manufacturing process.

However, when a user operates or carries the image acquisition device, when the image acquisition device is affected by extrusion, impact or drop, the lens may be displaced or rotated and other changes in spatial position may occur. Once the lens is displaced or deformed, the calibration parameters preset in the factory no longer meet the current application conditions, and the image acquisition device cannot acquire correct depth information. For example, if there is a problem of level imbalance between the two lenses of the stereoscopic image acquisition device, the level of the left and right images captured after the imbalance will not match, which will further lead to poor 3D stereoscopic shooting effect.

Contents of the invention

In view of this, the present invention provides an image acquisition device and an image distortion correction method thereof, which can adaptively adjust correction parameters for image rectification according to the displacement state of the image sensor.

The invention proposes an image distortion correction method, which is suitable for an image acquisition device with a first image sensor and a second image sensor. The image acquisition device has current calibration parameters associated with the first image sensor and the second image sensor, and the image distortion correction method includes the following steps. A plurality of image groups are acquired by the first image sensor and the second image sensor, wherein each image group includes a first image and a second image respectively, and these image groups include a reference image group. Detecting whether image deformation occurs in the first reference image and the second reference image in the reference image group. When it is detected that image deformation occurs in the reference image groups, the current correction parameters are updated according to the comparison values of the plurality of feature points corresponding to these image groups. The current correction parameters are used to perform image correction on each first image and each corresponding second image.

In an embodiment of the present invention, the above-mentioned step of detecting whether image deformation occurs in the reference image group includes the following steps. The first feature point of the first reference image and the second feature point of the second reference image are detected. It is judged whether the offset between the first feature point and the second feature point and the image coordinates of the first reference image and the second reference image respectively exceeds a threshold value. If the determination is yes, it is determined that image deformation occurs in the reference image group.

In an embodiment of the present invention, the above-mentioned step of detecting whether image deformation occurs in the reference image group includes the following steps. The 3D depth estimation is performed according to the first reference image and the second reference image to generate reference depth information of the reference focus object in the reference image group, and the depth focus position of the reference object is obtained according to the reference depth information. The auto-focus position with respect to the reference target is acquired through the auto-focus procedure. It is judged whether the depth focus position corresponding to the reference focus target conforms to the auto focus position. If the determination is negative, it is determined that image deformation occurs in the reference image group.

In an embodiment of the present invention, before the above-mentioned step of updating the current correction parameters according to the comparison values of the feature points corresponding to the image groups, the image correction method further includes the following steps. 3D depth estimation is performed on these image groups to generate depth information of each image group. Whether to keep the image group is determined according to the depth information of each image group.

In an embodiment of the present invention, the above-mentioned step of updating the current calibration parameters according to the comparison values of the feature points corresponding to the image groups further includes the following steps. Feature point detection is performed on the first images and the second images, and multiple first feature points of the first images and multiple second feature points of the second images are acquired. Comparing the coordinate positions of these first feature points and the coordinate positions of a plurality of second feature points respectively corresponding to these first feature points, so as to obtain multiple distances between these first feature points and these second feature points feature point comparison value. A plurality of feature point comparison values between the first feature points and the second feature points are recorded.

In an embodiment of the present invention, the above-mentioned step of recording the comparison values of the feature points between the first feature points and the second feature points further includes the following steps. The comparison values of each feature point are classified into a plurality of statistical slots according to the coordinate positions of the first feature points and/or the coordinate positions of the second feature points.

In an embodiment of the present invention, the above-mentioned step of updating the current calibration parameters according to the comparison values of these feature points corresponding to the image group includes the following steps. According to the number of comparison values of multiple feature points in the statistical slots and the multiple depth values corresponding to the comparison values of feature points, it is judged whether the comparison values of these feature points are sufficient for operation. If the judgment is yes, the current calibration parameters are updated according to the comparison values of these feature points. These depth values are obtained by performing 3D depth estimation on the first feature point and the corresponding second feature point.

From another point of view, the present invention provides an image acquisition device, the image acquisition device has a first image sensor and a second image sensor. The image acquisition device also includes a storage unit and a processing unit. The storage unit records a plurality of modules and stores current calibration parameters associated with the first image sensor and the second image sensor. The processing unit is coupled to the first image sensor, the second image sensor and the storage unit to access and execute the modules recorded in the storage unit. These modules include acquisition module, deformation detection module and parameter update module. The acquiring module acquires a plurality of image groups through the first image sensor and the second image sensor. Each image group includes a first image of the first image sensor and a second image of the second image sensor respectively, and these image groups include a reference image group. The deformation detection module detects whether image deformation occurs in the first reference image and the second reference image in the reference image group. When image deformation occurs in the reference image groups, the parameter update module updates the current correction parameters according to the comparison values of the plurality of feature points corresponding to these image groups. The current correction parameters are used to perform image correction on each first image and each corresponding second image.

Based on the above, in the embodiment of the image distortion correction method of the present invention, when the current correction parameters cannot perform accurate image correction, the feature point information on the image groups captured at different scenes and at different time points is used to establish A database to update current calibration information with complete information in the database. In this way, even if the left and right image sensors are displaced, the image acquisition device can dynamically and adaptively generate new correction parameters, so as to avoid inaccurate image correction by using correction parameters that do not conform to the current situation. In this way, the parameter updating operation can be performed automatically without the user's awareness, so as to ensure the shooting quality of the image acquisition device and improve user experience.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail with reference to the accompanying drawings.

Description of drawings

Fig. 1 is a block diagram of an image acquisition device shown in an embodiment of the present invention;

Fig. 2 is a flowchart of an image distortion correction method shown in an embodiment of the present invention;

3A to 3B are detailed flowcharts of step S202 shown in an embodiment of the present invention;

Fig. 4 is a flowchart of an image distortion correction method shown in an embodiment of the present invention;

FIG. 5A to FIG. 5B are schematic diagrams of an embodiment of comparing values of classification feature points to statistical slots shown in an embodiment of the present invention.

Explanation of reference signs:

100: image acquisition device;

110: a first image sensor;

120: the second image sensor;

130: focusing unit;

140: processing unit;

150: storage unit;

151: Obtain the module;

152: deformation detection module;

153: parameter update module;

154: depth selection module;

TH: preset threshold value;

A, B: the first feature point;

Z1～Z9: image blocks;

S1～S9: statistics slot;

Δd _A , Δd _B : feature point comparison value;

S201-S203: each step of the image distortion correction method described in an embodiment of the present invention;

S2011-S2023: each sub-step of step S202 in an embodiment of the present invention;

S2024～S2027: each sub-step of step S202 in an embodiment of the present invention;

S401-S409: each step of the image distortion correction method according to an embodiment of the present invention.

detailed description

When the image acquisition device leaves the factory, the spatial setting relationship between the two lenses has been precisely calculated and adjusted, and a set of factory preset correction parameters has been generated accordingly. The factory preset correction parameters are used to correct the images acquired by different lenses to a designed and fixed coordinate parameter relationship. In order to solve the situation that the factory preset correction parameters are no longer applicable due to the displacement or rotation of the dual lens, the present invention generates a database recording multiple feature point information according to the depth information and pixel position of the image, and uses the accumulated The information adaptive update correction parameters. In order to make the content of the present invention more clear, the following examples are listed as examples according to which the present invention can be implemented.

FIG. 1 is a block diagram of an image acquisition device according to an embodiment of the present invention. Please refer to FIG. 1 , the image acquisition device 100 of this embodiment is, for example, a digital camera, a digital video camera, or other handheld electronic devices with image acquisition functions, such as smart phones, tablet computers, etc., and is not limited to the above. The image acquisition device 100 includes a first image sensor 110 , a second image sensor 120 , a focusing unit 130 , a processing unit 140 and a storage unit 150 .

The first image sensor 110 and the second image sensor 120 may include lenses and photosensitive elements. The photosensitive element is, for example, a Charge Coupled Device (CCD for short), a Complementary Metal-Oxide Semiconductor (CMOS for short) element or other elements, and the first image sensor 110 and the second image sensor 120 may also include Aperture, etc., are not limited here. In addition, according to the positions of the lenses of the first image sensor 110 and the second image sensor 120 , the lenses of the first image sensor 110 and the second image sensor 120 can be divided into a left lens and a right lens.

In this embodiment, the focusing unit 130 is coupled to the first image sensor 110 , the second image sensor 120 and the processing unit 140 to control the focal length of the images captured by the first image sensor 110 and the second image sensor 120 . In other words, the focusing unit 130 controls the lens of the first image sensor 110 and the lens of the second image sensor 120 to move to the focusing position. The focusing unit 130 controls the step position of the lens through a Voice Coil Motor (VCM for short) or other different types of motors, so as to change the focal lengths of the first image sensor 110 and the second image sensor 120 .

The processing unit 140 may be, for example, a central processing unit (Central Processing Unit, referred to as CPU), a microprocessor (Microprocessor), a specific application integrated circuit (Application Specific Integrated Circuits, referred to as ASIC), a programmable logic device (Programmable Logic Device, referred to as PLD) Or other hardware devices with computing capabilities. The storage unit 150 is, for example, a random access memory (random access memory), a fast memory (Flash) or other memories, and is used to store current calibration parameters and multiple modules, and the processing unit 140 is coupled to the storage unit 150 and used to execute these modules . The above-mentioned modules include an acquisition module 151 , a deformation detection module 152 , a parameter update module 153 and a depth selection module 154 . These modules are, for example, computer programs that can be loaded into the processing unit 140 to perform the function of correcting image deformation.

FIG. 2 is a flow chart of an image distortion correction method shown in an embodiment of the present invention. The method of this embodiment is applicable to the image acquisition device 100 in FIG. 1 , and the detailed steps of the image distortion correction method of this embodiment will be described below in conjunction with each component in the image acquisition device 100 .

First, in step S201 , the acquisition module 151 acquires a plurality of image groups through the first image sensor 110 and the second image sensor 120 . Each image group includes a first image and a second image, and the image group includes at least one reference image group. That is to say, in this embodiment, a single image group has two photos, and the first image and the second image in the same image group are two photos of the same scene captured by the left camera and the right camera at the same time image. In other words, the first image is, for example, a left image captured through a left lens, and the second image is relatively a right image captured through a right lens. In this embodiment, the first image and the second image are, for example, live-view images acquired in a preview state.

Similarly, the reference image group is one of the image groups acquired by the image acquisition device 100, so the reference image group also has the first reference image and the second image corresponding to the first image sensor 110 and the second image sensor 120. Reference image. In step S202 , the deformation detection module 152 detects whether image deformation occurs in the first reference image and the second reference image in the reference image group. It should be noted that the deformation detection module 152 can detect image deformations for some image groups in a timing detection manner, and can also detect image deformations for all image groups, and the reference image group here represents the deformation detection module 152 is one of the objects used to detect whether image deformation occurs.

It should be noted that the factory preset correction parameters are suitable for image rectification of the two left and right images, so that the two real images only have horizontal aberration or only vertical aberration (because the lens position is placed caused by the relationship). For example, there will be a slight difference in the angle of elevation between the two lenses and so on. Image correction is performed by factory preset correction parameters, which can convert the real image into that the left and right lenses are placed on the same imaging plane, leaving only the difference in horizontal or vertical position. That is to say, on the premise that the left and right lenses are set horizontally, there should only be a difference in the horizontal position of each pixel on the left and right images after image correction. At this time, if the shooting directions of the left and right lenses change, the vertical positions of the pixels on the left and right images after image correction still have differences, and this phenomenon is called image distortion herein. Here, the deformation detection module 152 can determine the reference image group according to the offset of the corresponding feature points on the first reference image and the second reference image or perform 3D depth estimation on the first reference image and the second reference image Whether or not the group has image warping.

To be more clear, FIG. 3A is a detailed flowchart of step S202 shown in an embodiment of the present invention. In the embodiment shown in FIG. 3A , in step S2021 , the deformation detection module 152 detects the first feature point on the first reference image and the second feature point on the second reference image. After that, in step S2022, the deformation detection module 152 judges whether the offset between the first feature point and the second feature point between the image coordinates of the first reference image and the second reference image exceeds a threshold value. If the offset between the first feature point and the second feature point between the image coordinates of the first reference image and the second reference image exceeds the threshold value, in step S2023, the deformation detection module 152 determines the reference image group Image distortion occurs. That is to say, by analyzing and counting the displacement information between the first feature point and the second feature point, it can be known whether image deformation occurs between the first reference image and the second reference image.

In other words, the deformation detection module 152 can detect any feature point of the reference set of images according to the existing algorithm method of feature point detection. The algorithm of feature point detection is used to detect multiple feature points in the image, such as edge detection, corner detection (conner detection) or other feature point detection algorithm, which is not limited in the present invention. Afterwards, the deformation detection module 152 judges whether the offset of the coordinate position between the corresponding first feature point and the second feature point exceeds the above-mentioned threshold value, and accordingly detects whether image deformation occurs in the reference group of images. For example, the deformation detection module 152 can determine whether the vertical offset (displacement difference in the vertical direction) of the first feature point and the second feature point corresponding to each other exceeds the above-mentioned threshold value. When the deformation detection module 152 determines that the above-mentioned vertical offset exceeds the above-mentioned threshold value, it means that image deformation occurs in the reference image group.

In another embodiment, FIG. 3B is a detailed flowchart of step S202 shown in an embodiment of the present invention. In the embodiment shown in FIG. 3B , in step S2024, the deformation detection module 152 performs 3D depth estimation according to the first reference image and the second reference image, so as to generate reference depth information of the reference focusing target in the reference image group , and obtain the depth focus position of the reference target object according to the reference depth information. Next, in step S2025 , the deformation detection module 152 obtains the auto-focus position of the reference target through the auto-focus procedure. In step S2026 , the deformation detection module 152 determines whether the depth focus position corresponding to the reference focus target conforms to the auto focus position. If the determination is negative, in step S2027, the deformation detection module 152 determines that image deformation occurs in the reference image group.

Specifically, the deformation detection module 152 may perform image processing through stereo vision technology to obtain the three-dimensional coordinate position of the object in space and the depth information of each point in the image. Furthermore, the step of obtaining the depth focus position of the target object according to the depth information is, for example, querying a depth comparison table according to the depth information to obtain the focus position of the target object. Therefore, by obtaining the corresponding relationship between the number of steps of the stepping motor or the current value of the voice coil motor and the sharpness of the target object in advance, the stepping motor corresponding to the depth information can be queried based on the currently obtained depth information of the target object The number of steps or the current value of the voice coil motor, and based on this, the depth focus position of the target object can be obtained.

On the other hand, the process of executing the auto-focus program can be to automatically control the lens module to move in a large range through the focus unit 130, so as to adjust the lenses of the first image sensor 110 and the second image sensor 120 to the required focus positions, respectively. Get the autofocus position about the target. For example, the focusing unit 130 acquires the auto-focus position of the target object by using the hill-climbing method used in the auto-focus technology, but the present invention is not limited thereto. Therefore, under the condition that no image deformation occurs in the first reference image and the second reference image, the image acquisition device 100 can acquire ideal depth information, so that the depth focus position is consistent with the auto focus position. If the image acquisition device 100 cannot further obtain the ideal depth information according to the current calibration parameters, it will not be able to estimate the correct depth focus position by querying the depth comparison table between the depth information and the pre-stored depth information. The AF position obtained by the AF routine will vary. Accordingly, the deformation detection module 152 determines whether image deformation occurs in the reference image group according to the difference between the depth focus position and the auto focus position.

Referring to FIG. 2 again, when the deformation detection module 152 detects that the reference image group is deformed, in step S203, the parameter update module 153 updates the current correction parameters according to the comparison values of multiple feature points corresponding to the multiple image groups, wherein The current correction parameters are used to perform image correction on each first image and each corresponding second image. That is to say, when the image acquisition device 100 determines that the first image sensor 110 and the second image sensor 120 are deformed or displaced so that the parameter coordinates between the first reference image and the second reference image change, it represents the current correction parameter It is no longer possible to perform accurate image correction on the image.

Therefore, in one embodiment, the parameter update module 153 starts to collect the feature point comparison values of multiple image groups captured after the reference image group, so as to generate The acquired image after warping or shifting yields ideal current correction parameters. It should be noted that the parameter update module 153 can obtain these feature point comparison values by comparing the coordinate positions of the first feature points with the coordinate positions of the second feature points respectively corresponding to the first feature points. Furthermore, the parameter updating module 153 can also generate new current correction parameters according to the depth information of the image and the coordinate positions of the pixels. Another embodiment will be listed below to describe in detail.

FIG. 4 is a flow chart of an image distortion correction method shown in an embodiment of the present invention. Please refer to FIG. 4 , the method of this embodiment is applicable to the image acquisition device 100 of FIG. 1 , and the detailed steps of the image distortion correction method of this embodiment will be described below with each component in the image acquisition device 100 .

First, in step S401 , the acquisition module 151 acquires a plurality of image groups through the first image sensor 110 and the second image sensor 120 . Each image group includes a first image and a second image respectively, and the image group includes a reference image group. In step S402 , the deformation detection module 152 detects whether image deformation occurs in the first reference image and the second reference image in the reference image group. Step S401 and step S402 are similar or identical to step S201 and step S202 of the foregoing embodiment, and are not repeated here.

If the deformation detection module determines that image deformation occurs in the reference image group, in step S403, the depth selection module 154 performs three-dimensional depth estimation on the image group to generate depth information of each image group, and according to the depth of each image group The message determines whether to keep the group of pictures. Further, the depth selection module 154 can generate a 3D depth map associated with the first reference image and the second reference image through stereo vision image processing technology. Based on the depth information in the 3D depth map, the depth selection module 154 can obtain the depth range corresponding to the 3D depth map, and decide to keep or discard the image group according to the depth range.

In detail, assume that the minimum value of the depth value is set to 0 and the maximum value is set to 128, that is, the depth value of the image group falls within the range of 0-128. If the depth selection module 154 has collected image groups whose depth ranges from the depth value 100 to the depth value 128, the depth selection module 154 will not keep other image groups whose depth range falls within the depth value 100 to the depth value 128. On the contrary, if the depth selection module 154 determines that the depth of field range of the current image group falls outside the depth value 100 to the depth value 128, for example, the image group with the depth value 0 to the depth value 80, the depth selection module 154 will keep this image group, so as to further utilize the feature point information of this image group.

In other words, the depth selection module 154 determines whether each image group is a valid image group according to the depth information of each image group. If the depth-of-field range of the newly acquired image module has mostly overlapped with the depth-of-field range of the previous image group, the depth selection module 154 will filter it accordingly. Based on this, in an embodiment, in addition to retaining and filtering the image groups by judging whether the depth ranges overlap, the depth selection module 154 may also determine whether to retain the image groups according to the overlapping ratio of the depth ranges. Based on this, it can ensure that the depth selection module 154 collects information corresponding to all or most of the depth of field ranges, and at the same time filters redundant information according to the depth of field ranges corresponding to each image group, so as to reduce the amount of data processing and increase the speed of data processing .

After that, in step S404, the parameter update module 153 performs feature point detection on the first image and the second image, and acquires a plurality of first feature points of the first image and a plurality of second feature points of the second image. In step S405, the parameter update module 153 compares the coordinate position of the first feature point with the coordinate position of the second feature point corresponding to the first feature point to obtain the distance between the first feature point and the second feature point. The feature point comparison value of . In step S406, the parameter update module 153 records the comparison value of the feature points between the first feature point and the second feature point.

Further, the parameter update module 153 can also detect the feature points of the first image and the second image in each image group according to the existing algorithm method of feature point detection, so as to obtain the first feature point and A second feature point on the second image. Next, the parameter update module 153 judges the offset (offset) of the matched first feature point and the second feature point in the same coordinate system and uses the offset as a feature point comparison value. Wherein, the matched first feature point and the second feature point are projected to the same position in the scene to be shot. In other words, the feature point comparison value can also be regarded as the aberration between the first feature point and the second feature point. Afterwards, the parameter update module 153 records the comparison value of the feature points between the first feature point and the second feature point in a database, so as to establish a correction database for updating the current correction parameters. It is worth mentioning that when it is determined that the reference image group is deformed, the acquisition module 151 continues to acquire images to obtain multiple image groups, and the parameter update module 153 also continues to calculate the characteristics of each image group generated by calculation. Point comparison values are recorded to the calibration database.

In step S407, the parameter update module 153 classifies the comparison values of each feature point into a plurality of statistical slots according to the coordinate position of the first feature point and/or the coordinate position of the second feature point. That is to say, the parameter updating module 153 not only records the comparison value of the feature points in the calibration database, but also classifies the comparison values of the feature points into different statistical slots according to the coordinate positions in the coordinate system. Specifically, in one embodiment, the parameter update module 153 can divide the first image acquired by the first image sensor 110 into a plurality of image blocks, and each image block corresponds to a statistical slot. Therefore, according to the coordinate position of the first feature point, the parameter updating module 153 may sequentially map the feature point comparison value to one of the image blocks, thereby classifying the feature point comparison value into the corresponding statistical slot.

For example, FIG. 5A and FIG. 5B are schematic diagrams of an embodiment of comparing classification feature points to statistical slots according to an embodiment of the present invention. In this embodiment, first referring to FIG. 5A , the parameter updating module 153 divides the first image Img1 into nine image blocks Z1 - Z9 of three times three. The parameter updating module 153 further determines the image block where the first feature point falls according to the coordinate position of the first feature point. As shown in FIG. 5A , the parameter updating module 153 can know that the first feature point A is located in the image block Z2 according to the coordinate position of the first feature point A. Similarly, the parameter update module 153 can know that the first feature point B is located in the image block Z6 according to the coordinate position of the first feature point B.

Please refer to FIG. 5B , in this embodiment, the first image Img1 is divided into nine image blocks Z1 - Z9 , and the image blocks Z1 - Z9 correspond to statistical slots S1 - S9 respectively. Wherein, the image block Z1 corresponds to the statistical slot S1, and the image block Z2 corresponds to the statistical slot S2, and so on. Based on this, since the first feature point A is located in the image block Z2, the feature point comparison value Δd _A corresponding to the first feature point A is classified into the statistical slot S2. Since the first feature point B is located in the image block Z6, the feature point comparison value Δd _B corresponding to the first feature point B is classified into the statistical slot S6. It should be noted that, FIG. 5A and FIG. 5B are only exemplary implementations, and are not intended to limit the present invention.

In addition, in one embodiment, based on the fact that the first feature point and the corresponding second feature point are at the same position in the scene to be photographed, the corresponding feature points on the first image and the second image are calculated by coordinate conversion and projected to the same coordinate point in the 3D coordinate system. Therefore, the parameter update module 153 can classify the comparison values of the feature points into corresponding statistical slots according to the projected coordinate positions of the first feature point and the second feature point under the three-dimensional coordinate system. Furthermore, image processing is performed based on stereo vision technology, and the parameter updating module 153 can obtain depth information and corresponding three-dimensional coordinate positions of each point in the image. The parameter updating module 153 can use the horizontal component and the vertical component of the 3D projection points corresponding to the first feature point and the second feature point to determine the statistical slot corresponding to the comparison value of the feature point.

Therefore, in step S408, the parameter update module 153 judges whether the feature point comparison values are sufficient for calculation according to the number of feature point comparison values in each statistical slot and the multiple depth values corresponding to the feature point comparison values. The depth value is obtained by performing 3D depth estimation on the first feature point and the corresponding second feature point. It can be known that as the number of image groups increases, the amount of information recorded in the correction database also increases. Here, the parameter updating module 153 can determine whether the amount of data currently recorded in the calibration database is sufficient according to the number of comparison values of feature points in each statistical slot and the depth value of comparison values of feature points.

Based on the foregoing, it can be seen that these feature point comparison values have been classified into corresponding statistical slots by the parameter updating module 153 according to the feature point information. Therefore, the parameter update module 153 can determine whether the calibration database has sufficient data volume according to the total number of comparison values of feature points in each statistical slot. In detail, in order to generate the current correction parameters conforming to the current lens setting conditions to straighten the left and right images to an ideal state, the feature points providing feature point information should preferably be evenly distributed on the image. Through the feature point information provided by the feature points evenly distributed in each area of the image, the rotation amount or skew of the entire image can be calculated more accurately.

In this embodiment, these feature point comparison values are classified into corresponding statistical slots by the parameter update module 153 according to the coordinate positions of the feature points, so the total number of feature point comparison values corresponding to each statistical slot can represent a feature The spatial distribution of points. Therefore, the parameter updating module 153 judges whether the number of feature point comparison values in each statistical slot is sufficient, as a decision mechanism for determining whether the amount of data is enough to calculate an accurate current calibration parameter.

Taking Figure 5B as an example for illustration, the parameter update module 153 classifies the feature point comparison values into nine statistical slots S1-S9, wherein the statistical slot S1 includes at least the feature point comparison value ΔdA, and the statistical slot S6 includes at least the feature point ratio pair value ΔdB. When the parameter update module 153 keeps recording the comparison values of the feature points of different image groups in the statistics slots, the comparison values of the feature points in the statistics slots are also continuously accumulated (shown by the dotted line in FIG. 5B ). Once the comparison values of the feature points in the statistical slots S1 - S9 are sufficient, the parameter updating module 153 can start updating the current calibration parameters. For example, the parameter updating module 153 can determine whether the number corresponding to each statistical slot S1 - S9 exceeds a preset threshold TH to determine whether the amount of data in the calibration database is sufficient. However, the above-mentioned embodiment is only an exemplary implementation manner, and is not intended to limit the present invention. Those with ordinary knowledge in this technical field can select the classification method of the feature comparison value and the judgment condition for judging whether the amount of data is sufficient according to the statistical slot according to actual needs, and will not be repeated here.

In addition, the parameter update module 153 can also determine whether the data volume in the current calibration database is sufficient through the depth information. In detail, the depth values (depth values) corresponding to the comparison values of these feature points can be obtained by performing three-dimensional depth estimation on the first feature point and the corresponding second feature point. Based on the foregoing, it can be known that the depth selection module 154 has filtered out image groups with excessive depth range repeatability, and the parameter update module 153 will determine whether corresponding feature point comparison values have been collected for most of the depth values. In other words, in one embodiment, the parameter update module 153 also classifies each feature point comparison value according to the depth value corresponding to each feature point comparison value, and judges whether the number of feature point comparison values corresponding to each depth value is enough, and determine whether the amount of data in the calibration database is sufficient.

It is worth mentioning that if the spatial arrangement relationship between the first image sensor 110 and the second image sensor is changed during the establishment of the calibration database, it means that the recorded data in the calibration database is no longer available. In one embodiment, the parameter updating module 153 may also determine whether the comparison value of the feature point is significantly different from the data in the calibration database before recording the comparison value of the feature point into the calibration database. If so, the parameter updating module 153 may discard the previously recorded data and start building another calibration database, so as to obtain the most ideal current calibration parameters.

That is to say, once the parameter update module 153 determines that the amount of data in the calibration database is sufficient, the parameter update module 153 can stop recording and start calculating new current calibration parameters. On the contrary, the parameter update module 153 keeps recording new comparison values of feature points to the calibration database. Therefore, if the determination in step S408 is yes, in step S409, the parameter updating module 153 updates the current calibration parameters according to the feature point comparison value. The parameter update module 153, for example, uses the feature comparison value in the correction database to perform an optimization algorithm to find the best current correction parameters, so that the two images corrected by the current correction parameters can correspond to the ideal parameter coordinate relationship . The optimization algorithm is, for example, gradient descent method (gradient decent method), Levenberg-Marquardt method (Levenberg-Marquardt method, LM method for short) or Gauss-Newton algorithm (Gauss-Newton method), etc. This is unlimited.

To sum up, in one embodiment of the present invention, when the image acquisition device detects that the image is deformed in time, the current correction parameters pre-stored in the image acquisition device are adaptively corrected through the establishment of the correction database, so that the left and right The image is corrected to the ideal parametric coordinate relationship. In this way, the adjustment of the current calibration parameters can be performed without the user's awareness, so as to ensure the shooting quality of the image acquisition device. Furthermore, the embodiment of the present invention can further judge whether the feature point information in the database is collected completely according to the depth information and the feature point position. In this way, once a sufficient amount of data is collected, new current calibration parameters can be generated immediately by correcting the comparison values of feature points recorded in the database, thereby greatly shortening the time required for data collection and calibration.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (14)

1. An image distortion correction method applicable to an image acquisition device having a first image sensor and a second image sensor, wherein the image acquisition device has current correction parameters associated with the first image sensor and the second image sensor, It is characterized in that the image distortion correction method includes: A plurality of image groups captured at different time points are acquired by the first image sensor and the second image sensor, wherein each of the image groups includes a first image and a second image, and the image groups include a reference image group; Detecting whether image deformation occurs in the first reference image and the second reference image in the reference image group; and When it is detected that the image deformation occurs in the reference image group, the current correction parameter is updated according to a plurality of feature point comparison values corresponding to the image group, wherein the current correction parameter is used for each of the first image and Image correction is performed on each of the corresponding second images. 2. The image distortion correction method according to claim 1, wherein the step of detecting whether the image distortion occurs in the reference image group comprises: detecting a first feature point of the first reference image and a second feature point of the second reference image; and judging whether the offset between the first feature point and the second feature point between the image coordinates of the first reference image and the second reference image exceeds a threshold value, and if so, determining that the reference image group occurs The image is distorted. 3. The image distortion correction method according to claim 1, wherein the step of detecting whether the image distortion occurs in the reference image group comprises: Performing 3D depth estimation according to the first reference image and the second reference image to generate reference depth information of a reference focus object in the reference image group, and obtaining the depth of the reference object according to the reference depth information focus position; The auto-focus position of the reference object obtained through the auto-focus procedure; and It is determined whether the depth focus position corresponding to the reference focus target conforms to the auto focus position, and if not, it is determined that the image deformation occurs in the reference image group. 4. The image distortion correction method according to claim 1, wherein before the step of updating the current correction parameters according to the feature point comparison value corresponding to the image group, the image correction method further comprises: performing 3D depth estimation on the groups of images to generate depth information for each of the groups of images; and It is determined whether to keep the image group according to the depth information of each image group. 5. The image distortion correction method according to claim 1, wherein the step of updating the current correction parameters according to the comparison value of the feature points corresponding to the image group further comprises: performing feature point detection on the first image and the second image, and acquiring a plurality of first feature points of the first image and a plurality of second feature points of the second image; comparing the coordinate position of the first feature point with the coordinate position of the second feature point respectively corresponding to the first feature point to obtain the first feature point and the second feature point The comparison value of the feature points between; and Recording the feature point comparison value between the first feature point and the second feature point. 6. The image distortion correction method according to claim 5, wherein the step of recording the comparison value of the feature point between the first feature point and the second feature point further comprises: Classifying each feature point comparison value into a plurality of statistical slots according to the coordinate position of the first feature point and/or the coordinate position of the second feature point. 7. The image distortion correction method according to claim 6, wherein the step of updating the current correction parameters according to the comparison value of the feature points corresponding to the image group comprises: According to the number of the feature point comparison values in the statistical slot and the multiple depth values corresponding to the feature point comparison values, it is judged whether the feature point comparison values are sufficient for operation, and if so, according to the feature The point comparison value updates the current correction parameter, wherein the depth value is obtained by performing three-dimensional depth estimation on the first feature point and the corresponding second feature point. 8. An image acquisition device having a first image sensor and a second image sensor, characterized in that the image acquisition device comprises: a storage unit that records a plurality of modules and stores current calibration parameters associated with the first image sensor and the second image sensor; and A processing unit, coupled to the first image sensor, the second image sensor and the storage unit, to access and execute the modules recorded in the storage unit, the modules include: The acquisition module acquires a plurality of image groups captured at different time points through the first image sensor and the second image sensor, and each of the image groups includes the first image and the second image of the first image sensor respectively a second image of the sensor, wherein the group of images includes a group of reference images; The deformation detection module detects whether image deformation occurs in the first reference image and the second reference image in the reference image group; A parameter update module, when the image deformation occurs in the reference image group, the parameter update module updates the current correction parameter according to the comparison values of the plurality of feature points corresponding to the image group, wherein the current correction parameter is used for each Image correction is performed on the first image and the corresponding second images. 9. The image acquisition device according to claim 8, wherein the deformation detection module detects the first feature point of the first reference image and the second feature point of the second reference image, and the deformation detection module judges the Whether the offset between the first feature point and the second feature point between the image coordinates of the first reference image and the second reference image exceeds a threshold value, if so, the deformation detection module determines the reference image group Group that image warping occurs. 10. The image acquisition device according to claim 8, wherein the deformation detection module performs three-dimensional depth estimation according to the first reference image and the second reference image, so as to generate a reference focus in the reference image group The reference depth information of the target object, and obtain the depth focus position of the reference target object according to the reference depth information, the deformation detection module obtains the auto focus position of the reference target object through the auto focus program, and the deformation detection module It is judged whether the depth focus position corresponding to the reference focus target conforms to the auto focus position, if not, the deformation detection module determines that the image deformation occurs in the reference image group. 11. The image acquisition device according to claim 8, wherein the module further comprises: A depth selection module, the depth selection module performs three-dimensional depth estimation on the image groups to generate depth information of each of the image groups, and the depth selection module determines whether to The group of images is preserved. 12. The image acquisition device according to claim 8, wherein the parameter update module performs feature point detection on the first image and the second image, and acquires a plurality of first images of the first image. The feature point and the plurality of second feature points of the second image, the parameter update module for the coordinate position of the first feature point and the second feature point respectively corresponding to the first feature point Coordinate position, to obtain the feature point comparison value between the first feature point and the second feature point, the parameter update module records the difference between the first feature point and the second feature point The feature point comparison value of . 13. The image acquisition device according to claim 12, wherein the parameter updating module classifies each feature according to the coordinate position of the first feature point or/and the coordinate position of the second feature point Points compare values to multiple stat slots. 14. The image acquisition device according to claim 13, wherein the parameter update module is based on the number of the feature point comparison values in the statistical slot and the multiple depths corresponding to the feature point comparison values value, to determine whether the comparison value of the feature point is sufficient for calculation, if so, the parameter update module updates the current correction parameter according to the comparison value of the feature point, wherein the depth value is obtained by comparing the first feature point with the The corresponding second feature points are obtained by performing three-dimensional depth estimation.