CN107270875A - Visual signature three-dimensional rebuilding method under motion blur effects - Google Patents

Abstract

The three-dimensional reconstruction method of visual features under the motion blur effect of the present invention comprises the steps of: calibrating the camera to be used; arranging coded marking points on the surface of the object to be measured; obtaining a motion blurred image; identifying the identity of the coded marking points in the image; For the same coded point, with the help of time series images taken at different times, its spatial position at different times is roughly positioned and fitted into a spline curve as the initial value of the spatial motion trajectory; Fuzzy imaging model; within each exposure time, according to the fuzzy imaging model, optimize and solve the motion path and attitude. In the case of motion blur, the present invention restores the central position and attitude of the coded mark points within the exposure time, and obtains the three-dimensional information of the surface of the object to be measured and the motion information within the exposure time. The invention extends the vision-based measurement method to dynamic occasions, and plays an important role in the analysis, design and reverse engineering of high-speed moving parts.

Description

3D reconstruction method of visual features under the effect of motion blur

Technical field:

The invention relates to a three-dimensional reconstruction method of visual features under motion blur effect, belonging to the field of machine vision measurement.

Background technique:

Coded markers are widely used in machine vision-based industrial metrology and reverse engineering. Before the measurement, coded markers are placed on the surface of the measured object. According to a set of images of the measured object taken by a pair of or more calibrated cameras, the location information of the coded marker points in space can be reconstructed, so as to obtain the three-dimensional parameters of the measured object.

When the measured object is in high-speed motion, the acquired image is blurred. At this time, the traditional method of identifying the identity of the coded marker becomes invalid. Existing methods for locating the center of coded markers in clear images are completely unsuitable for identifying the identity of coded markers in fuzzy images, and are even more unsuitable for locating the centers of coded markers in fuzzy images.

Invention content:

The present invention provides a three-dimensional reconstruction method of visual features under the effect of motion blur in order to solve the problems existing in the above-mentioned prior art, which can still restore the center of the code mark point to Precise position at any moment during the exposure.

The technical solution adopted in the present invention includes: a method for three-dimensional reconstruction of visual features under the effect of motion blur, comprising the following steps:

Step 1: Calibrate the camera to be used;

Step 2: Arrange coded marking points on the surface of the measured object;

Step 3: Obtain a motion blurred image;

Step 4: identify the identity of the coded marker point in the image;

Step 5: For the same coded marker point, with the help of time series images taken at different times, roughly locate its spatial position at different times and fit it into a spline curve as the initial value of the spatial trajectory;

Step 6: Constructing a fuzzy imaging model for coding marker motion;

Step 7: In each exposure time, according to the fuzzy imaging model, optimize and solve the motion path and attitude.

The present invention has the following beneficial effects: the present invention can restore the central position and attitude of the coded marker point within the exposure time in the case of motion blur, thereby obtaining the three-dimensional information of the surface of the measured object and the motion information within the exposure time. The present invention extends vision-based measurement methods to dynamic situations. The analysis, design and reverse engineering of high-speed moving parts will play an important role.

Description of drawings:

Figure 1 is a schematic diagram of exposure timing.

Figure 2 is a schematic diagram of clear coding points.

Fig. 3 is a schematic diagram of motion blur coding points.

detailed description:

The present invention will be further described below in conjunction with the accompanying drawings.

The three-dimensional reconstruction method of visual features under the motion blur effect of the present invention comprises the following steps:

1. Calibrate a pair of cameras, respectively denoted as left camera C ₀ and right camera C ₁ , and their imaging matrices are respectively denoted as P ₀ and P ₁ . The distortion coefficient vectors of the two camera lenses are represented as d ^(c) , c=0,1, respectively.

2. Select the threshold T _p for epipolar constraint detection.

3. Select the ID number of the coding point to be used. The identity number is a natural number with a value between 1 and N ₀ , where N ₀ is the total number of coding points in the complete set. Record the set of selected coding points as N is the total number of selected coding points.

4. Right In each id _n , n=1,2,...,N. An image M _n of encoding points is prepared. All images have the same pixel size and the same number of width and height pixels, denoted as z.

5. Make actual coded marker stickers according to M _n , with side length l.

6. Paste the actual coded markers on the surface of the measured object. A coded point with the same number can appear at most once in a measurement.

7. Acquire a set of motion blurred images, ie pairs of images acquired by a pair of cameras at multiple times. Due to the motion blur effect, the image of the encoded markers in each picture is blurred to varying degrees. use respectively represent the kth image captured by the left (c=0) and right (c=1) cameras. K is the total number of shots. The duration of each shot is Δt, the start time of each shot is T _k , k=1, 2, . . . K, and the exposure times of two adjacent shots do not overlap. In two adjacent shots, the time interval between the end of the previous exposure and the start of the next exposure is the same, which is recorded as ΔT.

8. According to the distortion coefficient vector d ^(c) of the camera lens, correct The lens distortion effect, the result is recorded as

9. For each image Segmentation is performed so that each small block obtained after segmentation exactly contains a complete blurred image of a coded marker point. The number of image patches contained in is denoted as The segmented image blocks are denoted as Where c=0,1 correspond to the left and right cameras respectively, k=1,2,...K corresponds to the shooting sequence, correspond The sth small block in . at the center of The pixel coordinates in

10. In order to use the method in the remarks (remarks: 1. Use computer simulation to generate various motion blur images of different code points; 2. Construct a deep convolutional neural network; 3. Use the images generated by simulation to train deep convolutional neural networks Network; 4. use the network after training to identify the motion-blurred image of the coded marker point that is actually taken, and obtain its identity id) identify the identity of the fuzzy coded marker point in the image, and need to image small blocks Do preprocessing. This method uses a deep convolutional network MBCNet to identify the identity of the coded markers of motion blur. To construct the network, the width and height dimensions must be specified for the input layer, and w represents the width and height of the image required by the input layer, in pixels.

11. Perform size preprocessing on each small image block, assuming that the width of each small block is w _H pixels, and the height is w _V pixels. (1) No processing is required if w _H =w _V =w. (2) If max{w _H , w _V }≠w, scale the small image block times, and then symmetrically add blank areas of the same gray scale as the background to the top, bottom or left and right of the small image block, so that the width and height of the image are both w pixels. The image block after preprocessing is denoted as

12. For each Use the method in the above remarks to identify the identity of the fuzzy coded markers contained in it, denoted as place

13. Screen the identified coded markers, the steps are:

a) Filter all id∈ID, if there is a certain k∈{1,2,...,K},

Then mark this id as invalid.

b) Screen all id∈IDs that are not currently marked as invalid, if there exists a certain k∈{1,2,...,K}, it is not below the threshold T _p level and satisfies the epipolar constraint on the left and right images , it is marked as invalid.

c)

14. For each encoded marker point identity Calculate the initial value of the fitting start and end endpoints The steps are:

a) For each k∈{1,2,...,K}, for each c∈{0,1}, there exists some make according to And two camera matrices P ₁ , P ₂ reconstruct the initial value M _id,k of the spatial position of the encoding marker point id at the kth moment, and its three-dimensional coordinates are (x _id,k ,y _id,k ,z _id,k ) ^T.

b) Generate K-th degree B-spline curve SP _id according to M _id,k ,k=1,2,...,K interpolation. SP _id passed each M _id,k . The parametric equation of SP _id is expressed as

c) Calculate the arc length of SP _id , which is denoted as σ _id , and parameterize the approximate arc length of SP _id . The equation of the curve after reparameterization is denoted as V _id (t),t∈[0,σ _id ]. At this time, V _id (0)=M _id,1 , and V _id (σ _id )=M _id,K .

d) The parameter corresponding to each M _id,K on the SP _id is t _id,k , that is, M _id,k =V _id (t _id,k ),k=1,2,...,K.

e) Calculate the half window size

f) For k=1, calculate

g) For k=2,3,...,K-1, calculate

h) For k=K, calculate

15. Construct the static virtual imaging model of the code point moving in space, the steps are:

a) Set camera C and code point E to be located in the same three-dimensional space coordinate system.

b) Set the imaging matrix of camera C to P, without distortion.

c) The image of the coding point id is denoted as M, a binary image, the gray value is 0 or 1, and the length and width are 1 pixel. In its own plane, according to the counterclockwise direction, the homogeneous coordinates of the four vertices are respectively

d) The encoding point E to be imaged is a square plane with a side length l, and a pattern M is pasted on one side, filling the square without distortion.

e) M(u) is an image M of code points expressed in functional form. The parameter u is the homogeneous coordinate (u, v, s) ^T , and its corresponding non-homogeneous coordinate is M(u) represents the position on the image The gray value of the pixel at . If this coordinate is a non-integer value, the gray value is generated by interpolation. If the coordinate falls outside the image, the grayscale value returned by the function is 0.

f) The position of E in space is completely determined by the coordinates of the four vertices. When the image of the coding point faces the viewer, in the counterclockwise direction, the four vertices are Q ₁ (v,x), Q ₂ (v,x), Q ₃ (v,x), Q ₄ (v,x ), where the parameter vector v=(α,β,γ) ^T , x=(x,,y,z) ^T , determine the attitude and position respectively.

g) Each of Q ₁ (v,x), Q ₂ (v,x), Q ₃ (v,x), and Q ₄ (v,x) is composed of obtained through coordinate transformation.

α, β, γ determine the rotation matrix

x,y,z determine the translation matrix For i-1,2,3,4,

h) The homogeneous coordinates of the four vertices of E on the image plane of the camera C are respectively z _i =PQ _i (v,x).

i) Construct the homography matrix H such that in the sense of homogeneous coordinates

j) Then the encoding point is imaged in the camera C as I _M,v,x,P at this position and attitude, and its functional form is I _M,v,P (u)=M(Hu), where u=(u ,v,1) ^T is the homogeneous coordinate of the pixel position (u,v) ^T.

16. Use the I _M,v,x,P constructed in the previous step to construct a fuzzy imaging model for the motion of the encoding point. The steps are:

a) Select the discrete granularity N as a natural number, generally above 100 and below 1000. If the value of N is large, the blur effect is closer to the real effect.

b) Under the premise of short-time exposure, the attitude angle v=(α, β, γ) ^T of the encoding point remains unchanged during the limited movement process.

c) On the premise of short-time exposure, the movement is limited to a straight line segment with a uniform speed, the starting point is x ₁ =(x ₁ ,y ₁ ,z ₁ ) ^T , and the end point is x ₂ =(x ₂ ,y ₂ ,z ₂ ) ^T.

d) The fuzzy imaging result is

17. For each encoded marker point identity For each exposure number k=1, 2, . . . , K, the motion path is fitted. The steps are:

a) Record the image corresponding to the code point id as M. remember c=0, 1 is the left and right images captured for the kth time, record is the small image block containing the motion blur coding marker of the code point, where c=0, 1 correspond to the left and right cameras respectively, and s is the number of the small image block segmented from the image obtained by this shooting.

b) Select optimization variables as θ ₁ , θ ₂ , θ ₃ , λ ₁ , λ ₂ , λ ₃ , μ ₁ , μ ₂ , μ ₃ , ω ₁ , ω ₂ .

C) The initial values of θ ₁ , θ ₂ , θ ₃ are randomly selected within 0 to 2π.

d) The initial value of (λ ₁ ,λ ₂ ,λ ₃ ) is

e) The initial value of (μ ₁ ,μ ₂ ,μ ₃ ) is

f) ω ₁ is the image gain, the initial value is 1.

g) ω ₂ is the image offset, the initial value is 0.

h) Set v=(λ ₁ ,λ ₂ ,λ ₃ ) ^T , x ₁ =(λ ₁ ,λ ₂ ,λ ₃ ) ^T , x ₂ =(μ ₁ ,μ ₂ ,μ ₃ ) ^T .

i) Set mask function Where c, k, s mean the same as The corresponding symbols in have the same meaning. Parameter u=(u,v,s) ^T , is the pixel coordinate, when the coordinate falls on the small block of the image When within the range of pixel coordinates occupied in its parent image, Returns 1, otherwise returns 0.

j) Calculate and optimize the objective function

where |||| ² represents the square of the norm. When W is an image, ||W|| ² is the sum of the squares of the gray values of all pixels in the image.

k) By optimizing parameters θ ₁ , θ ₂ , θ ₃ , λ ₁ , λ ₂ , λ ₃ , μ ₁ , μ ₂ , μ ₃ , ω ₁ , ω ₂ to obtain the minimum optimal value of f.

l) Assign different random values to θ ₁ , θ ₂ , θ ₃ each time, repeat the above steps b) to k), and select the one with the smallest optimized value of f as the final optimization result. The number of repetitions is not less than 16 times.

m) After the operation is completed, within the exposure time, the motion track of the coding point is a straight line segment from (λ ₁ , λ ₂ , λ ₃ ) to (μ ₁ , μ ₂ , μ ₃ ). During this exposure time, the attitude parameters of the encoding point are (θ ₁ , θ ₂ , θ ₃ ,).

The above is only a preferred embodiment of the present invention, it should be pointed out that for those of ordinary skill in the art, some improvements can also be made without departing from the principle of the present invention, and these improvements should also be regarded as the invention. protected range.

Claims (10)

1. A visual feature three-dimensional reconstruction method under motion blur effect, is characterized in that: comprise the following steps Step 1: Calibrate the camera to be used; Step 2: Arrange coded marking points on the surface of the measured object; Step 3: Obtain a motion blurred image; Step 4: identify the identity of the coded marker point in the image; Step 5: For the same coded marker point, with the help of time series images taken at different times, roughly locate its spatial position at different times and fit it into a spline curve as the initial value of the spatial trajectory; Step 6: Constructing a fuzzy imaging model for coding marker motion; Step 7: In each exposure time, according to the fuzzy imaging model, optimize and solve the motion path and attitude. 2. The method for three-dimensional reconstruction of visual features under motion blur effect as claimed in claim 1, characterized in that: in said step 1, a pair of cameras are calibrated, respectively marked as left camera C ₀ and right camera C ₁ , their The imaging matrices are respectively denoted as P ₀ and P ₁ , and the distortion coefficient vectors of the two camera lenses are respectively denoted as d ^(c) , c=0,1; the threshold T _p is selected for epipolar constraint detection. 3. The visual feature three-dimensional reconstruction method under the motion blur effect as claimed in claim 2, is characterized in that: in the described step 2: (a). Select the identity number of the coded point to be used. The identity number is a natural number with a value between 1 and N _0. N ₀ is the total number of the complete set of coded points. Record the selected coded point The set of N is the total number of selected coding points; (b). Yes For each id _n , n=1,2,...,N, the image M _n of the encoding point is prepared, all images have the same pixel size, and the number of width and height pixels is recorded as z; (c). Make actual coded marking point stickers according to M _n , and the side length is l; (d). Paste the actual coded markers on the surface of the measured object. 4. The visual feature three-dimensional reconstruction method under the motion blur effect as claimed in claim 3, is characterized in that: in the described step 3: (a). Obtain a motion blur image group, that is, a pair of images acquired by a pair of cameras at multiple times. Due to the motion blur effect, the imaging of the coding markers in each picture has different degrees of blurring. Use respectively represent the kth image taken by the left (c=0) and right (c=1) cameras, K is the total number of shots, the duration of each shot is Δt, and the starting time of each shot is T _k , k= 1,2,...K, the exposure time of two adjacent shots does not overlap, and the interval between the end of the previous exposure and the start of the next exposure is the same, which is recorded as ΔT; (b). According to the distortion coefficient vector d ^(c) of the camera lens, correct The lens distortion effect, the result is recorded as (c). For each image Carry out segmentation so that each small block obtained after segmentation exactly contains a complete fuzzy image of a coded marker point, The number of image patches contained in is denoted as The segmented image blocks are denoted as Where c=0,1 correspond to the left and right cameras respectively, k=1,2,...K corresponds to the shooting sequence, correspond The sth small block in , at the center of The pixel coordinates in 5. The visual feature three-dimensional reconstruction method under the motion blur effect as claimed in claim 4, is characterized in that: in described step 4: (a). For small image blocks For preprocessing, use a deep convolutional network MBCNet to identify the identity of the motion-blurred coded markers. To build this network, you must specify the width and height dimensions for the input layer. Use w to represent the width and height of the image required by the input layer, in pixels ; (b). Perform size preprocessing on each small image block, assuming that the width of each small block is w _H pixels, and the height is w _V pixels, (1) if w _H =w _V =w then no processing is required, ( 2) If max{w _H ,w _V }≠w, scale the small image block times, and then symmetrically add a blank area of the same gray level as the background on the top, bottom or left and right of the small image, so that the width and height of the image are both w pixels, and the preprocessed image small block is denoted as (c). For each Identify the identity of the fuzzy coded markers contained in it, denoted as use Represents the set of all codepoints identified. 6. The visual feature three-dimensional reconstruction method under the motion blur effect as claimed in claim 5, characterized in that: in the step five: (a). Screening the identified coding markers; (b). Calculate the initial value of the fitting start and end endpoints for each coded marker identity id∈ID 7. The visual feature three-dimensional reconstruction method under the motion blur effect as claimed in claim 6, is characterized in that: the identified coded marker points are screened, the steps are: a) Filter all id∈ID, if there is a certain k∈{1,2,...,K}, Then mark this id as invalid; b) Screen all id∈IDs that are not currently marked as invalid, if there exists a certain k∈{1,2,...,K}, it is not below the threshold T _p level and satisfies the epipolar constraint on the left and right images , it is marked as invalid; c) 8. The visual feature three-dimensional reconstruction method under the motion blur effect as claimed in claim 7, is characterized in that: for each coding mark point identity Calculate the initial value of the fitting start and end endpoints The steps are: a) For each k∈{1,2,...,K}, for each c∈{0,1}, there exists some make according to And two camera matrices P ₁ , P ₂ reconstruct the initial value M _id,k of the spatial position of the encoding marker point id at the kth moment, and its three-dimensional coordinates are (x _id,k ,y _id,k ,z _id,k ) ^T ; b) Generate K times B-spline curve SP _id according to M _id,k ,k=1,2,...,K interpolation, SP _id passes each M _id,k , and the parameter equation of SP _id is expressed as c) Calculate the arc length of SP _id , denoted as σ _id , and parameterize the approximate arc length of SP _id , after reparameterization, the equation of the curve is denoted as V _id (t),t∈[0,σ _id ], here When V _id (0)=M _id,1 ,V _id (σ _id )=M _id,K ; d) The parameter corresponding to each M _id,K on SP _id is t _id,k , that is, M _id,k =V _id (t _id,k ),k=1,2,...,K; e) Calculate the half window size f) For k=1, calculate g) For k=2,3,...,K-1, calculate h) For k=K, calculate 9. The visual feature three-dimensional reconstruction method under the motion blur effect as claimed in claim 8, is characterized in that: In step six, construct a clear imaging model as follows: a) Set the camera C and the encoding point E to be located in the same three-dimensional space coordinate system; b) Set the imaging matrix of camera C to P, without distortion; c) The image of the coded point id is denoted as M, a binary image, the grayscale value is 0 or 1, and the length and width are l pixels. In its own plane, according to the counterclockwise direction, the homogeneous coordinates of the four vertices respectively d) The encoding point E to be imaged is a square plane with a side length l, and a pattern M is pasted on one side, filling the square without distortion; e) M(u) is the image M of the coding point expressed in the form of a function, where the parameter u is the homogeneous coordinate (u, v, s) ^T , and its corresponding non-homogeneous coordinate is M(u) represents the position on the image The gray value of the pixel at; f) The position of E in space is completely determined by the coordinates of the four vertices. When the image of the encoding point faces the observer, the four vertices are Q ₁ (v,x), Q ₂ (v,x) in the counterclockwise direction ), Q ₃ (v,x), Q ₄ (v,x), where the parameter vector v=(α,β,γ) ^T , x=(x,,y,z) ^T , respectively determine the attitude and position; g) Each of Q ₁ (v,x), Q ₂ (v,x), Q ₃ (v,x), and Q ₄ (v,x) is composed of After coordinate transformation, α, β, γ determine the rotation matrix x,y,z determine the translation matrix For i-1,2,3,4, h) The homogeneous coordinates of the four vertices of E on the image plane of the camera C are respectively z _i =PQ _i (v,x); i) Construct the homography matrix H such that in the sense of homogeneous coordinates j) Then the encoding point is imaged in the camera C as I _M,v,x,P at this position and attitude, and its functional form is I _M,v,x,P (u)=M(Hu), where u= (u,v,1) ^T is the homogeneous coordinate of the pixel position (u,v) ^T. 10. The method for three-dimensional reconstruction of visual features under motion blur effect as claimed in claim 9, characterized in that: in the step seven: (a). Utilize the I _M,v,x,P constructed in the previous step to construct the motion blur imaging model of the coding point; (b). For each coded point identity For each exposure number k=1, 2, . . . , K, the motion path is fitted.