CN114972626A - A system and method for 3D reconstruction of weld seam based on coded structured light binocular vision - Google Patents

Abstract

The present invention provides a system and method for 3D welding seam reconstruction based on coded structured light binocular vision, comprising a computer, a 3D welding seam reconstruction program, a left industrial camera, a structured light generator, a right industrial camera and a binocular mounting base; The computer processes the collected pictures through a welding seam 3D reconstruction program; the welding seam 3D reconstruction program includes coded structured light pattern generation, camera calibration, image acquisition, image processing, phase solution, epipolar correction, phase matching and 3D point cloud Generate function. The application of the technical solution does not require separate calibration of the projector and the structured light plane, reduces the calibration complexity of the three-dimensional reconstruction system, can generate dense point cloud data, and has high reconstruction accuracy.

Description

A system and method for 3D reconstruction of weld seam based on coded structured light binocular vision

technical field

The invention relates to the technical field of machine vision, in particular to a three-dimensional reconstruction system and method for welding seam based on coded structured light binocular vision.

Background technique

As an important branch in the field of machine vision, 3D reconstruction technology is widely used in the fields of robot navigation, 3D reconstruction of parts, reverse modeling of human skeleton in medicine, 3D reconstruction of cultural relics, 3D mapping of topography, and 3D reconstruction of real-time maps. application. With the continuous improvement of industrial production requirements, accurate and fast three-dimensional reconstruction technology of workpieces is particularly important. In the field of welding, traditional manual welding is limited by the technical level, physiology and other factors of the welder, and it is often difficult to meet the requirements of stability and efficiency of welding work. With the continuous development of welding technology and industrial robot technology, robot welding will gradually replace manual welding. Machine vision is added to the welding robot, so that the robot can operate through visual feedback information without touching the target workpiece, with high processing accuracy. , fast processing speed and other characteristics, adding 3D reconstruction technology to the welding robot system can improve the welding seam recognition accuracy and the automation degree of welding processing, and it is of great significance to expand the application field of robots.

The commonly used 3D reconstruction methods include binocular stereo vision and structured light method. The pure binocular method is greatly affected by the surface texture of the workpiece, indicating that it is difficult to find feature points for the workpiece to be welded with weak texture, and it is difficult to generate dense point cloud data; The solution of light plus monocular camera needs to calibrate the projector, the calibration process is complicated and difficult, and the error is difficult to eliminate.

Patent CN202110732138 (application date is June 30, 2021, publication date is September 10, 2020) discloses a method for feature extraction and three-dimensional reconstruction of weld shape based on laser scanning, according to the principle of laser scanning, optical triangulation The optical structure system is designed according to the principle of the method, and the change information of the horizontal and vertical positions of the weld is calculated through the geometric relationship; the deviation of the actual weld center point is deduced according to the position change of the center point of the weld image. 3D reconstruction restores the overall 3D image after welding. But there are still the following problems: 1) The scanning structured light projection period is long; 2) It is difficult to calibrate the structured light plane; 3) It is difficult to generate dense point cloud data.

Patent CN201710084398 (application date is February 16, 2017, publication date is July 21, 2017) discloses a three-dimensional reconstruction method of weld seam based on two-dimensional line structured light The target weld is photographed at regular intervals, and then the two-dimensional structured light image of the target weld is obtained. The actual length and width of a pixel in the structured light image are obtained by calibrating the structured light image. ; Preprocess the two-dimensional structured light image of the weld, and then use the contour averaging method to extract the center line of the weld to obtain the actual coordinates of each pixel of the center line of the weld; The actual coordinates of the center line of a weld are fitted with a B-spline-based interpolation method for 3D surface fitting, thereby reconstructing a complete 3D weld surface. The method has high reliability, simple method and high detection and recognition efficiency. However, there are still the following problems: 1) the structured light plane needs to be calibrated, and the process is complicated; 2) the scanning structured light projection period is long; 3) the dense point cloud data cannot be generated.

Patent CN201610280356 (application date is April 29, 2016, publication date is July 20, 2016) discloses a three-dimensional reconstruction method for curved welds based on line structured light visual inspection. The line structured light vision system includes a camera, a line A structured light projector, the linear structured light projector is fixed on one side of the camera and projected on the surface of the workpiece to be welded, and the steps of the three-dimensional reconstruction method for a curved weld include system calibration, weld detection, and weld reconstruction, Therefore, a complete three-dimensional curved weld seam is reconstructed, and the method can detect various weld seams, and the method is simple and the detection efficiency is high. But there are still some problems: 1) the structured light plane needs to be calibrated, and the calibration error is not easy to control; 2) the scanning projection period is long; 3) the dense point cloud data cannot be generated.

To sum up, most of the existing methods for 3D reconstruction of welds use line structured light, which requires the calibration of the structured light plane and is difficult to calibrate. Scanning line structured light often has a long projection period and low 3D reconstruction efficiency.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a three-dimensional reconstruction system and method for welding seam based on coded structured light binocular vision, to carry out three-dimensional reconstruction of the workpiece to be welded and the welding seam, and to convert the pixel coordinates of the image captured by the camera into the coordinates of the workpiece. 3D coordinates to generate dense point cloud data. The method does not require separate calibration of the projector and the structured light plane, which reduces the calibration complexity of the 3D reconstruction system, can generate dense point cloud data, and has high reconstruction accuracy.

In order to achieve the above object, the present invention adopts the following technical scheme: a three-dimensional reconstruction system for welding seam based on coded structured light binocular vision, including a computer, a three-dimensional welding seam reconstruction program, a left industrial camera, a structured light generator, a right industrial camera and The binocular mounting base; the spaced angle between the left industrial camera and the right industrial camera is installed on the mounting base; the structured light generator is mounted between the left industrial camera and the right industrial camera; the structured light generator is used for projection tape A structured light pattern with coded information; the left industrial camera and the right industrial camera are used to capture the image of the workpiece to be welded covered by the coded structured light pattern and transmit it to the computer through the data line; the computer uses the welding seam 3D reconstruction program The acquired pictures are processed; the three-dimensional reconstruction program of the weld seam includes the functions of coding structured light pattern generation, camera calibration, image acquisition, image processing, phase solution, epipolar correction, phase matching and three-dimensional point cloud generation.

The present invention also provides a three-dimensional reconstruction method for welding seam based on coded structured light binocular vision, comprising the following steps:

Step A1: Build a structured light binocular vision hardware platform including a computer, a left industrial camera, a structured light generator, a right industrial camera and a binocular mounting base;

Step A2: Adjust the relative position of the left industrial camera and the right industrial camera according to the working distance to maximize the public field of view, adjust the aperture and focal length of the camera, and fix the position, aperture and focal length of the camera, and the left and right cameras form a binocular vision system;

Step A3: Connect the computer and the binocular vision system, calibrate the binocular vision system through the welding seam 3D reconstruction program, and obtain the internal and external parameters of the two industrial cameras;

Step A4: Generate a coded structured light pattern through a welding seam 3D reconstruction program and import it into a projector, and control the projector to project the coded structured light image to the workpiece to be welded while the binocular vision system collects the image;

Step A5: The three-dimensional reconstruction program of the welding seam in the computer processes the collected images, obtains the three-dimensional point cloud data of the workpiece to be welded, and completes the three-dimensional reconstruction of the welding seam.

In a preferred embodiment, the three-dimensional reconstruction program of the weld seam in step A3 uses the Zhang Zhengyou calibration method to obtain the radial distortion parameters k1, k2, k3 and tangential distortion parameters p1, p2 of the camera for the binocular vision system, and the binocular joint The calibration obtains the rotation matrix R and translation matrix T of the right industrial camera relative to the left industrial camera.

In a preferred embodiment: in the step A4 welding seam three-dimensional reconstruction program, select 3 kinds of gratings with wavelengths of 28, 26 and 24 respectively, adopt four-step phase shift, and the phase shift formula for generating fringes is:

为相位值，

为相移值，f为相移频率，x为编码方向上的像素坐标，N为相移步数。where A is the background light intensity, B is the modulation intensity,

is the phase value,

is the phase shift value, f is the phase shift frequency, x is the pixel coordinate in the encoding direction, and N is the number of phase shift steps.

In a preferred embodiment:

The realization of step A5 includes the following steps:

Step C1: Obtain three kinds of wrapped phase maps of different frequencies through the collected images. The phase formula for the simultaneous solution of multiple equations is:

Step C2: Perform modulation filtering on the solved phase to eliminate noise areas:

Step C3: based on the rotation matrix R and the translation matrix T described in right 3, the left and right images are carried out polar correction by the polar correction algorithm, so that the left and right images are coplanar and aligned;

Step C4: Based on the corrected left and right unwrapped phases obtained in step C3, use a phase stereo matching algorithm to obtain the parallax d corresponding to each point in the left and right unwrapped phases;

Step C5: Establish the world coordinate system OW-XWYWZW with a certain reference point in the outside world, record the coordinates of the object point in the space as P=[XW, YW, ZW], and establish the camera coordinate system OC-XCYCZC with the camera optical center, so that its z axis Point to the positive direction of the camera, the coordinates of the object point in the camera coordinate system are P=[XC, YC, ZC], take the projection of the optical center on the imaging plane as the origin to establish the image coordinate system and establish the pixel coordinate system on the image, Use u and v to represent the columns and rows of image pixels;

Step C6: According to the conversion relationship of the four coordinate systems, the conversion relationship between pixel coordinates and world coordinates can be obtained as:

Step C7: Based on the calibration parameters obtained in claim 3, the pixel coordinates are corrected, and the radial distortion and the tangential distortion are combined, there are:

Get the known undistorted pixel coordinates (u, v);

Step C8: The correspondence between the pixel coordinates and the world coordinates obtained according to the parallax principle and step C5 is as follows:

The focal length f and the camera baseline b are obtained by calibration, the point coordinates Pl(ul,vl) of the point P in the space on the left camera and the corresponding point Pr(ur,vr) of the corresponding right camera;

Step C9: Generate a dense point cloud image according to the three-dimensional coordinates of the weld seam of the workpiece to be tested obtained in C8, and complete the three-dimensional reconstruction.

Compared with the prior art, the present invention has the following beneficial effects:

A system and method for three-dimensional reconstruction of welding seam based on coded structured light binocular vision proposed by the present invention utilizes a structured light coding method combining multi-frequency heterodyne principle and phase shift method to uniquely mark each point on the surface of the workpiece to be welded, Then, based on the binocular camera to capture the marked image of the workpiece to be welded, phase unwrapping is performed to obtain the absolute phase, and epipolar correction is performed according to the internal and external parameters obtained by the binocular camera to obtain left and right images with no distortion, parallel imaging planes and aligned rows. The phase of the left and right images is matched to obtain the corresponding parallax. Finally, the three-dimensional coordinates are calculated according to the principle of parallax to generate a dense point cloud image, and the three-dimensional reconstruction of the workpiece to be welded is completed.

The system and method for 3D welding seam reconstruction based on coded structured light binocular vision proposed by the present invention does not require calibration of the projector and the structured light plane, reduces the calibration difficulty of the vision system, and can generate dense point cloud data at the same time. , which improves the 3D reconstruction accuracy of the weld seam of the workpiece to be welded.

Description of drawings

FIG. 1 is a flow chart of performing three-dimensional reconstruction of a weld according to a preferred embodiment of the present invention;

Fig. 2 is the schematic diagram of the hardware platform of the preferred embodiment of the present invention;

FIG. 3 is a flowchart of a welding seam 3D reconstruction program according to a preferred embodiment of the present invention;

FIG. 4 is a flowchart of the phase matching algorithm in the three-dimensional reconstruction program of the weld according to the preferred embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and not intended to limit the exemplary embodiments in accordance with the present application; as used herein, unless the context clearly dictates otherwise, the singular forms are also intended to include Plural forms, furthermore, should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, they indicate the presence of features, steps, operations, devices, components and/or combinations thereof.

A welding seam 3D reconstruction system based on coded structured light binocular vision, referring to Figures 1 to 4, includes a computer 1, a welding seam 3D reconstruction program 2, a left industrial camera 3, a structured light generator 4, a right industrial camera 5 and a dual A mounting base 6; the left industrial camera 3 and the right industrial camera 5 are mounted on the mounting base 6 at a certain angle; the structured light generator 5 is installed in the middle of the left industrial camera 3 and the right industrial camera 4, and the left industrial camera The camera 3 and the right industrial camera 5 are distributed axially symmetrically; the structured light generator 4 is used for projecting a mechanism light pattern with coded information; the left industrial camera 3 and the right industrial camera 5 are used to collect the encoded structure. The image of the workpiece to be welded covered by the light pattern is transmitted to the computer 1 through the data line; the computer 1 processes the collected pictures through the three-dimensional reconstruction program of the welding seam; the three-dimensional reconstruction program of the welding seam includes coded structured light pattern generation, camera Calibration, image acquisition, image processing, dephasing, epipolar correction, phase matching and 3D point cloud generation.

The method for 3D reconstruction of weld seam based on coded structured light binocular vision includes the following steps:

Step A1: Build a structured light binocular including a computer, a left industrial camera (Hikvision MV-CE060-10UM), a structured light generator (UKCSIT9), a right industrial camera (Hikvision MV-CE060-10UM) and a binocular mounting base visual hardware platform;

Step A2: Adjust the relative position of the left industrial camera and the right industrial camera according to the working distance (400mm) to maximize the public field of view, and adjust the aperture and focal length of the camera to obtain a clear image, and fix the position, aperture and focal length of the camera, left and right The camera forms a binocular vision system;

Step A3: Connect the computer and the binocular vision system, calibrate the binocular vision system through the welding seam 3D reconstruction program, and obtain the internal and external parameters of the two industrial cameras;

Step A4: Generate a coded structured light pattern through a welding seam 3D reconstruction program and import it into a projector, and control the projector to project the coded structured light image to the workpiece to be welded while the binocular vision system collects the image;

Step A5: The three-dimensional reconstruction program of the welding seam in the computer processes the collected images, obtains the three-dimensional point cloud data of the workpiece to be welded, and completes the three-dimensional reconstruction of the welding seam.

Preferably, in step A3, the radial distortion parameters k1, k2, k3 and tangential distortion parameters p1, p2 of the camera are obtained by Zhang Zhengyou's calibration method, and the rotation matrix R and translation matrix T of the right camera relative to the left camera are obtained by the binocular joint calibration. .

Preferably, in step A4, three gratings with wavelengths of 28, 26 and 24 are selected according to the principle of multi-frequency heterodyne, and four-step phase shift is adopted, and the phase shift formula for generating fringes is:

为相位值，

为相移值，f为相移频率，x为编码方向上的像素坐标，N为相移步数。where A is the background light intensity, B is the modulation intensity,

is the phase value,

is the phase shift value, f is the phase shift frequency, x is the pixel coordinate in the encoding direction, and N is the number of phase shift steps.

Preferably, the realization of step A5 includes the following steps:

Step C1: Obtain three kinds of wrapped phase maps of different frequencies through the collected images. The phase formula for the simultaneous solution of multiple equations is:

Step C2: Perform modulation filtering on the solved phase to eliminate noise areas:

Step C3: based on the rotation matrix R and the translation matrix T described in right 3, polar correction is carried out to the left and right images by the polar correction algorithm, so that the left and right images are coplanar and aligned.

Step C4: Based on the corrected left and right unwrapped phases obtained in step C3, a phase stereo matching algorithm is used to obtain the parallax d corresponding to each point in the left and right unwrapped phases.

Step C5: Establish the world coordinate system OW-XWYWZW with a certain reference point in the outside world, record the coordinates of the object point in the space as P=[XW, YW, ZW], and establish the camera coordinate system OC-XCYCZC with the camera optical center, so that its z axis Point to the positive direction of the camera, the coordinates of the object point in the camera coordinate system are P=[XC, YC, ZC], take the projection of the optical center on the imaging plane as the origin to establish the image coordinate system and establish the pixel coordinate system on the image, Use u and v to represent the columns and rows of image pixels;

Step C6: According to the conversion relationship of the four coordinate systems, the conversion relationship between pixel coordinates and world coordinates can be obtained as:

Step C7: Based on the calibration parameters obtained in claim 3, the pixel coordinates are corrected, and the radial distortion and the tangential distortion are combined, there are:

Get the known undistorted pixel coordinates (u, v);

Step C8: The correspondence between the pixel coordinates and the world coordinates obtained according to the parallax principle and step C5 is as follows:

The focal length f and the camera baseline b are obtained by calibration, the point coordinates Pl(ul,vl) of the point P in the space on the left camera and the corresponding point Pr(ur,vr) of the corresponding right camera.

Step C9: Generate a dense point cloud image according to the three-dimensional coordinates of the weld seam of the workpiece to be tested obtained in C8, and complete the three-dimensional reconstruction.

This patent is not limited to the above-mentioned best embodiment, and any person skilled in the art can derive other various forms of three-dimensional reconstruction methods of welding seam under the inspiration of this patent. All equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of this patent.

Claims (5)

1. a welding seam three-dimensional reconstruction system based on coded structured light binocular vision, is characterized in that, comprises computer, welding seam three-dimensional reconstruction program, left industrial camera, structured light generator, right industrial camera and binocular mounting base; The interval angle between the left industrial camera and the right industrial camera is installed on the mounting base; the structured light generator is installed between the left industrial camera and the right industrial camera; the structured light generator is used for projecting structured light with coded information pattern; the left industrial camera and the right industrial camera are used to collect the image of the workpiece to be welded covered by the coded structured light pattern and transmit it to the computer through the data line; the computer processes the collected image through the welding seam three-dimensional reconstruction program; The three-dimensional reconstruction program of the welding seam includes the functions of coded structured light pattern generation, camera calibration, image acquisition, image processing, dephasing, epipolar correction, phase matching and three-dimensional point cloud generation. 2. a welding seam three-dimensional reconstruction method based on coded structured light binocular vision, is characterized in that comprising the following steps: Step A1: Build a structured light binocular vision hardware platform including a computer, a left industrial camera, a structured light generator, a right industrial camera and a binocular mounting base; Step A2: Adjust the relative position of the left industrial camera and the right industrial camera according to the working distance to maximize the public field of view, adjust the aperture and focal length of the camera, and fix the position, aperture and focal length of the camera, and the left and right cameras form a binocular vision system; Step A3: Connect the computer and the binocular vision system, calibrate the binocular vision system through the welding seam 3D reconstruction program, and obtain the internal and external parameters of the two industrial cameras; Step A4: Generate a coded structured light pattern through a welding seam 3D reconstruction program and import it into a projector, and control the projector to project the coded structured light image to the workpiece to be welded while the binocular vision system collects the image; Step A5: The three-dimensional reconstruction program of the welding seam in the computer processes the collected images, obtains the three-dimensional point cloud data of the workpiece to be welded, and completes the three-dimensional reconstruction of the welding seam. 3. a kind of welding seam three-dimensional reconstruction method based on coded structured light binocular vision according to claim 2, is characterized in that, the welding seam three-dimensional reconstruction procedure of step A3 adopts Zhang Zhengyou calibration method to obtain the diameter of camera to binocular vision system. The directional distortion parameters k1, k2, k3 and the tangential distortion parameters p1, p2 are jointly calibrated by binocular to obtain the rotation matrix R and translation matrix T of the right industrial camera relative to the left industrial camera. 4. a kind of welding seam three-dimensional reconstruction method based on coded structured light binocular vision according to claim 2, is characterized in that: in step A4 welding seam three-dimensional reconstruction program, select 3 kinds of wavelengths to be 28, 26 and 24 respectively The grating of , using four-step phase shift, the phase shift formula for generating fringes is:

where A is the background light intensity, B is the modulation intensity,

is the phase value,

is the phase shift value, f is the phase shift frequency, x is the pixel coordinate in the encoding direction, and N is the number of phase shift steps. 5. a kind of welding seam three-dimensional reconstruction method based on coded structured light binocular vision according to claim 2, is characterized in that: The realization of step A5 includes the following steps: Step C1: Obtain three kinds of wrapped phase maps of different frequencies through the collected images. The phase formula for the simultaneous solution of multiple equations is:

Step C2: Perform modulation filtering on the solved phase to eliminate noise areas:

Step C3: based on the rotation matrix R and the translation matrix T described in right 3, the left and right images are carried out polar correction by the polar correction algorithm, so that the left and right images are coplanar and aligned; Step C4: Based on the corrected left and right unwrapped phases obtained in step C3, use a phase stereo matching algorithm to obtain the parallax d corresponding to each point in the left and right unwrapped phases; Step C5: Establish the world coordinate system OW-XWYWZW with a certain reference point in the outside world, record the coordinates of the object point in the space as P=[XW, YW, ZW], and establish the camera coordinate system OC-XCYCZC with the camera optical center, so that its z axis Point to the positive direction of the camera, the coordinates of the object point in the camera coordinate system are P=[XC, YC, ZC], take the projection of the optical center on the imaging plane as the origin to establish the image coordinate system and establish the pixel coordinate system on the image, Use u and v to represent the columns and rows of image pixels; Step C6: According to the conversion relationship of the four coordinate systems, the conversion relationship between pixel coordinates and world coordinates can be obtained as:

Step C7: Based on the calibration parameters obtained in claim 3, the pixel coordinates are corrected, and the radial distortion and the tangential distortion are combined, there are:

Get the known undistorted pixel coordinates (u, v); Step C8: The correspondence between the pixel coordinates and the world coordinates obtained according to the parallax principle and step C5 is as follows:

The focal length f and the camera baseline b are obtained by calibration, the point coordinates Pl(ul,vl) of the point P in the space on the left camera and the corresponding point Pr(ur,vr) of the corresponding right camera; Step C9: Generate a dense point cloud image according to the three-dimensional coordinates of the weld seam of the workpiece to be tested obtained in C8, and complete the three-dimensional reconstruction.

Images