CN114998444B - A high-precision robot posture measurement system based on two-channel network - Google Patents

Abstract

The present invention relates to a high-precision posture measurement system for a robot based on a two-channel network, which belongs to the field of computer vision technology, and includes a monocular camera, a high-precision posture measurement module, a high-precision posture calibration mechanism, a static marker and an edge-end processor; when the monocular camera collects training data sets, it is fixed at the end of the high-precision posture calibration mechanism, and during online testing, it is packaged with the edge-end processor as a module and fixed at the end of a low-precision mechanical arm; the static marker is placed within the field of view in front of the monocular camera; a two-channel network model for posture measurement is implanted in the high-precision posture measurement module, and the model realizes the high-precision posture measurement of the robot by inputting the captured static marker image and the two steps of posture training and posture measurement. The present invention only collects an image and a reference image to form an image pair and inputs it into the model to complete the high-precision posture measurement, eliminating the need to design complex markers to obtain the three-dimensional distance information of the object, and greatly reducing the industrial cost.

Description

A high-precision robot posture measurement system based on two-channel network

Technical Field

The invention relates to a high-precision posture measurement system for a robot based on a two-channel network, and belongs to the technical field of computer vision.

Background Art

As robots are increasingly used, enterprises are also placing higher and higher demands on the performance of robots. Positioning accuracy is an important indicator that affects the performance of robots. The repetitive positioning accuracy of industrial robots on the market is generally high (up to 0.01 mm), but their absolute positioning accuracy is usually low (about a few millimeters). Therefore, to improve the performance of robots, it is necessary to improve the absolute positioning accuracy of robots. At present, there are many methods for absolute positioning of robots, such as improving the manufacturing accuracy of materials, kinematic calibration progress, and using lasers, ultrasonic waves and visual sensors for error compensation. Among them, visual sensors are low-cost compared to other measurement technologies, and can obtain rich information like human eyes, which is conducive to realizing intelligent control while improving positioning accuracy. However, the existing monocular vision positioning method mainly obtains the three-dimensional information of objects by designing complex markers; the binocular/multi-eye vision positioning method requires complex feature matching to obtain the target three-dimensional coordinates. Among them, the method of designing complex markers to obtain the three-dimensional information of objects can obtain the three-dimensional information of the workpiece in the world coordinate system without adding any auxiliary equipment. This type of method is more complicated in the design of markers and has higher requirements for the accuracy of markers. The method of calculating the three-dimensional coordinates of the target by binocular/multi-eye feature point matching is simple. For example, patent CN108388854A proposes a positioning method based on an improved FAST-SURF algorithm, which uses the SURF algorithm to match feature point pairs and RANSAC screening, and processes the matched feature point pairs according to the principle of triangulation to accurately locate the three-dimensional coordinates of the object. The main problem with this type of method is that it is necessary to establish sufficiently accurate matching pairs. When encountering repeated textures or textureless scenes, this type of method has poor robustness and the positioning accuracy is easily affected. In addition, since matching is time-consuming, the real-time performance of this type of method is also poor. Another type of method is a camera pose estimation method based on deep learning. This type of method mainly trains image pairs and achieves high-precision positioning by regressing the relative pose between two images. However, this method is based on photometric vision and pure color background, and has poor robustness to changing lighting conditions and the presence of irrelevant background problems. It is also a heavy quantitative model and is difficult to achieve industrial deployment.

Summary of the invention

The purpose of the present invention is to provide a high-precision robot posture measurement system based on a two-channel network, which solves the problems of the prior art that complex markers need to be designed, low positioning accuracy, low robustness to changing lighting conditions and irrelevant backgrounds, and large models that are difficult to deploy.

In order to achieve the above object, the technical solution adopted by the present invention is:

A high-precision posture measurement system for a robot based on a two-channel network, comprising a monocular camera, a high-precision posture measurement module, a high-precision posture calibration mechanism, a static marker and an edge processor; the monocular camera is fixed at the end of the high-precision posture calibration mechanism during the process of collecting a training data set; the monocular camera is connected to the edge processor via a connecting line during an online test, and is packaged into a module using an integrated packaging technology, and the module is fixed at the end of a low-precision robotic arm; the static marker is placed within the field of view in front of the monocular camera; a two-channel network model for posture measurement is implanted in the high-precision posture measurement module, and the two-channel network model realizes high-precision posture measurement of the robot by inputting a captured static marker image and two steps of posture training and posture measurement.

A further improvement of the technical solution of the present invention is that the specific process of the robot high-precision posture measurement system based on the two-channel network is as follows:

Step 1: Fix the monocular camera at the end of the high-precision posture calibration mechanism, and adjust the position of the static marker so that it can be clearly imaged within the field of view of the monocular camera;

Step 2: Control the high-precision posture calibration mechanism to collect N static marker images, and label each image with a six-dimensional posture label to form a training data set and a verification data set; input the labeled images into the two-channel network for training through the deep learning server. After the training is completed, perform an accuracy test. When the accuracy requirements are met, deploy the trained two-channel network model to the edge processor;

Step 3: Install the monocular camera on the edge processor, integrate the monocular camera and the edge processor into a single package, and then fix them at the end of the low-precision robotic arm;

Step 4: Posture measurement: Based on the trained two-channel network model, online posture measurement of the low-precision robotic arm is performed.

A further improvement of the technical solution of the present invention is that the step 2 specifically includes the following steps:

Step 2.1: Dataset construction and sample label generation

Control the high-precision pose calibration mechanism to uniformly sample N images in a vertical cylinder with a radius of 10mm and a height of 20mm. Use the six-dimensional pose coordinates of the high-precision pose calibration mechanism as sample labels, label each image, and use a random sampling algorithm to select some of the N images as validation data sets, and the rest as training data sets.

Step 2.2: Build the network architecture

In the data loading module, images are randomly paired to form image pairs, and the image pairs are preprocessed. The image pairs are superimposed along the channel axis, that is, two single-channel images are superimposed into a dual-channel image, thereby constructing a two-channel network architecture; depthwise separable convolution is used instead of traditional convolution, and full convolution is used instead of full connection layer to reduce the number of parameters of the two-channel network model and realize the lightweight of the two-channel network model. In addition, a lightweight attention mechanism is added to the two-channel network to make the two-channel network more focused on static landmarks;

Step 2.3: Select the loss function. The main purpose of the two-channel network is to estimate the relative six-dimensional posture between any two images. It is also used for the final refinement after the rough positioning of the low-precision robot arm. Therefore, the direction measurement is within 90°. Therefore, Euler angles are used to determine the direction. The transformation label is decomposed into translation and rotation according to the form of Euler angles. The final loss is calculated as:

Among them, _ti , are the true and estimated relative translation of the image pair, _ri , are the true value and estimated value of the relative rotation of the image pair, respectively, where w (0 < w < 1) is the weight to balance the metric size between translation (mm) and rotation (degrees). Rotation has a greater impact on the position and posture of static landmark images than translation, so the w value should be given a larger value to allow the network to better learn the translation variable; the RMS root mean square error is defined as:

Where m represents the number of parameters, for translation m=3, for rotation, expressed in Euler angles, m=3.

Step 2.4: Model training and deployment. Input the preprocessed images in step 2.2 into the two-channel network for training through the deep learning server, and set the two-channel network model to be saved in static mode. After the training is completed, perform an accuracy test. Preprocess the images in the test set according to the preprocessing steps in step 2.2 and input them into the saved static mode model. When the accuracy requirements are met, deploy the trained static model on the edge processor.

A further improvement of the technical solution of the present invention lies in that the specific steps of step 4 are: when the low-precision robotic arm reaches the specified position from any direction, the monocular camera fixed on the manipulator of the low-precision robotic arm collects an image of a static marker near the target position and its surrounding scene, and forms an input image pair with the pre-shot reference posture image of the target position. The image pair is input into a trained two-channel network model, and the two-channel network model outputs the relative posture of the two images and controls the low-precision robotic arm to adjust the posture, thereby realizing high-precision six-degree-of-freedom posture measurement of the robot.

Due to the adoption of the above technical solution, the technical effects achieved by the present invention are as follows:

The present invention is robust to the problem of changing lighting conditions and irrelevant background in the image affecting the measurement accuracy, and applies deep separable convolution and full convolution, so the model is lightweight and convenient for deployment on edge processors. At the same time, the present invention only needs to collect an image and a reference image to form an image pair and input it into the model to complete the robot's high-precision posture measurement.

The present invention simplifies the process of monocular camera positioning. When the robot reaches the command position, a single-frame image is captured and input into a reference image to form an input pair, which is then input into a trained model. The network model outputs the relative position and posture of the two images and controls the robot to adjust the position and posture, thereby achieving high-precision robot position and posture measurement, eliminating the need to design complex markers to obtain three-dimensional distance information of objects.

The present invention uses a lightweight model that takes up little memory and can be deployed on edge devices. It also incorporates an attention mechanism that is robust to changing lighting conditions and irrelevant backgrounds, making it more universal in industrial scenarios.

The present invention can significantly reduce industrial costs, deploy the model trained by the high-precision posture calibration mechanism to a low-precision robot, and achieve high-precision posture measurement of the low-precision robot using only an industrial monocular camera and static markers, which can significantly reduce industrial costs.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a flow chart of the system training process of the present invention;

FIG2 is a flow chart of an online testing process of the system of the present invention;

FIG3 is a schematic diagram of the structure of the system of the present invention;

FIG4 is a block diagram of the hardware system of the training process of the present invention;

5 is a hardware system structure diagram of the online testing process of the present invention;

Among them, 1. High-precision posture calibration mechanism, 2. Monocular camera, 3. Static markers, 4. Deep learning server, 5. Low-precision robotic arm, 6. Edge processor.

DETAILED DESCRIPTION

The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

A high-precision posture measurement system for a robot based on a two-channel network, comprising a monocular camera 2, a high-precision posture measurement module, a high-precision posture calibration mechanism 1, a static marker 3 and an edge processor 6; the monocular camera 2 is connected to the edge processor 6 via a connecting line, and is packaged into a module using an integrated packaging technology, and the module is fixed at the end of the high-precision posture calibration mechanism 1 or a low-precision robotic arm 5; the static marker 3 is placed in the field of view in front of the monocular camera 2; a two-channel network model for posture measurement is implanted in the high-precision posture measurement module, and the two-channel network model realizes high-precision posture measurement of the robot by inputting a captured static marker image and two steps of posture training and posture measurement.

The monocular camera 2 is fixed at a certain position at the end of the high-precision posture calibration mechanism 1, and the corresponding monocular camera 2 is under a static marker 3. The high-precision posture calibration mechanism 1 is controlled to uniformly collect 1800 static marker images in a vertical cylinder with a radius of 10mm and a height of 20mm, and each image is marked with a six-dimensional posture label. In the data loading module, the images are randomly paired to form image pairs, and the image pairs are input into the two-channel network for training through the deep learning server 4. After the training is completed, the accuracy of the test set is tested. When the accuracy requirements are met, the trained two-channel model is deployed to the edge processor 6, and the monocular camera 2 and the edge processor 6 are integrated and packaged and installed at the end of the low-precision robotic arm 5, and then an online test is performed, that is, when the low-precision robotic arm 5 reaches the instruction position from all directions, a static marker image is collected to form an input image pair with the reference posture static marker image, and the image pair is input into the trained two-channel network model. The model outputs the relative posture of the two images and controls the low-precision robotic arm 5 to adjust the posture, thereby realizing high-precision robot six-degree-of-freedom posture measurement.

Preferably, the monocular camera 2 is composed of a camera and a lens, and is mainly used to collect images of static markers 3. In this embodiment, the monocular camera is a Daheng MER-503-20GM-P monocular camera with a resolution of 2448 (H) * 2048 (V), a frame rate of 20fps, and a data interface of GIge.

Preferably, the high-precision pose measurement module is mainly composed of a two-channel network model, which has the characteristics of being lightweight, robust to problems such as changing lighting conditions and irrelevant background in the image that affects the measurement accuracy, and can output the relative pose between any two images with an accuracy of less than 0.1 mm.

Preferably, the high-precision posture calibration mechanism 1 is mainly used for data set collection and labeling, and the six-dimensional posture coordinates of the high-precision posture calibration mechanism 1 are used as posture labels of the training data set. In this embodiment, the high-precision posture calibration mechanism 1 is a high-precision four-degree-of-freedom serial-parallel hybrid mechanism with a positioning accuracy of plus or minus 0.03 mm.

Preferably, the static marker 3 is mainly used for the image of the two-channel network model for posture measurement. By capturing a static marker image and a reference posture image to form an input pair and inputting it into the two-channel network model, posture measurement is achieved. The static marker 3 can be a self-designed marker or a fixed object existing in the scene. In this embodiment, the static marker 3 is a checkerboard calibration plate.

Preferably, the edge processor 6 is mainly used to deploy a high-precision posture measurement module. In this embodiment, NVIDIA Jetson nano is used.

The specific process of the robot high-precision posture measurement system based on the two-channel network is as follows:

Step 1: Fix the monocular camera 2 at the end of the four-degree-of-freedom serial-parallel hybrid mechanism, and adjust the position of the checkerboard calibration plate so that it can clearly image within the field of view of the monocular camera 2. In this embodiment, the position of the checkerboard calibration plate is adjusted, and the checkerboard calibration plate is about 20 cm directly below the monocular camera.

Step 2: Control the four-degree-of-freedom serial-parallel hybrid mechanism to collect N (N≥1800) chessboard calibration plate images, and label each image with a six-dimensional posture label to form a training data set and a verification data set; input the labeled images into the two-channel network for training through the deep learning server 4. After the training is completed, the accuracy test is performed. When the accuracy requirements are met, the trained two-channel network model is deployed to the edge processor 6. In this embodiment, 1800 chessboard calibration plate images are collected, of which 1700 are used as training data sets, and the rest are used as verification data sets, and the posture coordinates of the four-degree-of-freedom serial-parallel hybrid mechanism are used as the posture labels of the training data sets and the verification data sets. The flow chart of the two-channel network training process is shown in Figure 1, and the hardware system structure diagram of the training process is shown in Figure 4.

The step 2 achieves its function through the following steps:

Step 2.1: Dataset construction and sample label generation

The four-degree-of-freedom serial-parallel hybrid mechanism is controlled to uniformly sample 1800 images in a vertical cylinder with a radius of 10mm and a height of 20mm. The six-dimensional posture coordinates of the four-degree-of-freedom serial-parallel hybrid mechanism (the other two rotation variables are set to 0) are used as sample labels to label each image. A random sampling algorithm is used to select 100 images from the 1800 images as a verification data set, and the rest are used as training data sets. In this embodiment, the image is a single-channel grayscale image, and the background in the image is not pure, including the support bracket and the ground reflection problem. As the posture of the four-degree-of-freedom serial-parallel hybrid mechanism changes, these conditions also change accordingly.

Step 2.2: Build the network architecture

In the data loading module, images are randomly paired to form image pairs, and the image pairs are preprocessed. The image pairs are superimposed according to the channel axis, that is, two single-channel images are superimposed into a dual-channel image, thereby constructing a two-channel network architecture. Depth-separable convolution is used instead of traditional convolution, and full convolution is used instead of full connection layer to reduce the number of parameters of the two-channel network model and realize the lightweight of the two-channel network model. In addition, a lightweight attention mechanism is added to the two-channel network to make the two-channel network more focused on the checkerboard calibration board. In this embodiment, the lightweight attention mechanism uses Squeeze-and-Excitation, which automatically obtains the importance of each feature channel through learning and suppresses less useful features. The network framework diagram is shown in Figure 3.

Step 2.3: Select the loss function. The main purpose of the two-channel network is to estimate the relative six-dimensional posture between any two images, and it is used for the final refinement process after the rough positioning of the low-precision robot 5. Therefore, the direction measurement is within 90°, so Euler angles are used to determine the direction, and the transformation label is decomposed into translation and rotation according to the form of Euler angles. The final loss is calculated as:

Among them, _ti , are the true and estimated relative translation of the image pair, _ri , They are the true value and estimated value of the relative rotation of the image pair, respectively, where w (0 < w < 1) is the weight to balance the metric size between translation (mm) and rotation (degrees). Rotation has a greater impact on the image pose of the checkerboard calibration plate than translation, so the w value should be given a larger value to allow the network to better learn the translation variable. The RMS root mean square error is defined as:

Where m represents the number of parameters, for translation m = 3, for rotation, expressed in Euler angles, m = 3. In this example, the weight parameter w = 0.99.

Step 2.4: Model training and deployment, input the preprocessed image in step 2.2 into the two-channel network for training through the deep learning server 4, and set the two-channel network model to be saved in static mode. After the training is completed, perform an accuracy test, preprocess the image in the test set according to the preprocessing steps in step 2.2, and input it into the saved static mode model, the static mode model outputs the relative error of the translation and rotation of the image pair, and the accuracy requirement of the static mode model is that the translation error is less than 0.2mm and the rotation error is less than 0.2°. When the accuracy requirements are met, the trained static model is deployed on the NVIDIA Jetson nano processor. The translation and rotation accuracy of the model in this embodiment are both less than 0.1mm/0.1°, which meets the accuracy requirements.

Step 3: Install the monocular camera 2 on the NVIDIA Jetson nano, integrate the monocular camera 2 and the NVIDIA Jetson nano into a single package, and fix them at the end of the low-precision robotic arm 5.

Step 4, posture measurement, based on the trained two-channel network model, the online posture measurement of the low-precision manipulator 5 is performed, that is: when the low-precision manipulator 5 reaches the specified position from any direction, the monocular camera 2 fixed on the manipulator of the low-precision manipulator 5 collects a chessboard calibration plate and its surrounding scene image near the target position, and forms an input image pair with the pre-shot target position reference posture image, and the image pair is input into the trained two-channel network model. The two-channel network model outputs the relative posture of the two images and controls the low-precision manipulator 5 to adjust the posture, thereby realizing high-precision robot six-degree-of-freedom posture measurement. The low-precision manipulator 5 in this embodiment is a low-precision robot, and the two-channel network performs final refinement after the robot makes a rough estimate, and the posture error is controlled within 0.1mm/0.1°. The online measurement process flow chart is shown in Figure 2, and the hardware system structure diagram of the online test process is shown in Figure 5.

The step 4 described above realizes its function through the following steps:

Step 4.1: Define the target position that the low-precision robot arm 5 is expected to reach as D _i (i=1, 2, ..., n), and collect the chessboard calibration board image of the corresponding position, which is defined as I _i (i=1, 2, ..., n). In this embodiment, the number of target positions is 5, and 5 chessboard calibration board images of the corresponding positions are collected, and the chessboard calibration board images are saved in the database.

Step 4.2: When the low-precision manipulator 5 reaches the target position D _i from all directions, the monocular camera 2 is turned on and the chessboard calibration plate image is obtained. Then, an input image pair is formed with the reference image I _i . After preprocessing, the dual-channel image is input into the trained two-channel network model. The two-channel network model outputs the relative pose of the two images and controls the low-precision manipulator 5 to adjust the pose, thereby realizing high-precision robot six-degree-of-freedom pose measurement.

The present invention is robust to the problem of changing lighting conditions and irrelevant background in the image affecting the measurement accuracy, and applies deep separable convolution and full convolution, so the model is lightweight and convenient to deploy on the edge processor 6. At the same time, the method only needs to collect a static marker image and a reference image to form an image pair and input it into the two-channel network model to complete the robot's high-precision posture measurement.

The present invention solves the problems of the prior art that complex markers need to be designed, positioning accuracy is low, robustness to changing lighting conditions and irrelevant backgrounds is low, and the model is large and difficult to deploy. The system controls the high-precision posture calibration mechanism 1 to collect static marker images to form a training set, inputs the training set into the network to train the network parameters, and performs accuracy testing after the training is completed. When the trained model meets the accuracy requirements, it is deployed to the edge processor 6, and the online robot posture measurement can be performed, that is: after the low-precision manipulator 5 reaches the specified position from all directions, a static marker image is collected to form an input image pair with the reference posture image, and the image pair is input into the trained two-channel network model. The two-channel network model automatically outputs the relative posture of the two images, and realizes the posture adjustment of the low-precision manipulator 5, thereby realizing high-precision robot six-degree-of-freedom posture measurement.

Claims (2)

1. A robot high-precision posture measurement system based on a two-channel network, characterized in that it comprises a monocular camera (2), a high-precision posture measurement module, a high-precision posture calibration mechanism (1), a static marker (3) and an edge processor (6); the monocular camera (2) is fixed at the end of the high-precision posture calibration mechanism (1) during the process of collecting a training data set; the monocular camera (2) is connected to the edge processor (6) through a connecting line during an online test, and is packaged into a module using an integrated packaging technology, and the module is fixed at the end of a low-precision mechanical arm (5); the static marker (3) is placed in the field of view in front of the monocular camera (2); a two-channel network model for posture measurement is implanted in the high-precision posture measurement module, and the two-channel network model realizes the high-precision posture measurement of the robot by inputting a captured static marker image and two steps of posture training and posture measurement; The specific process of the robot high-precision posture measurement system based on the two-channel network is as follows: Step 1: fix the monocular camera (2) at the end of the high-precision posture calibration mechanism (1), and adjust the position of the static marker (3) so that it can be clearly imaged within the field of view of the monocular camera (2); Step 2: Control the high-precision posture calibration mechanism (1) to collect N static marker images, and label each image with a six-dimensional posture label to form a training data set and a verification data set; input the labeled images into the two-channel network for training through the deep learning server (4); after the training is completed, perform an accuracy test, and when the accuracy requirements are met, deploy the trained two-channel network model to the edge processor (6); The specific steps include: Step 2.1: Dataset construction and sample label generation Control the high-precision posture calibration mechanism (1) to uniformly sample N images in a vertical cylinder with a radius of 10 mm and a height of 20 mm, use the six-dimensional posture coordinates of the high-precision posture calibration mechanism (1) as sample labels, label each image, use a random sampling algorithm to select a portion of the N images as a verification data set, and use the rest as a training data set; Step 2.2: Build the network architecture In the data loading module, images are randomly paired to form image pairs, and the image pairs are preprocessed. The image pairs are superimposed along the channel axis, that is, two single-channel images are superimposed into a dual-channel image, thereby constructing a two-channel network architecture; depthwise separable convolution is used instead of traditional convolution, and full convolution is used instead of full connection layer to reduce the number of parameters of the two-channel network model and realize the lightweight of the two-channel network model. In addition, a lightweight attention mechanism is added to the two-channel network to make the two-channel network more focused on static landmarks (3); Step 2.3: Select the loss function. The purpose of the two-channel network is to estimate the relative six-dimensional posture between any two images. It is used for the final refinement process after the rough positioning of the low-precision robot (5). Therefore, the direction measurement is within 90°. Therefore, Euler angles are used to determine the direction. The transformation label is decomposed into translation and rotation according to the form of Euler angles. The final loss is calculated as: Among them, _ti , are the true and estimated relative translation of the image pair, _ri , They are the true value and estimated value of the relative rotation of the image pair, respectively, where w is the weight. In order to balance the metric size between translation and rotation, rotation has a greater impact on the static landmark image pose than translation, so the w value should be given a larger value to allow the network to better learn the translation variable; the RMS root mean square error is defined as: Where m represents the number of parameters, for translation m = 3, for rotation, expressed in Euler angles, m = 3; Step 2.4: training and deployment of the model. The image preprocessed in step 2.2 is input into the two-channel network for training through the deep learning server (4), and the two-channel network model is saved in static mode. After the training is completed, an accuracy test is performed. The images in the test set are preprocessed according to the preprocessing steps in step 2.2 and input into the saved static mode model. When the accuracy requirements are met, the trained static model is deployed on the edge processor (6); Step 3: Install the monocular camera (2) on the edge processor (6), integrate the monocular camera (2) and the edge processor (6) into a package, and then fix them at the end of the low-precision robotic arm (5); Step 4: posture measurement: Based on the trained two-channel network model, online posture measurement of the low-precision robotic arm (5) is performed. 2. According to claim 1, a high-precision robot posture measurement system based on a two-channel network is characterized in that: the specific steps of step 4 are: when the low-precision robotic arm (5) reaches the specified position from any direction, the monocular camera (2) fixed on the manipulator of the low-precision robotic arm (5) collects a static marker (3) near the target position and its surrounding scene image, and forms an input image pair with a pre-shot target position reference posture image, and the image pair is input into a trained two-channel network model, and the two-channel network model outputs the relative posture of the two images and controls the low-precision robotic arm (5) to adjust the posture, thereby realizing high-precision robot six-degree-of-freedom posture measurement.