CN113989428B - A global three-dimensional reconstruction method and device for metallurgical storage area based on depth vision - Google Patents
A global three-dimensional reconstruction method and device for metallurgical storage area based on depth vision Download PDFInfo
- Publication number
- CN113989428B CN113989428B CN202111034419.5A CN202111034419A CN113989428B CN 113989428 B CN113989428 B CN 113989428B CN 202111034419 A CN202111034419 A CN 202111034419A CN 113989428 B CN113989428 B CN 113989428B
- Authority
- CN
- China
- Prior art keywords
- depth
- original image
- information
- global
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000006243 chemical reaction Methods 0.000 claims abstract description 6
- 230000009466 transformation Effects 0.000 claims description 56
- 239000011159 matrix material Substances 0.000 claims description 41
- 238000001514 detection method Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000000354 decomposition reaction Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 3
- 238000005259 measurement Methods 0.000 abstract 1
- 230000000694 effects Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/35—Determination of transform parameters for the alignment of images, i.e. image registration using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Probability & Statistics with Applications (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
本发明提供了一种基于深度视觉的冶金库区全局三维重建方法及装置，涉及智能仓储物流技术领域。包括：以相机采集的原始图像及深度图像作为输入，通过匹配相邻图像间的特征点结合深度信息计算出帧间位姿转换关系，从而将不同位置采集的局部空间三维信息进行拼接的全局三维重建系统。本发明通过使用融合深度视觉的冶金库区全局三维重建系统及方法，可以使用空间三维信息拼接的方式获取冶金库区全局的空间三维信息，从而对垛堆进行更完全的测量。
The present invention provides a method and device for global three-dimensional reconstruction of a metallurgical storage area based on depth vision, and relates to the field of intelligent warehousing and logistics technology. It includes: taking the original image and depth image captured by the camera as input, and calculating the inter-frame pose conversion relationship by matching the feature points between adjacent images and combining the depth information, thereby splicing the local spatial three-dimensional information collected at different positions. By using the global three-dimensional reconstruction system and method of the metallurgical storage area that integrates deep vision, the present invention can obtain the global spatial three-dimensional information of the metallurgical storage area by splicing the spatial three-dimensional information, so as to perform a more complete measurement of the stack.
Description
Technical Field
The invention relates to the technical field of intelligent warehouse logistics, in particular to a metallurgical warehouse area global three-dimensional reconstruction method and device based on depth vision.
Background
The positioning of the crown block in the unmanned crown block reservoir area to the crown block is mainly based on a Gray bus measuring device, and the current crown block position is obtained by a feedback callback system realized through a PLC in the crown block movement process. In the prior art, for three-dimensional reconstruction of piles and vehicles in large field of view metallurgical storage areas, the detection information obtained using fixed position detection is extremely limited.
Disclosure of Invention
Aiming at the problem that in the prior art, the pile and the vehicle in the large-view-field metallurgical storage area are three-dimensionally rebuilt, and the acquired detection information is extremely limited by using fixed position detection, the invention provides a global three-dimensional rebuilding method and device for the metallurgical storage area based on depth vision.
In order to solve the technical problems, the invention provides the following technical scheme:
In one aspect, a depth vision-based metallurgical reservoir global three-dimensional reconstruction method is provided, comprising:
s1, acquiring original images of continuous frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence;
s2, matching ORB characteristic points between two adjacent frames of original images to obtain initial matching point pairs, screening the initial matching point pairs to obtain characteristic point pair matching information and characteristic point pair pixel coordinates;
S3, extracting depth information of a depth image corresponding to the original image through a depth image sequence, and combining the matched characteristic point pair pixel coordinates and the corresponding depth information to obtain local space three-dimensional information;
s4, calculating a frame pose transformation matrix of the original image;
S5, repeatedly executing S1-S4 until all frame pose transformation matrixes in the original image sequence are calculated, and splicing the local space three-dimensional information acquired at different positions by combining the pose transformation matrixes to form global space three-dimensional information;
And S6, converting the coordinates of the global space three-dimensional information into a coordinate system of a crown block in a reservoir area through the coordinate system, and providing the coordinates for the crown block.
Optionally, in step S1, original images of continuous frames and depth images corresponding to the original images are acquired by a depth camera to form an original image sequence and a depth image sequence, and ORB characteristic points in the original images of two adjacent frames are detected respectively, wherein the ORB characteristic points comprise:
s11, moving along with the crown block through a depth camera arranged on the crown block, and collecting original images and depth images of continuous frames;
s12, selecting a current original image and a previous frame original image to extract ORB feature points, and obtaining pixel coordinates of the ORB feature points;
S13, taking ORB characteristic points as the center, selecting the size of a window as 13 pixel points, selecting 128 groups of comparison point pairs in the window, comparing the gray value of the center pixel with the gray value of the pixel points in the window, and calculating a binary code string of an ORB characteristic point descriptor through a formula (1):
wherein, I (p), I (q) represents the gray value of the central pixel and the pixels in the window, N p represents the pixel point contained in the neighborhood of the central pixel in the size range of the set window;
The binary string is marked as 0 if I (q) > I (p) and 1 if I (q) < I (p).
Optionally, in step S2, ORB feature points between two adjacent frames of original images are matched to obtain an initial matching point pair, the initial matching point pair is screened to obtain feature point pair matching information and feature point pair pixel coordinates, and the method comprises the following steps:
s21, matching ORB characteristic points detected in the current original image and the original image of the previous frame to form a matching point pair;
S22, substituting the initial matching point pair obtained in the S21 into a formula (2) through a random consistency algorithm RANSAC:
wherein, (x ', y') represents the initial matching point position in the current original image; (x, y) represents the initial matching point position of the original image of the previous frame;
determining an optimal transformation matrix meeting the most matching point pairs, and eliminating outliers;
S23, collecting position coordinate information of the crown block at different times and different images, calculating the movement direction of the crown block, eliminating characteristic point pairs, the relative positions of the current original image and the previous frame of original image of which do not accord with the movement trend of the crown block, and obtaining final characteristic point pair matching information and characteristic point pair pixel coordinates.
Optionally, in step S3, depth information of a depth image corresponding to the original image is extracted through a depth image sequence, and the matching feature point pair pixel coordinates and the corresponding depth information are combined to obtain local space three-dimensional information, which comprises the following steps:
s31, extracting a depth value D of a depth image corresponding to the original image through a depth image sequence;
S32, substituting the characteristic point pair pixel coordinates and the depth value D into a formula (3) according to the characteristic point pair pixel coordinates obtained in the step S2 and the depth value D, and calculating local space three-dimensional coordinates of the characteristic points to obtain local space three-dimensional information:
Wherein (u, v) represents the pixel coordinates of the feature points, C x,Cy represents the camera calibration parameters, and the position coordinates of the principal point of the image are respectively represented, f x,fy represents the camera calibration parameters, f x represents the component of the focal length along the X-axis direction under the image pixel coordinate system, f y represents the component of the focal length along the Y-axis direction under the image pixel coordinate system, and X, Y, D represent the calculated three-dimensional coordinates.
Optionally, in step S4, calculating a frame pose transformation matrix of the original image includes:
s41, constructing a space point set error term between the current original image and the previous frame original image, calculating a group of space transformation matrixes with minimum space position error through singular value decomposition SVD of a formula (4),
The method comprises the steps of p i, q i, E i, R, t and t, wherein p i represents a space three-dimensional point set obtained after characteristic point matching of a current original image, q i represents a space three-dimensional point set obtained after characteristic point matching of a previous original image, E i represents a space position error set of a characteristic point pair corresponding to a space transformation matrix [ R|t ];
S42, converting E i into a quaternion format for storage to obtain a frame pose transformation matrix.
Optionally, in step S5, the stitching is performed on the local spatial three-dimensional information acquired at different positions in combination with the pose transformation matrix to form global spatial three-dimensional information, including:
Combining the local space three-dimensional information under the images at different moments with a frame pose transformation matrix, and splicing the space three-dimensional information by using the space transformation matrix through a formula (5) to form global space three-dimensional information:
Pi=T*P' (5)
Wherein P i represents the set of spatial three-dimensional points in the current original image, P' i represents the set of spatial three-dimensional points in the previous frame image, and T represents the calculated frame pose transformation matrix.
Optionally, in step S6, the coordinates in the global space three-dimensional information are converted into the coordinates of the crown block in the reservoir area through the coordinate system, and are provided for the crown block, including:
And converting coordinates in the global space three-dimensional information into a reservoir area coordinate system and providing the reservoir area coordinate system for the crown block to use.
In one aspect, a metallurgical reservoir global three-dimensional reconstruction device based on depth vision is provided, the device is applied to the method of any one of the above items, and the device comprises:
the ORB detection module is used for acquiring original images of continuous frames through the depth camera and depth images corresponding to the original images to form an original image sequence and a depth image sequence;
The similarity matching module is used for matching ORB characteristic points between two adjacent frames of original images to obtain an initial matching point pair;
The local information calculation module is used for extracting depth information of a depth image corresponding to the original image through the depth image sequence, combining the matched characteristic point pair pixel coordinates and the corresponding depth information to obtain local space three-dimensional information;
the pose transformation calculation module is used for calculating a frame pose transformation matrix;
The global information splicing module is used for calculating all frame pose transformation matrixes in the original image sequence, splicing the local space three-dimensional information acquired at different positions by combining the pose transformation matrixes to form global space three-dimensional information;
The coordinate system conversion module is used for converting coordinates in the global space three-dimensional information into a coordinate system of the crown block in the reservoir area through the coordinate system and providing the coordinates for the crown block.
Optionally, the ORB detection module is used for acquiring original images and depth images of continuous frames by moving along with the crown block through a depth camera arranged on the crown block;
Selecting a current original image and a previous frame of original image to extract ORB feature points to obtain pixel coordinates of the ORB feature points;
After determining the pixel coordinate position of the ORB feature point, selecting the size of a window as 13 pixel points by taking the ORB feature point as the center, selecting 128 groups of comparison point pairs in the window, and calculating the binary code string of the ORB feature point descriptor by comparing the gray value of the center pixel with the gray value of the pixel point in the window.
Optionally, a similarity matching module is used for matching ORB characteristic points detected in the current original image and the previous frame original image to form a matching point pair;
selecting a matching point pair with a code exclusive OR value smaller than Yu Hanming distance percentage threshold in the binary code string, and setting the group of characteristic points as an initial matching point pair:
Determining an optimal transformation matrix meeting the most matching point pairs through a random consistency algorithm RANSAC, and eliminating outliers;
And acquiring position coordinate information of the crown block at different images and different moments, calculating the movement direction of the crown block, removing characteristic point pairs, the relative positions of the current original image and the previous frame of original image of which do not accord with the movement trend of the crown block, and obtaining final characteristic point pair matching information and characteristic point pair pixel coordinates.
The technical scheme provided by the embodiment of the invention has at least the following beneficial effects:
In the scheme, the global three-dimensional reconstruction method and the global three-dimensional reconstruction device for the metallurgical storage area, which are used for fusing the depth vision, acquire the original image and the depth image which cover the whole metallurgical storage area in real time in a surface scanning mode, acquire the global spatial three-dimensional information of the metallurgical storage area in a spatial three-dimensional information splicing mode, and therefore the effect of more comprehensively measuring the pile is achieved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a depth vision-based metallurgical reservoir global three-dimensional reconstruction method provided by an embodiment of the invention;
FIG. 2 is a feature point detection flow chart of a metallurgical reservoir global three-dimensional reconstruction method based on depth vision provided by the embodiment of the invention;
FIG. 3 is a feature point matching flow chart of a metallurgical reservoir global three-dimensional reconstruction method based on depth vision provided by an embodiment of the invention;
FIG. 4 is a feature point matching algorithm effect of the metallurgical reservoir global three-dimensional reconstruction method based on depth vision provided by the embodiment of the invention;
FIG. 5 is a flow chart of calculating local spatial three-dimensional information of a metallurgical reservoir global three-dimensional reconstruction method based on depth vision provided by the embodiment of the invention;
FIG. 6 is a frame pose transformation matrix calculation flow chart of a metallurgical reservoir global three-dimensional reconstruction method based on depth vision provided by an embodiment of the invention;
FIG. 7 is a block diagram of a device for global three-dimensional reconstruction of metallurgical storage areas based on depth vision according to an embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantages to be solved more apparent, the following detailed description will be given with reference to the accompanying drawings and specific embodiments.
As shown in fig. 1, the embodiment of the invention provides a metallurgical reservoir global three-dimensional reconstruction method based on depth vision, which comprises the following steps:
S1, acquiring original images of continuous frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence, and respectively detecting ORB (Oriented FAST and Rotated BRIEF, rapid feature point extraction and description algorithm) feature points in the original images of two adjacent frames.
And S2, matching ORB characteristic points between two adjacent frames of original images to obtain initial matching point pairs, screening the initial matching point pairs to obtain characteristic point pair matching information and characteristic point pair pixel coordinates.
And S3, extracting depth information of a depth image corresponding to the original image through the depth image sequence, and combining the matched characteristic point pair pixel coordinates and the corresponding depth information to obtain the local space three-dimensional information.
S4, calculating a frame pose transformation matrix of the original image.
S5, repeatedly executing S1-S4 until all frame pose transformation matrixes in the original image sequence are calculated, and splicing the local space three-dimensional information acquired at different positions by combining the pose transformation matrixes to form global space three-dimensional information.
And S6, converting the coordinates of the global space three-dimensional information into a coordinate system of a crown block in a reservoir area through the coordinate system, and providing the coordinates for the crown block.
In the embodiment, by using the system and the method for reconstructing the global three-dimensional of the metallurgical storage area by fusion depth vision, the global spatial three-dimensional information of the metallurgical storage area can be acquired in a spatial three-dimensional information splicing mode, so that the pile is more completely measured.
As shown in fig. 2, in step S1, original images of consecutive frames and depth images corresponding to the original images are collected by a depth camera to form an original image sequence and a depth image sequence, and the detecting of ORB feature points in two adjacent original images includes:
S11, acquiring original images and depth images of continuous frames by moving along with the crown block through a depth camera arranged on the crown block, and forming an original image sequence and a depth image sequence.
And S12, selecting the current original image and the previous frame of original image to extract ORB feature points, and obtaining pixel coordinates of the ORB feature points.
S13, taking ORB characteristic points as the center, selecting the size of a window as 13 pixel points, selecting 128 groups of comparison point pairs in the window, comparing the gray value of the center pixel with the gray value of the pixel points in the window, and calculating a binary code string of an ORB characteristic point descriptor through a formula (1):
Wherein, I (p) represents the gray value of the central pixel, I (q) represents the gray value of the pixel in the window, N p represents the pixel point contained in the neighborhood of the central pixel in the size range of the set window;
The binary string is marked as 0 if I (q) > I (p) and 1 if I (q) < I (p).
In this embodiment, it is assumed that an area-array CCD camera that can acquire an original image and a depth image simultaneously is used as the camera.
The corresponding relation between the image pixel plane and the space three-dimensional information can be established by calibrating the camera parameters and integrating the depth image information, and in the embodiment, the internal reference matrix of the depth camera can be expressed as follows:
Wherein f x,fy is a camera calibration parameter, f x represents a component of the focal length along the X-axis direction in the image pixel coordinate system, f y represents a component of the focal length along the Y-axis direction in the image pixel coordinate system, and C x,Cy represents camera calibration parameters, respectively representing position coordinates of the principal point of the image.
In this embodiment, the internal reference calibration result of the depth camera is:
and for ORB feature point extraction, firstly selecting a current original image and a previous frame original image to extract ORB feature points, and obtaining feature point pixel coordinates (u, v).
In this embodiment, if 16 neighborhood pixel points q on a circle with radius of 3 are selected corresponding to the center pixel point p, if the neighborhood meets the condition, the neighborhood can be used as a feature point, and the judgment expression is as follows;
Where N represents the number of coincident pixels and lambda d represents the pixel difference threshold.
The number of coincident pixels in this embodiment is set to 12 and the pixel difference threshold is set to 15.
In this embodiment, by extracting the ORB feature points in each frame of image, the number of ORB feature points in each frame of image is shown in table 1:
TABLE 1
As shown in fig. 3, in step S2, the ORB feature points between two adjacent frames of original images are matched to obtain an initial matching point pair, and the filtering of the initial matching point pair to obtain feature point pair matching information and feature point pair pixel coordinates includes:
s21, matching ORB characteristic points detected in the current original image and the original image of the previous frame to form a matching point pair, and measuring the consistency of the matching point pair by using the Hamming distance percentage.
S22, substituting the initial matching point pair obtained in the S21 into a formula (2) through RANSAC (RANdom SAmple Consensus, random consistency algorithm):
Wherein (x ', y') represents the initial matching point position in the current original image, (x, y) represents the initial matching point position of the previous frame image, and s represents the scale parameter.
And determining an optimal transformation matrix meeting the most matching point pairs, and eliminating outliers.
S23, collecting position coordinate information of the crown block at different times and different images, calculating the movement direction of the crown block, eliminating characteristic point pairs, the relative positions of the current original image and the previous frame of original image of which do not accord with the movement trend of the crown block, and obtaining final characteristic point pair matching information and characteristic point pair pixel coordinates.
In this embodiment, the hamming distance percentage threshold is set to 0.6, and feature points in the current original image and the previous frame original image are matched, and the hamming distance percentage is used to measure the consistency of the feature points:
η=Hamming(Sl(x,y),Sr(x',y'))
if the descriptor coding exclusive OR value is smaller than a certain threshold value, judging the group of characteristic points as matching point pairs;
Wherein S l (x, y) represents the current original image ORB feature point descriptor, S r (x ', y') represents the previous frame image ORB feature point descriptor, and η represents the Hamming distance percentage threshold.
The number of pairs of feature points of the ORB initially matched by hamming distance in this embodiment is shown in table 2:
TABLE 2
In the embodiment, the initial matching point pairs are brought into a matrix in an iterative mode through RANSAC, an optimal transformation matrix meeting the maximum matching point pairs is determined, outliers are removed, and final characteristic point pair matching information and characteristic point pair pixel coordinates are obtained through motion trend constraint. In this embodiment, the result of the ORB feature point matching algorithm based on the motion trend constraint is shown in fig. 4.
The number of pairs of completely consistent ORB feature point matching points obtained after RANSAC random consistency constraint and motion trend constraint in this embodiment is shown in table 3:
TABLE 3 Table 3
| Matching images | Matching feature points (pair) | Matching images | Matching feature points (pair) |
| 1-2 | 22 | 11-12 | 10 |
| 2-3 | 20 | 12-13 | 18 |
| 3-4 | 22 | 13-14 | 20 |
| 4-5 | 30 | 14-15 | 18 |
| 5-6 | 32 | 15-16 | 41 |
| 6-7 | 11 | 16-17 | 54 |
| 7-8 | 9 | 17-18 | 85 |
| 8-9 | 6 | 18-19 | 99 |
| 9-10 | 10 | 19-20 | 180 |
| 10-11 | 8 |
As shown in fig. 5, in step S3, depth information of a depth image corresponding to an original image is extracted through a sequence of depth images, and local spatial three-dimensional information is obtained by combining the matched feature point pair pixel coordinates and the corresponding depth information, including:
and S31, extracting a depth value D of a depth image corresponding to the original image through the depth image sequence.
S32, substituting the characteristic point pair pixel coordinates and the depth value D into a formula (3) according to the characteristic point pair pixel coordinates obtained in the step S2 and the depth value D, and calculating local space three-dimensional coordinates of the characteristic points to obtain local space three-dimensional information:
Wherein (u, v) represents the pixel coordinates of the feature points, C x,Cy represents the camera calibration parameters, and the position coordinates of the principal point of the image are respectively represented, f x,fy represents the camera calibration parameters, f x represents the component of the focal length along the X-axis direction under the image pixel coordinate system, f y represents the component of the focal length along the Y-axis direction under the image pixel coordinate system, and X, Y, D represent the calculated three-dimensional coordinates.
In this embodiment, the spatial transformation matrix is converted into a quaternion form using the rodgers formula and stored.
The frame pose conversion values between the respective groups of images can be expressed as shown in table 4 using quaternions:
TABLE 4 Table 4
As shown in fig. 6, in step S4, a frame pose transformation matrix of the original image is calculated, including:
S41, constructing a space point set error term between the current original image and the previous frame original image, calculating a group of space transformation matrixes with minimum space position errors through SVD (Singular Value Decomposition ) of a formula (4),
The method comprises the steps of p i, q i, E i, R and t, wherein p i represents a space three-dimensional point set obtained after characteristic point matching of a current original image, q i represents a space three-dimensional point set obtained after characteristic point matching of a previous frame of original image, E i represents a space position error set of a characteristic point pair corresponding to a space transformation matrix [ R|t ], R represents a space rotation matrix for minimizing space position errors, and t represents the space transformation matrix.
S42, converting E i into a quaternion format for storage to obtain a frame pose transformation matrix.
In step S5, the stitching is performed on the local spatial three-dimensional information acquired at different positions in combination with the pose transformation matrix to form global spatial three-dimensional information, which includes:
Combining the local space three-dimensional information under the images at different moments with a frame pose transformation matrix, and splicing the space three-dimensional information by using the space transformation matrix through a formula (5) to form global space three-dimensional information:
Pi=T*P' (5)
Wherein P i represents the set of spatial three-dimensional points in the current original image, P i' represents the set of spatial three-dimensional points in the previous frame image, and T represents the calculated frame pose transformation matrix.
In this embodiment, the image acquisition resolution is 1280×720, the number of the maximum spatial three-dimensional points that can be acquired by a single image is 921600, and the corresponding actual coverage area is 10m×7m according to the height calculation of the installation position.
The control three-dimensional points in each group of images are spliced, and the space three-dimensional point information under each group of images is counted as shown in table 5:
TABLE 5
In this embodiment, the global point cloud image obtained by using the spatial three-dimensional information stitching is used to raise the detection coverage to 20m×7m.
In step S6, the global space three-dimensional information is converted into a coordinate system of a crown block in a reservoir area through the coordinate system and provided for the crown block, and the method includes:
the three-dimensional information of the global space is converted into the coordinate system of the reservoir area and is provided for the crown block to use.
In this embodiment, coordinate information of the same detection point of the visual detection system and the space detection system in the same scene is obtained, coordinate system fitting is performed, a coordinate conversion relation is calculated, and finally, a detection result based on depth vision is converted into a reservoir coordinate system and provided for an overhead travelling crane to perform operation.
In this embodiment, coordinate values of the same point in the respective coordinate systems of the first image and the base region coordinate system in the continuous image sequence acquired by the depth camera are obtained, as shown in table 6:
TABLE 6
In this embodiment, the spatial transformation matrix fitting result of the visual coordinate system and the reservoir coordinate system is:
The embodiment of the invention provides a metallurgical reservoir global three-dimensional reconstruction device 700 based on depth vision, wherein the device 700 is used for realizing the metallurgical reservoir global three-dimensional reconstruction method based on depth vision, and as shown in fig. 7, the device comprises:
The ORB detection module 701 is configured to collect original images of consecutive frames and depth images corresponding to the original images by using a depth camera, and form an original image sequence and a depth image sequence;
The similarity matching module 702 is configured to match ORB feature points between two adjacent frames of original images to obtain an initial matching point pair, and screen the initial matching point pair to obtain feature point pair matching information and feature point pair pixel coordinates.
The local information calculation module 703 is configured to extract depth information of a depth image corresponding to the original image through the depth image sequence, and combine the matched feature point pair pixel coordinates and the corresponding depth information to obtain local spatial three-dimensional information.
The pose transformation calculation module 704 is configured to calculate a frame pose transformation matrix.
The global information stitching module 705 is configured to calculate all frame pose transformation matrices in the original image sequence, and stitch the local spatial three-dimensional information acquired at different positions together with the pose transformation matrices to form global spatial three-dimensional information.
The coordinate system conversion module 706 is configured to convert the global spatial three-dimensional information into a coordinate system of the crown block in the reservoir area through the coordinate system, and provide the three-dimensional information to the crown block.
In this embodiment, the depth camera is mounted on the crown block and connected to the ground server through the ethernet, and the original image and the depth image covering the whole metallurgical storage area are collected in real time in a surface scanning manner along with the movement of the crown block.
The ORB detection module 701 is used for acquiring original images and depth images of continuous frames by moving along with the crown block through a depth camera arranged on the crown block;
Selecting a current original image and a previous frame of original image to extract ORB feature points to obtain pixel coordinates of the ORB feature points;
After determining the pixel coordinate position of the ORB feature point, selecting the size of a window as 13 pixel points by taking the ORB feature point as the center, selecting 128 groups of comparison point pairs in the window, and calculating the binary code string of the ORB feature point descriptor by comparing the gray value of the center pixel with the gray value of the pixel point in the window.
The similarity matching module 702 is configured to match the ORB feature points detected in the current original image and the previous original image to form a matching point pair;
selecting a matching point pair with a code exclusive OR value smaller than Yu Hanming distance percentage threshold in the binary code string, and setting the group of characteristic points as an initial matching point pair:
Determining an optimal transformation matrix meeting the most matching point pairs through a random consistency algorithm RANSAC, and eliminating outliers;
And acquiring position coordinate information of the crown block at different images and different moments, calculating the movement direction of the crown block, removing characteristic point pairs, the relative positions of the current original image and the previous frame of original image of which do not accord with the movement trend of the crown block, and obtaining final characteristic point pair matching information and characteristic point pair pixel coordinates.
According to the system and the method for reconstructing the global three-dimensional of the metallurgical storage area by using the fusion depth vision, the global spatial three-dimensional information of the metallurgical storage area can be acquired by using a spatial three-dimensional information splicing mode, so that the pile is more completely measured.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.
Claims (8)
1. The global three-dimensional reconstruction method for the metallurgical reservoir area based on depth vision is characterized by comprising the following steps of:
s1, acquiring original images of continuous frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence;
In the step S1, original images of continuous frames and depth images corresponding to the original images are collected through a depth camera to form an original image sequence and a depth image sequence, ORB characteristic points in the original images of two adjacent frames are detected respectively, and the method comprises the following steps:
s11, moving along with the crown block through a depth camera arranged on the crown block, and collecting original images and depth images of continuous frames;
s12, selecting a current original image and a previous frame original image to extract ORB feature points, and obtaining pixel coordinates of the ORB feature points;
s13, selecting the size of a window as 13 pixel points by taking the ORB characteristic points as the center, selecting 128 groups of comparison point pairs in the window, and calculating binary code strings of ORB characteristic point descriptors according to a formula (1) by comparing the gray values of the center pixel and the gray values of the pixel points in the window:
Wherein, I (p) represents the gray value of the central pixel, I (q) represents the gray value of the pixel in the window, N p represents the pixel point contained in the neighborhood of the central pixel in the size range of the set window;
if I (q) is not less than I (p), the binary code string is marked as 0, and if I (q) is not less than I (p), the binary code string is marked as 1;
s2, matching ORB characteristic points between two adjacent frames of original images to obtain initial matching point pairs, screening the initial matching point pairs to obtain characteristic point pair matching information and characteristic point pair pixel coordinates;
S3, extracting depth information of a depth image corresponding to the original image through a depth image sequence, and combining the matched characteristic point pair pixel coordinates and the corresponding depth information to obtain local space three-dimensional information;
in the step S3, depth information of a depth image corresponding to the original image is extracted through a depth image sequence, and the matched characteristic point pair pixel coordinates and the corresponding depth information are combined to obtain local space three-dimensional information, wherein the method comprises the following steps:
s31, extracting a depth value D of a depth image corresponding to the original image through a depth image sequence;
S32, substituting the characteristic point pair pixel coordinates and the depth value D into a formula (3) according to the characteristic point pair pixel coordinates obtained in the step S2 and the depth value D, and calculating local space three-dimensional coordinates of the characteristic points to obtain local space three-dimensional information:
Wherein (u, v) represents the pixel coordinates of the feature points, C x,Cy represents the camera calibration parameters, and the position coordinates of the main points of the image are respectively represented, f x,fy represents the camera calibration parameters, f x represents the component of the focal length along the X-axis direction under the image pixel coordinate system, f y represents the component of the focal length along the Y-axis direction under the image pixel coordinate system, and X, Y, D represent the calculated three-dimensional coordinates of the space;
s4, calculating a frame pose transformation matrix of the original image;
S5, repeatedly executing S1-S4 until all frame pose transformation matrixes in the original image sequence are calculated, and splicing the local space three-dimensional information acquired at different positions by combining the pose transformation matrixes to form global space three-dimensional information;
And S6, converting the coordinates of the global space three-dimensional information into a coordinate system of a crown block in a reservoir area through the coordinate system, and providing the coordinates for the crown block.
2. The depth vision-based metallurgical reservoir global three-dimensional reconstruction method according to claim 1, wherein in the step S2, the ORB feature points between two adjacent frames of original images are matched to obtain an initial matching point pair, the initial matching point pair is screened to obtain feature point pair matching information and feature point pair pixel coordinates, and the method comprises the following steps:
s21, matching ORB characteristic points detected in the current original image and the original image of the previous frame to form a matching point pair;
S22, substituting the initial matching point pair obtained in the S21 into a formula (2) through a random consistency algorithm RANSAC:
Wherein, (x ', y') represents the initial matching point position in the current original image; (x, y) represents the initial matching point position of the previous frame of image;
determining an optimal transformation matrix meeting the most matching point pairs, and eliminating outliers;
S23, collecting position coordinate information of the crown block at different times and different images, calculating the movement direction of the crown block, eliminating characteristic point pairs, the relative positions of the current original image and the previous frame of original image of which do not accord with the movement trend of the crown block, and obtaining final characteristic point pair matching information and characteristic point pair pixel coordinates.
3. The depth vision-based metallurgical pool global three-dimensional reconstruction method according to claim 1, wherein in the step S4, a frame pose transformation matrix of the original image is calculated, and the method comprises:
S41, constructing a space point set error term between the current original image and the previous frame original image, and calculating a group of space transformation matrixes with minimum space position errors through Singular Value Decomposition (SVD) of a formula (4):
Wherein p i represents a space three-dimensional point set obtained after characteristic point matching of a current original image, qi represents a space three-dimensional point set obtained after characteristic point matching of a previous frame of original image, E i represents a space position error set of a characteristic point pair corresponding to a space transformation matrix [ R|t ];
S42, converting E i into a quaternion format for storage to obtain a frame pose transformation matrix.
4. The depth vision-based metallurgical reservoir global three-dimensional reconstruction method according to claim 3, wherein in the step S5, the local spatial three-dimensional information collected at different positions is spliced by combining with a pose transformation matrix to form global spatial three-dimensional information, and the method comprises the following steps:
Combining the local space three-dimensional information under the images at different moments with a frame pose transformation matrix, and splicing the space three-dimensional information by using the space transformation matrix through a formula (5) to form global space three-dimensional information:
Pi=T*P' (5)
Wherein P i represents the set of spatial three-dimensional points in the current original image, P' i represents the set of spatial three-dimensional points in the previous frame image, and T represents the calculated frame pose transformation matrix.
5. The depth vision-based metallurgical reservoir global three-dimensional reconstruction method according to claim 1, wherein in the step S6, coordinates in the global spatial three-dimensional information are converted into a reservoir crown block coordinate system by a coordinate system and provided for crown blocks, and the method comprises the following steps:
And converting coordinates in the global space three-dimensional information into a reservoir area coordinate system and providing the reservoir area coordinate system for the crown block to use.
6. A depth vision based metallurgical storage region global three-dimensional reconstruction device, characterized in that the device is applied to the method according to any one of claims 1-5, the device comprising:
the ORB detection module is used for acquiring original images of continuous frames through the depth camera and depth images corresponding to the original images to form an original image sequence and a depth image sequence;
The similarity matching module is used for matching ORB characteristic points between two adjacent frames of original images to obtain an initial matching point pair;
The local information calculation module is used for extracting depth information of a depth image corresponding to the original image through the depth image sequence, combining the matched characteristic point pair pixel coordinates and the corresponding depth information to obtain local space three-dimensional information;
the pose transformation calculation module is used for calculating a frame pose transformation matrix;
the global information splicing module is used for calculating all frame pose transformation matrixes in the original image sequence, splicing the local space three-dimensional information acquired at different positions by combining the pose transformation matrixes to form global space three-dimensional information;
The coordinate system conversion module is used for converting coordinates in the global space three-dimensional information into a coordinate system of the crown block in the reservoir area through the coordinate system and providing the coordinates for the crown block.
7. The depth vision-based metallurgical reservoir global three-dimensional reconstruction device according to claim 6, wherein the ORB detection module is configured to acquire an original image and a depth image of a continuous frame by moving with an overhead crane with a depth camera disposed on the overhead crane;
Selecting a current original image and a previous frame of original image to extract ORB feature points to obtain pixel coordinates of the ORB feature points;
After determining the pixel coordinate position of the ORB feature point, selecting 13 pixel points by taking the ORB feature point as the center, selecting 128 groups of comparison point pairs in the window, and calculating the binary code string of the ORB feature point descriptor by comparing the gray value of the center pixel with the gray value of the pixel point in the window.
8. The depth vision-based metallurgical reservoir global three-dimensional reconstruction device according to claim 7, wherein the similarity matching module is used for matching ORB characteristic points detected in a current original image and a previous original image to form matching point pairs;
Determining an optimal transformation matrix meeting the most matching point pairs through a random consistency algorithm RANSAC, and eliminating outliers;
And acquiring position coordinate information of the crown block at different images and different moments, calculating the movement direction of the crown block, removing characteristic point pairs, the relative positions of the current original image and the previous frame of original image of which do not accord with the movement trend of the crown block, and obtaining final characteristic point pair matching information and characteristic point pair pixel coordinates.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111034419.5A CN113989428B (en) | 2021-09-03 | 2021-09-03 | A global three-dimensional reconstruction method and device for metallurgical storage area based on depth vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111034419.5A CN113989428B (en) | 2021-09-03 | 2021-09-03 | A global three-dimensional reconstruction method and device for metallurgical storage area based on depth vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113989428A CN113989428A (en) | 2022-01-28 |
| CN113989428B true CN113989428B (en) | 2025-03-04 |
Family
ID=79735351
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111034419.5A Active CN113989428B (en) | 2021-09-03 | 2021-09-03 | A global three-dimensional reconstruction method and device for metallurgical storage area based on depth vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113989428B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114723978A (en) * | 2022-04-11 | 2022-07-08 | 南京拓控信息科技股份有限公司 | Train component positioning method based on three-dimensional axis information |
| CN116952413B (en) * | 2023-09-19 | 2023-11-24 | 徐州凌南生物科技有限公司 | Temperature measurement method and system for biomass fuel stack |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105303518A (en) * | 2014-06-12 | 2016-02-03 | 南京理工大学 | Region feature based video inter-frame splicing method |
| CN110021041A (en) * | 2019-03-01 | 2019-07-16 | 浙江大学 | Unmanned scene progressive mesh structural remodeling method based on binocular camera |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107481315A (en) * | 2017-06-29 | 2017-12-15 | 重庆邮电大学 | A kind of monocular vision three-dimensional environment method for reconstructing based on Harris SIFT BRIEF algorithms |
| CN111047620A (en) * | 2019-11-15 | 2020-04-21 | 广东工业大学 | A UAV Visual Odometry Method Based on Depth Point-Line Features |
| CN111223180A (en) * | 2020-01-08 | 2020-06-02 | 中冶赛迪重庆信息技术有限公司 | Three-dimensional modeling method and device for stock ground, storage medium and electronic terminal |
| CN112884848B (en) * | 2021-03-05 | 2022-06-28 | 河北工业大学 | Intelligent crown block control system, method and terminal based on binocular vision |
| CN113012212B (en) * | 2021-04-02 | 2024-04-16 | 西北农林科技大学 | Depth information fusion-based indoor scene three-dimensional point cloud reconstruction method and system |
-
2021
- 2021-09-03 CN CN202111034419.5A patent/CN113989428B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105303518A (en) * | 2014-06-12 | 2016-02-03 | 南京理工大学 | Region feature based video inter-frame splicing method |
| CN110021041A (en) * | 2019-03-01 | 2019-07-16 | 浙江大学 | Unmanned scene progressive mesh structural remodeling method based on binocular camera |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113989428A (en) | 2022-01-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110363858B (en) | Three-dimensional face reconstruction method and system | |
| CN112949478B (en) | Target detection method based on tripod head camera | |
| US9965870B2 (en) | Camera calibration method using a calibration target | |
| CN101794448B (en) | Full automatic calibration method of master-slave camera chain | |
| CN107481315A (en) | A kind of monocular vision three-dimensional environment method for reconstructing based on Harris SIFT BRIEF algorithms | |
| CN111996883B (en) | Method for detecting width of road surface | |
| CN112785654B (en) | Track geometry detection system calibration method and device | |
| CN113989428B (en) | A global three-dimensional reconstruction method and device for metallurgical storage area based on depth vision | |
| CN109671109B (en) | Dense point cloud generation method and system | |
| CN113313659A (en) | High-precision image splicing method under multi-machine cooperative constraint | |
| CN110660089A (en) | A satellite image registration method and device | |
| CN105139401A (en) | Depth credibility assessment method for depth map | |
| CN116228539A (en) | A method for stitching remote sensing images of drones | |
| CN114494039A (en) | A method for geometric correction of underwater hyperspectral push-broom images | |
| CN113808103A (en) | Method, device and storage medium for automatic detection of road potholes based on image processing | |
| CN113643363A (en) | Pedestrian positioning and trajectory tracking method based on video image | |
| CN112613107B (en) | Method, device, storage medium and equipment for determining construction progress of pole and tower engineering | |
| US20240242318A1 (en) | Face deformation compensating method for face depth image, imaging device, and storage medium | |
| CN114241276B (en) | A method for fusion of infrared and visible light under weak registration and binocular imaging device | |
| CN115588144A (en) | Real-time attitude capturing method, device and equipment based on Gaussian dynamic threshold screening | |
| CN115456870A (en) | Multi-image splicing method based on external parameter estimation | |
| CN119205875A (en) | A parameter measurement method for large-size lightning arresters based on robotic arm collaborative 3D reconstruction | |
| CN112686962A (en) | Indoor visual positioning method and device and electronic equipment | |
| CN118298120A (en) | Novel view angle synthesizing method of remote sensing image based on data driving | |
| CN117689813A (en) | A method and system for infrared three-dimensional modeling of high-precision power transformers in substations |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |