CN115018709A - Image splicing rock wall reconstruction method - Google Patents

Abstract

The invention discloses a method for reconstructing an image splicing rock wall, which comprises a video frame obtaining step, a feature point matching step and an image splicing step; a step of acquiring video frames, which is to intercept a plurality of video frames from a video and acquire images corresponding to the video frames; the method comprises the steps of obtaining characteristic points, wherein the step comprises an image preprocessing step, a rock point calibration step and a characteristic point obtaining step; matching the characteristic points, namely performing characteristic matching on the characteristic points extracted from the two images through a matching algorithm until the characteristic points of each image are matched with the characteristic points of the surrounding images; and image splicing, namely determining an overlapping area through perspective transformation, and transforming the original image to a target plane through a splicing method to perform splicing operation until a complete rock wall reconstruction image is obtained. The invention improves the detection speed, has less calculation amount and high accuracy.

Description

A method of image mosaic rock wall reconstruction

technical field

The invention relates to the technical field of image stitching, and more particularly, to a method for reconstructing rock wall by image stitching.

Background technique

Image stitching is a technique of stitching multiple overlapping images into a smooth image with wide viewing angle and high resolution. In the field of aviation, it is necessary to combine multiple photos of different angles of the ground to form a clear overall image; in the field of navigation, it is necessary to realize the full-scale rendering and three-dimensional reconstruction of the survey scene; in the field of medicine, it is necessary to combine the computer X-ray tomography technology The image data obtained by MRI, MRI and digital silhouette angiography can be combined to obtain complete information about the patient, which is convenient for doctors to diagnose, thereby improving the accuracy of pathological examination. Therefore, image stitching technology plays an important role in various modern scientific fields.

In the traditional image stitching process, firstly, SIFT features are extracted and matched in the input image, and a large number of feature points are obtained. The SIFT algorithm is used to detect and describe local features in images. It finds extreme points in the spatial scale and extracts its position, scale, and rotation invariants. Feature matching is then performed using the KNN algorithm, for each pair returned by iterative KNN and a distance test is performed. For each pair of features, we set a ratio value, if the distance between features is within a certain ratio, we keep it, otherwise, we discard it. Next, a single adaptation matrix is found to implement image stitching based on matching points, and outliers are filtered out in the process. Finally, a perspective transformation is performed, one or more of which can be combined, taking the image and the single adaptation matrix as input. Then, it transforms the source image onto the destination plane according to the single adaptation matrix to get a panorama. In feature extraction, since the SIFT method constructs a 128-dimensional vector from the feature points, and then matches the vectors, the image has to satisfy enough textures, so that the difference is not too large, and it is easy to cause mismatches. In rock wall reconstruction, due to the existence of many rock points, people and other irrelevant things, if all feature extraction is performed, it will lead to a large workload, and useful information will be affected by a huge number of key points. , These key points have many outliers. The RANSAC algorithm is used to remove irrelevant data, but the efficiency is low due to the large sample size and the long iteration time.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for reconstructing a rock wall by image splicing, so as to improve the detection speed, and the obtained feature points have stronger characteristics. In addition, due to the reduction of feature points, the number of iterations in the process of calculating a single adaptive matrix is reduced, and the characteristics of less calculation and high accuracy are realized.

In order to achieve the above purpose, a method for image mosaic rock wall reconstruction is provided, which includes the steps of acquiring video frames, acquiring feature points, matching feature points, and stitching images;

The step of acquiring video frames is to intercept several video frames from the video to obtain images corresponding to the video frames;

The step of acquiring feature points includes an image preprocessing step, a rock point calibration step and a feature point acquiring step; the image preprocessing step is to perform a preprocessing operation on the image; the rock point calibration step is to calibrate the rock points of the image; The feature point acquisition step is to extract feature points on the edge of rock points by an edge feature extraction method; the rock points are objects with irregular shapes and easy to be grasped in the image;

In the feature point matching step, the feature points extracted from the two images are matched by the matching algorithm until the feature points of each image are matched with the surrounding image feature points;

In the image splicing step, through perspective transformation, the overlapping area is determined, and the original image is transformed to the target plane through the splicing method for splicing operation until a complete reconstruction of the rock wall is obtained.

Particularly, the specific method for intercepting several video frames in the step of acquiring video frames is as follows: setting a value for intercepting video frames per unit, and intercepting several video frames per unit by using the above value.

Particularly, in the image preprocessing step, the specific method of performing the image preprocessing operation is to perform mean filtering on all video frame images to remove noise.

In particular, the specific method for calibrating the rock points of the image in the rock point calibration step is to calibrate each image one by one to obtain a label in each image; the label includes the coordinates of the rock point and the name of the rock point .

In particular, the edge feature extraction method includes the following steps:

Use the yolo network to train the calibration features, get the bounding box of each rock point and perform image grayscale for the rock points in the bounding box;

Use the Canny edge algorithm to detect the bounding box to get the image of the edge of each rock point;

Use the Shi-Tomasi corner detection method to obtain the corners on the edge of the rock point, and obtain the best corners through the adaptive matrix, which are the feature points;

Obtain the coordinates and scale of the feature points and the direction information, size and angle information of the feature points.

In particular, the specific method for obtaining the corner points on the edge of the rock point by using the Shi-Tomasi corner point detection method, and obtaining the best corner point through the adaptive matrix is:

A1. Observe the image containing the corners through the local window, move the local window in all directions (u,v) and calculate the sum of all grayscale differences:

E(u, v)=∑ _{x, y} w(x, y)[I(x+u, y+v)-I(x, y)] ² ; where x is and y is

where I(x, y) is the image grayscale of the local window, I(x+u, y+v) is the moved image grayscale, and w(x, y) is the window function;

A2. Perform corner detection to maximize the value of E(u, v), which is simplified to

设自适应矩阵

即

Let the adaptive matrix

which is

A3. Use the adaptive matrix M to find the corner points, set the eigenvalues of the adaptive matrix M as λ ₁ and λ ₂ , if the smaller one of the two eigenvalues of the matrix M is greater than the preset threshold, then the corner point R =min{λ ₁ ,λ ₂ };

A4. Calculate the quality of the corner points, set the maximum eigenvalue of the quality of the corner points as the threshold, and set the shortest Euclidean distance between the two corner points;

A5. Remove the corners below the threshold;

A6. Arrange the remaining corner points in descending order of quality;

A7. Select the first corner point in descending order as the best corner point and delete all the corner points whose Euclidean distance is less than the shortest Euclidean distance from the corner point;

A8. Repeat steps A6-A7 until the corner points cannot be deleted, and return to all the best corner points.

In particular, in the feature point matching step, the specific method for performing feature matching on the feature points extracted from the two images by the matching algorithm is: using the KNN matching algorithm to match the feature points between the two images, using the ratio test, That is, according to the method of comparing the nearest neighbor distance and the next nearest neighbor distance, when the distance ratio is less than a predetermined threshold, it can be considered that the two feature points are correctly matched.

In particular, before the image stitching step, it is also necessary to perform a step of removing false matching:

B1. Using the RANSAC algorithm, randomly select 4 pairs of feature points from the matched feature points as inner points, and other feature points as outer points;

B2. Obtain the single adaptation matrix of the interior point;

B3. Use the single adaptive matrix to test all the outer points, and set a threshold condition, and classify all the outer points that meet the threshold condition after the single adaptive matrix test as new inner points;

B4. Repeat steps B2-B3 until the final single adaptation matrix can be obtained when the number of interior points does not change.

Particularly, in the image splicing step, the splicing operation adopts a weighted fusion method for splicing; the weighted fusion method is to add and synthesize the pixel values of the overlapping areas of the images according to a certain weight.

Beneficial effects of the present invention:

The present invention performs individual feature extraction on rock points based on the YOLO target detection network. The detection speed is improved, and the useful feature information can be better used. The bounding box of the rock point after the YOLO network training is obtained, and then the edge detection of the rock point is performed by the Canny operator, and then the rock point is extracted based on the Shi-Tomasi algorithm. Feature matching is performed on the feature points, and the obtained feature points have strong features. In addition, due to the reduction of feature points, the number of iterations in the process of calculating a single adaptive matrix is reduced, and the characteristics of less calculation and high accuracy are realized.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

Fig. 1 is the overall framework flow chart of the method of the embodiment of the present invention;

FIG. 2 is a demonstration diagram of a weighted fusion method according to an embodiment of the present invention.

Detailed ways

The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, and the protection scope of the present invention can be more clearly defined.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

It should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. Based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of the invention is usually placed in use, it is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must be It has a specific orientation, is constructed and operates in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "first", "second", "third", etc. are only used to differentiate the description and should not be construed as indicating or implying relative importance.

Furthermore, the terms "horizontal", "vertical", "overhanging" etc. do not imply that a component is required to be absolutely horizontal or overhang, but rather may be slightly inclined. For example, "horizontal" only means that its direction is more horizontal than "vertical", it does not mean that the structure must be completely horizontal, but can be slightly inclined.

As shown in FIG. 1 , an image mosaic rock wall reconstruction method according to an embodiment of the present invention is performed in a speed rock climbing video. The method of the embodiment of the present invention is as follows:

(1) Steps of acquiring video frames: input the speed rock climbing video, set up a Z value (where Z is a positive integer) as a unit to intercept video frames, so that multiple scene graphs S _i (i=1, 2, . . . , N) Images as examples of the present invention. S _i is the image corresponding to the i-th video frame that has been cut in units of Z. The unit can be time or video frame interval.

(2) Image preprocessing step: perform mean filtering on images corresponding to all video frames to remove noise.

包含着岩点的位置坐标和尺度信息，

则是每个岩点的ID。(3) Rock point calibration step: calibrate N images one by one to obtain the information of each rock point on the left and right tracks in each image

Contains the location coordinates and scale information of the rock point,

is the ID of each move.

(4) Feature point acquisition step: feature points on the edge of rock points are extracted by an edge feature extraction method.

The edge feature extraction method includes the following steps:

Use the yolo network to train the calibration features, get the bounding box of each rock point and perform image grayscale for the rock points in the bounding box;

Use the Canny edge algorithm to detect the bounding box to get the image of the edge of each rock point;

The existing Canny edge detection algorithm includes the following steps: denoising the image; calculating the direction of the gradient intensity; retaining the pixels with maximum values in the edge direction; Bounds) track the boundaries through hysteresis techniques.

Use the Shi-Tomasi corner detection method to obtain the corner points on the edge of the rock point, and obtain the best corner point through the adaptive matrix, which is the feature point; the specific method is as follows:

A1. Observe the image containing the corners through the local window, move the local window in all directions (u,v) and calculate the sum of all grayscale differences:

E(u, v)=∑ _{x, y} w(x, y)[I(x+u, y+v)-I(x, y)] ² ; where x is and y is

where I(x, y) is the image grayscale of the local window, I(x+u, y+v) is the moved image grayscale, and w(x, y) is the window function;

A2. Perform corner detection to maximize the value of E(u, v), which is simplified to

设自适应矩阵

即

Let the adaptive matrix

which is

A3. Use the adaptive matrix M to find the corner points, set the eigenvalues of the adaptive matrix M as λ ₁ and λ ₂ , if the smaller one of the two eigenvalues of the matrix M is greater than the preset threshold, then the corner point R =min{λ ₁ ,λ ₂ };

A4. Calculate the quality of the corner points, set the maximum eigenvalue of the quality of the corner points as the threshold, and set the shortest Euclidean distance between the two corner points;

A5. Remove the corners below the threshold;

A6. Arrange the remaining corner points in descending order of quality;

A7. Select the first corner point in descending order as the best corner point and delete all the corner points whose Euclidean distance is less than the shortest Euclidean distance from the corner point;

A8. Repeat steps A6-A7 until the corner points cannot be deleted, and return to all the best corner points.

Obtain the coordinates and scale of the feature points and the direction information, size and angle information of the feature points.

(5) Feature matching step: The feature points extracted from the two images are matched by the matching algorithm until the feature points of each image are matched with the surrounding image feature points.

Use the KNN matching algorithm to match the feature points between the two images, and use the ratio test, that is, according to the method of comparing the nearest neighbor distance and the next nearest neighbor distance, when the distance ratio is less than a certain threshold, it can be considered that the two feature points are correctly matched. When the last image is matched, the feature points of each image are matched with the surrounding image feature points.

(6) Step of removing wrong matching: In the feature matching process of step (5), not all feature point pairings will be correct, and there may still be wrong matching pairs. The method for taking out an incorrect match according to an embodiment of the present invention includes the following steps:

B1. Using the RANSAC algorithm, randomly select 4 pairs of feature points from the matched feature points as inner points, and other feature points as outer points;

B2. Obtain the single adaptation matrix of the interior point;

B3. Use the single adaptive matrix to test all the outer points, and set a threshold condition, and classify all the outer points that meet the threshold condition after the single adaptive matrix test as new inner points;

B4. Repeat steps B2-B3 until the final single adaptation matrix can be obtained when the number of interior points does not change.

(h是重叠区域中某一个像素点距离拼接区域最上边的边界线的距离，P₁、P₂分别表示Picture 1和Picture 2的RGB取值分量)，得到一个拼接无裂缝的岩壁重建图。(7) Image stitching step: perform perspective transformation on the image, using the image and the single adaptation matrix as input, and then, it transforms the original image to the target plane according to the single adaptation matrix. At this time, the overlapping parts of a large area of the image are fused. During this process, stitching cracks will appear due to light color and other reasons. The embodiment of the present invention adopts the weighted fusion method. As shown in FIG. 2 , in the overlapping part, the previous image is slowly transitioned to the second image, that is, the pixel values P in the overlapping area of the images are added and synthesized according to a certain weight. which is

(h is the distance between a certain pixel in the overlapping area and the uppermost boundary line of the splicing area, P ₁ and P ₂ represent the RGB value components of Picture 1 and Picture 2 respectively), and obtain a spliced and crack-free rock wall reconstruction map .

Beneficial effects of the embodiments of the present invention:

The embodiment of the present invention performs individual feature extraction on rock points based on the YOLO target detection network. The detection speed is improved, and the useful feature information can be better used. The boundingbox of the rock points after the YOLO network training is obtained, and then the Canny operator is used to detect the edge of the rock points, and then the rock points are extracted based on the Shi-Tomasi algorithm. The feature points are matched, and the obtained feature points have stronger features. In addition, due to the reduction of feature points, the number of iterations in the process of calculating a single adaptive matrix is reduced, and the characteristics of less calculation and high accuracy are realized.

Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner can make various changes or modifications within the scope of the appended claims, as long as the protection scope described in the claims of the present invention is not exceeded, all should be within the protection scope of the present invention.

Claims (9)

1. a method for image splicing rock wall reconstruction, is characterized in that, comprises obtaining video frame step, obtaining feature point step, feature point matching step and image stitching step; The step of acquiring video frames is to intercept several video frames from the video to obtain images corresponding to the video frames; The step of acquiring feature points includes an image preprocessing step, a rock point calibration step and a feature point acquiring step; the image preprocessing step is to perform a preprocessing operation on the image; the rock point calibration step is to calibrate the rock points of the image; The feature point acquisition step is to extract feature points on the edge of rock points by an edge feature extraction method; the rock points are objects with irregular shapes and easy to be grasped in the image; In the feature point matching step, the feature points extracted from the two images are matched by the matching algorithm until the feature points of each image are matched with the surrounding image feature points; In the image splicing step, through perspective transformation, the overlapping area is determined, and the original image is transformed to the target plane through the splicing method for splicing operation until a complete reconstruction of the rock wall is obtained. 2. a kind of image mosaic rock wall reconstruction method according to claim 1 is characterized in that: the concrete method of intercepting several video frames in the described acquiring video frame step is: set the value of a unit to intercept video frame, The above value is used to intercept several video frames. 3. The method for reconstructing a rock wall by image splicing according to claim 1, wherein the specific method for performing a preprocessing operation on the image in the image preprocessing step is to perform mean filtering on all video frame images, remove the noise. 4. The method for reconstructing a rock wall by image splicing according to claim 1, wherein the specific method for calibrating the rock points of the image in the rock point calibration step is to calibrate each image one by one, A label in each image is obtained; the label includes the coordinates of the move and the name of the move. 5. a kind of image mosaic rock wall reconstruction method according to claim 1, is characterized in that: described edge feature extraction method comprises the following steps: Use the yolo network to train the calibration features, get the bounding box of each rock point and perform image grayscale for the rock points in the bounding box; Use the Canny edge algorithm to detect the bounding box to get the image of the edge of each rock point; Use the Shi-Tomasi corner detection method to obtain the corners on the edge of the rock point, and obtain the best corners through the adaptive matrix, which are the feature points; Obtain the coordinates and scale of the feature points and the direction information, size and angle information of the feature points. 6. The method for image mosaic rock wall reconstruction according to claim 5, characterized in that: the corner point on the edge of the rock point is obtained by using the Shi-Tomasi corner point detection method, and the optimal solution is obtained by an adaptive matrix. The specific method of the corner point is: A1. Observe the image containing the corners through the local window, move the local window in all directions (u, v) and calculate the sum of all grayscale differences: E(u,v)=∑ _x,y w(x,y)[I(x+u,y+v)-I(x,y)] ² ; where x is and y is where I(x,y) is the image grayscale of the local window, I(x+u,y+v) is the moved image grayscale, and w(x,y) is the window function; A2. Perform corner detection to maximize the value of E(u, v), which is simplified to

Let the adaptive matrix

which is

A3. Use the adaptive matrix M to find the corner points, set the eigenvalues of the adaptive matrix M as λ ₁ and λ ₂ , if the smaller one of the two eigenvalues of the matrix M is greater than the preset threshold, then the corner point R =min{λ ₁ ,λ ₂ }; A4. Calculate the quality of the corner points, set the maximum eigenvalue of the quality of the corner points as the threshold, and set the shortest Euclidean distance between the two corner points; A5. Remove the corners below the threshold; A6. Arrange the remaining corner points in descending order of quality; A7. Select the first corner point in descending order as the best corner point and delete all the corner points whose Euclidean distance is less than the shortest Euclidean distance from the corner point; A8. Repeat steps A6-A7 until the corner points cannot be deleted, and return to all the best corner points. 7. a kind of image mosaic rock wall reconstruction method according to claim 1, is characterized in that: in described feature point matching step, the specific method that the feature points that two images are extracted to carry out feature matching by matching algorithm is: Use the KNN matching algorithm to match the feature points between the two images, and use the ratio test, that is, according to the method of comparing the nearest neighbor distance and the next nearest neighbor distance, when the distance ratio is less than a predetermined threshold, it can be considered that the two feature points are correctly matched. 8. a kind of image splicing method for rock wall reconstruction according to claim 1, is characterized in that: before described image splicing step, also need to carry out the step of removing wrong matching: B1. Using the RANSAC algorithm, randomly select 4 pairs of feature points from the matched feature points as inner points, and other feature points as outer points; B2. Obtain the single adaptation matrix of the interior point; B3. Use the single adaptive matrix to test all the outer points, and set a threshold condition, and classify all the outer points that meet the threshold condition after the single adaptive matrix test as new inner points; B4. Repeat steps B2-B3 until the final single adaptation matrix can be obtained when the number of interior points does not change. 9. The method for reconstructing a rock wall by image splicing according to claim 1, wherein: in the image splicing step, the splicing operation adopts a weighted fusion method for splicing; The pixel values are added and synthesized according to a certain weight.

Images