CN113362363A - Automatic image annotation method and device based on visual SLAM and storage medium - Google Patents

Abstract

The invention discloses an image automatic labeling method, device and storage medium based on visual SLAM; the method first processes the current frame image to obtain a labeling frame, and then automatically labels other video images of different frames to correct the initial In order to obtain the final more accurate annotation data; this method does not require a priori data set, can realize online automatic annotation while collecting images, and supports multi-object and multi-category annotation. This method only needs a single annotation by the user. It can greatly reduce the manual image annotation work, improve the collection efficiency of image annotation data, and the operation is simple and easy to use. The invention can be widely used in the technical field of image processing.

Description

A kind of image automatic labeling method, device and storage medium based on visual SLAM

technical field

The invention relates to the technical field of image processing, in particular to a visual SLAM-based image automatic labeling method, device and storage medium.

Background technique

With the development of computer vision, artificial intelligence technologies such as machine learning and deep learning have been widely used in image classification, target detection and image segmentation and have achieved breakthrough results. However, the model training of such methods relies on a large number of For task-related image annotation data, image data collection and annotation of business scenarios must be performed manually before the model is applied. At present, this stage takes a long time and affects the overall research and development progress.

At present, the method of identifying images to be labeled based on deep learning is used for labeling, but this method needs to collect image data in advance for training and establish a corresponding candidate label set, so it cannot be directly used for the first labeling; The accuracy depends on the scale of the dataset and the probability distribution of each label, and the labeling objects are limited, so this method does not have universal applicability.

SUMMARY OF THE INVENTION

The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes an automatic image labeling method, device and storage medium based on visual SLAM.

The technical scheme adopted by the present invention is:

On the one hand, an embodiment of the present invention includes an automatic image labeling method based on visual SLAM, including:

Obtain at least two consecutive images from a real-time camera or video, and use the ORB-SLAM algorithm to construct a 3D point cloud space;

Obtain a new frame image as the current frame image, and obtain the camera pose by solving the PnP problem;

Taking the current frame image as a selection area, frame a marked area in the selection area, and mark corresponding semantic information in the marked area;

obtaining, according to the three-dimensional point cloud space, a first three-dimensional point cloud set corresponding to the two-dimensional feature points in the marked area;

Using a point cloud clustering algorithm to process the first three-dimensional point cloud set to obtain a second three-dimensional point cloud set, the second three-dimensional point cloud set is the point cloud cluster with the largest number of point clouds;

Calculate the minimum directed bounding box of the second three-dimensional point cloud set based on principal component analysis;

According to the camera pose and camera internal parameters, construct a camera perspective matrix;

According to the camera perspective matrix, project the minimum directed bounding box to the current frame image to form an augmented reality scene, and obtain a rectangular frame corresponding to the minimum directed bounding box;

image segmentation of the rectangular frame to identify the subject;

Adjust the rectangular frame according to the identified subject to obtain a labeling frame;

comparing the area of the rectangular frame and the callout frame;

If the ratio of the area of the callout frame to the area of the rectangular frame is less than the first threshold, the callout frame is used as the selection area, and the selection area is selected in the selection area, and the corresponding semantic information is marked in the callout area. step;

Obtain a target frame image from the video, where the target frame image is any subsequent frame image except the current frame image;

Obtain the first newly added point cloud data of the target frame image based on the visual SLAM algorithm;

Filtering the first newly added point cloud data to obtain second newly added point cloud data, where the second newly added point cloud data is the newly added point cloud data in the minimum directed bounding box;

If the quantity of the second newly added point cloud data is greater than the second threshold, the target frame image is used as a selection area, and the steps of selecting a marked area in the selection area and marking corresponding semantic information in the marked area are performed.

Further, the step of obtaining at least two frames of images in the real-time camera or the video, and constructing the three-dimensional point cloud space by using the ORB-SLAM algorithm, includes:

Calibrate the camera to obtain the camera internal parameter matrix and distortion parameters;

Obtain at least two consecutive images from a live camera or video;

Extracting ORB feature points from the continuous images of the two frames in turn and calculating the corresponding binary descriptors;

According to the ORB feature points, feature point matching is performed on the two consecutive images to obtain image feature points;

Using the epipolar geometric constraint to estimate the camera motion between the two consecutive images, the camera pose is calculated;

According to the camera internal parameter matrix, distortion parameters and camera pose, the three-dimensional coordinates of the image feature points are calculated by triangulation, and a three-dimensional point cloud space is constructed.

Further, the step of taking the current frame image as a selection area, selecting a marked area in the selection area, and marking corresponding semantic information in the marked area, includes:

Taking the current frame image as a selection area, select the first area in the selection area;

converting the first area into a pixel area according to the window size of the current frame image and the pixel aspect ratio of the current frame image;

The pixel area is used as an annotation area, and corresponding semantic information is annotated in the annotation area.

Further, the step of obtaining the first three-dimensional point cloud set corresponding to the two-dimensional feature points in the marked area according to the three-dimensional point cloud space includes:

acquiring two-dimensional feature points in the marked area;

According to the three-dimensional point cloud space, the two-dimensional feature points are back-projected to obtain a first three-dimensional point cloud set.

Further, the step of using a point cloud clustering algorithm to process the first three-dimensional point cloud set to obtain a second three-dimensional point cloud set includes:

Selecting a first point cloud from the first three-dimensional point cloud set as a target point cloud, and the first point cloud being any point cloud in the first three-dimensional point cloud set;

Determine whether the target point cloud is edge noise;

If the target point cloud is edge noise, then re-select a second point cloud from the first 3D point cloud set as the target point cloud, and the second point cloud is the first 3D point cloud set except Any point cloud other than the first point cloud mentioned above;

If the target point cloud is not an edge noise, find out all the point clouds that are density-reachable from the target point cloud from the first three-dimensional point cloud set to form a point cloud cluster;

Traverse all point clouds in the first three-dimensional point cloud set to obtain a plurality of point cloud clusters;

The point cloud cluster with the largest number of point clouds is selected as the second three-dimensional point cloud set.

Further, the step of constructing and obtaining the camera perspective matrix according to the camera pose and camera internal parameters includes:

世界坐标系为(R_w,t_w)，OpenGL坐标系为(R_o,t_o)；The data of the camera pose is transferred from the world coordinate system to the OpenGL coordinate system according to the first formula. The first formula is: R _o =R _ow R _w R _ow , to =R _{ow t w} ; wherein,

The world coordinate system is (R _w ,t _w ), and the OpenGL coordinate system is (R _o ,t _o );

According to the OpenGL coordinate system, combined with the camera internal parameters, the camera perspective matrix is constructed and obtained, and the camera perspective matrix is:

in,

In the formula, P represents the perspective matrix of the camera, and the internal parameters of the camera are (u ₀ , v ₀ , f _u , f _v ); near represents the distance from the camera optical center to the near section, far represents the distance from the camera optical center to the far section, and right represents The distance from the near section center to the right section, left indicates the distance from the near section center to the left section, top indicates the distance from the near section center to the upper section, bottom indicates the distance from the near section center to the lower section, w indicates the image pixel width, h indicates Image pixel height.

Further, the step of filtering the first newly added point cloud data to obtain the second newly added point cloud data includes:

Filtering the first newly added point cloud data based on a random sampling consistency algorithm to remove ground point cloud data;

Conditional filtering is performed according to the three-dimensional space range of the minimum directional bounding box to obtain second newly added point cloud data.

Further, if the number of the second newly added point cloud data is not greater than the second threshold, the target frame image is taken as the current frame image, and the minimum directed bounding box is executed according to the camera perspective matrix. The steps of projecting to the current frame image, forming an augmented reality scene, and obtaining a rectangular box corresponding to the minimum directed bounding box.

On the other hand, the embodiment of the present invention also includes an image automatic labeling device based on visual SLAM, including:

at least one processor;

at least one memory for storing at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the automatic image annotation method based on visual SLAM.

On the other hand, an embodiment of the present invention further includes a computer-readable storage medium on which a processor-executable program is stored, and the processor-executable program, when executed by the processor, is used to implement the vision-based SLAM method for automatic image annotation.

The beneficial effects of the present invention are:

The present invention proposes an automatic image labeling method based on visual SLAM, which includes: acquiring a video image, obtaining a three-dimensional point cloud space and camera pose based on constructing an ORB-SLAM algorithm, taking the current frame image as a selection area, and selecting a labeling area in the selection area , and label corresponding semantic information in the labeling area; obtain the first 3D point cloud set corresponding to the 2D feature points in the labeling area according to the 3D point cloud space; use the point cloud clustering algorithm to process the first 3D point cloud set , obtain the second three-dimensional point cloud set; calculate the minimum directed bounding box of the second three-dimensional point cloud set based on principal component analysis; construct the camera perspective matrix according to the camera pose and camera internal parameters; The directed bounding box is projected to the current frame image to form an augmented reality scene, and the rectangular frame corresponding to the smallest directed bounding box is obtained; the image segmentation of the rectangular frame is performed to identify the subject; the rectangular frame is adjusted according to the identified subject to obtain an annotation frame ; Compare the area of the rectangle and the callout; if the ratio of the area of the callout to the area of the rectangle is less than the first threshold, take the callout as the selection area, select the callout area in the selection area, and mark the corresponding semantics in the callout area information step; thus, the annotation result of the current frame image can be generated. Finally, combined with the newly added point cloud data of different video frames, the annotation frame is automatically adjusted to automatically generate the annotation results of each frame of image. This method does not require a priori data set, can realize online automatic annotation while collecting images, and supports multi-object and multi-category annotation. This method only needs a single annotation by the user, which can greatly reduce the manual image annotation work and improve the image annotation data. The collection efficiency is high, and the operation is simple and easy to use.

Additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a flowchart of the steps of the automatic image labeling method based on visual SLAM according to an embodiment of the present invention;

FIG. 2 is a specific flowchart of an image automatic labeling method based on visual SLAM according to an embodiment of the present invention.

3 is a flowchart of obtaining at least two frames of images in a video according to an embodiment of the present invention, and constructing a three-dimensional point cloud space by using an ORB-SLAM algorithm;

4 is a flowchart of processing the first three-dimensional point cloud set by using a point cloud clustering algorithm according to an embodiment of the present invention to obtain a second three-dimensional point cloud set;

5 is a specific flowchart of processing the first three-dimensional point cloud set by using a point cloud clustering algorithm according to an embodiment of the present invention to obtain a second three-dimensional point cloud set;

6 is a schematic diagram of a comparison between a rectangular frame and a marked frame according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of an apparatus for automatic image annotation based on visual SLAM according to an embodiment of the present invention.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, only used to explain the present invention, and should not be construed as a limitation of the present invention.

In the description of the present invention, it should be understood that the azimuth description, such as the azimuth or position relationship indicated by up, down, front, rear, left, right, etc., is based on the azimuth or position relationship shown in the drawings, only In order to facilitate the description of the present invention and simplify the description, it is not indicated or implied that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore should not be construed as limiting the present invention.

In the description of the present invention, the meaning of several is one or more, the meaning of multiple is two or more, greater than, less than, exceeding, etc. are understood as not including this number, above, below, within, etc. are understood as including this number. If it is described that the first and the second are only for the purpose of distinguishing technical features, it cannot be understood as indicating or implying relative importance, or indicating the number of the indicated technical features or the order of the indicated technical features. relation.

In the description of the present invention, unless otherwise clearly defined, words such as setting, installation, connection should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

The embodiments of the present application will be further described below with reference to the accompanying drawings.

1, an embodiment of the present invention provides a visual SLAM-based automatic image labeling method, including but not limited to the following steps:

S100. Acquire at least two consecutive frames of images from a real-time camera or video, and use the ORB-SLAM algorithm to construct a three-dimensional point cloud space;

S200 obtains a new frame image as the current frame image, and obtains the camera pose by solving the PnP problem;

S300. Take the current frame image as a selection area, select a marked area in the selection area, and mark corresponding semantic information in the marked area;

S400. According to the three-dimensional point cloud space, obtain the first three-dimensional point cloud set corresponding to the two-dimensional feature points in the marked area;

S500. Use a point cloud clustering algorithm to process the first three-dimensional point cloud set to obtain a second three-dimensional point cloud set, and the second three-dimensional point cloud set is the point cloud cluster with the largest number of point clouds;

S600. Calculate the minimum directed bounding box of the second three-dimensional point cloud set based on principal component analysis;

S700. According to the camera pose and camera internal parameters, construct the camera perspective matrix;

S800. Project the minimum directed bounding box to the current frame image according to the camera perspective matrix to form an augmented reality scene, and obtain a rectangular frame corresponding to the minimum directed bounding box;

S900. Perform image segmentation on the rectangular frame to identify the subject;

S1000. Adjust the rectangular frame according to the identified subject to obtain a label frame;

S1100. Compare the area of the rectangle frame and the callout frame;

S1200. If the ratio of the area of the marked frame to the area of the rectangular frame is less than the first threshold, the marked frame is used as a selection area, and the steps of selecting a marked area in the selection area and marking the corresponding semantic information in the marked area are performed;

S1300. Obtain the target frame image from the video, and the target frame image is any frame image except the current frame image;

S1400. Obtain the first newly added point cloud data of the target frame image based on the visual SLAM algorithm;

S1500. Filter the first newly added point cloud data to obtain the second newly added point cloud data, and the second newly added point cloud data is the newly added point cloud data in the minimum directed bounding box;

S1600. If the quantity of the second newly added point cloud data is greater than the second threshold, take the target frame image as a selection area, and perform the steps of selecting a marked area in the selection area, and marking corresponding semantic information in the marked area.

In this embodiment, in order to solve the problem that it takes a long time to manually label image data and restricts the progress of research and development, the embodiment of the present invention proposes an automatic image labeling method based on visual SLAM. It mainly includes the following steps: first, acquire video images, construct 3D point cloud space and camera pose based on visual SLAM algorithm, and construct augmented reality scene based on OpenGL coordinate system combined with camera internal parameter matrix; secondly, according to the user's single frame selection area The two-dimensional feature points of , back-project the corresponding three-dimensional point cloud data and cluster them, and then solve the minimum directed bounding box; Combined with the camera pose and projection matrix, the virtual bounding box of the wrapped entity can be projected to the two-dimensional A preliminary rectangular annotation frame is generated on the image; then, the subject is identified based on the image segmentation algorithm, and the frame selection area is adjusted reasonably to generate this annotation result. Finally, combined with the newly added point cloud data from different perspectives and the automatically adjusted two-dimensional frame selection area, the virtual bounding box will be further adjusted to automatically generate annotation data for each frame of image. The automatic image labeling method of the present embodiment performs data labeling from the image source, which can greatly reduce the manual image labeling work, improve the collection efficiency of image labeling data, and effectively promote the research and development progress of computer vision model training.

In this embodiment, the visual SLAM algorithm adopts the ORB-SLAM algorithm; the point cloud clustering algorithm adopts the density-based DBSCAN algorithm; the minimum directed bounding box algorithm adopts the oriented bounding box algorithm (Oriented Bounding Box); the image segmentation algorithm adopts the region growth algorithm.

Specifically, with regard to step S100, in this embodiment, by acquiring at least two consecutive images from a real-time camera or video, the ORB-SLAM algorithm can be used to construct a three-dimensional point cloud space. The three-dimensional point cloud space can be constructed and obtained. Referring to Figure 3, the specific operation process is as follows:

S101. Calibrate the camera to obtain the camera internal parameter matrix and distortion parameters;

S102. Acquire at least two consecutive frames of images from the real-time camera or video;

S103. Extract ORB feature points from two consecutive frames of images in turn and calculate corresponding binary descriptors;

S104. According to the ORB feature points, perform feature point matching on two consecutive images to obtain image feature points;

S105. Use epipolar geometric constraints to estimate the camera motion between two consecutive images, and calculate the camera pose;

S106. According to the camera internal parameter matrix, the distortion parameters and the camera pose, use triangulation to calculate the three-dimensional coordinates of the image feature points, and construct a three-dimensional point cloud space.

In this embodiment, the camera that collects video images is calibrated to obtain the internal parameter matrix and distortion parameters of the camera, ORB feature points are sequentially extracted from two consecutive frames of images and ORB descriptors are obtained, and then feature point matching is performed on the two consecutive frames of images. , obtain the image feature points, and finally solve the motion of the camera between two consecutive images according to the pole constraint, and use the triangulation measurement to calculate the three-dimensional coordinates of the image feature points to construct a three-dimensional point cloud space. If the acquired video images are more than two frames, such as randomly intercepting a video, at this time, the camera also needs to be calibrated to obtain the camera internal parameter matrix and distortion parameters, and then extract the ORB feature points from the video image in time sequence and find its descriptors , perform feature point matching between adjacent frames, solve the motion of the camera between adjacent frames according to the extreme point constraints, and use triangulation to calculate the three-dimensional coordinates of image feature points to construct a three-dimensional point cloud space.

In this embodiment, for step S200, for the subsequent image frames in the video, the PnP problem is solved to calculate the camera pose. In addition, the ORB-SLAM algorithm will perform closed-loop detection, relocation, and graph optimization of key frames, so as to obtain a more accurate camera pose, that is, the camera's rotation matrix R and translation vector t.

For step S300, that is, taking the current frame image as a selection area, selecting a marked area in the selection area, and marking the corresponding semantic information in the marked area, the step specifically includes:

S301. Take the current frame image as the selection area, and select the first area in the selection area;

S302. Convert the first area into a pixel area according to the window size of the current frame image and the pixel aspect ratio of the current frame image;

S303. Use the pixel area as the labeling area, and label corresponding semantic information in the labeling area.

In this embodiment, the user needs to perform the first frame selection on the target object of the current frame image. Specifically, the current frame image is used as the selection area, and the area where the frame selection target object is located in the selection area is the marked area. Then, according to the marked area selected by the user on the current frame image, according to the ratio of the window size and the image pixel width and height, it is converted into a pixel area, and the semantic information marked in the area is saved.

In this embodiment, step S400, that is, the step of acquiring the first three-dimensional point cloud set corresponding to the two-dimensional feature points in the marked area according to the three-dimensional point cloud space, specifically includes:

S401. Obtain two-dimensional feature points in the marked area;

S402. Back-project the two-dimensional feature points according to the three-dimensional point cloud space to obtain a first three-dimensional point cloud set.

In this embodiment, considering that the marked area on the current frame image is a two-dimensional plane, the point clouds in the marked area are all two-dimensional feature points. Therefore, it is necessary to back-project the two-dimensional feature points according to the previously constructed three-dimensional point cloud space to obtain the first three-dimensional point cloud set corresponding to the two-dimensional feature points.

In this embodiment, after the first three-dimensional point cloud set is obtained, it is necessary to further process the first three-dimensional point cloud set by using a point cloud clustering algorithm to obtain a second three-dimensional point cloud set, wherein the second three-dimensional point cloud set is: The point cloud cluster with the largest number of point clouds. That is to say, after obtaining the first three-dimensional point cloud set, step S500 needs to be performed. Referring to FIG. 4 , step S500 specifically includes:

S501. Select the first point cloud as the target point cloud from the first three-dimensional point cloud set, and the first point cloud is any point cloud in the first three-dimensional point cloud set;

S502. Determine whether the target point cloud is edge noise;

S503. If the target point cloud is edge noise, then re-select the second point cloud from the first three-dimensional point cloud set as the target point cloud, and the second point cloud is any arbitrary point cloud other than the first point cloud in the first three-dimensional point cloud set point cloud;

S504. If the target point cloud is not edge noise, find out all the point clouds that are density-reachable from the target point cloud from the first three-dimensional point cloud set to form a point cloud cluster;

S505. Traverse all point clouds in the first three-dimensional point cloud set to obtain multiple point cloud clusters;

S506. Select the point cloud cluster with the largest number of point clouds as the second three-dimensional point cloud set.

Specifically, referring to FIG. 5 , in this embodiment, according to the business scenario, the field radius and the threshold of the number of objects in the field are set; then a three-dimensional point cloud is arbitrarily selected from the first three-dimensional point cloud set obtained in step S400 as the core point P, Find all 3D point clouds that are density-reachable from point cloud P to form a point cloud cluster; if the selected point cloud is edge noise, select other 3D point clouds as core points, and repeat the above behavior until the first 3D point cloud is accessed. All point clouds within the point cloud collection. Finally, the point cloud cluster with the largest number of 3D point clouds is reserved as the second 3D point cloud set.

In this embodiment, if the number of points in the neighborhood of the point cloud P is not greater than the preset threshold, the point cloud P is marked as edge noise, and at this time, other point clouds are reselected from the first three-dimensional point cloud set as the core point. By selecting different core points, multiple different point cloud clusters will be created, and finally, the point cloud cluster with the largest number of point clouds needs to be selected from all the obtained point cloud clusters as the second three-dimensional point cloud set. In this embodiment, after the second three-dimensional point cloud set is obtained, it is necessary to remove a small amount of remaining external noise points.

For step S600, in this embodiment, after the second three-dimensional point cloud set is obtained, the minimum directed bounding box of the second three-dimensional point cloud set can be obtained based on principal component analysis (Principal Component Analysis). The center point position, size and rotation matrix to the bounding box.

In this embodiment, step S700, that is, the step of constructing and obtaining the camera perspective matrix according to the camera pose and camera internal parameters, specifically includes:

世界坐标系为(R_w,t_w)，OpenGL坐标系为(R_o,t_o)；S701. Transfer the camera pose data from the world coordinate system to the OpenGL coordinate system according to the first formula. The first formula is: R _o =R _ow R _w R _ow , t _o =R _ow t _w ; wherein,

The world coordinate system is (R _w ,t _w ), and the OpenGL coordinate system is (R _o ,t _o );

S702. According to the OpenGL coordinate system, combined with the internal parameters of the camera, the camera perspective matrix is constructed and obtained. The camera perspective matrix is:

in,

In the formula, P represents the perspective matrix of the camera, and the internal parameters of the camera are (u ₀ , v ₀ , f _u , f _v ); near represents the distance from the camera optical center to the near section, far represents the distance from the camera optical center to the far section, and right represents The distance from the near section center to the right section, left indicates the distance from the near section center to the left section, top indicates the distance from the near section center to the upper section, bottom indicates the distance from the near section center to the lower section, w indicates the image pixel width, h indicates Image pixel height.

In this embodiment, the camera pose data is transferred from the world coordinate system (R _w ,t _w ) to the OpenGL coordinate system (R _o ,t _o ) according to the first formula, combined with the camera internal parameters (u ₀ , v ₀ , f _{u )} , f _v ) can be constructed to obtain the camera perspective matrix P.

In this embodiment, after the camera perspective matrix P is constructed and obtained, step S800 is performed, that is, according to the camera perspective matrix, the minimum directed bounding box is projected to the current frame image to form an augmented reality scene, and the rectangle corresponding to the minimum directed bounding box is obtained. frame. Specifically, the minimum directed bounding box is projected onto the current image to form an augmented reality scene. Compare the projected pixel coordinates of the 8 vertices of the smallest directed bounding box, retain the largest and smallest u and v coordinate values, and form the coordinates of the upper left and lower right corners of the rectangular box.

In this embodiment, after the rectangular frame is obtained, considering that the rectangular frame contains a large range, the labeling object (the main body) may not be tightly wrapped, so the rectangular frame needs to be further processed, and steps S1000 and S1100 need to be further performed, that is, to The rectangular frame is image-segmented to identify the subject, and then the rectangular frame is adjusted according to the identified subject to obtain an annotation frame. Specifically, the rectangular frame is appropriately enlarged first, the image in the rectangular frame is segmented to identify the subject, the size of the rectangular frame is adjusted to compactly wrap the subject, and the adjusted rectangular frame is used as a labeling frame. Referring to FIG. 6 , FIG. 6 is a schematic diagram of a comparison between a rectangular frame and a marked frame.

In this embodiment, the image segmentation is carried out by using the region growing algorithm. Specifically, the center point of the rectangular frame is taken as the seed point, and is marked as the visited point at the same time, and 4 neighborhood expansion is performed at the seed point. If the absolute value of the pixel gray value difference is less than the threshold value, it will be merged into the same area, and the neighborhood point will be added to the collection and saved. Take unvisited points from the set as seed points and repeat the above behavior until the set is empty. The rectangle frame formed by the new area is used as the callout frame.

In this embodiment, after the callout frame is obtained, the areas of the rectangle frame and the callout frame are further compared; if the ratio of the area of the callout frame to the area of the rectangle frame is smaller than the first threshold, the callout frame is used as the selection area, and the selection in the selection area is performed. The step of labeling the area and labeling the corresponding semantic information in the labeling area, that is, repeating step S300 with the labeling frame as the selection area, until the ratio of the area of the labeling frame to the area of the rectangular frame obtained is not less than the first threshold. If the ratio of the area of the labeling frame to the area of the rectangular frame is not less than the first threshold, it means that the labeling frame is the labeling data required for the current frame image.

After completing the annotation of the current frame image, different 3D point cloud data will be added due to different perspectives, or for different video frame images, the corresponding perspectives are different. Therefore, for other different video frames When the image is processed, the correction of the annotation data is first completed through steps S1300-S1600. Specifically, select the next frame of video image, first perform step S1400, that is, obtain the first newly added point cloud data of this frame of video image based on the visual SLAM algorithm; then perform step S1500, that is, filter the first newly added point cloud data , obtain the second newly added point cloud data, and the second newly added point cloud data is the newly added point cloud data in the minimum directed bounding box; specifically, this step specifically includes:

S1501. Filter the first newly added point cloud data based on the random sampling consistency algorithm to remove the ground point cloud data;

S1502. Perform conditional filtering according to the three-dimensional space range of the minimum directed bounding box to obtain second newly added point cloud data.

After the second newly added point cloud data is obtained, step S1600 is performed again, that is, it is determined whether the second newly added point cloud data is greater than the second threshold, and if the quantity of the second newly added point cloud data is greater than the second threshold, the frame of video The image is a selection area, and the steps of selecting a marked area in the selection area and marking the corresponding semantic information in the marked area are performed. That is to say, if the number of the second newly added point cloud data is greater than the second threshold, it means that the video image of this frame is far from the current frame image processed before. S300, to obtain a new standard frame corresponding to the frame of video image.

In this embodiment, if the number of the second newly added point cloud data is not greater than the second threshold, the target frame image is taken as the current frame image, and the minimum directed bounding box is projected to the current frame image according to the camera perspective matrix, The steps of forming an augmented reality scene and obtaining the rectangular box corresponding to the minimum directed bounding box. That is to say, if the amount of the second newly added point cloud data is not greater than the second threshold, it means that the video image of this frame is not much different from the current frame image processed before. In the case of the projection of the minimum directed bounding box, step S800 is re-executed. In this embodiment, the specific process of the automatic image labeling method based on visual SLAM may refer to FIG. 2 .

An image automatic labeling method based on visual SLAM in the embodiment of the present invention has the following technical effects:

An embodiment of the present invention proposes an automatic image labeling method based on visual SLAM, which includes: acquiring a video image, obtaining a three-dimensional point cloud space and a camera pose based on constructing an ORB-SLAM algorithm, taking the current frame image as a selection area, and selecting a frame in the selection area. Mark the area, and mark the corresponding semantic information in the marked area; obtain the first three-dimensional point cloud set corresponding to the two-dimensional feature points in the marked area according to the three-dimensional point cloud space; use the point cloud clustering algorithm to process the first three-dimensional point cloud set , obtain the second three-dimensional point cloud set; calculate the minimum directed bounding box of the second three-dimensional point cloud set based on principal component analysis; construct the camera perspective matrix according to the camera pose and camera internal parameters; The directed bounding box is projected to the current frame image to form an augmented reality scene, and the rectangular frame corresponding to the smallest directed bounding box is obtained; the image segmentation of the rectangular frame is performed to identify the subject; the rectangular frame is adjusted according to the identified subject to obtain an annotation frame ; Compare the area of the rectangle and the callout; if the ratio of the area of the callout to the area of the rectangle is less than the first threshold, take the callout as the selection area, select the callout area in the selection area, and mark the corresponding semantics in the callout area information step; thus, the annotation result of the current frame image can be generated. Finally, combined with the newly added point cloud data of different video frames, the annotation frame is automatically adjusted to automatically generate the annotation results of each frame of image. This method does not require a priori data set, can realize online automatic annotation while collecting images, and supports multi-object and multi-category annotation. This method only needs a single annotation by the user, which can greatly reduce the manual image annotation work and improve the image annotation data. The collection efficiency is high, and the operation is simple and easy to use.

Referring to FIG. 7 , an embodiment of the present invention further provides a visual SLAM-based image automatic labeling apparatus 200, which specifically includes:

at least one processor 210;

at least one memory 220 for storing at least one program;

When at least one program is executed by at least one processor 210 , at least one processor 210 implements the method shown in FIG. 1 .

The memory 220, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Memory 220 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some implementations, the memory 220 may optionally include remote memory located remotely from the processor 210, which may be connected to the processor 210 through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

It can be understood that the device structure shown in FIG. 7 does not constitute a limitation to the device 200, and may include more or less components than those shown, or combine some components, or arrange different components.

In the apparatus 200 shown in FIG. 7 , the processor 210 can call the program stored in the memory 220 and execute but not limited to the steps of the embodiment shown in FIG. 1 .

The above-described embodiment of the apparatus 200 is only illustrative, and the units described as separate components may or may not be physically separated, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments.

An embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a program executable by a processor, and the program executable by the processor is used to implement the program as shown in FIG. 1 when executed by the processor Methods.

Embodiments of the present application further disclose a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method shown in FIG. 1 .

It will be understood that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit . Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data flexible, removable and non-removable media. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or may Any other medium used to store desired information and which can be accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and can include any information delivery media, as is well known to those of ordinary skill in the art .

The embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned embodiments, and within the scope of knowledge possessed by those of ordinary skill in the technical field, various changes can also be made without departing from the purpose of the present invention. .

Claims (10)

1. an image automatic labeling method based on visual SLAM, is characterized in that, comprises: Obtain at least two consecutive images from a real-time camera or video, and use the ORB-SLAM algorithm to construct a 3D point cloud space; Obtain a new frame image as the current frame image, and obtain the camera pose by solving the PnP problem; Taking the current frame image as a selection area, frame a marked area in the selection area, and mark corresponding semantic information in the marked area; obtaining, according to the three-dimensional point cloud space, a first three-dimensional point cloud set corresponding to the two-dimensional feature points in the marked area; Using a point cloud clustering algorithm to process the first three-dimensional point cloud set to obtain a second three-dimensional point cloud set, the second three-dimensional point cloud set is the point cloud cluster with the largest number of point clouds; Calculate the minimum directed bounding box of the second three-dimensional point cloud set based on principal component analysis; According to the camera pose and camera internal parameters, construct a camera perspective matrix; According to the camera perspective matrix, project the minimum directed bounding box to the current frame image to form an augmented reality scene, and obtain a rectangular frame corresponding to the minimum directed bounding box; image segmentation of the rectangular frame to identify the subject; Adjust the rectangular frame according to the identified subject to obtain a labeling frame; comparing the area of the rectangular frame and the callout frame; If the ratio of the area of the callout frame to the area of the rectangular frame is smaller than the first threshold, the callout frame is used as a selection area, and the selection of the callout area in the selection area is performed, and corresponding semantics are marked in the callout area information steps; Obtain a target frame image from the video, where the target frame image is any subsequent frame image except the current frame image; Obtain the first newly added point cloud data of the target frame image based on the visual SLAM algorithm; Filtering the first newly added point cloud data to obtain second newly added point cloud data, where the second newly added point cloud data is the newly added point cloud data in the minimum directed bounding box; If the quantity of the second newly added point cloud data is greater than the second threshold, take the target frame image as a selection area, and execute the step of selecting a marked area in the selection area, and marking corresponding semantic information in the marked area . 2. a kind of image automatic labeling method based on visual SLAM according to claim 1, is characterized in that, described acquisition real-time camera or at least two frames of images in the video, utilize ORB-SLAM algorithm to construct and obtain three-dimensional point cloud space this. steps, including: Calibrate the camera to obtain the camera internal parameter matrix and distortion parameters; Obtain at least two consecutive images from a live camera or video; Extracting ORB feature points from the continuous images of the two frames in turn and calculating the corresponding binary descriptors; According to the ORB feature points, feature point matching is performed on the two consecutive images to obtain image feature points; Using the epipolar geometric constraint to estimate the camera motion between the two consecutive images, the camera pose is calculated; According to the camera internal parameter matrix, distortion parameters and camera pose, the three-dimensional coordinates of the image feature points are calculated by triangulation, and a three-dimensional point cloud space is constructed. 3. a kind of image automatic labeling method based on visual SLAM according to claim 1, it is characterized in that, described taking the current frame image as a selection area, in the selection area frame selection label area, and label corresponding semantic information in the described label area This step includes: Taking the current frame image as a selection area, select the first area in the selection area; converting the first area into a pixel area according to the window size of the current frame image and the pixel aspect ratio of the current frame image; The pixel area is used as an annotation area, and corresponding semantic information is annotated in the annotation area. 4. A kind of image automatic labeling method based on visual SLAM according to claim 1, is characterized in that, according to described 3D point cloud space, obtain the first 3D point corresponding to 2D feature point in described labeling area This step of cloud collection includes: acquiring two-dimensional feature points in the marked area; According to the three-dimensional point cloud space, the two-dimensional feature points are back-projected to obtain a first three-dimensional point cloud set. 5. a kind of image automatic labeling method based on visual SLAM according to claim 1, is characterized in that, described utilizes point cloud clustering algorithm to process described first three-dimensional point cloud collection, obtains the second three-dimensional point cloud Assemble this step, including: Selecting a first point cloud from the first three-dimensional point cloud set as a target point cloud, and the first point cloud being any point cloud in the first three-dimensional point cloud set; Determine whether the target point cloud is edge noise; If the target point cloud is edge noise, then re-select a second point cloud from the first 3D point cloud set as the target point cloud, and the second point cloud is the first 3D point cloud set except Any point cloud other than the first point cloud mentioned above; If the target point cloud is not an edge noise, find out all the point clouds that are density-reachable from the target point cloud from the first three-dimensional point cloud set to form a point cloud cluster; Traverse all point clouds in the first three-dimensional point cloud set to obtain a plurality of point cloud clusters; The point cloud cluster with the largest number of point clouds is selected as the second three-dimensional point cloud set. 6. a kind of image automatic labeling method based on visual SLAM according to claim 1, is characterized in that, described according to described camera pose and camera internal parameter, construct and obtain the step of camera perspective matrix, comprising: The data of the camera pose is transferred from the world coordinate system to the OpenGL coordinate system according to the first formula. The first formula is: R _o =R _ow R _w R _ow , to =R _{ow t w} ; wherein,

The world coordinate system is (R _w ,t _w ), and the OpenGL coordinate system is (R _o ,t _o ); According to the OpenGL coordinate system, combined with the camera internal parameters, the camera perspective matrix is constructed and obtained, and the camera perspective matrix is:

in,

In the formula, P represents the perspective matrix of the camera, and the internal parameters of the camera are (u ₀ , v ₀ , f _u , f _v ); near represents the distance from the camera optical center to the near section, far represents the distance from the camera optical center to the far section, and right represents The distance from the near section center to the right section, left indicates the distance from the near section center to the left section, top indicates the distance from the near section center to the upper section, bottom indicates the distance from the near section center to the lower section, w indicates the image pixel width, h indicates Image pixel height. 7. a kind of image automatic labeling method based on visual SLAM according to claim 1, it is characterized in that, the described first newly added point cloud data is filtered, the step that obtains the second newly added point cloud data ,include: Filtering the first newly added point cloud data based on a random sampling consistency algorithm to remove ground point cloud data; Conditional filtering is performed according to the three-dimensional space range of the minimum directional bounding box to obtain second newly added point cloud data. 8. A kind of image automatic labeling method based on visual SLAM according to claim 1, is characterized in that, if the quantity of described second newly added point cloud data is not greater than the second threshold value, then described target frame image is used as. The current frame image, and executing the projection of the minimum directional bounding box to the current frame image according to the camera perspective matrix to form an augmented reality scene, and obtaining the minimum directional bounding box corresponding to the rectangular box step. 9. An image automatic labeling device based on visual SLAM, characterized in that, comprising: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor is caused to implement the method of any one of claims 1-8. 10. A computer-readable storage medium, characterized in that a program executable by a processor is stored thereon, and the program executable by the processor, when executed by the processor, is used to implement the program as claimed in any one of claims 1-8. method described.

Images