CN115115978A - Object recognition method, device, storage medium and processor - Google Patents

Abstract

The invention discloses an object identification method, device, storage medium and processor. Wherein, the method includes: acquiring a plurality of detection frames of an object in a video, wherein each detection frame is used to represent the position of the object in the video; respectively determining the number of time frames of the plurality of detection frames in the video, wherein the time frame The number is used to represent the length of time that each detection frame appears continuously in the video; based on the time frame number and the frame number difference threshold, the target detection frame is determined in multiple detection frames; based on the target detection frame, the object is identified in the video. The invention solves the technical problem of low accuracy of object recognition.

Description

Object recognition method, device, storage medium and processor

technical field

The present invention relates to the field of vehicles, and in particular, to an object recognition method, device, storage medium and processor.

Background technique

At present, in the process of autonomous driving, based on the results of the detection of objects on the traffic road, the information such as the location, size, direction and category of the objects in space can be accurately determined, so as to achieve 3D modeling, path planning and other purposes.

In the related art, for the detection of objects, the detection method of Non-Maximum Suppression (NMS for short) is usually used, but this method is only suitable for detection in two-dimensional scenes, so it is not suitable for spatial detection. Detection, there is a technical problem of low accuracy of object recognition.

For the problem of low accuracy of object recognition in the above-mentioned related art, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide an object recognition method, device, storage medium and processor, so as to at least solve the technical problem of low target recognition accuracy.

According to an aspect of the embodiments of the present invention, an object identification method is provided, including: acquiring a plurality of detection frames of an object in a video, wherein each detection frame is used to represent the position of the object in the video; The number of time frames of the detection frame in the video, where the number of time frames is used to represent the length of time that each detection frame appears continuously in the video; based on the number of time frames and the frame number difference threshold, target detection is determined in multiple detection frames Box; identifies objects in video based on object detection boxes.

Optionally, based on the time frame number and the frame number difference threshold, determining the target detection frame in the multiple detection frames includes: determining the intersection of the first detection frame in the multiple detection frames and the second detection frame in the multiple detection frames. ratio, wherein, the first detection frame and the second detection frame are any two detection frames in multiple detection frames at the same time; in response to the intersection ratio being greater than the intersection ratio threshold, the time frame number and the first detection frame of the first detection frame are obtained. 2. The time frame number of the detection frame; the target detection frame is determined in the first detection frame and the second detection frame based on the time frame number of the first detection frame, the time frame number of the second detection frame and the frame number difference threshold.

Optionally, based on the time frame number of the first detection frame, the time frame number of the second detection frame, and the frame number difference threshold, determining the target detection frame from the first detection frame and the second detection frame includes: determining the first detection frame. The first frame number difference between the time frame number of the frame and the time frame number of the second detection frame; the target is determined from the first detection frame and the second detection frame based on the first frame number difference and the frame number difference threshold Check box.

Optionally, based on the first frame number difference and the frame number difference threshold, determining the target detection frame from the first detection frame and the second detection frame, comprising: in response to the absolute value of the first frame number difference being greater than the frame number difference threshold, A detection frame with a long number of time frames in the first detection frame and the second detection frame is determined as a target detection frame.

Optionally, based on the first frame number difference and the frame number difference threshold, the target detection frame is determined from the first detection frame and the second detection frame, including: in response to the absolute value of the first frame number difference being not greater than the frame number difference threshold value , a detection frame with a large degree of matching between the first detection frame and the second detection frame is determined as a target detection frame, wherein the matching degree is used to represent the matching degree between the corresponding detection frame and the object.

Optionally, obtain the matching degree corresponding to each detection frame at each moment in the video, and obtain at least one matching degree of each detection frame; the sum of the at least one matching degree of the detection frame and the number of the at least one matching degree are both obtained. The quotient between is determined as the target matching degree of the detection frame.

Optionally, the frame number difference threshold is determined based on historical detection data of object processing.

According to another aspect of the embodiments of the present invention, there is also provided an object identification device, comprising: an acquisition unit configured to acquire a plurality of detection frames of an object in a video, wherein each detection frame is used to indicate that the object is in the video The first determining unit is used to respectively determine the number of time frames of multiple detection frames in the video, wherein the number of time frames is used to characterize the time length of each detection frame appearing continuously in the video; the second determining unit, It is used to determine the target detection frame in multiple detection frames based on the time frame number and the frame number difference threshold; the recognition unit is used to identify the object in the video based on the target detection frame.

According to another aspect of the embodiments of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored program, wherein when the program runs, the device where the computer-readable storage medium is located is controlled to execute the object identification method according to the embodiment of the present invention.

According to another aspect of the embodiments of the present invention, a processor is also provided. The processor is used for running a program, wherein the object identification method of the embodiment of the present invention is executed when the program is running.

In the embodiment of the present invention, a plurality of detection frames of an object in a video are obtained, wherein each detection frame is used to represent the position of the object in the video; the number of time frames of the plurality of detection frames in the video is determined respectively, wherein the time frame The number of frames is used to characterize the length of time that each detection frame appears continuously in the video; the target detection frame is determined in multiple detection frames based on the number of time frames and the difference threshold of the frame number; the object is identified in the video based on the target detection frame. That is to say, the embodiment of the present invention achieves the purpose of filtering some overlapping objects with a short appearance time by comparing the number of time frames between multiple detection frames, thereby achieving the technical effect of improving the accuracy of object recognition, and solving the problem of The technical problem of low accuracy of object recognition is solved.

Description of drawings

The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the attached image:

1 is a flowchart of a method for identifying an object according to an embodiment of the present invention;

2 is a flowchart of a method for acquiring a detection frame according to an embodiment of the present invention;

3 is a schematic diagram of a non-maximum suppression identification process according to the related art;

Fig. 4(a) is a schematic diagram of a processing result without non-maximum suppression according to an embodiment of the present invention,

Figure 4(b) is a schematic diagram of a processing result without non-maximum suppression according to an embodiment of the present invention;

5 is a flowchart of a processing method for non-maximum suppression according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of a device for recognizing an object according to an embodiment of the present invention.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only Embodiments are part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

Example 1

According to an embodiment of the present invention, an embodiment of an object identification method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and, Although a logical order is shown in the flowcharts, in some cases steps shown or described may be performed in an order different from that herein.

1 is a flowchart of a method for recognizing an object according to an embodiment of the present invention, and the flowchart of the method for recognizing an object as shown in FIG. 1 includes the following steps:

Step S102, acquiring multiple detection frames of the object in the video, wherein each detection frame is used to represent the position of the object in the video.

In the technical solution provided in the above step S102 of the present invention, an image for a period of time can be obtained by means of various sensors, etc., to obtain a video to be processed, and the video can be processed to achieve the purpose of obtaining multiple detection frames of objects in the video, The video may be an image of a period of time, or an image of a period of time collected by a collection device such as a camera. For example, it may be a picture of a continuous period of time captured by the vehicle's on-board camera during driving; the object may be The objects present in the video, for example, can be trucks on high-speed road sections, pedestrians at intersections, etc. There are no specific restrictions on the video acquisition method and the types of objects in the video; the detection frame can be a slanted frame ( rotate box, referred to as rbox), can be a three-dimensional three-dimensional detection box (3D rbox), for example, it can be a cube, a cuboid, or a two-dimensional plane two-dimensional detection box (2D rbox), for example, it can be a rectangle .

Optionally, multiple detection frames of objects in the video can be obtained through sensors installed in different positions. For example, three-dimensional detection frames can be obtained through lidar and vision (camera) sensors, and can be obtained through millimeter wave radar. 2D detection frame.

For example, when a vehicle is driving on a roundabout, a construction road, a downtown area, a straight road, etc., to obtain a video of the vehicle driving for a period of time, each frame in the video can be analyzed by devices such as lidar, millimeter-wave radar, and vision. The picture is processed to obtain a detection frame corresponding to an object in each frame of the video, and a plurality of detection frames are obtained.

Optionally, the three-dimensional detection frame can be obtained through vision (camera) and lidar (lidar), and the detected three-dimensional detection frame can be projected onto the ground to obtain a two-dimensional detection frame, thereby obtaining the center point, size, orientation, etc. of the detection frame. information, where the information of the detection frame can be used to represent the position of the object in the video in the video.

In the related art, the detection of the two-dimensional detection frame can only be realized by the non-maximum suppression detection method, but this method has limitations on the filtering of objects in the three-dimensional space in the scene. The three-dimensional detection frame of each object in the video in each frame, and the three-dimensional detection frame is projected to the plane, so as to achieve the purpose of obtaining multiple two-dimensional detection frames of the object in the video.

Step S104, respectively determining the number of time frames in the video for the multiple detection frames, where the number of time frames is used to represent the time length of each detection frame appearing continuously in the video.

In the technical solution provided by the above step S104 of the present invention, the number of time frames in which each detection frame appears continuously in the video can be determined respectively, wherein the number of time frames can be represented by frame_cnt, which can be used to represent each detection frame in the video. The length of time for continuous occurrence, for example, may be 1 second, 2 seconds, etc., which is only for illustration here, and is not specifically limited.

For example, if the detection frame corresponding to object 1 appears three frames consecutively from the second frame of the video, the number of time frames in which the detection frame corresponding to object 1 appears in the video may be three frames.

Step S106 , based on the time frame number and the frame number difference threshold, determine a target detection frame in a plurality of detection frames.

In the technical solution provided by the above step S106 of the present invention, based on the number of time frames corresponding to each detection frame, the difference in the number of frames between the two detection frames is determined, and the number of time frames is compared with the threshold of the difference in the number of frames. Using the comparison results, the target detection frame is determined from multiple detection frames, wherein the target detection frame can be the detection frame with the highest degree of matching with the object, and the frame number difference threshold can be called the time threshold, which can be represented by time_threshold, which can be based on experience The set value can also be the value set according to the actual situation.

For example, starting from the second frame of the video, the detection frame corresponding to object 1 appears in the video for three consecutive frames, then the number of time frames in the video for object 1 is three frames, and the detection frame corresponding to object 2 appears in the video. If five frames appear in a row, the number of time frames of object two in the video is five frames. Assuming that the frame number difference threshold is one frame, it can be based on the number of time frames of object one, the number of time frames of object two, and the set frame. The number difference threshold is used to filter between object 1 and object 2, so as to achieve the purpose of determining the target detection frame from multiple detection frames.

In the related art, by detecting images with a single time frame number, the influence of other time frame information on the perception result in a period of time is ignored. The number of time frames that appear sets a time threshold, thereby filtering out overlapping objects that appear for a short time, thereby improving the accuracy of object recognition.

Step S108, an object is identified in the video based on the target detection frame.

In the technical solution of the above step S108 of the present invention, information such as the position and direction of the object in each frame of the video is identified based on the target detection frame.

In the above steps S102 to S108 of the present application, multiple detection frames of the object in the video are obtained, wherein each detection frame is used to represent the position of the object in the video; the time frame numbers of the multiple detection frames in the video are respectively determined, wherein , the number of time frames is used to characterize the length of time that each detection frame appears continuously in the video; based on the number of time frames and the frame number difference threshold, the target detection frame is determined in multiple detection frames; based on the target detection frame, the target detection frame is identified in the video object, so as to achieve the technical effect of improving the accuracy of target recognition, and solve the technical problem of low accuracy of target recognition.

The above method of this embodiment will be further described below.

As an optional embodiment, determining the target detection frame in multiple detection frames based on the time frame number and the frame number difference threshold includes: determining a first detection frame in the multiple detection frames and a second detection frame in the multiple detection frames The intersection ratio of the detection frame, wherein the first detection frame and the second detection frame are any two detection frames at the same time in the multiple detection frames; in response to the intersection ratio being greater than the intersection ratio threshold, obtain the first detection frame. The number of time frames and the number of time frames of the second detection frame; based on the number of time frames of the first detection frame, the number of time frames of the second detection frame and the frame number difference threshold, the target is determined in the first detection frame and the second detection frame Check box.

In this embodiment, any two detection frames at the same time can be selected from multiple detection frames to obtain the first detection frame and the second detection frame, and the difference between the first detection frame and the second detection frame can be determined. The intersection ratio, judges whether the intersection ratio between the first detection frame and the second detection frame is greater than the intersection ratio threshold, and in response to the intersection ratio being greater than the intersection ratio threshold, indicating that the first detection frame and the second detection frame Overlapping, the number of time frames of the first detection frame and the second detection frame can be determined respectively, based on the number of time frames of the first detection frame, the number of time frames of the second detection frame and the frame number difference threshold, The target detection frame is determined in the second detection frame, wherein the intersection ratio can be used to characterize the degree of overlap between detection frames, which can be represented by IoU; the intersection ratio threshold can be a value set according to the actual situation, or can be based on The value obtained by experience can be represented by T; the first detection frame and the second detection frame can be any two detection frames in the same frame.

Optionally, selecting a first detection frame and a second detection frame from a plurality of detection frames, performing an operation on the intersection of the first detection frame and the second detection frame, and judging the area of the convex edge formed by the set of intersection points, Determine the intersection ratio between the two detection frames. If the intersection ratio is greater than the intersection ratio threshold, it means that the two detection frames overlap. Then the detection frames are screened by the number of time frames to determine the time frame of the first detection frame. and the number of time frames of the second detection frame, based on the number of time frames of the first detection frame, the number of time frames of the second detection frame and the frame number difference threshold, determine the target detection frame in the first detection frame and the second detection frame .

Optionally, select a first detection frame and a second detection frame from a plurality of detection frames, perform an intersection operation on the first detection frame and the second detection frame to obtain a corresponding intersection point set, and judge the convexity formed by the intersection point set. The area of the edge is obtained to obtain the intersection ratio between the two detection frames, that is, the intersection ratio can be calculated by the following formula:

IoU=area_r1r2/(area_r1+area_r2-arear1r2)

Wherein, area_r1 and area_r2 are the areas of the first detection frame and the second detection frame, respectively, and area_r1r2 is the area of the convex edge formed by the intersection of the two detection frames.

Optionally, if the IoU is greater than the threshold T, then determine the number of time frames of the first detection frame and the number of time frames of the second detection frame, based on the number of time frames of the first detection frame, the number of time frames of the second detection frame and the frame number of the second detection frame. The number difference threshold is used to determine the target detection frame in the first detection frame and the second detection frame.

As an optional embodiment, the target detection frame is determined from the first detection frame and the second detection frame based on the time frame number of the first detection frame, the time frame number of the second detection frame, and the frame number difference threshold, including : Determine the first frame number difference between the time frame number of the first detection frame and the time frame number of the second detection frame; The detection frame determines the target detection frame.

In this embodiment, the first detection frame and the second detection frame are selected from multiple detection frames, the time frame number of the first detection frame and the time frame number of the second detection frame are determined, and the first detection frame and the second detection frame are determined. The time frame number difference between the frames is obtained to obtain the first frame number difference, and the first frame number difference is judged based on the frame number difference threshold, so as to achieve the purpose of determining the target detection frame from the first detection frame and the second detection frame.

As an optional embodiment, determining the target detection frame from the first detection frame and the second detection frame based on the first frame number difference and the frame number difference threshold includes: in response to the absolute value of the first frame number difference being greater than The frame number difference threshold, and the detection frame with a long time frame number in the first detection frame and the second detection frame is determined as the target detection frame.

In this embodiment, the first detection frame and the second detection frame are selected from multiple detection frames, the time frame number of the first detection frame and the time frame number of the second detection frame are determined, and the first detection frame and the second detection frame are determined. Detect the time frame number difference between the frames, obtain the first frame number difference, and determine whether the absolute value of the first frame number difference is greater than the frame number difference threshold. In response to the absolute value of the first frame number difference being greater than the frame number difference threshold, then A detection frame with a long number of time frames in the first detection frame and the second detection frame is determined as a target detection frame.

Optionally, selecting a first detection frame and a second detection frame from a plurality of detection frames, performing an operation on the intersection of the first detection frame and the second detection frame, and judging the area of the convex edge formed by the set of intersection points, Determine the intersection ratio between the two detection frames. If the intersection ratio is greater than the intersection ratio threshold, it means that the two detection frames overlap, and the detection frames can be screened by the number of time frames; further, determine the first detection frame. The time frame number of the frame and the time frame number of the second detection frame, determine the frame number difference between the two detection frames, and obtain the first frame number difference. If the absolute value of the first frame number difference is greater than the frame number difference threshold, explain The two detection frames not only overlap but also overlap for a long time, and the detection frame with a long number of time frames in the two detection frames is determined as the target detection frame. Optionally, an iterative operation may be performed between multiple detection frames through the above description, so as to achieve the purpose of determining the target detection frame from the multiple detection frames.

As an optional embodiment, determining the target detection frame from the first detection frame and the second detection frame based on the first frame number difference and the frame number difference threshold includes: in response to the difference in the absolute value of the first frame number difference being different. If the frame number difference threshold is greater than the frame number difference threshold, the detection frame with a large target matching degree between the first detection frame and the second detection frame is determined as the target detection frame, wherein the target matching degree is used to represent the matching degree between the corresponding detection frame and the object.

In this embodiment, the first detection frame and the second detection frame are selected from multiple detection frames, the time frame number of the first detection frame and the time frame number of the second detection frame are determined, and the first detection frame and the second detection frame are determined. Detecting the time frame number difference between the frames, obtaining the first frame number difference, judging whether the absolute value of the first frame number difference is greater than the frame number difference threshold, and responding that the absolute value of the first frame number difference is not greater than the frame number difference threshold, Then, the detection frame with a large target matching degree between the first detection frame and the second detection frame is determined as the target detection frame, wherein the target matching degree can be used to represent the matching degree between the corresponding detection frame and the object, which can be the detection frame and the object. The matching mean over a period of time, for example, can be the confidence mean, which can be represented by the score mean.

Optionally, a first detection frame and a second detection frame are selected from multiple detection frames, the intersection of the first detection frame and the second detection frame is calculated, and the area of the convex polygon formed by the set of judging intersection points is determined. The intersection ratio between the two detection frames, if the intersection ratio is greater than the intersection ratio threshold, indicating that the two detection frames overlap, the detection frames can be screened by the number of time frames; further, determine the first detection frame The number of time frames and the number of time frames of the second detection frame, determine the frame number difference between the two detection frames, and obtain the first frame number difference. If the absolute value of the first frame number difference is not greater than the frame number difference threshold, explain Although the two detection frames overlap, but the overlapping time is not long, the detection frame with a large target matching degree between the first detection frame and the second detection frame is determined as the target detection frame. Optionally, an iterative operation is performed between multiple detection frames according to the above description, so as to achieve the purpose of determining the target detection frame from the multiple detection frames.

In the embodiment of the present invention, the overlapping relationship between the detection frames is determined, and the detection frames with low target matching degree are filtered out, so as to filter the corresponding objects in the three-dimensional space, thereby effectively improving the repetition, jumping, and association of the detection objects, resulting in The problem of unstable detection results can improve the fusion performance of multi-sensor three-dimensional space targets and improve the accuracy of object detection.

As an optional embodiment, the matching degree corresponding to each detection frame at each moment in the video is obtained, and at least one matching degree of each detection frame is obtained; the sum of the at least one matching degree of the detection frame is matched with at least one matching degree. The quotient between the number of degrees is determined as the target matching degree of the detection frame.

In this embodiment, the matching degree corresponding to each detection frame at each moment in the video is obtained, at least one matching degree of each detection frame is obtained, all the matching degrees corresponding to each detection frame are added up, and the detection frame corresponding to The quotient between the sum of all matching degrees and the number of matching degrees is determined as the target matching degree of the detection frame, where the matching degree can be used to indicate the matching degree of the detection frame and the object to be recognized in each frame, and score can be used to express.

In the related art, non-maximum suppression is a post-processing module in the target detection framework, which is mainly used to delete highly redundant detection frames. For multiple detection frames, the multiple detection frames of each target are de-redundant through non-maximum suppression, and the final target detection frame is obtained. In general, the detection frame is generally represented by a horizontal rectangular frame. Due to the three-dimensional output of the sensor Objects in space have orientation uncertainty. Therefore, in this embodiment of the present invention, Incline Non-Maximum Suppression (INMS) is adopted for a directional rectangular box (rotate box, rbox).

Optionally, in this embodiment of the present invention, multiple detection frames corresponding to the object may be obtained through multiple sensors, the detection library obtained by the millimeter-wave radar may be filtered by using the speed threshold, and the three-dimensional detection data obtained by the visual sensor and the lidar detection may be filtered. The frame is projected onto the ground to obtain a two-dimensional detection frame, and information such as the center point, size, and orientation of the object is obtained, and the number of frames (frame_cnt) and the matching degree of each frame (score) appearing in each detection frame are determined respectively, and multiple detection frames are obtained. Matching degree, add all the matching degrees corresponding to each detection frame, and determine the quotient between the sum of all matching degrees corresponding to the detection frame and the number of matching degrees as the target matching degree of the detection frame. The match for each frame changes to the target match.

Optionally, the detection frames of each frame can be sorted by the target matching degree to obtain the detection frame sequence (rbox_list), so as to compare the detection frames in the detection frame sequence to obtain the target detection frame, where the detection frames are sorted. It is only to ensure that the detection frames can be processed in an orderly manner so as to avoid omission, and therefore, no specific restrictions are imposed on the sorting method.

In the embodiment of the present invention, the target matching degree of the detection frame is set as the average matching degree of all the appearing frames, which effectively improves the problems of associated target jumping and unstable association results during fusion, thereby improving the results in the multi-sensor object detection process. fusion.

As an optional embodiment, the frame number difference threshold is determined based on historical detection data of object processing.

In this embodiment, selection can be made through multiple trials to achieve the purpose of determining an appropriate frame number difference threshold.

The embodiment of the present invention achieves the purpose of filtering some overlapping objects with a short appearance time by comparing the number of time frames between multiple detection frames, thereby achieving the technical effect of improving the accuracy of target recognition, and solving the problem of target recognition. Low accuracy technical issues.

Example 2

The technical solutions of the embodiments of the present invention are illustrated below with reference to the preferred embodiments.

The core functional modules in the field of autonomous driving can be divided into perception, decision-making and control modules. Among them, the perception module needs to achieve accurate perception of the surrounding environment to provide the basis for analysis for other subsequent modules. Therefore, it is important to improve the accuracy of object detection. At the same time, there are differences in the object attributes and sensing ranges perceived by different sensors. If different sensors of the same type are installed in different positions, the sensing areas will also be different. Therefore, it is necessary to fuse the sensing results of each sensor to obtain information about the sensor. More precise information about surrounding objects.

At present, 3D object detection, as one of the key technologies of the autonomous driving perception module, can provide information such as the position, size, direction and category of objects in 3D space, so as to complete 3D space modeling, path planning, etc. However, due to a single The perception of sensors has certain limitations, and the processing and fusion of multi-sensor (such as lidar, millimeter-wave radar, and vision) data has become the trend of object detection technology in the field of autonomous driving.

In related technologies, sensors detect three-dimensional objects, and there is a problem of overlapping frames, which leads to false detection and missed detection, which has a great impact on the correlation accuracy and fusion robustness of multi-sensor fusion. The overlapping frame problem of , non-maximum suppression is a commonly used detection filtering method, which is used to extract object detection boxes with high confidence and suppress false detection boxes with low confidence. Among them, the standard non-maximum suppression applies For the detection of horizontal two-dimensional rectangular boxes, smooth non-maximum suppression (soft NMS) and adaptive non-maximum suppression (adaptive NMS) can be used to optimize non-maximum suppression, which is prone to leaks when objects are dense and occluded. inspection problem.

In order to solve the above problem, an embodiment of the present invention proposes a detection method based on Incline Non-Maximum Suppression (INMS for short), which is based on an inclined frame and compares the currently traversed detection frame with the remaining other detection frames. The intersection operation is performed on the frame to obtain the corresponding intersection point set, and the intersection ratio of each two detection frames is calculated according to the area of the convex polygon composed of the intersection point set. However, most of this method is based on single-frame observation information for detection and filtering. The effect of the remaining timestamp information on the perception result is ignored, so there is a problem of low accuracy of object detection.

In a related art, a sensor data filtering and fusion method for automatic driving is provided. The method is divided into a spatial filtering and fusion method and a subsequent temporal filtering and fusion method. The spatial filtering and fusion method is used in the perception system of automatic driving. After obtaining a frame of raw data from the sensor, perform spatial clustering processing on the data points, identify valid clustering results and eliminate noise; perform correlation tracking on the clustering results, so as to obtain the position and speed information and historical information of an object. , prediction information; then estimate the feature information, calculate the length, width and orientation information of the object after clustering; then use the time filtering fusion method to distinguish the original target as a definite target, a suspicious target and a pseudo target. This method does not need to consider the type of sensor and the number of sensors. Although it can be used for any sensor data with large or small noise, the calculation amount is small, simple, flexible and effective. However, this method is mostly based on single-frame observation information for detection and filtering. Ignoring the effect of the remaining timestamp information on the perception results, there is still the problem of low accuracy of object detection.

In another related art, a non-maximum suppression method for multi-target detection is provided, the method obtains the category and confidence score of each detected target, and sorts each target according to the confidence score to obtain a detection queue; Obtain the target with the highest confidence score in the detection queue, record it as target A, and judge whether the confidence score of target A satisfies the category preset condition; if the confidence score of target A satisfies the category preset condition of its category, judge the detection frame of target A Whether there is overlap with the detection frame of other targets; if it is determined that the detection frame of target A and the detection frame of other targets overlap, the target whose detection frame overlaps with the detection frame of target A is marked as target B, and the target A and the target are judged to overlap. Whether B is in the same category, and according to the judgment result, the corresponding suppression algorithm is used to judge whether the target B is suppressed. Since most of the methods are also based on single-frame observation information for detection and filtering, the effect of other timestamp information on the perception result is ignored, so There is still a problem of low accuracy in detecting objects.

In another related technology, a 3D target detection method based on multi-information fusion is proposed. The method collects driving images and point cloud information respectively, and performs preprocessing. Based on the driving images, a 2D target detector is used to obtain 2D targets. frame and its score; based on the point cloud information, use the 3D target detector to obtain the candidate 3D target frame; use the correspondence between the 2D target frame and the candidate 3D target frame, as well as the 2D target frame score, filter the candidate 3D target frame, get 3D target frame, that is to say, this method makes up for the shortage of sparse point cloud by introducing visual information, and constrains the 3D target frame through the 2D detection result, which improves the recall rate of the 3D target frame and reduces false detection and leakage. However, this method does not consider the fusion of multi-sensor data, so there is still a technical problem of low accuracy of object detection.

In order to solve the above problems, the embodiment of the present invention proposes a multi-sensor 3D target detection filtering and fusion method based on INMS, aiming at the problem that overlapping frames are prone to reduce the fusion correlation accuracy in the process of multi-sensor 3D target detection. Including the detection frame obtained from the lidar, millimeter wave radar and vision sensor, the obtained detection frame is preprocessed and filtered, and the detection results of the vision sensor and the millimeter wave radar are processed according to a certain score threshold and speed threshold respectively. Filtering, in which the score threshold and speed threshold can be values set in advance according to the actual situation or experience; then project the detected 3D rbox to the ground to obtain a 2D rbox, so as to obtain the center point, size, and orientation information of the detection frame; In the technology, most of the detection and filtering are based on single-frame observation information, ignoring the effect of the remaining timestamp information on the perception result. In the INMS, the processing of a single detection frame introduces the information of the remaining timestamps, which is set according to the duration of the detection frame. Time threshold, filter some overlapping targets with a short appearance time, and set the matching degree (score) corresponding to the detection frame to the average matching degree of all the appearing frames, which can effectively improve the detection target overlapping, jumping, and unstable association results. to improve the fusion performance of multi-sensor object detection.

The embodiments of the present invention will be further introduced below.

In the embodiment of the present invention, it is mainly divided into two aspects: acquisition and processing of detection frames.

FIG. 2 is a flowchart of a method for acquiring a detection frame according to an embodiment of the present invention. As shown in FIG. 2 , the acquisition of a detection frame in an embodiment of the present invention may include the following steps:

Step S201, acquiring data of each sensor.

In this embodiment, the multi-sensor may include a lidar, a millimeter-wave radar (radar), and a vision sensor (camera), and detection data is acquired by using the lidar, the millimeter-wave radar, and the vision sensor.

Step S202, filtering the acquired detection data based on the score threshold or the speed threshold.

In this embodiment, preprocessing filtering is performed on the acquired target.

Optionally, the three-dimensional detection result (3Dbox) obtained by the lidar can be projected onto the ground to obtain a two-dimensional detection frame (2D rbox), and information such as the center point, size, and orientation information of the rbox can be obtained.

Optionally, the detection results obtained by the camera can be filtered based on the score threshold. For example, the three-dimensional detection frame can be filtered through the score threshold (3D rbox), and the filtered detection results can be projected onto the plane to obtain the corresponding plane detection. frame, to obtain information such as the center point, size, and orientation information of the detection frame, where the score threshold can be an empirical value, or a value obtained according to the actual situation or experimental inspection. There is no specific restriction on the determination of the score threshold here.

Optionally, for the detection data acquired by radar, the acquired detection results may be filtered through a speed threshold to obtain detection frames that need to be processed.

Step S203: Suppress the non-maximum value of the input inclination of the data after screening by all the sensors, and process the data.

Input the filtered detection box into the improved tilt non-maximum suppression, and use the improved tilt non-maximum suppression to process the data.

In the related art, non-maximum suppression is a post-processing module in the target detection framework, which can be used to delete highly redundant detection frames. FIG. 3 is a schematic diagram of a non-maximum suppression identification process according to the related art, As shown in Figure 3, for object A and object B, multiple detection frames will be generated during detection. The essence of non-maximum suppression is to remove redundancy from multiple detection frames of each object to obtain the final detection result, which can be In order to obtain the target detection frame with the maximum matching degree, the horizontal rectangular frame is generally used for target detection, and the three-dimensional target output by the sensor has direction uncertainty. At the same time, the information of the remaining time stamps is introduced for a single target in the slope non-maximum value suppression, and the time threshold is set according to its appearance time, and some overlapping targets with a short appearance time are filtered, and the target score is set to all its occurrences Frame score mean, FIG. 4( a ) is a schematic diagram of a processing result without non-maximum suppression according to an embodiment of the present invention, and FIG. 4( b ) is a non-maximum processed result according to an embodiment of the present invention. A schematic diagram of the processing result of value suppression, as shown in Fig. 4(a) and Fig. 4(b), the embodiment of the present invention can improve the problems of the jump of the associated target and the instability of the association result in the fusion, and at the same time, use the time frame The number of overlapping parts is filtered out, which effectively improves the fusion performance of multi-sensor 3D targets.

FIG. 5 is a flowchart of a processing method for non-maximum value suppression according to an embodiment of the present invention. As shown in FIG. 5 , the processing of the INSM proposed by the embodiment of the present invention may include the following steps:

Step S501, processing the detection frame filtered by each sensor.

Optionally, obtain the time frame number (frame_cnt) of each detection frame and the confidence of each frame, determine the average confidence of each detection frame, and change the score of each frame of the detection frame to the corresponding confidence average.

Optionally, sort all detection boxes under the same frame according to the confidence threshold to obtain a sorting table (rbox_list). It should be noted that the sorting can be in ascending or descending order. The main purpose of sorting is to ensure that all detections can be Frames are processed to prevent omission of detection frames during processing. Therefore, the sorting method is not specifically described here, as long as the method for obtaining the sorting order should be within the protection scope of the embodiments of the present invention.

In step S502, the detection frame is screened by using the intersection ratio threshold.

Optionally, starting from the first detection frame in the rbox_list, determine the intersection ratio between each detection frame in turn. The intersection operation can be performed on the detection frame and other detection frames to obtain a corresponding set of intersection points. According to the judgment The area of the convex polygon formed by the set of intersection points, so as to determine the intersection ratio between each two detection frames, can be determined by the following formula:

IoU=area_r1r2/(area_r1+area_r2-area1r2),

Among them, area_r1 and area_r2 are the areas of the two detection frames respectively, and area_r1r2 is the area of the convex edge formed by the intersection of the two detection frames.

Optionally, if the intersection ratio is greater than the intersection ratio threshold (T), step S503 is performed, and if the intersection ratio is not greater than the intersection ratio threshold (T), the detection frame is retained.

Step S503: Calculate the frame number difference between the detection frames, and filter the overlapping detection frames by using the frame number difference threshold.

Optionally, if the intersection ratio is greater than the intersection ratio threshold, it means that there is a possibility of overlapping between the two detection frames, then the difference in the number of frames between the two detection frames is judged, and the number of frames between the detection frames is calculated. Difference.

Optionally, determine the frame number difference threshold and the size of the frame number difference between the detection frames, if the frame number difference between the detection frames is not greater than the frame number difference threshold (time_threshold), then retain the detection frame with a larger degree of matching, and remove it. A detection frame with a small matching degree; if the frame number difference is greater than the frame number difference threshold, the detection frame with a relatively long corresponding time frame number is retained, and the detection frame with a relatively short corresponding time frame number is removed.

In the related art, by detecting images with a single time frame number, the influence of other time frame information on the perception result in a period of time is ignored. The number of time frames that appear sets a time threshold, thereby filtering out overlapping objects that appear for a short time, thereby improving the accuracy of object recognition.

Optionally, the multiple detection boxes are processed in steps S502 and S504, and the iterative operation is performed until all detection boxes in the rbox_list are screened.

Step S504, acquiring the filtered target, and performing multi-sensor data fusion.

Obtain the filtered target, perform multi-sensor data fusion, and process the detection frame that has been filtered many times to obtain the object to be recognized.

In the related art, the NMS algorithm can only realize the detection of a two-dimensional horizontal rectangular frame, which has limitations on the filtering of spatial objects in the scene. The embodiment of the present invention proposes an INMS-based detection algorithm to realize a multi-sensor three-dimensional space detection algorithm. A target detection filtering method, in which the three-dimensional detection frame generated by each target is projected onto the ground to generate a corresponding two-dimensional detection frame, the detection frame with low matching degree is filtered according to the judgment of the overlapping relationship of the detection frame, and the corresponding overlapping objects are filtered correspondingly , which can effectively improve the overlapping, jumping, and unstable correlation results of detected objects, so as to improve the fusion performance of multi-sensor spatial objects.

Further, in the related art, detection and filtering are performed based on the observation information of a single frame, and the effect of the remaining timestamp information on the perception result is ignored. The embodiment of the present invention improves the existing INMS, and introduces a single target For the information of the remaining time stamps, a time threshold is set according to the duration of its occurrence, some overlapping targets with a short duration of occurrence are filtered, and the matching degree of the detection frame is set to the average matching degree of all the occurrence frames, thereby improving the stability of filtering.

In the embodiment of the present invention, by comparing the number of time frames between multiple detection frames, the purpose of filtering some overlapping objects with a short appearance time is achieved, thereby achieving the technical effect of improving the accuracy of target recognition, and solving the problem of The technical problem of low accuracy of target recognition.

Example 3

According to an embodiment of the present invention, an object identification device is also provided. It should be noted that the object identification device can be used to execute the object identification method in Embodiment 1.

FIG. 6 is a schematic diagram of a device for recognizing an object according to an embodiment of the present invention. As shown in FIG. 6 , the object identification apparatus 600 may include: an acquisition unit 602 , a first determination unit 604 , a second determination unit 606 and an identification unit 608 .

The obtaining unit 602 is configured to obtain multiple detection frames of the object in the video, wherein each detection frame is used to represent the position of the object in the video.

The first determining unit 604 is configured to respectively determine the number of time frames of the multiple detection frames in the video, where the number of time frames is used to represent the time length that each detection frame appears continuously in the video.

The second determining unit 606 is configured to determine a target detection frame in a plurality of detection frames based on the time frame number and the frame number difference threshold.

The identifying unit 608 is configured to identify the object in the video based on the object detection frame.

Optionally, the second determination unit 606 includes: a first determination module, configured to determine the intersection ratio of the first detection frame in the multiple detection frames and the second detection frame in the multiple detection frames, wherein the first detection frame and The second detection frame is any two detection frames at the same moment in multiple detection frames; in response to the intersection ratio being greater than the intersection ratio threshold, the time frame number of the first detection frame and the time frame number of the second detection frame are obtained; based on The time frame number of the first detection frame, the time frame number of the second detection frame, and the frame number difference threshold are used to determine the target detection frame in the first detection frame and the second detection frame.

Optionally, the first determination module includes: a first determination submodule for determining the first frame number difference between the time frame number of the first detection frame and the time frame number of the second detection frame; The frame number difference and the frame number difference threshold are used to determine the target detection frame from the first detection frame and the second detection frame.

Optionally, the first determination submodule is also used to determine the detection frame with a long number of time frames in the first detection frame and the second detection frame as the target detection in response to the absolute value of the first frame number difference being greater than the frame number difference threshold. frame.

Optionally, the first determination sub-module is also used to determine the detection frame with a large degree of matching between the first detection frame and the second detection frame as the target detection in response to the absolute value of the first frame number difference being not greater than the frame number difference threshold. box, where the matching degree is used to characterize the matching degree between the corresponding detection frame and the object.

Optionally, the device further includes: a third determining unit, configured to obtain the matching degree corresponding to each detection frame at each moment in the video, and obtain at least one matching degree of each detection frame; The quotient between the sum and the number of at least one matching degree is determined as the target matching degree of the detection frame.

Optionally, the apparatus further includes: a fourth determining unit, configured to determine the frame number difference threshold based on historical detection data processed on the object.

In the embodiment of the present invention, the acquisition unit acquires multiple detection frames of the object in the video, wherein each detection frame is used to represent the position of the object in the video; The number of time frames in the video, where the number of time frames is used to characterize the time length that each detection frame appears continuously in the video; through the second determination unit, based on the number of time frames and the frame number difference threshold, in multiple detection frames Determine the target detection frame; through the recognition unit, identify the object in the video based on the target detection frame. That is to say, the embodiment of the present invention achieves the purpose of filtering some overlapping objects with a short appearance time by comparing the number of time frames between multiple detection frames, thereby achieving the technical effect of improving the accuracy of target recognition, and solving the problem of The technical problem of low accuracy of target recognition is solved.

Example 4

According to an embodiment of the present invention, a computer-readable storage medium is further provided, and the storage medium includes a stored program, wherein the program executes the object identification method described in Embodiment 1.

Example 5

According to an embodiment of the present invention, there is also provided a processor for running a program, wherein the method for identifying an object described in Embodiment 1 is executed when the program is run.

Example 6

According to an embodiment of the present invention, there is also provided a vehicle for running a program, wherein the method for recognizing an object described in Embodiment 1 is executed when the program is running.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are only illustrative, for example, the division of the units may be a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. It should be regarded as the protection scope of the present invention.

Claims (10)

1. a recognition method of an object, is characterized in that, comprises: Acquiring multiple detection frames of an object in the video, wherein each of the detection frames is used to represent the position of the object in the video; Determine the number of time frames of a plurality of the detection frames in the video respectively, wherein the number of time frames is used to represent the time length that each of the detection frames continuously appears in the video; determining target detection frames in a plurality of the detection frames based on the time frame number and the frame number difference threshold; The object is identified in the video based on the object detection frame. 2. The method according to claim 1, wherein, based on the time frame number and the frame number difference threshold, determining a target detection frame in a plurality of the detection frames, comprising: Determine the intersection ratio of the first detection frame in the plurality of detection frames and the second detection frame in the plurality of detection frames, wherein the first detection frame and the second detection frame are a plurality of the detection frames Any two detection frames at the same moment in the frame; In response to the cross-union ratio being greater than the cross-union ratio threshold, acquiring the time frame number of the first detection frame and the time frame number of the second detection frame; determining the first detection frame and the second detection frame based on the time frame number of the first detection frame, the time frame number of the second detection frame, and the frame number difference threshold Object detection box. 3. The method according to claim 2, characterized in that, based on the time frame number of the first detection frame, the time frame number of the second detection frame and the frame number difference threshold, from the first detection frame The detection frame and the second detection frame determine the target detection frame, including: Determine the first frame number difference between the time frame number of the first detection frame and the time frame number of the second detection frame; The target detection frame is determined from the first detection frame and the second detection frame based on the first frame number difference and the frame number difference threshold. 4. The method according to claim 3, wherein a target detection frame is determined from the first detection frame and the second detection frame based on the first frame number difference and the frame number difference threshold ,include: In response to the absolute value of the first frame number difference being greater than the frame number difference threshold, determining a detection frame with a long time frame number in the first detection frame and the second detection frame as the target detection frame. 5. The method according to claim 3, wherein a target detection frame is determined from the first detection frame and the second detection frame based on the first frame number difference and the frame number difference threshold ,include: In response to the absolute value of the first frame number difference not being greater than the frame number difference threshold, determining a detection frame with a large target matching degree between the first detection frame and the second detection frame as the target detection frame , wherein the target matching degree is used to represent the matching degree between the corresponding detection frame and the object. 6. The method of claim 1, wherein the method further comprises: Obtain the degree of matching corresponding to each of the detection frames at each moment in the video, and obtain at least one degree of matching of each of the detection frames; The quotient between the sum of the at least one matching degree of the detection frame and the quantity of the at least one matching degree is determined as the target matching degree of the detection frame. 7. The method of claim 1, wherein the method further comprises: The frame number difference threshold is determined based on historical detection data processed on the object. 8. A method for identifying an object, comprising: an obtaining unit, configured to obtain a plurality of detection frames of an object in a video, wherein each of the detection frames is used to represent the position of the object in the video; a first determining unit, configured to respectively determine the number of time frames of a plurality of the detection frames in the video, wherein the number of time frames is used to represent the time when each of the detection frames appears continuously in the video length; a second determining unit, configured to determine target detection frames in a plurality of the detection frames based on the time frame number and the frame number difference threshold; An identification unit, configured to identify the object in the video based on the object detection frame. 9. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored program, wherein, when the program is run, a device where the computer-readable storage medium is located is controlled to execute claims 1 to 7 any of the methods described in . 10. A vehicle for carrying out the method of any one of claims 1 to 7.

Images