CN116912788A - Attack detection method, device and equipment for automatic driving system and storage medium - Google Patents

Abstract

The invention discloses an attack detection method, device, equipment and storage medium for an automatic driving system. The method includes performing target detection on three-dimensional information data and two-dimensional information data obtained by three-dimensional sensors and two-dimensional sensors respectively, and obtains the first three-dimensional information data. detection frame, the first two-dimensional detection frame, and perform fusion target detection on the three-dimensional information data and the two-dimensional information data to obtain the second three-dimensional detection frame, and perform coordinate transformation on the second three-dimensional detection frame to obtain the second two-dimensional detection frame , then perform IoU numerical calculations on the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each object to obtain the first array, and perform IoU on the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each object. Numerical calculation is performed to obtain the second array, and the first array and the second array are analyzed to confirm whether the sensor has been attacked. The invention can detect attacks based on multi-modal data and locate the attacked sensor.

Description

Attack detection methods, devices, equipment and storage media for autonomous driving systems

Technical field

This application relates to the field of sensor detection technology, and in particular to an attack detection method, device, equipment and storage medium for an autonomous driving system.

Background technique

Autonomous driving technology is a comprehensive technology involving many cutting-edge technologies such as artificial intelligence, sensing technology, map technology, and computers. The function realization of driverless technology relies on the vehicle-mounted positioning sensor system. The positioning sensor system mainly consists of laser radar, visual camera, millimeter wave radar, global positioning system (GPS) and other equipment. The positioning sensor system provides rich positioning data for the planning and decision-making module of autonomous vehicles, that is, vehicle speed, attitude, position and other information. The safety of route planning and decision-making control of autonomous vehicles is based on the safety of the positioning sensor system. If the positioning sensor system of an autonomous vehicle is abnormal, it will cause the sensor to obtain wrong positioning information, and then plan the wrong driving control strategy. It poses a threat to the safety of other vehicles and the lives of drivers and pedestrians.

At present, the sensor attack detection method of unmanned driving technology is mainly combined with a specific single-sensor target detection algorithm, and attack detection is performed on this basis. However, this method is not universal. For example, lidar is used as the The attack detection method combined with the main target detection algorithm cannot be applied to the attack detection scenario of the camera-based target detection algorithm, and the accuracy of the attack detection results for a single type of sensor detection method is not high.

Contents of the invention

In view of this, this application provides an attack detection method, device, equipment and storage medium for an autonomous driving system to solve the problem that existing attack detection methods for autonomous driving systems are not universal and have low accuracy.

In order to solve the above technical problems, one technical solution adopted by this application is to provide an attack detection method for the automatic driving system, which includes: using three-dimensional sensors and two-dimensional sensors to respectively obtain three-dimensional information data and two-dimensional information data of the target area. ; Input the three-dimensional information data and the two-dimensional information data into the pre-trained three-dimensional detection model and the two-dimensional detection model respectively to obtain the first three-dimensional detection frame and the first two-dimensional detection frame of each target object in the target area; Information data and two-dimensional information data are input into the pre-trained fusion detection model to obtain the second three-dimensional detection frame of each target object in the target area, and the coordinate system of the second three-dimensional detection frame is changed to obtain the second two-dimensional detection frame. Frame; calculate the first IoU value of the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object, obtain the first array, and calculate the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object. The second IoU value of the detection frame is obtained to obtain the second array; based on the first array and the second array, attack detection is performed and the attacked sensor is located.

As a further improvement of this application, before inputting the three-dimensional information data and the two-dimensional information data into the pre-trained fusion detection model, it also includes: performing a chi-square comparison between the two-dimensional information data at the previous moment and the two-dimensional information data at the current moment. , obtain the chi-square comparison value; determine whether the chi-square comparison value exceeds the preset threshold; when the chi-square comparison value exceeds the preset threshold, perform histogram matching between the two-dimensional information data at the current moment and the two-dimensional information data at the previous moment. , obtain the enhanced two-dimensional information data at the current moment, and replace the two-dimensional information data at the current moment; when the chi-square comparison value does not exceed the preset threshold, the two-dimensional information data at the current moment is maintained.

As a further improvement of this application, changing the coordinate system of the second three-dimensional detection frame to obtain the second two-dimensional detection frame includes: obtaining the three-dimensional center point coordinates of the second three-dimensional detection frame; using the three-dimensional center point coordinates to confirm the second three-dimensional detection Homogeneous coordinates of four points on the frame on the same plane as the three-dimensional center point; four two-dimensional coordinate points are calculated using the pre-obtained camera projection matrix, camera rotation matrix, sensor-to-camera coordinate system rotation matrix and homogeneous coordinates ; Use four two-dimensional coordinate points to construct a second two-dimensional detection frame.

As a further improvement of this application, the first IoU value of the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object is calculated to obtain the first array, including: confirming the first target of each target object corresponding to The two-dimensional detection frame and the second two-dimensional detection frame of the target; respectively calculate the first area and the second area of the first two-dimensional detection frame of the target and the second two-dimensional detection frame of the target, as well as the relationship between the first two-dimensional detection frame of the target and the third two-dimensional detection frame of the target. The third area of the overlapping area of the two-dimensional detection frames; use the first area, the second area, and the third area to calculate the first IoU value corresponding to each target object; use the first IoU values corresponding to all target objects to construct the third area an array.

As a further improvement of this application, the second IoU value of the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object is calculated to obtain a second array, including: confirming the first three-dimensional detection of the target corresponding to each target object frame and the target second three-dimensional detection frame; calculate the first volume and second volume of the target first three-dimensional detection frame and the target second three-dimensional detection frame respectively; calculate the overlapping area of the target first three-dimensional detection frame and the target second three-dimensional detection frame The base area of Calculate the second IoU value corresponding to each target object; construct a second array using the second IoU values corresponding to all target objects.

As a further improvement of this application, performing attack detection and locating the attacked sensor based on the first array and the second array includes: respectively determining whether there are outliers in the first array and the second array; when there are outliers in the first array , when there are no outliers in the second array, it is confirmed that the three-dimensional sensor is attacked; when there are no outliers in the first array, and there are outliers in the second array, it is confirmed that the two-dimensional sensor is attacked; when the first array, the second array When there are outliers in both arrays, it is confirmed that the three-dimensional sensor and/or the two-dimensional sensor has been attacked; when there are no outliers in the first array and the second array, it is confirmed that the three-dimensional sensor and the two-dimensional sensor have not been attacked.

As a further improvement of this application, the three-dimensional sensor includes lidar, the two-dimensional sensor includes a camera, the three-dimensional detection model is built based on the PointPillar algorithm, the two-dimensional detection model is built based on the YOLOv3 algorithm, and the fusion detection model is built based on the AVOD algorithm.

In order to solve the above technical problems, another technical solution adopted by this application is to provide an attack detection device for an automatic driving system, which includes: an acquisition module for respectively acquiring three-dimensional information of the target area using three-dimensional sensors and two-dimensional sensors. data, two-dimensional information data; the first detection module is used to input the three-dimensional information data and the two-dimensional information data into the pre-trained three-dimensional detection model and the two-dimensional detection model respectively, and obtain the first image of each target object in the target area. The three-dimensional detection frame, the first two-dimensional detection frame; the second detection module is used to input the three-dimensional information data and the two-dimensional information data into the pre-trained fusion detection model to obtain the second three-dimensional detection of each target object in the target area. frame, and changes the coordinate system of the second three-dimensional detection frame to obtain the second two-dimensional detection frame; the calculation module is used to calculate the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object. IoU value, obtain the first array, and calculate the second IoU value of the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object, obtain the second array; the analysis module is used to calculate the first array and the second array based on the first three-dimensional detection frame and the second three-dimensional detection frame. The array performs attack detection and locates the attacked sensor.

In order to solve the above technical problems, another technical solution adopted by this application is to provide a computer device. The computer device includes a processor and a memory coupled to the processor. Program instructions are stored in the memory. When the program instructions are executed by the processor, the processor is caused to execute the steps of the attack detection method for the automatic driving system as described in any one of the above.

In order to solve the above technical problems, another technical solution adopted by this application is to provide a storage medium that stores program instructions that can implement any of the above attack detection methods for the automatic driving system.

The beneficial effects of this application are: the attack detection method of the automatic driving system of this application performs target detection on the three-dimensional information data and two-dimensional information data obtained by the three-dimensional sensor and the two-dimensional sensor respectively, and obtains the first three-dimensional detection frame, the first and second 3D detection frame, and perform fusion target detection on 3D information data and 2D information data to obtain a second 3D detection frame, perform coordinate transformation on the second 3D detection frame to obtain a second 2D detection frame, and then detect each object Perform IoU numerical calculation on the corresponding first two-dimensional detection frame and second two-dimensional detection frame to obtain the first array. Perform IoU numerical calculation on the first three-dimensional detection frame and second three-dimensional detection frame corresponding to each object to obtain the second array. Array, analyze the first array and the second array to confirm whether the sensor has been attacked. It uses the correlation of the three-dimensional sensor and the two-dimensional sensor in time and space to detect the attack, thereby improving the accuracy of the attack detection, and using it as an automatic The basic configuration of three-dimensional sensors and two-dimensional sensors used in driving cars can realize attack detection. It is no longer limited to using one of the sensor-based target detection algorithms for attack detection, and has higher universality.

Description of the drawings

Figure 1 is a schematic flow chart of an attack detection method for an autonomous driving system according to an embodiment of the present invention;

Figure 2 is a schematic diagram of a second three-dimensional detection frame according to an embodiment of the present invention;

Figure 3 is a schematic diagram of the overlapping area of two-dimensional detection frames according to the embodiment of the present invention;

Figure 4 is a functional module schematic diagram of the attack detection device of the automatic driving system according to the embodiment of the present invention;

Figure 5 is a schematic structural diagram of a computer device according to an embodiment of the present invention;

Figure 6 is a schematic structural diagram of a storage medium according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

The terms “first”, “second” and “third” in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Thus, features defined as "first", "second", and "third" may explicitly or implicitly include at least one of these features. In the description of this application, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically limited. All directional indications (such as up, down, left, right, front, back...) in the embodiments of this application are only used to explain the relative positional relationship between components in a specific posture (as shown in the drawings). , sports conditions, etc., if the specific posture changes, the directional indication will also change accordingly. Furthermore, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes Other steps or units inherent to such processes, methods, products or devices.

Reference herein to "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those skilled in the art understand, both explicitly and implicitly, that the embodiments described herein may be combined with other embodiments.

Figure 1 is a schematic flowchart of an attack detection method for an autonomous driving system according to an embodiment of the present invention. It should be noted that, if substantially the same results are obtained, the method of the present invention is not limited to the process sequence shown in Figure 1 . As shown in Figure 1, the attack detection method of the autonomous driving system includes the following steps:

Step S101: Use three-dimensional sensors and two-dimensional sensors to obtain three-dimensional information data and two-dimensional information data of the target area respectively.

In this embodiment, the three-dimensional sensor is preferably a lidar. It should be noted that the three-dimensional sensor in the embodiment of the present invention is not limited to lidar. Other sensor devices that can be used to obtain three-dimensional information data are also included in the protection scope of the present invention. . The two-dimensional sensor is preferably a camera. It should be noted that the two-dimensional sensor in the embodiment of the present invention is not limited to the camera. Other sensor devices that can be used to obtain two-dimensional information data are also included in the protection scope of the present invention.

Specifically, in this embodiment, when using the autonomous driving system to drive a vehicle, the three-dimensional sensor and the two-dimensional sensor equipped on the vehicle are used to simultaneously collect the three-dimensional information data and the two-dimensional information data of the target area. It should be noted that when performing attack detection, the detection range is limited to a target area, which is the overlapping field of view of the three-dimensional sensor and the two-dimensional sensor. For example, for lidar and cameras, the overlap The area is the FOV (Field of view, FOV) field of view of the lidar. By projecting the image size captured by the camera into the visual area of the lidar, the three-dimensional information data collected by the lidar and the two-dimensional information data collected by the camera are obtained. .

Step S102: Input the three-dimensional information data and the two-dimensional information data into the pre-trained three-dimensional detection model and the two-dimensional detection model respectively, and obtain the first three-dimensional detection frame and the first two-dimensional detection frame of each target object in the target area.

Among them, the three-dimensional detection model is built based on the PointPillar algorithm. Specifically, the three-dimensional sensor uses the PointPillars (3D Object Proposal Generation and Detection from Point Cloud) algorithm to perform three-dimensional target detection to obtain the first three-dimensional target detection frame. The input data is the original data point cloud data of the three-dimensional sensor, that is, the three-dimensional information data, and the format is [ x,y,z,intensity]. The original features of the three-dimensional sensor are point cloud collections, which can be expressed as vectors (64-line lidar, 1800 point clouds scanned per line) where X represents the abscissa of the point cloud on the horizontal plane, Y represents the ordinate of the point cloud on the horizontal plane, Z represents the height of the point cloud, and intensity represents the reflection intensity of the point cloud. . Using the two-stage method, using PointNet++ as the backbone network, first complete the segmentation task, determine the label of each three-dimensional point, and use feature generation boxes for each point divided into the foreground. Then optimize the box. Target object detection result vector/> express. Among them, x _lidar = [X _v Y _v Z _v ] ^T represents the center point value of the target object on the three-dimensional sensor coordinate system, and [L, W, H] ^T represents the length, width and height of the first three-dimensional detection frame.

Among them, the two-dimensional detection model is built based on the YOLOv3 algorithm. Specifically, the YOLOv3 algorithm is used to perform two-dimensional target detection on the two-dimensional information data obtained by the two-dimensional sensor to obtain the target center point. For example, taking a camera as an example, the collected pictures are input into a two-dimensional detection model in RGB format. The two-dimensional detection model calls a deep convolutional neural network algorithm to extract features of the RGB image. Common structures in neural network algorithms include convolutional layers. , pooling layer, activation layer, dropout layer, BN (batch normalization) layer, fully connected layer, etc. The final features extracted from the picture can effectively describe the information of the target object. The input data format is the RGB image of the camera, and the output is the center point coordinates on the image coordinate system and the detection frame width (width) and height (height), the format is Among them, X _camera represents the value in the x direction on the image coordinate system, Y _camera represents the value in the y direction, w represents the width of the first two-dimensional detection frame, and h represents the height of the first two-dimensional detection frame. After obtaining the first two-dimensional detection frame, the coordinates of the four points of the first two-dimensional detection frame can be further obtained as follows:

X _{camera, 1} , = X _{camera, 3} , = X _camera -w/2;

X _{camera, 2} ,=X _{camera, 4} ,=X _camera +w/2;

Y _{camera, 1} , = Y _{camera, 2} , = Y _camera + h/2;

Y _{camera, 3} , = Y _{camera, 4} , = Y _camera -h/2;

Among them, X _{camera, i} , i=1, 2, 3, and 4 respectively represent the x-direction coordinate values of the upper left corner, upper right corner, lower left corner, and lower right corner of the first two-dimensional detection frame of the image coordinate system. Y _{camera, i} , i=1, 2, 3, 4 respectively represent the coordinate values in the y direction of the image coordinate system in the upper left corner, upper right corner, lower left corner, and lower right corner of the first two-dimensional detection frame.

Step S103: Input the three-dimensional information data and the two-dimensional information data into the pre-trained fusion detection model to obtain the second three-dimensional detection frame of each target object in the target area, and change the coordinate system of the second three-dimensional detection frame to obtain The second two-dimensional detection frame.

Specifically, the fusion detection model is built based on the AVOD algorithm (Joint 3D Proposal Generation and Object Detection from View Aggregation). After obtaining the three-dimensional information data and the two-dimensional information data, the three-dimensional information data and the two-dimensional information data are input to the AVOD algorithm for target detection. , get the second three-dimensional detection frame. The AVOD algorithm combines three-dimensional information data and two-dimensional information data. When the three-dimensional information data is point cloud data obtained by lidar, the AVOD algorithm only uses the top view and front view of the point cloud, which can not only reduce the amount of calculation, but also eliminate the need for As for losing too much information. Then a three-dimensional candidate region is generated, and the final second three-dimensional detection frame is output after fusing the features and the candidate region.

Further, in order to improve the defense capability of the two-dimensional sensor, in some embodiments, before step S103, it also includes:

1. Perform a chi-square comparison between the two-dimensional information data at the previous moment and the two-dimensional information data at the current moment to obtain the chi-square comparison value.

2. Determine whether the chi-square comparison value exceeds the preset threshold.

3. When the chi-square comparison value exceeds the preset threshold, perform histogram matching between the two-dimensional information data at the current moment and the two-dimensional information data at the previous moment to obtain the enhanced two-dimensional information data at the current moment and replace the current moment. two-dimensional information data.

4. When the chi-square comparison value does not exceed the preset threshold, the two-dimensional information data at the current moment is maintained.

In this embodiment, the purpose of image enhancement is to improve the defense capability of the fusion detection model against two-dimensional sensor attacks. By performing a chi-square comparison between the two-dimensional information data at the previous moment and the two-dimensional information data at the current moment, the chi-square comparison value is obtained, and then it is determined whether the chi-square comparison value exceeds the preset threshold. When the chi-square comparison value exceeds the preset threshold, perform histogram matching between the two-dimensional information data at the current moment and the two-dimensional information data at the previous moment to obtain the enhanced two-dimensional information data at the current moment and replace the two-dimensional information data at the current moment. dimensional information data. When the chi-square comparison value does not exceed the preset threshold, the two-dimensional information data at the current moment is maintained.

Further, in step S103, the coordinate system of the second three-dimensional detection frame is changed to obtain the second two-dimensional detection frame, which specifically includes:

1. Obtain the three-dimensional center point coordinates of the second three-dimensional detection frame.

2. Use the coordinates of the three-dimensional center point to confirm the homogeneous coordinates of the four points on the second three-dimensional detection frame that are on the same plane as the three-dimensional center point.

3. Calculate four two-dimensional coordinate points using the pre-acquired camera projection matrix, camera rotation matrix, sensor-to-camera coordinate system rotation matrix and homogeneous coordinates.

4. Use four two-dimensional coordinate points to construct a second two-dimensional detection frame.

Specifically, in this embodiment, 4 corner points (including only x coordinates and y coordinates) and 2 heights are used to describe a second three-dimensional detection frame. As shown in Figure 2, the second three-dimensional detection frame is represented by a vector represents, where c _i = [x _ci y _ci ] ^T , i=1, 2, 3, 4 represents the coordinates of the four vertices of the second three-dimensional detection frame in the x and y directions, h ₁ represents the distance from the bottom of the second three-dimensional detection frame The height of the plane formed by the x-axis and the y-axis in the three-dimensional coordinate system, h ₂ represents the height of the top surface of the second three-dimensional detection frame from the plane formed by the x-axis and the y-axis in the three-dimensional coordinate system.

Specifically, after obtaining the second three-dimensional detection frame, the second three-dimensional detection frame is projected to the image coordinate system to obtain the second two-dimensional detection frame. Please refer to Figure 2 together. First, convert the vector form of the second three-dimensional detection frame into the conventional three-dimensional detection frame format Y _{3D fusion detection} = [X _v Y _v Z _v ] ^T , where:

Z _v =(h ₂ -h ₁ )/2;

X _v =x _c1 -x _c2 ;

Y _v =y _c1 -y _c2 ;

Among them, x _ci represents the x-axis coordinate of point c _i , y _ci represents the y-axis coordinate of point c _i , i=1, 2, 3, 4, [X _v Y _v Z _v ] ^T represents the center of the second three-dimensional detection frame The coordinates of the point.

Then, the four points near the center point of the second three-dimensional detection frame (the four points A, B, C, and D in Figure 2) are sequentially converted to the coordinate system to obtain four points in the image coordinate system. The specific conversion process as follows:

_{Homogeneous coordinates of point A} = [X _{v, 1} , Y _{v, 1} , Z _{v, 1} ] ^T = [X _v - (x _c2 - x _c1 )/2, Y _v , h2];

_{Homogeneous coordinates of point B} = [X _{v, 2} , Y _{v, 2} , Z _{v, 2} ] ^T = [X _v + (x _c2 -x _c1 )/2, Y _v , h2];

_{Homogeneous coordinates of point Y C} = [X _{v, 3} , Y _{v, 3} , Z _{v, 3} ] ^T = [X _v - (x _c2 - x _c1 )/2, Y _v , h1];

_{Homogeneous coordinates of Y point D} = [X _{v, 4} , Y _{v, 4} , Z _{v, 4} ] ^T = [X _v + (x _c2 -x _c1 )/2, Y _v , h1];

Y＝P*R*Tr _v elo _t o _c am*Y _{point n homogeneous coordinates} , n＝A, B, C, D;

Among them, Y represents the coordinates of the second two-dimensional detection frame, _{the homogeneous coordinates of the Y point n} represent the homogeneous vector of the target object, P is the camera projection matrix, R is the camera rotation matrix, and Tr _v elo _t o _c am is a 3x4 three-dimensional Rotation matrix from coordinate system to camera coordinate system.

Step S104: Calculate the first IoU value of the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object, obtain the first array, and calculate the first three-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object. The second IoU value of the three-dimensional detection frame is obtained to obtain the second array.

Specifically, the IoU value can be used to characterize the inconsistency between the two sets of detection frames. The first two-dimensional detection frame and the second two-dimensional detection frame are used to calculate the IoU value to obtain the first IoU value that represents the inconsistency between the first two-dimensional detection frame and the second two-dimensional detection frame. Multiple target objects correspond to multiple The first IoU value is constructed as the first array. Use the first three-dimensional detection frame and the second three-dimensional detection frame to perform IoU numerical calculations to obtain the second IoU numerical value that represents the inconsistency between the first three-dimensional detection frame and the second three-dimensional detection frame. Multiple target objects correspond to multiple second IoU values. Numeric value, construct it as a second array. It should be noted that both sets of detection frames correspond to the same target area, and the target objects in this target area are constant. Therefore, the number of elements in the first array and the second array is equal, and each target object is in the first array or There are corresponding elements in the second array.

Further, in step S104, the first IoU value of the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object is calculated to obtain the first array, which specifically includes:

1. Confirm the first 2D detection frame and the second 2D detection frame of the target corresponding to each target object.

Specifically, since there are often multiple target objects and multiple detection frames in the scene, when confirming the target first two-dimensional detection frame and the target second two-dimensional detection frame corresponding to each target object, first select one The first two-dimensional detection frame is used as the first two-dimensional detection frame of the target, and then the Euclidean distance between the center point of the first two-dimensional detection frame of the target and the center point of each second two-dimensional detection frame is calculated, and the Euclidean distance with the smallest Euclidean distance is selected. The two-dimensional detection frame is used as the second two-dimensional detection frame of the target.

2. Calculate respectively the first area and the second area of the target first two-dimensional detection frame, the target second two-dimensional detection frame, and the third area of the overlapping area of the target first two-dimensional detection frame and the target second two-dimensional detection frame. area.

Specifically, by obtaining the coordinates of four vertices of the two-dimensional detection frame, the first area and the second area of the target first two-dimensional detection frame and the target second two-dimensional detection frame are calculated based on the coordinates of the four vertices. Calculated as follows:

S _A =|x _a2 -x _a1 |×|y _a2 -y _a1 |;

S _B =|x _b2 -x _b1 |×|y _b2 -y _b1 |;

Among them, S _A represents the first area, A1(x _a1 ,y _a1 ), B1(x _a1 ,y _a2 ), C1(x _a2 ,y _a1 ), D1(x _a2 ,y _a2 ) represent the first two dimensions of the target. The coordinates of the four vertices of the detection box. S _B represents the second area, A2(x _b1 ,y _b1 ), B2(x _b1 ,y _b2 ), C2(x _b2 ,y _b1 ), D2(x _b2 ,y _b2 ) represent the second two-dimensional detection frame of the target The coordinates of the four vertices.

Taking Figure 3 as an example to illustrate, when two detection frames have an overlapping area, the overlapping area is also a rectangular frame. The four vertices of the rectangular frame can be determined according to the overlapping first two-dimensional detection frame of the target and the second two-dimensional target frame. The coordinates of the four vertices of the two-dimensional detection frame are obtained. The coordinates of the upper left corner are A2 (x _b1 , y _b1 ), the coordinates of the lower left corner are E (x _b1 , y _a2 ), and the coordinates of the upper right corner are F (x _a2 , y _b1 ). , the coordinates of the lower right corner are D1(x _a2 ,y _a2 ). Then use the four vertex coordinates of the overlapping area to calculate the third area of the overlapping area.

3. Use the first area, the second area, and the third area to calculate the first IoU value corresponding to each target object.

Specifically, the first IoU value=third area/(first area+second area-third area).

4. Construct the first array using the first IoU values corresponding to all target objects.

Specifically, the first array is expressed as: I=[i ₁ , i ₂ ,..., _in ], and n is the number of first IoU values.

Further, in step S104, the second IoU value of the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object is calculated to obtain the second array, which specifically includes:

1. Confirm the first three-dimensional target detection frame and the second three-dimensional target detection frame corresponding to each target object.

Specifically, the first three-dimensional detection frame of the target and the second three-dimensional detection frame of the target are also confirmed using the Euclidean distance between the center point coordinates of the first three-dimensional detection frame and the second three-dimensional detection frame.

2. Calculate the first volume and the second volume of the first three-dimensional detection frame of the target and the second three-dimensional detection frame of the target respectively.

Specifically, the length, width, and height of the three-dimensional detection frame can be determined based on the coordinates of the eight vertices of the three-dimensional detection frame, and the volume of the three-dimensional detection frame is calculated using the length, width, and height.

3. Calculate the bottom area of the overlapping area between the first three-dimensional detection frame of the target and the second three-dimensional detection frame of the target.

4. Confirm the height of the overlapping area between the first three-dimensional detection frame of the target and the second three-dimensional detection frame of the target.

5. Use the base area and height to calculate the third volume of the overlapping area.

It should be understood that the overlapping area of the first three-dimensional detection frame and the second three-dimensional detection frame is also a solid structure, and its volume can be calculated by multiplying the base area by the height, and the corresponding areas of the first three-dimensional detection frame and the second three-dimensional detection frame The target objects are all on the ground, that is, the bottom surfaces of the first three-dimensional detection frame and the second three-dimensional detection frame are on the same plane. Therefore, the vertex coordinates of the bottom surface of the overlapping area of the first three-dimensional detection frame and the second three-dimensional detection frame can be determined according to the first three-dimensional detection frame. Confirm the four vertex coordinates of the bottom surface of the detection frame and the four vertex coordinates of the second three-dimensional detection frame, and then calculate the bottom area of the overlapping area based on the vertex coordinates of the bottom surface of the overlapping area. This method is the same as that obtained in the two-dimensional detection frame. The method for overlapping areas is the same and will not be described again here. As for the height of the overlapping area, since the target objects corresponding to the first three-dimensional detection frame and the second three-dimensional detection frame are both on the ground, that is, the bottom surfaces of the first three-dimensional detection frame and the second three-dimensional detection frame are on the same plane, therefore, the first three-dimensional detection frame The height of the overlapping area between the three-dimensional detection frame and the second three-dimensional detection frame is the smaller height value of the first three-dimensional detection frame and the second three-dimensional detection frame. After obtaining the bottom area and height of the overlapping area of the first three-dimensional detection frame and the second three-dimensional detection frame, the third volume of the overlapping area can be calculated.

6. Calculate the second IoU value corresponding to each target object using the first volume, the second volume, and the third volume.

Specifically, the second IoU value = third volume/(first volume + second volume – third volume)

7. Construct the second array using the second IoU values corresponding to all target objects.

Specifically, the first array is expressed as: U=[u ₁ , u ₂ ,..., u _n ], and n is the number of second IoU values.

Step S105: Perform attack detection and locate the attacked sensor based on the first array and the second array.

Specifically, after obtaining the first array and the second array, inconsistency detection is performed on the two arrays to confirm whether the sensor is attacked and to locate whether the attacked three-dimensional sensor or the two-dimensional sensor is a two-dimensional sensor.

Further, step S105 specifically includes:

1. Determine whether there are outliers in the first array and the second array respectively.

2. When there are outliers in the first array and no outliers in the second array, it is confirmed that the three-dimensional sensor is attacked.

3. When there are no outliers in the first array and there are outliers in the second array, it is confirmed that the two-dimensional sensor is attacked.

4. When there are outliers in both the first array and the second array, confirm that the three-dimensional sensor and/or two-dimensional sensor is attacked.

5. When there are no outliers in the first array and the second array, confirm that the three-dimensional sensor and the two-dimensional sensor are not attacked.

The attack detection method of the autonomous driving system according to the embodiment of the present invention performs target detection on the three-dimensional information data and the two-dimensional information data obtained by the three-dimensional sensor and the two-dimensional sensor respectively, and obtains the first three-dimensional detection frame and the first two-dimensional detection frame, and Fusion target detection is performed on the three-dimensional information data and the two-dimensional information data to obtain the second three-dimensional detection frame, and the coordinate transformation of the second three-dimensional detection frame is performed to obtain the second two-dimensional detection frame, and then the first and second detection frames corresponding to each object are obtained. Perform IoU numerical calculations on the first three-dimensional detection frame and the second three-dimensional detection frame to obtain the first array. Perform IoU numerical calculations on the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each object to obtain the second array. Analyze the first array. The array and the second array confirm whether the sensor has been attacked. It uses the correlation of three-dimensional sensors and two-dimensional sensors in time and space for attack detection, thereby improving the accuracy of attack detection and using it as the basic configuration of self-driving cars. Three-dimensional sensors and two-dimensional sensors can realize attack detection, and are no longer limited to using one of the sensor-based target detection algorithms for attack detection, and have higher universality.

Figure 4 is a schematic functional module diagram of the attack detection device of the autonomous driving system according to the embodiment of the present invention. As shown in FIG. 4 , the attack detection device 20 of the automatic driving system includes an acquisition module 21 , a first detection module 22 , a second detection module 23 , a calculation module 24 and an analysis module 25 .

The acquisition module 21 is used to obtain three-dimensional information data and two-dimensional information data of the target area using three-dimensional sensors and two-dimensional sensors respectively;

The first detection module 22 is used to input the three-dimensional information data and the two-dimensional information data into the pre-trained three-dimensional detection model and the two-dimensional detection model respectively to obtain the first three-dimensional detection frame and the first three-dimensional detection frame of each target object in the target area. 2D detection frame;

The second detection module 23 is used to input the three-dimensional information data and the two-dimensional information data into the pre-trained fusion detection model to obtain the second three-dimensional detection frame of each target object in the target area, and perform the second three-dimensional detection frame on the second three-dimensional detection frame. The coordinate system changes to obtain the second two-dimensional detection frame;

The calculation module 24 is used to calculate the first IoU value of the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object, obtain the first array, and calculate the first three-dimensional detection frame corresponding to each target object. and the second IoU value of the second three-dimensional detection frame to obtain the second array;

The analysis module 25 is configured to perform attack detection and locate the attacked sensor based on the first array and the second array.

Optionally, before performing the operation of inputting the three-dimensional information data and the two-dimensional information data into the pre-trained fusion detection model, the second detection module 23 is also used to: combine the two-dimensional information data at the previous moment and the two-dimensional information data at the current moment. Perform chi-square comparison on the information data to obtain the chi-square comparison value; determine whether the chi-square comparison value exceeds the preset threshold; when the chi-square comparison value exceeds the preset threshold, compare the two-dimensional information data at the current moment with the two-dimensional information data at the previous moment. The information data performs histogram matching to obtain the enhanced two-dimensional information data at the current moment and replace the two-dimensional information data at the current moment; when the chi-square comparison value does not exceed the preset threshold, the two-dimensional information data at the current moment is maintained.

Optionally, the second detection module 23 performs an operation of changing the coordinate system of the second three-dimensional detection frame to obtain the second two-dimensional detection frame, which specifically includes: obtaining the three-dimensional center point coordinates of the second three-dimensional detection frame; using the three-dimensional center point Coordinates confirm the homogeneous coordinates of four points on the same plane as the three-dimensional center point on the second three-dimensional detection frame; calculated using the pre-obtained camera projection matrix, camera rotation matrix, sensor-to-camera coordinate system rotation matrix and homogeneous coordinates Four two-dimensional coordinate points; use the four two-dimensional coordinate points to construct a second two-dimensional detection frame.

Optionally, the calculation module 24 performs the operation of calculating the first IoU value of the first two-dimensional detection frame and the second two-dimensional detection frame corresponding to each target object to obtain the first array, which specifically includes: confirming that each target object corresponds to The first two-dimensional detection frame of the target and the second two-dimensional detection frame of the target; respectively calculate the first area and the second area of the first two-dimensional detection frame of the target, the second two-dimensional detection frame of the target, and the first two-dimensional detection frame of the target. The third area of the overlapping area between the frame and the target's second two-dimensional detection frame; use the first area, the second area, and the third area to calculate the first IoU value corresponding to each target object; use the first IoU value corresponding to all target objects IoU values construct the first array.

Optionally, the calculation module 24 performs the operation of calculating the second IoU value of the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object to obtain the second array, which specifically includes: confirming the target corresponding to each target object. The first three-dimensional detection frame and the second three-dimensional detection frame of the target; calculate the first volume and the second volume of the first three-dimensional detection frame of the target and the second three-dimensional detection frame of the target respectively; calculate the first three-dimensional detection frame of the target and the second three-dimensional detection frame of the target The bottom area of the overlapping area of the frame; confirm the height of the overlapping area of the first three-dimensional detection frame of the target and the second three-dimensional detection frame of the target; calculate the third volume of the overlapping area using the bottom area and height; use the first volume and the second volume , the third volume is calculated to obtain the second IoU value corresponding to each target object; the second IoU value corresponding to all target objects is used to construct the second array.

Optionally, the analysis module 25 performs an operation of detecting an attack and locating the attacked sensor based on the first array and the second array, specifically including: determining whether there are outliers in the first array and the second array respectively; when the first array When there are outliers and there are no outliers in the second array, it is confirmed that the three-dimensional sensor has been attacked; when there are no outliers in the first array and there are outliers in the second array, it is confirmed that the two-dimensional sensor has been attacked; when the first array has no outliers, it is confirmed that the two-dimensional sensor has been attacked. When there are outliers in both the array and the second array, it is confirmed that the three-dimensional sensor and/or the two-dimensional sensor is under attack; when there are no outliers in the first array and the second array, it is confirmed that the three-dimensional sensor and the two-dimensional sensor are not under attack .

Optionally, the three-dimensional sensor includes lidar, the two-dimensional sensor includes a camera, the three-dimensional detection model is built based on the PointPillar algorithm, the two-dimensional detection model is built based on the YOLOv3 algorithm, and the fusion detection model is built based on the AVOD algorithm.

For other details of the technical solution for implementing each module in the attack detection device of the automatic driving system in the above embodiment, please refer to the description of the attack detection method of the automatic driving system in the above embodiment, and will not be described again here.

It should be noted that each embodiment in this specification is described in a progressive manner. Each embodiment focuses on its differences from other embodiments. The same and similar parts between the various embodiments are referred to each other. Can. As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiment.

Please refer to FIG. 5 , which is a schematic structural diagram of a computer device according to an embodiment of the present invention. As shown in FIG. 5 , the computer device 30 includes a processor 31 and a memory 32 coupled to the processor 31 . The memory 32 stores program instructions. When the program instructions are executed by the processor 31 , the processor 31 executes any of the above. The attack detection method steps of the automatic driving system described in the embodiment.

The processor 31 may also be called a CPU (Central Processing Unit). The processor 31 may be an integrated circuit chip with signal processing capabilities. The processor 31 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. . A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

Refer to FIG. 6 , which is a schematic structural diagram of a storage medium according to an embodiment of the present invention. The storage medium in the embodiment of the present invention stores program instructions 41 that can implement the above-mentioned attack detection method of the automatic driving system. The program instructions 41 can be stored in the above-mentioned storage medium in the form of a software product and include several instructions to enable an A computer device (which may be a personal computer, a server, a network device, etc.) or a processor executes all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code. , or computer equipment such as computers, servers, mobile phones, tablets, etc.

In the several embodiments provided in this application, it should be understood that the disclosed computer equipment, apparatus and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

In addition, each functional unit in various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units. The above are only embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, All are similarly included in the patent protection scope of this application.

Claims (10)

1. An attack detection method for an autopilot system, comprising:

respectively acquiring three-dimensional information data and two-dimensional information data of the target area by using a three-dimensional sensor and a two-dimensional sensor;

respectively inputting the three-dimensional information data and the two-dimensional information data into a pre-trained three-dimensional detection model and a pre-trained two-dimensional detection model to obtain a first three-dimensional detection frame and a first two-dimensional detection frame of each target object in the target area;

Inputting the three-dimensional information data and the two-dimensional information data into a pre-trained fusion detection model to obtain a second three-dimensional detection frame of each target object in the target area, and carrying out coordinate system change on the second three-dimensional detection frame to obtain a second two-dimensional detection frame;

calculating first IoU values of a first two-dimensional detection frame and a second two-dimensional detection frame corresponding to each target object to obtain a first array, and calculating second IoU values of the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object to obtain a second array;

and performing attack detection and positioning of the attacked sensor based on the first array and the second array.

2. The method for detecting an attack on an autopilot system according to claim 1, wherein before the three-dimensional information data and the two-dimensional information data are input into a pre-trained fusion detection model, further comprising:

carrying out chi-square comparison on the two-dimensional information data at the previous moment and the two-dimensional information data at the current moment to obtain chi-square comparison values;

judging whether the chi-square comparison value exceeds a preset threshold value or not;

when the chi-square comparison value exceeds a preset threshold, performing histogram matching on the two-dimensional information data at the current moment and the two-dimensional information data at the previous moment to obtain enhanced two-dimensional information data at the current moment, and replacing the two-dimensional information data at the current moment;

And when the chi-square comparison value does not exceed the preset threshold value, maintaining the two-dimensional information data at the current moment.

3. The method for detecting an attack on an autopilot system according to claim 1, wherein said performing a coordinate system change on the second three-dimensional detection frame to obtain a second two-dimensional detection frame includes:

acquiring the coordinates of a three-dimensional center point of the second three-dimensional detection frame;

confirming homogeneous coordinates of four points on the same plane with the three-dimensional center point on the second three-dimensional detection frame by utilizing the three-dimensional center point coordinates;

calculating four two-dimensional coordinate points by using a camera projection matrix, a camera rotation matrix, a rotation matrix from a sensor to a camera coordinate system and the homogeneous coordinates which are acquired in advance;

and constructing the second two-dimensional detection frame by utilizing the four two-dimensional coordinate points.

4. The method for detecting an attack on an autopilot system according to claim 1, wherein the calculating a first IoU value of a first two-dimensional detection frame and a second two-dimensional detection frame corresponding to each target object to obtain a first array includes:

confirming a target first two-dimensional detection frame and a target second two-dimensional detection frame corresponding to each target object;

Respectively calculating a first area and a second area of the target first two-dimensional detection frame and the target second two-dimensional detection frame and a third area of an overlapping area of the target first two-dimensional detection frame and the target second two-dimensional detection frame;

calculating a first IoU value corresponding to each target object by using the first area, the second area and the third area;

and constructing the first array by using the first IoU values corresponding to all the target objects.

5. The method for detecting an attack on an autopilot system according to claim 1, wherein said calculating a first three-dimensional detection box corresponding to each target object and a second IoU value of the second three-dimensional detection box to obtain a second set of values includes:

confirming a target first three-dimensional detection frame and a target second three-dimensional detection frame corresponding to each target object;

respectively calculating a first volume and a second volume of the target first three-dimensional detection frame and the target second three-dimensional detection frame;

calculating the bottom surface area of the overlapping area of the target first three-dimensional detection frame and the target second three-dimensional detection frame;

confirming the height of an overlapping area of the target first three-dimensional detection frame and the target second three-dimensional detection frame;

Calculating a third volume of the overlapping region using the bottom surface area and the height;

calculating a second IoU value corresponding to each target object by using the first volume, the second volume and the third volume;

and constructing the second array by using the second IoU values corresponding to all the target objects.

6. The attack detection method of an autopilot system of claim 1 wherein the attack detection and localization of the attacked sensor based on the first array and the second array comprises:

judging whether the first array and the second array have outliers or not respectively;

when the first array has an outlier and the second array does not have an outlier, confirming that the three-dimensional sensor is attacked;

when the first array has no outlier and the second array has outlier, confirming that the two-dimensional sensor is attacked;

when the first array and the second array both have outliers, confirming that the three-dimensional sensor and/or the two-dimensional sensor are attacked;

and when the first array and the second array do not have outliers, confirming that the three-dimensional sensor and the two-dimensional sensor are not attacked.

7. The attack detection method of an autopilot system of claim 1 wherein the three-dimensional sensor includes a lidar, the two-dimensional sensor includes a camera, the three-dimensional detection model is constructed based on a pointpilar algorithm, the two-dimensional detection model is constructed based on a YOLOv3 algorithm, and the fusion detection model is constructed based on an AVOD algorithm.

8. An attack detection device of an automatic driving system, comprising:

the acquisition module is used for respectively acquiring three-dimensional information data and two-dimensional information data of the target area by using a three-dimensional sensor and a two-dimensional sensor;

the first detection module is used for inputting the three-dimensional information data and the two-dimensional information data into a pre-trained three-dimensional detection model and a pre-trained two-dimensional detection model respectively to obtain a first three-dimensional detection frame and a first two-dimensional detection frame of each target object in the target area;

the second detection module is used for inputting the three-dimensional information data and the two-dimensional information data into a pre-trained fusion detection model to obtain a second three-dimensional detection frame of each target object in the target area, and carrying out coordinate system change on the second three-dimensional detection frame to obtain a second two-dimensional detection frame;

The computing module is used for computing a first IoU value of a first two-dimensional detection frame and a second two-dimensional detection frame corresponding to each target object to obtain a first array, and computing a second IoU value of the first three-dimensional detection frame and the second three-dimensional detection frame corresponding to each target object to obtain a second array;

and the analysis module is used for carrying out attack detection and positioning the attacked sensor based on the first array and the second array.

9. A computer device comprising a processor, a memory coupled to the processor, the memory having stored therein program instructions that, when executed by the processor, cause the processor to perform the steps of the attack detection method of an autopilot system according to any one of claims 1 to 7.

10. A storage medium storing program instructions for implementing the attack detection method of the autopilot system according to any one of claims 1 to 7.