CN110298262B - Object identification method and device - Google Patents

Abstract

The present application relates to the field of artificial intelligence. The sensing network comprises a main network and a plurality of parallel Headers, wherein the parallel Headers are connected with the main network; the backbone network is used for receiving an input picture, carrying out convolution processing on the input picture and outputting feature images with different resolutions corresponding to the picture; each Header in the plurality of parallel headers is configured to detect a task object in a task according to the feature map output by the backbone network, and output a 2D frame of an area where the task object is located and a confidence level corresponding to each 2D frame; each parallel Header completes detection of different task objects; wherein, the task object is an object to be detected in the task; the higher the confidence is, the greater the probability that the task object corresponding to the task exists in the 2D frame corresponding to the confidence is.

Description

Object recognition method and device

Technical field

The present application relates to the field of artificial intelligence, and in particular to an object recognition method and device.

Background technique

Computer vision is an integral part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a subject about how to use cameras/video cameras and computers to obtain What we need is the knowledge of data and information about the subjects being photographed. To put it figuratively, it is to install eyes (cameras or video cameras) and brains (algorithms) on computers to replace human eyes in identifying, tracking, and measuring targets, so that the computer can perceive the environment. Because perception can be viewed as extracting information from sensory signals, computer vision can also be viewed as the science that studies how to make artificial systems "perceive" from images or multi-dimensional data. In general, computer vision uses various imaging systems to replace the visual organs to obtain input information, and then the computer replaces the brain to process and interpret the input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans do, and have the ability to adapt to the environment independently.

At present, the visual perception network can complete more and more functions, including image classification, 2D detection, semantic segmentation (Mask), key point detection, linear object detection (such as lane line or stop line detection in autonomous driving technology), drivable Area detection, etc. In addition, the visual perception system has the characteristics of low cost, non-contact, small size and large amount of information. As the accuracy of visual perception algorithms continues to improve, it has become a key technology for many artificial intelligence systems today and is increasingly widely used, such as: Advanced Driving Assistant System (ADAS) and Autonomous Driving System (ADS). , Autonomous Driving System), the recognition of dynamic obstacles (people or vehicles) and static objects (traffic lights, traffic signs or traffic cones) on the road is achieved by identifying the human body's Mask in the camera beauty function of the terminal vision. and key points to achieve slimming effects, etc.

Most of the current mainstream visual perception networks focus on one detection task, such as 2D detection, 3D detection, semantic segmentation, key point detection, etc. If multiple functions are to be implemented, different networks are often required to complete them. Running multiple networks at the same time will significantly increase the calculation amount and power consumption of the hardware, reduce the running speed of the model, and make it difficult to achieve real-time detection.

Contents of the invention

In order to reduce the calculation amount and power consumption of the hardware and improve the computing speed of the perceptual network model, embodiments of the present invention provide a perceptual network based on multiple heads (Headers). The perceptual network includes a backbone network and multiple parallel Headers. The multiple parallel headers are connected to the backbone network;

The backbone network is used to receive input pictures, perform convolution processing on the input pictures, and output feature maps with different resolutions corresponding to the pictures;

The parallel Header is used to detect the task object in a task based on the feature map output by the backbone network, and output the 2D box of the area where the task object is located and the confidence level corresponding to each 2D box; wherein, Each of the parallel headers completes the detection of different task objects; wherein the task object is an object that needs to be detected in the task; the higher the confidence level, it means that the 2D box corresponding to the confidence level contains the The greater the probability of the object corresponding to the task. The one parallel header is any one of the above multiple parallel headers, and the functions of each parallel header are similar.

Optionally, each parallel headend includes a region candidate generation network (RPN) module, a region of interest extraction (ROI-ALIGN) module, and a region convolutional neural network (RCNN) module. The RPN module of one parallel headend is independent of RPN modules of other parallel head ends; the ROI-ALIGN module of one parallel head end is independent of the ROI-ALIGN modules of other parallel head ends; the RCNN module of one parallel head end is independent of the RCNN modules of other parallel head ends, wherein, for each Parallel headend:

The RPN module is used to: predict the area where the task object is located on one or more feature maps provided by the backbone network, and output candidate 2D boxes that match the area;

The ROI-ALIGN module is used to: extract the characteristics of the area where the candidate 2D box is located from a feature map provided by the backbone network based on the area predicted by the RPN module;

The RCNN module is used to perform convolution processing on the characteristics of the area where the candidate 2D box is located through a neural network to obtain the confidence that the candidate 2D box belongs to each object category; each object category is the one parallel head The object category in the task corresponding to the end; adjust the coordinates of the candidate area 2D box through the neural network, so that the adjusted 2D candidate box matches the shape of the actual object better than the candidate 2D box, and selects a confidence level greater than The adjusted 2D candidate frame with a preset threshold is used as the 2D frame of the region.

Optionally, the 2D frame is a rectangular frame.

Optionally, in another aspect of the embodiment of the present application, the RPN module is configured to: based on the template box (Anchor) of the object corresponding to the task, identify the existence of the task on one or more feature maps provided by the backbone network The area of the object is predicted to obtain a candidate area, and a candidate 2D box matching the candidate area is output; wherein the template box is obtained based on the statistical characteristics of the task object to which it belongs, and the statistical characteristics include the object's Shape and size.

Optionally, in another aspect of the embodiment of this application, the perception network further includes at least one or more serial Headers; the serial Header is connected to the one parallel Header;

The serial header is used to: utilize the 2D box of the task object belonging to the task provided by the parallel header it is connected to, and extract the characteristics of the area where the 2D box is located on one or more feature maps on the backbone network. According to the The characteristics of the area where the 2D box is located predict the 3D information, Mask information or Keypiont information of the task object to which the task belongs.

Optionally, the RPN module predicts areas where objects of different sizes are located on feature maps of different resolutions.

Optionally, the RPN module completes the detection of the area where the large object is located on the low-resolution feature map, and the RPN module completes the detection of the area where the small object is located on the high-resolution feature map.

On the other hand, embodiments of the present invention also provide an object detection method, which method includes:

Receive input images;

Perform convolution processing on the input image and output feature maps with different resolutions corresponding to the image;

According to the feature map, the task object in each task is independently detected for different tasks, and the 2D box of the area where each task object is located and the confidence level corresponding to each 2D box are output; wherein, the task object is Objects that need to be detected in the task; the higher the confidence level, the greater the probability that the object corresponding to the task exists in the 2D box corresponding to the confidence level.

Optionally, based on the feature map, independently detect the task objects in each task for different tasks, and output the 2D box of the area where each task object is located and the confidence level corresponding to each 2D box, including:

Predict the area where the task object is located on one or more feature maps, and output candidate 2D boxes that match the area;

According to the area where the task object is located, extract the features of the area where the candidate 2D box is located from a feature map;

Perform convolution processing on the features of the area where the candidate 2D box is located to obtain the confidence that the candidate 2D box belongs to each object category; each object category is an object category in the one task;

The coordinates of the 2D box of the candidate area are adjusted through the neural network so that the adjusted 2D candidate box matches the shape of the actual object better than the candidate 2D box, and the adjusted 2D candidate whose confidence is greater than the preset threshold is selected. box as a 2D box for the region.

Optionally, the 2D frame is a rectangular frame.

Optionally, predict the area where the task object is located on one or more feature maps, and output the candidate 2D box matching the area as:

Based on the template box (Anchor) of the object corresponding to the task, predict the area where the task object exists on one or more feature maps provided by the backbone network to obtain the candidate area, and output the candidate 2D box matching the candidate area. ; Wherein, the template frame is obtained based on the statistical characteristics of the task object to which it belongs, and the statistical characteristics include the shape and size of the object.

Optionally, the method also includes:

Based on the 2D box of the task object to which the task belongs, the features of the area where the 2D box is located are extracted on one or more feature maps on the backbone network, and the task object of the associated task is calculated based on the characteristics of the area where the 2D box is located. 3D information, Mask information or Keypiont information for prediction.

Optionally, the detection of the area where the large object is located is completed on the low-resolution feature, and the RPN module is used to detect the area where the small object is located on the high-resolution feature map.

On the other hand, embodiments of the present application provide a method for training a multi-task perception network based on partially annotated data, which is characterized in that the perception network includes a backbone network and multiple parallel heads (Headers), and the method includes:

According to the annotated data type of each picture, determine the task to which each picture belongs; wherein, each picture is annotated with one or more data types, and the multiple data types are subsets of all data types, and one data type corresponds to a task;

According to the task to which each picture belongs, determine the Header that needs to be trained for each picture;

Calculate the loss value of the Header required to be trained for each picture;

For each picture, gradient backpropagation is performed through the Header required to be trained, and the parameters of the Header required to be trained and the backbone network are adjusted based on the loss value.

Optionally, perform data balancing on images belonging to different tasks.

Embodiments of the present invention also provide a device for training a multi-task perception network based on partial annotation data. The perception network includes a backbone network and multiple parallel heads (Headers). The device includes:

The task determination module is used to determine the task to which each picture belongs based on the annotated data type of each picture; wherein each picture is annotated with one or more data types, and the multiple data types are sub-categories of all data types. Set, one data type corresponds to one task;

The Header determination module is used to determine the Header required for each picture to be trained based on the task to which each picture belongs;

The loss value calculation module is used to calculate the loss value of the header determined by the header decision module for each picture;

The adjustment module performs gradient return for each picture by calculating the Header determined by the Header determination module, and adjusts the parameters of the Header to be trained and the backbone network based on the loss value obtained by the loss value calculation module.

Optionally, the device further includes: a data balancing module, configured to perform data balancing on pictures belonging to different tasks.

Embodiments of the present invention also provide a cognitive network application system, which includes at least one processor, at least one memory, at least one communication interface, and at least one display device. The processor, memory, display device and communication interface are connected and communicate with each other through the communication bus.

Communication interface, used to communicate with other devices or communication networks;

The memory is used to store the application code that executes the above scheme, and the processor controls the execution. The processor is configured to execute application code stored in the memory.

The code stored in the memory 2002 may execute the Multi-Header based object perception method provided above, or may be the method of training the perception network provided in the above embodiment.

The display device is used to display the image to be recognized, 2D, 3D, Mask, key points and other information of the object of interest in the image.

In the perceptual network provided by the embodiments of this application, each perceptual task shares the same backbone network, doubling the amount of calculation and improving the computing speed of the perceptual network model; the network structure is easy to expand, and only one or several Headers need to be added to expand the 2D Detection type. Each parallel Header has independent RPN and RCNN modules, and only needs to detect the objects of the task to which it belongs, so that during the training process, unlabeled objects of other tasks can be avoided.

These and other aspects of the application will be more clearly understood in the following description of the embodiments.

Description of drawings

In order to more clearly explain the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description These are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic structural diagram of the system architecture provided by the embodiment of the present application;

Figure 2 is a schematic diagram of the CNN feature extraction model provided by the embodiment of the present application;

Figure 3 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application;

Figure 4 is a schematic diagram of a perceptual network application system framework based on multiple parallel headers provided by an embodiment of the present application;

Figure 5 is a schematic structural diagram of a perception network based on multiple parallel headers provided by an embodiment of the present application;

Figure 6 is a schematic structural diagram of the ADAS/AD sensing system based on multiple parallel headers provided by the embodiment of the present application;

Figure 7 is a schematic flowchart of basic feature generation provided by an embodiment of the present application;

Figure 8 is a schematic structural diagram of another RPN layer provided by an embodiment of the present application;

Figure 9 is a schematic diagram of the corresponding Anchor of another object in the RPN layer provided by the embodiment of the present application;

Figure 10 is a schematic diagram of another ROI-ALIGN process provided by the embodiment of the present application;

Figure 11 is a schematic diagram of the implementation and structure of another RCNN provided by the embodiment of the present application;

Figure 12 is a schematic diagram of the implementation and structure of another serial header provided by the embodiment of the present application;

Figure 13 is a schematic diagram of the implementation and structure of another serial header provided by the embodiment of the present application;

Figure 14 is a schematic diagram of the implementation and structural diagram of a serial header provided by the embodiment of the present application;

Figure 15 is a schematic diagram of a training method for partially annotated data provided by an embodiment of the present application;

Figure 16 is a schematic diagram of another training method for partially annotated data provided by an embodiment of the present application;

Figure 17 is a schematic diagram of another training method for partially annotated data provided by an embodiment of the present application;

Figure 18 is a schematic diagram of another training method for partially annotated data provided by an embodiment of the present application;

Figure 19 is a schematic application diagram of a perception network based on multiple parallel headers provided by the embodiment of the present application;

Figure 20 is a schematic application diagram of a sensing network based on multiple parallel headers provided by the embodiment of the present application;

Figure 21 is a schematic flowchart of a sensing method provided by an embodiment of the present application;

Figure 22 is a schematic diagram of a 2D detection process provided by an embodiment of the present application;

Figure 23 is a schematic diagram of a 3D detection process of a terminal device provided by an embodiment of the present application;

Figure 24 is a schematic diagram of a Mask prediction process provided by an embodiment of the present application;

Figure 25 is a schematic flowchart of a key point coordinate prediction process provided by an embodiment of the present application;

Figure 26 is a schematic diagram of the training process of a perceptual network provided by an embodiment of the present application;

Figure 27 is a schematic structural diagram of a perception network based on multiple parallel headers provided by the embodiment of the present application;

Figure 28 is a schematic structural diagram of a perception network based on multiple parallel headers provided by the embodiment of the present application;

Figure 29 is a device diagram for training a multi-task perception network based on partial annotation data provided by an embodiment of the present application;

Figure 30 is a schematic flow chart of an object detection method provided by an embodiment of the present application;

Figure 31 is a flow chart for training a multi-task perception network based on partially annotated data provided by an embodiment of the present application.

Detailed ways

First, the abbreviations used in the embodiments of this application are listed as follows:

Table 1

It should be noted that some of the drawings of the embodiments of the present invention are described in English in order to be more consistent with the terminology description in the industry, and the corresponding Chinese definitions are also given in the embodiments. The embodiments of the present application are described below with reference to the accompanying drawings.

The embodiments of this application are mainly used in fields such as driving assistance, autonomous driving, and mobile phone terminals that require the completion of various sensing tasks. The application system framework of the present invention is shown in Figure 4. A single picture is obtained from the video through frame extraction. The picture is sent to the Mulit-Header perception network of the present invention to obtain the 2D, 3D and Mask of the object of interest in the picture. membrane), key points and other information. These detection results are output to the post-processing module for processing, such as sending them to the planning control unit in the autonomous driving system for decision-making, and sending them to the beautification algorithm in the mobile phone terminal for processing to obtain beautified pictures. The following is a brief introduction to the two application scenarios of ADAS/ADS visual perception system and mobile phone beauty.

Application scenario 1: ADAS/ADS visual perception system

As shown in Figure 19, in ADAS and ADS, multiple types of 2D target detection need to be performed in real time, including: dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks) (Truck, Bus), static obstacles (TrafficCone, TrafficStick, FireHydrant, Motorcycle, Bicycle), traffic signs ( TrafficSign, GuideSign, Billboard, TrafficLight_Red/TrafficLight_Yellow/TrafficLight_Green/TrafficLight_Black, RoadSign). In addition, in order to accurately obtain the area occupied by the dynamic obstacle in the 3D space, it is also necessary to perform 3D estimation of the dynamic obstacle and output a 3D box. In order to fuse it with the lidar data, it is necessary to obtain the Mask of the dynamic obstacle to filter out the laser point cloud hitting the dynamic obstacle; in order to carry out accurate parking space, it is necessary to detect the four key points of the parking space at the same time. ;In order to perform composition positioning, the key points of the static target need to be detected. Using the technical solutions provided by the embodiments of this application, all the above functions can be completed in a perception network.

Application scenario 2: mobile phone beauty function

As shown in Figure 20, in the mobile phone, the mask and key points of the human body are detected through the perception network provided by the embodiment of the present application, and the corresponding parts of the human body can be enlarged and reduced, such as waist tightening and buttock beautifying operations, thereby outputting beauty. picture of.

Application scenario 3: Image classification scenario:

After obtaining the image to be classified, the object recognition device uses the object recognition method of the present application to obtain the category of the object in the image to be classified, and then the image to be classified can be classified according to the category of the object in the image to be classified. For photographers, they take many photos every day, including animals, people, and plants. The method of this application can be used to quickly classify photos according to the content in the photos, which can be divided into photos containing animals, photos containing people, and photos containing plants.

For situations where the number of images is relatively large, the efficiency of manual classification is relatively low, and people are prone to fatigue when processing the same thing for a long time. At this time, the classification results will have large errors; and the method of this application is used Images can be classified quickly and without errors.

Application scenario 4 product classification:

After the object recognition device obtains the image of the commodity, it then uses the object recognition method of the present application to obtain the category of the commodity in the image of the commodity, and then classifies the commodity according to the category of the commodity. For a wide variety of goods in large shopping malls or supermarkets, using the object recognition method of this application can quickly complete the classification of goods, reducing time overhead and labor costs.

The methods and devices provided by the embodiments of the present application can also be used to expand the training database. As shown in Figure 1, the I/O interface 112 of the execution device 120 can process images processed by the execution device (such as image blocks or images containing objects). Together with the object categories input by the user, they are sent to the database 130 as a training data pair, so that the training data maintained by the database 130 is richer, thereby providing richer training data for the training work of the training device 130 .

The method provided by this application is described below from the model training side and the model application side:

The method for training a CNN feature extraction model provided by the embodiments of the present application involves computer vision processing, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning. The training data (such as images of objects in this application or image blocks and object categories) to carry out symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc., and finally obtain the trained CNN feature extraction model; and, the embodiment of this application will input data (such as this The image of the object in the application) is input into the trained CNN feature extraction model to obtain output data (for example, in this application, the 2D, 3D, Mask, key points and other information of the object of interest in the picture are obtained).

Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms involved in the embodiments of the present application and related concepts such as neural networks are first introduced below.

(1) Object recognition, using image processing, machine learning, computer graphics and other related methods to determine the category of image objects.

(2)Neural network

The neural network can be composed of neural units. The neural unit can refer to an arithmetic unit that takes xs and intercept 1 as input. The output of the arithmetic unit can be:

Among them, s=1, 2,...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

(3) Deep neural network

Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the division of DNN according to the position of different layers, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks very complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector,/> is the output vector,/> is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just a response to the input vector/> After such a simple operation, the output vector is obtained/> Since there are many DNN layers, the coefficient W and offset vector/> The number is also very large. The definitions of these parameters in DNN are as follows: Taking the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as /> The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third layer index 2 and the input second layer index 4. The summary is: the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as/> It should be noted that the input layer has no W parameter. in depth

In neural networks, more hidden layers make the network more capable of describing complex situations in the real world. Theoretically, a model with more parameters has higher complexity and greater "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix. The ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by the vectors W of many layers).

(4) Convolutional neural network

Convolutional Neural Network (CNN, Convolutional Neuron Network) is a deep neural network with a convolutional structure. The convolutional neural network contains a feature extractor composed of convolutional layers and subsampling layers. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). The convolutional layer refers to the neuron layer in the convolutional neural network that convolves the input signal. In the convolutional layer of a convolutional neural network, a neuron can be connected to only some of the neighboring layer neurons. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as a way to extract image information independent of position. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that the image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally, the greater the number of convolution kernels, the richer the image information reflected by the convolution operation.

The convolution kernel can be initialized in the form of a random-sized matrix. During the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

(5) Recurrent Neural Networks (RNN) are used to process sequence data. In the traditional neural network model, from the input layer to the hidden layer and then to the output layer, the layers are fully connected, while the nodes within each layer are unconnected. Although this ordinary neural network has solved many difficult problems, it is still incompetent for many problems. For example, if you want to predict the next word of a sentence, you generally need to use the previous word, because the preceding and following words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layer and the current layer are no longer unconnected but connected, and the input of the hidden layer not only includes The output of the input layer also includes the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. The error backpropagation algorithm is also used, but there is one difference: that is, if the RNN is expanded into a network, then the parameters, such as W, are shared; this is not the case with the traditional neural network as shown in the example above. And when using the gradient descent algorithm, the output of each step not only depends on the network of the current step, but also depends on the status of the network of several previous steps. This learning algorithm is called Back propagation Through Time (BPTT).

Now that we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In a convolutional neural network, there is a prerequisite assumption that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the changes in stocks over time. Another example is a person who said: I like traveling, and my favorite place is Yunnan. I must go there if I have the opportunity in the future. Fill in the blank here. Human beings should all know that it means "Yunnan". Because humans will make inferences based on the content of the context, but how do you get machines to do this? RNN came into being. RNN aims to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on current input information and historical memory information.

(6)Loss function

In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value that we really want to predict, we can compare the predicted value of the current network with the really desired target value, and then based on the difference between the two to update the weight vector of each layer of the neural network according to the difference (of course, there is usually an initialization process before the first update, that is, preconfiguring parameters for each layer in the deep neural network). For example, if the predicted value of the network If it is high, adjust the weight vector to make its prediction lower, and continue to adjust until the deep neural network can predict the really desired target value or a value that is very close to the really desired target value. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value". This is the loss function (loss function) or objective function (objective function), which is used to measure the difference between the predicted value and the target value. Important equations. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of reducing this loss as much as possible.

(7)Back propagation algorithm

The convolutional neural network can use the error back propagation (BP) algorithm to modify the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, forward propagation of the input signal until the output will produce an error loss, and the parameters in the initial super-resolution model are updated by back-propagating the error loss information, so that the error loss converges. The backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

The following introduces the system architecture provided by the embodiments of this application.

Referring to Figure 1, an embodiment of the present application provides a system architecture 110. As shown in the system architecture 110, the data collection device 170 is used to collect training data. In the embodiment of the present application, the training data includes: images or image blocks of objects and categories of objects; and the training data is stored in the database 130. The training device 130 is trained based on the training data maintained in the database 130 to obtain the CNN feature extraction model 101 (explanation: 101 here is the model trained in the training stage introduced earlier, which can be a perceptual network used for feature extraction, etc.). The following will describe in more detail how the training device 130 obtains the CNN feature extraction model 101 based on the training data in Embodiment 1. The CNN feature extraction model 101 can be used to implement the perceptual network provided by the embodiment of the present application, that is, to convert the image to be recognized or After the image block is input into the CNN feature extraction model 101 through relevant preprocessing, the 2D, 3D, Mask, key points and other information of the object of interest in the image or image block to be recognized can be obtained. The CNN feature extraction model 101 in the embodiment of the present application may specifically be a CNN convolutional neural network. It should be noted that in actual applications, the training data maintained in the database 130 may not all be collected by the data collection device 170, and may also be received from other devices. In addition, it should be noted that the training device 130 does not necessarily train the CNN feature extraction model 101 entirely based on the training data maintained by the database 130. It is also possible to obtain training data from the cloud or other places for model training. The above description should not be used as a limitation of this application. Limitations of Examples.

The CNN feature extraction model 101 trained according to the training device 130 can be applied to different systems or devices, such as to the execution device 120 shown in Figure 1. The execution device 120 can be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, AR/VR, vehicle terminals, etc., or servers or clouds, etc. In Figure 1, the execution device 120 is configured with an I/O interface 112 for data interaction with external devices. The user can input data to the I/O interface 112 through the client device 150. The input data is used in the embodiment of the present application. may include: images, image blocks, or pictures to be recognized.

When the execution device 120 preprocesses the input data, or when the calculation module 111 of the execution device 120 performs calculation and other related processing (such as implementing the function of the perception network in this application), the execution device 120 can call the data storage system 160 The data, codes, etc. in the system can be used for corresponding processing, and the data, instructions, etc. obtained by corresponding processing can also be stored in the data storage system 160.

Finally, the I/O interface 112 returns the processing results, such as the image or image block obtained above, or 2D, 3D, Mask, key points and other information of the object of interest in the picture, to the client device 150, thereby providing it to the user.

Optionally, the client device 150 may be a planning control unit in an autonomous driving system or a beauty algorithm module in a mobile phone terminal.

It is worth mentioning that the training device 130 can generate corresponding target models/rules 101 based on different training data for different goals or different tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete the The above tasks, thereby providing the user with the desired results.

In the situation shown in FIG. 1 , the user can manually set the input data, and the manual setting can be operated through the interface provided by the I/O interface 112 . In another case, the client device 150 can automatically send input data to the I/O interface 112. If requiring the client device 150 to automatically send the input data requires the user's authorization, the user can set corresponding permissions in the client device 150. The user can view the results output by the execution device 120 on the client device 150, and the specific presentation form may be display, sound, action, etc. The client device 150 can also be used as a data collection terminal to collect input data from the input I/O interface 112 and output results from the output I/O interface 112 as new sample data, and store them in the database 130 . Of course, it is also possible to collect data without going through the client device 150. Instead, the I/O interface 112 directly uses the input data input to the I/O interface 112 and the output result of the output I/O interface 112 as a new sample as shown in the figure. The data is stored in database 130.

It is worth noting that Figure 1 is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 1 , the data storage system 160 is an external memory relative to the execution device 120. In other cases, the data storage system 160 can also be placed in the execution device 120.

As shown in Figure 1, a CNN feature extraction model 101 is obtained by training according to the training device 130. In the embodiment of the present application, the CNN feature extraction model 101 can be a CNN convolutional neural network or based on multiple Headers to be introduced in the following embodiments. perception network.

As mentioned in the introduction to the basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture. The deep learning architecture refers to the algorithm of machine learning. Multiple levels of learning at different levels of abstraction. As a deep learning architecture, CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the image input into it.

As shown in FIG. 2 , a convolutional neural network (CNN) 210 may include an input layer 220 , a convolutional/pooling layer 230 (where the pooling layer is optional), and a neural network layer 230 .

Convolutional layer/pooling layer 230:

Convolutional layer:

As shown in Figure 2, the convolution layer/pooling layer 230 may include layers 221-226 as examples. For example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, and layer 223 is a convolution layer. Product layer, 224 is a pooling layer, 225 is a convolution layer, and 226 is a pooling layer; in another implementation, 221 and 222 are convolution layers, 223 is a pooling layer, and 224 and 225 are convolutions. layer, 226 is the pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or can be used as the input of another convolutional layer to continue the convolution operation.

The following will take convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

The convolution layer 221 can include many convolution operators. The convolution operator is also called a kernel. Its role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator is essentially It can be a weight matrix. This weight matrix is usually predefined. During the convolution operation on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. ...This depends on the value of the step size) to complete the process of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. During the convolution operation, the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a convolved output with a single depth dimension, but in most cases, instead of using a single weight matrix, multiple weight matrices of the same size (rows × columns) are applied, That is, multiple matrices of the same type. The output of each weight matrix is stacked to form the depth dimension of the convolution image. The dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to remove unnecessary noise in the image. Perform blurring, etc. The multiple weight matrices have the same size (row × column), and the feature maps extracted by the multiple weight matrices with the same size are also the same size. The extracted multiple feature maps with the same size are then merged to form a convolution operation. output.

The weight values in these weight matrices require a lot of training in practical applications. Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, thereby allowing the convolutional neural network 210 to make correct predictions. .

When the convolutional neural network 210 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features; as the convolutional neural network As the depth of the network 210 deepens, the features extracted by subsequent convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. In each layer 221-226 as shown at 230 in Figure 2, it can be a convolution layer followed by a layer The pooling layer can also be a multi-layer convolution layer followed by one or more pooling layers. During image processing, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling. The max pooling operator can take the pixel with the largest value in a specific range as the result of max pooling. In addition, just like the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the size of the image. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average or maximum value of the corresponding sub-region of the image input to the pooling layer.

Neural network layer 230:

After being processed by the convolutional layer/pooling layer 230, the convolutional neural network 210 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 230 will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (required class information or other related information), the convolutional neural network 210 needs to use the neural network layer 230 to generate an output or a set of required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (231, 232 to 23n as shown in Figure 2) and an output layer 240. The parameters contained in the multiple hidden layers may be based on specific task types. Relevant training data are pre-trained. For example, the task type can include image recognition, image classification, image super-resolution reconstruction, etc...

After the multi-layer hidden layer in the neural network layer 230, that is, the last layer of the entire convolutional neural network 210 is the output layer 240. The output layer 240 has a loss function similar to classification cross entropy and is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 210 (the propagation from the direction 220 to 240 in Figure 2 is forward propagation) is completed, the back propagation (the propagation from the direction 240 to 220 in Figure 2 is back propagation) will Start updating the weight values and biases of each layer mentioned above to reduce the loss of the convolutional neural network 210 and the error between the result output by the convolutional neural network 210 through the output layer and the ideal result.

It should be noted that the convolutional neural network 210 shown in Figure 2 is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models.

The following introduces a chip hardware structure provided by the embodiment of the present application.

Figure 3 is a chip hardware structure provided by an embodiment of the present invention. The chip includes a neural network processor 30. The chip can be disposed in the execution device 120 as shown in Figure 1 to complete the calculation work of the calculation module 111. The chip can also be provided in the training device 130 as shown in Figure 1 to complete the training work of the training device 130 and output the target model/rules 101. The algorithms of each layer in the convolutional neural network shown in Figure 2 can be implemented in the chip shown in Figure 3.

Neural network processor NPU 30, the NPU is mounted on the main CPU (Host CPU) as a co-processor, and the Host CPU allocates tasks. The core part of the NPU is the arithmetic circuit 303. The controller 304 controls the arithmetic circuit 303 to extract data in the memory (weight memory or input memory) and perform operations.

In some implementations, the computing circuit 303 internally includes multiple processing units (Process Engine, PE). In some implementations, arithmetic circuit 303 is a two-dimensional systolic array. The arithmetic circuit 303 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 303 is a general-purpose matrix processor.

For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit obtains the corresponding data of matrix B from the weight memory 302 and caches it on each PE in the arithmetic circuit. The operation circuit takes matrix A data and matrix B from the input memory 301 to perform matrix operations, and the partial result or final result of the matrix is stored in an accumulator (accumulator) 308 .

The vector calculation unit 307 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional/non-FC layers in neural networks, such as pooling, batch normalization, local response normalization, etc. .

In some implementations, the vector computation unit can 307 store the processed output vectors to the unified buffer 306 . For example, the vector calculation unit 307 may apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, vector calculation unit 307 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 303, such as for use in a subsequent layer in a neural network.

The operation of the perception network provided by the embodiment of this application can be performed by 303 or 307.

The unified memory 306 is used to store input data and output data.

The weight data directly transfers the input data in the external memory to the input memory 301 and/or unified memory 306 through the storage unit access controller 305 (Direct Memory Access Controller, DMAC), stores the weight data in the external memory into the weight memory 302, and Store the data in the unified memory 306 into the external memory.

The Bus Interface Unit (BIU) 310 is used to realize the interaction between the main CPU, the DMAC and the instruction memory 309 through the bus.

An instruction fetch buffer 309 connected to the controller 304 is used to store instructions used by the controller 304;

The controller 304 is used to call instructions cached in the memory 309 to control the working process of the computing accelerator.

Optionally, the input data here in this application is a picture, and the output data is 2D, 3D, Mask, key points and other information of the object of interest in the picture.

Generally, the unified memory 306, the input memory 301, the weight memory 302 and the instruction memory 309 are all on-chip memories, and the external memory is a memory external to the NPU. The external memory can be double data rate synchronous dynamic random access. Memory (Double Data Rate Synchronous Dynamic Random AccessMemory, referred to as DDR SDRAM), high bandwidth memory (High Bandwidth Memory, HBM) or other readable and writable memory.

The program algorithms in Figures 1 and 2 are completed by the main CPU and NPU.

Among them, the operations of each layer in the convolutional neural network shown in FIG. 2 can be performed by the operation circuit 303 or the vector calculation unit 307.

Refer to Figure 5, which is a schematic structural diagram of a multi-head sensing network provided by an embodiment of the present application. As shown in Figure 5, the perception network includes:

It mainly consists of two parts: the backbone network (Backbone) 401 and multiple parallel Headers0~N.

The backbone network 401 is used to receive input pictures, perform convolution processing on the input pictures, and output feature maps with different resolutions corresponding to the pictures; that is to say, output feature maps of different sizes corresponding to the pictures;

In other words, Backbone completes the extraction of basic features and provides corresponding features for subsequent detection.

Any parallel head end is used to detect the task object in a task based on the feature map output by the backbone network, and output the 2D box of the area where the task object is located and the confidence level corresponding to each 2D box; Wherein, each parallel Header completes the detection of different task objects; wherein, the task object is an object that needs to be detected in the task; the higher the confidence level, it means that there is a 2D box corresponding to the confidence level. The greater the probability of the object corresponding to the task.

That is to say, parallel Header completes different 2D detection tasks. For example, parallel Header0 completes car detection and outputs the 2D box and confidence of Car/Truck/Bus; parallel Header1 completes human detection and outputs the 2D box and confidence level of Pedestrian/Cyclist/Tricyle. Confidence; Parallel Header2 completes the detection of traffic lights and outputs the 2D box and confidence of Red_Trafficligh/Green_Trafficlight/Yellow_TrafficLight/Black_TrafficLight.

Optionally, as shown in Figure 5, the perception network may also include multiple serial Headers, and the perception network may also include at least one or more serial heads; the serial heads are connected to a parallel Head-end connection; what needs to be emphasized here is that although Figure 5 draws multiple serial headers for better display, in fact, serial headers are not necessary. For scenes that only need to detect 2D boxes, it is not necessary Need to include serial header.

The serial header is used to: utilize the 2D box of the task object belonging to the task provided by the parallel header it is connected to, and extract the characteristics of the area where the 2D box is located on one or more feature maps on the backbone network. According to the The characteristics of the area where the 2D box is located predict the 3D information, Mask information or Keypiont information of the task object to which the task belongs.

The serial header is optionally connected in series behind the parallel header. Based on detecting the 2D box of the task, the 3D/Mask/Keypoint detection of objects inside the 2D box is completed.

For example, serial 3D_Header0 completes the estimation of the vehicle's orientation, centroid, length, width and height, thereby outputting the 3D box of the vehicle; serial Mask_Header0 predicts the fine mask of the vehicle, thereby segmenting the vehicle; serial Keypont_Header0 completes the key points of the vehicle estimate.

The serial header is not necessary. Some tasks do not require 3D/Mask/Keypoint detection, so there is no need to connect the serial header in series. For example, the detection of traffic lights only needs to detect the 2D frame, and there is no need to connect the serial header in series. In addition, some tasks can choose to connect one or more serial headers in series according to the specific needs of the task. For example, the detection of parking lots (Parkingslot) requires not only the 2D frame, but also the key points of the parking spaces. Therefore, in this task You only need to connect a serial Keypoint_Header in series, and there is no need for 3D and Mask Headers.

Each module is described in detail below.

Backbone: The backbone network performs a series of convolution processes on the input images to obtain feature maps at different scales. These feature maps will provide basic features for subsequent detection modules. The backbone network can take many forms, such as VGG (Visual Geometry Group, visual geometry group), Resnet (Residual Neural Network, residual neural network), Inception-net (the core structure of GoogLeNet), etc.

Parallel Header: The parallel Header mainly completes the detection of the 2D box of a task based on the basic features provided by Backbone, and outputs the 2D box of the object of the task and the corresponding confidence.

Optionally, the parallel header of each task includes three modules: RPN, ROI-ALIGN and RCNN.

RPN module: used to predict the area where the task object is located on one or more feature maps provided by the backbone network, and output candidate 2D boxes that match the area;

Or it can be understood this way. The full name of RPN is Region Proposal Network. It predicts the areas where the task object may exist on one or more feature maps of Backbone, and gives the frames of these areas. These areas are called is the candidate area (Proposal).

For example, when parallel Header0 is responsible for detecting cars, its RPN layer predicts candidate boxes where cars may exist; when parallel Header1 is responsible for detecting people, its RPN layer predicts candidate boxes where people may exist. Of course, these proposals are inaccurate. On the one hand, they do not necessarily contain the objects of the task. On the other hand, these boxes are not compact.

ROI-ALIGN module: used to deduct the features of the area where the candidate 2D box is located from a feature map provided by the backbone network based on the area predicted by the RPN module;

In other words, the ROI-ALIGN module mainly extracts the features of the area where each Proposal is located on a feature map of Backbone based on the Proposal provided by the RPN module, and resizes it to a fixed size to obtain the features of each Proposal. . It can be understood that the ROI-ALIGN module can be used but is not limited to ROI-POOLING (region of interest pooling)/ROI-ALIGN (region of interest extraction)/PS-ROIPOOLING (position-sensitive region of interest pooling)/ Feature extraction methods such as PS-ROIALIGN (position-sensitive region of interest extraction).

RCNN module: used to perform convolution processing on the features of the area where the candidate 2D box is located through a neural network to obtain the confidence that the candidate 2D box belongs to each object category; each object category corresponds to the one parallel head end The object category in the task; adjust the coordinates of the 2D box in the candidate area through the neural network, so that the adjusted 2D candidate box matches the shape of the actual object better than the candidate 2D box, and select a confidence level greater than the preset The adjusted 2D candidate box of the threshold is used as the 2D box of the region.

In other words, the RCNN module mainly refines the features of each Proposal proposed by the ROI-ALIGN module to obtain the confidence of each Proposal belonging to each category (for example, for the car task, Backgroud/Car/Truck will be given /Bus 4 points), and at the same time adjust the coordinates of the Proposal's 2D box to output a tighter 2D box. These 2D boxes are merged by NMS (Non Maximum Suppression) and output as the final 2D box.

As mentioned above, in some practical application scenarios, the perception network may also include a serial header. The serial header is mainly connected in series behind the parallel header. On the basis of detecting the 2D frame, further 3D/Mask/Keypoint detection is performed. Therefore, there are 3 types of serial headers:

Serial 3D Header: The serial 3D Header uses the ROI-ALIGN module to convert these 2D boxes on a feature map of Backbone based on the 2D boxes provided by the front-end parallel Header (the 2D boxes at this time are accurate compact 2D boxes). The features of the area are extracted, and then the centroid point coordinates, orientation angle, length, width and height of the object inside the 2D box are returned through a small network (3D_Header in Figure 5), thereby obtaining complete 3D information.

Serial Mask Header: The serial Mask Header uses the ROI-ALIGN module to map these 2D boxes on a feature map of Backbone based on the 2D boxes provided by the front-end parallel header (the 2D boxes at this time are accurate compact 2D boxes). The features of the area are extracted, and then a small network (Mask_Header in Figure 5) is used to regress the mask of the object inside the 2D box, thereby segmenting the object.

Serial Keypoint Header: The serial Keypoint Header uses the ROI-ALIGN module to convert these 2D boxes on a feature map of Backbone based on the 2D boxes provided by the front-end parallel Header (the 2D boxes at this time are accurate compact 2D boxes). The features of the area are extracted, and then the key point coordinates of the object inside the 2D box are returned through a small network (Keypoint_Header in Figure 5).

Optionally, as shown in Figure 29, this embodiment of the present invention also provides a device for training a multi-task perception network based on partially annotated data. The perception network includes a backbone network and multiple parallel heads (Headers). The structure has been described in detail in the previous embodiments and will not be repeated here. The device includes:

The task determination module 2900 is used to determine the task to which each picture belongs based on the annotated data type of each picture; wherein each picture is annotated with one or more data types, and the multiple data types are all data types. Subset, one data type corresponds to one task;

Header determination module 2901 is used to determine the header that each picture needs to train according to the task to which each picture belongs determined by the task determination module 2900;

The loss value calculation module 2902 is used to calculate the loss value of the header determined by the header decision module 2901 for each picture;

The adjustment module 2903 performs gradient return for each picture by calculating the Header determined by the Header determination module 2901, and adjusts the parameters of the required training Header and the backbone network based on the loss value obtained by the loss value calculation module 2902. .

Optionally, in one embodiment, as shown in the dotted box in Figure 29, the device may further include:

The data balancing module 2904 is used to perform data balancing on pictures belonging to different tasks.

As shown in Figure 6, the following takes the ADAS/AD visual perception system as an example to introduce the embodiment of the present invention in detail.

In the ADAS/AD visual perception system, multiple types of 2D target detection need to be performed in real time, including: dynamic obstacles (Pedestrian, Cyclist, Tricycle, Car, Truck, Bus), static obstacles (TrafficCone, TrafficStick, FireHydrant, Motorcycle , Bicycle), traffic signs (TrafficSign, GuideSign, Billboard). In addition, in order to accurately obtain the area occupied by the vehicle in the 3D space, it is also necessary to perform 3D estimation of dynamic obstacles and output a 3D box. In order to fuse it with the lidar data, it is necessary to obtain the Mask of the dynamic obstacle to filter out the laser point cloud hitting the dynamic obstacle; in order to carry out accurate parking space, it is necessary to detect the four key points of the parking space at the same time. . Using the technical solution provided by this embodiment, all the above functions can be completed in a network. This embodiment will be described in detail below.

1. Division of each Header task and overall block diagram of the network

According to the similarity of the objects that need to be detected and the abundance and scarcity of training samples, in this embodiment, the 20 types of objects that need to be detected are divided into 8 major categories, as shown in Table 2.

Table 2 The object categories and extended functions that each Header needs to detect

According to business needs, Header0 needs to complete 3D and Mask detection in addition to 2D detection of cars; Header1 needs to further complete Mask detection in addition to 2D detection of people; Header2 needs to complete 2D parking space box detection in addition to In addition to detection, it is also necessary to complete the detection of key points of the parking space.

It should be noted that the task division in Table 2 is just an example in this embodiment. Different task divisions can be carried out in other embodiments and are not limited to the task division in Table 2.

According to the task division in Table 2, the overall structure of the perception network in this embodiment is shown in Figure 6.

The perceptual network mainly consists of three parts: Backbone, parallel Header and serial Header. It should be noted that, as mentioned in the previous embodiment, the serial header is not necessary. The reason has been described in the above embodiment and will not be repeated here. Among them, 8 parallel headers simultaneously complete the 2D detection of the 8 major categories in Table 1, and several serial headers are connected in series behind Header0~2 to further complete the 3D/Mask/Keypoint detection. It can be seen from Figure 6 that the present invention can flexibly add and delete Headers according to business needs, thereby realizing different functional configurations.

2. Basic feature generation

The basic feature generation process is implemented by Backbone in Figure 6, which performs convolution processing on the input image to generate several convolution feature maps of different scales. Each feature map is a matrix of H*W*C, where H is the feature map. The height of W is the width of the feature map, and C is the number of channels of the feature map.

Backbone can use a variety of existing convolutional network frameworks, such as VGG16, Resnet50, Inception-Net, etc. The following uses Resnet18 as Backbone to illustrate the basic feature generation process. The process is shown in Figure 7.

Assume that the resolution of the input image is H*W*3 (height H, width W, and the number of channels is 3, that is, three RBG channels). The input image undergoes a convolution operation through the first convolution module of Resnet18 (Res18-Conv1 in the figure. The convolution module consists of several convolution layers, and the subsequent convolution modules are similar) to generate Featuremap (feature map) C1. This feature map is downsampled twice relative to the input image, and the number of channels is expanded to 64, so the resolution of C1 is H/4*W/4*64; C1 passes through the second convolution module of Resnet18 (Res18- Conv2) performs a convolution operation to obtain Featuremap C2. The resolution of this feature map is the same as C1; C2 continues to be processed by the third convolution module (Res18-Conv3) of Resnet18 to generate Featuremap C3. This feature map is further lower than C2. Sampling, the number of channels is doubled, and its resolution is H/8*W/8*128; finally, C3 is processed by Res18-Conv4 to generate Featuremap C4, its resolution is H/16*W/16*256.

As can be seen from Figure 7, Resnet18 performs multiple levels of convolution processing on the input image to obtain feature maps of different scales: C1/C2/C3/C4. The width and height of the low-level feature map are relatively large and the number of channels is small. They are mainly low-level features of the image (such as image edges and texture features). The width and height of the high-level feature map are relatively small and the number of channels is large. They are mainly low-level features of the image (such as image edges and texture features). It is the high-level feature of the image (such as shape and object features). The subsequent 2D detection process will make further predictions based on these feature maps.

3. 2D candidate area prediction process

The 2D candidate area prediction process is implemented by the RPN module of each parallel Header in Figure 6. Based on the feature map (C1/C2/C3/C4) provided by Backbone, it predicts the areas where the task object may exist and gives these areas. candidate box (also called candidate area, Proposal). In this embodiment, parallel Header0 is responsible for detecting cars, and its RPN layer predicts candidate boxes where cars may exist; parallel Header1 is responsible for detecting people, and its RPN layer predicts candidate boxes where people may exist, and so on. Repeat.

The basic structure of the RPN layer is shown in Figure 8. Through a 3*3 convolution on C4, the feature map RPNHidden is generated. The RPN layer of each subsequent parallel Header will predict the Proposal from the RPN Hidden. Specifically, the RPN layer of parallel Header0 predicts the coordinates and confidence of the Proposal at each position of the RPN Hidden through two 1*1 convolutions. The higher the confidence level, the greater the probability that this Proposal has an object for this task. For example, the larger the score of a Proposal in parallel Header0, the greater the probability that it has a car. The proposals predicted by each RPN layer need to go through the proposal merging module. The redundant proposals are removed according to the degree of overlap between proposals (this process can be used but is not limited to the NMS algorithm), and the one with the largest score is selected from the remaining K proposals. N (N < K) Proposals are used as candidate areas where objects may exist. As can be seen from Figure 8, these Proposals are inaccurate. On the one hand, they do not necessarily contain the objects of the task, and on the other hand, these boxes are not compact. Therefore, the RPN module is only a rough detection process and requires subsequent RCNN module for subdivision.

When the RPN module returns the coordinates of the Proposal, it does not directly return the absolute value of the coordinates, but returns the coordinates relative to the Anchor. The higher the match between these Anchors and actual objects, the greater the probability that PRN can detect the object. The present invention uses a framework of multiple Headers to design corresponding Anchors according to the scale and aspect ratio of objects in each RPN layer, thereby improving the recall rate of each PRN layer. As shown in Figure 9.

For parallel Header1, it is responsible for the detection of people, and the main shape of people is elongated. Therefore, the Anchor can be designed to be elongated. For parallel Header4, it is responsible for the detection of traffic signs, and the main shape of traffic signs is square. , therefore, the Anchor can be designed as a square.

4. 2D candidate region feature extraction process

The 2D candidate region feature extraction process is mainly implemented by the ROI-ALIGN module in each parallel header in Figure 6. It extracts the features of each Proposal on a feature map provided by Backbone based on the coordinates of the Proposal provided by the PRN layer. come out. The process of ROI-ALIGN is shown in Figure 10.

In this embodiment, features are extracted from Backbone's C4 feature map. Each proposal is in the dark area pointed by the arrow in the C4 area. Interpolation and sampling methods are used in this area to extract a fixed resolution. rate characteristics. Assume that the number of proposals is N and the width and height of the features extracted by ROI-ALIGN are 14, then the size of the features output by ROI-ALIGN is N*14*14*256 (the number of channels of the features extracted by ROI-ALIGN The number of channels is the same as that of C4, 256 channels). These features will be sent to the subsequent RCNN module for segmentation.

5. Subdivision of 2D candidate areas

The subdivision of the 2D candidate area is mainly implemented by the RCNN module of each parallel Header in Figure 6. It further regresses the more compact 2D frame coordinates based on the features of each Proposal extracted by the ROI-ALIGN module, and at the same time performs on this Proposal. Classify and output the confidence that it belongs to each category.

RCNN can be implemented in many forms, one of which is shown in Figure 11. It is analyzed below.

The feature size output by the ROI-ALIGN module is N*14*14*256, which is first processed by the fifth convolution module (Res18-Conv5) of Resnet18 in the RCNN module, and the output feature size is N*7*7* 512, and then processed through a Global Avg Pool (average pooling layer) to average the 7*7 features in each channel of the input features to obtain N*512 features, each of which has a 1*512-dimensional feature The vector represents the characteristics of each Proposal. Next, two fully connected layers FC are used to respectively return the precise coordinates of the frame (output N*4 vector, these 4 numerical tables represent the x/y coordinates of the center point of the frame, the width and height of the frame), and the confidence of the category of the frame. degree (in Header0, the score that this box is Backgroud/Car/Truck/Bus needs to be given). Finally, through the frame merging operation, several frames with the largest scores are selected, and the duplicate frames are removed through the NMS operation, thereby obtaining a compact frame output.

6. 3D inspection process

The 3D detection process is completed by the serial 3D_Header0 in Figure 6. Based on the 2D box provided by the "2D detection" process and the feature map provided by Backbone, it predicts the centroid point coordinates, orientation angle, length, and width of the object inside each 2D box. , advanced 3D information. A possible implementation of serial 3D_Header is shown in Figure 12.

The ROI-ALIGN module extracts the features of the area where each 2D box is located on C4 based on the accurate 2D box provided by the parallel header. Assuming that the number of 2D boxes is M, then the feature size output by the ROI-ALIGN module is M*14 *14*256, which is first processed by the fifth convolution module (Res18-Conv5) of Resnet18, and the output feature size is N*7*7*512, and then processed by a Global Avg Pool (average pooling layer) , average the 7*7 features of each channel in the input features to obtain M*512 features, in which each 1*512-dimensional feature vector represents the characteristics of each 2D box. Next, three fully connected layers FC are used to respectively return the orientation angle of the object in the frame (orientation in the figure, M*1 vector) and center of mass point coordinates (centroid in the figure, M*2 vector. These two values represent the center of mass. x/y coordinates) and length, width and height (dimention in the picture)

7. Mask detection process

The Mask detection process is completed by the serial Mask_Header0 in Figure 6, which predicts the fine mask of the object inside each 2D box based on the 2D box provided by the "2D detection" process and the feature map provided by Backbone. A possible implementation of serial Mask_Header is shown in Figure 13.

The ROI-ALIGN module extracts the features of the area where each 2D box is located on C4 based on the accurate 2D box provided by the parallel header. Assuming that the number of 2D boxes is M, then the feature size output by the ROI-ALIGN module is M*14 *14*256, which is first processed by the fifth convolution module (Res18-Conv5) of Resnet18, and the output feature size is N*7*7*512, and then further convolved through the deconvolution layer Deconv to obtain The features of M*14*14*512 are finally passed through a convolution to obtain the Mask confidence output of M*14*14*1. Each 14*14 matrix in this output represents the confidence of the mask of the object in each 2D box. Each 2D box is equally divided into 14*14 areas. This 14*14 matrix marks each area. The possibility of an object existing. Thresholding this confidence matrix (for example, outputting 1 if it is greater than the threshold 0.5, otherwise outputting 0) can obtain the mask of the object.

8. Keypoint detection process

The Keypoint detection process is completed by the serial Keypoint_Header2 in Figure 6, which predicts the key point coordinates of the objects inside each 2D box based on the 2D box provided by the "2D detection" process and the feature map provided by Backbone. A possible implementation of serial Keyponit_Header is shown in Figure 14.

The ROI-ALIGN module extracts the features of the area where each 2D box is located on C4 based on the accurate 2D box provided by the parallel header. Assuming that the number of 2D boxes is M, then the feature size output by the ROI-ALIGN module is M*14 *14*256, which is first processed by the fifth convolution module (Res18-Conv5) of Resnet18, and the output feature size is N*7*7*512, and then processed through a Global Avg Pool, and each input feature is The 7*7 features of each channel are averaged to obtain M*512 features, in which each 1*512-dimensional feature vector represents the characteristics of each 2D box. Next, a fully connected layer FC is used to respectively return the key point coordinates of the object in the frame (Keypoint in the picture, M*8 vector, these 8 values represent the x/y coordinates of the four corner points of the parking space).

Based on the task division in Table 2, the embodiment of this application also describes in detail the training process of the perception network.

A. Preparation of training data

According to the task division in Table 2, we need to provide annotation data for each task. For example, it is necessary to provide vehicle annotation data for the training of Header0, and label the 2D box and class label of Car/Truck/Bus on the data set; it is necessary to provide human annotation data for the training of Header1, and label Pedestrian/Cyclist on the data set. /Tricycle’s 2D box and class label; you need to provide the traffic light annotation data for Header3, and mark the 2D box and class label of TrafficLight_Red/Yellow/Green/Black on the data set, and so on.

Each type of data only needs to be labeled with a specific type of object, so that targeted collection can be carried out instead of labeling all objects of interest in every picture, thus reducing the cost of data collection and labeling. In addition, preparing data in this way has very flexible scalability. When increasing the detection type of detected objects, you only need to add one or more Headers and provide the annotation data type of the new objects. There is no need to add the original Mark the newly added objects on the data.

In addition, in order to train the 3D detection function in Header0, independent 3D annotation data needs to be provided, and the 3D information of each vehicle (center of mass point coordinates, orientation angle, length, width, and height) is annotated on the data set; in order to train Header0 The Mask detection function in Header2 needs to provide independent Mask annotation data and mark the mask of each vehicle on the data set; in particular, the Parkingslot detection in Header2 needs to detect key points. This task requires the data set to simultaneously label the parking spaces. The 2D box and key points are marked (actually, you only need to mark the key points, and the 2D box of the parking space can be automatically generated based on the coordinates of the key points)

Generally, you only need to provide independent training data for each task, but you can also provide mixed annotation data. For example, you can label the 2D boxes and classes of Car/Truck/Bus/Pedestrian/Cyclist/Tricycle on the data set at the same time. label, so that you can use this data to train the parallel Header of Header0 and Header1 at the same time; you can also mark the 2D/3D/Mask data of Car/Truck/Bus on the data set at the same time, so that this data can train the parallel Header0 at the same time. , Serial3D_Header0 and SerialMask_Header0.

We can assign a label to each image. This label determines which headers in the network this image can be used to train. This will be described in detail in the subsequent training process.

In order to ensure that each Header gets equal training opportunities, the data needs to be balanced. Specifically, the small amount of data is expanded. The expansion method includes but is not limited to copy expansion. The balanced data is randomly scrambled and then sent to the network for training, as shown in Figure 15.

B. Train a full-featured network based on part of the annotated data

In the process of training a full-featured network based on partially annotated data, the Loss of the corresponding Header is calculated according to the type of task to which each input picture belongs, and the gradient is returned through this Loss, and the gradient of the parameters on the corresponding Header and Backbone is calculated. Then adjust the corresponding Header and Backbone according to the gradient. Headers that are not in the annotation task of the current input image will not be adjusted.

If a picture is only labeled with 2D data for a task, then when the picture is sent to the network for training, only the corresponding parallel Header will be trained. As shown in Figure 16.

The current picture only has the 2D frame of the traffic light marked, so during training, the prediction result of the traffic light of this input picture is only obtained through parallel Header3, and compared with the true value, the loss cost of this Header 2D_Loss3 is obtained. Since only one cost loss is generated, the overall cost loss value Final Loss=2D_Loss3. In other words, the input image of the traffic light only flows through Backbone and parallel Header3, and the other Headers do not participate in the training, as shown by the thick arrow without an "X" in Figure 16. After getting the Final Loss, calculate the gradients in parallel Header3 and Backbone in the opposite direction of the thick arrow without "X" in Figure 16, and then use this gradient to update the parameters of Header3 and Backbone to adjust the network so that the network Better predict traffic lights.

If a picture is annotated with 2D data for multiple tasks, then when the picture is sent to the network for training, multiple corresponding parallel headers will be trained. As shown in Figure 17.

The current picture is marked with 2D boxes of people and cars at the same time. Then during training, the prediction results of people and cars in this input picture will be obtained through parallel Header0 and parallel Header1, and compared with the true values, the prediction results of these two Headers will be obtained. The loss cost is 2D_Loss0/2D_Loss1. Since multiple cost losses occur, the overall cost loss value is the average of each loss, that is, Final Loss=(2D_Loss0+2D_Loss1)/2. In other words, the input images labeled with people and cars only flow through Backbone and parallel Header0/1, and the rest of the Headers do not participate in the training, as shown by the thick arrow without an "X" in Figure 17. After obtaining the Final Loss, calculate the gradients in parallel Header0/1 and Backbone in the opposite direction of the thick arrow without "X" in Figure 17, and then use this gradient to update the parameters of Header0/1 and Backbone to achieve network optimization. Adjustments are made to make the network better at predicting people and vehicles.

The training of serial header requires an independent data set for training. It needs to be explained below using the 3D training of the car as an example. As shown in Figure 18.

The input image at this time is labeled with the 2D and 3D true values of the car. During training, the flow of data is shown in the figure by the thick arrow without "X", and the thick arrow with "X" indicates the header where the data cannot flow. When this picture is sent to the network, the 2D and 3D loss functions will be calculated simultaneously, and the final Final Loss=(2D_Loss0+3D_Loss0)/2 is obtained. Then calculate the gradient in the serial 3D Header0, parallel Header0 and Backbone in the opposite direction of the thick arrow without "X", and then use this gradient to update its parameters to adjust the network so that the network can better predict the 2D shape of the car. and 3D.

When each picture is sent to the network for training, only the corresponding Header and Backbone will be adjusted to make the corresponding task perform better. In this process, the performance of other tasks will become worse, but when it is the subsequent turn of the picture for the task, it will become worse. The Header can be adjusted again. Since the training data of all tasks are balanced in advance, each task gets an equal training opportunity, so there will be no overtraining of a certain task. Through this training method, Backbone learns the common features of each task, and each Header learns its task-specific features.

At present, perception needs to implement more and more functions, and using multiple networks to implement single-point functions will lead to excessive overall computing load. The embodiment of the present invention proposes a high-performance and scalable sensing network based on Multi-Header. Each sensing task shares the same backbone network, doubling the amount of calculation and network parameters. Table 3 shows the calculation amount and parameter amount statistics of Single-Header network to implement a single function.

Single-Header-Model@720p GFlops Parameters(M) Vehicle(Car/Truck/Tram) 235.5 17.76 Vehicle+Mask+3D 235.6 32.49 Person(Pedestrian/Cyclist/Tricycle) 235.5 17.76 Person+Mask 235.6 23.0 Motorcycle/Bicycle 235.5 17.76 TrafficLight(Red/Green/Yellow/Black) 235.6 17.76 TrafficSign(Trafficsign/Guideside/Billboard) 235.5 17.75 TrafficCone/TrafficStick/FireHydrant 235.5 17.75 Parkingslot(with keypoint) 235.6 18.98 Full-featured network (multiple single-Header networks) 1648.9 145.49

Table 3 Statistics of calculation amount and parameter amount of Single-Header network

It can be seen from the table that if 8 networks are used to implement all the functions in this embodiment, the total amount of calculation required is 1648.9GFlops, and the total amount of network parameters is 145.49M. This amount of calculation and network parameters is very huge, which will put a lot of pressure on the hardware.

Table 4 shows the calculation amount and parameter amount required to implement all functions of this embodiment using a Multi-Header network.

Multi-Header-Model@720p GFlops Parameters(M) Fully functional network (single Multi-Header network) 236.6 42.16

Table 4 Statistics of calculation amount and parameter amount of Multi-Header network

As can be seen from the table, the calculation amount and parameter amount of the Mulit-Header network are only 1/7 and 1/3 of the Single-Header network, which greatly reduces the calculation consumption.

In addition, the Multi-Header network can achieve the same detection performance as the Single-Header. Table 5 shows the performance comparison between Multi-Header and Single-Header in some categories.

category Single-Header Multi-Header Car 91.7 91.6 Tram 81.8 80.1 Pedestrian 73.6 75.2 Cyclist 81.8 83.3 TrafficLight 98.3 97.5 TrafficSign 95.1 94.5 Parkingslot(point precision/recall) 94.01/80.61 95.17/78.89 3D(mean_orien_err/mecentroid_dist_err) 2.95/6.78 2.88/6.34

Table 5 Comparison of detection performance between Single-Header and Multi-Header networks

As can be seen from the table, the performance of the two is equivalent. Therefore, the Multi-Header network will not cause performance degradation while saving computing power and video memory.

The embodiment of the present invention proposes a high-performance scalable perception network based on Multi-Header, which simultaneously implements different perception tasks (2D/3D/key points/semantic segmentation, etc.) on the same network. Each perception task in the network is shared The same backbone network saves computation and the network structure is easy to expand. You only need to add a Header to add a function. In addition, embodiments of the present invention also propose a method for training a multi-task perception network based on partial annotation data. Each task uses an independent data set, and there is no need to perform full-task annotation on the same picture. The training data of different tasks can be easily balanced. Data from different tasks do not inhibit each other.

As shown in Figure 30, an embodiment of the present invention also provides an object detection method, which method includes:

S3001, receive the input image;

S3002, perform convolution processing on the input image, and output feature maps with different resolutions corresponding to the image;

S3003, according to the feature map, independently detect the task objects in each task for different tasks, and output the 2D box of the area where each task object is located and the confidence level corresponding to each 2D box; wherein, the task object is the object that needs to be detected in the task; the higher the confidence level, the greater the probability that the object corresponding to the task exists in the 2D box corresponding to the confidence level.

Optionally, in one embodiment, S3002 may include the following four steps:

1. Predict the area where the task object is located on one or more feature maps, and output candidate 2D boxes that match the area;

Alternatively, based on the template box (Anchor) of the object corresponding to the task, the area where the task object exists can be predicted on one or more feature maps provided by the backbone network to obtain a candidate area, and output matching the candidate Candidate 2D boxes of the region; wherein, the template box is obtained based on the statistical characteristics of the task object to which it belongs, and the statistical characteristics include the shape and size of the object.

2. According to the area where the task object is located, extract the features of the area where the candidate 2D box is located from a feature map;

3. Perform convolution processing on the features of the area where the candidate 2D box is located to obtain the confidence that the candidate 2D box belongs to each object category; each object category is an object category in the one task;

4. Adjust the coordinates of the 2D box in the candidate area through the neural network so that the adjusted 2D candidate box matches the shape of the actual object better than the candidate 2D box, and select the adjusted 2D box with a confidence greater than the preset threshold. The 2D candidate box serves as the 2D box of the region.

Optionally, the above-mentioned 2D frame may be a rectangular frame.

Optionally, the method also includes:

S3004. Based on the 2D box of the task object to which the task belongs, extract the characteristics of the area where the 2D box is located on one or more feature maps on the backbone network, and classify the task to which the task belongs based on the characteristics of the area where the 2D box is located. The object’s 3D information, Mask information or Keypiont information is used for prediction.

Optionally, the detection of the area where the large object is located can be completed on low-resolution features, and the RPN module is used to detect the area where the small object is located on the high-resolution feature map.

As shown in Figure 31, an embodiment of the present invention also provides a method for training a multi-task perception network based on partial annotation data. The method includes:

S3101. Determine the task to which each picture belongs based on the annotated data type of each picture; wherein each picture is annotated with one or more data types, and the multiple data types are subsets of all data types. One data type The type corresponds to a task;

S3102: Determine the Header required to be trained for each picture according to the task to which each picture belongs;

S3103, calculate the loss value of the Header required for training in each picture;

S3104: For each picture, perform gradient backtransmission through the Header that needs to be trained, and adjust the parameters of the Header that needs to be trained and the backbone network based on the loss value.

Optionally, as shown in the dotted box in Figure 31, before step S3102, the method further includes:

S31020, perform data balancing on pictures belonging to different tasks.

Embodiments of the present invention also provide a multi-Header-based object sensing method. The process of the sensing method provided by the embodiment of the present invention includes two parts: a "reasoning" process and a "training" process. They are introduced below.

1. Perception process:

The flow of the sensing method provided by the embodiment of the present invention is shown in Figure 21.

In step S210, the image is input into the network;

In step S220, the "basic feature generation" process is entered.

In this process, the image is extracted from basic features through the Backbone in Figure 5 to obtain feature maps of different scales. After the basic features are generated, the core process located in the dotted box in Figure 21 will be entered. In the core process, each task has an independent "2D detection" process and optional "3D detection" process, "Mask detection" process and "Keypoint detection" process. The core process is described below.

1. 2D inspection process

The "2D detection" process predicts the 2D box and confidence of each task based on the feature map generated by the "basic feature generation" process. Specifically, the "2D detection" process can be subdivided into the "2D candidate area prediction" process, the "2D candidate area feature extraction" process and the "2D candidate area subdivision" process, as shown in Figure 22.

The "2D candidate area prediction" process is implemented by the RPN module in Figure 5, which predicts areas where the task object may exist on one or more feature maps provided by the "basic feature generation" process, and gives the frames of these areas. (Proposal).

The "2D candidate area feature extraction" process is implemented by the ROI-ALIGN module in Figure 5. According to the Proposal provided by the "2D candidate area prediction" process, each Proposal is generated on a feature map provided by the "Basic feature generation" process. The features of the area are extracted and resized to a fixed size to obtain the features of each Proposal.

The "2D candidate area subdivision" process is implemented by the RCNN module in Figure 5, which uses a neural network to further predict the characteristics of each Proposal, output the confidence of each category belonging to each Proposal, and at the same time calculate the coordinates of the Proposal's 2D box Make adjustments to output a tighter 2D frame.

2. 3D inspection process

The "3D detection" process predicts the 3D information such as the centroid point coordinates, orientation angle, length, width and height of the objects inside each 2D box based on the 2D box provided by the "2D detection" process and the feature map generated by the "basic feature generation" process. Specifically, "3D detection" consists of two sub-processes, as shown in Figure 23.

The sub-process analysis is as follows:

The "2D candidate area feature extraction" process is implemented by the ROI-ALIGN module in Figure 5, which deducts the features of the area where each 2D box is located on a feature map provided by the "basic feature generation" process based on the coordinates of the 2D box. Take it out and resize it to a fixed size to get the characteristics of each 2D box.

The "3D centroid/orientation/length, width, and height prediction" process is implemented by the 3D_Header in Figure 5. It mainly returns the 3D information such as the centroid point coordinates, orientation angle, length, width, and height of the objects inside the 2D box based on the characteristics of each 2D box.

3. Mask detection process

The "Mask Detection" process predicts detailed masks of objects inside each 2D box based on the 2D boxes provided by the "2D Detection" process and the feature maps generated by the "Basic Feature Generation" process. Specifically, "Mask Detection" consists of two sub-processes, as shown in Figure 24.

The sub-process analysis is as follows:

The "2D candidate area feature extraction" process is implemented by the ROI-ALIGN module in Figure 5, which deducts the features of the area where each 2D box is located on a feature map provided by the "basic feature generation" process based on the coordinates of the 2D box. Take it out and resize it to a fixed size to get the characteristics of each 2D box.

The "mask prediction" process is implemented by the Mask_Header in Figure 5, which mainly returns the mask where the object inside the 2D box is located based on the characteristics of each 2D box.

4. Keypoint detection process

The "Keypoint prediction" process predicts the mask of the object inside each 2D box based on the 2D box provided by the "2D detection" process and the feature map generated by the "basic feature generation" process. Specifically, "Keypoint prediction" consists of two sub-processes, as shown in Figure 25.

The sub-process analysis is as follows:

The "2D candidate area feature extraction" process is implemented by the ROI-ALIGN module in Figure 5, which deducts the features of the area where each 2D box is located on a feature map provided by the "basic feature generation" process based on the coordinates of the 2D box. Take it out and resize it to a fixed size to get the characteristics of each 2D box.

The "keypoint coordinate prediction" process is implemented by the Keypoint_Header in Figure 5, which mainly returns the coordinates of the key points of the objects inside the 2D box based on the characteristics of each 2D box.

2. Training process

The training process of the embodiment of the present invention is shown in Figure 26.

The red box part is the core training process. The core training process is introduced below.

1. Data balancing process between tasks

The amount of data for each task is extremely unbalanced. For example, the number of pictures containing people will be much larger than that of traffic signs. In order to allow the headers of each task to get equal training opportunities, the data between tasks must be balanced. Specifically, a small amount of data is expanded. The expansion methods include but are not limited to copy expansion.

2. Loss calculation process based on the task to which the picture belongs

Each picture can belong to one or more tasks according to the data type it is annotated. For example, if a picture only annotates traffic signs, then the image only belongs to the task of traffic signs; if a picture also annotates people and vehicles, , then this kind of picture belongs to both the human and the car tasks. When calculating Loss, only the Loss of the Header corresponding to the task to which the current picture belongs is calculated, and the Loss of other tasks is not calculated. For example, if the currently input training image belongs to the task of people and cars, then only the loss of the header of people and cars will be calculated at this time, and the loss of the rest (such as traffic lights and traffic signs) will not be calculated.

3. Return the gradient process according to the task to which the picture belongs

After calculating Loss, gradient backhaul needs to be performed. At this time, gradient return is only performed through the Header of the current task, and Headers that are not in the current task do not participate in gradient return. In this way, the current Header can be adjusted for the current picture, so that the current Header can better learn the current task. Since the data of each task has been balanced, each Header can get equal training opportunities. Therefore, in this repeated adjustment process, each Header learns task-related features, while Backbone learns features shared between tasks.

The embodiments of this application comprehensively consider the shortcomings of existing methods and propose a high-performance scalable perception network based on Multi-Header, which can simultaneously realize different perception tasks (2D/3D/key points/semantic segmentation, etc.) on the same network. ), each sensing task in this network shares the same backbone network, which significantly saves computational effort. In addition, its network structure is easy to expand, and the functions only need to be expanded by adding one or several Headers.

In addition, embodiments of this application also propose a method for training a multi-task perception network based on partial annotation data. Each task uses an independent data set, and there is no need to annotate the entire task on the same picture. The training data of different tasks can be easily balanced. Data from different tasks do not inhibit each other.

The perception network shown in Figure 5 can be implemented with the structure in Figure 27. Figure 27 is a schematic diagram of an application system of the perception network. As shown in the figure, the perception network 2000 includes at least one processor 2001, at least one memory 2002, At least one communication interface 2003 and at least one display device 2004. The processor 2001, the memory 2002, the display device 2004 and the communication interface 2003 are connected through a communication bus and complete communication with each other.

Communication interface 2003 is used to communicate with other devices or communication networks, such as Ethernet, wireless access network (radioaccess network, RAN), wireless local area networks (WLAN), etc.

The memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (random access memory (RAM)) or other type that can store information and instructions. The dynamic storage device can also be electrically erasable programmable read-only memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disk storage, optical disc storage ( Including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program code in the form of instructions or data structures and can be stored by a computer. any other medium, but not limited to this. The memory can exist independently and be connected to the processor through a bus. Memory can also be integrated with the processor.

The memory 2002 is used to store the application code for executing the above solution, and the processor 2001 controls the execution. The processor 2001 is configured to execute application program codes stored in the memory 2002 .

The code stored in the memory 2002 can execute the Multi-Header based object perception method provided above.

The display device 2004 is used to display the image to be recognized, 2D, 3D, Mask, key points and other information of the object of interest in the image.

The processor 2001 may also use one or more integrated circuits to execute relevant programs to implement the Multi-Header-based object perception method or model training method in the embodiment of the present application.

The processor 2001 may also be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the recommended method of this application can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 2001 . During the implementation process, each step of the training method in the embodiment of the present application can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 2001. The above-mentioned processor 2001 can also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, an off-the-shelf programmable gate array (fieldprogrammable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic. devices, discrete hardware components. Each method, step and module block diagram disclosed in the embodiment of this application can be implemented or executed. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the memory 2002. The processor 2001 reads the information in the memory 2002 and completes the object perception method or model training method in the embodiment of the present application in combination with its hardware.

The communication interface 2003 uses a transceiver device such as but not limited to a transceiver to implement communication between the recommendation device or training device and other devices or communication networks. For example, the picture to be recognized or the training data can be obtained through the communication interface 2003.

A bus may include a path that carries information between various components of the device (eg, memory 2002, processor 2001, communication interface 2003, display device 2004). In a possible embodiment, the processor 2001 specifically performs the following steps: receiving an input picture; performing convolution processing on the input picture, and outputting feature maps with different resolutions corresponding to the picture; according to the backbone network The provided feature map independently detects the object corresponding to each task for different tasks, and outputs the 2D box of the candidate area of the object corresponding to each task and the confidence level corresponding to each 2D box.

In a possible embodiment, after executing the feature map provided by the backbone network, the object corresponding to each task is independently detected for different tasks, and the 2D box of the candidate area of the object corresponding to each task is output. and the confidence level corresponding to each 2D box, the processor 2001 specifically performs the following steps: predict the area where the task object exists on one or more feature maps to obtain a candidate area, and output a matching candidate area candidate 2D box; according to the candidate area obtained by the RPN module, extract the characteristics of the area where the candidate area is located from a feature map; refine the characteristics of the candidate area to obtain the candidate area corresponding to each The confidence of the object category; each object is an object in a corresponding task; the coordinates of the candidate area are adjusted to obtain a second candidate 2D box, and the second 2D candidate box is closer to the actual candidate 2D box than the candidate 2D box. The objects are more closely matched, and the 2D candidate box with a confidence greater than the preset threshold is selected as the 2D box of the candidate area.

In a possible embodiment, when performing prediction on the area where the task object exists on one or more feature maps to obtain a candidate area, and outputting a candidate 2D box matching the candidate area, the processor 2001 specifically Perform the following steps:

Based on the template box (Anchor) of the object of the corresponding task, predict the area where the object of the task exists on one or more feature maps to obtain a candidate area, and output a candidate 2D box that matches the candidate area; where: The template frame is obtained based on the statistical characteristics of the task object to which it belongs, and the statistical characteristics include the shape and size of the object.

In a possible embodiment, the processor 2001 also performs the following steps:

Based on the 2D box of the object corresponding to the task, the features of the object are extracted from one or more feature maps on the backbone network, and the 3D, Mask or Keypiont of the object is predicted.

In a possible embodiment, the detection of candidate areas for large objects is completed on a low-resolution feature map, and the detection of candidate areas for small objects is completed on a high-resolution feature map.

In a possible embodiment, the 2D frame is a rectangular frame.

Optionally, as shown in Figure 28, the structure of the perception network can be implemented by a server. The server can be implemented with the structure in Figure 28. The server 2110 includes at least one processor 2101, at least one memory 2102 and at least one communication interface 2103. . The processor 2101, the memory 2102 and the communication interface 2103 are connected through a communication bus and communicate with each other.

Communication interface 2103 is used to communicate with other devices or communication networks, such as Ethernet, RAN, WLAN, etc.

The memory 2102 can be ROM or other types of static storage devices that can store static information and instructions, RAM or other types of dynamic storage devices that can store information and instructions, or it can be EEPROM, CD-ROM or other optical disk storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), disk storage media or other magnetic storage devices, or can be used to carry or store desired program code in the form of instructions or data structures and can be used by a computer Any other medium for access, but not limited to this. The memory can exist independently and be connected to the processor through a bus. Memory can also be integrated with the processor.

The memory 2102 is used to store the application code for executing the above solution, and the processor 2101 controls the execution. The processor 2101 is used to execute application codes stored in the memory 2102.

The code stored in the memory 2102 can execute the Multi-Header based object perception method provided above.

The processor 2101 may also use one or more integrated circuits to execute related programs to implement the Multi-Header-based object perception method or model training method in the embodiment of the present application.

The processor 2101 may also be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the recommended method of this application can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 2101. During the implementation process, each step of the training method in the embodiment of the present application can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 2101. The above-mentioned processor 2001 can also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. Each method, step and module block diagram disclosed in the embodiment of this application can be implemented or executed. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the memory 2102. The processor 2101 reads the information in the memory 2102 and completes the object perception method or model training method in the embodiment of the present application in combination with its hardware.

The communication interface 2103 uses a transceiver device such as but not limited to a transceiver to implement communication between the recommendation device or training device and other devices or communication networks. For example, the picture to be recognized or the training data can be obtained through the communication interface 2103.

A bus may include a path that carries information between various components of a device (eg, memory 2102, processor 2101, communication interface 2103). In a possible embodiment, the processor 2101 specifically performs the following steps: predict the area where the task object exists on one or more feature maps to obtain a candidate area, and output a candidate 2D box that matches the candidate area. ; According to the candidate area obtained by the RPN module, extract the characteristics of the area where the candidate area is located from a feature map; refine the characteristics of the candidate area to obtain the confidence of each object category corresponding to the candidate area degree; each object is an object in a corresponding task; the coordinates of the candidate area are adjusted to obtain a second candidate 2D box, and the second 2D candidate box is more consistent with the actual object than the candidate 2D box, And select the 2D candidate box with a confidence greater than the preset threshold as the 2D box of the candidate area.

The present application provides a computer-readable medium that stores program code for device execution. The program code includes a method for executing the object sensing method as shown in the embodiment as shown in Figure 21, 22, 23, 24 or 25. related content.

The present application provides a computer-readable medium that stores program code for device execution. The program code includes relevant content for executing the training method of the embodiment shown in Figure 26.

This application provides a computer program product containing instructions. When the computer program product is run on a computer, it causes the computer to execute the relevant content of the sensing method of the embodiment shown in Figure 21, 22, 23, 24 or 25.

The present application provides a computer program product containing instructions. When the computer program product is run on a computer, it causes the computer to execute the relevant content of the training method of the embodiment shown in FIG. 26 .

This application provides a chip. The chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface and executes them as shown in Figure 21, 22, 23, 24, 25 or 26. Relevant content of the sensing method of the embodiment.

The present application provides a chip. The chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface and executes relevant contents of the training method of the embodiment shown in Figure 26.

Optionally, as an implementation manner, the chip may further include a memory, in which instructions are stored, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the The processor is configured to execute relevant contents of the sensing method of the embodiment shown in Figure 21, 22, 23, 24 or 25, or execute relevant contents of the training method of the embodiment shown in Figure 26.

It should be noted that for the sake of simple description, the foregoing method embodiments are expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence. Because in accordance with the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present invention.

In the above embodiments, each embodiment is described with its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

To sum up, the beneficial effects of the embodiments of the present application are summarized as follows:

(1) Each sensing task shares the same backbone network, doubling the amount of calculation; the network structure is easy to expand, and 2D detection types can be expanded by adding one or several Headers. Each parallel Header has independent RPN and RCNN modules, and only needs to detect the objects of the task to which it belongs, so that during the training process, unlabeled objects of other tasks can be avoided. In addition, by using an independent RPN layer, a special Anchor can be customized according to the scale and aspect ratio of the object of each task, thereby increasing the overlap ratio between the Anchor and the object, thereby improving the object recall rate of the RPN layer.

(2) The detection functions of 3D, Mask and Keypoint can be realized in a flexible and convenient way. And these functional extensions share the same backbone network as the 2D part and will not significantly increase the computational load. Use one network to implement multiple functions, easy to implement on the chip

(3) Each task uses an independent data set, and there is no need to label all tasks on the same picture, saving labeling costs. Task expansion is flexible and simple. When adding a new task, you only need to provide the data of the new task. Do not mark new objects on the original data. The training data of different tasks can be easily balanced so that each task gets equal training opportunities and avoids a large amount of data from overwhelming a small amount of data.

In the several embodiments provided in this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or may be Integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention is essentially or contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, It includes several instructions to cause a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention. The aforementioned memory includes: U disk, ROM, RAM, mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable memory, and the memory can include: a flash disk. , ROM, RAM, magnetic disk or CD, etc.

The embodiments of the present application have been introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and the core idea of the present application; at the same time, for Those of ordinary skill in the art will make changes in the specific implementation and application scope based on the ideas of the present invention. In summary, the contents of this description should not be understood as limiting the present invention.

Claims (17)

1. A perceptual network based on multiple heads (Headers), characterized in that the perceptual network includes a backbone network and multiple parallel Headers, and the multiple parallel Headers are connected to the backbone network; The backbone network is used to receive input pictures, perform convolution processing on the input pictures, and output feature maps with multiple different resolutions corresponding to the pictures; Each of the multiple parallel headers is used to detect a task object in a task based on multiple feature maps of different resolutions output by the backbone network, and output a 2D box of the area where the task object is located. And the confidence that each 2D box belongs to each object category; wherein each parallel header completes the detection of task objects of different categories; wherein the task object is an object that needs to be detected in the task; the confidence The higher it is, the greater the probability that the task object corresponding to the task exists in the 2D box corresponding to the confidence level, and each object category is the object category in the task corresponding to the parallel head end; The perception network also includes one or more serial headers, which are connected to one of the parallel headers. The serial headers are used to analyze the task objects of the task according to the characteristics of the area where the 2D box is located. 3D information, Mask information or Keypiont information for prediction. 2. The perceptual network according to claim 1, characterized in that each parallel head end includes a region candidate generation network (RPN) module, a region of interest extraction (ROI-ALIGN) module and a region convolutional neural network ( RCNN) module, the RPN module of each parallel head end is independent of the RPN modules of other parallel head ends; the ROI-ALIGN module of each parallel head end is independent of the ROI-ALIGN modules of other parallel head ends; each parallel head end The RCNN module of the end is independent of the RCNN modules of other parallel heads, where, for each parallel head: The RPN module is used to: predict the area where the task object is located on one or more feature maps provided by the backbone network, and output candidate 2D boxes matching the area; The ROI-ALIGN module is used to: extract the characteristics of the area where the candidate 2D box is located from a feature map provided by the backbone network based on the area predicted by the RPN module; The RCNN module is used to: perform convolution processing on the features of the area where the candidate 2D box is located through a neural network to obtain the confidence that the candidate 2D box belongs to each object category; each object category is the parallel head end The corresponding object category in the task; adjust the coordinates of the candidate 2D box through the neural network so that the adjusted 2D candidate box matches the shape of the actual object better than the candidate 2D box, and select a confidence level greater than the preset The adjusted 2D candidate box of the threshold is used as the 2D box of the region. 3. The perceptual network according to claim 1 or 2, characterized in that the 2D box is a rectangular box. 4. The perception network according to claim 2, characterized in that, The RPN module is used to: based on the template box (Anchor) of the object corresponding to the task, predict the area where the task object exists on one or more feature maps provided by the backbone network to obtain the candidate area, and output the matching results. The candidate 2D box of the candidate area; wherein the template box is obtained based on the statistical characteristics of the task object to which it belongs, and the statistical characteristics include the shape and size of the object. 5. The perception network according to claim 1, characterized in that, The serial header is specifically used to: utilize the 2D box of the task object belonging to the task provided by the parallel header it is connected to, and extract the region where the 2D box is located on one or more feature maps on the backbone network. Features: predict the 3D information, Mask information or Keypiont information of the task object belonging to the task according to the characteristics of the area where the 2D frame is located. 6. The perceptual network according to claim 4, characterized in that the RPN module is used to predict areas where objects of different sizes are located on feature maps of different resolutions. 7. The perception network according to claim 6, characterized in that the RPN module is used to complete the detection of the area where the large object is located on the low-resolution feature map, and to complete the detection of the area where the small object is located on the high-resolution feature map. Area detection. 8. An object detection method, characterized in that it is applied to a perceptual network. The perceptual network includes a backbone network and multiple parallel Headers. The perceptual network also includes one or more serial Headers, and the serial Header and One of the parallel Header connections; the method includes: The backbone network receives an input picture; performs convolution processing on the input picture, and outputs feature maps with multiple different resolutions corresponding to the picture; According to the multiple feature maps of different resolutions output by the backbone network, each parallel headend (Header) detects the task object in a task, outputs the 2D box of the area where the task object is located and the location of each 2D box. Confidence of each object category; wherein each parallel headend (Header) completes the detection of task objects of different categories; the task object is an object of the category that needs to be detected in the task; the higher the confidence, the The greater the probability that the task object corresponding to the task exists in the 2D box corresponding to the confidence level, the each object category is the object category in the task corresponding to the parallel head end; The serial header predicts the 3D information, Mask information or Keypiont information of the task object belonging to the task based on the characteristics of the area where the 2D box is located. 9. The object detection method according to claim 8, characterized in that, according to the feature map, the task objects in each task are independently detected for different tasks, and the 2D image of the area where each task object is located is output. Frames and the confidence corresponding to each 2D frame, including: Predict the area where the task object is located on one or more feature maps, and output candidate 2D boxes that match the area; According to the area where the task object is located, extract the features of the area where the candidate 2D box is located from a feature map; Perform convolution processing on the features of the area where the candidate 2D box is located to obtain the confidence that the candidate 2D box belongs to each object category; each object category is an object category in the one task; The coordinates of the candidate 2D box are adjusted through the neural network so that the adjusted 2D candidate box matches the shape of the actual object better than the candidate 2D box, and the adjusted 2D candidate box with a confidence greater than the preset threshold is selected. as a 2D box for the region. 10. The object detection method according to claim 9, wherein the 2D frame is a rectangular frame. 11. The object detection method according to claim 9, characterized in that, predicting the area where the task object is located on one or more feature maps, and outputting the candidate 2D box matching the area is: Based on the template frame (Anchor) of the object corresponding to the task, predict the area where the task object exists on one or more feature maps provided by the backbone network to obtain candidate areas, and output candidates matching the candidate areas. 2D box; wherein, the template box is obtained based on the statistical characteristics of the task object to which it belongs, and the statistical characteristics include the shape and size of the object. 12. The object detection method according to any one of claims 8-11, characterized in that the method further includes: Based on the 2D box of the task object belonging to the task, the features of the area where the 2D box is located are extracted from one or more feature maps on the backbone network. 13. The object detection method according to claim 8, characterized in that the detection of the area where the large object is located is completed on the low-resolution feature, and the detection of the area where the small object is located is completed on the high-resolution feature map. 14. A method for training a multi-task perception network based on partially annotated data, characterized in that it is applied to the perception network according to any one of claims 1 to 7, and the method includes: Determine the task to which each sample picture belongs according to the labeled data type of each sample picture; wherein, each sample picture is labeled with one or more data types, and the plurality of data types are subsets of all data types , each data type among all the data types corresponds to a task; According to the task to which each sample picture belongs, determine the Header that needs to be trained for each sample picture; Calculate the loss value of the Header required for training for each sample image; For each sample picture, gradient backhaul is performed through the Header required to be trained, and the parameters of the Header required to be trained and the backbone network are adjusted based on the loss value, and the remaining Headers are not involved. In training. 15. The method of training a multi-task perception network according to claim 14, characterized in that before calculating the loss value of the Header required for training for each sample picture, the method further includes: Perform data balancing on sample images belonging to different tasks. 16. A device for training a multi-task perception network based on partially annotated data, characterized in that it is applied to the perception network according to any one of claims 1 to 7, and the device includes: A task determination module, configured to determine the task to which each sample picture belongs based on the annotated data type of each sample picture; wherein each sample picture is annotated with one or more data types, and the plurality of data types are A subset of all data types, each of which corresponds to a task; A Header determination module is used to determine the Header required for training of each sample picture according to the task to which each sample picture belongs; A loss value calculation module, for each sample picture, used to calculate the loss value of the Header determined by the Header decision module; The adjustment module performs gradient return for each sample picture by calculating the Header determined by the Header determination module, and adjusts the required training Header and backbone network based on the loss value obtained by the loss value calculation module. parameters, the rest of the Headers do not participate in the training. 17. The device for training a multi-task perception network based on partial annotation data as claimed in claim 16, characterized in that the device further includes: The data balancing module is used to balance the data of sample images belonging to different tasks.