CN115880483A - Neural network model distillation method, device and storage medium - Google Patents

Abstract

The present disclosure relates to a neural network model distillation method, device and storage medium. The neural network model distillation method includes: determining the target image, and obtaining the object condition feature of the target image, the object condition feature is the feature information of the target object that is manually input; performing feature detection on the target image based on the teacher model, and obtaining the teacher model feature map, And based on the initial student model, perform feature detection on the target image to obtain the student model feature map; determine the correlation between the object features at each pixel position in the teacher model feature map and the object condition features, and based on the correlation, determine the attention of the teacher model Force distribution; based on the attention distribution of the teacher model and the feature difference between the feature map of the teacher model and the feature map of the student model, the initial student model is trained to obtain the target student model. A student model with higher object detection accuracy can be obtained through the present disclosure.

Description

A neural network model distillation method, device and storage medium

technical field

The present disclosure relates to the technical field of image recognition, in particular to a neural network model distillation method, device and storage medium.

Background technique

Knowledge distillation is a method to improve network performance. It uses a pre-trained large model (teacher model) to teach the model to be trained (student model) to improve model performance. In related technologies, knowledge distillation is also widely used in the field of image recognition to improve the sampling and detection effects of images.

In related technologies, the easily identifiable regions learned by the model network do not necessarily match human observation and cognition, which limits the detection performance of the model.

Contents of the invention

In order to overcome the problems existing in related technologies, the present disclosure provides a neural network model distillation method, device and storage medium.

According to a first aspect of an embodiment of the present disclosure, a neural network model distillation method is provided, including:

Determining the target image, and obtaining the object condition feature of the target image, the object condition feature is manually input and characterizes feature information of the target object; performing feature detection on the target image based on the teacher model to obtain a teacher model feature map, and Perform feature detection on the target image based on the initial student model to obtain a student model feature map; determine the correlation between the object feature at each pixel position in the teacher model feature map and the object condition feature, and based on the correlation Determine the attention distribution of the teacher model; based on the attention distribution of the teacher model, and the feature difference between the teacher model feature map and the student model feature map, the initial student model is Train to get the target student model.

In one embodiment, determining the correlation between the object feature at each pixel position in the teacher model feature map and the object condition feature includes: decomposing the object feature at each pixel position in the teacher model feature map into A plurality of first sub-feature spaces; for each pixel position in the feature map of the teacher model, perform feature weighting on the plurality of first sub-feature spaces based on different weights, to obtain object features after feature weighting; determine the The correlation between the feature-weighted object feature and the object condition feature, and according to the correlation between the feature-weighted object feature and the object condition feature, a plurality of first pixel positions for each pixel position is obtained The first sub-feature space correlation of a sub-feature space; for each pixel position in the feature map of the teacher model, normalize the first sub-feature space correlations of the plurality of first sub-feature spaces, The correlation between the object feature corresponding to each pixel position and the object condition feature is obtained.

In one embodiment, based on the attention distribution of the teacher model and the degree of feature difference between the feature map of the teacher model and the feature map of the student model, the initial student model is trained to obtain a target student model, Including: optimizing the attention distribution of the teacher model; based on the optimized attention distribution and the feature difference between the teacher model feature map and the student model feature map, the initial student model is Train to get the target student model.

In one embodiment, optimizing the attention distribution of the teacher model includes: decomposing the object features of each pixel position in the feature map of the teacher model into a plurality of second sub-feature spaces; Attention distribution, performing feature weighting on the plurality of second sub-feature spaces; controlling the teacher model to perform target detection based on the weighted features, and adjusting the attention of the teacher model based on the target detection result and object condition characteristics distribution, so that the teacher model can obtain target detection results consistent with the object condition characteristics.

In one embodiment, the decomposition methods used in the first sub-feature space and the second sub-feature space are different; wherein, the decomposition method includes a key-value decomposition method and a content decomposition method.

In an embodiment, obtaining a target detection result consistent with the object condition feature includes: obtaining a target detection result consistent with one or a combination of the category feature, position feature, and scale feature corresponding to the object condition feature.

In one embodiment, based on the correlation, determining the attention distribution of the teacher model includes: based on the correlation size relationship between the object feature at each pixel position in the feature map of the teacher model and the object condition feature , assigning weights to each pixel position in the feature map of the teacher model, and obtaining the attention distribution of the teacher model according to the assigned weights; wherein, for each pixel position in the feature map of the teacher model, the correlation The higher the pixel position, the greater the corresponding weight.

In one embodiment, performing feature detection on the target image based on the teacher model to obtain a feature map of the teacher model, and performing feature detection on the target image based on the initial student model to obtain the feature map of the student model, including: using the teacher model performing multi-scale feature detection on the target image to obtain a teacher model feature map including multi-scale features; and performing multi-scale feature detection on the target image through the student model to obtain a student model feature map including multi-scale features.

According to a second aspect of an embodiment of the present disclosure, a neural network model distillation device is provided, including:

A determination unit, configured to determine the target image, and to determine the correlation between the object feature at each pixel position in the teacher model feature map and the object condition feature, and based on the correlation, determine the attention of the teacher model Force distribution; acquisition unit, used to acquire the object condition feature of the target image, the object condition feature is manually input and characterizes the feature information of the target object; detection unit, used to perform the target image based on the teacher model feature detection, to obtain a teacher model feature map, and perform feature detection on the target image based on the initial student model, to obtain a student model feature map; a processing unit for attention distribution based on the teacher model, and the teacher model features The degree of feature difference between the graph and the feature map of the student model, the initial student model is trained to obtain the target student model.

In one embodiment, the determination unit determines the correlation between the object feature at each pixel position in the teacher model feature map and the object condition feature in the following manner: the pixel position in the teacher model feature map The object features of the object feature are decomposed into multiple first sub-feature spaces; for each pixel position in the feature map of the teacher model, feature weighting is performed on the multiple first sub-feature spaces based on different weights, and the feature weighted Object features; determining the correlation between the feature-weighted object features and the object condition features, and according to the correlation between the feature-weighted object features and the object condition features, for each The first sub-feature space correlation of multiple first sub-feature spaces of the pixel position; for each pixel position in the feature map of the teacher model, the first sub-feature space correlation of the multiple first sub-feature spaces, Perform normalization processing to obtain the correlation between the object feature corresponding to each pixel position and the object condition feature.

In one embodiment, the processing unit uses the following method to process the initial student model based on the attention distribution of the teacher model and the feature difference between the teacher model feature map and the student model feature map. Perform training to obtain the target student model: optimize the attention distribution of the teacher model; based on the optimized attention distribution, and the feature difference between the teacher model feature map and the student model feature map, the The initial student model is trained to obtain a target student model.

In one embodiment, the processing unit optimizes the attention distribution of the teacher model in the following manner: decompose the object features at each pixel position in the feature map of the teacher model into multiple second sub-feature spaces; Based on the attention distribution of the teacher model, perform feature weighting on the plurality of second sub-feature spaces; control the teacher model to perform target detection based on the weighted features, and adjust the target detection results and object condition characteristics based on the target detection results. The attention distribution of the teacher model is adjusted so that the teacher model can obtain a target detection result that is consistent with the conditional characteristics of the object.

In an implementation manner, the decomposition methods used in the first sub-feature space are different from those used in the second sub-feature space; wherein, the decomposition methods include a key-value decomposition method and a content decomposition method.

In one embodiment, the processing unit obtains a target detection result that is consistent with the conditional feature of the object in the following manner: Obtaining a result that is consistent with one or a combination of the category feature, position feature, and scale feature corresponding to the object conditional feature target detection results.

In one embodiment, the determination unit determines the attention distribution of the teacher model based on the correlation in the following manner: based on the relationship between the object feature at each pixel position in the teacher model feature map and the object condition feature Correlation size relationship between, carry out weight distribution for each pixel position in the feature map of the teacher model, and according to the assigned weight, obtain the attention distribution of the teacher model; wherein, the first weight corresponding to the first correlation greater than the second weight corresponding to the second correlation, the first correlation is greater than the second correlation.

In one embodiment, the detection unit performs feature detection on the target image based on the teacher model in the following manner to obtain a feature map of the teacher model, and performs feature detection on the target image based on the initial student model to obtain the feature map of the student model : controlling the teacher model to perform multi-scale feature detection on the target image to obtain a teacher model feature map including multi-scale features; and controlling the student model to perform multi-scale feature detection on the target image to obtain multi-scale features The feature map of the student model of .

According to a third aspect of an embodiment of the present disclosure, a neural network model distillation device is provided, including:

processor; memory for storing instructions executable by the processor;

Wherein, the processor is configured to: execute the neural network model distillation method described in the first aspect or any one implementation manner of the first aspect.

According to the fourth aspect of the embodiments of the present disclosure, there is provided a storage medium, the storage medium stores instructions, and when the instructions in the storage medium are executed by the processor, the processor can execute the first aspect or the first aspect The neural network model distillation method described in any one of the implementations.

The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: determine the target image, and acquire object condition characteristics of the target image. The object condition feature is manually input and characterizes the feature information of the target object. Therefore, based on the correlation between the object feature at each pixel position in the teacher model feature map and the object condition feature, the attention distribution of the obtained teacher model is determined by It is obtained by manually inputting object condition characteristics. Therefore, using this attention distribution for target detection can obtain target detection results that are more in line with human observation perspectives, and improve the target detection accuracy of the student model to a certain extent.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

Fig. 1 is a flow chart of a neural network model distillation method according to an exemplary embodiment.

Fig. 2 is a flow chart showing another neural network model distillation method according to an exemplary embodiment.

Fig. 3 is a flow chart of a method for determining the correlation between object features and object condition features at each pixel position in the teacher model feature map according to an exemplary embodiment.

Fig. 4 is a flow chart showing another neural network model distillation method according to an exemplary embodiment.

Fig. 5 is a flowchart showing a method for training an initial student model according to an exemplary embodiment.

Fig. 6 is a flowchart of a method for optimizing the attention distribution of a teacher model according to an exemplary embodiment.

Fig. 7 is a flowchart of another method for optimizing the attention distribution of a teacher model according to an exemplary embodiment.

Fig. 8 is a schematic flowchart of training an initial student model according to an exemplary embodiment.

Fig. 9 is a block diagram of a neural network model distillation device according to an exemplary embodiment.

Fig. 10 is a schematic diagram of an electronic device according to an exemplary embodiment.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as recited in the appended claims.

In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of the disclosure. The embodiments described below by referring to the figures are exemplary and are intended to explain the present disclosure and should not be construed as limiting the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present disclosure. Embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings.

In recent years, artificial intelligence-based computer vision, deep learning, machine learning, image processing, image recognition and other technologies have made important progress. Artificial Intelligence (AI) is an emerging science and technology that researches and develops theories, methods, technologies and application systems for simulating and extending human intelligence. The subject of artificial intelligence is a comprehensive subject that involves many technologies such as chips, big data, cloud computing, Internet of Things, distributed storage, deep learning, machine learning, and neural networks. As an important branch of artificial intelligence, computer vision is specifically to allow machines to recognize the world. Computer vision technology usually includes face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian detection, etc. Recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, three-dimensional reconstruction, virtual reality, augmented reality, simultaneous localization and map construction (SLAM), computational photography, Robot navigation and positioning technology. With the research and progress of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, face access, face attendance, logistics management, warehouse management, robots , smart marketing, computational photography, mobile imaging, cloud services, smart home, wearable devices, unmanned driving, automatic driving, smart medical care, face payment, face unlock, fingerprint unlock, witness verification, smart screen, smart TV, Cameras, mobile Internet, webcasting, beauty, cosmetics, medical beauty, intelligent temperature measurement and other fields.

With the development of machine learning technology, it has become more and more common to use models for image processing or recognition. Generally speaking, larger, deeper and more complex models have better fitting effects and better predictive capabilities, but at the same time, their computational efficiency is low, time-consuming, and the number of parameters is large, which is not conducive to applications such as mobile terminals and chip terminals. layer deployment. Although the simple model has weak fitting ability, it has higher computational efficiency and fewer parameters, which makes it easier to deploy. As an important model compression method, knowledge distillation can transfer the knowledge in the complex model (teacher, also known as the teacher model) to the simple model (student, also known as the student model) to make the fitting of the student model The ability can approach or even exceed the teacher model, so that similar prediction effects can be obtained with less time and space complexity.

In related technologies, knowledge distillation systems are mostly designed for classification problems, which cannot be well applied to target detection and instance segmentation. Moreover, the knowledge distillation system in related technologies needs to rely on the prediction results of the network itself, which makes the recognizable areas learned by the network not necessarily consistent with human observation and cognition, which in turn affects the target detection results of the model to a certain extent.

An embodiment of the present disclosure provides a neural network model distillation method, which makes the attention distribution obtained by the teacher model more suitable for human observation by manually inputting the object condition characteristics used to represent the characteristic information of the target object. Through the attention distribution and the feature difference between the feature map of the teacher model and the feature map of the student model, training the initial student model can improve the training effect of the student model, thereby improving the target detection ability of the student model.

Fig. 1 is a flow chart of a neural network model distillation method according to an exemplary embodiment, as shown in Fig. 1 , including the following steps.

In step S11, the target image is determined, and the object condition characteristics of the target image are acquired.

Among them, the object condition feature can be understood as the feature information that is manually input and characterizes the target object. And, it can be understood that the target object is an object existing in the target image.

In step S12, perform feature detection on the target image based on the teacher model to obtain a feature map of the teacher model, and perform feature detection on the target image based on the initial student model to obtain a feature map of the student model.

In the embodiment of the present disclosure, the initial student model can be understood as a student model to be trained. In one example, the teacher model feature map includes the object features obtained by the teacher model for each pixel position of the target image, and the student model feature map includes the initial student model for each pixel position of the target image. The obtained object characteristics.

In step S13, the correlation between the object feature and the object condition feature at each pixel position in the teacher model feature map is determined, and based on the correlation, the attention distribution of the teacher model is determined.

For example, the correlation between object features and object conditional features can be understood as the degree of similarity between the feature sequence corresponding to the object feature and the feature sequence corresponding to the object conditional feature. The attention distribution of the teacher model can be understood as the detection weight assigned to each pixel position by the teacher model in the process of object detection.

In step S14, based on the attention distribution of the teacher model and the feature difference between the feature map of the teacher model and the feature map of the student model, the initial student model is trained to obtain the target student model.

Wherein, the target student model can be understood as a trained student model.

In the embodiment of the present disclosure, the index direction for determining the attention distribution is provided for the teacher model by manually inputting prior features (ie, object condition features). The attention distribution determined by the teacher model in this way is more in line with the human observation perspective. Further, training the initial student model with the attention distribution can improve the recognition accuracy of the student model.

In an example, training the initial student model to obtain the target student model may be to determine the degree of feature difference for each pixel position between the feature map of the teacher model and the feature map of the student model. Further, the attention distribution of the teacher model is used as the weight assigned when calculating the distillation loss between the teacher model and the initial student model, and the distillation loss between the teacher model and the initial student model is calculated. For example, calculating the distillation loss between the teacher model and the initial student model may be calculating the mean square error between the feature map of the teacher model and the feature map of the student model. Introducing the distillation loss between the teacher model and the initial student model into the initial student model, as an auxiliary parameter for the initial student model for object detection, can make the object detection results of the initial student model close or the same as those of the teacher model. That is, the target student model that has been trained is obtained.

In one example, a Feature Pyramid Network (FPN for short) may be introduced into the teacher model and the student model. On this basis, the multi-scale feature detection of the target image can be carried out through the teacher model, and the teacher model feature map including multi-scale features can be obtained, and the multi-scale feature detection of the target image can be obtained through the student model, and the student model including multi-scale features can be obtained feature map.

Fig. 2 is a flow chart of another neural network model distillation method shown according to an exemplary embodiment. As shown in Fig. 2 , the implementation process of step S21, step S23 and step S24 in the embodiment of the present disclosure is the same as that in Fig. 1 The execution methods of step S11 , step S13 and step S14 shown are similar and will not be repeated here.

In step S22, the multi-scale feature detection is performed on the target image through the teacher model to obtain the teacher model feature map including multi-scale features, and the multi-scale feature detection is performed on the target image through the student model to obtain the student model features including multi-scale features picture.

The neural network model distillation method provided by the embodiments of the present disclosure can perform feature detection on a target image at multiple different scales (resolutions) through a feature pyramid network. Among them, for large-scale (lower resolution) images, the relevant features of small objects in the target image can be more accurately identified, and for small-scale (higher resolution) images, the features of large objects in the target image can be more accurately identified relevant features. This method can meet the feature detection requirements for objects of different scales in the target image, thereby improving the target detection accuracy of the teacher model and/or student model.

Generally, for the same image, there are various feature regions that contribute to target detection, such as foreground regions, target object edge regions, overlapping targets, and interrelationships between targets.

The object detection distillation method provided by the embodiments of the present disclosure can decompose the feature map of the teacher model, and then map the decomposed feature map of the teacher model to multiple sub-feature spaces through a multi-head attention network. Furthermore, for each sub-feature space, assigning different weights enables the teacher model to identify features that are helpful for target detection in different regions.

In the present disclosure, for the convenience of description, the sub-feature space obtained by decomposing the feature map of the teacher model in the process of calculating the correlation between the object feature and the object condition feature at each pixel position of the target image is called the first sub-feature space.

Fig. 3 is a flow chart of a method for determining the correlation between object features and object condition features at each pixel position in the teacher model feature map according to an exemplary embodiment, as shown in Fig. 3 , including the following steps.

In step S31, the object features at each pixel position in the teacher model feature map are decomposed into a plurality of first sub-feature spaces.

In step S32 , for each pixel position in the feature map of the teacher model, feature weighting is performed on a plurality of first sub-feature spaces based on different weights respectively, to obtain object features after feature weighting.

In step S33, the correlation between the feature-weighted object feature and the object condition feature is determined, and according to the correlation between the feature-weighted object feature and the object condition feature, a plurality of first A first sub-feature space correlation of a sub-feature space.

In step S34, for each pixel position in the feature map of the teacher model, the first sub-feature space correlation of multiple first sub-feature spaces is normalized to obtain the object feature and object condition corresponding to each pixel position Correlations between features.

As an example, assume that the number of pixel rows in the feature map of the teacher model is W, the number of pixel columns is H, and the feature dimension corresponding to each pixel position is C. When decomposing the teacher model feature map into multiple first sub-feature spaces, and performing feature weighting on the multiple sub-feature spaces with different weights, the teacher model feature map can be decomposed into the number of pixel rows W, the number of pixel columns H, and The feature dimension corresponding to each pixel position is N first sub-feature spaces (M*N=C). Wherein, the number N of sub-feature spaces can be any value, that is, the feature map of the teacher model can be decomposed into any number of first sub-feature spaces. Further, for each first sub-feature space in the N first sub-feature spaces, different weights are used for weighting. For example, for the first sub-feature space A, the foreground region of the image may be assigned a higher weight than other regions except the foreground region. For the first sub-feature space B (different from the first sub-feature space A), other areas of the image except the foreground area can be assigned higher weights than the foreground area of the image, so as to set different weights for different first sub-feature spaces . Of course, multiple different ways may be used to perform feature weighting for different first sub-feature spaces, which is not specifically limited in the present disclosure.

In the neural network model distillation method provided by the embodiments of the present disclosure, by assigning different weights to different first sub-feature spaces, the teacher model respectively uses different regions in the target image as the main identification regions. Further, by normalizing the attention distribution obtained for different sub-feature spaces, the teacher model can identify the object features that contribute to the target detection in each region of the target image, thereby improving the teacher model's identification Accuracy.

In one example, the correlation of each pixel position in the teacher model feature map can be used to assign weights to the correlation of each pixel position in the teacher model feature map, and then determine the attention distribution of the teacher model.

Fig. 4 is a flowchart of another neural network model distillation method according to an exemplary embodiment. As shown in Fig. 4, the implementation process of step S41, step S42 and step S44 in the embodiment of the present disclosure is the same as that in Fig. 1 The execution methods of step S11 , step S12 and step S14 shown are similar and will not be repeated here.

In step S43, determine the correlation between the object feature and the object condition feature of each pixel position in the teacher model feature map, and based on the correlation size between the object feature and the object condition feature of each pixel position in the teacher model feature map relationship, assign weights to each pixel position in the feature map of the teacher model, and determine the attention distribution of the teacher model according to the assigned weights.

Among them, for each pixel position in the feature map of the teacher model, the pixel position with greater correlation has a greater corresponding weight. For example, based on the correlation size relationship between each pixel position in the teacher model feature map, when performing weight distribution for the correlation of each pixel position in the teacher model feature map, it can be for the larger correlation value. Larger pixel locations are assigned higher weights, and pixel locations with smaller correlation values are assigned lower weights.

In one example, if the two different correlations between pixel positions are referred to as the first correlation and the second correlation, where the first correlation is greater than the second correlation, then the first correlation can be assigned a corresponding , assigning a corresponding second weight to the second correlation, and the first weight corresponding to the first correlation is greater than the second weight corresponding to the second correlation.

In one embodiment, when assigning weights for the correlation of each pixel position in the teacher model feature map through the correlation size relationship between the pixel positions in the teacher model feature map, the distribution of each pixel position in the teacher model feature map The sum of the weights is 1.

In one example, the attention distribution of the teacher model can be optimized, and based on the optimized attention distribution, the initial student model can be trained to obtain the target student model, so as to further improve the identification accuracy of the target student model.

Fig. 5 is a flow chart of a method for training an initial student model according to an exemplary embodiment, as shown in Fig. 5 , including the following steps.

In step S51, the attention distribution of the teacher model is optimized.

In step S52, based on the optimized attention distribution and the degree of feature difference between the feature map of the teacher model and the feature map of the student model, the initial student model is trained to obtain the target student model.

The neural network model distillation method provided by the embodiments of the present disclosure can optimize the attention distribution of the teacher model. Using the optimized attention distribution to train the initial student model to obtain the target student model can further improve the identification accuracy of the target student model.

In one example, the teacher model can be controlled to perform object detection on the target image, and then the attention distribution of the teacher model can be adjusted through the object detection results of the teacher model and the object condition characteristics, so as to realize the optimization of the attention distribution.

In the present disclosure, for the convenience of description, the sub-feature space obtained by decomposing the feature map of the teacher model in the process of optimizing the attention distribution of the teacher model is called the second sub-feature space.

Fig. 6 is a flow chart of a method for optimizing the attention distribution of a teacher model according to an exemplary embodiment, as shown in Fig. 6 , including the following steps.

In step S61, the object features at each pixel position in the teacher model feature map are decomposed into a plurality of second sub-feature spaces.

In step S62, based on the attention distribution of the teacher model, perform feature weighting on multiple second sub-feature spaces.

In step S63, the teacher model is controlled to perform target detection based on the weighted features, and based on the target detection result and object condition characteristics, the attention distribution of the teacher model is adjusted so that the teacher model obtains a target detection result consistent with the object condition characteristics.

In one example, to perform target detection based on weighted features, multiple second sub-feature spaces may be aggregated into feature vectors used to characterize object features of the target image, and the teacher model may perform Target detection, get the target detection result.

In one embodiment, a multi-layer perceptron (Multi Layer Percepron, MLP) may be used to perform target detection on the feature vectors obtained after weighting and aggregation, to obtain a target detection result.

In the embodiment of the present disclosure, the first sub-feature space and the second sub-feature space may be obtained by decomposing the teacher model feature map through different decomposition methods. Wherein, for example, the decomposition method used may include a key-value decomposition method and a content decomposition method.

In one embodiment, the feature map of the teacher model may be decomposed by key-value decomposition to obtain multiple first sub-feature spaces, and the feature map of the teacher model may be decomposed by content decomposition to obtain multiple second feature spaces. Due to the feature map of the same teacher model, different decomposition methods can be decomposed to obtain different features for object detection. Therefore, this method can enrich feature samples and improve the identification accuracy of the teacher model. Of course, the present disclosure can also perform feature decomposition on the teacher model feature map through other decomposition methods, and the present disclosure does not specifically limit the decomposition method used.

In the embodiment of the present disclosure, the object condition features manually input into the teacher model may include one or a combination of category features, location features, and scale features of the target object. In an example, obtaining a target detection result consistent with object conditional features may be obtaining a target detection result consistent with one or a combination of category features, position features, and scale features corresponding to the object conditional features.

Fig. 7 is a flow chart of another method for optimizing the attention distribution of the teacher model according to an exemplary embodiment. The implementation process of step S71 and step S72 in the embodiment of the present disclosure is the same as that shown in Fig. 6 The execution methods of step S61 and step S62 are similar and will not be repeated here.

In step S73, control the teacher model to perform target detection based on the weighted features, and adjust the attention distribution of the teacher model based on the target detection results and object condition features, so that the teacher model can obtain the category features corresponding to the object condition features, Target detection results consistent with one or a combination of location features and scale features.

In the embodiment of the present disclosure, the position feature of the target object may be coded by way of position embedding (PositionEmbedding) and input into the teacher model. The category features of the target object and/or the scale features of the target object can be encoded by one-hot vectors and input to the teacher model.

In one embodiment, the position feature of the input target object may be the coordinates of the center point of the input target object in the target image. Of course, it is also possible to input the coordinates of any point within the area where the target object is located.

In another embodiment, inputting the category features of the target object may be based on human observation to determine the object category of the target object. Further, a feature sequence that matches the object category of the target object can be found in the category dataset preset by the teacher model, and the feature sequence can be used as the category feature of the target object and input into the teacher model.

In yet another embodiment, the scale feature of the input target object may be a predefined object scale range for the teacher model. For example, a target object with an object scale smaller than 64-bit pixels can be defined as a small object, an object with an object scale within the range of 64-bit pixels to 256-bit pixels can be defined as a medium object, and an object with an object scale larger than 256-bit pixels Defined as large objects. Further, the scale of the target object is determined according to the results of human observation, and the feature sequence used to characterize the scale range of the scale is input into the teacher model.

In the embodiment of the present disclosure, the teacher model can output a target detection result that is consistent with the condition characteristics of the object by adjusting related parameters of the teacher model. In one embodiment, a corresponding loss function can be set for the teacher model, and the teacher model can adaptively adjust the attention distribution through the loss function.

In the neural network model distillation method provided by the embodiments of the present disclosure, the method of adjusting the attention distribution of the teacher model through a loss function can be understood as assigning auxiliary tasks to the teacher model. As an example, it can be understood that the following auxiliary tasks are assigned to the teacher model.

Auxiliary task 1: Determine whether there is a target object in the target image.

Auxiliary task 2: Determine whether the location area marked by the teacher model matches the area where the target object is located in the target image.

In one embodiment, the teacher model may be controlled to perform target detection, and determine whether there is a target object in the target image that matches the type feature input to the teacher model. For example, the output of the teacher model can be 1 if it exists, and the output of the teacher model can be 0 if it does not exist. Of course, the output of the teacher model can also be a floating-point number between 0 and 1, and in the case that the output value of the teacher model is a floating-point number, the output result that the output value is close to 1 can be determined as the existence of the object, and the output Outputs with values approaching 0 determine the absence of the object. For example, if the output result does not match the result represented by the category feature, adjust the attention distribution of the teacher model in the direction of decreasing the output value of the loss function of the teacher model until the target detection result of the teacher model is consistent with the representation of the category feature consistent.

In another embodiment, when it is determined that there is a target object in the target image, it can be judged that the position area of the target object marked by the teacher model in the target image is different from the location feature and scale feature represented in the target image. Whether the location area matches. For example, the image position represented by the position feature of the target object is determined as the random point coordinates of the target object in the area where the target object is located, and the teacher model is controlled to mark the position area where the target object is located in the target image . Further, the distances between the position coordinates represented by the position features and the coordinates of each edge vertex of the position area marked by the teacher model are respectively determined, and the obtained multiple different distance values are normalized, and then converted into 0 A floating point number between 1 and 1. Further, the output result whose output value is close to 1 is determined as the location area marked by the two in the target image, and the output result whose output value is close to 0 is determined to be the position area marked by the two in the target image The location areas do not match. For example, if the output result does not match the result represented by the position feature and scale feature, adjust the attention distribution of the teacher model in the direction of decreasing the output value of the loss function of the teacher model until the target detection result of the teacher model is consistent with the position The representations of features and scale features are consistent.

In the embodiment of the present disclosure, the teacher model can be controlled to perform target detection according to the previously obtained attention distribution. Further, the object condition characteristics input into the teacher model are used as the target detection results expected by the teacher model, so that the teacher model can perform adaptive adjustment of the attention distribution, so that the teacher model can obtain the target detection results consistent with the object condition characteristics. This method can improve the object detection accuracy of the teacher model, and training the initial student model with the adjusted attention distribution can improve the object detection accuracy of the target student model.

Fig. 8 is a schematic flow diagram of training an initial student model according to an exemplary embodiment. As shown in Fig. 8, for the same target image, the object condition feature 1 used to characterize the target object in the target image (for Characterize the relevant features of "person" in the target image) and object condition features 2 (used to characterize the relevant features of "horse" in the target image) are input into the teacher model. For example, feature detection can be performed on the target image to obtain the feature map of the teacher model. Further, the feature map of the teacher model may be decomposed into multiple first sub-feature spaces by means of key-value decomposition, and the feature map of the teacher model may be decomposed into multiple second sub-feature spaces by means of content decomposition. In this case, for example, the object condition feature 1 (of course, the object condition feature 2 can also be selected) can be used as an index feature to calculate the correlation between the weighted first sub-feature space and the object condition feature 1, and then Determine the attention distribution of the teacher model for object conditional feature 1. Moreover, in the case of determining the attention distribution of the teacher model, the teacher model may be controlled to perform object detection in a manner that the obtained attention distribution weights a plurality of second sub-feature spaces. Further, the representation of the object condition features (object condition feature 1 and/or object condition feature 2) input to the teacher model is used as the target output result of the teacher model to optimize the attention distribution of the teacher model. Then, the initial student model is trained with the optimized attention distribution, and the trained target student model is obtained. For example, after the training of the initial student model is completed, only the feature extraction network and the final detection head of the target student model can be reserved for testing and deployment related to target detection. In addition, the target student model trained in this disclosure has a relatively obvious increase in target detection and distillation of commonly used data sets.

The neural network model distillation method provided in this disclosure can divide the teacher model feature map obtained by the teacher model from detecting the target image into multiple first sub-feature spaces by means of key-value decomposition, and assign different first sub-feature spaces to Different weights, so that the teacher model can identify various types of features that are helpful for object detection. And the object condition feature input used to characterize the target object can be input into the teacher model as an auxiliary parameter. Since the object condition feature is manually calibrated and input, the attention distribution of the teacher model obtained through the object condition feature is more accurate. It fits the observation perspective of human beings, thereby improving the target detection accuracy of the teacher model. In addition, since the feature map of the teacher model can be a multi-scale feature map obtained through a feature pyramid network, better object detection results can be obtained no matter for objects with larger scales or objects with smaller scales. On this basis, the neural network model distillation method provided by the embodiment of the present disclosure introduces a special loss function to adjust the attention distribution obtained by the teacher model, which can realize the optimization of the attention distribution and significantly improve the teacher's performance. The object detection accuracy of the model. Furthermore, the initial student model is trained with the optimized attention distribution, and the obtained target student model also has higher target detection accuracy. Using the target student model trained by this method for target detection can get better target detection results, which meets the user's target detection needs.

Based on the same idea, an embodiment of the present disclosure also provides a neural network model distillation device.

It can be understood that, in order to realize the above-mentioned functions, the neural network model distillation apparatus provided by the embodiments of the present disclosure includes corresponding hardware structures and/or software modules for performing various functions. Combining the units and algorithm steps of each example disclosed in the embodiments of the present disclosure, the embodiments of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the technical solutions of the embodiments of the present disclosure.

Fig. 9 is a block diagram of a neural network model distillation apparatus 100 according to an exemplary embodiment. Referring to FIG. 9 , the device 100 includes a determination unit 101 , an acquisition unit 102 , a detection unit 103 and a processing unit 104 .

The determination unit 101 is used to determine the target image, and to determine the correlation between the object feature and the object condition feature at each pixel position in the teacher model feature map, and determine the attention distribution of the teacher model based on the correlation. The acquisition unit 102 is configured to acquire object condition features of the target image, where the object condition features are feature information that is manually input and characterizes the target object. The detection unit 103 is configured to perform feature detection on the target image based on the teacher model to obtain a feature map of the teacher model, and perform feature detection on the target image based on the initial student model to obtain a feature map of the student model. The processing unit 104 is configured to train the initial student model based on the attention distribution of the teacher model and the feature difference between the feature map of the teacher model and the feature map of the student model to obtain a target student model.

In one embodiment, the determination unit 101 determines the correlation between the object features at each pixel position in the teacher model feature map and the object condition features in the following manner: decompose the object features at each pixel position in the teacher model feature map into multiple a first sub-feature space. For each pixel position in the feature map of the teacher model, feature weighting is performed on multiple first sub-feature spaces based on different weights respectively, to obtain object features after feature weighting. Determining the correlation between the feature-weighted object features and the object conditional features, and calculating the correlation between the feature-weighted object features and the object conditional features, to obtain the first multiple first sub-feature spaces for each pixel position A sub-feature space correlation is aimed at each pixel position in the feature map of the teacher model, and the first sub-feature space correlation of multiple first sub-feature spaces is normalized to obtain the object feature and object corresponding to each pixel position. Correlations between conditional features.

In one embodiment, the processing unit 104 trains the initial student model based on the attention distribution of the teacher model and the feature difference between the feature map of the teacher model and the feature map of the student model in the following manner to obtain the target student model: The attention distribution of the teacher model is optimized. Based on the optimized attention distribution and the degree of feature difference between the feature map of the teacher model and the feature map of the student model, the initial student model is trained to obtain the target student model.

In one embodiment, the processing unit 104 optimizes the attention distribution of the teacher model in the following manner: decompose the object features at each pixel position in the teacher model feature map into multiple second sub-feature spaces. Based on the attention distribution of the teacher model, feature weighting is performed on multiple second sub-feature spaces. Control the teacher model to perform target detection based on the weighted features, and adjust the attention distribution of the teacher model based on the target detection results and object condition characteristics, so that the teacher model can obtain the target detection results consistent with the object condition characteristics.

In one embodiment, the decomposition methods used in the first sub-feature space and the second sub-feature space are different. Wherein, the decomposition manner includes a key-value decomposition manner and a content decomposition manner.

In one embodiment, the processing unit 104 obtains the target detection result consistent with the object conditional feature in the following manner: Obtain the target detection result consistent with one or a combination of the category feature, position feature, and scale feature corresponding to the object conditional feature.

In one embodiment, the determining unit 101 determines the attention distribution of the teacher model based on the correlation in the following manner: based on the correlation size relationship between the object feature and the object condition feature at each pixel position in the teacher model feature map, the teacher model Weights are assigned to each pixel position in the model feature map, and the attention distribution of the teacher model is obtained according to the assigned weights. Wherein, the first weight corresponding to the first correlation is greater than the second weight corresponding to the second correlation, and the first correlation is greater than the second correlation.

In one embodiment, the detection unit 103 performs feature detection on the target image based on the teacher model in the following manner to obtain a feature map of the teacher model, and performs feature detection on the target image based on the initial student model to obtain a feature map of the student model: control the teacher model to The target image is subjected to multi-scale feature detection, and a teacher model feature map including multi-scale features is obtained. And control the student model to perform multi-scale feature detection on the target image, and obtain the student model feature map including multi-scale features.

As shown in FIG. 10 , an embodiment of the present disclosure provides an electronic device 200 . Wherein, the electronic device 200 includes a memory 201 , a processor 202 , and an input/output (Input/Output, I/O) interface 203 . Wherein, the memory 201 is used for storing instructions. The processor 202 is configured to invoke the instructions stored in the memory 201 to execute the neural network model distillation method of the embodiment of the present disclosure. Wherein, the processor 202 is respectively connected to the memory 201 and the I/O interface 203, for example, may be connected through a bus system and/or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including the program of the neural network model distillation method involved in the embodiments of the present disclosure, and the processor 202 executes various functional applications and data processing of the electronic device 200 by running the programs stored in the memory 201 .

In the embodiment of the present disclosure, the processor 202 may adopt at least one hardware form in a digital signal processor (Digital Signal Processing, DSP), a field-programmable gate array (Field-Programmable Gate Array, FPGA), or a programmable logic array (Programmable Logic Array, PLA) In implementation, the processor 202 may be one or a combination of a central processing unit (Central Processing Unit, CPU) or other forms of processing units with data processing capabilities and/or instruction execution capabilities.

The memory 201 in the embodiments of the present disclosure may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, a random access memory (Random Access Memory, RAM) and/or a cache memory (cache). The non-volatile memory may include, for example, a read-only memory (Read-OnlyMemory, ROM), a flash memory (FlashMemory), a hard disk (HardDiskDrive, HDD) or a solid-state disk (Solid-StateDrive, SSD) and the like.

In the embodiment of the present disclosure, the I/O interface 203 can be used to receive input instructions (such as digital or character information, and generate key signal input related to user settings and function control of the electronic device 200, etc.), and can also output various commands to the outside. information (for example, images or sounds, etc.). In the embodiment of the present disclosure, the I/O interface 203 may include one or more of a physical keyboard, a function button (such as a volume control button, a switch button, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel. indivual.

In some embodiments, the present disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, perform any of the methods described above .

Although operations are depicted in the drawings in a particular order, this should not be understood as requiring that operations be performed in the particular order shown, or in a serial order, or that all illustrated operations be performed, to achieve desirable results . In certain circumstances, multitasking and parallel processing may be advantageous.

The methods and apparatus of the present disclosure can be accomplished using standard programming techniques, using rule-based logic or other logic to implement the various method steps. It should also be noted that the words "means" and "module" as used herein and in the claims are intended to include an implementation using one or more lines of software code and/or a hardware implementation and/or a device for receiving input.

Any steps, operations or procedures described herein can be performed or realized using one or more hardware or software modules alone or in combination with other devices. In one embodiment, a software module is implemented using a computer program product comprising a computer readable medium containing computer program code executable by a computer processor for performing any or all of the described steps, operations or procedures.

The foregoing description of an implementation of the disclosure has been presented for purposes of illustration and description. The foregoing description is not intended to be exhaustive or to limit the disclosure to the precise form disclosed, and variations and modifications are possible in light of the above teachings or may be acquired from practice of the disclosure. The embodiments were chosen and described in order to illustrate the principles of the disclosure and its practical application, to enable others skilled in the art to utilize the disclosure in various embodiments and with various modifications as are suited to the particular use contemplated.

Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

It can be understood that "plurality" in the present disclosure refers to two or more, and other quantifiers are similar. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. The character "/" generally indicates that the contextual objects are an "or" relationship. The singular forms "a", "said" and "the" are also intended to include the plural unless the context clearly dictates otherwise.

It can be further understood that the terms "first", "second", etc. are used to describe various information, but the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not imply a specific order or degree of importance. In fact, expressions such as "first" and "second" can be used interchangeably. For example, without departing from the scope of the present disclosure, first information may also be called second information, and similarly, second information may also be called first information.

It can be further understood that, unless otherwise specified, "connection" includes a direct connection without other components between the two, and also includes an indirect connection between the two with other elements.

It can be further understood that although operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring that these operations be performed in the specific order shown or in a serial order, or that Perform all operations shown to obtain the desired result. In certain circumstances, multitasking and parallel processing may be advantageous.

Other embodiments of the disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the disclosure herein. This application is intended to cover any modification, use or adaptation of the present disclosure, and these modifications, uses or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure . The specification and examples are to be considered as illustrative only, with the true scope and spirit of the disclosure indicated by the following claims.

It should be understood that the present disclosure is not limited to the precise constructions which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims (11)

1. a neural network model distillation method, is characterized in that, described method comprises: Determining the target image, and acquiring object condition features of the target image, the object condition features are feature information that is manually input and characterizes the target object; performing feature detection on the target image based on the teacher model to obtain a feature map of the teacher model, and performing feature detection on the target image based on the initial student model to obtain a feature map of the student model; Determining the correlation between the object feature at each pixel position in the feature map of the teacher model and the object condition feature, and based on the correlation, determine the attention distribution of the teacher model; Based on the attention distribution of the teacher model and the feature difference between the feature map of the teacher model and the feature map of the student model, the initial student model is trained to obtain a target student model. 2. neural network model distillation method according to claim 1, is characterized in that, determines the correlation between the object feature of each pixel position in the feature map of the teacher model and the object condition feature, comprising: Decomposing the object features of each pixel position in the teacher model feature map into a plurality of first sub-feature spaces; For each pixel position in the feature map of the teacher model, perform feature weighting on the plurality of first sub-feature spaces based on different weights respectively, to obtain object features after feature weighting; Determining the correlation between the feature-weighted object features and the object condition features, and according to the correlation between the feature-weighted object features and the object condition features, obtaining the first sub-feature space correlation of the plurality of first sub-feature spaces; For each pixel position in the feature map of the teacher model, the first sub-feature space correlation of the plurality of first sub-feature spaces is normalized to obtain the object feature and the object feature corresponding to each pixel position. Correlations between conditional features. 3. The neural network model distillation method according to claim 1 or 2, characterized in that, based on the attention distribution of the teacher model, and the feature difference between the teacher model feature map and the student model feature map degree, the initial student model is trained to obtain the target student model, including: Optimizing the attention distribution of the teacher model; Based on the optimized attention distribution and the degree of feature difference between the feature map of the teacher model and the feature map of the student model, the initial student model is trained to obtain a target student model. 4. neural network model distillation method according to claim 3, is characterized in that, optimizing the attention distribution of described teacher model, comprises: Decomposing the object features of each pixel position in the teacher model feature map into a plurality of second sub-feature spaces; performing feature weighting on the plurality of second sub-feature spaces based on the attention distribution of the teacher model; Controlling the teacher model to perform target detection based on the weighted features, and adjusting the attention distribution of the teacher model based on the target detection results and object condition characteristics, so that the teacher model obtains a target consistent with the object condition characteristics Test results. 5. The neural network model distillation method according to claim 4, characterized in that, the first sub-feature space is different from the decomposition method used in the second sub-feature space; Wherein, the decomposition manner includes a key-value decomposition manner and a content decomposition manner. 6. neural network model distillation method according to claim 4, is characterized in that, obtains the target detection result consistent with described object condition characteristic, comprises: A target detection result consistent with one or a combination of the category feature, position feature, and scale feature corresponding to the object condition feature is obtained. 7. according to the neural network model distillation method described in claim 1 or 5 or 6, it is characterized in that, based on described correlation, determine the attention distribution of described teacher model, comprising: Based on the correlation size relationship between the object feature of each pixel position in the teacher model feature map and the object condition feature, carry out weight distribution for each pixel position in the teacher model feature map, and according to the assigned weight, Obtain the attention distribution of the teacher model; Wherein, for each pixel position in the feature map of the teacher model, the pixel position with greater correlation has a greater corresponding weight. 8. The neural network model distillation method according to claim 1 or 2, wherein the target image is subjected to feature detection based on the teacher model to obtain a teacher model feature map, and the target image is performed based on the initial student model Feature detection, to obtain the feature map of the student model, including: performing multi-scale feature detection on the target image through the teacher model to obtain a teacher model feature map including multi-scale features; and Using the student model to perform multi-scale feature detection on the target image, a student model feature map including multi-scale features is obtained. 9. A neural network model distillation device, characterized in that the device comprises: A determination unit, configured to determine the target image, and to determine the correlation between the object feature at each pixel position in the teacher model feature map and the object condition feature, and based on the correlation, determine the attention of the teacher model force distribution; An acquisition unit, configured to acquire object condition features of the target image, where the object condition features are feature information that is manually input and characterizes the target object; A detection unit, configured to perform feature detection on the target image based on the teacher model to obtain a feature map of the teacher model, and perform feature detection on the target image based on the initial student model to obtain a feature map of the student model; A processing unit, configured to train the initial student model based on the attention distribution of the teacher model and the feature difference between the feature map of the teacher model and the feature map of the student model to obtain a target student model. 10. An electronic device, characterized in that it comprises: processor; memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the neural network model distillation method described in any one of claims 1-8. 11. A storage medium, characterized in that instructions are stored in the storage medium, and when the instructions in the storage medium are executed by a processor, the processor is able to execute any one of claims 1 to 8. Distillation methods for neural network models.

Images