CN108108657A - A kind of amendment local sensitivity Hash vehicle retrieval method based on multitask deep learning - Google Patents

Abstract

A kind of amendment local sensitivity Hash vehicle retrieval method based on multitask deep learning, identification is carried out at the same time to vehicle vehicle, vehicle system, logo, color, car plate using multitask convolutional neural networks segmentation end to end parallel, feature based pyramid extracts the network module of vehicle image example aspects, the algorithm being ranked up using local sensitivity Hash-Sorting Algorithm is corrected to vehicle characteristics in database can not obtain the cross-module state text searching method of retrieval vehicle image.The present invention proposes a kind of multitask convolutional neural networks and amendment local sensitivity Hash vehicle retrieval method end to end, effectively improve automation and the intelligent level of vehicle retrieval, and with less memory space, faster retrieval rate meets the image retrieval requirement in big data epoch.

Description

A Modified Local Sensitive Hashing Vehicle Retrieval Method Based on Multi-task Deep Learning

technical field

The invention relates to the application of computer vision, pattern recognition, information retrieval, multi-task learning, similarity measurement, deep self-encoding convolutional neural network and deep learning technology in the field of image retrieval, especially to a modified local Sensitive Hash Vehicle Retrieval Method.

Background technique

With the rapid development of society and economy, motor vehicles have become an essential tool for criminals and terrorists to engage in illegal activities while increasingly becoming an indispensable means of transportation for people's daily life. All provinces and inter-city expressways and main roads, city entrances and exits, and major traffic arteries have deployed bayonet equipment to collect information from passing vehicles. However, current bayonets are generally based on license plate recognition technology. Once a suspected vehicle uses fake License plates, sets of cards, no license plates, and constant replacement of license plates can evade the tracking and identification of existing bayonets.

Image-based vehicle feature recognition involves image processing, pattern recognition, computer vision and other related technical fields. At present, the research on this technology at home and abroad can be roughly divided into three directions: (1) Vehicle type recognition based on license plate. The license plate information is recognized in the image, and the type of the vehicle is not directly analyzed. The classification granularity is coarse, and it is impossible to distinguish the license plate vehicles; The existence of objective factors such as light and occlusion cannot achieve the desired effect; (3) car model recognition based on appearance features, this technology has better robustness than the previous two methods, and the types of recognition are more refined. It can be accurate to the brand, series, model, year, etc. of the vehicle.

The vehicle feature recognition technology based on appearance features is mainly completed through the following three steps: vehicle segmentation, vehicle feature extraction and vehicle classification. The traditional vehicle identification methods mainly include: template matching method, statistical pattern recognition method, neural network recognition method, bionic pattern (topological pattern) recognition method and support vector machine method. However, these methods have their own defects, and cannot meet the two most important indicators of speed and accuracy of vehicle classification at the same time. However, the most important factors affecting these two indicators are the extracted vehicle features and the rapid positioning of the vehicle, so feature extraction and rapid target positioning are the keys to the entire recognition process. The extraction of vehicle features is affected by many factors, such as many types of vehicles but no obvious distinguishing features, the movement of vehicles, and the height and angle of the camera lead to large differences in the characteristics of vehicle models, the influence of weather, and the influence of light.

The development of deep learning technology has promoted the ability of image structure and feature extraction. The bayonet system built in the early stage has weak intelligent analysis ability, low picture quality and low accuracy of license plate recognition. It is often necessary to manually search for the target vehicle from a large number of passing pictures or videos based on the inherent information of the vehicle itself, such as the brand, model and color. Due to factors such as limited police force, high labor intensity, various vehicle types, and uncertain light angles, the accuracy and timeliness of the search cannot be guaranteed, especially in emergencies, which often delay the best time to deal with them. By using the deep learning system of vehicle characteristics, the characteristic structural analysis and identification of the passing pictures obtained by the front-end bayonet or simple bayonet are carried out, and the valuable information in the massive bayonet passing pictures can be fully excavated, which can not only improve the accuracy of the license plate model , and increase the identification information of vehicle characteristics, realize the identification and detection functions of vehicle sub-brand, body color, not wearing seat belts, drivers answering and calling, sun visor status, etc., finely correct the passing data, get rid of the traditional The single method of analyzing and judging purely relying on the license plate provides a richer and more practical vehicle prevention and control application for the bayonet electric police data. Effectively lock suspect vehicles in cases involving vehicles and driving, improve the efficiency of criminal investigations, and transform public security prevention and control methods from post-passive investigations to pre-active early warnings.

Chinese invention patent application number CN201510744990.4 discloses a vehicle retrieval method based on similarity learning. Given a vehicle area, after obtaining SIFT feature points and descriptions, a clustering algorithm is used to discretize SIFT features. In order to make up for the lack of position information of SIFT features, the discrete SIFT feature distribution in the neighborhood is further used to generate neighborhood features as the final feature point description. Each vehicle picture is represented by a batch of features, and a pair of features of similar vehicle pictures form a Positive samples, a pair of features of different vehicle images form a negative sample. After collecting a large number of positive and negative samples in this way, the random forest method is used for similarity learning, and the obtained classifier can be used to judge whether two vehicles are similar, so as to achieve the purpose of similar vehicle retrieval. This technique cannot sufficiently extract vehicle features using SIFT features.

China Invention Patent Application No. CN201610711333.4 discloses a vehicle retrieval method and device based on big data. The method includes: extracting each vehicle inspection mark of the target vehicle from the target vehicle image; Relational fusion of each vehicle inspection mark to obtain a plurality of fusion areas, the fusion area includes at least one of the vehicle inspection marks; determine the shape and color of each vehicle inspection mark contained in each fusion area; according to the number of vehicle inspection marks, the fusion area The number, the number, shape and color of the vehicle inspection marks contained in each fusion area, and the target vehicle is retrieved layer by layer in multiple vehicle images to be retrieved. The technique uses only a single feature to retrieve vehicles.

China Invention Patent Application No. CN201710451957.1 discloses a machine vision-based retrieval and recognition system for licensed vehicles. The system mainly includes a vehicle image acquisition system, a database system, and a retrieval system. The present invention proposes a machine vision-based retrieval and recognition system for licensed vehicles, which uses the vehicle decorations of suspicious vehicles, such as vehicle decorations, annual inspection labels, etc. Carry out the retrieval of suspected vehicles, through the feature collection of the image of the vehicle decoration area, and use the sparse coding method based on the image of the vehicle decoration area to perform vehicle retrieval, solve the problem of searching for the target vehicle from a large number of traffic scene images, and realize the identification of licensed vehicles accurate identification and discovery. This technique has high time complexity in a large number of databases.

To sum up, there are still several thorny problems in the image search technology using the deep self-encoded convolutional neural network and the modified local sensitive hash reordering method: 1) How to accurately extract information from the complex background Segment the overall image of the vehicle under test; how to use as little label image data as possible to learn and train the characteristic data of the vehicle model; 2) how to classify the vehicle model more subdivided, and identify the brand, series, and body color of the vehicle Wait for more information. On the other hand, how to process the model, license plate and logo in parallel in the same deep convolutional neural network, that is, to realize the multi-task parallel computing of deep learning, so as to improve the level of vehicle identification; 3) how to design a vehicle The method of extracting instance features from images is used to retrieve similar types of vehicle models; 4) How to use the extracted features to establish a hierarchical deep search to obtain more accurate retrieval results; 5) How to reduce the storage space of image retrieval systems in the era of big data Large consumption, slow retrieval speed and other issues.

Contents of the invention

Aiming at the low level of automation and intelligence in the existing vehicle retrieval technology, lack of deep learning, difficulty in obtaining accurate retrieval results, large storage space consumption of retrieval technology, slow retrieval speed and difficulty in meeting the image retrieval needs in the era of big data, this paper The invention proposes an end-to-end vehicle image retrieval method through layered depth search based on deep self-encoded convolutional neural network, which improves the automation and intelligence level of the retrieval system by using deep learning methods and simultaneously enables image recognition, feature acquisition, The perfect combination of retrieval efficiency enables the entire retrieval system to obtain accurate retrieval results, and the use of sparse coding reduces the system's dependence on memory and speeds up retrieval, thereby meeting the image retrieval needs in the era of big data.

In order to solve the above technical problems, the present invention provides the following technical solutions:

A modified local sensitive hash vehicle retrieval method based on multi-task deep learning, comprising the following steps:

1) Construct a multi-task end-to-end convolutional neural network for deep learning and training recognition. The training data and layer-by-layer progressive network structure can deeply learn various attribute information of vehicles, including model, car series, car logo, color and license plate;

2) Utilize the multi-task convolutional neural network of step 1) to adopt segmented parallel learning and encoding strategies to construct vehicle attribute hash codes;

3) Use the pyramid pooling layer and the vector compression layer to construct a feature pyramid module to adapt to the convolution feature map input of different sizes to extract the instance features of the vehicle;

4) Using the instance features obtained in step 3) to construct a locally sensitive reordering algorithm;

5) Construct a cross-modal retrieval method under the condition that the retrieved vehicle image cannot be obtained to realize vehicle retrieval.

Further, the multi-task end-to-end convolutional neural network for deep learning and training recognition contains a shared convolution module, a region of interest coordinate regression and recognition module, a multi-task learning module, and an instance feature extraction module;

Shared convolution module: The shared network consists of 5 convolution modules, where the last layer of conv2_x to conv5_x is {4 ² ,8 ² ,16 ² ,16 ² } as the output size of the feature map, conv1 as the input layer only Contains a single convolutional layer;

After the shared convolution module, the coordinate regression and recognition module of the region of interest is connected. This module can take an image of any size as input, and output a set of rectangular prediction frames of the target area, including the position coordinates of each prediction frame and the category in the data set. The probability score of , in order to generate a region proposal box, first input the image through the convolution shared layer to generate a feature map, and then perform a multi-scale convolution operation on the feature map. The implementation process is: use 3 scales and 3 kinds of aspect ratios, centered on the center of the current sliding window, and corresponding to one scale and aspect ratio, then 9 candidate regions of different scales can be mapped on the original image; for example, for a shared convolution with a size of w×h feature map, there are a total of w×h×9 candidate regions; finally, the classification layer outputs the scores of w×h×9×2 candidate regions, that is, the estimated probability of each region being a target/non-target, and the regression layer outputs w×h×9×4 parameters, that is, the coordinate parameters of the candidate area;

When training the RPN network, assign a binary label to each candidate area to mark whether the area is an object target. The operation is as follows: 1) It has the highest IoU (Intersection- over-Union, the ratio of intersection and union) overlapping candidate regions; 2) candidate regions that have an IoU overlap greater than 0.7 with any GT bounding box. Assign negative labels to candidate regions whose IoU ratios to all GT bounding boxes are lower than 0.3; 3) discard in between.

With these definitions, the objective function is minimized. The loss function for an image is defined as:

where i is the index of the ith candidate region, and P _i is the probability that the candidate region is the ith class. If the label of the candidate region is positive, is 1, if the candidate region label is 0, It is 0. t _i is a vector representing the 4 parameterized coordinates of the predicted bounding box, is the coordinate vector of the corresponding GT bounding box. N _cls and N _reg are the normalization coefficients of the classification loss function and the position regression loss function respectively, λ is the weight parameter between the two, the classification loss function L _cls is the logarithmic loss of the two categories, and the two categories are the target and non-target:

For the positional regression loss function L _reg , the following function is defined:

Among them, R is a robust loss function (smooth L1).

However, training a multi-task deep learning network is not an easy process because different task-level information has different learning difficulties and convergence speeds. Therefore, it is crucial to design a good multi-task objective function. The multi-task joint training process is as follows: Assume that the total number of tasks is T, and the training data for the t-th task is recorded as Where t∈(1,T), i∈(1,N), N is the total number of training samples, are the feature vector and label of the i-th sample respectively, then the multi-task objective function is expressed as:

In the formula is the input feature vector and the mapping function of the weight parameter w ^t , L( ) is the loss function, and Φ(w ^t ) is the regularization value of the weight parameter;

For the loss function, the features of the last layer are trained using softmax and the log-likelihood cost function to achieve image classification. The softmax loss function is defined as follows:

In the formula, x _i is the i-th depth feature, W _j is the j-th column of the weight in the last fully connected layer, b is the bias item, m, n are the number of processing samples and the number of categories respectively;

Convolutional neural network training is a backpropagation process, similar to the BP algorithm, through the backpropagation of the error function, using the stochastic gradient descent method to optimize and adjust the convolution parameters and bias until the network converges or reaches the maximum number of iterations to stop;

The neural network training is a backpropagation process, through the backpropagation of the error function, the convolution parameters and bias are optimized and adjusted by the stochastic gradient descent method, until the network converges or reaches the maximum number of iterations to stop;

Backpropagation needs to compare the training samples with labels, and use the square error cost function to identify multiple categories of c categories and N training samples. The final output error function of the network uses formula (7) to calculate the error.

In the formula, E ^N is the square error cost function, is the k-th dimension of the label corresponding to the n-th sample, The nth sample corresponds to the kth output predicted by the network;

When backpropagating the error function, a calculation method similar to the traditional BP algorithm is used, and the specific formula form is shown in formula (8),

δ ^l ＝(W ^l+1 ) ^T δ ^l+1 ×f'(u ^l )(u ^l ＝W ^l x ^l-1 +b ^l ) (8)

In the formula, δ ^l represents the error function of the current layer, δ ^l+1 represents the error function of the previous layer, W ^l+1 is the mapping matrix of the previous layer, f' represents the inverse function of the activation function, that is, upsampling, u ^l Represents the output of the previous layer that has not passed the activation function, x ^l-1 represents the input of the next layer, and W ^l is the mapping weight matrix of this layer.

Furthermore, there is a correlation between tasks in the multi-task learning process, that is, there is information sharing between tasks. When training multiple tasks at the same time, the network uses the shared information between tasks to enhance the system's inductive bias ability and classification. The generalization ability of the device; the multi-task network is divided into five subtasks by adding five fully connected layers after the region of interest module, and each fully connected softmax activation function normalizes the threshold in [0,1] In between, the normalized value is sent to the segmentation function to promote the output of the binary code, and the redundancy between the hash codes is reduced through segmentation learning and coding strategies to enhance the robustness of the learned features;

The multi-task learning network is divided into T tasks, each task contains c ^t categories, and the one-dimensional vector output of the fully connected layer of each task is represented by m ^t ; firstly, the output of the fully connected layer is normalized by using the softmax activation function between [0,1], the specific expression of the formula is as follows:

Among them, θ represents a random hyperplane; the normalized value is sent to the threshold segmentation function for binarization, and the binary output of the fully connected layer is obtained. The specific expression of the formula is as follows:

Finally, in order to obtain the vehicle attribute hash code learned in parallel by the multi-task convolutional network, the H ^t vector obtained by the formula (10) is fused again in a certain proportion, and expressed by the vector f _A. The specific expression of the formula is as follows:

f _A = [α ¹ H ¹ ; α ² H ² ; . . . ; α ^t H ^t ] (11)

The specific form of α ^t in formula (11) is as follows:

Multiplying the penalty factor α ^t before each H ^t makes up for the error caused by different classification numbers between different tasks.

Furthermore, the age of hand-designed features makes heavy use of figurative image pyramids, so that object detectors like DPM require high-density sampling to achieve good results (e.g., 10 octaves); for recognition tasks, Engineered features have been mostly replaced by features computed by deep convolutional networks. In addition to being able to represent higher-level semantics, deep convolutional networks have more robust scale variability, thereby facilitating recognition from features computed at a single input scale; however, even with this robustness, Pyramids are still needed to get the most accurate results; all recent top entries in the ImageNet and COCO detection challenges use multi-scale testing of featurized image pyramids; the main advantage of having features at each level of an image pyramid is that it A multi-level feature representation is produced, where all levels are semantically strong, including the high-resolution level;

Utilizes the pyramid shape of the convolutional feature hierarchy while simultaneously creating a feature pyramid that is semantically strong at all scales; to achieve this goal relies on a structure that combines low-resolution, semantically powerful features with high-resolution, semantically Weak features are combined via top-down paths and lateral connections, and can be quickly constructed from a single input image scale, and can be used instead of a featurized image pyramid without sacrificing representative features, speed, or memory; in order to obtain vehicle The instance features of the image and adapt to the input of the convolution feature map of any size, select the last layer of each unit of the shared module conv2_x to conv5_x and combine the output of the region of interest module, and then add a pyramid pooling layer and a vector compression layer will be The three-dimensional features are compressed into a one-dimensional feature vector. This choice is because the feature map information obtained by the feature pyramid can be enriched, and the deepest layer of each stage has the most powerful feature representation function;

Take the last layer of each module as the input of the feature pyramid, and select {4 ² , 8 ² , 16 ² , 16 ² } as the input feature map size of the feature pyramid for the last layer of the network conv2_x to conv5_x defined above; Use I to represent the input image, and its length and width are represented by letters h and w respectively. The shared convolution module of the xth layer is represented by convx_x. After the input image I is activated into a three-dimensional feature vector T, the dimension is h′×w′ ×d is a set of a series of two-dimensional feature maps, the length and width of the two-dimensional feature maps are h′×w′, T contains d two-dimensional feature maps, and the set S=S{S _n },n∈(1, d) indicates that S _n corresponds to the feature map of the nth channel; then the three-dimensional feature vector T is sent into the feature pyramid, and the three-dimensional feature vector T′ is obtained after convolution with multiple scale convolution kernels, and the dimension is l×l× d, also contains a set of two-dimensional feature maps, which can be represented by S′=S′{S′ _n }, n∈(1,d), where S _n ′ corresponds to the nth channel feature map, and the size of each feature map It is l×l, including d in total; then use the sliding window of k×k size and select the maximum pooling to perform logistic regression on the feature map to obtain a set of feature maps of l/k×l/k size, and then for each Channel S _n ′ is fused to obtain a one-dimensional vector, and the same operation is performed on d channels in turn, and the finally obtained personality feature vector f _B has a size of (1,l/k×d). The final retrieval feature vector f is shown in formula (13):

f=[f _A ; f _B ] (13).

The basic idea of locality-sensitive hashing algorithm is: After transforming two adjacent data points in the original data space through the same mapping or projection, the probability that these two data points are still adjacent in the new data space is very high. The probability that non-adjacent data points are mapped to the same bucket is very small. That is to say, if we perform some hash mapping on the original data, we hope that two adjacent data can be hashed into the same bucket with the same bucket number. After hash mapping is performed on all the data in the original data set, a hash table is obtained. These original data sets are scattered into the buckets of the hash table, and each bucket will fall into some original data belonging to the same bucket. The data in a bucket is likely to be adjacent. Of course, there are also non-adjacent data that are hashed into the same bucket. Therefore, if some hash functions can be found so that after their hash map transformation, the adjacent data in the original space fall into the same bucket, then it becomes easy to search for neighbors in the data set , you only need to perform hash mapping on the query data to obtain its bucket number, then take out all the data in the bucket corresponding to the bucket number, and then perform linear matching to find the data adjacent to the query data. In other words, through the hash function mapping transformation operation, the original data set is divided into multiple sub-sets, and the data in each sub-set is adjacent and the number of elements in the sub-set is small, so one in The problem of finding adjacent elements in a very large set is transformed into the problem of finding adjacent elements in a small set. This algorithm can greatly reduce the amount of search calculations;

For the hash function that two adjacent data points fall into the same bucket after hash transformation, the following two conditions need to be met:

If d(x,y)≤d1, then h(x)=h(y) with probability at least p1;

If d(x,y)≥d2, then the probability of h(x)=h(y) is at most p2;

Where d(x,y) represents the distance between x and y, d1<d2, h(x) and h(y) represent the hash transformation of x and y, respectively.

A hash function that satisfies the above two conditions is called (d1,d2,p1,p2)-sensitive. The process of generating one or more hash tables by hashing the original data set through one or more (d1, d2, p1, p2)-sensitive hash functions is called locality-sensitive hashing.

The process of using local sensitive hash to index massive data, that is, hash table and perform approximate nearest neighbor search through index is as follows:

Offline indexing

(1) Select a hash function that satisfies (d1,d2,p1,p2)-sensitive locality-sensitive hashing;

(2) Determine the number L of hash tables, the number K of hash functions in each hash table, and local sensitivity according to the accuracy of the search results, that is, the probability that adjacent data is found parameters related to the hash function itself;

(3) Hash all the data into corresponding buckets through the hash function of local sensitive hashing, forming one or more hash tables;

find online

(1) Hash the query data through the hash function of local sensitive hash to obtain the corresponding bucket number;

(2) Take out the corresponding data in the bucket number; in order to ensure the search speed, only take out the first 2L data;

(3) Calculate the similarity or distance between the query data and the 2L data, and return the nearest neighbor data;

The local sensitive hash online search time consists of two parts: ① the time to calculate the hash value through the hash function of the local sensitive hash, that is, the time to calculate the bucket number; ② the time to compare the query data with the data in the bucket. Therefore, the lookup time of locality-sensitive hashing is at least a sublinear time. This is because the matching speed is accelerated by indexing the belongings in the bucket. At this time, the time consumption of part ② changes from O(N) to O(logN) or O(1), which greatly reduces the calculation quantity.

A key of the locality-sensitive hashing is: to map similar samples to the same bucket with high probability; the hash function h(.) of the locality-sensitive hashing satisfies the following conditions:

s{h(f _Aq )=h(f _A )}=sim(f _Aq ,f _A ) (14)

In the formula, sim(f _Aq , f _A ) represents the similarity between f _Aq and f _A , h(f _A ) represents the hash function of f _A , h(f _Aq ) represents the hash function of f _A , where The similarity measure is directly related to a distance function σ, such as:

A typical classification of locality-sensitive hash functions is given by random projections and thresholds, as shown in Equation (16),

h(f _A )=sign(Wf _A +b) (16)

where W is a random hyperplane vector and b is a random intercept.

The locality-sensitive hash is composed of a preprocessing algorithm and a nearest neighbor search algorithm, and the search image features will be represented as a string of fixed-length binary codes through these two algorithm processing;

The process of the preprocessing algorithm is:

Input a set of extracted image features p and the number l ₁ of the hash table, use the random hash function g(.) to map the image features, and store the point p _j in the corresponding bucket number g _{i (} p _j ); output hash table T _i , i=1,...,l ₁ ;

The process of the nearest neighbor search algorithm is:

Input a retrieval image feature q, access the hash table T _i generated by the preprocessing algorithm, i=1,...,l ₁ the number K of the nearest neighbors, and return the K nearest neighbor data of the retrieval point q in the data set S ;

Let Γ={I ₁ ,I ₂ ,…,In } be the retrieved data set consisting of _n images, and the binary code corresponding to each image is Γ _H ＝{H ₁ ,H ₂ ,…,H _n }, H _i ∈ {0,1} ^h ; given a search image I _q and a binary code H _q , put those images whose Hamming distance between H _q and H _i ∈ Γ _H is smaller than the threshold T _H into In the candidate pool P, for candidate images.

Using instance features to construct a local-sensitive reordering algorithm; in the traditional local-sensitive hashing algorithm, the returned images are mainly similar in distance, that is, the similarity between the retrieved image and the image in the candidate pool is close to 1; mainly because of the low-dimensional Vehicles of the same model can be obtained after mapping the vehicle attribute hash code, but there are still some indistinguishable situations between vehicles of the same model. There are obvious differences in human subjective judgment, but only through the vehicle attribute hash code cannot be effective Distinguish these differences; in order to find out the vehicles in the candidate pool that have the same personality characteristics as the retrieved pictures, after the retrieved images are mapped to each bucket through the vehicle attribute hash code, and then use the obtained image instance features to compare the images in the bucket Sorting is performed again to reduce the intra-class error, and the expression of the re-sorting formula is as follows:

In formula (17), k represents the kth image in the bucket selected by the vehicle attribute hash code mapping, represents the penalty factor and cos represents the cosine distance formula for measuring the characteristics of image instances; in order to eliminate the wrong mapping of vehicle attribute hash codes, y represents the retrieval image f _Aq before mapping and the image in the bucket Whether the types of are equal, if they are equal, y is 1, otherwise it is 0;

In further sorting, those images whose Hamming distance between H _q and H _i ∈ Γ _H are smaller than the threshold T _H have been put into the candidate pool P. In order to obtain more accurate search results, the present invention selects On the basis of further adopting the reordering method;

The reranking method, given a search image I _q and a candidate pool P, uses instance features to determine the top-k ranked images from the images in the candidate pool P; the similarity between them is calculated using Equation (17),

Further, regarding the evaluation of re-ranking, a ranking-based criterion is used for evaluation; given a search image I _q and a similarity measure, a ranking is performed for each dataset image; here, the top k Ranking images to represent the retrieval accuracy of a search image I _q , expressed by formula (18);

In the formula, Rel(i) represents the true correlation between the search image I _q and the i-th ranked image, k represents the number of ranked images, Precision@k search accuracy; when calculating the true correlation, only the part with classification labels is considered, Rel(i) ∈ {0,1}, if the search image has the same label as the i-th ranked image, set Rel(i)=1, otherwise set Rel(i)=0, traverse the top k candidates in the pool P Rank images to get search accuracy;

In the step 5), when the retrieved image cannot be obtained, the text retrieval method is used for auxiliary retrieval, and without additional training, the retrieval features obtained through the text and the text features obtained by the convolutional network can be shared. Set retrieval method, its text acquisition feature method is as follows:

Initialization: parse the text file into entry vectors; remove small words and repeated words; check the entries to ensure the correctness of the analysis;

5.1) Take out the word segmentation minimum vector R=(r ₁ ,r ₂ ,...,r _n ) of random combination from the input text O;

5.2) Integrate R and f _A sequence and vehicle attribute hash code to get the text attribute characteristic At this time, the dimension of _fATxt is smaller than that of R;

5.3) Retrieve using locality-sensitive reordering hash algorithm;

5.4) Return similar image group I.

To realize the above invention, several core problems must be solved: 1) Aiming at the difficult problem of image feature extraction, using the powerful feature representation ability of deep self-encoding convolutional neural network to realize feature adaptive extraction; 2) Aiming at large-scale image retrieval For the problem of slow speed, design a multi-task layering method, use the query image to quickly compare with the image in the database; 3) design a method for extracting instance features from vehicle images for retrieval of similar types of vehicles; 4) design a Modify the locality-sensitive hash reordering code to increase the difference between vehicle images in the class; 5) take advantage of the end-to-end deep network to design an end-to-end deep self-encoding convolutional neural network that combines detection, recognition, and feature extraction into one network.

The modified local sensitive hash reordering vehicle retrieval method based on multi-task deep learning of the present invention includes the following processes: 1) sending the image into a deep self-encoding convolutional neural network, performing logistic regression on the feature map, and performing a logical regression on the retrieved image 2) Use the multi-task deep self-encoding convolutional neural network to extract the vehicle attribute hash code learned in parallel by image segmentation; 3) Use the pyramid shape of the convolutional feature hierarchy Extract the instance features of each vehicle; 4) use the modified local sensitive hashing method to retrieve the extracted features; 5) use cross-modal retrieval when the vehicle image cannot be obtained;

The beneficial effects of the present invention are mainly manifested in:

1) A multi-task end-to-end convolutional neural network is provided to identify vehicle models, car series, car logos, colors, and license plates;

2) Use the powerful feature representation ability of deep convolutional neural network to realize feature adaptive extraction;

3) A modified local sensitive hash reordering code is constructed to retrieve the features extracted by the convolutional network;

4) This design takes both versatility and specificity into account. In terms of versatility, retrieval speed, accuracy and practicability meet the needs of various users; After the network parameters are fine-tuned, an application-specific vehicle retrieval system is realized.

Description of drawings

Figure 1 is the overall retrieval flow chart.

Figure 2 is a flowchart of the overall training network.

Figure 3 is an expanded view of the RPN network.

Fig. 4 is a schematic diagram of vehicles that cannot be distinguished by vehicle attribute hash codes.

Figure 5 is a graph of text feature vector generation.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention.

Referring to Figures 1 to 5, a modified local sensitive hash vehicle retrieval method based on multi-task deep learning. The overall flow chart is shown in Figure 1. First, the pictures in the database are sent to multiple Task end-to-end convolutional neural network, a large number of training and layer-by-layer progressive network structure to deeply learn various attribute information of vehicles, including model, car series, car logo, color, license plate; and then use this convolutional network to extract vehicles The vehicle attribute hash code learned in parallel by image segmentation is used to extract the instance features of the vehicle from the constructed feature pyramid module; the retrieved vehicle image is compared with the images in the database using the modified local sensitive hash reordering method.

The multi-task end-to-end convolutional neural network used for deep learning and training recognition contains a shared convolution module, a region of interest coordinate regression and recognition module, a multi-task learning module and an instance feature extraction module. The overall flow chart is shown in Figure 2 As shown, there are a total of 4 shared convolution modules and 4 layers of feature pyramid modules; the double dotted line in Figure 2 is the vehicle instance feature extracted by the compression layer; the dotted line in Figure 2 is the proposed division and encoding module , learn the compact features of the vehicle through different tasks, and finally fuse the two extracted feature vectors;

The present invention comprises the following steps:

1) Shared convolution module: The shared network consists of 5 convolution modules, where the last layer of conv2_x to conv5_x is {4 ² , 8 ² , 16 ² , 16 ² } as the output size of the feature map, and conv1 as the input The layer contains only a single-layer convolutional layer;

After the shared convolution module, the coordinate regression and recognition module of the region of interest is connected. This module can take an image of any size as input, and output a set of rectangular prediction frames of the target area, including the position coordinates of each prediction frame and the category in the data set. The probability score of , in order to generate a region suggestion box, first input the image through the convolution shared layer to generate a feature map, and then perform a multi-scale convolution operation on the feature map. The specific implementation is: use 3 scales and 3 kinds of aspect ratios, centered on the center of the current sliding window, and corresponding to one scale and aspect ratio, then 9 candidate regions of different scales can be mapped on the original image; for example, for a shared convolution with a size of w×h feature map, there are a total of w×h×9 candidate regions. Finally, the classification layer outputs the scores of w×h×9×2 candidate regions, that is, the estimated probability of being a target/non-target for each region, and the regression layer outputs w×h×9×4 parameters, which are the coordinates of the candidate regions Parameters, the specific form is shown in Figure 3;

When training the RPN network, a binary label is assigned to each candidate region to mark whether the region is an object target. The specific operation is as follows: 1) It has the highest IoU (Intersection-over-Union, ratio of intersection and union) overlapping candidate area with a real target area (Ground Truth, GT); Candidate regions for IoU overlap. Assign negative labels to candidate regions whose IoU ratios to all GT bounding boxes are lower than 0.3; 3) discard in between.

With these definitions, the objective function is minimized. The loss function for an image is defined as:

where i is the index of the ith candidate region, and P _i is the probability that the candidate region is the ith class. If the label of the candidate region is positive, is 1, if the candidate region label is 0, It is 0. t _i is a vector representing the 4 parameterized coordinates of the predicted bounding box, is the coordinate vector of the corresponding GT bounding box. N _cls and N _reg are the normalization coefficients of the classification loss function and the position regression loss function respectively, and λ is the weight parameter between the two. The classification loss function L _cls is the log loss for two classes (target vs. non-target):

For the positional regression loss function L _reg , the following function is defined:

Among them, R is a robust loss function (smooth L1).

However, training a multi-task deep learning network is not an easy process, because information at different task levels has different learning difficulties and convergence speeds. Therefore, it is crucial to design a good multi-task objective function. The multi-task joint training process is as follows: Assume that the total number of tasks is T, and the training data for the t-th task is recorded as Where t∈(1,T), i∈(1,N), N is the total number of training samples. are the feature vector and label of the i-th sample, respectively. Then the multi-task objective function can be expressed as:

In the formula is the input feature vector and the mapping function of the weight parameter w ^t , L( ) is the loss function, and Φ(w ^t ) is the regularization value of the weight parameter.

For the loss function, the features of the last layer are trained using softmax and the log-likelihood cost function to achieve image classification. The softmax loss function is defined as follows:

In the formula, x _i is the i-th depth feature, W _j is the j-th column of the weight in the last fully connected layer, b is the bias item, m, n are the number of processing samples and the number of categories respectively;

Convolutional neural network training is a backpropagation process, similar to the BP algorithm, through the backpropagation of the error function, using the stochastic gradient descent method to optimize and adjust the convolution parameters and bias until the network converges or reaches the maximum number of iterations to stop;

The neural network training is a backpropagation process, through the backpropagation of the error function, the convolution parameters and bias are optimized and adjusted by the stochastic gradient descent method, until the network converges or reaches the maximum number of iterations to stop;

Backpropagation needs to compare the training samples with labels, and use the square error cost function to identify multiple categories of c categories and N training samples. The final output error function of the network uses formula (7) to calculate the error.

In the formula, E ^N is the square error cost function, is the k-th dimension of the label corresponding to the n-th sample, The nth sample corresponds to the kth output predicted by the network;

When backpropagating the error function, a calculation method similar to the traditional BP algorithm is used, and the specific formula form is shown in formula (8),

δ ^l ＝(W ^l+1 ) ^T δ ^l+1 ×f'(u ^l )(u ^l ＝W ^l x ^l-1 +b ^l ) (8)

In the formula, δ ^l represents the error function of the current layer, δ ^l+1 represents the error function of the previous layer, W ^l+1 is the mapping matrix of the previous layer, f' represents the inverse function of the activation function, that is, upsampling, u ^l Represents the output of the previous layer that has not passed the activation function, x ^l-1 represents the input of the next layer, and W ^l is the mapping weight matrix of this layer;

2) There is a correlation between tasks in the learning process of multi-tasks, that is, there is information sharing between tasks. When training multiple tasks at the same time, the network uses the shared information between tasks to enhance the system's inductive bias ability and classifier The generalization ability; the multi-task network is divided into five subtasks by adding five fully connected layers after the region of interest module, and each fully connected softmax activation function normalizes the threshold between [0,1] Then, the normalized value is sent to the segmentation function to promote the output of the binary code, and the redundancy between the hash codes is reduced through segmentation learning and coding strategies to enhance the robustness of the learned features;

The multi-task learning network is divided into T tasks, each task contains c ^t categories, and the one-dimensional vector output of the fully connected layer of each task is represented by m ^t ; firstly, the output of the fully connected layer is normalized by using the softmax activation function between [0,1], the specific expression of the formula is as follows:

Among them, θ represents a random hyperplane; the normalized value is sent to the threshold segmentation function for binarization, and the binary output of the fully connected layer is obtained. The specific expression of the formula is as follows:

Finally, in order to obtain the vehicle attribute hash code learned in parallel by the multi-task convolutional network, the H ^t vector obtained by the formula (10) is fused again in a certain proportion, and expressed by the vector f _A. The specific expression of the formula is as follows:

f _A = [α ¹ H ¹ ; α ² H ² ; . . . ; α ^t H ^t ] (11)

The specific form of α ^t in formula (11) is as follows:

Multiplying the penalty factor α ^t before each H ^t makes up for the error caused by different classification numbers between different tasks;

3) Utilize the pyramidal shape of the convolutional feature hierarchy while creating a feature pyramid that is semantically strong at all scales; to achieve this goal relies on a structure that combines low-resolution, semantically strong features with high-resolution, Semantically weak features are combined via top-down paths and lateral connections, and can be quickly constructed from a single input image scale, and can be used to replace featurized image pyramids without sacrificing representative features, speed, or memory; for Get the instance features of the vehicle image and adapt to the input of the convolution feature map of any size, select the last layer of each unit of the shared module conv2_x to conv5_x and combine the output of the region of interest module, and then add a pyramid pooling layer and vector compression The layer compresses the three-dimensional features into a one-dimensional feature vector. This choice is because the feature map information obtained by the feature pyramid can be enriched, and the deepest layer of each stage has the most powerful feature representation function;

Take the last layer of each module as the input of the feature pyramid, and select {4 ² , 8 ² , 16 ² , 16 ² } as the input feature map size of the feature pyramid for the last layer of the network conv2_x to conv5_x defined above; Use I to represent the input image, and its length and width are represented by letters h and w respectively. The shared convolution module of the xth layer is represented by convx_x. After the input image I is activated into a three-dimensional feature vector T, the dimension is h′×w′ ×d is a set of a series of two-dimensional feature maps, the length and width of the two-dimensional feature maps are h′×w′, T contains d two-dimensional feature maps, and the set S=S{S _n },n∈(1, d) indicates that S _n corresponds to the feature map of the nth channel; then the three-dimensional feature vector T is sent into the feature pyramid, and the three-dimensional feature vector T′ is obtained after convolution with multiple scale convolution kernels, and the dimension is l×l× d, also contains a set of two-dimensional feature maps, which can be represented by S′=S′{S′ _n }, n∈(1,d), where S _n ′ corresponds to the nth channel feature map, and the size of each feature map It is l×l, including d in total; then use the sliding window of k×k size and select the maximum pooling to perform logistic regression on the feature map to obtain a set of feature maps of l/k×l/k size, and then for each Channel S _n ′ is fused to obtain a one-dimensional vector, and the same operation is performed on d channels in turn, and the finally obtained personality feature vector f _B has a size of (1,l/k×d). The final retrieval feature vector f is shown in formula (13):

f=[f _A ; f _B ] (13)

The basic idea of locality-sensitive hashing algorithm is: After transforming two adjacent data points in the original data space through the same mapping or projection, the probability that these two data points are still adjacent in the new data space is very high. The probability that non-adjacent data points are mapped to the same bucket is very small. That is to say, if we perform some hash mapping on the original data, we hope that two adjacent data can be hashed into the same bucket with the same bucket number. After hash mapping is performed on all the data in the original data set, a hash table is obtained. These original data sets are scattered into the buckets of the hash table, and each bucket will fall into some original data belonging to the same bucket. The data in a bucket is likely to be adjacent, and of course there are non-adjacent data that are hashed into the same bucket. Therefore, if some hash functions can be found so that after their hash map transformation, the adjacent data in the original space fall into the same bucket, then it becomes easy to search for neighbors in the data set , you only need to perform hash mapping on the query data to obtain its bucket number, then take out all the data in the bucket corresponding to the bucket number, and then perform linear matching to find the data adjacent to the query data. In other words, through the hash function mapping transformation operation, the original data set is divided into multiple sub-sets, and the data in each sub-set is adjacent and the number of elements in the sub-set is small, so one in The problem of finding adjacent elements in a very large set is transformed into the problem of finding adjacent elements in a small set. This algorithm can greatly reduce the amount of search calculations;

For the hash function that two adjacent data points fall into the same bucket after hash transformation, the following two conditions need to be met:

If d(x,y)≤d1, then h(x)=h(y) with probability at least p1;

If d(x,y)≥d2, then the probability of h(x)=h(y) is at most p2;

Where d(x,y) represents the distance between x and y, d1<d2, h(x) and h(y) represent the hash transformation of x and y, respectively.

A hash function that satisfies the above two conditions is called (d1,d2,p1,p2)-sensitive. The process of generating one or more hash tables by hashing the original data set through one or more (d1, d2, p1, p2)-sensitive hash functions is called locality-sensitive hashing.

The process of using local sensitive hash to index massive data, that is, hash table and perform approximate nearest neighbor search through index is as follows:

Offline indexing

(1) Select a hash function that satisfies (d1,d2,p1,p2)-sensitive locality-sensitive hashing;

(2) Determine the number L of hash tables, the number K of hash functions in each hash table, and local sensitivity according to the accuracy of the search results, that is, the probability that adjacent data is found parameters related to the hash function itself;

(3) Hash all the data into corresponding buckets through the hash function of local sensitive hashing, forming one or more hash tables;

find online

(1) Hash the query data through the hash function of local sensitive hash to obtain the corresponding bucket number;

(2) Take out the corresponding data in the bucket number; in order to ensure the search speed, only take out the first 2L data;

(3) Calculate the similarity or distance between the query data and the 2L data, and return the nearest neighbor data;

The local sensitive hash online search time consists of two parts: ① the time to calculate the hash value through the hash function of the local sensitive hash, that is, the time to calculate the bucket number; ② the time to compare the query data with the data in the bucket. Therefore, the lookup time of locality-sensitive hashing is at least a sublinear time. This is because the matching speed is accelerated by indexing the belongings in the bucket. At this time, the time consumption of part ② changes from O(N) to O(logN) or O(1), which greatly reduces the calculation quantity;

A key of the locality-sensitive hashing is: to map similar samples to the same bucket with high probability; the hash function h(.) of the locality-sensitive hashing satisfies the following conditions:

s{h(f _Aq )=h(f _A )}=sim(f _Aq ,f _A ) (14)

In the formula, sim(f _Aq , f _A ) represents the similarity between f _Aq and f _A , h(f _A ) represents the hash function of f _A , h(f _Aq ) represents the hash function of f _A , where The similarity measure is directly related to a distance function σ, such as:

A typical classification of locality-sensitive hash functions is given by random projections and thresholds, as shown in Equation (16),

h(f _A )=sign(Wf _A +b) (16)

where W is a random hyperplane vector and b is a random intercept.

The locality-sensitive hash is composed of a preprocessing algorithm and a nearest neighbor search algorithm, and the search image features will be represented as a string of fixed-length binary codes through these two algorithm processing;

The process of the preprocessing algorithm is:

Input a set of extracted image features p and the number l ₁ of the hash table, use the random hash function g(.) to map the image features, and store the point p _j in the corresponding bucket number g _{i (} p _j ); output hash table T _i , i=1,...,l ₁ ;

The process of the nearest neighbor search algorithm is:

Input a retrieval image feature q, access the hash table T _i generated by the preprocessing algorithm, i=1,...,l ₁ the number K of the nearest neighbors, and return the K nearest neighbor data of the retrieval point q in the data set S ;

Let Γ={I ₁ ,I ₂ ,…,In } be the retrieved data set consisting of _n images, and the binary code corresponding to each image is Γ _H ＝{H ₁ ,H ₂ ,…,H _n }, H _i ∈ {0,1} ^h ; given a search image I _q and a binary code H _q , put those images whose Hamming distance between H _q and H _i ∈ Γ _H is smaller than the threshold T _H into In the candidate pool P, for candidate images.

4) Using instance features to build a local-sensitive reordering algorithm; in the traditional local-sensitive hashing algorithm, the returned images are mainly close in distance, that is, the similarity between the retrieved image and the image in the candidate pool is close to 1; the main reason is that through The same type of vehicle can be obtained after low-dimensional vehicle attribute hash code mapping, but there are still some indistinguishable situations between vehicles of the same model. There are obvious differences in human subjective judgment, but only through the vehicle attribute hash code These differences cannot be effectively distinguished, as shown in Figure 4; in order to find out the vehicles in the candidate pool that have the same personality characteristics as the retrieved pictures, after the retrieved images are mapped to each bucket through the vehicle attribute hash code, then use the obtained images Instance features, reorder the pictures in the bucket to reduce the intra-class error, and the reordering formula is expressed as follows:

In formula (17), k represents the kth image in the bucket selected by the vehicle attribute hash code mapping, represents the penalty factor and cos represents the cosine distance formula for measuring the characteristics of image instances; in order to eliminate the wrong mapping of vehicle attribute hash codes, y represents the retrieval image f _Aq before mapping and the image in the bucket Whether the types of are equal, if they are equal, y is 1, otherwise it is 0;

In further sorting, those images whose Hamming distance between H _q and H _i ∈ Γ _H are smaller than the threshold T _H have been put into the candidate pool P. In order to obtain more accurate search results, the present invention selects On the basis of further adopting the reordering method;

The reranking method, given a search image I _q and a candidate pool P, uses instance features to determine the top-k ranked images from the images in the candidate pool P; the similarity between them is calculated using Equation (17),

Further, regarding the evaluation of re-ranking, a ranking-based criterion is used for evaluation; given a search image I _q and a similarity measure, a ranking is performed for each dataset image; here, the top k Ranking images to represent the retrieval accuracy of a search image I _q , expressed by formula (18);

In the formula, Rel(i) represents the true correlation between the search image I _q and the i-th ranked image, k represents the number of ranked images, Precision@k search accuracy; when calculating the true correlation, only the part with classification labels is considered , Rel(i)∈{0,1}, if the search image has the same label as the i-th ranked image, set Rel(i)=1, otherwise set Rel(i)=0, traverse the top k in the candidate pool P The search accuracy can be obtained by ranking images;

5) When the retrieved image cannot be obtained, the text retrieval method is used for auxiliary retrieval. Without additional training, the retrieval features obtained through the text and the text features obtained by the convolutional network can share a set of retrieval methods. If A certain text contains a vehicle description information identification tag, as shown in Figure 5, the method for obtaining the features of the text is as follows:

The initialization process is: the text file is parsed into entry vectors; small words and repeated words are removed; the entries are checked to ensure the correctness of the analysis;

5.1) Take out the word segmentation minimum vector R=(r ₁ ,r ₂ ,...,r _n ) of random combination from the input text O;

5.2) Integrate R and f _A sequence and vehicle attribute hash code to get the text attribute characteristic At this time, the dimension of _fATxt is smaller than that of R;

5.3) Retrieve using locality-sensitive reordering hash algorithm;

5.4) return similar image group I;

The above descriptions are only examples of the preferred implementation of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention within.

Claims (8)

1. A modified locality sensitive hashing vehicle retrieval method based on multitask deep learning is characterized by comprising the following steps:

1) Constructing a multi-task end-to-end convolutional neural network for deep learning and training recognition, and deeply learning various attribute information of the vehicle, including vehicle type, vehicle system, vehicle logo, color and license plate, by training data and a network structure which progresses layer by layer;

2) Constructing a vehicle attribute hash code by using the multitask convolutional neural network in the step 1) and adopting a segmented parallel learning and coding strategy;

3) Constructing a characteristic pyramid module by utilizing a pyramid pooling layer and a vector compression layer so as to adapt to convolution feature maps of different sizes and input example characteristics of extracted vehicles;

4) Constructing a local sensitive reordering algorithm by using the example characteristics obtained in the step 3);

5) A cross-mode retrieval method under the condition that a retrieval vehicle image cannot be obtained is constructed, and vehicle retrieval is realized.

2. The revised locality-sensitive hashing vehicle retrieval method based on multitask deep learning as claimed in claim 1, wherein: the multi-task end-to-end convolutional neural network for deep learning and training identification comprises a shared convolution module, an interesting region coordinate regression and identification module and a multi-task learning module;

a shared convolution module: the shared network consists of 5 convolution modules, where the final layers conv2_ x to conv5_ x are {4 } respectively ² ,8 ² ,16 ² ,16 ² Output size as characteristic diagram, conv1 as input layer only contains single layer convolution layer;

the method comprises the following steps that a region-of-interest coordinate regression and identification module is connected behind a shared convolution module, the module takes an image with any size as input, outputs a set of rectangular prediction frames of a target region, and comprises position coordinates of each prediction frame and probability scores of categories in a data set, in order to generate a region suggestion frame, firstly, the input image generates a feature map through a convolution sharing layer, and then, multi-scale convolution operation is carried out on the feature map, and the implementation process is as follows: using 3 scales and 3 length-width ratios at the position of each sliding window, taking the center of the current sliding window as the center and corresponding to one scale and length-width ratio, and then mapping to obtain 9 candidate regions with different scales on an original image; if the shared convolution characteristic diagram with the size of w × h is used, w × h × 9 candidate regions are totally obtained; finally, the classification layer outputs scores of w × h × 9 × 2 candidate regions, namely, the estimation probability that each region is a target/non-target, and the regression layer outputs w × h × 9 × 4 parameters, namely, coordinate parameters of the candidate regions;

when the RPN is trained, each candidate region is allocated with a binary label so as to mark whether the region is an object target, and the operation is as follows: 1) An IoU overlapping candidate region highest to a certain real target region GT; 2) Candidate regions with IoU overlap of more than 0.7 with any GT bounding box. Assigning a negative label to a candidate region for which the IoU ratio of all GT bounding boxes is below 0.3; 3) Between the two.

With these definitions, the objective function is minimized, and the loss function for an image is defined as:

where i is the index of the ith candidate region, P _i Is the probability that the candidate region is of the ith class. If the label of the candidate region is positive,is 1, if the candidate area label is 0,is exactly 0,t _i Is a vector, representing the 4 parameterized coordinates of the predicted bounding box,is the coordinate vector of the corresponding GT bounding box, N _cls And N _reg Normalization of classification loss function and position regression loss function respectivelyCoefficient, λ is a weighting parameter between the two, a classification loss function L _cls Is a log loss of two classes, target and non-target:

regression loss function L for position _reg Defined by the following function:

where R is a robust loss function (smooth L1).

However, training a multitask deep learning network is not an easy process to implement because information at different task levels has different learning difficulties and convergence rates, and the multitask joint training process is as follows: assuming that the total number of tasks is T, the training data for the T-th task is recorded asWherein t is the number of total training samples, i is the number of training samples (1, T), i is the number of training samples (1, N).Respectively, the feature vector and the label of the ith sample, and then the multitask objective function is expressed as:

in the formulaIs an input feature vectorAnd a weight parameter w ^t L (-) is a loss function, phi (w) ^t ) Is a regularization value of a weight parameter;

and for the loss function, training the characteristics of the last layer by utilizing softmax in cooperation with a log-likelihood cost function to realize image classification. The softmax loss function is defined as follows:

in the formula, x _i Is the ith depth feature, W _j B is a bias term, m and n are the number of processed samples and the category number respectively;

the convolutional neural network training is a back propagation process, and the convolutional parameters and the bias are optimized and adjusted by using a random gradient descent method through back propagation of an error function until the network is converged or the maximum iteration times is reached;

the neural network training is a back propagation process, the convolution parameters and the bias are optimized and adjusted by a random gradient descent method through back propagation of an error function until the network is converged or the maximum iteration times are reached;

the back propagation needs to compare the training samples with labels, adopt a square error cost function to identify multiple classes of the c classes and the N training samples, calculate the error by the formula (7) according to the final output error function of the network,

in the formula, E ^N In order to be a function of the squared error cost,for the kth dimension of the label for the nth sample,corresponding to the k output of the network prediction for the n sample;

when the error function is reversely propagated, a calculation method similar to the traditional BP algorithm is adopted, and a specific formula form is shown as a formula (8),

δ ^l ＝(W ^l+1 ) ^T δ ^l+1 ×f'(u ^l )(u ^l ＝W ^l x ^l-1 +b ^l ) (8)

in the formula, delta ^l Error function, δ, representing the current layer ^l+1 Representing the error function of the previous layer, W ^l+1 For the previous layer of the mapping matrix, f' represents the inverse of the activation function, i.e. upsampling, u ^l Output, x, representing the layer above the failed activation function ^l-1 Denotes the input of the next layer, W ^l The weight matrix is mapped for this layer.

3. The revised locality-sensitive hash-ordered vehicle retrieval method based on multitask deep learning according to claim 1 or 2, characterized by comprising: the multi-task has relevance between tasks in the learning process, namely information sharing exists between the tasks,

when a plurality of tasks are trained simultaneously, the network utilizes shared information among the tasks to enhance the induction bias capability of the system and the generalization capability of the classifier; the multitask network is divided into five subtasks by adding five full-connection layers behind an interested region module, each full-connection post-connection softmax activation function normalizes a threshold value between [0 and 1], then sends the normalized value to a segmentation function to promote the output of binary codes, and reduces the redundancy among the hash codes through a segmentation learning and coding strategy so as to enhance the robustness of the learned characteristics;

dividing the multi-task learning network into T tasks, wherein each task contains c ^t One class, full connected layer one-dimensional vector output per task uses m ^t Indicating that the full connection layer is first imported by utilizing softmax activation functionNormalized to [0,1]]The formula is expressed in the following form:

wherein θ represents a random hyperplane; and sending the normalized value into a threshold segmentation function for binarization to obtain binary output of the full-link layer, wherein the formula is specifically expressed as follows:

finally, H obtained by the formula (10) is compared with H in order to obtain the vehicle attribute hash code which is obtained by the segmentation and parallel learning of the multi-task convolutional network ^t The vectors are fused again in a certain proportion, using the vector f _A The formula is expressed in the following concrete form:

f _A ＝[α ¹ H ¹ ；α ² H ² ；...；α ^t H ^t ] (11)

α in the formula (11) ^t The concrete expression form is as follows:

at each H ^t Is previously multiplied by a penalty factor alpha ^t The error caused by different classification numbers among different tasks is compensated.

4. The revised locality-sensitive hashing vehicle retrieval method based on multitask deep learning as claimed in claim 3, wherein: simultaneously creating a characteristic pyramid with strong semantics on all scales by utilizing the pyramid shape of the convolution characteristic hierarchical structure; to achieve this goal, low resolution, semantically strong features are combined with high resolution, semantically weak features by top-down paths and transverse connections by means of a structure; the feature pyramid has rich semantics at all levels, can be quickly constructed from a single input image scale, and can be used to replace a characterized image pyramid without sacrificing representative features, speed or memory. In order to obtain example features of the vehicle image and adapt to the input of a convolution feature map with any size, the last layer of each unit of the sharing modules conv2_ x to conv5_ x is selected and combined with the output of the region-of-interest module, and a pyramid pooling layer and a vector compression layer are added to compress three-dimensional features into a one-dimensional feature vector, so that the selection is that feature map information obtained by a feature pyramid can be enriched, and the deepest layer of each stage has the strongest feature representation function;

the last layer of each module is taken as input of the feature pyramid, and {4 } is selected in turn for the last layer of the networks conv2_ x to conv5_ x defined above ² ,8 ² ,16 ² ,16 ² The size of an input feature map of the feature pyramid is used as the size of the feature pyramid; the input image is represented by I, the length and the width of the input image are respectively represented by letters h and w, the shared convolution module of the x-th layer is represented by convx _ x, the input image I is activated into a three-dimensional feature vector T, the dimension h 'multiplied by w' multiplied by d is a set of a series of two-dimensional feature maps, the length and the width of the two-dimensional feature map are h 'multiplied by w', the T contains d two-dimensional feature maps, and the set S = S { S } is used _n Is represented by ∈ (1, d), S _n Corresponding to the nth channel characteristic diagram; and then, sending the three-dimensional feature vector T into a feature pyramid, and obtaining a three-dimensional feature vector T ' after convolution by a plurality of scale convolution kernels, wherein the dimension is l multiplied by d, the three-dimensional feature vector T ' also comprises a group of two-dimensional feature maps, and can be used as S ' = S ' { S ' _n Denotes n ∈ (1, d), where S _n ' corresponding to the nth channel feature map, each feature map is l × l in size, and the total number of the feature maps is d; then, a sliding window with the size of k multiplied by k is utilized and the maximum pooling is selected to carry out logistic regression on the characteristic graph to obtain a group of characteristic graphs with the size of l/k multiplied by l/k, and then S of each channel is carried out _n ' conducting fusion to obtain a one-dimensional vector, conducting the same operation on d channels in sequence, and finally obtaining an individual characteristic vector f _B The size was (1,l/kXd). The final search feature vector f is shown in equation (13):

f＝[f _A ；f _B ] (13)。

5. the revised locality-sensitive hashing vehicle retrieval method based on multitask deep learning as claimed in claim 4, wherein: performing fast image comparison on the compact binary code by using a hash method and a Hamming distance; the Hash method adopts a locality sensitive Hash algorithm, namely, a Hash bit is constructed by adopting random projection transformation;

one key of the locality sensitive hash is: mapping similar samples to the same bucket with high probability; the hash function h () of the locality sensitive hash satisfies the following condition:

s{h(f _Aq )＝h(f _A )}＝sim(f _Aq ,f _A ) (14)

in the formula, sim (f) _Aq ,f _A ) Denotes f _Aq And f _A Similarity of (c), h (f) _A ) Denotes f _A Hash function of h (f) _Aq ) Denotes f _A The hash function of (1), wherein the similarity measure is directly associated with a distance function σ, such as:

a typical classification of a locality-sensitive hash function is given by a random projection and a threshold, as shown in equation (16),

h(f _A )＝sign(Wf _A +b) (16)

where W is a random hyperplane vector and b is a random intercept.

6. The revised locality-sensitive hashing vehicle retrieval method based on multitask deep learning as claimed in claim 5, wherein: the locality sensitive hash is composed of a preprocessing algorithm and a nearest neighbor search algorithm, and the search image features are represented into a string of binary codes with fixed length through the processing of the two algorithms;

the preprocessing algorithm comprises the following processes:

inputting a set of extracted image features p and a number of hash tables l ₁ Mapping the image features using a random hash function g _j Store to hash table T _i Corresponding barrel number g _i (p _j ) The preparation method comprises the following steps of (1) performing; output hash table T _i ,i＝1,…,l ₁ ；

The nearest neighbor search algorithm comprises the following processes:

inputting a search image feature q, accessing a hash table T generated by a preprocessing algorithm _i ,i＝1,…,l ₁ The number K of nearest neighbors returns K nearest neighbor data of the retrieval point q in the data set S;

let Γ = { I ₁ ,I ₂ ,…,I _n Is a data set composed of n images, and the binary code corresponding to each image is Γ _H ＝{H ₁ ,H ₂ ,…,H _n }，H _i ∈{0,1} ^h (ii) a Given search image I _q And binary code H _q Is prepared from H _q And H _i ∈Γ _H The Hamming distance between is less than the threshold value T _H Are put into the candidate pool P,are candidate images.

7. The revised locality-sensitive hashing vehicle retrieval method based on multitask deep learning as claimed in claim 6, wherein: in the step 4), a local sensitive reordering algorithm is constructed by using the example characteristics; in the traditional locality sensitive hashing algorithm, returned images which are close in distance are mainly used, namely the similarity between a retrieval image and an image in a candidate pool is close to 1; the vehicle attribute hash codes are mapped to obtain vehicles of the same model, but the vehicles of the same model are still difficult to distinguish, and the vehicle attribute hash codes are obviously distinguished from subjective judgment of people, but the differences cannot be effectively distinguished only through the vehicle attribute hash codes; in order to find out vehicles in the candidate pool which have the same individual characteristics as the retrieval pictures, after the retrieval images are mapped into each barrel through the vehicle attribute hash codes, the images in the barrels are sorted again by using the acquired image example characteristics to reduce the errors in the classes, and the representation form of a re-sorting formula is as follows:

in equation (17), k represents the kth image in the bucket selected by the vehicle attribute hash code mapping,represents a penalty factor andcos represents a cosine distance formula for measuring the characteristics of the image instance; to exclude the vehicle attribute hash code from being mapped incorrectly, y represents the pre-mapping search image f _Aq And images in bucketIs equal, y is 1 if equal, otherwise is 0.

In further ranking, H has already been assigned _q And H _i ∈Γ _H Hamming distance therebetween is less than a threshold value T _H The images are put into a candidate pool P, and in order to obtain a more accurate search result, a re-ordering method is further adopted on the basis of the candidate pool;

reordering method, given search image I _q And a candidate pool P, using the instance features to determine top k ranked images from the images in the candidate pool P; the degree of similarity between them is calculated using the formula (17),

further, with respect to the re-ranking evaluation, a ranking-based criterion is used for evaluation; for a given search image I _q And a similarity measure, one ranking for each dataset image; here, a search image I is represented by evaluating the top k ranking images _q Search essence ofDegree, expressed by equation (18);

where Rel (I) denotes a search image I _q The real correlation between the ith ranking image and k represents the number of the ranking images and precision @ k searching precision; when the real correlation is calculated, only a part with a classification label is considered, rel (i) ∈ {0,1}, if the search image and the ith ranking image have the same label setting Rel (i) =1, otherwise, rel (i) =0 is set, and the search precision can be obtained by traversing the first k ranking images in the candidate pool P.

8. The revised locality-sensitive hashing vehicle retrieval method based on multitask deep learning as claimed in claim 1, wherein: in the step 5), when the retrieved image cannot be obtained, a text retrieval mode is adopted for auxiliary retrieval, and under the condition of not adding extra training, the retrieval characteristics obtained through the text and the text characteristics obtained through the convolutional network can share one set of retrieval mode, and the text acquisition characteristic method comprises the following steps:

5.1 Parsing the input text O into entry vectors, removing small words and repeated words, and finally taking out the minimum word segmentation vector R = (R) of random combination ₁ ,r ₂ ,...,r _n )；

5.2 Pair of R and f _A Integrating the sequence and the vehicle attribute Hash codes to obtain text attribute characteristicsAt this time f _ATxt A dimension less than R;

5.3 Using a locality sensitive reordering hash algorithm;

5.4 ) return a similar group of images I.