CN110717402B - Pedestrian re-identification method based on hierarchical optimization metric learning - Google Patents

Abstract

The invention discloses a pedestrian re-identification method based on hierarchical optimization metric learning, which belongs to the field of pedestrian re-identification. The invention proposes a hierarchical optimization strategy, optimizes the parameters of the deep neural network by using the batch gradient descent method, and realizes the convex optimization of the mapping matrix through mathematical transformation. The hierarchical optimization mode can effectively realize the complementary advantages of deep metric learning and traditional metric learning, thereby improving the performance of the algorithm and the performance of pedestrian recognition. It is proposed to construct strong sample constraints, that is, only the samples with the same label with the farthest distance and the samples with different labels with the closest distance are used to construct sample constraints, which can effectively reduce the number of sample constraints, ensure the validity of each sample constraint, and realize the rapid optimization of the model. A new regular term is proposed to measure the difference of the parameter vector by calculating the cosine value of the parameter vector. The given regular form is the summation of the inner product of the parameter vector and its norm normalization loss, which can avoid parameter convergence during the training process. Increase diversity.

Description

A pedestrian re-identification method based on hierarchical optimization metric learning

technical field

The invention belongs to the field of pedestrian re-identification, and more particularly, relates to a pedestrian re-identification method based on hierarchical optimization metric learning.

Background technique

In life, many scenes are installed with a large number of cameras, and these video capture devices can generate massive amounts of data. Pedestrian re-identification technology is to use massive visual information to analyze the characteristics of pedestrians. Pedestrian re-identification technology has a wide range of applications, which can be used for pedestrian tracking, pedestrian age recognition, and pedestrian behavior judgment. In recent years, pedestrian re-identification technology has developed rapidly, among which, the pedestrian re-identification technology based on metric learning is the more mainstream method.

The essence of metric learning can be simply described as: supervised learning of a mapping mechanism, using this mechanism to map samples into a new embedding space; in this embedding space, the sample constraints need to be satisfied: samples with the same label The distance is smaller than the distance between samples with different labels. This complete metric learning has three parts: mapping samples, building sample constraints, computing losses, and updating parameters. The development of metric learning can be divided into two stages: traditional metric learning stage and deep metric learning stage. The modeling process of traditional metric learning can be described as: using a positive semi-definite Mahalanobis matrix to define the Mahalanobis distance, and transforming the problem into a convex optimization problem to obtain the optimal solution of the matrix. The modeling process of deep metric learning is relatively simple, mainly using the strong expressive ability and strong nonlinear mapping ability of deep neural network to practice the goal of metric learning.

Traditional metric learning and deep metric learning are only the same in the loss function, but there are great differences in other aspects. Structurally, traditional metric learning is a single-layer matrix model. Therefore, in terms of optimization, it has advantages that deep metric learning does not have. For example, algorithm convergence is guaranteed, optimization parameters are few, mathematical background is easy to explain, constraints Conditions are easy to convert. However, during the training process of traditional network models, the parameter vectors (or parameter matrices) tend to converge, which makes the network fall into a local optimum and ultimately affects the performance of the model. Compared with traditional metric learning, the optimization algorithms of deep metric learning with multi-layer structure mainly include gradient descent method and its improved optimization algorithm, and the performance effect is generally better than that of traditional metric learning. However, regardless of traditional metric learning or deep metric learning, constructing the ternary sample constraint set of the model will generate a lot of time overhead. In addition, existing metric learning-based person re-identification techniques can only use backpropagation alone to optimize deep models or use mathematical transformations to ensure convex optimization of shallow models.

SUMMARY OF THE INVENTION

In view of the defects and improvement requirements of the prior art, the present invention provides a pedestrian re-identification method based on hierarchical optimization metric learning, the purpose of which is to realize the organic combination of deep neural network and traditional metric learning, and improve the construction of ternary sample constraints. strategy, a new regular term and a hierarchical optimization strategy are proposed, which can effectively improve the distribution of samples and achieve data dimensionality reduction.

In order to achieve the above object, according to an aspect of the present invention, there is provided a pedestrian re-identification method based on hierarchical optimization metric learning, the method comprising the following steps:

S1. Select a training sample set, initialize the dimension _Dime of the embedding space, the learning rate η of the back-propagation algorithm, and the number of iterations N _iter ;

S2. Use the training sample set to pre-train the deep neural network, initialize the deep neural network weight parameter W that removes the loss layer according to the trained deep neural network parameters, and initialize the mapping matrix L with random numbers;

S3. Select some training samples, and map the batch of samples to the embedding space of dimension _Dime under the current W and L;

S4. Construct ternary sample constraints in the embedding space to obtain a ternary sample constraint set;

S5. Traverse all ternary sample constraint sets, and update the mapping matrix L using the maximum marginal nearest neighbor method;

S6. Fix the updated mapping matrix L, and use the back-propagation algorithm to update the network parameter W;

S7. Determine whether the number of iterations is less than _Niter , if so, add 1 to the number of iterations, and go to step S3; otherwise, go to step S8;

S8. Input the sample to be tested into the deep neural network with the loss layer removed under the current W and L to obtain the re-identification result.

其中，Dim_Net表示网络输出向量维度。Specifically, in step S2, a random number in the range of [-1, 1] is used to initialize the mapping matrix

Among them, Dim _Net represents the network output vector dimension.

Specifically, the mapping of the batch of samples to the embedding space of dimension _Dime is as follows:

x′ _i =L×f(W, x _i )

Among them, f(W, x _i ) represents the mapping result of the sample _xi in the neural network model with parameter W, and x′ _i represents the feature vector of the sample in the embedding space.

Specifically, step S4 includes the following steps:

S41. In the embedding space, calculate the distance between the sample x′ _i and other samples with the same label, and select the sample with the farthest distance as x′ _j ;

S42. In the embedding space, calculate the distance between the sample x′ _i and other samples with different labels, and select the sample with the farthest distance as x′ _k ;

S43. Return ternary sample constraints (x′ _i , x′ _j , x′ _k ).

Specifically, the loss function of metric learning is as follows:

Among them, (W, L) represents the loss function of the deep neural network under the weight parameter W and the mapping matrix L, x′ _i , x′ _j , x′ _k represent the samples x _i , x _j , x _k in the embedding space The feature vector in , the samples x _i , x _j belong to the same pedestrian, the samples x _i , x _k do not belong to the same pedestrian, S represents the ternary sample constraint set, D _M (,) represents the Mahalanobis distance between the two, ρ represents the interval parameter, and R(W) represents the regular term corresponding to the weight parameter W.

Specifically, the calculation formula of the regular term of the weight of the hth layer of the neural network is as follows:

表示W_h的第i分量，<,>表示向量的内积运算，λ、γ为超参数。Among them, R(W _h ) represents the regular term of the h-th layer, W _h represents the weight of the h-th layer of the deep neural network, N _h represents the number of weight components of the h-th layer,

Represents the i-th component of W _h , <,> represents the inner product operation of vectors, and λ and γ are hyperparameters.

In order to achieve the above object, according to another aspect of the present invention, a computer-readable storage medium is provided, and a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the first aspect is implemented The described pedestrian re-identification method based on hierarchical optimization metric learning.

In general, through the above technical solutions conceived by the present invention, the following beneficial effects can be achieved:

(1) Aiming at the existing pedestrian re-identification technology based on metric learning, the deep model can only be optimized by the back-propagation technology alone or the convex optimization problem of the shallow model can be ensured by using mathematical transformation. The present invention proposes a hierarchical optimization strategy , using the pre-trained model to optimize the network parameters, using the batch gradient descent method to optimize the parameters of the deep neural network, and realizing the convex optimization of the mapping matrix through mathematical transformation. The hierarchical optimization mode can effectively realize the complementary advantages of deep metric learning and traditional metric learning, thereby improving the performance of the algorithm and the performance of pedestrian recognition.

(2) When constructing the sample constraints of the model, the traditional technology requires a lot of time overhead, and many of the generated sample constraints are invalid (the resulting loss is zero), the present invention proposes a method to construct strong sample constraints method, that is, only the samples with the same label at the farthest distance and the samples with different labels at the nearest distance are used to construct sample constraints. This method can effectively reduce the number of sample constraints, and can also ensure the effectiveness of each sample constraint, thereby achieving rapid model optimization.

(3) In the training process of the traditional network model, the parameter vector (or parameter matrix) is easy to converge, which makes the network fall into the local optimal problem. Above, the difference of the parameter vector is measured by calculating the cosine value of the parameter vector (or matrix), and the final regular form is the sum of the inner product of the parameter vector (or matrix) and its norm normalization loss. This regularization method can avoid parameter convergence and improve diversity during training.

Description of drawings

1 is a flowchart of a pedestrian re-identification method based on hierarchical optimization metric learning provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of a process of mapping a sample to an embedding space according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

The inventive concept of the present invention is as follows: using a deep neural network to give a new representation feature of the sample, then passing the new representation to a mapping matrix, and finally mapping the sample into a new embedding space. In this embedding space, an improved strategy is used to construct a ternary sample constraint set. Finally, according to the ternary sample constraint set, a hierarchical optimization strategy is used to update the model parameters.

As shown in FIG. 1 , the present invention provides a pedestrian re-identification method based on hierarchical optimization metric learning, and the method includes the following steps:

Step S1. Select a training sample set, and initialize the dimension _{Dime of the embedding space, the learning rate η of the back-propagation algorithm, and the number of iterations Niter .}

The training sample set is {x ₁ , x ₂ , ..., x _i , ..., x _n }, the sample x _i represents a pedestrian image from one perspective, and each person corresponds to a label, that is, the same person's sample labels from different perspectives The same, therefore, the number of labels is equal to the number of pedestrians.

In this embodiment, the dimension _Dime of the embedding space is used in step S3 and is initialized to 64; the learning rate η is used in step S6 and is initialized to 0.001; the number of iterations N _iter is used in step S7 and is initialized to 3000.

Step S2. Use the training sample set to pre-train the deep neural network, initialize the deep neural network weight parameter W that removes the loss layer according to the trained deep neural network parameters, and initialize the mapping matrix L with random numbers.

The deep neural network used in this example is a conventional convolutional neural network. The network structure is: convolutional layer, maximum pooling layer, convolutional layer, maximum pooling layer, fully connected layer, and fully connected layer. The size of the product kernel is 3*3, and the step size is 1. Due to the loss function of metric learning, the deep neural network in the present invention does not have the last loss layer in the conventional network. When initializing the weights of the network, first complete the network to train a traditional network; then remove the last layer to initialize the network part of the model of the present invention.

其中，Dim_Net表示网络输出向量维度。Initialize the mapping matrix with random numbers in the range [-1, 1]

Among them, Dim _Net represents the network output vector dimension.

Step S3. Select some training samples, and map the batch of samples to the embedding space of dimension _Dime under the current W and L.

As shown in Figure 2, the original space refers to the original visual image, and the mapping process refers to the process of mapping the image to the embedding space. In traditional metric learning, a mapping matrix is needed for samples from the original space to the embedding space. The representation sample of the embedding space is mapped into a new space by the mapping matrix. Generally speaking, the dimension of the embedding space is lower than the dimension of the original space.

x′ _i =L×f(W, x _i )

Among them, f(W, _xi ) represents the mapping result of _xi in the neural network model whose parameter is W.

Step S4. Construct ternary sample constraints in the embedding space to obtain a ternary sample constraint set.

The ternary sample constraint means that the distance relationship between the three samples should satisfy the constraint specified in the present invention—the distance between samples with the same label is smaller than the distance between samples with different labels. It traverses the samples in the batch sample set, and constructs sample constraints with the most distant similar samples and the nearest heterogeneous samples. Its beneficial effect is to reduce the number of ternary sample constraint sets and obtain more constrained sample constraints.

Step S4 includes the following steps:

S41. In the embedding space, calculate the distance between the sample x′ _i and other samples with the same label, and select the sample with the farthest distance as x′ _j .

The distance in the embedding space is chosen as Euclidean distance.

S42. In the embedding space, calculate the distance between the sample x′ _i and other samples with different labels, and select the sample with the farthest distance as x′ _k .

S43. Return ternary sample constraints (x′ _i , x′ _j , x′ _k ).

A ternary sample constraint is constructed in the embedding space, and a ternary sample constraint set is obtained.

Step S5. Traverse all ternary sample constraint sets, and update the mapping matrix L by using the maximum margin nearest neighbor method.

The goal of Large Margin Nearest Neighbor (LMNN) is to learn a Mahalanobis distance metric

Among them, the positive semi-definite symmetric matrix M can be expressed as M=L ^T L

The update process is as follows:

(1) Calculate the gradient G _t+1 , the calculation formula is as follows:

Among them, G _t represents the gradient of the t-th iteration, μ represents the weight coefficient, which is generally 0.5, S _t represents the t-th iteration ternary sample constraint set, and D _M represents the Mahalanobis distance.

(2) Update the Mahalanobis matrix

M _t+1 =SDP(M _t -αG _t+1 )

Among them, M _t represents the t-th iteration of the Mahalanobis matrix, α represents the weight coefficient, and SDP (Semi-definite Programming, semi-definite regularization) is a convex function, which ensures that the Mahalanobis matrix is a semi-positive definite matrix.

(3) Update the mapping matrix L

The model is trained using ternary constraints for constructing samples, and the mapping matrix is optimized using traditional metric learning algorithms.

Step S6. The updated mapping matrix L is fixed, and the network parameter W is updated by using the back-propagation algorithm.

Loss function in metric learning.

Among them, Φ(W, L) represents the loss function of the deep neural network under the weight parameter W and the mapping matrix L, x′ _i , x′ _j , x′ _k represent the samples x _i , x _j , x _k in the embedding space The feature vector in , the samples x _i , x _j belong to the same pedestrian, the samples x _i , x _k do not belong to the same pedestrian, S represents the ternary sample constraint set, ρ represents the interval parameter, generally 1, R(W) represents The regular term corresponding to the weight parameter W.

On the basis of the loss function Φ(W, L), the network parameter W is updated by using the back-propagation algorithm. The learning rate η of the backpropagation algorithm is 0.001.

Preferably, in order to make each component of the weight of the same layer as different as possible, and increase the difference of neurons in each layer by maximizing the cosine similarity, the present invention designs a special regular term - the regular term contains The cosine term and the norm constraint term are combined with two weight coefficients λ and γ. The beneficial effect of the regular term is to avoid the appearance of approximate neurons in the optimization process, which is beneficial to increase the richness of new features.

The regular term of the weight of the h-th layer of the neural network is expressed as follows:

表示W_h的第i分量，<，>表示向量的内积运算，超参数λ、γ需要事先定好，本实施例中设置为0.5、0.5。Among them, R(W _h ) represents the regular term of the h-th layer, W _h represents the weight of the h-th layer of the deep neural network, N _h represents the number of weight components of the h-th layer,

represents the i-th component of W _h , and <, > represents the inner product operation of vectors. The hyperparameters λ and γ need to be determined in advance, and are set to 0.5 and 0.5 in this embodiment.

Step S7. Determine whether the number of iterations is less than N _iter , if so, add 1 to the number of iterations, and go to step S3; otherwise, go to step S8.

Step S8. Input the sample to be tested into the deep neural network with the loss layer removed under the current W and L to obtain a re-identification result.

The optimized parameters are output to obtain the model of the algorithm. So far, the present invention has obtained a mapping mechanism, which can map pictures obtained from various perspectives to the embedding space through this mapping mechanism, and match pedestrians in this space.

The strong expressive ability of the deep neural network is used to extract the strong features of the samples, and then traditional metric learning is used to optimize a mapping matrix based on the strong features, and to ensure that the mapping matrix is optimal under the current strong features. This optimization strategy combines deep neural networks with traditional metric learning.

Those skilled in the art can easily understand that the above are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, etc., All should be included within the protection scope of the present invention.

Claims (4)

1. A pedestrian re-identification method based on hierarchical optimization metric learning is characterized by comprising the following steps:

s1, selecting a training sample set, and initializing dimension Dim of an embedding space _e Learning rate eta and iteration number N of back propagation algorithm _iter The training sample set is { x ₁ ，x ₂ ，…，x _i ，…，x _n }, sample x _i The method comprises the steps that a pedestrian picture under one visual angle is represented, each person corresponds to one label, namely the labels of samples of the same person under different visual angles are the same, and therefore the number of the labels is equal to the number of pedestrians;

s2, pre-training a deep neural network by using a training sample set, initializing a deep neural network weight parameter W for removing a loss layer according to the trained deep neural network parameters, and initializing a mapping matrix L by using a random number;

s3, selecting part of training samples, and mapping the batch of training samples to Dim with dimension under the current W and L _e The embedding space of (a);

s4, constructing a ternary sample constraint in the embedding space to obtain a ternary sample constraint set;

s5, traversing all the ternary sample constraint sets, and updating a mapping matrix L by using a maximum marginal nearest neighbor method;

s6, fixing the updated mapping matrix L, and updating the weight parameter W by using a back propagation algorithm;

s7, judging whether the iteration frequency is less than N _iter If so, adding 1 to the iteration number, and turning to the step S3, otherwise, turning to the step S8;

s8, inputting the sample to be detected into the current W, L deep neural network with the loss layer removed, and obtaining a re-identification result;

step S4 includes the following steps:

s41. in the embedding space, calculating a sample x' _i Distance from other same label samples, selecting the sample with the farthest distance as x' _j ；

S42, in an embedding space, calculating a sample x' _i Distance from other different label samples, selecting the sample with the farthest distance as x' _k ；

S43, returning ternary sample constraint (x' _i ，x′ _j ，x′ _k )；

The loss function for metric learning is as follows:

where Φ (W, L) represents a loss function, x 'of the deep neural network under the weight parameter W and the mapping matrix L' _i ，x′ _j ，x′ _k Represents a sample x _i ，x _j ，x _k Feature vector in embedding space, sample x _i ，x _j Belonging to the same pedestrian, sample x _i ，x _k Not belonging to the same pedestrian, S represents a ternary sample constraint set, D _M (,) represents the mahalanobis distance between the two, ρ represents the interval parameter, and R (W) represents the regular term corresponding to the weight parameter W;

the regular term calculation formula of the h-th layer weight of the neural network is as follows:

wherein R (W) _h ) Regular term, W, representing the h-th layer _h H-th layer weight, N, representing a deep neural network _h Indicates the number of weight components of the h-th layer,

represents W _h The (i) th component of (a),<，>represents the inner product operation of the vector, and lambda and gamma are hyper-parameters.

2. The method of claim 1, wherein in step S2, use is made of [ -1, 1 [ -1 [ ]]Random number in range, initializing mapping matrix

Wherein, Dim _Net Representing the network output vector dimension.

3. The method of claim 1, wherein mapping the batch of samples to a dimension of Dim _e The embedding space of (a) is:

x′ _i ＝L×f(W，x _i )

wherein, f (W, x) _i ) Represents a sample x _i Mapping result, x 'of neural network model with weight parameter W' _i Representing the feature vector of the sample in the embedding space.

4. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, implements the pedestrian re-identification method based on hierarchical optimization metric learning according to any one of claims 1 to 3.

Images