CN116012955A - Infrared gait recognition method for improving GaitSet - Google Patents

Abstract

The invention discloses an infrared gait recognition method for improving GaitSet, which belongs to the field of computer vision and comprises the following steps: 1) extracting CASIA-C using a lossless silhouette extraction scheme combining Faster R-CNN detection algorithm and Deeplab v3+ semantic segmentation algorithm Human silhouette in the infrared gait data set, and construct a sample set; 2) Introduce silhouette difference fusion module, residual unit, multi-scale feature fusion technology and multi-scale pyramid mapping module on the basis of GaitSet network to improve; 3) Use steps 1) The obtained training samples are used for training; 4) The gait features of pedestrians in the infrared video sequence are extracted to complete the human body identification under the infrared state. This method can fully capture and utilize different granularity and spatio-temporal gait information in infrared images, and does not need to calculate the gait cycle, and can take any sequence and number of infrared gait frame sequences as input.

Description

An infrared gait recognition method based on improved GaitSet

Technical Field

The invention relates to an infrared gait recognition method for improving GaitSet, and belongs to the technical field of computer vision.

Background Art

In recent years, with the development of big data of surveillance videos, gait feature recognition technology with characteristics such as non-contact and difficult to disguise has attracted much attention. Researchers have proposed many effective gait recognition methods, but most of them collect gait information and conduct relevant algorithm performance tests in an environment with sufficient lighting in advance, and pay less attention to infrared gait images collected at night. Since the light is sufficient and stable in the laboratory environment during the day, it is relatively simple to extract the human silhouette of the detection object, and the noise influence is small. However, in the infrared state, due to the low clarity and contrast of the collected gait images, the pre-processed human silhouette often has holes and incomplete limbs, which will have a great impact on the subsequent gait feature extraction. At the same time, the gait recognition models designed in many research methods are mostly for environments with good lighting conditions. Under infrared conditions, these network models will greatly reduce the extraction and expression capabilities of different granularity information in the contour features and deep spatiotemporal features. Therefore, how to eliminate the influence of external redundant information in the gait silhouette and improve the feature extraction ability and recognition expression of the gait recognition model has become the focus of subsequent research.

At present, there are still few related studies in the field of gait recognition, and the existing recognition methods can be roughly divided into two categories: appearance-based and model-based. Among them, the appearance-based method refers to directly extracting features such as human silhouettes or contours from gait video sequences for gait recognition. Although this type of method can obtain gait information such as gait cycle, it is easily affected by the quality of the silhouette. For example, Kale et al. proposed a hidden Markov model method, which can achieve compact and effective gait recognition, but requires frame-by-frame processing and has a large amount of calculation. Han et al. proposed a classic recognition method based on gait energy image (GEI), which can make full use of the information of human appearance shape changing over time. Sokolova and Castro et al. proposed to extract optical flow containing time information from gait contour sequences to obtain the difference change information between adjacent frames, but the extraction process is complex and extremely time-consuming. Yu et al. proposed a recognition method suitable for multiple perspectives. This method is implemented using a stacked autoencoder, which can convert GEI sequences at any perspective and state into GEI sequences at a specified perspective and normal state. Chao et al. proposed the GaitSet method, which can take the gait silhouette image without temporal constraints as input and extract the internal relationship in the gait image through CNN. The model-based method refers to modeling the human limb structure or behavioral habits. Although it can effectively reduce the influence of factors such as noise and occlusion, most models have poor sensitivity to spatiotemporal information and information of different granularity, and the applicable scenarios are very limited. Under infrared conditions, the recognition effect is not ideal. For example, Cunado et al. proposed a pendulum model, using Hough transform to extract the bone lines of the human leg as gait features. Lu et al. proposed a method based on a link model, using image threshold and contour automatic adjustment method to extract human silhouettes, and using SVM (Support Vector Machine) classifier for final recognition. Zhu et al. proposed the LFN and LFN V2 methods, among which the LFN V2 method is fine-tuned on the basis of the LFN method, expanding the scope of obtaining gait contour information, and can effectively eliminate the redundant spatial information contained in the gait contour features. Mei et al. used long short-term memory networks to focus on extracting leg change information and time dimension characteristics within each human gait cycle, effectively improving the recognition accuracy when some features are missing.

Summary of the invention

In order to solve the problems that low-quality infrared gait images have blurred details and low contrast, resulting in holes in the extracted gait silhouettes, and that the existing gait recognition model based on convolutional neural networks has a shallow network layer and cannot fully utilize and capture gait information of different granularities and temporal and spatial dimensions in infrared images, the purpose of the present invention is to provide a gait recognition method that can fully capture and utilize temporal and spatial information of different granularities and dimensions in infrared gait images, and to achieve efficient recognition of human identity in infrared state by designing a lossless extraction scheme for human silhouettes and constructing a dual-stream deep network model with residual multi-scale fusion.

To achieve the above object, an embodiment of the present invention provides an infrared gait recognition method based on an improved GaitSet, which is characterized by comprising the following steps:

1) Split the video sequences in the initial CASIA-C infrared gait dataset into image sequences according to the frame rate;

2) The Faster R-CNN algorithm is used to accurately locate the human body area in the infrared gait image, and the minimum bounding box of the human body is returned while indicating the categories of the content in different areas. The Deeplab v3+ algorithm is used to extract the human silhouette and construct a sample set;

3) Build an improved GaitSet gait recognition network and introduce a silhouette difference fusion module to highlight the dynamic change information between adjacent gait frames;

4) Use the residual unit Bottleneck to replace the convolutional layer to deepen the network layer and improve the model convergence speed;

5) A multi-scale feature fusion technique is proposed to capture gait information of different granularities, making the extracted feature information more representative and discriminative;

6) A multi-scale pyramid mapping module is used to effectively integrate gait information in the horizontal and vertical directions, further enhancing the feature extraction capability of the model;

7) Use the training samples obtained in step 2) to train the network model and complete identity recognition through similarity measurement.

A further technical solution is: first, split the pedestrian gait video stream in the initial CASIA-C dataset into image sequences according to the frame rate; then, use the Faster R-CNN algorithm to detect targets in the image sequence and form human snapshots; finally, input the human snapshot sequence into the Deeplab v3+ semantic segmentation model for human silhouette extraction, and normalize the human silhouette to a size of 64×64, further process the normalized segmentation results, set the pixel values of the human area in the segmented image to 255, and set the pixel values of the rest (background) to 0.

帧的初始步态剪影可以得到

帧的剪影差分融合图，计算为：A further technical solution is to design a gait silhouette difference fusion module.

The initial gait silhouette of the frame can be obtained

The silhouette difference fusion map of the frame is calculated as:

与残差映射

的关系为：A further technical solution is to introduce the residual unit Bottleneck, so that jump connections can be formed between convolutional layers during the network forward propagation process, achieving the purpose of equal mapping of input and output, feature dimensionality reduction, deepening the network layer and accelerating the model convergence speed without causing model performance degradation.

With residual mapping

The relationship is:

，

,

的输出为：When the residual map is 0, the deep layer is calculated

The output is:

，

,

是损失函数的梯度下降：The reverse gradient calculation is as follows, where

is the gradient descent of the loss function:

。

.

A further technical solution is to add two 1×1 convolution branch structures to the network trunk consisting of 4 Bottleneck layers, 2 pooling layers, 2 SP layers and 2 3×3 convolution layers to perform feature sampling at different depths of the trunk to obtain two feature maps, and then fuse the feature maps of the two branches with the trunk feature map through the Add feature map addition method, and finally obtain an output feature map with rich granular feature information. Its calculation expression is:

，

,

SP (Set Pooling) is a set pooling operation that can aggregate frame-level features into separate set-level features containing global information to compress gait information. The calculation process is as follows:

，

,

，

,

表示set-level特征，

表示frame-level特征。In the formula,

Represents set-level features,

Indicates frame-level features.

A further technical solution is to propose a multi-scale pyramid mapping module to introduce gait feature information in both horizontal and vertical directions into the final discrimination task. The number of feature blocks obtained by horizontal and vertical segmentation is calculated as follows:

，

,

，

,

和

代表分割尺度，其中

，

，经纵向、横向分割后，对得到的特征子块

和

进行全局池化操作：In the formula

and

represents the segmentation scale, where

,

After vertical and horizontal segmentation, the feature sub-blocks are obtained

and

Perform global pooling operations:

。

.

和当前快照目标特征

是否一致，计算如下：A further technical solution is to match pedestrian targets, obtain a snapshot of the person target among many category labels, and use cosine similarity measurement to determine the previous target feature.

and the current snapshot target feature

Whether it is consistent is calculated as follows:

。

.

The beneficial effects of adopting the above technical solution are: in order to avoid the occurrence of holes when extracting human silhouettes in low-quality infrared images, the present invention proposes a fine-grained semantic segmentation solution, combining the Faster R-CNN target detection algorithm and the Deeplab v3+ semantic segmentation algorithm to achieve lossless human silhouette extraction; in addition, in order to deepen the hierarchy of the network model, accelerate the model convergence speed, and improve the model's ability to extract and learn different granularity information and spatiotemporal gait features, the residual unit Bottleneck, silhouette difference fusion module, multi-scale feature fusion technology and multi-scale pyramid mapping module are introduced. Experimental results show that the comprehensive performance of the present invention is better than many currently advanced methods.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG1 is a framework flow chart of an improved GaitSet infrared gait recognition method provided by an embodiment of the present invention;

FIG2 is a flow chart of a lossless human silhouette extraction solution provided by an embodiment of the present invention;

FIG3 is a schematic diagram of a lossless human body silhouette provided by an embodiment of the present invention;

FIG4 is a structural diagram of an infrared gait recognition network model of an improved GaitSet provided by an embodiment of the present invention;

FIG5 is a schematic diagram of silhouette difference fusion provided by an embodiment of the present invention;

FIG6 is a schematic diagram of a Bottleneck structure of a residual unit provided in an embodiment of the present invention;

FIG7 is a schematic diagram of a multi-scale feature fusion technology provided by an embodiment of the present invention;

FIG8 is a schematic diagram of the structure of a multi-scale pyramid mapping module provided by an embodiment of the present invention.

DETAILED DESCRIPTION

The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

As shown in FIG1 , a framework flow chart of an improved GaitSet infrared gait recognition method provided by an embodiment of the present invention includes:

1) A lossless silhouette extraction scheme combining the Faster R-CNN detection algorithm and the Deeplab v3+ semantic segmentation algorithm is used to extract human silhouettes from the CASIA-C infrared gait dataset and construct a sample set;

2) Based on the GaitSet network, the silhouette difference fusion module, residual unit, multi-scale feature fusion technology and multi-scale pyramid mapping module are introduced to improve it;

3) Using the training samples obtained in step 1) for training;

4) Extract pedestrian gait features from infrared video sequences and complete human identity recognition in infrared state.

Among them, the lossless silhouette extraction scheme combining the Faster R-CNN detection algorithm with the Deeplab v3+ semantic segmentation algorithm can realize lossless human silhouette extraction. The process is shown in Figure 2, including:

1) Split the CASIA-C infrared gait dataset into gait image sequences according to the frame rate;

2) Use the target detection algorithm Faster R-CNN to detect moving targets in infrared images, accurately locate the human body area in the gait image, return the minimum bounding box of the human body and indicate the category to which it belongs, and form a target snapshot;

和当前快照目标特征

是否一致，如果一致便保存至相似目标集合中，不一致则丢弃，计算为：3) After obtaining the target snapshot, the cosine similarity metric is used to determine the previous target feature

and the current snapshot target feature

Are they consistent? If they are consistent, they are saved in the similar target set. If they are inconsistent, they are discarded. The calculation is:

，

,

4) After the human snapshot matching is completed, the fine-grained semantic segmentation algorithm Deeplab v3+ is used to segment the human silhouette, and the pixel value of the human area in the segmented image is set to 255, and the pixel value of the rest (background) is set to 0. The final silhouette is shown in Figure 3.

As shown in Figure 4, the infrared gait recognition network model of the improved GaitSet takes a 64×64-sized sequential silhouette sequence as input, and then processes it through two branch networks, where the upper branch is a silhouette difference fusion flow, which is composed of a silhouette difference fusion module, a feature extraction and multi-scale feature fusion module, a multi-scale pyramid mapping module, and a recognition classification module. The lower branch is a silhouette flow. After removing the silhouette difference fusion module, the remaining structure is highly consistent with the upper branch. The present invention uses silhouette difference fusion flow and silhouette flow to capture the dynamic change information and spatial information between adjacent gait feature graphs respectively; introduces residual units to deepen the network hierarchy and accelerate the convergence speed of the model; uses multi-scale fusion technology to obtain different granularity features of deep and shallow layers to improve the representativeness and expressiveness of gait information; uses a multi-scale pyramid mapping module to further enhance the model's ability to represent local and global feature information. In the network framework, SP (SetPooling) is a set pooling operation that can aggregate frame-level features into separate set-level features containing global information to achieve gait information compression. The calculation process is as follows:

，

,

，

,

表示set-level特征，

表示frame-level特征。In the formula,

Represents set-level features,

Indicates frame-level features.

As shown in FIG5 , the silhouette difference fusion module compares the grayscale value of the pixel in the image with the threshold value, thereby highlighting the changed part in the image frame and ignoring the similar part. The module can capture the difference information of texture and boundary in adjacent gait silhouettes. This information focuses on the contour edge changes of the moving human body, which can effectively solve the problem that the subtle movements of the human body and the change information of clothing are difficult to capture. The process is defined as:

，

,

，

代表剪影帧数；

和

分别为第

和

帧的剪影图；

为剪影图像中的像素坐标；

表示逆向输出的差分图；

表示正向输出的差分图；

表示按照0.5倍权重相加得到的剪影差分融合图。Where:

,

Represents the number of silhouette frames;

and

Respectively

and

Silhouette drawing of the frame;

is the pixel coordinate in the silhouette image;

A difference diagram showing the reverse output;

A differential graph showing the forward output;

It represents the silhouette difference fusion image obtained by adding the weights by 0.5.

和

分别表示特征输入与残差映射，其关系为：As shown in Figure 6, the present invention uses the residual unit Bottleneck in resnet50 to replace the convolution layer. By adding Bottleneck, a jump connection is formed between the convolution layers during the forward propagation of the network. While achieving the purpose of equal mapping of input and output, feature dimensionality reduction, deepening the network level and accelerating the convergence speed of the model, it will not cause the performance degradation of the model, and greatly optimizes the feature extraction method. The structural design idea of Bottleneck is to first reduce the dimension of the feature map through a 1×1 convolution; then perform a 3×3 convolution operation, and the channel dimension of the feature map remains unchanged; finally, restore the feature dimension again through a 1×1 convolution operation. The whole process uses three batch normalizations (BatchNormalization, BN) to find the balance point between linearity and nonlinearity, so that the output data falls within the nonlinear region, thereby accelerating the convergence speed of the network.

and

Represent the feature input and residual mapping respectively, and their relationship is:

，

,

的输出为：When the residual map is 0, the deep layer is calculated

The output is:

，

,

是损失函数的梯度下降：The reverse gradient calculation is as follows, where

is the gradient descent of the loss function:

。

.

As shown in Figure 7, in order to obtain more accurate and effective features, identify as many different pedestrian identities as possible in the infrared state, and improve the recognition accuracy, the present invention proposes a multi-scale feature fusion technology, which can perform feature sampling on feature maps of different scales, while containing rich semantic information and reducing the loss of detail features. The main road is composed of 4 Bottlenecks, 2 pooling layers, 2 SP layers, and 2 3×3 convolution layers. The 1×1 convolution of the upper and lower double branches performs feature sampling at different depths of the main road to obtain 2 feature maps, and the feature map of the double branches is fused with the feature map of the main road through the Add feature map addition method, and finally an output feature map with rich granular feature information is obtained. The process is defined as:

，

,

、

和

分别为3条通路的输出特征；

为融合后的输出特征；

和

分别为经1个以及2个Bottleneck(2)&Pooling&SP模块处理后的输出特征；

、

分别是卷积核大小为1×1和3×3的卷积层。In the formula,

,

and

They are the output features of the three pathways;

is the output feature after fusion;

and

They are the output features after being processed by 1 and 2 Bottleneck (2) & Pooling & SP modules respectively;

,

They are convolutional layers with kernel sizes of 1×1 and 3×3 respectively.

。其中，

、

和

分别为特征图的高度、宽度及通道数。假设

代表横向分割、

表示分割尺度，

，则在横向分割后会得到

个特征子块

。其中，

的大小与

有关，受下式关系约束，

为横向分割后的特征子块编号，

。As shown in FIG8 , the multi-scale pyramid mapping module is used to perform multi-scale vertical and horizontal segmentation on the feature map F.

.in,

,

and

are the height, width and number of channels of the feature map respectively. Assume

Represents horizontal division,

represents the segmentation scale,

, then after horizontal segmentation we get

Feature sub-block

.in,

The size and

Related, subject to the following relationship,

is the feature sub-block number after horizontal segmentation,

.

。

.

代表纵向分割、

为分割尺度，则在纵向分割后会到

个特征子块

。为了避免分割得到的特征子块中包含重复的特征信息，纵向分割尺度从

开始，即

，定义如下：Similarly, assuming

Represents vertical segmentation,

is the segmentation scale, then after vertical segmentation,

Feature sub-block

In order to avoid repeated feature information in the segmented feature sub-blocks, the vertical segmentation scale is

Start, that is

, defined as follows:

。

.

和

进行全局池化操作，计算过程如下，以便获取特征子块的全局信息。After vertical and horizontal segmentation, the obtained feature sub-blocks

and

A global pooling operation is performed, and the calculation process is as follows to obtain the global information of the feature sub-block.

，

,

和

分别表示全局最大池化操作(Global Max Pooling，GMP)和全局平均池化操作(Global Average Pooling，GAP)。In the formula

and

They represent the global maximum pooling operation (Global Max Pooling, GMP) and the global average pooling operation (Global Average Pooling, GAP) respectively.

In order to verify the effectiveness of the above embodiments, we compare the present invention with other advanced methods by setting the average recognition accuracy under four walking conditions. Specifically, we use the CASIA-C data set to evaluate our invention. The CASIA-C data set contains 153 pedestrians (130 men and 23 women), each of whom has four walking scenes: fast walking, slow walking, normal walking and backpack walking, totaling 1530 infrared gait videos. Four groups of experiments, Test A, Test B, Test C and Test D, were set up for testing. Among them, the purpose of Test A is to evaluate the performance of the algorithm under normal walking conditions, TestB and Test C are experiments to test the influence of walking speed on recognition effect, and Test D is an experiment to test the influence of backpack situation on recognition effect. The training samples are randomly selected from 3 of the 4 normal walking sequences in each sample, totaling 36720 training images. The test set is composed of the remaining 1 normal walking sequence, 2 fast walking sequences, 2 slow walking sequences and 2 backpack walking sequences, totaling 9180 test images. The results are shown in Table 1.

Table 1 Comparison of recognition accuracy of different methods under four walking conditions (unit: %)

As can be seen from Table 1, in Test A, the recognition rate of the present invention is 97.84%, which is 0.75%, 0.39%, 0.21% and 0.05% higher than that of the LFN method, LFN V2 method, CNN method and LSTM method, respectively; in Test B, the recognition rate of the present invention is 96.89%, which is 2.76%, 1.68%, 0.89% and 0.69% higher than that of the other four methods, respectively; in Test C, the recognition rate of the present invention is 97.04%, which is 3.01%, 2.19%, 1.13% and 0.66% higher than that of the other four methods, respectively; in Test D, the recognition rate of the present invention is 93.32%, which is 3.2% higher than that of the LFN method and LFN V2 method, respectively. The V2 method improved by 4.07% and 0.63% respectively, and decreased by 0.74% and 0.86% compared with the CNN method and the LSTM method; in terms of the final average recognition rate, the present invention improved by 2.64%, 1.22%, 0.37% and 0.13% respectively compared with the other four methods, indicating that the present invention has the best comprehensive performance.

The specific embodiments of the present invention are described in detail above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims (7)

1. An infrared gait recognition method for improving GaitSet is characterized by comprising the following steps:

1) Splitting a video sequence in an initial CASIA-C infrared gait data set into an image sequence according to a frame rate;

2) The method comprises the steps of adopting a Faster R-CNN algorithm to accurately locate the region where a human body is located in an infrared gait image, returning to a minimum bounding box of the human body, marking the category to which different region contents belong, extracting human body silhouettes by adopting a deep v3+ algorithm, and constructing a sample set;

3) Constructing a gait recognition network of an improved GaitSet, and highlighting dynamic change information between adjacent gait frames by introducing a silhouette differential fusion module;

4) The residual error unit Bottleneck is used for replacing a convolution layer, so that the network hierarchy is deepened, and the convergence speed of a model is improved;

5) Providing a multi-scale feature fusion technology to capture gait information with different granularities, so that the extracted feature information is more representative and distinguishable;

6) The multi-scale pyramid mapping module is adopted to effectively fuse gait information in the transverse direction and the longitudinal direction, so that the feature extraction capability of the model is further enhanced;

7) Training the network model by using the training sample obtained in the step 2), and completing identity recognition through similarity measurement.

2. The improved GaitSet infrared gait recognition method of claim 1, wherein the sample data generation is specifically performed as follows: firstly, splitting pedestrian gait video stream in an initial CASIA-C data set into an image sequence according to a frame rate; then, performing target detection on the image sequence by adopting a Faster R-CNN algorithm, and forming a humanoid snapshot; finally, inputting the human snapshot sequence into the deep v3+ semantic segmentation model to extract the human silhouette, and normalizing the human silhouette to 64×64.

3. The method for improving GaitSet's infrared gait recognition of claim 1, wherein a gait silhouette differential fusion module is designed, and the calculation formula is:

。

4. the method for infrared gait recognition by improving GaitSet according to claim 1, wherein a residual unit Bottleneck is introduced to form jump connection between convolution layers in the forward propagation process of the network, so that the purposes of constant mapping of input and output, feature dimension reduction, network hierarchy deepening and model convergence speed acceleration are achieved without causing model performance degradation, and input

Mapping with residual->

The relation of (2) is:

，

when the residual map is 0, the deep layer is calculated

The outputs of (2) are:

。

5. the improved GaitSet infrared gait recognition network of claim 1, wherein two 1 x 1 convolved branch structures are added to a network trunk consisting of 4 Bottleneck, 2 pooling layers, 2 SP layers and 23 x 3 convolved layers for feature sampling at different depths of the trunk, the calculation formula is:

。/>

6. the improved GaitSet infrared gait recognition method of claim 1, wherein a multi-scale pyramid mapping module is provided for introducing gait feature information in both the horizontal and vertical directions together into a final discrimination task; the calculation formula of the number of the feature blocks obtained by transverse and longitudinal segmentation is as follows:

，

，

in the middle ofiAndjrepresenting a segmentation scale, wherein

，

After longitudinal and transverse segmentation, the obtained characteristic sub-blocks are +.>

And->

And performing global pooling operation, wherein the calculation formula is as follows:

。

7. the improved GaitSet infrared gait recognition method of claim 2, wherein the pedestrian target is matched, a snapshot of the person's target is taken from a plurality of classes of tags, and the cosine similarity measure is used to determine the last target feature

And current snapshot target feature->

Whether the two calculation formulas are consistent is as follows:

。/>

Images