CN108021869A - A kind of convolutional neural networks tracking of combination gaussian kernel function - Google Patents

Abstract

A kind of convolutional neural networks tracking of combination gaussian kernel function disclosed by the invention, this method step include：First two field picture is normalized first and clusters extraction target information, with reference to target background information during tracking collectively as each rank wave filter in convolutional network structure, convolution algorithm speed is improved by gaussian kernel function, extract target simple abstract feature, then the convolution results for being superimposed simple layer obtain the profound expression of target, finally realize and track with reference to particle filter tracking frame.Convolutional network structure after the present invention is simplified, departs from the depth abstract characteristics of harsh deep learning running environment extraction, can effectively cope with low resolution, the scene such as target occlusion and deformation, improves the tracking efficiency under complex background.

Description

A Convolutional Neural Network Tracking Method Combined with Gaussian Kernel Function

technical field

The invention relates to the field of computer vision target tracking, in particular to a convolutional neural network tracking method combined with a Gaussian kernel function.

Background technique

Visual tracking is a research hotspot in the field of computer vision, and has important research and application value in scenarios such as virtual reality, human-computer interaction, intelligent monitoring, augmented reality, and machine perception. Visual tracking mainly analyzes the video image sequence, matches each detected candidate target area, and locates the position of the tracking target in the video sequence. At present, many research results have been achieved in tracking algorithms, but they still face great challenges in dealing with various complex scenes in practice, such as how to achieve more robustness and accuracy in the face of many factors such as occlusion, deformation, and low resolution of video sequences. The tracking of is still the core of current research.

Most of the traditional tracking algorithms directly use the pixel value features in the video image sequence for modeling. When there are big challenges such as complex scenes in the tracking process, shallow pixel-level features cannot cope well.

Therefore, the inventors further explored and studied it, and proposed a convolutional neural network tracking method combined with a Gaussian kernel function.

Contents of the invention

In order to solve the problem of deep learning in the tracking field, the present invention proposes a convolutional neural network tracking method combined with a Gaussian kernel function:

The present invention proposes a kind of convolutional neural network tracking method combined with Gaussian kernel function, comprising the following steps:

Step 1. Initialization: including the normalization of each frame image, particle filter, network size and sample capacity, and the parameters set include the filter network slice size w*w, the number of filters P, normalization Size n*n, the standard deviation σ _x , σ _y , σ _s of the target state of the particle filter and the use of N particles;

Step 2. Initial filter extraction: For the target of the first frame image, an initial filter bank is extracted through sliding window and K-means clustering for subsequent network filters. This filter bank remains unchanged during the tracking process. Change;

Step 3. According to the convolutional neural network structure, first extract the deep abstract features of each candidate sample, and then use the Gaussian kernel function to accelerate the convolution calculation, which specifically includes:

Step 31. Simple layer feature extraction: For the input image frame, normalize the image to n*n size through preprocessing, and sample the target area using a sliding window of w*w size to obtain an image block group X with a length L ;

Step 32, use k-means clustering method to cluster from L=(n-w+1)×(n-w+1) image blocks to obtain d image block filters as convolution kernels, and convolve Nuclear as

Step 33, the corresponding response to the input image I is shown in formula (1):

Among them, S is the convolution result of the first layer, and F is the convolution kernel;

Step 34, randomly sample the area around the target of the image to obtain l samples, and also perform k-means clustering to obtain the background template of the image:

Step 35. Process the background templates of all images by mean pooling to obtain the average background:

Among them, F ^b is the background convolution kernel, b is marked as the background, d is the total number of background templates obtained, and m is the parameter of the average pooling operation;

Step 36, the feature expression of the simple layer is shown in formula (2):

Step 37, complex layer feature extraction: stack the features of d simple layers to form a three-dimensional tensor to represent the complex layer features of the target, and record the complex layer features as C∈R ^{(n-w+1)× (n-w+1)×d} ;

Step 38, using sparse expression to represent features to obtain the sparse expression of C of the feature tensor, and

Step 39, obtain the target feature expression according to the soft-shrinking method as shown in formula (3):

Step 310, use the Gaussian kernel function to perform convolution operation, the expression of which is shown in the following formula (4):

Among them, * represents the complex conjugate, k(x, x′) represents the Gaussian kernel function;

Step 311, set is a map of a high-dimensional kernel Hilbert space, then the kernel function The weight can be expressed as Among them, the coefficient vector is α, and the element is α _i ;

Step 312, the parameter to be solved is changed from v to α, then the closed-form solution of the kernel regularized least squares classifier (KernelizedRegularized Least Square, KRLS) can be expressed as:

α＝(K+λI) ^-1 y (5)

Wherein, K is a kernel function matrix, and the matrix element is K _ij =k( _xi , x _j ), I is an identity matrix, and the element of vector y is y _i ,

Since K is a circular matrix, the above formula (5) can be converted to the DFT domain,

in, is a vector composed of elements in the first row of the kernel function matrix K, and the symbol ∧ represents Fourier transform;

Step 4. Feature matching and positioning: Use the particle filter tracking framework to perform feature matching and positioning for target tracking.

Described step 4 specifically comprises:

Step 41. Set the total observation sequence at frame t as O _t ={o ₁ ,...,o _t }, and calculate the maximum value of the posterior probability p according to Bayesian theory,

p(S _t |O _t )∝p(O _t |S _t )∫p(S _t |S _t-1 )p(S _t-1 |O _t-1 )dS _t-1 (7)

where S _t = [x _t , y _t , s _t ] ^T , where x _t , y _t are the position of the target, s _t is the scale parameter, p(S _t |S _t-1 ) is the motion model, p(S _t |O _t ) is the observation model.

Step 42. For the motion model p(S _t |S _t-1 ), assuming that the target state parameters are independent of each other and described by three Gaussian distributions, the motion model is Brownian motion.

p(S _t |S _t-1 )=N(S _t |S _t-1 , ∑S) (8)

Where ∑S=diag(σ _x , σ _y , σ _t ) is the diagonal covariance matrix;

Step 43. For the observation model p(S _t |O _t ); calculate by measuring the similarity between sample targets:

Step 44, finally track the target according to formula (9):

After the step 4, if the last frame of image is processed, the result is output, if it is not the last frame of image, then step 5 and 6 are entered in sequence;

Step 5, network update: including adopting the method of limiting the threshold, that is, when the highest confidence value of all particles is lower than the threshold, the network is updated, using the initial filter set, combined with the current foreground filter set obtained during the tracking process, through different The weights are added to obtain a new convolutional network filter;

Step 6. Template update: use a template matching scheme to update the template.

Described step 6 specifically comprises:

Step 61: Take the central point of the object in the first frame as the center and perform equal-sized sampling within the range of ±1 offset to form a set of positive samples.

Step 62: Sampling with the near and far distances of the current frame to form a negative sample set; the positive templates are the same in the entire sequence; preset an update threshold, and update the template once when the threshold is reached.

Compared with the prior art, the present invention has the following advantages:

The method of the present invention introduces a Gaussian kernel function to accelerate calculation, adopts a simplified convolutional neural network, breaks away from the harsh operating environment of a deep learning algorithm, and extracts deep abstract features of a target. The first layer uses K-means to extract normalized image blocks in the first frame as the filter bank to extract the simple layer features of the target, and the second layer stacks simple unit feature maps to form a complex feature map and encodes the target Local structure position information, enabling robust tracking.

The present invention will be further described below in conjunction with the accompanying drawings.

Description of drawings

Fig. 1 is a processing flowchart of a convolutional neural network tracking method combined with a Gaussian kernel function in the present invention.

Detailed ways

As shown in Figure 1, a convolutional neural network tracking method combined with a Gaussian kernel function disclosed in this embodiment specifically includes the following steps:

Step 1. Initialization: including the normalization of each frame image, particle filter, network size and sample size, and the parameters set include the filter network slice size w*w (6×6), the number of filters P=100, normalization size n*n(32×32), the standard deviation of the target state of the particle filter is set as follows: _σx =4, _σy =4, _σs =0.01, using N=300 particles ;

Step 2, initial filter extraction: for the target of the first frame image (i.e. the first frame), an initial filter bank is extracted through sliding window and K-means clustering for the filter use of the subsequent network. During the tracking process, this The filter bank remains unchanged;

Step 3. According to the convolutional neural network structure, first extract the deep abstract features of each candidate sample, and then use the Gaussian kernel function to accelerate the convolution calculation, which specifically includes:

Step 31. Simple layer feature extraction: For the input image frame, normalize the image to n*n size through preprocessing, and sample the target area using a sliding window of w*w size to obtain an image block group X with a length L , where w=6;

Step 32, use k-means clustering method to obtain d image blocks by clustering from L=(n-w+1)×(n-w+1) (this L is the same as L in step 31) image blocks The filter is used as a convolution kernel, and the convolution kernel is denoted as

Step 33, the corresponding response to the input image I is shown in formula (1):

Among them, S is the convolution result of the first layer, and F is the convolution kernel;

Step 34, randomly sample the area around the target of the image to obtain l samples, and also perform k-means clustering to obtain the background template of the image:

Step 35. Process the background templates of all images by mean pooling to obtain the average background:

Among them, F ^b is the background convolution kernel, b is marked as the background, d is the total number of background templates obtained, and m is the parameter of the average pooling operation;

Step 36, the feature expression of the simple layer is shown in formula (2):

Step 37, complex layer feature extraction: stack the features of d simple layers to form a three-dimensional tensor to represent the complex layer features of the target, and record the complex layer features as C∈R ^{(n-w+1)× (n-w+1)×d} ;

Step 38, using sparse expression to represent features to obtain the sparse expression of C of the feature tensor, and

Step 39, obtain the target feature expression according to the soft-shrinking method as shown in formula (3):

Step 310, use the Gaussian kernel function to perform convolution operation, the expression of which is shown in the following formula (4):

Among them, * represents the complex conjugate, k(x, x′) represents the Gaussian kernel function;

Step 311, set is a map of a high-dimensional kernel Hilbert space, then the kernel function The weight can be expressed as Among them, the coefficient vector is α, and the element is α _i ;

Step 312, the parameter to be solved is changed from v to α, then the closed-form solution of the kernel regularized least squares classifier (KernelizedRegularized Least Square, KRLS) can be expressed as:

α＝(K+λI) ^-1 y (5)

Wherein, K is a kernel function matrix, and the matrix element is K _ij =k( _xi , x _j ), I is an identity matrix, and the element of vector y is y _i ,

Since K is a circular matrix, the above formula (5) can be converted to the DFT domain,

in, is a vector composed of elements in the first row of the kernel function matrix K, and the symbol ∧ represents Fourier transform; the closed-form solution of the KRLS classifier can be obtained quickly by using FFT.

Step 4. Feature matching and positioning: use the particle filter tracking framework to perform feature matching and positioning for target tracking, which specifically includes:

Step 41. Set the total observation sequence at frame t as O _t ={o ₁ ,...,o _t }, and calculate the maximum value of the posterior probability p according to Bayesian theory,

p(S _t |O _t )∝p(O _t |S _t )∫p(S _t |S _t-1 )p(S _t-1 |O _t-1 )dS _t-1 (7)

where S _t = [x _t , y _t , s _t ] ^T , where x _t , y _t are the position of the target, s _t is the scale parameter, p(S _t |S _t-1 ) is the motion model, p(S _t |O _t ) is the observation model.

Step 42. For the motion model p(S _t |S _t-1 ), assuming that the target state parameters are independent of each other and described by three Gaussian distributions, the motion model is Brownian motion.

p(S _t |S _t-1 )=N(S _t |S _t-1 , ∑S) (8)

Where ∑S=diag(σ _x , σ _y , σ _t ) is the diagonal covariance matrix;

Step 43. For the observation model p(S _t |O _t ); calculate by measuring the similarity between sample targets:

Step 44, finally track the target according to formula (9):

After the step 4, if the last frame of image is processed, the result is output, if it is not the last frame of image, then step 5 and 6 are entered in sequence;

Step 5, network update: including adopting the method of limiting the threshold, that is, when the highest confidence value of all particles is lower than the threshold, the network is updated, using the initial filter set, combined with the current foreground filter set obtained during the tracking process, through different The weights are added to obtain a new convolutional network filter;

Step 6. Template update: use a template matching scheme to update the template, which specifically includes:

Step 61: Take the central point of the object in the first frame as the center and perform equal-sized sampling within the range of ±1 offset to form a set of positive samples.

Step 62: Sampling with the near and far distances of the current frame to form a negative sample set; the positive templates are the same in the entire sequence; preset an update threshold, and update the template once when the threshold is reached.

The above description shows and describes the preferred embodiments of the present invention, it should be understood that the present invention is not limited to the form disclosed herein, should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments , and can be modified within the scope of the inventive concept herein through the above teachings or techniques or knowledge in related fields. However, changes and changes made by those skilled in the art do not depart from the spirit and scope of the present invention, and should all be within the protection scope of the appended claims of the present invention.

Claims (4)

1. a convolutional neural network tracking method in conjunction with Gaussian kernel function, is characterized in that, comprises the following steps: Step 1. Initialization: including the normalization of each frame image, particle filter, network size and sample capacity, and the parameters set include the filter network slice size w*w, the number of filters P, normalization Size n*n, the standard deviation σ _x , σ _y , σ _s of the target state of the particle filter and the use of N particles; Step 2. Initial filter extraction: For the target of the first frame image, an initial filter bank is extracted through sliding window and K-means clustering for subsequent network filters. This filter bank remains unchanged during the tracking process. Change; Step 3. According to the convolutional neural network structure, first extract the deep abstract features of each candidate sample, and then use the Gaussian kernel function to accelerate the convolution calculation, which specifically includes: Step 31. Simple layer feature extraction: For the input image frame, normalize the image to n*n size through preprocessing, and sample the target area using a sliding window of w*w size to obtain an image block group X with a length L ; Step 32, use k-means clustering method to cluster from L=(n-w+1)×(n-w+1) image blocks to obtain d image block filters as convolution kernels, and convolve Nuclear as <mrow><mi>F</mi><mo>=</mo><mo>{</mo><msub><mi>F</mi><mn>1</mn></msub><mo>,</mo><msub><mi>F</mi><mn>2</mn></msub><mo>,</mo><mn>...</mn><mo>,</mo><msub><mi>F</mi><mi>d</mi></msub><mo>,</mo><mo>}</mo><mo>&amp;Subset;</mo><mi>X</mi><mo>;</mo></mrow> Step 33, the corresponding response to the input image I is shown in formula (1): <mrow><msubsup><mi>S</mi><mi>i</mi><mi>o</mi></msubsup><mo>=</mo><msubsup><mi>F</mi><mi>i</mi><mi>o</mi></msubsup><mo>&amp;CircleTimes;</mo><mi>I</mi><mo>,</mo><msubsup><mi>S</mi><mi>i</mi><mi>o</mi></msubsup><mo>&amp;Element;</mo><msup><mi>R</mi><mrow><mo>(</mo><mi>n</mi><mo>-</mo><mi>w</mi><mo>+</mo><mn>1</mn><mo>)</mo><mo>&amp;times;</mo><mo>(</mo><mi>n</mi><mo>-</mo><mi>w</mi><mo>+</mo><mn>1</mn><mo>)</mo></mrow></msup><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>1</mn><mo>)</mo></mrow></mrow> Among them, S is the convolution result of the first layer, and F is the convolution kernel; Step 34, randomly sampling the area around the target of the image to obtain samples, also perform k-means clustering to obtain the background template of the image: Step 35. Process the background templates of all images by mean pooling to obtain the average background: <mrow><msup><mi>F</mi><mi>b</mi></msup><mo>=</mo><mo>{</mo><msubsup><mi>F</mi><mn>1</mn><mi>b</mi></msubsup><mo>=</mo><mfrac><mn>1</mn><mi>m</mi></mfrac><msubsup><mi>&amp;Sigma;</mi><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>m</mi></msubsup><msubsup><mi>F</mi><mrow><mi>i</mi><mo>,</mo><mn>1</mn></mrow><mi>b</mi></msubsup><mo>,</mo><msubsup><mi>F</mi><mn>2</mn><mi>b</mi></msubsup><mo>=</mo><mfrac><mn>1</mn><mi>m</mi></mfrac><msubsup><mi>&amp;Sigma;</mi><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>m</mi></msubsup><msubsup><mi>F</mi><mrow><mi>i</mi><mo>,</mo><mn>2</mn></mrow><mi>b</mi></msubsup><mo>,</mo><mn>...</mn><mo>,</mo><msubsup><mi>F</mi><mi>d</mi><mi>b</mi></msubsup><mo>=</mo><mfrac><mn>1</mn><mi>m</mi></mfrac><msubsup><mi>&amp;Sigma;</mi><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>m</mi></msubsup><msubsup><mi>F</mi><mrow><mi>i</mi><mo>,</mo><mi>d</mi></mrow><mi>b</mi></msubsup><mo>}</mo></mrow> Among them, F ^b is the background convolution kernel, b is marked as the background, d is the total number of background templates obtained, and m is the parameter of the average pooling operation; Step 36, the feature expression of the simple layer is shown in formula (2): <mrow><msub><mi>S</mi><mi>i</mi></msub><mo>=</mo><msubsup><mi>S</mi><mi>i</mi><mi>o</mi></msubsup><mo>-</mo><msubsup><mi>S</mi><mi>i</mi><mi>b</mi></msubsup><mo>=</mo><mrow><mo>(</mo><msubsup><mi>F</mi><mi>i</mi><mi>o</mi></msubsup><mo>-</mo><msubsup><mi>F</mi><mi>i</mi><mi>d</mi></msubsup><mo>)</mo></mrow><mo>&amp;CircleTimes;</mo><mi>I</mi><mo>,</mo><mi>i</mi><mo>=</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>...</mn><mo>,</mo><mi>d</mi><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>2</mn><mo>)</mo></mrow></mrow> Step 37, complex layer feature extraction: stack the features of d simple layers to form a three-dimensional tensor to represent the complex layer features of the target, and record the complex layer features as C∈R ^{(n-w+1)× (n-w+1} ) ^×d ; Step 38, using sparse expression to represent features to obtain the sparse expression of C of the feature tensor, and Step 39, obtain the target feature expression according to the soft-shrinking method as shown in formula (3): <mrow><mover><mi>c</mi><mo>^</mo></mover><mo>=</mo><mi>s</mi><mi>i</mi><mi>g</mi><mi>n</mi><mrow><mo>(</mo><mi>v</mi><mi>e</mi><mi>c</mi><mo>(</mo><mi>C</mi><mo>)</mo><mo>)</mo></mrow><mi>m</mi><mi>a</mi><mi>x</mi><mrow><mo>(</mo><mn>0</mn><mo>,</mo><mi>a</mi><mi>b</mi><mi>s</mi><mo>(</mo><mrow><mi>v</mi><mi>e</mi><mi>c</mi><mrow><mo>(</mo><mi>C</mi><mo>)</mo></mrow></mrow><mo>)</mo><mo>-</mo><mi>&amp;rho;</mi><mo>)</mo></mrow><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>3</mn><mo>)</mo></mrow></mrow> Step 310, use the Gaussian kernel function to perform convolution operation, the expression of which is shown in the following formula (4): Among them, * represents the complex conjugate, k(x, x′) represents the Gaussian kernel function; Step 311, set is a map of a high-dimensional kernel Hilbert space, then the kernel function The weight can be expressed as Among them, the coefficient vector is α, and the element is α _i ; Step 312, the parameter to be solved is changed from v to α, then the closed-form solution of the kernel regularized least squares classifier can be expressed as: α＝(K+λI) ^-1 y (5) Wherein, K is a kernel function matrix, and the matrix element is K _ij =k( _xi , x _j ), I is an identity matrix, and the element of vector y is y _i , Since K is a circular matrix, the above formula (5) can be converted to the DFT domain, <mrow><msup><mover><mi>&amp;alpha;</mi><mo>^</mo></mover><mo>*</mo></msup><mo>=</mo><mover><mi>y</mi><mo>^</mo></mover><mo>&amp;times;</mo><msup><mrow><mo>(</mo><msup><mover><mi>k</mi><mo>^</mo></mover><mrow><mi>x</mi><mi>x</mi></mrow></msup><mo>+</mo><mi>&amp;lambda;</mi><mo>)</mo></mrow><mrow><mo>-</mo><mn>1</mn></mrow></msup><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>6</mn><mo>)</mo></mrow></mrow> in, is a vector composed of elements in the first row of the kernel function matrix K, and the symbol eight represents Fourier transform; Step 4. Feature matching and positioning: Use the particle filter tracking framework to perform feature matching and positioning for target tracking. 2. a kind of convolutional neural network tracking method in conjunction with Gaussian kernel function as claimed in claim 1, is characterized in that, described step 4 specifically comprises: Step 41. Set the total observation sequence at frame t as O _t ={o ₁ ,...,o _t }, and calculate the maximum value of the posterior probability p according to Bayesian theory, p(S _t |O _t )∝p(O _t |S _t )∫p(S _t |S _t-1 )p(S _t-1 |O _t-1 dS _t-1 (7) where S _t = [x _t , y _t , s _t ] ^T , where x _t , y _t are the position of the target, s _t is the scale parameter, p(S _t |S _t-1 ) is the motion model, p(S _t |O _t ) is the observation model. Step 42. For the motion model p(S _t |S _t-1 ), assuming that the target state parameters are independent of each other and described by three Gaussian distributions, the motion model is Brownian motion. p(S _t |S _t-1 )=N(S _t |S _t-1 , ∑S) (8) Where ∑s=diag(σ _x , σ _y , σ _t ) is the diagonal covariance matrix; Step 43. For the observation model p(S _t |O _t ); calculate by measuring the similarity between sample targets: Step 44, finally track the target according to formula (9): <mrow><msub><mover><mi>S</mi><mo>^</mo></mover><mi>t</mi></msub><mo>=</mo><msub><mi>argmax</mi><msubsup><mrow><mo>{</mo><msubsup><mi>s</mi><mi>t</mi><mi>i</mi></msubsup><mo>}</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></msubsup></msub><mi>p</mi><mrow><mo>(</mo><msub><mi>o</mi><mi>t</mi></msub><mo>|</mo><msubsup><mi>s</mi><mi>t</mi><mi>i</mi></msubsup><mo>)</mo></mrow><mi>p</mi><mrow><mo>(</mo><msubsup><mi>s</mi><mi>t</mi><mi>i</mi></msubsup><mo>|</mo><msub><mover><mi>s</mi><mo>^</mo></mover><mrow><mi>t</mi><mo>-</mo><mn>1</mn></mrow></msub><mo>)</mo></mrow><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>9</mn><mo>)</mo></mrow><mo>.</mo></mrow> 3. a kind of convolutional neural network tracking method in conjunction with Gaussian kernel function as claimed in claim 1, is characterized in that, after described step 4, if what process is last frame image, then output result, if not last For one frame of image, proceed to steps 5 and 6 in sequence; Step 5, network update: including adopting the method of limiting the threshold, that is, when the highest confidence value of all particles is lower than the threshold, the network is updated, using the initial filter set, combined with the current foreground filter set obtained during the tracking process, through different The weights are added to obtain a new convolutional network filter; Step 6. Template update: use a template matching scheme to update the template. 4. a kind of convolutional neural network tracking method in conjunction with Gaussian kernel function as claimed in claim 3, is characterized in that, described step 6 specifically comprises: Step 61: Take the central point of the object in the first frame as the center and perform equal-sized sampling within the range of ±1 offset to form a set of positive samples. Step 62: Sampling with the near and far distances of the current frame to form a negative sample set; the positive template is the same in the whole sequence; preset an update threshold, and update the template once when the threshold is reached.