CN109815911B - Video moving object detection system, method and terminal based on depth fusion network - Google Patents

Abstract

The present invention provides a video moving object detection system based on a deep fusion network, comprising: a video feature extraction module, which receives video sequence input, performs feature extraction on the video content, and obtains the feature expression about scene information in the video, that is, the video scene Feature expression, and send to the deep fusion module; basic result detection module, receive video sequence input, use basic detectors to detect moving objects, get the corresponding basic detection results, and send to the deep fusion module; deep fusion module, receive video The scene feature expression and basic detection results are optimally fused using deep neural networks to output the final detection results. At the same time, a video moving object detection method and terminal are provided. The present invention can obtain high-accuracy detection results.

Description

Video moving object detection system, method and terminal based on deep fusion network

technical field

The present invention relates to the technical field of video moving object detection, and in particular, to a video moving object detection system, method and terminal based on a deep fusion network.

Background technique

Video moving object detection can be used as the first link of video image processing and video content analysis to provide preliminary analysis results for subsequent operations and help to improve the performance of the entire video processing and analysis system. Therefore, video moving object detection is a crucial step. important technology.

For the video moving object detection problem, researchers have proposed a large number of methods. However, most of these research results are designed for a certain or a certain type of specific scenarios, based on feature engineering, using the method of hand-designed operators. These traditional methods are classified into statistical model-based, cluster-based, and sparse representation-based types. At present, there is no traditional method that can robustly deal with various scenarios, and most of them are only efficient for some scenarios and poor for other scenarios.

Recently, a small number of video moving object detection methods based on deep learning have appeared. The biggest difference between these methods and traditional methods is that no manual parameter adjustment is required, but the detection model is automatically learned from the data. For example, Wang et al. used a deep convolutional network to design a semi-automatic video moving object detection algorithm. This method needs to manually mark the detection results of some key frames, and then the deep convolutional neural network is trained according to the marked results. After the training is completed, the remaining video frames are automatically analyzed to obtain the moving object detection results of these frames. This method can obtain high-accuracy detection results, but requires manual intervention and cannot be completed automatically.

The biggest difficulty in using deep learning to obtain detection models is the lack of training data. Without enough labeled data, the neural network cannot be effectively trained. At present, there is no description or report of the technology similar to the present invention, and no similar materials at home and abroad have been collected.

SUMMARY OF THE INVENTION

Aiming at the above-mentioned deficiencies in the prior art, the present invention provides a video moving object detection system, method and terminal based on a deep fusion network. Combined with the traditional method and deep learning technology, very robust detection can be achieved in various scenarios. result.

The present invention is achieved through the following technical solutions.

According to one aspect of the present invention, a video moving object detection system based on a deep fusion network is provided, including the following modules:

The video feature extraction module receives the input of the video sequence, performs feature extraction on the video content, obtains the feature representation of the scene information in the video, that is, the feature representation of the video scene, and sends it to the deep fusion module;

The basic result detection module receives the video sequence input, uses the basic detector to detect the moving object, obtains the corresponding basic detection result, and sends it to the deep fusion module;

The deep fusion module receives the feature expression of the video scene and the basic detection results, uses the deep neural network for optimal fusion, and outputs the final detection result.

Preferably, the video feature extraction module uses a pre-trained VGG-16 network as a feature extractor, extracts the features of each frame of video, and then stacks the features of each frame of video together to form a set of features for describing the video scene. Descriptor, that is, the representation of video scene features.

Preferably, there are a plurality of basic detectors, wherein each basic detector adopts a traditional motion detection method to detect a moving object to obtain a plurality of corresponding basic detection results.

Preferably, there are four basic detectors, and correspondingly, each basic detector adopts the following traditional motion detection methods:

- Pixel-based adaptive semantic association segmentation method;

- Front and back background segmentation method based on edge detection;

- Background segmentation method based on shared model;

- Background segmentation method based on sampling point weighting. .

Preferably, the deep fusion module receives the feature expression of the video scene as input, obtains the optimal fusion weight map through four convolution layers and one Soft-Max layer, and then performs pixel-by-pixel processing on the basic detection results according to the optimal fusion weight map. Linear weighting.

According to another aspect of the present invention, a video moving object detection method based on a deep fusion network is provided, comprising the following steps:

S1: Sequentially read the current frame in the video and the multiple frames before the current frame as the video sequence input;

S2: Use the feature extractor to analyze each frame of video in the input video sequence to obtain multiple sets of video frame features, and stack the multiple sets of video features in the channel direction to form a descriptor describing the characteristics of the video scene. That is, the video scene descriptor; use the traditional motion detection method to analyze the moving objects of the input video sequence, and obtain the basic detection results;

S3: Input the video scene descriptor and the basic detection result obtained in S2 into the deep fusion network; the deep fusion network analyzes the video scene descriptor to obtain the optimal fusion weight map, and uses the optimal fusion weight map to analyze the The basic detection results are linearly weighted and fused.

Preferably, the deep fusion network is based on a deep convolutional network, and the input video scene descriptor is passed through four convolution layers and one Soft-Max layer to obtain an optimal fusion weight map, and then based on the optimal fusion weight map The detection results are linearly weighted pixel by pixel.

Preferably, the method for detecting moving objects based on the deep fusion network further includes offline training of the feature extractor and the deep fusion network, and the steps are as follows:

Randomly sample video clips in the training video as the predicted motion mask, and together with the annotation mask of the real moving object, that is, the real motion mask, as a training pair, multiple training pairs constitute a training set; Randomly crop to obtain training samples, and then randomly flip the training samples left and right and up and down to expand the training set;

Using one training pair as input, the parameters of the feature extractor and the deep fusion network are jointly optimized using a stochastic gradient descent algorithm, and multiple rounds of learning are performed on all training pairs in the training set until the loss converges.

Preferably, the loss function used in the stochastic gradient descent algorithm is the average variance of the predicted motion mask and the real motion mask.

Preferably, the parameter update rate of the deep fusion network is set to be 100-10000 times the update rate of the parameters of the feature extractor.

According to a third aspect of the present invention, a detection terminal is provided, comprising: a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor can be used to execute the above-mentioned program when the processor executes the program Video moving object detection method based on deep fusion network.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention makes full use of a variety of existing traditional video moving object detection systems to improve the effectiveness for different scenarios;

2. The present invention makes full use of deep learning technology to improve the ability to describe high-level semantic features of video images;

3. The present invention automatically learns the parameters in the system from the data, and does not need to use parameter adjustment based on feature engineering;

4. The present invention combines the traditional method and the deep learning method to obtain a robust and high-performance video moving object detection system and method, which has high detection accuracy for various scenarios;

5. The present invention combines the high-efficiency performance of traditional methods for specific scenes and the powerful expressive ability of deep learning to extract video image content features, and utilizes a deep fusion network to perform optimal fusion of various traditional detection results according to video scene characteristics, so that Robust detection results can be obtained in various scenarios.

Description of drawings

Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

1 is a structural block diagram of a video moving object detection system based on a deep fusion network provided by an embodiment of the present invention;

FIG. 2 is a flowchart of a method for detecting moving objects based on a deep fusion network provided by an embodiment of the present invention.

Detailed ways

The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

An embodiment of the present invention provides a video moving object detection system based on a deep fusion network, including the following modules:

Module 1: Video feature extraction module, which receives video sequence input, performs feature extraction on video content, and obtains the feature expression of scene information in the video, that is, the video scene feature expression, and sends it to the deep fusion module for the deep fusion module. Optimal fusion of basic detection results;

Module 2: Basic result detection module, receives video sequence input, uses basic detectors to detect moving objects, obtains corresponding basic detection results, and sends them to the deep fusion module;

Module 3: Deep fusion module, which receives video scene feature expression and basic detection results, uses deep neural network for optimal fusion, and outputs final detection results.

In some preferred embodiments, the video feature extraction module uses a pre-trained VGG-16 network as a feature extractor, extracts the features of each frame of video, and then stacks the features of each frame of video together to form a group that can describe the video scene. descriptor.

Further, the basic result detection module has a plurality of basic detectors, wherein each basic detector adopts a traditional motion detection method to detect the moving object to obtain a plurality of corresponding basic detection results. In an embodiment, the number of basic detectors may be four, or may be other numbers. For example, when there are four basic detectors, correspondingly, the motion detection method may adopt the following methods, but is not limited to the following methods: PWACS, EFIC, SharedModel and WeSamBE. Among them, PWACS is a pixel-based adaptive semantic association segmentation method; EFIC is a front and back background segmentation method based on edge detection; ShareModel is a background segmentation method based on a shared model; WeSamBE is a background segmentation method based on sampling point weighting. The above methods are all traditional methods based on non-deep learning for moving object detection. Of course, in different embodiments, different motion detection methods may be used, and in this embodiment of the present invention, a more robust detection result can be obtained by fusing the results of several traditional motion detection methods.

In some preferred embodiments, the deep fusion module receives the video scene feature expression (video feature descriptor) as input, obtains the optimal fusion weight map through four convolution layers and one Soft-Max layer, and then obtains the optimal fusion weight map according to the optimal fusion The weight map performs pixel-by-pixel linear weighting on the underlying detection results.

The embodiment of the present invention also provides a video moving object detection method based on a deep fusion network, the steps of which include:

Step 1: Sequentially read the current frame in the video and its previous multiple frames (for example, 16 frames, the number is determined according to the input format of the specific implementation, if the implementation is different, the number here will also change) as the video sequence input;

Step 2: Use the feature extractor to analyze each frame of video in the input video sequence to obtain the features of multiple groups (when it is 16 frames, 16 groups of video frames are obtained here) video frame features, The features are stacked together in the channel direction to form a descriptor that describes the characteristics of the video scene; at the same time, the traditional motion detection method is used to analyze the moving objects of the input video sequence to obtain the basic detection results;

Step 3: Input the video scene descriptor and basic detection results obtained in step 2 into the deep fusion network; the deep fusion network further analyzes the video scene descriptor to obtain the optimal fusion weight map, and finally uses the optimal fusion weight. Figure, linearly weighted fusion of basic detection results.

In step 1, the video sequence input is used as the system input, which is a video clip including the current frame and its previous multiple frames.

In step 2, the output of the feature extractor is a set of feature maps based on deep learning.

In step 3, the optimal fusion is based on a deep convolutional network. The final step of fusion is a linear weighting operation based on the optimal fusion graph.

Further, the method may also include an offline training step for the feature extractor and the deep fusion network, as follows:

Step 1: Randomly sample video clips in the training video as the predicted motion mask, and together with the annotation mask of the real moving object, that is, the real motion mask, as a training pair, multiple training pairs constitute a training set; The training video is randomly cropped to obtain training samples, and then the samples are randomly flipped left and right and up and down to expand the training set;

Step 2: Using a training pair as input, use the stochastic gradient descent algorithm to jointly optimize the parameters of the feature extractor and the deep fusion network, and perform multiple rounds of learning on all training pairs in the training set until the loss converges.

In step 1, the size of the training samples can be 128x128 or other sizes, depending on the computing resources. If the computing resources allow, a larger size, such as 256x256, or 512x512, can be used.

In step 2, the loss function used in the stochastic gradient descent algorithm may be the average variance of the predicted motion mask and the real motion mask. Further, the update rate of the parameters of the deep fusion network is set to be 100-10000 times the update rate of the parameters of the feature extractor. The joint optimization method in step 2 is to perform a gradient descent method on the error of the basic detection result, and gradually iteratively optimize. After the optimal model parameters after training are saved, they are directly used in the video moving object detection method.

Based on the above, the technical solutions of the present invention are further described in detail below with reference to the accompanying drawings and specific examples.

As shown in FIG. 1, a video moving object detection system based on a deep fusion network in an embodiment of the present invention includes three types of modules: a video feature extraction module (video feature extraction network), a basic result detection module and a deep fusion module (Deep Fusion Networks).

In this embodiment, the system includes a video feature extraction module and a deep fusion module, and the type and number of basic detection systems in the basic result detection module can be flexibly selected according to specific scene characteristics and the performance of the processing platform.

In this embodiment, the video feature extraction module uses a pre-trained VGG-16 network as a feature extractor, analyzes all video frames in a video segment in turn, and stacks the obtained feature maps as video feature descriptors.

In this embodiment, the basic result detection module adopts four basic detection systems: PWACS, EFIC, SharedModel and WeSamBE, which have complementary performance in dynamic background, night scene, lens shake and infrared scene.

In this embodiment, the deep fusion module is mainly composed of four convolution layers and one Soft-Max layer concatenated. The module receives the video descriptor as input, further analyzes to obtain the optimal fusion weight map, and then performs pixel-by-pixel linear weighting on the basic detection results according to the weight map.

As shown in Figure 2, in a specific embodiment, the method for detecting video moving objects by using a video moving object detection system based on a deep fusion network, the steps are as follows:

Step 1, read sequentially including the current frame and its previous 16 frames as system input (video sequence input);

Step 2: Use the VGG-16 network to analyze each frame, and stack the last layer of feature maps obtained by each frame through the network in the channel direction to form a descriptor describing the video features;

At the same time, the basic result detection module is used to analyze the system input, and 4 basic detection results are obtained, denoted as B(n), n=1, 2, 3, 4, which represent the detection results of the four basic detection methods;

Step 3: Input the video descriptor and basic detection result in Step 2 into the deep fusion network. The deep fusion network further analyzes the video descriptor to obtain the optimal fusion weight map, and finally uses the weight map M to perform linear weighted fusion of the basic detection results as shown in formula (1);

In formula (1), B(n) represents the nth basic detection result, and M(n) represents the weighting coefficient corresponding to the nth basic detection result. They are all two-dimensional images with the same size as the input video frame. ⊙ represents the element-wise multiplication. Therefore, formula (1) indicates that the pixel-by-pixel weighted average of the four basic detection results is used as the final prediction result P.

In this embodiment, the offline training steps for the parameters in the feature extractor and the deep fusion network are as follows:

Step 1: Randomly sample video clips in the training video, and use them as training pairs together with the annotation masks of real moving objects. Randomly crop the training video to obtain 128x128 training samples, and then randomly flip the samples left and right and up and down to expand the training set;

Step 2: Use the stochastic gradient descent algorithm to jointly optimize the parameters in the entire system until the loss converges;

The optimization method in step 2 is the Adam optimization method. The loss function is set to formula (2):

In formula (2), H and W represent the height and width of the image, and G represents the real motion annotation mask;

In step 2, the learning rate of the parameters in the video feature extraction module is set to 10 ^-7 , and the learning rate of the deep fusion network is set to 10 ^-4 . After the training converges, save the parameters and directly load and use them in actual use.

Based on the above method, an embodiment of the present invention further provides a detection terminal, including: a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor can be used to execute the program when the processor executes the program The above-mentioned video moving object detection method based on deep fusion network.

In the video moving object detection system and method based on the deep fusion network, and the detection terminal provided by the above embodiments of the present invention, after the video sequence is input into the system, the video feature extraction operation and the basic result detection operation are simultaneously performed, and then the deep fusion module is used. Video features perform optimal fusion of multiple basic detection results. The above-mentioned embodiments of the present invention use a deep convolutional network to construct a feature extraction module and a deep fusion module, use a large amount of data for training to obtain optimal model parameters, and can automatically detect moving objects in practical applications; the experimental results show that the system can achieve high accuracy degree test results.

The specific parameters in the above embodiments of the present invention are only examples to illustrate the implementation of the technical solutions of the present invention. The present invention may also adopt other specific parameters in other embodiments, which have no essential impact on the implementation of the present invention.

It should be noted that the steps in the method provided by the present invention can be implemented by using the corresponding modules, devices, units, etc. in the system, and those skilled in the art can refer to the technical solutions of the system to implement the method. The step flow, that is, the embodiments in the system can be understood as a preferred example for implementing the method, and details are not described here.

Those skilled in the art know that, in addition to implementing the system provided by the present invention and its various modules, devices, and units in the form of purely computer-readable program codes, the system provided by the present invention and its various devices can be implemented by logically programming method steps. The same function is implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by the present invention can be regarded as a kind of hardware components, and the devices for realizing various functions included in the system can also be regarded as structures in the hardware components; The means for implementing various functions can be regarded as either a software module implementing a method or a structure within a hardware component.

The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and those skilled in the art can make various variations or modifications within the scope of the claims, which do not affect the essential content of the present invention.

Claims (9)

1. A video moving object detection system based on a depth fusion network is characterized by comprising:

the video feature extraction module receives video sequence input, performs feature extraction on video content to obtain feature expression about scene information in a video, namely video scene feature expression, and sends the feature expression to the depth fusion module;

the basic result detection module receives video sequence input, detects the moving object by using the basic detector to obtain a corresponding basic detection result and sends the basic detection result to the depth fusion module;

the depth fusion module is used for receiving the video scene feature expression and the basic detection result, performing optimal fusion by using a depth neural network and outputting a final detection result;

the depth fusion module receives video scene feature expression as input, obtains an optimal fusion weight map through four convolutional layers and a Soft-Max layer, and performs pixel-by-pixel linear weighting on a basic detection result according to the optimal fusion weight map.

2. The system of claim 1, wherein the video feature extraction module uses a pre-training-based VGG-16 network as a feature extractor to extract features of each frame of video, and then stacks the features of each frame of video together to form a set of descriptors, i.e., video scene feature expressions, for describing video scenes.

3. The video moving object detection system based on the deep convergence network as claimed in claim 1, wherein the number of the basic detectors is plural, and each of the basic detectors respectively detects the moving object by using a conventional motion detection method to obtain plural corresponding basic detection results.

4. The system of claim 3, wherein the number of the basic detectors is four, and accordingly, each basic detector respectively adopts the following conventional motion detection methods:

-a pixel-based adaptive semantic relevance segmentation method;

-a context segmentation method based on edge detection;

-a shared model based background segmentation method;

-background segmentation method based on sample point weighting.

5. A video moving object detection method based on a depth fusion network is characterized by comprising the following steps:

s1: reading a current frame and a plurality of frames before the current frame in the video in sequence as video sequence input;

s2: analyzing each frame of video in an input video sequence by using a feature extractor to obtain a plurality of groups of video frame features, and stacking the plurality of groups of video features in the channel direction to form a descriptor for describing video scene features, namely a video scene descriptor; carrying out moving object analysis on an input video sequence by utilizing a traditional motion detection method to obtain a basic detection result;

s3: inputting the video scene descriptor and the basic detection result obtained in the S2 into a depth fusion network; the deep fusion network analyzes the video scene descriptors to obtain an optimal fusion weight map, and linear weighted fusion is performed on basic detection results by using the optimal fusion weight map.

6. The method as claimed in claim 5, wherein the depth fusion network is based on a depth convolution network, the input video scene descriptor is processed through four convolution layers and a Soft-Max layer to obtain an optimal fusion weight map, and the basic detection result is subjected to pixel-by-pixel linear weighting according to the optimal fusion weight map.

7. The method of claim 5 or 6, further comprising offline training of the feature extractor and the deep fusion network, wherein:

randomly sampling video segments in a training video to serve as a prediction motion mask, and using the video segments and a marking mask of a real moving object, namely a real motion mask, as training pairs, wherein a plurality of training pairs form a training set; randomly cutting the training video in the training pair to obtain a training sample, and then randomly turning the training sample left and right and/or up and down to expand the training set;

and using a training pair as input, performing joint optimization on parameters of the feature extractor and the deep fusion network by using a random gradient descent algorithm, and performing multiple rounds of learning on all training pairs in a training set until loss convergence.

8. The method according to claim 7, wherein the loss function used in the stochastic gradient descent algorithm is the mean variance of the predicted motion mask and the true motion mask; and/or the presence of a gas in the gas,

the parameter updating rate of the deep fusion network is set to be 100-10000 times of the parameter updating rate of the feature extractor.

9. A detection terminal, comprising: memory, processor and computer program stored on the memory and executable on the processor, characterized in that the program, when executed by the processor, is operable to perform the method of any of the preceding claims 5 to 8.

Images