CN110490073A - Object detection method, device, equipment and storage medium - Google Patents

Abstract

The present application discloses a target detection method, device, equipment, and storage medium. Video data is obtained, and a first image sequence of the video data is preprocessed to obtain a second image sequence in which background images are removed. The image sequence is input into the trained detection model for target detection, and the target detection result is obtained. On the one hand, only the foreground object is reserved for the image with the background removed, and there is no interference from other background images. The detection model pays more attention to the foreground object during learning and reasoning, which can improve the accuracy of object detection; on the other hand, due to the removal of the input image Background pixels, the detection model only sees foreground pixels, and will not be affected by video or picture sequence scenes at all, thus improving the scene migration performance of target detection.

Description

Target detection method, device, equipment and storage medium

technical field

The present application relates to the technical field of computer vision, and in particular to an object detection method, device, equipment and storage medium.

Background technique

As we all know, vision is the most direct and effective means of obtaining information. However, most monitoring systems are in the working mode of "only recording without judgment". judgment. However, there is a great waste of human resources in this way. With the emergence of computer vision intelligent video processing systems, image processing technology and machine learning methods are used to achieve video analysis such as target detection and tracking.

The task of object detection is to find all the objects of interest in the image, determine their position and size. Object detection has always been the most challenging problem in the field of machine vision due to the different appearance, shape, and posture of various objects, coupled with the interference of factors such as illumination and occlusion during imaging.

Existing target detection is prone to false detection for scenes with complex backgrounds in static pictures, so the accuracy of target detection needs to be improved. In addition, the existing target detection also has certain limitations in realizing the generalization performance of monitoring complex scenes. In order to improve the scene transfer performance of the target detection algorithm, a large number of data sets need to be trained, which is highly dependent on data.

Contents of the invention

The purpose of the present application is to provide an object detection method, device, equipment and storage medium, so as to improve the accuracy of object detection and scene transition performance.

In the first aspect, the embodiment of the present application provides a target detection method, including:

Get video data;

Preprocessing the first image sequence of the video data to obtain a second image sequence from which the background image is removed;

The second image sequence is input into the trained detection model to perform target detection, and a target detection result is obtained.

In a possible implementation manner, the above-mentioned method provided in the embodiment of the present application includes:

performing moving target detection on the first image sequence of the video data by using a background subtraction method;

The pixels of the area where the moving object is located are retained, and the pixels of the area where the moving object is located are segmented by using a morphological method, and divided into independent units of the moving object, so as to obtain a second image sequence in which the background image is removed.

In a possible implementation manner, in the above method provided by the embodiment of the present application, the detection model adopts an SSD framework, and the SSD framework includes: a feature extraction network and a target detection network.

In a possible implementation, in the above method provided by the embodiment of the present application, the method further includes training the SSD framework, which includes:

Preprocessing the image sequence of the sample video data to obtain a sample image sequence from which the background image is removed;

Carrying out manual target labeling on the sample image sequence to obtain a training data set;

The SSD framework is trained based on the training data set: first, initialize the parameters and hyperparameters to be trained in the network, input the training data into the initialized network for forward propagation of the network, and obtain the actual output results, and combine the loss function with the inverse Adjust the network parameters to the propagation BP algorithm, perform iterative training, and end the training until the loss value of the loss function is less than the set threshold or reaches the maximum number of iterations, and a trained SSD framework is obtained.

In a possible implementation manner, in the above method provided in the embodiment of the present application, the loss function is a weighted sum of a position error and a confidence error.

In a possible implementation, in the above method provided in the embodiment of the present application, the calculation formula of the confidence error is as follows:

in, Indicates that the predicted box i matches the ground truth box j with respect to the category.

In the second aspect, the embodiment of the present application provides a target detection device, including:

An acquisition module, used to acquire video data;

A preprocessing module, configured to preprocess the first image sequence of the video data to obtain a second image sequence from which the background image is removed;

A target detection module, configured to input the second image sequence into a trained detection model for target detection and obtain a target detection result.

In a possible implementation manner, in the above device provided in the embodiment of the present application, the preprocessing module is specifically used for:

performing moving target detection on the first image sequence of the video data by using a background subtraction method;

Retaining the pixels in the region where the moving target is located, and segmenting the pixels in the region where the moving target is located by using a morphological method, and dividing them into independent moving target units, so as to obtain a second image sequence in which the background image is removed;

Obtain a second sequence of images with the background image removed.

In a possible implementation manner, in the above device provided by the embodiment of the present application, the detection model adopts an SSD framework, and the SSD framework includes: a feature extraction network and a target detection network.

In a possible implementation, in the above device provided in the embodiment of the present application, a training module is also included for:

Preprocessing the image sequence of the sample video data to obtain a sample image sequence from which the background image is removed;

Carrying out manual target labeling on the sample image sequence to obtain a training data set;

The SSD framework is trained based on the training data set: first, initialize the parameters and hyperparameters to be trained in the network, input the training data into the initialized network for forward propagation of the network, and obtain the actual output results, and combine the loss function with the inverse Adjust the network parameters to the propagation BP algorithm, perform iterative training, and end the training until the loss value of the loss function is less than the set threshold or reaches the maximum number of iterations, and a trained SSD framework is obtained.

In a possible implementation manner, in the above-mentioned apparatus provided by the embodiment of the present application, the loss function is a weighted sum of a position error and a confidence error.

In a possible implementation, in the above device provided in the embodiment of the present application, the calculation formula of the confidence error is as follows:

in, Indicates that the predicted box i matches the ground truth box j with respect to the category.

In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor;

The memory is used to store computer programs;

Wherein, the processor executes the computer program in the memory, so as to implement the above first aspect and the method described in each implementation manner of the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned first aspect and the first aspect are implemented. The methods described in the various embodiments of the aspect.

Compared with the prior art, the object detection method, device, equipment and storage medium provided by the present application acquire video data, perform preprocessing on the first image sequence of the video data, and obtain the second image sequence with the background image removed, The second image sequence is input into the trained detection model to perform target detection, and a target detection result is obtained. On the one hand, only the foreground object is reserved for the image with the background removed, without the interference of other background images, the detection model pays more attention to the foreground object during learning and reasoning, which can improve the accuracy of object detection; on the other hand, due to the removal of the input image Background pixels, the detection model only sees foreground pixels, and will not be affected by video or picture sequence scenes at all, thus improving the scene migration performance of target detection.

Description of drawings

FIG. 1 is a schematic flow chart of a target detection method provided in Embodiment 1 of the present application;

FIG. 2 is a flow chart of the background removal method provided by the embodiment of the present application;

Fig. 3 is the overall structure of the target detection system based on the SSD framework provided by the embodiment of the present application;

FIG. 4 is a schematic structural diagram of a target detection device provided in Embodiment 2 of the present application;

FIG. 5 is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present application.

Detailed ways

The specific embodiments of the present application will be described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present application is not limited by the specific embodiments.

Unless expressly stated otherwise, throughout the specification and claims, the term "comprise" or variations thereof such as "includes" or "includes" and the like will be understood to include the stated elements or constituents, and not Other elements or other components are not excluded.

The problem to be solved in target detection is to find certain categories of objects in images or videos through the target frame, and give the probability that the object belongs to a certain category, that is, a task combining position coordinate regression and category prediction.

SSD: Full name in Chinese: Single Shot MultiBox Detector, full name in English: Single Shot MultiBoxDetector, the SSD framework includes a feature extraction network and a target detection network, where the feature extraction network is used to extract features from images, and the target detection network is used to extract The features of the image are used to perform position regression and target category prediction, thereby identifying the object category in the image.

Figure 1 is a schematic flow chart of the target detection method provided by Embodiment 1 of the present application. In practical applications, the subject of execution in this embodiment may be a target detection device, and the target detection device may be implemented by a virtual device, such as software code, or by It can be implemented by a physical device with relevant execution codes written therein, such as a USB flash drive, or it can also be realized by a physical device integrated with relevant execution codes, such as a chip, a computer, a robot, and the like.

As shown in Figure 1, the method includes the following steps S101-S103:

S101. Acquire video data.

S102. Perform preprocessing on the first image sequence of the video data to obtain a second image sequence with background images removed.

S103. Input the second image sequence into the trained detection model to perform target detection, and obtain a target detection result.

In this embodiment, the video data can be collected in real time by the camera or stored in advance. It can be understood that the video data is composed of multiple frames of images, and the video data includes objects to be identified, such as people and vehicles. After the video image to be detected is obtained, the background is removed from the image sequence in the video image, and only the foreground object is retained, that is, only the pixels in the area where the object is located are retained, and the background pixel area is set to zero to obtain an image sequence with the background image removed.

Specifically, step S102 can be specifically implemented as: using the background subtraction method to detect the moving object in the first image sequence of the video data; retaining the pixels in the area where the moving object is located, and segmenting the pixels in the area where the moving object is located using a morphological method The processing is divided into independent moving target units to obtain the second image sequence with the background image removed. Figure 2 is a flowchart of the background removal method. The algorithm mainly calculates the pixel stability. During the operation of the algorithm, it will record the gray value of the pixel with the longest stable time from the beginning of operation to the current moment, and use the stability of several adjacent frames and historical pixels as the basis for judgment. , When a new frame comes, the stability of the pixel is judged through a series of threshold comparison operations, so as to judge whether it is a background point, thereby removing the background pixel and retaining the foreground pixel.

Then input the image sequence with the background image removed into the trained detection model for target detection, and obtain the target detection result. The detection model is also trained with background-removed samples during training.

The present application will be introduced in a specific implementation manner below.

In this embodiment, the detection model adopts an SSD framework, and the SSD framework includes: a feature extraction network and a target detection network. The way to train this SSD framework is as follows:

S201. Perform preprocessing on an image sequence of sample video data to obtain a sample image sequence with background images removed.

S202. Perform manual target labeling on the sample image sequence to obtain a training data set.

S203. Train the SSD framework based on the training data set: first initialize the parameters and hyperparameters to be trained in the network, input the training data into the initialized network for forward propagation of the network, obtain the actual output result, and pass the loss function Combined with the backpropagation BP algorithm to adjust the network parameters, iterative training is carried out, and the training ends when the loss value of the loss function is less than the set threshold or reaches the maximum number of iterations, and the trained SSD framework is obtained.

Specifically, first make a training data set: use traditional image processing algorithms to detect moving objects through the background subtraction method, retain the pixel mask of the area where the moving object is located, and use the morphological method to retain the pixels of the area where the moving object is located as much as possible, so that Obtain the image sequence with the background removed, use the labeling tool to manually label the data set, and obtain the training data set.

Detection model design: The design of the detection model is based on the existing SSD target detection network structure, and the loss function is mainly modified to remove the background loss function item in the category loss. In this embodiment, the loss function is the weighted sum of the position error and the confidence error. For the confidence error, Softmax Loss is used, and for the position error, SmoothL1loss is used.

The loss function is as follows:

Among them, the first item L _conf is the confidence error, the second item L _loc is the position error, N is the number of matching default borders, α is the balance factor (weight coefficient), and the value is 1 during cross-validation. c is the category confidence prediction value. l is the position prediction value of the corresponding bounding box of the prior box, and g is the position parameter of the real target.

in,

Among them, due to the singleness of the background sample, the feature learning of the background can be ignored, so that the network pays more attention to the foreground sample learning, Indicates that the predicted frame i matches the real frame j with respect to the category, then the higher the probability prediction of p, the smaller the loss, Obtained by Softmax, if the prediction box has no target, the higher the probability of being the background, the smaller the loss.

in,

where, using the positional regression function, Indicates whether the i-th predicted frame matches the j-th real frame with respect to category k, and represent the predicted frame and the real frame, respectively, Indicates the midpoint of the i-th ground-truth box, represents the midpoint of the i-th default box, Indicates the width of the i-th default box.

Detection model training: Use the labeled background image dataset as the training dataset, and use the SSD network framework to train the detection model. Figure 3 shows the overall structure of the target detection system based on the SSD framework. Specifically, first adjust the size of the input image to the input required by the network (for example, 300x300), and use the background image obtained by preprocessing as the input data of the training model. As shown in B in Figure 3, the multi-layer image features are extracted through the forward propagation of the main network, the image features of different layers are fused, and the IoU (Intersection over Union) is compared with the real value data to obtain the error value, and the model objective function is modified , calculate the loss value, so that the network learns more foreground information and ignores the background; use error backpropagation to adjust network parameters, use stochastic gradient descent method in the process of backpropagation error, and set network learning rate lr=0.001, gradient momentum momentum =0.9, complete one iteration. The training ends when the loss value of the loss function is less than the set threshold or reaches the maximum number of iterations, and the trained SSD framework is obtained.

As shown in Figure 3, a data preprocessing layer is added based on the SSD network structure. The data layer acquires video data, uses a background modeling algorithm to remove the background, and retains the foreground image and sends it to the SSD framework for detection. Feature extraction is realized as shown in Figure 3 C, and then feature maps of different scales are extracted through some convolution and pooling operations, and candidate target frames are proposed for feature maps of different scales. For example, if the feature map is 8*8 in size, the candidate target The number of boxes is 8*8*9, and each feature point position generates nine types of candidate targets, namely, 3 scales and 3 areas. When the SSD framework infers the image, it generates a series of fixed-size candidate boxes, and the possibility that each candidate box contains an object instance. A forward processing generates a large number of target frames, and it is necessary to use Non-maximum suppression (NMS) to filter out most of the target frames. The method used is when the confidence threshold of the target frame is less than the threshold ct (such as 0.01) and When the IoU is less than lt (such as 0.45), it is discarded and only the first N prediction results are kept. Match the features obtained by fusion with the true features to constrain the loss function, so that the loss function pays more attention to the foreground target features, so as to realize the target detection.

The object detection method provided in this embodiment preprocesses the first image sequence of the video data to obtain a second image sequence with the background image removed, and inputs the second image sequence into a trained detection model for object detection , to obtain the target detection result. On the one hand, only the foreground object is reserved for the image with the background removed, and there is no interference from other background images. The detection model pays more attention to the foreground object during learning and reasoning, which can improve the accuracy of object detection; on the other hand, due to the removal of background pixels, The detection model sees only foreground pixels, and is completely unaffected by video or picture sequence scenes, thereby improving the scene migration performance of target detection.

The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

FIG. 4 is a schematic structural diagram of a target detection device provided in Embodiment 2 of the present application. As shown in FIG. 4, the device may include:

Obtaining module 410, for obtaining video data;

A preprocessing module 420, configured to preprocess the first image sequence of the video data to obtain a second image sequence from which the background image is removed;

The object detection module 430 is configured to input the second image sequence into the trained detection model for object detection, and obtain an object detection result.

The target detection device provided in this embodiment preprocesses the first image sequence of the video data to obtain a second image sequence with the background image removed, and inputs the second image sequence into a trained detection model for target detection , to obtain the target detection result. On the one hand, only the foreground object is reserved for the image with the background removed, and there is no interference from other background images. The detection model pays more attention to the foreground object during learning and reasoning, which can improve the accuracy of object detection; on the other hand, due to the removal of background pixels, The detection model sees only foreground pixels, and is completely unaffected by video or picture sequence scenes, thereby improving the scene migration performance of target detection.

In a possible implementation, in the above device provided in the embodiment of the present application, the preprocessing module 420 is specifically used to:

performing moving target detection on the first image sequence of the video data by using a background subtraction method;

The pixels of the area where the moving object is located are retained, and the pixels of the area where the moving object is located are segmented by using a morphological method, and divided into independent units of the moving object, so as to obtain a second image sequence in which the background image is removed.

In a possible implementation manner, in the above device provided by the embodiment of the present application, the detection model adopts an SSD framework, and the SSD framework includes: a feature extraction network and a target detection network.

In a possible implementation, in the above device provided in the embodiment of the present application, a training module is also included for:

Preprocessing the image sequence of the sample video data to obtain a sample image sequence from which the background image is removed;

Carrying out manual target labeling on the sample image sequence to obtain a training data set;

The SSD framework is trained based on the training data set: first, initialize the parameters and hyperparameters to be trained in the network, input the training data into the initialized network for forward propagation of the network, and obtain the actual output results, and combine the loss function with the inverse Adjust the network parameters to the propagation BP algorithm, perform iterative training, and end the training until the loss value of the loss function is less than the set threshold or reaches the maximum number of iterations, and a trained SSD framework is obtained.

In a possible implementation manner, in the above-mentioned apparatus provided by the embodiment of the present application, the loss function is a weighted sum of a position error and a confidence error.

In a possible implementation, in the above device provided in the embodiment of the present application, the calculation formula of the confidence error is as follows:

in, Indicates that the predicted box i matches the ground truth box j with respect to the category.

FIG. 5 is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present application. As shown in FIG. 5 , the device includes: a memory 501 and a processor 502;

Memory 501, for storing computer programs;

Wherein, the processor 502 executes the computer program in the memory 501, so as to implement the methods provided by the above method embodiments.

In the embodiment, an electronic device is used as an example of the target detection device provided in this application. The processor may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include random access memory (RAM) and/or cache memory (cache), etc., for example. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the above methods and/or other desired functions in the various embodiments of the present application. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

Embodiment 4 of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the computer program is used to implement the methods provided in the foregoing method embodiments.

In practical applications, the computer programs in this embodiment can be written in any combination of one or more programming languages to execute the program codes for performing the operations of the embodiments of the present application. The programming languages include object-oriented programming languages, such as Java, C++, etc., also includes conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.

In practical applications, any combination of one or more readable media may be used as the computer-readable storage medium. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

The foregoing descriptions of specific exemplary embodiments of the present application have been presented for purposes of illustration and description. These descriptions are not intended to limit the application to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. The exemplary embodiments were chosen and described in order to explain certain principles of the application and their practical application, thereby enabling those skilled in the art to implement and utilize various exemplary embodiments of the application, as well as various Choose and change. It is intended that the scope of the application be defined by the claims and their equivalents.

Claims (10)

1. A target detection method, characterized in that, comprising: Get video data; Preprocessing the first image sequence of the video data to obtain a second image sequence from which the background image is removed; The second image sequence is input into the trained detection model to perform target detection, and a target detection result is obtained. 2. The method according to claim 1, wherein the preprocessing the first image sequence of the video data to obtain the second image sequence for removing the background image comprises: performing moving target detection on the first image sequence of the video data by using a background subtraction method; The pixels of the area where the moving object is located are retained, and the pixels of the area where the moving object is located are segmented by using a morphological method, and divided into independent units of the moving object, so as to obtain a second image sequence in which the background image is removed. 3. The method according to claim 1, wherein the detection model adopts an SSD framework, which comprises a feature extraction network and a target detection network. 4. method according to claim 3, is characterized in that, described method also comprises training SSD framework, and it comprises: Preprocessing the image sequence of the sample video data to obtain a sample image sequence from which the background image is removed; Carrying out manual target labeling on the sample image sequence to obtain a training data set; The SSD framework is trained based on the training data set: first, initialize the parameters and hyperparameters to be trained in the network, input the training data into the initialized network for forward propagation of the network, and obtain the actual output results, and combine the loss function with the inverse Adjust the network parameters to the propagation BP algorithm, perform iterative training, and end the training until the loss value of the loss function is less than the set threshold or reaches the maximum number of iterations, and a trained SSD framework is obtained. 5. The method according to claim 4, wherein the loss function is a weighted sum of position error and confidence error. 6. The method according to claim 5, wherein the calculation formula of the confidence error is as follows: in, Indicates that the predicted box i matches the ground truth box j with respect to the category. 7. A target detection device, characterized in that, comprising: An acquisition module, used to acquire video data; A preprocessing module, configured to preprocess the first image sequence of the video data to obtain a second image sequence from which the background image is removed; A target detection module, configured to input the second image sequence into a trained detection model for target detection and obtain a target detection result. 8. The device according to claim 7, wherein the preprocessing module is specifically used for: performing moving target detection on the first image sequence of the video data by using a background subtraction method; Keep the pixel mask of the area where the moving target is located, and use the morphological method to extract the pixels of the area where the moving target is located; Obtain a second sequence of images with the background image removed. 9. An electronic device, comprising: a memory and a processor; The memory is used to store computer programs; Wherein, the processor executes the computer program in the memory, so as to realize the method according to any one of claims 1-6. 10. A computer-readable storage medium, characterized in that, a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it is used to implement any one of claims 1-6. described method.