CN116434160A - Expressway casting object detection method and device based on background model and tracking - Google Patents

Abstract

The invention discloses a method and a device for detecting a highway casting object based on a background model and tracking, wherein the method comprises the following steps: acquiring a video stream to be detected; background modeling and moving object detection are carried out, so that a moving object in a video frame image is detected, and according to the detecting result of the moving object, the GMM generates a binary image with the same size as the frame image; processing the foreground detection binary image by using mathematical morphology; tracking each moving target based on a tracking algorithm matched with the IOU and the Hungary, and associating the detection frames of the same moving target in the video in different frames based on the IOU values of the target detection frame of the current frame and the target detection frame of the history frame, so that the tracking of the target is completed; fusing a perception loss function, binarizing the perception images of the whole image and the images in the candidate frames by using an otsu threshold value, performing AND operation to obtain a candidate target of the throwing object, and tracking the candidate target; road surface interference is eliminated by using the two-class network, and accuracy of detecting the throwing objects is improved.

Description

Method and device for detection of sprinkled objects on expressway based on background model and tracking

technical field

The invention belongs to the technical field of moving target detection, and in particular relates to a method for detecting sprinkled objects on expressways based on background models and tracking.

Background technique

With the development of economy, the transportation of passengers and goods on expressways has developed rapidly, and the safety of driving on expressways has also received more attention. Among them, spilled objects are one of the important reasons for expressway accidents. Generally, the detection of spilled objects mostly relies on manual naked eye observation, which is inefficient and unsustainable. In order to improve the detection efficiency, there is also a 7*24-hour high-efficiency detection through the highway camera video detection in the prior art to solve this problem. However, due to the wide variety of spills, they do not have uniform characteristics in appearance. Therefore, most of the existing detection algorithms for spills choose to detect moving objects in the video first, and then detect them from the moving objects according to the motion characteristics of the throwing objects that move first and then remain still. Spill out. Although these spill detection algorithms have a certain detection effect on spills, they are limited by the detection effect of moving targets and the tracking effect of targets in the algorithm, and there is still room for improvement in the detection accuracy of the algorithm. At the same time, these algorithms do not have strong anti-interference ability to people, vehicles and interference items outside the road in the highway scene, and are easily affected and cause false alarms.

Contents of the invention

The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, to provide a method and device for detecting sprinkled objects on highways based on background models and tracking, and to solve the problem of low recognition accuracy when the characteristics of training samples are not obvious and easily confused problem.

In order to achieve the above object, the present invention adopts the following technical solutions:

In a first aspect, the present invention provides a method for detecting sprinkled objects on expressways based on background models and tracking, comprising the following steps:

Obtain a video stream to be detected, and obtain a current video frame image after decoding the video stream to be detected, and the current video frame image includes a background image and a moving target;

Background modeling and moving target detection, using the pre-established mixed Gaussian model GMM to use K Gaussian distributions for each pixel in the background image for background modeling, divide the background image, and combine the image in the current video frame with the background image pixel Matching is performed to detect the moving target in the video frame image, and according to the result of the moving target detection, the GMM generates a binary image with the same size as the frame image;

Use mathematical morphology to process the foreground detection binary image to remove the noise in the binary image;

Track each moving target based on the tracking algorithm based on IOU and Hungary matching, and associate the detection frames of the same moving target in different frames in the video based on the IOU value of the target detection frame of the current frame and the target detection frame of the historical frame, so as to complete tracking of targets;

Integrate the perceptual loss function, binarize the entire image and the perceptual image of the image in the candidate frame using the otsu threshold, and perform the sum operation to obtain the candidate target of the sprinkler, and track the candidate target;

Use the binary classification network to eliminate road interference and improve the accuracy of the detection of spilled objects.

As a preferred technical solution, the background modeling and moving target detection are specifically:

Parameter initialization: For the first frame image of the input video stream to be detected, take the value of each pixel in the first frame image as the mean value μ _1,1 of the first Gaussian distribution of the corresponding pixel in the background image, and at the same time Set up a larger weight ω _1,1 and variance ∑ _1,1 for the Gaussian distribution. For the remaining K-1 Gaussian distributions of each pixel, set their mean value to 0 and initialize the variance to a larger value, the weights are all set to (1-ω _1,1 )/(K-1);

Background division: The formula for calculating the respective priorities p _i,t of K Gaussian distributions of a certain pixel in the tth frame of the video stream to be detected is p _i,t =ω _i,t /|σ _i,t | , where the larger the value of pi _,t, the larger the weight of the Gaussian distribution, which is more suitable for describing the background content; on the other hand, the smaller the variance, the larger the value of pi _,t , which means that in the historical frame The value of the pixel does not change much, so the probability of the tth frame belonging to the background is relatively high;

Moving object detection: After the background is divided, GMM compares each pixel of the input frame image with the first K ₀ Gaussian distributions to determine whether the pixel belongs to a moving object. If a pixel in the frame image and the corresponding pixel in the background image If it matches one of the first K ₀ Gaussian distributions, the pixel is considered to belong to the background pixel; otherwise, the pixel is judged as a moving object pixel; according to the result of moving object detection, GMM generates a binary image with the same size as the frame image, which belongs to The value of the pixel of the moving target is 255, which is displayed as a white area, and the value of the pixel belonging to the background area is 0, which is displayed as a black area, and a binary image is obtained;

GMM parameter update: In order to enable the background model to keep up with the background changes in the video, the parameters of the GMM are updated. If a pixel of the frame image successfully matches the i-th Gaussian distribution of the corresponding pixel in the background image, then the Gaussian distribution If the pixel does not match the corresponding K Gaussian distributions, it is considered that it belongs to a new moving target in the frame image, and a new Gaussian distribution is established for the corresponding pixel in the background image.

其中D为预先设立的阈值。As a preferred technical solution, in the step of background division, the top _K Gaussian distributions after sorting are taken, and they are considered to describe the background content in the video, so that:

Where D is a preset threshold.

As a preferred technical solution, the use of mathematical morphology to process the foreground detection binary image to remove noise in the binary image is specifically:

In view of the possible existence of small lighting changes and the noise generated by the shooting equipment during video recording, the mathematical description is as follows:

In the formula, A' is the result of mathematical morphology processing, A is the binary image to be processed, B1 is the probe used in the opening operation and closing operation, which is a 3×3 square structure, and B2 is the one used in the expansion operation probe;

Aiming at the interference to the detection of spilled objects such as moving vehicles and pedestrians getting off the vehicle under special circumstances, YOLOx detection is used to extract a frame image every set frame and input it to the pre-established YOLO network model for human detection. Vehicle detection, to obtain the specific position of pedestrians and vehicles in the image, and generate a binary image for human-vehicle detection through the position information; take the center point of each moving target detection frame of the current frame image, if the center point is located in the second level of human-vehicle detection In the black area in the value map, the detection frame is marked as a human-vehicle detection frame in the algorithm program, and in the subsequent steps of detecting spilled objects, these marked detection frames will not be judged as spilled objects;

For the interference items that may exist in the area outside the road of the expressway, use semantic segmentation to get rid of the road area, and modify the output part of the semantic segmentation network model, so that the pixel value of the area classified as road in the semantic segmentation result image becomes [255, 255, 255 ], and the pixel values in the remaining areas become [0,0,0], and the image is converted into a single-channel grayscale image, thus obtaining a road segmentation binary image. In the road segmentation binary image, the white area is the road area, and the black area is the non-road area.

As a preferred technical solution, the tracking algorithm based on IOU and Hungary matching tracks each moving target, based on the IOU value of the target detection frame of the current frame and the target detection frame of the historical frame, the same moving target in the video is displayed in different frames The detection frame is associated to complete the tracking of the target, specifically:

It is assumed that the moving object detection frame of the current frame image of the video is matched with the moving object detection frame of the historical frame, and the set of the moving object detection frame in the current frame image is Sc, wherein the number of detection frames is m, and the moving object detection frame of the historical frame image The set is Sh, the number of detection frames is n, and the cost matrix C is constructed. The element c(i,j) in the matrix is the opposite number of the IOU value of the i-th detection frame in Sc and the j-th detection frame in Sh, where 1 ≤i≤m, 1≤j≤n; if n≠m, use 0 to fill the vacant columns or rows in the cost matrix; match the detection frames of Sc and Sh, and the goal is to obtain the minimum matching cost, that is, the total The sum of the opposite numbers of the IOU value is the smallest, which means the sum of the IOU value is the largest; then use the Hungarian matching algorithm to find the optimal solution under the cost matrix; in the case of a high frame rate, the moving target is in the adjacent frame image The overlapping degree of the detection frames in is relatively high, so the optimal solution that maximizes the sum of IOU values can achieve correct association of each moving target, thereby completing the tracking of the target. As a preferred technical solution, the fusion perceptual loss function uses the otsu threshold to binarize the entire image and the perceptual image of the image in the candidate frame, and performs a summation operation to obtain the candidate target of the spilled objects, and then tracks the candidate target, In order to eliminate the interference of the road area, specifically:

For the current video frame, track the edge contours of two consecutive still frames for matching to obtain a long background image and a short background image; then use otsu to binarize the perceptual map within 4 times the area of the candidate frame, and then use otsu to binarize the perceptual of the entire image map, and perform morphological processing on the perceptual map, perform a sum operation on the perceptual map in the candidate frame and the perceptual map of the entire image, extract a rectangular frame from the summed map graphic, and judge Lou (total frame, map frame) The size of the set threshold, if it is less than the set threshold, it will be discarded, and if it is greater than the set threshold, the new scatter will be added to the scatter box array.

As a preferred technical solution, the two classification network adopts ResNet-34, specifically:

Input the collected samples into the ResNet-34 model for training. The two classification refers to road and non-road. The non-road samples in the selected training samples include: boxes, tires, road cones, bottles, bags and paper; road samples include : Roads with lane lines, roads with water marks in rainy days, reflective roads, waterlogged roads and shadow roads.

In the second aspect, the present invention provides a highway spill detection system based on background model and tracking, which is applied to the highway spill detection method based on background model and tracking, including a data acquisition module, background modeling and A moving target detection module, a noise processing module, a target tracking module, a sprinkler detection module, and a classification module;

The data acquisition module is used to obtain a video stream to be detected, and obtain a current video frame image after decoding the video stream to be detected, and the current video frame image includes a background image and a moving target;

The background modeling and moving object detection module is used for background modeling and moving object detection, using the pre-established mixed Gaussian model GMM to perform background modeling on each pixel in the background image using K Gaussian distributions respectively, and dividing the background Image, the image in the current video frame is matched with the background image pixels, thereby detecting the moving target in the video frame image, according to the result of the moving target detection, GMM generates a binary image with the same size as the frame image;

The noise processing module is used to process the foreground detection binary image by using mathematical morphology to remove noise in the binary image;

The target tracking module is used to track each moving target based on the tracking algorithm based on IOU and Hungary matching, based on the IOU value of the target detection frame of the current frame and the target detection frame of the historical frame, the same moving target in the video is displayed in different frames The detection frame is associated to complete the tracking of the target;

The spill detection module is used to fuse the perceptual loss function, binarize the entire image and the perceptual image of the image in the candidate frame using the otsu threshold, and perform a sum operation to obtain the candidate target of the spill, and track the candidate target ;

The classification module is used to use a binary classification network to eliminate road interference and improve the accuracy of detection of spilled objects.

In a third aspect, the present invention provides an electronic device, the electronic device comprising:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores computer program instructions executable by the at least one processor, the computer program instructions are executed by the at least one processor, so that the at least one processor can perform the background model-based and Tracking detection method for freeway spills.

In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, and when the program is executed by a processor, the method for detecting sprinkled objects on a highway based on background model and tracking is implemented.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

Compared with the existing methods, the present invention provides a method for detecting sprinkled objects on highways based on background models and tracking. The sprinkled objects are detected based on the motion characteristics of the first moving and then stationary, and the mixed Gaussian model is used to detect the video first. The moving target in the frame image, and then use the tracking algorithm based on IOU and Hungary to obtain the moving state information of the moving target, and finally detect the throwing objects in the moving target according to the movement characteristics of the throwing objects. In addition, the present invention also uses the YOLO network model and the semantic segmentation network model to remove the interference items outside the human, vehicle and road areas, uses the threshold binarization method to sum the candidate frame and the perceptual map of the original picture, and uses the binary classification network to exclude road surface Interference effectively improves the accuracy of the algorithm for detecting spilled objects. It can effectively detect various types of spills in highway scenes, and has high robustness to environmental noise.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

Fig. 1 is the flow chart of the detection method of expressway spillage based on background model and tracking according to the embodiment of the present invention;

Fig. 2 is the flowchart of background modeling and moving object detection of the embodiment of the present invention;

Fig. 3 is a flow chart of comparing the difference between perceptual maps according to the embodiment of the present invention;

FIG. 4 is a schematic structural diagram of an expressway sprinkler detection system based on a background model and tracking according to an embodiment of the present invention;

FIG. 5 is a structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Apparently, the described embodiments are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of this application.

Reference in this application to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrences of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is understood explicitly and implicitly by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

The invention provides a detection method based on the background model and tracking of highway spills. The background model is used to obtain the moving targets in the video frame images, and then the moving targets are tracked using the tracking algorithm based on IOU and Hungary matching, and their moving states are analyzed. Information, through the movement characteristics of the sprinkled objects and the background difference method selected in this paper for moving target detection.

Please refer to Fig. 1, the present embodiment is based on the background model and the detection method of the freeway spilled object that tracks and includes six parts: Data acquisition, background modeling and moving target detection, noise processing and interference item exclusion, moving target tracking and spilled object detection , comparison of perceptual map differences, and two-category recognition, as follows:

S1. Acquire a video stream to be detected, and decode the video stream to obtain a current video frame image, where the current video frame image includes a background image and a moving target.

Exemplarily, the video stream to be detected may be a video stream extracted from a traffic surveillance video, and it is necessary to detect whether there is a spilled object from the surveillance video stream, and if there is a spilled object, perform target tracking on the spilled object.

S2. Background modeling and moving target detection;

To detect the spilled objects on the expressway, it is first necessary to detect the moving objects in the video, and then select the scattered objects therein through its motion characteristics. In this embodiment, a Gaussian Mixture Model (GMM, Gaussian Mixture Model) is used for background meeting and moving object detection. Please refer to Figure 2, the specific process of background modeling and moving target detection is as follows:

S2.1. GMM uses K Gaussian distributions for background modeling for each pixel in the background image. K generally takes a value between 3 and 5. The larger the value of K, the more comprehensive the description of the pixel by the background model. The stickiness is stronger, each Gaussian distribution describes the pixel value range of its corresponding pixel in the background image, and multiple Gaussian distributions can more accurately represent the complex situation in the background.

S2.2. Parameter initialization: For the first frame image of the input video, take the value of each pixel in the image as the mean value μ _1,1 of the first Gaussian distribution of the corresponding pixel in the background image, and at the same time be the Gaussian distribution Set up a larger weight ω _1,1 (between 0.5-0.9) and variance Σ _1,1 . For the remaining K-1 Gaussian distributions of each pixel, set their mean to 0, initialize their variance to a larger value, and set their weights to (1-ω _1,1 )/(K-1).

(D为设立的阈值，定为0.8)。S2.3. Background division: the calculation formula of the respective priorities p _i,t of K Gaussian distributions of a certain pixel in the tth frame of the surveillance video is p _i,t =ω _i,t /|σ _{i, t} |, where the larger the value of p _i,t, the larger the weight of the Gaussian distribution, which is more suitable for describing the background content of the video; on the other hand, the smaller the variance, the larger the value of p _i,t , which means that in The value of the pixel in the historical frame image does not change much, so it has a higher probability of belonging to the background. The present embodiment takes the _top K Gaussian distributions after sorting, and considers that they describe the background content in the video, so that:

(D is the established threshold, set at 0.8).

S2.4. Moving object detection: After the background is divided, GMM compares each pixel of the input frame image with the first K ₀ Gaussian distributions to determine whether the pixel belongs to a moving object. If a pixel in the frame image is consistent with the background image If one of the first K ₀ Gaussian distributions of the corresponding pixel matches, the pixel is considered to belong to the background pixel. Otherwise, the pixel is determined as a moving target pixel.

Further, according to the result of moving object detection, GMM generates a binary image with the same size as the frame image, the value of the pixels belonging to the moving object is 255, which is displayed as a white area, and the value of pixels belonging to the background area is 0, which is displayed as black area to get a binary image.

S2.5, GMM parameter update: In order to enable the background model to keep up with the background changes in the video, update the parameters of the GMM; if a certain pixel of the frame image successfully matches the ith Gaussian distribution of the corresponding pixel in the background image, then The parameters of the Gaussian distribution are updated. If the pixel does not match the corresponding K Gaussian distributions, it is considered that it belongs to a new moving target in the frame image, and a new Gaussian distribution is established for the corresponding pixel in the background image.

S3. Noise processing and interference elimination, details are as follows:

S3.1. Use mathematical morphology to process the foreground detection binary image and remove the noise in the binary image. This part is mainly aimed at situations such as possible slight lighting changes and noise generated by the shooting equipment during video recording. The mathematical description is as follows :

Among them, A' is the result of mathematical morphology processing, A is the binary image to be processed, B1 is the probe used in the opening operation and closing operation, which is a 3×3 square structure, and B2 is the one used in the expansion operation The probe is a 5×5 square structure.

S3.2. Use YOLOx to detect the interference to the detection of spilled objects such as vehicles in motion and pedestrians who get off the vehicle in a few special cases. A frame image is extracted every 25 frames and input to the YOLO network model for human-vehicle detection. The specific positions of pedestrians and vehicles in the image can be obtained. Through this position information, a binary image for human-vehicle detection is generated. Take the center point of each moving target detection frame of the current frame image, if the center point is located in the black area in the binary image of human-vehicle detection, mark the detection frame as a human-vehicle detection frame in the algorithm program, and detect spilled objects in the subsequent In the step of , these marked detection frames will not be judged as spilled objects.

S3.3. For the interference items that may exist in the off-road area of the expressway, the road area obtained by semantic segmentation is used to exclude it. Modify the output part of the semantic segmentation network model, so that the pixel values of the regions classified as roads in the semantic segmentation result image become [255,255,255], and the pixel values of other regions become [0,0,0], and the image becomes A single-channel grayscale image is used to obtain a road segmentation binary image. In the road segmentation binary image, the white area is the road area, and the black area is the non-road area.

S4, moving target tracking;

Use the tracking algorithm based on IOU and Hungarian matching to track each moving target. Based on the IOU value of the target detection frame of the current frame and the target detection frame of the historical frame, the detection frames of the same moving target in different frames in the video are associated, so that Finish tracking them.

Further, the specific process of moving target tracking is as follows:

Assume that the moving object detection frame of the current frame image of the video is matched with the moving object detection frame of the historical frame, where the historical frame is set as 5 frames before the current frame. Let the set of moving object detection frames in the current frame image be Sc, where the number of detection frames is m, the set of moving object detection frames in historical frame images is Sh, and the number of detection frames is n. Construct the cost matrix C, the element c(i, j) in the matrix is the opposite number of the IOU value of the i-th detection frame in Sc and the j-th detection frame in Sh, where 1≤i≤m, 1≤j≤n . If n≠m, use 0 to fill the vacant columns or rows in the cost matrix. To match the detection frames of Sc and Sh, the goal is to obtain the minimum matching cost, that is, the sum of the opposite numbers of the total IOU value is the minimum, which means that the sum of the IOU values is the largest. Then use the Hungarian matching algorithm to find the optimal solution under the cost matrix. In the case of a high frame rate, the detection frames of the moving target in adjacent frame images have a high degree of overlap, so the optimal solution that maximizes the sum of the IOU values allows each moving target to achieve correct association, thereby completing the detection their tracking.

S5. Spill detection;

Integrating the perceptual loss function, the perceptual map of the entire image and the image in the candidate frame is binarized using the otsu threshold, and the sum operation is performed to obtain the candidate target of the sprinkler, and the candidate target is tracked to eliminate the interference of the road area.

Further, please refer to Figure 3. For the current video frame, track the edge contours of 2 consecutive still frames for matching to obtain a long background image and a short background image; then use otsu to binarize the perceptual map within 4 times the area of the candidate frame, and then otsu Binarize the perceptual map of the entire image, and perform morphological processing on the perceptual map, perform an AND operation on the perceptual map in the candidate frame and the perceptual map of the entire image, extract a rectangular frame from the summed map graphic, and judge Lou (total frame, map frame) and the size of the set threshold. If it is less than the set threshold, it will be discarded. If it is greater than the set threshold, the new throwing object will be added to the throwing object frame array.

S6, road and non-road binary classification model;

Using ResNet-34 structure network training,

Input the collected samples into the ResNet-34 model for training. The two categories refer to road surface and non-road surface. The non-pavement samples in the training samples selected in this paper include: boxes, tires, road cones, bottles, bags, and paper. , stagnant water pavement, shadow pavement.

The sprinkler is detected based on the motion characteristics that first move and then stay still. First, the mixed Gaussian model is used to detect the moving target in the video frame image, and then the tracking algorithm based on IOU and Hungary matching is used to obtain the moving state information of the moving target. Finally, according to The motion feature of the splatter detects the splatter in the moving target. In addition, the algorithm also uses the YOLO network model and the semantic segmentation network model to remove the interference items outside the human, vehicle and road areas, uses the threshold binarization method to sum the candidate frame and the original image perceptual map, and uses the ResNet34 binary classification network to exclude Road interference effectively improves the accuracy of the algorithm for detecting spilled objects.

It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because Certain steps may be performed in other orders or simultaneously in accordance with the present invention.

Based on the same idea as the detection method for freeway spills based on background model and tracking in the above embodiments, the present invention also provides a detection system for freeway spills based on background model and tracking, which can be used to implement the above-mentioned method based on background model A detection method for freeway spills with tracking. For the sake of illustration, in the schematic structural diagram of the embodiment of the highway sprinkler detection system based on the background model and tracking, only the parts related to the embodiment of the present invention are shown, and those skilled in the art can understand that the illustrated structure does not constitute a reference Device definitions may include more or fewer components than shown, or combinations of certain components, or different arrangements of components.

Please refer to FIG. 4 , in another embodiment of the present application, a highway spill detection system 100 based on background model and tracking is provided, the system includes a data acquisition module 101, a background modeling and moving object detection module 102 , a noise processing module 103, a target tracking module 104, a spill detection module 105, and a classification module 106;

The data acquisition module 101 is configured to acquire a video stream to be detected, and obtain a current video frame image after decoding the video stream to be detected, and the current video frame image includes a background image and a moving object;

The background modeling and moving object detection module 102 is used for background modeling and moving object detection, using a pre-established mixed Gaussian model GMM to perform background modeling on each pixel in the background image using K Gaussian distributions, and divides The background image matches the image in the current video frame with the pixels of the background image, thereby detecting the moving target in the video frame image, and according to the result of the moving target detection, GMM generates a binary image with the same size as the frame image;

The noise processing module 103 is used to process the foreground detection binary image by using mathematical morphology to remove noise in the binary image;

The target tracking module 104 is used to track each moving target based on the tracking algorithm based on IOU and Hungary matching, and based on the IOU value of the target detection frame of the current frame and the target detection frame of the historical frame, the same moving target in the video is displayed in different frames. The detection frame in is associated to complete the tracking of the target;

The spill detection module 105 is used to fuse the perceptual loss function, binarize the entire picture and the perceptual map of the image in the candidate frame using the otsu threshold, and perform a summation operation to obtain the candidate target of the spill, and perform a calculation on the candidate target track;

The classification module 106 is configured to use a binary classification network to eliminate road interference and improve the accuracy of detection of spilled objects.

It should be noted that the freeway spill detection system based on the background model and tracking of the present invention corresponds to the freeway spill detection method based on the background model and tracking of the present invention. The technical features and beneficial effects described in the embodiment of the spill detection method are applicable to the embodiment of the detection of expressway spills based on the background model and tracking. For details, please refer to the description in the method embodiment of the present invention, which is not described here. Again, hereby declare.

In addition, in the implementation of the highway sprinkler detection system based on the background model and tracking in the above-mentioned embodiment, the logical division of each program module is only an example. Considering the convenience of software implementation, the above-mentioned function allocation is completed by different program modules, that is, the internal structure of the highway sprinkler detection system based on the background model and tracking is divided into different program modules, so as to complete all or all of the above-described Some functions.

Please refer to FIG. 5 , in one embodiment, there is provided an electronic device for realizing a detection method for highway spills based on a background model and tracking, and the electronic device 200 may include a first processor 201, a first memory 202 and The bus may also include a computer program stored in the first memory 202 and operable on the first processor 201 , such as a highway spill detection program 203 .

Wherein, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example: SD or DX memory, etc.), a magnetic memory, Disk, CD, etc. The first storage 202 may be an internal storage unit of the electronic device 200 in some embodiments, such as a mobile hard disk of the electronic device 200 . The first memory 202 may also be an external storage device of the electronic device 200 in other embodiments, such as a plug-in mobile hard disk equipped on the electronic device 200, a smart memory card (Smart Media Card, SMC), a secure digital ( SecureDigital, SD) card, flash memory card (Flash Card), etc. Further, the first memory 202 may also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various data installed in the electronic device 200, such as the code of the expressway litter detection program 203, but also can be used to temporarily store data that has been output or will be output .

In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same function or different functions packaged, including one or A combination of multiple central processing units (Central Processing unit, CPU), microprocessors, digital processing chips, graphics processors and various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect various components of the entire electronic device, and runs or executes programs stored in the first memory 202 or modules, and call data stored in the first memory 202 to execute various functions of the electronic device 200 and process data.

FIG. 5 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. components, or combinations of certain components, or different arrangements of components.

The highway spill detection program 203 stored in the first memory 202 in the electronic device 200 is a combination of multiple instructions, and when running in the first processor 201, it can realize:

Obtain a video stream to be detected, and obtain a current video frame image after decoding the video stream to be detected, and the current video frame image includes a background image and a moving target;

Background modeling and moving target detection, using the pre-established mixed Gaussian model GMM to use K Gaussian distributions for each pixel in the background image for background modeling, divide the background image, and combine the image in the current video frame with the background image pixel Matching is performed to detect the moving target in the video frame image, and according to the result of the moving target detection, the GMM generates a binary image with the same size as the frame image;

Use mathematical morphology to process the foreground detection binary image to remove the noise in the binary image;

Track each moving target based on the tracking algorithm based on IOU and Hungary matching, and associate the detection frames of the same moving target in different frames in the video based on the IOU value of the target detection frame of the current frame and the target detection frame of the historical frame, so as to complete tracking of targets;

Integrate the perceptual loss function, binarize the entire image and the perceptual image of the image in the candidate frame using the otsu threshold, and perform the sum operation to obtain the candidate target of the sprinkler, and track the candidate target;

Use the binary classification network to eliminate road interference and improve the accuracy of the detection of spilled objects.

Further, if the integrated modules/units of the electronic device 200 are realized in the form of software function units and sold or used as independent products, they may be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM, Read-Only Memory) .

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be realized through computer programs to instruct related hardware, and the programs can be stored in a non-volatile computer-readable storage medium When the program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include non-volatile and/or volatile memory. Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiment is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited by the above-mentioned embodiment, and any other changes, modifications, substitutions, combinations, Simplifications should be equivalent replacement methods, and all are included in the protection scope of the present invention.

Claims (10)

1. The highway casting detection method based on the background model and tracking is characterized by comprising the following steps of:

obtaining a video stream to be detected, and decoding the video stream to be detected to obtain a current video frame image, wherein the current video frame image comprises a background image and a moving target;

background modeling and moving object detection, wherein a pre-established Gaussian Mixture Model (GMM) is adopted to perform background modeling on each pixel in a background image by using K Gaussian distributions respectively, the background image is divided, the image in a current video frame is matched with the background image pixels, so that a moving object in the video frame image is detected, and the GMM generates a binary image with the same size as the frame image according to the moving object detection result;

Processing the foreground detection binary image by using mathematical morphology to remove noise in the binary image;

tracking each moving target based on a tracking algorithm matched with the IOU and the Hungary, and associating the detection frames of the same moving target in the video in different frames based on the IOU values of the target detection frame of the current frame and the target detection frame of the history frame, so that the tracking of the target is completed;

fusing a perception loss function, binarizing the perception images of the whole image and the images in the candidate frames by using an otsu threshold value, performing AND operation to obtain a candidate target of the throwing object, and tracking the candidate target;

road surface interference is eliminated by using the two-class network, and accuracy of detecting the throwing objects is improved.

2. The method for detecting highway casts based on background model and tracking according to claim 1, wherein the background modeling and moving object detection are specifically as follows:

parameter initialization: for a first frame image of an input video stream to be detected, taking the value of each pixel in the first frame image as the mean value mu of the first Gaussian distribution of corresponding pixel points in a background image _1,1 At the same time, a larger weight omega is set for the Gaussian distribution _1,1 Variance sigma _1,1 For the remaining K-1 Gaussian distributions of each pixel, the mean value is set to 0, the variance is initialized to a larger value, and the weights are all set to (1-omega) _1,1 )/(K-1)；

Background division: k Gaussian components arranged at a pixel point at the t-th frame of a video stream to be detectedCloth respective priority p _i,t The calculation formula of (2) is p _i,t ＝ω _i,t /|σ _i,t I, wherein p _i,t The larger the value of (2), the larger the Gaussian distribution weight is, and the more suitable for describing background content is described; on the other hand, the smaller the variance, p _i,t The larger the value of (2) is, the less the value change of the pixel point in the historical frame image is represented, so the probability that the t frame belongs to the background is larger;

moving object detection: after background division, the GMM associates each pixel of the input frame image with the previous K ₀ The Gaussian distribution is compared to judge whether the pixel belongs to a moving object, if the pixel belongs to a front K of a pixel point in a frame image and a corresponding pixel point in a background image ₀ One of the gaussian distributions matches, then the pixel is considered to belong to the background pixel; otherwise, the pixel is judged as a motion target pixel; according to the detection result of the moving object, the GMM generates a binary image with the same size as the frame image, the value of the pixel belonging to the moving object is 255, the pixel is displayed as a white area, the value of the pixel belonging to the background area is 0, and the pixel is displayed as a black area, so as to obtain a binary image;

GMM parameter update: in order to enable the background model to keep up with background changes in the video, parameter updating is carried out on the GMM, if a certain pixel of the frame image is successfully matched with the ith Gaussian distribution of a corresponding pixel in the background image, the parameter of the Gaussian distribution is updated, and if the pixel is not matched with the corresponding K Gaussian distributions, the pixel is considered to belong to a newly-appearing moving object in the frame image, and a new Gaussian distribution is established for the corresponding pixel in the background image.

3. The method for detecting highway casting based on background model and tracking according to claim 2, wherein in the step of background division, the front K after sorting is taken ₀ A gaussian distribution, which is considered to describe the background content in the video, let:

where D is a pre-established threshold.

4. The method for detecting the highway casting based on the background model and the tracking according to claim 1, wherein the mathematical morphology is used for processing the foreground detection binary image, and the noise in the binary image is removed, specifically:

for the situations of possible tiny illumination change and noise generated by shooting equipment during video recording, the mathematics are described as follows:

Wherein A 'is the result of the morphological processing, A is a binary image to be processed, B1 is a probe used in the open operation and the close operation, B2 is a probe used in the expansion operation, and A' is a square structure of 3×3;

for the interference of the running vehicles and pedestrians from the vehicles to the detection of the throwing objects under a few special conditions, the YOLOx detection is used, a frame image is extracted every set frame and is input into a pre-established YOLO network model to carry out the detection of the vehicles, the specific positions of the pedestrians and the vehicles in the image are obtained, and a binary image of the detection of the vehicles is generated through the position information; taking the center point of each moving object detection frame of the current frame image, if the center point is positioned in a black area in the human-vehicle detection binary image, marking the detection frame as the human-vehicle detection frame in an algorithm program, and judging that the marked detection frames are not to be the casting objects in the subsequent step of detecting the casting objects;

for possible interference items in the road outer region of the expressway, the road region is obtained by semantic segmentation, the road region is eliminated, the output part of the semantic segmentation network model is modified, the pixel values of the region classified as the road in the semantic segmentation result image are changed to [255,255 ], the pixel values of the other regions are changed to [0, 0], and the image is changed to a single-channel gray image, so that a road segmentation binary image is obtained, wherein in the road segmentation binary image, the white region is the road region, and the black region is the non-road region.

5. The method for detecting the highway casting based on the background model and tracking according to claim 1, wherein the tracking algorithm based on the matching between the IOU and the hungarian tracks each moving target, and associates the detection frames of the same moving target in the video in different frames based on the IOU values of the target detection frame of the current frame and the target detection frame of the history frame, thereby completing the tracking of the target, specifically:

setting a moving target detection frame of a current frame image of a video and a moving target detection frame of a history frame to be matched, setting a set of the moving target detection frames in the current frame image as Sc, wherein the number of the detection frames is m, the set of the moving target detection frames of the history frame image is Sh, the number of the detection frames is n, and constructing a cost matrix C, wherein an element C (i, j) in the matrix is the opposite number of IOU values of an ith detection frame in Sc and a jth detection frame in Sh, wherein i is more than or equal to 1 and less than or equal to m, and j is more than or equal to 1 and less than or equal to n; if n is not equal to m, filling the vacant columns or rows in the cost matrix by using 0; the detection frames of Sc and Sh are matched, and the aim is that the obtained total cost of matching is minimum, namely the sum of the opposite numbers of the total IOU values is minimum, which means that the sum of the IOU values is maximum; then using a Hungary matching algorithm to obtain an optimal solution under the cost matrix; under the condition of higher frame rate, the overlapping degree of detection frames of the moving targets in the adjacent frame images is higher, so that the optimal solution with the maximum sum of IOU values can enable each moving target to achieve correct association, and the target tracking is completed.

6. The method for detecting the casting object on the expressway based on the background model and tracking according to claim 1, wherein the fusion perception loss function is used for binarizing the perception map of the whole image and the image in the candidate frame by using an otsu threshold value, performing an AND operation to obtain a casting object candidate target, and tracking the candidate target so as to eliminate the interference of a road surface area, and specifically comprises the following steps:

for the current video frame, tracking the edge contour of the continuous still 2 frames to match so as to obtain a long background image and a short background image; then, the perservical map in the 4-time area of the otsu binarization candidate frame is utilized, then the perservical map of the whole image is binarized by the otsu, morphological processing is carried out on the perservical map and the perservical map of the whole image in the candidate frame, the sum operation is carried out on the perservical map and the perservical map of the whole image in the candidate frame, a rectangular frame is extracted from the map image after the sum operation, the size of Lou (total frame, map frame) and a set threshold value is judged, if the smaller threshold value is set, the smaller threshold value is discarded, and if the smaller threshold value is larger than the set threshold value, new throws are added into a throws frame array.

7. The method for detecting highway casting objects based on the background model and tracking according to claim 1, wherein the two classification networks adopt res net-34, specifically:

The collected samples are input into a ResNet-34 model for training, the two classes refer to road surfaces and non-road surfaces, and the non-road surface samples in the selected training samples comprise: boxes, tires, road cones, bottles, bags and paper; the pavement sample comprises: road surface with lane line, rainy day water mark road surface, reflection of light road surface, ponding road surface and shadow road surface.

8. The expressway casting detection system based on the background model and tracking is characterized by being applied to the expressway casting detection method based on the background model and tracking, which is disclosed in any one of claims 1-7, and comprises a data acquisition module, a background modeling and moving target detection module, a noise processing module, a target tracking module, a casting detection module and a classification module;

the data acquisition module is used for acquiring a video stream to be detected, decoding the video stream to be detected to obtain a current video frame image, wherein the current video frame image comprises a background image and a moving target;

the background modeling and moving object detection module is used for background modeling and moving object detection, a pre-established Gaussian Mixture Model (GMM) is adopted to carry out background modeling on each pixel in a background image by using K Gaussian distributions respectively, a background image is divided, an image in a current video frame is matched with background image pixels, so that a moving object in the video frame image is detected, and the GMM generates a binary image with the same size as the frame image according to the moving object detection result;

The noise processing module is used for processing the foreground detection binary image by using mathematical morphology and removing noise in the binary image;

the target tracking module is used for tracking each moving target based on a tracking algorithm matched with the IOU and the Hungary, and associating the detection frames of the same moving target in the video in different frames based on the IOU values of the target detection frame of the current frame and the target detection frame of the history frame, so that the tracking of the target is completed;

the casting object detection module is used for fusing the perception loss function, binarizing the perception images of the whole image and the images in the candidate frames by using an otsu threshold value, performing AND operation to obtain casting object candidate targets, and tracking the candidate targets;

the classification module is used for eliminating road surface interference by utilizing a classification network and improving the accuracy of detecting the casting objects.

9. An electronic device, the electronic device comprising:

at least one processor; the method comprises the steps of,

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores computer program instructions executable by the at least one processor to enable the at least one processor to perform the background model and tracking based highway casting detection method according to any one of claims 1-7.

10. A computer-readable storage medium storing a program, wherein the program, when executed by a processor, implements the method for detecting highway handoffs based on a background model and tracking as claimed in any one of claims 1 to 7.

Images