CN112287805B - Method and device for detecting moving object, readable storage medium and electronic equipment - Google Patents

Abstract

The present disclosure provides a method for detecting a moving object, including: determining inter-frame difference information between the current frame image and the previous frame image based on the current frame image and the previous frame image; determining a multi-frame image set based on the current frame image and a predetermined number of historical frame images before the current frame image; determining a first moving area in the current frame image based on the inter-frame difference information of each frame image in the multi-frame image set; determining at least one detected area of a predetermined scale based on the first moving area; and determining a moving target in the detected area based on the detected area in the current frame image. The technical solution provided by the present disclosure performs image scaling processing and moving target detection on the first moving area, which can reduce the amount of calculation in the moving object detection process and improve calculation efficiency.

Description

Moving object detection method, device, readable storage medium and electronic device

Technical Field

The present disclosure relates to the field of image processing, and in particular to a moving object detection method, device, readable storage medium and electronic device.

Background Art

The traditional image pyramid algorithm downsamples the input image and then performs image recognition based on the downsampled result. Since the scale supported by the module for detecting moving objects (such as the SOC chip that can perform neural network operations) is fixed, but the system on chip (SOC) chip does not know the scale of the target object in advance, the pyramid result of each layer will be detected by the hardware module for detecting moving objects in the system on chip, and the module for detecting moving objects will determine one of the frames for detecting moving objects based on the supported scale. In the scenario of large image scale, the problem of this method is that the amount of calculation is large. It is necessary to perform multi-layer pyramid processing on each input frame image, and at the same time, to achieve a processing frame image rate of 30 frames per second or 60 frames per second or even higher, the computing power and bandwidth of the system on chip (SOC) chip are put forward. In practical applications, especially in the fields of security monitoring, the background of multiple frames of continuous images is basically the same. To some extent, it can be considered that only a small part of the image area is changing. If each frame of the image is repeatedly processed, it will cause a lot of waste of bandwidth and computing power.

Summary of the invention

In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a moving object detection method, device, readable storage medium and electronic device, which detect moving targets based on the detected area of the first motion area, avoid detecting moving objects in areas outside the first motion area in the image, thereby greatly reducing the amount of calculation in the moving object detection process and improving the calculation efficiency.

According to one aspect of the present disclosure, a method for detecting a moving object is provided, comprising:

Based on the current frame image and the previous frame image, determining inter-frame difference information between the current frame image and the previous frame image;

Determine a set of multiple frame images based on the current frame image and a predetermined number of historical frame images before the current frame image;

Determining a first motion region in the current frame image based on the inter-frame difference information of each frame image in the multi-frame image set;

Based on the first motion area, determining at least one detected area of a predetermined size;

Based on the detected area in the current frame image, a moving target in the detected area is determined.

According to a second aspect of the present disclosure, there is provided a moving object detection device, comprising:

The differential information acquisition module is used to determine the inter-frame differential information between the current frame image and the previous frame image based on the current frame image and the previous frame image;

A multi-frame image set determination module: used to determine a multi-frame image set based on the current frame image and a predetermined number of frame images before the current frame image;

A motion region acquisition module: used to determine a first motion region in the current frame image based on the inter-frame difference information of each frame image in the multi-frame image set;

A scale conversion module: used to determine at least one detected area of a predetermined scale based on the first motion area;

The moving target detection module is used to determine the moving target in the detected area based on the detected area in the current frame image.

According to a third aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute any of the above-mentioned moving object detection methods.

According to a fourth aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

processor;

a memory for storing instructions executable by the processor;

The processor is used to read the executable instructions from the memory and execute the instructions to implement any of the above-mentioned moving object detection methods.

In the above four technical solutions disclosed in the present invention, the first motion area in the current frame image is determined by the differential information in the multi-frame image set, and then calculations are performed based on the first motion area to obtain a detected area of a predetermined scale, and the moving target is detected based on the detected area. In the present invention, the first motion area is scaled and detected. Since the part outside the first motion area does not participate in the calculation, the calculation amount of the detected area is greatly reduced, thereby improving the calculation efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

FIG. 1 is a schematic flow chart of a moving object detection method provided by an exemplary embodiment of the present disclosure.

FIG. 2 is a schematic diagram of a flow chart of differential information confirmation of a moving object detection method provided by another exemplary embodiment of the present disclosure.

FIG. 3 is a schematic diagram of a flow chart of first motion region confirmation in a moving object detection method provided by another exemplary embodiment of the present disclosure.

FIG. 4 is a schematic diagram of a flow chart of first motion region confirmation in a moving object detection method provided by another exemplary embodiment of the present disclosure.

FIG. 5 is a schematic diagram of a flow chart of second motion region merging of a moving object detection method provided by another exemplary embodiment of the present disclosure.

FIG. 6 is a schematic diagram of a flow chart of second motion region merging of a moving object detection method provided by another exemplary embodiment of the present disclosure.

FIG. 7 is a schematic diagram of a flow chart of determining a detected reference image in a moving object detection method provided by another exemplary embodiment of the present disclosure.

FIG. 8 is a schematic diagram of a flow chart of reference image determination in a moving object detection method provided by another exemplary embodiment of the present disclosure.

FIG. 9 is a schematic diagram of a moving object detection device provided by an exemplary embodiment of the present disclosure.

FIG. 10 is a schematic diagram of a differential information acquisition module of a moving object detection device provided by another exemplary embodiment of the present disclosure.

FIG. 11 is a schematic diagram of a first motion region acquisition module of a moving object detection device provided by another exemplary embodiment of the present disclosure.

FIG. 12 is a schematic diagram of a first motion region submodule of a moving object detection device provided by another exemplary embodiment of the present disclosure.

FIG. 13 is a schematic diagram of a first merging subunit of a moving object detection device provided by another exemplary embodiment of the present disclosure.

FIG. 14 is a schematic diagram of a first motion region determining unit of a moving object detection device provided by another exemplary embodiment of the present disclosure.

FIG. 15 is a schematic diagram of various modules for determining a reference image target object in a moving object detection device provided by another exemplary embodiment of the present disclosure.

FIG. 16 is a schematic diagram of a reference image determination module of a moving object detection device provided by another exemplary embodiment of the present disclosure.

FIG. 17 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure.

DETAILED DESCRIPTION

Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

Application Overview

For the detection of moving objects, especially in the process of detecting moving objects in the video using neural networks, due to the different resolutions of the sensors of the image acquisition module or the different distances of the photographed objects, the imaging sizes of the objects will be different. It is usually necessary to scale each frame of the video captured by the image acquisition module to form an image of a specified scale before detecting the moving objects.

In the technical solution disclosed in the present invention, first, the inter-frame difference information is obtained by performing a difference calculation between the current frame image and the previous frame image, and then a multi-frame image set is determined by multiple historical frame images before the current frame image, and then a first motion area is determined based on the inter-frame difference information of the multi-frame image set. In the subsequent image recognition process, the first motion area of the current frame image is scaled to obtain a detected area of a predetermined scale, and a moving target is determined based on the detected area. In the present invention, the first motion area is scaled, and the part outside the first motion area does not participate in the calculation, thereby greatly reducing the amount of calculation in the image scaling process, thereby greatly improving the calculation efficiency.

Exemplary Methods

FIG1 is a moving object detection method provided by an exemplary embodiment of the present disclosure, comprising:

Step 101, based on a current frame image and a previous frame image of the current frame image, determining inter-frame difference information between the current frame image and the previous frame image;

In some embodiments, the current frame image refers to an image frame being processed during the image processing process; the previous frame image refers to an image frame that is before the current frame image and adjacent to the current frame image; the inter-frame difference information refers to the difference image obtained by performing a difference calculation between the current frame image and the previous frame image and the brightness or grayscale information of each pixel in the difference image.

Step 102, determining a multi-frame image set based on the current frame image and a predetermined number of historical frame images before the current frame image;

In some embodiments, the historical frame image refers to multiple frame images before the current frame image. In some preferred embodiments, the historical frame image refers to multiple frame images before the current frame image; a multi-frame image set refers to a set of multiple frame images before the current frame image. The number of historical frame images can be selected according to the real-time requirements in the video processing process. For example, when the real-time requirements are high, one frame of historical frame images is used. When the real-time requirements are low, two or three frames of historical frame images can be selected. In the actual implementation process, the number of historical frame images can also be set according to the bandwidth of the system on chip (SOC) chip. For example, when the system on chip (SOC) chip is sensitive to bandwidth, the historical frame image can be set to one frame image. Each frame image will calculate and update the motion area. When the system on chip (SOC) chip has sufficient bandwidth, the differential result accumulated by the continuous multiple frames of images can be calculated.

Step 103, determining a first motion region in the current frame image based on the inter-frame difference information of each frame image in the multi-frame image set;

In some embodiments, the inter-frame difference information refers to the difference image obtained by performing a difference calculation between the current frame image and the previous frame image, and the brightness or grayscale information of each pixel in the difference image. Multiple continuous inter-frame difference information in a multi-frame image set is accumulated to obtain a cumulative result of multiple continuous inter-frame difference information. When the cumulative result of a specific pixel point exceeds a specified threshold in the cumulative result, the corresponding pixel point in the last frame image in the multi-frame image set is considered to be a moving pixel point, and the position of the moving pixel point and each pixel point set within a predetermined distance around the moving pixel point is the first motion area. Since the moving pixels of the multi-frame images are accumulated when the inter-frame difference information of the multi-frame image set is accumulated, the obtained result includes the moving pixels in each frame image, so that the moving target object will be ghosted along its moving direction in the accumulated result. Therefore, the number of frame images in the multi-frame image set in the previous step can be selected as a number not greater than 3, for example, to avoid the generation of ghosts.

Step 104, determining at least one detected area of a predetermined size based on the first motion area;

In some embodiments, the first motion region is scaled to obtain at least one detected region of a predetermined scale. In the subsequent processing, the at least one detected region of a predetermined scale is sent to a module for detecting moving targets. The module for detecting moving targets may be a SOC chip, an FPGA chip, an ASIC chip, etc., such as a SOC chip capable of performing neural network operations. The predetermined scale is determined by the scale supported by the module for detecting moving targets, and the number of detected regions is determined by the scale of the first motion region and the scale supported by the module for detecting moving targets.

Step 105: Determine a moving target in the detected area based on the detected area in the current frame image.

In some embodiments, the detected area is a portion of the current frame image that has a moving target. Detecting the moving target based on the detected area can reduce the amount of calculation while ensuring the accuracy of the detection. A moving target refers to an object that has changed position between the current frame image and the previous frame image before the current frame image, such as a moving vehicle, a running animal, or a walking person.

In the technical solution disclosed in the present invention, the first motion area in the current frame image is determined by the differential information in the multi-frame image set, and then calculations are performed based on the first motion area to obtain a detected area of a predetermined scale, and the moving target is detected based on the detected area. Since the detected area is a local image in the current frame image, calculations are performed on the detected area, which can greatly reduce the amount of calculations of the detected area and improve calculation efficiency.

As shown in FIG. 2 , based on the embodiment of FIG. 1 , step 101 may further include the following steps:

Step 1011, determining a corresponding pixel difference value of a predetermined color channel in a predetermined color space based on the pixel values corresponding to the current frame image and the previous frame image;

In some embodiments, each frame image has a pixel value in a predetermined color channel in a predetermined color space, and when calculating the pixel difference value, the pixel value of the predetermined color channel in the predetermined color space of the current frame image is differentiated from that of the previous frame image, so as to obtain the pixel difference value. Taking the calculation of the current frame image and the previous frame image in the YUV space as an example, the pixel value of the Y channel of the current frame image in the YUV space is differentiated from the pixel value of the Y channel of the previous frame image in the YUV space, so as to obtain the pixel difference value of the Y channel of the current frame image and the previous frame image.

Taking the calculation of the current frame image and the previous frame image in YUV space as an example, the calculation process is as follows:

The calculation formula for the pixel difference value corresponding to the Y channel is as follows:

|Y(x,y,t)–Y(x,y,t-1)|;

The calculation formula for the pixel difference value corresponding to the U channel is as follows:

|U(x,y,t)–U(x,y,t-1)|;

The calculation formula for the pixel difference value corresponding to the V channel is as follows:

|V(x,y,t)–V(x,y,t-1)|;

Among them, x, y are the coordinates of the image; t, t-1 are the frame image numbers of the current frame image and the previous frame image; Y, U, and V represent three channels respectively. Specifically, Y(x, y, t) represents the pixel value at the coordinate (x, y) in the Y channel of the current frame image in the YUV space.

Those skilled in the art should be able to understand that the above only takes the calculation process in the YUV space as an example, and during the calculation process, calculation can also be performed in the RGB, HSV or HSL space.

Step 1012: Determine the inter-frame difference information based on the pixel difference value and a predetermined weight value.

In some embodiments, the pixel difference value is the pixel difference value of the predetermined color channel in the predetermined color space calculated in the previous step, and the above-mentioned difference value is weighted and calculated according to the predetermined weight value to obtain the inter-frame difference information. The above-mentioned weight value is set according to the moving target that needs to be paid attention to in the video picture and the background of the picture in the video. For example, the information of the moving target that needs to be paid attention to in the video picture is mainly concentrated in the Y channel. At this time, the information recognition requirement for the Y channel is relatively high. The weight value corresponding to the Y channel is set to a relatively high weight value. At this time, the pixel difference value of the Y channel will have a relatively large impact on the inter-frame difference information, which is conducive to the identification of the information of the Y channel. For another example, the background information of the picture in the video is mainly concentrated in the V channel. At this time, during the detection process, it is not necessary to pay attention to the background information. Therefore, the information recognition requirement for the V channel is relatively high. The weight value corresponding to the V channel is set to a relatively high weight value. At this time, the pixel difference value of the V channel will have a relatively small impact on the inter-frame difference information, which is conducive to the identification of the information of the Y channel and the V channel.

Still taking the calculation of the current frame image and the previous frame image in the YUV space as an example, the calculation process is as follows:

D(x,y,t)=((|Y(x,y,t)–Y(x,y,t-1)|<<W0)+

(|U(x,y,t)–U(x,y,t-1)|<<W1)+

(|V(x,y,t)–V(x,y,t-1)|<<W2))

Among them, W0+W1+W2=100%.

In the above formula, D(x, y, t) represents the inter-frame differential information of the pixel at the coordinate (x, y) of the current frame image; Y(x, y, t) represents the pixel differential value at the coordinate (x, y) of the Y channel in the YUV space of the current frame image, and the same applies to Y(x, y, t-1), U(x, y, t), U(x, y, t-1), V(x, y, t), and V(x, y, t-1); W0, W1, and W2 represent the weight values of the pixel differential values in the three color channels. In some embodiments, the above weight values are set according to the moving target that needs to be paid attention to in the video and the background of the video. For example, the information of the moving target that needs to be paid attention to in the video is mainly concentrated in the Y channel. At this time, the information recognition requirements for the Y channel are relatively high, and the weight value corresponding to the Y channel is set to a higher weight value. At this time, the pixel differential value of the Y channel will have a greater impact on the inter-frame differential information, which is conducive to the recognition of the information of the Y channel. For another example, the background information of the picture in the video is mainly concentrated in the V channel. At this time, during the detection process, there is no need to pay attention to the background information. Therefore, the information recognition requirements for the V channel are relatively high. The weight value corresponding to the V channel is set to a higher weight value. At this time, the pixel differential value of the V channel will have less impact on the formation of inter-frame differential information, which is conducive to the recognition of information from the Y channel and the V channel.

By using the method in the above embodiment to confirm the inter-frame difference information, the pixel difference values of all color channels can be paid attention to, so that the calculation result is sensitive to the grayscale of the moving object. Of course, in some scenarios, for example, in order to adapt to the scene of traffic lights, it is also possible to only pay attention to one or more components in YUV, so that the calculation result is sensitive to the color change of the object in the image.

As shown in FIG3 , based on the embodiment shown in FIG1 , the above step 103 may further include the following steps:

Step 1031, accumulating the inter-frame difference information of each frame image in the multi-frame image set to determine the accumulated difference information of the multi-frame image set;

In some embodiments, the inter-frame differential information of each frame image of the multi-frame image set is accumulated, and the accumulated result is used as the accumulated differential information of the multi-frame image set; the differential information will be used to determine the first motion area of the current frame image later. The specific calculation formula of the differential information is as follows:

C(x,y,Δt)=D(x,y,t1)+D(x,y,t1+1)+…+D(x,y,t2);

Among them, C(x, y, Δt) represents the accumulated result of the pixel difference value at the point (x, y); D(x, y, t1) represents the pixel difference value of the t1 frame image at (x, y).

Step 1032, determining the moving pixel points in the current frame image based on the comparison result between the accumulated difference information and a predetermined threshold;

In some embodiments, the predetermined threshold is set based on the stray noise of the sensor that collects the image. The greater the stray noise of the sensor, the greater the predetermined threshold is usually set, and the purpose is to filter out recognition errors caused by stray noise. For example, two frames of images are substantially the same. When the threshold is set too small, due to the influence of the stray noise of the sensor, the motion area is recognized in the two frames of images.

Step 1033: Determine a first motion region of the current frame image based on the motion pixel points in the current frame image.

In some embodiments, after the accumulated differential information of all pixels in the current frame image is determined, pixels whose accumulated differential information is greater than a predetermined threshold are screened out, that is, moving pixels are screened out, and the positions occupied by the moving pixels are the first moving area. Usually, in order to ensure the integrity of the moving target, the pixels within a certain distance around the moving pixels are also taken as the pixel values in the first moving area.

In the above-mentioned embodiment, the inter-frame differential information in a multi-frame image set is accumulated, and the moving pixels are identified based on the accumulated results. In the multi-frame image set, the moving area of the current frame is identified, thereby reducing the computational complexity of the moving area identification and the subsequent moving target detection, thereby greatly improving the efficiency of identifying moving objects.

As shown in FIG. 4 , based on the embodiment shown in FIG. 3 , step 1033 may further include the following steps:

Step 10331, based on the moving pixel point and the predetermined distance, determining a set of pixel points whose distances from the moving pixel point are within the predetermined distance range;

In some embodiments, the predetermined distance is set based on the scene of each frame of image. For example, when the target objects in the scene of each frame of image are relatively large, the predetermined distance is set to be larger, and when the target objects in the scene of each frame of image are relatively small, the predetermined distance is set to be smaller. The pixel point set includes moving pixels and pixels whose distance from the moving pixels is less than the predetermined distance. The purpose of this setting is to include the entire moving target object in the first moving area. For example, when a part of a moving target object moves, it is obviously impossible to include the entire moving target object in the first moving area by using the moving pixels as the moving area. However, by setting a reasonable predetermined distance and determining the first moving area by using the pixels within the predetermined distance around the moving pixels as elements of the pixel point set, the moving target object can usually be included in the first moving area.

Step 10332: determine the first motion area based on the pixel point set.

In some embodiments, the position of the moving pixel in the current frame image and the position of the pixel within a predetermined distance around the moving pixel in the image are merged to obtain the first motion region. For example, when the predetermined distance is set to 16 pixels, the scale of the first motion region determined by a moving pixel is a 32*32 first motion region. At this time, the first motion region includes both the moving pixel and the pixel within a predetermined distance around the moving pixel.

By adopting the technical solution in the above embodiment, the position of the moving pixel point and the pixel points within a predetermined distance around the moving pixel point in the current frame image is taken as the first moving area, which can ensure that the entire moving target is included in the first moving area, which is facilitating the subsequent identification of the moving target in the first moving area.

As shown in FIG. 5 , based on the embodiment shown in FIG. 4 , step 10332 may further include the following steps:

Step 103321, determining a second motion area based on the pixel point set;

In some embodiments, each moving pixel point can determine a pixel point set, and the positions occupied by each pixel point in each pixel point set are combined to form the second moving area; since each second moving area includes not only the moving pixel point, but also the pixels within a predetermined distance around the moving pixel point, when the distance between two moving pixels is less than the predetermined distance, the second moving area determined by the pixel point sets corresponding to the two moving pixels will inevitably have overlapping pixels.

Step 103322: Based on the overlapping pixels of the two or more second motion regions, merge the two or more second motion regions into a first motion region.

In some embodiments, when two second motion areas have overlapping pixels, one of the motion pixels is within the second motion area corresponding to the other motion pixel. At this time, the two motion pixels can be considered as two motion pixels of the same moving target. Therefore, the two second motion areas are merged into the first motion area, thereby ensuring that the same moving target is not divided, which facilitates the subsequent identification of the moving target.

As shown in FIG. 6 , based on the embodiment shown in FIG. 4 , step 10332 may further include the following steps:

Step 103321, determining a second motion area based on the pixel point set;

In some embodiments, each moving pixel can determine a pixel set, and the positions occupied by each pixel in each pixel set are combined to form a second moving area. When the distance between two adjacent second moving areas is small, the two second moving areas can usually be considered to be moving areas on the same target object.

Step 103323, determining a merging result of two adjacent second motion regions based on a distance between the two adjacent second motion regions;

In some embodiments, when a part of a moving target is close to the image of a background area, it will cause discontinuity between the two second motion areas identified for the same moving target. At this time, in order to ensure that the same moving target is divided into the same first motion area, whether to merge the two second motion areas is determined based on the distance between the two adjacent second motion areas. When the distance is less than a specific distance, the two second motion areas are merged. When the distance is not less than a specific distance, the two second motion areas are respectively used as two first motion areas. The above-mentioned specific distance can be set according to the scene of the picture in the video, and can also be set according to the moving target that needs to be identified.

Step 103324: Based on the merging result, two adjacent second motion areas whose distances meet a preset condition are merged into a first motion area.

In some embodiments, the above-mentioned predetermined condition may be, for example, that the distance between the two second motion areas is less than a specific distance. Based on the above steps, the two second motion areas in the current frame image whose distance is less than the specific distance are merged into the first motion area. When setting the above-mentioned specific distance, it is necessary to set it according to the scene of the picture in the video, and it can also be set according to the moving target to be identified. For example, when the moving target occupies a large proportion in the video image, the specific distance can be set to a larger value. For example, when the scene color of the picture in the video is relatively single, the specific distance can be set to a smaller value. The purpose of merging the two second motion areas whose distance in the current frame image is less than the specific distance is to ensure that the same moving target is divided into the same first motion area. If the specific distance is set too small, the complete moving object may be segmented, resulting in the inability of subsequent modules to correctly identify it; if the specific distance is set too large, the scale of the first motion area will be larger, and the subsequent scaling of the first motion area and the calculation amount of the recognition of the detected area will increase.

As shown in FIG. 7 , based on the embodiment shown in FIG. 1 , the method further includes the following steps:

Step 106, determining a reference image based on a comparison result between the current frame image and a predetermined condition;

In some embodiments, the picture in the video expresses a scene with a slowly changing background. In this case, reference images need to be used at intervals to help the system grasp the slowly changing background. The predetermined condition can be determined based on the background of the picture in the video, or it can be determined based on the moving target of the picture in the video. For example, the current frame image can be selected as the reference image every fixed number of historical frame images based on the background change speed of the picture in the video. When the background change speed of the picture in the video is fast, the number of historical frame images at intervals is small, and when the background change speed of the picture in the video is slow, the number of historical frame images at intervals is large. It is also possible to determine whether to use the current frame image as a reference image based on the proportion of the first motion area in the current frame image.

Step 107, downsampling the reference image one by one to obtain at least one detected reference image of a predetermined scale;

In some embodiments, the reference image is also limited by the scale supported by the module for detecting moving objects (e.g., a SOC chip capable of performing neural network operations). Therefore, the reference image needs to be downsampled at least once to obtain at least one detected reference image of a predetermined scale. Since the minimum image scale that can be obtained by a single downsampling is 1/2 of the original image scale, when the scale of the reference image differs greatly from the predetermined scale, the reference image needs to be downsampled multiple times, and each downsampling obtains a frame of a detected reference image of a predetermined scale until a detected reference image of a scale supported by the module for detecting moving objects is obtained. Therefore, the number of detected reference images needs to be determined by the scale supported by the module for detecting moving objects and the scale of the reference image. The predetermined scale can be determined by the scale supported by the module for detecting moving objects.

Step 108: Determine the moving target in the reference image based on the detected reference image of the at least one predetermined scale.

In some embodiments, after obtaining the detected reference image, the detected reference image is sent to a module for detecting moving objects, such as a SOC chip capable of performing neural network operations. The module is used to detect moving objects in the reference image.

In the above-mentioned embodiment, by selecting reference images at intervals, the background changes of the video can be identified, thereby eliminating the influence of the background changes on the recognition of moving objects. For example, as time changes, the illumination changes caused by the rising and setting of the sun will affect the brightness of each frame of the video. By adopting the method of this embodiment, the influence of the background brightness changes on the recognition of moving objects can be eliminated.

Based on the embodiment shown in FIG. 7 , step 106 includes: determining a reference image based on the number of frame images between the current frame image and the previous reference image.

In some embodiments, the above-mentioned method of determining the reference image is a static determination method, and the specific method is to take out a frame image as a reference image at every fixed number of frame images. For example, the reference image is called an A frame image, and the frame image that scales the first motion area is called a B frame image. By setting a period T, the reference image is periodically scaled to obtain the detected reference image every T period and sent to the moving target recognition module for processing. For example, taking a video with a frame rate of 50Hz as an example, when T is 100 milliseconds, that is, a frame image is taken as a reference image at every 100 millisecond interval. At this time, the arrangement order of the reference image and the frame image that scales the first motion area is ABBBBABBBB. Since in the same video, the frame image rate is fixed, that is, the time occupied by each frame image is fixed, and the fixed period is specified to determine the number of frame images at the interval. Of course, you can also directly set a fixed number of interval frame images, for example, set it to obtain a frame image as a reference image every 3 frames. At this time, the frame images of the full image zoom and the frame images of the first motion area zoom will be arranged as follows: ABBBABBB.

By setting the reference image in the manner of this embodiment, the influence caused by the slow changes in the background environment can be eliminated, which is conducive to the accurate recognition of the moving target object.

As shown in FIG8 , based on the embodiment shown in FIG7 , step 106 may further include the following steps:

Step 1061, determining the proportion of the first motion region in the current frame image to the current frame image;

In some embodiments, the ratio of the first motion region to the current frame image refers to the ratio of the number of pixels in the first motion region to the number of all pixels in the current frame image. In some embodiments, there may be multiple first motion regions. When there are multiple first motion regions, the ratio refers to the ratio of the sum of the number of pixels in the multiple first motion regions to the number of all pixels in the current frame image.

Step 1062: Determine a reference image based on the comparison result of the ratio and a predetermined threshold range.

In some embodiments, when the detected motion area exceeds a certain proportion of the whole image, the whole image is automatically scaled and sent to the motion target detection module for processing, and the proportion of the motion area can be configured according to different application scenarios. For example, in a scene with dense flow of people, vehicles or animals, setting the proportion to be smaller helps to identify the moving details of the picture in the video. At this time, the proportion of the A frame image and the B frame image changes dynamically. For example, when the proportion of the first motion area of the first frame image, the fourth frame image to the sixth frame image and the twelfth frame image exceeds a certain proportion of the whole image, the first frame image, the fourth frame image to the sixth frame image and the twelfth frame image are used as reference frames, and the first motion area of other frame images does not exceed a certain proportion of the whole image, and the first motion area is scaled, so that the reference image and the frame image for scaling the first motion area are arranged in the following order: ABBAAABBBBBABB.

By adopting the technical solution of this embodiment, the reference image can be determined according to the proportion of the moving target in the current frame image, so that the activity details of the moving target in the video can be fully recognized.

Those skilled in the art should understand that the above two methods of setting reference images, static and dynamic, can be switched. For example, the ratio (or absolute number) of moving pixels per unit time is detected. When the ratio is low, the reference image is selected in a static manner. For another example, according to the scene, it is specified that the reference image is selected in a static manner at night and in a dynamic manner during the day.

Based on the embodiment shown in FIG. 7 above, the following steps may be further included after step 107:

Based on the detected area of the current frame image and the detected reference image of the reference image of a frame image before the current frame image, a detected image of a predetermined scale corresponding to the current frame image is determined.

In some embodiments, this step can be implemented in two ways, one of which is as follows: according to the coordinates of the first motion motion area, the position of the motion area relative to the starting point of the image is obtained, and the relative position can be converted into an address offset relative to the starting point of the image. This method stores the scaling result of the first motion area, that is, at least one detected area of a predetermined scale, in a storage unit. When the module for detecting the moving target needs the scaling result of the entire image of the current frame image, the scaling result of the detected area and a frame of reference image before the current frame image, that is, at least one frame of detected reference image, can be read out at the same time. According to the address offset of the first motion area, the detected reference image and the detected area are summed, and the summed result is the scaling result of the entire image of the current frame image. The second is as follows: if the module for detecting moving targets needs to directly obtain the full image scaling result of the current frame image instead of transferring it through a storage unit, the previously detected reference image stored in the storage unit can be read out during the calculation process of the detected area, and whether the detected area and the previously detected reference image are summed and output to the subsequent processing module or the previously detected reference image in the storage unit is output to the subsequent processing module is determined based on whether the current image pixel is a moving area. If the pixel is in the first moving area, the detected area and the previously detected reference image are summed and output to the subsequent module, otherwise, the previously detected reference image is directly output to the subsequent module for processing. This processing transmits the detected area of the predetermined scale obtained in the moving area of the current frame image to the module for detecting moving targets, eliminating the process of storing the detected area of the predetermined scale in the storage unit and reading it out from the storage unit, thereby greatly reducing the delay of image processing.

Exemplary Devices

FIG9 is a moving object detection device provided by an exemplary embodiment of the present disclosure, comprising:

The difference information acquisition module 901, the user determines the inter-frame difference information between the current frame image and the previous frame image based on the current frame image and the previous frame image of the current frame image;

In some embodiments, the current frame image refers to an image frame being processed during the image processing process; the previous frame image refers to an image frame that is before the current frame image and adjacent to the current frame image; the inter-frame difference information refers to the difference image obtained by performing a difference calculation between the current frame image and the previous frame image and the brightness or grayscale information of each pixel in the difference image.

A multi-frame image set determination module 902, configured to determine a multi-frame image set based on the current frame image and a predetermined number of historical frame images before the current frame image;

In some embodiments, the historical frame image refers to multiple frame images before the current frame image. In some preferred embodiments, the historical frame image refers to multiple frame images before the current frame image; a multi-frame image set refers to a set including multiple frame images before the current frame image. The number of historical frame images can be selected according to the real-time requirements in the video processing process. For example, when the real-time requirements are high, one frame of historical frame images is used. When the real-time requirements are low, two or three frames of historical frame images can be selected. In the actual implementation process, the number of historical frame images can also be set according to the bandwidth of the system on chip (SOC). For example, when the system on chip (SOC) is sensitive to bandwidth, the historical frame image can be set to one frame image. Each frame image will calculate and update the motion area. When the system on chip (SOC) has sufficient bandwidth, the differential result accumulated by the continuous multiple frames of images can be calculated.

A first motion region acquisition module 903, configured to determine a first motion region in the current frame image based on the inter-frame difference information of each frame image in the multi-frame image set;

In some embodiments, the inter-frame difference information refers to the difference image obtained by performing a difference calculation between the current frame image and the previous frame image, and the brightness or grayscale information of each pixel in the difference image. Multiple continuous inter-frame difference information in a multi-frame image set is accumulated to obtain a cumulative result of multiple continuous inter-frame difference information. When the cumulative result of a specific pixel point exceeds a specified threshold in the cumulative result, the corresponding pixel point in the last frame image in the multi-frame image set is considered to be a moving pixel point, and the position of the moving pixel point and each pixel point set within a predetermined distance around the moving pixel point is the first motion area. Since the moving pixels of the multi-frame images are accumulated when the inter-frame difference information of the multi-frame image set is accumulated, the obtained result includes the moving pixels in each frame image, so that the moving target object will be ghosted along its moving direction in the accumulated result. Therefore, the number of frame images in the multi-frame image set in the previous step can be selected as a number not greater than 3, for example, to avoid the generation of ghosts.

A scale conversion module 904 is used to determine at least one detected area of a predetermined scale based on the first motion area;

In some embodiments, the first motion region is scaled to obtain at least one detected region of a predetermined scale. In the subsequent processing, the at least one detected region of a predetermined scale is sent to a module for detecting moving targets. The module for detecting moving targets may be a SOC chip, an FPGA chip, an ASIC chip, etc., such as a SOC chip capable of performing neural network operations. The predetermined scale is determined by the scale supported by the module for detecting moving targets, and the number of detected regions is determined by the scale of the first motion region and the scale supported by the module for detecting moving targets.

The moving object detection module 905 determines the moving object in the detected area based on the detected area in the current frame image.

In some embodiments, the detected area is a portion of the current frame image that has a moving target. Detecting the moving target based on the detected area can reduce the amount of calculation while ensuring the accuracy of the detection. A moving target refers to an object that has changed position between the current frame image and the previous frame image before the current frame image, such as a moving vehicle, a running animal, or a walking person.

In the technical solution disclosed in the present invention, the first motion area in the current frame image is determined by the differential information in the multi-frame image set, and then calculations are performed based on the first motion area to obtain a detected area of a predetermined scale, and the moving target is detected based on the detected area. Since the detected area is a local image in the current frame image, calculations are performed on the detected area, which can greatly reduce the amount of calculations of the detected area and improve calculation efficiency.

As shown in FIG. 10 , based on the embodiment of FIG. 9 , the differential information acquisition module 901 may further include the following submodules:

A pixel difference submodule 9011 is used to determine a corresponding pixel difference value of a predetermined color channel in a predetermined color space based on the pixel values corresponding to the current frame image and the previous frame image;

In some embodiments, each frame image has a pixel value in a predetermined color channel in a predetermined color space, and when calculating the pixel difference value, the pixel value of the predetermined color channel in the predetermined color space of the current frame image is differentiated from that of the previous frame image, so as to obtain the pixel difference value. Taking the calculation of the current frame image and the previous frame image in the YUV space as an example, the pixel value of the Y channel of the current frame image in the YUV space is differentiated from the pixel value of the Y channel of the previous frame image in the YUV space, so as to obtain the pixel difference value of the Y channel of the current frame image and the previous frame image.

Taking the calculation of the current frame image and the previous frame image in YUV space as an example, the calculation process is as follows:

The calculation formula for the pixel difference value corresponding to the Y channel is as follows:

|Y(x,y,t)–Y(x,y,t-1)|;

The calculation formula for the pixel difference value corresponding to the U channel is as follows:

|U(x,y,t)–U(x,y,t-1)|;

The calculation formula for the pixel difference value corresponding to the V channel is as follows:

|V(x,y,t)–V(x,y,t-1)|;

Among them, x, y are the coordinates of the image; t, t-1 are the frame image numbers of the current frame image and the previous frame image; Y, U, and V represent three channels respectively. Specifically, Y(x, y, t) represents the pixel value at the coordinate (x, y) in the Y channel of the current frame image in the YUV space.

Those skilled in the art should be able to understand that the above only takes the calculation process in the YUV space as an example, and during the calculation process, calculation can also be performed in the RGB, HSV or HSL space.

The inter-frame difference submodule 9012 is used to determine the inter-frame difference information based on the pixel difference value and a predetermined weight value.

In some embodiments, the pixel difference value is the pixel difference value of the predetermined color channel in the predetermined color space calculated in the previous step, and the above-mentioned difference value is weighted and calculated according to the predetermined weight value to obtain the inter-frame difference information. The above-mentioned weight value is set according to the moving target that needs to be paid attention to in the video picture and the background of the picture in the video. For example, the information of the moving target that needs to be paid attention to in the video picture is mainly concentrated in the Y channel. At this time, the information recognition requirement for the Y channel is relatively high. The weight value corresponding to the Y channel is set to a relatively high weight value. At this time, the pixel difference value of the Y channel will have a relatively large impact on the inter-frame difference information, which is conducive to the identification of the information of the Y channel. For another example, the background information of the picture in the video is mainly concentrated in the V channel. At this time, during the detection process, it is not necessary to pay attention to the background information. Therefore, the information recognition requirement for the V channel is relatively high. The weight value corresponding to the V channel is set to a relatively high weight value. At this time, the pixel difference value of the V channel will have a relatively small impact on the inter-frame difference information, which is conducive to the identification of the information of the Y channel and the V channel.

Still taking the calculation of the current frame image and the previous frame image in the YUV space as an example, the calculation process is as follows:

D(x,y,t)=((|Y(x,y,t)–Y(x,y,t-1)|<<W0)+

(|U(x,y,t)–U(x,y,t-1)|<<W1)+

(|V(x,y,t)–V(x,y,t-1)|<<W2))

Among them, W0+W1+W2=100%.

In the above formula, D(x, y, t) represents the inter-frame differential information of the pixel at the coordinate (x, y) of the current frame image; Y(x, y, t) represents the pixel differential value at the coordinate (x, y) of the Y channel in the YUV space of the current frame image, and the same applies to Y(x, y, t-1), U(x, y, t), U(x, y, t-1), V(x, y, t), and V(x, y, t-1); W0, W1, and W2 represent the weight values of the pixel differential values in the three color channels. In some embodiments, the above weight values are set according to the moving target that needs to be paid attention to in the video and the background of the video. For example, the information of the moving target that needs to be paid attention to in the video is mainly concentrated in the Y channel. At this time, the information recognition requirements for the Y channel are relatively high, and the weight value corresponding to the Y channel is set to a higher weight value. At this time, the pixel differential value of the Y channel will have a greater impact on the inter-frame differential information, which is conducive to the recognition of the information of the Y channel. For another example, the background information of the picture in the video is mainly concentrated in the V channel. At this time, during the detection process, there is no need to pay attention to the background information. Therefore, the information recognition requirements for the V channel are relatively high. The weight value corresponding to the V channel is set to a higher weight value. At this time, the pixel differential value of the V channel will have less impact on the formation of inter-frame differential information, which is conducive to the recognition of information from the Y channel and the V channel.

By using the method in the above embodiment to confirm the inter-frame difference information, the pixel difference values of all color channels can be paid attention to, so that the calculation result is sensitive to the grayscale of the moving object. Of course, in some scenarios, for example, in order to adapt to the scene of traffic lights, it is also possible to only pay attention to one or more components in YUV, so that the calculation result is sensitive to the color change of the object in the image.

As shown in FIG. 11 , based on the embodiment shown in FIG. 9 , the first motion region acquisition module 903 may further include the following submodules:

The difference accumulation submodule 9031 is used to accumulate the inter-frame difference information of each frame image in the multi-frame image set to determine the accumulated difference information of the multi-frame image set;

In some embodiments, the inter-frame differential information of each frame image of the multi-frame image set is accumulated, and the accumulated result is used as the accumulated differential information of the multi-frame image set; the differential information will be used to determine the first motion area of the current frame image later. The specific calculation formula of the differential information is as follows:

C(x,y,Δt)=D(x,y,t1)+D(x,y,t1+1)+…+D(x,y,t2);

Among them, C(x, y, Δt) represents the accumulated result of the pixel difference value at the point (x, y); D(x, y, t1) represents the pixel difference value of the t1 frame image at (x, y).

A moving pixel point determination submodule 9032 is used to determine the moving pixel points in the current frame image based on the comparison result between the accumulated difference information and a predetermined threshold value;

In some embodiments, the predetermined threshold is set based on the stray noise of the sensor that collects the image. The greater the stray noise of the sensor, the greater the predetermined threshold is usually set, and the purpose is to filter out recognition errors caused by stray noise. For example, two frames of images are substantially the same. When the threshold is set too small, due to the influence of the stray noise of the sensor, the motion area is recognized in the two frames of images.

The first motion region submodule 9033 determines a first motion region of the current frame image based on the motion pixel points in the current frame image.

In some embodiments, after the accumulated differential information of all pixels in the current frame image is determined, pixels whose accumulated differential information is greater than a predetermined threshold are screened out, that is, moving pixels are screened out, and the positions occupied by the moving pixels are the first moving area. Usually, in order to ensure the integrity of the moving target, the pixels within a certain distance around the moving pixels are also taken as the pixel values in the first moving area.

In the above-mentioned embodiment, the inter-frame differential information in a multi-frame image set is accumulated, and the moving pixels are identified based on the accumulated results. In the multi-frame image set, the moving area of the current frame is identified, thereby reducing the computational complexity of the moving area identification and the subsequent moving target detection.

As shown in FIG. 12 , based on the embodiment shown in FIG. 11 , the first motion region submodule 9033 may further include the following units:

A pixel point set determining unit 90331 is used to determine, based on the moving pixel point and the predetermined distance, a pixel point set whose distance from the moving pixel point is within the predetermined distance range;

In some embodiments, the predetermined distance is set based on the scene of each frame of image. For example, when the target objects in the scene of each frame of image are relatively large, the predetermined distance is set to be larger, and when the target objects in the scene of each frame of image are relatively small, the predetermined distance is set to be smaller. The pixel point set includes moving pixels and pixels whose distance from the moving pixels is less than the predetermined distance. The purpose of this setting is to include the entire moving target object in the first moving area. For example, when a part of a moving target object moves, it is obviously impossible to include the entire moving target object in the first moving area by using the moving pixels as the moving area. However, by setting a reasonable predetermined distance and determining the first moving area by using the pixels within the predetermined distance around the moving pixels as elements of the pixel point set, the moving target object can usually be included in the first moving area.

The first motion region determining unit 90332 is configured to determine the first motion region based on the pixel point set.

In some embodiments, the position of the moving pixel in the current frame image and the position of the pixel within a predetermined distance around the moving pixel in the image are merged to obtain the first motion region. For example, when the predetermined distance is set to 16, the scale of the first motion region determined by a moving pixel is a 32*32 first motion region. At this time, the first motion region includes both the moving pixel and the pixel within a predetermined distance around the moving pixel.

By adopting the technical solution in the above embodiment, the position of the moving pixel point and the pixel points within a predetermined distance around the moving pixel point in the current frame image is taken as the first moving area, which can ensure that the entire moving target is included in the first moving area, which is facilitating the subsequent identification of the moving target in the first moving area.

As shown in FIG. 13 , based on the embodiment shown in FIG. 12 , the first motion region determining unit 90332 may further include the following subunits:

A second motion region determining subunit 903321 is configured to determine a second motion region based on the pixel point set;

In some embodiments, each moving pixel point can determine a pixel point set, and the positions occupied by each pixel point in each pixel point set are combined to form the second moving area; since each second moving area includes not only the moving pixel point, but also the pixels within a predetermined distance around the moving pixel point, when the distance between two moving pixels is less than the predetermined distance, the second moving area determined by the pixel point sets corresponding to the two moving pixels will inevitably have overlapping pixels.

The first merging subunit 903322 is configured to merge the two or more second motion regions into a first motion region based on overlapping pixels in the two or more second motion regions.

In some embodiments, when two second motion areas have overlapping pixels, one of the motion pixels is within the second motion area corresponding to the other motion pixel. At this time, the two motion pixels can be considered as two motion pixels of the same moving target. Therefore, the two second motion areas are merged into the first motion area, thereby ensuring that the same moving target is not divided, which facilitates the subsequent identification of the moving target.

As shown in FIG. 14 , based on the embodiment shown in FIG. 12 , the first motion region acquisition module 90332 may further include the following subunits:

A second motion region determining subunit 903321 is configured to determine a second motion region based on the pixel point set;

In some embodiments, each moving pixel can determine a pixel set, and the positions occupied by each pixel in each pixel set are combined to form a second moving area. When the distance between two adjacent second moving areas is small, the two second moving areas can usually be considered to be moving areas on the same target object.

A merging result confirmation subunit 903323 is used to determine a merging result of two adjacent second motion areas based on a distance between the two adjacent second motion areas;

In some embodiments, when a part of a moving target is close to the image of a background area, it will cause discontinuity between the two second motion areas identified for the same moving target. At this time, in order to ensure that the same moving target is divided into the same first motion area, whether to merge the two second motion areas is determined based on the distance between the two adjacent second motion areas. When the distance is less than a specific distance, the two second motion areas are merged. When the distance is not less than a specific distance, the two second motion areas are respectively used as two first motion areas. The above-mentioned specific distance can be set according to the scene of the picture in the video, and can also be set according to the moving target that needs to be identified.

The second merging subunit 903324 is configured to merge two adjacent second motion regions whose distances meet a preset condition into a first motion region based on the merging result.

In some embodiments, the above-mentioned predetermined condition may be, for example, that the distance between the two second motion areas is less than a specific distance. Based on the above steps, the two second motion areas in the current frame image whose distance is less than the specific distance are merged into the first motion area. When setting the above-mentioned specific distance, it is necessary to set it according to the scene of the picture in the video, and it can also be set according to the moving target to be identified. For example, when the moving target occupies a large proportion in the video image, the specific distance can be set to a larger value. For example, when the scene color of the picture in the video is relatively single, the specific distance can be set to a smaller value. The purpose of merging the two second motion areas whose distance in the current frame image is less than the specific distance is to ensure that the same moving target is divided into the same first motion area. If the specific distance is set too small, the complete moving object may be segmented, resulting in the inability of subsequent modules to correctly identify it; if the specific distance is set too large, the scale of the first motion area will be larger, and the subsequent scaling of the first motion area and the calculation amount of the recognition of the detected area will increase.

As shown in FIG. 15 , based on the embodiment shown in FIG. 9 , the device further includes the following modules:

A reference image determination module 906, configured to determine a reference image based on a comparison result between the current frame image and a predetermined condition;

In some embodiments, in some embodiments, the picture in the video expresses a scene with a slowly changing background. At this time, it is necessary to use reference images at intervals to help the system grasp the slowly changing background. The predetermined condition can be determined based on the background of the picture in the video, or it can be determined based on the moving target of the picture in the video. For example, based on the background change speed of the picture in the video, the current frame image can be selected as the reference image every fixed number of historical frame images. When the background change speed of the picture in the video is fast, the number of historical frame images at intervals is small, and when the background change speed of the picture in the video is slow, the number of historical frame images at intervals is large. It is also possible to determine whether to use the current frame image as a reference image based on the proportion of the first motion area in the current frame image.

A detected reference image determination module 907 is used to successively downsample the reference image to obtain at least one detected reference image of a predetermined scale;

In some embodiments, the reference image is also limited by the scale supported by the module for detecting moving objects (e.g., a SOC chip capable of performing neural network operations). Therefore, the reference image needs to be downsampled at least once to obtain at least one detected reference image of a predetermined scale. Since the minimum image scale that can be obtained by a single downsampling is 1/2 of the original image scale, when the scale of the reference image differs greatly from the predetermined scale, the reference image needs to be downsampled multiple times, and each downsampling obtains a frame of a detected reference image of a predetermined scale until a detected reference image of a scale supported by the module for detecting moving objects is obtained. Therefore, the number of detected reference images needs to be determined by the scale supported by the module for detecting moving objects and the scale of the reference image. The predetermined scale can be determined by the scale supported by the module for detecting moving objects.

The reference image target determination module 908 is used to determine the moving target in the reference image based on the detected reference image of at least one predetermined scale.

In some embodiments, after obtaining the detected reference image, the detected reference image is sent to a module for detecting moving objects, such as a SOC chip capable of performing neural network operations. The module is used to detect moving objects in the reference image.

In the above-mentioned embodiment, by selecting reference images at intervals, the background changes of the video can be identified, thereby eliminating the influence of the background changes on the recognition of moving objects. For example, as time changes, the illumination changes caused by the rising and setting of the sun will affect the brightness of each frame of the video. By adopting the method of this embodiment, the influence of the background brightness changes on the recognition of moving objects can be eliminated.

Based on the embodiment shown in FIG. 15 , the reference image determination module 906 is used to determine the reference image based on the number of frame images between the current frame image and the previous reference image.

In some embodiments, the above-mentioned method of determining the reference image is a static determination method, and the specific method is to take out a frame image as a reference image at every fixed number of frame images. For example, the reference image is called an A frame image, and the frame image that scales the first motion area is called a B frame image. By setting a period T, the reference image is periodically scaled to obtain the detected reference image every T period and sent to the moving target recognition module for processing. For example, taking a video with a frame rate of 50Hz as an example, when T is 100 milliseconds, that is, a frame image is taken as a reference image at every 100 millisecond interval. At this time, the arrangement order of the reference image and the frame image that scales the first motion area is ABBBBABBBB. Since in the same video, the frame image rate is fixed, that is, the time occupied by each frame image is fixed, and the fixed period is specified to determine the number of frame images at the interval. Of course, you can also directly set a fixed number of interval frame images, for example, set it to obtain a frame image as a reference image every 3 frames. At this time, the frame images of the full image zoom and the frame images of the first motion area zoom will be arranged as follows: ABBBABBB.

By setting the reference image in the manner of this embodiment, the influence caused by the slow changes in the background environment can be eliminated, which is conducive to the accurate recognition of the moving target object.

As shown in FIG. 16 , based on the embodiment shown in FIG. 15 , the reference image determination module 906 may further include the following modules:

A ratio confirmation submodule 9061 is used to determine the ratio of the first motion area in the current frame image to the current frame image;

In some embodiments, the ratio of the first motion region to the current frame image refers to the ratio of the number of pixels in the first motion region to the number of all pixels in the current frame image. In some embodiments, there may be multiple first motion regions. When there are multiple first motion regions, the ratio refers to the ratio of the sum of the number of pixels in the multiple first motion regions to the number of all pixels in the current frame image.

The reference image confirmation submodule 9062 is used to determine a reference image based on the comparison result of the ratio and a predetermined threshold range.

In some embodiments, when the detected motion area exceeds a certain proportion of the whole image, the whole image is automatically scaled and sent to the motion target detection module for processing, and the proportion of the motion area can be configured according to different application scenarios. For example, in a scene with dense flow of people, vehicles or animals, setting the proportion to be smaller helps to identify the moving details of the picture in the video. At this time, the proportion of the A frame image and the B frame image changes dynamically. For example, when the proportion of the first motion area of the first frame image, the fourth frame image to the sixth frame image and the twelfth frame image exceeds a certain proportion of the whole image, the first frame image, the fourth frame image to the sixth frame image and the twelfth frame image are used as reference frames, and the first motion area of other frame images does not exceed a certain proportion of the whole image, and the first motion area is scaled, so that the reference image and the frame image for scaling the first motion area are arranged in the following order: ABBAAABBBBBABB.

By adopting the technical solution of this embodiment, the reference image can be determined according to the proportion of the moving target in the current frame image, so that the activity details of the moving target in the video can be fully recognized.

Those skilled in the art should understand that the above two methods of setting reference images, static and dynamic, can be switched. For example, the ratio (or absolute number) of moving pixels per unit time is detected. When the ratio is low, the reference image is selected in a static manner. For another example, according to the scene, it is specified that the reference image is selected in a static manner at night and in a dynamic manner during the day.

Based on the embodiment shown in FIG. 15 above, the device may further include the following modules:

The current frame image full image scaling module is used to determine a detected image of a predetermined scale corresponding to the current frame image based on the detected area of the current frame image and the detected reference image of the reference image of a frame image before the current frame image.

In some embodiments, this module can be implemented in two ways, one of which is as follows: according to the coordinates of the first motion motion area, the position of the motion area relative to the starting point of the image is obtained, and this relative position can be converted into an address offset relative to the starting point of the image. This method stores the scaling result of the first motion area, that is, at least one detected area of a predetermined scale, in a storage unit. When the module for detecting the moving target needs the scaling result of the entire image of the current frame image, the scaling result of the detected area and a frame of reference image before the current frame image, that is, at least one frame of detected reference image, can be read out at the same time. According to the address offset of the first motion area, the detected reference image and the detected area are summed, and the summed result is the scaling result of the entire image of the current frame image. The second is as follows: if the module for detecting moving targets needs to directly obtain the full image scaling result of the current frame image instead of transferring it through a storage unit, the previously detected reference image stored in the storage unit can be read out during the calculation process of the detected area, and whether the detected area and the previously detected reference image are summed and output to the subsequent processing module or the previously detected reference image in the storage unit is output to the subsequent processing module is determined based on whether the current image pixel is a moving area. If the pixel is in the first moving area, the detected area and the previously detected reference image are summed and output to the subsequent module, otherwise, the previously detected reference image is directly output to the subsequent module for processing. This processing transmits the detected area of the predetermined scale obtained in the moving area of the current frame image to the module for detecting moving targets, eliminating the process of storing the detected area of the predetermined scale in the storage unit and reading it out from the storage unit, thereby greatly reducing the delay of image processing.

Exemplary Electronic Devices

Hereinafter, an electronic device according to an embodiment of the present disclosure will be described with reference to Fig. 17. Fig. 17 illustrates a block diagram of an electronic device according to an embodiment of the present disclosure.

As shown in FIG. 17 , the electronic device 11 includes one or more processors 111 and a memory 112 .

The processor 111 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 11 to perform desired functions.

The memory 112 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 111 may run the program instructions to implement the moving object detection method of each embodiment of the present disclosure described above and/or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

In one example, the electronic device 11 may further include: an input device 113 and an output device 114 , and these components are interconnected via a bus system and/or other forms of connection mechanisms (not shown).

For example, the input device 113 may be a communication network connector for receiving input signals.

In addition, the input device 113 may also include, for example, a keyboard, a mouse, and the like.

The output device 114 can output various information to the outside. The output device 114 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

Of course, for simplicity, FIG17 only shows some of the components related to the present disclosure in the electronic device 11, omitting components such as a bus, an input/output interface, etc. In addition, the electronic device 17 may also include any other appropriate components according to specific application scenarios.

Exemplary computer program products and computer-readable storage media

In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the motion object detection method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the motion object detection method according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and/or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and/or recombined. Such decomposition and/or recombination should be regarded as equivalent solutions of the present disclosure.

The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims (10)

1. A method for detecting a moving object, comprising: Based on a current frame image and a frame image before the current frame image, determining inter-frame difference information between the current frame image and the frame image before the current frame image; Determine a set of multiple frame images based on the current frame image and a predetermined number of historical frame images before the current frame image; Determining a first motion region in the current frame image based on the inter-frame difference information of each frame image in the multi-frame image set; Based on the first motion area, determining at least one detected area of a predetermined size; Determining a reference image based on a comparison result between the current frame image and a predetermined condition; Downsampling the reference image successively to obtain at least one detected reference image of a predetermined scale; According to the address offset of the first motion region, the detected reference image and the detected region are summed to determine the detected image of the predetermined scale corresponding to the current frame image; or, according to whether the pixel of the current frame image is in the motion region, it is determined whether to sum the detected region and the detected reference image; if the current pixel is in the first motion region, the detected region and the detected reference image are summed; if the current pixel is not in the first motion region, the detected reference image is retained to determine the detected image of the predetermined scale corresponding to the current frame image; Based on the detected image of the current frame image, a moving target in the detected area is determined. 2. The method according to claim 1, wherein, based on the current frame image and the previous frame image, determining the inter-frame difference information between the current frame image and the previous frame image comprises: Determine a corresponding pixel difference value of a predetermined color channel in a predetermined color space based on the pixel values corresponding to the current frame image and the previous frame image; The inter-frame difference information is determined based on the pixel difference value and a predetermined weight value. 3. The method according to claim 1, wherein determining the first motion area in the current frame image based on inter-frame difference information of each frame image in the multi-frame image set comprises: Accumulating inter-frame difference information of each frame image in the multi-frame image set to determine accumulated difference information of the multi-frame image set; Determine the moving pixel points in the current frame image based on the comparison result between the accumulated difference information and a predetermined threshold; Based on the moving pixel points in the current frame image, a first moving area of the current frame image is determined. 4. The method according to claim 3, wherein determining the first motion area of the current frame image based on the motion pixel points in the current frame image comprises: Based on the moving pixel point and the predetermined distance, determining a set of pixel points whose distances from the moving pixel point are within the predetermined distance range; Based on the pixel point set, the first motion area is determined. 5. The method according to claim 4, wherein determining the first motion area based on the pixel point set comprises: Determining a second motion area based on the pixel point set; Based on the overlapping pixel points of the two or more second motion regions, the two or more second motion regions are merged into a first motion region. 6. The method according to claim 4, wherein determining the first motion region frame image based on the pixel point set comprises: Determining a second motion area based on the pixel point set; determining a merging result of two adjacent second motion regions based on a distance between the two adjacent second motion regions; Based on the merging result, two adjacent second motion regions whose distances meet a preset condition are merged into a first motion region. 7. The method according to claim 1, wherein the method further comprises: Based on the detected reference image of the at least one predetermined scale, the moving target in the reference image is determined. 8. A moving object detection device, comprising: The differential information acquisition module is used to determine the inter-frame differential information between the current frame image and the previous frame image based on the current frame image and the previous frame image; A multi-frame image set determination module: used to determine a multi-frame image set based on the current frame image and a predetermined number of frame images before the current frame image; A motion region acquisition module: used to determine a first motion region in the current frame image based on the inter-frame difference information of each frame image in the multi-frame image set; A scale conversion module: used to determine at least one detected area of a predetermined scale based on the first motion area; A reference image determination module: used to determine a reference image based on a comparison result between the current frame image and a predetermined condition; A detected reference image determination module is used to successively downsample the reference image to obtain at least one detected reference image of a predetermined scale; The current frame image full image scaling module is used to sum the detected reference image and the detected area according to the address offset of the first motion area to determine the detected image of the predetermined scale corresponding to the current frame image; or, according to whether the pixel of the current frame image is in the motion area, determine whether to sum the detected area and the detected reference image. If the current pixel is in the first motion area, the detected area and the detected reference image are summed. If the current pixel is not in the first motion area, the detected reference image is retained to determine the detected image of the predetermined scale corresponding to the current frame image; The moving target detection module is used to determine the moving target in the detected area based on the detected image of the current frame image. 9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the moving object detection method according to any one of claims 1 to 7. 10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the moving object detection method described in any one of claims 1-7.