CN113850879A - Method for improving compression ratio of static background video based on background modeling technology - Google Patents

Abstract

The present invention provides a method for improving the compression rate of static background video based on background modeling technology. In the present invention, background modeling is performed on the video picture, and continuous differential encoding is performed according to the probability that each pixel is determined to be background, so that pixels with a high probability of background tend to obtain fewer bits in the rate control. encoding, thereby increasing the compression rate. Since the probability function is continuous, the coding strategy can be continuously adjusted accordingly, so as to eliminate the visual difference in image quality and improve user perception on the basis of ensuring the compression performance.

Description

A method for improving the compression ratio of static background video based on background modeling technology

technical field

The present invention relates to video coding, in particular to a method for improving the compression rate of static background video based on background modeling technology.

Background technique

The amount of information obtained by humans through vision accounts for about 70% of the total amount of information, and video information has a series of advantages such as intuition and credibility. However, with the continuous expansion of the application scope of video technology, such as the widespread application of cameras in home scenarios, the amount of data transmitted is also increasing. However, the method of simply relying on expanding the memory capacity and increasing the transmission rate of the communication trunk is expensive and difficult to achieve. Therefore, the video coding and compression technology is an effective solution.

At present, the mainstream video encoders are mainly divided into 3 series: VPx (VP8, VP9), H.26x (H.264, H.265, H.266), AVS (AVS1.0, AVS2.0), but They only propose standards and specifications, and do not distinguish application scenarios. Therefore, in practical applications, in order to further improve the video compression efficiency, it is necessary to improve and deeply optimize the scene characteristics. For application scenarios such as home cameras and road cameras with relatively static backgrounds and small changes, if only mainstream video coding technologies are used, the compression rate has basically reached the bottleneck. Only by adopting more adaptive optimization solutions for video characteristics can the compression be improved. efficiency.

The patent "ROI-based video coding method and system, and video transmission and coding system" (CN202010249206.3A) discloses a ROI-based video coding method, including: acquiring a video frame of a video to be coded, the video frame including a plurality of coding block; dividing the video frame into a ROI area and a non-ROI area; generating a mask of the video frame, the mask can distinguish the ROI area and the non-ROI area; obtaining the color space of the video frame The difference value of the quantization parameter of at least one channel; for each coding block of the video frame, the prediction mode of the at least one channel is selected according to the mask; for each coding block of the video frame, according to the the difference value of the quantization parameter of the at least one channel, according to the coding block including the ROI area and/or the non-ROI area, adjust the quantization parameter of the at least one channel; and according to the prediction mode and the quantization of the at least one channel parameter to encode the video frame.

The patent "ROI-based video coding method and video coding system" (CN202010366816.1A) discloses a ROI-based video coding method, including: S101: acquiring video frames of the video to be encoded; S102: extracting the The ROI area of the video frame; S103: For the ROI area of the video frame, use the first encoding mode for encoding; for the non-ROI area of the video frame, use the second encoding mode to encode, wherein the encoding of the first encoding mode The image quality level is higher than the encoded image quality level of the second encoding method.

However, the above two schemes only divide the picture into two types of areas, namely, ROI (interesting) area and non-ROI area. Performing differentiated coding based on segmentation function for the two areas will make the two areas in the same picture. The quality difference is obvious, thereby reducing the visual experience of the user's viewing.

Therefore, how to improve the compression rate of videos with small background changes, such as home cameras, without affecting the subjective visual quality of the picture, is a problem worthy of further optimization and solution.

SUMMARY OF THE INVENTION

This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

According to an embodiment of the present invention, there is provided a method for improving the compression rate of a static background video based on a background modeling technology, including: using a foreground extraction algorithm to establish a background model, where the background model is used to represent each pixel in an image frame point features; initialize the established background model; update the background model based on the acquired new image frame; perform foreground or background estimation on each pixel in the new image frame; Describe the probability that the pixel point in each coding block in the new image frame is determined as the background, adjust the dynamic rate control factor for each coding block; and based on the dynamic rate control factor for each coding block, for each coding block The encoding block is differentially encoded.

According to another embodiment of the present invention, there is provided a system for improving the compression rate of static background video based on background modeling technology, including a background modeling module and a dynamic encoding module. The background modeling module is configured to: use a foreground extraction algorithm to establish a background model, the background model is used to characterize the characteristics of each pixel in the image frame; initialize the established background model; Image frame, update the background model; perform foreground or background estimation on each pixel in the new image frame. The dynamic coding module is configured to: adjust the dynamic rate control factor for each coding block according to the probability that the pixel point in each coding block in the new image frame is determined as the background; and based on the probability for each coding block The dynamic rate control factor of the block, which performs differential coding for each coded block.

According to yet another embodiment of the present invention, there is provided a computing device for image super-resolution reconstruction, comprising: a processor; a memory, where the memory stores instructions that can be executed when executed by the processor method as described above.

These and other features and advantages will become apparent upon reading the following detailed description with reference to the associated drawings. It is to be understood that both the foregoing general description and the following detailed description are illustrative only and not restrictive of the claimed aspects.

Description of drawings

In order that the manner in which the above-described features of the present invention can be understood in detail, what has been briefly summarized above may be described in more detail with reference to various embodiments, some aspects of which are illustrated in the accompanying drawings. It is to be noted, however, that the appended drawings illustrate only certain typical aspects of the invention and are therefore not to be considered limiting of its scope, for the description may admit to other equally effective aspects.

FIG. 1 shows a block diagram of a video coding system 100 based on background modeling techniques according to one embodiment of the present invention;

FIG. 2 shows a flowchart of a video coding method 200 based on a background modeling technique according to an embodiment of the present invention;

3 illustrates a block diagram of a computing device 300 of a hardware device applicable to aspects of the invention, according to an embodiment of the invention.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings, and the features of the present invention will be further revealed in the following detailed description.

The purpose of the present invention is to combine the background modeling technology to obtain a higher compression rate than mainstream coding (such as H.264, H.265, etc.), and to improve the subjective visual quality of the picture. In the present invention, the background modeling is performed on the video picture, and continuous differential encoding is performed according to the probability that each pixel is judged to be the background, so that the part with a high probability of the background tends to obtain fewer bits in the rate control. encoding, thereby increasing the compression rate. Since the probability function is continuous, the coding strategy can be continuously adjusted accordingly, so as to eliminate the visual difference in image quality and improve user perception on the basis of ensuring the compression performance.

FIG. 1 shows a block diagram of a video coding system 100 for background modeling techniques according to one embodiment of the present invention. As shown in FIG. 1 , the system 100 is divided into modules, and each module communicates and exchanges data in a manner known in the art. In the present invention, each module can be implemented by software or hardware or a combination thereof. The system 100 may include a background modeling module 101 and a dynamic encoding module 102 .

In general, referring to FIG. 1 , the background modeling module 101 is configured to obtain the probability that a pixel belongs to the background based on background subtraction/foreground extraction based on the background modeling method. According to an embodiment of the present invention, the background subtraction/foreground extraction based on the background modeling method mainly includes: modeling the background environment of the video picture, extracting the foreground of the picture through background subtraction, and calculating the value of each pixel in the model The probability of being identified as the background in the middle.

The dynamic encoding module 102 is configured for differential encoding based on dynamic rate control. According to an embodiment of the present invention, the differential coding based on the dynamic rate control mainly includes: based on the probability that the pixels output by the background modeling module 101 are judged to be backgrounds, adopting a dynamic rate control strategy for the pixels, so that the maximum Pixels whose probability is the background tend to obtain fewer bits of coding in the rate control, which can improve the video compression rate while ensuring the visual quality.

The present invention is especially suitable for application scenarios with static backgrounds, such as application scenarios using monitoring equipment such as home cameras, conference video equipment, road cameras and the like. Such monitoring equipment can take pictures and video of the scene, store the acquired image data in the local machine for processing (for example, encoding), and send the processed data to a remote device (for example, a smart home control platform) when needed. , central control platform, other computing devices, etc.) for subsequent processing (eg, playback, editing, etc.). According to one embodiment of the present invention, the background modeling module 101 and the dynamic encoding module 102 may be implemented in the above-mentioned monitoring device or other computing device 300 as described in FIG. 3 .

FIG. 2 shows a flowchart of a video encoding method 200 based on background modeling techniques according to one embodiment of the present invention. The method 200 mainly includes two stages, a background modeling stage 201 and a dynamic coding stage 202 . According to an embodiment of the present invention, the background modeling stage 201 may be implemented by the background modeling module 101 shown in FIG. 1 , and the dynamic encoding stage 202 may be implemented by the dynamic encoding module 102 shown in FIG. 1 .

In the present invention, for the pixel points determined as the background and the pixels determined as the foreground in the image frame, different rate control factors λ are used for subsequent conventional encoding (such as H.264, H.265, etc.), The pixels with a high probability of background tend to obtain fewer bits of coding in the rate control, thereby improving the overall compression rate of the image frame, reducing the storage pressure of the device and the transmission pressure of the communication trunk. Further detailed description is given below with reference to FIG. 2 .

The background modeling stage 201 adopts a method of obtaining the probability that a pixel belongs to the background using a background modeling method. In this stage 201, the background modeling method is used, and after identifying the foreground, the relevant parameters of the background model are used to obtain parameters that can represent the probability that the pixel belongs to the background, so that conventional coding (such as H.264, H.264, H. 265, etc.), the adjustment function for the rate control factor is continuous. As known to those skilled in the art, the rate-distortion optimization strategy is often used in conventional coding to balance the bit rate and image quality, that is, the Lagrangian method is used to obtain the cost function J=D+λ·R, where D represents the image distortion, R represents the code rate, and the Lagrangian factor λ is also called the code rate control factor, which is used to control the proportion between distortion and code rate. The larger the λ, the greater the proportion of the code rate, and the more inclined to sacrifice more when encoding. Video quality for a smaller bitrate.

First, the background modeling phase 201 begins at step 201-1. In step 201-1, a foreground extraction algorithm is used to establish a background model, and the background model is used to characterize the feature of each pixel in the image frame. Based on the general assumption that the background image without the intrusion target can be described by a statistical model, the background modeling is performed for each pixel in the picture. The modeling method can refer to and is not limited to the current mainstream foreground extraction algorithms, such as GMM, ViBe , SACON, PBAS and other algorithms. Of course, other types of foreground extraction algorithms are also within the scope of the present invention.

According to an embodiment of the present invention, step 201-1 is described by using a Gaussian mixture model (GMM) as an example. As shown in equation (1), K mixed Gaussian distribution models with weights w are used to characterize the features of each pixel in the image frame:

Where X is the historical value of any pixel, ω, μ, Σ are the weight, mean and covariance of each Gaussian distribution, respectively.

In step 201-2, the background model established in step 201-1 is initialized. In theory, if there is an image with only background and no foreground, you only need to subtract the background from the new image to get the foreground object. However, in many cases, there is no such background image, so according to different algorithms, the first frame or the first N frame images are usually used to initialize the selected/established background model.

According to an embodiment of the present invention, continuing the above example of using the GMM model, in step 201-2, the background model is initialized by using the first frame of image, each Gaussian distribution uses the pixel value of the first frame as the expectation, and the weights are equal to each other. is 1/K, and the standard deviation is large. Those skilled in the art can understand that the standard deviation represents the degree of dispersion of the data. There is only one value during initialization, and the degree of dispersion cannot be calculated. Therefore, it is assumed that it is very large, and then the corresponding data is performed on it when there is new data in the next frame. renew.

In step 201-3, the background model is updated based on the acquired new image frame. According to an embodiment of the present invention, when a new image frame is acquired, the model parameters are updated according to the update strategy of each model algorithm and the pixel values in the new image frame. Those skilled in the art can understand that parameters used in different models are different (such as thresholds used in matching conditions, etc.), and which parameters are updated are not specifically limited here.

According to an embodiment of the present invention, the above-mentioned example of using the GMM model is continued. In step 201-3, when a new image frame is acquired, the model parameters are updated according to the following update strategy:

ω _i,t =(1-α)ω _i,t-1 +αM _i,t (2)

μ _i,t =(1-ρ)μ _i,t-1 +ρX _t (3)

in,

ρ=αη(X _t |μ _i,t ,σ _i,t ) (6)

And α is the learning rate.

In step 201-4, perform foreground/background estimation on each pixel in the new image frame acquired in step 201-3. According to an embodiment of the present invention, the current pixel value of each pixel in the new image frame is matched with its corresponding background model (for example, each pixel's own GMM background model), and the pixels that fail to match are For the pixels that are successfully matched, the probability (p) that the pixel belongs to the background is calculated according to the updated algorithm model parameters in step 201-3. According to one embodiment of the present invention, a matching threshold may be used in the above-mentioned matching.

排序，取前B个高斯分布作为当前的背景模型：According to an embodiment of the present invention, the above-mentioned example of using the GMM model is continued. In step 201-4, K Gaussian distributions are

Sort, take the first B Gaussian distributions as the current background model:

Where T is the minimum proportion of the background in the picture. Equation (7) is to obtain the parameter B. As mentioned above, for the GMM model updated in step 201-3, the model composed of the first B Gaussian distributions is taken as the current background model. Match the current pixel value of each pixel with the current background model. If the matching fails, the pixel is judged as the foreground; otherwise, the B values of a are normalized to obtain a', and the pixel that can represent the pixel is calculated. The parameter p of the probability size of the point belonging to the background:

p=a'(X _t -μ) (8)

After calculating the probability that the pixel is determined to be the background, the dynamic encoding stage 202 is entered. The dynamic encoding stage 202 adopts a method of adjusting the encoding strategy using the probability that the pixel points belong to the background. According to one embodiment of the present invention, the dynamic coding stage 202 is performed on a coding block-by-block basis. The following steps 202-1 to 202-2 are repeated for each coding block in the current image frame until all coding blocks are coded. Those skilled in the art can fully understand that the division of the coding blocks of the image frame and the coding sequence of the coding blocks can be configured based on the specific coding mode (such as H.264, H.265, etc.) adopted. The division manner and/or coding sequence of .

In this stage 202, based on the user-acceptable background picture quality, explore the mapping relationship between the trade-off value between the attenuation and the bit rate in the rate-distortion optimization strategy and the probability that the pixel points are background points, and obtain an adjustment strategy that is more in line with the human eye's attention mechanism. .

In step 202-1, the dynamic rate control factor is adjusted for the current coding block based on the probability that the pixels in the current coding block are determined as the background. According to an embodiment of the present invention, if all the pixels in the current coding block are background points, the average value p_avg of the p values of all the pixels in the coding block is taken, and the probability that the pixels belong to the background is found to be acceptable to the user. The mapping relationship between the background picture quality of , obtains the rate control factor λ=f(p_avg) in the rate-distortion optimization strategy, where f( ) represents the trade-off between attenuation and bit rate in the rate-distortion optimization strategy and the pixel point The correspondence between the probabilities belonging to the background is a continuous function rather than a discrete piecewise function in the prior art. For example, in a conventional x264 encoder, λ has a corresponding relationship with the quantization parameter QP value. The QP value is one of the inputs of the encoder and is an integer, and λ is obtained by looking up the QP value in a table. Therefore, the correspondence between λ and QP values in conventional coding is not a continuous function. In addition, in the present invention, based on the trade-off between the attenuation and the bit rate in the rate-distortion optimization strategy and the correspondence between the probability that the pixel belongs to the background, the greater the probability of the pixel belonging to the background, the larger the λ, and the more inclined it is to encode. to sacrifice more video quality for a smaller bitrate.

In step 202-2, the current coding block is differentially coded using the dynamic rate control factor. According to an embodiment of the present invention, if the pixels in the current coding block contain or all are foreground, the normal coding is performed according to the original conventional coding method (such as H.264, H.265, etc.); If all the pixels are background points, the coding is performed according to the λ obtained in step 202-1. Since the pixels with a larger p value can be coded with a larger λ, the background part of the video picture will be further compressed.

In step 202-3, it is judged whether there are coding blocks that need to be coded in the current image frame, if yes, go back to step 202-1, if not, go to step 203.

In step 203, it is judged whether there is a new image frame, if yes, return to step 201-3 to continue to acquire the next new image frame, if not, end the process. According to an embodiment of the present invention, it may be set to acquire image frames within a predetermined time period for encoding. According to another embodiment of the present invention, if the background changes significantly (such as the difference between the background of the new image frame and the background of the previous image frame reaches a certain threshold), the process may end.

Compared with the prior art, the present invention has the following advantages:

(1) Improve the video compression rate. On the premise of ensuring the quality of the foreground image, by further compressing the images in the video that are more likely to belong to the background, more bit rates are saved, thereby improving the compression rate of the video.

(2) The compression efficiency can be dynamically adjusted. The dynamic adjustment of compression efficiency can be achieved by modifying the parameters in the formula for calculating the rate control factor according to the requirements of specific scenarios.

(3) Improve the subjective visual quality of the picture. Based on the human eye attention mechanism, a more reasonable coding strategy is adopted to encode and compress the part identified as background in the picture according to the probability that it is indeed a background point, so that the adjustment formula for the rate control factor is a continuous function rather than a discrete one. A piecewise function that avoids visual quality layering of foreground and background regions.

(4) High practicability and easy implementation in a static background. For scenes where the background of the picture is mostly static or with small changes, such as home cameras, conference recordings, road cameras, etc., it has outstanding compression performance improvement and visual effect improvement.

FIG. 3 shows a block diagram 300 of an exemplary computing device, which is one example of a hardware device applicable to aspects of the invention, according to one embodiment of the invention.

3, a computing device 300 will now be described, which is one example of a hardware device applicable to aspects of the present invention. Computing device 300 can be any machine that can be configured to perform processing and/or computation, and can be, but is not limited to, workstations, servers, desktops, laptops, tablets, personal digital processors, smartphones , on-board computers, home cameras, conference recording equipment, road cameras, or any combination thereof. The aforementioned various methods/apparatuses/servers/client devices may be implemented in whole or at least in part by the computing device 300 or similar device or system.

Computing device 300 may include components that may connect or communicate with bus 302 via one or more interfaces. For example, computing device 300 may include a bus 302 , one or more processors 304 , one or more input devices 306 , and one or more output devices 308 . The one or more processors 304 may be any type of processor and may include, but are not limited to, one or more general-purpose processors and/or one or more special-purpose processors (eg, specialized processing chips). Input device 306 may be any type of device capable of inputting information to a computing device and may include, but is not limited to, a mouse, keyboard, touch screen, microphone, and/or remote controller. Output device 308 may be any type of device capable of presenting information and may include, but is not limited to, displays, speakers, video/audio output terminals, vibrators, and/or printers. Computing device 300 may also include or be connected to a non-transitory storage device 310, which may be any storage device that is non-transitory and capable of data storage, and that is non-transitory. Transient storage devices may include, but are not limited to, magnetic disk drives, optical storage devices, solid state memory, floppy disks, floppy disks, hard disks, magnetic tape or any other magnetic medium, optical disks or any other optical medium, ROM (read only memory), RAM (random memory) fetch memory), cache memory, and/or any memory chip or tape cartridge, and/or any other medium from which a computer may read data, instructions, and/or code. Non-transitory storage device 310 may be detached from the interface. The non-transitory storage device 310 may have data/instructions/code for implementing the methods and steps described above. Computing device 300 may also include communication device 312 . Communication device 312 may be any type of device or system capable of communicating with internal devices and/or with a network and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication devices, and/or chipsets, such as Bluetooth devices , IEEE 1302.11 devices, WiFi devices, WiMax devices, cellular communication devices and/or similar devices.

The bus 302 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Microchannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

Computing device 300 may also include working memory 314, which may be any type of working memory capable of storing instructions and/or data that facilitates the operation of processor 304 and may include, but is not limited to, random access memory and/or read-only storage device.

Software components may be located in working memory 314, including, but not limited to, operating system 316, one or more application programs 318, drivers, and/or other data and code. Instructions for implementing the above-described methods and steps of the present invention may be contained in the one or more application programs 318, and the present invention may be implemented by the processor 304 reading and executing the instructions of the one or more application programs 318 The method 200 described above.

It should also be recognized that variations may be made according to specific needs. For example, custom hardware may also be used, and/or certain components may be implemented in hardware, software, firmware, middleware, microcode, hardware description voice, or any combination thereof. Additionally, connections to other computing devices, such as network input/output devices, etc., may be employed. For example, programmable logic circuits (eg, programmable logic circuits including field programmable gate arrays (FPGA) and/or programmable logic arrays (PLA)) may be programmed with assembly language or hardware programming languages (eg, VERILOG, VHDL, C++). ) utilize logic and algorithms in accordance with the present invention to implement some or all of the disclosed methods and apparatus.

Although aspects of the present invention have so far been described with reference to the accompanying drawings, the above-described methods, systems, and apparatus are merely examples, and the scope of the present invention is not limited to these aspects, but only by the appended claims and their equivalents . Various components may be omitted or may be replaced by equivalent components. Additionally, the steps may also be performed in an order different from that described in this disclosure. Furthermore, the various components can be combined in various ways. It is also important that, as technology develops, many of the components described may be replaced by equivalent components that appear later.

Claims (10)

1. A method for improving the compression ratio of static background video based on background modeling technology, comprising: A foreground extraction algorithm is used to establish a background model, and the background model is used to characterize the feature of each pixel in the image frame; Initialize the established background model; updating the background model based on the acquired new image frame; performing foreground or background estimation on each pixel in the new image frame; Adjusting the dynamic rate control factor for each coding block according to the probability that the pixel point in each coding block in the new image frame is determined as the background; and Each coding block is differentially encoded based on a dynamic rate control factor for each coding block. 2. The method of claim 1, wherein the foreground extraction algorithm comprises one of GMM, ViBe, SACON, or PBAS. 3. The method of claim 1, wherein initializing the established background model further comprises: The background model is initialized by using the first frame image or the first N frame images. 4. The method of claim 1, wherein performing foreground or background estimation on each pixel in the new image frame further comprises: matching the current pixel value of each pixel in the new image frame to its corresponding updated background model; If the match fails, the pixel is determined to be the foreground; If the matching is successful, the probability p that the pixel belongs to the background is calculated according to the updated background model. 5 . The method according to claim 4 , wherein, according to the probability that a pixel point in each coding block in the new image frame is determined as the background, adjusting the dynamic rate control factor for each coding block further. 6 . include: If all the pixels in the current coding block are all determined as the background, take the average value p_avg of the p values of all the pixels in the current coding block, and obtain the rate control factor λ=f(p_avg in the rate-distortion optimization strategy ). 6. The method of claim 5, wherein the differentially encoding each coding block based on a dynamic rate control factor for each coding block further comprises: If all the pixels in the current coding block are all determined to be the background, coding is performed using the rate control factor λ. 7. A system for improving the compression ratio of static background video based on background modeling technology, comprising: A background modeling module configured to: A foreground extraction algorithm is used to establish a background model, and the background model is used to characterize the feature of each pixel in the image frame; Initialize the established background model; updating the background model based on the acquired new image frame; performing a foreground or background estimate for each pixel in the new image frame; and A dynamic encoding module, the dynamic encoding module is configured to: Adjusting the dynamic rate control factor for each coding block according to the probability that the pixel point in each coding block in the new image frame is determined as the background; and Each coding block is differentially encoded based on a dynamic rate control factor for each coding block. 8. The system of claim 7, wherein performing foreground or background estimation on each pixel in the new image frame further comprises: matching the current pixel value of each pixel in the new image frame to its corresponding updated background model; If the match fails, the pixel is determined to be the foreground; If the matching is successful, the probability p that the pixel belongs to the background is calculated according to the updated background model. 9. The method of claim 8, wherein Wherein, according to the probability that the pixels in each coding block in the new image frame are determined as the background, adjusting the dynamic rate control factor for each coding block further includes: if all the pixels in the current coding block are all determined as In the background, take the average value p_avg of the p values of all pixels in the current coding block, and obtain the rate control factor λ=f(p_avg) in the rate-distortion optimization strategy; Wherein, based on the dynamic rate control factor for each coding block, performing differential coding on each coding block further includes: if all the pixels in the current coding block are determined as backgrounds, use the rate control factor λ to perform the differential coding. coding. 10. A computing device for image super-resolution reconstruction, comprising: processor; a memory storing instructions which, when executed by the processor, are capable of performing the method of claims 1-6.

Images