CN117201796A - Video encoding method, apparatus, computing device and storage medium - Google Patents

Abstract

The application provides a video encoding method, a video encoding apparatus, a computing device and a computer-readable storage medium. The method comprises the following steps: acquiring generation information of a video; dividing a first frame of the video according to the generated information to obtain a plurality of coding blocks, wherein the plurality of coding blocks of the first frame comprise a current block, and target blocks are included in the plurality of coding blocks of the first frame or in a second frame adjacent to the first frame of the video; predicting predicted pixel information of the target block according to the current pixel information of the current block; the video is encoded based on the predicted pixel information. According to the application, the division of the coding blocks can be rapidly realized with less calculation amount, thereby accelerating the video coding process.

Description

Video encoding method, device, computing device and storage medium

Technical field

The present application relates to the field of image processing technology, and in particular to a video encoding method, a video encoding device, a computing device and a computer-readable storage medium.

Background technique

In the field of video coding and compression technology, there are intra-frame prediction and inter-frame prediction technologies. Both technologies require dividing the video frame into multiple coding blocks for better predictive coding. However, the coding block division process in the existing technology often requires multiple attempts to obtain the optimal division method, which requires a large amount of calculations, resulting in increased calculation costs and time consumption, making it increasingly difficult to adapt to the increasing amount of current video data. The current situation is that the transmission rate requirements are getting higher and higher. Therefore, there is an urgent need in this field for a technology that can quickly divide video frames into coding blocks with less calculation, thereby accelerating the video coding process.

Contents of the invention

To this end, the present application is committed to providing a video encoding method, a video encoding device, a computing device, and a computer-readable storage medium, which can quickly implement encoding block division of video frames with less calculation, thereby accelerating the video encoding process.

In one aspect, the present application provides a video encoding method, which includes: obtaining generation information of the video; dividing the first frame of the video according to the generation information to obtain multiple coding blocks. The multiple coding blocks of the first frame include the current block, the target block is included in multiple coding blocks of the first frame or in the second frame adjacent to the first frame of the video; according to the current pixel information of the current block, the predicted pixel information of the target block is predicted; according to the predicted pixel information, the video Encode.

According to this aspect, the video generation information is used to determine the encoding block division method of the video frame, which is conducive to directly and quickly making optimal divisions based on the nature of the video content (material, foreground and background, etc.) and avoiding calculations caused by repeated attempts. The increased volume greatly improves the coding efficiency of generative videos, allowing generative videos to have clearer images and greater transmission rates.

In a special embodiment of the present application, the generated information includes position information and material information of the foreground object in the first frame.

According to this embodiment, the position information and material information of the foreground object often cannot be obtained directly from the video picture itself, but the video generation tool can often provide the distinction between the foreground object and the background object, thereby determining the position information of the foreground object. In addition, the generation tool can determine the material information of the foreground when rendering the surface color and material of the foreground object. Based on these two kinds of information, it can help to determine different division strategies of coding blocks, making the subsequent prediction and coding process more accurate and efficient.

In a special embodiment of the present application, the first frame of the video is divided according to the generated information to obtain multiple coding blocks, including: determining the foreground area and background area in the first frame according to the position information of the foreground object; The foreground area is divided according to the first mode, and the background area is divided according to the second mode, wherein the coding blocks in the first mode are smaller than the coding blocks in the second mode.

According to this embodiment, using different division methods for the foreground area and the background area is beneficial to utilizing the difference in the change amplitude of the display content between the foreground area and the background area to adopt different division methods, thereby improving prediction accuracy and compression efficiency.

In a special embodiment of the present application, dividing the first frame of the video according to the generated information to obtain multiple coding blocks also includes: determining the edge area of the foreground object in the first frame according to the position information of the foreground object; The edge area is divided according to the first mode.

According to this embodiment, a more detailed coding block division method is adopted for the edge area, and the change pattern that is more likely to occur in the edge area can be used to more clearly retain the display details of the edge of the foreground object and maintain the image content of the edge area as much as possible. accuracy and fidelity.

In a special embodiment of the present application, the first frame of the video is divided according to the generated information to obtain multiple coding blocks, including: determining the foreground area and background area in the first frame according to the position information of the foreground object; Based on the material information of the foreground object, it is judged whether the foreground object is a rigid object or a flexible object; if the foreground object is a rigid object, the foreground area is divided according to the third mode; if the foreground object is a flexible object, the foreground area is divided according to the fourth mode. The coding blocks in the third mode are larger than the coding blocks in the fourth mode.

According to this embodiment, different division methods are adopted for rigid objects and flexible objects, which is conducive to utilizing the different display rules of the two objects in the video picture to compress the video data more efficiently, while maintaining the display details of the flexible objects as much as possible, and improving The clarity and restoration of the picture.

In a special embodiment of the present application, the video includes a virtual scene video generated by a video generation tool, and obtaining the generation information of the video includes: obtaining the generation information from the video generation tool.

According to this embodiment, a large number of virtual scene videos exist in today's video products, including games, movies, and TV series that contain a large number of virtual scene videos produced through manual production and special effects technology. The encoding acceleration of virtual scene videos is beneficial to greatly promote Dissemination and widespread use of video content. In addition, obtaining generation information from video generation tools that produce virtual scene videos is conducive to quickly obtaining information helpful for video encoding directly from the source, improving the accuracy and timeliness of information acquisition.

In a special embodiment of the present application, the video generation tool includes a renderer or an AI tool.

According to this embodiment, renderers and AI tools are common video generation tools that can provide a large amount of hidden information about video content, which is beneficial to quickly determine the division method by referring to this information when dividing encoding blocks.

In a special embodiment of the present application, the coding block includes one or more of a macroblock, a sub-block, a coding tree unit CTU, a coding unit CU, a prediction unit PU, and a transformation unit TU.

According to this embodiment, there are various specific forms of coding blocks in the field, and the coding block division method provided by this application can be adopted, which is beneficial to realizing the acceleration of coding block division at each coding level, thereby greatly improving the overall Encoding compression rate.

On the other hand, the present application provides a video encoding device, including: an acquisition module, used to obtain the generation information of the video; a dividing module, used to divide the first frame of the video according to the generation information to obtain multiple encoding blocks. , the multiple coding blocks of the first frame include the current block, and the multiple coding blocks of the first frame or the second frame adjacent to the first frame of the video include the target block; the prediction module is used to predict the current block according to the current pixel information of the current block. , predict the predicted pixel information of the target block; the encoding module is used to encode the video based on the predicted pixel information.

In another aspect, the present application provides a computing device, characterized in that the computing device includes a processor and a memory, and the processor is configured to execute a computer program stored in the memory to implement the above video encoding method.

On the other hand, the present application provides a computer-readable storage medium, which is characterized in that the computer-readable storage medium stores a computer program, and the computer program is used to execute the above video encoding method.

In another aspect, the present application provides a computer program product, including program code, which causes the computer to implement the above video encoding method when the computer runs the computer program product.

Any video encoding device, computing device, computer-readable storage medium or computer program product provided above is used to execute the video encoding method provided above. Therefore, the beneficial effects it can achieve can be referred to the above provided The beneficial effects of the corresponding solution in the corresponding method will not be described again here.

Description of the drawings

Below, specific embodiments of the present application are described in detail with reference to the accompanying drawings, wherein:

Figure 1 shows a schematic diagram of an application scenario of a video encoding method according to an embodiment of the present application;

Figure 2 shows a schematic diagram of inter-frame prediction according to the video encoding method according to the embodiment of Figure 1;

Figure 3 shows a schematic diagram of the coding block division method according to the video coding method according to the embodiment of Figure 1;

Figure 4 shows a schematic flowchart of a video encoding method according to another embodiment of the present application;

Figure 5 shows a schematic diagram of the dividing steps of the video encoding method according to the embodiment of Figure 4;

Figure 6 shows another schematic diagram of the dividing steps of the video encoding method according to the embodiment of Figure 4;

Figure 7 shows another schematic diagram of the dividing steps of the video encoding method according to the embodiment of Figure 4;

Figure 8 shows another schematic diagram of the dividing steps of the video encoding method according to the embodiment of Figure 4;

Figure 9 shows another schematic diagram of the dividing steps of the video encoding method according to the embodiment of Figure 4;

Figure 10 shows another schematic diagram of the dividing steps of the video encoding method according to the embodiment of Figure 4;

Figure 11 shows a schematic structural diagram of a video encoding device according to an embodiment of the present application;

Figure 12 shows a schematic structural diagram of a computing device according to an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to understand the concepts and ideas of the present application more clearly, the present application is described in detail below with reference to specific embodiments. It should be understood that the embodiments given herein are only a portion of all possible embodiments of the present application. After reading the description of this application, those skilled in the art are capable of making improvements, transformations, or substitutions to part or all of the following embodiments, and these improvements, transformations, or substitutions are also included in the scope of protection claimed by this application.

In this article, the terms "a", "an" and other similar words are not intended to indicate that there is only one of the things, but rather that the description is directed to only one of the things, which may have one or more . In this article, the terms "comprising", "includes" and other similar words are intended to indicate logical interrelationships and cannot be regarded as indicating spatial structural relationships. For example, "A includes B" is intended to mean that B belongs to A logically, but does not mean that B is located inside A spatially. In addition, the terms "includes," "includes," and other similar words are to be considered open-ended rather than closed-ended. For example, "A includes B" is intended to mean that B belongs to A, but B does not necessarily constitute all of A. A may also include other elements such as C, D, and E.

In this document, the terms "first", "second" and other similar words are not intended to imply any order, quantity or importance, but are merely used to distinguish different elements. In this article, the terms "embodiment", "this embodiment", "an embodiment" and "an embodiment" do not mean that the relevant description is only applicable to a specific embodiment, but mean that the description may also be applicable to in one or more embodiments. Those skilled in the art should understand that any description of a certain embodiment herein can be replaced, combined, or otherwise combined with the relevant description of one or more embodiments. New embodiments generated by combining or combining in other ways can be easily thought of by those skilled in the art, and belong to the protection scope of this application.

In various embodiments of the present application, video may refer to various technologies that capture, record, process, store, transmit and reproduce a series of still images in the form of electrical signals. For example, video uses the principle of persistence of human vision to give the human eye a sense of movement by playing a series of pictures (also called frames). In some embodiments, the video may include generative video. Generative video can refer to video that is different from traditional video. Traditional video is produced by video collection equipment, such as cameras, mobile phones with camera functions, video recorders, etc., which capture scenes in the real world. Compared with traditional video, generative video is not produced by video collection. Device generation is usually generated by computer software, such as renderers, AI, etc., and generates videos composed of corresponding continuous images according to user needs. The most common videos are games, movies, etc.

In various embodiments of the present application, video encoding may refer to a method of converting an original video format file into another video format file through compression technology. Video is a continuous image sequence, consisting of continuous frames. Due to the extremely high similarity between continuous frames, in order to facilitate storage and transmission, the original video needs to be encoded and compressed to remove redundancy in the spatial and temporal dimensions. Video image data has a strong correlation, which means there is a lot of redundant information. The redundant information can be divided into spatial domain redundant information and time domain redundant information. Compression technology is to remove the redundant information in the data. Compression technology includes intra-frame image data compression technology, inter-frame image data compression technology and entropy coding compression technology. In this article, "video compression" and "video encoding" are interchangeable in some scenarios.

In this field, the video images of games and most movies are generated by computers through renderers, especially in non-local rendering scenarios, such as cloud rendering. After the renderer generates the video, it needs to be compressed by a video encoder and then pushed to The terminal performs decoding and display. Non-local rendering scenes generally follow the following workflow: 1) The renderer generates video images according to the set program; 2) The video images are encoded (compressed) by the video encoder, which greatly reduces network transmission bandwidth; 3) The compressed and encoded video stream is passed through The network transmits it to the end user device; 4) The end user device decodes the compressed and encoded video stream through the decoder, and displays the video image generated by the renderer on the display after decoding.

The amount of video data before compression is very large. Taking 1080P in many current application scenarios as an example, each pixel includes brightness and chroma, represented by 12 bits in total, and the frame rate is 60 (that is, 60 images are output per second). In this way, the network transmission bandwidth requirement is approximately 1.39Gbps. . 4K video bandwidth requirements will be greater. Such a large demand for Internet bandwidth is difficult to meet. A key technology for video streaming applications is video coding, also known as video compression. The video encoder that completes the video compression function is the same as text compression. The purpose is to remove redundant data in the video data and reduce the coded and compressed data. quantity. Video compression is sometimes a type of lossy compression. Although video lossy compression will introduce certain video picture distortions, these distortions are usually difficult to detect by the human eye.

In videos, there is usually a strong spatial correlation between pixels in adjacent ranges within a single image (also called a single frame). There is a strong temporal correlation between consecutive image data. Video compression coding technology mainly uses intra-frame prediction and inter-frame prediction methods, using the coded pixels of the image to predict spatially and temporally adjacent pixels, thereby effectively compressing the temporal and spatial redundancy of video image pixels.

As the video picture increases, such as 4K and 8K pictures, in order to perform better compression and encoding, video encoders usually divide the picture into many small blocks for compression and encoding, such as 16x16 pixels of AVC (advanced video coding, advanced video coding) size macroblock. In order to more efficiently compress the details of different textures and motion changes in video scenes, HEVC (high efficiency video coding, high efficiency video coding) defines a coding tree unit (coding tree unit, CTU) of 64x64 pixels for the image and codes of different sizes. Unit (coding unit, CU). The VVC (versatile video coding, flexible video coding) standard CU division is more flexible and rich.

During the video encoding process, it is necessary to try different CU division depths. As the video picture increases, the video encoding delay requirements increase, and predictions between CUs are interdependent, putting increasing pressure on the video encoder.

A technology in this field. When compressing in units of CTU, for each CU division depth, that is, the four division methods of 64x64, 32x32, 16x16, and 8x8, motion evaluation is performed to obtain the motion vector, and the motion vector is obtained based on the motion vector. Inter-frame prediction, discrete cosine transform (DCT), quantization, bit number evaluation entropy coding and distortion calculation, and then compared layer by layer according to the rate-distortion optimized (RDO) algorithm. For a 16x16 pixel area, there are four division methods: 4 8x8 CUs, 2 4x8 CUs, 2 8x4 CUs, or 1 16x16 CU. Motion evaluation is performed on these four division methods to obtain motion vectors, and then prediction is made based on the motion vectors. Finally, the division with the smallest rate distortion is selected. Evaluate various division methods in a similar manner as above.

This technology can be used for traditional videos and generative videos. The problem is that it needs to try all CU division methods, then perform motion evaluation, prediction calculation, DCT, quantization, entropy coding process, and finally obtain the final division method through rate-distortion calculation. , the amount of calculation is huge, and only one division method is finally selected, resulting in a lot of calculation waste. Especially for 4K and 8K videos, the latest VCC encoding standard is difficult to meet the needs of real-time video encoding and decoding.

Another technology in this field is used for generative video coding. The rendering engine will provide the motion vector of each pixel to the video encoder. The encoder calculates the similarity of the motion vector of each pixel in different division methods, such as calculating 16x16 The variance of the motion vectors of 256 pixels in the pixel block, the variance of the motion vectors of 128 pixels in the 16x8 pixel block, the variance of the motion vector of 64 pixels in the 8x8 pixel block, and finally the block size of the block with the smallest variance is selected as the final division method.

This technology can be used for generative videos. The problem is that although motion evaluation and RDO calculation processes are not required, different CU partition attempts are still required, the amount of calculation is still huge, and there is computational waste.

Some embodiments of the present application are dedicated to solving the problem of: for generative videos, how to use the information provided by the renderer to reduce the attempt to divide the size of the CU (or other blocks), thereby reducing the waste of encoder computing power.

In some embodiments of this application, for generative videos, while minimizing the loss of image quality caused by compression, the waste of computing power of the video encoder is reduced, and the encoding delay of the encoder is reduced to meet the requirements of high image quality and low delay. Coding requirements. Specifically, current video encoders need to try different CU division methods, and finally choose one method as the final division method. Different CU divisions require a lot of calculations. Since only one of the division methods is selected for the final video stream, The encoder has a lot of invalid calculations. In fact, in the background area that occupies the main range of the video image, the motion vectors of the two frames before and after are usually the same, and the motion vectors of the foreground objects may also be the same. In theory, there is no need to try all division methods. Especially for generative video, image generation tools such as renderers know which pixels belong to the background area and which belong to the foreground area. Therefore, the embodiment of the present application uses the information of the renderer to assist the video encoder in encoding, thereby minimizing invalid calculations.

Some embodiments of this application are applied to generative video encoders, using video generators, such as renderers, AI tools, etc., to generate videos and video-related information, including but not limited to layer information and foreground moving object pixels in the picture. Coordinate information, material of foreground objects such as rigid body, flexible object information, etc. The video encoder performs different CU division processing on the background area and foreground moving objects based on the layer and moving object coordinate information, that is, quickly implements inter-frame mode encoding, thereby saving the video encoder computing power and reducing the video encoder encoding delay. .

In some embodiments of this application, the CU division of inter-frame prediction is divided into fixed CU sizes according to the background area and foreground moving objects. Compared with trying four different CU division methods to select a final division method through the RDO algorithm, the embodiments of this application It can reduce the waste of computing power by 3 times. At the same time, the background area is often not an important area of the video and has basically no impact on the end user experience. In addition, some embodiments of this application divide foreground moving objects into different CUs according to rigid and flexible materials. The motion vectors of rigid objects are usually consistent, and the motion vectors of various parts of flexible objects are usually inconsistent. Rigid objects can be divided into large CUs of 64x64 and 32x32. In this way, flexible objects are divided into CUs of 8x8 size, which saves computing power to the greatest extent while ensuring video quality.

Figure 1 shows a schematic diagram of an application scenario of a video encoding method according to an embodiment of the present application.

This embodiment includes a video generator and a video encoder. The video encoder quickly compresses the generative video based on the video and related information generated by the video generator. Specifically, it involves the video encoder's rapid division of intra-frame or inter-frame coding blocks. . In this embodiment, the video generation tool generates a video and related generation information, and both the video and the generation information are input to the video encoder. Video encoders can be set and adjusted based on the encoder parameters. The video encoder receives the input of the video generation tool, compresses the received video according to the content or parameters of the generated information, and finally generates a compressed code stream for data transmission or further processing. In this embodiment, the video used for video encoding is a video generated by a video generation tool, rather than a natural scene video captured directly by a shooting device. The encoding process of the video encoder requires not only the original video data, but also Generated information that can represent information about video content is needed. The information about video content provided by the generated information plays an important role in accelerating the video encoding process. Video encoders can compress and encode videos through intra-frame prediction and inter-frame prediction. Both intra-frame and inter-frame prediction require dividing video frames into coding blocks.

As an example, FIG. 2 shows a schematic diagram of an inter prediction process in the video encoding method according to the embodiment of FIG. 1 . As shown in Figure 2, it shows two consecutive frames in the video. The dotted frame in the figure is only for illustration, not the content of the picture. In these two consecutive frames, only the position of object 1 has changed, then the pixels of object 1 area (dashed square area) in the previous frame can be used to predict the pixels of object 1 area (dashed square area) in the second frame, before and after The second frame can be predictively encoded by subtracting the residual of the corresponding pixels in the two areas and adding the motion vector information of object 1. When performing inter-frame prediction in the next frame, it is necessary to search in the previous frame to find the matching area. This search process is called motion estimation (ME). The position change of an object or pixel between the two frames before and after is Motion vector (moving vector, MV).

As an example, FIG. 3 shows a schematic diagram of a coding block dividing process in the video encoding method according to the embodiment of FIG. 1 . As shown in Figure 3, in this embodiment, a CTU of 64x64 pixels is defined for the image. Each image in the video is divided into several non-overlapping CTUs. Inside the CTU, a cyclic hierarchical structure based on a quadtree is adopted. For example, the top layer is a 64x64 pixel unit, which can be divided down one level into four CUs of 32x32 pixel size. Each 32x32 pixel size can be further divided into It is divided into 16x16 coding blocks until the minimum 8x8 coding unit.

In this embodiment, both intra-frame prediction and inter-frame prediction can be performed sequentially from left to right and top to bottom in CU units. Subtract the prediction value from the original pixel value of the video in the CU range to obtain the residual. After DCT, the transformation result is quantized. The quantized value and related syntax elements, such as CU division depth, motion vector corresponding to the pixel block, etc., are entropy encoded, that is, Compressed video streams are available. At the same time, the quantization result is dequantized and inverse discrete cosine transform can be used to obtain the decoding residual value, and then added to the predicted value to obtain the pixel reconstruction value. The pixel reconstruction value of the previous CU block, CU division depth and other information need to be used as input for the next CU prediction, so in some cases all CUs cannot be compressed and encoded in parallel at the same time.

Figure 4 shows a schematic flowchart of a video encoding method according to an embodiment of the present application.

According to this embodiment, the video encoding method includes steps S410 to S440, and each step is described in detail below.

S410. Obtain video generation information.

In this embodiment, the generation information may refer to information related to subsequent encoding operations generated during the generation process of the generative video. In some embodiments, the generated information of the video is different from the information of the video image itself (such as the pixel value information of the image or the image texture, color and other information that can be directly reflected by the pixel value), and the generated information is a tool for generating the video. Information provided that cannot or is difficult to determine based only on the image content or pixel value information of the video.

As an example, the generated information includes position information and material information of the foreground object in the first frame. For example, the generated information includes layer information and coordinate information of foreground moving object pixels in the picture.

In this example, the position information of the foreground object may refer to the position or coordinate information of the foreground object in the video screen, and the material information of the foreground object may refer to the material given to the fictitious foreground object by the video producer when making the video. For example, metal, wood, human skin, etc.

As an example, the video includes a virtual scene video generated by a video generation tool. At this time, in order to obtain the generation information of the video, the generation information can be obtained from the video generation tool.

In this example, the video generation tool may refer to a tool used to generate videos based on the user's manual video production methods such as modeling, rendering, and color correction. Such video generation tools are different from those that shoot actual objective objects. Photography, video or video recording tools. Virtual scene videos can refer to videos in which the objects displayed do not really exist, but are artificially created through image editing methods such as sketching, modeling and rendering. Of course, the virtual scene video generated by the video generation tool can contain actual objects. For example, a video mixed with virtual objects and real objects generated in AR (augmented reality) glasses also belongs to the virtual scene video defined in this application. .

In this example, in order to obtain the video generation information, the relevant information or data can be directly exported from the video generation tool for subsequent video encoding. For example, some video generation tools can provide three-dimensional coordinate information of objects, from which it can be concluded which objects are foreground objects and which objects are background objects. For another example, some video generation tools can provide material information of an object, so that its hardness or rigidity can be judged based on the material information. Different encoding block division methods are used for encoding materials with different rigidities.

As examples, video generation tools include renderers or AI (artificial intelligence, artificial intelligence) tools.

In this example, the renderer can refer to a software or hardware tool that converts a 3D (3-dimension, three-dimensional) model into a 2D (2-dimension, two-dimensional) image. It can convert the geometric shapes, Information such as textures, lighting, and materials are converted into 2D images for display on the screen or printing. Renderers are often used in movies, games, architectural design, and other fields to create realistic visual effects. For example, a renderer can refer to a software used to process and display three-dimensional graphics. It can draw a three-dimensional scene onto a two-dimensional display device, such as a monitor, mobile phone, or TV. Its main function is to convert a digital model into An image for the device to display.

In this example, the AI tool can refer to a software tool that uses artificial intelligence technology to automatically generate videos. The tool can be based on the user's drawing operations or based on the picture materials, video materials, added text, dubbing, background music, etc. provided by the user. , intelligently generate video works through deep learning technology and algorithms.

S420. According to the generated information, divide the first frame of the video to obtain multiple coding blocks. The multiple coding blocks of the first frame include the current block. Among the multiple coding blocks of the first frame or adjacent to the first frame of the video, The target block is included in the second frame.

In this embodiment, the frame of the video may refer to an image in a continuous image sequence that constitutes the video. The second frame adjacent to the first frame can refer to two video frames that are closely adjacent in the continuous image sequence of the video, or it can refer to two video frames that are one or more frames apart. These two video frames There are temporal or spatial correlations that can be used for prediction and encoding.

In this embodiment, the coding block may refer to a video frame divided into small blocks for better encoding during the video compression encoding process. Each small block is an independent unit and is used for different encoding methods. In this field, compression includes inter-frame prediction and intra-frame prediction. The small blocks that need to be divided into inter-frame and intra-frame prediction are coding blocks. The coding blocks can have different levels. A large coding block can contain many small coding blocks. These large or small coding blocks are all coding blocks described in this application.

As an example, the coding block includes one or more of a macroblock, a sub-block, a coding tree unit CTU, a coding unit CU, a prediction unit PU (prediction unit), and a transformation unit TU (transform unit).

In this example, macroblocks and sub-blocks are, for example, coding block names in the AVC coding standard, and CTU, CU, PU, and TU are, for example, coding block names in the HEVC and VVC standards. The division, prediction, and encoding of these coding blocks can all be The video encoding method provided by the embodiment of this application is adopted. Those skilled in the art should know that there are other coding standards in this field. Concepts in these standards that are the same as or similar to the coding blocks in this application are within the scope of protection of this application. This application is not limited to the coding listed above. standard.

S430. Predict the predicted pixel information of the target block based on the current pixel information of the current block.

In this embodiment, the current pixel information may refer to the known pixel information in the current block, including the pixel value of each pixel point in the current block. The predicted pixel information may refer to the pixel information in the target block predicted based on the current block, including the pixel value of each pixel in the predicted block.

In this embodiment, the predicted pixel information of the target block is predicted based on the current pixel information of the current block, and a prediction method known in the art can be used. For example, based on the spatial and temporal similarity and redundancy between the current block and the target block, the pixel value of each pixel point of the target block is predicted based on the known pixel information of the current block.

S440. Encode the video according to the predicted pixel information.

In this embodiment, video encoding based on predicted pixel information can be performed using techniques known in the art, such as DCT, quantization, entropy coding, and other means. Specifically, the original pixel value (current pixel information) of the video within the coding block range can be subtracted from the predicted value (predicted pixel information) to obtain the residual. After DCT, the transformation result is quantized, and the quantized value and related syntax elements, such as The CU divides the depth, the motion vector corresponding to the pixel block, etc., and then performs entropy coding to obtain the compressed video stream.

As an example, FIG. 5 shows a schematic diagram of the dividing step in the video encoding method according to the embodiment of FIG. 4 . As shown in Figure 5, S420 can be specifically implemented as follows: dividing the first frame of the video to obtain multiple coding blocks. The multiple coding blocks of the first frame include the current block and the target block. In this example, the encoding method is intra-frame prediction, the current block and the target block are both in the same frame, and the pixel information of the target block in adjacent or other positions is predicted based on the pixel information of the current block in the frame. Intra-frame prediction is mainly based on the spatial correlation of picture content in the same frame, and adjacent coding blocks usually have spatially related pixel information.

As an example, FIG. 6 shows a schematic diagram of the dividing step in the video encoding method according to the embodiment of FIG. 4 . As shown in Figure 6, S420 can be specifically implemented as follows: dividing the first frame of the video to obtain multiple coding blocks, the multiple coding blocks of the first frame include the current block, and the second frame includes the target block. In this example, the encoding method is inter-frame prediction. The current block and the target block are in two adjacent or adjacent frames, and the pixel information of the target block in the other frame is predicted based on the current block in one frame. Inter-frame prediction is mainly based on the temporal correlation of the contents of two temporally adjacent frames. Two adjacent frames in a video frame sequence usually have temporally correlated pixel information.

As an example, FIG. 7 shows a schematic diagram of the dividing step in the video encoding method according to the embodiment of FIG. 4 . As shown in Figure 7, S420 can be specifically implemented as: determining the foreground area and background area in the first frame according to the position information of the foreground object; dividing the foreground area according to the first mode, and dividing the background area according to the second mode. Partitioning, where the coding blocks in the first mode are smaller than the coding blocks in the second mode.

In this example, the generated information includes position information of the foreground object. Based on this position information, the position and boundary of the foreground object in the current picture can be determined, thereby delimiting the foreground area and the background area. Due to the shape limitation of the coding block, the foreground area used to divide the coding block may not be an area that completely fits the outline of the foreground object, but contains the outline of the foreground object within it. For example, as shown in Figure 7, the foreground object is circular, and the foreground area is a square area containing the circle, which is delineated by a black box. For the foreground area, coding blocks with a smaller area (less number of pixels) can be divided. For the background area, coding blocks with a larger area (a larger number of pixels) can be divided. This is because the amount of movement or change of foreground objects is usually larger. Large, detailed division is required, while the background area has less changing content, so the division can be less detailed. For example, for the background area, coding blocks (such as CUs) are directly divided according to a fixed pattern, such as all divided into 64x64 size CUs for prediction and coding. In some embodiments, for the background area, a user interface may be provided, and the user may specify a encoding block division method. For example, in order to further improve the video quality, the background area may be divided into smaller encoding blocks such as 32x32 or 16x16.

As an example, FIG. 8 shows a schematic diagram of the dividing step in the video encoding method according to the embodiment of FIG. 4 . As shown in Figure 8, S420 can also be specifically implemented as follows: determining the edge area of the foreground object in the first frame according to the position information of the foreground object; and dividing the edge area according to the first mode.

In this example, the generated information still includes the position information of the foreground object. Based on this position information, the position and edge lines of the foreground object in the current picture can be determined, so that the edge area of the foreground object can be determined based on the position and size of the edge lines. As shown in Figure 7, the foreground object is circular, and its edge lines are circle lines. The coding area along the circular circle line can be the zigzag area between the large black box and the small black box in Figure 7. For the back-shaped area, a more detailed division method can be adopted, such as using smaller (less number of pixels) coding blocks for division, and for other areas, including areas outside the large black box and inside the small black box, A less detailed division method can be used, for example, coding blocks with a larger area (larger number of pixels) can be used for division. For example, the edge of the foreground area, such as the edge of the area where the character is located, is divided according to a fixed mode, such as only the 8x8 mode.

As an example, FIG. 9 shows a schematic diagram of two dividing methods according to the dividing step in the video encoding method of the embodiment of FIG. 4 . As shown in Figure 9, S420 can also be specifically implemented as: determining the foreground area and background area in the first frame based on the position information of the foreground object; determining whether the foreground object is a rigid object or a flexible object based on the material information of the foreground object; if If the foreground object is a rigid object, the foreground area is divided according to the third mode; if the foreground object is a flexible object, the foreground area is divided according to the fourth mode, in which the coding blocks in the third mode are larger than the coding blocks in the fourth mode.

As shown on the left side of Figure 9, the foreground object is a rigid object, so the third mode is used to divide the foreground area. As shown on the right side of Figure 9, the foreground object is a flexible object, so the fourth mode is used to divide the foreground area. As can be seen from the figure, the third mode and the fourth mode are both division modes for foreground objects, so compared to the background area (that is, the area outside the black box), the division of the two modes is more detailed. Between the third and fourth modes, the division of the fourth mode is more detailed, using coding blocks with smaller areas (fewer number of pixels) to divide the circular foreground objects in the picture. The division of the third mode It is relatively rough and uses coding blocks with a larger area (more pixels) to divide foreground objects. This is because the motion directions of various parts of rigid objects are relatively consistent and their spatial and temporal correlations are stronger. Therefore, in order to improve compression efficiency, larger encoding blocks can be used, while the motion directions of various parts of flexible objects usually are inconsistent. Or motion vector, its spatial and temporal correlation is weaker, so in order to retain the image details as much as possible and make the picture restoration higher, smaller coding blocks can be used. For example, for the inside of the foreground area, if the foreground area is made of flexible material, it will be divided according to the fixed mode of the smaller coding block, such as only 8x8 mode division; if the foreground area is made of rigid material, it will be divided according to the fixed mode of the larger coding block, such as Only 64x64 mode partitioning is performed.

As an example, FIG. 10 shows a schematic diagram of the dividing step in the video encoding method according to the embodiment of FIG. 4 . As shown in Figure 10, the foreground object is in the shape of a human body, and the edge shape is relatively complex. Various division methods can be adopted for such foreground objects. First, the foreground area can be determined, that is, the black rectangular frame in the picture, and the roughest background area coding block division method is used for the parts outside the rectangular frame, such as a 64x64 sized coding block. For the image area in the foreground area involving the edge of the human body outline, the most detailed division method can be adopted, such as 8x8-sized coding blocks. For the part of the foreground area that does not involve the edge of the foreground object, a mid-level division method can be adopted, such as a 32x32-sized coding block. For the interior of the foreground object, considering the rigidity of the human body, a mid-level division method can be adopted, such as a 32x32-sized coding block. For other parts, flexible partitioning strategies can be used, such as 24x32, 8x32, 16x16, etc.

Specifically, based on the foreground moving object information provided by the video generator, the encoder selects a 64x64 aligned minimum area to wrap the foreground moving object, such as the black rectangular area shown in Figure 10. If there are multiple foreground moving objects, multiple rectangular areas are generated. For the background area outside the rectangular area, which is less important than the foreground moving objects, they are all divided according to the maximum coding block specification, that is, 64x64, and prediction and production code streams are performed according to 64x64. For the area within the rectangular area, coding block division methods such as 32x32, 16x16, 24x32, 8x32 (allowed by the video coding protocol) that do not contain any foreground moving objects can be further performed. For foreground moving objects, if it is a rigid object, the motion is often consistent, and the maximum block division that does not include the edges of the foreground moving object can also be performed, such as the methods allowed by video encoding protocols such as 64x64 and 32x32. For foreground moving objects, if they are flexible objects, their movements are often inconsistent and should be divided according to the minimum 8x8 method. For the edge area of the foreground moving object, it is the image detail information, which is divided into coding blocks according to the minimum 8x8.

Based on the method embodiment described in Figure 4, an embodiment of the present application also provides a video encoding device 1100, the schematic structural diagram of which is shown in Figure 11. This device is used to perform the various steps in Figure 4 mentioned above.

According to this embodiment, the video encoding device 1100 includes an acquisition module 1110, a partitioning module 1120, a prediction module 1130 and an encoding module 1140. The acquisition module 1110 is used to acquire video generation information. The dividing module 1120 is used to divide the first frame of the video according to the generated information to obtain multiple coding blocks. The multiple coding blocks of the first frame include the current block. Among the multiple coding blocks of the first frame or the adjacent coding blocks of the video, The second frame of a frame includes the target block. The prediction module 1130 is configured to predict the predicted pixel information of the target block according to the current pixel information of the current block. The encoding module 1140 is used to encode the video according to the predicted pixel information.

In one embodiment, the generated information includes position information and material information of the foreground object in the first frame.

In one embodiment, the partitioning module 1120 is further configured to:

Determine the foreground area and background area in the first frame based on the position information of the foreground object;

The foreground area is divided according to the first mode, and the background area is divided according to the second mode, wherein the coding blocks in the first mode are smaller than the coding blocks in the second mode.

In one embodiment, the partitioning module 1120 is further configured to:

According to the position information of the foreground object, determine the edge area of the foreground object in the first frame;

The edge area is divided according to the first mode.

In one embodiment, the partitioning module 1120 is further configured to:

Determine the foreground area and background area in the first frame based on the position information of the foreground object;

Based on the material information of the foreground object, determine whether the foreground object is a rigid object or a flexible object;

If the foreground object is a rigid object, the foreground area is divided according to the third mode;

If the foreground object is a flexible object, the foreground area is divided according to the fourth mode, where the coding blocks in the third mode are larger than the coding blocks in the fourth mode.

In one embodiment, the video includes a virtual scene video generated by a video generation tool, and obtaining the video generation information includes:

Get generation information from the video generation tool.

In one embodiment, the video generation tool includes a renderer or AI tool.

In an embodiment, the coding block includes one or more of a macroblock, a sub-block, a coding tree unit CTU, a coding unit CU, a prediction unit PU, and a transformation unit TU.

It should be noted that when the video encoding device 1100 provided in the embodiment shown in FIG. 11 performs the method, only the division of the above functional modules is used as an example. In actual applications, the above functions can be allocated to different functional modules as needed. Completion means dividing the internal structure of the device into different functional modules to complete all or part of the functions described above. In addition, the video encoding device 1100 provided in the above embodiments and the video encoding method embodiment shown in FIG. 4 respectively belong to the same concept. Please refer to the method embodiment for details of the specific implementation process, which will not be described again here.

FIG. 12 is a schematic diagram of the hardware structure of a computing device 1200 according to an embodiment of the present application.

Referring to FIG. 12 , the computing device 1200 includes a processor 1210 , a memory 1220 , a communication interface 1230 , and a bus 1240 through which the processor 1210 , the memory 1220 , and the communication interface 1230 are connected to each other. The processor 1210, the memory 1220 and the communication interface 1230 may also be connected using other connection methods besides the bus 1240.

Among them, the memory 1220 can be various types of storage media, such as random access memory (randomaccess memory, RAM), read-only memory (read-only memory, ROM), non-volatile RAM (non-volatileRAM, NVRAM), Programmable ROM (PROM), erasable PROM (erasable PROM, EPROM), electrically erasable PROM (electrically erasable PROM, EEPROM), flash memory, optical memory, hard disk, etc.

The processor 1210 may be a general-purpose processor, and the general-purpose processor may be a processor that performs specific steps and/or operations by reading and executing content stored in a memory (eg, the memory 1220). For example, the general-purpose processor may be a central processing unit (CPU). The processor 1210 may include at least one circuit to perform all or part of the steps of the video encoding method provided by the embodiment shown in FIG. 4 .

Among them, the communication interface 1230 includes an input/output (I/O) interface, a physical interface, a logical interface, and other interfaces for realizing device interconnection within the computing device 1200, and for realizing the interconnection between the computing device 1200 and other devices. (such as other computing devices or user equipment) interconnection interfaces. The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc. The communication interface 1230 can be externally connected to input devices and output devices. For example, the input device may be a microphone or microphone array for capturing voice input signals; it may be a communication network connector for receiving collected input signals from the cloud or other devices; it may also include, for example, a keyboard, a mouse, and so on. The output device can output various information to the outside, including determined distance information, direction information, etc. Output devices may include, for example, displays, speakers, printers, and communication networks and remote output devices to which they are connected, among others.

The bus 1240 may be any type of communication bus used to interconnect the processor 1210, the memory 1220 and the communication interface 1230, such as a system bus.

The above-mentioned devices may be arranged on separate chips, or at least part or all of them may be arranged on the same chip. Whether each device is independently installed on different chips or integrated on one or more chips often depends on the needs of product design. The embodiments of this application do not limit the specific implementation forms of the above devices.

The computing device 1200 shown in FIG. 12 is only exemplary. During the implementation process, the computing device 1200 may also include other components, which are not listed here.

Embodiments of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are run by a processor, the processor causes the processor to perform various methods described above in this specification according to the present application. Steps in the video encoding method of the embodiment.

The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

The concepts, principles and ideas of the present application are described in detail above in conjunction with specific implementations (including embodiments and examples). Those skilled in the art should understand that the embodiments of the present application are more than the forms given above. After reading the documents of this application, those skilled in the art can make decisions about the steps, methods, devices and components in the above-mentioned embodiments. Any possible improvements, substitutions and equivalent forms shall be deemed to fall within the scope of this application. The scope of protection of this application shall be determined only by the claims.

Claims (11)

1. A video encoding method, characterized in that the method includes: Obtain the generation information of the video; According to the generated information, the first frame of the video is divided to obtain multiple coding blocks. The multiple coding blocks of the first frame include the current block, and any of the multiple coding blocks of the first frame or all A second frame of the video adjacent to the first frame includes a target block; Predict predicted pixel information of the target block according to the current pixel information of the current block; The video is encoded based on the predicted pixel information. 2. The video encoding method according to claim 1, wherein the generated information includes position information and material information of the foreground object in the first frame. 3. The video encoding method according to claim 2, wherein the first frame of the video is divided according to the generated information to obtain a plurality of coding blocks, including: Determine the foreground area and background area in the first frame according to the position information of the foreground object; The foreground area is divided according to a first mode, and the background area is divided according to a second mode, wherein the coding blocks in the first mode are smaller than the coding blocks in the second mode. 4. The video encoding method according to claim 3, characterized in that, dividing the first frame of the video according to the generated information to obtain a plurality of coding blocks, further comprising: Determine the edge area of the foreground object in the first frame according to the position information of the foreground object; The edge area is divided according to the first mode. 5. The video encoding method according to claim 2, wherein the first frame of the video is divided according to the generated information to obtain a plurality of coding blocks, including: Determine the foreground area and background area in the first frame according to the position information of the foreground object; Determine whether the foreground object is a rigid object or a flexible object according to the material information of the foreground object; If the foreground object is a rigid object, divide the foreground area according to the third mode; If the foreground object is a flexible object, the foreground area is divided according to a fourth mode, wherein the coding blocks in the third mode are larger than the coding blocks in the fourth mode. 6. The video encoding method according to claim 1, wherein the video includes a virtual scene video generated by a video generation tool, and the obtaining the generation information of the video includes: The generation information is obtained from the video generation tool. 7. The video encoding method according to claim 6, wherein the video generation tool includes a renderer or an AI tool. 8. The video encoding method according to claim 1, wherein the coding block includes one or more of a macroblock, a sub-block, a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), and a transformation unit (TU). kind. 9. A video encoding device, characterized in that the device includes: An acquisition module, used to acquire the generation information of the video; A dividing module, configured to divide the first frame of the video according to the generated information to obtain a plurality of coding blocks. The plurality of coding blocks of the first frame include the current block. The plurality of coding blocks of the first frame The target block is included in the encoding block or in a second frame of the video adjacent to the first frame; A prediction module, configured to predict the predicted pixel information of the target block based on the current pixel information of the current block; An encoding module, configured to encode the video according to the predicted pixel information. 10. A computing device, characterized in that the computing device includes a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the method of any one of claims 1 to 8 Video encoding methods. 11. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program is used to execute the video encoding method according to any one of claims 1 to 8.