CN101150719B - Method and device for parallel video coding - Google Patents

Abstract

The invention relates to a parallel video encoding method and device. The present invention mainly includes a main processor and a plurality of encoders, the main processor is used to divide the current frame in the video sequence into macroblocks, and allocate all the macroblocks to one or more slice groups according to predetermined rules; and Divide the slice group of the current frame into one or more sub-slice groups according to the principle of balancing the processing load between multiple encoders and in the order of raster scanning; and divide all the sub-slice groups and encoding The configuration parameters are respectively transmitted to multiple encoders, and then the multiple encoders encode the sub-strip groups of the current frame in parallel, and output code streams and parameters respectively; finally, the code streams output by each encoder And parameter aggregation, further complete the encoding of slice groups, frames and sequences, and output the entire sequence code stream. Therefore, the method and device of the present invention are suitable for real-time high-definition video coding.

Description

Method and device for parallel video coding

technical field

The present invention relates to the technical field of video coding, in particular to a parallel video coding technology.

Background technique

Video coding technology is to compress digital video information so that it can be transmitted and stored more efficiently. Video coding is the core technology of multimedia data processing.

At present, video compression coding standards mainly include: the moving picture coding standards H.261 and H.263 formulated by the Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU-T); the moving picture expert group ( MPEG) video coding standards MPEG1, MPEG4-Part2; video coding standards MPEG2/H.262, H.264/ AVC (MPEG4-Part10); In addition, there are VC-1 (formerly WMV-9) and the video coding standard AVS1.0-P2 formulated by the Audio Video Standards Group (AVS).

The basic framework of MPEG (Moving Picture Experts Group), JVT (MPEG Joint Experts Group) and VCEG (Video Coding Experts Group) series of video coding standards is shown in Figure 1, using a hybrid coding architecture based on block-based motion compensation and transform coding, including Intra prediction, inter prediction, transform, quantization and entropy coding, etc. Among them, inter-frame prediction uses block-based motion vectors to eliminate redundancy between images; intra-frame prediction uses spatial prediction modes to eliminate image redundancy. Then, the visual redundancy in the image is eliminated by transforming and quantizing the prediction residual. Finally, motion vectors, prediction modes, quantization parameters and transform coefficients are compressed with entropy coding. The basic processing unit in the video decoding process is a macroblock, and a macroblock usually includes a 16×16 luma sample block and a corresponding chrominance sample block.

There are certain differences in the tools used by different standards. For the new generation standard H.264/AVC (MPEG4-Part10), VC-1, AVS1.0-P2, deblocking filter is a necessary module, called loop filter; while in MPEG2, H.263 , In the MPEG4-Part2 standard, the deblocking filter is only an optional post-processing link in the decoder.

The input of the real-time video encoder is a high-definition video signal, and the video compression coding is completed in real time, and the output stream is output. The real-time video encoder is the basic equipment of the digital TV headend (Headend) system, and is also widely used in equipment such as video conferencing, digital camera, and recordable DVD player.

H.264/AVC (MPEG4-Part10), VC-1, and AVS1.0-P2 are called new-generation video compression coding standards. Compared with the previous generation standard represented by MPEG2, the compression rate of the new generation standard provides a higher times, but the complexity also increases by more than 2 times, and the difficulty of implementation is greatly increased.

High-definition television (HDTV) usually refers to moving images whose scanning lines per frame are 720 progressive lines or 1080 interlaced lines and above. Currently common high-definition formats are: 720p (resolution 1280×720, frame frequency 24, 30, 60), 1080i (interlaced scanning, resolution 1920×1088 per frame, field frequency 60), 1080p (resolution 1920× 1088, the frame rate is 24, 30). Higher resolution video will also be available in the future. High-definition video can provide higher video quality, and at the same time, the realization of high-definition video compression coding is more difficult.

Taking the new-generation standard H.264/AVC as an example, due to the introduction of multi-reference frame motion compensation, variable block size prediction with a minimum of 4×4, rich intra-frame prediction modes, loop filtering, and context-adaptive variable length Tools such as coding (CAVLC) or context-adaptive arithmetic coding (CABAC) greatly increase the complexity of the encoder. According to estimates, in the case of full search motion estimation, the computational complexity of the H.264/AVC HDTV720p encoder About 3600giga-instructions per second(GIPS), Memory access traffic about 5570giga-bytes per second(GBytes/s). It's even bigger at 1080i and 1080p.

Due to the enormous computational complexity of high-definition video encoders, real-time encoding is often difficult to achieve with a single processor. Especially in applications such as the headend (Headend), which need to support multiple channels, multiple video formats and compression coding standards, and transcoding application environments, the difficulty of using a single encoder to achieve encoding is more prominent. Therefore, high-definition encoders often need to use multiple chips (that is, multiple encoders) to implement encoding processing in parallel.

However, currently there is no parallel video coding solution in the industry that can well meet the high requirements of head-end applications such as H.264/AVC (MPEG4-Part10), VC-1, and AVS1.0-P2. Performance real-time HD encoding requirements.

Contents of the invention

The purpose of the present invention is to provide a parallel video encoding method and device, thereby providing an efficient implementation of parallel video encoding to meet high-performance real-time high-definition video encoding.

The purpose of the present invention is achieved through the following technical solutions:

The present invention provides a method for parallel video coding, the method comprising:

Divide the current frame in the video sequence into macroblocks, and assign all macroblocks to one or more slice groups according to predetermined rules;

Divide the slice group into one or more sub-slice groups according to the order of raster scanning; during the division process, each macroblock must and can only be allocated to one sub-slice group;

Corresponding sub-strip groups to respective encoders in a balanced manner according to their corresponding processing loads, each sub-strip group corresponds to the same encoder, and each encoder corresponds to one or more sub-strip groups;

According to the corresponding relationship between sub-slice groups and encoders, transmit all sub-slice groups and encoding configuration parameters of the current frame to multiple encoders;

When the current frame is an I frame, each encoder discards all reconstructed sub-slice group data;

When the current frame is not an I frame, each encoder reconstructs the sub-slice group from other encoders to obtain the reference data required for motion estimation of the sub-slice group; The image area and storage area overlapping relationship of the sub-slice group, as well as the maximum search area for motion estimation in the current frame sub-slice group, and the multi-reference frame attribute determine the minimum reference data exchanged between the encoders, and each encoder Obtain the reference data through an exchange operation, and use the reference data to update the reference data of each sub-slice group motion estimation search area cached in each encoder;

The sub-slice groups of the current frame are encoded in parallel by the multiple encoders, and each sub-strip group is divided into one or more slices in the order of raster scanning during the encoding process, and the Coding of all macroblocks in the slice group, generating a reconstructed sub-slice group, and outputting the coded code stream and parameters;

Gather the code streams and parameters output by each encoder to complete the encoding of the slice group, frame and sequence, and output the entire sequence code stream.

The predetermined rules include: interlaced scan mode, random scan mode, foreground and background scan mode, open box Box-out scan mode, raster sweep scan mode, handkerchief scan mode, and the need to use numbers to indicate that each macroblock belongs to The explicit scan mode of the slice group, or, divide the current frame into a slice group.

The sub-strip group is one or more continuous whole macroblock rows whose width is equal to the frame width.

The processing of corresponding the sub-strip group to each encoder in a balanced manner according to its corresponding processing load includes:

According to the processing capability of each encoder and the reference data transmission cost, each sub-slice included in the current frame is corresponding to each encoder, so that multiple encoders simultaneously complete the encoding process of the assigned sub-slice group.

During the encoding process of the group of pictures, the processing of dividing into multiple sub-slice groups includes:

For an I frame, the encoding processing load is predicted according to the number of macroblocks, and the sub-slice group is corresponding to each encoder according to the processing capability of the encoder and the encoding processing load, so that multiple encoders can simultaneously complete the assigned sub-slice group encoding processing;

For the first non-I frame, when the slice group division is unchanged, the sub-slice group division method of the I frame is used to divide the sub-slice group;

For the second non-I frame, predict the encoding processing load of each sub-slice group of the second non-I frame according to the actual statistics of the encoding processing load of each sub-slice group of the first non-I frame, and according to the reference Data loading cost, adjusting the division of the second non-I frame sub-strip group, so that the processing time of multiple encoders is consistent when encoding the second non-I frame in parallel;

For any non-I frame after the second non-I frame, predict the encoding processing load of each sub-slice group of the non-I frame according to the encoding processing amount of each sub-slice group in the previous frame or several previous frames of the frame, and According to the frame data loading cost, the division of the non-I frame sub-slice group is adjusted, so that the processing time of multiple encoders is consistent when the non-I frame is encoded in parallel.

The slice is one or more consecutive rows of full macroblocks with a width equal to the frame width.

The encoding process performed by each encoder on each macroblock in the sub-slice group includes:

Motion estimation and motion compensation using multiple backward or forward reference frames;

Intra-frame prediction selection, that is, when the current macroblock is intra-frame predicted, use the same strip to reconstruct the left macroblock and the upper macroblock data that have not been loop-filtered; intra-frame/inter-frame selection, find the residual;

Bit rate control;

Integer transformation, quantization;

Reordering, entropy coding, entropy coding is context-based adaptive variable-length coding or context-based adaptive binary arithmetic coding;

Inverse quantization, inverse transformation;

reconstruction;

loop filtering.

The encoding process performed by each encoder on each macroblock in the sub-slice group further includes: setting and selecting the boundary loop filtering mode of all slices in the current frame to no filtering, and each encoder independently completes each Each slice is loop-filtered, and information is not exchanged between each encoder and each slice.

The independently completed loop filtering includes: the loop filtering of the slice is started after the reconstruction of the first macroblock of the slice is completed, or the loop filtering of the slice is started after all reconstructions of the slice are completed.

The present invention also provides a device for parallel video encoding, which includes a main processor and multiple encoders, the main processor is used to divide the current frame to be encoded into sub-slice groups, and transmit them to the multiple encoders respectively. An encoder encodes each sub-strip group in parallel, and outputs each coded code stream to the main processor, and the main processor generates a serial code stream;

The main processor includes a slice group determination unit, a sub-slice group determination unit and a top-level coding unit, wherein:

A slice group determination unit, configured to divide the current frame in the video frame sequence into macroblocks, and assign all macroblocks to one or more slice groups according to predetermined rules;

The sub-slice group determination unit is used to divide all slice groups of the current frame into one or more sub-slice groups according to the order of raster scanning, and each macroblock must and can only be allocated To a sub-strip group, each encoder corresponds to one or more sub-strip groups, and each sub-strip group corresponds to an encoder;

The sub-slice group data transmission unit transmits all sub-slice groups and encoding configuration parameters of the current frame to each encoder according to the corresponding relationship between the sub-slice group and the encoder;

The top-level encoding unit gathers the code streams and parameters output by each encoder, completes the encoding of strip groups, frames and sequences, and outputs the entire sequence code stream;

The encoder includes a sub-strip group receiving unit, a reference data input and output unit, and an encoding unit, wherein:

The sub-strip group receiving unit is used to receive the sub-strip group and encoding configuration parameters;

The reference data input and output unit is used to exchange reference data between the encoders. When the current frame is not an I frame, control the exchange of the reconstructed reference frame sub-strip group data between the encoders, and update the cache in each encoder Each sub-slice group motion estimation search area reference data;

The encoding unit is used for encoding processing of the assigned sub-strip group. During the encoding process, each sub-strip group is divided into one or more slices in the order of raster scanning, and all macros in the assigned sub-strip group are completed. Coding of the block generates a reconstructed sub-strip group, and outputs coded code stream and parameters.

The processing of the main processor includes: dividing the entire frame into a slice group, and further dividing it into a plurality of sub-strip groups, and all the sub-strip groups are one or more continuous whole macros whose width is equal to the frame width block row.

Each of the plurality of encoders further includes a data storage unit for buffering current and reconstructed sub-slice group data.

The strip is one or more continuous whole macroblock rows whose width is equal to the frame width, and the same macroblock can only be divided into the same strip.

The main processor of the device further includes a control unit for initializing, configuring and controlling the main processor and multiple encoders to complete the encoding of the entire video sequence.

Each encoder in the plurality of encoders also includes:

Set the boundary loop filtering mode of all slices in the current frame to no filtering, each encoder completes the loop filtering process of each slice independently, no information is exchanged between each encoder and each slice, and the loop of the slice The in-loop filtering of the slice starts after the reconstruction of the first macroblock of the slice is completed, or the in-loop filtering of the slice starts after all reconstructions of the slice are completed.

The multiple encoders only complete macroblock-level entropy coding to output macroblock code streams and parameters, and the top-level coding module of the main processor completes slice-level entropy coding.

It can be seen from the above-mentioned technical solution provided by the present invention that the present invention provides a parallel video coding processing solution, so that a parallel video coding method can be selected in the video coding process, so that the present invention can also well meet high-performance real-time The processing power required for high-definition video encoding. At the same time, in the present invention, due to the principle of load balancing, the data to be encoded with moderate granularity is provided to a plurality of encoders, and the video data is encoded in parallel by a plurality of encoders, so that the processors are as close as possible to each other. The waiting time for each other can be reduced, and the interaction overhead between encoders can be reduced as much as possible. Therefore, the parallel video encoding processing solution provided by the present invention also has higher video encoding efficiency.

Description of drawings

FIG. 1 is a schematic diagram of a video coding framework in the prior art;

Fig. 2 is a schematic diagram of the specific implementation process of the method of the present invention;

Fig. 3 is a schematic structural diagram of a specific implementation of the device of the present invention.

Detailed ways

The present invention mainly divides the slice group into a plurality of sub-slice groups in the video encoding process, each sub-strip group includes one or more slices, and uses multiple encoders to perform parallel processing based on the sub-slice group. coding.

In order to facilitate the understanding of the technical concepts in the existing encoding implementation schemes involved in the present invention, some concepts in the existing encoding schemes are described first.

H.264 is divided into sequence, picture group, picture (ie frame), field, slice group, slice, macroblock, sub-macroblock and other different levels according to time and space from high to low.

(1) field, frame, image

A field or frame of video can be used to generate an encoded picture. In general, video frames can be divided into two types: continuous or interlaced video. In the case of interlaced video, two fields (referred to as the top field and the bottom field respectively) form a frame;

Current frame: the frame being encoded;

Reconstructed frame: the encoder reconstructs the output frame through local decoding;

Reference image (frame): In order to improve prediction accuracy, H264 encoding can select one or more images that best match the current image from a set of forward or backward encoded reconstruction images as reference images for inter-frame encoding. In H.264, you can choose from up to 16 reference images to select the best matching image.

(2) Macroblock (Macroblock, MB) and sub-macroblock

A coded image is usually divided into several macroblocks, and a macroblock may consist of a 16×16 luminance pixel and an additional 8×8Cb and an 8×8Cr color pixel block. A macroblock is the basic unit of size for video encoding. A macroblock can be further divided into blocks: it can be divided into 16×8, 8×16 or 8×8 luminance pixel blocks (and accompanying color pixels); for 8×8 sub-macroblocks, it can be further divided into Various sub-macroblocks: 8x4, 4x8 or 4x4 luma pixel blocks (and accompanying color pixels). Two corresponding macroblocks of the top and bottom fields of an interlaced frame form a macroblock pair.

According to whether the prediction mode adopted is intra frame or inter frame, the macro block is divided into intra frame prediction block (I macro block) and inter frame prediction block (P macro block). Pixels are used as a reference for intra prediction. The P macroblock uses the previously coded picture as a reference picture for intra prediction.

(3) Strip (Slice)

In each picture, several macroblocks are arranged in the form of slices. An intra-coded slice (I slice) contains only I macroblocks, and an inter-coded slice (P slice) can contain both P and I macroblocks. The coding of the slices is independent of each other, and the intra prediction of a certain slice cannot use the macroblocks in other slices as reference images.

A frame composed entirely of I slices is called an I frame; a frame not entirely composed of I slices is called a non-I frame.

(4) Strip group

Image macroblocks in the H.264 standard can be divided into multiple slice groups (slicegroups) in a flexible macroblock organization order (FMO); a slice group is a subset of several MBs in a coded image, which can contain one or several strips. Through the use of slice groups, FMO changes the way images are divided into slices and macroblocks, which can further improve the error recovery capability of slices.

The macroblock to slice group mapping defines which slice group the macroblock belongs to. Using FMO technology, H.264 defines seven macroblock scanning modes, the seven scanning modes include: interlaced, scattered, foreground and background, open box (Box-out), raster scan, handkerchief and explicit ( The slice group to which each macroblock belongs is indicated by number).

Different standards and profiles support FMO differently. H.264 basic profile (BaselineProfile) and extended profile support FMO7 scanning modes. H.264 main profile (Main Profile), VC-1, AVS1.0-P2 do not support FMO mode, there is only one scan mode of "raster scan"; there is only one strip group, and its size is equal to the frame.

(5) Group of Pictures (GOP)

For multiple consecutive images (frames), the starting frame is the I frame.

(6) sequence

A video sequence (sequence), the highest-level syntax structure of a coded bitstream, includes one or more consecutive coded images.

A profile is a subset of syntax, semantics, and algorithms specified in the encoding process. A decoder conforming to a profile must fully support the subset defined by that profile. The H.264/AVC standard is divided into 3 grades (Profile) and 4 kinds of high-fidelity extensions (High Extended).

1. Baseline Profile:

Utilize I slice and P slice to support intra-frame and inter-frame coding, and support entropy coding (CAVLC) based on context adaptive variable length coding. It is mainly used in real-time video communication such as videophone, conference TV, and wireless communication.

2. Main Profile:

Supports interlaced video, inter-frame coding using B slices and intra-frame coding using weighted prediction; supports context-based adaptive arithmetic coding (CABAC). It is mainly used for digital broadcast TV and digital video storage.

3. Extended Profile:

Support effective switching between code streams (SP and SI strips), improve bit error performance (data segmentation), but do not support interlaced video and CABAC, mainly used in streaming media.

In the specific encoding process, in order to facilitate the adaptation of various network standard protocols, the function of H.264/AVC is divided into two layers, namely the video coding layer (VCL) and the network abstraction layer (NAL, Network Abstraction Layer). The VCL data is the output after encoding, which represents the compressed and encoded video data sequence. Before VCL data transmission or storage, these coded VCL data are first mapped or encapsulated into NAL units.

The video coding layer of the H.264/AVC (MPEG4-Part10) video coding standard adopts a hybrid coding method of transformation and prediction, and the corresponding block diagram is still shown in FIG. 1 . If intra-frame prediction coding is used, the predicted value PRED (indicated by P in the figure) is obtained by motion compensation (MC) of the coded reference image in the current slice, where the reference image is F'n-1 express. In order to improve the prediction accuracy and thus the compression ratio, the actual reference image can be selected from the encoded, decoded, reconstructed and filtered frames in the past or in the future (referring to the display order). After the predicted value PRED is subtracted from the current block, a residual block Dn is generated. After block transformation and quantization, a set of quantized transformation coefficients X is generated, and then entropy encoded, and some side information required for decoding (such as prediction mode Quantization parameters, motion vectors, etc.) together form a compressed code stream, which is used for transmission and storage through NAL (Network Adaptation Layer).

As mentioned above, in order to provide reference images for further prediction, the encoder must have the function of reconstructing images. Therefore, it is necessary to add the D'n obtained after inverse quantization and inverse transformation of the residual image to the predicted value P to obtain uF'n (unfiltered frame). In order to remove the noise generated in the encoding and decoding loop, improve the image quality of the reference frame, and thus improve the performance of the compressed image, a loop filter is set, and the filtered output is the reconstructed image, which can be used as a reference image.

Motion estimation accounts for more than 50% of the computation of the encoder and is the bottleneck of the encoder implementation. The so-called motion estimation is to find out the block most similar to the current block according to a certain matching criterion for each block (luminance macroblock and its sub-macroblock) in the current frame to a given search range of the previous frame or the next frame. That is, the matching block, and the motion vector (Motion Vector) is calculated from the relative displacement between the matching block and the current block. The commonly used criterion here is the minimum sum of absolute errors (SAD). The more accurate the motion estimation, the smaller the compensation residual, the higher the coding efficiency, and the better the quality of the coded image. For block motion estimation, it is necessary to read in reference frame data (also referred to as reference data) corresponding to the search window of the block. For a 16×16 macroblock, if the motion estimation search position range is horizontal [-64, +63]/vertical [-32, +31], then it is necessary to read in reference data corresponding to this macroblock and The size of the surrounding location image area is (64+16+64)×(32+16+32)=144×80. In the case of multiple reference frames, it may be necessary to read in the search window data of multiple reference frames. Due to the large number of macroblocks per frame, the above-mentioned storage access is huge, which becomes the bottleneck of video coding implementation. For this reason, it is necessary to reuse the reference frame search window data between adjacent macroblocks, which can greatly reduce the amount of reference frame data read in. When many adjacent macroblocks perform motion estimation in one unit, the required read-in Reference data is only slightly larger than these macroblock areas.

The H.264/MPEG-4AVC standard defines a deblocking filtering process for 16X16 macroblocks and 4X4 block boundaries. In the case of macroblocks, the purpose of filtering is to remove artifacts caused by adjacent macroblocks having different intra or inter predictions or different quantization parameters. In the case of block boundaries, the purpose of filtering is to remove artifacts that may arise from transform/quantization and differences in motion vectors from neighboring blocks. Loop filtering modifies two pixels on the same side of a macroblock/block boundary through a content-adaptive non-linear algorithm.

Other next-generation video coding standards, namely VC-1 and AVS1.0-P2, have the same coding framework as H.264/AVC, only some module details are different; such as loop filtering: H.264/AVC can choose to Band boundary filtering or no filtering, VC-1 and AVS1.0-P2 strip boundaries are always not filtered, and the specific filtering algorithms are also different. Similarly, the previous generation of video coding standards, including MPEG4, H.263, MPEG4-Part2, and H.264/AVC, VC-1, AVS1.0-P2 have a similar coding framework, only some modules are different, such as MPEG4, H .263, MPEG4-Part2 encoder has no loop filter, and there is only one reference frame for motion estimation.

In order to facilitate the understanding of the present invention, the specific implementation process of the present invention will be described below.

The concrete realization process of the present invention is as shown in Figure 2, specifically comprises:

Step 21: Divide the current frame into at least one slice group;

Specifically, the current frame in the video frame sequence is divided into macroblocks, and all the macroblocks are assigned to one or more slice groups according to predetermined rules;

In this step, the video frame can be divided into slice groups, which can be divided according to various scanning modes of flexible macroblock organization order (FMO). The various scanning modes include: interlaced scanning mode, random scanning mode, Foreground and background scanning mode, open box (Box-out) scanning mode, raster scanning scanning mode, handkerchief scanning mode and explicit scanning mode, wherein, in the explicit scanning mode, it is necessary to use the number to indicate which macroblock belongs to Slice group; if the encoder does not support FMO, all macroblocks in one frame can be divided into one slice group.

Moreover, in the raster scanning scanning mode, the strip group can be selected as one or more continuous entire macroblock rows whose size and position are fixed in the image and whose width is equal to the frame width; In the scanning mode, the shapes of the strip groups may all be selected as rectangles.

Step 22: Divide the current frame into multiple sub-slice groups based on the slice group;

In the H.264 coding scheme, it is divided into different levels such as sequence, group of pictures (GOP), picture (frame), slice group, slice, macroblock, sub-macroblock; in the present invention, in order to use multiple encoders in parallel For encoding, it is necessary to select an appropriate granularity to divide the entire encoding task of a sequence into multiple sub-tasks (modules), that is, to further divide the slice group into one or more sub-slice groups, or the current frame has only one slice group, the slice group needs to be divided into multiple sub-slice groups, so that subsequent parallel encoding processing of load balancing can be performed based on the multiple sub-slice groups of the current frame.

In this step, all the slice groups of the current frame are further divided into a plurality of sub-strip groups, specifically divided into one or more sub-strip groups in the order of normalized raster scanning;

Wherein, all the sub-slice groups may be one or more continuous whole macroblock rows whose width is equal to the frame width.

It should be noted that, in the process of dividing sub-slice groups, each macroblock must and can only be assigned to one sub-slice group, and each encoder corresponds to one or more sub-slice groups, and each sub-slice group It can only correspond to one encoder, but each encoder can correspond to one or more sub-slice groups, that is, it can be used to encode one or more sub-slice groups;

In addition, in this step, one encoder can also correspond to multiple sub-slice groups. For example, if a certain slice group and sub-slice group division scheme causes the size of some sub-strip groups to be small, several small sub-slice groups can be Take components to an encoder.

The principle of processing load balancing between multiple encoders is as follows: divide the sub-strip group according to the principle of matching the processing capabilities of multiple encoders, and combine the reference data transmission cost during the division process to make multiple encoders complete all tasks simultaneously. Allocate encoding processing for sub-slice groups.

In this step, in the process of encoding a group of pictures (GOP), the specific sub-slice group division processing scheme includes the following processing methods:

For an I frame, the encoding processing load is predicted according to the number of macroblocks, and the sub-slice groups are divided according to the principle of matching the processing capabilities of multiple encoders, so that multiple encoders simultaneously complete the encoding processing of the allocated sub-slice groups;

For the first non-I frame, when the slice group division is unchanged, the sub-slice group division method of the I frame is used to divide the sub-slice group;

For the second non-I frame, predict the encoding processing load of each sub-slice group of the second non-I frame according to the actual statistics of the encoding processing load of each sub-slice group of the first non-I frame, and according to the reference Data loading cost, adjusting the division of the second non-I frame sub-strip group, so that the processing time of multiple encoders is consistent when encoding the second non-I frame in parallel;

For any non-I frame after the second non-I frame, predict the encoding processing load of each sub-slice group of the non-I frame according to the encoding processing amount of each sub-slice group in the previous frame or several previous frames of the frame, and According to the frame data loading cost, adjust the division of the sub-strip group of the non-I frame, so that the processing time of multiple encoders is consistent when encoding the non-I frame in parallel;

Step 23: Distributing the data to be encoded divided into sub-slice groups to each encoder;

The specific distribution method is carried out according to the corresponding relationship between sub-slice groups and encoders described in step 22. The specific distribution content not only includes all sub-slice groups in the current frame, but also includes related encoding configuration parameter information. Descriptive information, such as position, number of macroblocks, etc.; the encoding configuration parameter information specifically includes but not limited to: encoding standard information (standard, grade), FMO and scanning mode, number of reference frames, motion estimation search range, code rate control requirements, and the loop filter mode, etc.

Step 24: perform the operation of exchanging reference data between the encoders;

The specific process of exchanging reference data can be as follows: when the current frame is not an I frame, exchange the reconstructed reference frame sub-slice group data between encoders, and update the motion estimation of each sub-slice group cached in each encoder search area reference data;

Further, the process of exchanging reference data is as follows: when the current frame is not an I frame, the encoder performs inter-frame prediction, and the core process is motion estimation; before the motion estimation, the encoder needs to obtain a search Region reference data; the reference data comes from the previously reconstructed reference frame, which is composed of the reconstructed sub-strips output by each encoder; the reading of reference data is the largest part of video coding storage access, and it is important for coding The performance impact is significant; this requires minimizing the amount of reference data read in, which requires various reference data reuse strategies.

That is, in this step, if the current frame is not an I frame, then according to the image area and storage area overlapping relationship of the current frame sub-slice group and the reconstructed frame sub-strip group of each encoder, and the current frame sub-slice group For motion estimation maximum search area and reference frame, determine the minimum reference data exchanged between encoders, and use the exchanged reference data to update each sub-slice group motion estimation search area reference data cached in each encoder.

That is to say, in the present invention, the reconstructed sub-slice group generated during the encoding process of an encoder is stored in the local memory of the encoder. All of the data can be used as reference data for motion estimation in the next frame, but part of the reference data in the search area beyond the sub-slice group area allocated by this encoder is generated by other encoders and must be obtained from other encoders.

For example, a fixed-size search window and a reference frame are used in the motion estimation process, and the minimum reference data exchanged between encoders is determined according to the size of the search window, and the corresponding reference data is exchanged between encoders, so that it can be used to update the motion estimation search area reference data of each sub-slice group cached in each encoder.

Step 25: Encoding the macroblocks included in all the sub-slice groups of the current frame to be encoded is performed in parallel by each encoder;

The processing of encoding all sub-slice groups in the current frame in parallel with multiple encoders may specifically be as follows: each encoder completes the encoding of the assigned sub-slice group, and during the encoding process, each sub-slice group is encoded in a raster-scanned Sequentially divided into one or more slices, each macroblock must and can only be allocated to one slice; each encoder completes the encoding of all macroblocks in the assigned sub-slice group, generates a reconstructed sub-slice group, and outputs Encoding stream and parameters.

In this step, each encoder's encoding process for all macroblocks in the sub-slice group may specifically include: motion estimation, motion compensation; intra prediction selection, intra prediction, intra/inter selection, residual error calculation ; Rate Control; Transformation, Quantization; Reordering, Entropy Coding; Inverse Quantization, Inverse Transformation; and Reconstruction.

According to the H.264, VC-1, AVS1.0-P2 standards, the encoding process of each encoder for all macroblocks in the sub-slice group may specifically include the following steps:

(1) Use multiple backward or forward reference frames for motion estimation and motion compensation;

(2) Intra-frame prediction selection, that is, when the current macroblock is intra-frame predicted, use the same strip to reconstruct the left macroblock and the upper macroblock data that have not been loop-filtered; intra-frame/inter-frame selection, and find the residual;

(3) code rate control processing;

(4) Integer transformation and quantization processing;

(5) Reordering, entropy coding, entropy coding is context-based adaptive variable-length coding or context-based adaptive binary arithmetic coding;

(6) Inverse quantization and inverse transformation processing;

(7) Reconstruction processing;

(8) Loop filter processing.

Based on the H.264 standard, loop filtering is a necessary link in the encoding process to generate reconstructed frames. Whether to perform loop filtering at the boundary of each H.264 strip can be selected by setting the "strip boundary loop filtering mode". In the present invention, it is preferable to set the "boundary loop filtering mode" of all strips in the current frame to "no filtering". When the "boundary loop filter mode" of all slices in the current frame is set to "no filter", each encoder completes the loop filter processing of each slice independently, and no data is exchanged between encoders and slices.

Since the loop filtering of each slice is independent, in order to save the processing time of loop filtering, in the present invention, the loop filtering of the macroblocks contained in a slice is started after the reconstruction of the first macroblock of the slice is completed. The macroblock reconstruction and loop filtering constitute a macroblock-level pipeline. In order to reduce the access frequency of the loop filtering process to the main memory, the pixel values and block values of the bottom 4 rows of the entire row of macroblocks above can be stored on the on-chip cache. information and the pixel values and block information of the rightmost 4 columns of the left macroblock, only this on-chip register needs to be accessed during the loop filtering process. Of course, the loop filtering of a slice can also be started after the rebuilding of the slice is completed. Similarly, each encoder performs the loop filtering of each slice independently, and there is no need to switch between each encoder and each slice. data.

Step 26: Perform aggregation processing and top-level encoding on the encoded output code stream and parameters of the encoder to generate an output sequence code stream;

Specifically: the main processor collects the code streams and related parameters output by each encoder, generates slice groups and frame code streams, merges them to generate sequence code streams, and outputs them; for example, for H.264, it is necessary to complete the code streams including NAL All encoding is handled.

After the processing of the above steps 21 to 26, the parallel encoding process for video data can be realized, and the steps 21 to 26 need to be repeatedly executed during the encoding process for video data, so as to repeat the next step Frame encoding is processed until the end of the encoding process.

In the specific implementation process of the present invention, the process of dividing the subbands into groups is the key to the realization of the present invention. The process of dividing the subbands into groups will be described below in conjunction with specific application examples.

In the present invention, in order to obtain greater encoding processing capability through parallel processing and meet the requirements of real-time high-definition video encoding, it is necessary to use multiple encoders to perform encoding processing in parallel, and the architecture and encoding processing capabilities of the multiple encoders may be different. , but the total processing capacity of the encoder should be greater than the required encoding processing, and there is a certain margin to cope with the additional overhead of the parallel encoding processor.

Take high-definition video 720p (1280×720, 30fps) H.264 real-time encoding as an example. The encoder architecture is designed to support 1 or 2 reference frames, and the motion estimation search range is horizontal [-64, +63]/vertical [-32, +31]. A single encoder is realized by VLIW DSP or FPGA. According to the pre-evaluation, about 5 encoders are required to achieve the total encoding processing capacity of 720p with a certain margin. The encoder number is encoder 1 ~5.

In the present invention, in order to obtain the best parallel processing performance, the following two aspects are mainly considered:

(1) The load is balanced among the encoders, and the mutual waiting is as little as possible;

(2) Communication overhead between encoders should be as small as possible.

The method of the present invention enables multiple processors (ie, encoders) in the parallel video coding system to meet the above two aspects of processing through flexible division of sub-slice groups, so as to achieve the best performance of the entire parallel video coding system.

In order to meet the requirement described in (1) above, it is necessary to match the expected encoding processing load of the sub-slice group with the allocated processing capacity of the encoder. The encoding processing load of a sub-slice group is related to the size of the sub-slice group (that is, the number of macroblocks included), image content characteristics, and encoding configuration parameters (intra-frame prediction, inter-frame prediction, number of reference frames, minimum block, quantization level , entropy coding methods, etc.), generally, the more the number of macroblocks, the greater the encoding processing load, so the number of macroblocks in the sub-slice group can be adjusted to match the processing capability of the corresponding encoder.

For the requirements described in (2) above, the communication overhead of multi-encoder parallel processing is mainly the exchange of reference data. If the motion estimation parameters (number of reference frames, search range, etc.) are fixed, when the division of sub-strip groups remains unchanged, The communication overhead remains unchanged, and if the sub-strip group division of the current frame changes, the communication overhead of the reference data exchange increases significantly. Therefore, the adjustment of sub-stripe group division needs to calculate the cost impact of the increase of reference data communication overhead.

Specifically, taking a high-definition video 720p (1280×720) as an example, the method of dividing the slice group and the sub-strip group provided by the present invention will be described. That is, as shown in Table 1, Table 3 is the macroblock division of a 720p frame. There are 3600 16×16 macroblocks in total, which are divided into 45 macroblock rows (the rightmost column in the table is the number of the macroblock row, MBR1～ 45), each macroblock row has 80 macroblocks, and the number in each cell is the macroblock number, which is incremented according to the raster scan (from left to right, from top to bottom);

Table 1

0 1 2 36 37 38 39 40 41 42 43 … 77 78 79 MBR1 80 81 82 116 117 118 119 120 121 122 123 … 157 158 159 MBR2 … … … … … … … … … … … … … … … … MBR3~7 560 561 562 … 596 597 598 599 600 601 602 603 … 637 638 639 MBR8 640 641 642 … 676 677 678 679 680 691 692 693 … 717 718 719 MBR9 720 721 722 … 756 757 758 759 760 761 762 763 … 797 798 799 MBR10 800 801 802 … 836 837 838 839 840 841 842 843 … 877 878 879 MBR11 … … … … … … … … … … … … … … … … MBR12~16 1280 … … 1359 MBR17 1360 … … 1439 MBR18 1440 … … 1519 MBR19 1520 … … 1599 MBR20 … … … … … … … … … … … … … … … … MBR21~25 2000 … … 2039 MBR26 2080 … … 2159 MBR27 2160 … … 2239 MBR28 2240 … … 2319 MBR29 … … … … … … … … … … … … … … … … MBR30~34 2720 … … 2799 MBR35 2800 … … 2879 MBR36 2880 … … 2959 MBR37

2960 … … 3039 MBR38 … … … … … … … … … … … … … … … … MBR39~43 3440 … … 3519 MBR44 3520 … … 3599 MBR45

For a frame shown in Table 1, it is taken as an example to divide strip groups by foreground and background. Assume that there is only one foreground slice group, with a total of 216 macroblocks (8 macroblock columns, 27 macroblock rows). The remainder of the remaining macroblocks belong to the background slice group. In practical applications, such as video conferencing, people (or faces) in the field of view are often regarded as the most concerned part, and this part is divided into foreground strip groups for separate encoding. The two strip groups are further divided into a total of six sub-strip groups, wherein the background strip group is divided into five sub-strip groups, and the foreground strip group is divided into one sub-strip group.

As shown in Table 2, the corresponding divided sub-stripe groups may specifically be divided into the following division results:

Sub-strip group 1: macroblock numbers 0-719;

Sub-strip group 2: macroblock numbers 720-1439, except the foreground part (area in bold with double underlines);

Sub-strip group 3: macroblock numbers 1440-2159, except the foreground part (area in bold with double underlines);

Sub-strip group 4: macroblock numbers 2160-2879, except the foreground part (area in bold with double underlines);

Sub-strip group 5: macroblock numbers 2880-3599;

Sub-slice group 6: all 216 macroblocks of the foreground slice group (area in bold with double underline).

This division method of the sub-stripe group is applicable to the situation that the H.264 basic profile and the extended profile support FMO.

Table 2

Still taking the frame shown in Table 1 as an example, and as shown in Table 3, a frame is divided into 4 slice groups and further divided into a total of 5 sub-strip groups.

Slice group 1 is the italic font area in the upper left corner, with a total of 1600 macroblocks (40 macroblock columns and 40 macroblock rows); this strip group is further divided into two sub-strip groups (sub-strip group 1 , 2), each sub-strip group has a total of 800 macroblocks (40 macroblock columns, 20 macroblock rows).

Stripe group 2 is the underlined bold font area in the upper right corner, with a total of 1600 macroblocks (40 macroblock columns and 40 macroblock rows); this strip group is further divided into two sub-stripe groups (sub-stripe groups) strip groups 3 and 4), each sub-strip group has a total of 800 macroblocks (40 macroblock columns, 20 macroblock rows).

Slice group 3 is the bottom 5 full macroblock rows, with a total of 400 macroblocks (80 macroblock columns, 5 macroblock rows); this slice group is entirely divided into a sub-strip group (sub-strip group 5) .

This division is applicable to the situation that H.264 basic profile and extended profile support FMO.

table 3

0 1 2 … 36 37 38 39 40 41 42 43 … 77 78 79 MBR1 80 81 82 … 116 117 118 119 120 121 122 123 … 157 158 159 MBR2

0 1 2 … 36 37 38 39 40 41 42 43 … 77 78 79 MBR1 ... ... ... … ... ... ... ... … … … … … … … … MBR3~7 560 561 562 … 596 597 598 599 600 601 602 603 … 637 638 639 MBR8 640 641 642 … 676 677 678 679 680 691 692 693 … 717 718 719 MBR9 720 721 722 … 756 757 758 759 760 761 762 763 … 797 798 799 MBR10 800 801 802 … 836 837 838 839 840 841 842 843 … 877 878 879 MBR11 ... ... ... … ... ... ... ... … … … … … … … … MBR12~16 1280 … … 1359 MBR17 1360 … … 1439 MBR18 1440 … … 1519 MBR19 1520 … … 1599 MBR20 ... ... ... … ... ... ... ... … … … … … … … … MBR21~25 2000 … … 2039 MBR26 2080 … … 2159 MBR27 2160 … … 2239 MBR28 2240 … … 2319 MBR29 ... ... ... … ... ... ... ... … … … … … … … … MBR30~34 2720 … … 2799 MBR35 2800 … … 2879 MBR36 2880 … … 2959 MBR37 2960 … … 3039 MBR38 ... ... ... … ... ... ... ... ... ... ... ... … ... ... ... MBR39~43 3440 … … 3519 MBR44 3520 … … 3599 MBR45

Taking the division of stripe groups by raster scanning as an example, the corresponding division results are shown in Table 4. One frame is divided into one stripe group, which is further divided into a total of five sub-strip groups, and each sub-strip group is 9 The whole macroblock row has 720 macroblocks in total.

Sub-strip group 1: macroblock numbers 0-719;

Sub-strip group 2: macroblock numbers 720-1439;

Sub-strip group 3: macroblock numbers 1440-2159;

Sub-strip group 4: macroblock numbers 2160-2879;

Sub-strip group 5: macroblock numbers 2880-3599;

This division scheme is applicable to H.264, VC-1, AVS1.0-P2, H.263, MPEG2, MPEG4-Part2 standards and their grades.

Table 4

The exchange reference data described in step 24 among the present invention is illustrated as follows here:

Take the sub-band grouping method shown in Table 4 as an example, and take 720p, motion estimation search range horizontal [-64, +63]/vertical [-32, +31] as an example, assuming that encoder 3 corresponds to sub-strips In group 3, the required reference data area is one or more previous reference frame image areas corresponding to macroblock numbers 1280-2319.

For the motion estimation of a reference frame, if the division of sub-slice groups encoded by the current frame and the previous frame of the encoder 3 remains unchanged, that is, the same macroblock number is 1440-2159 areas, because the required reference frame area The area of macroblocks numbered 1440-2159 is the sub-strip group reconstructed by the encoder in the last frame, so the reference data that needs to be read in from the encoder 3 is only the following two blocks:

Macroblocks numbered 1280-1439 from reconstructed sub-slice group 2 (output by encoder 2); and,

The macroblocks are numbered 2160-2319 and come from the reconstructed sub-slice group 4 (outputted by the encoder 4).

Similarly, in the case of motion estimation of a reference frame, if the sub-slice group division between the current frame and the previous frame of the encoder 2 remains unchanged, that is, the same macroblock number is 720-1439, because the required reference In the frame area, the macroblock numbered from 720 to 1439 is the sub-strip group reconstructed in the previous frame of the encoder, so the reference data that needs to be read from the outside of the encoder 3 is only the following two blocks:

The macroblocks are numbered 1440-1559 from the reconstructed sub-slice group 3 (output by the encoder 3); and,

The macroblocks are numbered 560-719, and come from the reconstructed sub-slice group 1 (outputted by the encoder 1).

Here, the two encoders 2 and 3 adjacent to the current encoding sub-slice group transmit the reconstructed sub-slice group data to each other as reference data for encoding operations.

If multi-reference frame motion estimation is used, it is necessary to load the required multiple reconstructed sub-slice group data parts from other encoders.

The method provided by the present invention can be applicable to various video formats, including high-definition video or standard-definition video, and can be progressive or interlaced scanning; wherein, for interlaced scanning video, the top field and bottom field of an interlaced scanning frame Two corresponding macroblocks form a macroblock pair, and one macroblock pair needs to be allocated to the same slice group, sub-slice group, and slice coding.

Based on the previous examples of slice group and sub-slice group division, the following describes an application example for the division of slices in the sub-slice group encoding process:

Application example one

An example of dividing sub-slice groups into slice partitions is shown in Table 5. In Table 5, sub-strip group 1 is divided into 7 slices during the encoding process, numbered as slices 1-7 according to the raster scanning sequence, slices 1 and 2 are two macroblock rows, and slices 3-7 Each is a macroblock row;

table 5

0 1 2 … 36 37 38 39 40 41 42 43 … 77 78 79 MBR1 80 81 82 … 116 117 118 119 120 121 122 123 … 157 158 159 MBR2 160 161 162 … 196 197 198 199 200 201 202 203 … 237 238 239 MBR3 240 241 242 … 276 277 278 279 280 281 282 283 … 317 318 319 MBR4 320 321 322 … 356 357 358 359 360 361 362 363 … 397 398 399 MBR5 400 401 402 … 416 417 418 419 420 421 422 423 … 477 478 479 MBR6 480 481 482 … 516 517 518 519 520 521 522 523 557 558 559 MBR7 560 561 562 … 596 597 598 599 600 601 602 603 … 637 638 639 MBR8 640 641 642 676 677 678 679 680 691 692 693 717 718 719 MBR9

Application example two

The second example of slice division is shown in Table 6. In Table 2, sub-strip group 1 is divided into 5 slices during the encoding process, and they are numbered as slices 1 to 5 according to the raster scanning order. Each slice contains The macroblocks are as follows:

Slice 1: macroblock code 0~123;

Slice 2: Macroblock coding 124-237;

Slice 5: macroblock coding 238-319;

Slice 3: macroblock coding 320-521;

Slice 4: macroblock coding 522-719;

Table 6

0 1 2 … 36 37 38 39 40 41 42 43 44 … 77 78 79 MBR1 80 81 82 … 116 117 118 119 120 121 122 123 124 … 157 158 159 MBR2 160 161 162 … 196 197 198 199 200 201 202 203 204 … 237 238 239 MBR3 240 241 242 … 276 277 278 279 280 281 282 283 284 … 317 318 319 MBR4

0 1 2 … 36 37 38 39 40 41 42 43 44 … 77 78 79 MBR1 320 321 322 … 356 357 358 359 360 361 362 363 364 … 397 398 399 MBR5 400 401 402 … 416 417 418 419 420 421 422 423 424 … 477 478 479 MBR6 480 481 482 … 516 517 518 519 520 521 522 523 524 … 557 558 559 MBR7 560 561 562 … 596 597 598 599 600 601 602 603 604 … 637 638 639 MBR8 640 641 642 … 676 677 678 679 680 691 692 693 694 … 717 718 719 MBR9

The present invention also includes a device for parallel video encoding. The specific implementation structure of the device is shown in Figure 3, including a main processor and a plurality of encoders, and the main processor is used to divide the current frame to be encoded into sub-strip groups , and transmit them to multiple encoders respectively, and the multiple encoders encode each sub-strip group in parallel, and output respective encoded code streams to the main processor, and the main processor generates and outputs serial code streams.

(1) Main processor

The main processor further includes a slice group determination unit, a sub-stripe group determination unit, a sub-stripe group data transfer unit, and a top-level encoding unit, and each unit is specifically:

The slice group determination unit is used to receive digital video sequence data from the data transmission unit, divide the current frame in the video frame sequence into macroblocks, and assign all macroblocks to one or more slice groups according to predetermined rules;

The sub-slice group determination unit is used to divide all slice groups of the current frame into one or more sub-slice groups in the order of raster scanning, and each macroblock must and can only be assigned to one sub-slice group;

A sub-strip group data transfer unit, configured to control multiple encoders to read in the sub-strip group data according to the corresponding configuration between the sub-strip group and the encoder, and each encoder reads in one or more sub-stripes The data to be encoded in each sub-strip group is transmitted to the same encoder; the bus signal formats that can be used in the sub-strip group data transmission process include but are not limited to: parallel interface, high-speed serial interface, BT656 or CCIR601 digital video signal format, HD-SDI signal, signal format corresponding to AHB or AXI bus or Ethernet interface in AMBA bus specification;

Top-level encoding unit: used to collect the code streams and related parameters output by each encoder, generate slice groups and frame code streams, and combine them to generate sequence code streams and output them;

Specifically, in the main processor, the entire frame can be divided into a strip group and divided into multiple sub-strip groups, and all sub-strip groups are one or more consecutive ones whose size and position are fixed in the image and whose width is equal to the frame width. entire macroblock row.

In addition, the main processor also performs image preprocessing on the received current frame in advance, including: two-dimensional denoising processing, and or, scaling processing, and/or, digital video 4:4:4 to 4:2: 2 Format conversion processing, and/or format conversion processing of digital video from 4:2:2 to 4:2:0, etc.

Moreover, the main processor of the present invention may specifically be a computer system, including: a CPU, a memory, an interface with a data transfer unit (an interface with a data transfer unit and an external communication network input interface), and an output interface for output coded code stream, and configuration control information interface for receiving external configuration control information.

(2) encoder, the structure of described encoder specifically can comprise:

(1) a sub-strip group receiving unit, configured to receive sub-strip groups and encoding configuration parameters;

(2) The reference data input and output unit is used for exchanging reference data between encoders, specifically: when the current frame is not an I frame, each encoder exchanges reference data units set in the data transfer unit to exchange Obtain the required reference data from other encoders, that is, control the exchange of reconstructed reference frame sub-slice group data between encoders, and update the reference data of each sub-slice group motion estimation search area cached in each encoder.

(3) The coding unit is used to encode all sub-strip groups in the current frame in parallel: each encoder completes the coding of the assigned sub-strip group, and divides each sub-strip group into raster scanning order during the coding process One or more slices, each macroblock must and can only be allocated to one slice; each encoder completes the encoding of all macroblocks in the allocated sub-slice group, generates a reconstructed sub-slice group, generates and outputs code stream and related parameters; moreover, the slice is preferably one or more consecutive whole macroblock rows with a width equal to the frame width;

Each encoder in the plurality of encoders described in the present invention can be:

General-purpose processors, including: superscalar processors, very long instruction word signal processors (VLIW DSP); or

Field Programmable Large Scale Array (FPGA); or

Custom VLSI chips; or

The instruction set configures the processor.

Each encoder is also configured with a data random access memory (RAM), that is, a storage unit for caching information such as current and reconstructed sub-slice group data. When the capacity of the data RAM is not large, the on-chip large-capacity RAM can be used, and when the required capacity is large, the off-chip synchronous static random access memory (SSRAM) or synchronous dynamic random access memory (SDRAM) can be configured.

Further, in order to obtain better parallel processing, when being used for H.264, VC-1, AVS1.0-P2 standard encoding, the loop filtering mode in the described encoder encoding is:

Set the boundary loop filtering mode of all slices in the current frame to no filtering, each encoder completes the loop filtering process independently, and no data is exchanged between each encoder and each slice;

Specifically: the loop filtering of a slice is started after the reconstruction of the first macroblock of the slice is completed, and the reconstruction of the macroblock and the loop filtering constitute a macroblock-level pipeline; or, the loop filtering of a slice starts After all band reconstruction is completed, each encoder performs loop filtering for each band independently, and there is no need to exchange data between each encoder and each band.

During the encoding process of the encoder, the corresponding reference data exchange process is:

When the current frame is not an I frame, the reference frame sub-slice group data is exchanged between the encoders, according to the overlapping relationship between the image area and the storage area of the current frame sub-slice group and the reconstruction frame sub-slice group of each encoder, and Considering the maximum motion estimation search area and reference frame in the sub-slice group of the current frame, determine the minimum reference data exchanged between encoders, and update the reference data of each sub-slice group motion estimation search area cached in each encoder; wherein, Motion estimation can use a fixed-size search window and a reference frame, determine the minimum reference data exchanged between encoders according to the size of the search window, and update the reference data of each sub-slice group motion estimation search area cached in each encoder.

It should be noted that during the encoding operation of the encoder, it is usually necessary to perform preprocessing on the current frame before video encoding, including: two-dimensional denoising, scaling, and format conversion. Format conversion refers to converting the input digital video format to It is processed for the format required by the encoder; the common encoded digital video format is 4:2:0, and the common input digital video format is 4:2:2 specified by BT656 or CCIR601. H.264 high-fidelity extended encoding supports 4:2:2 and 4:4:4, and the input digital video is 4:2:2 and 4:4:4. HD video physical interface is usually HD-SDI.

Also, the current frame data preprocessing required by multiple encoders can take one of two forms:

1. The main processor receives the current frame, preprocesses it centrally, buffers it, and then forwards it to multiple encoders; this solution requires a strong main processor processing capability and a large data frame storage capacity;

2. Each of the multiple encoders pre-processes the allocated sub-strip group.

The main processor of the device according to the present invention may also be provided with a control unit for initializing, configuring and controlling other processing units and multiple encoders in the main processor to complete the encoding of the entire video sequence.

In the device of the present invention, in order to simplify the start-up and configuration circuits of each encoder and reduce costs, the start-up and configuration programs or data of the encoder can come from the main processor; in the case that the encoder is an FPGA, it can The FPGA is configured by the main processor through the parallel or serial configuration interface; for the case where the encoder is a general-purpose processor, parallel port or serial port boot (Boot) can be selected, and the BootROM program used during the boot process comes from the main processor.

The device of the present invention is typically an embedded computer system composed of one or more circuit boards; or, it can also be a system in which personal computers or servers are connected together through a communication network, wherein the encoder can be a PC or a server .

The method and device described in the present invention are applicable to various encoding standards and their grades.

The device provided by the present invention is applicable to various video formats, including high-definition video or standard-definition video, which can be progressive or interlaced. For interlaced video, two corresponding macroblocks of the top field and the bottom field of an interlaced frame To form a macroblock pair, it is necessary to assign a macroblock pair to the same slice group, sub-slice group, and slice coding.

The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present invention can easily think of changes or Replacement should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (16)

1. a method of parallel video coding is characterized in that, comprising:

Present frame in the video sequence is divided into macro block, presses pre-defined rule and give one or more slice-group all macroblock allocation;

Order according to raster scan is divided into one or more sub-slice-group respectively with described slice-group; In described partition process, each macro block must and can only be assigned to a sub-slice-group;

With sub-slice-group according to the processing load balancing of its correspondence correspond to each encoder, the corresponding one and same coding device of each sub-slice-group, the corresponding one or more sub-slice-group of each encoder;

By sub-slice-group and encoder corresponding relation, send all sub-slice-group of present frame and coding configuration parameter to each encoder;

When present frame is not the I frame, exchange the sub-slice-group data of reference frame of having rebuild between the encoder, upgrade each sub-slice-group motion estimation search district reference data of buffer memory in each encoder;

By described each encoder to the parallel encoding process of carrying out of the sub-slice-group of present frame, in cataloged procedure, each sub-slice-group is divided into one or more bands by the order of raster scan, finish all macroblock encoding in the sub-slice-group that each encoder is distributed, produce and rebuild sub-slice-group, output encoder code stream and parameter;

The code stream and the parameter of each encoder output are converged, finish the coding of slice-group, frame and sequence, output whole sequence code stream.

2. method according to claim 1 is characterized in that, described pre-defined rule comprises:

Interlacing pattern, random scanning pattern, prospect and background scans pattern, Box-out scan pattern, raster scan scan pattern, handkerchief scan pattern and needs adopt numbering to indicate the explicit scan pattern of the affiliated slice-group of each macro block, perhaps, present frame is divided into a slice-group.

3. method according to claim 1 is characterized in that, described sub-slice-group is one or more continuous whole macro-block line that width equals the frame width.

4. according to each described method of claim 1 to 3, it is characterized in that, described with sub-slice-group according to the processing load balancing of its correspondence correspond to each encoder processing comprise:

According to each coder processes ability, and reference data transmits cost, and each sub-band that present frame is comprised corresponds to each encoder, make a plurality of encoders finish simultaneously divide the encoding process of gamete slice-group.

5. method according to claim 1 is characterized in that, in the image sets cataloged procedure, the described processing that is divided into a plurality of sub-slice-group comprises:

For the I frame, handle load according to the macroblock number predictive coding, and sub-slice-group corresponded to each encoder according to the disposal ability and the described encoding process load of encoder, make a plurality of encoders finish simultaneously divide the encoding process of gamete slice-group;

For first non-I frame, divide when constant when slice-group, adopt the sub-slice-group dividing mode of I frame to carry out the division of sub-slice-group;

For the 2nd non-I frame, each sub-slice-group encoding process amount of the 1st the non-I frame that obtains according to actual count is predicted the encoding process load of described the 2nd non-each sub-slice-group of I frame, and be written into cost according to reference data, adjust the division of described the 2nd non-I frame slice-group, a plurality of coder processes time unanimities when making described the 2nd the non-I frame of parallel encoding;

For the 2nd the arbitrary non-I frame that non-I frame is later, predict the encoding process load of described non-each sub-slice-group of I frame according to this frame former frame or each sub-slice-group encoding process amount of former frame, and be written into cost according to reference frame data, adjust the division of described non-I frame slice-group, a plurality of coder processes time unanimities when making the described non-I frame of parallel encoding.

6. method according to claim 1 is characterized in that, described band is one or more continuous whole macro-block line that width equals the frame width.

7. method according to claim 1 is characterized in that, the encoding process process that each macro block in described each encoder antithetical phrase slice-group carries out comprises:

Use a plurality of backward or forward reference frames to carry out estimation, and motion compensation;

Infra-frame prediction is selected, and promptly during the current macro infra-frame prediction, uses same band to rebuild as yet the not left side macro block and the top macro block data of loop filtering; In the frame/and the interframe selection, ask residual error;

Rate Control;

Integer transform, quantification;

Reorder, entropy coding, entropy coding are that context is based on adaptive variable-length encoding or based on contextual adaptive binary arithmetic coding;

Inverse quantization, inverse transformation;

Rebuild;

Loop filtering.

8. method according to claim 7 is characterized in that, the encoding process process that each macro block in described each encoder antithetical phrase slice-group carries out also comprises:

Be provided with and select the border loop filter patterns of all bands of present frame to be not filtering, each encoder is independently finished each band loop Filtering Processing separately, and exchange message not between each encoder, each band.

9. method according to claim 8 is characterized in that, the described loop filtering of independently finishing separately comprises:

The loop filtering of band starts after first macro block reconstruction of this band is finished, and perhaps, the loop filtering of band begins after this band reconstruction is all finished.

10. the device of a parallel video coding, it is characterized in that, comprise primary processor and a plurality of encoder, primary processor is used for present frame to be encoded is divided into sub-slice-group, and pass to a plurality of encoders respectively, each sub-slice-group of a plurality of encoder parallel encodings is exported separately encoding code stream and is given primary processor, by primary processor formation sequence code stream and output;

Described primary processor comprises slice-group determining unit, sub-slice-group determining unit and top layer coding unit, wherein:

The slice-group determining unit is used for the present frame of sequence of frames of video is divided into macro block, presses pre-defined rule and gives one or more slice-group with all macroblock allocation;

Sub-slice-group determining unit, be used for all slice-group to present frame, order according to raster scan is divided into one or more sub-slice-group respectively with described slice-group, and each macro block must and can only be assigned to a sub-slice-group, the corresponding one or more sub-slice-group of each encoder, the corresponding encoder of each sub-slice-group;

Sub-slice-group data passes unit sends all sub-slice-group of present frame and coding configuration parameter to each encoder by sub-slice-group and encoder corresponding relation;

The top layer coding unit, code stream and parameter that each encoder is exported converge, and finish the coding of slice-group, frame and sequence, output whole sequence code stream;

Described encoder comprises sub-slice-group receiving element, reference data input-output unit and coding unit, wherein:

Sub-slice-group receiving element is used to receive sub-slice-group and coding configuration parameter;

The reference data input-output unit, be used between each encoder, exchanging reference data, when present frame is not the I frame, exchange the sub-slice-group data of reference frame of having rebuild between the controlled encoder, upgrade each sub-slice-group motion estimation search district reference data of buffer memory in each encoder;

Coding unit, be used for divide the gamete slice-group to carry out encoding process, in cataloged procedure, each sub-slice-group is divided into one or more bands by the order of raster scan, finish all macroblock encoding in the branch gamete slice-group, produce and rebuild sub-slice-group, output encoder code stream and parameter.

11. device according to claim 10 is characterized in that, the processing of described primary processor comprises:

Entire frame is divided into a slice-group, and further is divided into a plurality of sub-slice-group, and all sub-slice-group are one or more continuous whole macro-block line that width equals the frame width.

12. device according to claim 10 is characterized in that, each encoder in described a plurality of encoders also comprises data storage cell, is used for the sub-slice-group data that buffer memory is current and rebuild.

13. device according to claim 10 is characterized in that, described band is one or more continuous whole macro-block line that width equals the frame width, and described same macro block only can be divided in the same band.

14. device according to claim 10 is characterized in that, described device primary processor also comprises control unit, is used for initialization, configuration and control primary processor and a plurality of encoder and finishes the whole video sequence coding.

15., it is characterized in that each encoder in described a plurality of encoders also comprises according to each described device of claim 10 to 13 in carrying out cataloged procedure:

The border loop filter patterns of all bands of present frame is set to not filtering, each encoder is independently finished each band loop Filtering Processing separately, exchange message not between each encoder, each band, and the loop filtering of band starts after first macro block reconstruction of this band is finished, perhaps, the loop filtering of band begins after this band reconstruction is all finished.

16., it is characterized in that described a plurality of encoders are only finished macro-block level entropy coding output macro block code stream and parameter according to each described device of claim 10 to 13, primary processor top layer coding module is finished the slice level entropy coding.

Images