CN104168481B - Real-time AVS soft coding method for Intel mobile platform - Google Patents

Abstract

The invention discloses a kind of soft coding methods of real-time AVS for Intel mobile platforms.This method is：1) AVS encoders detect the type of current macroblock to be encoded, call corresponding I frames coding unit or P frame coding units to be encoded；2) when carrying out I frame macroblock codings, the termination in advance of predictive mode traversal is realized as threshold value by using the Matching power flow average of predictive mode；3) when carrying out P frame macroblock codings, the termination in advance that threshold value realizes reference frame and predictive mode traversal is used as by using the minimum value of Matching power flow.During Matching power flow (sad value) before current macro and predicted macroblock is calculated, realized using the distinctive SSE instruction techniques of Intel mobile platforms.The present invention substantially increases processor utilization, realizes the video acquisition and coding of 10 frame per second.

Description

Real-time AVS soft coding method for Intel mobile platform

technical field

The method involves AVS video coding technology, Android application program development technology, NDK technology, program optimization technology and SSE technology. The method is directly applied to the field of mobile platform video coding, and realizes real-time AVS soft coding on the Intel mobile platform.

Background technique

AVS video coding technology:

The AVS video standard is formulated to meet the needs of moving image compression technology in applications such as digital TV broadcasting, digital storage media, Internet streaming media, and multimedia communications. The scope of application of this standard includes but is not limited to the following fields: digital terrestrial television broadcasting (DTTB, Digital terrestrial television broadcasting), cable television (CATV, Cable TV), interactive storage media, direct broadcast satellite video services (DBS, Direct broadcastSatellite video services ), broadband video services, multimedia mail, multimedia services on packet networks (MSPN, Multimedia services on packet networks), real-time communication services (video conferencing, videophone, etc.), remote video surveillance.

This standard adopts a series of techniques to achieve high-efficiency video coding, including intra prediction, inter prediction, transformation, quantization and entropy coding. Inter prediction uses block-based motion vectors to remove redundancy between images; intra prediction uses spatial prediction modes to remove redundancy within images. Remove visual redundancy within images by transforming and quantizing prediction residuals. Finally, motion vectors, prediction modes, quantization parameters and transform coefficients are compressed with entropy coding.

Android application development technology:

Android is a Linux-based free and open-source operating system primarily used on mobile devices such as smartphones and tablets. Android's system architecture, like its operating system, adopts a layered architecture and is divided into four layers. From high to low, they are the application layer, the application framework layer, the system runtime layer and the Linux kernel layer. The four major components of Android development are: Activity (Activity): used to express functions. Service: The service runs in the background and does not provide interface presentation. Broadcast Receiver (BroadcastReceiver): used to receive broadcasts. Content Provider: It supports storing and reading data in multiple applications, which is equivalent to a database. The basic Android application development can be completed by using the above components in the Android Integrated Development Environment (IDE).

NDK technology:

Android NDK (Android Native Development Kit) is a series of development tools that allow program developers to embed code written in C/C++ language in Android applications. NDK allows program developers to use C/C++ language to develop library files, and provides convenient tools to package library files into apk files.

Program optimization technology:

Program optimization technology refers to improving the program by modifying the algorithm and structure of the original program or using software development tools according to the characteristics of the processor and the system without changing the program function. Make the modified program run faster or take up less space or consume the least amount of energy. The principles of optimization are: equivalence principle, effective principle and economic principle. The optimization methods mainly include: program multi-threading, using processor-specific compiler, program structure optimization, code optimization, etc.

SSE technology:

SSE is the abbreviation of the instruction set, which includes 70 instructions, including single instruction multiple data floating point calculations, and additional SIMD integer and cache control instructions. Its advantages include: higher resolution image browsing and processing, high-quality audio, MPEG2 video, and MPEG2 encryption and decryption at the same time; speech recognition takes up less CPU resources; higher precision and faster response speed.

Existing similar applications have disadvantages such as low AVS encoding efficiency and low processor utilization.

Contents of the invention

Aiming at the problems of low AVS coding efficiency and low processor utilization in existing similar technical solutions, the present invention proposes a real-time AVS soft coding method for Intel mobile platforms.

In this method, the camera frame is firstly developed by using the Android application development technology. Then use NDK technology to port the AVS encoder to the Android camera project. Referring to the AVS video coding principle and technology, the algorithm optimization of the AVS encoder is completed; using the program optimization technology, the multi-thread optimization of the camera program and the structure optimization and code optimization of the AVS encoder C program are completed. Finally, use a processor-specific C compiler and use the SSE instruction set supported by the processor to optimize calculations and improve processor utilization.

The key of AVS video coding is to the coding of I, P, B three kinds of pictures (the optimization that this method involves has only considered I picture and P picture, does not involve B picture). The coding of I picture and P picture is based on macroblock, and all key technologies are embodied in the coding process of one macroblock. The optimization of macroblock coding is introduced below.

Technical scheme of the present invention is:

A kind of real-time AVS soft coding method for Intel mobile platform, its step is:

1) AVS encoder detects the type of the macroblock to be encoded at present, if it is an I frame macroblock, then calls the I frame coding unit to encode; if it is a P frame macroblock, then calls the P frame coding unit to encode;

2) The P-frame encoding unit performs inter-frame prediction on the current P-frame macroblock, selects the inter-frame macroblock mode with the smallest inter-block absolute error and SAD as the best inter-frame macroblock mode, and obtains prediction in this inter-frame macroblock mode The macroblock is used as the best predicted macroblock; the residual is calculated with the current P frame macroblock, and the resulting residual matrix is transformed, quantized, and entropy encoded;

3) The I-frame encoding unit performs intra-frame prediction on the current I-frame macroblock, obtains the best intra-frame prediction block of the current I-frame macroblock, performs residual calculation with the current I-frame macroblock, and calculates the obtained residual The matrix is transformed, quantized, and entropy encoded; wherein, during each traversal of the intra prediction mode, it is judged whether the rate-distortion cost of the 8x8 block calculated in the current intra prediction mode is less than the average value of the rate-distortion cost, and if so, it is As the best intra frame prediction block of the current I frame macroblock, the traversal is terminated;

The average value of the rate-distortion cost=the sum of the rate-distortion costs of all calculated intra-frame prediction modes for 8×8 blocks/the number of calculated intra-frame prediction modes.

Further, the method for obtaining the best inter-frame macroblock mode of the current P-frame macroblock is as follows: for each type of inter-frame macroblock mode inter-frame prediction, when performing inner-layer cycle reference frame traversal, reference frame traversal After the first frame, if the cost corresponding to the current sub-block in the current inter-frame macroblock mode is less than the previous minimum cost, the current reference frame will be used as the best reference frame for the current sub-block in the current inter-frame macroblock mode. At this time, the current sub-block The cost corresponding to the block is taken as the minimum cost, and the inner loop reference frame traversal is terminated; for the outer inter macroblock mode traversal, after the inter macroblock mode has traversed the first type, if the total cost in the current inter macroblock mode is less than the previous minimum total cost, the current inter-frame macroblock mode is taken as the best inter-frame macroblock mode for the current P-frame macroblock.

Further, the calculation method of the sum of absolute errors (SAD) between blocks is as follows: all the blocks to be encoded are divided into units of 8x8 blocks, and then the _mm_loadl_epi64() instruction in the SSE technology is used to obtain the 8x8 blocks to be encoded and the predicted 8x8 A whole row of pixel values in each block, and then use the _mm_sad_epu8() instruction in the SSE technology to calculate the absolute error sum of the two rows of pixel values, and use the _mm_extract_epi16() instruction in the SSE technology to extract the absolute error sum and accumulate it to the total absolute error sum; when all rows are traversed, the absolute error sum between the 8x8 block to be encoded and the predicted 8x8 block is obtained.

Further, the inter-frame macroblock modes include four inter-frame macroblock modes: 16x16, 16x8, 8x16, and 8x8.

Further, the UMHexagons motion estimation algorithm is used for inter-frame prediction.

Further, the intra-frame prediction includes: mean value prediction, horizontal prediction, vertical prediction, planar prediction, bottom-left diagonal prediction, and bottom-right diagonal prediction.

The beneficial effect of this method is:

Improve the efficiency of AVS encoding on the mobile platform, enabling the camera to complete real-time acquisition and encoding at 10 frames per second, and improve processor utilization. The following is combined with the test case for analysis.

Run the camera application equipped with the pre-optimized AVS encoder and the camera application equipped with the optimized AVS encoder on the mobile device respectively, and obtain test data by encoding three different YUV files. Analyze the test data to obtain the test data analysis results.

The mobile device configuration is shown in Table 1.

Table 1 Mobile device configuration

project value target Android4.2.2–API Level16 CPU/ABI INTEL (x86-atom) SD Card support yes

SD Card size 10GB Camera Facing Front support yes Camera Facing Back support yes Keyboard support yes

The detailed parameters of the three YUV test files are shown in Table 2:

Table 2 YUV test file detailed parameters

Use the optimized and optimized AVS encoders to encode three YUV test files respectively, and the number of encoded frames is 100 frames. The output video bit rate, the average signal-to-noise ratio of the three components of Y, U, and V, and the encoding time are obtained. The test data for the three YUV test files are shown in Table 3, Table 4, and Table 5.

Table 3 Test data of YUV test file

Table 4 Test data of YUV test file

Table 5 Test data of YUV test file

Comprehensive test data analysis, after optimization compared with before optimization: the encoding time is saved by more than 80% compared with before optimization, the average signal-to-noise ratio is not lower than 98% before optimization, the output file bit rate is not higher than 172% before optimization, and the frame rate Basically reach 10fps.

Description of drawings

The method will be further described below in conjunction with the accompanying drawings and embodiments.

Figure 1 is a flow chart of macroblock encoding.

Fig. 2 is a flowchart of inter-frame prediction before optimization.

Fig. 3 is a flowchart of optimized inter-frame prediction.

Fig. 4 is a flow chart of selecting the best intra prediction mode for an 8x8 block before optimization.

Fig. 5 is a flow chart of selecting the best intra prediction mode for an optimized 8x8 block.

Fig. 6 is the flow chart of absolute error and calculation before optimization.

Fig. 7 is a flow chart of absolute error and calculation after optimization.

Fig. 8 is a schematic diagram of the _mm_loadl_epi64() instruction.

Fig. 9 is a schematic diagram of the _mm_sad_epu8() instruction.

Fig. 10 is a schematic diagram of the _mm_extract_epi16() instruction.

detailed description

Compile the optimized AVS video encoder with a processor-specific C compiler, making it an external reference library for Android camera application projects. Compile the camera application and generate the camera application. Install the Android camera application corresponding to this method on the mobile platform. Click the app icon to open the app. Click the "Start Recording" button, the program enters the recording state. During the recording process, the program processes each frame of image: the mobile platform AVS camera application captures a frame of YUV format image and passes it to the optimized AVS encoder C program. The AVS encoder C program completes the AVS encoding of the YUV image, and finally saves the encoded AVS image data to the SD memory card. After the recording is completed, click the "End Recording" button, and the program enters the recording end state. The recorded video files are stored in the SD memory card of the Android mobile platform.

The main process of AVS macroblock coding is shown in Figure 1.

For I-frame macroblocks, the encoding steps are mainly: intra-frame chroma prediction, intra-frame luma prediction, calculating the rate-distortion cost in all intra-frame prediction modes, and selecting the intra-frame prediction mode with the smallest rate-distortion cost as the best intra-frame Prediction mode, setting macroblock coding parameters, transformation, quantization, entropy coding.

For P-frame macroblocks, the encoding steps mainly include: inter-frame prediction, selection of the best inter-frame macroblock mode, setting macroblock encoding parameters, transformation, quantization, and entropy encoding.

The following points need to be explained:

1. The macroblock coding parameters include the residual matrix of the macroblock to be coded and the predicted macroblock, the optimal macroblock coding mode parameter, and the like.

2. For the intra-frame mode, since the intra-frame macroblock mode is fixed (8x8), only several intra-frame prediction modes (average prediction, horizontal prediction, vertical prediction, planar prediction, bottom-left diagonal prediction, bottom-right In the diagonal prediction), the key parameters of the best intra-macroblock coding mode can be determined by selecting the best intra-frame prediction mode according to the value of the rate-distortion cost.

3. For the inter-frame mode, since the inter-frame prediction algorithm is fixed (UMHexagons motion estimation algorithm is used in this method), it only needs to be selected from several inter-frame macroblock modes (16x16, 16x8, 8x16, 8x8, SKIP) The best inter-frame macroblock mode can determine the key parameters of the best inter-frame macroblock coding mode.

4. Before encoding the macroblock, the AVS encoder will detect the type of the macroblock to be encoded. If it is an I-frame macroblock, it will call the I-frame encoding unit for encoding; if it is a P-frame macroblock, it will call the P-frame encoding unit to encode.

5. In the "Inter Prediction" section, the sum of absolute errors (SAD) of the block to be encoded and the predicted block is used as the characterization of the cost, and the corresponding inter macroblock mode is selected as the best inter macroblock according to the minimum value of SAD model.

Inter prediction optimization:

The purpose of inter-frame prediction is to find the best inter-frame prediction block for the current P-frame macroblock to be coded, so that the current P-frame macroblock to be coded can make a residual with it, and the residual matrix is transformed, quantized, entropy coded, etc. encoding operations.

Through the execution of inter-frame prediction, the following parameters need to be clarified: the best reference frame, motion vector and minimum cost of all sub-blocks in a specific inter-frame macroblock mode, the best inter-frame macroblock mode and the total minimum cost in this mode .

In this method, the inter-frame prediction mainly includes the inter-frame prediction under the four inter-frame macroblock modes (16x16, 16x8, 8x16, 8x8) and the inter-frame prediction under the Skip mode (wherein, the inter-frame prediction under the Skip mode is relatively independent , not within the scope of optimization).

The main body of the inter-frame prediction under the four inter-frame macroblock modes (16x16, 16x8, 8x16, 8x8) is a two-layer loop, from the outside to the inside is the traversal of the four inter-frame macroblock modes and the current inter-frame macroblock mode block traversal. In the inner loop, first perform the motion search of the current sub-block in the current inter-macroblock mode, and then find the best reference frame and the minimum cost of the current sub-block in the current inter-macroblock mode by completing the traversal of the reference frames. Before the traversal of each inter-frame macroblock mode ends, the minimum cost of all sub-blocks of the current inter-frame macroblock mode obtained by the inner loop is summed to obtain the total cost of the current inter-frame macroblock mode, and then the previous minimum total cost The best inter-macroblock mode of the macroblock to be encoded and the minimum total cost in the macroblock mode are updated. The process is shown in Figure 2.

The optimization of inter prediction under four inter macroblock modes (16x16, 16x8, 8x16, 8x8) uses the idea of early termination. On the basis of keeping the original structure basically unchanged, two early terminations were added. The first is added to the inner loop reference frame traversal to terminate the reference frame traversal early. The second part is added to the traversal of the four inter-frame macroblock modes in the outer layer, and is used to terminate the traversal of the four inter-frame macroblock modes in advance. The condition for the first early termination is: after the reference frame has traversed the first frame, and under the current reference frame condition, the cost corresponding to the current sub-block in the current inter-frame macroblock mode is less than the current inter-frame macroblock mode under the previous reference frame condition. The minimum cost of a subblock. The second condition for early termination is: after the inter-macroblock mode has traversed the first type, and the total cost in the current inter-macroblock mode is smaller than the minimum total cost in the previous inter-macroblock mode. Its process is shown in Figure 3. The guiding ideas of the two early terminations are: only need to find a better reference frame or a better inter-frame macroblock mode, instead of finding the optimal one, which can improve the execution efficiency of the program.

Intra prediction optimization:

The intra-frame prediction process corresponding to the I-frame macroblock is as described above, including: intra-frame chroma prediction, intra-frame luma prediction, calculating the rate-distortion cost in all intra-frame prediction modes, and selecting the intra-frame prediction with the smallest rate-distortion cost The modes are optimal intra prediction mode, setting macroblock encoding parameters, transformation, quantization, and entropy encoding.

The unit of intra prediction is an 8x8 block. In the original process, it is necessary to traverse all intra prediction modes (mean prediction, horizontal prediction, vertical prediction, planar prediction, lower left diagonal prediction, lower right diagonal prediction) to determine the current 8x8 block in all intra prediction modes rate-distortion cost, and then select the intra-frame prediction mode with the smallest rate-distortion cost as the best intra-frame prediction mode. Its process is shown in Figure 4.

The optimization of this part of the operation uses the idea of early termination. Under the premise of keeping the original structure of the optimal intra-frame prediction mode selection of 8x8 blocks unchanged, a judgment is added in each intra-frame prediction mode traversal. If the judgment is true, the best intra-frame prediction mode of 8x8 blocks is updated and Terminates the walk through all intra prediction modes early. The judgment is: judging whether the rate-distortion cost of the 8x8 block calculated in the current intra-frame prediction mode is smaller than the average value of the rate-distortion cost.

The calculation method of the mean value of the rate-distortion cost is:

Average rate-distortion cost = sum of rate-distortion costs of all calculated intra-frame prediction modes for 8x8 blocks/number of calculated intra-frame prediction modes.

After the optimization is completed, the number of traversals to the intra-frame prediction mode is reduced, and the program execution efficiency is improved. Its process is shown in Figure 5.

AVS video encoder SSE optimization:

Install a processor-specific C compiler that supports the SSE instruction set. Replace the C language instructions in the inter-frame prediction SAD calculation part with appropriate SSE instructions, so as to complete the optimization and improve the program execution efficiency and processor utilization.

The object of SSE optimization is the calculation part of the sum of absolute errors (SAD) between two blocks in inter-frame prediction. Its calculation formula is:

∑ _i=0 ^i<M |P(i)-Q(i)|,

Where M is the total number of block pixels, P(i) is the Y component value of the i pixel point of the first block, and Q(i) is the Y component value of the i pixel point of the second block.

The code before optimization is all expressed in C language. The specific implementation method is to perform two-layer loop according to the X and Y coordinates of the pixels in the block. The innermost loop calculates the absolute error of the current position and adds it to the absolute error sum. Finally, the purpose of calculating the sum of the absolute errors of all pixels is achieved. The optimization is made when the X coordinate is cycled, and the calculation of four consecutive pixels is taken as a group, which is convenient to use the pipeline of the processor. Its process is shown in Figure 6.

The idea of optimization is to divide all incoming 16x16, 16x8, 8x16 or 8x8 blocks into units of 8x8 blocks, and then process the 8x8 blocks in turn. In the processing of 8x8 blocks, several functions such as _mm_loadl_epi64(), _mm_sad_epu8(), _mm_extract_epi16() in SSE technology are used. First, use the _mm_loadl_epi64() instruction to obtain a whole row of pixel values in the 8x8 block to be encoded and the predicted 8x8 block, and then use the _mm_sad_epu8() instruction to calculate the absolute error sum of the two rows of pixel values, and finally use the _mm_extract_epi16() instruction to extract Just compute the sum of absolute errors and add it to the total sum of absolute errors. After traversing all rows, the sum of absolute errors between the 8x8 block to be coded and the predicted 8x8 block is obtained. Its process is shown in Figure 7.

Several functions used in the optimization process are explained in detail as follows.

(1) s0=_mm_loadl_epi64((__m128i*)ref_block):

As shown in Figure 8, this instruction extracts the 64-bit (8-byte) data pointed to by the ref_block pointer and stores it in the lower 64 bits of s0 (the total length is 128 bits).

(2) s2=_mm_sad_epu8(s0,s1):

As shown in Figure 9, this instruction calculates the absolute error and sums the lower 64 bits of S0 and S1 (both data lengths are 128 bits) in bytes (8 bits), and sums the obtained absolute error sum (SAD) is stored in 0-15 bits of S2 (stored as a word); at the same time, the upper 64 bits of S0 and S1 are calculated and summed in units of bytes (8 bits), and the obtained absolute The sum of errors (SAD) is stored in bits 64-79 of S2 (stored as a word). This function only uses the low 64-bit sum of absolute errors (SAD).

(3) mcost+=_mm_extract_epi16(s2,0):

As shown in Figure 10, this instruction extracts the lower 16-bit data of S2 (that is, the absolute error sum of the required lower 64 bits of S1 and S0 in bytes) and accumulates it to the mcost variable of type int middle. _mm_extract_epi16(__m128i, int) determines the position of the extracted data in S2 through the second parameter (0 means 0-15 bits, 1 means 16-31 bits, and so on), and the extraction unit is 16 bits (one word).

Claims (5)

1. A kind of real-time AVS soft coding method for Intel mobile platform, its step is: 1) AVS encoder detects the type of the macroblock to be encoded at present, if it is an I frame macroblock, then calls the I frame coding unit to encode; if it is a P frame macroblock, then calls the P frame coding unit to encode; 2) The P-frame encoding unit performs inter-frame prediction on the current P-frame macroblock, selects the inter-frame macroblock mode with the smallest inter-block absolute error and SAD as the best inter-frame macroblock mode, and obtains prediction in this inter-frame macroblock mode The macroblock is used as the best predicted macroblock; the residual is calculated with the current P frame macroblock, and the resulting residual matrix is transformed, quantized, and entropy encoded; 3) The I-frame encoding unit performs intra-frame prediction on the current I-frame macroblock, obtains the best intra-frame prediction block of the current I-frame macroblock, performs residual calculation with the current I-frame macroblock, and calculates the obtained residual The matrix is transformed, quantized, and entropy encoded; wherein, during each traversal of the intra prediction mode, it is judged whether the rate-distortion cost of the 8x8 block calculated in the current intra prediction mode is less than the average value of the rate-distortion cost, and if so, it is As the best intra frame prediction block of the current I frame macroblock, the traversal is terminated; The average value of the rate-distortion cost = the sum of the rate-distortion costs of all calculated intra-frame prediction modes for 8x8 blocks/the number of calculated intra-frame prediction modes; Wherein, the method for obtaining the best inter-frame macroblock mode of the macroblock of the current P frame is: in the inter-frame prediction of each inter-frame macroblock mode, when performing inner-layer cycle reference frame traversal, the reference frame traversed After the first frame, if the cost corresponding to the current sub-block in the current inter-macroblock mode is less than the previous minimum cost, the current reference frame is used as the best reference frame for the current sub-block in the current inter-macroblock mode. At this time, the current sub-block The corresponding cost is taken as the minimum cost, and the inner loop reference frame traversal is terminated; for the outer inter macroblock mode traversal, after the inter macroblock mode has traversed the first type, if the total cost in the current inter macroblock mode is less than If the previous minimum total cost is used, the current inter-frame macroblock mode is used as the best inter-frame macroblock mode for the current P-frame macroblock. 2. The method according to claim 1, wherein the calculation method of the absolute error between the blocks and the SAD is: all blocks to be coded are all divided into units of 8x8 blocks, and then _mm_loadl_epi64 () in the SSE technology is adopted The instruction obtains a whole row of pixel values in the 8x8 block to be encoded and the predicted 8x8 block, and then uses the _mm_sad_epu8() instruction in the SSE technology to calculate the absolute error sum of the two rows of pixel values, and uses the _mm_extract_epi 16() in the SSE technology The instruction extracts the sum of absolute errors and adds it to the total sum of absolute errors; when all rows are traversed, the sum of absolute errors between the 8x8 block to be encoded and the predicted 8x8 block is obtained. 3. The method according to claim 1, wherein the inter-frame macroblock modes include four inter-frame macroblock modes: 16x16, 16x8, 8x16, and 8x8. 4. The method according to claim 1, characterized in that UMHexagons motion estimation algorithm is used for inter-frame prediction. 5. The method according to claim 1, wherein the intra-frame prediction comprises: mean value prediction, horizontal prediction, vertical prediction, planar prediction, bottom-left diagonal prediction, and bottom-right diagonal prediction.