CN103442226B - The multiple views color video fast encoding method of distortion just can be perceived based on binocular - Google Patents

Abstract

The invention discloses a multi-view point color video fast encoding method based on binocular just detectable distortion, which firstly uses the disparity information of the left view point video and the right view point video to determine the non-boundary of each frame of the right view point image in the right view point video The binocular just perceptible distortion value of each macroblock in the area, and secondly terminate the macroblock mode selection in advance according to the size of the binocular just perceptible distortion value. This fast coding method can achieve Effectively improve the coding efficiency of multi-viewpoint color video, the saving of coding time can reach 66.48% to 71.90%, and the average saving of coding time is 68.46%.

Description

Multi-view color video fast coding method based on binocular just perceivable distortion

technical field

The invention relates to a method for processing multi-viewpoint color video signals, in particular to a fast encoding method for multi-viewpoint color video based on binocularly detectable distortion.

Background technique

3D and free-viewpoint television make extensive use of multi-viewpoint color video for scene description. Multi-viewpoint color video includes multiple viewpoints in color, and after encoding, transmission, and decoding, virtual viewpoint rendering is performed on the display terminal. In multi-view color plus depth video, multi-view color video coding has been extensively studied, and the more applicable coding platforms include joint multi-view video model and joint multi-view video coding. However, because the research on the characteristics of human vision is developing, the use of the perceptual characteristics of the human visual system in the compression of multi-viewpoint color video needs further research.

At present, among the many characteristics of the human visual system, just perceptible distortion is one of the characteristics that most researchers tend to. Just perceivable distortion represents the visible threshold that the human eye can perceive changes in image pixels when watching an image, and it mainly depends on the brightness and contrast of the image. Liu et al. utilize a just perceptible distortion model to distinguish the border and texture regions of images. Recently, some researches on the visible threshold of human eyes perceiving 3D images and videos, such as depth just perceptible distortion and binocular just perceptible distortion, are popular. Depth just perceivable distortion represents the minimum perceivable threshold in the depth video, so the pixels below the threshold in the depth video can be further compressed. Binocular just perceptible distortion is a model obtained based on binocular brightness masking and contrast masking experiments, which means that when the distortion of an image or video at one viewpoint is lower than binocular just perceptible distortion, binoculars will not perceive the distortion. Distortion of images or videos.

In order to further compress the huge data volume of multi-viewpoint color plus depth video, the applicable coding platform adopts full search mode selection to determine the minimum rate-distortion cost of macroblocks, so as to determine the best prediction mode. In view of the high computational complexity of the full search mode, researchers have proposed some fast mode selection algorithms. Shen et al. proposed a low-complexity mode selection algorithm, including four effective mode selection techniques, pre-determined SKIP mode, adaptive early termination, fast mode size selection and selective intra-frame coding methods, this method is effective It greatly saves the encoding time of multi-viewpoint color video, and at the same time can maintain almost the same encoding result as the full search mode selection. Zeng et al. used the relationship between the quantization step size and the rate-distortion cost as a threshold, and calculated the motion vector of the current block using the motion vectors of adjacent blocks. The above methods can effectively save coding complexity without reducing the coding quality. However, in the coding process of multi-viewpoint color video, the visual characteristics of the human eye cannot be fully utilized in these methods. Viewpoint color video still has a lot of room for research.

Contents of the invention

The technical problem to be solved by the present invention is to provide a fast encoding method for multi-viewpoint color video based on binocular just detectable distortion, which can effectively reduce the encoding time of multi-viewpoint color video while maintaining the performance of reconstructed viewpoint video.

The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a multi-viewpoint color video fast encoding method based on binocular just detectable distortion, which is characterized in that it includes the following steps:

①Record the left view video of the multi-viewpoint color video as {C _L (k)}, and record the right view video of the multi-viewpoint color video as {C _R (k)}, where C _L (k) means {C _L (k)} in the k-th frame of the left-view image, C _R (k) represents the k-th frame of the right-view image in {C _R (k)}, 1≤k≤K, K represents the left-view video and the right-view video The number of frames of the image contained in;

② Divide each frame of right-view image in {C _R (k)} into boundary area and non-boundary area, where the boundary area consists of the first row of macroblocks, the last row of macroblocks, and the first column of macroblocks in the right-view image. block and the last column of macroblocks; then calculate the binocular perceivable distortion value of each macroblock in the non-boundary area of each frame of right-view image in {C _R (k)}, assuming {C _R (k )} in the k-th frame of right-viewpoint image C _R (k) in the coordinate position (i, j) of the macroblock belongs to {C _R (k)} in the k-th frame of right-viewpoint image C _R (k) in In the non-boundary area, the binocular just perceivable distortion value of the macroblock at the coordinate position (i, j) in the k-th frame of the right view image C _R (k) in {C _R (k)} is recorded as Among them, 2≤i≤W/16-1, 2≤j≤H/16-1, W represents each frame of left view image in { _CL (k)} and each frame in { _CR (k)} The width of the right view image, H represents the height of each frame of the left view image in { _CL (k)} and the height of each frame of the right view image in {C _R (k)};

③ On the multi-view video coding verification model JMVC, use the HBP predictive coding structure to pair each macroblock in each frame of the left view image in { _CL (k)} and each frame in { _CR (k)} Each macroblock in the right view image is encoded, and the process of selecting the optimal macroblock encoding mode for each macroblock in the encoding process is:

③-1. Define the current macroblock to be encoded as the current macroblock;

③-2. When the current macroblock is a macroblock in { _CL (k)}, the encoder uses the H.264 mode selection process to search for SKIP, Inter16×16, Inter16×8, Inter8×16, Inter8×8 , Inter8×8Frext, Intra16×16, Intra8×8, and Intra4×4 macroblock coding modes, select the macroblock coding mode with the smallest rate-distortion cost from these macroblock coding modes as the optimal macroblock coding mode for the current macroblock to encode;

When the current macroblock is a macroblock in { _CR (k)}, it is judged whether the current macroblock belongs to the boundary area or the non-boundary area. If the current macroblock belongs to the boundary area, the encoder adopts the mode selection process of H.264 Search for SKIP, Inter16×16, Inter16×8, Inter8×16, Inter8×8, Inter8×8Frext, Intra16×16, Intra8×8, and Intra4×4 macroblock coding modes, and select rate-distortion from these macroblock coding modes The macroblock coding mode with the least cost is used as the optimal macroblock coding mode for the current macroblock; if the current macroblock belongs to the non-boundary area, then judge whether the binocular detectable distortion value of the current macroblock is greater than or equal to the set If yes, the encoder uses the H.264 mode selection process to search for SKIP, Inter16×16, Inter16×8, Inter8×16, Inter8×8, Inter8×8Frext, Intra16×16, Intra8×8 and Intra4 ×4 macroblock coding mode, select the macroblock coding mode with the smallest rate-distortion cost from these macroblock coding modes as the optimal macroblock coding mode of the current macroblock for coding; otherwise, the encoder adopts the mode selection of H.264 The process only searches for SKIP and Inter16×16 macroblock coding modes, and selects the macroblock coding mode with the smallest rate-distortion cost from these two macroblock coding modes as the optimal macroblock coding mode of the current macroblock for coding;

③-3. Use the next macroblock to be encoded as the current macroblock, and then return to step ③-2 to continue until each macroblock in each frame of the left view image in { _CL (k)} and {C Each macroblock in each frame of right-view image in _R (k)} is coded.

The binocular just perceivable distortion value of the macroblock whose coordinate position is (i, j) in the k-th frame of the right viewpoint image C _R (k) in {C _R (k)} in the step ② Among them, d represents the disparity value of the macroblock whose coordinate position is (i, j) in the k-th frame right view image C _R (k) in {C _R (k)}, bg _L (i+d, j) Represents the average value of the luminance values of all pixels in the macroblock whose coordinate position is (i+d,j) in the k-th frame left-viewpoint image _CL (k) in { _CL (k)}, bg _L ( i+d,j)∈[0,255], bg _L (i×16+d×16+m, j×16+n) means that the coordinate position in the kth frame left viewpoint image C _L (k) in _{ CL (k)} is (i×16+d ×16+m, j×16+n), 0≤m≤15, 0≤n≤15, eh _L (i+d, j) represents the luminance value of the pixel in {C _L (k)} The average value of the edge intensity values of all pixels in the macroblock whose coordinate position is (i+d, j) in the left view point image C _L (k) of k frames, bg _L (i×16+d×16+m-3+h, j×16+n-3+v) represents the coordinates in the k-th frame left viewpoint image C _L (k) in _{ CL (k)} The brightness value of the pixel at the position (i×16+d×16+m-3+h, j×16+n-3+v), G _H (h, v) represents the horizontal Sobel operator of 5×5 The element at the coordinate position (h, v) in G _H , G _V (h, v) represents the element at the coordinate position (h, v) in the vertical Sobel operator G _V of 5×5, _{1≤h≤5,1≤v≤5} , A limit (bg _L (i+d,j),eh _L (i+d,j))＝A _limit (bg _L (i+d,j))+ K(bg _L (i+d,j))×eh _L (i+d,j), K(bg _L (i+d,j))＝-10 ^-6 ×(0.7×(bg _L (i+d,j)) ² +32×bg _L (i+d,j))+0.07，λ Indicates the parameter to control the influence of right viewpoint noise, n _L (i+d,j) indicates that the coordinate position in the kth frame left viewpoint image C _L (k) in _{ CL (k)} is (i+d,j) The noise magnitude of the macroblock.

The parameter λ for controlling the influence of noise on the right viewpoint takes a value of 1.25.

The noise magnitude n _{L (i+d, j) of the macroblock whose coordinate position is (i+d, j) in the k-th frame left view image C L ( k} ) in the {CL (k)} The value is 0.3.

The determination threshold set in step ③-2 is 5.

Compared with the prior art, the present invention has the advantages that: firstly, the binocular value of each macroblock in the non-boundary area of each frame of the right-view image in the right-view video is determined by using the disparity information of the left-view video and the right-view video. Just detectable distortion value, and secondly terminate the macroblock mode selection in advance according to the size of binocular just detectable distortion value. This fast coding method can effectively improve the coding efficiency of multi-viewpoint color video without causing rate-distortion performance degradation. , the saving of coding time can reach 66.48% to 71.90%, and the average saving of coding time is 68.46%.

Description of drawings

Fig. 1 is the block flow diagram of the inventive method;

Figure 2a is the first frame color image of the color video sequence corresponding to the fourth viewpoint in the "Book Arrival" test video sequence;

Figure 2b is the color image of the first frame of the color video sequence corresponding to the fifth viewpoint in the "Book Arrival" test video sequence;

Fig. 2c is the image formed after the binocular just detectable distortion value of each macroblock in the non-boundary area in the color image shown in Fig. 2b is enlarged by 20 times;

Figure 3a shows the non-boundary area in the right view image of the multi-view color video sequence "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", "Xmas9", for the same For all macroblocks with binocular just perceivable distortion values, the percentage of the number of macroblocks whose SKIP mode is selected as the optimal macroblock coding mode accounts for the number of all macroblocks with the same binocular just perceivable distortion value;

Figure 3b shows the non-boundary area in the right view image of the multi-view color video "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", "Xmas9" sequences, for the same For all macroblocks with binocular just perceivable distortion values, the percentage of the number of macroblocks for which Inter16×16 mode is selected as the optimal macroblock coding mode to the number of all macroblocks with the same binocular just perceivable distortion value;

Figure 3c shows the non-boundary area in the right view image of the multi-view color video "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", "Xmas9" sequences, for the same For all macroblocks with binocular just perceivable distortion values, the percentage of the number of macroblocks for which Inter16×8 mode is selected as the optimal macroblock coding mode to the number of all macroblocks with the same binocular just perceivable distortion value;

Figure 3d shows the non-boundary areas in the right view images of the multi-view color video sequences "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9". For all macroblocks with binocular just perceivable distortion values, the percentage of the number of macroblocks for which Inter8×16 mode is selected as the optimal macroblock coding mode to the number of all macroblocks with the same binocular just perceivable distortion value;

Figure 3e shows the non-boundary areas in the right view images of the multi-view color video sequences "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9". For all macroblocks with binocular just perceivable distortion values, the percentage of the number of macroblocks for which the Inter8×8 mode is selected as the optimal macroblock coding mode to the number of all macroblocks with the same binocular just perceivable distortion value;

Figure 3f shows the non-boundary areas in the right view images of the multi-view color video sequences "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9". For all macroblocks whose binocular distortion value can be detected, select Others (including Inter8×8Frext, Intra16×16, Intra8×8 and Intra4×4) mode as the optimal macroblock coding mode. The number of macroblocks with the same binocular correctness Percentage of the number of all macroblocks with perceivable distortion values;

Fig. 4a is the coding rate-distortion performance comparison diagram of the multi-viewpoint color video sequence obtained after the "Exit" test sequence is respectively encoded by the multi-viewpoint standard verification platform and the method of the present invention;

Fig. 4 b is the coding rate-distortion performance comparison diagram of the multi-viewpoint color video sequence obtained after the "Vassar" test sequence is encoded by the multi-viewpoint standard verification platform and the method of the present invention respectively;

Fig. 4c is a comparison chart of the encoding rate-distortion performance of the multi-viewpoint color video sequence obtained after the "Champagne tower" test sequence is coded by the multi-viewpoint standard verification platform and the method of the present invention respectively;

Figure 5a is the 10th frame image of the second viewpoint reconstructed by using the multi-viewpoint standard verification platform in the "Exit" test sequence;

Fig. 5b is the 10th frame image of the second viewpoint reconstructed by the fast encoding method of the present invention in the "Exit" test sequence;

Figure 5c is the 10th frame image of the second viewpoint reconstructed by the multi-viewpoint standard verification platform of the "Vassar" test sequence;

Figure 5d is the 10th frame image of the second viewpoint reconstructed by the fast encoding method of the present invention in the "Vassar" test sequence;

FIG. 6 is a schematic diagram of the relationship between the coordinate position of a macroblock and the coordinate position of a pixel in the macroblock.

detailed description

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

Since the multi-viewpoint color video coding does not fully utilize the human visual perception characteristic compression, there is a large amount of redundancy in the multi-viewpoint color video, so in order to further reduce the coding time of the multi-viewpoint color video without affecting the human visual perception , the present invention proposes a fast coding method for multi-viewpoint color video based on binocular just perceivable distortion, which calculates the corresponding binocular just perceivable distortion for the macroblocks in the right-viewpoint video before encoding the multi-viewpoint color video value to reduce encoding time for encoding multiview color video.

A fast encoding method for multi-viewpoint color video based on binocular just detectable distortion proposed by the present invention, its flow chart is shown in Figure 1, which includes the following steps:

①Record the left view video of the multi-viewpoint color video as {C _L (k)}, and record the right view video of the multi-viewpoint color video as {C _R (k)}, where C _L (k) means {C _L (k)} in the k-th frame of the left-view image, C _R (k) represents the k-th frame of the right-view image in {C _R (k)}, 1≤k≤K, K represents the left-view video and the right-view video The number of frames of the image contained in .

②Because there are occlusion exposure areas in the left view image and right view image, the disparity value between the left view image and the right view image cannot be calculated, and the occlusion exposure area is generally located in the boundary area between the left view image and the right view image, so in When calculating the binocular just perceptible distortion value of the macroblocks in each frame of the right view image in {C _R (k)}, it is necessary to remove the border area of the right view image (the top row and the bottom row of macroblocks and the leftmost and The rightmost column of macroblocks), that is, the specific process is: divide each frame of right-viewpoint image in {C _R (k)} into boundary area and non-boundary area, where the boundary area is defined by the first row of macroblocks in the right-viewpoint image block, the last row of macroblocks, the first column of macroblocks and the last column of macroblocks. The non-boundary area is composed of the abscissa in the right view image within the range of 2≤i≤W/16-1 and the ordinate in the range of 2≤j≤ All macroblocks in the range of H/16-1 are composed; then calculate the binocular perceivable distortion value of each macroblock in the non-boundary area of each frame of right view image in { _CR (k)}, assuming The macroblock whose coordinate position is (i, j) in the k-th frame of right-viewpoint image C _R (k) in { _{CR (k)} belongs to the k-th frame of right-viewpoint image C R in} { _CR (k)} In the non-boundary area in (k), the binocular distortion of the macroblock whose coordinate position is (i, j) in the k-th frame of right view image C _R (k) in {C _R (k)} is just perceivable value recorded as Among them, 2≤i≤W/16-1, 2≤j≤H/16-1, W represents each frame of left view image in { _CL (k)} and each frame in { _CR (k)} The width of the right view image, H represents the height of each frame of the left view image in {C _L (k)} and the height of each frame of the right view image in {C _R (k)}, d represents the height of each frame of {C _R (k)} The disparity value of the macroblock whose coordinate position is (i,j) in the k-th frame of the right view image C _R (k) of , using the existing disparity estimation algorithm based on the Sum of Square Differences (SSD) Calculate the disparity between each macroblock in the non-boundary area of each frame of right view image in {C _R (k)} and the corresponding macro block in the left view image at the same moment in {C _L (k)} value, bg _L (i+d, j) represents all pixels in the macroblock whose coordinate position is (i+d, j) in the k-th frame left view image C _L (k) in _{ CL (k)} The average value of the brightness value of the point, bg _L (i+d,j)∈[0,255], bg _L (i×16+d×16+m,j×16+n) means that the coordinate position in the kth frame left viewpoint image C _L (k) in _{ CL (k)} is (i×16+d ×16+m, j×16+n) pixel brightness value, Figure 6 shows the schematic diagram of the relationship between the coordinate position of the macroblock and the coordinate position of the pixel point in the macroblock, 0≤m≤15, 0≤n≤15, eh _L (i+d,j) represents the macroblock whose coordinate position is (i+d,j) in the k-th frame left view image _CL (k) in { _CL (k)} The average of the edge intensity values of all pixels in bg _L (i×16+d×16+m-3+h, j×16+n-3+v) represents the coordinates in the k-th frame left viewpoint image C _L (k) in _{ CL (k)} The brightness value of the pixel at the position (i×16+d×16+m-3+h, j×16+n-3+v), G _H (h, v) represents the horizontal Sobel operator of 5×5 The element at the coordinate position (h, v) in G _H , G _V (h, v) represents the element at the coordinate position (h, v) in the vertical Sobel operator G _V of 5×5, _{1≤h≤5,1≤v≤5} , A limit (bg _L (i+d,j),eh _L (i+d,j))＝A _limit (bg _L (i+d,j))+ K(bg _L (i+d,j))×eh _L (i+d,j), K(bg _L (i+d,j))＝-10 ^-6 ×(0.7×(bg _L (i+d,j)) ² +32×bg _L (i+d,j))+0.07，λ Represents the parameter for controlling the influence of right viewpoint noise. In this embodiment, the value of λ is 1.25, and n _L (i+d, j) represents the kth frame left viewpoint image C _L (k) in { _CL (k)} The noise magnitude of the macroblock whose middle coordinate position is (i+d, j), in this embodiment, n _L (i+d, j) takes a value of 0.3.

Figure 2a shows the first frame of color image (left view image) corresponding to the 4th viewpoint in the "Book Arrival" test video sequence; Fig. 2b shows the 5th viewpoint in the "Book Arrival" test video sequence The first frame color image (right viewpoint image) corresponding to the color video sequence; because the H.264 encoding process is carried out based on the size of the macroblock, the present invention is for each non-boundary area in the color image shown in Figure 2b The macroblock calculates the binocular just perceptible distortion value, and since the just perceivable distortion value of each macroblock in the non-boundary area in the color image shown in Figure 2b is small, the human eye cannot directly view it, so for convenience Looking at the just perceivable distortion value of each macroblock in the non-boundary area in the color image shown in Figure 2b is magnified by 20 times, Figure 2c shows the value of each macroblock in the non-boundary area in the color image shown in Figure 2b The binoculars of a macroblock can just perceive the image formed by magnifying the distortion value by 20 times.

③ On the multi-view video coding verification model JMVC, use the HBP predictive coding structure to pair each macroblock in each frame of the left view image in { _CL (k)} and each frame in { _CR (k)} Each macroblock in the right view image is encoded, and the process of selecting the optimal macroblock encoding mode for each macroblock in the encoding process is:

③-1. Define the current macroblock to be encoded as the current macroblock;

③-2. When the current macroblock is a macroblock in { _CL (k)}, the encoder uses the existing H.264 mode selection process to search for SKIP, Inter16×16, Inter16×8, Inter8×16, Inter8×8, Inter8×8Frext, Intra16×16, Intra8×8 and Intra4×4 macroblock coding modes, select the macroblock coding mode with the smallest rate-distortion cost from these macroblock coding modes as the optimal macroblock for the current macroblock Block encoding mode for encoding;

When the current macroblock is a macroblock in { _CR (k)}, it is judged whether the current macroblock belongs to the boundary area or the non-boundary area. If the current macroblock belongs to the boundary area, the encoder uses the existing H.264 The mode selection process searches for SKIP, Inter16×16, Inter16×8, Inter8×16, Inter8×8, Inter8×8Frext, Intra16×16, Intra8×8 and Intra4×4 macroblock coding modes, and selects from these macroblock coding modes The macroblock coding mode with the smallest rate distortion cost is used as the optimal macroblock coding mode for the current macroblock; if the current macroblock belongs to the non-boundary area, then judge whether the binocular just detectable distortion value of the current macroblock is greater than or Equal to the set decision threshold, if yes, the encoder uses the existing H.264 mode selection process to search for SKIP, Inter16×16, Inter16×8, Inter8×16, Inter8×8, Inter8×8Frext, Intra16×16 , Intra8×8 and Intra4×4 macroblock coding modes, from these macroblock coding modes, select the macroblock coding mode with the smallest rate-distortion cost as the optimal macroblock coding mode for the current macroblock; otherwise, the encoder uses The existing H.264 mode selection process only searches for SKIP and Inter16×16 macroblock coding modes, and selects the macroblock coding mode with the smallest rate-distortion cost from these two macroblock coding modes as the optimal macroblock coding mode for the current macroblock. Block encoding mode for encoding;

③-3. Use the next macroblock to be encoded as the current macroblock, and then return to step ③-2 to continue until each macroblock in each frame of the left view image in { _CL (k)} and {C Each macroblock in each frame of right-view image in _R (k)} is coded.

In this embodiment, the value of the decision threshold set in step ③-2 is 5, and the specific value of the set decision threshold is obtained through experiments, and the experiment process is as follows:

③-2a. 46 frames in the left-viewpoint video of the sequence "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", "Xmas9" using multi-viewpoint color video, multi-viewpoint color For the 46 frames in the right view video sequence of the video "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", "Xmas9", calculate "Alt Moabit", "Balloons", " The binocular perceivable distortion value of each macroblock in the non-boundary area of each frame of the right view video in the right view video of the Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9" sequences , record the minimum and maximum values of binocular just perceivable distortion values of each macroblock in the non-boundary area of each frame of right view image as BJND_min and BJND_max respectively, and then count "Alt Moabit", "Balloons ", "Ballroom", "Kendo", "Race1", "Xmas3", "Xmas9" sequences in the 46 frames of right-view video images, the binocular just detectable distortion value is equal to the value of each value in the range from BJND_min to BJND_max The number of macroblocks is recorded as {N _{BJND_min} ,...,N _{BJND_max} } in the form of a set, where N _{BJND_min} represents a macroblock whose binocular distortion value is equal to BJND_min in 46 frames of images in a sequence of right-view video N _{BJND_max} represents the number of macroblocks whose binocularly detectable distortion value is equal to BJND_max in 46 frames of images in a sequence of right-view video.

③-2b. 46 frames of the left-viewpoint video of the sequence "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", "Xmas9" using multi-viewpoint color video, multi-viewpoint color For the 46 frames in the right-view video sequences of "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9", on the multi-view video coding verification model JMVC, adopt HBP predictive coding structure for multi-view color video "AltMoabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", "Xmas9" sequences of left-view video and multi-view color video "Alt Moabit" , "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9" sequences to encode right-view video, and obtain "Alt Moabit", "Balloons", "Ballroom", The optimal macroblock coding mode for all macroblocks in the 46 frames of images in the right-view video of the sequences "Kendo", "Race1", "Xmas3", and "Xmas9".

③-2c. For the multi-viewpoint color video "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", "Xmas9" sequences of the right-viewpoint video in the 46-frame image All macroblocks in the boundary area, where when the binocularly detectable distortion value of the macroblock is equal to a certain value BJND_x in the range from BJND_min to BJND_max, a value in the corresponding set {N _{BJND_min} ,...,N _{BJND_max} } , denoted as N _{BJND_x} , that is, the number of macroblocks whose perceivable distortion value is equal to a certain value BJND_x in the range from BJND_min to BJND_max is N _{BJND_x} , wherein, among these N _{BJND_x} macroblocks, the SKIP mode is selected as The number of macroblocks in the optimal macroblock coding mode is recorded as N _SKIP , the number of macroblocks in which the Inter16×16 mode is selected as the optimal macroblock coding mode is recorded as N _Inter16×16 , and the Inter16×8 mode is selected as the optimal macroblock coding mode The number of macroblocks is recorded as N _Inter16×8 , the number of macroblocks with Inter8×16 mode as the optimal macroblock coding mode is recorded as N _Inter8×16 , and the number of macroblocks with Inter8×8 mode as the optimal macroblock coding mode It is denoted as N _Inter8×8 , and the number of macroblocks that select the Others (including Inter8×8Frext, Intra16×16, Intra8×8 and Intra4×4) mode as the optimal macroblock coding mode is denoted as N _Others .

③-2d. Calculate the value of N _SKIP /N _{BJND_x} , which is recorded as P _SKIP , that is, among all macroblocks whose perceivable distortion value is equal to a certain value BJND_x in the range from BJND_min to BJND_max is N _{BJND_x} , select The SKIP mode is the percentage of the number N _SKIP of macroblocks in the optimal macroblock coding mode to the N _{BJND_x} macroblocks.

Figure 3a shows the non-boundary regions in the respective 46-frame right-view images of the multi-view color video sequences "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9" Inside, for all macroblocks with the same binocular just perceptible distortion value, the percentage of the number of macroblocks with the same binocular just perceivable distortion value as the number of macroblocks that select SKIP mode as the optimal coding mode, that is, Fig. 3a is a schematic diagram of the size change of P _SKIP in the right view images of the multi-view color video sequences "AltMoabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9", as shown in Figure 3a As shown, the value of P _SKIP is generally greater than 50%, and the percentage is relatively large, that is, for all macros whose distortion values are just in the range of BJND_min to BJND_max in the non-boundary area of the multi-viewpoint color video right-viewpoint video image Block, in the mode selection process of the existing H.264 encoding, the percentage of selecting the SKIP mode is relatively large, so for the non-boundary area of the multi-viewpoint color video right-viewpoint video image, the perceivable distortion value is between BJND_min and BJND_max All macroblocks within the range must be searched in SKIP mode.

Calculate the value of N _Inter16×16 /N _{BJND_x} , which is recorded as P _Inter16×16 , that is, among all the macroblocks whose perceivable distortion value is equal to a certain value BJND_x in the range from BJND_min to BJND_max is N _{BJND_x} , select The Inter16×16 mode is the percentage of the number N _Inter16×16 macroblocks of the optimal macroblock coding mode to the N _{BJND_x} macroblocks.

Figure 3b shows the non-boundary regions in the 46 right-view images of the multi-view color video sequences "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9" respectively Inside, for all macroblocks with the same binocular just perceivable distortion value, the percentage of the number of macroblocks that choose Inter16×16 mode as the optimal coding mode to the number of all macroblocks with the same binocular just perceivable distortion value, such as As shown in Figure 3b, the value of P _Inter16×16 is generally greater than 20%, and the percentage is relatively large, that is, for the multi-viewpoint color video in the non-boundary area of the right-viewpoint video image, the perceivable distortion value is in the range of BJND_min to BJND_max For all macroblocks in the existing H.264 encoding mode selection process, the percentage of SKIP mode is relatively large, so the distortion value can be perceived in the non-boundary area of the right-viewpoint video image for multi-viewpoint color video All macroblocks within the range from BJND_min to BJND_max must be searched in Inter16×16 mode.

Calculate the value of N _Inter16×8 /N _{BJND_x} , which is denoted as P _Inter16×8 , that is, among all macroblocks whose perceivable distortion value is equal to a certain value BJND_x in the range from BJND_min to BJND_max is N _{BJND_x} , select The number N _Inter16×8 of macroblocks in Inter16×8 mode as the optimal macroblock coding mode accounts for the percentage of the N _{BJND_x} macroblocks.

Figure 3c shows the non-boundary regions in the respective 46-frame right-view images of the multi-view color video sequences "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9" Inside, for all macroblocks with the same binocular just perceptible distortion value, the percentage of the number of macroblocks that choose Inter16×8 mode as the optimal coding mode to the number of all macroblocks with the same binocular just perceivable distortion value, such as As shown in Figure 3c, the value of P _Inter16×8 generally accounts for about 5%. As the binocularly detectable distortion value increases, the distortion that the human eye can tolerate is smaller, and the macroblock needs to be further refined. , as shown in Figure 3c, P _Inter16×8 also increases accordingly, specifically when 0≤BJND≤5, P _Inter16×8 remains at a small value, about 2%, which shows that when the macroblock When the binocular just detectable distortion value is less than a certain threshold, the macroblock mode selection can no longer perform the Inter16×8 macroblock coding mode search process, and when 6<BJND<n, n>6, P _Inter16×8 is The increasing trend increases to approximately 5%, which indicates that when the binocularly detectable distortion value of the macroblock is greater than a certain threshold, the macroblock needs to perform the Inter16×8 macroblock coding mode search process, where BJND means Distortion values are just perceptible to both eyes.

Calculate the value of N _Inter8×16 /N _{BJND_x} , which is denoted as P _Inter8×16 , that is, among all macroblocks whose perceivable distortion value is equal to a certain value BJND_x in the range from BJND_min to BJND_max is N _{BJND_x} , select The number of macroblocks N _Inter8×16 in Inter16×8 mode as the optimal macroblock coding mode accounts for the percentage of the N _{BJND_x} macroblocks.

Figure 3d shows the non-boundary regions in the respective 46-frame right view images of the multi-view color video sequences "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9" Inside, for all macroblocks with the same binocular just perceivable distortion value, the percentage of the number of macroblocks that choose Inter8×16 mode as the optimal coding mode to the number of all macroblocks with the same binocular just perceivable distortion value, such as As shown in Figure 3d, the value of P _Inter8×16 generally accounts for about 6%. As the binocularly detectable distortion value increases, the distortion that the human eye can tolerate is smaller, and the macroblock needs to be further refined. , as shown in Figure 3d, P _Inter8×16 also increases; specifically, when 0≤BJND≤5, P _Inter8×16 remains at a small value, about 3%, which shows that when the macroblock When the binocular just detectable distortion value is less than a certain threshold, the macroblock mode selection can no longer perform the Inter8×16 macroblock coding mode search process, and when 6<BJND<n, n>6, P _Inter8×16 is The increasing trend increases to approximately 6%, which indicates that when the binocularly perceptible distortion value of the macroblock is greater than a certain threshold, the macroblock needs to perform an Inter8×16 macroblock coding mode search process.

Calculate the value of N _Inter8×8 /N _{BJND_x} , which is denoted as P _Inter8×8 , that is, among all macroblocks whose perceivable distortion value is equal to a certain value BJND_x in the range from BJND_min to BJND_max is N _{BJND_x} , select The number N _Inter8×8 of macroblocks in Inter16×8 mode as the optimal macroblock coding mode accounts for the percentage of the N _{BJND_x} macroblocks.

Figure 3e shows the non-boundary regions in the respective 46-frame right-view images of the multi-view color video sequences "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9" Inside, for all macroblocks with the same binocular just perceivable distortion value, the percentage of the number of macroblocks that choose Inter8×8 mode as the optimal coding mode to the number of all macroblocks with the same binocular just perceivable distortion value, such as As shown in Figure 3e, the value of P _Inter8×8 generally accounts for about 4%. As the binocularly detectable distortion value increases, the distortion that the human eye can tolerate is smaller, and the macroblock needs to be further refined. , as shown in Figure 3e, P _Inter8×8 also increases; specifically, when 0≤BJND≤5, P _Inter8×8 remains at a small value, about 1.5%, which shows that when the macroblock When the binocular just detectable distortion value of is less than a certain threshold, the macroblock mode selection can no longer perform the Inter8×8 macroblock coding mode search process, and when 6<BJND<n, n>6, P _Inter8×8 is The increasing trend increases to approximately 4%, which indicates that when the binocularly detectable distortion value of the macroblock is greater than a certain threshold, the macroblock needs to perform an Inter8×8 macroblock coding mode search process.

Calculate the value of N _Others /N _{BJND_x} , denoted as P _Others , that is, among all the macroblocks whose perceivable distortion value is equal to a certain value BJND_x in the range from BJND_min to BJND_max is N _{BJND_x} , select Inter16×8 mode The number of macroblocks N _Others as the optimal macroblock coding mode accounts for the percentage of the N _{BJND_x} macroblocks.

Figure 3f shows the non-boundary regions in the respective 46-frame right-view images of the multi-view color video sequences "Alt Moabit", "Balloons", "Ballroom", "Kendo", "Race1", "Xmas3", and "Xmas9" Inside, for all macroblocks with the same binocularly detectable distortion value, the number of macroblocks with the same The percentage of the number of all macroblocks whose binocular distortion value is just perceivable, as shown in Figure 3f, the value of P _Others generally accounts for about 6%. The smaller the distortion that can be tolerated, the macroblock needs to be further refined. As shown in Figure 3f, P _Others also increases; when 0≤BJND≤5, P _Others is kept at a small value, about It is 5%, which shows that when the binocular distortion value of the macroblock is less than a certain threshold, the macroblock mode selection can no longer be carried out Others (including Inter8×8Frext, Intra16×16, Intra8×8 and Intra4×4 ) macroblock coding mode search process, and when 6<BJND<n, n>6, P _Others shows an increasing trend, increasing to about 8%, which shows that when the binocular of the macroblock can detect the distortion When the value is greater than a certain threshold, the macroblock needs to go through the Others (including Inter8×8Frext, Intra16×16, Intra8×8 and Intra4×4) macroblock coding mode search process.

Based on the above analysis, it can be concluded that the judgment threshold value is 5.

In order to verify the effectiveness and feasibility of the fast coding method of the present invention, at first the "Exit", "Vassar" 0 viewpoint and 2 viewpoint color video test sequence of MERL laboratory and the 39th viewpoint of "Champagnetower" of Nogaya University in Japan were selected. Viewpoint and the color video test sequence of the 41st viewpoint, these color videos are used as the original color video sequence, and then the "Exit", "Vassar" second viewpoint and "Champagne tower" of these sequences are processed by the fast coding method proposed by the present invention The color video of the 41st viewpoint is fast encoded to obtain the reconstructed color video sequence after fast coding.

Here, the performance of the method of the present invention will be measured from the encoding time of the color video sequence, the encoding bit rate and the peak signal-to-noise ratio (Peak Signal to Noise Ratio, PSNR) of the reconstructed color video sequence.

Table 1 lists the original color video sequences of the above-mentioned sequences through the JMVC verification platform and the color video sequences encoded by the method of the present invention under the same conditions (Basis QP=22) encoding time and encoding bit rate and reconstruction quality comparison. From the data listed in Table 1, it can be drawn that in terms of color video sequence coding time, the method of the present invention can save 66.48% to 71.90% of coding time compared with the multi-viewpoint standard verification platform, and save 68.46% of coding time on average. Compared with the PSNR of the color video sequence reconstructed by the fast encoding of the present invention and the color video sequence reconstructed by the JMVC verification platform, the average reduction of the video sequence is 0.037dB, and the maximum reduction is 0.04dB. Compared with the code rate encoded by the JMVC verification platform, the code rate after encoding increases by an average of 0.46%, the maximum increase is 1.27%, and the minimum increase is -0.20%, which is almost negligible.

Fig. 4a, Fig. 4b and Fig. 4c have respectively provided " Exit ", " Vassar " and " Champagne tower " test sequence respectively by JMVC verification platform and the coding rate-distortion performance comparison chart of the color video sequence obtained after coding by the method of the present invention The rate-distortion performance comparison includes the PSNR of the color video sequence encoded by the JMVC verification platform and the method of the present invention and the comparison of the code rate used. As can be seen from Figure 4a, in the direction of the abscissa, when the code rate of the right view image of the coded "Exit" sequence is the same, the PSNR of the reconstructed color video sequence encoded by the JMVC verification platform and the method of the present invention is basically maintained Consistent; in the direction of the ordinate, when the PSNR of the reconstructed color video sequence encoded by the JMVC verification platform and the method of the present invention is the same, the code rate used for encoding the right viewpoint image of the "Exit" sequence is basically the same. It can be seen from Figure 4b that in the direction of the abscissa, when the code rate of the right view image of the encoded "Vassar" sequence is the same, the PSNR of the reconstructed color video sequence encoded by the JMVC verification platform and the method of the present invention is basically consistent ; On the ordinate direction, when the PSNR of the color video sequence reconstructed by the JMVC verification platform and the inventive method encoding is the same, the code rate used by the right viewpoint image of the encoding "Vassar" sequence is basically the same. It can be seen from Fig. 4c that, in the direction of the abscissa, when the code rate of the right view image of the coded "Champagnetower" sequence is constant, the PSNR of the reconstructed color video sequence encoded by the JMVC verification platform and the method of the present invention is basically consistent. On the ordinate direction, when the PSNR of the color video sequence reconstructed by the JMVC verification platform and the inventive method encoding is the same, the code rate used by the right viewpoint image of the encoding "Champagne tower" sequence is basically the same. From the above analysis, it can be concluded that the fast encoding method of the present invention can basically maintain the same rate-distortion performance as that of the JMVC verification platform.

The original color video sequence of table 1 passes through JMVC verification platform and the encoding time of the present invention method and coding bit rate and the quality comparison of reconstruction color video sequence

Subjectively, comparing the quality of the color video sequence reconstructed by the JMVC verification platform and the method of the present invention, Fig. 5a shows the 10th frame image of the 2nd viewpoint reconstructed by the multi-viewpoint standard verification platform for the "Exit" test sequence, Figure 5b shows the image of the 10th frame of the second viewpoint reconstructed by the fast encoding method of the present invention in the "Exit" test sequence, and there is almost no difference between Figure 5a and Figure 5b intuitively; Figure 5c shows the "Vassar" The test sequence uses the multi-viewpoint standard verification platform to reconstruct the image of the 10th frame of the 2nd viewpoint. Figure 5d shows the image of the 10th frame of the 2nd viewpoint reconstructed by the fast coding method of the present invention in the "Vassar" test sequence, intuitively almost No difference between Figure 5c and Figure 5d can be seen. Therefore, there is basically no difference intuitively between the fast encoding method of the present invention and the right view image of the multi-view video sequence reconstructed by the JMVC verification platform.

In summary, for the right-view image of a multi-view video sequence, the method of the present invention can greatly save the encoding time of the right-view image of a multi-view video sequence on the premise of keeping the rate-distortion performance consistent with the JMVC verification platform, thereby saving multi-view The overall encoding time of the video sequence.

Claims (5)

1. A multi-viewpoint color video fast encoding method based on binocular just perceptible distortion, is characterized in that comprising the following steps: ①Record the left view video of the multi-viewpoint color video as {C _L (k)}, and record the right view video of the multi-viewpoint color video as {C _R (k)}, where C _L (k) means {C _L (k)} in the k-th frame of the left-view image, C _R (k) represents the k-th frame of the right-view image in {C _R (k)}, 1≤k≤K, K represents the left-view video and the right-view video The number of frames of the image contained in; ② Divide each frame of right-view image in {C _R (k)} into boundary area and non-boundary area, where the boundary area consists of the first row of macroblocks, the last row of macroblocks, and the first column of macroblocks in the right-view image. block and the last column of macroblocks; then calculate the binocular perceivable distortion value of each macroblock in the non-boundary area of each frame of right-view image in {C _R (k)}, assuming {C _R (k )} in the k-th frame of right-viewpoint image C _R (k) in the coordinate position (i, j) of the macroblock belongs to {C _R (k)} in the k-th frame of right-viewpoint image C _R (k) in In the non-boundary area, the binocular just perceivable distortion value of the macroblock at the coordinate position (i, j) in the k-th frame of the right view image C _R (k) in {C _R (k)} is recorded as Among them, 2≤i≤W/16-1, 2≤j≤H/16-1, W represents each frame of left view image in { _CL (k)} and each frame in { _CR (k)} The width of the right view image, H represents the height of each frame of the left view image in { _CL (k)} and the height of each frame of the right view image in {C _R (k)}; ③ On the multi-view video coding verification model JMVC, use the HBP predictive coding structure to pair each macroblock in each frame of the left view image in { _CL (k)} and each frame in { _CR (k)} Each macroblock in the right view image is encoded, and the process of selecting the optimal macroblock encoding mode for each macroblock in the encoding process is: ③-1. Define the current macroblock to be encoded as the current macroblock; ③-2. When the current macroblock is a macroblock in { _CL (k)}, the encoder uses the H.264 mode selection process to search for SKIP, Inter16×16, Inter16×8, Inter8×16, Inter8×8 , Inter8×8Frext, Intra16×16, Intra8×8, and Intra4×4 macroblock coding modes, select the macroblock coding mode with the smallest rate-distortion cost from these macroblock coding modes as the optimal macroblock coding mode for the current macroblock to encode; When the current macroblock is a macroblock in { _CR (k)}, it is judged whether the current macroblock belongs to the boundary area or the non-boundary area. If the current macroblock belongs to the boundary area, the encoder adopts the mode selection process of H.264 Search for SKIP, Inter16×16, Inter16×8, Inter8×16, Inter8×8, Inter8×8Frext, Intra16×16, Intra8×8, and Intra4×4 macroblock coding modes, and select rate-distortion from these macroblock coding modes The macroblock coding mode with the least cost is used as the optimal macroblock coding mode for the current macroblock; if the current macroblock belongs to the non-boundary area, then judge whether the binocular detectable distortion value of the current macroblock is greater than or equal to the set If yes, the encoder uses the H.264 mode selection process to search for SKIP, Inter16×16, Inter16×8, Inter8×16, Inter8×8, Inter8×8Frext, Intra16×16, Intra8×8 and Intra4 ×4 macroblock coding mode, select the macroblock coding mode with the smallest rate-distortion cost from these macroblock coding modes as the optimal macroblock coding mode of the current macroblock for coding; otherwise, the encoder adopts the mode selection of H.264 The process only searches for SKIP and Inter16×16 macroblock coding modes, and selects the macroblock coding mode with the smallest rate-distortion cost from these two macroblock coding modes as the optimal macroblock coding mode of the current macroblock for coding; ③-3. Use the next macroblock to be encoded as the current macroblock, and then return to step ③-2 to continue until each macroblock in each frame of the left view image in { _CL (k)} and {C Each macroblock in each frame of right-view image in _R (k)} is coded. 2. The multi-viewpoint color video fast coding method based on binocular just perceivable distortion according to claim 1, characterized in that the k-th frame right viewpoint image C in {C _R (k)} in the described step 2. The binocular just detectable distortion value of the macroblock at the coordinate position (i, j) in _R (k) Among them, d represents the disparity value of the macroblock whose coordinate position is (i, j) in the k-th frame right view image C _R (k) in {C _R (k)}, bg _L (i+d, j) Represents the average value of the luminance values of all pixels in the macroblock whose coordinate position is (i+d,j) in the k-th frame left-viewpoint image _CL (k) in { _CL (k)}, bg _L ( i+d,j)∈[0,255], bg _L (i×16+d×16+m,j×16+n) means that the coordinate position in the kth frame left viewpoint image C _L (k) in _{ CL (k)} is (i×16+d ×16+m, j×16+n), 0≤m≤15, 0≤n≤15, eh _L (i+d, j) represents the luminance value of the pixel in {C _L (k)} The average value of the edge intensity values of all pixels in the macroblock whose coordinate position is (i+d, j) in the left view point image C _L (k) of k frames, bg _L (i×16+d×16+m-3+h, j×16+n-3+v) represents the coordinates in the k-th frame left viewpoint image C _L (k) in _{ CL (k)} The brightness value of the pixel at the position (i×16+d×16+m-3+h, j×16+n-3+v), G _H (h, v) represents the horizontal Sobel operator of 5×5 The element at the coordinate position (h, v) in G _H , G _V (h, v) represents the element at the coordinate position (h, v) in the vertical Sobel operator G _V of 5×5, _{1≤h≤5,1≤v≤5} , A limit (bg _L (i+d,j),eh _L (i+d,j))＝A _limit (bg _L (i+d,j))+ K(bg _L (i+d,j))×eh _L (i+d,j), K(bg _L (i+d,j))＝-10 ^-6 ×(0.7×(bg _L (i+d,j)) ² +32×bg _L (i+d,j))+0.07，λ Indicates the parameter to control the influence of right viewpoint noise, n _L (i+d,j) indicates that the coordinate position in the kth frame left viewpoint image C _L (k) in _{ CL (k)} is (i+d,j) The noise magnitude of the macroblock. 3. The multi-view point color video fast coding method based on binocular just perceivable distortion according to claim 2, characterized in that the value of the parameter λ for controlling the influence of noise on the right view point is 1.25. 4. the multi-viewpoint color video fast coding method based on binocular just perceivable distortion according to claim 3, is characterized in that the kth frame left viewpoint image C _L (k in the described { _CL (k)} ) The noise amplitude n _L (i+d, j) of the macroblock whose coordinate position is (i+d, j) is 0.3. 5. The multi-viewpoint color video fast encoding method based on binocular just detectable distortion according to any one of claims 1 to 4, characterized in that the determination threshold set in the step ③-2 is 5.