CN103379349B - A kind of View Synthesis predictive coding method, coding/decoding method, corresponding device and code stream - Google Patents

Abstract

The present invention provides a viewpoint synthesis predictive coding method applied in the field of multimedia communication. In the process of coding an image I in a viewpoint V1 in a three-dimensional video sequence, a synthetic precision X is adopted to generate an image I from another viewpoint V1 in the three-dimensional video sequence. The reconstructed image and the reconstructed depth in the coded viewpoint V2 are synthesized into a viewpoint composite image P of a region H1 in the viewpoint V1, and the viewpoint composite image P is used to generate the prediction pixels required for viewpoint composite prediction in the process of generating the coded image I; the composite precision X It is written into the code stream of the three-dimensional video sequence; the invention also discloses a viewpoint synthesis predictive decoding method, a viewpoint synthesis predictive encoding device, a decoding device and a corresponding code stream. The invention can improve the coding efficiency of the three-dimensional video sequence.

Description

A View Synthesis Predictive Encoding Method, Decoding Method, Corresponding Device, and Code Stream

technical field

The present invention relates to the field of multimedia communication, in particular to a viewpoint synthesis predictive coding method, a decoding method, a corresponding device and a code stream.

Background technique

A 3D video sequence includes a multi-channel (usually 2-channel) image sequence (including texture information) and a corresponding depth (depth) sequence (including depth information, that is, each pixel in the image corresponds to the distance between the object in the three-dimensional space and the camera. The distance), usually also known as MVD (multi-viewvideoplusdepth) format. A 3D video sequence includes several access units, and one access unit includes multiple (two or more than two) images at a certain moment and their corresponding depths. The coded 3D video sequence forms a 3D video code stream (bitstream), and the code stream consists of bits. The encoding method may be an encoding method based on video encoding standards such as MPEG-2, H.264/AVC, HEVC, AVS, and VC-1. It should be noted that the video coding standard specifies the syntax of the code stream and the decoding method for the code stream that conforms to the coding standard, but does not specify the encoding method for generating the code stream, but the encoding method used must be consistent with the decoding method specified in the standard. The method is matched to form a code stream conforming to the encoding standard, and the decoder can complete the decoding of these code streams, otherwise the decoding process may go wrong. The decoded image is called decoded image or reconstructed image, and the decoded depth is called decoded depth or reconstructed depth; the (uncoded) image input to the encoder is called original )image. That is to say, the encoder also completes the decoding process of generating a reconstructed image (or reconstructed depth) from the encoded information, and this decoding process is the same as the decoding process of the code stream in the decoder. The currently encoded image (or currently encoded image) refers to the original image of the frame where the macroblock (macroblck) is currently being encoded, and the currently decoded image (or currently decoded image) refers to the reconstruction of the frame where the macroblock is currently being decoded image. The current coding viewpoint refers to the viewpoint where the macroblock currently being coded is located, and the current decoding viewpoint refers to the viewpoint where the macroblock currently being decoded is located. The coded viewpoint refers to the viewpoint where the macroblock that is not currently being coded is located, and the frame in this viewpoint is coded before the frame at the same time in the current coded viewpoint; the decoded viewpoint refers to the viewpoint where the macroblock that is not currently being decoded is located, and Frames in this view are decoded before frames at the same moment in the currently decoded view.

Most of the current video coding standards, such as H.264/AVC, are based on a hybrid coding framework including prediction coding and transform coding. Each frame of image in the video sequence is coded frame by frame, and one frame of image is divided into several macroblocks (Macroblock), and each macroblock can be subdivided into several blocks (block). The usual coding process can be summarized as: for each macroblock, a prediction method (corresponding to a prediction mode) is used to obtain a prediction image (composed of a group of prediction pixels) of the macroblock, wherein the prediction method usually includes Intra prediction (intraprediction, the predicted image of the block is obtained from the reconstructed pixels around the current coded block) and inter prediction (interprediction, the predicted image of the block is obtained from the reconstructed image of another coded frame); for multi-view video The (multi-viewvideo) sequence also includes inter-view prediction (inter-view prediction, which obtains the predicted image of the block from the reconstructed image of another coded viewpoint); for the 3D video sequence containing depth information, it can also include view synthesis prediction (viewsynthesis prediction) , the predicted image of the block is obtained from the view synthesis image generated from the reconstructed image of another encoded viewpoint and the reconstructed depth). Then, the original image of the macroblock is subtracted from the predicted image to obtain a residual; after that, the residual is transformed to obtain a transform coefficient; the transform coefficient is quantized, and the quantized transform coefficient and the predicted Entropy encoding is performed on the side information of the mode to form a code stream. Due to the introduction of quantization, there is a certain difference between the encoded image (ie, the reconstructed image) and the unencoded input image (ie, the original image, or the current encoded image), which is usually called distortion.

The decoding process can be regarded as the reverse process of the encoding process, which usually includes the following steps: for each macroblock, analyze the prediction mode information corresponding to the macroblock and the quantized transform coefficient from the code stream; through the prediction mode, use the corresponding A prediction method obtains a prediction image of the macroblock; dequantizes and inversely transforms quantized transformation coefficients to generate a residual image, and adds the residual image and the prediction image to obtain a reconstructed image.

View synthesis prediction (VSP) is a predictive coding technique for 3D video sequences. Viewpoint synthesis prediction is based on the reconstructed image of a viewpoint V2 at a certain moment and the corresponding reconstruction depth, using depth-image-based rendering (DIBR for short) technology to project the reconstructed image of V2 to another viewpoint V1, thus Generate a projected image; and then generate a synthetic view image P in the viewpoint V1 through one or more of processing such as hole filling, filtering, and resampling. The view synthesis image P has a high degree of similarity with the corresponding original image O in the viewpoint V1 at the same time, so it is used as a predictive image when encoding the original image O in the viewpoint V1. The currently encoded macroblock (that is, the macroblock undergoing encoding processing) can select some pixels from the view synthesis image as the predicted image in the macroblock, and then perform transformation, quantization, and entropy encoding to obtain the code stream; correspondingly, the current decoding A macroblock (that is, a macroblock undergoing decoding processing) can obtain a reconstructed image by adding the predicted image and corresponding residual information. In general, view synthesis prediction is similar to traditional inter prediction, the main difference is that the prediction image used by view synthesis prediction is a reconstructed image from a coded (or decoded) viewpoint different from the current coded (or decoded) viewpoint and The view synthesis image generated by depth is reconstructed, while the prediction image used in inter prediction is the reconstructed image at another moment of the currently coded (or decoded) viewpoint.

A key point in view synthesis prediction is to generate high-quality view synthesis images. The closer the view synthesis image is to the original image (that is, the higher the similarity), the higher the prediction efficiency and the corresponding residual error will be reduced. The projection in view synthesis can use integer pixel precision, that is, the pixels in the viewpoint V2 are projected onto the pixel grid of the projected image in the viewpoint V1, and the resolution of the projected image and the image in viewpoint V2 are the same at this time. The method of projecting each pixel in the viewpoint V2 onto the pixel grid of the projected image in the viewpoint V1 is usually to project the projection points of the pixels in the viewpoint V2 in the viewpoint V1 (obtained by depth-based and image rendering techniques, and the projection points are usually located at Between two pixels in the viewpoint V1) is rounded to the pixel with the closest horizontal distance to generate a projected pixel. When there is only one projected pixel at a pixel on the pixel grid in V1, the projected pixel is taken as the pixel of the pixel; when multiple projected pixels correspond to the same pixel on the pixel grid in V1, the depth is minimized The projected pixel (closest to the camera) is used as the pixel of the pixel; when there is no projected pixel corresponding to a pixel on the pixel grid in V1, the pixel is called a hole.

Projection in view synthesis can also use sub-pixel synthesis accuracy, which is recorded as K/L pixel accuracy (K and L are positive integers, and L is usually a multiple of 2, and K is usually 1, such as 1/2 pixel synthesis accuracy and 1/ 4 pixel compositing precision). When K/L pixel precision is used, the pixels in viewpoint V2 are projected onto the pixel grid of the projected image in viewpoint V1, and the horizontal resolution of the projected image is L/K times the horizontal resolution of the image in viewpoint V2 . The method of projecting a pixel in the viewpoint V2 onto the pixel grid of the projected image in the viewpoint V1 usually has one of the following two methods or a combination thereof: 1) Sampling the image (and its depth) of the viewpoint V2 in the horizontal direction by L/ K times (that is, the horizontal resolution of the upsampled image is L/K times that of the original image), and then project the upsampled image to the viewpoint V1 to form a horizontal resolution that is L/K of the horizontal resolution of the image in the viewpoint V2 2) set the horizontal resolution of the projected image in V1 to L/K times of the image in V2, and multiply the projection position obtained by calculating the pixel points in the V2 image with integer pixel precision by L/K to obtain The projection position under K/L pixel precision (that is, the point whose coordinates are (a, b) is projected on the projection image with integer pixel precision, and the projected coordinates under K/L pixel precision are (aK/L, b)); Then, similar to the viewpoint synthesis with integer pixel synthesis accuracy, the projected position is rounded to the nearest pixel grid in the projected image to obtain the projected pixels, and then the horizontal resolution of the image with the horizontal resolution of V2 is generated. L /K times the projected image. The projected image is subjected to processing such as hole filling to form a view synthesis image. During the encoding or decoding process, the macroblock or block predicted by view synthesis will obtain some pixels from the view synthesis image (the resolution may be different from the resolution of the current encoded image) to generate the predicted image of the macroblock or block (for example, from Extract M×K pixels from an M×L area in the view synthesis image as the prediction image of the current coding block); or, when the resolution of the projected image is different from the resolution of the current coding image, project the hole-filled image Up-sampling or down-sampling to obtain a view synthesis image with the same resolution as the current encoding, and the current encoding (or decoding) block obtains prediction pixels from the view synthesis image formed by variable sampling (for example, taking a block from the view synthesis image The pixels are used as the prediction image of the current coding block).

View synthesis images generated with different synthesis precision are usually different, and the prediction efficiency when used for prediction is also different. Generally speaking, a higher-quality view synthesis image can be obtained by adopting sub-pixel synthesis precision than by adopting integer-pixel synthesis precision, but the computational complexity is also higher. In order to improve the compression coding performance, the encoder can select the view synthesis image with the highest similarity to the current coded image (ie the original image) among multiple view synthesis images with different synthesis precisions through an adaptive method as the view synthesis image, which is used to generate The predicted pixels in the view synthesis prediction, and write the corresponding synthesis accuracy information into the code stream to inform the decoder to use the corresponding synthesis accuracy to generate a view synthesis image that matches the encoder.

The information generated by encoding (such as prediction mode, transformation coefficient, etc.) is converted into a code stream through entropy encoding. A piece of information (such as synthetic precision) can be numericalized, that is, the content of the information description is represented by a value in a certain range (usually an integer field). For example, if one piece of information describes three situations, it can be represented by 0, 1, and 2 respectively These three situations. The information is numerically converted into a syntax element (syntax element), and the syntax element can be encoded with a suitable code according to the range and distribution of its value to form a code word (codeword, consisting of one or more bits), that is, the syntax element Encoded into a string of bits. Commonly used codes include n-ratio specific long codes, exponential Golomb codes, arithmetic coding, etc.; more specifically, 1-bit unsigned integer codes include two codewords 0 and 1, and 2-bit unsigned integer codes include four codewords 00 , 01, 10 and 11, the codewords of the 0th-order Exponential Golomb code include codewords 1, 010, 011 and so on. The codeword is restored to the value of the syntax element corresponding to the codeword by a corresponding decoding method (such as a look-up method, that is, looking up the value of the syntax element corresponding to the codeword from the code table). A piece of information can also be jointly numericalized together with other information into a syntax element, thus corresponding to a codeword. For example, the combination of two pieces of information can be numbered, and this number can be used as a syntax element. To write information into the code stream, the usual method is to digitize the information into syntax elements, and write the code words corresponding to the syntax elements into the code stream.

Contents of the invention

In order to overcome the above-mentioned defects of the prior art, the purpose of the present invention is to provide a view synthesis predictive encoding method, decoding method, corresponding device and code stream that can improve the efficiency of encoding and prediction.

The first technical solution of the present invention is to provide a predictive coding method for viewpoint synthesis. When coding an image I in a viewpoint V1 in a three-dimensional video sequence, a synthetic precision X is used, and another coded image I in the three-dimensional video sequence is used. The reconstructed image and the reconstructed depth in the viewpoint V2 are synthesized into a viewpoint synthesis image P of an area H1 in the viewpoint V1, and the viewpoint synthesis image P is used to generate prediction pixels required for viewpoint synthesis prediction in the process of generating the coded image I; at the same time, Writing the synthesis precision X into the code stream of the 3D video sequence, wherein the viewpoint V1 is the current encoding viewpoint.

Preferably, the method for obtaining the synthetic precision X includes one of the following methods:

1) Synthesize the view synthesis image Pn of a region H2 in the current coded view V1 by using N kinds of synthesis precision, where N≥2, 1≤n≤N, N and n are integers; The similarity between the images of the corresponding regions, selecting the synthesis precision corresponding to the viewpoint synthesis image with the highest similarity from all viewpoint synthesis images Pn as the synthesis precision X;

2) Calculate the variance Vk of the pixels in the K regions H3k in the image I, take the average value V of the variance Vk of the pixels, and derive the synthesis accuracy X through the function f(V) about the average value V, where 1≤ k≤K, k is an integer; the function f(V) is for example Where E is a constant, such as E=15, Indicates that x is rounded down, and the synthesis precision X is 1/D pixel precision.

Preferably, "calculating the similarity between each view synthesis image Pn and the image of the corresponding region in the image I" described in 1) includes one of the following methods:

Method 1: For the M1 pixels Q1m in the viewpoint synthesis image Pn, calculate the difference D1m between each pixel Q1m and the pixel R1m at the corresponding position in the image I, where 1≤n≤N, 1≤m ≤M1, m and M1 are positive integers, M1 is less than or equal to the total number of pixels in the area H2; calculate the average value ValA of the absolute value of D1m or the average value ValB of the square value of D1m, the average value ValA or the average value ValB The smaller the value, the higher the similarity between the view synthesis image Pn and the image of the corresponding region in the image I;

Method 2: For the M2 pixels Q2m in the viewpoint synthesis image Pn, calculate the linear correlation coefficient C between each pixel Q2m and the corresponding pixel R2m in the image I, the larger the linear correlation coefficient C, the The higher the similarity between the viewpoint synthesis image Pn and the image of the corresponding area in the image I, where 1≤n≤N, 1≤m≤M2, n, N, m, and M2 are positive integers, and M2 is less than or equal to The total number of pixels in the region H2.

The second technical solution of the present invention is to provide a viewpoint synthesis predictive decoding method. When decoding an image I in a viewpoint V1 in a three-dimensional video sequence, the corresponding image I is parsed from the code stream of the three-dimensional video sequence. Synthesizing accuracy X; using the synthesis accuracy X, synthesizing a view synthesis image P of a region H1 in the view V1 from the reconstructed image and the reconstruction depth in another decoded view V2 in the 3D video sequence, the view synthesis The picture P is used to generate the prediction pixels needed for view synthesis prediction in the process of decoding the picture I.

The third technical solution of the present invention is to provide a view synthesis predictive encoding device, which is characterized in that it includes the following two modules:

The view synthesis image generating module adopts a synthetic precision X to generate a synthetic view image, and its input includes a synthetic precision X, a reconstructed image and a reconstructed depth in a coded viewpoint V2 in a 3D video sequence, and the coded viewpoint V2 and the current encoding camera parameter information of a viewpoint V1, the output of which comprises a view synthesis image P of an area H1 of said viewpoint V1, the processing of which includes applying said synthesis precision X when encoding an image I of said viewpoint V1, From the reconstructed image and the reconstructed depth in the coded viewpoint V2, synthesize a viewpoint synthesis image P of a region H1 in the viewpoint V1, and the viewpoint synthesis image P is used for the viewpoint synthesis prediction required in the process of generating the coded image I predicted pixels;

The composite precision writing module writes the composite precision X into the code stream of the three-dimensional video sequence, its input includes the composite precision X and the code stream of the three-dimensional video sequence, and its output includes the composite precision X The code stream of the 3D video sequence, the completed processing includes writing the synthesis precision X into the code stream of the 3D video sequence.

Preferably, the view synthesis predictive encoding device further includes one of the following modules:

Synthesis accuracy decision module based on image similarity, its input includes the reconstructed image and reconstruction depth in the coded viewpoint V2, and the camera parameter information of the coded viewpoint V2 and the current coded viewpoint V1, and its output is the composite Accuracy X, the completed processing includes synthesizing the viewpoint synthesis image Pn of a region H2 in the current coded viewpoint V1 by using N kinds of synthesis precisions, where N≥2, n and N are integers, 1≤n≤N; calculate each viewpoint synthesis The similarity between the image Pn and the image of the corresponding area in the image I, select the synthesis precision corresponding to the viewpoint synthesis image with the highest similarity from Pn as the synthesis precision X;

The synthetic accuracy decision module based on texture complexity, its input includes the image I, its output is the synthetic accuracy X, and its completed processing includes calculating the variance Vk of the pixels in the K regions H3k in the image I, taking Vk The average value V of the average value V, through the function f(V) about the average value V, derives the composite precision X, wherein 1≤k≤K, k, K are integers; the function f(V) is for example Where E is a constant, such as E=15, Indicates that x is rounded down, and the synthesis precision X is 1/D pixel precision.

Preferably, in the image similarity-based synthesis accuracy decision-making module, the calculation of the similarity between each viewpoint synthesis image Pn and the image of the corresponding region in the image I includes one of the following processes:

Process 1: For the M1 pixels Q1m in the viewpoint synthesis prediction image Pn, calculate the difference D1m between each pixel Q1m and the pixel R1m in the corresponding position in the image I, where 1≤n≤N, 1≤ m≤M1, m and M1 are positive integers, M1 is less than or equal to the total number of pixels in the area H2; calculate the average value ValA of the absolute value of D1m or the average value ValB of the square value of D1m, the average value ValA or the average value The smaller the ValB, the higher the similarity between the view synthesis image Pn and the image of the corresponding area in the image I;

Processing 2: For the M2 pixels Q2m in the view synthesis image Pn, calculate the linear correlation coefficient C between each pixel Q2m and the corresponding pixel R2m in the image I, the larger the linear correlation coefficient C, the The higher the similarity between the viewpoint synthesis image Pn and the image of the corresponding area in the original image O, where 1≤n≤N, 1≤m≤M2, m and M2 are positive integers, and M2 is less than or equal to the Total number of pixels in region H2.

The fourth technical solution of the present invention is to provide a viewpoint synthesis predictive decoding device, which includes the following two modules:

The synthetic accuracy analysis module parses the synthetic precision X from the three-dimensional video sequence code stream, its input is the three-dimensional video sequence code stream, and its output is the synthetic precision X, and its completed function is when decoding a viewpoint V1 in the three-dimensional video sequence When an image I in the image I, analyze the composite accuracy X corresponding to the image I from the code stream of the three-dimensional video sequence;

The viewpoint synthesis image generation module synthesizes the viewpoint synthesis image P in the viewpoint V1 with the synthesis accuracy X, and its input includes synthesis accuracy X, a reconstructed image and reconstruction depth of a decoded viewpoint V2, the decoded viewpoint V2 and The camera parameters of the viewpoint V1, the output of which is a view synthesis image P of an area H1 in the viewpoint V1, and the completed processing includes synthesizing the reconstructed image and the reconstructed depth in the decoded viewpoint V2 in the viewpoint V1 A view synthesis image P of a region H1, the view synthesis image P is used to generate prediction pixels required for view synthesis prediction in the process of decoding the image I.

The fifth technical solution of the present invention is to provide a code stream for view synthesis prediction, which includes information corresponding to the synthesis accuracy X; the synthesis accuracy is the synthesis accuracy of view synthesis images generated in view synthesis prediction.

Beneficial effects: Compared with the prior art, the view synthesis predictive encoding method, decoding method, corresponding device and code stream of the present invention generate a view synthesis image with an optimal synthesis accuracy, which is used to provide the required view synthesis prediction. Predict pixels, thereby improving the efficiency of encoding prediction.

Description of drawings

Other features and advantages of the invention will become apparent from the following description of preferred embodiments, taken by way of example to explain the principles of the invention, taken in conjunction with the accompanying drawings.

FIG. 1 is a schematic structural diagram of an embodiment of a viewpoint synthesis predictive encoding device in the present invention;

FIG. 2 is a schematic structural diagram of another embodiment of a view synthesis predictive encoding device in the present invention;

FIG. 3 is a schematic structural diagram of another embodiment of a view synthesis predictive encoding device in the present invention;

FIG. 4 is a schematic structural diagram of an embodiment of a view synthesis predictive decoding device in the present invention.

detailed description

Embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings:

Example 1

The first embodiment of the present invention relates to a view synthesis predictive coding method. For a three-dimensional video sequence (comprising at least two viewpoints, each viewpoint including an image sequence and a depth sequence), an image I in one of the viewpoint V1 is encoded (the image I is the Tth frame of the image sequence in the viewpoint V1 In the process of the original image), a synthesis precision X is adopted, and depth-based and image rendering techniques are used to generate the same moment (or another moment, such as T-1th) from another encoded viewpoint V2 in the 3D video sequence. The reconstructed image and the reconstructed depth of the frame) are synthesized to synthesize the viewpoint synthesis image P of an area H1 in the viewpoint V1. Described area H1 can be whole described image I, also can be a designated area in image I (for example a center coincides with image I center and size is the matrix window of 1/2 of image I size), also It can be a macroblock or block in image I, and so on. The view synthesis image P is used to generate prediction pixels required for view synthesis prediction in the process of encoding the image I, that is, there is at least one macroblock or block that adopts view synthesis prediction in the process of encoding the image I, The pixels in the view synthesis image P are selected as the prediction pixels of the macroblock or block. For example, when using integer pixel synthesis precision, a pixel in a rectangular area with the same size as the macroblock or block can be taken from P as the predicted pixel of the macroblock or block; when P adopts sub-pixel precision, from P Extract M×J pixels from an M×L area in P, or change sampling (that is, upsampling or downsampling) into M×J pixels in an M×L area in P, as the macroblock or block (size is M×J pixels) of predicted pixels.

At the same time, writing the synthesis precision X into the code stream of the three-dimensional video sequence; the synthesis precision X can be integer pixel precision or sub-pixel precision, such as 1/2 pixel precision, 1/2 pixel precision, etc. 4 pixel precision, 1/3 pixel precision, k/l pixel precision (k, l are positive integers), etc.

When the synthesis precision X used in viewpoint synthesis is integer pixel precision, the pixels in viewpoint V2 are projected onto the pixel grid of the projected image in viewpoint V1, and the resolutions of the projected image and the viewpoint V2 image are the same. The method of projecting each pixel in the viewpoint V2 onto the pixel grid of the projected image in the viewpoint V1 is usually to project the projection point of each pixel in the viewpoint V2 in the viewpoint V1 (obtained by depth-based and image rendering techniques, usually located in the viewpoint V1 Between two pixels) is rounded to the nearest pixel horizontally, resulting in a projected pixel. When there is only one projected pixel at a pixel on the pixel grid in V1, the projected pixel is taken as the pixel of the pixel; when multiple projected pixels correspond to the same pixel on the pixel grid in V1, the depth is minimized The projected pixel (closest to the camera) is used as the pixel of the pixel.

If the synthesis precision X used in view synthesis is sub-pixel synthesis precision, then the synthesis precision X is recorded as K/L pixel precision (K and L are positive integers, and L is usually a multiple of 2, and K is usually 1, for example, 1/ 2-pixel compositing precision and 1/4-pixel compositing precision). When K/L pixel precision is used, the pixels in viewpoint V2 are projected onto the pixel grid of the projected image in viewpoint V1, and the horizontal resolution of the projected image is L/K times the horizontal resolution of the image in viewpoint V2 . The method of projecting a pixel in the viewpoint V2 onto the pixel grid of the projected image in the viewpoint V1 usually has one of the following two methods or a combination thereof: 1) Sampling the image (and its depth) of the viewpoint V2 in the horizontal direction by L/ K times (that is, the horizontal resolution of the upsampled image is L/K times that of the original image), and then project the upsampled image to the viewpoint V1 to form a horizontal resolution L/K of the horizontal resolution of the image of the viewpoint V2 2) the horizontal resolution of the projected image in V1 is set as L/K times of the image in V2, and the projection position calculated by each pixel in the V2 image with integer pixel precision is multiplied by L/K, Get the projection position under K/L pixel precision (that is, the projected coordinates of a point (a, b) on the projection image with integer pixel precision, the projected coordinates under K/L pixel precision are (aK/L, b)) , and then the projected position is rounded to the nearest pixel grid in the projected image to obtain the projected pixels, and then the projected image whose horizontal resolution is L/K times the horizontal resolution of the image of viewpoint V2 is generated.

When there are holes in the projected image (that is, some pixels in the projected image do not have projected pixels), hole filling is also required to fill the missing pixels in the hole area to obtain a view synthesis image.

Example 2

The second embodiment of the present invention relates to a view synthesis predictive coding method. For a 3D video sequence, in the process of encoding an image I in one of the viewpoints V1 (the image I is the original image of the Tth frame of the image sequence in the viewpoint V1), a synthetic precision X is used, and the depth and image The rendering technique is to synthesize the viewpoint composite image P of an area H1 in the viewpoint V1 from the reconstructed image and the reconstructed depth at the same moment in another coded viewpoint V2 in the 3D video sequence.

Among them, the synthetic accuracy X is determined by the following method. According to the reconstructed image and reconstructed depth (and other information such as camera parameters) at the same moment in the viewpoint V2, using depth-based and image rendering technology, with N synthesis precision (for example, N=2, respectively, the synthesis precision of the integer number and 1/ 2 pixel synthesis precision) respectively synthesize N view synthesis images Pn (1≤n≤N, n is an integer) of an area H2 in the current coded view V1.

Synthesize view synthesis images Pn (1≤n≤N, n is an integer) of a region H2 in the current coded viewpoint V1 using N (N≥2, N is an integer) synthesis precisions. The area H2 may be the entire image I, or the area H1, or a collection of multiple scattered areas (such as macroblocks) whose area is smaller than the image I.

For the N synthetic view images Pn, calculate the similarity between each synthetic view image Pn and the image corresponding to the region U in the image I. When the integer pixel synthesis precision is adopted, the position of the region U in the image I is the same as the position of the view synthesis image Pn in the projected image. When the sub-pixel synthesis precision is used, the corresponding region U is the same position as the viewpoint synthesis image Pn in the projected image in the image after the image I is correspondingly sampled (up-sampled or down-sampled) according to the adopted synthesis precision. area.

The synthesis accuracy corresponding to the view synthesis image with the highest similarity is selected from all Pn as the synthesis accuracy X. The similarity calculation method includes one of the following two methods:

Method 1: For the M1 pixels Q1m (1≤m≤M1, m is a positive integer, M1 is a positive integer, and M1 is less than or equal to the area H2) in the viewpoint synthesis image Pn (1≤n≤N). the total number of pixels), calculate the difference between each pixel Q1m (1≤m≤M) and the pixel R1m (1≤m≤M1) at the corresponding position in the image I (that is, the same coordinates in the region U) D1m (1≤m≤M1); calculate the average value ValA of the absolute value of D1m (1≤m≤M1) or the average value ValB of the square value of D1m (1≤m≤M1), the average value ValA or the average value ValB The smaller the value, the higher the similarity between the viewpoint synthesis image Pn and the image corresponding to the region U in the image I;

Method 2: For M2 pixels Q2m (1≤m≤M2) in the viewpoint synthesis image Pn (1≤n≤N), m is a positive integer, M2 is a positive integer, and M2 is less than or equal to the area H2 The total number of pixels), calculate the linear correlation coefficient between each pixel Q2m (1≤m≤M2) and the pixel R2m (1≤m≤M2) in the corresponding position in the image I (that is, the same coordinates in the region U) C, that is:

$\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; mean</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; mean</span>}}<span\; class="notranslate">\; )</span>}{\sqrt{\underset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; mean</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}\sqrt{\underset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; mean</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}$

Among them, Q _mean and R _mean are the mean values of Qm (1≤m≤M) and Rm (1≤m≤M). The larger the linear correlation coefficient C is, the higher the similarity between the view synthesis image Pn and the image corresponding to the region U in the image I is.

At the same time, the synthesis accuracy corresponding to the view synthesis image with the highest similarity is written into the code stream of the 3D video sequence. The synthetic precision can be quantified as a syntax element including two values of 0 and 1, where 0 and 1 represent integer pixel projection precision and 1/2 pixel projection precision respectively. Synthetic accuracy D can be coded by 0-order exponential Columbus code, or by 1-bit unsigned integer code.

Example 3

The third embodiment of the present invention relates to a view synthesis predictive coding method. For a three-dimensional video sequence, in the process of encoding an image I in one of the viewpoints V1 (the pixels in the image I described in this embodiment are original pixels), a synthetic precision X is adopted, and the depth and image based The rendering technique is to synthesize the viewpoint composite image P of an area H1 in the viewpoint V1 from the reconstructed image and the reconstructed depth at the same moment in another coded viewpoint V2 in the 3D video sequence.

Among them, the synthetic accuracy X is determined by the following method.

Calculate the variance Vk (1≤k≤K) of pixels in the K regions H3k (1≤k≤K, k is an integer) in the image I, take the average value V of Vk, and pass a function f(V) about V, Deriving the synthetic accuracy X described; the f(V) such as

1) Where E is a constant, such as E=15, Indicates that x is rounded down, and the synthesis precision X is 1/D pixel precision;

2) Wherein F is a constant, such as F=10, and the synthesis precision X is 1/D pixel precision.

At the same time, write the composite precision X into the code stream of the 3D video sequence. The composite precision can be numericalized as a syntax element including N non-negative integers, and its value Y respectively indicates that the composite precision X is 1/Y pixel precision.

Example 4

A fourth embodiment of the present invention relates to a view synthesis predictive decoding method. When decoding an image I in a viewpoint V1 in a three-dimensional video sequence (the pixels in the image I described in this embodiment are reconstructed pixels), the image I is parsed from the code stream of the three-dimensional video sequence Corresponding composite accuracy X; using the composite accuracy X, using depth-based and image rendering techniques, the reconstructed image and the reconstructed depth (and the corresponding camera parameters, such as focal length, camera coordinates, etc.) to synthesize a view synthesis image P of an area H1 in the view V1. Described area H1 can be whole described image I, also can be a designated area in image I (for example a center coincides with image I center and size is the matrix window of 1/2 of image I size), also It can be a macroblock or block in image I, and so on. The view synthesis image P is used to generate prediction pixels required for view synthesis prediction in the process of decoding the image I, that is, there is at least one macroblock or block that adopts view synthesis prediction in the process of decoding the image I, The pixels in the view synthesis image P are selected as the prediction pixels of the macroblock or block. For example, when using integer pixel synthesis precision, a pixel in a rectangular area with the same size as the macroblock or block can be taken from P as the predicted pixel of the macroblock or block; when P adopts sub-pixel precision, from P Extract M×J pixels from an M×L area in P, or change sampling (that is, upsampling or downsampling) into M×J pixels in an M×L area in P, as the macroblock or block (size is M×J pixels) of predicted pixels.

The synthetic precision X can be integer pixel precision or sub-pixel precision, sub-pixel precision such as 1/2 pixel precision, 1/4 pixel precision, 1/3 pixel precision, 3/8 pixel precision, k/l pixel precision ( k, l are positive integers) and so on.

Example 5

A fifth embodiment of the present invention relates to a view synthesis predictive encoding device. FIG. 1 is a schematic structural diagram of an embodiment of a view synthesis predictive encoding device. The device includes the following two modules: a synthesis-of-view image generation module for generating a synthesis-of-view image with a synthesis precision, and a synthesis precision writing module for writing the synthesis precision information into a three-dimensional video sequence code stream.

The input of the viewpoint synthesis image generation module includes information such as the reconstructed image and reconstruction depth of a coded viewpoint V2 in the 3D video sequence, the synthesis precision X, the coded viewpoint V2 and the camera parameters of the current coded viewpoint V1, and its output includes information such as the current coded viewpoint V1 The viewpoint synthesis image P, its completed function and implementation mode are the same as those in the above-mentioned viewpoint synthesis predictive coding method using a synthesis precision X, using the depth-based and image rendering technology, from the encoded viewpoint V2 in the 3D video sequence The function of synthesizing the reconstructed image and the reconstructed depth at a moment (or another moment, such as frame T-1 ) with the viewpoint synthesis image P of an area H1 in the viewpoint V1 is the same as that in the embodiment.

The input of the composite precision writing module includes the composite precision X and the 3D video sequence code stream, and its output includes the 3D video sequence code stream containing the composite precision X. In the encoding method, the function of writing the synthesis precision X into the code stream of the 3D video sequence is the same as that in the embodiment.

Example 6

A sixth embodiment of the present invention relates to a view synthesis predictive encoding device. Fig. 2 is a schematic structural diagram of another embodiment of a view synthesis predictive encoding device. The difference between this device and the device in Embodiment 5 is that before the viewpoint synthesis image generation module, it also includes a synthesis accuracy decision module based on image similarity, and its input includes the reconstructed image in the encoded viewpoint V2 and reconstructed depth, as well as information such as the camera parameters of the coded viewpoint V2 and the current coded viewpoint V1, the currently coded image I in the current coded viewpoint V1 (the pixels in the image I described in this embodiment are original pixels) , its output is the synthesis accuracy X (as the input of the view synthesis image generation module), its completed function and implementation are the same as those in the above view synthesis predictive coding method using different synthesis accuracy to generate the current coded viewpoint V1 respectively A plurality of view synthesis images Pn (1≤n≤N) in an area H2, calculate the similarity between each of the view synthesis images Pn and the image of the corresponding area in the image I, and select from all Pn The function of the synthesis accuracy corresponding to the view synthesis image with the highest similarity as the synthesis accuracy X is the same as that in the embodiment.

Example 7

A seventh embodiment of the present invention relates to a view synthesis predictive encoding device. Fig. 3 is a schematic structural diagram of another embodiment of a view synthesis predictive encoding device. The difference between this device and the device in Embodiment 5 is that, before the viewpoint synthesis image generation module, it also includes a synthesis accuracy decision module based on texture complexity, and its input includes the currently encoded image I (the one in this embodiment) The pixel in the image I described above is the original pixel), and its output is the synthesis accuracy X described above. Its completed functions and implementations are the same as those in the above-mentioned view synthesis predictive coding method for calculating K regions H3k in the image I (1 ≤k≤K, k is an integer), the variance Vk (1≤k≤K) of the pixel in the pixel), take the average value V of Vk, through a function f(V) about V, derive the function sum of the synthetic precision X The implementation is the same.

Example 8

An eighth embodiment of the present invention relates to a view synthesis predictive decoding device. Fig. 4 is a schematic structural diagram of an embodiment of a view synthesis predictive decoding device. The device includes the following two modules: a synthesis accuracy analysis module that analyzes the synthesis accuracy X from the three-dimensional video sequence code stream, and a view synthesis image generation module that uses the synthesis accuracy X to synthesize the view synthesis image of the viewpoint V1 .

The input of the synthesis accuracy analysis module is the 3D video sequence code stream, and its output is the synthesis accuracy X. Its completed functions and implementation methods are the same as those in the above viewpoint synthesis predictive decoding method when decoding one of the viewpoints V1 in the 3D video sequence. When an image I (the pixels in the image I described in this embodiment are reconstructed pixels), the function sum of the synthesis precision X corresponding to the image I is analyzed from the code stream of the three-dimensional video sequence The implementation is the same;

The input of the view synthesis image generation module includes the synthesis accuracy information X, the reconstructed image and the reconstruction depth of a decoded viewpoint V2, the camera parameters of the decoded viewpoint V2 and the current decoded viewpoint V1, and its output is The viewpoint synthesis image P of an area H1 in the viewpoint V1, its completed functions and implementation methods are the same as those described above in the viewpoint synthesis predictive decoding method by synthesizing the viewpoint from the reconstructed image and the reconstructed depth in the decoded viewpoint V2 The function of the view synthesis image P of an area H1 in V1 is the same as that in the embodiment. The view synthesis image P is used to generate prediction pixels required for view synthesis prediction in the process of decoding the image I.

The view synthesis predictive encoding device and decoding device can be implemented in various ways, for example:

Method 1: use a computer as hardware to add a software program with the same functions as the viewpoint synthesis predictive encoding method and decoding method to implement.

Method 2: use a single-chip microcomputer as hardware and add a software program with the same functions as the viewpoint synthesis predictive encoding method and decoding method to implement.

Method 3: use a digital signal processor as hardware and add a software program with the same functions as the viewpoint synthesis predictive encoding method and decoding method to implement.

Method 4: Design a circuit with the same function as the viewpoint synthesis predictive encoding method and decoding method to implement.

There may also be other methods for implementing the view synthesis predictive encoding device and decoding device, and are not limited to the above four methods.

Example 9

The ninth embodiment of the invention relates to a code stream for view synthesis prediction, which includes information corresponding to synthesis accuracy; the synthesis accuracy is the synthesis accuracy of view synthesis images generated in view synthesis prediction.

The information corresponding to a synthesis precision in the code stream can be used for view synthesis prediction of all frames of all viewpoints in a 3D video sequence (indicating the synthesis accuracy of video synthesis images generated in view synthesis prediction), or can be used for a 3D video sequence Viewpoint synthesis prediction of all frames (or several frames) of a certain viewpoint in a certain viewpoint, or can be used for viewpoint synthesis prediction of a certain frame of a certain viewpoint in a 3D video sequence, or can be used for a certain viewpoint in a 3D video sequence View synthesis prediction of a certain area (such as a macroblock) in a certain frame of .

Although the embodiments of the present invention have been described with reference to the accompanying drawings, various variations or modifications may be made by those skilled in the art within the scope of the appended claims.

Claims (7)

1. A predictive coding method for viewpoint synthesis, characterized in that, when encoding an image I in a viewpoint V1 in a three-dimensional video sequence, a synthetic accuracy X is used to generate an encoded viewpoint V2 in the three-dimensional video sequence The reconstructed image and the reconstructed depth in the viewpoint V1 are synthesized into a view synthesis image P of an area H1 in the viewpoint V1, and the view synthesis image P is used to generate the prediction pixels required for view synthesis prediction in the process of generating the coded image I; at the same time, the area The synthesis precision X of H1 is written into the code stream of the 3D video sequence, wherein the viewpoint V1 is the current encoding viewpoint; the method for obtaining the synthesis precision X includes one of the following methods: 1) Synthesize the view synthesis image Pn of a region H2 in the current coded view V1 by using N kinds of synthesis precision, where N≥2, 1≤n≤N, N and n are integers; The similarity between the images of the corresponding regions, selecting the synthesis precision corresponding to the viewpoint synthesis image with the highest similarity from all viewpoint synthesis images Pn as the synthesis precision X; 2) Calculate the variance Vk of the pixels in the K regions H3k in the image I, take the average value V of the variance Vk of the pixels, and derive the synthesis accuracy X through the function f(V) about the average value V, where 1≤ k≤K, k is an integer, the function f(V) maps V into an integer value, D=f(v), and the synthesis precision X is 1/D pixel precision. 2. A view synthesis predictive coding method as claimed in claim 1, characterized in that, in 1), "calculate the similarity between each view synthesis image Pn and the image of the corresponding region in the image I" includes the following methods one: Method 1: For the M1 pixels Q1m in the viewpoint synthesis image Pn, calculate the difference D1m between each pixel Q1m and the pixel R1m at the corresponding position in the image I, where 1≤n≤N, 1≤m ≤M1, m and M1 are positive integers, M1 is less than or equal to the total number of pixels in the area H2; calculate the average value ValA of the absolute value of D1m or the average value ValB of the square value of D1m, the average value ValA or the average value ValB The smaller the value, the higher the similarity between the view synthesis image Pn and the image of the corresponding region in the image I; Method 2: For the M2 pixels Q2m in the viewpoint synthesis image Pn, calculate the linear correlation coefficient C between each pixel Q2m and the corresponding pixel R2m in the image I, the larger the linear correlation coefficient C, the The higher the similarity between the viewpoint synthesis image Pn and the image of the corresponding area in the image I, where 1≤n≤N, 1≤m≤M2, n, N, m, and M2 are positive integers, and M2 is less than or equal to The total number of pixels in the region H2. 3. A view synthesis predictive decoding method, characterized in that, when decoding an image I in a viewpoint V1 in a three-dimensional video sequence, the synthesis accuracy corresponding to the image I is parsed from the code stream of the three-dimensional video sequence X: Synthesize a view synthesis image P of a region H1 in the view V1 from the reconstructed image and the reconstruction depth in another decoded view V2 in the 3D video sequence using the synthesis precision X, the view synthesis image P It is used to generate the predicted pixels required for view synthesis prediction in the process of generating the decoded image I. For different synthesis precisions X, the methods for generating the predicted pixels are different. 4. A view synthesis predictive encoding device, characterized in that it comprises the following two modules: The view synthesis image generating module adopts a synthetic precision X to generate a synthetic view image, and its input includes a synthetic precision X, a reconstructed image and a reconstructed depth in a coded viewpoint V2 in a 3D video sequence, and the coded viewpoint V2 and the current encoding camera parameter information of a viewpoint V1, the output of which comprises a view synthesis image P of an area H1 of said viewpoint V1, the processing of which includes applying said synthesis precision X when encoding an image I of said viewpoint V1, From the reconstructed image and the reconstructed depth in the coded viewpoint V2, synthesize a viewpoint synthesis image P of a region H1 in the viewpoint V1, and the viewpoint synthesis image P is used for the viewpoint synthesis prediction required in the process of generating the coded image I predicted pixels; The composite precision writing module writes the composite precision X into the code stream of the three-dimensional video sequence, its input includes the composite precision X and the code stream of the three-dimensional video sequence, and its output includes the composite precision X The code stream of the 3D video sequence, the completed processing includes writing the synthesis precision X into the code stream of the 3D video sequence. 5. A view synthesis predictive encoding device as claimed in claim 4, further comprising one of the following modules: Synthesis accuracy decision module based on image similarity, its input includes the reconstructed image and reconstruction depth in the coded viewpoint V2, and the camera parameter information of the coded viewpoint V2 and the current coded viewpoint V1, and its output is the composite Accuracy X, the completed processing includes synthesizing the viewpoint synthesis image Pn of a region H2 in the current coded viewpoint V1 by using N kinds of synthesis precisions, where N≥2, n and N are integers, 1≤n≤N; calculate each viewpoint synthesis The similarity between the image Pn and the image of the corresponding area in the image I, select the synthesis precision corresponding to the viewpoint synthesis image with the highest similarity from Pn as the synthesis precision X; The synthetic accuracy decision module based on texture complexity, its input includes the image I, its output is the synthetic accuracy X, and its completed processing includes calculating the variance Vk of the pixels in the K regions H3k in the image I, taking Vk The average value V of the average value V is derived from the function f(V) of the average value V, and the synthetic precision X is derived, where 1≤k≤K, k and K are integers, and the function f(V) maps V into an integer value , D=f(v), the synthesis precision X is 1/D pixel precision. 6. A view synthesis predictive coding device as claimed in claim 5, wherein in the said image similarity-based synthesis precision decision module, said calculation of images in the corresponding regions of each view synthesis image Pn and image I The similarity between, including one of the following treatments: Process 1: For the M1 pixels Q1m in the view synthesis image Pn, calculate the difference D1m between each pixel Q1m and the corresponding pixel R1m in the image I, where 1≤n≤N, 1≤m ≤M1, m and M1 are positive integers, M1 is less than or equal to the total number of pixels in the area H2; calculate the average value ValA of the absolute value of D1m or the average value ValB of the square value of D1m, the average value ValA or the average value ValB The smaller the value, the higher the similarity between the view synthesis image Pn and the image of the corresponding region in the image I; Processing 2: For the M2 pixels Q2m in the view synthesis image Pn, calculate the linear correlation coefficient C between each pixel Q2m and the corresponding pixel R2m in the image I, the larger the linear correlation coefficient C, the The higher the similarity between the viewpoint synthesis image Pn and the image of the corresponding area in the image I, where 1≤n≤N, 1≤m≤M2, m and M2 are positive integers, and M2 is less than or equal to the area Total number of pixels in H2. 7. A view synthesis predictive decoding device, characterized in that it comprises the following two modules: The synthetic accuracy analysis module parses the synthetic precision X from the three-dimensional video sequence code stream, its input is the three-dimensional video sequence code stream, and its output is the synthetic precision X, and its completed function is when decoding a viewpoint V1 in the three-dimensional video sequence When an image I in the image I, analyze the composite accuracy X corresponding to the image I from the code stream of the three-dimensional video sequence; The viewpoint synthesis image generation module synthesizes the viewpoint synthesis image P in the viewpoint V1 with the synthesis accuracy X, and its input includes synthesis accuracy X, a reconstructed image and reconstruction depth of a decoded viewpoint V2, the decoded viewpoint V2 and The camera parameters of the viewpoint V1, the output of which is a view synthesis image P of an area H1 in the viewpoint V1, and the completed processing includes synthesizing the reconstructed image and the reconstructed depth in the decoded viewpoint V2 in the viewpoint V1 A view synthesis image P of a region H1, the view synthesis image P is used to generate prediction pixels required for view synthesis prediction in the process of decoding the image I.