CN104010180B - Three-dimensional video filtering method and device - Google Patents

Abstract

The embodiment of the invention provides a three-dimensional video filtering method and device. The three-dimensional video filtering method comprises the following steps: projecting pixels in an image plane into three-dimensional space; according to the coordinate values of the pixel to be filtered and the reference pixel in the three-dimensional space, calculating the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space; calculating the similarity of the texture pixel values of the pixel to be filtered and the reference pixel according to the texture pixel values of the pixel to be filtered and the reference pixel; calculating the consistency of the motion characteristics of the pixel to be filtered and the reference pixel according to the texture pixel values of the pixel to be filtered, the reference pixel and the pixel at the same position in the previous frame of image; and determining a filtering weight value according to the spatial proximity, the similarity of the texture pixel values and the consistency of the motion characteristics, and respectively carrying out weighted average on the pixel values of the reference pixels in the reference pixel set to obtain a filtering result of the pixel to be filtered. The embodiment of the invention improves the filtering accuracy of the three-dimensional video.

Description

Three-dimensional video filtering method and device

technical field

Embodiments of the present invention relate to image processing technologies, and in particular, to a three-dimensional video filtering method and device.

Background technique

With the continuous development of visual multimedia technology, 3D video has gradually entered people's lives with its unique depth-of-field effect, and has been applied in many fields such as education, military affairs, entertainment and medical treatment. The current 3D video is mainly divided into two categories according to video content: pure color 3D video and depth-based 3D video. Pure color 3D video directly presents multi-channel color video to users, and its viewpoint position and parallax are fixed, which brings certain limitations to people's viewing. Compared with pure color 3D video, due to the introduction of depth map, depth-based 3D video can synthesize virtual images of any viewpoint through the rendering technology based on depth image. People can choose viewpoint and adjust parallax according to personal preference, so as to enjoy better The joy of 3D video. This free and flexible feature makes depth-based 3D video a widely accepted 3D video format.

The depth-based 3D video content consists of a texture map sequence and a depth map sequence. The texture map intuitively presents the texture features of the object surface, and the depth map reflects the distance between the object and the camera. The specified virtual viewpoint texture image can be synthesized by using the above video content and the rendering technology based on the depth image. However, depth maps and texture maps will introduce a lot of noise in the process of acquisition, encoding and transmission. In the view synthesis stage, the noise in the depth map and texture map will cause geometric distortion and texture distortion of the synthesized image respectively, which will seriously affect people's visual experience. Filtering technology can effectively remove these noises and effectively improve the quality of 3D video.

In the prior art, the denoising method for the texture image is mainly a bilateral filter, which uses the surrounding pixels of the pixel to be filtered as a reference and performs weighted average on them to obtain the filtering result. In the process of weight calculation, it mainly refers to the similarity of pixel position proximity and pixel value in the image. The filtering method believes that: the closer the distance between two pixel points on the image plane, the stronger the correlation; the more similar the pixel values of the two pixel points, the stronger the correlation.

FIG. 1 is a schematic diagram of computing proximity of a bilateral filter in the prior art. The problem of the existing technology is that since the pixels in the image are the reproduction of the points in the real three-dimensional space on the two-dimensional image plane, however, the bilateral filter does not start from the real three-dimensional scene when considering the proximity of the pixels, and its calculation results Not accurate, as shown in Figure 1, where A', B', and C' are three points in the real scene, and their positions in the image plane are A, B, and C after being collected by the camera, and A and C are on the plane The distance on is equal to the distance between B and C on the plane. If C is filtered, A and B are reference pixels. As shown in Figure 1, it can be clearly seen that in the three-dimensional space, the proximity of B and C is stronger, while the bilateral filter believes that the proximity of A and C and B and C are consistent, so the accuracy of the filtering results is not good. high.

Contents of the invention

Embodiments of the present invention provide a three-dimensional video filtering method and device to overcome the problem of low accuracy of filtering results in the prior art.

In a first aspect, an embodiment of the present invention provides a three-dimensional video filtering method, including:

Projecting pixels in the image plane to a three-dimensional space; the pixels include pixels to be filtered and a set of reference pixels;

Calculate the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; wherein, the The set of reference pixels and the pixel to be filtered are in the same frame of image;

calculating the similarity of texel values between the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set;

Calculate the pixel to be filtered and the reference pixel according to the texture pixel value of the pixel to be filtered, the reference pixel in the reference pixel set, and the pixel at the same position in the previous frame image of the frame where the pixel to be filtered is located The consistency of the movement characteristics;

The filtering weight is determined according to the spatial proximity, texture pixel value similarity and motion feature consistency, and the pixel values of the reference pixels in the reference pixel set are respectively weighted and averaged to obtain the filtering result of the pixel to be filtered.

With reference to the first aspect, in the first implementation of the first aspect, when filtering the depth image, the filtering is determined according to the spatial proximity, the similarity of the texel value corresponding to the depth pixel, and the consistency of the motion feature Weighted values of the depth pixel values of the reference pixels in the reference pixel set are respectively weighted and averaged to obtain a filtering result of the depth pixel values of the pixels to be filtered in the depth image; or,

When filtering a texture image, determine the weight of filtering according to the spatial proximity, the similarity of the texel and the consistency of the motion feature, and perform weighted average of the texture pixel values of the reference pixels in the reference pixel set respectively , obtaining a filtering result of the texel value of the pixel to be filtered in the texture image.

In combination with the first aspect, or the first implementation of the first aspect, in the second implementation of the first aspect, the projecting the pixels in the image plane to a three-dimensional space includes:

Using the depth image information, viewpoint position information and reference camera parameter information provided by the 3D video, project the pixel from the image plane to the 3D space; the depth image information includes the depth pixel value of the pixel.

With reference to the second implementation of the first aspect, in the third implementation of the first aspect, the pixel is projected from the image plane to the three-dimensional space, including:

According to the formula P=R ^-1 (dA ^-1 pt), calculate the coordinate value after the pixel is projected into the three-dimensional space;

Among them, R and t are the rotation matrix and translation vector of the reference camera, A is the parameter matrix of the reference camera, $A=\left(\begin{array}{ccc}{f}_x& r& o_x\\ 0& f_{\mathrm{the\; y}}& o_{\mathrm{the\; y}}\\ 0& 0& 1\end{array}\right),$ $p=\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)$ is the coordinate value of the pixel in the image plane, $p=\left(\begin{array}{c}x\\ \mathrm{the\; y}\\ z\end{array}\right)$ For the coordinate value of the pixel in the three-dimensional space, d is the depth pixel value of the pixel; f _x and f _y are respectively the normalized focal lengths in the horizontal and vertical directions, and r is the radial distortion coefficient, ( o _x , o _y ) are coordinate values of a reference point on the image plane; the reference point is the intersection of the optical axis of the reference camera and the image plane.

In combination with the first aspect, or any one of the first-third implementation manners of the first aspect, in the fourth implementation manner of the first aspect, the spatial proximity is determined by the pixel to be filtered and the pixel to be filtered in three-dimensional space The distance of the reference pixel is calculated as the input value of the function; the output value of the function increases as the input value decreases;

The texel value similarity is calculated by using the difference between the texture pixel values of the pixel to be filtered and the reference pixel as an input value of a function; the output value of the function increases as the input value decreases;

The motion feature consistency is obtained by calculating whether the motion features of the pixel to be filtered and the reference pixel are consistent, including:

When the difference between the texture pixel value of the pixel to be filtered and the pixel at the corresponding position in the previous frame, and the difference between the texture pixel value between the reference pixel and the pixel at the corresponding position in the previous frame are greater than or less than the predetermined When the threshold is set, it is determined that the motion state of the pixel to be filtered is consistent with the reference pixel; otherwise, it is determined that the motion state of the pixel to be filtered is inconsistent with the reference pixel.

In combination with the first aspect, or any one of the first to fourth implementations of the first aspect, in the fifth implementation of the first aspect, according to the spatial proximity, texel value similarity and motion feature Consistency determines the weight of the filter, respectively performs weighted average on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered, including:

According to formula (1): ${{\mathrm{D.}}_p}^{\′}=\frac{\underset{q\∈K}{\mathrm{\Σ}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q){\mathrm{D.}}_q}{\underset{q\∈K}{\mathrm{\Σ}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)}$ Calculate and obtain the filtering result of the depth pixel value of the pixel to be filtered; or,

According to formula (2): ${T_p}^{\′}=\frac{\underset{q\∈K}{\mathrm{\Σ}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)T_q}{\underset{q\∈K}{\mathrm{\Σ}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)}$ Calculate and obtain the filtering result of the texel value of the pixel to be filtered;

in, for calculating the spatial proximity between the pixel to be filtered and the reference pixel;

f _T (T _p , T _q )=f _T (||T _p −T _q ||) is used to calculate the texture pixel value similarity between the pixel to be filtered and the reference pixel;

$f_m(m_p,m_q)=f_m\left(\right||m_p-m_q|\left|\right)=\left\{\begin{array}{c}1,\mathrm{if}\left(\right|T_p-T_{p^{\&prime;}}|<\mathrm{the\; th})\&CirclePlus;\left(\right|T_q-T_{q^{\&prime;}}|<\mathrm{the\; th})=0\\ 0,\mathrm{else}\end{array}\right.$ used to calculate the motion feature consistency between the pixel to be filtered and the reference pixel;

Among them, p is the pixel to be filtered, q is the reference pixel, K is the reference pixel set, D _p ' is the depth pixel value after p filtering, D _q is the depth pixel value of q, and P and Q are p and q in the three-dimensional space The coordinate values in , T _p , T _q are the texture pixel values of p and q, T _p' , T _q' are the texture pixel values of p and q at the same position in the previous frame, T _p ' is the texture after p filtering Pixel value, th is the preset texture pixel difference threshold.

In a second aspect, an embodiment of the present invention provides a three-dimensional video filtering method, including:

Projecting pixels in the image plane to a three-dimensional space; the pixels include pixels to be filtered and a set of reference pixels;

Calculate the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; wherein, the The reference pixels are set in the same frame of image and adjacent multi-frame images where the pixel to be filtered is located;

calculating the similarity of texel values between the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set;

calculating the temporal proximity of the pixel to be filtered and the reference pixel according to the time interval of the frame where the pixel to be filtered and the reference pixel in the set of reference pixels are located;

The filtering weight is determined according to the spatial proximity, texture pixel value similarity and temporal proximity, and the pixel values of the reference pixels in the reference pixel set are respectively weighted and averaged to obtain the filtering result of the pixel to be filtered.

With reference to the second aspect, in the first implementation of the second aspect, when filtering the depth image, the filtering is determined according to the spatial proximity, the similarity of the texel value corresponding to the depth pixel, and the temporal proximity Weighted values of the depth pixel values of the reference pixels in the reference pixel set are respectively weighted and averaged to obtain a filtering result of the depth pixel values of the pixels to be filtered in the depth image; or,

When filtering the texture image, determine the weight of filtering according to the spatial proximity, the texture pixel similarity and the temporal proximity, and perform weighted average on the texture pixel values of the reference pixels in the reference pixel set respectively , obtaining a filtering result of the texel value of the pixel to be filtered in the texture image.

In combination with the second aspect, or the first implementation manner of the second aspect, in the second implementation manner of the second aspect, the projecting the pixels in the image plane to a three-dimensional space includes:

Using the depth image information, viewpoint position information and reference camera parameter information provided by the 3D video, project the pixel from the image plane to the 3D space; the depth image information includes the depth pixel value of the pixel.

With reference to the second implementation of the second aspect, in the third implementation of the second aspect, the depth image information, viewpoint position information and reference camera parameter information provided by the 3D video are used to convert the pixels from the image plane to Projection into 3D space, including:

According to the formula P=R ^-1 (dA ^-1 pt), calculate the coordinate value after the pixel is projected into the three-dimensional space;

Among them, R and t are the rotation matrix and translation vector of the reference camera, A is the parameter matrix of the reference camera, $A=\left(\begin{array}{ccc}{f}_x& r& o_x\\ 0& f_{\mathrm{the\; y}}& o_{\mathrm{the\; y}}\\ 0& 0& 1\end{array}\right),$ $p=\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)$ is the coordinate value of the pixel in the image plane, $p=\left(\begin{array}{c}x\\ \mathrm{the\; y}\\ z\end{array}\right)$ For the coordinate value of the pixel in the three-dimensional space, d is the depth pixel value of the pixel; f _x and f _y are respectively the normalized focal lengths in the horizontal and vertical directions, and r is the radial distortion coefficient, ( o _x , o _y ) are coordinate values of a reference point on the image plane; the reference point is the intersection of the optical axis of the reference camera and the image plane.

In combination with the second aspect, or any of the first-third implementation manners of the second aspect, in the fourth implementation manner of the second aspect, the spatial proximity is determined by the pixel to be filtered and the pixel to be filtered in three-dimensional space The distance of the reference pixel is calculated as the input value of the function; the output value of the function increases as the input value decreases;

The texel value similarity is calculated by using the difference between the texture pixel values of the pixel to be filtered and the reference pixel as an input value of a function; the output value of the function increases as the input value decreases;

The temporal proximity is calculated by using the time interval of the frame where the pixel to be filtered and the reference pixel are located as an input value of a function; the output value of the function increases as the input value decreases.

In combination with the second aspect, or any of the first-fourth implementations of the second aspect, in the fifth implementation of the second aspect, according to the spatial proximity, texel value similarity and time domain Proximity determines the weight of the filter, and performs weighted average on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered, including:

According to formula (3): ${{\mathrm{D.}}_p}^{\&prime;}=\frac{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N){\mathrm{D.}}_{q_i}\right)}{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)\right)}$ Calculate and obtain the filtering result of the depth pixel value of the pixel to be filtered; or,

According to formula (4): ${T_p}^{\&prime;}=\frac{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)T_{q_i}\right)}{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)\right)}$ Calculate and obtain the filtering result of the texel value of the pixel to be filtered;

in, for calculating the spatial proximity between the pixel to be filtered and the reference pixel;

being used to calculate the similarity of texel values between the pixel to be filtered and the reference pixel;

f _tem (i, N)=f _tem (||iN||) is used to calculate the temporal proximity of the pixel to be filtered and the reference pixel;

Among them, N is the frame number of the frame where the pixel to be filtered is located, i is the frame number of the frame where the reference pixel is located, and the value of i is an integer in the interval [Nm, N+n], m and n are respectively before the frame where the pixel to be filtered is located , the number of subsequent reference frames, m and n are non-negative integers, p is the pixel to be filtered, q _i is the reference pixel in the i-th frame, K _i is the reference pixel set in the i-th frame, and D _p ' is p Filtered depth pixel values, The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the i-th frame in three-dimensional space, T _p , are the texture pixel values of p and q in the i-th frame, respectively, and T _p ' is the texture pixel value of p after filtering.

In a third aspect, an embodiment of the present invention provides a three-dimensional video filtering device, including:

A projection module, configured to project pixels in the image plane to a three-dimensional space; the pixels include a pixel to be filtered and a set of reference pixels;

A calculation module, configured to calculate the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space property; wherein, the set of reference pixels and the pixel to be filtered are in the same frame of image;

The calculation module is further configured to calculate the similarity of the texel value of the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set;

The calculation module is further configured to calculate the pixel to be filtered, the reference pixel in the reference pixel set, and the texture pixel value of the pixel at the same position in the previous frame image of the frame where the pixel to be filtered is located. The motion feature consistency of the pixel to be filtered and the reference pixel;

A filtering module, configured to determine a filtering weight according to the spatial proximity, texture pixel value similarity, and motion feature consistency, and perform weighted average on the pixel values of the reference pixels in the reference pixel set to obtain the to-be-filtered The result of filtering for pixels.

With reference to the third aspect, in the first implementation manner of the third aspect, the filtering module is specifically used for:

When filtering the depth image, the weight of the filter is determined according to the spatial proximity, the similarity of the texture pixel value corresponding to the depth pixel and the consistency of the motion feature, and the depth of the reference pixel in the reference pixel set is respectively Performing weighted average of the pixel values to obtain a filtering result of the depth pixel values of the pixels to be filtered in the depth image; or,

When filtering a texture image, determine the weight of filtering according to the spatial proximity, the similarity of the texel and the consistency of the motion feature, and perform weighted average of the texture pixel values of the reference pixels in the reference pixel set respectively , obtaining a filtering result of the texel value of the pixel to be filtered in the texture image.

In combination with the third aspect, or the first implementation manner of the third aspect, in the second implementation manner of the third aspect, the projection module is specifically used for:

Using the depth image information, viewpoint position information and reference camera parameter information provided by the 3D video, project the pixel from the image plane to the 3D space; the depth image information includes the depth pixel value of the pixel.

In combination with the second implementation of the third aspect, in the third implementation of the third aspect, the projection module is specifically used for:

According to the formula P=R ^-1 (dA ^-1 pt), calculate the coordinate value after the pixel is projected into the three-dimensional space;

Among them, R and t are the rotation matrix and translation vector of the reference camera, A is the parameter matrix of the reference camera, $A=\left(\begin{array}{ccc}{f}_x& r& o_x\\ 0& f_{\mathrm{the\; y}}& o_{\mathrm{the\; y}}\\ 0& 0& 1\end{array}\right),$ $p=\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)$ is the coordinate value of the pixel in the image plane, $p=\left(\begin{array}{c}x\\ \mathrm{the\; y}\\ z\end{array}\right)$ For the coordinate value of the pixel in the three-dimensional space, d is the depth pixel value of the pixel; f _x and f _y are respectively the normalized focal lengths in the horizontal and vertical directions, and r is the radial distortion coefficient, ( o _x , o _y ) are coordinate values of a reference point on the image plane; the reference point is the intersection of the optical axis of the reference camera and the image plane.

In combination with the third aspect, or any of the first-third implementation manners of the third aspect, in the fourth implementation manner of the third aspect, the spatial proximity is determined by the pixel to be filtered and the pixel to be filtered in three-dimensional space The distance of the reference pixel is calculated as the input value of the function; the output value of the function increases as the input value decreases;

The texel value similarity is calculated by using the difference between the texture pixel values of the pixel to be filtered and the reference pixel as an input value of a function; the output value of the function increases as the input value decreases;

The motion feature consistency is obtained by calculating whether the motion features of the pixel to be filtered and the reference pixel are consistent, including:

When the difference between the texture pixel value of the pixel to be filtered and the pixel at the corresponding position in the previous frame, and the difference between the texture pixel value between the reference pixel and the pixel at the corresponding position in the previous frame are greater than or less than the predetermined When the threshold is set, it is determined that the motion state of the pixel to be filtered is consistent with the reference pixel; otherwise, it is determined that the motion state of the pixel to be filtered is inconsistent with the reference pixel.

In combination with the third aspect, or any one of the first-fourth implementation manners of the third aspect, in the fifth implementation manner of the third aspect, the filtering module is specifically used for:

According to formula (1): ${{\mathrm{D.}}_p}^{\&prime;}=\frac{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q){\mathrm{D.}}_q}{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)}$ Calculate and obtain the filtering result of the depth pixel value of the pixel to be filtered; or,

According to formula (2): ${T_p}^{\&prime;}=\frac{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)T_q}{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)}$ Calculate and obtain the filtering result of the texel value of the pixel to be filtered;

in, for calculating the spatial proximity between the pixel to be filtered and the reference pixel;

f _T (T _p , T _q )=f _T (||T _p −T _q ||) is used to calculate the texture pixel value similarity between the pixel to be filtered and the reference pixel;

$f_m(m_p,m_q)=f_m\left(\right||m_p-m_q|\left|\right)=\left\{\begin{array}{c}1,\mathrm{if}\left(\right|T_p-T_{p^{\&prime;}}|<\mathrm{the\; th})\&CirclePlus;\left(\right|T_q-T_{q^{\&prime;}}|<\mathrm{the\; th})=0\\ 0,\mathrm{else}\end{array}\right.$ used to calculate the motion feature consistency between the pixel to be filtered and the reference pixel;

Among them, p is the pixel to be filtered, q is the reference pixel, K is the reference pixel set, D _p ' is the depth pixel value after p filtering, D _q is the depth pixel value of q, and P and Q are p and q in the three-dimensional space The coordinate values in , T _p , T _q are the texture pixel values of p and q, T _p' , T _q' are the texture pixel values of p and q at the same position in the previous frame, T _p ' is the texture after p filtering Pixel value, th is the preset texture pixel difference threshold.

In a fourth aspect, an embodiment of the present invention provides a three-dimensional video filtering device, including:

A projection module, configured to project pixels in the image plane to a three-dimensional space; the pixels include a pixel to be filtered and a set of reference pixels;

A calculation module, configured to calculate the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space property; wherein, the set of reference pixels is located in the same frame of image as the pixel to be filtered and in adjacent multiple frames of images;

The calculation module is further configured to calculate the similarity of the texel value of the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set;

The calculation module is further configured to calculate the temporal proximity of the pixel to be filtered and the reference pixel according to the time interval of the frame where the pixel to be filtered and the reference pixel in the set of reference pixels are located;

A filtering module, configured to determine a filtering weight according to the spatial proximity, texture pixel value similarity, and temporal proximity, and perform a weighted average on the pixel values of the reference pixels in the reference pixel set to obtain the to-be-filtered The result of filtering for pixels.

With reference to the fourth aspect, in the first implementation manner of the fourth aspect, the filtering module is specifically used for:

When filtering the depth image, the weight of filtering is determined according to the spatial proximity, the similarity of the texture pixel value corresponding to the depth pixel and the temporal proximity, and the depth of the reference pixel in the reference pixel set is respectively Performing weighted average of the pixel values to obtain a filtering result of the depth pixel values of the pixels to be filtered in the depth image; or,

When filtering the texture image, determine the weight of filtering according to the spatial proximity, the texture pixel similarity and the temporal proximity, and perform weighted average on the texture pixel values of the reference pixels in the reference pixel set respectively , obtaining a filtering result of the texel value of the pixel to be filtered in the texture image.

With reference to the fourth aspect, or the first implementation manner of the fourth aspect, in the second implementation manner of the fourth aspect, the projection module is specifically used for:

Using the depth image information, viewpoint position information and reference camera parameter information provided by the 3D video, project the pixel from the image plane to the 3D space; the depth image information includes the depth pixel value of the pixel.

In combination with the second implementation of the fourth aspect, in the third implementation of the fourth aspect, the projection module is specifically used for:

According to the formula P=R ^-1 (dA ^-1 pt), calculate the coordinate value after the pixel is projected into the three-dimensional space;

Among them, R and t are the rotation matrix and translation vector of the reference camera, A is the parameter matrix of the reference camera, $A=\left(\begin{array}{ccc}{f}_x& r& o_x\\ 0& f_{\mathrm{the\; y}}& o_{\mathrm{the\; y}}\\ 0& 0& 1\end{array}\right),$ $p=\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)$ is the coordinate value of the pixel in the image plane, $p=\left(\begin{array}{c}x\\ \mathrm{the\; y}\\ z\end{array}\right)$ For the coordinate value of the pixel in the three-dimensional space, d is the depth pixel value of the pixel; f _x and f _y are respectively the normalized focal lengths in the horizontal and vertical directions, and r is the radial distortion coefficient, ( o _x , o _y ) are coordinate values of a reference point on the image plane; the reference point is the intersection of the optical axis of the reference camera and the image plane.

In combination with the fourth aspect, or any of the first-third implementation manners of the fourth aspect, in the fourth implementation manner of the fourth aspect, the spatial proximity is determined by the pixel to be filtered and the pixel to be filtered in three-dimensional space The distance of the reference pixel is calculated as the input value of the function; the output value of the function increases as the input value decreases;

The texel value similarity is calculated by using the difference between the texture pixel values of the pixel to be filtered and the reference pixel as an input value of a function; the output value of the function increases as the input value decreases;

The temporal proximity is calculated by using the time interval of the frame where the pixel to be filtered and the reference pixel are located as an input value of a function; the output value of the function increases as the input value decreases.

In combination with the fourth aspect, or any one of the first-fourth implementation manners of the fourth aspect, in the fifth implementation manner of the fourth aspect, the filtering module is specifically used for:

According to formula (3): ${{\mathrm{D.}}_p}^{\&prime;}=\frac{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N){\mathrm{D.}}_{q_i}\right)}{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)\right)}$ Calculate and obtain the filtering result of the depth pixel value of the pixel to be filtered; or,

According to formula (4): ${T_p}^{\&prime;}=\frac{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)T_{q_i}\right)}{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)\right)}$ Calculate and obtain the filtering result of the texel value of the pixel to be filtered;

in, for calculating the spatial proximity between the pixel to be filtered and the reference pixel;

being used to calculate the similarity of texel values between the pixel to be filtered and the reference pixel;

f _tem (i, N)=f _tem (||iN||) is used to calculate the temporal proximity of the pixel to be filtered and the reference pixel;

Among them, N is the frame number of the frame where the pixel to be filtered is located, i is the frame number of the frame where the reference pixel is located, and the value of i is an integer in the interval [Nm, N+n], m and n are respectively before the frame where the pixel to be filtered is located , the number of subsequent reference frames, m and n are non-negative integers, p is the pixel to be filtered, q _i is the reference pixel in the i-th frame, K _i is the reference pixel set in the i-th frame, and D _p ' is p Filtered depth pixel values, The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the i-th frame in three-dimensional space, T _p , are the texture pixel values of p and q in the i-th frame, respectively, and T _p ' is the texture pixel value of p after filtering.

The three-dimensional video filtering method and device of the embodiment of the present invention use the relationship between the pixel to be filtered and the reference pixel in the real three-dimensional space to calculate the spatial proximity and texture pixel of the pixel to be filtered and the reference pixel in the three-dimensional space Value similarity, motion feature consistency, and temporal proximity; determine the weight of filtering based on spatial proximity, texture pixel value similarity, and motion feature consistency, or determine the weight of filtering based on spatial proximity, texture pixel value similarity, and temporal proximity To determine the weight of the filter, the pixel values of the reference pixels in the reference pixel set are weighted and averaged to obtain the filtering result of the pixel to be filtered, and the spatial proximity and texture pixel value similarity are comprehensively considered when calculating the weight , motion feature consistency, and time-domain proximity, because the location in the real 3D space is used to calculate the spatial proximity, and since the 3D video is composed of a series of images collected at different times, the pixels between different frames are also There is a correlation, so the weight value considers the continuity between frames after temporal proximity filtering, and adds consideration of the consistency of motion features between pixels, which improves the accuracy of filtering results and solves the problem of existing problems. The accuracy of the filtering results in the technology is not high.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 is a schematic diagram of computing proximity of a bilateral filter in the prior art;

FIG. 2 is a flow chart of Embodiment 1 of the three-dimensional video filtering method of the present invention;

Fig. 3 is a schematic diagram of pixel projection according to Embodiment 1 of the method of the present invention;

FIG. 4 is a flow chart of Embodiment 2 of the three-dimensional video filtering method of the present invention;

5 is a schematic diagram of reference pixel selection in the second embodiment of the method of the present invention;

FIG. 6 is a schematic structural diagram of an embodiment of a three-dimensional video filtering device according to the present invention;

FIG. 7 is a schematic structural diagram of an embodiment of a three-dimensional video filtering device according to the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

FIG. 2 is a flowchart of Embodiment 1 of the three-dimensional video filtering method of the present invention, and FIG. 3 is a schematic diagram of pixel projection of Embodiment 1 of the method of the present invention. As shown in Figure 2, the method of this embodiment may include:

Step 201. Project the pixels in the image plane to a three-dimensional space; the pixels include pixels to be filtered and a set of reference pixels.

Optionally, project pixels in the image plane to 3D space, including:

Using the depth image information, viewpoint position information and reference camera parameter information provided by the 3D video, project the pixel from the image plane to the 3D space; the depth image information includes the depth pixel value of the pixel.

Optionally, projecting pixels from an image plane to a three-dimensional space using depth image information, viewpoint position information and reference camera parameter information provided by a three-dimensional video includes:

According to the formula P=R ^-1 (dA ^-1 pt), calculate the coordinate value after the pixel is projected into the three-dimensional space;

Among them, R and t are the rotation matrix and translation vector of the reference camera, A is the parameter matrix of the reference camera, $A=\left(\begin{array}{ccc}{f}_x& r& o_x\\ 0& f_{\mathrm{the\; y}}& o_{\mathrm{the\; y}}\\ 0& 0& 1\end{array}\right),$ $p=\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)$ is the coordinate value of the pixel in the image plane, $p=\left(\begin{array}{c}x\\ \mathrm{the\; y}\\ z\end{array}\right)$ For the coordinate value of the pixel in the three-dimensional space, d is the depth pixel value of the pixel; f _x and f _y are respectively the normalized focal lengths in the horizontal and vertical directions, and r is the radial distortion coefficient, ( o _x , o _y ) are coordinate values of a reference point on the image plane; the reference point is the intersection of the optical axis of the reference camera and the image plane.

As shown in Figure 3, the plane where the uv coordinates are located is the image plane, the pixel position in the three-dimensional space is represented by the coordinates in the world coordinate system, p is the pixel in the image plane, and the coordinate value of the pixel in the image plane for $p=\left(\begin{array}{c}u\\ v\\ 1\end{array}\right),$ Apply the three-dimensional projection technology to project the pixel to point P in the world coordinate system, and the coordinate value of the pixel in the world coordinate system is $p=\left(\begin{array}{c}x\\ \mathrm{the\; y}\\ z\end{array}\right),$ The coordinate value can be calculated by the formula P=R ^-1 (dA ^-1 pt), where R and t are the rotation matrix and translation vector of the reference camera, d is the depth pixel value of the pixel, which can be obtained from the depth map provided by the 3D video The information is obtained, and A is the reference camera parameter matrix $A=\left(\begin{array}{ccc}{f}_x& r& o_x\\ 0& f_{\mathrm{the\; y}}& o_{\mathrm{the\; y}}\\ 0& 0& 1\end{array}\right),$ f _x and f _y are the normalized focal lengths in the horizontal and vertical directions respectively, r is the radial distortion coefficient, (o _x , o _y ) is the coordinate value of the reference point on the image plane; the reference point is The intersection of the optical axis of the reference camera and the image plane.

Step 202, according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space, calculate the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space; wherein, the reference pixel set and the pixel to be filtered are in in the same image frame.

Optionally, the spatial proximity is calculated by taking the distance between the pixel to be filtered and the reference pixel in three-dimensional space as an input value of a function; the output value of the function increases as the input value decreases.

Specifically, the spatial distance between two points can reflect their spatial proximity. The closer the distance, the stronger the correlation, and the greater the spatial proximity, that is, the pixel to be filtered and the reference pixel in the reference pixel set can be compared in the three-dimensional space. The coordinate value calculates the spatial distance, and takes the spatial distance as an input value. For example, the spatial proximity is calculated through the Gaussian function. The function of calculating the spatial proximity can also be other functions, but it is necessary to ensure that the output value of the function decreases with the input value. Small and large; wherein, the reference pixel set in this embodiment is in the same frame as the pixel to be filtered.

Step 203 , according to the pixel to be filtered and the texel value of the reference pixel in the reference pixel set, calculate the similarity of the texel value of the pixel to be filtered and the reference pixel.

Optionally, the texel value similarity is calculated by using the difference between the texel values of the pixel to be filtered and the reference pixel as an input value of a function; the output value of the function decreases as the input value And increase.

Specifically, the degree of difference in texture features between two points reflects their similarity. The more similar the texture, the stronger the correlation, and the greater the similarity of the texture pixel value, that is, the texture of the pixel to be filtered and the reference pixel can be Calculate the difference of pixel values, and use the difference as an input value. For example, the similarity of texture pixel values is calculated by a Gaussian function. The function of calculating the similarity of texture pixel values can also be other functions, but it is necessary to ensure that the output value of this function follows the increase as the input value decreases.

Step 204, according to the pixel to be filtered, the reference pixel in the reference pixel set, and the texel value of the pixel at the same position in the previous frame image where the pixel to be filtered is located, calculate the consistency of motion characteristics between the pixel to be filtered and the reference pixel.

Optionally, the motion feature consistency is obtained by calculating whether the motion features of the pixel to be filtered and the reference pixel are consistent, including:

When the difference between the texture pixel value of the pixel to be filtered and the pixel at the corresponding position in the previous frame, and the difference between the texture pixel value between the reference pixel and the pixel at the corresponding position in the previous frame are greater than or less than the predetermined When the threshold is set, it is determined that the motion characteristics of the pixel to be filtered and the reference pixel are consistent; otherwise, it is determined that the motion characteristics of the pixel to be filtered and the reference pixel are inconsistent.

Specifically, the relative movement relationship between two points also reflects their movement similarity, and the more similar the movement, the stronger the correlation. Since it is difficult to obtain the motion information of a pixel point from a three-dimensional video sequence, the embodiment of the present invention uses the difference between the pixels of the two frames before and after the texel at the same position in the image plane to determine whether the pixel is moving. When a preset threshold is reached, the motion feature of the pixel is considered to be motion, otherwise, the motion feature of the pixel is considered to be no motion; further, the pixel to be filtered and the reference pixel of the two frames before and after are used in the same position texel in the image plane The difference between the pixel to be filtered and the reference pixel to determine whether the motion characteristics are consistent, when the difference is greater than or less than a preset threshold, it is considered that the motion characteristics of the pixel to be filtered and the reference pixel are consistent, otherwise, the motion Features are inconsistent. If the motion characteristics of the pixels are consistent, it is considered that there is correlation, otherwise it is considered that there is no correlation.

Step 205: Determine the filtering weight according to the spatial proximity, texture pixel value similarity and motion feature consistency, and perform weighted average on the pixel values of the reference pixels in the reference pixel set respectively to obtain the filtering result of the pixel to be filtered.

Optionally, when filtering the depth image, the weight of the filter is determined according to the spatial proximity, the similarity of the texel value corresponding to the depth pixel, and the consistency of the motion feature, respectively for the reference pixel set performing weighted averaging on the depth pixel values of the reference pixels to obtain a filtering result of the depth pixel values of the to-be-filtered pixels in the to-be-depth image; or,

When filtering a texture image, determine the weight of filtering according to the spatial proximity, the similarity of the texel and the consistency of the motion feature, and perform weighted average of the texture pixel values of the reference pixels in the reference pixel set respectively , obtaining a filtering result of the texel value of the pixel to be filtered in the image to be textured.

Optionally, according to the spatial proximity, texture pixel value similarity and motion feature consistency to determine the weight of the filter, the pixel values of the reference pixels in the reference pixel set are weighted and averaged to obtain the filtering result of the pixel to be filtered, including:

According to formula (1): ${{\mathrm{D.}}_p}^{\&prime;}=\frac{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q){\mathrm{D.}}_q}{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)}$ calculating a filtering result to obtain the depth pixel value of the pixel to be filtered; or,

According to formula (2): ${T_p}^{\&prime;}=\frac{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)T_q}{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)}$ Calculate and obtain the filtering result of the texture pixel value of the pixel to be filtered;

in, Used to calculate the spatial proximity of the pixel to be filtered and the reference pixel;

f _T (T _p , T _q )=f _T (||T _p -T _q ||) is used to calculate the texture pixel value similarity between the pixel to be filtered and the reference pixel;

$f_m(m_p,m_q)=f_m\left(\right||m_p-m_q|\left|\right)=\left\{\begin{array}{c}1,\mathrm{if}\left(\right|T_p-T_{p^{\&prime;}}|<\mathrm{the\; th})\&CirclePlus;\left(\right|T_q-T_{q^{\&prime;}}|<\mathrm{the\; th})=0\\ 0,\mathrm{else}\end{array}\right.$ It is used to calculate the motion feature consistency between the pixel to be filtered and the reference pixel;

Among them, p is the pixel to be filtered, q is the reference pixel, K is the reference pixel set, D _p ' is the depth pixel value after p filtering, D _q is the depth pixel value of q, and P and Q are p and q in the three-dimensional space The coordinate values in , T _p , T _q are the texture pixel values of p and q, T _p' , T _q' are the texture pixel values of p and q at the same position in the previous frame, T _p ' is the texture after p filtering Pixel value, th is the preset texture pixel difference threshold.

Specifically, according to the formula ${{\mathrm{D.}}_p}^{\&prime;}=\frac{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q){\mathrm{D.}}_q}{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)}$ Carry out weighted average on the reference pixels in the reference pixel set, calculate and obtain the filtering result of the depth pixel value of the pixel to be filtered; according to the formula ${T_p}^{\&prime;}=\frac{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)T_q}{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)}$ Carrying out weighted averaging on the reference pixels in the reference pixel set, and calculating and obtaining the filtering result of the texel value of the pixel to be filtered;

in, Used to calculate the spatial proximity of the pixel to be filtered and the reference pixel; the input value of the function is the spatial distance between the pixel to be filtered and the reference pixel; the output value of the function increases as the input value decreases ;

f _T (T _p , T _q )=f _T (||T _p -T _q ||) is used to calculate the texture pixel value similarity between the pixel to be filtered and the reference pixel; the input value of this function is the pixel to be filtered and the difference between the texel value of the reference pixel; the output value of the function increases as the input value decreases;

$f_m(m_p,m_q)=f_m\left(\right||m_p-m_q|\left|\right)=\left\{\begin{array}{c}1,\mathrm{if}\left(\right|T_p-T_{p^{\&prime;}}|<\mathrm{the\; th})\&CirclePlus;\left(\right|T_q-T_{q^{\&prime;}}|<\mathrm{the\; th})=0\\ 0,\mathrm{else}\end{array}\right.$ It is used to calculate the motion feature consistency between the pixel to be filtered and the reference pixel; that is, when the difference between the texture pixel value of the pixel to be filtered and the pixel at the corresponding position in the previous frame, and the corresponding pixel value between the reference pixel and the previous frame If the difference between the texture pixel values of the pixel at the position is greater than or less than a preset threshold, it is determined that the motion characteristics of the pixel to be filtered and the reference pixel are consistent; otherwise, it is determined that the pixel to be filtered is consistent with the reference pixel The movement characteristics are inconsistent.

Among them, p is the pixel to be filtered, q is the reference pixel, and K is the set of reference pixels. This set usually takes a square area centered on the pixel to be filtered, with a size of 5*5 or 7*7. D _p ' is p after filtering D _q is the depth pixel value of q, P and Q are the coordinate values of p and q in three-dimensional space, T _p and T _q are the texture pixel values of p and q, T _p' and T _q' is the texel value of p and q at the same position in the previous frame, where the same position refers to the same position in the image plane, T _p ' is the texel value after p filtering, and th is the preset texel difference threshold. th is the threshold value for judging whether the motion characteristics of pixels are consistent. It can be selected according to the content of the 3D video sequence, generally 6-20. When the selection is appropriate, the boundary of the moving object can be better distinguished, making the boundary of the filtered object more accurate. obvious.

It should be noted that step 202, step 203, and step 204 are not sequenced in this embodiment.

In this embodiment, the relationship between the pixel to be filtered and the reference pixel in the real three-dimensional space is used to calculate the spatial proximity, texture pixel value similarity, and motion feature consistency between the pixel to be filtered and the reference pixel in the three-dimensional space property; according to the spatial proximity, texture pixel value similarity and motion feature consistency to determine the weight of the filter, the pixel values of the reference pixels in the reference pixel set are respectively weighted and averaged to obtain the filtering result of the pixel to be filtered, When calculating the weight value, the spatial proximity, texture pixel value similarity, and motion feature consistency are comprehensively considered. Since the position in the real three-dimensional space is used to calculate the spatial proximity, and the motion feature consistency between pixels is considered The accuracy of the filtering result is improved, and the problem of low accuracy of the filtering result in the prior art is solved.

FIG. 4 is a flowchart of Embodiment 2 of the three-dimensional video filtering method of the present invention, and FIG. 5 is a schematic diagram of reference pixel selection in Embodiment 2 of the method of the present invention. As shown in FIG. 4 , the method of this embodiment may include:

Step 401. Project the pixels in the image plane to a three-dimensional space; the pixels include pixels to be filtered and a set of reference pixels.

Optionally, project pixels in the image plane to 3D space, including:

Using the depth image information, viewpoint position information and reference camera parameter information provided by the 3D video, project the pixel from the image plane to the 3D space; the depth image information includes the depth pixel value of the pixel.

Optionally, using the depth image information, viewpoint position information and reference camera parameter information provided by the 3D video, projecting the pixels from the image plane to the 3D space includes:

According to the formula P=R ^-1 (dA ^-1 pt), calculate the coordinate value after the pixel is projected into the three-dimensional space;

Among them, R and t are the rotation matrix and translation vector of the reference camera, A is the parameter matrix of the reference camera, $A=\left(\begin{array}{ccc}{f}_x& r& o_x\\ 0& f_{\mathrm{the\; y}}& o_{\mathrm{the\; y}}\\ 0& 0& 1\end{array}\right),$ $p=\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)$ is the coordinate value of the pixel in the image plane, $p=\left(\begin{array}{c}x\\ \mathrm{the\; y}\\ z\end{array}\right)$ For the coordinate value of the pixel in the three-dimensional space, d is the depth pixel value of the pixel; f _x and f _y are respectively the normalized focal lengths in the horizontal and vertical directions, and r is the radial distortion coefficient, ( o _x , o _y ) are coordinate values of a reference point on the image plane; the reference point is the intersection of the optical axis of the reference camera and the image plane.

Step 402: Calculate the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; Wherein, the reference pixels are set in the same frame of image as the pixel to be filtered and multiple adjacent frames of images.

Optionally, the spatial proximity is calculated by taking the distance between the pixel to be filtered and the reference pixel in three-dimensional space as an input value of a function; the output value of the function increases as the input value decreases.

Step 403: Calculate the similarity of texel values between the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set.

Optionally, the texel value similarity is calculated by using the difference between the texel values of the pixel to be filtered and the reference pixel as an input value of a function; the output value of the function decreases as the input value and increased;

The implementation principles in step 401, step 402, and step 403 are similar to those in the first embodiment, and will not be repeated here.

Step 404 , according to the time interval of the frame where the pixel to be filtered and the reference pixel in the set of reference pixels are located, calculate the temporal proximity between the pixel to be filtered and the reference pixel.

Optionally, the temporal proximity is calculated by using the time interval between the frame where the pixel to be filtered and the reference pixel is located as an input value of a function; the output value of the function increases as the input value decreases .

Specifically, since the pixels between different frames jointly reflect the state of the object within a period of time, there is a certain correlation between them, and this correlation can also be used in the image filtering method in the 3D video. Therefore, on the basis of the weight calculation method in the first embodiment above, the present invention expands the selection range of reference pixels from the frame where the current pixel to be filtered is located to its adjacent frame (filtered reference frame), so as to increase the number of frames after filtering. continuity between. As shown in Figure 5, in each filtered reference frame, the selection range of reference pixels is consistent with the selection range of reference pixels in the frame to be filtered, wherein the Nth frame is the frame where the current pixel to be filtered is located, and the first m frames and the next n frames are selected. The frame is used as a filtering reference frame, and the reference pixel window of each frame has the same coordinates and size as the reference pixel window of the Nth frame (here, the coordinates and size on the image plane are the same), K _i is the reference pixel set in the i-th frame, and the value of i It is an integer in the interval [Nm, N+n], m and n are non-negative integers, and when n=0, it means that it is only acquired from the encoded or decoded frame before the frame to be filtered.

The distance between two points in the time domain reflects their temporal proximity. The closer the time domain distance is, the stronger the correlation is, and the greater the temporal proximity is, that is, the distance between the pixel to be filtered and the reference pixel can be Calculate the time interval of the frame, and use this time interval as an input value to calculate the time domain proximity through the Gaussian function. The function of calculating the time domain proximity can also be other functions, but it is necessary to ensure that the output value of the function follows the input value decrease and increase.

Step 405: Determine the weight of filtering according to the spatial proximity, texture pixel value similarity and temporal proximity, respectively perform weighted average on the pixel values of the reference pixels in the reference pixel set to obtain the pixel value of the pixel to be filtered filter results.

Optionally, when filtering the depth image, the filtering weight is determined according to the spatial proximity, the similarity of the texel value corresponding to the depth pixel, and the temporal proximity, and respectively for the reference pixel set performing a weighted average on the depth pixel values of the reference pixels to obtain a filtering result of the depth pixel values of the pixels to be filtered in the depth image; or,

When filtering the texture image, determine the weight of filtering according to the spatial proximity, the texture pixel similarity and the temporal proximity, and perform weighted average on the texture pixel values of the reference pixels in the reference pixel set respectively , obtaining a filtering result of the texel value of the pixel to be filtered in the texture image.

Optionally, the filtering weight is determined according to the spatial proximity, texture pixel value similarity and temporal proximity, and the pixel values of the reference pixels in the reference pixel set are respectively weighted and averaged to obtain the pixel to be filtered The filtering results, including:

According to formula (3): ${{\mathrm{D.}}_p}^{\&prime;}=\frac{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N){\mathrm{D.}}_{q_i}\right)}{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)\right)}$ Calculate and obtain the filtering result of the depth pixel value of the pixel to be filtered; or,

According to formula (4): ${T_p}^{\&prime;}=\frac{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)T_{q_i}\right)}{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)\right)}$ Calculate and obtain the filtering result of the texel value of the pixel to be filtered;

in, For calculating the spatial proximity of the pixel to be filtered and the reference pixel;;

being used to calculate the similarity of texel values between the pixel to be filtered and the reference pixel;

f _tem (i, N)=f _tem (||iN||) is used to calculate the temporal proximity of the pixel to be filtered and the reference pixel;

Among them, N is the frame number of the frame where the pixel to be filtered is located, i is the frame number of the frame where the reference pixel is located, and the value of i is an integer in the interval [Nm, N+n], m and n are respectively before the frame where the pixel to be filtered is located , the number of subsequent reference frames, m and n are non-negative integers, p is the pixel to be filtered, q _i is the reference pixel in the i-th frame, K _i is the reference pixel set in the i-th frame, and D _p ' is p Filtered depth pixel values, The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the i-th frame in three-dimensional space, T _p , are the texture pixel values of p and q in the i-th frame, respectively, and T _p ' is the texture pixel value of p after filtering.

Specifically, according to the formula ${{\mathrm{D.}}_p}^{\&prime;}=\frac{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N){\mathrm{D.}}_{q_i}\right)}{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)\right)}$ Carry out weighted average on the reference pixels in the reference pixel set, calculate and obtain the filtering result of the depth pixel value of the pixel to be filtered; according to the formula ${T_p}^{\&prime;}=\frac{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)T_{q_i}\right)}{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)\right)}$ Carrying out weighted averaging on the reference pixels in the reference pixel set, and calculating and obtaining the filtering result of the texel value of the pixel to be filtered;

in, Used to calculate the spatial proximity of the pixel to be filtered and the reference pixel; the input value of the function is the spatial distance between the pixel to be filtered and the reference pixel; the output value of the function increases as the input value decreases ;

f _T (T _p , T _q )=f _T (||T _p -T _q ||) is used to calculate the texture pixel value similarity between the pixel to be filtered and the reference pixel; the input value of this function is the pixel to be filtered and the difference between the texel value of the reference pixel; the output value of the function increases as the input value decreases;

f _tem (i, N)=f _tem (||iN||) is used to calculate the temporal proximity of the pixel to be filtered and the reference pixel; the input value of this function is the pixel to be filtered and the pixel to be filtered the time interval framed by the reference pixel; the output value of the function increases as the input value decreases;

Among them, N is the frame number of the frame where the pixel to be filtered is located, i is the frame number of the frame where the reference pixel is located, m and n are the number of reference frames before and after the frame where the pixel to be filtered is located, and usually m and n can be 1 ~3, because as the time interval increases, the correlation between frames will become very small and can be ignored, p is the pixel to be filtered, q _i is the reference pixel in the i-th frame, and K _i is the i-th frame A set of reference pixels in the frame, which usually takes a square area centered on the pixel to be filtered, with a size of 5*5 or 7*7, D _p ' is the depth pixel value after p filtering, The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the i-th frame in three-dimensional space, T _p , are the texture pixel values of p and q in the i-th frame, respectively, and T _p ' is the texture pixel value of p after filtering.

It should be noted that, in this embodiment, step 402, step 403, and step 404 are not sequenced.

In this embodiment, the relationship between the pixel to be filtered and the reference pixel in the real three-dimensional space is used to calculate the spatial proximity, texture pixel value similarity, and time domain Proximity: determine the weight of filtering according to spatial proximity, texture pixel value similarity and time domain proximity, respectively perform weighted average on the pixel values of the reference pixels in the reference pixel set to obtain the filtering result of the pixel to be filtered , the spatial proximity, texture pixel value similarity and temporal proximity are considered comprehensively when calculating the weight, because the position in the real 3D space is used to calculate the spatial proximity, and because the 3D video consists of a series of images collected from different times There is also a correlation between pixels between different frames. Therefore, the continuity between frames is stronger after the weight is considered in the time-domain proximity filter, which improves the accuracy of the filtering results and solves the problem in the prior art. The accuracy of filtering results is not high.

FIG. 6 is a schematic structural diagram of an embodiment of a three-dimensional video filtering device of the present invention. As shown in FIG. 6, the three-dimensional video filtering device 60 of this embodiment may include: a projection module 601, a calculation module 602, and a filtering module 603;

Wherein, the projection module 601 is configured to project pixels in the image plane to a three-dimensional space; the pixels include a pixel to be filtered and a set of reference pixels;

Calculation module 602, configured to calculate the space between the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space Proximity; wherein, the set of reference pixels and the pixel to be filtered are in the same frame image;

The calculation module 602 is further configured to calculate the similarity of the texel value of the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set;

The calculation module 602 is further configured to calculate the pixel to be filtered, the reference pixel in the reference pixel set, and the texture pixel value of the pixel at the same position in the previous frame image of the frame where the pixel to be filtered is located. The motion feature consistency of the pixel to be filtered and the reference pixel;

The filtering module 603 is configured to determine the weight of filtering according to the spatial proximity, texture pixel value similarity and motion feature consistency, and perform weighted average on the pixel values of the reference pixels in the reference pixel set respectively to obtain the to-be The filtered result of the filtered pixel.

Optionally, the filtering module 603 is specifically configured to:

When filtering the depth image, the weight of the filter is determined according to the spatial proximity, the similarity of the texture pixel value corresponding to the depth pixel and the consistency of the motion feature, and the depth of the reference pixel in the reference pixel set is respectively Performing weighted average of the pixel values to obtain a filtering result of the depth pixel values of the pixels to be filtered in the depth image; or,

When filtering a texture image, determine the weight of filtering according to the spatial proximity, the similarity of the texel and the consistency of the motion feature, and perform weighted average of the texture pixel values of the reference pixels in the reference pixel set respectively , obtaining a filtering result of the texel value of the pixel to be filtered in the texture image.

Optionally, the projection module 601 is specifically used for:

Using the depth image information, viewpoint position information and reference camera parameter information provided by the 3D video, project the pixel from the image plane to the 3D space; the depth image information includes the depth pixel value of the pixel.

Optionally, the projection module 601 is specifically used for:

According to the formula P=R ^-1 (dA ^-1 pt), calculate the coordinate value after the pixel is projected into the three-dimensional space;

Among them, R and t are the rotation matrix and translation vector of the reference camera, A is the parameter matrix of the reference camera, $A=\left(\begin{array}{ccc}{f}_x& r& o_x\\ 0& f_{\mathrm{the\; y}}& o_{\mathrm{the\; y}}\\ 0& 0& 1\end{array}\right),$ $p=\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)$ is the coordinate value of the pixel in the image plane, $p=\left(\begin{array}{c}x\\ \mathrm{the\; y}\\ z\end{array}\right)$ For the coordinate value of the pixel in the three-dimensional space, d is the depth pixel value of the pixel; f _x and f _y are respectively the normalized focal lengths in the horizontal and vertical directions, and r is the radial distortion coefficient, ( o _x , o _y ) are coordinate values of a reference point on the image plane; the reference point is the intersection of the optical axis of the reference camera and the image plane.

Optionally, the spatial proximity is calculated by using the distance between the pixel to be filtered and the reference pixel in three-dimensional space as an input value of a function; the output value of the function increases as the input value decreases;

The texel value similarity is calculated by using the difference between the texture pixel values of the pixel to be filtered and the reference pixel as an input value of a function; the output value of the function increases as the input value decreases;

The motion feature consistency is obtained by calculating whether the motion features of the pixel to be filtered and the reference pixel are consistent, including:

When the difference between the texture pixel value of the pixel to be filtered and the pixel at the corresponding position in the previous frame, and the difference between the texture pixel value between the reference pixel and the pixel at the corresponding position in the previous frame are greater than or less than the predetermined When the threshold is set, it is determined that the motion state of the pixel to be filtered is consistent with the reference pixel; otherwise, it is determined that the motion state of the pixel to be filtered is inconsistent with the reference pixel.

Optionally, the filtering module 603 is specifically configured to:

According to formula (1): ${{\mathrm{D.}}_p}^{\&prime;}=\frac{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q){\mathrm{D.}}_q}{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)}$ Calculate and obtain the filtering result of the depth pixel value of the pixel to be filtered; or,

According to formula (2): ${T_p}^{\&prime;}=\frac{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)T_q}{\underset{q\&Element;K}{\mathrm{\&Sigma;}}{f}_S(P,Q)f_T(T_p,T_q)f_m(m_p,m_q)}$ Calculate and obtain the filtering result of the texel value of the pixel to be filtered;

in, for calculating the spatial proximity between the pixel to be filtered and the reference pixel;

f _T (T _p , T _q )=f _T (||T _p −T _q ||) is used to calculate the texture pixel value similarity between the pixel to be filtered and the reference pixel;

$f_m(m_p,m_q)=f_m\left(\right||m_p-m_q|\left|\right)=\left\{\begin{array}{c}1,\mathrm{if}\left(\right|T_p-T_{p^{\&prime;}}|<\mathrm{the\; th})\&CirclePlus;\left(\right|T_q-T_{q^{\&prime;}}|<\mathrm{the\; th})=0\\ 0,\mathrm{else}\end{array}\right.$ used to calculate the motion feature consistency between the pixel to be filtered and the reference pixel;

Among them, p is the pixel to be filtered, q is the reference pixel, K is the reference pixel set, D _p ' is the depth pixel value after p filtering, D _q is the depth pixel value of q, and P and Q are p and q in the three-dimensional space The coordinate values in , T _p , T _q are the texture pixel values of p and q, T _p' , T _q' are the texture pixel values of p and q at the same position in the previous frame, T _p ' is the texture after p filtering Pixel value, th is the preset texture pixel difference threshold.

The device of this embodiment can be used to implement the technical solution of the method embodiment shown in FIG. 2 , and its implementation principle and technical effect are similar, and will not be repeated here.

In the second embodiment of the three-dimensional video filtering device of the present invention, the device in this embodiment is based on the structure of the device shown in FIG. Pixels in the plane are projected into a three-dimensional space; the pixels include pixels to be filtered and a set of reference pixels;

Calculation module 602, configured to calculate the space between the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space Proximity; wherein, the set of reference pixels is located in the same frame of image and adjacent multi-frame images as the pixel to be filtered;

The calculation module 602 is further configured to calculate the similarity of the texel value of the pixel to be filtered and the reference pixel according to the texel value of the pixel to be filtered and the reference pixel in the reference pixel set;

The calculation module 602 is further configured to calculate the temporal proximity of the pixel to be filtered and the reference pixel according to the time interval of the frame where the pixel to be filtered and the reference pixel in the set of reference pixels are located;

The filtering module 603 is configured to determine the weight of filtering according to the spatial proximity, texture pixel value similarity and temporal proximity, and perform weighted average on the pixel values of the reference pixels in the reference pixel set respectively to obtain the to-be The filtered result of the filtered pixel.

Optionally, the filtering module 603 is specifically configured to:

When filtering the depth image, the weight of filtering is determined according to the spatial proximity, the similarity of the texture pixel value corresponding to the depth pixel and the temporal proximity, and the depth of the reference pixel in the reference pixel set is respectively Performing weighted average of the pixel values to obtain a filtering result of the depth pixel values of the pixels to be filtered in the depth image; or,

When filtering the texture image, determine the weight of filtering according to the spatial proximity, the texture pixel similarity and the temporal proximity, and perform weighted average on the texture pixel values of the reference pixels in the reference pixel set respectively , obtaining a filtering result of the texel value of the pixel to be filtered in the texture image.

Optionally, the projection module 601 is specifically used for:

Using the depth image information, viewpoint position information and reference camera parameter information provided by the 3D video, project the pixel from the image plane to the 3D space; the depth image information includes the depth pixel value of the pixel.

Optionally, the projection module 601 is specifically used for:

According to the formula P=R ^-1 (dA ^-1 pt), calculate the coordinate value after the pixel is projected into the three-dimensional space;

Among them, R and t are the rotation matrix and translation vector of the reference camera, A is the parameter matrix of the reference camera, $A=\left(\begin{array}{ccc}{f}_x& r& o_x\\ 0& f_{\mathrm{the\; y}}& o_{\mathrm{the\; y}}\\ 0& 0& 1\end{array}\right),$ $p=\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)$ is the coordinate value of the pixel in the image plane, $p=\left(\begin{array}{c}x\\ \mathrm{the\; y}\\ z\end{array}\right)$ For the coordinate value of the pixel in the three-dimensional space, d is the depth pixel value of the pixel; f _x and f _y are respectively the normalized focal lengths in the horizontal and vertical directions, and r is the radial distortion coefficient, ( o _x , o _y ) are coordinate values of a reference point on the image plane; the reference point is the intersection of the optical axis of the reference camera and the image plane.

Optionally, the spatial proximity is calculated by using the distance between the pixel to be filtered and the reference pixel in three-dimensional space as an input value of a function; the output value of the function increases as the input value decreases;

The texel value similarity is calculated by using the difference between the texture pixel values of the pixel to be filtered and the reference pixel as an input value of a function; the output value of the function increases as the input value decreases;

The temporal proximity is calculated by using the time interval of the frame where the pixel to be filtered and the reference pixel are located as an input value of a function; the output value of the function increases as the input value decreases.

Optionally, the filtering module 603 is specifically configured to:

According to formula (3): ${{\mathrm{D.}}_p}^{\&prime;}=\frac{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N){\mathrm{D.}}_{q_i}\right)}{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)\right)}$ Calculate and obtain the filtering result of the depth pixel value of the pixel to be filtered; or,

According to formula (4): ${T_p}^{\&prime;}=\frac{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)T_{q_i}\right)}{\underset{i=N-m}{\overset{N+\mathrm{no}}{\mathrm{\&Sigma;}}}\left(\underset{q_i\&Element;K_i}{\mathrm{\&Sigma;}}{f}_S(P,Q_i)f_T(T_p,T_{q_i})f_{\mathrm{tem}}(i,N)\right)}$ Calculate and obtain the filtering result of the texel value of the pixel to be filtered;

in, For calculating the spatial proximity of the pixel to be filtered and the reference pixel;;

being used to calculate the similarity of texel values between the pixel to be filtered and the reference pixel;

f _tem (i, N)=f _tem (||iN||) is used to calculate the temporal proximity of the pixel to be filtered and the reference pixel;

Among them, N is the frame number of the frame where the pixel to be filtered is located, i is the frame number of the frame where the reference pixel is located, and the value of i is an integer in the interval [Nm, N+n], m and n are respectively before the frame where the pixel to be filtered is located , the number of subsequent reference frames, m and n are non-negative integers, p is the pixel to be filtered, q _i is the reference pixel in the i-th frame, K _i is the reference pixel set in the i-th frame, and D _p ' is p Filtered depth pixel values, The depth pixel value of q in the i-th frame, P, Q _i is p, the coordinate value of q in the i-th frame in three-dimensional space, T _p , are the texture pixel values of p and q in the i-th frame, respectively, and T _p ' is the texture pixel value of p after filtering.

The device of this embodiment can be used to implement the technical solution of the method embodiment shown in FIG. 4 , and its implementation principle and technical effect are similar, and details are not repeated here.

FIG. 7 is a schematic structural diagram of an embodiment of a three-dimensional video filtering device according to the present invention. As shown in FIG. 7 , a three-dimensional video filtering device 70 provided in this embodiment includes a processor 701 and a memory 702 . The memory 702 is used to store execution instructions. When the three-dimensional video filtering device 70 is running, the processor 701 communicates with the memory 702, and the processor 701 calls the execution instructions in the memory 702 to execute any method described in the embodiment. The implementation principle and technical effect of the technical solution are similar and will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units or modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or modules can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or modules may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or may also be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by program instructions and related hardware. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other various media that can store program codes.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (20)

1. A method for filtering three-dimensional video, comprising:

projecting pixels in an image plane into three-dimensional space; the pixels comprise pixels to be filtered and reference pixel sets;

calculating the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; wherein the reference pixel set and the pixel to be filtered are in the same frame of image;

calculating the similarity of the texture pixel values of the pixel to be filtered and the reference pixel according to the texture pixel values of the pixel to be filtered and the reference pixel in the reference pixel set;

calculating the consistency of the motion characteristics of the pixel to be filtered and the reference pixel according to the texture pixel values of the pixels at the same position in the previous frame image of the frame where the pixel to be filtered, the reference pixel in the reference pixel set and the pixel to be filtered are located;

when the depth image is filtered, determining a filtering weight value according to the spatial proximity, the similarity of the texture pixel values corresponding to the depth pixels and the motion characteristic consistency, and respectively carrying out weighted average on the depth pixel values of the reference pixels in the reference pixel set to obtain a filtering result of the depth pixel values of the pixels to be filtered of the depth image.

2. The method of claim 1, wherein projecting pixels in an image plane into a three-dimensional space comprises:

projecting the pixels from an image plane to a three-dimensional space by using depth image information, viewpoint position information and reference camera parameter information provided by a three-dimensional video; the depth image information includes a depth pixel value of the pixel.

3. The method of claim 2, wherein projecting the pixel from the image plane into the three-dimensional space using the depth image information, the viewpoint position information, and the reference camera parameter information provided by the three-dimensional video comprises:

according to the formula P ═ R^-1(dA^-1p-t) calculating a coordinate value of the pixel projected to the three-dimensional space;

wherein R and t are a rotation matrix and a translation vector of the reference camera, A is a reference camera parameter matrix,is the pixel is atThe coordinate values in the image plane are,the coordinate value of the pixel in the three-dimensional space is shown, and d is the depth pixel value of the pixel; f. of_xAnd f_yNormalized focal lengths in the horizontal and vertical directions, respectively, r is the radial distortion coefficient, (o)_x,o_y) Coordinate values of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

4. A method according to any one of claims 1 to 3, characterized in that said spatial proximity is calculated from the distance of said pixel to be filtered and said reference pixel in three-dimensional space as an input value of a function; the output value of the function increases with decreasing input value;

the texture pixel value similarity is obtained by calculating the difference value of the texture pixel values of the pixel to be filtered and the reference pixel as the input value of a function; the output value of the function increases with decreasing input value;

the motion characteristic consistency is obtained by calculating whether the motion characteristics of the pixel to be filtered and the reference pixel are consistent, and the motion characteristic consistency comprises the following steps:

when the difference value of the texture pixel values of the pixel to be filtered and the pixel at the corresponding position in the previous frame and the difference value of the texture pixel values of the pixel of the reference pixel and the pixel at the corresponding position in the previous frame are simultaneously greater than or less than a preset threshold value, determining that the motion states of the pixel to be filtered and the reference pixel are consistent; otherwise, determining that the motion states of the pixel to be filtered and the reference pixel are inconsistent.

5. The method of claim 1, wherein the determining a filtered weight value according to the spatial proximity, the similarity of texture pixel values, and the consistency of motion features, and performing weighted average on pixel values of reference pixels in the reference pixel set to obtain a filtering result of the pixel to be filtered respectively comprises:

according to formula (1):calculating a filtering result of the depth pixel value of the pixel to be filtered;

wherein,for calculating the spatial proximity of the pixel to be filtered and the reference pixel;

f_T(T_p,T_q)＝f_T(||T_p-T_q| |) is used for calculating the similarity of the texture pixel values of the pixel to be filtered and the reference pixel;

the motion characteristic consistency of the pixel to be filtered and the reference pixel is calculated;

wherein p is a pixel to be filtered, q is a reference pixel, K is a reference pixel set, and D_pIs a p-filtered depth pixel value, D_qA depth pixel value of q, P, Q coordinate values of p and q in three-dimensional space, T_p、T_qTexel values p, q, T_p'、T_q'The texel values of p and q at the same position in the previous frame are obtained, and th is a preset texel difference threshold value.

6. A method for filtering three-dimensional video, comprising:

projecting pixels in an image plane into three-dimensional space; the pixels comprise pixels to be filtered and reference pixel sets;

calculating the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; the reference pixel set is in the same frame image and adjacent multi-frame images where the pixel to be filtered is located;

calculating the similarity of the texture pixel values of the pixel to be filtered and the reference pixel according to the texture pixel values of the pixel to be filtered and the reference pixel in the reference pixel set;

calculating the time domain proximity of the pixel to be filtered and the reference pixel according to the time interval of the frame where the pixel to be filtered and the reference pixel in the reference pixel set are located;

when the depth image is filtered, determining a filtering weight value according to the spatial proximity, the similarity of the texture pixel values corresponding to the depth pixels and the time domain proximity, and respectively performing weighted average on the depth pixel values of the reference pixels in the reference pixel set to obtain a filtering result of the depth pixel values of the pixels to be filtered of the depth image.

7. The method of claim 6, wherein projecting pixels in an image plane into a three-dimensional space comprises:

projecting the pixels from an image plane to a three-dimensional space by using depth image information, viewpoint position information and reference camera parameter information provided by a three-dimensional video; the depth image information includes a depth pixel value of the pixel.

8. The method of claim 7, wherein projecting the pixel from the image plane into the three-dimensional space using the depth image information, the viewpoint position information, and the reference camera parameter information provided by the three-dimensional video comprises:

according to the formula P ═ R^-1(dA^-1p-t) calculating a coordinate value of the pixel projected to the three-dimensional space;

wherein R and t are a rotation matrix and a translation vector of the reference camera, A is a reference camera parameter matrix,for the pixel in the figureThe coordinate values in the image plane are,the coordinate value of the pixel in the three-dimensional space is shown, and d is the depth pixel value of the pixel; f. of_xAnd f_yNormalized focal lengths in the horizontal and vertical directions, respectively, r is the radial distortion coefficient, (o)_x,o_y) Coordinate values of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

9. The method according to any one of claims 6 to 8, wherein the spatial proximity is calculated from an input value of a function of a distance between the pixel to be filtered and the reference pixel in three-dimensional space; the output value of the function increases with decreasing input value;

the texture pixel value similarity is obtained by calculating the difference value of the texture pixel values of the pixel to be filtered and the reference pixel as the input value of a function; the output value of the function increases with decreasing input value;

the time domain proximity is obtained by calculating the time interval of the frame where the pixel to be filtered and the reference pixel are located as the input value of a function; the output value of the function increases with decreasing input value.

10. The method of claim 6, wherein the determining a filtered weight value according to the spatial proximity, the texel value similarity and the temporal proximity, and performing weighted averaging on pixel values of reference pixels in the reference pixel set to obtain a filtering result of the pixel to be filtered respectively comprises:

according to formula (3):calculating a filtering result of the depth pixel value of the pixel to be filtered;

wherein,for calculating the spatial proximity of the pixel to be filtered and the reference pixel;

the texture pixel value similarity of the pixel to be filtered and the reference pixel is calculated;

f_tem(i,N)＝f_tem(| i-N |) is used for calculating the temporal proximity of the pixel to be filtered and the reference pixel;

wherein, N is the frame number of the frame where the pixel to be filtered is located, i is the frame number of the frame where the reference pixel is located, and the value of i is [ N-m, N + N ]]The integer of the interval, m and n are respectively the number of reference frames before and after the frame of the pixel to be filtered, m and n are non-negative integers, p is the pixel to be filtered, q is the integer of the interval_iIs a reference pixel in the ith frame, K_iFor a set of reference pixels in the ith frame, D_pIs' a p-filtered depth pixel value,depth pixel value of q in the ith frame, P, Q_iIs the coordinate value of p, q in the ith frame in three-dimensional space, T_p、The texel values of p and q in the ith frame are respectively.

11. A three-dimensional video filtering apparatus, comprising:

a projection module for projecting pixels in an image plane into a three-dimensional space; the pixels comprise pixels to be filtered and reference pixel sets;

the calculation module is used for calculating the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; wherein the reference pixel set and the pixel to be filtered are in the same frame of image;

the calculating module is further configured to calculate, according to the to-be-filtered pixel and the texture pixel value of the reference pixel in the reference pixel set, the similarity between the texture pixel values of the to-be-filtered pixel and the reference pixel;

the computing module is further configured to compute motion feature consistency of the pixel to be filtered and the reference pixel according to texture pixel values of pixels at the same position in a previous frame image of a frame where the pixel to be filtered, the reference pixel in the reference pixel set, and the pixel to be filtered are located;

and the filtering module is used for determining a filtering weight value according to the spatial proximity, the similarity of texture pixel values corresponding to the depth pixels and the motion characteristic consistency when filtering the depth image, and respectively carrying out weighted average on the depth pixel values of the reference pixels in the reference pixel set to obtain a filtering result of the depth pixel values of the pixels to be filtered of the depth image.

12. The apparatus of claim 11, wherein the projection module is specifically configured to:

projecting the pixels from an image plane to a three-dimensional space by using depth image information, viewpoint position information and reference camera parameter information provided by a three-dimensional video; the depth image information includes a depth pixel value of the pixel.

13. The apparatus of claim 12, wherein the projection module is specifically configured to:

according to the formula P ═ R^-1(dA^-1p-t) calculating a coordinate value of the pixel projected to the three-dimensional space;

wherein R and t are a rotation matrix and a translation vector of the reference camera, A is a reference camera parameter matrix,is a coordinate value of the pixel in the image plane,the coordinate value of the pixel in the three-dimensional space is shown, and d is the depth pixel value of the pixel; f. of_xAnd f_yNormalized focal lengths in the horizontal and vertical directions, respectively, r is the radial distortion coefficient, (o)_x,o_y) Coordinate values of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

14. The apparatus according to any one of claims 11-13, wherein the spatial proximity is calculated from an input value of a distance between the pixel to be filtered and the reference pixel in a three-dimensional space as a function; the output value of the function increases with decreasing input value;

the texture pixel value similarity is obtained by calculating the difference value of the texture pixel values of the pixel to be filtered and the reference pixel as the input value of a function; the output value of the function increases with decreasing input value;

the motion characteristic consistency is obtained by calculating whether the motion characteristics of the pixel to be filtered and the reference pixel are consistent, and the motion characteristic consistency comprises the following steps:

when the difference value of the texture pixel values of the pixel to be filtered and the pixel at the corresponding position in the previous frame and the difference value of the texture pixel values of the pixel of the reference pixel and the pixel at the corresponding position in the previous frame are simultaneously greater than or less than a preset threshold value, determining that the motion states of the pixel to be filtered and the reference pixel are consistent; otherwise, determining that the motion states of the pixel to be filtered and the reference pixel are inconsistent.

15. The apparatus of claim 11, wherein the filtering module is specifically configured to:

according to formula (1):calculating a filtering result of the depth pixel value of the pixel to be filtered;

wherein,for calculating the spatial proximity of the pixel to be filtered and the reference pixel;

f_T(T_p,T_q)＝f_T(||T_p-T_q| |) is used for calculating the similarity of the texture pixel values of the pixel to be filtered and the reference pixel;

the motion characteristic consistency of the pixel to be filtered and the reference pixel is calculated;

wherein p is a pixel to be filtered, q is a reference pixel, K is a reference pixel set, and D_pIs a p-filtered depth pixel value, D_qA depth pixel value of q, P, Q coordinate values of p and q in three-dimensional space, T_p、T_qTexel values p, q, T_p'、T_q'The texel values of p and q at the same position in the previous frame are obtained, and th is a preset texel difference threshold value.

16. A three-dimensional video filtering apparatus, comprising:

a projection module for projecting pixels in an image plane into a three-dimensional space; the pixels comprise pixels to be filtered and reference pixel sets;

the calculation module is used for calculating the spatial proximity of the pixel to be filtered and the reference pixel in the three-dimensional space according to the coordinate values of the pixel to be filtered and the reference pixel in the reference pixel set in the three-dimensional space; the reference pixel set is in the same frame image and adjacent multi-frame images where the pixel to be filtered is located;

the calculating module is further configured to calculate, according to the to-be-filtered pixel and the texture pixel value of the reference pixel in the reference pixel set, the similarity between the texture pixel values of the to-be-filtered pixel and the reference pixel;

the calculating module is further configured to calculate time domain proximity of the pixel to be filtered and the reference pixel according to a time interval of a frame where the pixel to be filtered and the reference pixel in the reference pixel set are located;

and the filtering module is used for determining a filtering weight value according to the spatial proximity, the similarity of texture pixel values corresponding to the depth pixels and the time domain proximity when filtering the depth image, and respectively carrying out weighted average on the depth pixel values of the reference pixels in the reference pixel set to obtain a filtering result of the depth pixel values of the pixels to be filtered of the depth image.

17. The apparatus of claim 16, wherein the projection module is specifically configured to:

projecting the pixels from an image plane to a three-dimensional space by using depth image information, viewpoint position information and reference camera parameter information provided by a three-dimensional video; the depth image information includes a depth pixel value of the pixel.

18. The apparatus of claim 17, wherein the projection module is specifically configured to:

according to the formula P ═ R^-1(dA^-1p-t) calculating a coordinate value of the pixel projected to the three-dimensional space;

wherein R and t are a rotation matrix and a translation vector of the reference camera, A is a reference camera parameter matrix,is a coordinate value of the pixel in the image plane,the coordinate value of the pixel in the three-dimensional space is shown, and d is the depth pixel value of the pixel; f. of_xAnd f_yNormalized focal lengths in the horizontal and vertical directions, respectively, r is the radial distortion coefficient, (o)_x,o_y) Coordinate values of a reference point on the image plane; the reference point is an intersection of an optical axis of the reference camera and the image plane.

19. The apparatus according to any one of claims 16-18, wherein the spatial proximity is calculated from an input value of a distance between the pixel to be filtered and the reference pixel in a three-dimensional space as a function; the output value of the function increases with decreasing input value;

the texture pixel value similarity is obtained by calculating the difference value of the texture pixel values of the pixel to be filtered and the reference pixel as the input value of a function; the output value of the function increases with decreasing input value;

the time domain proximity is obtained by calculating the time interval of the frame where the pixel to be filtered and the reference pixel are located as the input value of a function; the output value of the function increases with decreasing input value.

20. The apparatus of claim 16, wherein the filtering module is specifically configured to:

according to formula (3):calculating a filtering result of the depth pixel value of the pixel to be filtered;

wherein,for calculating the spatial proximity of the pixel to be filtered and the reference pixel;

the texture pixel value similarity of the pixel to be filtered and the reference pixel is calculated;

f_tem(i,N)＝f_tem(| i-N |) is used for calculating the temporal proximity of the pixel to be filtered and the reference pixel;

wherein, N is the frame number of the frame where the pixel to be filtered is located, i is the frame number of the frame where the reference pixel is located, and the value of i is [ N-m, N + N ]]The integer of the interval, m and n are respectively the number of reference frames before and after the frame of the pixel to be filtered, m and n are non-negative integers, p is the pixel to be filtered, q is the integer of the interval_iIs a reference pixel in the ith frame, K_iFor a set of reference pixels in the ith frame, D_pIs' a p-filtered depth pixel value,depth pixel value of q in the ith frame, P, Q_iIs the coordinate value of p, q in the ith frame in three-dimensional space, T_p、The texel values of p and q in the ith frame are respectively.