RU2831969C1 - Coding scheme for video data using down/up sampling and non-linear filtering of depth map - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to video encoding. Result is achieved by the fact that source video data containing one or more source types are encoded into a binary video stream. Before encoding, nonlinear filtering and downsampling are applied to depth data of at least one of the initial types. After decoding, the decoded depth data is upsampled and non-linearly filtered.

EFFECT: high quality and efficiency of immersive video encoding.

13 cl, 9 dwg, 1 tbl

Description

AREA OF TECHNOLOGY

The present invention relates to video coding. In particular, it relates to methods and devices for coding and decoding immersive video.

LEVEL OF TECHNOLOGY

Immersive video, also known as six degrees of freedom (6DoF) video, is a video of a three-dimensional (3D) scene that allows for the reconstruction of views of the scene for viewpoints that differ in position and orientation. It is a further development of three degrees of freedom (3DoF) video, which allows for the reconstruction of views for viewpoints with arbitrary orientations, but only at one fixed point in space. In 3DoF, the degrees of freedom are angular, namely pitch, roll, and yaw. 3DoF video supports head rotation, in other words, the user viewing the video content can look in any direction in the scene, but cannot move to another location in the scene. 6DoF video supports head rotation and additionally supports the selection of the position in the scene from which the scene is viewed.

Generating 6DoF video to record a scene requires multiple cameras. Each camera produces image data (often called texture data in this context) and corresponding depth data. For each pixel, the depth data represents the depth at which the corresponding image pixel data is observed by a given camera. Each of the multiple cameras provides a corresponding view of the scene. Providing all the texture data and depth data for all views may be impractical or inefficient in many applications.

To reduce redundancy between views, it has been proposed to crop views and pack them into a “texture atlas” for each frame of the video stream. This approach is intended to reduce or eliminate overlapping parts between multiple views and thereby improve efficiency. The non-overlapping parts of different views that remain after cropping can be called “overlays”. An example of this approach is described in Alvaro Collet et al., “High-quality streamable free-viewpoint video,” ACM Trans. Graphics (SIGGRAPH), 34(4), 2015.

DISCLOSURE OF THE ESSENCE OF THE INVENTION

It would be desirable to improve the quality and efficiency of immersive video encoding. The approach of using cropping (i.e., eliminating redundant texture overlays) to create texture atlases as described above can help reduce the pixel rate. However, cropping views often requires detailed analysis that is not error-free and can result in lower quality for the end user. Therefore, there is a need for reliable and simple ways to reduce the pixel rate.

The invention is defined by the claims.

As shown in the examples, in accordance with an aspect of the present invention, a method is proposed for encoding video data comprising one or more source views, wherein each source view comprises a texture map and a depth map, wherein the method includes:

receiving video data;

processing a depth map of at least one source view to generate a processed depth map, including:

nonlinear filtering and

downsampling; and

encoding the processed depth map and the texture map of at least one original view to form a binary video stream.

Preferably, at least some of the nonlinear filtering is performed before downsampling.

The inventors have found that nonlinear filtering of the depth map before downsampling can help to eliminate, reduce or mitigate the errors introduced by downsampling. In particular, nonlinear filtering can help to prevent small or thin foreground objects from partially or completely disappearing from the depth map due to downsampling. It has been found that nonlinear filtering can be preferable to linear filtering in this regard, since linear filtering can introduce intermediate depth values at the boundaries between foreground and background objects. This makes it difficult for the decoder to distinguish between object boundaries and large depth gradients.

Video data may contain immersive 6DoF video.

Nonlinear filtering may involve increasing the region of at least one foreground object in the depth map.

Enlarging the foreground object before upsampling can help ensure that the foreground object survives the downsampling process better, in other words, that it is better preserved in the processed depth map.

The foreground object can be identified as a local group of pixels at a relatively shallow depth. The background can be identified as pixels at a relatively deep depth. The peripheral pixels of foreground objects can be locally distinguished from the background, for example by applying a threshold to the depth values in the depth map.

Nonlinear filtering may include morphological filtering, in particular grayscale morphological filtering, such as a maximum filter, a minimum filter, or another filter based on ordinal statistics. When the depth map contains depth levels with a special meaning, such as a zero depth level indicating an invalid depth, such depth levels are preferably considered as foreground, regardless of their actual meaning. Accordingly, these levels are preferably preserved after subsampling. Consequently, their area can also be increased.

Nonlinear filtering may involve the application of a filter designed using a machine learning algorithm.

A machine learning algorithm can be trained to reduce or minimize the reconstruction error of the reconstructed depth map after the processed depth map has been encoded and decoded.

A trained filter can similarly help preserve foreground objects in the processed (downsampled) depth map.

The method may also include developing a filter using a machine learning algorithm, wherein the filter is developed to reduce the reconstruction error of the reconstructed depth map after the processed map has been encoded and decoded, and wherein the nonlinear filtering includes applying the developed filter.

Nonlinear filtering may involve neural network processing, and filter design may involve neural network training.

Nonlinear filtering can be performed by a neural network containing multiple layers, and downsampling can be performed between two of the said layers.

Downsampling can be performed by a max-pooling (or min-pooling) layer of a neural network.

The method may include processing a depth map in accordance with a plurality of sets of processing parameters to generate a corresponding plurality of processed depth maps, wherein the method includes: selecting a set of processing parameters that reduces a reconstruction error of the reconstructed depth map after the corresponding processed depth map has been encoded and decoded; and generating a binary metadata stream identifying the selected set of parameters.

This may allow you to optimize the parameters for a given application or a given video sequence.

The processing parameters may include a definition of nonlinear filtering and/or a definition of the downsampling performed. Alternatively or additionally, the processing parameters may include a definition of the processing operations to be performed in the decoder when reconstructing the depth map.

For each set of processing parameters, the method may include: generating a corresponding processed depth map in accordance with the set of processing parameters; encoding the processed depth map to generate a coded depth map; decoding the coded depth map; reconstructing a depth map from the decoded depth map and comparing the reconstructed depth map with a depth map of at least one original view to determine a reconstruction error.

According to another aspect, a method is proposed for decoding video data containing one or more source views, comprising:

receiving a binary video stream containing an encoded depth map and an encoded texture map for at least one source view;

decoding the encoded depth map to obtain a decoded depth map;

decoding the encoded texture map to obtain a decoded texture map, and

processing the decoded depth map to form a reconstructed depth map, wherein the processing includes:

upsampling and

nonlinear filtering.

The method may also include: before the step of processing the decoded depth map to form the reconstructed depth map, detecting that the decoded map has a lower resolution than the decoded texture map.

In some encoding schemes, the depth map may be downsampled only in certain cases or only for certain views. By comparing the resolution of the decoded depth map with the resolution of the decoded texture map, the decoding method can determine whether downsampling has been applied at the encoder. This can avoid the need for metadata in the metadata bitstream to signal which depth maps have been downsampled and to what extent they have been downsampled (this example assumes that the texture map is encoded at full resolution).

To generate a reconstructed depth map, the decoded depth map can be upsampled to the same resolution as the decoded texture map.

Preferably, the nonlinear filtering in the decoding method is designed to compensate for the effect of the nonlinear filtering that was used in the coding method.

The nonlinear filtering may involve reducing the region of at least one foreground object in the depth map. This may be appropriate when the nonlinear filtering during encoding involved increasing the region of at least one foreground object.

Nonlinear filtering may include morphological filtering, in particular grayscale morphological filtering, such as a maximum filter, a minimum filter, or another filter based on ordinal statistics.

Nonlinear filtering during decoding preferably compensates or reverses the effect of nonlinear filtering during encoding. For example, if nonlinear filtering during encoding includes a "maximum" filter (grayscale dilation), then nonlinear filtering during decoding may include a "minimum" filter (grayscale erosion), and vice versa. When a depth map contains depth levels with a special meaning, such as a zero depth level indicating an invalid depth, such depth levels are preferably considered foreground, regardless of their actual meaning.

Preferably, at least part of the nonlinear filtering is performed after upsampling. Not necessarily, all of the nonlinear filtering is performed after upsampling.

The processing of the decoded depth map can be based at least in part on the decoded texture map. The inventors realized that the texture map contains useful information that helps reconstruct the depth map. In particular, when the boundaries of foreground objects have been changed by nonlinear filtering during encoding, analyzing the texture map can help compensate for or reverse the changes.

The method may include: upsampling a decoded depth map; identifying peripheral pixels of at least one foreground object in the upsampled depth map; determining, based on the decoded texture map, whether the peripheral pixels are more similar to the foreground object or to the background; and applying nonlinear filtering only to the peripheral pixels that are determined to be more similar to the background.

Thus, the texture map is used to help identify pixels that have been converted from background to foreground by nonlinear filtering during encoding. Nonlinear filtering during decoding can help return these identified pixels to the background.

Nonlinear filtering may include smoothing the edges of at least one foreground object.

Smoothing may include: identifying peripheral pixels of at least one foreground object in an upsampled depth map; for each peripheral pixel, analyzing the number and/or location of foreground and background pixels in the vicinity of that peripheral pixel; based on the result of the analysis, identifying outlying peripheral pixels that protrude from the object into the background; and applying nonlinear filtering only to the identified peripheral pixels.

The analysis may include counting the number of background pixels in the vicinity, wherein a peripheral pixel is identified as outlier from the object if the number of background pixels in the vicinity is above a specified threshold.

Alternatively or additionally, the analysis may include identifying a spatial structure of foreground and background pixels in the vicinity, wherein a peripheral pixel is identified as an outlier if the spatial structure in its vicinity matches one or more specified spatial structures.

The method may also include receiving a binary metadata stream associated with the binary video stream, wherein the binary metadata stream identifies a set of parameters, and the method optionally also includes processing the decoded depth map in accordance with the identified set of parameters.

Processing parameters may include defining nonlinear filtering and/or defining the upsampling to be performed.

Nonlinear filtering may involve the application of a filter designed using a machine learning algorithm.

A machine learning algorithm can be trained to reduce or minimize the reconstruction error of the reconstructed depth map after the processed depth map has been encoded and decoded.

The filter can be defined in the binary metadata stream associated with the video binary stream.

A computer program is also proposed, containing a computer code for causing the processing system to implement the method described above, when the said program is launched in the processing system.

The computer program may be stored on a computer-readable storage medium. This may be a non-transitory storage medium.

According to another aspect, a video encoder is provided that is configured to encode video data comprising one or more source views, wherein each source view comprises a texture map and a depth map, wherein the video encoder comprises:

an input capable of receiving video data;

a video processor configured to process a depth map of at least one source view to generate a processed depth map, including:

nonlinear filtering and

downsampling;

an encoder configured to encode a texture map of at least one original view and a processed depth map to form a binary video stream; and

an output capable of outputting a binary video stream.

According to another aspect, a video decoder is provided, configured to decode video data comprising one or more source views, wherein the video decoder comprises:

a binary stream input configured to receive a binary video stream, wherein the binary video stream comprises an encoded depth map and an encoded texture map for at least one source view;

a first decoder configured to decode an encoded depth map from a binary video stream to obtain a decoded depth map;

a second decoder configured to decode an encoded texture map from a binary video stream to obtain a decoded texture map;

a reconstruction processor configured to process the decoded depth map to generate a reconstructed depth map, wherein the processing includes:

upsampling and

nonlinear filtering,

and an output configured to output the reconstructed depth map.

These and other aspects of the invention will be apparent and elucidated with reference to the embodiment(s) described below.

BRIEF DESCRIPTION OF DRAWINGS

For a better understanding of the invention and a clearer demonstration of the method of its implementation in practice, references are made further, only as an example, to the accompanying drawings, in which:

Fig. 1 shows an example of encoding and decoding immersive video using existing video codecs;

Fig. 2 is a block diagram showing a method for encoding video data in accordance with an embodiment;

Fig. 3 shows a block diagram of a video encoder in accordance with an embodiment;

Fig. 4 is a block diagram illustrating a method for encoding video data in accordance with another embodiment;

Fig. 5 is a block diagram showing a method for decoding video data in accordance with an embodiment;

Fig. 6 shows a block diagram of a video decoder in accordance with an embodiment;

Fig. 7 shows a method for selectively applying nonlinear filtering to specific pixels in a decoding method according to an embodiment;

Fig. 8 is a block diagram illustrating a method for decoding video data in accordance with another embodiment; and

Fig. 9 shows the use of neural network processing for encoding and decoding video data in accordance with an embodiment.

IMPLEMENTATION OF THE INVENTION

The invention is described below with reference to the figures of the drawings.

It should be understood that the detailed description and specific examples, while showing individual embodiments of the device, systems and methods, serve only as an illustration and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the device, systems and methods of the present invention will become more apparent from the following description, the appended claims and the accompanying drawings. It should be understood that the figures are only schematic and are not drawn to scale. In addition, it should be understood that the same reference numbers are used throughout the figures to designate the same or similar parts.

Methods for encoding and decoding immersive video are disclosed. In the encoding method, source video data comprising one or more source views is encoded into a binary video stream. Before encoding, nonlinear filtering and downsampling are applied to the depth data of at least one of the source views. Downsampling of the depth map helps to reduce the amount of data to be transmitted and, therefore, helps to reduce the bit rate. However, the inventors have found that simply downsampling can lead to the disappearance of thin or small foreground objects, such as wires, from the downsampled depth map. Embodiments of the present invention are aimed at mitigating this effect and preserving small and thin objects on the depth map.

Embodiments of the present invention may be suitable for implementing a part of a technical standard, such as ISO/IEC 23090-12 MPEG-I Part 12, Immersive Video. Where possible, the terminology used in this document is chosen to be consistent with the terms used in MPEG-I Part 12. However, it should be understood that the scope of the present invention is not limited to MPEG-I Part 12 or any other technical standard.

It may be helpful to provide the following definitions/explanations:

"3D scene" refers to visual content in a global reference coordinate system.

An "atlas" is a collection of overlays of one or more view representations after a process of packing into a pair of images that contains a texture component image and a corresponding depth component image.

The "Satin Component" is the textural or depth component of the satin.

Camera Settings define the projection used to generate a view representation of a 3D scene.

"Cropping" is the process of identifying and extracting overlapping areas in different views, resulting in overlaps.

A "renderer" is a process for creating a viewport or omnidirectional view from a 3D scene view that matches the view's position and orientation.

"Original view" is the original video material before encoding, which corresponds to the view representation format, which can be obtained by capturing a 3D scene with a real camera or by projecting a virtual camera onto a surface using the original camera's parameters.

The "target view" is defined as either a spatial viewport or an omnidirectional view at the desired viewing position and orientation.

The "View View" contains arrays of 2D texture component samples and the corresponding depth component, representing a projection of the 3D scene onto the surface using camera parameters.

A machine learning algorithm is any self-learning algorithm that processes input data to produce or predict output data. In some embodiments of the present invention, the input data comprises one or more views decoded from a bitstream, and the output data comprises a prediction/reconstruction of the target view.

Suitable machine learning algorithms for use in the present invention are obvious to those skilled in the art. Examples of suitable machine learning algorithms include decision tree and artificial neural network algorithms. Other machine learning algorithms, such as logistic regression, support vector machines or naive Bayesian models, are suitable alternatives.

The structure of an artificial neural network (or simply neural network) is inspired by the human brain. Neural networks are composed of layers, with each layer containing many neurons. Each neuron contains a mathematical operation. Specifically, each neuron may contain a different weighted combination of transformations of the same type (e.g., transformations of the same type, sigmoid, etc., but with different weights). During input processing, each neuron's mathematical operation is performed on the input to produce a numerical output, and the output of each layer in the neural network is fed to one or more other layers (e.g., sequentially). The final layer provides the output.

Methods for training a machine learning algorithm are well known. Typically, such methods involve obtaining a training data set containing training input items and corresponding training output items. An initialized machine learning algorithm is applied to each input item to produce predicted output items. The error between the predicted output items and the corresponding training output items is used to modify the machine learning algorithm. This process may be repeated until the error converges and the predicted output items are sufficiently similar (e.g., ±1%) to the training output items. This is commonly known as a supervised learning method.

For example, when a sequence of operations for machine learning is formed from a neural network, the mathematical operation (its weight) can be changed for each neuron until the error converges. Known methods for modifying a neural network include gradient descent, backpropagation algorithms, etc.

A convolutional neural network (CNN or ConvNet) is a class of deep neural networks most commonly used for visual image analysis. CNNs are regularized versions of multilayer perceptrons.

Fig. 1 shows a simplified system for encoding and decoding immersive video. An array of cameras 10 is used to capture multiple views of a scene. Each camera captures regular images (called "texture maps" herein) and a depth map of the view in front of it. The set of views containing texture and depth data is fed to an encoder 300. The encoder encodes both the texture data and the depth data into a regular video bitstream - in this case, a high efficiency video coding (HEVC) bitstream. It is accompanied by a metadata bitstream to inform the decoder 400 what the various parts of the video bitstream mean. For example, the metadata indicates to the decoder which parts of the video bitstream correspond to texture maps and which correspond to depth maps. Depending on the complexity and flexibility of the coding scheme, more or less metadata may be required. For example, a very simple scheme may very strictly define the structure of the bitstream, so that little or no metadata is required to decompress it at the decoder end. The more additional features a bitstream has, the more metadata is required.

The decoder 400 decodes the encoded views (texture and depth). It transmits the decoded views to the synthesizer 500. The synthesizer 500 is connected to a display device, such as a virtual reality headset 550. The headset 550 requests the synthesizer 500 to synthesize and render a specific view of the 3D scene using the decoded views in accordance with the current position and orientation of the headset 550.

The advantage of the system shown in Fig. 1 is that it is designed to use conventional 2D video codecs to encode and decode texture and depth data. However, a disadvantage is that a large amount of data must be encoded, transmitted, and decoded. It would therefore be desirable to reduce the data rate as much as possible with the least possible detriment to the quality of the reconstructed views.

Fig. 2 shows a coding method according to a first embodiment. Fig. 3 shows a video encoder that can be configured to implement the method shown in Fig. 2. The video encoder comprises an input 310 configured to receive video data. A video processor 320 is connected to the input and is configured to receive depth maps received by the input. An encoder 330 is configured to receive processed depth maps from the video processor 320. An output 370 is configured to output a binary video stream generated by the encoder 330. The video encoder 300 also comprises a depth decoder 340, a reconstruction processor 350, and an optimizer 360. These components will be described in more detail in connection with a second embodiment of the coding method, which is described below with reference to Fig. 4.

With reference to Fig. 2 and 3, the method according to the first embodiment begins at step 110 with receiving at the input 310 video data comprising a texture map and a depth map. At steps 120 and 130, the video processor 320 processes the depth map to form a processed depth map. The processing includes non-linear filtering of the depth map at step 120 and downsampling of the filtered depth map at step 130. At step 140, the encoder 330 encodes the processed depth map and the texture map to form a binary video stream. Then, the formed binary video stream is output through the output 370.

The original views received at the input 310 may be views captured by the camera array 10. However, this is not essential, and the original views do not necessarily have to be identical to the views captured by the camera. Some of the original views received at the input 310 may be synthesized or otherwise processed original views. The number of original views received at the input 310 may be greater or less than the number of views captured by the camera array 10.

In the embodiment of Fig. 2, the nonlinear filtering 120 and downsampling 130 are combined into a single step. A 2x2 max-subsampling downsampling filter is used. This means that each pixel in the processed depth map takes the maximum pixel value in a 2x2 neighborhood of four pixels in the original input depth map. This choice of nonlinear filtering and downsampling follows from two considerations:

1. The downsampling result should not include intermediate, i.e. "transitional", depth levels. Such intermediate depth levels would be obtained, for example, when using a linear filter. The inventors realized that intermediate depth levels often give incorrect results after view synthesis on the decoder side.

2. Thin foreground objects represented in depth maps should be preserved. Otherwise, for example, a relatively thin object will disappear into the background. Note that this assumes that the foreground, i.e., close objects, are encoded as high (bright) levels, and the background, i.e., far objects, are encoded as low (dark) levels (disparity convention). An alternative implementation would be to use a 2×2 min-subsampling downsampler to achieve the same effect, provided that the z-coordinate encoding convention is used (z-coordinate increases with increasing distance from the lens).

This processing operation essentially increases the size of all local foreground objects and therefore preserves small and thin objects. However, the decoder should preferably know what operation was applied, since it should preferably undo the introduced offset and compress all objects to align the depth map with the texture again.

According to the present embodiment, the memory requirements for the video decoder are reduced. The original pixel rate was 1Y + 0.5CrCb + 1D, where Y is the luminance channel, CrCb are the color channels, and D is the depth channel. According to the present example, which uses downsampling by a factor of four (2×2), the pixel rate becomes 1Y + 0.5CrCb + 0.25D. Consequently, a 30% reduction in the pixel rate can be achieved. The most practical video decoders have a 4:2:0 ratio and do not include monochrome modes. In this case, a 37.5% reduction in pixels is achieved.

Fig. 4 is a flow chart illustrating a method for encoding according to a second embodiment. This method begins similarly to the method shown in Fig. 2, with the input 310 of the video encoder receiving source images in step 110. In steps 120a and 130a, the video processor 320 processes the depth map in accordance with a plurality of sets of processing parameters to generate a corresponding plurality of processed depth maps (each depth map corresponds to a set of processing parameters). In this embodiment, the system is designed to try each of these depth maps to determine which of them will give the best quality on the decoder side. In step 140a, each of the processed depth maps is encoded by the encoder 330. In step 154, the depth decoder 340 decodes each encoded depth map. The decoded depth maps are transmitted to the reconstruction processor 350. In step 156, the reconstruction processor 350 reconstructs the depth maps from the decoded depth maps. Then, in step 158, the optimizer 360 compares each reconstructed depth map with the original depth map of the original view to determine a reconstruction error. The reconstruction error quantifies the difference between the original depth map and the reconstructed depth map. Based on the comparison result, the optimizer 360 selects a set of parameters that resulted in a reconstructed depth map that has the smallest reconstruction error. This set of parameters is selected for use in generating a binary video stream. The output 370 outputs a binary video stream corresponding to the selected set of parameters.

It should be noted that the operation of the depth decoder 340 and the reconstruction processor 350 will be described in more detail below with reference to the decoding method (see Figs. 5-8).

In essence, video encoder 300 implements a decoder in a loop so that it can predict how a bitstream will be decoded at a far-end decoder. Video encoder 300 selects a set of parameters that will give the best performance at the far-end decoder (in terms of minimizing reconstruction error for a given target bit rate or pixel rate). The optimization may be performed iteratively, as suggested by the flow chart of Fig. 4, wherein the parameters of nonlinear filtering 120 and/or downsampling 130 are updated at each iteration after comparison 158 by optimizer 360. In an alternative implementation, video decoder may try a fixed set of parameter sets, wherein this may be done sequentially or in parallel. For example, in a highly parallel implementation, video encoder 300 may have N encoders (and decoders), each of which is configured to try a set of parameters for encoding a depth map. This may increase the number of parameter sets that can be tried in a given time, at the expense of increasing the complexity and/or size of the encoder 300.

The parameters that may be tested may include the nonlinear filtering parameters 120a, the downsampling parameters 130a, or both. For example, the system may experiment with downsampling by different factors in one or both dimensions. Similarly, the system may experiment with different nonlinear filters. For example, instead of a max filter (which assigns each pixel the maximum value in the local neighborhood), a different type of filter based on ordinal statistics may be used. For example, a nonlinear filter may analyze the local neighborhood around a given pixel and may assign the pixel the second highest value in the neighborhood. This may provide an effect similar to the max filter while helping to avoid sensitivity to individual outliers. The kernel size of the nonlinear filter is another parameter that may be varied.

It should be noted that the processing parameters in the video decoder can also be included in the parameter set (as will be described in more detail below). Thus, the video encoder can choose a set of parameters for both encoding and decoding that helps optimize quality as a function of the bit rate/pixel rate. The optimization can be performed for a given scene or for a given video sequence, or, more generally, on a training set of different scenes and video sequences. Thus, the best set of parameters can change for each sequence, for each bit rate, and/or for each allowed pixel rate.

Parameters that are useful or necessary for correct decoding of the video bitstream by the video decoder may be embedded in the metadata bitstream associated with the video bitstream. The metadata bitstream may be transmitted/transported to the video decoder together with the video bitstream or separately from it.

Fig. 5 shows a block diagram showing a method for decoding video data according to an embodiment. Fig. 6 shows a block diagram of a corresponding video decoder 400. The video decoder 400 comprises an input 410, a texture decoder 424, a depth decoder 426, a reconstruction processor 450 and an output 470. The input 410 is connected to the texture decoder 424 and the depth decoder 426. The reconstruction processor 450 is configured to receive decoded texture maps from the texture decoder 424 and to receive decoded depth maps from the depth decoder 426. The reconstruction processor 450 is configured to supply the reconstructed depth maps to the output 470.

The method of Fig. 5 begins at step 210 with the reception by the input 410 of a binary video stream and optionally a binary metadata stream. At step 224, the texture decoder 424 decodes a texture map from the binary video stream. At step 226, the depth decoder 426 decodes a depth map from the binary video stream. At steps 230 and 240, the video processor 450 processes the decoded depth map to generate a reconstructed depth map. This processing includes upsampling 230 and nonlinear filtering 240. The processing, in particular the nonlinear filtering 240, may also depend on the content of the decoded texture map, as will be described in more detail below.

An example of the method shown in Fig. 5 will now be described in more detail with reference to Fig. 8. In this embodiment, the upsampling 230 includes a nearest neighbor upsampling, in which each pixel in a 2x2 block of pixels in the upsampled map is assigned the value of one of the pixels from the decoded depth map. This 2x2 nearest neighbor upsampler scales the depth map to its original size. Like the max-pooling operation in the encoder, this procedure in the decoder avoids the creation of intermediate depth levels. The characteristics of the upsampled depth map compared to the original depth map in the encoder can be predicted in advance: the max-pooling downsampling filter tends to increase the area of foreground objects. Therefore, some depth pixels in the upsampled depth map are foreground pixels that should instead be background; However, there are usually no background depth pixels, which should instead be foreground. In other words, after upsampling, objects are sometimes too big, but usually not too small.

In the present invention, in order to cancel the displacement (of foreground objects that have increased in size), the non-linear filtering 240 of the upsampled depth maps includes a color adaptive conditional filter "erosion" (steps 242, 244 and 240a in Fig. 8). The erosion-related part (the minimum operator) provides a decrease in the size of the object, while the color adaptation provides the correct spatial position of the depth edge, i.e., the transitions in the full-scale texture map indicate where the edges should be. Due to the non-linearity with which the "erosion" filter operates (i.e., pixels are either eroded or not), the resulting edges of the object can be noisy. Pixels adjacent to the edge can give different results of the classification " erosion or not erosion " with minimally different input. Such noise negatively affects the smoothness of the edge of the object. The inventors realized that such smoothness is an important requirement for obtaining view synthesis results with sufficient perceptual quality. Therefore, the nonlinear filtering 240 also includes a contour smoothness filter (step 250) for smoothing edges in the depth map.

The nonlinear filtering 240 according to the present embodiment will now be described in more detail. Fig. 7 shows a small enlarged region of the upsampled depth map representing the filter kernel before the linear filtering 240. The gray squares indicate foreground pixels; the black squares indicate background pixels. The peripheral pixels of the foreground object are marked with an X. These are pixels that may represent an extended/enlarged region of the foreground object caused by the nonlinear filtering in the encoder. In other words, there is uncertainty as to whether the peripheral pixels X are actually foreground or background pixels.

The steps taken to carry out adaptive erosion are as follows:

1. Finding local foreground edges, i.e., peripheral pixels of foreground objects (marked with X in Fig. 7). This may be done by applying a local threshold to distinguish foreground pixels from background pixels. The peripheral pixels are then identified as those foreground pixels that are adjacent to background pixels (in this example, in a 4-connectivity sense). This is done by the reconstruction processor 450 at step 242. The depth map may, for efficiency, contain packed regions from multiple camera views. Edges at the boundaries of such regions are ignored since they do not indicate edges of objects.

2. For the identified edge pixels (e.g. the central pixel in the 5x5 kernel in Fig. 7), determine the average color of the foreground and background texture in the 5x5 kernel. This is done based on only the "trusted" pixels (marked with a dot •), in other words, the undefined edge pixels X are excluded from the calculation of the average foreground and background texture. Pixels from possible neighboring overlay regions, which are used, for example, by other types of camera, are also excluded.

3. Determining the similarity with the foreground, i.e. the reliability of the foreground : ,

where: D indicates the (e.g., Euclidean) color distance between the color of the central pixel and the average color of the background or foreground pixels. This confidence metric will be close to 1 if the central pixel is relatively more similar to the average foreground color in the neighborhood. It will be close to zero if the central pixel is relatively more similar to the average background color in the neighborhood. At step 244, the reconstruction processor 450 determines the similarity of the identified peripheral pixels to the foreground.

4. Mark with an X all peripheral pixels for which C _foreground < threshold value (e.g. 0.5).

5. Eroding all marked pixels, i.e. taking the minimum value in a local (e.g., 3x3) neighborhood. At step 240a, the reconstruction processor 450 applies this nonlinear filtering to the marked peripheral pixels (which are more similar to the background than to the foreground).

As mentioned above, this process can introduce noise and can result in jagged edges in the depth map. The steps taken to smooth the edges of an object represented in the depth map are as follows:

1. Finding local foreground edges, i.e. peripheral pixels of foreground objects (like those marked with X in Fig. 7).

2. For these edge pixels (e.g. the center pixel in Fig. 7), count the number of background pixels in a 3x3 kernel around the pixel of interest.

3. Mark all edge pixels for which the counted number > threshold value.

4. Eroding all marked pixels, i.e. taking the minimum value in a local (e.g. 3×3) neighborhood. This step is performed by the reconstruction processor 450 at step 250.

This antialiasing tends to convert outlying or protruding foreground pixels into background pixels.

In the above example, the method for determining whether an outlier peripheral pixel is protruding from a foreground object uses the number of background pixels in a 3x3 kernel. Other methods may be used. For example, as an alternative or in addition to counting the number of pixels, the positions of the foreground and background pixels in the kernel may be analyzed. If the background pixels are all on one side of the pixel in question, then it is more likely to be a foreground pixel. On the other hand, if the background pixels are scattered all around the pixel in question, then that pixel may be an outlier or noise, and is more likely to actually be a background pixel.

The pixels in the kernel can be classified binary as foreground or background. A binary flag encodes this for each pixel, with a logical "1" indicating background and a logical "0" indicating foreground. The neighborhood (i.e. the pixels in the kernel) can then be described by an n-digit binary number, where n is the number of pixels in the kernel surrounding the pixel of interest. One example of how to construct a binary number is given in the following table:

b ₇ = 1 b ₆ = 0 b ₅ = 1 _b4 =0 b ₃ = 0 _b2 =1 b ₁ = 0 b ₀ = 1

In this example, b = b ₇ b ₆ b ₅ b ₄ b ₃ b ₂ b ₁ b ₀ = 10100101 ₂ = 165 (it should be noted that the algorithm described above with reference to Fig. 5 corresponds to counting the number of non-zero bits in b (= 4)).

Training involves calculating, for each value of b, how often the pixel of interest (the central pixel of the kernel) is foreground or background. Assuming equal costs for false positives and misses, a pixel is defined as foreground if it is more likely (in the training set) to be a foreground pixel than a background pixel, and vice versa.

The decoder implementation will construct b and extract the answer (the pixel of interest is the foreground pixel or the pixel of interest is the background pixel) from a look up table (LUT).

The approach of nonlinearly filtering the depth map in both the encoder and decoder (e.g., dilation and erosion, respectively, as described above) is counterintuitive, since it would naturally be expected to remove information from the depth map. However, the inventors surprisingly found that the smaller the depth maps produced using the nonlinearly downsampled approach, the higher the quality they can be encoded with (using a conventional video codec) at a given bit rate. This quality gain outweighs the loss in reconstruction; therefore, the net effect is to improve end-to-end quality while reducing the pixel rate.

As described above with reference to Fig. 3 and 4, a decoder can be implemented within the video encoder to optimize the parameters of nonlinear filtering and downsampling and thereby reduce the reconstruction error. In this case, the depth decoder 340 in the video encoder 300 is substantially identical to the depth encoder 426 in the video decoder 400, and the reconstruction processor 350 in the video encoder 300 is substantially identical to the reconstruction processor 450 in the video decoder 400. These respective components perform substantially identical processes.

If the parameters of the nonlinear filtering and downsampling in the video encoder have been selected so as to reduce the reconstruction error, as described above, the selected parameters can be signaled in a binary metadata stream, which is the input to the video decoder. The reconstruction processor 450 can use the parameters signaled in the binary metadata stream as an aid for the correct reconstruction of the depth map. The reconstruction processing parameters can include, but are not limited to: an upsampling factor in one or both dimensions; a kernel size for identifying peripheral pixels of foreground objects; a kernel size for erosion; a type of nonlinear filtering to be applied (for example, whether to use a "minimum" filter or another type of filter); a kernel size for identifying foreground pixels and a kernel size for smoothing.

An alternative embodiment will now be described with reference to Fig. 9. In this embodiment, a neural network architecture is used instead of hand-coded nonlinear filters for the encoder and decoder. The neural network is divided to model the depth downscaling operation and the depth upscaling operation. This network is trained end-to-end and learns both the optimal downscaling and the optimal upscaling. However, during development (i.e., for encoding and decoding real sequences), the first part is used before the video encoder, and the second part is used after the video decoder. Thus, the first part provides nonlinear filtering 120 for the encoding method, and the second part provides nonlinear filtering 240 for the decoding method.

The network parameters (weights) of the second part of the network can be transmitted as metadata using a binary stream. It should be noted that different sets of network parameters can be created corresponding to different encoding configurations (different downscaling factors, different target bit rates, etc.). This means that the upscaling filter for the depth map will behave optimally for a given texture map bit rate. This can improve performance, since texture encoding artifacts change the luminance and chroma characteristics, and especially at object boundaries, this change will lead to different weights of the depth upscaling neural network.

Fig. 9 shows an example architecture for this embodiment, in which the neural network is a convolutional neural network (CNN). The symbols in the diagram mean the following:

- full resolution 3-channel input texture map;

- decoded full resolution texture map;

D - input 1-channel full resolution depth map;

- reduced scale depth map;

- decoded scaled depth map;

- convolution with kernel ;

- coefficient zoom out;

-coefficient upsampling.

Each vertical black bar in the diagram represents a tensor of input or intermediate data, in other words, the input to a neural network layer. The dimensions of each tensor are described by a triple ( p, w, h ), where w and h are the width and height of the image, respectively, and p is the number of planes or channels of data. Accordingly, the input texture map has dimensions (3, w , h ) — three planes corresponding to the three color channels. The downsampled depth map has dimensions (1, w /2, h /2).

The downscaling of P _k may involve an averaging downscaling operation with a factor of k or a max-pooling (or min-pooling) operation with a kernel size of k. The averaging downscaling operation may introduce some intermediate values, but subsequent layers of the neural network may correct for this (e.g. based on texture information).

It should be noted that during the training phase, the decoded depth map, , do not use. Instead, use an uncompressed map downscaled depth. The reason for this is that the neural network training phase requires derivative calculations, which is not possible for a nonlinear video encoder function. In practice, this approximation is likely to be valid, especially for higher quality (higher bit rates). In the inference phase (i.e. when processing real video data), the uncompressed map the reduced scale depth is obviously not available to the video decoder. Therefore, a decoded map is used depth. It should also be noted that the decoded texture map full resolution is used in the training phase as well as in the inference phase. There is no need to calculate derivatives, since this is auxiliary information and not data processed by the neural network.

The second part of the neural network (after video decoding) will typically contain only a few convolutional layers due to the complexity of constraints that are possible on the client device.

For the deep learning approach to be used, it is important to have training data. In this case, it is easy to obtain. The uncompressed texture image and the full-resolution depth map are used on the input side before video encoding. The second part of the network uses the decoded texture and the downscaled depth map (via the first half of the network as input for training), and the error is estimated from the ground truth full-resolution depth map, which was also used as input. Therefore, in essence, the overlays from the original high-resolution depth map serve as both input and output for the neural network. Therefore, the network has some aspects of both the autoencoder and UNet architectures. However, the proposed architecture is not just a combination of these approaches. For example, the decoded texture map is fed to the second part of the network as auxiliary data for optimal reconstruction of the high-resolution depth map.

In the example shown in Fig. 9, the input data to the neural network in video encoder 300 comprises a texture map and a depth map D. The downsampling P ₂ is performed between the two layers of the neural network. There are three layers of the neural network before the downsampling and two layers after it. The output of the neural network portion of the video encoder 300 contains a map reduced scale depth. It is encoded by the encoder 320 at step 140.

The encoded depth map is transmitted to the video decoder 400 in a binary video stream. It is decoded using the depth decoder 426 in step 226. The result is a decoded map downscaled depth. It is upsampled (U ₂ ) for use in the neural network portion of video decoder 400. Another part of the input to this portion of the neural network is the decoded texture map full resolution, which is generated by the 424 texture decoder. The second part of the neural network has three layers. It produces as output a reconstructed estimate , which is compared with the original depth map D to obtain the final error e .

From the above, it is clear that neural network processing can be implemented in the video encoder 300 using the video processor 320 and in the video decoder 400 using the reconstruction processor 450. In the example shown, nonlinear filtering 120 and downsampling 130 are performed in an integrated manner using a portion of the neural network in the video encoder 300. In the video decoder 400, upsampling 230 is performed separately before the nonlinear filtering 240, which is performed using the neural network.

It is understood that the arrangement of the neural network layers shown in Fig. 9 is not restrictive and can be changed in other implementations. In this example, the network produces downsampled depth maps of size 2×Of course, other scaling factors can also be used.

Several of the implementations described above mentioned maximum filtering, maximum subsampling, dilation, or similar operations in the encoder. It is understood that these implementations assume that depth is encoded as 1/d (or some other similar inverse relationship), where d is the distance from the camera. Under this assumption, higher values in the depth map indicate foreground objects, and lower values in the depth map indicate the background. Thus, by applying a maximum or dilation-type operation, the method tends to increase foreground objects. The corresponding inverse process in the decoder may consist of applying a minimum or erosion-type operation.

Of course, in other implementations, depth may be encoded as d or log d (or some other variable that is positively correlated with d). This means that foreground objects are represented by low d values and background objects by high d values. In such implementations, min-selection filtering, min-pooling, erosion, or some similar operation may be performed in the encoder. Again, this will tend to increase foreground objects, which is the goal. The corresponding inverse process in the decoder may be to apply a max-type operation or a dilation.

The encoding and decoding methods of Figs. 2, 4, 5, 8 and 9, and the encoder and decoder of Figs. 3 and 6 may be implemented in hardware or software, or a combination of both (e.g., as firmware running on a hardware device). To the extent that an embodiment is partially or completely implemented in software, the functional steps shown in the process flow charts may be performed by suitably programmed physical computing devices, such as one or more central processing units (CPUs), graphic processing units (GPUs), or neural network accelerators (NNAs). Each process and its individual constituent steps, as shown in the flow charts, may be performed by the same or different computing devices. According to embodiments, a computer-readable data storage medium stores a computer program containing a computer program code configured to cause one or more physical computing devices to perform an encoding or decoding method as described above when the program is executed on one or more physical computing devices.

The storage medium may comprise volatile and nonvolatile computer memory, such as RAM, PROM, EPROM, and EEPROM. The various storage media may be located within the computing device or may be movable so that one or more programs stored on the storage media may be loaded into the processor.

According to one embodiment, the metadata may be stored on a data storage medium. According to one embodiment, the bitstream may be stored on the same data storage medium or on a different data storage medium. The metadata may be embedded in the bitstream, but this is not essential. Similarly, the metadata and/or the bitstreams (with or without the metadata in the bitstream) may be transmitted as a signal modulated on an electromagnetic carrier wave. The signal may be defined in accordance with a digital communication standard. The carrier wave may be an optical carrier, a radio frequency wave, a millimeter wave, or a near-field communication wave. It may be transmitted by wire or wirelessly.

To the extent that the embodiment is partially or completely implemented in hardware, the blocks shown in the block diagrams of Figs. 3 and 6 may be separate physical components or logical units of separate physical components, or they may all be implemented in an integrated manner in a single physical component. The functions of one block shown in the drawings may be divided between several components when implemented, or the functions of several blocks shown in the drawings may be combined into separate components when implemented. For example, although the texture decoder 424 and the depth decoder 426 are shown as separate components in Fig. 6, their functions may be provided by a single unified decoder component.

In general, examples of methods for encoding and decoding data, a computer program that implements these methods, and video encoders and video decoders are indicated in the following embodiments.

IMPLEMENTATION OPTIONS

1. A method for encoding video data containing one or more source views, wherein each source view contains a texture map and a depth map, wherein the method includes:

receiving (110) video data;

processing a depth map of at least one source view to generate a processed depth map, including:

nonlinear filtering (120) and

downsampling (130); and

encoding (140) the processed depth map and the texture map of at least one original view to form a binary video stream.

2. The method according to embodiment 1, wherein the nonlinear filtering includes increasing the region of at least one foreground object in the depth map.

3. The method according to embodiment 1 or 2, wherein the nonlinear filtering includes the use of a filter developed using a machine learning algorithm.

4. The method according to any of the previous embodiments, in which the nonlinear filtering is performed using a neural network containing multiple layers, and the downsampling is performed between two of the said layers.

5. The method according to any of the previous embodiments, which includes processing (120a, 130a) a depth map in accordance with a plurality of sets of processing parameters to generate a corresponding plurality of processed depth maps,

the method also includes:

selecting a set of processing parameters that reduces the reconstruction error of the reconstructed depth map after the corresponding processed depth map has been encoded and decoded; and

generating a binary metadata stream identifying the selected set of parameters.

6. A method for decoding video data containing one or more source types, comprising:

receiving (210) a binary video stream containing an encoded depth map and an encoded texture map for at least one source view;

decoding (226) the encoded depth map to obtain a decoded depth map;

decoding (224) the encoded texture map to obtain a decoded texture map; and

processing the decoded depth map to form a reconstructed depth map, wherein the processing includes:

upsampling (230) and

nonlinear filtering (240).

7. The method according to embodiment 6, also including, before the step of processing the decoded depth map to form the reconstructed depth map, detecting that the decoded map has a lower resolution than the decoded texture map.

8. The method according to embodiment 6 or embodiment 7, wherein the nonlinear filtering comprises reducing the area of at least one foreground object in the depth map.

9. The method according to any one of embodiments 6-8, wherein the processing of the decoded depth map is based at least in part on the decoded texture map.

10. A method according to any of embodiments 6-9, comprising:

upsampling (230) the decoded depth map;

identifying (242) peripheral pixels of at least one foreground object in the upsampled depth map;

determining (244) based on the decoded texture map whether the peripheral pixels are more similar to the foreground object or to the background; and

applying (240a) nonlinear filtering only to the peripheral pixels that are determined to be more similar to the background.

11. The method according to any of the embodiments, wherein the nonlinear filtering includes smoothing (250) the edges of at least one foreground object.

12. The method according to any one of embodiments 6-11, further comprising receiving a binary metadata stream associated with the binary video stream, wherein the binary metadata stream identifies a set of parameters,

and the method also includes processing the decoded depth map in accordance with the identified set of parameters.

13. A computer program containing computer code for calling the implementation by the processing system of any of the implementation options 1-12 when the specified program is launched in the processing system.

14. A video encoder (300) configured to encode video data comprising one or more source views, wherein each source view comprises a texture map and a depth map, wherein the video encoder comprises:

an input (310) capable of receiving video data;

a video processor (320) configured to process a depth map of at least one source view to generate a processed depth map, including:

nonlinear filtering (120) and

downsampling (130);

an encoder (330) configured to encode a texture map of at least one original view and a processed depth map to form a binary video stream; and

output (360) capable of outputting a binary video stream.

15. A video decoder (400) configured to decode video data containing one or more source views, wherein the video decoder comprises:

a binary stream input (410) configured to receive a binary video stream, wherein the binary video stream comprises an encoded depth map and an encoded texture map for at least one source view;

a first decoder (426) configured to decode from the binary video stream an encoded depth map to obtain a decoded depth map;

a second decoder (424) configured to decode from the binary video stream an encoded texture map to obtain a decoded texture map;

a reconstruction processor (450) configured to process the decoded depth map to generate a reconstructed depth map, wherein the processing includes:

upsampling (230) and

nonlinear filtering (240),

and an output (470) configured to output the reconstructed depth map.

Hardware components for use in embodiments of the present invention include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs). One or more units may be implemented as a combination of specialized hardware to perform some functions and one or more programmed microprocessors and associated circuits to perform other functions.

More specifically, the present invention is defined by the appended claims.

Other variations of the described embodiments may be understood and effected by a person skilled in the art in practicing the claimed invention after reading the drawings, description and appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the grammatical means of expression of the singular does not exclude a plurality. A single processor or other unit may perform the functions of several articles specified in the claims. The mere fact that some measures are listed in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. In the case of a computer program discussed above, it may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied with or as part of other hardware, but may also be distributed in other forms, for example via the Internet or other wired or wireless telecommunication systems. When the term "made so that" is used in the claims or description, it should be noted that the term "made so that" should be considered equivalent to the term "made with the possibility of". No reference signs in the claims should be considered as limiting their scope.

Claims (54)

1. A method for encoding video data containing one or more source views, wherein each source view contains a texture map and a depth map, wherein the method includes: receiving (110) video data; processing a depth map of at least one source view to generate a processed depth map; and encoding (140) the processed depth map and the texture map of at least one original view to form a binary video stream, characterized in that the processing includes: nonlinear filtering (120) of the depth map to form a nonlinearly filtered depth map and downsampling (130) the nonlinearly filtered depth map to form a processed depth map; wherein the nonlinear filtering comprises increasing the region of at least one foreground object in the depth map. 2. The method according to claim 1, wherein the nonlinear filtering includes applying a filter developed using a machine learning algorithm. 3. The method according to any of the preceding claims, wherein the nonlinear filtering is performed using a neural network containing a plurality of layers, and the downsampling is performed between two of said layers. 4. The method according to any one of the preceding claims, which includes processing (120a, 130a) a depth map in accordance with a plurality of sets of processing parameters to generate a corresponding plurality of processed depth maps, wherein the processing parameters include at least one of: determining the nonlinear filtering performed, determining the downsampling performed, and determining the processing operations to be performed in the decoder during reconstruction of the depth map, the method also includes: selecting a set of processing parameters that reduces the reconstruction error of the reconstructed depth map after the corresponding processed depth map has been encoded and decoded; and generating a binary metadata stream identifying the selected set of parameters. 5. A method for decoding video data containing one or more source types, comprising: receiving (210) a binary video stream containing an encoded depth map and an encoded texture map for at least one source view; decoding (226) the encoded depth map to obtain a decoded depth map; decoding (224) the encoded texture map to obtain a decoded texture map, and processing the decoded depth map to form a reconstructed depth map, wherein the processing includes: upsampling (230) the decoded depth map to form an upsampled depth map, and the method differs in that the processing also includes nonlinear filtering (240) the upsampled depth map to form a reconstructed depth map, wherein the nonlinear filtering comprises reducing the area of at least one foreground object in the depth map. 6. The method according to claim 5, also comprising, before the step of processing the decoded depth map to form the reconstructed depth map, detecting that the decoded depth map has a lower resolution than the decoded texture map. 7. The method according to any one of claims 5, 6, wherein the processing of the decoded depth map is based at least in part on the decoded texture map. 8. The method according to any of paragraphs 5-7, including: upsampling (230) the decoded depth map; identifying (242) peripheral pixels of at least one foreground object in the upsampled depth map; determining (244) based on the decoded texture map whether the peripheral pixels are more similar to the foreground object or to the background; and applying (240a) nonlinear filtering only to the peripheral pixels that are determined to be more similar to the background. 9. The method according to any one of claims 5-8, wherein the nonlinear filtering comprises smoothing (250) the edges of at least one foreground object. 10. The method according to any one of paragraphs. 5-9, further comprising receiving a binary metadata stream associated with the binary video stream, wherein the binary metadata stream identifies a set of parameters including a definition of nonlinear filtering and/or a definition of upsampling to be performed, and the method also includes processing the decoded depth map in accordance with the identified set of parameters. 11. A computer-readable data carrier storing a computer program containing computer code for calling the implementation by the processing system of the method according to any of paragraphs 1-10 upon launching the specified program in the processing system. 12. A video encoder (300) configured to encode video data comprising one or more source views, wherein each source view comprises a texture map and a depth map, wherein the video encoder comprises: an input (310) capable of receiving video data; a video processor (320) configured to process a depth map of at least one source view to generate a processed depth map, an encoder (330) configured to encode a texture map of at least one original view and a processed depth map to form a binary video stream; and output (360) configured to output a binary video stream, characterized in that the processing includes: nonlinear filtering (120) of the depth map to form a nonlinearly filtered depth map and downsampling (130) the nonlinearly filtered depth map to form a processed depth map; wherein the nonlinear filtering comprises increasing the region of at least one foreground object in the depth map. 13. A video decoder (400) configured to decode video data containing one or more source views, wherein the video decoder comprises: a binary stream input (410) configured to receive a binary video stream, wherein the binary video stream comprises an encoded depth map and an encoded texture map for at least one source view; a first decoder (426) configured to decode from the binary video stream an encoded depth map to obtain a decoded depth map; a second decoder (424) configured to decode from the binary video stream an encoded texture map to obtain a decoded texture map; a reconstruction processor (450) configured to process the decoded depth map to generate a reconstructed depth map, and an output (470) configured to output a reconstructed depth map, characterized in that the processing includes: upsampling (230) the decoded depth map to form an upsampled depth map and nonlinear filtering (240) the upsampled depth map to form a reconstructed depth map, wherein the nonlinear filtering comprises reducing the area of at least one foreground object in the depth map.