CN111937041A - Video encoding by providing geometry proxies - Google Patents

Abstract

Compressing a video frame includes receiving a video frame, identifying a three-dimensional (3D) object in the frame, matching the 3D object to a stored 3D object, compressing the video frame using a color prediction scheme based on the 3D object and the stored 3D object, and storing the compressed frame with metadata that identifies the 3D object, indicates a location of the 3D object in the video frame, and indicates an orientation of the 3D object in the video frame.

Description

Video encoding by providing geometry proxies

Related applications

This application is a continuation of and claims the benefit of US Non-Provisional Application No. 16/143,165, filed September 26, 2018, the entire contents of which are incorporated herein by reference.

technical field

Embodiments relate to compression and decompression of three-dimensional (3D) video data.

Background technique

The techniques used for video compression are all related by a common method. Typically, frames of video are temporarily compressed by defining blocks of the frame as residuals (eg, in terms of displacements from previous or future frames). For objects within the frame that have residuals that can be characterized by in-plane rigid transformations (e.g. objects that are shifted and rotated in the image plane over time), this compression technique is generally acceptable (e.g., when decompressed with minimal artifacts or errors). Although this model captures many video dynamic sources (eg, camera or scene translation), there are common scenarios for which this is not the best model (inefficient when decompressing or contains too many artifacts or errors).

In other words, typical prediction schemes can reliably predict pixels/blocks/patches in previous and/or future frames (e.g., keyframes) for use when objects move mostly linearly and/or have transitions from frame to frame of predictable movements. However, when objects have dynamic non-linear motion from frame to frame, typical prediction schemes may not be able to reliably predict pixels/blocks/patches in previous and/or future frames (eg, keyframes) for computation residual. Therefore, using a displacement prediction model will likely result in very little compression when objects have dynamic non-linear motion from frame to frame.

SUMMARY OF THE INVENTION

Example embodiments describe systems and methods for compressing frames of video using color prediction by a geometric proxy.

In general aspects, a method and a non-transitory computer readable storage medium having computer executable program code stored thereon that, when executed on a computer system, cause the computer system to perform steps. The steps include, receiving a frame of the video, identifying a three-dimensional (3D) object in the frame, matching the 3D object to a stored 3D object, compressing the video frame using a color prediction scheme based on the 3D object and the stored 3D object, and storing the compressed frame with metadata identifying the 3D object, indicating the position of the 3D object in the video frame and indicating the orientation of the 3D object in the video frame.

Implementations may include one or more of the following features. For example, compressing the frames of the video using a color prediction scheme based on the 3D object and the stored 3D object may include generating a first 3D object proxy based on the stored 3D object, transforming the first 3D object based on the 3D object identified in the frame Proxy, generate a second 3D object proxy based on the stored 3D object, identify the 3D object in keyframes of the video, transform the second 3D object proxy based on the 3D object identified in the keyframe, map color attributes from the 3D object to the a transformed first 3D object proxy, mapping color properties from the 3D object identified in the keyframe to the transformed second 3D object proxy, and based on the color properties of the transformed first object and the transformed second 3D object The color properties of the proxy generate residuals for 3D objects.

For example, compressing the frames of the video using a color prediction scheme based on the 3D objects and the stored 3D objects may include generating a first 3D object proxy based on the stored 3D objects, transforming the first 3D object proxy based on the 3D objects identified in the frame , generate a second 3D object proxy based on the stored 3D object, identify the 3D object in a key frame of the video, transform the second 3D object proxy based on the 3D object identified in the key frame, map color attributes from the 3D object to the transformed and a residual of the 3D object is generated based on a color attribute of the transformed first 3D object proxy and a default color attribute of the transformed second 3D object proxy.

For example, compressing the frames of the video using a color prediction scheme based on the 3D object and the stored 3D object may include generating a first 3D object proxy based on the stored 3D object, encoding the first 3D object proxy using an auto-encoder, Transform-encoded first 3D object proxy based on the 3D object identified in the frame, generate a second 3D object proxy based on the stored 3D object, encode the second 3D object proxy using an autoencoder, identify the 3D object proxy in the key frame of the video object, transform-coded second 3D object proxy based on the 3D object identified in the keyframe, maps color properties from the 3D object to the transformed first 3D object proxy, converts the color properties from the 3D object identified in the keyframe to the transformed first 3D object proxy The color properties of the are mapped to the transformed second 3D object proxy, and a residual is generated for the 3D object based on the color properties for the transformed first 3D object proxy and the color properties for the transformed second 3D object proxy.

For example, compressing the frames of the video using a color prediction scheme based on the 3D objects and the stored 3D objects may include generating a first 3D object proxy based on the stored 3D objects, encoding the first 3D object proxy using an auto-encoder, based on Transform the encoded first 3D object proxy with the 3D object identified in the frame, generate a second 3D object proxy based on the stored 3D object, encode the second 3D object proxy using an auto-encoder, identify in key frames of the video a 3D object, transforming the encoded second 3D object proxy based on the 3D object identified in the keyframe, mapping color properties from the 3D object to the transformed first 3D object proxy, and based on the transformed first 3D object The color properties of the proxy and the default color properties of the transformed second 3D object proxy generate the residual for the 3D object.

For example, prior to storing the 3D object, the steps may further include: identifying at least one 3D object of interest associated with the video, determining a plurality of mesh attributes associated with the 3D object of interest, determining Locations associated with objects, determine orientations associated with 3D objects of interest, determine multiple color properties associated with 3D objects of interest, and reduce mesh properties associated with 3D objects of interest using autoencoders number of variables. Compressing the frames of the video may include determining the positional coordinates of the 3D object relative to the origin coordinates of the background 3D object in the keyframe. The stored 3D objects may include default color properties, and the color prediction scheme may use the default color properties. The steps may further include: identifying at least one 3D object of interest associated with the video; generating at least one stored 3D object, each of the at least one stored 3D object, based on the at least one 3D object of interest A grid definition comprising a collection of points connected by faces, each point storing at least one attribute including the positional coordinates of the corresponding point, and at least one stored 3D object being stored in association with the video.

In another general aspect, a method and non-transitory computer-readable storage medium having computer-executable program code stored thereon that, when executed on a computer system, causes the computer system to perform steps. These steps include receiving frames of the video, identifying three-dimensional (3D) objects in the frames, matching the 3D objects to stored 3D objects, and decompressing the frames of the video using a color prediction scheme based on the 3D objects and the stored 3D objects , and the frame of the rendered video.

Implementations may include one or more of the following features. For example, decompressing the frame of the video using a color prediction scheme based on the 3D object and the stored 3D object may include generating a first 3D object proxy based on the stored 3D object; transforming the first 3D object based on the 3D object identified in the frame An object proxy that identifies the 3D object in the keyframes of the video, transforms the second 3D object proxy based on the 3D object identified in the keyframe, maps the color properties from the 3D object to the transformed first 3D object proxy, converts the color properties Mapping from the 3D object identified in the keyframe to the transformed second 3D object proxy, and generating the color of the 3D object based on the color properties of the transformed first 3D object proxy and the transformed color properties of the second 3D object proxy Attributes.

For example, decompressing a frame of a video using a color prediction scheme based on the 3D object and the stored 3D object may include generating a first 3D object proxy based on the stored 3D object, transforming the first 3D object proxy based on the 3D object identified in the frame , identify the 3D object in the key frame of the video, transform the second 3D object proxy based on the 3D object identified in the key frame, map the color attributes from the 3D object to the transformed first 3D object proxy, based on the transformed first 3D object proxy The color properties of the 3D object proxy and the transformed default color properties of the second 3D object proxy generate the color properties of the 3D object.

For example, decompressing the frame of the video using a color prediction scheme based on the 3D object and the stored 3D object may include generating a first 3D object proxy based on the stored 3D object, decoding the first 3D object proxy using an auto-encoder, Transform the decoded first 3D object proxy based on metadata associated with the 3D object, generate a second 3D object proxy based on the stored 3D object, decode the second 3D object proxy using an auto-encoder, at keyframes of the video identify the 3D object in the keyframe, transform the decoded second 3D object proxy based on the metadata associated with the 3D object identified in the keyframe, map the color attributes from the 3D object to the transformed first 3D object proxy, The color properties are mapped from the 3D object identified in the keyframe to the transformed second 3D object proxy, and the 3D object is generated based on the transformed color properties of the first 3D object proxy and the transformed color properties of the second 3D object proxy color properties.

For example, decompressing the frame of the video using a color prediction scheme based on the 3D object and the stored 3D object may include generating a first 3D object proxy based on the stored 3D object, decoding the first 3D object proxy using an auto-encoder, Transforming the decoded first 3D object proxy based on metadata associated with the 3D object, generating a second 3D object proxy based on the stored 3D object, decoding the second 3D object proxy using an autoencoder, Identifying the 3D object in the keyframe, transforming the decoded second 3D object proxy based on metadata associated with the 3D object identified in the keyframe, mapping color attributes from the 3D object to the transformed first 3D object proxy , and the color properties of the 3D object are generated based on the transformed color properties of the first 3D object proxy and the default properties of the transformed second 3D object proxy.

For example, the steps may further include using machine-trained generative modeling techniques to receive at least one potential representation of the 3D shape to: determine a plurality of mesh attributes associated with the 3D shape; determine a location associated with the 3D shape; determine a location associated with the 3D shape an orientation associated with the shape; and determining a plurality of color properties associated with the 3D shape and storing the 3D shape as a stored 3D object. Rendering the frame of the video may include receiving position coordinates of the 3D object relative to the origin coordinates of the background 3D object in the keyframe, and using the position coordinates to place the 3D object in the frame. The steps may further include: receiving a neural network used by an encoder of the auto-encoder to reduce the number of variables associated with mesh properties, position, orientation, and color properties of the at least one 3D object of interest, in the auto-encoder The decoder uses a neural network to regenerate points associated with the mesh of at least one 3D object of interest, the regenerated points include regenerating position attributes, orientation attributes, and color attributes, and stores the at least one 3D object of interest as stored 3D objects.

In yet another general aspect, a method and non-transitory computer-readable storage medium having computer-executable program code stored thereon, the computer-executable program code, when executed on a computer system, causes the computer system to execute for use Steps for the agent to predict color variance. The steps include: generating a first 3D object proxy based on the stored 3D objects; generating a second 3D object proxy based on the stored 3D objects; transforming the first 3D object proxy based on the 3D objects identified in the frames of the video; The 3D object identified in the keyframe of mapping to the transformed second 3D object proxy, and generating color data for the 3D object based on the transformed color properties of the first 3D object proxy and the transformed color properties of the second 3D object proxy.

Implementations may include one or more of the following features. For example, the steps may further comprise: encoding the first 3D object proxy using an autoencoder prior to transforming the first 3D object proxy, and using the autoencoder to encode the second 3D object proxy prior to transforming the second 3D object proxy to encode. The steps may further include decoding the first 3D object proxy using an autoencoder after transforming the first 3D object proxy, and decoding the second 3D object proxy using the autoencoder after transforming the second 3D object proxy. Generating color data for the 3D object may include subtracting the color attribute of the transformed first 3D object proxy from the color attribute of the transformed second 3D object proxy. Generating color data for the 3D object may include adding color attributes of the transformed first 3D object proxy to color attributes of the transformed second 3D object proxy.

Description of drawings

Example embodiments will become more fully understood from the following detailed description and accompanying drawings, wherein like elements are denoted by like reference numerals, which are given by way of illustration only and are therefore not Example embodiments are limited and where:

1 illustrates a block diagram of a signal flow for compressing video, according to an example embodiment.

2 illustrates a block diagram of a signal flow for storing compressed video, according to an example embodiment.

3A illustrates a block diagram of an encoder prediction module according to an example embodiment.

3B illustrates a block diagram of another encoder prediction module according to an example embodiment.

4A illustrates a block diagram of a decoder prediction module according to an example embodiment.

4B illustrates a block diagram of another decoder prediction module according to an example embodiment.

5A illustrates a block diagram of a signal flow for encoding a 3D object, according to an example embodiment.

5B illustrates a block diagram of a signal flow for decoding a 3D object, according to an example embodiment.

6A illustrates a block diagram of a signal flow for streaming video and rendering the video on a client device.

6B illustrates a block diagram of another signal flow for streaming video and rendering the video on a client device.

7 illustrates a block diagram of a method for compressing frames of video, according to at least one example embodiment.

8 illustrates a block diagram of another method for compressing frames of video, according to at least one example embodiment.

9 illustrates a block diagram of a method for decompressing and rendering frames of a video, according to at least one example embodiment.

10 illustrates a block diagram of a method for compressing a 3D object according to at least one example embodiment.

11 illustrates a block diagram of a method for decompressing a 3D object according to at least one example embodiment.

12 illustrates a video encoder system in accordance with at least one example embodiment.

13 illustrates a video decoder system in accordance with at least one example embodiment.

14 illustrates an example of a computer device and a mobile computer device in accordance with at least one example embodiment.

It should be noted that these figures are intended to illustrate the general characteristics of the methodologies and/or structures utilized in certain example embodiments and are intended to supplement the written description provided below. However, these drawings are not to scale and may not accurately reflect the precise structural or performance characteristics of any given embodiment, and should not be construed as defining or limiting the range of values or properties encompassed by example embodiments. The use of similar or identical reference numbers in the various figures is intended to indicate the presence of similar or identical elements or features.

Detailed ways

While example embodiments may include various modifications and alternative forms, embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intention to limit example embodiments to the particular forms disclosed, but on the contrary, example embodiments are to cover all modifications, equivalents, and alternatives falling within the scope of the claims. Throughout the description of the drawings, the same reference numerals refer to the same elements.

In a 3D dynamic environment, 3D objects undergoing complex 3D transformations may not allow simple rigid transformations based on previous and/or future frames, previously compressed images or frames, and the like. For example, a human actor undergoing a transformation consisting of rigidity, sharpness, and deformation components will result in highly nonlinear transformations in pixel space. Subsequently, there may be no suitable corresponding blocks in the nearby keyframes. These 3D objects are referred to herein as dynamic 3D objects.

Non-dynamic can include 3D objects that appear to move from frame to frame due to camera or scene translation, which can be encoded/decoded using color prediction by geometry proxies. In this example, a stationary 3D object may appear to be moving from frame to frame because the camera capturing the scene is moving (eg, in a predictable manner and/or direction). In this example, a 3D object (eg, a vehicle, ie, a train, car, or airplane) may move from frame to frame in a predictable manner (eg, with a constant speed and/or direction). These objects are sometimes referred to herein as translation 3D objects. Another example of a non-dynamic 3D object is a 3D object that does not appear to move within the scene. In this example, a stationary or stationary 3D object (eg, the background of the scene, furniture in a fixed position within the scene, or a slowly moving object in the distance) may appear stationary from frame to frame (eg, without any cameras) or scene pan). These objects are sometimes referred to herein as stationary or background 3D objects.

Example embodiments use stored 3D objects as geometric proxies for objects that appear in the video. The stored 3D objects may be deformable 3D shape models (eg, meshes with given properties that can be manipulated if necessary) that can be used in the prediction scheme. For example, the positions of pixels associated with dynamic 3D objects in the frames of the video may be predicted based on the stored 3D objects. Similarly, prediction schemes may be used to compress background images in frames of video, 3D objects that move (eg, predictably move) within the image plane of frames of video, portions of 3D objects, layers of video frames that include 3D objects (eg, as part of a Z-order image), a container that includes 3D objects (eg, standard mesh objects), etc.

One or more implementations may include using stored 3D objects to locate pixels, blocks, and/or patches in keyframes. One or more implementations may include matching pixels, blocks, and/or patches in a frame of a video to pixels, blocks, and/or patches of a stored 3D object. One or more embodiments may include compressing the stored 3D object prior to storing the stored 3D object, and decompressing the stored 3D object during the prediction process. One or more implementations may include compressing the stored 3D objects prior to storing them; compressing the 3D objects associated with the frames of the video; and using the compressed 3D objects in the prediction process.

1 illustrates a block diagram of a signal flow for compressing video, according to an example embodiment. As shown in FIG. 1 , encoder 105 includes frame 110 , prediction module 115 and encoding module 120 . In addition, video storage 125 includes metadata 130 , stored 3D objects 135 and compressed frames 140 .

Video data 5 is input to encoder 105 , where frame 110 is selected from a plurality of frames included in video data 5 . Video data 5 may correspond to 3D video (eg, a monocular view), 2D video, a portion of a video (eg, less than all frames of a video), or the like. Accordingly, frames 110 may include data corresponding to frames of video. The encoder 105 may be configured to use a color prediction scheme based on using 3D objects as geometric proxies. The encoder 105 may compress the frame 110 using a color prediction scheme based on the use of 3D objects as geometric proxies. Compressing the frame 110 may reduce the amount of data used to store and/or communicate the frame 110 . The compressed frame 110 may include prediction steps, quantization steps, transformation steps, and entropy coding steps.

Inter prediction can exploit spatial redundancy (eg, correlation between pixels between frames) by computing delta values expressed in terms of one or more adjacent frames. Incremental coding may include locating relevant pixels/blocks/patches in keyframes (eg, previous adjacent keyframes, upcoming adjacent keyframes), and then computing delta values for pixels in the frame being encoded . The incremental value can be called residual. Thus, inter prediction can produce residuals of pixels/blocks/patches in a frame (eg, 3D objects). In an example embodiment, the delta value may be expressed in terms of an explicit texture (color), a default texture, a predefined texture, the texture of the identified 3D object, or the like.

Prediction module 115 may be configured to locate 3D objects in frame 110 and/or keyframes. For example, machine vision, computer vision and/or computer image recognition techniques may be used to identify and locate 3D objects. Once the 3D object has been identified, the 3D object can be positioned using a coordinate system (eg, 2D Cartesian, 3D Cartesian, polar, etc.) as an attribute of the points associated with the mesh of the identified 3D object.

In an example embodiment, a computer image recognition technique based on a trained (machine learning) convolutional neural network using a number of known images may be used to identify and locate 3D objects. For example, a block, blocks and/or patches are selected and/or identified from the selected frame. The trained convolutional neural network may operate on the selected block, multiple blocks, and/or patches. Results can be tested (eg, error testing, loss testing, divergence testing, etc.). If the test yields a value below the threshold (or alternatively, depending on the type of test, above the threshold), the selected block, blocks and/or patches may be identified as a 3D object.

In an example embodiment, a frame of the video may include a tag indicating that a previously identified 3D object of interest is included in the frame. Labels may include the identity and location of the 3D object. For example, a computer-generated image (CGI) tool (eg, a computer-animated film) can be used to generate the video. Computer-generated characters can be identified and tagged in each frame. Additionally, a model for each identified 3D object of interest (eg, identified character) may be stored as stored 3D object 135 .

In an example embodiment, the stored 3D objects 135 and 3D objects may be defined by a triangular mesh. A triangular mesh can be a collection of points connected by triangular faces. Each point can store various properties. For example, attributes can include the location, color, texture coordinates, etc. of each point. The attributes may include and/or indicate (eg, a plurality of attributes may indicate) the orientation of the corresponding 3D object and/or the position of the corresponding 3D object in a frame (eg, frame 110 ) and/or image of the video.

Thus, in an example real-time manner, the mesh properties of the 3D object may be sufficient to identify and locate the 3D object. A model including mesh properties for a plurality of 3D objects of interest may be stored as stored 3D objects 135 . The model can be normalized. For example, a model for a man, woman, teen, child, or more generally a human being or a part of a human being (eg, body, head, hand, etc.) may be stored as 3D object 135 . For example, a model for a dog, cat, deer, or more generally a quadruped or part of a quadruped (eg, body, head, legs, etc.) may be stored as the stored 3D object 135 . The properties of the model can then be used to search the frame for 3D objects with similar properties.

The 3D object located in the frame 110 (hereinafter referred to as the 3D object) may be matched with one of the stored 3D objects 135 . In an example embodiment, the identities of the 3D objects generated by computer image recognition techniques may be used to search the stored 3D objects 135 . In an example embodiment, the label of the 3D object found in the frame may be matched to the label of one of the stored 3D objects 135 . In an example embodiment, a model with one of the stored 3D objects 135 having similar properties to the 3D object may be identified as a match.

The matched 3D objects in the stored 3D objects 135 (hereinafter referred to as stored 3D objects) may then be used in the color prediction scheme for the frame 110 . For example, a grid corresponding to a stored 3D object may be translated and oriented to align with the orientation of the 3D object. The points corresponding to the translated and oriented stored 3D object may then be matched to points of the corresponding (eg, identical) 3D object located in a nearby (temporary), previously encoded keyframe. The prediction module 115 may then use the matching points of the corresponding 3D object to select or predict pixels/blocks/patches in the keyframe for use in computing the residual of the 3D object in frame 110 (eg, relative to the keyframe's color shift).

Additionally, prediction module 115 may generate metadata 130 associated with frame 110 . The metadata 130 may include data associated with at least one 3D object located in the frame 110 that has been predicted using one of the stored 3D objects 135 . Metadata 130 may include attributes (eg, grid point attributes) associated with the positioning and/or orientation of 3D objects in frame 110 . The metadata 130 is stored with respect to (corresponds to) one of the compressed frames 140 of the video data 5 .

The encoding module 120 may be configured to perform a series of encoding processes on the residual. For example, the data corresponding to the residuals may be transformed, quantized, and entropy encoded.

Transform residuals may include transforming data (eg, pixel values) from the spatial domain into transform coefficients in the transform domain. The transform coefficients may correspond to a two-dimensional matrix of coefficients that are initially the same size as the original block and/or patch in frame 110 . In other words, there may be as many transform coefficients in the original block and/or patch in frame 110 as there are data points (eg, pixels). However, due to the transform, a portion of the transform coefficients may have a value equal to zero. Typically, transforms include Karhunen-Loève transform (KLT), discrete cosine transform (DCT), singular value decomposition transform (SVD), and asymmetric discrete sine transform (ADST).

Vector coordinates (eg, representing transform coefficients) are usually given by float or double precision. However, such representations are often more precise than actually needed. For example, the video data 5 may originate from a video capture device (eg, camera, scanner, computer-generated image program) with some measurement error. Therefore, a relatively large number of lower bits may be noise. Quantize converts the given floating-point number to a b-bit long integer representation. Therefore, quantization can reduce the data in each transform coefficient. Quantization may involve mapping a relatively large range of values to a relatively small range of values, thereby reducing the amount of data required to represent the quantized transform coefficients. Quantization can convert transform coefficients into discrete quantum values, which are called quantized transform coefficients or quantization levels. For example, quantization may add zeros to data associated with transform coefficients. For example, a coding standard may define 128 quantization levels in a scalar quantization process.

The expected value of the information contained in the quantized transform coefficients (eg, a dataset of discrete quantum values). The higher the number of unique values a dataset contains, the higher the entropy. Repeated values reduce entropy and get better compression. Therefore, the quantized transform coefficients can be entropy encoded. Entropy coding is performed using one of a set of techniques for compressing datasets based on data entropy. For example, entropy coding techniques may include Huffman coding, arithmetic coding, or the use of asymmetric number systems.

The entropy coded coefficients (for all pixels/blocks/patches in frame 110) are then output as compressed frame 140 along with the information needed to decode frame 110 (eg prediction type used, motion vector and quantization value) One of the compressed frames of the video data 5 and stored with and/or in association with the other compressed frames of the video data 5 .

Typically, encoder 105 and video storage 125 are elements in the cloud or the World Wide Web. For example, the encoder 105 may be one of a number of encoders implemented as computer hardware and/or computer software implemented in a cloud computing device that is configured to use data provided by one or more standards (eg, H.264, H.265, HEVC, VP8, VP9, VP10, AV1, etc.) to compress video data (eg, video data 5). For example, video storage 125 may be implemented as at least one type of non-volatile memory, non-transitory computer-readable medium, etc. located in a cloud computing device (eg, a streaming server). In at least one embodiment, the encoder 105 compresses the video data 5 into video, and stores the compressed video data for future (eg, later in time) playback when streaming from the cloud computing device to the client device video.

In this embodiment, the client device includes a decoder (eg, decoder 145). In some implementations, metadata 130 and/or stored 3D objects 135 may be communicated to the client device. Metadata 130 and/or stored 3D objects 135 may be communicated to the client device on demand and/or as an initialization process. Metadata 130 and/or stored 3D objects 135 may be compressed when stored in video storage 125 and/or on demand before being communicated to the client device. In an example embodiment, the stored 3D objects 135 may be compressed using machine-trained generative modeling techniques to generate a reduced number of variables associated with the properties and positions of the meshes for the stored 3D objects (herein referred to as latent representation or reduced latent representation) (as described in more detail below).

Accordingly, as shown in FIG. 1 , the decoder 145 includes a frame reconstruction module 150 , a prediction module 155 and a decoding module 160 . The decoding module 160 may be configured to perform the inverse operation of the encoding module 120 . Decoding module 160 may receive compressed frames 140 (eg, representing a 3D movie selected for streaming or download by a user of a client device including decoder 145). The compressed frames 140 may be received one at a time, multiple frames or blocks of frames at a time, or as a complete 3D movie. The decoder 145 may select one of the compressed frames 140, and the decoding module 160 may entropy decode, dequantize, and inverse transform the selected compressed frame.

In an example embodiment, decoding module 160 uses the selected data elements within the compressed frame and performs entropy decoding (eg, inverse transform using Huffman coding, arithmetic coding, or Asymmetric Numerical System coding) on the data. Decompress to produce a set of quantized transform coefficients. The quantized transform coefficients are dequantized, and the dequantized transform coefficients are inversely transformed (eg, using an inverse transform of KLT, DCT, SVD, or ADST) to generate residuals that may be the same (or similar to those generated by prediction module 115 ) the same) derivative residuals.

The prediction module 155 may be configured to determine whether the selected compressed frame includes one of the stored 3D objects 135 . In an example embodiment, metadata 130 may be used to determine whether the selected compressed frame includes one of the stored 3D objects. For example, prediction module 155 may query metadata 130 in the selected compressed frame, and if metadata is returned, determine that the selected compressed frame includes one of the stored 3D objects 135 .

The prediction module 155 may be configured to identify, for the selected compressed frame, the 3D object, the location (eg, positioning) of the 3D object, and the orientation of the 3D object based on the returned metadata. The prediction module 155 may be configured to select a stored 3D object (hereinafter referred to as a stored 3D object) from the stored 3D objects 135 and use the stored 3D object to decompress the selected frame.

In an example embodiment, the grid corresponding to the stored 3D object may be translated and oriented based on the location (eg, positioning) of the identified 3D object and the orientation of the identified 3D object. The points corresponding to the translated and oriented stored 3D object can then be matched with points of the corresponding (eg, the same) 3D object located in a nearby (temporary), previously decoded keyframe. The prediction module 155 may then use the matching points of the corresponding 3D object to select or predict pixels/blocks/patches in the keyframe for regeneration based on the residuals at the identified locations in the selected frame (eg, Compute) the translated and oriented color values and/or color properties of the stored 3D object.

The prediction module 155 may be further configured to regenerate (eg, calculate) color values and/or for the remainder of the selected frame based on the residual or remainder of the selected frame and the corresponding pixels/blocks/patches of the keyframe. or color properties. The frame reconstruction module 150 may be configured to be based on reproduced color values and/or color attributes for the translated and oriented stored 3D objects and reproduced color values and/or color attributes for the remainder of the selected frame to reconstruct the selected frame. In an example embodiment, the frame reconstruction module 150 may be configured to stitch the reproduced color values and/or color attributes for the translated and oriented stored 3D objects to the values and/or color properties for the selected 3D objects based on the positions of the identified 3D objects. The regenerated color value and/or color attribute for the remainder of the frame.

Example embodiments may include identifying two or more 3D objects, regenerating color values and/or color attributes of each of the two or more 3D objects, and using each of the two or more 3D objects. reconstructed selected frames. The frame reconstruction module 150 may be configured to reproduce the video data 5 based on the plurality of reconstructed frames. Video data 5 (eg, texture data and color data) may be rendered and color corrected for display on the client device's display.

Compressing video (or video frames) using an encoder (eg, encoder 105 ) configured to use a color prediction scheme based on the use of 3D objects as geometric proxies may not result in the highest performance on the video, video frame, or multiple video frames. compression ratio (eg, minimum data size). Thus, example embodiments may include compressing a video, a frame of video, or multiple frames of video using two or more encoders, each encoder capable of compressing video data using an identified color prediction scheme.

2 illustrates a block diagram of a signal flow for storing compressed video (and/or compressed frames of video) according to an example embodiment. As shown in Figure 2, the first encoder 205 is an encoder (eg, encoder 105) that uses a color prediction scheme based on using 3D objects as geometric proxies. Furthermore, at least one second encoder 210-1 is an encoder using color prediction scheme 1, at least one second encoder 210-2 is an encoder using color prediction scheme 2, and at least one second encoder 210-i is the encoder using color prediction scheme i. Each of the first encoder 205 and the at least one second encoder 210-1, 210-2, . . . , 210-i may be configured to generate and communicate n frames having x bits in total to the Size comparator 215 . Color prediction schemes 1, 2, ..., i can be default prediction schemes for coding standards, configurable prediction schemes for coding standards, custom prediction schemes based on temporal displacement, alternative prediction schemes based on using 3D objects as geometric proxies Wait.

Compression size comparator 215 may be configured to select one of the outputs of first encoder 205, at least one second encoder 210-1, 210-2, . . . or 210-i for saving in video storage 125 ( For example, for later streaming to client devices). In an example embodiment, the encoder output may be selected based on compression efficiency. For example, compressed video (and/or compressed frames of video) with the fewest number of bits (eg, minimum value of x) may be saved.

In another example embodiment, the color prediction scheme may be the preferred prediction scheme. In this embodiment, unless some conditions exist, the compressed video (and/or the compressed frames of the video) of the preferred color prediction scheme is saved. For example, the condition may be based on compression efficiency. Unless the output of at least one of the second encoders 210-1, 210-2, . . . , 210-i is at least a threshold percentage more efficient than the output of the first encoder 205 (eg 20%, etc.), otherwise the output of the first encoder 205 can be selected. If more than one of the at least one second encoder 210-1, 210-2, . . . , 210-i is at least a threshold percent more efficient than the output of the first encoder 205 (eg, 10%, 15 %, 20%, etc.), the highest efficiency (eg, minimum value of x) of at least one second encoder 210-1, 210-2, . . . , 210-i can be saved.

In an example embodiment, the signal flow illustrated in FIG. 2 for storing compressed video (and/or compressed frames of video) may be performed on frame-by-frame video and/or on frame sets. For example, each frame of the video may be encoded as described above. The compression size comparator 215 may then select the encoder output based on efficiency or based on a conditional preferred color prediction scheme. In an example embodiment, for example, keyframes may be used to select between frame-by-frame or frameset compression.

In an example embodiment, one of the encoder outputs may always be saved. For example, the default color prediction scheme can be saved to ensure backward compatibility. In Figure 2, this embodiment is illustrated as a dashed line from at least one second encoder 210-i.

The signal flow illustrated in FIG. 2 for storing compressed video (and/or compressed frames of video) may be performed multiple times using different coding standards. For example, the signal flow illustrated in FIG. 2 may be performed using two or more of H.264, H.265, HEVC, VP8, VP9, VP10, AV1, etc. coding standards. Thus, a color prediction scheme (eg, encoder 105) based on the use of 3D objects as geometric proxies can be implemented in two or more coding standards and on the same video (eg, video data 5). Thus, video store 125 may store multiple instances of video (and/or compressed frames of video), each instance having been compressed using a different standard and/or a different configuration of the standard. As a result, the streaming server may serve the video based on the capabilities of the requesting client device and/or the network over which the video is to be communicated.

3A illustrates a block diagram of an encoder prediction module according to an example embodiment. As shown in Figure 3A, prediction module 115 includes frame 110, 3D object locator module 305, 3D object matching module 310, stored 3D objects 135, stored 3D object translation module 315, keyframes 320, block matching module 325 and a residual generation module 330.

In an example embodiment, the stored 3D objects 135 and the 3D objects in frame 110 may be defined by a triangular mesh. A triangular mesh can be a collection of points connected by triangular faces. Each point can store various properties. For example, attributes can include the location, color, texture coordinates, etc. of each point. The attributes may include and/or indicate (eg, a plurality of attributes may indicate) the orientation of the corresponding 3D object and/or the position of the corresponding 3D object in the frame 110 and/or the image of the video.

The 3D object locator module 305 may be configured to identify and locate 3D objects (hereinafter referred to as 3D objects) in frames 110 and keyframes 320 . For example, machine vision techniques, computer vision techniques, computer image recognition techniques, and the like may be used to identify and locate 3D objects in frame 110 and key frame 320 . Once the 3D object has been identified, the 3D object can be positioned using a coordinate system (eg, 2D Cartesian, 3D Cartesian, polar, etc.) as an attribute of the points associated with the mesh of the identified 3D object.

In an example embodiment, a computer image recognition technique based on a trained (machine learning) convolutional neural network using a number of known images may be used to identify 3D objects. For example, a block, blocks and/or patches are selected and/or identified from the selected frame. The trained convolutional neural network may operate on the selected block, multiple blocks, and/or patches. Results can be tested (eg, error testing, loss testing, scatter testing, etc.). If the test yields a value below the threshold (or alternatively, a value above the threshold depending on the type of test) the threshold, the selected block, blocks and/or patches may be identified as a 3D object.

In an example embodiment, a video frame that includes a previously identified 3D object of interest may also include (eg, in a header, frame metadata, etc.) a tag indicating that the previously identified 3D object of interest is included in the frame. Labels may include the identity and location of the 3D object (eg, coordinate attributes of points associated with the grid). For example, a computer-generated image (CGI) tool (eg, a computer-animated film) can be used to generate the video. Computer-generated characters can be identified and tagged in each frame. Additionally, a model (eg, defined by a triangular mesh) for each identified 3D object of interest (eg, a character in an animated movie) may be stored as stored 3D object 135 .

In an example embodiment, the mesh properties of the 3D object may be sufficient to identify and locate the 3D object. A model including mesh properties for a number of generic 3D objects of interest may be stored in (or associated with) a 3D object locator module. The model can be normalized. For example, models for men, women, adolescents, children, or more generally humans or parts of humans (eg, bodies, heads, hands, etc.) may be stored. For example, models for dogs, cats, deer, or more generally quadrupeds or parts of quadrupeds (eg, body, head, legs, etc.) can be stored. The properties of the model can then be used to search for 3D objects with similar properties in that frame.

The 3D object matching module 310 may be configured to match a 3D object (hereinafter, referred to as a 3D object) located in the frame 110 with one of the stored 3D objects 135 . In an example embodiment, computer image recognition techniques are used by 3D object locator module 305 to identify 3D objects. Identifying the 3D object may also include assigning a unique ID from the ID database to the 3D object. The unique ID can be used to search the stored 3D objects 135 for 3D objects. If the unique ID is found in the stored 3D objects 135, then the corresponding one of the stored 3D objects matches the 3D object.

In an example embodiment, the tags of the 3D objects found in the frame may be unique to the 3D objects (eg, have a one-to-one relationship between tags and 3D objects). Tags can be used to search for 3D objects in stored 3D objects 135 . If a tag is found in the stored 3D objects 135, then the corresponding one of the stored 3D objects is matched with the 3D object.

In an example embodiment, a model in which one of the stored 3D objects 135 has similar properties to the 3D object may be identified as a partial match. The stored 3D objects 135 may then be filtered based on the partial matches. The stored 3D object 135 may then be searched for the 3D object using one or more properties or a combination of properties of the 3D object. If one or more attributes or combination of attributes are found in the stored 3D objects 135, then the corresponding one of the stored 3D objects is a match for that 3D object. One or more attributes, or combination of attributes, may uniquely identify the 3D object relative to the stored 3D object 135 with a relatively high degree of certainty. For example, the shape or relative position of body parts (e.g., face shape, nose shape, eyes, relative position of nose and mouth, etc.), the color of a cartoon character's hair or skin, the shape of objects worn by the character (e.g., jewelry), and Relative positioning, vehicle type (eg, car, tractor, etc.), etc., may uniquely identify the 3D object relative to the stored 3D object 135 (eg, if the stored 3D object 135 corresponds to the video's set of 3D objects of interest).

If the 3D object matching module 310 does not find a match for the 3D object among the stored 3D objects, the prediction module 115 may use standard prediction techniques defined by coding standards. In other words, a color prediction scheme based on using a 3D object as a geometric proxy may not be used for the 3D object.

If the 3D object matching module 310 does not find a match for the 3D object in the stored 3D objects 135 , the 3D object matching module 310 may be configured to add the 3D object to the stored 3D objects 135 . For example, 3D object matching module 310 may be configured to assign a unique ID or unique label to a 3D object and define a model (eg, a triangular mesh including points and corresponding attributes) for the 3D object. The model can then be stored as one of the stored 3D objects 135 and identified by a unique ID or unique tag.

In an example embodiment, the stored 3D objects 135 may have a predetermined grid representation and/or data structure. Thus, the model may have a predefined size (eg, number of points, number of faces, number of vertices, etc.), number of attributes, types of attributes, etc. that are predefined based on the stored design of 3D object 135 . Additionally, matching the 3D object to 3D objects in the stored 3D objects 135 may include redefining by the 3D object matching module 310 based on the design of the stored 3D objects 135 before searching for a match in the stored 3D objects 135 3D objects.

The stored 3D object translation module 315 may be configured to translate (or transform) a matched one of the stored 3D objects 135 (hereinafter, referred to as the stored 3D object). For example, a grid corresponding to a stored 3D object may be translated and oriented to align with the orientation of the 3D object. Translation and orientation together may be referred to as transformations. Stored 3D object translation module 315 may be configured to transform a matched one of stored 3D objects 135 associated with frame 110 and keyframe 320 . Accordingly, the stored 3D object translation module 315 may be configured to generate and transform the first 3D object proxy based on the stored 3D object associated with the frame 110 for the 3D object. Additionally, the stored 3D object translation module 315 may be configured to generate and transform a second 3D object proxy based on the stored 3D object associated with the keyframe 320 for the 3D object.

Transforming 3D proxies can include generating latent representations using autoencoders (described below) by articulating mesh points (like skeletons, for example) and locating values associated with latent representations, sending for Bezier, or NURBS surfaces or subdivisions Control points and values for the surface. Each grid may have predetermined connectivity. The predetermined connectivity may allow direct correspondence between points of two meshes in different poses. Therefore, we can use the same texture parameterization for both meshes.

The stored 3D object translation module 315 may be configured to generate the metadata 20 . The metadata 20 includes information identifying the stored 3D object and information related to the translation and orientation of the stored 3D object. Information about the stored translation and orientation of the 3D object can be used to perform the same translation and orientation of the stored 3D object in the future (eg, by decoder 145). In an example embodiment, metadata is generated and stored for both the first 3D object proxy and the second 3D object proxy.

Block matching module 325 may be configured to match points of the translated and oriented stored 3D object of frame 110 corresponding to the translated and oriented stored 3D object of key frame 320 . In an example embodiment, the block matching module 325 may be configured to wrap a mesh (eg, a 3D mesh) representing the translated and oriented stored 3D object with color and/or texture attributes. For example, block matching module 325 may be configured to map color attributes from the 3D object identified in frame 110 to the transformed first 3D object proxy. Additionally, block matching module 325 may be configured to map color attributes from the 3D object identified in keyframe 320 to the transformed second 3D object proxy. Mapping the color properties from the 3D object identified in frame 110 to the transformed first 3D object proxy may include converting the 3D object from 3D space (eg, XYZ space) to 2D space (eg, UV space) and/or converting the 3D object from 3D space (eg, XYZ space) to 2D space (eg, UV space) The 3D objects identified in 110 are converted from 2D space (eg, UV space) to 3D space (eg, XYZ space).

Each grid may have predetermined connectivity. The predetermined connectivity may allow direct correspondence between points of two meshes in different poses. Therefore, we can use the same texture parameterization for both meshes. Thus, mapping color properties may include identifying pixels in a frame (eg, frame 110 and/or keyframe 320 ) based on pixel coordinates in the frame, and then mapping pixels in a grid representation of the transformed 3D object proxy with the same coordinates The color property of the point is set to the same color value of the identified pixel. In example embodiments, more than one frame may be used to generate pixel attributes. Additionally, pixel attributes from regenerated frames (eg, from regeneration loops in the encoder) can be used to generate pixel attributes.

In an example embodiment, pixel attributes may be blended. Blending pixel attributes (or textures) can be done using per-texel averages of corresponding texels from those textures that have information about that texel, texels from different textures may have confidence values (e.g. the temporal distance between the current frame and the frame from which this texture was generated), also predict unobserved texels in a particular texture (reduce their confidence), for the current frame, use the new pose of the mesh and its textures to get predictions for (part of) the current frame, etc.

In some cases, frame 110 and/or keyframe 320 may not have pixels corresponding to points on the meshed representation of the transformed 3D object proxy. Thus, the mesh representation of the transformed 3D object proxy may retain the default color properties of the points. Therefore, any residual color calculations can be based on the default color properties.

The block matching module 325 may then use the matching points of the corresponding 3D objects to select or predict pixels/blocks/patches in the key frame 320 for use in generating the residuals 15 for the 3D objects in the frame 110 . In an example embodiment, the stored 3D objects and/or 3D object proxies may be encoded using an auto-encoder (described in more detail below) prior to translation and matching. Encoding the stored 3D object or 3D object proxy converts the 3D object, the stored 3D object and/or the 3D object proxy into a latent representation. The latent representation includes fewer values (eg, points) than the mesh representing the 3D object, the stored 3D object, and/or the 3D object proxy. Thus, translating a 3D object, stored 3D object and/or 3D object proxy into a latent representation involves manipulating fewer points. Furthermore, mapping color properties includes mapping fewer points when encoding 3D objects, stored 3D objects, and/or 3D object proxies as latent representations.

Residual generation module 330 may be configured to generate (or calculate) residuals 15 (eg, color shifts relative to keyframes 320 ) for the 3D objects in frame 110 . For example, the residual generation module 330 may subtract the value of each point in the triangular mesh for the 3D object from the pixel attribute value for each matching point of the predicted pixel/block/patch in the keyframe 320 pixel attribute values to generate residuals 15. In an example embodiment, the first 3D object proxy may be subtracted from the color attribute of the corresponding point in the grid representation of the second 3D object proxy (eg, having the same point identification or being in the same position in the grid sequence) The color property of the points in the grid representation.

In an example embodiment, the encoder 105 includes a reconstruction path (not shown). The reconstruction path includes several components or software implementations that decode the frame together using at least one inverse of the encoding process described above. For example, the reconstruction path may include at least an inverse quantization process and an inverse transform process. When encoding the next consecutive frame, the reconstructed frame generated in the reconstruction path may be used in place of the key frame 320 . There is some loss in encoding raw frames. Therefore, the reconstructed frame generated in the reconstruction path includes some artifacts (eg, errors) compared to the original frame. These artifacts may or may not be corrected in prediction module 115 . In some embodiments, artifacts generated using reconstructed frames may be corrected by the color correction module 635 described below.

3B shows a block diagram of another encoder prediction module according to an example embodiment. As shown in Figure 3B, prediction module 115 includes frame 110, 3D object locator module 305, 3D object matching module 310, stored 3D objects 135, stored 3D object translation module 315, block matching module 325, and residual generation Module 330.

The implementation of the prediction module 115 shown in FIG. 3B is basically the same as the prediction module 115 shown in FIG. 3A . However, the prediction module 115 shown in Figure 3B does not generate the residual 15 based on the keyframes. In contrast, residual generation module 330 of prediction module 115 shown in FIG. 3B uses 3D objects and the translated and oriented stored 3D objects to generate residuals 15 .

In an example embodiment, the translated and oriented stored 3D object has a direct point-to-point relationship with the 3D object. In other words, a point in the triangle mesh that defines the translated and oriented stored 3D object (eg, the position attributes are x ₁ , y ₁ , z ₁ ) have a corresponding triangle mesh in the triangle mesh that defines the 3D object Points (for example, the location attributes are x ₁ , y ₁ , z ₁ ). Thus, the color property can be used to determine the relative color displacement between the 3D object and the stored 3D object. Accordingly, residual generation module 330 may be configured to generate (or calculate) residuals 15 (eg, color shifts) for 3D objects in frame 110 based on the stored objects. For example, residual generation module 330 may subtract the pixel for each point in the triangular mesh of the 3D object from the translated and oriented stored pixel attribute value of the corresponding point in the triangular mesh of the 3D object attribute values to generate residuals 15.

In an example embodiment, the residuals may be generated using a translated 3D object proxy (eg, the first 3D object proxy described above) based on and frame 110 and a stored 3D object with default color properties The associated stored 3D object for the 3D object that has been mapped with the colors of frame 110 . Thus, the grid representation of the first 3D object proxy can be subtracted from the color properties of the corresponding points in the stored grid representation of the 3D object (eg, having the same point identification or being in the same position in the grid sequence) The color property of the point in .

4A shows a block diagram of a decoder prediction module according to an example embodiment. As shown in FIG. 4A, prediction module 155 includes frame 405, metadata module 410, 3D object matching module 415, stored 3D objects 135, stored 3D object translation module 420, keyframes 320, block matching module 325, and pixels Regeneration module 425.

Frame 405 is a compressed frame selected from compressed frames 140 as a frame to be decompressed. Keyframe 320 is the decompressed keyframe associated with frame 405 (nearby, earlier in time, and/or later in time). Frame 405 has been entropy decoded, inverse quantized, and inverse transformed by decoding module 160 . Accordingly, frame 405 may include a derivative residual, which may be the same (or approximately the same) as the residual generated by prediction module 115 .

Metadata module 410 may be configured to determine whether frame 405 includes metadata and/or has associated metadata (eg, metadata 130). Metadata module 410 may be configured to read metadata from headers associated with frame 405 . The metadata may include metadata 20 . The metadata may include data associated with at least one 3D object located in frame 405 that has been predicted using one of the stored 3D objects 135 . The metadata may include attributes (eg, grid point attributes) associated with the positioning and/or orientation of the 3D object in frame 405 . The metadata may include information identifying the stored 3D object and information related to the translation and orientation of the stored 3D object. Information about the stored translation and orientation of the 3D object can be used to perform the same translation and orientation of the stored 3D object as performed by the stored 3D object translation module 315 .

The 3D object matching module 415 may be configured to match a 3D object (hereinafter, referred to as a 3D object) located in the frame 405 with one of the stored 3D objects 135 . In an example embodiment, the metadata module 410 outputs a unique ID or tag that can be used to search the stored 3D objects 135 for 3D objects. If a unique ID or tag is found in the stored 3D objects 135, a corresponding one of the stored 3D objects 135 (hereinafter, referred to as stored 3D objects) is a match of the 3D objects.

The stored 3D object translation module 420 may be configured to translate the stored 3D objects. For example, prior to compressing frame 405, when orienting a 3D object in frame 405, a grid corresponding to the stored 3D object may be translated and oriented to align with the orientation of the 3D object. In other words, when a 3D object is oriented in frame 110, the grid corresponding to the stored 3D object may be translated and oriented to align with the orientation of the 3D object. The stored 3D object translation module 420 may be configured to translate the stored 3D object based on information about the translation and orientation of the stored 3D object included in the metadata.

Stored 3D object translation module 420 may be configured to translate a matched one of stored 3D objects 135 associated with frame 405 and keyframe 320 . Accordingly, the stored 3D object translation module 315 may be configured to generate a first 3D object proxy based on the stored 3D object for the 3D object associated with the frame 405 . Additionally, the stored 3D object translation module 315 may be configured to generate a second 3D object proxy based on the stored 3D object for the 3D object associated with the keyframe 320 . In an example embodiment, the second 3D object proxy may be generated once per keyframe 320 and then used for each associated frame 405 .

Block matching module 325 may be configured to match points of the translated and oriented stored 3D object of frame 405 corresponding to the translated and oriented stored 3D object of keyframe 320 . In an example embodiment, the block matching module 325 may be configured to map color attributes from the 3D object identified in the frame 110 to the transformed first 3D object proxy. Additionally, block matching module 325 may be configured to map color attributes from the 3D object identified in keyframe 320 to the transformed second 3D object proxy. Mapping the color properties from the 3D object identified in frame 110 to the transformed first 3D object proxy may include converting the 3D object from 3D space (eg, XYZ space) to 2D space (eg, UV space) and/or converting the 3D object from 3D space (eg, XYZ space) to 2D space (eg, UV space) at frame 110 The 3D objects identified in are converted from 2D space (eg, UV space) to 3D space (eg, XYZ space).

In an example embodiment, mapping color attributes may include identifying pixels in a frame (eg, frame 405 and/or keyframe 320 ) based on pixel coordinates in the frame, and then mapping pixels in a mesh representation of the transformed 3D object proxy The color properties of points with the same coordinates are set to the same color value of the identified pixel. In some cases, frame 405 and/or keyframe 320 may not have pixels corresponding to points on the meshed representation of the transformed 3D object proxy. Thus, the mesh representation of the transformed 3D object proxy may retain the default color properties of the points. Therefore, any residual color calculation can be based on the default color.

The block matching module 325 may then use the matching points of the corresponding 3D object to select or predict the pixels/blocks/patches in the key frame 320 for regenerating the pixels of the 3D object in the frame 405 . Pixel regeneration module 425 may be configured to generate (or calculate) pixel values for 3D object 25 in frame 405 . For example, the pixel regeneration module 425 may, based on the residuals at the identified locations in the selected frame, generate a The pixel value is generated by adding the pixel attribute value for each point in the triangular mesh of the 3D object to the stored color value and/or color attribute for the translated and oriented 3D object.

In example embodiments, the 3D objects, stored 3D objects and/or 3D object proxies have been encoded using an auto-encoder. Thus, the 3D objects, stored 3D objects and/or 3D object proxies may be decoded using an auto-encoder (described in more detail below) prior to translation and matching. Decoding the stored 3D object or 3D object proxy converts the 3D object, the underlying representation of the stored 3D object and/or the 3D object proxy into a regenerated mesh representation.

The pixel regeneration module 425 may be further configured to regenerate (e.g., calculate) the color values and/or the remaining portion of the selected frame based on the residual or remainder of the selected frame and the corresponding pixels/blocks/patches of the key frame. or color properties.

4B shows a block diagram of another decoder prediction module according to an example embodiment. As shown in Figure 4B, prediction module 155 includes frame 405, metadata module 410, 3D object matching module 415, stored 3D objects 135, stored 3D object translation module 420, keyframes 320, block matching module 325, and pixels Regeneration module 425.

The implementation of the prediction module 155 shown in FIG. 4B is basically the same as the prediction module 155 shown in FIG. 4A . However, the prediction module 155 shown in FIG. 4B does not reproduce the color values of the 3D object based on the key frames. In contrast, the pixel regeneration module 425 of the prediction module 155 shown in FIG. 4B uses the 3D object and the translated and oriented stored 3D object to regenerate the color values of the 3D object.

In an example embodiment, the translated and oriented stored 3D object has a direct point-to-point relationship with the 3D object. In other words, points in the triangle mesh that define the translated and oriented stored 3D object (eg, position attributes are x ₁ , y ₁ , z ₁ ) have corresponding points in the triangle mesh that defines the 3D object (For example, the position attributes are x ₁ , y ₁ , z ₁ ). Thus, the translated and oriented stored color properties of the 3D object can be used to determine the color value of the 3D object.

Accordingly, pixel regeneration module 425 may be configured to generate (or calculate) color values for 3D objects in frame 405 based on the stored objects. For example, the pixel regeneration module 425 may compare the pixel value of each point in the triangular mesh of the 3D object read from frame 405 with the corresponding point in the triangular mesh of the stored 3D object for the translated and oriented The color attribute values of the s are added to reproduce the color value. The translated and oriented stored 3D object with the calculated color attribute values is then output as 3D object 25 as a regenerated 3D object. The remainder of frame 30 is reproduced as described above with reference to Figure 4A.

As described above, each stored 3D object 135 may be defined by a triangular mesh. A triangular mesh can be a collection of points connected by triangular faces. Each point can store various properties. For example, attributes can include the location, color, texture coordinates, etc. of each point. Although the triangular mesh structure is referenced above, other polygons may also be used to define the stored 3D objects 135 . Additionally, the properties of each point may include other properties that may or may not be useful in the context of the present disclosure.

In an example embodiment, the stored 3D objects 135 may include 3D objects of interest for CGI movies. Thus, stored 3D objects 135 may include 3D model or mesh data and corresponding attribute values for a number of 3D characters. Typically, 3D characters may include CGI actors (eg, main characters, supporting characters, and other characters), CGI pets, CGI creatures, CGI monsters, and the like. As mentioned above, each of these 3D characters may be a 3D object of interest stored as a 3D model or mesh data stored in the stored 3D object 135 . Each 3D character can include a unique ID or tag.

In some embodiments, a portion of the 3D object of interest may be stored as a 3D model or mesh data stored in the stored 3D object 135 . Each of the parts of the 3D character may include a unique ID or tag. For example, at least one 3D character (eg, main and supporting characters) may have an associated 3D model or mesh data representing at least one 3D character's head or face, and another associated 3D model representing at least one 3D character's body or mesh data (note that the head, face or body can also be divided into smaller parts). By dividing the character into multiple parts, there can be more dynamic parts of the character (eg, head, arms, and/or legs) than relatively less dynamic character parts (eg, torso and/or shoulders) Stores a relatively high level of detail.

Additionally, mesh data can be configured to fit standard 3D models. For example, the first cubic model may be a standard size designed to fit a head model, while the second cubic model may be a standard size designed to fit a hand model. 3D models of other standard shapes can include spheres, right-angled prisms, rectangular prisms, pyramids, cylinders, etc. For example, an arm or leg can use a right-angled prism 3D model or a cylindrical 3D model. Standard 3D models may also place limits (maximum and/or minimum) on a number of points (eg, vertices) used to define the 3D object.

Dividing the parts of the character and storing each part as a unique 3D object of interest in the stored 3D objects 135 may allow the size of the stored 3D objects 135 to grow exponentially. However, 3D model or mesh data representing part of a character can be used for many characters. For example, a torso model representing the torso of character A can also be used for character B. The stored 3D objects 135 may also include metadata identifying 3D models or mesh data representing standard parts of the character and some deformation information corresponding to the standard parts for the character. Therefore, 3D models or mesh data represent standard parts. For example, one 3D model representing the torso may be used as the 3D model representing the torso of character A as a tall, thin man, and as the 3D model representing the torso of character B as a squat man.

As the number of stored 3D objects 135 increases (eg, for a full-length CGI animated movie with many characters using a large number of 3D standard models), the amount of resources (eg, memory) required to store the stored 3D objects 135 also increases increased accordingly. For example, a streaming server can store 100,000, 1 million, and millions of videos. As demand for video including 3D video increases (eg, in virtual reality, augmented reality, 3D CGI movies, etc.), the percentage of video stored at the streaming server (usually stored as left-eye and right-eye 2D video) It will definitely increase as well. Therefore, when implementing the techniques described herein, for use with the growing number of videos. The number of stored 3D objects 135 must increase, so the amount of resources (eg, memory) required to store the stored 3D objects 135 will also increase. Furthermore, transferring the stored 3D objects 135 from the streaming server to the client device may require significant bandwidth during a streaming activity. Therefore, for streaming operations, it may become desirable to efficiently encode and decode stored 3D objects 135 .

5A shows a block diagram of a signal flow for encoding a 3D object, according to an example embodiment. As shown in FIG. 5A, encoding a 3D object may include at least the stored 3D object 135, a neural network encoder 505, and a latent representation 510 of the 3D object.

As described above, each stored 3D object 135 may be defined by a triangular mesh. A triangular mesh can be a collection of points or vertices connected by triangular faces. Each point can store various properties. In example embodiments, the number of points or vertices may be limited to a certain value.

The neural network encoder 505 may use generative modeling techniques to compress 3D objects. Example implementations of generative modeling techniques may include variational and/or convolutional autoencoders (VAEs), generative adversarial networks (GANs), and/or combined VAE-GANs. While these generative modeling techniques are mentioned and/or discussed, example embodiments are not so limited. Furthermore, although a neural network-type encoder is discussed as an implementation for compressing 3D objects, other implementations of encoding 3D objects, including triangular meshes, are also within the scope of the present disclosure.

Neural network encoder 505 can compress 3D objects defined by meshes with a fixed number of vertices and fixed connectivity into a relatively small number of variables sometimes referred to as latent spaces. For example, a VAE can be configured to learn a compact latent space for 3D shapes (eg, 3D shapes on which the VAE has been trained). Both the neural network of neural network encoder 505 and the neural network of neural network decoder 515 may be machine-trained neural networks and/or machine-trained convolutional neural networks.

The neural network of neural network encoder 505 may include coefficients associated with one or more convolutions and/or filters that are applied to the trellis to encode the trellis. Each convolution can have C filters, a K×K mask (K for convolution) and a stride factor. The coefficients may correspond to one or more of the C filters, KxK masks, and stride factors. For example, the KxK mask can be a 3x3 mask. The 3x3 mask contains nine (9) variables used to perform convolution on the grid. In other words, a 3x3 mask has nine (9) blocks, each block containing variables. Each variable can be one of the coefficients individually.

The neural network of the neural network encoder 505 may include layers with different numbers of neurons. The KxK spatial extent may include K columns and K (or L) rows. The KxK spatial extent can be 2x2, 3x3, 4x4, 5x5, (KxL)2x4, etc. Convolution involves centering the KxK spatial extent at a grid point, and convolving all grid points in the spatial extent, and generating the The new value of the grid point. Then based on the stride, the spatial extent is moved to the new grid point, and the convolution is repeated for the new grid point. The step size can be, for example, one (1) or two (2), where a step size of 1 moves to the next grid point, and a step size of 2 skips a grid point.

The VAE may use the position coordinates for each point of the grid (eg, for any mostly 3D shape of interest or a relatively large amount of data with a visually considerable amount of detail) as input to the neural network encoder 505, And by convolving the neural network with the position coordinates of each point of the grid, a reduced (preferably relatively small) number of variables (eg from 64 to 128 variables) is generated. The configuration of the C filters, KxK masks, and stride factor for the neural network may determine the number of variables in the latent space for the latent representation of the 3D object 510 .

5B shows a block diagram of a signal flow for decoding a 3D object, according to an example embodiment. As shown in FIG. 5B , decoding a 3D object may include at least a latent representation 510 for the 3D object, a neural network decoder 515 , and the stored 3D object 135 . The latent representation 510 of the 3D object may be generated by using the neural network encoder 505 A latent representation 510 of the 3D object generated by encoding the stored 3D object 135 .

The neural network of neural network decoder 515 may include coefficients associated with one or more convolutions and/or filters that are applied to the trellis to encode the trellis. Each convolution can have C filters, a K×K mask (K for convolution) and a stride factor. The coefficients may correspond to one or more of the C filters, KxK masks, and stride factors. For example, the KxK mask can be a 3x3 mask. The 3x3 mask contains nine (9) variables used to perform convolution on the grid. In other words, a 3x3 mask has nine (9) blocks, each block containing variables. Each variable can be one of the coefficients individually.

The neural network of the neural network decoder 515 may include layers with different numbers of neurons. The KxK spatial extent may include K columns and K (or L) rows. The KxK spatial extent can be 2x2, 3x3, 4x4, 5x5, (KxL)2x4, etc. Convolution involves centering the KxK spatial extent at a grid point, and convolving all grid points in the spatial extent, and generating the The new value of the grid point. Then based on the upsampling, the spatial extent is moved to the new grid point, and the convolution is repeated on the new grid point. The upsampling factor can be, for example, one (1) or two (2), where a step factor of 1 moves to the next grid point and a step factor of 2 skips grid points.

Convolution may include one or more zero-padded convolution operations and coefficient reorganization. In an example embodiment, zero-padding includes padding zeros between non-zero pixels, and coefficient reorganization may include convolution with a KxK mask rotated 180 degrees around the zero-padded pixels.

The VAE may use the variables in the latent space for the latent representation 510 of the 3D object as input to the neural network decoder 515 and regenerate the position coordinates for each point of the grid. The configuration of the neural network's C filters, KxK mask, and step factor can determine the number of regenerated grid points.

Accordingly, the neural network decoder 515 may be configured to use the variables for the respective one of the latent representations 510 of the 3D object to reproduce an approximation of the shape and/or model of the 3D object prior to compression of the 3D object by the neural network encoder 505 . The neural network decoder 515 may be configured to generate an approximation of the 3D model corresponding to one of the stored 3D objects 135 , including generating an approximation of the 3D model having the same number as one of the stored 3D objects 135 prior to compression by the neural network encoder 505 The location coordinates of the vertices and points in the connectivity mesh.

According to an example embodiment, each stored 3D object 135 may be defined by a grid having the same number of points, each point having position coordinates. Additionally, the VAE may include a neural network encoder 505 and a neural network decoder 515, each including a neural network with the same configuration of C filters, KxK masks, and stride factors used for the neural network. As a result, the number of variables in the latent space for each of the latent representations 510 for 3D objects generated by the neural network encoder 505 is the same. Furthermore, the number of points in the grid reproduced by the neural network decoder 515 is the same.

In the example embodiments described above, the stored 3D objects 135 may include 3D objects of interest for CGI movies. Thus, the stored 3D objects 135 are likely to include 3D model or mesh data and corresponding attribute values for a large number of 3D characters. As mentioned above, the 3D model may represent the entire 3D object of interest and/or a portion of the 3D object of interest. The entire 3D object of interest and/or a portion of the 3D object of interest may be represented using a standard 3D model.

In this example embodiment, the stored 3D objects 135 correspond to 3D objects of interest to the CGI movie (eg, as video data 5), and may not include any other 3D objects not included in the CGI movie. Typically, supervised machine learning methods machine learn one or more rules or functions to map between example inputs and desired outputs as predetermined by an operator. Supervised machine learning methods and semi-supervised learning methods can use labeled datasets. Thus, machine training or training neural network encoder 505 and neural network decoder 515 may use a method that uses stored 3D objects 135 (eg, each 3D object is labeled with a unique ID and/or label) as input data and comparison data Supervised machine learning methods or semi-supervised learning methods are done.

The many training iterations used during the training process can produce an approximate logarithmic gain in the accuracy of the reconstructed grid. Thus, in semi-supervised learning methods, thresholds based on time, error (eg, loss), and/or number of iterations can be used to stop further training. For example, a threshold can be set to the reconstruction error or loss based on the number of points and position coordinates of the reconstructed grid. In an example embodiment, the reconstruction error may be a loss based on a shape reconstruction loss (Lr) and a regularization prior loss (Lp).

The shape reconstruction loss (Lr) may be based on obtaining a near-identical transformation between the neural network encoder 505 and the neural network decoder 515 . The shape reconstruction loss (Lr) may be calculated based on the pointwise distance between the stored 3D object 135 generated in FIG. 5B and the stored 3D object 135 of FIG. 5A . The regularization prior loss (Lp) may be calculated on the latent variables using a divergence algorithm based on the variation of the vector based on the stored 3D object 135 of FIG. 5A or a previous iterative distribution, and the divergence algorithm is further based on the latent variable Distribution. The VAE loss or total loss becomes L=Lr+λLp, where λ≧0, and controls the similarity of the variation distribution to the previous variation distribution.

In an example embodiment, the training process includes training the neural network encoder 505 by sequentially iterating through the signal flow for encoding 3D objects as shown in FIG. 5A and the signal flow for decoding 3D objects as shown in FIG. 5B . Neural Network Decoder 515. Continuing the above example, the stored 3D objects 135 correspond to the 3D objects of interest for the CGI movie. After each iteration (eg, encoding and decoding each stored 3D object 135 ), the stored 3D object 135 generated in FIG. 5B (eg, as a reconstructed 3D object) is compared with the stored 3D object 135 of FIG. 5A . 3D object 135 for comparison. In an example embodiment, the reconstruction error may be calculated based on a loss based on a shape reconstruction loss (Lr) and a regularization prior loss (Lp). If the reconstruction error is above a threshold (as described above), the variables for the coefficients corresponding to the C filters, KxK, for at least one of the neural network encoder 505 and the neural network decoder 515 may be modified One or more of mask and stride factor. In addition, the variable λ associated with calculating the VAE loss can be corrected. In an example embodiment, the variables may be modified based on the variable values and results for the previous iteration and the variable values and results for the current modification. After the variables are corrected, the next training iteration can be started.

In an example embodiment, streaming and/or downloading the video may include transmitting initialization data for use by the requesting client device (eg, to decompress the compressed frames) prior to transmitting or streaming the compressed frames of the video. 6A shows a block diagram of a signal flow for streaming video and rendering the video on a client device. As shown in FIG. 6A , streaming server 605 - 1 includes video storage 125 , active streaming module 610 , streaming initialization module 615 , and transceiver 620 . Video storage 125 includes compressed frames 140, metadata 130, and potential representations 510 for 3D objects. Streaming server 605 - 1 may be communicatively coupled to client device 650 via transceiver 625 . As shown in FIG. 6A , client device 650 includes transceiver 625 , decoder 145 , renderer 630 , color correction module 635 , and display 640 .

Active streaming module 610 may be configured to stream selected frames of video to a requesting client device (eg, client device 650). Active streaming module 610 may receive a request for the next frame from client device 650 . The active streaming module 610 may then select the next frame from the compressed frames 140 and select the corresponding metadata from the metadata 130 . In an example embodiment, the selected next frame may be a plurality of compressed frames, and the selected metadata may be a plurality of corresponding metadata elements. For example, multiple compressed frames may be multiple frames bound by (and may include) a previous (eg, temporally, before the frame) keyframe and a future (eg, temporally, after the frame) keyframe .

The active streaming module 610 may then transmit the selected next frame (or frames) and the selected metadata (or metadata elements) to the transceiver 620 . The transceiver 620 may then construct one or more data packets including the selected next frame and the selected metadata, assign the address of the client device 650 to the data packets, and transmit the data packets to the client device 650 .

Streaming initialization module 615 may be configured to select data associated with streaming video in response to a first request for video by client device 650 . For example, the user of client device 650 may download the video for future playback and/or begin streaming playback. The streaming initialization module 615 may select the stored 3D object 135 for the video as the set of potential representations 510 for the 3D object and select the metadata corresponding to the video from the metadata 130 . The streaming initialization module 615 may then transmit the selected stored 3D objects 135 and the selected metadata to the transceiver 620 . The transceiver 620 may then construct a data packet including the selected stored 3D object 135 and the selected metadata, assign the address of the client device 650 to the data packet, and transmit the data packet to the client device 650. In an example embodiment, transceiver 620 may construct more than one data packet.

Client device 650 receives data packets including stored 3D objects 135 and metadata via transceiver 625 . The transceiver 625 transmits the stored 3D object 135 and metadata to the decoder 145 . The decoder 145 may be configured to store the stored 3D object 135 and metadata in association with the requested video. Client device 650 then receives via transceiver 625 a data packet including the selected next frame and the selected metadata. The next frame selected may be the first frame on initial playback. The transceiver 625 transmits the selected next frame and the selected metadata to the decoder 145 .

The decoder 145 then decodes (as described in more detail above) the next frame using the stored 3D object 135 and metadata associated with the next frame. In example embodiments, the decoder 145 may be implemented as a graphics card and/or chip including a graphics processing unit (GPU) (eg, an ASIC on a computer motherboard) or as a graphics card and/or chip including a graphics processing unit (GPU) and/or implemented in a chip (eg, an ASIC on a computer motherboard) configured to remove the burden from the central processing unit (CPU) of the device 650 . GPUs can be configured to process large blocks of video data in parallel. The GPU may be configured to process (eg, decompress) mesh data and generate pixel data from the mesh data. The decoder transmits texture data and color data to the renderer 630 .

In 3D rendering systems, a Cartesian coordinate system (x-(left/right), y-(up/down) and z-axis (near/far)) is usually used. The Cartesian coordinate system provides a precise mathematical way to locate and represent objects in space. For simplicity, the image of the frame can be thought of as the space the object will be in. The space can be oriented based on the camera position. For example, you can place the camera at the origin and look directly at the z-axis. Thus, the translational movement (relative to the camera position) is z forward/backward, y up/down, and x left/right. The object is then projected into space based on its coordinates relative to the origin of space and any camera repositioning. Note that objects and/or the camera can move frame by frame.

Renderer 630 may be configured to render texture data and color data for display. Rendering or drawing can be implemented in the geometry stage and the graphics pipeline in the render stage. The GPU can be configured to handle the geometry stage and the rendering stage. The geometry stage, executed by the CPU or GPU, is configured to handle all polygon activity and convert 3D spatial data to pixels. Geometry stage processes may include, but are not limited to, scene (eg background) geometry generation, camera motion based object motion, object motion, object transformation (eg rotation, translation and/or scaling), object visibility (eg occlusion and culling) and Generation of polygons (eg triangles).

The rendering stage performed by the GPU's 3D hardware accelerator is configured to manage memory and pixel activity, and to process pixels to be drawn to the display 640 . Render stage processes may include, but are not limited to, shading, texturing, depth buffering, and display.

Rendering the texture data and color data for display may include using a 3D shader (eg, vertex shader, geometry shader, etc.) to draw a mesh associated with the frame. Shaders can be configured to generate primitives. The shader may be configured to transform the 3D position and texture of each vertex in the mesh into 2D coordinates (eg, primitives) that appear on a display (eg, display 640 ). Rendering texture data and color data for display may also include performing rasterization. Rasterization may include assigning pixel (eg, color) values to primitives using texture data and color data.

As mentioned above, the geometry phase involves building a 3D mesh in a 2D coordinate system, while the rendering phase involves adding color and texture to the mesh. Thus, the output of decoder 145 may include object position data and color/texture data. Accordingly, compressed frame 140 may include compressed object position data and compressed color/texture data. Compressed object position data and compressed color/texture data may be based on displacement from the previous or next frame, based on camera translation motion data, based on absolute data (e.g., x, y, z coordinates, RGB values and/or the like) and/ or a combination or variation thereof. Thus, the encoder 105 may encode the scene of the frame to generate a compressed frame 140 including data with the compressed object position data and compressed color/texture data.

Data associated with the next frame to be rendered may be passed to the color correction module 635 . Color correction module 635 may be configured to color correct data associated with the next frame to be rendered. Color correction may include compensating for chromatic aberration between frames, compensating for chromatic aberration between multiple views of the same scene, correcting object distortion (warping), correcting object boundary distortion, and the like.

Color correction module 635 may transmit the color correction frame to display 640 . Display 640 may be configured to display the frames as a sequence of frames representing the requested video. In an example embodiment, display 640 includes and/or has associated buffers and/or queues. The buffers and/or queues may be first-in/first-out buffers and/or queues. Accordingly, the color correction module 635 may transmit the color corrected frames to a buffer and/or queue. Display 640 includes components configured to select the next frame from a buffer and/or queue.

In example embodiments, streaming and/or downloading video may include transmitting data for use by a requesting client device (eg, for decompressing the compressed frames) in parallel (eg, nearly simultaneously) with transmitting or streaming compressed frames of the video. . 6B shows a block diagram of another signal flow for streaming video and rendering the video on a client device. As shown in FIG. 6B , streaming server 605 - 2 includes video storage 125 , active streaming module 610 and transceiver 620 .

In the example embodiment shown in FIG. 6B, the active streaming module 610 is further configured to select, from the latent representations 510 for 3D objects, at least one 3D object of interest for use in compressing the selected next frame as a At least one of the stored 3D objects 135 . The active streaming module 610 transmits at least one of the stored 3D objects 135 to the transceiver 620 along with the selected next frame and the selected metadata. The transceiver 620 may then construct a data packet including at least one of the selected next frame, the selected metadata, and the stored 3D object 135, assign the address of the client device 650 to the data packet, and The data packet is transmitted to the client device 650 .

The client device 650 then receives via the transceiver 625 a data packet including at least one of the selected next frame, the selected metadata, and the stored 3D object 135 . The transceiver 625 transmits at least one of the selected next frame, the selected metadata, and the stored 3D object 135 to the decoder 145 . The decoder 145 may be further configured to add and/or use at least one of the stored 3D objects 135 to the stored 3D objects 135 associated with the requested video. The stored 3D object 135 for the requested video is initialized.

Furthermore, while encoding/decoding 3D objects in video with dynamic non-linear and/or random motion from frame to frame (sometimes referred to herein as dynamic 3D objects), color through the geometric proxy techniques described herein Prediction may be most advantageous, but other 3D objects or non-dynamic 3D objects can also be encoded/decoded advantageously using color prediction through geometric proxy techniques.

For example, color prediction can be used by geometry proxies to encode/decode 3D objects that appear to move from frame to frame due to camera or scene translation. In this example, a stationary 3D object may appear to be moving from frame to frame because the camera capturing the scene is moving (eg, in a predictable manner and/or direction). For example, color prediction can be used to encode/decode 3D objects that appear to move in a predictable manner within the scene through a geometric proxy. In this example, a 3D object (eg, a vehicle, ie, a train, car, or plane) may move from frame to frame in a predictable manner (eg, with a constant speed and/or direction). These objects are sometimes referred to herein as translation 3D objects. As another example, color prediction can be used to encode/decode 3D objects that do not appear to be moving within the scene through a geometric proxy. In this example, a stationary or stationary 3D object (eg, the background of the scene, furniture at a fixed location within the scene, or objects moving slowly in the distance) may appear stationary from frame to frame (eg, without any cameras or scene pan). These objects are sometimes referred to herein as stationary or background 3D objects.

The other types of 3D objects exemplified above can be advantageously encoded/decoded using color prediction through geometric proxy techniques for a number of reasons. For example (this is not intended to be an exhaustive list), at least one position (eg, a position in a frame) of the panning 3D object, the stationary 3D object, and/or the background 3D object may be downloaded from the streaming server (eg, the streaming server 605 ). -1, 605-2) to the client device (eg, client device 650) to be used in the renderer and/or rendering operations; due to the presence of geometry proxies (eg, stored 3D objects 135), the critical Number of frames between frames; Z-level layering techniques can be used for both dynamic 3D objects and non-dynamic 3D objects; temporally earlier and later frames (e.g. stored in a queue before being rendered) can be used and geometry proxies, to recreate lost (eg, not retransmitted) frames; geometry proxies can be used to encode/decode 3D objects that appear and disappear between keyframes and/or out-of-frame background cameras can be encoded/decoded or scene panning.

Accordingly, the above-described metadata 20 may be used in the renderer 630 to render panning 3D objects, stationary 3D objects, and/or background 3D objects more efficiently. In an example embodiment, the first object may be identified as a background 3D object. The metadata 20 may identify the first object as, for example, one of the stored 3D objects 135 . The metadata 20 may also identify the origin coordinates (eg, x ₀ , y ₀ , z ₀ ) of the first object (eg, as the background of the frame) as a location attribute representing one of the points in the grid of the first object .

Furthermore, the at least one second object may be identified as a translating 3D object and/or a stationary 3D object. The metadata 20 may identify the at least one second object as, for example, one of the stored 3D objects 135 . The metadata 20 may also identify the location (eg, x _n , y _n , z _n ) of the at least one second object as a location attribute representing one of the points in the grid of the at least one second object. The identified location of the at least one second object may be relative to another object. The relative position of the at least one second object may vary from frame to frame (eg, as a translating 3D object). The relative position of the at least one second object may be fixed from frame to frame (eg, as a fixed 3D object).

For example, the identified location of the at least one second object may be relative to the identified origin coordinates for the background (eg, the first object) for the frame. Using the relative position for the at least one second object may allow six degrees of freedom when positioning the at least one second object relative to the background of the frame. In other words, the at least one second object may have translational motion (eg, forward/backward, up/down, and/or left/right) when positioned relative to the background of the frame and rotational motion (eg, pitch, roll, and/or yaw).

As described above, the first object may be identified as a background 3D object, and may be one of the stored 3D objects 135 . Furthermore, at least one second object may be identified as a translation 3D object and/or a stationary 3D object, which may also be one of the stored 3D objects 135 . Accordingly, the first object and/or the at least one second object may be encoded by an autoencoder (implementing a neural network) into a corresponding stored latent representation of the 3D object 135 (as described in more detail above). Latent representations (eg, as one or more latent representations 510 for 3D objects) and information (eg, metadata) related to the structure of the neural network may be communicated from the streaming servers 605-1, 605-2 to the client 650. The client 650 may use the auto-encoder's decoder (implementing a neural network) to reconstruct one or more corresponding stored 3D objects 135 (discussed in more detail above).

7-11 are flowcharts of methods according to example embodiments. The steps described with reference to Figures 7-11 may be implemented as a result of executing software code stored in memory (eg, at least one memory 1210, 1310) associated with the apparatus (eg, as shown in Figures 12 and 13) or by The device is associated with at least one processor (eg, at least one processor 1205, 1305) to execute.

However, alternative embodiments are contemplated, such as systems embodied as special purpose processors. The dedicated processor may be a graphics processing unit (GPU). A GPU can be a component of a graphics card. The graphics card may also include video memory, random access memory digital to analog converters (RAMDACs), and driver software. The video memory may be a frame buffer that stores digital data representing a scene of an image or frame. The RAMDAC can be configured to read the contents of video memory, convert the contents to an analog RGB signal, and send the analog signal to a display or monitor. The driver software may be software codes stored in the memory described above (eg, at least one memory 1210, 1310). The software code may be configured to implement the steps described below (and/or the components, modules and signal flows described above).

Although the steps described below are described as being performed by a processor and/or special purpose processors, these steps are not necessarily performed by the same processor. In other words, at least one processor and/or at least one dedicated processor may perform the steps described below with reference to FIGS. 7 to 11 .

7 illustrates a block diagram of a method for compressing frames of video, according to at least one example embodiment. As shown in FIG. 7, in step S705, a file including a plurality of frames of video is received. For example, the file may be saved or transferred to a server (eg, a streaming server). The file can include video. The video can be a CGI 3D movie. The file may include a number of 3D objects of interest (eg, characters in a 3D movie).

In an example embodiment, each of the plurality of 3D objects of interest may be defined by a triangular mesh. A triangular mesh can be a collection of points connected by triangular faces. Each point can store various properties. For example, attributes can include the location, color, texture coordinates, etc. of each point. The mesh for each of the multiple 3D objects of interest may have the same number of points, each point having the same properties. Thus, the grids for each of the plurality of 3D objects of interest may be approximately the same size (eg, bits) when stored in memory.

In step S710, one of the plurality of frames is selected. For example, each of a plurality of frames can be a compression target. Multiple frames can be compressed in chronological order. Therefore, in an initial step, the first frame in time is selected. The next frame can then be selected sequentially.

In step S715, 3D objects are identified in the selected frames and keyframes. The identified 3D objects may be dynamic 3D objects, non-dynamic 3D objects, fixed 3D objects, background 3D objects, and the like. For example, machine vision, computer vision and/or computer image recognition techniques may be used to identify and locate 3D objects.

In an example embodiment, computer image recognition techniques based on training (machine learning) convolutional neural networks using a number of known images may be used to identify 3D objects. For example, a block, blocks and/or patches are selected and/or identified from the selected frame. The trained convolutional neural network can operate on the selected block, multiple blocks, and/or patches. Results can be tested (eg, error tests, loss tests, divergence tests, etc.). If the test produces a value below the threshold (or alternatively, depending on the type of test, above the threshold), the selected block, blocks and/or patches may be identified as a 3D object.

In an example embodiment, a frame of the video may include a tag indicating that a previously identified 3D object of interest is included in the frame. Labels may include the identity and location of the 3D object. For example, a computer-generated image (CGI) tool, such as a computer-animated film, may be used to generate the video. Computer-generated characters can be identified and tagged in each frame. Additionally, a model of each of the identified 3D objects of interest (eg, identified characters) may be stored as stored 3D objects 135 .

In an example embodiment, a 3D object may be defined by a triangular mesh. A triangular mesh can be a collection of points connected by triangular faces. Each point can store various properties. For example, attributes can include the location, color, texture coordinates, etc. of each point. The attributes may include and/or indicate (eg, multiple attributes may indicate) the orientation of the corresponding 3D object and/or the position of the corresponding 3D object in the selected frame.

Thus, in example embodiments, the mesh properties of the 3D object may be sufficient to identify and locate the 3D object. A model including mesh properties for multiple 3D objects of interest may be stored as stored 3D objects 135 . The model can be normalized. For example, a model for a man, woman, teenager, child, or more generally a human being or a part of a human being (eg, body, head, hand, etc.) may be stored as stored 3D objects 135 . For example, a model for a dog, cat, deer, or more generally a quadruped or part of a quadruped (eg, body, head, legs, etc.) may be stored as the stored 3D object 135 . The model's properties can then be used to search the frame for 3D objects with similar properties.

In step S720, the position and orientation of the 3D object within the selected frame and keyframe are determined. For example, a 3D object can be defined by a grid with multiple points, each point having at least one property. The position of the 3D object may be the position of the 3D object within the frame. Thus, the position of the 3D object within the frame may be based on the 3D Cartesian coordinate system used for the frame. For example, at least one point of the grid may be located at an x,y,z position within the frame. The orientation of the 3D object may be based on the location coordinate properties of each point in the grid used to define the 3D object. If the 3D object is a moving character, the orientation of the 3D object may be the pose of the character in frames and/or keyframes.

In step S725, the 3D objects are matched with the stored 3D objects. For example, the 3D object may be matched to one of the 3D objects of interest (eg, stored 3D objects 135). In an example embodiment, the stored 3D object 135 may be searched using the identity of the 3D object as generated by computer image recognition techniques. In an example embodiment, the label of the 3D object found in the frame may be matched to the label of one of the stored 3D objects 135 . In an example embodiment, a model with one of the stored 3D objects 135 having similar properties to the 3D object may be identified as a match. The matched stored 3D objects can be used as geometric proxies.

In step S730, the stored 3D objects are transformed based on the 3D objects. For example, the stored 3D objects can be deformed based on the 3D objects. The stored 3D objects can be resized based on the 3D objects. The stored 3D objects may be oriented based on the 3D objects. The stored 3D objects can be rotated based on the 3D objects. The stored 3D objects can be translated based on the 3D objects. Transforming the stored 3D object may cause the stored 3D object (if rendered side-by-side with the 3D object on the display) to appear substantially similar (eg, pose-wise) to the 3D object. Note that the stored 3D objects may be defined by different meshes and may have different color properties than the 3D objects.

In an example embodiment, the grid corresponding to the stored 3D object may be rotated and translated about and along the x, y and z axes to align with the orientation of the 3D object. The matched one of the stored 3D objects 135 associated with the frame and keyframe may be transformed. Thus, the stored 3D object may be used to generate a first 3D object proxy based on the stored 3D object for the 3D object associated with the frame. Furthermore, the stored 3D object may be used to generate a second 3D object proxy based on the stored 3D object for the 3D object associated with the keyframe. Thus, the first 3D object proxy can be transformed based on the 3D object identified in the frame, and the second 3D object proxy can be transformed based on the 3D object identified in the keyframe.

In step S735, the selected frame is compressed by a color prediction scheme using the transformed stored 3D object. For example, points in a grid defining a transformed stored 3D object may match points of a corresponding (eg, the same) 3D object in a nearby (temporally), previously encoded keyframe. The prediction technique may then use the matching points of the corresponding 3D object to select or predict pixels/blocks/patches in the keyframes for use in computing the residuals of the 3D objects in the selected frames (eg, relative to the keyframes) color shift).

In an example embodiment, the transformed first 3D object proxy may be wrapped (with color and/or texture) corresponding to the 3D object identified in the frame, and may correspond (with color and/or texture) to the 3D object identified in the keyframe and/or texture) wraps the transformed second 3D object proxy. Mapping the color properties from the 3D object identified in frame 110 to the transformed first 3D object proxy may include converting the 3D object from 3D space (eg, XYZ space) to 2D space (eg, UV space) and/or converting the 3D object from 3D space (eg, XYZ space) to 2D space (eg, UV space) The 3D objects identified in 110 are converted from 2D space (eg, UV space) to 3D space (eg, XYZ space). can be obtained by subtracting the color attribute in the mesh representation of the first 3D object proxy from the color attribute of the corresponding point in the mesh representation of the second 3D object proxy (eg, having the same point identification or being at the same position in the mesh sequence) Color properties of points to generate residuals for 3D objects.

In an example embodiment, two or more 3D objects are identified in the selected frame. Each identified 3D object is predicted using the prediction techniques described above. The remainder of the frame uses standard prediction techniques to generate residuals. Compressing the selected frames may include performing a series of encoding processes on the residuals. For example, the residual can be transformed, quantized and entropy encoded.

Additionally, the prediction scheme may generate metadata associated with the selected frame. The metadata may include data associated with at least one 3D object located in the selected frame that has been predicted using one of the 3D objects of interest (eg, stored 3D object 135). The metadata may include attributes (eg, grid point attributes) associated with the positioning and/or orientation of the 3D object in selected frames and keyframes.

In step S740, the compressed selected frame is stored together with metadata identifying the 3D object and the position and orientation of the 3D object. For example, the compressed selected frames and metadata may be stored in memory associated with the streaming server. The compressed selected frame and metadata may be stored in relation to a plurality of compressed frames corresponding to the video (or a portion thereof).

8 illustrates a block diagram of another method for compressing frames of video, according to at least one example embodiment. As shown in FIG. 8, in step S805, a file including a plurality of frames of video is received. For example, the file may be saved or transferred to a server (eg, a streaming server). The file can include video. The video can be a CGI 3D movie. The file may include a number of 3D objects of interest (eg, characters in a 3D movie).

In an example embodiment, each of the plurality of 3D objects of interest may be defined by a triangular mesh. A triangular mesh can be a collection of points connected by triangular faces. Each point can store various properties. For example, attributes can include the location, color, texture coordinates, etc. of each point. The mesh for each of the multiple 3D objects of interest may have the same number of points, each point having the same properties. Thus, the grids for each of the plurality of 3D objects of interest may have approximately the same size (eg, number of bits) when stored in memory.

In an example embodiment, the mesh for each of the plurality of 3D objects of interest may be compressed (eg, using the techniques described above with reference to FIG. 5A ). For example, the meshes for each of the plurality of 3D objects of interest may be compressed using generative modeling techniques (eg, using neural networks, convolutional neural networks, VAEs, etc.). The mesh for each of the plurality of 3D objects of interest can be compressed using a neural network encoder with a convolutional neural network with a machine-trained-based generative modeling technique selected and trained. Element, the machine-trained generative modeling technique is configured to generate a reduced number of variables associated with mesh properties and positions for each 3D object. The resulting reduced number of variables associated with mesh properties and positions for 3D objects is sometimes referred to as a compact latent representation for 3D objects.

In step S810, one of the plurality of frames is selected. For example, each of a plurality of frames can be a compression target. Multiple frames can be compressed in chronological order. Therefore, in an initial step, the first frame in time is selected. The next frame can then be selected in sequence.

In step S815, 3D objects are identified in the selected frames and keyframes. The identified 3D objects may be dynamic 3D objects, non-dynamic 3D objects, stationary 3D objects, background 3D objects, and the like. For example, machine vision, computer vision and/or computer image recognition techniques may be used to identify and locate 3D objects.

In an example embodiment, computer image recognition techniques based on training (machine learning) convolutional neural networks using a number of known images may be used to identify 3D objects. For example, a block, blocks and/or patches are selected and/or identified from the selected frame. The trained convolutional neural network can operate on the selected block, multiple blocks, and/or patches. Results can be tested (eg, error tests, loss tests, divergence tests, etc.). If the test produces a value below the threshold (or alternatively, depending on the type of test, above the threshold), the selected block, blocks and/or patches may be identified as a 3D object.

In an example embodiment, a frame of the video may include a tag indicating that a previously identified 3D object of interest is included in the frame. Labels may include the identity and location of the 3D object. For example, a computer-generated image (CGI) tool, such as a computer-animated film, may be used to generate the video. Computer-generated characters can be identified and tagged in each frame. Additionally, a model for each identified 3D object of interest (eg, identified character) may be stored as stored 3D object 135 .

In an example embodiment, a 3D object may be defined by a triangular mesh. A triangular mesh can be a collection of points connected by triangular faces. Each point can store various properties. For example, attributes can include the location, color, texture coordinates, etc. of each point. The attributes may include and/or indicate (eg, multiple attributes may indicate) the orientation of the corresponding 3D object and/or the position of the corresponding 3D object in the selected frame.

Thus, in example embodiments, the mesh properties of the 3D object may be sufficient to identify and locate the 3D object. A model including mesh properties for multiple 3D objects of interest may be stored as stored 3D objects 135 . The model can be normalized. For example, a model for a man, woman, teenager, child, or more generally a human being or a part of a human being (eg, body, head, hand, etc.) may be stored as stored 3D objects 135 . For example, a model for a dog, cat, deer, or more generally a quadruped or part of a quadruped (eg, body, head, legs, etc.) may be stored as the stored 3D object 135 . The model's properties can then be used to search the frame for 3D objects with similar properties.

In step S820, the position and orientation of the 3D object within the selected frame and keyframe is determined. For example, a 3D object can be defined by a grid with multiple points, each point having at least one property. The position of the 3D object may be the position of the 3D object within the frame. Thus, the position of the 3D object within the frame may be based on the 3D Cartesian coordinate system used for the frame. For example, at least one point of the grid may be located at an x,y,z position within the frame. The orientation of the 3D object may be based on the location coordinate properties of each point in the grid used to define the 3D object. If the 3D object is a moving character, the orientation of the 3D object may be the pose of the character in frames and/or keyframes.

In step S825, the 3D object is matched with a latent representation for the 3D object. For example, the 3D object may be matched with one of the 3D objects of interest (eg, stored 3D object 135). Then, as described above, the matched 3D objects can be encoded into latent representations for the 3D objects. For example, the 3D object can be matched to one of the 3D objects of interest (eg, the latent representations 510 for the stored 3D objects). In an example embodiment, the identities of the 3D objects as generated by computer image recognition techniques may be used to search 510 the stored potential representations of the 3D objects. In an example embodiment, the label of the 3D object found in the frame may be matched with the label of one of the stored potential representations 510 of the 3D object. In an example embodiment, a model whose one of the stored potential representations 510 of the 3D object has similar properties to the compressed mesh properties of the 3D object may be identified as a match.

In step S830, the latent representation is transformed based on the 3D object. For example, the stored 3D objects can be deformed based on the 3D objects. The stored 3D objects can be resized based on the 3D objects. The stored 3D objects may be oriented based on the 3D objects. The stored 3D objects can be rotated based on the 3D objects. The stored 3D objects can be translated based on the 3D objects. Transforming the stored 3D object may cause the stored 3D object (if rendered side-by-side with the 3D object on the display) to appear substantially similar to the 3D object. Note that the stored 3D objects may be defined by different meshes and may have different color properties than the 3D objects.

In an example embodiment, the coordinates of each point of the underlying representation of the 3D object may be rotated and translated about and along the x, y and z axes to align with the orientation of the 3D object. The underlying representations of the matched 3D objects associated with the frames and keyframes can be transformed. Accordingly, a potential representation of the 3D object may be generated as the first 3D object proxy based on the stored 3D object for the 3D object associated with the frame. Furthermore, a latent representation of the 3D object may be generated as a second 3D object proxy based on the stored 3D object for the 3D object associated with the keyframe. Thus, the first 3D object proxy can be transformed based on the 3D object identified in the frame, and the second 3D object proxy can be transformed based on the 3D object identified in the keyframe.

In an example embodiment, the stored 3D objects are stored as compact latent representations based on the 3D objects. Accordingly, the stored 3D object may be regenerated (eg, decompressed using the techniques described above with reference to FIG. 5B ) using a compact latent representation based on the 3D object prior to transforming the stored 3D object. For example, variables in the compact latent space for the stored 3D object can be input to a neural network decoder to reproduce the points defining the grid of the stored 3D object and the location coordinates for each point of the grid .

In step S835, the selected frame is compressed with a color prediction scheme using the transformed stored 3D object with the compact latent representation for the 3D object. As described above, the transformed stored 3D object with a compact latent representation for the 3D object may be a transformed reproduced stored 3D object. For example, points in the grid defining the transformed reproduced stored 3D object may be matched with points of the corresponding (eg, the same) 3D object located in a nearby (temporally), previously encoded keyframe. The prediction technique may then use the matching points of the corresponding 3D object to select or predict pixels/blocks/patches in the keyframes for use in computing the residuals of the 3D objects in the selected frames (eg, relative to the keyframes) color shift).

In an example embodiment, the transformed first 3D object proxy may be wrapped corresponding to the 3D object identified in the frame (with color and/or texture), and may correspond to the 3D object identified in the keyframe (with color and/or texture) or texture) wraps the transformed second 3D object proxy. Mapping the color properties from the 3D object identified in frame 110 to the transformed first 3D object proxy may include converting the 3D object from 3D space (eg, XYZ space) to 2D space (eg, UV space) and/or converting the 3D object from 3D space (eg, XYZ space) to 2D space (eg, UV space) The 3D objects identified in 110 are converted from 2D space (eg, UV space) to 3D space (eg, XYZ space). In an example embodiment, the value of a point in the latent representation of the first 3D object proxy may be subtracted from the color attribute of the corresponding point in the latent representation of the second 3D object proxy (eg, having the same coordinates or at the same location) Color properties to generate residuals for 3D objects.

In an example embodiment, the latent representation for the transformed wrapped first 3D object proxy and the latent representation of the transformed wrapped second 3D object proxy may be decoded. As described above, decoding the underlying representation of a 3D object can reproduce a grid representation that includes the color properties of the 3D object. The regenerated mesh representation of the first 3D object proxy may be obtained by subtracting the regenerated mesh representation of the first 3D object proxy from the color properties of the corresponding points in the regenerated mesh representation of the second 3D object proxy (eg, having the same point identification or being in the same position in the mesh sequence) Color properties of points in to generate residuals for 3D objects

In an example embodiment, two or more 3D objects are identified in the selected frame. Each identified 3D object is predicted using the prediction techniques described above. The remainder of the frame uses standard prediction techniques to generate residuals. Compressing the selected frames may include performing a series of encoding processes on the residuals. For example, the residual can be transformed, quantized and entropy encoded.

Additionally, the prediction scheme may generate metadata associated with the selected frame. The metadata may include data associated with at least one 3D object located in the selected frame that has been predicted using one of the 3D objects of interest (eg, stored 3D object 135). The metadata may include attributes (eg, grid point attributes) associated with the positioning and/or orientation of the 3D object in selected frames and keyframes. The metadata may include information associated with a neural network for an autoencoder used to generate latent representations of 3D objects (e.g., encode) and regenerate grid representations (e.g., decode) that include color properties of 3D objects. ).

In step S840, the compressed selected frame is stored together with metadata identifying the 3D object and the position and orientation of the 3D object. For example, the compressed selected frames and metadata may be stored in memory associated with the streaming server. The compressed selected frame and metadata may be stored in relation to a plurality of compressed frames corresponding to the video (or a portion thereof).

9 illustrates a block diagram of a method for decompressing and rendering frames of a video, according to at least one example embodiment. As shown in FIG. 9, in step S905, a data packet of at least one compressed frame of the packet video is received. For example, the client device may request the next frame of video from the streaming server. In response to the request, the streaming server may select the next frame(s), determine if the selected next frame(s) has associated metadata and select the associated metadata data. The streaming server may generate a packet (or packets) including the selected next frame(s) and the selected associated metadata, and the packet(s) ) to the requesting client device.

In step S910, at least one frame is selected for decompression. For example, a frame may be selected from the received packet (or packets). In step S915, it is determined whether the frame includes metadata. In response to determining that the frame includes metadata, processing proceeds to step S925. Otherwise, processing continues to step S920 and the selected frame is decoded by some other prediction scheme (eg, a prediction scheme not based on using 3D objects as geometric proxies).

For example, a header associated with the data packet for the selected frame may be configured to contain metadata when the data packet is transmitted from the streaming server to the client. Accordingly, the header associated with the data packet for the selected frame may be read (eg, via transceiver 625) and transmitted to a decoder (eg, decoder 145) along with the selected frame. Thus, determining that the selected frame includes metadata may include determining that metadata has been transmitted with the selected frame. Alternatively, determining that the selected frame includes metadata may include determining that metadata is stored in association with the selected frame (eg, in a decoder).

In step S925, a 3D object is identified in the selected frame based on the metadata. For example, the metadata may include data associated with at least one 3D object (eg, at least one of the stored 3D objects 135 ) located in the selected frame that has been predicted using the 3D object as a geometric proxy. Metadata may include information identifying 3D objects used as geometric proxies.

In step S930, based on the metadata, the position and orientation of the 3D object within the selected frame and keyframe is determined. For example, the metadata may include attributes (eg, grid point attributes) associated with the positioning and/or orientation of the 3D object in selected frames and keyframes.

In step S935, the 3D objects are matched with the stored 3D objects. For example, the metadata may include information identifying that a 3D object has a corresponding stored 3D object. In an example embodiment, the metadata includes a unique ID or tag that can be used to search the stored 3D objects 135 for 3D objects. If a unique ID or tag is found in the stored 3D objects 135, then the corresponding one of the stored 3D objects 135 is a match for the 3D object.

In step S940, the stored 3D object is transformed based on the metadata. For example, the metadata may include information identifying that the 3D object has a corresponding stored 3D object and information related to transformations performed on the stored 3D object during the encoding process. Information about the transformations performed on the stored 3D objects during the encoding process can be used to perform the same transformations on the stored 3D objects as performed on the stored 3D objects to encode the selected frame.

The stored 3D object may be deformed based on information related to transformations performed on the stored 3D object during the encoding process. The stored 3D objects may be resized based on information related to transformations performed on the stored 3D objects during the encoding process. The stored 3D objects may be oriented based on information related to transformations performed on the stored 3D objects during the encoding process. The stored 3D objects may be rotated based on information related to transformations performed on the stored 3D objects during the encoding process. The stored 3D objects may be translated based on information related to transformations performed on the stored 3D objects during the encoding process. Transforming the stored 3D object may cause the stored 3D object (if rendered side-by-side with the 3D object on the display) to appear substantially similar to the 3D object because the 3D object appears at the selected frame before compressing the selected frame middle.

In an example embodiment, the matched 3D objects of the stored 3D objects associated with the frames and keyframes may be transformed. Accordingly, the stored 3D object may be used to generate a first 3D object proxy based on the stored 3D object for the 3D object associated with the frame. Furthermore, the stored 3D object may be used to generate a second 3D object proxy based on the stored 3D object for the 3D object associated with the keyframe. Thus, the first 3D object proxy may be transformed based on the metadata identified in the frame for the 3D object, and the second 3D object proxy may be transformed based on the metadata identified in the keyframe for the 3D object.

In an example embodiment, the first 3D object proxy and the second 3D object proxy are latent representations of the 3D object. Thus, the first 3D object proxy can be used to reproduce the mesh representation for the first 3D object proxy, and the second 3D object proxy can be used to reproduce the mesh representation for the second 3D object proxy. As described above, decoding a latent representation for a 3D object using an autoencoder can regenerate a mesh representation that includes color properties for the 3D object. Autoencoders can use neural networks with structures read from metadata.

In step S945, the selected frame is decompressed with a color prediction scheme using the stored 3D objects for the 3D objects (as geometric proxies). Initially, selected frames may be entropy decoded, inverse quantized, and inverse transformed to generate the same (or approximately the same) residuals as generated by the encoder when encoding the selected frames and prior to transforming, quantizing, and entropy encoding the residuals. ) of the derivative residuals.

In an example embodiment, the prediction scheme includes matching points corresponding to the transformed (eg, translated and oriented) stored 3D objects with points of keyframes. The matching points are then used to select or predict pixels/blocks/patches in the keyframe for reproducing the color values of the pixels of the 3D object in the selected frame.

The color value of the reproduced pixel may include the pixel attribute used for each point in the triangular mesh of the 3D object from the pixel attribute value used for each matching point of the predicted pixel/block/patch in the keyframe. The value is added to the stored color value and/or color attribute of the 3D object for translation and orientation based on the residual at the identified location in the selected frame. Color values and/or color attributes for the remainder of the selected frame may be reproduced based on the residual or remainder of the selected frame and the corresponding pixels/blocks/patches of the keyframe.

In an example embodiment, the transformed first 3D object proxy may be wrapped (with color and/or texture) corresponding to the 3D object identified in the frame. In other words, the transformed first 3D object proxy may be wrapped with a residual corresponding to the 3D object identified in the frame. Furthermore, the transformed second 3D object proxy may be wrapped (with color and/or texture) corresponding to the 3D object identified in the keyframe. This can be done by comparing the color properties of points in the mesh representation of the first 3D object proxy with the corresponding points in the mesh representation of the second 3D object proxy (eg, having the same point identity or being in the same position in the mesh sequence) The color properties are added to reproduce the color values of the pixels used for the 3D object. Mapping the color properties from the 3D object identified in the frame to the transformed first 3D object proxy may include converting the 3D object from 3D space (eg, XYZ space) to 2D space (eg, UV space) and/or converting the 3D object in frame 110 The identified 3D objects are converted from 2D space (eg, UV space) to 3D space (eg, XYZ space).

In an example embodiment, the first 3D object proxy and the second 3D object proxy are latent representations of the 3D object. Thus, a first 3D object proxy can be used to reproduce a mesh representation that includes color attributes for the first 3D object proxy, and a second 3D object proxy can be used to reproduce a mesh representation that includes color attributes for the second 3D object proxy grid representation. As described above, decoding a latent representation of a 3D object using an autoencoder can regenerate a mesh representation that includes color attributes for the 3D object. Autoencoders can use neural networks with structures read from metadata.

In step S950, the transformed stored 3D object is stitched into the decompressed frame. The selected frame may be reconstructed based on the translated and oriented stored reproduced color values and/or color properties of the 3D object and the reproduced color values and/or color properties of the remainder of the selected frame. In an example embodiment, the translated and oriented stored regenerated color values and/or color properties of the 3D object may be stitched to the The selected frame is reconstructed from the reproduced color values and/or color attributes of the remainder of the selected frame.

In step S955, the frame including the 3D object is rendered. For example, regenerated texture data and regenerated color data may be rendered for display. Rendering the texture data and color data for display may include using a 3D shader (eg, vertex shader, geometry shader, etc.) to draw a mesh associated with the frame. Shaders can be configured to generate primitives. The shader may be configured to transform the 3D position and texture of each vertex in the mesh into 2D coordinates (eg, primitives) that appear on a display (eg, display 640 ). Rendering texture data and color data for display may also include performing rasterization. Rasterization may include assigning pixel (eg, color) values to primitives using texture data and color data.

In step S960, color correction is performed on the rendered frame. For example, color correction may include compensating for chromatic aberration between frames, compensating for chromatic aberration between multiple views of the same scene, correcting object distortion (warping), correcting object boundary distortion, and the like.

Example embodiments may include identifying two or more 3D objects; regenerating the color value and/or color attribute of each of the two or more 3D objects; and using each of the two or more 3D objects. A reconstructed frame selected. The video data 5 may be reproduced based on a plurality of reconstructed frames. Video data 5 (eg, texture data and color data) may be rendered and color corrected for display on the client device's display.

As described above, the techniques described herein (eg, the methods described above) are intended for use with an increased number of videos when implemented. The number of stored 3D objects of interest (eg, stored 3D objects 135) must increase, and the amount of resources (eg, memory) required to store the 3D objects of interest will also increase. Furthermore, during a streaming activity, transferring the 3D object of interest from the streaming server to the client device may require significant bandwidth. Thus, for streaming operations, it may become desirable to efficiently encode and decode 3D objects of interest.

10 illustrates a block diagram of a method for compressing a 3D object according to at least one example embodiment. As shown in FIG. 10, in step S1005, at least one 3D object of interest of the video is identified. For example, objects of interest may include 3D characters from CGI movies, including CGI actors (eg, lead characters, supporting characters, and extras), CGI pets, CGI creatures, CGI monsters, and the like. Objects of interest may include vehicles (eg, trains, cars, or airplanes) or objects that may move in a predictable manner (eg, with a constant speed and/or direction) from frame to frame. Objects of interest can include stationary or stationary 3D objects (e.g., the background of the scene, furniture at fixed locations within the scene, or objects moving slowly in the distance) from frame to frame (e.g., without any camera or scene panning). bottom) may appear stationary.

3D objects of interest can be predetermined and stored in association with the video. A 3D object of interest may be determined as needed (eg, when a keyframe is selected, as part of an initialization operation) and stored in association with the video or added to a previously stored 3D object of interest.

In step S1010, mesh properties (eg, vertices and connectivity) and locations of the 3D object of interest are determined. According to an example embodiment, each 3D object of interest may be defined by a grid with the same number of points, each point having position coordinates. Other attributes of the point can be added as needed. Thus, the mesh properties for each 3D object of interest may include the same number of vertices with varying connectivity.

In step S1015, a machine-trained generative modeling technique is used to generate a reduced number of variables associated with mesh properties and positions of the 3D object of interest. For example, a VAE can be used to generate a reduced number of variables. This reduced number of variables is sometimes referred to as a latent representation for 3D objects or a reduced latent representation for 3D objects. A VAE can include a neural network encoder and a neural network decoder, each of which includes a neural network with the same configuration of C filters, KxK masks and stride factors for the neural network . Therefore, the number of variables in the latent space generated by the neural network encoder for each latent representation of a 3D object of interest is the same for each 3D object of interest. Furthermore, the number of points in the grid regenerated by the neural network decoder is the same for each 3D object of interest.

In an example embodiment, the mesh for each of the plurality of 3D objects of interest may be compressed (eg, using the techniques described above with reference to FIG. 5A ). For example, the meshes for each of the plurality of 3D objects of interest may be compressed using generative modeling techniques (eg, using neural networks, convolutional neural networks, VAEs, etc.). The mesh for each of the plurality of 3D objects of interest can be compressed using a neural network encoder with a convolutional neural network with a machine-trained-based generative modeling technique selected and trained. Element, the machine-trained generative modeling technique is configured to generate a reduced number of variables associated with mesh properties and positions for each 3D object. The resulting reduced number of variables associated with mesh properties and positions for 3D objects is sometimes referred to as a compact latent representation for 3D objects.

In step S1020, the variable is stored as a compact decodable representation (or compact latent representation) of the 3D object of interest associated with the 3D object of interest and associated with the video. For example, variables may be stored as latent representations for 3D object 510 . The underlying representation for 3D object 510 may be stored on a streaming server and/or a device that includes at least one encoder (eg, encoder 105).

11 illustrates a block diagram of a method for decompressing a 3D object according to at least one example embodiment. As shown in FIG. 11, in step S1105, a compact decodable representation of the 3D object associated with the video is received. For example, variables stored as potential representations of 3D object 510 may be stored on a streaming server and/or a device that includes at least one encoder (eg, encoder 105). During a video streaming operation, at least one set of variables stored as a potential representation of a 3D object may be received from a streaming server.

In step S1110, a machine-trained generative modeling technique may be used to generate mesh properties (eg, vertices and connectivity) and locations of the 3D object. For example, the VAE can use variables in a latent space of latent representations of 3D objects that can be used as input to the neural network decoder, and the VAE can regenerate the location coordinates of each point of the grid. The configuration of the C filters, KxK mask and step factor for the neural network can determine the number of points of the regenerated grid. When regenerating the points of the mesh, the neural network decoder can regenerate the mesh properties and positions of the 3D objects.

In step S1115, the mesh properties, position, orientation and color properties of the 3D object as the stored 3D object are stored in association with the video. For example, the 3D object may be stored as stored 3D object 135 when the 3D object is regenerated as a mesh comprising a collection of points connected by faces. Each regenerated point can store various attributes. For example, attributes can include the location, color, texture coordinates, etc. of each point. Accordingly, in a color prediction scheme based on the use of 3D objects as geometric proxies as implemented in the 3D encoder and/or 3D decoder, the reproduced 3D objects included in the stored 3D objects 135 may be used.

12 illustrates a video encoder system 1200 in accordance with at least one example embodiment. As shown in FIG. 12 , video encoder system 1200 includes at least one processor 1205 , at least one memory 1210 , controller 1220 and video encoder 105 . At least one processor 1205 , at least one memory 1210 , controller 1220 , video encoder 105 , and video encoder 105 are communicatively coupled through bus 1215 .

In the example of FIG. 12, video encoder system 1200 may be or include at least one computing device, and should be understood to represent virtually any computing device configured to perform the methods described herein. As such, video encoder system 1200 may be understood to include various components that may be used to implement the techniques described herein or different or future versions thereof. For example, video encoder system 1200 is illustrated as including at least one processor 1205, and at least one memory 1210 (eg, a non-transitory computer-readable storage medium).

It will be appreciated that at least one processor 1205 may be utilized to execute instructions stored on at least one memory 1210 to thereby implement various features and functions described herein, or additional or alternative features and functions. Of course, the at least one processor 1205 and the at least one memory 1210 may be used for various other purposes. In particular, it is to be understood that at least one memory 1210 may be understood to represent examples of various types of memory and associated hardware and software that may be used to implement any of the modules described herein.

At least one memory 1210 may be configured to store data and/or information associated with video encoder system 1200 . At least one memory 1210 may be a shared resource. For example, video encoder system 1200 may be an element of a larger system (eg, server, personal computer, mobile device, etc.). Accordingly, at least one memory 1210 may be configured to store data and/or information associated with other elements within the larger system (eg, image/video services, web browsing, or wired/wireless communications).

Controller 1220 may be configured to generate and communicate various control signals to various blocks in video encoder system 1200 . Controller 1220 may be configured to generate control signals to implement the techniques described above. According to example embodiments, the controller 1220 may be configured to control the video encoder 1225 to encode images, image sequences, video frames, video sequences, and the like. For example, the controller 1220 may generate a control signal corresponding to the video quality.

Video encoder 105 may be configured to receive video stream input 5 and output compressed (eg, encoded) video bits 10 . The video encoder 105 may convert the video stream input 5 into discrete video frames. The video stream input 5 can also be an image, so the compressed (eg, encoded) video bits 10 can also be compressed image bits. Video encoder 105 may further convert each discrete video frame (or image) into a matrix of blocks (hereinafter referred to as blocks). For example, a video frame (or image) may be converted into a 16x16, 16x8, 8x8, 4x4, or 2x2 matrix of blocks of multiple pixels, respectively. Although five example matrices are listed, example embodiments are not so limited.

Compressed video bits 10 may represent the output of video encoder system 1200 . For example, compressed video bits 10 may represent encoded video frames (or encoded images). For example, compressed video bits 10 may be ready for transmission to a receiving device (not shown). For example, video bits may be transmitted to a system transceiver (not shown) for transmission to a receiving device.

At least one processor 1205 may be configured to execute computer instructions associated with controller 1220 and/or video encoder 105 . At least one processor 1205 may be a shared resource. For example, video encoder system 1200 may be an element of a larger system (eg, mobile device, server, streaming server, etc.). Accordingly, at least one processor 1205 may be configured to execute computer instructions (eg, image/video services, web browsing, or wired/wireless communications) associated with other elements within the larger system.

In example embodiments, the video encoder system 1200 may be implemented as or in a graphics card and/or chip including a graphics processing unit (GPU) (eg, an ASIC on a computer motherboard), The graphics processing unit (GPU) is configured to offload the central processing unit (CPU). At least one processor 1205 may be implemented as a GPU configured to process large blocks of video data in parallel. The GPU may be configured to process (eg, compress) mesh data and generate pixel data from the mesh data. At least one memory 1210 may include video memory and driver software. Video memory may be a frame buffer that stores digital data representing frame images or scenes. Video memory can store digital data before and after GPU processing. The driver software may include a codec configured to decompress the video data. The codec may include implementing a color prediction scheme based on the use of 3D objects as geometric proxies as described herein.

13 illustrates a video decoder system 1300 in accordance with at least one example embodiment. As shown in FIG. 13 , the video decoder system 1300 includes at least one processor 1305 , at least one memory 1310 , a controller 1320 and a video decoder 145 . At least one processor 1305 , at least one memory 1310 , controller 1320 , video decoder 145 , and video decoder 145 are communicatively coupled through bus 1315 .

In the example of FIG. 13, video decoder system 1300 may be at least one computing device and should be understood to represent virtually any computing device configured to perform the methods described herein. As such, video decoder system 1300 may be understood to include various components that may be used to implement the techniques described herein or different or future versions thereof. For example, video decoder system 1300 is illustrated as including at least one processor 1305, and at least one memory 1310 (eg, a computer-readable storage medium).

Thus, it will be appreciated that at least one processor 1305 may be utilized to execute instructions stored on at least one memory 1310 to implement various features and functions described herein, or additional or alternative features and functions. Of course, the at least one processor 1305 and the at least one memory 1310 may be used for various other purposes. In particular, it is to be understood that the at least one memory 1310 may be understood to represent an example of various types of memory and associated hardware and software that may be used to implement any of the modules described herein. According to example embodiments, video encoder system 1200 and video decoder system 1300 may be included in the same larger system (eg, personal computer, mobile device, etc.).

At least one memory 1310 may be configured to store data and/or information associated with video decoder system 1300 . At least one memory 1310 may be a shared resource. For example, video decoder system 1300 may be an element of a larger system (eg, personal computer, mobile device, etc.). Accordingly, at least one memory 1310 may be configured to store data and/or information associated with other elements within the larger system (eg, web browsing or wireless communications).

Controller 1320 may be configured to generate and communicate various control signals to various blocks in video decoder system 1300 . The controller 1320 may be configured to generate control signals to implement the video encoding/decoding techniques described below. According to example embodiments, the controller 1320 may be configured to control the video decoder 145 to decode video frames.

Video decoder 145 may be configured to receive input and output compressed (eg, encoded) video bits 10 of video stream 5 . Video decoder 145 may convert discrete video frames of compressed video bits 10 into video stream 5 . The compressed (eg, encoded) video bits 10 may also be compressed image bits, and thus, the video stream 5 may also be an image.

At least one processor 1305 may be configured to execute computer instructions associated with controller 1320 and/or video decoder 145 . At least one processor 1305 may be a shared resource. For example, video decoder system 1300 may be an element of a larger system (eg, personal computer, mobile device, etc.). Accordingly, at least one processor 1305 may be configured to execute computer instructions associated with other elements within the larger system (eg, web browsing or wireless communication).

In example embodiments, video decoder system 1300 may be implemented as or include a graphics processing unit (GPU) on or in a graphics card and/or chip (eg, an ASIC on a computer motherboard) GPU) is configured to remove the load from the central processing unit (CPU). At least one processor 1305 may be implemented as a GPU configured to process large blocks of video data in parallel. The GPU may be configured to process (eg, decompress) mesh data and generate pixel data from the mesh data. At least one memory 1310 may include video memory and driver software. Video memory may be a frame buffer that stores digital data representing frame images or scenes. Video memory can store digital data before and after GPU processing. The driver software may include a codec configured to decompress the video data. The codec may include implementing a color prediction scheme based on the use of 3D objects as geometric proxies as described herein.

14 shows an example of a computer device 1400 and a mobile computer device 1450 that may be used with the techniques described herein. Computing device 1400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1450 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit implementations of the invention described and/or claimed in this document.

Computing device 1400 includes processor 1402 , memory 1404 , storage device 1406 , high-speed interface 1408 connected to memory 1404 and high-speed expansion port 1410 , and low-speed interface 1412 connected to low-speed bus 1414 and storage device 1406 . Each of the components 1402, 1404, 1406, 1408, 1410, and 1412 are interconnected using various buses, and may be mounted on a common motherboard or otherwise as appropriate. Processor 1402 may process instructions for execution within computing device 1400 , including storing in memory 1404 or on storage device 1406 for display for a GUI on an external input or output device such as display 1416 coupled to high-speed interface 1408 instruction for graphic information. In other implementations, multiple processors and/or multiple buses may be used along with multiple memories and multiple types of memory, as appropriate. Additionally, multiple computing devices 1400 may be connected, with each device providing a portion of the necessary operations (eg, as a server group, a group of blade servers, or a multi-processor system).

Memory 1404 stores information within computing device 1400 . In one embodiment, memory 1404 is one or more volatile memory cells. In another embodiment, memory 1404 is one or more non-volatile memory cells. Memory 1404 may also be another form of computer-readable medium, such as a magnetic or optical disk.

Storage device 1406 is capable of providing mass storage for computing device 1400 . In one embodiment, storage device 1406 may be or include a computer-readable medium such as a floppy disk device, hard disk device, optical disk device or tape device, flash memory or other similar solid state storage devices or arrays of devices, including storage area networks or devices in other configurations. The computer program product may be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as memory 1404 , storage device 1406 , or memory on processor 1402 .

High-speed controller 1408 manages bandwidth-intensive operations of computing device 1400, while low-speed controller 1412 manages lower bandwidth-intensive operations. This distribution of functions is exemplary only. In one embodiment, high-speed controller 1408 is coupled to memory 1404, display 1416 (eg, through a graphics processor or accelerator), and to high-speed expansion port 1410, which can accept various expansion cards (not shown) out). In this embodiment, low-speed controller 1412 is coupled to storage device 1406 and low-speed expansion port 1414 . Low-speed expansion ports, which may include various communication ports (eg, USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input or output devices, such as keyboards, pointing devices, scanners, or Networked devices such as switches or routers.

As shown in the figure, computing device 1400 may be implemented in many different forms. For example, it may be implemented as a standard server 1420, or multiple times in a group of such servers. It can also be implemented as part of rack server system 1424. Furthermore, it may be implemented in a personal computer such as laptop computer 1422. Alternatively, components from computing device 1400 may be combined with other components in a mobile device (not shown) such as device 1450 . Each of such devices may contain one or more of the computing devices 1400, 1450, and the overall system may consist of multiple computing devices 1400, 1450 in communication with each other.

Computing device 1450 includes processor 1452, memory 1464, input or output devices such as display 1454, communication interface 1466, and transceiver 1468, among other components. Device 1450 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 1450, 1452, 1464, 1454, 1466, and 1468 are interconnected using various buses, and several components may be mounted on a common motherboard or otherwise as appropriate.

Processor 1452 may execute instructions within computing device 1450 , including instructions stored in memory 1464 . The processor may be implemented as a chipset of chips including individual and multiple analog and digital processors. The processor may, for example, provide coordination for other components of the device 1450 , such as control of the user interface, applications run by the device 1450 , and wireless communications by the device 1450 .

Processor 1452 may communicate with a user through control interface 1458 and display interface 1456 coupled to display 1454 . Display 1454 may be, for example, a TFT LCD (Thin Film Transistor Liquid Crystal Display) or OLED (Organic Light Emitting Diode) display or other suitable display technology. Display interface 1456 may include appropriate circuitry for driving display 1454 to present graphics and other information to a user. The control interface 1458 may receive commands from the user and convert them for submission to the processor 1452 . Additionally, an external interface 1462 may be provided in communication with the processor 1452 to enable near area communication of the device 1450 with other devices. The external interface 1462 may be provided, for example, for wired communication in some embodiments, or for wireless communication in other embodiments, and multiple interfaces may also be used.

Memory 1464 stores information within computing device 1450 . Memory 1464 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. Expansion memory 674 may also be provided and connected to device 1450 through expansion interface 1472, which may include, for example, a SIMM (Single Inline Memory Module) card interface. Such expanded memory 1474 may provide additional storage space for device 1450, or may also store applications or other information for device 1450. Specifically, expansion memory 1474 may include instructions for performing or supplementing the above-described processes, and may also include security information. Thus, for example, expansion memory 1474 may be provided as a security module for device 1450, and may be programmed with instructions to allow secure use of device 1450. Furthermore, security applications and additional information may be provided via the SIMM card, such as placing identification information on the SIMM card in an uncrackable manner.

The memory may include, for example, flash memory and/or NVRAM memory, as discussed below. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 1464 , expansion memory 1474 , or memory on processor 1452 , such as memory 1464 , expansion memory 1474 , or a computer- or machine-readable medium that may be received, eg, through transceiver 1468 or external interface 1462 .

Device 1450 may communicate wirelessly through communication interface 1466, which may include digital signal processing circuitry as desired. Communication interface 1466 may provide for communication in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, and the like. Such communication may occur through radio frequency transceiver 1468, for example. Additionally, short-range communication may occur, such as using Bluetooth, WiFi, or other such transceivers (not shown). Additionally, GPS (Global Positioning System) receiver module 570 may provide additional navigation and location-related wireless data to device 1450, which may be used by applications running on device 1450 as appropriate.

Device 1450 may also communicate audibly using audio codec 1460, which may receive spoken information from the user and convert it into usable digital information. Audio codec 1460 may likewise generate audible sound for the user, such as through speakers - eg, in a headset of device 1450. Such sounds may include sounds from voice phone calls, may include recorded sounds (eg, voice messages, music files, etc.) and may also include sounds generated by applications operating on device 1450.

As shown in the figure, computing device 1450 may be implemented in many different forms. For example, it can be implemented as a cellular phone 1480. It can also be implemented as part of a smartphone 1482, personal digital assistant or other similar mobile device.

Various implementations of the systems and techniques described herein may be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include embodiments in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose are coupled to receive data and instructions from a storage system, at least one input device and at least one output device, and to send data and instructions to the storage system, at least one input device and at least one output device. Various implementations of the systems and techniques described herein may be implemented as and/or generally referred to herein as circuits, modules of combinable software and hardware aspects as circuits, modules, blocks or systems of combinable software and hardware aspects , block or system. For example, a module may include functions/acts/computer program instructions executed on a processor (eg, a processor formed on a silicon substrate, GaAs substrate, etc.) or some other programmable data processing device.

Some of the above-described example embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations as a sequential process, many of the operations can be performed in parallel, concurrently, or concurrently. Additionally, the order of operations may be rearranged. These processes may be terminated when their operations are complete, but may also have additional steps not included in the figure. These procedures may correspond to methods, functions, programs, subroutines, subroutines, and the like.

The methods discussed above, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a machine- or computer-readable medium, such as a storage medium. The processor can perform the necessary tasks.

Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments. Example embodiments, however, are embodied in many alternative forms and should not be construed as limited to only the embodiments set forth herein.

It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the terms and/or include any and all combinations of one or more of the associated listed items.

It will be understood that when an element is referred to as being connected or coupled to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being directly connected or coupled to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted in a like fashion (eg, between versus directly between, adjacent versus directly adjacent, etc.).

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit example embodiments. As used herein, the singular forms a (a), an (an) and the (the) are intended to include the plural forms as well, unless the context clearly dictates otherwise. It is to be further understood that the terms containing, containing, including and/or including when used herein designate the presence of stated features, integers, steps, operations, elements and/or components, but do not preclude the presence or addition of one or Various other features, integers, steps, operations, elements, components and/or groups thereof.

It should also be noted that, in some alternative implementations, the functions/acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed concurrently or may sometimes be executed in the reverse order, depending upon the functionality or behavior involved.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which example embodiments belong. It should be further understood that, unless expressly so defined herein, terms (eg, those defined in commonly used dictionaries) should be construed as having meanings consistent with their meanings in the context of the relevant art, not Explain in an idealized or overly formal sense.

Portions of the above-described example embodiments and corresponding detailed descriptions have been presented in terms of software or algorithms and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are those used by those of ordinary skill in the art to effectively convey the substance of their work to others of ordinary skill in the art. An algorithm (as the term is used here, and when it is used generally) is considered to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

In the above-described illustrative embodiments, references to acts and symbolic representations of operations (eg, in the form of flowcharts) that may be implemented as program modules or functional processes include performing particular tasks or implementing particular abstract data types and may use existing Routines, programs, objects, components, data structures, etc., are described and/or implemented with existing hardware at the structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate array (FPGA) computers, and the like.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, or as apparent from the discussion, terms such as processing or computing or calculating or determining display refer to the acts and processes of a computer system or similar electronic computing device that Computing devices manipulate and transform data represented as physical, electronic quantities within the registers and memory of a computer system into other physical quantities similarly represented as computer system memory or registers or other such information storage, transmission or display devices data.

Note also that the software-implemented aspects of the example embodiments are typically encoded on some form of non-transitory program storage medium or implemented over some type of transmission medium. The program storage medium may be magnetic (eg, a floppy disk or hard drive) or optical (eg, a compact disk read only memory or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, fiber optics, or some other suitable transmission medium known in the art. Example embodiments are not limited by these aspects of any given implementation.

Finally, it should also be noted that although the appended claims recite specific combinations of features described herein, the scope of the present disclosure is not limited to the specific combinations claimed hereinafter, but instead extends to encompass the features disclosed herein or any combination of embodiments, whether or not at this time the specific combination is specifically recited in the appended claims.

Claims (22)

1. A method, comprising:

receiving a frame of a video;

identifying a three-dimensional (3D) object in the frame;

matching the 3D object with the stored 3D object;

compressing frames of the video using a color prediction scheme based on the 3D object and the stored 3D object; and

storing compressed frames having metadata identifying the 3D object, indicating a location of the 3D object in a frame of the video and indicating an orientation of the 3D object in the frame of the video.

2. The method of claim 1, wherein compressing frames of the video using the color prediction scheme based on the 3D object and the stored 3D object comprises:

generating a first 3D object proxy based on the stored 3D object;

transforming the first 3D object proxy based on the 3D object identified in the frame;

generating a second 3D object proxy based on the stored 3D object;

identifying the 3D object in a key frame of the video;

transforming the second 3D object proxy based on the 3D object identified in the key frame;

mapping color attributes from the 3D object to a transformed first 3D object proxy;

mapping color attributes from the 3D object identified in the keyframe to a transformed second 3D object proxy; and

generating a residual of the 3D object based on the color attributes of the transformed first 3D object proxy and the color attributes of the transformed second 3D object proxy.

3. The method of claim 1, wherein compressing frames of the video using the color prediction scheme based on the 3D object and the stored 3D object comprises:

generating a first 3D object proxy based on the stored 3D object;

transforming the first 3D object proxy based on the 3D object identified in the frame;

generating a second 3D object proxy based on the stored 3D object;

identifying the 3D object in a key frame of the video;

transforming the second 3D object proxy based on the 3D object identified in the key frame;

mapping color attributes from the 3D object to a transformed first 3D object proxy; and

generating a residual of the 3D object based on the color attribute of the transformed first 3D object proxy and a default color attribute of the transformed second 3D object proxy.

4. The method of claim 1, wherein compressing frames of the video using the color prediction scheme based on the 3D object and the stored 3D object comprises:

generating a first 3D object proxy based on the stored 3D object;

encoding the first 3D object proxy using an auto-encoder;

transform the encoded first 3D object proxy based on the 3D object identified in the frame;

generating a second 3D object proxy based on the stored 3D object;

encoding the second 3D object proxy using an auto-encoder;

identifying the 3D object in a key frame of the video;

transform the encoded second 3D object proxy based on the 3D object identified in the key frame;

mapping color attributes from the 3D object to a transformed first 3D object proxy;

mapping color attributes from the 3D object identified in the keyframe to a transformed second 3D object proxy; and

generating a residual of the 3D object based on the color attributes of the transformed first 3D object proxy and the color attributes of the transformed second 3D object proxy.

5. The method of claim 1, wherein compressing frames of the video using the color prediction scheme based on the 3D object and the stored 3D object comprises:

generating a first 3D object proxy based on the stored 3D object;

encoding the first 3D object proxy using an auto-encoder;

transform the encoded first 3D object proxy based on the 3D object identified in the frame;

generating a second 3D object proxy based on the stored 3D object;

encoding the second 3D object proxy using an auto-encoder;

identifying the 3D object in a key frame of the video;

transform the encoded second 3D object proxy based on the 3D object identified in the key frame;

mapping color attributes from the 3D object to a transformed first 3D object proxy; and

generating a residual of the 3D object based on the color attribute of the transformed first 3D object proxy and a default color attribute of the transformed second 3D object proxy.

6. The method of claim 1, further comprising:

prior to storing the 3D object:

identifying at least one 3D object of interest associated with the video;

determining a plurality of mesh attributes associated with the 3D object of interest;

determining a location associated with the 3D object of interest;

determining an orientation associated with the 3D object of interest;

determining a plurality of color attributes associated with the 3D object of interest; and

reducing a number of variables associated with a mesh attribute of the 3D object of interest using an auto-encoder.

7. The method of claim 1, wherein compressing frames of the video comprises determining location coordinates of the 3D object relative to origin coordinates of a background 3D object in a keyframe.

8. The method of claim 1, wherein:

the stored 3D object includes default color attributes, and

the color prediction scheme uses the default color attribute.

9. The method of claim 1, further comprising:

identifying at least one 3D object of interest associated with the video;

generating at least one stored 3D object based on the at least one 3D object of interest, each of the at least one stored 3D object being defined by a mesh comprising a set of points connected by a face, each point storing at least one attribute comprising location coordinates of the respective point; and

storing the at least one stored 3D object in association with the video.

10. A method, comprising:

receiving a frame of a video;

identifying a three-dimensional (3D) object in the frame;

matching the 3D object with the stored 3D object;

decompressing frames of the video using a color prediction scheme based on the 3D object and the stored 3D object; and

rendering frames of the video.

11. The method of claim 10, wherein decompressing frames of the video using the color prediction scheme based on the 3D object and the stored 3D object comprises:

generating a first 3D object proxy based on the stored 3D object;

transforming the first 3D object proxy based on the 3D object identified in the frame;

identifying the 3D object in a key frame of the video;

transforming the second 3D object proxy based on the 3D object identified in the key frame;

mapping color attributes from the 3D object to a transformed first 3D object proxy;

mapping color attributes from the 3D object identified in the keyframe to a transformed second 3D object proxy; and

generating color attributes of the 3D object based on the color attributes of the transformed first 3D object proxy and the color attributes of the transformed second 3D object proxy.

12. The method of claim 10, wherein decompressing frames of the video using the color prediction scheme based on the 3D object and the stored 3D object comprises:

generating a first 3D object proxy based on the stored 3D object;

transforming the first 3D object proxy based on the 3D object identified in the frame;

identifying the 3D object in a key frame of the video;

transforming the second 3D object proxy based on the 3D object identified in the key frame;

mapping color attributes from the 3D object to a transformed first 3D object proxy;

generating color attributes of the 3D object based on the color attributes of the transformed first 3D object proxy and the default color attributes of the transformed second 3D object proxy.

13. The method of claim 10, wherein decompressing frames of the video using the color prediction scheme based on the 3D object and the stored 3D object comprises:

generating a first 3D object proxy based on the stored 3D object;

decoding the first 3D object proxy using an auto-encoder;

transforming the decoded first 3D object proxy based on metadata associated with the 3D object;

generating a second 3D object proxy based on the stored 3D object;

decoding the second 3D object proxy using an auto-encoder;

identifying the 3D object in a key frame of the video;

transforming the decoded second 3D object proxy based on metadata associated with the 3D object identified in the key frame;

mapping color attributes from the 3D object to the transformed first 3D object proxy;

mapping color attributes from the 3D object identified in the keyframe to the transformed second 3D object proxy; and

generating color attributes of the 3D object based on the color attributes of the transformed first 3D object proxy and the color attributes of the transformed second 3D object proxy.

14. The method of claim 10, wherein decompressing frames of the video using the color prediction scheme based on the 3D object and the stored 3D object comprises:

generating a first 3D object proxy based on the stored 3D object;

decoding the first 3D object proxy using an auto-encoder;

transforming the decoded first 3D object proxy based on metadata associated with the 3D object;

generating a second 3D object proxy based on the stored 3D object;

decoding the second 3D object proxy using an auto-encoder;

identifying the 3D object in a key frame of the video;

transforming the decoded second 3D object proxy based on metadata associated with the 3D object identified in the key frame;

mapping color attributes from the 3D object to a transformed first 3D object proxy; and

generating color attributes of the 3D object based on the color attributes of the transformed first 3D object proxy and default attributes of the transformed second 3D object proxy.

15. The method of claim 10, further comprising:

receiving at least one potential representation of a 3D shape;

using machine-trained generative modeling techniques to:

determining a plurality of mesh attributes associated with the 3D shape;

determining a location associated with the 3D shape;

determining an orientation associated with the 3D shape; and

determining a plurality of color attributes associated with the 3D shape, an

Storing the 3D shape as a stored 3D object.

16. The method of claim 10, wherein rendering the frame of the video comprises:

receiving location coordinates of the 3D object relative to origin coordinates of a background 3D object in a keyframe; and

placing the 3D object in the frame using the location coordinates.

17. The method of claim 10, further comprising:

receiving a neural network used by an encoder of an auto-encoder to reduce a number of variables associated with mesh properties, position, orientation, and color properties of at least one 3D object of interest;

regenerating, in a decoder of the auto-encoder, points associated with a mesh of at least one 3D object of interest using the neural network, the regenerating the points including regenerating a position attribute, an orientation attribute, and a color attribute; and

storing the at least one 3D object of interest as a stored 3D object.

18. A method for predicting color changes using a proxy:

generating a first 3D object proxy based on the stored 3D object;

generating a second 3D object proxy based on the stored 3D object;

transforming the first 3D object proxy based on 3D objects identified in frames of a video;

transforming the second 3D object proxy based on the 3D objects identified in key frames of the video;

mapping color attributes from the 3D object identified in the frame of the video to a transformed first 3D object proxy;

mapping color attributes from the 3D object identified in the keyframe to a transformed second 3D object proxy; and

generating color data of the 3D object based on the color attributes of the transformed first 3D object proxy and the color attributes of the transformed second 3D object proxy.

19. The method of claim 18, further comprising:

encoding the first 3D object agent using an auto-encoder prior to transforming the first 3D object agent; and

encoding the second 3D object agent using the auto-encoder prior to transforming the second 3D object agent.

20. The method of claim 18, further comprising:

decoding the first 3D object proxy using an auto-encoder after transforming the first 3D object proxy; and

decoding the second 3D object proxy using the auto-encoder after transforming the second 3D object proxy.

21. The method of claim 18, wherein generating color data for the 3D object comprises subtracting the color attribute of the transformed first 3D object proxy from the color attribute of the transformed second 3D object proxy.

22. The method of claim 18, wherein generating color data for the 3D object comprises adding color attributes of the transformed first 3D object proxy to color attributes of the transformed second 3D object proxy.

Images