WO2024060161A1 - Encoding method, decoding method, encoder, decoder and storage medium - Google Patents

Abstract

Provided in the embodiments of the present application are an encoding method and a decoding method. The encoding method comprises: inputting geometric information of a node of an ith layer into a first embedded network, so as to determine an embedding feature of the node of the ith layer; inputting the embedding feature of the node of the ith layer into an encoding network, so as to determine a latent variable of the node of the ith layer; according to the latent variable of the node of the ith layer, determining a reconstruction value of the residual of the node of the ith layer, and according to the reconstruction value of the residual of the node of the ith layer, determining a reconstruction value of the latent variable of the node of the ith layer; and inputting the reconstruction value of the latent variable of the node of the ith layer into a first decoding network, so as to generate a code stream of the geometric information of the node of the ith layer. The decoding method comprises: decoding a code stream, so as to determine a reconstruction value of the residual of a node of an ith layer; according to the reconstruction value of the residual of the node of the ith layer, determining a reconstruction value of a latent variable of the node of the ith layer; and inputting the reconstruction value of the latent variable of the node of the ith layer into a second decoding network, so as to determine geometric information of the node of the ith layer.

Description

Coding and decoding methods, encoders, decoders and storage media Technical field

The embodiments of the present application relate to the field of point cloud compression technology, and in particular, to a coding and decoding method, an encoder, a decoder, and a storage medium.

Background technique

Currently, OctSqueeze is a common point cloud entropy model used in Laser Radar (LIDAR). Among them, OctSqueeze is a LIDAR point cloud octree entropy model based on ancestor nodes, and OctAttention is an expansion of OctSqueeze technology. A LIDAR point cloud octree entropy model based on ancestor sibling nodes.

However, the OctSqueeze scheme assumes that neighbor nodes are conditionally independent when their parent nodes are known. This is usually a wrong assumption and will affect the encoding and decoding efficiency. In the OctAttention scheme, since the context window keeps sliding during the decoding process, the model needs to be continuously run to update the semantic information of the context window. This process is not parallel and therefore has extremely high time complexity.

Contents of the invention

The embodiments of the present application provide a coding and decoding method, an encoder, a decoder and a storage medium, which can improve the efficiency of point cloud coding and decoding while improving the compression performance.

The technical solutions of the embodiments of this application can be implemented as follows:

In a first aspect, embodiments of the present application provide an encoding method applied to an encoder. The encoder includes a first embedding network, an encoding network, and a first decoding network. The method includes:

Input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network, and determine the embedding characteristics of the i-th layer node; wherein i is greater than 2;

Input the embedded features of the i-th layer node into the encoding network to determine the latent variables of the i-th layer node;

Determine the reconstruction value of the residual of the i-th layer node based on the latent variable of the i-th layer node, and determine the reconstruction of the latent variable of the i-th layer node based on the reconstruction value of the residual of the i-th layer node value;

The reconstructed value of the latent variable of the i-th layer node is input to the first decoding network to generate a code stream of the geometric information of the i-th layer node.

In a second aspect, embodiments of the present application provide a decoding method, applied to a decoder, where the decoder includes a second decoding network, and the method includes:

Decode the code stream and determine the reconstruction value of the residual of the i-th layer node of the point cloud; where i is greater than 2;

Determine a reconstruction value of a latent variable of the i-th layer node according to the reconstruction value of the residual of the i-th layer node;

The reconstructed value of the latent variable of the i-th layer node is input to the second decoding network to determine the geometric information of the i-th layer node.

In a third aspect, embodiments of the present application provide an encoder, which includes a first determining unit, a coding unit, and a generating unit; wherein,

The first determination unit is configured to input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network to determine the embedding features of the i-th layer node; wherein i is greater than 2; input the embedding features of the i-th layer node into the encoding network to determine the latent variables of the i-th layer node; determine the reconstructed value of the residual of the i-th layer node according to the latent variable of the i-th layer node, and determine the reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node;

The coding unit is configured to write the reconstructed value of the residual of the i-th layer node into the code stream;

The generating unit is configured to input the reconstructed value of the latent variable of the i-th layer node to the first decoding network and generate a code stream of the geometric information of the i-th layer node.

In a fourth aspect, embodiments of the present application provide an encoder, which includes a first memory and a first processor; wherein,

The first memory is used to store a computer program capable of running on the first processor;

The first processor is configured to execute the encoding method as described above when running the computer program.

In a fifth aspect, embodiments of the present application provide a decoder, which includes a decoding unit and a second determination unit; wherein,

The decoding unit is configured to decode the code stream;

The second determination unit is configured to determine the reconstruction value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; determined based on the reconstruction value of the residual of the i-th layer node The reconstructed value of the latent variable of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node to the second decoding network to determine the geometric information of the i-th layer node.

In a sixth aspect, embodiments of the present application provide a decoder, the decoder including a second memory and a second processor; wherein,

The second memory is used to store a computer program capable of running on the second processor;

The second processor is configured to execute the decoding method as described above when running the computer program.

In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the decoding method as described above, or implements the encoding method as described above.

Embodiments of the present application provide an encoding and decoding method, an encoder, a decoder, and a storage medium. On the encoding side, the encoder includes a first embedding network, an encoding network, and a first decoding network. The encoder converts point clouds into octrees. The geometric information of the i-th layer node obtained by the division is input to the first embedding network, and the embedding characteristics of the i-th layer node are determined; where i is greater than 2; the embedding characteristics of the i-th layer node are input to the Encoding network, determine the latent variable of the i-th layer node; determine the reconstruction value of the residual of the i-th layer node according to the latent variable of the i-th layer node, and determine the reconstruction value of the residual of the i-th layer node according to the The reconstruction value determines the reconstruction value of the latent variable of the i-th layer node; the reconstruction value of the latent variable of the i-th layer node is input to the first decoding network to generate a code of the geometric information of the i-th layer node. flow. At the decoding end, the decoder includes a second decoding network. The decoder decodes the code stream and determines the reconstruction value of the residual of the i-th layer node of the point cloud; where i is greater than 2; according to the residual of the i-th layer node The reconstruction value determines the reconstruction value of the latent variable of the i-th layer node; the reconstruction value of the latent variable of the i-th layer node is input to the second decoding network to determine the geometric information of the i-th layer node. It can be seen that in the embodiments of the present application, when encoding and decoding point clouds, based on the constructed octree structure, latent variables can be used to represent the spatial correlation between nodes at a unified level in the octree. , ultimately achieving higher compression performance; at the same time, based on the constructed network, parallel encoding and decoding processes are performed on nodes at the same level of the octree, which improves encoding and decoding efficiency and further improves compression performance.

Description of the drawings

Figure 1 is a schematic diagram of a point cloud encoding and decoding network architecture provided by an embodiment of the present application;

Figure 2 is a schematic flowchart 1 of the implementation of the encoding method proposed in the embodiment of the present application;

Figure 3 is a schematic diagram of the network structure of the embedded network;

Figure 4 is a schematic diagram of the implementation of the embedded network;

Figure 5 is a schematic diagram of the network structure of the coding network;

Figure 6 is a schematic diagram of the implementation of the encoding network;

Figure 7 is a schematic diagram of the network structure of the subtraction network;

Figure 8 is a schematic diagram of the implementation of the subtraction network;

FIG9 is a schematic diagram of the network structure of an addition network;

Figure 10 is a schematic diagram of the implementation of the addition network;

Figure 11 is a schematic diagram of the network structure of the decoding network;

Figure 12 is a schematic diagram of the implementation of the decoding network;

FIG13 is a second schematic diagram of the implementation flow of the encoding method proposed in the embodiment of the present application;

Figure 14 is a schematic diagram 2 of the network structure of the decoding network;

Figure 15 is the implementation diagram 2 of the decoding network;

Figure 16 is a schematic diagram of the overall framework for executing the encoding method;

FIG17 is a schematic diagram of a first implementation flow of a decoding method proposed in an embodiment of the present application;

Figure 18 is a schematic flow chart 2 of the implementation of the decoding method proposed by the embodiment of the present application;

Figure 19 is a schematic diagram of the overall framework for executing the decoding method;

FIG20 is a schematic diagram of the structure of the encoder;

Figure 21 is a schematic diagram 2 of the structure of the encoder;

Figure 22 is a schematic diagram of the structure of the decoder;

Figure 23 is a schematic diagram 2 of the structure of the decoder.

Detailed ways

In order to understand the characteristics and technical content of the embodiments of the present application in more detail, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not intended to limit the embodiments of the present application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or a different subset of all possible embodiments, and Can be combined with each other without conflict.

It should also be pointed out that the terms "first\second\third" involved in the embodiments of this application are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understandable that "first\second\ The third "specific order or sequence may be interchanged where permitted, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein.

Point Cloud is a three-dimensional representation of the surface of an object. Through collection equipment such as photoelectric radar, lidar, laser scanner, and multi-view camera, the point cloud (data) of the surface of the object can be collected.

Point cloud is a set of discrete points randomly distributed in space that expresses the spatial structure and surface properties of a three-dimensional object or scene. Figure 1A shows a three-dimensional point cloud image and Figure 1B shows a partial enlargement of the three-dimensional point cloud image. It can be seen that the point cloud surface is composed of densely distributed points.

Two-dimensional images have information expression at each pixel point, and the distribution is regular, so there is no need to record its position information additionally; however, the distribution of points in point clouds in three-dimensional space is random and irregular, so it is necessary to record the position of each point in space in order to fully express a point cloud. Similar to two-dimensional images, each position in the acquisition process has corresponding attribute information, usually RGB color values, and the color value reflects the color of the object; for point clouds, in addition to color information, the attribute information corresponding to each point is also commonly the reflectance value, which reflects the surface material of the object. Therefore, the points in the point cloud can include the position information of the point and the attribute information of the point. For example, the position information of the point can be the three-dimensional coordinate information (x, y, z) of the point. The position information of the point can also be called the geometric information of the point. For example, the attribute information of the point can include color information (three-dimensional color information) and/or reflectance (one-dimensional reflectance information r), etc. For example, the color information can be information on any color space. For example, the color information can be RGB information. Among them, R represents red (Red, R), G represents green (Green, G), and B represents blue (Blue, B). For another example, the color information may be luminance and chrominance (YCbCr, YUV) information, where Y represents brightness (Luma), Cb (U) represents blue color difference, and Cr (V) represents red color difference.

According to the point cloud obtained based on the principle of laser measurement, the points in the point cloud can include the three-dimensional coordinate information of the point and the reflectivity value of the point. For another example, in a point cloud obtained according to the principle of photogrammetry, the points in the point cloud may include the three-dimensional coordinate information of the point and the three-dimensional color information of the point. For another example, a point cloud is obtained by combining the principles of laser measurement and photogrammetry. The points in the point cloud may include the three-dimensional coordinate information of the point, the reflectivity value of the point, and the three-dimensional color information of the point.

Point clouds can be divided into:

Static point cloud: that is, the object is stationary and the device that obtains the point cloud is also stationary;

Dynamic point cloud: The object is moving, but the device that obtains the point cloud is stationary;

Dynamically acquire point clouds: The device that acquires point clouds is in motion.

For example, point clouds are divided into two categories according to their uses:

Category 1: Machine perception point cloud, which can be used in scenarios such as autonomous navigation systems, real-time inspection systems, geographic information systems, visual sorting robots, and rescue and disaster relief robots;

Category 2: Human eye perception point cloud, which can be used in point cloud application scenarios such as digital cultural heritage, free-viewpoint broadcasting, three-dimensional immersive communication, and three-dimensional immersive interaction.

Point clouds can flexibly and conveniently express the spatial structure and surface properties of three-dimensional objects or scenes, and because point clouds are obtained by directly sampling real objects, they can provide a strong sense of reality while ensuring accuracy, so they are widely used and their scope Including virtual reality games, computer-aided design, geographic information systems, automatic navigation systems, digital cultural heritage, free-viewpoint broadcasting, three-dimensional immersive telepresence, three-dimensional reconstruction of biological tissues and organs, etc.

Point cloud collection mainly has the following methods: computer generation, 3D laser scanning, 3D photogrammetry, etc. Computers can generate point clouds of virtual three-dimensional objects and scenes; 3D laser scanning can obtain point clouds of static real-world three-dimensional objects or scenes, and can obtain millions of point clouds per second; 3D photogrammetry can obtain dynamic real-world three-dimensional objects or scenes Point clouds can obtain tens of millions of point clouds per second. These technologies reduce the cost and time period of point cloud data acquisition and improve the accuracy of the data. Changes in the way of obtaining point cloud data have made it possible to obtain large amounts of point cloud data. With the growth of application requirements, the processing of massive 3D point cloud data has encountered bottlenecks limited by storage space and transmission bandwidth.

For example, taking a point cloud video with a frame rate of 30 frames per second (fps), the number of points in each frame of the point cloud is 700,000, and each point has coordinate information xyz (float) and color information RGB (uchar), Then the data volume of 10s point cloud video is approximately 0.7 million×(4Byte×3+1Byte×3)×30fps×10s=3.15GB, where 1Byte is 8bit, and the YUV sampling format is 4:2:0, and the frame rate The data volume of 1280×720 2D video at 24fps for 10s is about 1280×720×12bit×24fps×10s≈0.33GB, and the data volume of two-view 3D video for 10s is about 0.33×2=0.66GB. It can be seen that the data volume of point cloud video far exceeds the data volume of 2D video and 3D video of the same length. Therefore, in order to better realize data management, save server storage space, and reduce transmission traffic and transmission time between the server and the client, point cloud compression has become a key issue to promote the development of the point cloud industry.

In other words, since the point cloud is a collection of massive points, storing the point cloud will not only consume a lot of memory, but is also not conducive to transmission. There is not such a large bandwidth to support the direct transmission of the point cloud at the network layer without compression. Therefore, , the point cloud needs to be compressed.

Currently, the point cloud coding framework that can compress point clouds can be the Geometry-based Point Cloud Compression (G-PCC) codec framework provided by the Moving Picture Experts Group (MPEG) Or the Video-based Point Cloud Compression (V-PCC) codec framework, or the AVS-PCC codec framework provided by AVS. The G-PCC encoding and decoding framework can be used to compress the first type of static point cloud and the third type of dynamic point cloud, and the V-PCC encoding and decoding framework can be used to compress the second type of dynamic point cloud. The G-PCC encoding and decoding framework is also called point cloud codec TMC13, and the V-PCC encoding and decoding framework is also called point cloud codec TMC2.

This embodiment of the present application provides a network architecture of a point cloud encoding and decoding system that includes a decoding method and an encoding method. Figure 1 is a schematic diagram of the network architecture of a point cloud encoding and decoding system provided by an embodiment of the present application. As shown in Figure 1, the network architecture includes one or more electronic devices 13 to 1N and a communication network 01, wherein the electronic devices 13 to 1N can perform video interaction through the communication network 01. During the implementation process, electronic devices may be various types of devices with point cloud encoding and decoding functions. For example, the electronic devices may include mobile phones, tablet computers, personal computers, personal digital assistants, navigators, digital phones, and video phones. , televisions, sensing equipment, servers, etc., are not limited by the embodiments of this application. Among them, the decoder or encoder in the embodiment of the present application can be the above-mentioned electronic device.

Among them, the electronic device in the embodiment of the present application has a point cloud encoding and decoding function, and generally includes a point cloud encoder (ie, encoder) and a point cloud decoder (ie, decoder).

In the embodiment of this application, OctSqueeze is a LIDAR point cloud octree entropy model based on ancestor nodes. This technology is applied to the LIDAR point cloud entropy model and mainly consists of the following parts:

(1), encoder

The encoder first performs octree construction on the point cloud. Then, for each node of the octree, its depth, parent node occupancy code, and coordinates are selected as context information, and its corresponding features are obtained through a multi-layer perceptron network. After that, k iterations are performed. For the k-th iteration, the characteristics of the current node and the characteristics of its parent node are spliced, and the characteristics of the current node at the k-th iteration are obtained through the k-th multi-layer perceptron. For the features after the k-th iteration of each node, the probability of each occupancy code is obtained through a 256-dimensional softmax layer, and the current occupancy code is losslessly encoded using arithmetic coding and the probability of the current occupancy code.

(2) Code stream

The code stream of OctSqueeze technology consists of the code stream corresponding to the occupied code of each layer of nodes.

(3), decoder

The decoder decodes the octree sequentially from lower layers to higher layers in units of layers. The decoding process is exactly the same as the encoding process except for the node order.

(4), loss function

The loss function is the cross entropy of the predicted node probability and the real occupancy code, as follows:

Among them, N is the number of octree nodes, _xi represents the i-th node, j is the dimension in 256 dimensions, and p _ij is the probability of the corresponding occupancy code.

In the embodiment of this application, OctAttention is a LIDAR point cloud octree entropy model based on ancestor sibling nodes. This technology is applied to the LIDAR point cloud entropy model and is an expansion of the OctSqueeze technology. It mainly consists of the following parts:

(1), encoder

This technique encodes and decodes point clouds in breadth-first order, using a context window to store encoded/decoded nodes. For the current node to be encoded, an attention-based network is used to extract the information of the context window, and the softmax function is used to obtain the probability of each occupied code of the current node. Losslessly encode the current node using arithmetic coding and add the current node to the context window.

(2), code stream

The code stream of this technology consists of the code streams of each node arranged in breadth-first order.

(3), decoder

The decoding process is the same as the encoding process.

(4), loss function

The loss function is the cross entropy of the predicted node probability and the real occupancy code, as follows:

Where N is the number of octree nodes, _xi represents the i-th node, j is the dimension in 256 dimensions, and _pij is the probability of the corresponding occupied code.

It is understandable that the above-mentioned OctSqueeze scheme assumes that neighbor nodes are conditionally independent when the parent node is known. This is usually a wrong assumption and will lead to suboptimal coding efficiency. In the above OctAttention scheme, since the context window keeps sliding during the decoding process, the model needs to be continuously run to update the semantic information of the context window. This process is not parallel and therefore has extremely high time complexity.

It can be seen that common point cloud encoding and decoding methods have the disadvantage of high complexity, which affects the compression performance and reduces the encoding and decoding efficiency to a certain extent.

In order to solve the above problems, embodiments of the present application provide a coding and decoding method, an encoder, a decoder, and a storage medium. On the encoding side, the encoder includes a first embedding network, a coding network, and a first decoding network. The encoder will The geometric information of the i-th layer node obtained by Yuanyi octree division is input to the first embedding network to determine the embedding characteristics of the i-th layer node; where i is greater than 2; the embedding characteristics of the i-th layer node are input to the encoding network, Determine the latent variable of the i-th layer node; determine the reconstructed value of the residual of the i-th layer node based on the latent variable of the i-th layer node, and determine the reconstructed value of the residual of the i-th layer node based on the reconstructed value of the residual of the i-th layer node. Reconstruction value; input the reconstruction value of the latent variable of the i-th layer node to the first decoding network to generate a code stream of the geometric information of the i-th layer node. At the decoding end, the decoder includes a second decoding network. The decoder decodes the code stream and determines the reconstruction value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; where i is greater than 2; according to the i-th layer node The reconstructed value of the residual determines the reconstructed value of the latent variable of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input to the second decoding network to determine the geometric information of the i-th layer node. It can be seen that in the embodiments of the present application, when encoding and decoding point clouds, based on the constructed octree structure, latent variables can be used to represent the spatial correlation between nodes at a unified level in the octree. , ultimately achieving higher compression performance; at the same time, based on the constructed network, parallel encoding and decoding processes are performed on nodes at the same level of the octree, which improves encoding and decoding efficiency and further improves compression performance.

Each embodiment of the present application will be described in detail below with reference to the accompanying drawings.

The embodiment of the present application proposes a point cloud encoding method, which can be applied to an encoder, where the encoder can include a first embedding network, an encoding network, and a first decoding network.

It should be noted that in embodiments of the present application, the encoder may be provided with a multi-layer network, where at least one layer of the multi-layer network may include a first embedding network, an encoding network, and a first decoding network.

Exemplarily, in the embodiment of the present application, the encoder may be provided with a 5-layer network, where one layer of the network or a multi-layer network in the 5-layer network may include a first embedding network, a coding network and a first Decoding network.

Figure 2 is a schematic flow chart of the implementation of the encoding method proposed by the embodiment of the present application. As shown in Figure 2, the method for the encoder to perform encoding processing may include the following steps:

Step 101: Input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network to determine the embedding characteristics of the i-th layer node; where i is greater than 2.

In the embodiment of the present application, the encoder can first input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network, so that the embedding characteristics of the i-th layer node can be determined.

It should be noted that, in the embodiment of the present application, i may be an integer greater than 2.

It can be understood that, in the embodiment of the present application, the i-th layer network in the multi-layer network can perform parallel encoding processing on the geometric information of the i-th layer node.

Further, in the embodiment of the present application, when performing geometric encoding processing on the point cloud, an octree may be constructed first. Among them, the octree structure is used to recursively divide the point cloud space into eight sub-blocks of the same size, and the occupied code words of each sub-block are judged. When the sub-block does not contain points, it is recorded as empty, otherwise it is recorded as empty. is non-empty, the occupied codeword information of all blocks is recorded in the last layer of recursive division, and geometric encoding is performed; on the one hand, the geometric information expressed through the octree structure can further form a geometric code stream, on the other hand, it can be geometrically encoded During the reconstruction process, the reconstructed geometric information is used as additional information for attribute encoding.

It can be understood that in the embodiment of the present application, for the constructed octree, the parent node layer can be defined as a high-level layer and the child node layer as a low-level layer; the parent node layer can also be defined as a low-level layer and the child node layer can be defined as a low-level layer. A layer is defined as a high-level layer.

The encoding sequence in which the encoder performs the encoding process may be in the order from the root node to the leaf node of the constructed octree. Correspondingly, on the decoding side, the decoding order may be in the order from the leaf nodes to the root node of the constructed octree. For example, the root node layer of the constructed octree is the i-th layer, and the leaf node layers are from the i-1 to the first layer in sequence. Then the order of encoding processing is from the i-th layer to the first layer, and the order of decoding processing is The opposite of encoding, that is, from the first layer to the i-th layer.

It can be understood that in an embodiment of the present application, the geometric information of the i-th layer nodes of the point cloud can represent the geometric information of the i-th layer nodes, such as the position coordinates of the nodes, and can also represent the occupancy codeword situation of the i-th layer nodes, that is, the occupancy situation (placeholder information).

That is to say, in the embodiment of the present application, the geometric information of the i-th layer node includes both the position coordinate information of the node and the occupancy information of the node.

It can be understood that in the embodiment of the present application, at the encoding end, the input of the first embedding network is the geometric information of the i-th layer node, wherein the geometric information of the node as the input of the first embedding network may also include nodes The position coordinate information and the placeholder information of the node; and the code stream of the geometric information output by the first decoding network mainly includes the code stream of the placeholder information of the node, which may include the code stream of the node's position coordinate information, or may not include Code stream of node position coordinate information.

It can be seen that in the embodiment of the present application, the embedding network and the encoding network need to refer to the place information and position coordinate information of the node at the same time, while the decoding network focuses on generating a code stream of the place information.

Further, in the embodiment of the present application, the first embedding network may be an occupied embedding network embedding. Among them, the first embedding network can be used to map the octree nodes from the 256-dimensional one-hot vector to the 16-dimensional continuous feature space, so that similar occupancy conditions in the space are also similar in the feature space.

It should be noted that in the embodiment of the present application, the first embedding network may include a sparse convolutional network.

For example, in the embodiment of the present application, the network structure of the first embedding network may be composed of a multi-layer sparse convolutional network. For example, Figure 3 is a schematic diagram of the network structure of the embedding network. As shown in Figure 3, the network structure of the first embedding network can be a three-layer sparse convolutional network.

Further, in embodiments of the present application, after the encoder inputs the geometric information of the i-th layer node in the point cloud to the first embedding network, the encoder can output the embedding feature of the i-th layer node. Among them, the embedded features of the i-th layer node can determine the occupancy status of the i-th layer node after it is mapped to the feature space.

That is to say, in the embodiment of the present application, the embedded characteristics of the i-th layer node can determine the occupancy status of the i-th layer node.

Exemplarily, in the embodiment of the present application, Figure 4 is a schematic diagram of the implementation of the embedding network. As shown in Figure 4, x ^(l) represents the geometric information of the l-th layer node, and e ^(l) represents the first embedding network output. The embedded features of the l-th layer node. For the l-th layer node, the encoder can input the geometric information x ^(l) of the l-th layer node into the first embedding network, so that the corresponding l-th layer node can be obtained through the first embedding network. Embedding features e ^(l) of layer nodes.

Step 102: Input the embedded features of the i-th layer node into the encoding network to determine the latent variables of the i-th layer node.

In the embodiment of the present application, after the encoder inputs the geometric information of the i-th layer node in the point cloud to the first embedding network and outputs the embedding feature of the i-th layer node, it can further add the embedding feature of the i-th layer node Input to the encoding network to determine the latent variables of the i-th layer node.

Further, in the embodiment of the present application, the encoding network may be an encoder network. Among them, the encoding network can be used to determine the spatial correlation between the latent variables of the previous layer node of the current layer and the embedded features of the current layer node, thereby obtaining the latent variables of the current layer node. That is, through the encoding network, the latent variable of the i-th layer node can be determined based on the embedding characteristics of the i-th layer node output by the first embedding network.

It should be noted that, in the embodiment of the present application, the latent variable of the i-th layer node can represent the correlation between the i-th layer sibling nodes. That is to say, in this application, the encoder can make full use of hierarchical latent variables to capture the correlation of LIDAR point clouds.

It should be noted that in the embodiment of the present application, the encoding network can be implemented through sparse convolution, which may specifically include sparse convolution network, ReLU activation function, initial residual network (Inception Resnet, IRN), and feature fusion network Concatenate .

For example, in the embodiment of the present application, the network structure of the encoding network may be composed of a sparse convolution network, a ReLU activation function, an initial residual network, and a Concatenate network. For example, Figure 5 is a schematic diagram of the network structure of the encoding network. As shown in Figure 5, the network structure of the encoding network can be a sparse convolution layer with a convolution kernel size of 2 and a stride of 2, followed by a ReLU activation function, and then Three layers of IRN are connected, and after passing through the Concatenate network, a layer of sparse convolutional network is finally connected.

Furthermore, in an embodiment of the present application, after the first embedding network outputs the embedding features of the i-th layer nodes, the encoder can continue to input the embedding features of the i-th layer nodes into the encoding network, and then the latent variables of the i-th layer nodes can be output through the encoding network.

It should be noted that in the embodiment of the present application, when the coding network uses the embedded features of the current layer node to determine the latent variables of the current layer node, it needs to be combined with the latent variables of the previous layer node. Among them, the previous layer of the current layer can be understood as the parent node layer of the current node.

It can be understood that in the embodiment of the present application, for the layer above the current layer, after the coding network in the previous layer determines the corresponding latent variable, the latent variable of the node in the previous layer can be input to the current layer. The coding network of the current layer can determine the spatial correlation between the latent variables of the previous layer nodes and the embedded features of the current layer nodes, thereby obtaining the latent variables of the current layer nodes.

That is to say, in the embodiment of the present application, the input of the coding network of the i-th layer network may include the latent variable of the i+1-th layer node output by the coding network of the i+1-th layer network.

Correspondingly, in the embodiment of the present application, after determining the latent variables of the i-th layer node, the coding network of the i-th layer can also input the latent variables of the i-th layer node to the coding network of the i-1th layer network, That is, the input of the coding network of the i-1th layer network may include the latent variables of the i-th layer node output by the coding network of the i-th layer network.

That is to say, in the embodiment of the present application, the encoder can input the embedded features of the i-th layer node and the latent variables of the i+1-th layer node into the encoding network of the i-th layer network, so that the i-th layer node can be determined latent variables. Among them, the embedded features of the i-th layer node are output by the first embedding unit of the i-th layer network, and the latent variables of the i+1-th layer node are output by the encoding network of the i+1-th layer network.

Exemplarily, in the embodiment of the present application, Figure 6 is a schematic diagram of the implementation of the coding network. As shown in Figure 6, e ^(l) represents the embedding feature of the l-th layer node output by the first embedding network, f ^(l) Represents the latent variable of the l-th layer node, f ^(l+1) represents the latent variable of the l+1-th layer node, for the l-th layer node, the encoder can explore the spatial correlation between f ^(l+1) and e ^(l) property, and output the latent variable f ^(l) of the lower layer (layer l). For example, a sparse tensor can be used to represent an octree, where the coordinate matrix of the sparse tensor represents the node coordinates of the octree; the attribute matrix of the sparse tensor represents the node occupancy code of the octree. The encoding network can be implemented using sparse convolution, where the downsampling of f ^(l) is implemented by a sparse convolution layer with a convolution kernel size of 2 and a stride of 2, followed by a ReLU activation function. After that, the initial residual network is used to aggregate the sibling node features and output to the Concatenate network for feature fusion with e ^(l) . Finally, a sparse convolution layer is used to fuse e ^(l) with the downsampled f ^{(l+ 1)} , obtain the latent variable of the low-level node, that is, the latent variable f ^(l) of the l+1th level node.

Step 103: Determine the reconstructed value of the residual of the i-th node based on the latent variable of the i-th node, and determine the reconstructed value of the latent variable of the i-th node based on the reconstructed value of the residual of the i-th node.

In the embodiment of the present application, after the encoder inputs the embedded features of the i-th layer node into the encoding network and determines the latent variables of the i-th layer node, it can further determine the i-th layer node's latent variables based on the latent variables of the i-th layer node. The reconstructed value of the residual can then determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node. At the same time, the reconstructed value of the residual of the i-th layer node can be written into the code stream.

It should be noted that in the embodiment of the present application, when the encoder determines the reconstructed value of the residual of the i-th layer node based on the latent variable of the i-th layer node, the encoder may first determine the reconstructed value of the residual of the i-th layer node based on the predicted value of the latent variable of the i-th layer node. and the latent variables of the i-th layer node to determine the residual of the i-th layer node; then the residual of the i-th layer node can be quantified, and then the reconstructed value of the residual of the i-th layer node can be determined.

It can be understood that in the embodiment of the present application, the encoder can write the reconstructed value of the residual of the i-th layer node into the code stream and transmit it to the decoder. In other words, the encoder can decompose and compress the latent variables using residual coding.

Further, in the embodiment of the present application, the predicted value of the latent variable of the i-th layer node can be obtained first. Among them, the predicted value of the latent variable of the i-th layer node can be predicted by the first decoding network of the next layer network of the current layer, that is, it can be the i-th layer output by the first decoding network in the i-1 layer network. The predicted value of the latent variable of the layer node.

It should be noted that in the embodiment of the present application, the encoder may also include a subtraction network, where the subtraction network may be used to determine the residual of the latent variable.

That is to say, in the embodiment of the present application, at least one layer of the multi-layer network set by the encoder may also include a subtraction network. Among them, the subtraction network is a soft subtraction network, which can be used to replace common hard subtraction processing.

For example, in the embodiment of the present application, based on the subtraction network, the residual can be determined through the following formula:

代表第l层节点的潜变量的预测值。 Among them, _gs represents the operation of a sparse convolutional layer after concatenation, r ^(l) represents the residual of the l-th layer node, and f ^(l) represents the latent variable of the l-th layer node.

Represents the predicted value of the latent variable of the l-th layer node.

For example, in the embodiment of the present application, when the encoder determines the residual of the i-th layer node, the predicted value of the latent variable of the i-th layer node and the latent variable of the i-th layer node can be input to the subtraction network, Then the residual of the i-th layer node can be obtained.

It should be noted that in the embodiment of the present application, the subtraction network may include a sparse convolution network and a feature fusion network Concatenate.

Exemplarily, in the embodiment of the present application, Figure 7 is a schematic diagram of the network structure of the subtraction network. As shown in Figure 7, the network structure of the subtraction network can be a sparse convolution with a convolution kernel size of 3 and a step size of 1. Accumulated layer, followed by Concatenate network.

代表第l层节点的潜变量的预测值，r ^(l)代表第l层节点的残差。其中，f ^(l)和

在经过Concatenate网络的融合后，可以通过稀疏卷积层的操作，最终输出第l层节点的残差r ^(l)。 Exemplarily, in the embodiment of the present application, Figure 8 is a schematic diagram of the implementation of the subtraction network. As shown in Figure 8, f ^(l) represents the latent variable of the l-th layer node,

represents the predicted value of the latent variable of the l-th layer node, and r ^(l) represents the residual of the l-th layer node. Among them, f ^(l) and

After the fusion of the Concatenate network, the residual r ^(l) of the l-th layer node can be finally output through the operation of the sparse convolution layer.

Further, in the embodiment of the present application, after the encoder performs quantization processing on the obtained residual of the i-th layer node and determines the reconstruction value of the residual of the i-th layer node, it can further calculate the residual value of the i-th layer node based on the residual value of the i-th layer node. The difference between the reconstructed value and the predicted value of the latent variable of the i-th layer node determines the reconstructed value of the latent variable of the i-th layer node. Wherein, the predicted value of the latent variable of the i-th layer node may be output by the first decoding network in the i-1th layer.

It should be noted that in the embodiment of the present application, the encoder may also include an additive network, where the additive network may be used to determine the reconstructed value of the latent variable.

That is to say, in the embodiment of the present application, at least one layer of the multi-layer network set by the encoder may also include an additive network. Among them, the addition network is a soft addition network, which can be used to replace common hard addition processing.

Illustratively, in the embodiment of the present application, based on the additive network, the determination of the predicted value of the latent variable can be achieved through the following formula:

代表第l层节点的残差的重建值，

代表第l层节点的潜变量的重建值，

代表第l层节点的潜变量的预测值。 Among them, g _a represents the operation of a sparse convolution layer after splicing,

Represents the reconstructed value of the residual of the l-th layer node,

Represents the reconstructed value of the latent variable of the l-th layer node,

Represents the predicted value of the latent variable of the l-th layer node.

For example, in the embodiment of the present application, when the encoder determines the reconstructed value of the latent variable of the i-th layer node, the encoder may first combine the reconstructed value of the residual of the i-th layer node with the reconstructed value of the latent variable of the i-th layer node. The predicted value is input to the additive network, and the reconstructed value of the latent variable of the i-th layer node can be obtained.

It should be noted that in the embodiment of the present application, the additive network may include a sparse convolutional network and a feature fusion network Concatenate.

Exemplarily, in the embodiment of the present application, Figure 9 is a schematic diagram of the network structure of the additive network. As shown in Figure 9, the network structure of the additive network can be a sparse convolution with a convolution kernel size of 3 and a step size of 1. Accumulated layer, followed by Concatenate network.

代表第l层节点的残差的重建值，

代表第l层节点的潜变量的重建值，

代表第l层节点的潜变量的预测值。其中，

和

在经过Concatenate网络的融合后，可以通过稀疏卷积层的操作，最终输出第l层节点的潜变量的重建值

Exemplarily, in the embodiment of the present application, Figure 10 is a schematic diagram of the implementation of the addition network, as shown in Figure 10,

Represents the reconstructed value of the residual of the l-th layer node,

Represents the reconstructed value of the latent variable of the l-th layer node,

Represents the predicted value of the latent variable of the l-th layer node. in,

and

After the fusion of the Concatenate network, the reconstructed value of the latent variable of the l-th layer node can be finally output through the operation of the sparse convolution layer.

It can be understood that in the embodiment of the present application, soft addition/subtraction networks (addition network and subtraction network) are used instead of the original hard addition and subtraction, which greatly improves the network flexibility.

Step 104: Input the reconstructed value of the latent variable of the i-th layer node to the first decoding network to generate a code stream of the geometric information of the i-th layer node.

In an embodiment of the present application, after the encoder determines the reconstructed value of the latent variable of the i-th layer node based on the latent variable of the i-th layer node, the encoder may further input the reconstructed value of the latent variable of the i-th layer node to the first decoder. network, so that the code stream of the geometric information of the i-th layer node can be generated.

It should be noted that in the embodiment of the present application, before the code stream of the geometric information of the i-th layer node is generated through the first decoding network, the first embedding network in the i-th layer network to the first layer network has been completed. For the corresponding output of embedded features, the encoding network in the i-th layer network to the first layer network has completed the output of the corresponding latent variables. At the same time, the i-1th layer network to the first layer network has completed the corresponding latent variable output. The output of the reconstructed value.

Further, in the embodiment of the present application, the input of the first decoding network of the i-th layer network may include: the embedded features of the i-2th layer node output by the first embedding network of the i-2th layer network, the i-th layer node The embedding feature of the i-1th layer node output by the first embedding network of the -1 layer network, the reconstructed value of the latent variable of the i-1th layer node obtained by the i-1th layer network; the first decoding of the i-th layer network The input of the network can also include the reconstructed value of the latent variable of the i-th layer node.

It should be noted that, in an embodiment of the present application, when the encoder generates a code stream of geometric information of the i-th layer nodes according to the reconstructed values of the latent variables of the i-th layer nodes, the reconstructed values of the latent variables of the i-th layer nodes, the embedded features of the i-2 layer nodes, the embedded features of the i-1 layer nodes, and the reconstructed values of the latent variables of the i-1 layer nodes can be first input into the first decoding network of the i-th layer network, thereby generating a code stream of geometric information of the i-th layer nodes.

It can be understood that in the embodiment of the present application, based on the first decoding network, the probability parameters of the i-th layer node can be determined first; and then the geometric information of the i-th layer node can be further generated based on the probability parameters of the i-th layer. code stream.

That is to say, in the embodiment of the present application, the embedding features of the i-2th layer node, the embedding features of the i-1th layer node, and the reconstructed values of the latent variables of the i-1th layer node can be obtained first. Among them, the embedded features of the i-2th layer node can be obtained from the first embedding network of the two lower layers of the current layer, that is, the i-2th layer node can be output by the first embedding network in the i-2th layer network. The embedded features of the layer node; the embedded features of the i-1th layer node can be obtained by the first embedding network of the next layer network of the current layer, that is, it can be output by the first embedding network in the i-1th layer network. The embedded features of the i-1th layer node; the reconstructed value of the latent variable of the i-1th layer node can be obtained from the network of the next layer of the current layer.

It should be noted that in the embodiment of the present application, the first decoding network may include a sparse convolution network, a deconvolution layer, an initial residual network, a feature fusion network, and a ReLU activation function.

Exemplarily, in the embodiment of the present application, Figure 11 is a schematic diagram of the network structure of the decoding network. As shown in Figure 11, the network structure of the first decoding network can be a Concatenate network followed by two layers of convolution kernels with a size of 3. A sparse convolution layer with a stride of 1, followed by an autoencoder, such as Binary AE.

代表第l层节点的潜变量的重建值，

代表第l-1层节点的潜变量的重建值，e ^(l-1)代表第一嵌入网络输出的第l-1层节点的嵌入特征，e ^(l-2)代表第一嵌入网络输出的第l-2层节点的嵌入特征。在获得

e ^(l-1)，e ^(l-2)之后，可以在通过Concatenate网络对

e ^(l-1)，e ^(l-2)进行拼接之后，通过一个二层的稀疏卷积网络，预测出第l层各节点的概率p ^(l)，最后通过算术编码器，如Binary AE，熵编码生成x ^(l)对应的二进制码流。 Exemplarily, in the embodiment of the present application, Figure 12 is a schematic diagram 1 of the implementation of the decoding network. As shown in Figure 12,

Represents the reconstructed value of the latent variable of the l-th layer node,

Represents the reconstructed value of the latent variable of the l-1th layer node, e ^(l-1) represents the embedding feature of the l-1th layer node output by the first embedding network, and e ^(l-2) represents the embedding feature output by the first embedding network. Embedded features of nodes in layer l-2. in getting

e ^(l-1) , e ^(l-2) , you can use the Concatenate network to

After e ^(l-1) and e ^(l-2) are spliced, the probability p ^(l) of each node in the l-th layer is predicted through a two-layer sparse convolution network, and finally through an arithmetic encoder, such as Binary AE , entropy coding generates the binary code stream corresponding to x ^(l) .

Further, in the embodiment of the present application, Figure 13 is a schematic diagram 2 of the implementation flow of the encoding method proposed in the embodiment of the present application. As shown in Figure 13, the latent variable of the i-th layer node is determined according to the latent variable of the i-th layer node. After the reconstructed value of the variable, that is, after step 103, the encoder's encoding method may also include the following steps:

Step 105: Input the reconstructed value of the latent variable of the i-th layer node to the first decoding network, and determine the predicted value of the latent variable of the i+1-th layer node.

In an embodiment of the present application, after the encoder determines the reconstructed value of the latent variable of the i-th layer node based on the latent variable of the i-th layer node, the encoder may further input the reconstructed value of the latent variable of the i-th layer node to the first decoder. network, so that the predicted value of the latent variable of the i+1th layer node can be determined.

It should be noted that in the embodiment of the present application, before the predicted value of the latent variable of the i+1-th layer node is generated by the first decoding network, the first embedding network in the i-th layer network to the first layer network has already The output of the corresponding embedded features has been completed. The coding network from the i-th layer network to the first layer network has completed the output of the corresponding latent variables. At the same time, the i-1 layer network to the first layer network has completed the corresponding output. Output of the reconstructed values of the latent variable.

Further, in the embodiment of the present application, the input of the first decoding network of the i-th layer network may include: the embedded features of the i-1th layer node output by the first embedding network of the i-1th layer network, the i-th layer node The reconstructed value of the latent variable of the i-1th layer node obtained by the -1 layer network; the input of the first decoding network of the i-th layer network may also include the reconstructed value of the latent variable of the i-th layer node and the i-th embedded feature.

It should be noted that in the embodiment of the present application, when the encoder determines the predicted value of the latent variable of the i+1-th layer node based on the reconstructed value of the latent variable of the i-th layer node, the encoder may convert the latent variable of the i-th layer node into The reconstructed value of the variable, the embedded feature of the i-th layer node, the embedded feature of the i-1th layer node, and the reconstructed value of the latent variable of the i-1th layer node are input to the first decoding network of the i-th layer network, so that it can be determined The predicted value of the latent variable of the i+1th layer node.

That is to say, in the embodiment of the present application, the embedding features of the i-1th layer node and the reconstructed values of the latent variables of the i-1th layer node can be obtained first. Among them, the embedded features of the i-1th layer node can be obtained by the first embedding network of the next layer network of the current layer, that is, the i-1th layer node can be output by the first embedding network in the i-1th layer network. Embedding features of layer nodes; the reconstructed value of the latent variable of the i-1th layer node can be obtained from the network of the next layer of the current layer.

It should be noted that in the embodiment of the present application, the first decoding network may include a sparse convolution network, a deconvolution layer, an initial residual network, a feature fusion network, and a ReLU activation function.

Exemplarily, in the embodiment of the present application, Figure 14 is a schematic diagram 2 of the network structure of the decoding network. As shown in Figure 14, the network structure of the first decoding network can be a Concatenate network followed by a layer of convolution kernel size. 2. A sparse convolution layer with a stride of 2, followed by three initial residual networks, followed by a sparse convolution layer with a convolution kernel size of 3 and a stride of 1.

代表第l层节点的潜变量的重建值，

代表第l-1层节点的潜变量的重建值，e ^(l)代表第一嵌入网络输出的第l层节点的嵌入特征，e ^(l-1)代表第一嵌入网络输出的第l-1层节点的嵌入特征。在获得

e ^(l)，e ^(l-1)之后，可以在通过Concatenate网络对

e ^(l-1)，e ^(l-2)进行拼接之后，然后通过一个卷积核大小为2，步长为2的反卷积层，再通过初始残差网络得到第l+1层节点的潜变量的预测值

Exemplarily, in the embodiment of the present application, Figure 15 is a schematic diagram 2 of the implementation of the decoding network. As shown in Figure 15,

Represents the reconstructed value of the latent variable of the l-th layer node,

Represents the reconstructed value of the latent variable of the l-1th layer node, e ^(l) represents the embedding feature of the l-th layer node output by the first embedding network, e ^(l-1) represents the l-1th layer output by the first embedding network Embedding features of layer nodes. in getting

e ^(l) , e ^(l-1) , you can use the Concatenate network to

After e ^(l-1) and e ^(l-2) are spliced, a deconvolution layer with a convolution kernel size of 2 and a step size of 2 is passed, and then the l+1th layer node is obtained through the initial residual network The predicted value of the latent variable

It can be seen that in the embodiment of the present application, for the first decoding network, two different branches can be output based on different inputs. One branch is the code corresponding to the geometric information of the i-th layer node of the octree. flow, and the other branch is the predicted value of the latent variable of the i+1th layer node of the octree.

It can be understood that in the embodiment of the present application, the code stream corresponding to the geometric information of the i-th layer node of the octree can be transmitted to the decoding end, and the predicted value of the latent variable of the i+1-th layer node of the octree is It can be input into the i+1th layer network and used to determine the residual of the i+1th layer node and the reconstruction value of the latent variable of the i+1th layer node.

Further, in the embodiment of the present application, for the first layer, the encoder can first perform quantization processing on the latent variables of the first layer nodes output by the coding unit of the first layer network, and determine the latent variables of the first layer nodes. The reconstructed value; then the reconstructed value of the latent variable and the preset embedding feature of the first layer node can be output to the first decoding unit of the first layer network, and then the code stream of the geometric information of the first layer node and the second The predicted value of the latent variable of the layer node.

For example, in the embodiment of the present application, the first level may be a level in an octree in which all leaf nodes are empty. In other words, when constructing an octree for a point cloud, the last level of the octree that cannot be further divided is the first level.

For example, in the embodiment of the present application, the first level may also be a level in an octree that is divided into minimum units, such as divided into minimum unit blocks of 1x1x1. In other words, when constructing an octree for a point cloud, the last level of the octree divided into the smallest unit blocks is the first level.

Further, in the embodiment of the present application, the encoder may use a factorized variational auto-encoder style entropy encoder to compress the residual r ^(l) .

则如下公式： For example, in the embodiment of the present application, for the variable y to be compressed, since the quantization process is not differentiable, uniform noise U (-0.5, 0.5) is used instead of quantization in the training stage. The quantified variable is recorded as

Then the following formula:

各维度的独立性，则有如下公式： Among them, p(x) is defined as the probability distribution of x, and the probability assumption

The independence of each dimension has the following formula:

c(y)应具有如下性质： Among them, the cumulative distribution c(y) fitted by the neural network is based on

c(y) should have the following properties:

·y∈(-∞,∞)

·c(y)increments

·c(-∞)=0,c(∞)=1

Therefore, the following neural network is used to fit c(y):

f _k (x)＝g _k (H ^(k) x+b ^(k) ),1≤k≤K

f _K (x)＝sigmoid(H ^(K) x+b ^(K) )

的方法如下公式： Calculated from p(y)

The method is as follows:

概率分布后，使用算术编码器对

进行熵编解码。 get

After the probability distribution, use the arithmetic encoder to

Perform entropy encoding and decoding.

的码流，通过算术编码器产生的第l层节点的几何信息x ^(l)的码流，结构信息。 Further, in the embodiment of the present application, at the encoding end, the final output code stream may be composed of the code stream corresponding to the output of at least one layer of the multi-layer network. Among them, for one layer of the network, such as the l-th layer network, the output code stream includes the l-th layer node generated by the factorial variation autoencoder.

The code stream is the code stream of the geometric information x ^(l) of the l-th layer node generated by the arithmetic encoder, and the structural information.

To sum up, the encoding method proposed in steps 101 to 105 above can rely on the octree structure, use the constructed embedding network, encoding network and decoding network, and introduce latent variables to capture brothers in the same layer in the octree The correlation between nodes can achieve higher compression performance; at the same time, parallel encoding and decoding processing based on the level of the octree can effectively shorten the encoding and decoding time, greatly improve the encoding and decoding efficiency, and further improve the compression performance.

The embodiment of the present application provides an encoding method. At the encoding end, the encoder includes a first embedding network, a coding network, and a first decoding network. The encoder divides the point cloud according to the octree to obtain the geometric information of the i-th layer node. Input to the first embedding network to determine the embedding features of the i-th layer node; where i is greater than 2; input the embedding features of the i-th layer node to the encoding network to determine the latent variables of the i-th layer node; according to the i-th layer node The latent variable determines the reconstructed value of the residual of the i-th layer node, and determines the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node To the first decoding network, a code stream of geometric information of the i-th layer node is generated. It can be seen that in the embodiments of the present application, when encoding and decoding point clouds, based on the constructed octree structure, latent variables can be used to represent the spatial correlation between nodes at a unified level in the octree. , ultimately achieving higher compression performance; at the same time, based on the constructed network, parallel encoding and decoding processes are performed on nodes at the same level of the octree, which improves encoding and decoding efficiency and further improves compression performance.

Based on the above embodiments, the encoding method proposed in the embodiment of this application can be understood as an end-to-end LIDAR point cloud depth entropy model, or as an end-to-end self-supervised dynamic point cloud compression technology based on deep learning. Among them, it relies on the octree structure, introduces latent variables to capture the correlation of octree sibling nodes, and uses sparse convolution to construct the network, thus achieving the optimal effect in the LIDAR point cloud lossless entropy model.

At the same time, the encoding method proposed in the embodiment of this application can improve the encoding and decoding efficiency of LIDAR point cloud. Among them, the embodiments of the present application have a lower encoding and decoding time. This is because the encoding and decoding of each node in the embodiments of the present application are processed in parallel on a layer basis, and efficient sparse convolution is used to construct the network. Therefore, it has Much lower encoding and decoding time compared to common encoding and decoding schemes.

It should be noted that the network that performs the coding method proposed in the embodiment of the present application can make full use of hierarchical latent variables to capture the LIDAR point cloud correlation, and can also use residual coding to decompose and compress the latent variables. Furthermore, it can also The soft addition/subtraction network is used to replace the original hard addition/subtraction, which improves the flexibility of the network.

Further, in the embodiment of the application, Figure 16 is a schematic diagram of the overall framework for executing the encoding method. As shown in Figure 16, the encoder can be provided with a multi-layer network, wherein at least one layer of the multi-layer network can include a third layer. An embedding network (embedding), encoding network (encoder), and first decoding network (decoder). For example, each layer consists of an encoding network (encoder), a decoding network (decoder), and an embedding network (embedding).

It can be understood that, in the embodiments of the present application, the l-th layer network in the multi-layer network can perform parallel encoding processing on the geometric information of the l-th layer nodes.

代表第l层残差的重建值，

代表第l层节点的潜变量的重建值，

代表第l层节点的潜变量的预测值。图中+/-符号代表软加/减网络(加法网络/减法网络)，Q代表量化操作。 Among them, as shown in the figure, x ^(l) represents the geometric information of the l-th layer node, e ^(l) represents the embedded feature of the l-th layer node after occupying the embedded network, and f ^(l) represents the latent variable of the l-th layer node. ,

Represents the reconstructed value of the l-th layer residual,

Represents the reconstructed value of the latent variable of the l-th layer node,

Represents the predicted value of the latent variable of the l-th layer node. The +/- symbols in the figure represent soft addition/subtraction networks (addition networks/subtraction networks), and Q represents quantization operations.

其中，第l层节点的潜变量的预测值

与真实值f ^(l)间的第l层节点的残差为r ^(l)，经过量化后为第l层节点的残差的重建值

由因式变分自编码器无损熵编码。 For example, in the embodiment of the present application, as shown in the figure, the network provided in the encoder may be composed of a 5-layer network. Among them, for the l-th layer network, the geometric information x ^(l) of the l-th layer node passes through the embedding network to obtain the occupancy embedding e ^(l) (embedded feature). The encoding network captures the spatial correlation between e ^(l) and the latent variable f ^(l+1) of the previous layer (layer l+1), and outputs the latent variable f ^(l) of the current layer node. The decoding network is divided into two paths, which generate the code stream of the current layer x ^(l) of the octree and the predicted value of the latent variable of the node in the previous layer.

Among them, the predicted value of the latent variable of the l-th layer node

The residual of the l-th layer node between the real value f ^(l) is r ^(l) . After quantization, it is the reconstructed value of the residual of the l-th layer node.

Lossless entropy coding by factorial variational autoencoders.

然后可以将第一层节点的潜变量的重建值和预设嵌入特征e ^(l-6)输出至第一层网络的第一解码单元，进而可以分别输出第一层节点的几何信息的码流和第二层节点的潜变量的预测值

It should be noted that in the embodiment of the present application, for the first layer (layer 1-5 as shown in the figure), the encoder may first perform The latent variable f ^(l-5) is quantified to determine the reconstructed value of the latent variable of the first layer node.

Then the reconstructed value of the latent variable of the first layer node and the preset embedding feature e ^(l-6) can be output to the first decoding unit of the first layer network, and then the code stream of the geometric information of the first layer node can be output respectively. and the predicted value of the latent variable of the second-level node

Further, in the embodiment of the present application, as shown in the figure, the input of the encoding network of the l-1th layer network may include the latent variables of the l-th layer nodes output by the encoding network of the l-th layer network. That is to say, in the embodiment of the present application, the encoder can input the embedded features of the l-1th layer node and the latent variable of the l-th layer node into the encoding network of the l-1th layer network, so that the l-th layer node can be determined -Latent variables of layer 1 nodes. Among them, the embedded features of the l-1th layer node are output by the first embedding unit of the l-1th layer network, and the latent variables of the l-th layer node are output by the encoding network of the l-th layer network.

Further, in the embodiment of the present application, as shown in the figure, the input of the first decoding network of the l-1th layer network may include: the l-3th layer output of the first embedding network of the l-3th layer network The embedding features of the node, the embedding features of the l-2th layer node output by the first embedding network of the l-2th layer network, the reconstructed value of the latent variable of the l-2th layer node obtained by the l-2th layer network; The input of the first decoding network of the l-1 layer network may also include the reconstructed value of the latent variable of the l-1 layer node. Correspondingly, when the encoder generates the code stream of the geometric information of the l-1th layer node, it can first combine the reconstructed value of the latent variable of the l-1th layer node, the embedded feature of the l-3th layer node, the The embedded features of the l-2 layer nodes and the reconstructed values of the latent variables of the l-2 layer nodes are input to the first decoding network of the l-1 layer network, thereby generating a code stream of the geometric information of the l-1 layer nodes. .

It should be noted that in the embodiment of the present application, when the encoder determines the predicted value of the latent variable of the l-th layer node, the reconstructed value of the latent variable of the l-1th layer node, the l-1th layer node The embedded features of the layer node, the embedded features of the l-2th layer node, and the reconstructed value of the latent variable of the l-2th layer node are input to the first decoding network of the l-1th layer network, so that the l-th layer node can be determined. Predictive value of the latent variable.

Further, in the embodiment of the present application, as shown in the figure, after the first decoding network of the l-1 layer network outputs the predicted value of the latent variable of the l-th layer node, the subsequent encoding of the l-th layer can be During the processing, the predicted value of the latent variable of the l-th layer node is used, and the addition network and the subtraction network are used to determine the residual of the l-th layer node and the reconstruction value of the latent variable of the l-th layer node, respectively.

Illustratively, in the embodiment of the present application, the encoding network (encoder) can be used to explore the spatial correlation between e ^(l) and the latent variable f ^(l+1 ) of the previous layer (l+1th layer), And output the latent variable f ^(l) of the current layer node. For example, a sparse tensor can be used to represent an octree, where the coordinate matrix of the sparse tensor represents the node coordinates of the octree; the attribute matrix of the sparse tensor represents the node occupancy code of the octree. The encoding network can be implemented using sparse convolution, where the downsampling of f ^(l) is implemented by a sparse convolution layer with a convolution kernel size of 2 and a stride of 2, followed by a ReLU activation function. After that, the initial residual network is used to aggregate the sibling node features and output to the Concatenate network for feature fusion with e ^(l) . Finally, a sparse convolution layer is used to fuse e ^(l) with the downsampled f ^{(l+ 1)} , obtain the latent variable of the low-level node, that is, the latent variable f ^(l) of the l+1th level node.

其中，一个分支包括：在获得

e ^(l-1)，e ^(l-2)之后，可以在通过Concatenate网络对

e ^(l-1)，e ^(l-2)进行拼接之后，通过一个二层的稀疏卷积网络，预测出第l层各节点的概率p ^(l)，最后通过算术编码器，熵编码生成x ^(l)对应的二进制码流。另一个分支包括：在获得

e ^(l)，e ^(l-1)之后，可以在通过Concatenate网络对

e ^(l-1)，e ^(l-2)进行拼接之后，然后通过一个卷积核大小为2，步长为2的反卷积层，再通过初始残差网络得到第l+1层节点的潜变量的预测值

Illustratively, in the embodiment of the present application, the decoder network (decoder) is used to generate the code stream of the current layer x ^(l) of the octree and the predicted value of the latent variable of the node in the previous layer.

Among them, one branch includes: after obtaining

e ^(l-1) , e ^(l-2) , you can use the Concatenate network to

After splicing e ^{(l-1) and} e ^(l-2) , the probability p ^(l) of each node in the l-th layer is predicted through a two-layer sparse convolution network, and finally generated by entropy coding through an arithmetic encoder The binary code stream corresponding to x ^(l) . Another branch includes: after obtaining

e ^(l) , e ^(l-1) , you can use the Concatenate network to

After e ^(l-1) and e ^(l-2) are spliced, a deconvolution layer with a convolution kernel size of 2 and a step size of 2 is passed, and then the l+1th layer node is obtained through the initial residual network The predicted value of the latent variable

For example, in the embodiment of the present application, the embedding network (embedding) can be used to map the octree nodes from the 256-dimensional one-hot vector to the 16-dimensional continuous feature space, so that they occupy similar spaces in the space. The situation is similar in feature space.

Exemplarily, in an embodiment of the present application, at least one layer of the multi-layer network set by the encoder may also include a subtraction network and an addition network. Among them, the use of soft addition/subtraction networks (addition networks and subtraction networks) instead of the original hard addition and subtraction greatly improves the network flexibility.

To sum up, the encoding method proposed in the embodiment of this application can rely on the octree structure, use the built embedding network, encoding network and decoding network, and introduce latent variables to capture the differences between sibling nodes in the same layer in the octree. The correlation can achieve higher compression performance; at the same time, parallel encoding and decoding processing based on the level of the octree can effectively shorten the encoding and decoding time, greatly improve the encoding and decoding efficiency, and further improve the compression performance.

The embodiment of the present application provides an encoding method. At the encoding end, the encoder includes a first embedding network, a coding network, and a first decoding network. The encoder divides the point cloud according to the octree to obtain the geometric information of the i-th layer node. Input to the first embedding network to determine the embedding features of the i-th layer node; where i is greater than 2; input the embedding features of the i-th layer node to the encoding network to determine the latent variables of the i-th layer node; according to the i-th layer node The latent variable determines the reconstructed value of the residual of the i-th layer node, and determines the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node To the first decoding network, a code stream of geometric information of the i-th layer node is generated. It can be seen that in the embodiments of the present application, when encoding and decoding point clouds, based on the constructed octree structure, latent variables can be used to represent the spatial correlation between nodes at a unified level in the octree. , ultimately achieving higher compression performance; at the same time, based on the constructed network, parallel encoding and decoding processes are performed on nodes at the same level of the octree, which improves encoding and decoding efficiency and further improves compression performance.

The embodiment of the present application proposes a point cloud decoding method, which can be applied to a decoder, where the decoder can include a second embedding network and a second decoding network.

It should be noted that in embodiments of the present application, the decoder may be provided with a multi-layer network, wherein at least one layer of the multi-layer network may include a second embedding network and a second decoding network.

For example, in the embodiment of the present application, the decoder may be provided with a 5-layer network, wherein one layer of the 5-layer network or a multi-layer network may include a second embedding network and a second decoding network.

Figure 17 is a schematic flowchart 1 of the implementation of the decoding method proposed by the embodiment of the present application. As shown in Figure 17, the method for the decoder to perform decoding processing may include the following steps:

Step 201: Decode the code stream and determine the reconstruction value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; where i is greater than 2.

In the embodiment of the present application, the decoder can determine the reconstruction value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree by decoding the code stream.

It should be noted that, in the embodiment of the present application, i may be an integer greater than 2.

It can be understood that, in the embodiment of the present application, the i-th layer network in the multi-layer network can perform parallel decoding processing on the geometric information of the i-th layer nodes.

Further, in the embodiment of the present application, when performing geometric encoding processing on the point cloud, an octree may be constructed first. Among them, the octree structure is used to recursively divide the point cloud space into eight sub-blocks of the same size, and the occupied code words of each sub-block are judged. When the sub-block does not contain points, it is recorded as empty, otherwise it is recorded as empty. is non-empty, the occupied codeword information of all blocks is recorded in the last layer of recursive division, and geometric encoding is performed; on the one hand, the geometric information expressed through the octree structure can further form a geometric code stream, on the other hand, it can be geometrically encoded During the reconstruction process, the reconstructed geometric information is used as additional information for attribute encoding.

Correspondingly, in the embodiment of the present application, by analyzing the geometric code stream transmitted on the encoding side, the reconstruction value of the residual of the i-th layer node of the point cloud can be determined.

It can be understood that in the embodiment of the present application, for the constructed octree, the parent node layer can be defined as a high-level layer and the child node layer as a low-level layer; the parent node layer can also be defined as a low-level layer and the child node layer can be defined as a low-level layer. A layer is defined as a high-level layer.

The encoding sequence in which the encoder performs the encoding process may be in the order from the root node to the leaf node of the constructed octree. Correspondingly, on the decoding side, the decoding order may be in the order from the leaf nodes to the root node of the constructed octree. For example, the root node layer of the constructed octree is the i-th layer, and the leaf node layers are from the i-1 to the first layer in sequence. Then the order of encoding processing is from the i-th layer to the first layer, and the order of decoding processing is The opposite of encoding, that is, from the first layer to the i-th layer.

It should be noted that in the embodiment of the present application, at the encoding end, the residual of the i-th layer node can be determined first based on the predicted value of the latent variable of the i-th layer node and the latent variable of the i-th layer node; and then The residual of the i-th layer node is quantized, and then the reconstructed value of the residual of the i-th layer node can be determined.

It can be understood that in the embodiment of the present application, the encoder can write the reconstructed value of the residual of the i-th layer node into the code stream and transmit it to the decoder. In other words, the encoder can decompose and compress the latent variables using residual coding.

Step 202: Determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node.

In the embodiment of the present application, after the decoder determines the reconstructed value of the residual of the i-th layer node after decoding the code stream, it can determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node. .

Further, in the embodiment of the present application, the predicted value of the latent variable of the i-th layer node can be obtained first. Among them, the predicted value of the latent variable of the i-th layer node can be predicted by the second decoding network of the next layer network of the current layer, that is, it can be the output of the i-th layer by the second decoding network in the i-1 layer network. The predicted value of the latent variable of the layer node.

Further, in the embodiment of the present application, after the decoder determines the reconstruction value of the residual of the i-th layer node, the decoder can further calculate the reconstruction value of the residual of the i-th layer node and the prediction of the latent variable of the i-th layer node. value, determines the reconstructed value of the latent variable of the i-th layer node. Wherein, the predicted value of the latent variable of the i-th layer node may be output by the second decoding network in the i-1th layer.

It should be noted that in the embodiment of the present application, the decoder may also include an addition network, where the addition network may be used to determine the reconstructed value of the latent variable.

That is to say, in the embodiment of the present application, at least one layer of the multi-layer network set up by the decoder may also include an additive network. Among them, the addition network is a soft addition network, which can be used to replace common hard addition processing.

For example, in the embodiment of the present application, based on the additive network, the determination of the predicted value of the latent variable can be achieved through the above formula (2).

For example, in the embodiment of the present application, when the decoder determines the reconstructed value of the latent variable of the i-th layer node, the decoder may first combine the reconstructed value of the residual of the i-th layer node with the reconstructed value of the latent variable of the i-th layer node. The predicted value is input to the additive network, and the reconstructed value of the latent variable of the i-th layer node can be obtained.

It should be noted that in the embodiment of the present application, the additive network may include a sparse convolutional network and a feature fusion network Concatenate.

For example, in the embodiment of the present application, as shown in Figure 9, the network structure of the additive network can be a sparse convolution layer with a convolution kernel size of 3 and a step size of 1, followed by a Concatenate network.

代表第l层节点的残差的重建值，

代表第l层节点的潜变量的重建值，

代表第l层节点的潜变量的预测值。其中，

和

在经过Concatenate网络的融合后，可以通过稀疏卷积层的操作，最终输出第l层节点的潜变量的重建值

Illustratively, in the embodiment of the present application, as shown in Figure 10,

Represents the reconstructed value of the residual of the l-th layer node,

Represents the reconstructed value of the latent variable of the l-th layer node,

Represents the predicted value of the latent variable of the l-th layer node. in,

and

After the fusion of the Concatenate network, the reconstructed value of the latent variable of the l-th layer node can be finally output through the operation of the sparse convolution layer.

It can be understood that, in the embodiments of the present application, a soft addition network (addition network) is used to replace the original hard addition, which greatly improves the network flexibility.

It should be noted that, in the embodiment of the present application, the latent variable of the i-th layer node can represent the correlation between the i-th layer sibling nodes. That is to say, in this application, the decoder can make full use of hierarchical latent variables to capture the correlation of LIDAR point clouds.

Step 203: Input the reconstructed value of the latent variable of the i-th layer node to the second decoding network to determine the geometric information of the i-th layer node.

In an embodiment of the present application, after the decoder determines the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node, the decoder may input the reconstructed value of the latent variable of the i-th layer node to the i-th layer node. 2 decoding network, so that the geometric information of the i-th layer node can be determined.

It can be understood that in the embodiments of the present application, the geometric information of the i-th layer node of the point cloud can represent the geometric information of the i-th layer node, such as the position coordinate of the node, and can also represent the occupied codeword of the i-th layer node. situation, that is, occupancy status (occupancy information).

That is to say, in the embodiment of the present application, the geometric information of the i-th layer node includes both the position coordinate information of the node and the occupancy information of the node.

It can be understood that in the embodiment of the present application, at the decoding end, the input of the second embedding network is the geometric information of the i-th layer node, wherein the geometric information of the node as the input of the second embedding network may also include nodes The position coordinate information and the occupancy information of the node; and the geometric information output by the second decoding network mainly includes the occupancy information of the node, which may or may not include the position coordinate information of the node.

It can be seen that in the embodiment of the present application, the embedding network needs to refer to the occupancy information and position coordinate information of the node at the same time, while the decoding network focuses on generating the occupancy information.

It should be noted that in the embodiment of the present application, before the geometric information of the i-th layer node is determined through the second decoding network, the second embedding network in the i-1th layer network to the first layer network has completed the corresponding The output of the embedded feature, at the same time, the i-1th layer network to the first layer network have completed the output of the corresponding reconstruction value of the latent variable.

Further, in the embodiment of the present application, the input of the second decoding network of the i-th layer network may include: the embedded features of the i-2-th layer node output by the second embedding network of the i-2-th layer network, the i-th layer node The embedding feature of the i-1th layer node output by the second embedding network of the -1 layer network, the reconstructed value of the latent variable of the i-1th layer node obtained by the i-1th layer network; the second decoding of the i-th layer network The input of the network can also include the reconstructed value of the latent variable of the i-th layer node.

It should be noted that in the embodiment of the present application, when the decoder generates the geometric information of the i-th layer node based on the reconstructed value of the latent variable of the i-th layer node, it may first convert the latent variable of the i-th layer node into The reconstructed value, the embedded feature of the i-2th layer node, the embedded feature of the i-1th layer node, and the reconstructed value of the latent variable of the i-1th layer node are input to the second decoding network of the i-th layer network, thereby generating The geometric information of the i-th layer node.

It can be understood that in the embodiment of the present application, based on the second decoding network, the probability parameters of the i-th layer nodes can be determined first; then the geometric information of the i-th layer nodes can be further generated according to the probability parameters of the i-th layer.

That is to say, in the embodiment of the present application, the embedding features of the i-2th layer node, the embedding features of the i-1th layer node, and the reconstructed values of the latent variables of the i-1th layer node can be obtained first. Among them, the embedded features of the i-2th layer node can be obtained by the second embedding network of the two-layer network below the current layer, that is, the i-2th layer node can be output by the second embedding network in the i-2th layer network. The embedded features of the layer node; the embedded features of the i-1th layer node can be obtained by the second embedding network of the next layer network of the current layer, that is, it can be output by the second embedding network in the i-1th layer network. The embedded features of the i-1th layer node; the reconstructed value of the latent variable of the i-1th layer node can be obtained from the network of the next layer of the current layer.

It should be noted that in the embodiment of the present application, the second decoding network may include a sparse convolution network, a deconvolution layer, an initial residual network, a feature fusion network, and a ReLU activation function.

Exemplarily, in the embodiment of the present application, as shown in Figure 11, the network structure of the second decoding network may be a Concatenate network followed by two sparse convolution layers with a convolution kernel size of 3 and a step size of 1. , followed by an autodecoder, such as Binary AE.

代表第l层节点的潜变量的重建值，

代表第l-1层节点的潜变量的重建值，e ^(l-1)代表第二嵌入网络输出的第l-1层节点的嵌入特征，e ^(l-2)代表第二嵌入网络输出的第l-2层节点的嵌入特征。在获得

e ^(l-1)，e ^(l-2)之后，可以在通过Concatenate网络对

e ^(l-1)，e ^(l-2)进行拼接之后，通过一个二层的稀疏卷积网络，预测出第l层各节点的概率p ^(l)，最后通过算术编码器，如Binary AE，熵解码生成x ^(l)。 Illustratively, in the embodiment of the present application, as shown in Figure 12,

Represents the reconstructed value of the latent variable of the l-th layer node,

Represents the reconstructed value of the latent variable of the l-1 layer node, e ^(l-1) represents the embedding feature of the l-1 layer node output by the second embedding network, and e ^(l-2) represents the embedding feature output by the second embedding network. Embedded features of nodes in layer l-2. in getting

e ^(l-1) , e ^(l-2) , you can use the Concatenate network to

After e ^(l-1) and e ^(l-2) are spliced, the probability p ^(l) of each node in the l-th layer is predicted through a two-layer sparse convolution network, and finally through an arithmetic encoder, such as Binary AE , entropy decoding generates x ^(l) .

Further, in the embodiment of the present application, after determining the geometric information of the i-th layer node, the decoder can continue to input the geometric information of the i-th layer node to the second embedding network, so that the i-th layer node can be determined embedded features.

Further, in the embodiment of the present application, the second embedding network may be an occupied embedding network embedding. Among them, the second embedding network can be used to map the octree nodes from the 256-dimensional one-hot vector to the 16-dimensional continuous feature space, so that similar occupancy conditions in the space are also similar in the feature space.

It should be noted that, in the embodiment of the present application, the second embedding network may include a sparse convolutional network.

For example, in the embodiment of the present application, the network structure of the second embedding network may be composed of a multi-layer sparse convolutional network. For example, as shown in Figure 3, the network structure of the second embedding network may be a three-layer sparse convolutional network.

Further, in embodiments of the present application, after the decoder inputs the geometric information of the i-th layer node in the point cloud to the second embedding network, it can output the embedding feature of the i-th layer node. Among them, the embedded features of the i-th layer node can determine the occupancy status of the i-th layer node after it is mapped to the feature space.

That is to say, in the embodiment of the present application, the embedded characteristics of the i-th layer node can determine the occupancy status of the i-th layer node.

Exemplarily, in the embodiment of the present application, as shown in Figure 4, x ^(l) represents the geometric information of the l-th layer node, e ^(l) represents the embedding feature of the l-th layer node output by the second embedding network, For the l-th layer node, the decoder can input the geometric information x ^(l) of the l-th layer node into the second embedding network, so that the corresponding embedding feature e ^(l) of the l-th layer node can be obtained through the second embedding network. .

Further, in the embodiment of the present application, Figure 18 is a schematic diagram 2 of the implementation flow of the decoding method proposed in the embodiment of the present application. As shown in Figure 18, the i-th layer is determined based on the reconstructed value of the residual of the i-th layer node. After the reconstructed value of the latent variable of the node, that is, after step 202, the decoding method of the decoder may also include the following steps:

Step 204: input the reconstructed value of the latent variable of the i-th layer node into the second decoding network to determine the predicted value of the latent variable of the i+1-th layer node.

In the embodiment of the present application, after the decoder determines the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node, the decoder can further input the reconstructed value of the latent variable of the i-th layer node into The second decoding network can determine the predicted value of the latent variable of the i+1th layer node.

It should be noted that in the embodiment of the present application, before the second decoding network generates the predicted value of the latent variable of the i+1-th layer node, the second embedding network in the i-th layer network to the first layer network has already The output of the corresponding embedded features is completed. At the same time, the i-1th layer network to the first layer network have completed the output of the corresponding reconstructed values of the latent variables.

Further, in the embodiment of the present application, the input of the second decoding network of the i-th layer network may include: the embedded features of the i-1-th layer node output by the second embedding network of the i-1-th layer network, the i-th layer node The reconstructed value of the latent variable of the i-1th layer node obtained by the -1 layer network; the input of the second decoding network of the i-th layer network may also include the reconstructed value of the latent variable of the i-th layer node and the i-th embedded feature.

It should be noted that in the embodiment of the present application, when the decoder determines the predicted value of the latent variable of the i+1-th layer node based on the reconstructed value of the latent variable of the i-th layer node, the decoder may convert the latent variable of the i-th layer node into The reconstructed value of the variable, the embedded feature of the i-th layer node, the embedded feature of the i-1th layer node, and the reconstructed value of the latent variable of the i-1th layer node are input to the second decoding network of the i-th layer network, so that it can be determined The predicted value of the latent variable of the i+1th layer node.

That is to say, in the embodiment of the present application, the embedding features of the i-1th layer node and the reconstructed values of the latent variables of the i-1th layer node can be obtained first. Among them, the embedded features of the i-1th layer node can be obtained by the second embedding network of the next layer network of the current layer, that is, the i-1th layer output can be obtained by the second embedding network in the i-1th layer network. Embedding features of layer nodes; the reconstructed value of the latent variable of the i-1th layer node can be obtained from the network of the next layer of the current layer.

It should be noted that in the embodiment of the present application, the second decoding network may include a sparse convolution network, a deconvolution layer, an initial residual network, a feature fusion network, and a ReLU activation function.

Exemplarily, in the embodiment of the present application, as shown in Figure 14, the network structure of the second decoding network may be a sparse convolution layer with a convolution kernel size of 2 and a step size of 2 followed by the Concatenate network. , followed by three initial residual networks, followed by a sparse convolution layer with a convolution kernel size of 3 and a stride of 1.

代表第l层节点的潜变量的重建值，

代表第l-1层节点的潜变量的重建值，e ^(l)代表第二嵌入网络输出的第l层节点的嵌入特征，e ^(l-1)代表第二嵌入网络输出的第l-1层节点的嵌入特征。在获得

e ^(l)，e ^(l-1)之后，可以在通过Concatenate网络对

e ^(l-1)，e ^(l-2)进行拼接之后，然后通过一个卷积核大小为2，步长为2的反卷积层，再通过初始残差网络得到第l+1层节点的潜变量的预测值

Illustratively, in the embodiment of the present application, as shown in Figure 15,

Represents the reconstructed value of the latent variable of the l-th layer node,

Represents the reconstructed value of the latent variable of the l-1th layer node, e ^(l) represents the embedding feature of the l-th layer node output by the second embedding network, e ^(l-1) represents the l-1 th layer output by the second embedding network Embedding features of layer nodes. in getting

e ^(l) , e ^(l-1) , you can use the Concatenate network to

After e ^(l-1) and e ^(l-2) are spliced, a deconvolution layer with a convolution kernel size of 2 and a step size of 2 is passed, and then the l+1th layer node is obtained through the initial residual network The predicted value of the latent variable

It can be seen that in the embodiment of the present application, for the second decoding network, two different branches can be output based on different inputs. One branch is the geometric information of the i-th layer node of the octree, and the other branch is the geometric information of the i-th layer node of the octree. The branch is the predicted value of the latent variable of the i+1th level node of the octree.

It can be understood that in the embodiment of the present application, the predicted value of the latent variable of the i+1th layer node of the octree can be input into the i+1th layer network for performing the i+1th layer node calculation. Determination of reconstructed values of latent variables.

Further, in an embodiment of the present application, for the first layer, after the decoder determines the reconstructed value of the latent variable of the first layer node using the reconstructed value of the residual of the first layer node obtained by decoding the code stream; then the reconstructed value of the latent variable of the first layer node and the preset embedded feature can be output to the second decoding unit of the first layer network, and then the geometric information of the first layer node and the predicted value of the latent variable of the second layer node can be output respectively.

For example, in the embodiment of the present application, the first level may be a level in an octree in which all leaf nodes are empty. In other words, when constructing an octree for a point cloud, the last level of the octree that cannot be further divided is the first level.

For example, in the embodiment of the present application, the first level may also be a level in an octree that is divided into minimum units, such as divided into minimum unit blocks of 1x1x1. That is to say, when constructing an octree for a point cloud, the last level of the octree divided into the smallest unit block is the first level.

Further, in embodiments of the present application, the decoder may use a factorized variational auto-encoder style entropy codec to compress the residual r ^(l) .

的码流，通过算术编码器产生的第l层节点的几何信息x ^(l)的码流，结构信息。 Further, in the embodiment of the present application, at the decoding end, the finally received code streams may be composed of code streams corresponding to at least one layer of the multi-layer network. Among them, for one layer of the network, such as the l-th layer network, the corresponding code stream includes the l-th layer node generated by the factorial variation autoencoder.

The code stream is the code stream of the geometric information x ^(l) of the l-th layer node generated by the arithmetic encoder, and the structural information.

In summary, the decoding method proposed in steps 201 to 204 above can rely on the octree structure, use the built embedding network and decoding network, and introduce latent variables to capture the interactions between sibling nodes in the same layer in the octree The correlation can achieve higher compression performance; at the same time, parallel encoding and decoding processing based on the level of the octree can effectively shorten the encoding and decoding time, greatly improve the encoding and decoding efficiency, and further improve the compression performance.

Embodiments of the present application provide a decoding method. At the decoding end, the decoder includes a second decoding network. The decoder decodes the code stream and determines the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; Among them, i is greater than 2; determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node into the second decoding network to determine the i-th layer node. Geometry information of layer nodes. It can be seen that in the embodiments of the present application, when encoding and decoding point clouds, based on the constructed octree structure, latent variables can be used to represent the spatial correlation between nodes at a unified level in the octree. , ultimately achieving higher compression performance; at the same time, based on the constructed network, parallel encoding and decoding processes are performed on nodes at the same level of the octree, which improves encoding and decoding efficiency and further improves compression performance.

Based on the above embodiments, the decoding method proposed in the embodiment of this application can be understood as an end-to-end LIDAR point cloud depth entropy model, or as an end-to-end self-supervised dynamic point cloud compression technology based on deep learning. Among them, it relies on the octree structure, introduces latent variables to capture the correlation of octree sibling nodes, and uses sparse convolution to construct the network, thus achieving the optimal effect in the LIDAR point cloud lossless entropy model.

At the same time, the decoding method proposed in the embodiment of this application can improve the encoding and decoding efficiency of LIDAR point cloud. Among them, the embodiments of the present application have a lower encoding and decoding time. This is because the encoding and decoding of each node in the embodiments of the present application are processed in parallel on a layer basis, and efficient sparse convolution is used to construct the network. Therefore, it has Much lower encoding and decoding time compared to common encoding and decoding schemes.

It should be noted that the network that performs the decoding method proposed in the embodiment of the present application can make full use of hierarchical latent variables to capture LIDAR point cloud correlation, and can use residual coding to decompose and compress the latent variables. Furthermore, it can also The soft addition/subtraction network is used to replace the original hard addition/subtraction, which improves the flexibility of the network.

Further, in an embodiment of the application, FIG19 is a schematic diagram of an overall framework for executing a decoding method. As shown in FIG19 , the decoder may be provided with a multi-layer network, wherein at least one layer of the multi-layer network may include a second embedding network (embedding) and a second decoding network (decoder). For example, each layer is composed of a decoding network (decoder) and an embedding network (embedding).

It can be understood that, in the embodiments of the present application, the l-th layer network in the multi-layer network can perform parallel decoding processing on the geometric information of the l-th layer nodes.

代表第l层残差的重建值，

代表第l层节点的潜变量的重建值，

代表第l层节点的潜变量的预测值。图中+符号代表软加法网络(加法网络)。 Among them, as shown in the figure, x ^(l) represents the geometric information of the l-th layer node, e ^(l) represents the embedded feature of the l-th layer node after occupying the embedded network, and f ^(l) represents the latent variable of the l-th layer node. ,

Represents the reconstructed value of the l-th layer residual,

Represents the reconstructed value of the latent variable of the l-th layer node,

Represents the predicted value of the latent variable of the l-th layer node. The + symbol in the figure represents a soft additive network (additive network).

For example, in the embodiment of the present application, as shown in the figure, the network provided in the decoder may be composed of a 5-layer network. Among them, for the l-th layer network, the geometric information x ^(l) of the l-th layer node passes through the embedding network to obtain the occupancy embedding e ^(l) (embedded feature). The decoding network is divided into two paths, which generate the current layer x ^(l) of the octree and the predicted value of the latent variable of the node in the previous layer.

然后可以将第一层节点的潜变量的重建值和预设嵌入特征e ^(l-6)输出至第一层网络的第二解码单元，进而可以分别输出第一层节点的几何信息和第二层节点的潜变量的预测值

It should be noted that in the embodiment of the present application, for the first layer (layer 1-5 as shown in the figure), the decoder can first determine the reconstructed value of the latent variable of the node in the first layer.

Then the reconstructed value of the latent variable of the first layer node and the preset embedded feature e ^(l-6) can be output to the second decoding unit of the first layer network, and then the geometric information of the first layer node and the second decoding unit can be output respectively. The predicted value of the latent variable of the layer node

Further, in an embodiment of the present application, as shown in the figure, the input of the second decoding network of the l-1 layer network may include: the embedded features of the l-3 layer nodes output by the second embedding network of the l-3 layer network, the embedded features of the l-2 layer nodes output by the second embedding network of the l-2 layer network, and the reconstructed values of the latent variables of the l-2 layer nodes obtained by the l-2 layer network; the input of the second decoding network of the l-1 layer network may also include the reconstructed values of the latent variables of the l-1 layer nodes. Accordingly, when the decoder generates the geometric information of the l-1 layer nodes, it may first input the reconstructed values of the latent variables of the l-1 layer nodes, the embedded features of the l-3 layer nodes, the embedded features of the l-2 layer nodes, and the reconstructed values of the latent variables of the l-2 layer nodes into the second decoding network of the l-1 layer network, thereby generating the geometric information of the l-1 layer nodes.

It should be noted that in the embodiment of the present application, when the decoder determines the predicted value of the latent variable of the l-th layer node, the reconstructed value of the latent variable of the l-1th layer node, the l-1th layer node The embedded features of the layer node, the embedded features of the l-2th layer node, and the reconstructed value of the latent variable of the l-2th layer node are input to the second decoding network of the l-1th layer network, so that the l-th layer node can be determined. Predictive value of the latent variable.

Further, in the embodiment of the present application, as shown in the figure, after the second decoding network of the l-1th layer network outputs the predicted value of the latent variable of the l-th layer node, the subsequent l-th layer node can be During the decoding process, the predicted value of the latent variable of the l-th layer node is used, and the additive network is used to determine the reconstructed value of the latent variable of the l-th layer node.

其中，一个分支包括：在获得

e ^(l-1)，e ^(l-2)之后，可以在通过Concatenate网络对

e ^(l-1)，e ^(l-2)进行拼接之后，通过一个二层的稀疏卷积网络，预测出第l层各节点的概率p ^(l)，最后通过算术编码器，熵解码生成x ^(l)。另一个分支包括：在获得

e ^(l)，e ^(l-1)之后，可以在通过Concatenate网络对

e ^(l-1)，e ^(l-2)进行拼接之后，然后通过一个卷积核大小为2，步长为2的反卷积层，再通过初始残差网络得到第l+1层节点的潜变量的预测值

Exemplarily, in an embodiment of the present application, the decoding network (decoder) is used to generate the predicted value of the latent variable of the current layer x ^(l) of the octree and the previous layer node.

One branch includes: obtaining

e ^(l-1) , e ^(l-2) can be connected through the Concatenate network

After e ^(l-1) and e ^(l-2) are concatenated, a two-layer sparse convolutional network is used to predict the probability p ^(l) of each node in the lth layer, and finally an arithmetic encoder is used to entropy decode and generate x ^(l) . Another branch includes:

e ^(l) , e ^(l-1) can be used in the Concatenate network.

After e ^(l-1) and e ^(l-2) are concatenated, they are passed through a deconvolution layer with a convolution kernel size of 2 and a step size of 2, and then the predicted value of the latent variable of the l+1th layer node is obtained through the initial residual network.

For example, in the embodiment of the present application, the embedding network (embedding) can be used to map the octree nodes from the 256-dimensional one-hot vector to the 16-dimensional continuous feature space, so that they occupy similar spaces in the space. The situation is similar in feature space.

For example, in the embodiment of the present application, at least one layer of the multi-layer network set up by the decoder may also include an additive network. Among them, the soft addition network (additive network) is used to replace the original hard addition, which greatly improves the flexibility of the network.

To sum up, the decoding method proposed in the embodiment of this application can rely on the octree structure, use the built embedding network, encoding network and decoding network, and introduce latent variables to capture the differences between sibling nodes in the same layer in the octree The correlation can achieve higher compression performance; at the same time, parallel encoding and decoding processing based on the level of the octree can effectively shorten the encoding and decoding time, greatly improve the encoding and decoding efficiency, and further improve the compression performance.

Embodiments of the present application provide a decoding method. At the decoding end, the decoder includes a second decoding network. The decoder decodes the code stream and determines the reconstructed value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; Among them, i is greater than 2; determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node into the second decoding network to determine the i-th layer node. Geometry information of layer nodes. It can be seen that in the embodiments of the present application, when encoding and decoding point clouds, based on the constructed octree structure, latent variables can be used to represent the spatial correlation between nodes at a unified level in the octree. , ultimately achieving higher compression performance; at the same time, based on the constructed network, parallel encoding and decoding processes are performed on nodes at the same level of the octree, which improves encoding and decoding efficiency and further improves compression performance.

Based on the above embodiments, in yet another embodiment of the present application, based on the same inventive concept as in the previous embodiments, Figure 20 is a schematic structural diagram of an encoder. As shown in Figure 20, the encoder 10 may include: a first determination Unit 11, encoding unit 12, generation unit 13; where,

The first determination unit 11 is configured to input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network, and determine the embedding characteristics of the i-th layer node; wherein i is greater than 2 ; Input the embedded features of the i-th layer node into the encoding network to determine the latent variable of the i-th layer node; determine the reconstructed value of the residual of the i-th layer node based on the latent variable of the i-th layer node , and determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node;

The encoding unit 12 is configured to write the reconstructed value of the residual of the i-th layer node into the bitstream;

The generating unit 13 is configured to input the reconstructed value of the latent variable of the i-th layer node to the first decoding network and generate a code stream of the geometric information of the i-th layer node.

It can be understood that in this embodiment, the "unit" may be part of a circuit, part of a processor, part of a program or software, etc., and of course may also be a module, or may be non-modular. Moreover, each component in this embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software function modules.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially either The part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions to make a computer device (can It is a personal computer, server, or network device, etc.) or processor that executes all or part of the steps of the method described in this embodiment. The aforementioned storage media include: U disk, mobile hard disk, Read Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk and other media that can store program code.

Therefore, embodiments of the present application provide a computer-readable storage medium for use in the encoder 10. The computer-readable storage medium stores a computer program. When the computer program is executed by the first processor, any of the foregoing embodiments can be implemented. The method described in one item.

Based on the composition of the above-mentioned encoder 10 and the computer-readable storage medium, Figure 21 is a schematic diagram 2 of the composition structure of the encoder. As shown in Figure 21, the encoder 10 may include: a first memory 14 and a first processor 15. Communication interface 16 and first bus system 17 . The first memory 14 , the first processor 15 , and the first communication interface 16 are coupled together through a first bus system 17 . It can be understood that the first bus system 17 is used to realize connection communication between these components. In addition to the data bus, the first bus system 17 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the various buses are labeled as first bus system 17 in FIG. 9 . in,

The first communication interface 16 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

The first memory 14 is used to store computer programs that can run on the first processor;

The first processor 15 is configured to input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network when running the computer program, and determine the i-th layer The embedded features of the node; where i is greater than 2; input the embedded features of the i-th layer node into the encoding network to determine the latent variable of the i-th layer node; determine based on the latent variable of the i-th layer node The reconstructed value of the residual of the i-th layer node, and determine the reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node; The reconstructed value of the variable is input to the first decoding network to generate a code stream of the geometric information of the i-th layer node.

It can be understood that the first memory 14 in the embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically removable memory. Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. Volatile memory may be Random Access Memory (RAM), which is used as an external cache. By way of illustration, but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) and Direct Rambus RAM (DRRAM). The first memory 14 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

The first processor 15 may be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by instructions in the form of hardware integrated logic circuits or software in the first processor 15 . The above-mentioned first processor 15 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or an off-the-shelf programmable gate array (Field Programmable Gate Array, FPGA). or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Each method, step and logical block diagram disclosed in the embodiment of this application can be implemented or executed. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the first memory 14. The first processor 15 reads the information in the first memory 14 and completes the steps of the above method in combination with its hardware.

It will be understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more Application Specific Integrated Circuits (ASIC), Digital Signal Processing (DSP), Digital Signal Processing Device (DSP Device, DSPD), programmable Logic device (Programmable Logic Device, PLD), Field-Programmable Gate Array (FPGA), general-purpose processor, controller, microcontroller, microprocessor, and other devices used to perform the functions described in this application electronic unit or combination thereof. For software implementation, the technology described in this application can be implemented through modules (such as procedures, functions, etc.) that perform the functions described in this application. Software code may be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

Optionally, as another embodiment, the first processor 15 is further configured to perform the method described in any one of the preceding embodiments when running the computer program.

FIG. 22 is a schematic diagram of a structure of a decoder. As shown in FIG. 22 , the decoder 20 may include: a decoding unit 21 and a second determining unit 22; wherein:

The decoding unit 21 is configured to decode the code stream;

The second determination unit 22 is configured to determine the reconstruction value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; according to the reconstruction value of the residual of the i-th layer node Determine the reconstructed value of the latent variable of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node to the second decoding network to determine the geometric information of the i-th layer node.

It can be understood that in this embodiment, a "unit" can be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional module.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially either The part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions to make a computer device (can It is a personal computer, server, or network device, etc.) or processor that executes all or part of the steps of the method described in this embodiment. The aforementioned storage media include: U disk, mobile hard disk, Read Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk and other media that can store program code.

Therefore, embodiments of the present application provide a computer-readable storage medium for use in the decoder 20 . The computer-readable storage medium stores a computer program. When the computer program is executed by the first processor, any of the foregoing embodiments can be implemented. The method described in one item.

Based on the composition of the decoder 20 and the computer-readable storage medium described above, Figure 23 is a schematic diagram 2 of the composition of the decoder. As shown in Figure 23, the decoder 20 may include: a second memory 23 and a second processor 24. Communication interface 25 and second bus system 26 . The second memory 23, the second processor 24, and the second communication interface 25 are coupled together through a second bus system 26. It can be understood that the second bus system 26 is used to realize connection communication between these components. In addition to the data bus, the second bus system 26 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the various buses are labeled as second bus system 26 in FIG. 11 . in,

The second communication interface 25 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

The second memory 23 is used to store computer programs that can run on the second processor;

The second processor 24 is used to decode the code stream when running the computer program and determine the reconstruction value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein, i is greater than 2; according to The reconstructed value of the residual of the i-th layer node determines the reconstructed value of the latent variable of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input to the second decoding network, and the reconstructed value of the latent variable of the i-th layer node is determined. The geometric information of the i-th layer node.

It can be understood that the second memory 23 in the embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically removable memory. Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. Volatile memory may be Random Access Memory (RAM), which is used as an external cache. By way of illustration, but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) and Direct Rambus RAM (DRRAM). The second memory 23 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

The second processor 24 may be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by instructions in the form of hardware integrated logic circuits or software in the second processor 24 . The above-mentioned second processor 24 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or an off-the-shelf programmable gate array (Field Programmable Gate Array, FPGA). or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Each method, step and logical block diagram disclosed in the embodiment of this application can be implemented or executed. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the second memory 23. The second processor 24 reads the information in the second memory 23 and completes the steps of the above method in combination with its hardware.

It will be understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more Application Specific Integrated Circuits (ASIC), Digital Signal Processing (DSP), Digital Signal Processing Device (DSP Device, DSPD), programmable Logic device (Programmable Logic Device, PLD), Field-Programmable Gate Array (FPGA), general-purpose processor, controller, microcontroller, microprocessor, and other devices used to perform the functions described in this application electronic unit or combination thereof. For software implementation, the technology described in this application can be implemented through modules (such as procedures, functions, etc.) that perform the functions described in this application. Software code may be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

Embodiments of the present application provide an encoder and a decoder. When encoding and decoding point clouds, based on the constructed octree structure, latent variables can be used to represent the space between nodes at a unified level in the octree. Correlation can ultimately achieve higher compression performance; at the same time, based on the constructed network, parallel encoding and decoding is performed on nodes at the same level of the octree, which improves encoding and decoding efficiency and further improves compression performance.

It should be noted that in the embodiments of the present application, the terms "comprising", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only Includes those elements and also includes other elements not expressly listed or inherent in such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article or apparatus that includes that element.

The above serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

The methods disclosed in several method embodiments provided in this application can be combined arbitrarily to obtain new method embodiments without conflict.

The features disclosed in several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Industrial applicability

Embodiments of the present application provide an encoding and decoding method, an encoder, a decoder, and a storage medium. On the encoding side, the encoder includes a first embedding network, an encoding network, and a first decoding network. The encoder converts point clouds into octrees. The geometric information of the i-th layer node obtained by the division is input to the first embedding network to determine the embedding characteristics of the i-th layer node; where i is greater than 2; the embedding characteristics of the i-th layer node are input to the encoding network to determine the i-th layer node of the latent variable; determine the reconstruction value of the residual of the i-th layer node based on the latent variable of the i-th layer node, and determine the reconstruction value of the latent variable of the i-th layer node based on the reconstruction value of the residual of the i-th layer node; The reconstructed value of the latent variable of the i-layer node is input to the first decoding network to generate a code stream of the geometric information of the i-th layer node. At the decoding end, the decoder includes a second decoding network. The decoder decodes the code stream and determines the reconstruction value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; where i is greater than 2; according to the i-th layer node The reconstructed value of the residual determines the reconstructed value of the latent variable of the i-th layer node; the reconstructed value of the latent variable of the i-th layer node is input to the second decoding network to determine the geometric information of the i-th layer node. It can be seen that in the embodiments of the present application, when encoding and decoding point clouds, based on the constructed octree structure, latent variables can be used to represent the spatial correlation between nodes at a unified level in the octree. , ultimately achieving higher compression performance; at the same time, based on the constructed network, parallel encoding and decoding processes are performed on nodes at the same level of the octree, which improves encoding and decoding efficiency and further improves compression performance.

Claims (37)

An encoding method, applied to an encoder. The encoder includes a first embedding network, a coding network, and a first decoding network. The method includes: Input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network, and determine the embedding characteristics of the i-th layer node; wherein i is greater than 2; Input the embedded features of the i-th layer node into the encoding network to determine the latent variables of the i-th layer node; Determine the reconstructed value of the residual of the i-th layer node according to the latent variable of the i-th layer node, and determine the reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node; The reconstructed value of the latent variable of the i-th layer node is input to the first decoding network to generate a code stream of the geometric information of the i-th layer node. The method of claim 1, further comprising: The encoder is provided with a multi-layer network, wherein at least one layer of the multi-layer network includes the first embedding network, the encoding network, and the first decoding network. The method according to claim 2, wherein the i-th layer network in the multi-layer network performs parallel encoding processing on the geometric information of the i-th layer node. The method of claim 3, further comprising: The input of the encoding network of the i-th layer network includes the latent variable of the i+1-th layer node output by the encoding network of the i+1-th layer network. The method of claim 4, further comprising: The embedded features of the i-th layer node and the latent variables of the i+1-th layer node are input into the encoding network of the i-th layer network to determine the latent variables of the i-th layer node. The method according to any one of claims 3-4, wherein the method further includes: Determine the residual of the i-th layer node according to the predicted value of the latent variable of the i-th layer node and the latent variable of the i-th layer node; The residual of the i-th layer node is quantized to determine the reconstruction value of the residual of the i-th layer node. The method of claim 6, further comprising: The reconstructed value of the latent variable of the i-th layer node is determined according to the reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node. The method of claim 6, wherein the encoder further comprises a subtraction network, the method further comprising: The predicted value of the latent variable of the i-th layer node and the latent variable of the i-th layer node are input to the subtraction network to determine the residual of the i-th layer node. The method of claim 7, wherein the encoder further comprises an additive network, the method further comprising: The reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node are input to the addition network, and the reconstructed value of the latent variable of the i-th layer node is determined. The method of claim 3, further comprising: The input of the first decoding network of the i-th layer network includes: the embedding feature of the i-2 layer node output by the first embedding network of the i-2 layer network, the first embedding network output of the i-1 layer network The embedded features of the i-1th layer node, and the reconstructed value of the latent variable of the i-1th layer node obtained by the i-1th layer network. The method of claim 10, wherein the method further includes: The reconstructed value of the latent variable of the i-th layer node is input to the first decoding network of the i-th layer network, and the predicted value of the latent variable of the i+1-th layer node is determined. The method of claim 11, wherein the method further includes: The reconstructed values of the latent variables of the i-th layer nodes, the embedded features of the i-th layer nodes, the embedded features of the i-1-th layer nodes, and the reconstructed values of the latent variables of the i-1-th layer nodes are input into the first decoding network of the i-th layer network to determine the predicted values of the latent variables of the i+1-th layer nodes. The method of claim 10, wherein the method further includes: The reconstructed value of the latent variable of the i-th layer node, the embedded feature of the i-2th layer node, the embedded feature of the i-1th layer node, and the latent variable of the i-1th layer node are The reconstructed value is input to the first decoding network of the i-th layer network to generate a code stream of the geometric information of the i-th layer node. The method of claim 13, wherein the method further includes: Based on the first decoding network, the probability parameter of the i-th layer node is determined, and a code stream of the geometric information of the i-th layer node is generated according to the probability parameter of the i-th layer node. The method of claim 3, further comprising: For the first-layer nodes obtained by octree division, the latent variables of the first-layer nodes output by the coding network of the first-layer network are quantified, and the reconstructed values of the latent variables of the first-layer nodes are determined. ; Output the reconstructed value of the latent variable and the preset embedding feature of the first layer node to the first decoding network of the first layer network, and output the code stream of the geometric information of the first layer node and the second The predicted value of the latent variable of the layer node. The method of claim 3, wherein, The encoding network includes a sparse convolution network, a ReLU activation function, an initial residual network, and a feature fusion network. The method of claim 3, wherein, The first embedding network includes a sparse convolutional network. The method of claim 3, wherein, The first decoding network includes a sparse convolutional network, a deconvolution layer, an initial residual network, a feature fusion network, a ReLU activation function, and a deconvolution layer. A decoding method, applied to a decoder, the decoder including a second decoding network, the method includes: Decode the code stream and determine the reconstruction value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; where i is greater than 2; Determine the reconstructed value of the latent variable of the i-th layer node according to the reconstructed value of the residual of the i-th layer node; The reconstructed value of the latent variable of the i-th layer node is input to the second decoding network to determine the geometric information of the i-th layer node. The method of claim 19, wherein the decoder further includes a second embedding network, the method further comprising: The geometric information of the i-th layer node is input to the second embedding network, and the embedding characteristics of the i-th layer node are determined. The method of claim 20, wherein the method further includes: The decoder is provided with a multi-layer network, wherein at least one layer of the multi-layer network includes the second embedding network and the second decoding network. The method according to claim 21, wherein the i-th layer network in the multi-layer network performs decoding processing on the i-th layer node. The method of claim 22, wherein the method further includes: The reconstructed value of the latent variable of the i-th layer node is determined according to the reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node. The method of claim 23, wherein the decoder further comprises an additive network, the method further comprising: The reconstructed value of the residual of the i-th layer node and the predicted value of the latent variable of the i-th layer node are input to the addition network, and the reconstructed value of the latent variable of the i-th layer node is determined. The method of claim 22, wherein the method further includes: The input of the second decoding network of the i-th layer network includes: the embedding feature of the i-2 layer node output by the second embedding network of the i-2 layer network, the second embedding network output of the i-1 layer network The embedded features of the i-1th layer node, and the reconstructed value of the latent variable of the i-1th layer node obtained by the i-1th layer network. The method of claim 25, wherein the method further includes: The reconstructed values of the latent variables of the i-th layer nodes, the embedded features of the i-2 layer nodes, the embedded features of the i-1 layer nodes, and the reconstructed values of the latent variables of the i-1 layer nodes are input into the second decoding network of the i-th layer network to determine the geometric information of the i-th layer nodes. The method according to claim 26, wherein the method further comprises: Based on the second decoding network, the probability parameter of the i-th layer node is determined, and the geometric information of the i-th layer node is generated according to the probability parameter of the i-th layer node. The method according to claim 25, wherein the method further comprises: The reconstructed value of the latent variable of the i-th layer node is input to the second decoding network, and the predicted value of the latent variable of the i+1-th layer node is determined. The method of claim 28, wherein the method further includes: The reconstructed value of the latent variable of the i-th layer node, the embedded feature of the i-th layer node, the embedded feature of the i-1th layer node, and the reconstructed value of the latent variable of the i-1th layer node The second decoding network input to the i-th layer network determines the predicted value of the latent variable of the i+1-th layer node. The method according to claim 22, wherein the method further comprises: For the first layer node obtained by octree division, the reconstructed value of the latent variable and the preset embedded feature of the first layer node are output to the second decoding network of the first layer network, and the first layer node is output. The geometric information of the layer nodes and the predicted values of the latent variables of the second layer nodes. The method of claim 22, wherein: The second embedding network includes a sparse convolutional network. The method of claim 22, wherein: The second decoding network includes a sparse convolution network, a deconvolution layer, an initial residual network, a feature fusion network, a ReLU activation function, and a deconvolution layer. An encoder, the encoder includes a first determination unit, a coding unit, and a generating unit; wherein, The first determination unit is configured to input the geometric information of the i-th layer node obtained by dividing the point cloud according to the octree into the first embedding network, and determine the embedding characteristics of the i-th layer node; wherein i is greater than 2; Input the embedded features of the i-th layer node into the encoding network to determine the latent variable of the i-th layer node; determine the reconstructed value of the residual of the i-th layer node based on the latent variable of the i-th layer node, And determine the reconstructed value of the latent variable of the i-th layer node based on the reconstructed value of the residual of the i-th layer node; The coding unit is configured to write the reconstructed value of the residual of the i-th layer node into the code stream; The generating unit is configured to input the reconstructed value of the latent variable of the i-th layer node to the first decoding network and generate a code stream of the geometric information of the i-th layer node. An encoder comprises a first memory and a first processor; wherein: The first memory is used to store a computer program capable of running on the first processor; The first processor is configured to execute the method according to any one of claims 1 to 18 when running the computer program. A decoder, the decoder includes a decoding unit and a second determination unit; wherein, The decoding unit is configured to decode the code stream; The second determination unit is configured to determine the reconstruction value of the residual of the i-th layer node obtained by dividing the point cloud according to the octree; wherein i is greater than 2; according to the reconstruction value of the residual of the i-th layer node Determine the reconstructed value of the latent variable of the i-th layer node; input the reconstructed value of the latent variable of the i-th layer node to the second decoding network to determine the geometric information of the i-th layer node. A decoder, the decoder includes a second memory and a second processor; wherein, The second memory is used to store a computer program that can be run on the second processor; The second processor is configured to perform the method according to any one of claims 19 to 32 when running the computer program. A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed, the method of any one of claims 1 to 18 is implemented, or the method of claim 19 is implemented. The method described in any one of to 32.

Images