CN110070096B - Local frequency domain descriptor generation method and device for non-rigid shape matching - Google Patents

Abstract

The invention belongs to the field of computer vision, and in particular relates to a method and device for generating local frequency domain descriptors for non-rigid shape matching, aiming to solve the problem of obtaining local frequency domain descriptors under the influence of different spatial resolutions, sampling and various transformations. Bad stick problem. The method of the invention includes: based on the Laplace-Beltramian operator, for each vertex in the surface triangular mesh model, extracting the local point features of the three-dimensional shape in the frequency domain; A vertex frequency domain image corresponding to each vertex of the three-dimensional shape is obtained, and a local frequency domain descriptor for non-rigid shape matching is obtained through a ternary neural network. The local point features extracted by the present invention have good robustness to three-dimensional shapes under various resolutions, triangulations, scales and rotations, and based on this, the ternary neural network of the present invention can accurately and robustly Rod acquisition of local frequency-domain descriptors for non-rigid shape matching.

Description

Method and device for generating local frequency-domain descriptors for non-rigid shape matching

technical field

The invention belongs to the field of computer vision, and in particular relates to a method and device for generating local frequency domain descriptors for non-rigid shape matching.

Background technique

Because of the simplicity, effectiveness, and flexibility of three-dimensional shapes (mesh models), they have become a widely used form of discrete representation of three-dimensional data in the field of computer vision. With the development of 3D scanning equipment and computer vision reconstruction technology, it has become easier to obtain detailed 3D shapes. Consequently, the importance of 3D shape analysis such as shape matching, segmentation, correspondence and retrieval has been significantly increased. Designing a local descriptor for surface shape is a fundamental problem in computer vision and is the key and foundation of other advanced applications.

The methods for generating local descriptors in the prior art mainly include methods in the spatial domain, methods based on embedding, and methods based on deep learning.

The method in the space domain mainly obtains a statistical histogram to represent the local vertex features by counting the vertex feature information (such as quantity, angle, direction, etc.) in the local space. For example, 3DSC (3D shape context, three-dimensional shape environment) descriptor counts the number of vertices in each histogram block, SHOT (Signature of histogram of orientations, orientation histogram feature) counts the angle of the normal vector of the neighborhood points, Mesh-HOG (Mesh-HOG (Mesh) Histogram of oriented gradients) describes the direction of the sub-statistic gradient, and RoPS (The rotational projection statistics) describes multiple features of the sub-statistic projection two-dimensional plane, etc. Obviously, these traditional methods are all based on spatial domain features, which are susceptible to various transformations and deformations of resolution and shape.

Most of the embedding-based methods propose intrinsic descriptors, which aim to solve the isometric deformation problem. The methods of generating embedded descriptors are generally based on the Laplace-Beltrami operator. Shape-DNA uses the eigenvalues of the Laplace-Beltrami operator as features. GPS (Global Point Signature) combines eigenvalues and eigenfunctions to generate features. HKS (Heat Kernel Signature), scale-invariant HKS (scale-invariant Heat Kernel Signature) and OSD (optimal spectral descriptors) are proposed based on diffusion geometry. However, most of these embedding methods are based on global intrinsic properties, are not robust to local detail descriptions, and are mostly scale-sensitive.

Deep learning-based methods have recently been used to extract shape descriptors. Deep learning-based methods mainly include multi-view, voxelization, and methods that learn directly on 3D shapes. However, due to the information learned by these methods and the structure of the shape (eg grid scale, spatial resolution, sampling, etc.) are correlated, so their generalization ability is flawed.

Therefore, how to provide a method to solve the above problems (different spatial resolutions, sampling and various transformations) is a problem that those skilled in the art need to solve at present.

SUMMARY OF THE INVENTION

In order to solve the above problems in the prior art, that is, to solve the problem of poor robustness of the local frequency domain descriptors obtained under the influence of different spatial resolutions, sampling and various transformations, the first aspect of the present invention proposes a A local frequency-domain descriptor generation method for rigid shape matching, which includes the following steps:

Step S10, based on the Laplacian-Beltramian operator, for each vertex in the surface triangular mesh model, extract the three-dimensional shape local point feature in the frequency domain;

Step S20, based on the local feature points, obtain a vertex frequency domain image corresponding to each vertex of the three-dimensional shape, and obtain a local frequency domain descriptor for non-rigid shape matching through a ternary neural network.

In some preferred embodiments, "based on the Laplacian-Beltramian operator, for each vertex in the surface triangular mesh model, the local point features of the three-dimensional shape are extracted in the frequency domain", and the method is:

Based on the surface continuous function f of the surface triangular mesh model of the three-dimensional shape, calculate the Laplace-Beltrium matrix L of the surface triangular mesh model;

Perform eigendecomposition on the matrix L to obtain eigenvectors and eigenvalues;

并将其在所述矩阵L的特征向量下展开，得到频域展开系数σ_j；Extend the surface continuous function f to a discrete function

and expand it under the eigenvectors of the matrix L to obtain the frequency domain expansion coefficient σ _j ;

According to the eigendecomposition of the matrix L and the frequency domain expansion coefficient σ _j , the discrete Dirichlet energy is calculated

计算离散狄利克雷能量

并将所述能量

在频域中展开得到通用的频域特征；Extend the high-dimensional continuous function F based on the surface continuous function f to high-dimensional discrete functions

Calculate discrete Dirichlet energies

and the energy

Expand in the frequency domain to obtain general frequency domain features;

设置为局部面片三维坐标信息X，根据连续的能量E(X)求解离散能量

取所述离散能量

在频域中展开的前Q维得到所述局部点特征；其中Q为第一设定值。high-dimensional discrete functions

Set to the three-dimensional coordinate information X of the local patch, and solve the discrete energy according to the continuous energy E(X)

Take the discrete energy

The first Q dimension expanded in the frequency domain obtains the local point feature; wherein Q is the first set value.

In some preferred embodiments, "calculate the Laplacian-Beltramian matrix L of the surface triangular mesh model based on the surface continuous function f of the three-dimensional shape surface triangular mesh model", and the method is:

通过下式计算所述表面三角网格模型的拉普拉斯－贝尔特拉米矩阵L中的元素L_ij：Get the discrete function of the surface continuous function f

Elements L _ij in the Laplacian- Beltrami matrix L of the surface triangular mesh model are calculated by:

Among them, α _ij and β _ij are the two angles opposite to the edge {i, j} in the surface triangular mesh model, α _i is the Voronoi polygon area of the vertex v _i , and k is the number of adjacent vertices.

In some preferred embodiments, "decompose the matrix L to obtain eigenvectors and eigenvalues", the method is:

Decompose matrix L into two symmetric matrices T and A,

TΦ _i =λ _i AΦ _i ,i=0,1,...,N-1;

in,

A _ii =a _i

The ARPACK method is used to solve the problem, and the eigenvector Φ _i and the eigenvalue λ _i are obtained again, and N is the number of vertices of the surface triangular mesh model.

并将其在所述矩阵L的特征向量下展开，得到频域展开系数σ_j”，其方法为：In some preferred embodiments, "extend the surface continuous function f to a discrete function

And expand it under the eigenvector of the matrix L to obtain the frequency domain expansion coefficient σ _j ", the method is:

Wherein, Φ _j is the j-th eigenvector of the matrix L.

”，其方法为：In some preferred embodiments, "According to the eigendecomposition of the matrix L, the frequency domain expansion coefficient σ _j , the discrete Dirichlet energy is calculated

", the method is:

为连续实函数f的狄利克雷能量对应的离散的能量形式，N为顶点的个数，λ_j为矩阵L的第j个特征值。in,

is the discrete energy form corresponding to the Dirichlet energy of the continuous real function f, N is the number of vertices, and λ _j is the jth eigenvalue of the matrix L.

计算离散狄利克雷能量

并将所述能量

在频域中展开得到通用的频域特征”，其方法为：In some preferred embodiments, "Extend a high-dimensional continuous function F based on a surface continuous function f to a high-dimensional discrete function

Calculate discrete Dirichlet energies

and the energy

Expand in the frequency domain to get general frequency domain features", the method is:

Among them, sf is a general frequency domain feature, λ _N-1 is an eigenvalue, and σ _iN-1 is a high-dimensional frequency domain expansion coefficient.

设置为局部面片三维坐标信息X，根据连续的能量E(X)求解离散能量

取所述离散能量

在频域中展开的前Q维得到所述局部点特征”，其方法为：In some preferred embodiments, "the high-dimensional discrete function

Set to the three-dimensional coordinate information X of the local patch, and solve the discrete energy according to the continuous energy E(X)

Take the discrete energy

The first Q-dimension unrolled in the frequency domain obtains the local point feature", and the method is:

Among them, LPS is the obtained local point feature.

In some preferred embodiments, "based on the local feature points, a vertex frequency domain image corresponding to each vertex of the three-dimensional shape is obtained, and a local frequency domain descriptor for non-rigid shape matching is obtained through a ternary neural network. ", the method is:

encoding the local point feature into Q images of the first set size;

For each vertex on the three-dimensional shape surface triangular mesh model, generate the corresponding vertex frequency domain image, and obtain the vertex frequency domain image set;

Based on the vertex frequency domain image set, a local frequency domain descriptor for non-rigid shape matching is obtained through a preset ternary neural network.

In some preferred embodiments, "encode the local point features into Q images of a first set size", the method is:

Take Q as 16, and the first set size is 8*8;

The local point features are encoded into 16 geometric images of size 8*8 using the geometric image method.

In some preferred embodiments, "for each vertex on the three-dimensional shape surface triangular mesh model, a corresponding vertex frequency domain image is generated", and the method is:

Initialize a 32*32 empty image, fill 16 feature-encoded images of 8*8 local point features into the initialized empty image, and place the minimum and maximum eigenvalues in the upper left and lower right corners, respectively, to generate 32 *32 size vertex frequency domain image.

In some preferred embodiments, the ternary neural network is composed of three identical ConvNet convolutional networks, and the training samples include three-dimensional shapes and corresponding descriptors; when the ternary neural network is trained, the three-dimensional The shape adopts the method in step S10 and step S20 to obtain the corresponding vertex frequency domain image set and input it to the ternary neural network.

In a second aspect of the present invention, a device for generating a local frequency domain descriptor for non-rigid shape matching is proposed, the device includes a local point feature generation module and a local frequency domain descriptor acquisition module;

The local point feature generation module is configured to obtain the local point feature in the frequency domain of each vertex in the three-dimensional shape surface triangular mesh model based on the Laplace-Beltrami operator;

The local frequency domain descriptor obtaining module is configured to obtain a vertex frequency domain image corresponding to each vertex of the three-dimensional shape based on the local feature points, and obtain a local frequency domain image for non-rigid shape matching through a ternary neural network. frequency domain descriptor.

In a third aspect of the present invention, a storage device is provided, wherein a plurality of programs are stored, and the programs are adapted to be loaded and executed by a processor to realize the above-mentioned method for generating local frequency-domain descriptors for non-rigid shape matching.

In a fourth aspect of the present invention, a processing device is provided, including a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded by the processor And execute to achieve the above-mentioned local frequency-domain descriptor generation method for non-rigid shape matching.

Beneficial effects of the present invention:

The local point features extracted by the method of the present invention have better robustness to three-dimensional shapes under various resolutions, triangulation (sampling), scales and rotations, and based on this, the ternary neural network of the present invention Local frequency-domain descriptors for non-rigid shape matching can be obtained accurately and robustly.

Description of drawings

Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

1 is a schematic flowchart of a method for generating local frequency domain descriptors for non-rigid shape matching according to an embodiment of the present invention;

Fig. 2 is an example diagram of matching results for different resolutions, samplings, scales and rotations of descriptors obtained by adopting the present invention;

3 is an example diagram of an angle and a Voronoi area in a discrete Laplace-Beltrami operator in an embodiment of the present invention;

FIG. 4 is an exemplary diagram of a structure model of different shapes generated in an embodiment of the present invention;

5 is an exemplary diagram of a vertex frequency domain image VSI at different positions in an embodiment of the present invention;

6 is a graph showing the results of LPS features at different resolutions and triangulation models in an embodiment of the present invention;

7 is a graph showing the results of descriptors in different resolutions and triangulation models in an embodiment of the present invention;

8 is a graph showing the comparison of the results of descriptors and LPS features at 6890 and 8K resolutions and other methods in an embodiment of the present invention;

FIG. 9 is a graph comparing the results of descriptors and LPS features at 6890 and 12K resolutions and other methods in an embodiment of the present invention;

FIG. 10 is a graph comparing the results of descriptors and LPS features on the rotation model set and other methods in an embodiment of the present invention;

FIG. 11 is a graph comparing the results of descriptors and LPS features on different scale model sets and other methods in an embodiment of the present invention;

FIG. 12 is a result comparison graph of a corresponding frame and other corresponding frames in an embodiment of the present invention;

FIG. 13 is a schematic diagram of a framework of an apparatus for generating a local frequency domain descriptor for non-rigid shape matching according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not All examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict.

A method for generating local frequency domain descriptors for non-rigid shape matching of the present invention includes the following steps:

Step S10, based on the Laplace-Beltrami operator, for each vertex in the surface triangular mesh model, extract the local point feature of the three-dimensional shape in the frequency domain;

Step S20, based on the local feature points, obtain a vertex frequency domain image corresponding to each vertex of the three-dimensional shape, and obtain a local frequency domain descriptor for non-rigid shape matching through a ternary neural network.

FIG. 1 is a schematic flowchart of a method for generating a local frequency-domain descriptor for non-rigid shape matching according to an embodiment of the present invention, and the descriptor obtained in the schematic diagram has 256 dimensions. 2 is a schematic diagram of the matching results for different resolutions, sampling, scale and rotation of descriptors obtained by the present invention. The model on the far left is a reference model with 6890 points, followed by models with 12K high resolution and different samplings, From the scaling model and the rotation model, it can be seen that the present invention can perform robust matching on different shape structures.

In order to describe the present invention more clearly, each step in the present invention will be described in detail below with reference to the accompanying drawings.

A method for generating local frequency domain descriptors for non-rigid shape matching according to an embodiment of the present invention includes steps S10 and S20.

Step S10, based on the Laplacian-Beltrium operator, for each vertex in the surface triangular mesh model, extract the local point features of the three-dimensional shape in the frequency domain. The local point features obtained by this step method are robust to each resolution, triangulation (sampling), scale and rotation of the 3D shape. This step can be further subdivided into steps S101-S106.

Step S101 , based on the surface continuous function f of the surface triangular mesh model of the three-dimensional shape, calculate the Laplacian-Beltrium matrix L of the surface triangular mesh model.

该函数中

为连续的表面、R为实数域，则可以求出该函数的Dirichlet(狄利克雷)能量，如式(1)所示：The surface continuous real function f on the surface of the known three-dimensional shape:

in this function

is a continuous surface and R is a real number field, then the Dirichlet energy of the function can be obtained, as shown in formula (1):

为梯度，f(v)为顶点v上的特征，v为连续表面上的点。in,

is the gradient, f(v) is the feature on the vertex v, and v is the point on the continuous surface.

中，若存在表面形状的连续实函数f:

的离散形式

V→R，则Δf在顶点v_i的离散形式如式(2)所示：As shown in Figure 3, in the discrete grid

, if there is a continuous real function f of the surface shape:

discrete form of

_V →R, then the discrete form of Δf at vertex vi is shown in formula (2):

Among them, V is the discrete vertex, a _i is the Voronoi polygon area of the vertex v _i , N(vi ) _represents a circle of neighborhood points of the vertex v _i , α _ij and β _ij represent two and edges {i, j} Opposite angles; V _i and V _j in Fig. 3 are two discrete vertices.

Therefore, the discrete Laplace-Beltrami matrix L can approximate the Laplace-Beltrami operator Δ, as shown in equation (3):

where k is the number of adjacent vertices.

Step S102, eigendecomposition is performed on the matrix L to obtain eigenvalues λ _i and eigenvectors Φ _i .

上一组连续的正交基函数{φ_i|i＝0,1,...,N-1}最小化Dirichlet(狄利克雷)能量，那么问题的解(这组正交基函数)是Laplace-Beltrami算子Δ的前N个特征函数(N为表面三角网格模型顶点个数)，且满足式(4)：If there is a continuous surface

The previous set of continuous orthonormal basis functions {φ _i |i=0,1,...,N-1} minimizes the Dirichlet energy, then the solution to the problem (this set of orthonormal basis functions) is The first N characteristic functions of the Laplace-Beltrami operator Δ (N is the number of vertices of the surface triangular mesh model), and satisfy the formula (4):

Δφ _i =λ _i φ _i ,i=0,1,...,N-1 (4)

In formula (4), Δ is the Laplace-Beltrami operator, φ _i is the i-th feature function, {λ _i |i=0,1,...,N-1} is the first N features arranged in increasing order value, it corresponds to the discrete case as shown in formula (5):

LΦ _i =λ _i Φ _i ,i=0,1,...,N-1 (5)

Among them, Φ _i is a discrete orthonormal basis (that is, an eigenvector), and λ _i is a decomposed eigenvalue.

The Laplace-Beltrami matrix L is decomposed into two symmetric matrices T and A, so that equation (6) is obtained from equation (5),

TΦ _i =λ _i AΦ _i ,i=0,1,...,N-1 (6)

Among them, A _ii =a _i , and T _ij is shown in formula (7)

The solution is used to obtain the eigenvector Φ _i and the eigenvalue λ _i .

When the eigenvector Φ _i and the eigenvalue λ _i are calculated by using the ARPACK method for solving the generalized eigenvalues of the matrix, the eigenvector Φ _i is orthogonal to A, specifically <Φ _i ,Φ _j > _A =Φ _i ^T AΦ _j , Among them, Φ _i and Φ _j are two different eigenvectors respectively.

并将其在所述矩阵L的特征向量下展开，得到频域展开系数σ_j。Step S103, extending the surface continuous function f to a discrete function

And expand it under the eigenvectors of the matrix L to obtain the frequency domain expansion coefficient σ _j .

Similar to the Fourier expansion, the continuous function f can be expanded by the basis function, as shown in equation (8):

Among them, σ _j is the frequency domain expansion coefficient, and φ _j is the jth characteristic function. The expansion coefficient σ _j can be obtained by extending to the discrete case, as shown in equation (9).

Step S104, according to the eigendecomposition of the matrix L and the frequency domain expansion coefficient σ _j , calculate the discrete Dirichlet energy

根据矩阵L的特征分解和频域展开系数σ_j求得所述离散Dirichlet(狄利克雷)能量

的公式如式(10)所示。According to the continuous Dirichlet energy formula, the discrete energy form of the continuous real function f is obtained as

The discrete Dirichlet energy is obtained according to the eigendecomposition of the matrix L and the frequency domain expansion coefficient σ _j

The formula is shown in formula (10).

Among them, N is the number of vertices, and λj is the _jth eigenvalue.

计算离散狄利克雷能量

并将所述能量

在频域中展开得到通用的频域特征。Step S105, extending the high-dimensional continuous function F based on the surface continuous function f to a high-dimensional discrete function

Calculate discrete Dirichlet energies

and the energy

Expand in the frequency domain to get general frequency domain features.

For a high-dimensional continuous function F=(f ₁ , f ₂ ,...,f _d ):S→R ^d , the Dirichlet energy formula is shown in equation (11):

为高维函数的梯度，

为低维函数的梯度。Among them, f ₁ , f ₂ ,...,f _d are respectively d low-dimensional continuous functions,

is the gradient of the high-dimensional function,

is the gradient of the low-dimensional function.

其离散Dirichlet能量

如式(12)所示：extension to higher dimensional functions

its discrete Dirichlet energy

As shown in formula (12):

Among them, σ _ij is the j-th frequency domain expansion coefficient under the i-th dimension, and λ _j is the j-th eigenvalue.

The specific formula of the general frequency domain feature obtained by expanding it in the frequency domain is shown in formula (13):

Among them, sf is a general frequency domain feature, λ _N-1 is the Nth eigenvalue, and σ _iN-1 is the Nth frequency domain expansion coefficient under the ith dimension.

设置为局部面片三维坐标信息X，根据连续的能量E(X)求解离散能量

取所述离散能量

在频域中展开的前Q维得到所述局部点特征；其中Q为第一设定值。Step S106, the high-dimensional discrete function

Set to the three-dimensional coordinate information X of the local patch, and solve the discrete energy according to the continuous energy E(X)

Take the discrete energy

The first Q dimension expanded in the frequency domain obtains the local point feature; wherein Q is the first set value.

The local patch is obtained by the local geodesic radius of the vertex, and the local three-dimensional coordinate information is shown in equation (14),

X=(x ₁ , x ₂ , x ₃ ): V→R ³ (14)

Among them, x ₁ , x ₂ , and x ₃ are three-dimensional coordinates.

For the case where the X function is continuous, the continuous energy E(X) is shown in equation (15),

为X函数的梯度，

为x函数的梯度，P为局部面片。in,

is the gradient of the X function,

is the gradient of the x function, and P is the local patch.

Extending it to the discrete case, as shown in Eq. (16),

Taking the first Q dimension of the discrete energy expansion in the frequency domain, the LPS feature is obtained, as shown in equation (17), where the value of Q is 16.

Existing methods of frequency domain features such as GPS and HKS avoid using extrinsic features to obtain a global intrinsic feature, and the local descriptiveness is not strong. Although the intrinsic feature is isometric invariant, it cannot be effectively solved for non-isometric deformation. Through experiments, it is found that many methods use non-intrinsic descriptors such as SHOT as input to achieve better results. In the present invention, vertex coordinate information is effectively introduced, and a new frequency domain feature is designed through the Dirichlet framework, so the present invention combines the intrinsic and extrinsic information to obtain a more discriminative local frequency domain descriptor. In addition, some extrinsic (such as SHOT) or intrinsic (such as HKS) methods are extremely sensitive to scale transformation, and the descriptor of the present invention maintains invariance on scale transformation.

In this embodiment, in order to test the effectiveness of the method, models of local structures with different shapes are generated. As shown in Figure 4, the three separate models on the left are models with different resolutions (6890, 8K, 15K) and triangulation. , the models on the right show models with different scales and different rotation directions.

The FAUST model library is a widely used standard dataset in the field of shape matching, which has rich details and more accurate correspondences. This example generates models of different shape structures (resolution, triangulation (sampling), scale, and rotation) based on this dataset.

Barthe,and MarioBotsch.Adaptive remeshing for real-time mesh deformation.Eurographics.2013；2、Yiqun Wang,Dong-Ming Yan,Xiaohan Liu,Chengcheng Tang,Jianwei Guo,XiaopengZhang,and Peter Wonka.Isotropic surface remeshing without large and smallangles.IEEE Trans.on Vis.and Comp.Graphics,2018.Since all FAUST model libraries have the same triangulation and resolution of 6890 points, in addition, a large number of models need an automated method to generate, this example refers to the existing remeshing method, and generates a The library of multi-resolution models is optimized and ensures that each shape contains a user-specified number of vertices. The remeshing method can be found in: 1. Marion Dunyach, David Vanderhaeghe,

Barthe,and MarioBotsch.Adaptive remeshing for real-time mesh deformation.Eurographics.2013; 2. Yiqun Wang,Dong-Ming Yan,Xiaohan Liu,Chengcheng Tang,Jianwei Guo,XiaopengZhang,and Peter Wonka.Isotropic surface remeshing without large and smallangles .IEEE Trans.on Vis.and Comp.Graphics, 2018.

In order to keep the corresponding relationship accurately, each original vertex is marked as "locked" in this embodiment, which means that the original vertex cannot be moved or deleted. Afterwards, vertices are added or subtracted in iterations through edge splitting and edge folding operations. In iterations, this embodiment applies a random edge flipping operation to obtain an irregular triangulation model. The next few steps of smoothing are done to avoid most cases where the triangle shape is too bad. At the same time, these operations will keep local details from being destroyed. For the generation of low-resolution models, this method uses Jing Ren, Adrien Poulenard, Peter Wonka, and Maks Ovsjanikov. Continuous and orientation-preserving correspondences via functional maps. ACM Trans. on Graphics, 37(6):248:1-248 :16, the 5K model provided in Dec.2018, this model is independently generated by a CVT (Centroidal Voronoi Tessellation, based on the center of gravity Voronoi diagram) method. Ultimately, six different resolution models were obtained (5K, 6890, 8K, 10K, 12K and 15K). For the CVT method used in this example, please refer to: Dong-Ming Yan, Guanbo Bao, Xiaopeng Zhang, and Peter Wonka. Low-resolution remeshing using the localized restricted voronoi diagram. IEEE Trans.on Vis.and Comp.Graphics, 20(10) : 1418-1427, 2014.

For models with different scales, this embodiment randomly selects one of five scaling factors (0.2, 0.5, 1.0, 2.0 and 4.0) for scaling. In addition, this embodiment also randomly rotates the model to generate model libraries with different rotation directions, and the effect is shown in FIG. 4 .

Step S20, based on the local feature points, obtain a vertex frequency domain image corresponding to each vertex of the three-dimensional shape, and obtain a local frequency domain descriptor for non-rigid shape matching through a ternary neural network. This step specifically includes steps S201-S203.

Based on the acquired characteristic LPS, the present invention designs a new and compact geometric image (geometry image)—VSI (vertex spectral image, vertex frequency domain image). VSI can effectively encode LPS and be used for neural network training, using existing The triplet (ternary) neural network can effectively use VSI to learn local descriptors for non-rigid matching, shown in Figure 2 as the stage of LPS features to 256-dimensional descriptors.

Step S201, encoding the local point features into Q images of a first set size.

Take Q as 16, and the first set size is 8*8;

The local point features are encoded into 16 geometric images of size 8*8 using the geometry image method. Different from previous methods, the previous methods usually encode low-level information (curvature, normal, etc.), which is very sensitive to different shape structures, and low-level features require large size (32*32) to obtain sufficient local information. , while LPS features are high-level features that can represent local details, thereby reducing the amount of data without reducing the amount of information.

Step S202, for each vertex on the three-dimensional shape surface triangular mesh model, generate a corresponding vertex frequency domain image, and obtain a vertex frequency domain image set.

Initialize a 32*32 empty image, fill 16 feature-encoded images of 8*8 local point features into the initialized empty image, and place the minimum and maximum eigenvalues in the upper left and lower right corners, respectively, to generate 32 *32 size vertex frequency domain image.

In the example of Fig. 5, vertex frequency domain images of three different parts (finger, navel, knee) are displayed. Each VSI consists of 16 small geometric images. This method can make full use of the characteristics of convolution in the training process of convolutional neural network, and learn useful information from 16 different frequency bands. In addition, this method can only be applied to features that are not related to coordinates (such as LPS in the present invention). If the features are related to coordinates, there will be problems related to the order in the encoding process. This embodiment uses multi-scale VSIs as input, and at the same time, this embodiment generates 12 VSIs with different rotation directions as training data to learn the rotational invariance of geometric images.

Step S203, based on the vertex frequency domain image set, obtain a local frequency domain descriptor for non-rigid shape matching through a preset ternary neural network.

In this implementation, the ternary neural network consists of three identical ConvNet convolutional networks, and each ConvNet convolutional network consists of the following network structures: CONV128-3x3-/2+CONV256-3x3-/2+CONV512-3x3-/ 2+FC512+FC256, where CONVx represents the output of the convolutional layer with an x-dimensional feature map, 3x3 represents the size of the convolution kernel, /2 represents the stride of the pooling operation, and FCx represents a fully connected layer whose output is an x-dimensional vector.

In this embodiment, the training samples of the ternary neural network include three-dimensional shapes and corresponding descriptors; during the training of the ternary neural network, the three-dimensional shapes in the training samples use the methods in steps S10 and S20 to obtain the corresponding vertex frequency domain The set of images is input to the ternary neural network.

After the trained ternary neural network is obtained, the generated VSI image data set is sent to the ternary neural network to obtain a 256-dimensional descriptor.

Existing methods can only learn through key point data, but in this embodiment, through VSI, a compact geometric image representation, all vertices can be used as a training set, thereby learning more discriminative descriptors.

In practical applications, the robustness of the descriptor provided by the present invention can be evaluated through experiments. The experimental platform of this embodiment is an Intel i7-7700 4.20GH CPU, 16GB RAM and a computer with a 64-bit Windows 10 operating system. , and use open source software to design a non-rigid shape matching local frequency domain descriptor generation system on this platform. Offline training runs on an NVIDIA GeForce GTX 1080Ti (11GB RAM) GPU.

In practical applications, standard evaluation frameworks can be used, namely CMC (cumulative matchcharacteristic) and PP (Princeton protocol). CMC evaluates the probability of finding a correct match among the k-nearest neighbors. PP measures the quality of the match by plotting the percentage of nearest neighbor matches within r geodesic distance.

In describing the invariance of resolution and triangulation, in order to demonstrate the invariance of the frequency domain feature proposed by the present invention to resolution and triangulation, this embodiment selects four types of shapes, namely thin man, fat woman, Fat men and skinny girls. Each shape has five models of different resolutions and different triangulations. The 16-dimensional LPS frequency domain features are calculated for each model, and then the classical PCA (principal component analysis) method is used to reduce the dimension. The dimensionality reduction results are shown in the PCA results on the left of Figure 6. Five different resolutions and the same category of results are clustered together, showing that the method of the present invention is insensitive to spatial resolution and triangulation. The right side of Figure 6 shows the results of shape matching. In the two-dimensional map of geodesic radius and accuracy, OUR-LPS, OUR-LPS Ori-8000, OUR-LPS Ori-10000, OUR-LPS Ori-12000, OUR-LPS LPS Ori-15000 is the result of matching the LPS feature of the original vertex, the matching result of the original vertex and the LPS feature of the 8000 resolution vertex, the matching result of the original vertex and the LPS feature of the 10000 resolution vertex, the original vertex and the 12000 resolution vertex It can be seen from the LPS feature matching results of the original vertices and the LPS feature matching results of the 15,000-resolution vertices that multiple curves of different resolutions and triangulations overlap, showing the frequency domain feature proposed by the present invention. robustness.

In order to illustrate the discriminative, resolution and triangulation robustness of the learned descriptors, as shown in Fig. 7, the two-dimensional maps of matching times and hit rates, the geodesic radius and the correct corresponding two-dimensional maps respectively show that only in The performance of descriptors learned at the original 6890 resolution tested at low and high resolutions. In the figure, "OURS" represents the matching of the shape between the original shape, "OURS Ori-8K" represents the matching result between the original shape and the high-resolution 8K shape, and so on. This embodiment is applied to two extreme tests, namely the matching between the 10K model and the 100K model and the matching between the 5K CVT model and the 10K model. For the CVT model, this embodiment selects the nearest geodesic distance as the true matching corresponding point. In addition, a comparison with the state-of-the-art geometry method LDGI is carried out. Figure 7 shows that the performance of the original FAUST resolution test is higher than that of the LDGI method, in addition, the performance of the present invention does not degrade much for low-resolution and other high-resolution tests, thus demonstrating the robustness of the learned descriptor. Awesomeness. For the geometry method LDGI, please refer to: Hanyu Wang,Jianwei Guo,Dong-Ming Yan,Weize Quan,and Xiaopeng Zhang.Learning 3d keypoint descriptors for non-rigid shape matching.In European Conference on Computer Vision(ECCV),pages 3-20. Springer, 2018.

Further comparisons are made with multiple state-of-the-art methods, including three deep learning descriptors (OSD, CGF32, and LDGI), and four handcrafted descriptors (SI, SHOT, RoPS, and HKS). Figures 8 and 9 show the matching results between the original 6890 model and the 8K and 12K models, respectively. All trainable methods are trained on the original dataset and applied to high-resolution models. The experimental results show that many methods cannot handle the situation at different resolutions, on the contrary, the method of the present invention exhibits good performance.

In order to illustrate the discriminative and scale and rotation robustness of the learned descriptors, it can be directly tested under different rotation, translation and different scale models. Figures 10 and 11 show that the present invention exhibits performance that exceeds the state-of-the-art methods in terms of Rotation and Scale, respectively.

Shape correspondence and matching are two different tasks. Many methods (such as GCNN, MoNet, etc.) transform the correspondence problem into a learning label problem. The present invention compares through two configurations. The GCNN(OURS) configuration means using LPS features as input , while using the GCNN framework for learning, which also shows that LPS can be used as input features for other frameworks. Furthermore, the OURS-CORR configuration indicates that the present invention transforms the learned triplet loss into a cross-entropy loss to learn the labeling problem in order to compare with the rest of the methods. Figure 12 shows the comparison results of shape correspondence on the FAUST dataset, showing that the performance of both configurations is on par with the current state-of-the-art methods.

As shown in FIG. 13 , a device 100 for generating local frequency domain descriptors for non-rigid shape matching according to the second embodiment of the present invention includes a local point feature generation module 101 and a local frequency domain descriptor acquisition module 102;

The local point feature generation module 101 is configured to obtain the local point feature in the frequency domain of each vertex in the three-dimensional shape surface triangular mesh model based on the Laplace-Beltrumian operator;

The local frequency domain descriptor obtaining module 102 is configured to obtain a vertex frequency domain image corresponding to each vertex of the three-dimensional shape based on the local feature points, and obtain a ternary neural network for non-rigid shape matching. Local frequency domain descriptor.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process and related description of the device described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here.

It should be noted that the device for generating local frequency domain descriptors for non-rigid shape matching provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be allocated by It can be completed by different functional modules, that is, the modules or steps in the embodiments of the present invention are decomposed or combined. For example, the modules in the above-mentioned embodiments can be combined into one module, and can also be further split into multiple sub-modules, so as to complete the above description. All or part of the functionality. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing each module or step, and should not be regarded as an improper limitation of the present invention.

A storage device according to a third embodiment of the present invention stores a plurality of programs, wherein the programs are adapted to be loaded and executed by a processor to implement the above-mentioned method for generating local frequency domain descriptors for non-rigid shape matching.

A processing device according to a fourth embodiment of the present invention includes a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded and executed by the processor In order to realize the above-mentioned local frequency-domain descriptor generation method for non-rigid shape matching.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and relevant description of the storage device and processing device described above can refer to the corresponding process in the foregoing method embodiments, which is not repeated here. Repeat.

Those skilled in the art should be aware that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two, and the programs corresponding to the software modules and method steps Can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or as known in the art in any other form of storage medium. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been described generally in terms of functionality in the foregoing description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods of implementing the described functionality for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The terms "first," "second," etc. are used to distinguish between similar objects, and are not used to describe or indicate a particular order or sequence.

The term "comprising" or any other similar term is intended to encompass a non-exclusive inclusion such that a process, method, article or device/means comprising a list of elements includes not only those elements but also other elements not expressly listed, or Also included are elements inherent to these processes, methods, articles or devices/devices.

So far, the technical solutions of the present invention have been described with reference to the preferred embodiments shown in the accompanying drawings, however, those skilled in the art can easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims (14)

1. a local frequency domain descriptor generation method for non-rigid shape matching, is characterized in that, this method comprises the following steps: Step S10, based on the surface continuous function f of the surface triangular mesh model of the three-dimensional shape, calculate the Laplace-Beltram matrix L of the surface triangular mesh model; Perform eigendecomposition on the matrix L to obtain eigenvectors and eigenvalues; Extend the surface continuous function f to a discrete function

and expand it under the eigenvectors of the matrix L to obtain the frequency domain expansion coefficient σ _j ; According to the eigendecomposition of the matrix L and the frequency domain expansion coefficient σ _j , the discrete Dirichlet energy is calculated

Extend the high-dimensional continuous function F based on the surface continuous function f to high-dimensional discrete functions

Calculate discrete Dirichlet energies

and the energy

Expand in the frequency domain to obtain general frequency domain features; high-dimensional discrete functions

Set to the three-dimensional coordinate information X of the local patch, and solve the discrete energy according to the continuous energy E(X)

Take the discrete energy

The first Q dimension is expanded in the frequency domain, and the local point feature is obtained; wherein Q is the first set value; Step S20, based on the local point features, obtain a vertex frequency domain image corresponding to each vertex of the three-dimensional shape, and obtain a local frequency domain descriptor for non-rigid shape matching through a ternary neural network. 2. The method for generating local frequency-domain descriptors for non-rigid shape matching according to claim 1, characterized in that, "calculate the surface triangular mesh model based on the surface continuous function f of the three-dimensional shape surface triangular mesh model. The Laplace-Beltrami matrix L", the method is: Get the discrete function of the surface continuous function f

Elements L _ij in the Laplacian- Beltrami matrix L of the surface triangular mesh model are calculated by:

Among them, α _ij and β _ij are the two angles opposite to the edge {i, j} in the surface triangular mesh model, α _i is the Voronoi polygon area of the vertex v _i , and k is the number of adjacent vertices. 3. The method for generating local frequency-domain descriptors for non-rigid shape matching according to claim 2, characterized in that, "decomposing the matrix L to obtain eigenvectors and eigenvalues", the method is: Decompose matrix L into two symmetric matrices T and A, TΦ _i =λ _i AΦ _i ,i=0,1,...,N-1; where, A _ii =a _i

The ARPACK method is used to solve the problem, and the eigenvector Φ _i and the eigenvalue λ _i are obtained, and N is the number of vertices of the surface triangular mesh model. 4. The method for generating local frequency-domain descriptors for non-rigid shape matching according to claim 3, characterized in that "extend the surface continuous function f to a discrete function

And expand it under the eigenvector of the matrix L to obtain the frequency domain expansion coefficient σ _j ", the method is:

Among them, Φ _j is the j-th eigenvector. 5. The method for generating local frequency-domain descriptors for non-rigid shape matching according to claim 4, characterized in that "According to the eigendecomposition of the matrix L and the frequency-domain expansion coefficient σ _j , the discrete Dirichlet energy is calculated.

", the method is:

in,

is the discrete energy form corresponding to the Dirichlet energy of the continuous real function f, N is the number of vertices, and λ _j is the jth eigenvalue. 6. The method for generating local frequency-domain descriptors for non-rigid shape matching according to claim 5, characterized in that "expanding a high-dimensional continuous function F based on a surface continuous function f to a high-dimensional discrete function

Calculate discrete Dirichlet energies

and the energy

Expand in the frequency domain to get general frequency domain features", the method is:

Among them, sf is a general frequency domain feature, λ _N-1 is the Nth eigenvalue, and σ _iN-1 is the Nth frequency domain expansion coefficient under the ith dimension. 7. The method for generating local frequency-domain descriptors for non-rigid shape matching according to claim 6, characterized in that "the high-dimensional discrete function is

Set to the three-dimensional coordinate information X of the local patch, and solve the discrete energy according to the continuous energy E(X)

Take the discrete energy

The first Q-dimension unrolled in the frequency domain obtains the local point feature", and the method is:

Among them, LPS is the obtained local point feature. 8. The method for generating local frequency-domain descriptors for non-rigid shape matching according to claim 1, wherein "based on the local point features, a vertex frequency-domain image corresponding to each vertex of the three-dimensional shape is obtained. , and obtain a local frequency domain descriptor for non-rigid shape matching through a ternary neural network", the method is: encoding the local point feature into Q images of the first set size; For each vertex on the three-dimensional shape surface triangular mesh model, generate the corresponding vertex frequency domain image, and obtain the vertex frequency domain image set; Based on the vertex frequency domain image set, a local frequency domain descriptor for non-rigid shape matching is obtained through a preset ternary neural network. 9. The method for generating local frequency-domain descriptors for non-rigid shape matching according to claim 8, characterized in that "encode the local point features into Q images of the first set size", the method of for: Take Q as 16, and the first set size is 8*8; The local point features are encoded into 16 geometric images of size 8*8 using the geometric image method. 10. The method for generating local frequency domain descriptors for non-rigid shape matching according to claim 9, characterized in that "for each vertex on the three-dimensional shape surface triangular mesh model, a corresponding vertex frequency domain image is generated", Its method is: Initialize an empty image of 32*32, fill 16 feature-encoded images of local point features of size 8*8 into the initialized empty image, and place the minimum and maximum eigenvalues in the upper left and lower right corners, respectively, to generate 32 *32 size vertex frequency domain image. 11. The method for generating local frequency domain descriptors for non-rigid shape matching according to any one of claims 1-10, wherein the ternary neural network consists of three identical ConvNet convolutional networks, which The training samples include three-dimensional shapes and corresponding descriptors; during the training of the ternary neural network, the three-dimensional shapes in the training samples are obtained by using the methods in steps S10 and S20 to obtain the corresponding vertex frequency domain image set and input into the ternary neural network . 12. A local frequency-domain descriptor generation device for non-rigid shape matching, characterized in that the device comprises a local point feature generation module and a local frequency-domain descriptor acquisition module; The local point feature generation module is configured to obtain the local point feature in the frequency domain of each vertex in the three-dimensional shape surface triangular mesh model based on the Laplace-Beltrami operator; The local frequency domain descriptor obtaining module is configured to obtain a vertex frequency domain image corresponding to each vertex of the three-dimensional shape based on the local point feature, and obtain a local frequency domain image for non-rigid shape matching through a ternary neural network. frequency domain descriptor; Among them, based on the Laplace-Beltramian operator, for each vertex in the surface triangular mesh model, the local point features of the three-dimensional shape are extracted in the frequency domain, and the method is as follows: Based on the surface continuous function f of the three-dimensional shape surface triangular mesh model, calculate the Laplace-Beltrami matrix L of the surface triangular mesh model; Perform eigendecomposition on the matrix L to obtain eigenvectors and eigenvalues; Extend the surface continuous function f to a discrete function

and expand it under the eigenvectors of the matrix L to obtain the frequency domain expansion coefficient σ _j ; According to the eigendecomposition of the matrix L and the frequency domain expansion coefficient σ _j , the discrete Dirichlet energy is calculated

Extend the high-dimensional continuous function F based on the surface continuous function f to high-dimensional discrete functions

Computing discrete Dirichlet energies

and transfer the energy

Expand in the frequency domain to obtain general frequency domain features; high-dimensional discrete function

Set as the three-dimensional coordinate information X of the local patch, and solve the discrete energy according to the continuous energy E(X)

Take the discrete energy

The first Q dimension is expanded in the frequency domain to obtain local point features; where Q is the first set value. 13. A storage device, wherein a plurality of programs are stored, wherein the programs are adapted to be loaded and executed by a processor to implement the local frequency matching for non-rigid shape matching according to any one of claims 1-11 Domain descriptor generation method. 14. A processing device, comprising a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store a plurality of programs; characterized in that the programs are adapted to be loaded and executed by the processor to The method for generating local frequency domain descriptors for non-rigid shape matching according to any one of claims 1 to 11 is implemented.

Images