CN114581293A - Perspective transformation method and device, electronic equipment and readable storage medium - Google Patents

Abstract

The application provides a perspective transformation method, a perspective transformation device, electronic equipment and a readable storage medium, wherein a face region image in a face image is obtained by intercepting the face image collected by shooting equipment, and the face region image comprises a plurality of face key points. And constructing a virtual perspective camera, and obtaining the pose parameters of the virtual perspective camera according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each face key point in the face region image and the rigid transformation information of the face during perspective transformation. And finally, transforming the 3D coordinate information of each face key point into virtual 3D coordinate information in a virtual perspective camera space based on the pose parameters. In the scheme, the reprojection error and the rigid transformation information between the 3D point and the 2D point are comprehensively considered in perspective transformation, and a good effect can be achieved on avoiding face transformation deformation and improving the projection accuracy of key points.

Description

Perspective transformation method, apparatus, electronic device and readable storage medium

technical field

The present application relates to the technical field of image processing, and in particular, to a perspective transformation method, apparatus, electronic device and readable storage medium.

Background technique

In recent years, face 3D keypoint detection has been widely used in the fields of face tracking, emotion recognition, and face VR special effects. In the field of face 3D key point detection, it usually involves cropping and other processing of the collected original image, resulting in changes in the internal and external parameters of the camera. Therefore, in order to accurately restore the position information of key points in the processed image, a virtual camera is generally constructed and the three-dimensional face information in the original world coordinate system is perspective-transformed into the virtual perspective camera space.

However, in the prior art, when performing perspective transformation, a fitting method such as the least squares method is generally used to perform the fitting transformation, which is prone to problems such as inaccurate calculation and distortion of the human face.

SUMMARY OF THE INVENTION

The objectives of the present application include, for example, to provide a perspective transformation method, apparatus, electronic device, and readable storage medium, which can avoid face transformation and deformation during virtual perspective transformation and improve the accuracy of key point projection.

The embodiments of the present application can be implemented as follows:

In a first aspect, the present application provides a perspective transformation method, the method comprising:

acquiring a face image collected by a photographing device, and intercepting and obtaining a face area image in the face image, where the face area image includes a plurality of face key points;

Build a virtual perspective camera, according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face area image, and the rigid transformation information of the face during perspective transformation, obtain the virtual The pose parameters of the perspective camera; based on the pose parameters, the 3D coordinate information of each of the face key points is transformed into virtual 3D coordinate information in the virtual perspective camera space.

In an optional implementation manner, the step of obtaining a face region image in the face image by intercepting includes:

Determine a plurality of face key points that constitute the face contour in the face image;

obtaining the minimum bounding frame of the multiple face key points;

The face image is intercepted based on the minimum bounding frame to obtain a face region image.

In an optional implementation manner, the step of intercepting the face image based on the minimum bounding frame to obtain a face region image includes:

determining the center position of the minimum bounding box;

The interception length is calculated based on the position information of multiple corner points of the minimum bounding frame;

An interception range is determined according to the center position and the interception length, and the face image is intercepted based on the interception range to obtain a face area image.

In an optional embodiment, the method further includes the step of obtaining the 2D coordinate information of each of the face key points in the face region image, and the step includes:

Obtain the 3D coordinate information of each of the face key points in the world coordinate system;

According to the device parameters of the photographing device, project the 3D coordinate information of each of the face key points as 2D coordinate information under a first image coordinate system, and the first image coordinate system is constructed with the face image;

The 2D coordinate information is updated based on the interception information when the face area image is intercepted to obtain 2D coordinate information in the second image coordinate system constructed with the face area image.

In an optional implementation manner, the clipping information includes clipping length and clipping center position;

The step of updating the 2D coordinate information based on the interception information when intercepting the face region image, includes:

Subtract the abscissa of the interception center position from the abscissa in the 2D coordinate information, and add half of the interception length;

The ordinate of the clipping center position is subtracted from the ordinate in the 2D coordinate information, and the half of the clipping length is added.

In an optional implementation manner, the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face region image, and the rigid transformation information of the face during perspective transformation , the steps of obtaining the pose parameters of the virtual perspective camera include:

Constructing the first optimization term consisting of the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face region image, and constructing the rigid transformation information of the face during perspective transformation. The second optimization term of ;

assigning a weight to the second optimization term based on the set weight coefficient;

The first optimization term and the weighted second optimization term are added to construct an optimized model, and the optimized model is minimized to obtain the pose parameters of the virtual perspective camera.

In an optional implementation manner, the step of constructing a first optimization term consisting of the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face region image includes the following steps: :

constructing a projection item obtained from the 3D coordinate information of each face key point in the face region image and the pose parameter;

The first optimization term is obtained by subtracting the projection item from the 2D coordinate information of each face key point in the face region image.

In an optional implementation manner, the pose parameter includes a rotation matrix, and the step of constructing a second optimization term composed of rigid transformation information of the face during perspective transformation includes:

Multiplying the rotation matrix by the transpose matrix of the rotation matrix to obtain rigid feature information;

The rigid feature information is subtracted from the constructed identity matrix to obtain the second optimization term.

In an optional embodiment, the method further includes:

Constructing a face grid based on the plurality of face key points;

The grid information of the face grid is obtained according to the virtual 3D coordinate information of each of the face key points.

In a second aspect, the present application provides a perspective transformation device, the device comprising:

an acquisition module, which acquires a face image collected by a photographing device, and intercepts and obtains a face region image in the face image, where the face image includes a plurality of face key points;

A determination module, for constructing a virtual perspective camera, according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face area image, and the rigid transformation information of the face during perspective transformation , obtain the pose parameters of the virtual perspective camera;

A transformation module, configured to transform the 3D coordinate information of each of the face key points into virtual 3D coordinate information in the virtual perspective camera space based on the pose parameters.

In a third aspect, the present application provides an electronic device, comprising one or more storage media and one or more processors in communication with the storage media, wherein the one or more storage media stores machine-executable instructions executable by the processor, When the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the preceding embodiments.

In a fourth aspect, the present application provides a computer-readable storage medium, where the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, implements the method described in any one of the foregoing embodiments step.

The beneficial effects of the embodiments of the present application include, for example:

The present application provides a perspective transformation method, device, electronic device, and readable storage medium. For a face image collected by a photographing device, a face region image in the face image is obtained by intercepting, and the face region image contains a plurality of human faces. key point. A virtual perspective camera is constructed, and the pose parameters of the virtual perspective camera are obtained according to the reprojection error between the 3D coordinate information and 2D coordinate information of each face key point in the face area image, and the rigid transformation information of the face during perspective transformation. . Finally, based on the pose parameters, the 3D coordinate information of each face key point is transformed into the virtual 3D coordinate information in the virtual perspective camera space. In this scheme, the reprojection error and rigid transformation information between 3D points and 2D points are comprehensively considered in perspective transformation, which can achieve good results in avoiding face transformation and deformation and improving the accuracy of key point projection.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following drawings will briefly introduce the drawings that need to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore do not It should be regarded as a limitation of the scope, and for those of ordinary skill in the art, other related drawings can also be obtained according to these drawings without any creative effort.

1 is a schematic diagram of an application scenario of a perspective transformation method provided by an embodiment of the present application;

2 is a flowchart of a perspective transformation method provided by an embodiment of the present application;

3 is a schematic diagram of a face region image interception provided by an embodiment of the present application;

4 is a schematic diagram of a face grid provided by an embodiment of the present application;

5 is a schematic diagram of another perspective of a face grid provided by an embodiment of the present application;

6 is a flowchart of a method for acquiring 2D coordinate information provided by an embodiment of the present application;

FIG. 7 is a flowchart of the sub-steps included in step S101 in FIG. 2;

FIG. 8 is a schematic diagram of a minimum bounding frame determined in an embodiment of the application;

FIG. 9 is a flowchart of the sub-steps included in step S1015 in FIG. 7;

10 is a flowchart of the sub-steps included in step S103 in FIG. 2;

11 is a flowchart of a method for acquiring grid information provided by an embodiment of the present application;

12 is a structural block diagram of an electronic device provided by an embodiment of the present application;

FIG. 13 is a block diagram of functional modules of a perspective transformation apparatus provided by an embodiment of the present application.

Icons: 110-storage medium; 120-processor; 130-perspective transformation device; 131-acquisition module; 132-determination module; 133-transformation module; 140-communication interface.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of the present application, but not all of the embodiments. The components of the embodiments of the present application generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

Thus, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the application as claimed, but is merely representative of selected embodiments of the application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

In the description of the present application, it should be noted that if the terms "first", "second" etc. appear, they are only used to distinguish the description, and should not be construed as indicating or implying relative importance.

It should be noted that the features in the embodiments of the present application may be combined with each other under the condition of no conflict.

Please refer to FIG. 1 , which is a schematic diagram of an application scenario of the perspective transformation method provided by an embodiment of the present application. The application scenario includes a server and a photographing device that is communicatively connected to the server. There may be one or more photographing devices, and the photographing device may include a device for collecting two-dimensional images, such as a camera, and also a device for collecting depth images, such as a depth camera. In this embodiment, the photographing device may send the collected image information or video information to the server, and the server may analyze and process the received image information or video information.

Taking the live broadcast application scenario as an example, the shooting device may be a shooting device set on the live broadcast provider or the live broadcast receiver. The face image of the host or the audience watching the live broadcast of the host can be collected, and then the face image is sent to the server for analysis and processing.

It should be noted that the perspective transformation method provided in this embodiment can also be applied to other processes that need to analyze and process the three-dimensional information of the key points of the face, and use the three-dimensional information to perform processing such as facial emotion recognition and facial special effects. in application scenarios.

With reference to FIG. 2 , an embodiment of the present application provides a perspective transformation method applicable to an electronic device, where the electronic device may be the above-mentioned server. The method steps defined by the flow related to the perspective transformation method can be implemented by the electronic device. The specific flow shown in FIG. 2 will be described in detail below.

S101: Acquire a face image collected by a photographing device, and intercept and obtain a face region image in the face image, where the face region image includes a plurality of face key points.

S103, constructing a virtual perspective camera, according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face area image, and the rigid transformation information of the face during perspective transformation, obtain the Describe the pose parameters of the virtual perspective camera.

S105 , transform the 3D coordinate information of each of the face key points into virtual 3D coordinate information in a virtual perspective camera space based on the pose parameters.

Taking the live broadcast application scenario as an example, when the host broadcasts the live broadcast, the camera device set on the host side can collect a face image including the host's face. The face image collected by the shooting device can be sent to the server together with the live video stream. Wherein, the face images collected by the photographing device may include multiple face images of the host from different angles and different expression states.

In addition to the main face area, the captured face image may also include the image of the environment area at the time of shooting, that is, the image of the background area. In order to facilitate the analysis of the face information in the image, the face region image in the face image is often cut out, as shown in FIG. 3 .

The face image is cut out to cut out the face area image contained therein, and the coordinate information of each pixel in the face area image will be changed under the coordinate system of the original face image. That is, the image coordinate system of the face region image is changed with respect to the image coordinate system of the face image.

Moreover, due to the change of the image coordinate system, the viewpoint of each pixel in the face area image also changes, which means that each pixel in the face area image needs to be transformed into the camera space under the changed viewpoint. Therefore, in this embodiment, a virtual perspective camera needs to be constructed, and the viewpoint of the virtual perspective camera is established based on the changed viewpoint, that is, the virtual perspective camera is a perspective camera corresponding to the three-dimensional camera space after the viewpoint is changed. It is necessary to transform the coordinate information of the pixel points in the face area image into the virtual perspective camera space. That is, it is necessary to transform the coordinate information of the pixel points originally in the world coordinates into the constructed virtual perspective camera space.

In this embodiment, the pixel points that mainly need to perform perspective transformation processing are the key points of each face in the image of the face region. The face key points may be key points that embody face contour information, facial feature information, etc., such as key points constituting the face contour, key points constituting the shape of the mouth, key points constituting the shape of the eyebrows, and the like.

In order to accurately construct a virtual perspective camera, the pose parameters of the virtual perspective camera need to be determined. The pose parameters of the virtual perspective camera include rotation matrix and translation vector. When the pose parameters of the virtual perspective camera are determined, the virtual perspective camera can be fixed. When the perspective transformation of the coordinates of the face key points is performed, the perspective transformation of the face key points may be performed based on the determined pose parameters. The pose parameters of the virtual perspective camera will affect the transformation effect of the face key point perspective transformation. Therefore, it is necessary to obtain pose parameters that can make the face key points have better effects after perspective transformation.

Coordinate information of each face key point in the face area image, including 3D coordinate information and 2D coordinate information. The 3D coordinate information is the coordinate information in the world coordinate system, which can be understood as the coordinate information in the three-dimensional camera space before the viewpoint changes. The 2D coordinate information is the coordinate information in the image coordinate system constructed with the face area image. After the perspective transformation, the 3D coordinate information and the 2D coordinate information of each face pixel should be accurately projected, that is, the reprojection error should be as small as possible. The reprojection error represents the error between the 2D coordinates of the pixel points and the position points obtained by projecting the 3D coordinates under the currently estimated pose parameters. Under a virtual perspective camera with a certain pose parameter, if the reprojection error between the 3D coordinate information and the 2D coordinate information of the face key points is smaller, it indicates that the alignment effect between the 3D coordinate information and the 2D coordinate information is better. . The effect of the subsequent perspective transformation based on the pose parameter is better.

In addition, in this embodiment, it is also considered that the human face is easily distorted after perspective transformation. Therefore, in this embodiment, the pose parameters of the virtual perspective camera are determined by combining the reprojection error between the 3D coordinate information and the 2D coordinate information of the key points of the face, and the rigid transformation information of the face during perspective transformation. Optimization. Therefore, it is guaranteed to have a better effect on the reprojection error and the rigid transformation of the face.

Among them, the rigid transformation information of the face reflects the information such as position, orientation, and shape transformation between the key points of the face before and after transformation. Among them, the rigid transformation information of the face can be controlled by the data features of the pose parameters themselves. If the rigid features presented by the data features of the pose parameters themselves are stronger, it indicates that the transformed face has stronger rigid features. And the embodiment of the rigid features of the pose parameters themselves, for example, if the rotation matrix in the pose parameters is closer to the orthogonal matrix, it indicates that the rigid features of the pose parameters are stronger, and the rigid features of the rigid transformation information of the face are correspondingly stronger.

The optimization of the pose parameters is carried out under the two indicators of the reprojection error and the rigid transformation information of the face. When the pose parameters of the virtual perspective camera are obtained, the 3D coordinate information of each face key point is calculated based on the pose parameters. Transform to virtual 3D coordinate information in virtual perspective camera space. Based on the obtained virtual 3D coordinate information of key points of each face, the 3D information of the face can be constructed, so that application processing such as face tracking and emotion recognition can be realized.

The perspective transformation method provided in this embodiment comprehensively considers the reprojection error between the 3D coordinate information and the 2D coordinate information of the face key points and the rigidity of the face reflected in the face key point transformation when performing the perspective transformation. The transformation information can avoid the distortion of the face after transformation, and can improve the accuracy of the face key point projection.

In this embodiment, for the face area image collected and intercepted by the photographing device, the corresponding numbers of each face key point can be set in advance according to unified rules. For example, 1220 face key points in the face part can be used to synthesize reflect face information. Then, among the multiple face key points, each face key point may correspond to one of the numbers 1-1220 respectively. The 1220 face key points can constitute a face mesh, and the face mesh can be constituted by a plurality of triangle slices. Each triangle slice is composed of three adjacent face key points as vertices, combined with the connection lines between them, so that 2304 triangle slices can be formed, as shown in Figures 4 and 5.

The 3D coordinate information of each face key point in the world coordinate system can be recorded as (x, y, z), and each triangle slice can be recorded by the numbers corresponding to its three vertices, such as (i, j, k) .

The 3D coordinate information of multiple face key points contained in the face region image can be represented by an N×3 matrix P _w , where N represents the number of face key points.

It can be seen from the above that the 2D coordinate information under the face area image needs to be used during perspective transformation. Therefore, the method provided by this embodiment also includes the step of obtaining the 2D coordinate information of each face key point in the face area image. , see Figure 6, this step can be achieved by:

S1021, obtain 3D coordinate information of each of the face key points in the world coordinate system.

S1023: Project the 3D coordinate information of each key point of the face into 2D coordinate information in a first image coordinate system according to the equipment parameters of the photographing equipment, and the first image coordinate system is based on the face image. Construct.

S1025 , based on the interception information when intercepting the face area image, update the 2D coordinate information to obtain 2D coordinate information in the second image coordinate system constructed with the face area image.

In this embodiment, when the device parameters of the photographing device are determined, the 3D coordinate information of each face key point in the world coordinate system in the face image obtained by the photographing device can be determined.

For each face key point in the face image, there is a projection relationship between its 3D coordinate information and 2D coordinate information. Based on the device parameters of the shooting device, the 2D coordinate information under the projection of the 3D coordinate information of each face key point can be obtained. . It should be noted that what is obtained here is the 2D coordinate information under the coordinates of the first image constructed by the face image. The obtained 2D coordinate information of multiple face key points can be represented by an N×2 matrix P _uv .

When the face area image is cut out from the face image, the image coordinate system is changed from the first image coordinate system under the face image to the second image coordinate system under the face area image. Correspondingly, the 2D coordinate information of each face key point will also be changed, and needs to be changed to the coordinate information in the second image coordinate system.

In this embodiment, the 2D coordinate information of each face key point can be updated and revised based on the interception information when the face area image is intercepted from the face image, so as to be converted into a second image coordinate system based on the face area image 2D coordinate information. The clipping information is mainly the position information of the face region image relative to the face image, such as the clipping length of the face region image, the position of the center point of the face region image in the first image coordinate system, and the like.

In this embodiment, when the face area image is cut out from the face image, and the subsequent perspective transformation processing is performed based on the face area image, the 2D coordinate information under the face area image is revised, thereby ensuring the 2D coordinate information. The accuracy of the coordinate information.

Referring to FIG. 7 , in this embodiment, when the face region image is intercepted from the face image, the following methods can be used:

S1011: Determine a plurality of face key points that constitute a face contour in the face image.

S1013: Obtain the minimum bounding frame of the multiple face key points.

S1015: Intercept the face image based on the minimum bounding frame to obtain a face region image.

In this embodiment, the determined multiple face key points constituting the face contour are multiple key points located on the periphery of the face in the image. For example, as shown in FIG. 8 , 68 persons numbered 1 to 68 can be determined. face key points. A minimum bounding frame may be determined based on the multiple face key points, and the minimum bounding frame is the smallest bounding frame that can enclose the multiple face key points. The shape of the minimum bounding frame can be a rectangle, a square, etc., which is mainly determined by the face contour formed by the key points of the face.

Based on the determined minimum bounding frame, the face area framed by the minimum bounding frame may be cut out from the face image to obtain a face area image.

Referring to FIG. 9, in this embodiment, when the face image is intercepted based on the minimum bounding frame, the following methods can be used:

S10151: Determine the center position of the minimum bounding frame.

S10153: Calculate the interception length based on the position information of multiple corner points of the minimum bounding frame.

S10155: Determine an interception range according to the center position and the interception length, and intercept the face image based on the interception range to obtain a face region image.

In this embodiment, the minimum bounding box may be a rectangular box, and the minimum bounding box may be represented by a coordinate vector (u _tl , v _tl , u _br , v _br ), where (u _tl , v _tl ) is the minimum bounding box The coordinate value of the corner point of the upper left corner of the frame in the image coordinate system of the face image, (u _br , v _br ) is the coordinate value of the corner point of the lower right corner of the minimum bounding box in the image coordinate system of the face image.

Based on the position information of the corner points of the minimum bounding box, such as the above-mentioned coordinate values of the upper left corner and the lower right corner, the center position of the minimum bounding box can be obtained, which can be recorded as (u _c , _vc ). The abscissa of the center position may be half the sum of the abscissa of the upper left corner and the abscissa of the lower right corner: _uc =0.5(u _tl +u _br ). The ordinate of the center position may be half the sum of the ordinate of the upper left corner and the ordinate of the lower right corner: v _c =0.5(v _tl +v _br ).

In addition, the interception length L can be calculated based on the position information of the corner points of the minimum bounding frame, and the calculation formula can be as follows:

和

可以将截取范围记为一个向量

基于确定出的截取范围可对人脸图像进行截取，获得人脸区域图像。After obtaining the center position of the minimum bounding frame and the clipping length, the clipping range can be determined, and the clipping range is also a rectangular frame. The coordinate values of the upper left corner and the lower right corner of the intercepted range are respectively

and

The intercept range can be recorded as a vector

Based on the determined clipping range, the face image can be clipped to obtain a face region image.

In this embodiment, the above-mentioned minimum bounding frame constituting the outline of the human face is determined, and a certain deformation is performed on the basis of the minimum bounding frame to determine the interception range. On the basis of ensuring that the complete face area in the face image is intercepted, the interception of too many background areas can be avoided, thereby causing interference to subsequent face information processing.

In the case of intercepting the face area image in the above manner, when revising and updating the 2D coordinate information of the face key points in the intercepted face area image, the revision can be performed based on the interception center position and the interception length of the minimum bounding frame .

In a possible implementation manner, the abscissa in the 2D coordinate information of the face key points may be subtracted from the abscissa of the interception center position, and half of the interception length may be added.

In addition, the ordinate in the 2D coordinate information of the face key point may be subtracted from the ordinate of the clipping center position, and half of the clipping length may be added.

In this embodiment, in order to facilitate the processing of the face region image, the resolution of the cutout face region image may be unified to a set resolution, such as PxP, where P may be 120 or 160 and so on.

Correspondingly, after revising and updating the 2D coordinate information of the face key points in the face region image, the abscissa and the ordinate in the 2D coordinate information may be multiplied by P/L.

After the above processing, the 3D coordinate information and the 2D coordinate information of each key point of the face can be obtained under the image of the face region after being intercepted. In order to change the 3D coordinate information into the virtual 3D coordinate information under the virtual perspective camera, in this embodiment, the reprojection error between the 3D coordinate information and the 2D coordinate information, and the rigid transformation information of the face during perspective transformation are used as indicators Transform.

Referring to FIG. 10 , in this embodiment, the above-mentioned determination of the pose parameters of the virtual perspective camera can be implemented in the following manner:

S1031, constructing a first optimization term composed of the 3D coordinate information of each of the face key points in the face region image and the reprojection error of the 2D coordinate information, and constructing a first optimization term composed of the rigid transformation information of the face during perspective transformation the second optimization term.

S1033, assign a weight value to the second optimization item based on the set weight coefficient.

S1035, adding the first optimization item and the second optimization item after the weighted value is added to construct the optimization model, and performing a minimization process on the optimization model to obtain the pose parameters of the virtual perspective camera .

In this embodiment, the reprojection error between the 3D coordinate information and the 2D coordinate information, and the rigid transformation information of the face during perspective transformation are considered when optimizing the pose parameters of the virtual perspective camera. If the reprojection error between the 3D coordinate information and the 2D coordinate information is smaller, it indicates that the projection of the 2D key points after transformation is more accurate. The more the relationship can be kept the same as before the transformation, that is, the face will not be deformed and distorted.

In order to control the inclination in rigid transformation when the above two are combined to perform perspective transformation processing, a weighting coefficient may be set to give the second optimal weighting value composed of rigid transformation information. Therefore, the degree of attention on rigid features can be adjusted based on the adjustment of the weight coefficients while ensuring the reprojection error.

In this embodiment, when constructing the first optimization term based on the reprojection error, it can be implemented in the following manner:

Build a projection item obtained from the 3D coordinate information of each face key point in the face area image and the pose parameter, and subtract the 2D coordinate information of each face key point in the face area image by subtracting the The projection term obtains the first optimization term.

The pose parameters of the virtual perspective camera include a rotation matrix and a translation vector of the virtual perspective camera, the rotation matrix may be a 3×3 matrix, and the translation vector may be a vector with a dimension of 3.

When constructing the projection item obtained from the 3D coordinate information of the face key point and the pose parameters, it is equivalent to converting the 3D coordinate information of the face key point into the virtual perspective camera space, and then projecting it into 2D coordinate information. The error between reprojections can be obtained by subtracting the 2D coordinate information of the projection from the 2D coordinate information of the face key points in the face region image.

In addition, when constructing the second optimization term composed of the rigid transformation information of the face during perspective transformation, it can be achieved in the following ways:

Multiplying the rotation matrix and the transposed matrix of the rotation matrix to obtain rigid feature information, and subtracting the multiplied rigid feature information from the constructed identity matrix to obtain the second optimization term.

The rotation matrix is a 3×3 matrix, and if the rotation matrix is closer to an orthogonal matrix, the transformation performed based on the rotation matrix exhibits more rigid characteristics. The orthogonality of the rotation matrix can be represented by the product between the rotation matrix and the transpose of the rotation matrix. Therefore, rigid feature information can be obtained based on the product between the rotation matrix and the transpose of the rotation matrix.

When the rotation matrix is an orthogonal matrix, the rotation matrix is multiplied by the transpose of the rotation matrix to obtain an identity matrix. Therefore, the constructed identity matrix can be reused to subtract the multiplied result. If the subtraction result is closer to 0, it indicates that the rotation matrix is closer to the orthogonal matrix, and the rigidity of the transformation based on the rotation matrix is better.

In this embodiment, in a possible implementation manner, the constructed optimization item may be as follows:

为第一优化项，

为第二优化项。P_uv表示人脸区域图像中的人脸关键点的2D坐标信息，P_w表示3D坐标信息，R表示旋转矩阵，t表示平移向量，Z表示各个人脸关键点在虚拟透视相机空间中z轴的坐标信息，为一个1220维的向量。K表示虚拟透视相机的内参矩阵。|| ||₂表示取L2泛函数，0:2表示取矩阵的前2列元素，w表示权重系数，例如可取值为0.2。I表示单位矩阵，维度为3×3。in,

is the first optimization term,

is the second optimization term. P _uv represents the 2D coordinate information of the face key points in the face area image, P _w represents the 3D coordinate information, R represents the rotation matrix, t represents the translation vector, and Z represents the z-axis of each face key point in the virtual perspective camera space The coordinate information is a 1220-dimensional vector. K represents the intrinsic parameter matrix of the virtual perspective camera. || || ₂ means to take the L2 functional function, 0:2 means to take the first 2 columns of the matrix, w means the weight coefficient, for example, the value can be 0.2. I represents the identity matrix with dimensions 3×3.

When performing the minimization process based on the optimization model to determine the pose parameters of the virtual perspective transformation camera, the gradient descent method, the BFGS method, or other optimization methods may be used for the optimization process, which is not limited in this embodiment.

On the basis of obtaining the pose parameters of the virtual perspective transformation camera, the 3D coordinate information of the face key points can be transformed into virtual 3D coordinate information in the virtual perspective camera space based on the pose parameters.

In this embodiment, the obtained rotation matrix can be multiplied by the 3D coordinate information, and then the translation vector can be added, so as to realize the transformation to the virtual 3D coordinate information. In this embodiment, the transformation can be realized by the following calculation formula:

P _cam =R·P _w +t

Among them, P _cam represents the transformed virtual 3D coordinate information, and similarly, it can be a 1220×3-dimensional matrix, and each row in the matrix represents the 3-dimensional coordinate information of a face key point.

When performing face tracking and face special effects processing based on the 3-dimensional information of the face, it usually needs to be implemented based on the relevant information of the face mesh. Therefore, in the case of obtaining the virtual 3D coordinate information of the face key points in the virtual perspective transformation camera space, the following steps may also be performed, please refer to FIG. 11 in conjunction:

S107, constructing a face grid based on the plurality of face key points.

S109, obtaining grid information of the face grid according to the virtual 3D coordinate information of each of the face key points.

In this embodiment, the constructed face mesh is the above-mentioned face mesh composed of multiple face key points and triangular slices formed by each face key point as vertices, as shown in Figures 4 and 5 Show.

Based on the virtual 3D coordinate information of each face key point, the mesh information of the face mesh can be obtained. Therefore, based on the grid information of the face grid, face tracking, face special effect processing, etc. can be realized.

The perspective transformation method provided in this embodiment comprehensively considers the reprojection error between the 3D coordinate information and the 2D coordinate information, and the rigid transformation information of the face during transformation. To achieve the effect of reducing the reprojection error and improving the rigid transformation characteristics of the face. It can avoid face transformation and deformation, and can improve the accuracy of key point projection.

Please refer to FIG. 12 , which is a schematic diagram of an exemplary component of an electronic device provided in an embodiment of the present application, and the electronic device may be the server shown in FIG. 1 . The electronic device may include a storage medium 110 , a processor 120 , a perspective transforming device 130 and a communication interface 140 . In this embodiment, the storage medium 110 and the processor 120 are both located in the electronic device and are provided separately. However, it should be understood that the storage medium 110 may also be independent of the electronic device, and may be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, may be a cache and/or a general purpose register.

The perspective transformation device 130 can be understood as the above electronic device or the processor 120 of the electronic device, and can also be understood as a software function module independent of the above electronic device or processor 120 that implements the above perspective transformation method under the control of the electronic device.

As shown in FIG. 13 , the above perspective transformation apparatus 130 may include an acquisition module 131 , a determination module 132 and a transformation module 133 . The functions of each functional module of the perspective transformation device 130 will be described in detail below.

The acquiring module 131 acquires a face image collected by the photographing device, and intercepts and obtains a face region image in the face image, where the face image includes a plurality of face key points.

It can be understood that the obtaining module 131 may be configured to execute the above-mentioned step S101, and for the detailed implementation of the obtaining module 131, reference may be made to the above-mentioned content related to the step S101.

The determination module 132 is used to construct a virtual perspective camera, according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face area image, and the rigid transformation of the face during perspective transformation information to obtain the pose parameters of the virtual perspective camera.

It can be understood that the determining module 132 may be configured to execute the above-mentioned step S103, and for the detailed implementation of the determining module 132, reference may be made to the above-mentioned content related to the step S103.

The transformation module 133 is configured to transform the 3D coordinate information of each of the face key points into virtual 3D coordinate information in the virtual perspective camera space based on the pose parameter.

It can be understood that the transformation module 133 can be used to execute the above-mentioned step S105, and for the detailed implementation of the transformation module 133, please refer to the above-mentioned content related to the step S105.

In a possible implementation manner, the above acquisition module 131 may be used for:

Determine a plurality of face key points that constitute the face contour in the face image;

obtaining the minimum bounding frame of the multiple face key points;

The face image is intercepted based on the minimum bounding frame to obtain a face region image.

In a possible implementation manner, the above acquisition module 131 may be used for:

determining the center position of the minimum bounding box;

The interception length is calculated based on the position information of multiple corner points of the minimum bounding frame;

An interception range is determined according to the center position and the interception length, and the face image is intercepted based on the interception range to obtain a face area image.

In a possible implementation manner, the perspective transformation device 130 obtains a module for obtaining the 2D coordinate information of each of the key points of the face in the face region image, and the obtaining module can be used for:

Obtain the 3D coordinate information of each of the face key points in the world coordinate system;

According to the device parameters of the photographing device, project the 3D coordinate information of each of the face key points as 2D coordinate information under a first image coordinate system, and the first image coordinate system is constructed with the face image;

The 2D coordinate information is updated based on the interception information when the face area image is intercepted to obtain 2D coordinate information in the second image coordinate system constructed with the face area image.

In a possible implementation manner, the clipping information includes clipping length and clipping center position, and the above obtaining module can be used for:

Subtract the abscissa of the interception center position from the abscissa in the 2D coordinate information, and add half of the interception length;

The ordinate of the clipping center position is subtracted from the ordinate in the 2D coordinate information, and the half of the clipping length is added.

In a possible implementation manner, the above-mentioned transformation module 133 may be used for:

Constructing the first optimization term consisting of the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face region image, and constructing the rigid transformation information of the face during perspective transformation. The second optimization term of ;

assigning a weight to the second optimization term based on the set weight coefficient;

The first optimization term and the weighted second optimization term are added to construct an optimized model, and the optimized model is minimized to obtain the pose parameters of the virtual perspective camera.

In a possible implementation manner, the above-mentioned transformation module 133 may be used for:

constructing a projection item obtained from the 3D coordinate information of each face key point in the face region image and the pose parameter;

The first optimization term is obtained by subtracting the projection item from the 2D coordinate information of each face key point in the face region image.

In a possible implementation manner, the above-mentioned transformation module 133 may be used for:

Multiplying the rotation matrix by the transpose matrix of the rotation matrix to obtain rigid feature information;

The rigid feature information is subtracted from the constructed identity matrix to obtain the second optimization term.

In a possible implementation, the perspective transformation device further includes a building module, which can be used for:

Constructing a face grid based on the plurality of face key points;

The grid information of the face grid is obtained according to the virtual 3D coordinate information of each of the face key points.

For the description of the processing flow of each module in the apparatus and the interaction flow between the modules, reference may be made to the relevant descriptions in the foregoing method embodiments, which will not be described in detail here.

Further, the embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the perspective transformation method provided by the foregoing embodiments is implemented.

Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk, a hard disk, etc., when the computer program on the computer-readable storage medium is executed, the above-mentioned perspective transformation method can be executed. For the processes involved when the computer-readable storage medium and its executable instructions are executed, reference may be made to the relevant descriptions in the foregoing method embodiments, which will not be described in detail here.

To sum up, the perspective transformation method, device, electronic device, and readable storage medium provided by the embodiments of the present application, for the face image collected by the photographing device, intercept the face region image in the face image, and the face region image is obtained. The image contains multiple face keypoints. A virtual perspective camera is constructed, and the pose parameters of the virtual perspective camera are obtained according to the reprojection error between the 3D coordinate information and 2D coordinate information of each face key point in the face area image, and the rigid transformation information of the face during perspective transformation. . Finally, based on the pose parameters, the 3D coordinate information of each face key point is transformed into the virtual 3D coordinate information in the virtual perspective camera space. In this scheme, the reprojection error and rigid transformation information between 3D points and 2D points are comprehensively considered in perspective transformation, which can achieve good results in avoiding face transformation and deformation and improving the accuracy of key point projection.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may also be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architectures, functions and possible implementations of apparatuses, methods and computer program products according to various embodiments of the present application. operate. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

In addition, each functional module in each embodiment of the present application may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

If the functions are implemented in the form of software function modules and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims (12)

1. A perspective transformation method, wherein the method comprises: acquiring a face image collected by a photographing device, and intercepting and obtaining a face area image in the face image, where the face area image includes a plurality of face key points; Build a virtual perspective camera, according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face area image, and the rigid transformation information of the face during perspective transformation, obtain the virtual The pose parameters of the perspective camera; Based on the pose parameters, the 3D coordinate information of each of the face key points is transformed into virtual 3D coordinate information in the virtual perspective camera space. 2. perspective transformation method according to claim 1, is characterized in that, described intercepting the step of obtaining the face area image in described face image, comprising: Determine a plurality of face key points that constitute the face contour in the face image; obtaining the minimum bounding frame of the multiple face key points; The face image is intercepted based on the minimum bounding frame to obtain a face region image. 3. The perspective transformation method according to claim 2, wherein the step of intercepting the face image based on the minimum bounding frame to obtain a face region image comprises: determining the center position of the minimum bounding box; The interception length is calculated based on the position information of multiple corner points of the minimum bounding frame; An interception range is determined according to the center position and the interception length, and the face image is intercepted based on the interception range to obtain a face area image. 4. The perspective transformation method according to claim 2, wherein the method further comprises the step of obtaining the 2D coordinate information of each of the key points of the face in the face region image, the step comprising: Obtain the 3D coordinate information of each of the face key points in the world coordinate system; According to the device parameters of the photographing device, project the 3D coordinate information of each of the face key points as 2D coordinate information under a first image coordinate system, and the first image coordinate system is constructed with the face image; The 2D coordinate information is updated based on the interception information when the face area image is intercepted to obtain 2D coordinate information in the second image coordinate system constructed with the face area image. 5. The perspective transformation method according to claim 4, wherein the clipping information comprises clipping length and clipping center position; The step of updating the 2D coordinate information based on the interception information when intercepting the face region image, includes: Subtract the abscissa of the interception center position from the abscissa in the 2D coordinate information, and add half of the interception length; The ordinate of the clipping center position is subtracted from the ordinate in the 2D coordinate information, and the half of the clipping length is added. 6. The perspective transformation method according to claim 1, wherein the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face region image, and The rigid transformation information of the face during perspective transformation, and the steps of obtaining the pose parameters of the virtual perspective camera, including: Constructing the first optimization term consisting of the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face region image, and constructing the rigid transformation information of the face during perspective transformation. The second optimization term of ; assigning a weight value to the second optimization term based on the set weight coefficient; The first optimization term and the weighted second optimization term are added to construct an optimized model, and the optimized model is minimized to obtain the pose parameters of the virtual perspective camera. 7. The perspective transformation method according to claim 6, wherein the construction is composed of the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face area image The steps of the first optimization term include: constructing a projection item obtained from the 3D coordinate information of each face key point in the face region image and the pose parameter; The first optimization term is obtained by subtracting the projection item from the 2D coordinate information of each face key point in the face region image. 8. The perspective transformation method according to claim 6, wherein the pose parameter comprises a rotation matrix, and the step of constructing the second optimization item formed by the rigid transformation information of the face during perspective transformation comprises: Multiplying the rotation matrix by the transpose matrix of the rotation matrix to obtain rigid feature information; The rigid feature information is subtracted from the constructed identity matrix to obtain the second optimization term. 9. The perspective transformation method according to any one of claims 1-8, wherein the method further comprises: Constructing a face grid based on the plurality of face key points; The grid information of the face grid is obtained according to the virtual 3D coordinate information of each of the face key points. 10. A perspective transformation device, characterized in that the device comprises: an acquisition module, which acquires a face image collected by a photographing device, and intercepts and obtains a face region image in the face image, where the face image includes a plurality of face key points; A determination module for constructing a virtual perspective camera, based on the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the face key points in the face area image, and the rigid transformation information of the face during perspective transformation , obtain the pose parameters of the virtual perspective camera; A transformation module, configured to transform the 3D coordinate information of each of the face key points into virtual 3D coordinate information in the virtual perspective camera space based on the pose parameters. 11. An electronic device, characterized by comprising one or more storage media and one or more processors in communication with the storage media, wherein the one or more storage media stores machine-executable instructions executable by the processor, when When the electronic device is running, the processor executes the machine-executable instructions to perform the method steps of any one of claims 1-9. 12. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, implement the method described in any one of claims 1-9. method steps.

Images