CN115147889A - Method, device, device and storage medium for processing face pose in video - Google Patents

Abstract

The application provides a method, a device and equipment for processing a face gesture in a video and a computer readable storage medium; the method comprises the following steps: detecting human face characteristic points of a plurality of video frames containing a target object in a target video to obtain a first characteristic point set comprising two-dimensional coordinates of the human face characteristic points; performing three-dimensional reconstruction on the target object based on the first feature point set to obtain the human face posture of the target object and the three-dimensional coordinates of the human face feature points of the target object; projecting the three-dimensional coordinates of the human face characteristic points to a camera imaging surface to obtain a second characteristic point set comprising the two-dimensional coordinates of the human face characteristic points; and comparing the second characteristic point set with the first characteristic point set, and when the error between the first characteristic point set and the second characteristic point set meets an error condition, taking the human face posture obtained by three-dimensional reconstruction as the target human face posture of the target object in the target video. By the method and the device, the stability and the accuracy of the determination of the target face posture can be improved.

Description

Method, device, device and storage medium for processing face pose in video

technical field

The present application relates to the technical field of artificial intelligence, and in particular, to a method, apparatus, device, and computer-readable storage medium for processing face gestures in videos.

Background technique

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

Computer Vision (CV) is one of the artificial intelligence technologies. It uses cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphics processing to make computer processing more suitable for human eyes. An image that is observed or transmitted to an instrument for inspection, where computer vision techniques include video processing.

In video processing, it is an important technology to recognize the face pose in the video frame contained in the video. In the related art, the face pose in the frequency frame is usually estimated by training a deep learning model, although this method can identify the position and direction of the face in the video frame, but the accuracy and stability are poor.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a method, apparatus, device, and computer-readable storage medium for processing a face pose in a video, which can improve the stability and accuracy of determining the target face pose.

The technical solutions of the embodiments of the present application are implemented as follows:

An embodiment of the present application provides a method for processing a face gesture in a video, including:

Perform facial feature point detection on multiple video frames that contain the target object in the target video, and obtain a first feature point set including the two-dimensional coordinates of the facial feature points;

Based on the first feature point set, three-dimensional reconstruction is performed on the target object to obtain the face pose of the target object and the three-dimensional coordinates of the face feature points of the target object;

Projecting the three-dimensional coordinates of the facial feature points to obtain a second feature point set comprising the two-dimensional coordinates of the facial feature points;

Comparing the second feature point set with the first feature point set, when the error between the first feature point set and the second feature point set satisfies the error condition, obtain the three-dimensional reconstruction The face pose of the target video is taken as the target face pose of the target object in the target video.

An embodiment of the present application provides a device for processing a face gesture in a video, including:

Detection module, for carrying out facial feature point detection to a plurality of video frames comprising target object in the target video, obtains the first feature point set comprising the two-dimensional coordinates of facial feature points;

a reconstruction module, configured to perform three-dimensional reconstruction on the target object based on the first feature point set, to obtain the face pose of the target object and the three-dimensional coordinates of the face feature points of the target object;

Projection module, for projecting the three-dimensional coordinates of the facial feature points, obtains the second feature point set comprising the two-dimensional coordinates of the facial feature points;

The comparison module is used to compare the second feature point set with the first feature point set, and when the error between the first feature point set and the second feature point set satisfies the error condition, the The face pose obtained by the three-dimensional reconstruction is used as the target face pose of the target object in the target video.

In the above-mentioned scheme, the detection module is also used for, from the target video, in a plurality of video frames comprising the target object, intercepts the face region of the target object and obtains a plurality of face region images;

Perform face feature point detection on each face region image respectively, and obtain the two-dimensional coordinates of the face feature points in each face region image;

Perform coordinate transformation on the two-dimensional coordinates of the face feature points in the face area image to obtain the two-dimensional coordinates of the face feature points in each video frame;

The two-dimensional coordinates of the face feature points in each video frame form a first feature point set.

In the above scheme, the detection module is also used to carry out face re-identification to the target video to obtain the face track of the target object;

According to the face track of the target object, it is determined that the target video includes a plurality of video frames of the target object and the face position of the target object in the video frame;

Perform face feature point detection on the face located at the face position in the video frame.

In the above scheme, the detection module is also used to re-identify the face of the target video, determine the video frame that recognizes the target object as a video frame containing the target object, and determine the face of the target object in the video frame. Location;

For the first video frame and the second video frame containing the target object, when the target object is not identified in at least one third video frame between the first video frame and the second video frame, and the third video frame When the quantity is less than the first quantity threshold, obtain the first face position of the target object in the first video frame and the second face position of the target object in the second video frame;

When the distance between the first face position and the second face position is less than a distance threshold, determine that the third video frame contains a target object, and

According to the position of the first face and the position of the second face, interpolation processing is performed to obtain the face position of the target object in at least one of the third video frames, so as to generate the face track of the target object.

In the above scheme, the detection module is also used to perform face re-identification on a plurality of video frames in the target video, determine the video frame that does not contain the target object by identifying the video frame of the target object, and determine the video frame in the video frame. The face position of the target object;

For the first video frame and the second video frame containing the target object, when the target object is not identified in at least one third video frame between the first video frame and the second video frame, and the third video frame When the number reaches the second number threshold, based on the first face position of the target object in the first video frame and the second face position of the target object in the second video frame, generate at least two paragraphs of the target object. Two face trajectories.

In the above-mentioned scheme, the reconstruction module is also used to carry out face pose estimation based on the first feature point set to obtain the face pose of the target object;

Based on the facial posture of the target object, three-dimensional reconstruction is performed on the target object through triangulation processing, and the three-dimensional coordinates of the facial feature points of the target object are determined.

In the above scheme, the reconstruction module is also used to obtain a basic matrix based on the positional relationship of each face feature point in a plurality of described video frames based on the first feature point set;

Normalize the fundamental matrix to obtain an essential matrix;

Perform singular value decomposition on the essential matrix to obtain the face pose of the target object.

In the above solution, the reconstruction module is also used to obtain the target camera internal parameters of the target video;

Based on the internal parameters of the target camera and the first feature point set, three-dimensional reconstruction is performed on the target object;

In the process of carrying out three-dimensional reconstruction to the target object, obtain the camera internal parameters of the target video;

When the error between the first feature point set and the second feature point set does not satisfy the error condition, based on the acquired in-camera parameters, update the target in-camera parameters to

Based on the updated internal parameters of the target camera and the first feature point set, three-dimensional reconstruction of the target object is performed.

In the above scheme, the reconstruction module is also used to obtain a fundamental matrix and an essential matrix in the process of carrying out three-dimensional reconstruction to the target object;

According to the conversion relationship between the fundamental matrix and the essential matrix, the in-camera parameters of the target video are acquired.

In the above scheme, the reconstruction module is also used for when the error between the first feature point set and the second feature point set does not satisfy the error condition, and the difference between the facial feature points in each of the video frames is When the number is multiple, for each target face feature point in the multiple face feature points, the two-dimensional coordinates of at least one face feature point are removed from the first feature point set to obtain a third feature point set;

Based on the third feature point set, three-dimensional reconstruction is carried out to the target object, and the three-dimensional coordinates of the facial feature points of the target object are obtained;

Projecting the three-dimensional coordinates obtained by performing three-dimensional reconstruction based on the third feature point set on the imaging plane of the camera to obtain a fourth feature point set including the two-dimensional coordinates of the face feature points;

The first feature point set is updated based on the third feature point set and the fourth feature point set.

In the above solution, the projection module is further configured to rotate and translate the three-dimensional coordinates of the facial feature points on the imaging plane of the camera according to the facial posture, so as to obtain the facial feature points in the camera coordinate system. coordinate of;

Obtain the target camera internal parameters corresponding to the target video;

According to the internal parameters of the target camera, the coordinates under the camera coordinate system are converted to the coordinates under the image coordinate system to obtain a second feature point set including the two-dimensional coordinates of the facial feature points.

In the above scheme, the device further comprises: a processing module for generating a picture corresponding to the information to be displayed based on the information to be displayed;

Obtain the original two-dimensional coordinates of the multiple vertices of the picture and the three-dimensional coordinates of the multiple vertices of the picture;

Based on the gesture of the target face, the three-dimensional coordinates of the multiple vertices of the picture are projected onto the camera imaging surface to obtain the two-dimensional coordinates of the multiple vertices of the picture;

Determine the perspective transformation matrix according to the mapping relationship between the original two-dimensional coordinates and the two-dimensional coordinates obtained by projection;

Through the perspective transformation matrix, affine transformation is performed on the picture to obtain the transformed target picture;

The target picture is superimposed on the video frame to obtain the video frame to which the information to be displayed is added.

Embodiments of the present application provide a computer device, including:

memory for storing executable instructions;

The processor is configured to, when executing the executable instructions stored in the memory, implement the method for processing the face gesture in the video provided by the embodiment of the present application.

The embodiments of the present application provide a computer-readable storage medium, which stores executable instructions for causing a processor to execute the method for processing a face gesture in a video provided by the embodiments of the present application.

The embodiments of the present application have the following beneficial effects:

Applying the above embodiment, by performing facial feature point detection on multiple video frames containing the target object in the target video, a first feature point set including the two-dimensional coordinates of the facial feature points is obtained; based on the first feature point set , perform three-dimensional reconstruction on the target object to obtain the face pose of the target object and the three-dimensional coordinates of the face feature points of the target object; project the three-dimensional coordinates of the face feature points to obtain the The second feature point set of the two-dimensional coordinates of the face feature points; comparing the second feature point set with the first feature point set, when the first feature point set and the second feature point set are When the error between the two satisfies the error condition, the face pose obtained by the three-dimensional reconstruction is used as the target face pose of the target object in the target video; in this way, the second feature point set is obtained through projection, and the first feature point set is obtained by projection. The second feature point set is compared with the first feature point set, which realizes the verification of the three-dimensional reconstruction result and improves the stability and accuracy of the determination of the target face pose.

Description of drawings

Fig. 1 is an optional framework schematic diagram of the processing system of the facial gesture in the video provided by the embodiment of the present application;

2 is a schematic flowchart of a method for processing a face gesture in a video provided by an embodiment of the present application;

3 is a schematic diagram of a screening process of facial feature points provided by an embodiment of the present application;

4 is a schematic diagram of a text picture provided in an embodiment of the present application in a face coordinate system;

5A-5B are schematic diagrams of playback interfaces provided by embodiments of the present application;

6 is a schematic flowchart of a method for processing a face gesture in a video provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a three-dimensional reconstruction process provided by an embodiment of the present application;

8 is a schematic diagram of a process of adding information to be displayed in a video frame provided by an embodiment of the present application;

9 is a schematic structural diagram of an apparatus for processing a face gesture in a video provided by an embodiment of the present application;

FIG. 10 is a schematic structural diagram of a computer device provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings. All other embodiments obtained under the premise of creative work fall within the scope of protection of the present application.

In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same or a different subset of all possible embodiments, and Can be combined with each other without conflict.

In the following description, the term "first\second\third" is only used to distinguish similar objects, and does not represent a specific ordering of objects. It is understood that "first\second\third" in the Where permitted, the specific order or sequence may be interchanged to enable the embodiments of the application described herein to be practiced in sequences other than those illustrated or described herein.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are suitable for the following explanations.

1) Face pose: refers to the position (three-dimensional coordinates) and direction of the face relative to the camera coordinate system in the picture captured by the camera.

2) In-camera parameters: In-camera parameters recorded in matrix form, including the camera focal length in the horizontal and vertical directions, the position of the optical center on the sensor plane and other parameters, which can be expressed as:

Wherein, f _x is the focal length parameter in the horizontal direction, f _y is the focal length parameter in the vertical direction, and (c _x , _cy ) is the position coordinate of the optical center on the camera sensor plane.

3) Camera external parameters: parameters used to record the rotation and translation of the camera relative to the world coordinate system. Generally speaking, a matrix is used to record the rotation of the camera, and a vector is used to record the translation of the camera. When we use the face coordinate system as the world coordinate system, the external parameter of the camera is the face pose.

4) Fundamental matrix: F is a 3×3 matrix, which expresses the correspondence between the image points of the stereo pair.

5) Essential Matrix: It is a matrix relationship used to connect two corresponding points in a three-dimensional coordinate system.

Referring to FIG. 1, FIG. 1 is a schematic diagram of an optional architecture of a system for processing a face gesture in a video provided by an embodiment of the present application. In order to support an exemplary application, a terminal (exemplarily shows terminal 400-1 and a terminal) 400-2) Connect the server 200 through the network 300, and the network 300 may be a wide area network or a local area network, or a combination of the two. The number of servers and terminals is not limited here. In actual implementation, the terminal is provided with a client, such as a video client, a browser client, an information flow client, an education client, etc., for uploading and playing videos.

The terminal is used to receive the upload instruction for the target video and send the target video to the server;

The server 200 is configured to perform face feature point detection on multiple video frames including the target object in the target video, and obtain a first feature point set including two-dimensional coordinates of the face feature points; based on the first feature point set, the target The object is reconstructed in three dimensions to obtain the face pose of the target object and the three-dimensional coordinates of the face feature points of the target object; the three-dimensional coordinates of the face feature points are projected to the imaging surface of the camera, and the two-dimensional coordinates including the face feature points are obtained. the second feature point set; compare the second feature point set with the first feature point set, when the error between the first feature point set and the second feature point set satisfies the error condition, the face pose obtained by the three-dimensional reconstruction as the target face pose of the target object in the target video;

The terminal is further configured to receive a playback instruction for the target video, and send an acquisition request for the target video, the information to be displayed, and the target face posture of the target object to the server;

The server 200 is used for delivering the target video, the information to be displayed and the target face posture of the target object to the terminal;

The terminal is used to include the target video in the playback interface; in the process of playing the target video, when the target object is included in the video picture of the target video and there is information to be displayed associated with the target object, the person who is associated with the target object The display area associated with the face displays the information to be displayed; in the process of displaying the information to be displayed, when it is determined that the target face posture of the target object changes, the display posture of the information to be displayed is adjusted along with the change of the target face posture.

In some embodiments, the server 200 may be an independent physical server, or may be a server cluster or distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, Cloud servers for basic cloud computing services such as network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a TV terminal, a vehicle terminal, etc., but is not limited to this. The terminal and the server may be directly or indirectly connected through wired or wireless communication, which is not limited in this embodiment of the present application.

Based on the above description of the processing system for the facial posture in the video according to the embodiment of the present application, the following describes the processing method for the facial posture in the video provided by the embodiment of the present application. Referring to FIG. 2, FIG. 2 is a schematic flowchart of a method for processing a face gesture in a video provided by an embodiment of the present application; in some embodiments, the method for processing a face gesture in the video may be implemented by a terminal or a server alone, or by a terminal. Implemented in collaboration with the server, taking the server as an example, the method for processing a face gesture in a video provided by the embodiment of the present application includes:

Step 201: The server performs facial feature point detection on multiple video frames containing the target object in the target video, and obtains a first feature point set including the two-dimensional coordinates of the facial feature points.

In actual implementation, the server obtains the target video and decodes the target video to obtain multiple video frames included in the target video, and selects multiple video frames including the target object from the multiple video frames included in the target video. Face feature point detection to obtain two-dimensional coordinates including face feature points. Among them, the first feature point set is composed of a plurality of face feature points, and the face feature points are represented by the two-dimensional coordinates of the face feature points.

In practical applications, for each video frame containing the target object, the face feature point detection is performed on the video frame, and the two-dimensional coordinates of the face feature point in the video frame are obtained. In this way, the face feature point can be obtained. Two-dimensional coordinates in a plurality of video frames, these two-dimensional coordinates constitute the first feature point set.

For example, the 68-point feature point detection algorithm can be used to detect the feature points of the face. Then, for each video frame, the two-dimensional coordinates of the 68 face feature points will be obtained. For example, the face attention mechanism network ( FAN, FaceAttention Network) detects feature points on faces.

In some embodiments, the terminal may perform facial feature point detection on multiple video frames including the target object in the target video by: intercepting the face of the target object from the multiple video frames including the target object in the target video obtain multiple face region images; perform face feature point detection on each face region image respectively, and obtain the two-dimensional coordinates of the face feature points in each face region image; Coordinate transformation is performed on the two-dimensional coordinates to obtain the two-dimensional coordinates of the face feature points in each video frame; the two-dimensional coordinates of the face feature points in each video frame form a first feature point set.

In practical applications, in order to improve the detection accuracy of face feature points, the face region image corresponding to the face region of the target object can be intercepted from the video frame, and then the face region image is detected by face feature points; here , the two-dimensional coordinates of the face feature points obtained when the face region image is detected by the face feature points are the two-dimensional coordinates under the coordinate system corresponding to the face region image. Based on this, it needs to be transformed into the corresponding video frame. in the coordinate system.

In actual implementation, the position of the face area image in the video frame is obtained to determine the conversion relationship between the coordinate system of the corresponding face area image and the coordinate system of the corresponding video frame. For example, when the upper left vertex of the face area image is The position in the video frame is (20, 20), then when transforming the two-dimensional coordinates of the face feature points in the face area image, the abscissa of the two-dimensional coordinates of the face feature points in the face area image Add 20 to both the ordinate and the ordinate to get the two-dimensional coordinates of the face feature points in the video frame.

In some embodiments, before performing facial feature point detection on multiple video frames in which the target video contains the target object, the target video may also be re-identified to obtain the facial trajectory of the target object; correspondingly, the terminal may The facial feature point detection is performed on multiple video frames containing the target object in the target video in the following manner: according to the face trajectory of the target object, the multiple video frames containing the target object in the target video and the target object in the video frame are determined. Face position; perform face feature point detection on the face located at the face position in the video frame.

Here, since the video frame may include multiple objects, that is, the video frame including the target object may also include other objects. Based on this, it is necessary to determine which object is the target object, so as to determine the facial feature points of the target object. test.

In actual implementation, the face trajectory of the target object can be obtained by means of face re-recognition, where the face trajectory can identify the face position of the target object in each video frame; in this way, after obtaining the face trajectory , it can determine which video frames contain the target object, and the face position of the target object in the corresponding video frame; Other objects are determined as target objects, and the facial feature points of other objects are detected.

In practical applications, the server may perform face detection on each video frame included in the target video to obtain all faces in each video frame; then, based on the detected faces, the face regions are intercepted to obtain faces Region image, re-identify each face region image to obtain the object corresponding to each face region image; and then determine the face trajectory of the target object according to the position of the face region image corresponding to the target object in the corresponding video frame. Among them, the position of the face area image in the corresponding video frame is the face position of the target object in the video frame.

In some embodiments, face re-identification can be performed on the target video to obtain the face trajectory of the target object: face re-identification is performed on the target video, and the video frame in which the target object is recognized is determined as a video containing the target object. frame, and determine the face position of the target object in the video frame; for the first video frame and the second video frame containing the target object, when in at least one third video frame between the first video frame and the second video frame When the target object is not recognized and the number of the third video frames is less than the first number threshold, obtain the first face position of the target object in the first video frame and the second face position of the target object in the second video frame When the distance between the first human face position and the second human face position is less than the distance threshold, determine that the third video frame comprises the target object, and according to the first human face position and the second human face position, carry out interpolation processing, obtain face position of the target object in at least one third video frame to generate a face trajectory of the target object.

In actual implementation, the server may perform face detection on each video frame included in the target video to obtain all faces in each video frame; then, based on the detected faces, intercept the face region to obtain the face Region image, re-identify each face region image to obtain the object corresponding to each face region image; and then determine the face trajectory of the target object according to the position of the face region image corresponding to the target object in the corresponding video frame. Among them, the position of the face area image in the corresponding video frame is the face position of the target object in the video frame.

In practical applications, when determining the face position of the target object according to the position of the face region image corresponding to the target object in the corresponding video frame, if the target object is not recognized in a certain M video frames (third video frame) , but the target object is recognized in the two video frames before and after M video frames (ie, the first video frame and the second video frame), and in the two video frames before and after, the distance between the face positions of the target object is less than The distance threshold indicates that in the two video frames before and after, the face position of the target object is close enough, and interpolation can be used here to supplement the face position of the intermediate frame (third video frame). Wherein, M is a natural number, and M is smaller than the first quantity threshold, for example, the first quantity threshold may be 3.

Exemplarily, a linear interpolation method can be used here to obtain the face position of the target object in the third video frame, that is, a linear function is constructed according to the first face position and the second face position, and then based on the linear function Determine the face position of the target object in the third video frame. For example, the first face position is (200, 200) and the second face position is (260, 260). When there are two third video frames between the first video frame and the second video frame, the first The face position of the target object in the third video frame (one video frame after the first video frame) is (220, 220), and the face position of the target object in the second third video frame (two video frames after the first video frame) The face position is (240, 240).

Here, since the target object is not detected in the third video frame, when determining the face pose of the target object in the third face frame, interpolation can also be used to determine, that is, the person who obtains the target object in the first video frame face pose and the face pose of the target object in the second video frame, and then determine the third video frame by interpolation according to the face pose of the target object in the first video frame and the face pose of the target object in the second video frame face pose of the target object.

In practical applications, if the target object is not detected in a certain M video frame (the third video frame), but the two video frames before and after the M video frames (ie the first video frame and the second video frame) are identified The target object, and the distance between the target object's face position in the two video frames before and after reaches the distance threshold, it means that in the two video frames before and after the target object's face position is not close enough, then it is considered that the video picture When switching occurs, the face trajectory is divided into two segments according to the moment when the video screen is switched, that is, the face position in the first video frame is taken as the end position of the previous face trajectory, and the face in the second video frame is used. The position is used as the starting position of the next face trajectory.

In some embodiments, the face re-identification of the target video can be performed to obtain the face trajectory of the target object: face re-identification is performed on multiple video frames in the target video, and the video frame of the identified target object is determined as The video frame containing the target object, and determining the face position of the target object in the video frame; for the first video frame and the second video frame containing the target object, when at least one video frame between the first video frame and the second video frame When the target object is not recognized in the third video frame and the number of the third video frames reaches the second number threshold, based on the position of the first face of the target object in the first video frame and the position of the target object in the second video frame The second face position generates at least two face trajectories of the target object.

In practical applications, if the target object is not identified in a certain N video frames (the third video frame), but the target object is identified in the two video frames before and after the N video frames (ie, the first video frame and the second video frame) For the target object, compare N with the second number threshold. If N reaches the second number threshold, it is considered that the video screen is switched. According to the moment when the video screen is switched, the face trajectory is divided into two segments, that is, the first video. The face position in the frame is taken as the end position of the previous face trajectory, and the face position in the second video frame is taken as the start position of the next face trajectory. Wherein, N is a positive integer, and the second quantity threshold is greater than or equal to the first quantity threshold.

In some embodiments, the target object is identified in the first video frame, the second video frame, and the third video frame, wherein the first video frame is a previous video frame adjacent to the third video frame, and the second video frame The frame is the next video frame adjacent to the third video frame. When the third video frame is relative to the first video frame and the second video frame, the face position of the target object jumps, that is, the target object in the first video frame jumps. The distance between the face position of the target object in the third video frame and the face position of the target object in the third video frame exceeds the distance threshold, and the distance between the face position of the target object in the second video frame and the face position of the target object in the third video frame The distance exceeds the distance threshold, for example, the distance threshold can be set to 5% of the image height, then it is considered that the target object detection in the third video frame fails, through the face position in the first video frame and the person in the second video frame. The face position is obtained by interpolation to obtain the correct face data in the video frame.

Step 202: Based on the first feature point set, perform three-dimensional reconstruction on the target object to obtain the face pose of the target object and the three-dimensional coordinates of the face feature points of the target object.

In actual implementation, the first feature point set includes the two-dimensional coordinates of face feature points in multiple video frames, and three-dimensional reconstruction is performed on the target object according to the positional relationship of each face feature point in different video frames. For example, when When the face feature points include the left corner of the left eye, the right corner of the right eye, the left corner of the mouth, the right corner of the mouth, the tip of the nose, and the chin, the two-dimensional coordinates of the left corner of the left eye in each video frame are obtained, and the position of the left corner of the left eye in each video frame is determined. Positional relationship, based on the positional relationship, three-dimensional reconstruction of the target object is performed, and the above-mentioned operation for the left corner of the left eye is performed for each face feature point to obtain the face pose of the target object and the face feature point of the target object. three-dimensional coordinates.

In some embodiments, three-dimensional reconstruction of the target object may be performed based on the first feature point set in the following manner to obtain the face pose of the target object and the three-dimensional coordinates of the face feature points of the target object: Based on the first feature point set The face pose estimation is performed to obtain the face pose of the target object; based on the face pose of the target object, the three-dimensional reconstruction of the target object is performed through triangulation processing, and the three-dimensional coordinates of the face feature points of the target object are determined.

In actual implementation, first, according to the positional relationship of each face feature point in the first feature point set in multiple video frames, the face pose estimation of the target object is performed to obtain the face pose of the target object; Face pose, three-dimensional reconstruction of the target object is performed through triangulation processing, and the three-dimensional coordinates of the face feature points of the target object are obtained.

Here, the triangulation process will be described. When the face coordinate system of the target object is used as the world coordinate system, the change in the face pose of the target object is equivalent to observing the face of the target object at different positions, and it is known that different In the case of the two-dimensional coordinates of the face observed at the position, the three-dimensional reconstruction of the target object is performed by using the triangular relationship in the projected perspective model.

In some embodiments, the server may perform face pose estimation with the first feature point set to obtain the face pose of the target object: based on the positional relationship of each face feature point in the first feature point set in multiple video frames , obtain the basic matrix; normalize the basic matrix to obtain the essential matrix; perform singular value decomposition on the essential matrix to obtain the face pose of the target object.

计算基础矩阵，其中，F为基础矩阵，

和

是视频帧中的 一对匹配点在相应视频帧中的二维坐标，例如，相邻两个视频帧中同一人脸特 征点的二维坐标，如第一视频帧中下巴的二维坐标为

第二视频帧中下巴的二 维坐标为

In actual implementation, the fundamental matrix is a 3x3 matrix with a rank of 2 and 7 degrees of freedom, which can be determined according to

Calculate the fundamental matrix, where F is the fundamental matrix,

and

is the two-dimensional coordinates of a pair of matching points in the video frame in the corresponding video frame, for example, the two-dimensional coordinates of the same face feature point in two adjacent video frames, for example, the two-dimensional coordinates of the chin in the first video frame are

The two-dimensional coordinates of the chin in the second video frame are

Here, when calculating the fundamental matrix, the two-dimensional coordinates of multiple facial feature points can be used to make the determined fundamental matrix more accurate.

After the fundamental matrix is obtained, since the relationship between the fundamental matrix and the essential matrix is: F=K- ^T EK', where F is the fundamental matrix, K is the camera internal parameter, and E is the essential matrix. After normalization, the in-camera parameters become the identity matrix, and at this time, the fundamental matrix is the essential matrix. Based on this, the essential matrix is obtained by normalizing the fundamental matrix.

After obtaining the essential matrix, perform singular value decomposition on the essential matrix. Here, by performing singular value decomposition on the essential matrix, four sets of translation vectors and rotation angles can be obtained, of which only one set is correct. Based on this, according to the triangulation process, determine The three-dimensional coordinates corresponding to the face feature points are substituted into each set of translation vectors and rotation angles to obtain the three-dimensional coordinates in the corresponding camera coordinate system. Only the depth value of one set of results is positive, and the correct translation vector is filtered out. and rotation angle to get the face pose of the target object.

Step 203: Project the three-dimensional coordinates of the face feature points to obtain a second feature point set including the two-dimensional coordinates of the face feature points.

In actual implementation, according to the face pose of the target object, the face feature points are projected onto the camera presentation surface. Here, the face poses of the target objects in different video frames are different. When the three-dimensional coordinates of the face feature points are projected, the two-dimensional coordinates of the face feature points corresponding to each video frame are obtained. For example, when each face is represented by 68 face feature points, the two-dimensional coordinates of the face feature points corresponding to each video frame are the two-dimensional coordinates of the 68 face feature points obtained by projection.

In some embodiments, the projection of the three-dimensional coordinates of the facial feature points can be obtained in the following manner to obtain a second feature point set including the two-dimensional coordinates of the facial feature points: The three-dimensional coordinates are rotated and translated for the camera imaging plane to obtain the coordinates of the face feature points in the camera coordinate system; the internal parameters of the target camera corresponding to the target video are obtained; according to the internal parameters of the target camera, the coordinates in the camera coordinate system are converted to the image The coordinates in the coordinate system are obtained, and the second feature point set including the two-dimensional coordinates of the face feature points is obtained.

Here, the face pose is represented by the translation vector of the face relative to the camera and three rotation angles (pitch angle, yaw angle, and roll angle). In actual implementation, the three-dimensional coordinates of the face feature points are translated according to the translation vector. , and rotate the three-dimensional coordinates of the face feature points according to the three rotation angles to obtain the coordinates of the face feature points in the camera coordinate system; then, obtain the internal parameters of the target camera and the coordinates of the face feature points in the camera coordinate system The product is taken as the coordinates of the face feature points in the image coordinate system, so as to realize the projection of the three-dimensional coordinates of the face feature points.

Step 204: Compare the second feature point set with the first feature point set, and when the error between the first feature point set and the second feature point set satisfies the error condition, use the face pose obtained by the three-dimensional reconstruction as the target video The target face pose of the target object in .

In actual implementation, the mean square error between the second feature point set and the first feature point set can be obtained, and when the mean square error between the second feature point set and the first feature point set is less than the error threshold, determine the first The error between the first feature point set and the second feature point set satisfies the error condition; otherwise, it is determined that the error between the first feature point set and the second feature point set does not satisfy the error condition.

In some embodiments, the three-dimensional reconstruction of the target object may be performed based on the first feature point set in the following ways: acquiring target camera internal parameters of the target video; based on the target camera internal parameters and the first feature point set, performing three-dimensional reconstruction on the target object reconstruction; the method further includes: in the process of three-dimensional reconstruction of the target object, acquiring the in-camera parameters of the target video; when the error between the first feature point set and the second feature point set does not meet the error condition, based on the acquired In-camera parameters, the target camera internal parameters are updated, so as to perform three-dimensional reconstruction of the target object based on the updated target camera internal parameters and the first feature point set.

Here, the target camera internal parameters of the target video acquired for the first time are the initial camera internal parameters, and the initial camera internal parameters can be manually given or obtained based on the camera internal parameter estimation algorithm. Here, the initial in-camera parameters are <focal length is set to 50mm> <optical center position is set to the center of the imaging plane>, when the initial in-camera parameters are given manually, they can be set according to the height of the image. When the error between the first feature point set and the second feature point set does not meet the error condition, for example, when the mean square error between the second feature point set and the first feature point set reaches the error threshold, it indicates that the parameters in the target camera are It may not be accurate enough. Here, the internal parameters of the target camera are updated according to the internal parameters of the camera obtained during the three-dimensional reconstruction.

In actual implementation, after obtaining the internal parameters of the target camera, the three-dimensional coordinates of the facial feature points in each video frame in the corresponding camera coordinate system can be determined according to the internal parameters of the target camera; then, based on p ^T Ep′=0, obtain The calculation essential matrix, where E is the basic matrix, p and p' are the three-dimensional coordinates of a pair of matching points in the video frame in the camera coordinate system, for example, the same face feature point in two adjacent video frames is in the corresponding The three-dimensional coordinates in the camera coordinate system, for example, the three-dimensional coordinate of the chin in the camera coordinate system in the first video frame is p, and the two-dimensional coordinate of the chin in the second video frame is p'; after obtaining the essential matrix, the essential matrix Perform singular value decomposition to obtain the face pose of the target object.

Here, the internal parameters of the target camera can be updated in an iterative manner until the error between the first feature point set and the second feature point set satisfies the error condition.

In some embodiments, the in-camera parameters of the target video may be acquired during the 3D reconstruction of the target object in the following manner: during the 3D reconstruction of the target object, a fundamental matrix and an essential matrix are acquired; according to the fundamental matrix The conversion relationship with the essential matrix to obtain the in-camera parameters of the target video.

In the actual implementation, in the process of 3D reconstruction, the fundamental matrix and the essential matrix will be obtained, as F=K ^{- T} EK', where F is the fundamental matrix, K is the internal parameter of the camera, and E is the essential matrix, so, The in-camera parameter K can be calculated.

In some embodiments, when the error between the first feature point set and the second feature point set does not satisfy the error condition, and the number of face feature points in each video frame is multiple Eliminate the two-dimensional coordinates of at least one face feature point from the feature point set to obtain a third feature point set; based on the third feature point set, perform three-dimensional reconstruction on the target object to obtain the three-dimensional coordinates of the face feature points of the target object; The three-dimensional coordinates obtained by the three-dimensional reconstruction of the third feature point set are projected onto the imaging surface of the camera, and the fourth feature point set including the two-dimensional coordinates of the face feature points is obtained; based on the third feature point set and the fourth feature point set, update The first feature point set.

In actual implementation, when 3D reconstruction is performed for the first time, the two-dimensional coordinates of all detected facial feature points are used as the first feature point set. If the error between the first feature point set and the second feature point set does not satisfy Error conditions, the facial feature points can be screened to eliminate the facial feature points that reduce the accuracy of the 3D reconstruction. For example, due to the behavior of people speaking, the position of the facial feature point corresponding to the upper lip relative to other facial feature points is not fixed. Based on this, the facial feature point corresponding to the upper lip can be eliminated from the facial feature points. And after culling, the culling results are verified, that is, based on the third feature point set, three-dimensional reconstruction of the target object is performed to obtain the three-dimensional coordinates of the face feature points of the target object; The three-dimensional coordinates obtained by the three-dimensional reconstruction are projected to the imaging surface of the camera to obtain the fourth feature point set; the error between the third feature point set and the fourth feature point set is determined, and when the error meets a certain error condition, the third feature point set is used. The feature point set updates the first feature point set.

For example, the two-dimensional coordinates of 68 face feature points are obtained by performing feature point detection on the target object, then, the first feature point set contains the two-dimensional coordinates of 68 face feature points; the face corresponding to the lips is eliminated from the first feature point set. The two-dimensional coordinates of the feature points, since the number of face feature points corresponding to the lips is 16, then, the two-dimensional coordinates of the remaining 52 face feature points, the two-dimensional coordinates of these 52 face feature points form the third feature point set Then, based on the third feature point set, three-dimensional reconstruction is performed on the target object to obtain the three-dimensional coordinates of the 52 facial feature points; the three-dimensional coordinates of the 52 facial feature points are projected to the camera imaging surface to obtain the fourth feature point set, The fourth feature point set here includes the two-dimensional coordinates of the 52 face feature points obtained by projection; the two-dimensional coordinates in the third feature point set are compared with the two-dimensional coordinates in the fourth feature point set, and the third feature point set and The error between the fourth feature point set, when the error satisfies the error condition, the first feature point set is updated with the third feature point set.

In some embodiments, multiple third feature point sets and corresponding fourth feature point sets may be obtained, and by comparing the errors between the multiple third feature point sets and the corresponding fourth feature point sets, the first feature point set to update. As an example, FIG. 3 is a schematic diagram of a screening process of facial feature points provided by an embodiment of the present application. Referring to FIG. 3 , for each facial feature point, the two-dimensional coordinates of the facial feature point are removed from the first feature point set to obtain The corresponding third feature point set, such as removing the two-dimensional coordinates of the face feature point 1 from the first feature point set, obtains the corresponding third feature point set, and obtains the third feature point set 1; when the number of face feature points is When it is N, N third feature point sets can be obtained. After N third feature point sets are obtained, for each third feature point set, based on the third feature point set, three-dimensional reconstruction of the target object is performed to obtain the three-dimensional coordinates of the face feature points of the target object; The three-dimensional coordinates obtained by the three-dimensional reconstruction of the third feature point set are projected onto the imaging plane of the camera, and the fourth feature point set including the two-dimensional coordinates of the face feature points is obtained; after obtaining the fourth feature points corresponding to the N third feature point sets After the set, based on the error between each third feature point set and the corresponding fourth feature point set, the third feature point set with the smallest corresponding error is selected to update the first feature point set.

Here, the face feature points can be screened in an iterative manner, until after the first feature point set is updated, the obtained error satisfies the error condition, or the number of face feature points is lower than the number threshold.

In some embodiments, after obtaining the target face pose, the server may further perform filtering processing on the target face pose based on the face trajectory to obtain a jitter-free target face pose.

In actual implementation, for each face track, the target face pose sequence corresponding to the face track is obtained; here, the face pose in the target face pose sequence includes 6-dimensional data, respectively for the target face The data of each dimension in the pose sequence is filtered to filter out jitter. Among them, the extended Kalman filter can be used to filter the data of each dimension, and the value of the start time point of the face track is used as the initial value of the extended Kalman filter to ensure that the filter results are normally available at the start time of each track.

In some embodiments, the terminal may also generate a picture corresponding to the information to be displayed based on the information to be displayed; obtain the original two-dimensional coordinates of multiple vertices of the image and the three-dimensional coordinates of multiple vertices of the image; based on the target face posture, Projecting the three-dimensional coordinates of the multiple vertices of the picture onto the imaging surface of the camera to obtain the two-dimensional coordinates of the multiple vertices of the picture; determining the perspective transformation matrix according to the mapping relationship between the original two-dimensional coordinates and the projected two-dimensional coordinates; Through the perspective transformation matrix, affine transformation is performed on the picture to obtain the transformed target picture; the target picture is superimposed on the video frame to obtain the video frame to which the information to be displayed is added.

In actual implementation, the information to be displayed can be displayed based on the determined target face pose, so that in the process of displaying the information to be displayed, when the target face pose of the target object changes, along with the change of the target face pose, Adjust the display posture of the information to be displayed. Here, the information to be displayed may be text information, image information, etc., and the specific form of the information to be displayed is not limited here.

In practical applications, when the information to be displayed is image information, the information to be displayed is directly used as a picture; when the information to be displayed is text information, a text picture is drawn to obtain a picture corresponding to the information to be displayed. Here, when drawing a text picture, automatic line wrapping is optional, that is, after the character length reaches a given threshold, a newline symbol is automatically added, the text has a transparent channel, and the non-text area is transparent.

In practical applications, multiple vertices refer to at least two vertices. Here, the original two-dimensional coordinates of the picture can be determined according to the shape, size and display position of the picture. For example, when the picture is a rectangle, its length is w, and its width is is H, when the height of the text image from the nose of the target object is D, the four vertices of the text image are the upper left corner (0, 0), the lower left corner (0, h-1), the lower right corner (w-1, h-1) ), upper right corner (w-1, 0).

And, the three-dimensional coordinates of the multiple vertices of the picture here refer to the three-dimensional coordinates of the multiple vertices of the picture in the face coordinate system, which can be determined according to the size of the picture and the relationship between the original coordinates and the face of the target object. Positional relationship, determine the original coordinates of multiple vertices, such as obtaining the length and width of the picture, based on the length and width of the picture, and the positional relationship between the display area of the picture and the face of the target object, determine that the multiple vertices of the picture are in 3D coordinates in the face coordinate system.

As an example, taking the image displayed on the head of the target object as an example, FIG. 4 is a schematic diagram of a text image provided by an embodiment of the present application in the face coordinate system. Referring to FIG. 4 , when the image is a rectangle, its length is w, and its width is H, when the height of the image from the nose of the target object is D, determine the coordinates of the four vertices of the image in the face coordinate system as the upper left corner (-W/2, -D-H, 0), the lower left corner (-W/2 , -D, 0), lower right corner (W/2, -D, 0), upper right corner (W/2, -D-H, 0).

After acquiring the original two-dimensional coordinates of the multiple vertices of the picture and the three-dimensional coordinates of the multiple vertices of the image, based on the target face posture, the three-dimensional coordinates of the multiple vertices of the image are projected on the imaging surface of the camera to obtain the multi-dimensional coordinates of the image. Two-dimensional coordinates of each vertex; mark the two-dimensional coordinates of multiple vertices as point combination A, mark the original two-dimensional coordinates of multiple vertices as point combination B, and calculate the perspective transformation matrix M that transforms point combination B to point combination A ; Then, perform affine transformation on the picture through the perspective transformation matrix, that is, multiply the coordinates of each point in the picture with the perspective transformation matrix to obtain the coordinates of the modified point in the target picture, and then obtain the target picture. After the target picture is obtained, the target picture is superimposed on the video frame. After receiving the playback instruction for the target video, the terminal performs rendering based on the video frame to which the information to be displayed is added, so that the display posture of the rendered information to be displayed matches the facial posture of the target object.

Referring to FIGS. 5A-5B , FIGS. 5A-5B are schematic diagrams of a playback interface provided by an embodiment of the present application. The display posture of the information to be displayed 501 is consistent with the facial posture of the target object 502 , both of which are rotated to the left; and the display posture of the information to be displayed 503 The display posture is consistent with the face posture of the target object 504, and both are rotated to the left; and, since the face of the target object 504 is rotated to the left by a larger angle relative to the face of the target object 502, correspondingly, the information to be displayed 503 The angle of rotation to the left relative to the information 501 to be displayed is larger.

Applying the above-mentioned embodiment, by performing facial feature point detection on multiple video frames including the target object in the target video, a first feature point set including the two-dimensional coordinates of the facial feature points is obtained; Perform three-dimensional reconstruction of the target object to obtain the face pose of the target object and the three-dimensional coordinates of the face feature points of the target object; project the three-dimensional coordinates of the face feature points to obtain a second coordinate including the two-dimensional coordinates of the face feature points. Feature point set; compare the second feature point set with the first feature point set, when the error between the first feature point set and the second feature point set satisfies the error condition, take the face pose obtained by 3D reconstruction as the target The target face pose of the target object in the video; in this way, the second feature point set is obtained through projection, and the second feature point set is compared with the first feature point set, which realizes the verification of the three-dimensional reconstruction results and improves the target Stability and accuracy of facial pose determination.

Below, an exemplary application of the embodiments of the present application in a practical application scenario will be described.

FIG. 6 is a schematic flowchart of a method for processing a face gesture in a video provided by an embodiment of the present application. Referring to FIG. 6 , in some embodiments, the method for processing a face gesture in the video may be implemented by a terminal or a server alone, or by a terminal Implemented in collaboration with the server, taking the server as an example, the method for processing a face gesture in a video provided by the embodiment of the present application includes:

Step 601: The server receives the target video uploaded by the user.

Step 602: The server performs face detection on each video frame in the target video through a face detection algorithm.

For each video frame, face detection is performed on the video frame, and all faces in the video frame are detected. Here, a multi-task convolutional neural network (MTCNN, Multi-task convolutional neural network) can be used to perform face detection on the video frame. .

Step 603: The server re-identifies the human face in each video frame.

Here, the star recognition algorithm can be used to re-identify the face to obtain the face trajectory. It can be understood that the face trajectory here corresponds to the object, that is, when the target video contains multiple objects, the face trajectory of each object can be obtained.

In actual implementation, face re-identification is performed on the faces in each video frame to determine the object to which each face belongs, and the face trajectory is obtained according to the position of the same object in consecutive video frames.

In practical applications, if the face of an object is not detected in a certain M video frames, but the face of the object is detected in the two video frames before and after the M video frames, and the two video frames before and after , the face position of the object (the position of the face frame) is close enough, and the interpolation method is used to supplement the face data (face position, face pose, etc.) of the intermediate frame. Among them, M is a natural number, such as M can take 2.

If the face of a certain object is detected in a video frame, but the face position of the two video frames before and after the video frame jumps (the distance between the face positions in the adjacent two video frames exceeds the distance threshold , such as 5% of the image height), it is considered that the face detection of the video frame fails, and the correct face data in the video frame is obtained by interpolation through the face data of the preceding and following frames.

If there is no face of an object in N consecutive video frames, but the face of the object is detected in the two video frames before and after the N video frames, it is considered that the video screen is switched, and the face trajectory is switched according to the switch. The moment is divided into 2 segments, where N is a natural number, and N is greater than M, such as M is 2, N is 3.

Step 604: The server detects the facial feature points.

Here, based on the face trajectory, the feature points of the face are detected. Here, the feature point detection algorithm with 68 points can be used to detect the feature points of the face, such as using the face attention mechanism network (FAN, Face Attention Network) Detect facial feature points.

In actual implementation, based on the detected face, the face region image corresponding to each video frame is intercepted, and the face region image sequence matching each face track is obtained according to the face track. Here, for the face region For each face area image in the image sequence, perform face feature point detection on the face area image to determine the coordinates of each face feature point in the face area image; then based on the position of the face area image in the video frame , transform the coordinates of the facial feature points to determine the coordinates of each facial feature point in the video frame.

Step 605: Obtain initial in-camera parameters.

Here, an initial in-camera parameter for the initialization of the subsequent algorithm is obtained, and the initial in-camera parameter can be manually given or obtained based on the estimation algorithm of the camera's internal parameter. Here, the initial in-camera parameters are <focal length is set to 50mm> <optical center position is set to the center of the imaging plane>, when the initial in-camera parameters are given manually, they can be set according to the height of the image.

Step 606: Perform three-dimensional reconstruction based on the initial in-camera parameters and facial feature points.

Here, a monocular camera 3D reconstruction algorithm is used for 3D reconstruction, for example, a structure from motion (SFM, Structure From Motion) algorithm is used to obtain the face pose, accurate camera internal parameters and the 3D coordinates of the face feature points.

The three-dimensional reconstruction process will be specifically described below. Fig. 7 is a schematic diagram of a three-dimensional reconstruction process provided by an embodiment of the present application. Referring to Fig. 7, the three-dimensional reconstruction process provided by an embodiment of the present application includes:

Step 701: Acquire face coordinates in each video frame.

Step 702: Screen the face feature points to obtain a first feature point set.

Here, the face feature points can be screened in an iterative manner, until after the first feature point set is updated, the obtained error satisfies the error condition, or the number of face feature points is lower than the number threshold. For the first time, all face feature points are retained. In the iterative process, the method shown in Figure 3 is used to screen the face feature points. See Figure 3. For each face feature point, the face is removed from the first feature point set. The two-dimensional coordinates of the feature points are obtained to obtain the corresponding third feature point set. For example, the two-dimensional coordinates of the face feature point 1 are removed from the first feature point set to obtain the corresponding third feature point set, that is, the third feature point set 1. ; When the number of face feature points is N, N third feature point sets can be obtained. After obtaining N third feature point sets, for each third feature point set, input the third feature point set into the SMF module, carry out three-dimensional reconstruction of the target object, and obtain the three-dimensional coordinates of the face feature points of the target object; The three-dimensional coordinates obtained by performing three-dimensional reconstruction based on the third feature point set are projected to the imaging plane of the camera to obtain a fourth feature point set including the two-dimensional coordinates of the face feature points; After the four feature point sets are set, based on the error between each third feature point set and the corresponding fourth feature point set, the third feature point set with the smallest corresponding error is selected to update the first feature point set.

Step 703: Acquire internal parameters of the target camera.

Here, the target camera internal parameters acquired for the first time are the initial camera internal parameters, and the initial camera internal parameters can be set manually. After the update operation of the camera internal parameters is performed, the target camera internal parameters are the updated camera internal parameters.

Step 704: Input the internal parameters of the target camera and the first feature point set into the SMF module, and output the facial posture, the internal parameters of the camera, and the three-dimensional coordinates of the facial feature points.

Here, through the SMF module, based on the positional relationship of each face feature point in the first feature point set in multiple video frames, the target object is reconstructed in 3D, so as to obtain the 3D representation of the face pose, the parameters in the camera and the face feature points. coordinate.

Step 705: Project the three-dimensional coordinates of the facial feature points onto the imaging surface of the camera according to the facial posture and the internal parameters of the camera to obtain a second feature point set.

Step 706: Obtain the mean square error between the first feature point set and the second feature point set.

Step 707: Determine whether the mean square error is less than the error threshold, if so, go to Step 708; otherwise, go to Step 709 and Step 702.

Step 708 : Output the obtained face pose, internal parameters of the camera, and three-dimensional coordinates of the face feature points.

Step 709: Update the target camera in-camera parameters using the obtained in-camera parameters.

Here, after the obtained face pose is output, the face pose can also be filtered based on the face trajectory to obtain a face pose without jitter.

For each face track, obtain the face pose sequence corresponding to the face track; here, the face pose in the face pose sequence includes data of 6 dimensions, and the data of each dimension in the face pose sequence are respectively A filtering process is performed to filter out jitter. Among them, the extended Kalman filter can be used to filter the data of each dimension, and the value of the start time point of the face track is used as the initial value of the extended Kalman filter to ensure that the filter results are normally available at the start time of each track.

After obtaining the face pose without shaking, according to the sequence of video frames, the detected face position, face pose, face ID (identification object), video frame serial number, video ID, lens focal length, etc. The information is stored in the face pose database for subsequent use by the terminal.

Here, after obtaining the facial posture, the information to be displayed can be bound with the human face, so that when the information to be displayed is displayed, the presentation posture of the information to be displayed changes synchronously with the change of the facial posture of the object.

Here, taking the information to be displayed as the bullet screen information as an example, referring to FIG. 5A , the bullet screen information (information to be displayed) 501 associated with the target object 502 is displayed in the head area of the target object 502 . The display posture of the bullet screen matches the display posture of the bullet screen. During the playback of the video, as the face posture of the target object changes, the display posture of the bullet screen changes accordingly.

The association between the target object and the bullet screen may be determined by matching the bullet screen with the object in the video screen, or it may be associated by the user when the user publishes the bullet screen.

Taking the information to be displayed as subtitles as an example, referring to FIG. 5B , subtitles 503 corresponding to the current dialogue content of the target object are displayed in the head area of the target object 504 , wherein the facial posture of the target object matches the display posture of the subtitles, and in the During the playback of the video, as the face posture of the target object changes, the display posture of the bullet screen changes accordingly.

The specific process of adding the information to be displayed in the video frame will be described below. The specific process of adding the information to be displayed in the video frame can be performed by the terminal, the server, or the server and the terminal jointly. Taking execution as an example, FIG. 8 is a schematic diagram of a process of adding information to be displayed in a video frame provided by an embodiment of the present application. Referring to FIG. 8 , when the information to be displayed is text, the process of adding the information to be displayed in the video frame by the terminal includes:

Step 801: Draw a text picture.

Draw text pictures according to the text content of the information to be displayed; among them, automatic line wrapping is optional during drawing, that is, after the character length reaches a given threshold, a line break symbol is automatically added, the text has a transparent channel, and the non-text area is transparent.

Step 802: Determine the coordinates of the text picture in the face coordinate system.

Here, taking the drawing on the top of the target object as an example, in the face coordinate system, the height D (y-axis direction) of the text image to the nose is fixed, and the height H of the text image is fixed, according to the aspect ratio r of the text image, Calculate the width W of the text picture in the face coordinate system, where W=Hr. For example, D can be set to 125 and H can be set to 40.

Correspondingly, the coordinates of the four vertices of the text picture in the face coordinate system can be determined, that is, the upper left corner (-W/2, -D-H, 0), the lower left corner (-W/2, -D, 0), the right Bottom (W/2, -D, 0), top right (W/2, -D-H, 0).

Step 803: Based on the posture of the face, using the method of point cloud projection, project the coordinates of the text picture in the face coordinate system onto the imaging surface of the camera.

In actual implementation, the method of point cloud projection is used to project the coordinates of the four vertices of the text image in the face coordinate system onto the imaging surface of the camera, and the four vertices of the text image in the video frame are obtained in the video frame. The coordinates of these 4 vertices in the video frame form point combination A.

In practical applications, the three-dimensional coordinates of the text image in the camera coordinate system are calculated based on the face pose; then the three-dimensional coordinates of the text image in the camera coordinate system are projected onto the camera imaging surface.

Step 804: Calculate the transformation matrix for transforming the text image to the camera imaging surface.

In actual implementation, the original two-dimensional coordinates of the four vertices of the text image are the upper left corner (0, 0), the lower left corner (0, h-1), the lower right corner (w-1, h-1), the upper right corner ( w-1, 0), the original two-dimensional coordinates of these 4 vertices are composed of point combination B. Here, the transformation matrix M for transforming the point combination B to the point combination A is calculated.

Step 805: Using the transformation matrix, perform affine transformation on the text picture to obtain the transformed text picture.

Step 806: The transformed text picture is superimposed on the video frame to obtain the video frame to which the information to be displayed is added.

By applying the above embodiment, more accurate parameters in the camera can be obtained, the effect of adding information to be displayed in the future can be ensured, and the accuracy of the determination of the face pose can be improved; moreover, while the face pose is determined, the face can also be obtained. The three-dimensional coordinates of the feature points to obtain the three-dimensional shape of the face.

The following continues to describe the apparatus for processing a face gesture in a video provided by the embodiment of the present application. Referring to Fig. 9, Fig. 9 is a schematic structural diagram of a processing device of a human face gesture in a video provided by an embodiment of the present application, and the processing device of a human face gesture in a video provided by an embodiment of the present application includes:

Detection module 901, for carrying out facial feature point detection to multiple video frames comprising target object in the target video, obtains the first feature point set comprising the two-dimensional coordinates of facial feature points;

Reconstruction module 902 is used to carry out three-dimensional reconstruction to the target object based on the first feature point set, to obtain the facial posture of the target object and the three-dimensional coordinates of the facial feature points of the target object;

Projection module 903 is used to project the three-dimensional coordinates of the facial feature points, and obtains the second feature point set comprising the two-dimensional coordinates of the facial feature points;

A comparison module 904, configured to compare the second feature point set with the first feature point set, when the error between the first feature point set and the second feature point set does not satisfy an error condition , using the face pose obtained by the three-dimensional reconstruction as the target face pose of the target object in the target video.

In some embodiments, the detection module 901 is also used to intercept the face region of the target object and obtain multiple face region images from the target video including multiple video frames of the target object;

Perform face feature point detection on each face region image respectively, and obtain the two-dimensional coordinates of the face feature points in each face region image;

Perform coordinate transformation on the two-dimensional coordinates of the face feature points in the face area image to obtain the two-dimensional coordinates of the face feature points in each video frame;

The two-dimensional coordinates of the face feature points in each video frame form a first feature point set.

In some embodiments, the detection module 901 is also used to carry out face re-identification to the target video to obtain the face track of the target object;

The described detection of facial feature points on multiple video frames containing the target object in the target video, including:

According to the face track of the target object, it is determined that the target video includes a plurality of video frames of the target object and the face position of the target object in the video frame;

Perform face feature point detection on the face located at the face position in the video frame.

In some embodiments, the detection module 901 is further configured to perform face re-identification on the target video, determine the video frame in which the target object is recognized as a video frame containing the target object, and determine the target object in the video frame face position;

For the first video frame and the second video frame containing the target object, when the target object is not identified in at least one third video frame between the first video frame and the second video frame, and the third video frame When the quantity is less than the first quantity threshold, obtain the first face position of the target object in the first video frame and the second face position of the target object in the second video frame;

When the distance between the first face position and the second face position is less than a distance threshold, determine that the third video frame contains a target object, and

According to the position of the first face and the position of the second face, interpolation processing is performed to obtain the face position of the target object in at least one of the third video frames, so as to generate the face track of the target object.

In some embodiments, the detection module 901 is further configured to perform face re-recognition on multiple video frames in the target video, determine the video frames that recognize the target object as video frames that do not contain the target object, and determine the The face position of the target object in the video frame;

For the first video frame and the second video frame containing the target object, when the target object is not identified in at least one third video frame between the first video frame and the second video frame, and the third video frame When the number reaches the second number threshold, based on the first face position of the target object in the first video frame and the second face position of the target object in the second video frame, generate at least two paragraphs of the target object. Two face trajectories.

In some embodiments, the reconstruction module 902 is further configured to perform face pose estimation based on the first feature point set to obtain the face pose of the target object;

Based on the facial posture of the target object, three-dimensional reconstruction is performed on the target object through triangulation processing, and the three-dimensional coordinates of the facial feature points of the target object are determined.

In some embodiments, the reconstruction module 902 is further configured to obtain a fundamental matrix based on the positional relationship of each face feature point in the first feature point set in a plurality of the video frames;

Normalize the fundamental matrix to obtain an essential matrix;

Perform singular value decomposition on the essential matrix to obtain the face pose of the target object.

In some embodiments, the reconstruction module 902 is further configured to obtain the target in-camera parameters of the target video;

Based on the internal parameters of the target camera and the first feature point set, three-dimensional reconstruction is performed on the target object;

In the process of carrying out three-dimensional reconstruction to the target object, obtain the camera internal parameters of the target video;

When the error between the first feature point set and the second feature point set satisfies the error condition, based on the acquired in-camera parameters, the target in-camera parameters are updated to

Based on the updated internal parameters of the target camera and the first feature point set, three-dimensional reconstruction of the target object is performed.

In some embodiments, the reconstruction module 902 is further configured to obtain a fundamental matrix and an essential matrix in the process of three-dimensional reconstruction of the target object;

According to the conversion relationship between the fundamental matrix and the essential matrix, the in-camera parameters of the target video are acquired.

In some embodiments, the reconstruction module 902 is further configured to, when the error between the first feature point set and the second feature point set does not satisfy an error condition, and the face in each of the video frames When the number of feature points is multiple, for each target face feature point in the multiple face feature points, the two-dimensional coordinates of at least one face feature point are removed from the first feature point set to obtain a third feature point. set;

Based on the third feature point set, three-dimensional reconstruction is carried out to the target object, and the three-dimensional coordinates of the facial feature points of the target object are obtained;

Projecting the three-dimensional coordinates obtained by performing the three-dimensional reconstruction based on the third feature point set to the imaging plane of the camera to obtain a fourth feature point set corresponding to the third feature point set and including the two-dimensional coordinates of the face feature points;

The first feature point set is updated based on the third feature point set and the fourth feature point set.

In some embodiments, the projection module 903 is further configured to rotate and translate the three-dimensional coordinates of the facial feature points on the imaging plane of the camera according to the facial posture, to obtain the facial feature points on the camera. the coordinates in the coordinate system;

Obtain the target camera internal parameters corresponding to the target video;

According to the internal parameters of the target camera, the coordinates under the camera coordinate system are converted to the coordinates under the image coordinate system to obtain a second feature point set including the two-dimensional coordinates of the facial feature points.

In some embodiments, the apparatus further comprises: a processing module for generating a picture corresponding to the information to be displayed based on the information to be displayed;

Obtain the original two-dimensional coordinates of the multiple vertices of the picture and the three-dimensional coordinates of the multiple vertices of the picture;

Based on the gesture of the target face, the three-dimensional coordinates of the multiple vertices of the picture are projected onto the camera imaging surface to obtain the two-dimensional coordinates of the multiple vertices of the picture;

Determine the perspective transformation matrix according to the mapping relationship between the original two-dimensional coordinates and the two-dimensional coordinates obtained by projection;

Through the perspective transformation matrix, affine transformation is performed on the picture to obtain the transformed target picture;

The target picture is superimposed on the video frame to obtain the video frame to which the information to be displayed is added.

By applying the above embodiment, the second feature point set is obtained by projection, and the second feature point set is compared with the first feature point set, so as to realize the verification of the three-dimensional reconstruction result and improve the stability of the determination of the target face pose and accuracy.

An embodiment of the present application also provides a computer device, which can be a terminal or a server. Referring to FIG. 10, FIG. 10 is a schematic structural diagram of the computer device provided by the embodiment of the present application. The computer device provided by the embodiment of the present application includes:

memory 550 for storing executable instructions;

The processor 510 is configured to implement the information display method provided by the embodiments of the present application when executing the executable instructions stored in the memory.

Here, the processor 510 may be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gate or transistor logic devices, Discrete hardware components, etc., where a general purpose processor may be a microprocessor or any conventional processor or the like.

Memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard drives, optical drives, and the like. Memory 450 optionally includes one or more storage devices that are physically remote from processor 510.

Memory 550 includes volatile memory or non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be Read Only Memory (ROM, Read Only Memory), and the volatile memory may be Random Access Memory (RAM, Random Access Memory). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

At least one network interface 520 and user interface 530 may also be included in some embodiments. The various components in computer device 500 are coupled together by bus system 540. It can be understood that the bus system 540 is used to implement the connection communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. For clarity, however, the various buses are labeled as bus system 540 in Figure 10.

Embodiments of the present application provide a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above-mentioned method for processing the facial gesture in the video in the embodiment of the present application.

The embodiments of the present application provide a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored, and when the executable instructions are executed by a processor, the processor will cause the processor to execute the human in the video provided by the embodiments of the present application. How to deal with face poses.

In some embodiments, the computer-readable storage medium may be memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; it may also include one or any combination of the foregoing memories of various equipment.

In some embodiments, executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and which Deployment may be in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

As an example, executable instructions may, but do not necessarily correspond to files in a file system, may be stored as part of a file that holds other programs or data, eg, a Hyper Text Markup Language (H TML) document One or more scripts in , stored in a single file dedicated to the program in question, or in multiple cooperating files (eg, files that store one or more modules, subroutines, or code sections).

As an example, executable instructions may be deployed to be executed on one computer device, or on multiple computer devices located at one site, or alternatively, distributed across multiple sites and interconnected by a communication network execute on.

The above descriptions are merely examples of the present application, and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements and improvements made within the spirit and scope of this application are all included within the protection scope of this application.

Claims (15)

1. a processing method of human face posture in a video, is characterized in that, comprises: Performing facial feature point detection on multiple video frames containing the target object in the target video to obtain a first feature point set including the two-dimensional coordinates of the facial feature points; Based on the first feature point set, three-dimensional reconstruction is performed on the target object to obtain the face pose of the target object and the three-dimensional coordinates of the face feature points of the target object; Projecting the three-dimensional coordinates of the face feature points to obtain a second feature point set including the two-dimensional coordinates of the face feature points; Comparing the second feature point set with the first feature point set, when the error between the first feature point set and the second feature point set satisfies the error condition, obtain the three-dimensional reconstruction The face pose of the target video is taken as the target face pose of the target object in the target video. 2. method as claimed in claim 1 is characterized in that, described in the target video, comprises the multiple video frame of target object and carries out facial feature point detection, obtains the first feature that comprises the two-dimensional coordinates of facial feature point point set, including: From multiple video frames containing the target object in the target video, intercept the face region of the target object to obtain multiple face region images; Perform face feature point detection on each face region image respectively, and obtain the two-dimensional coordinates of the face feature points in each face region image; Perform coordinate transformation on the two-dimensional coordinates of the face feature points in the face area image to obtain the two-dimensional coordinates of the face feature points in each video frame; The two-dimensional coordinates of the face feature points in each video frame form a first feature point set. 3. The method according to claim 1, wherein before the multiple video frames of the target video including the target object are subjected to facial feature point detection, the method further comprises: Perform face re-identification on the target video to obtain the face trajectory of the target object; The described detection of facial feature points on multiple video frames containing the target object in the target video, including: According to the face trajectory of the target object, determine a plurality of video frames including the target object in the target video and the face position of the target object in the video frame; Perform face feature point detection on the face located at the face position in the video frame. 4. method as claimed in claim 3, is characterized in that, described carrying out face re-identification to described target video, obtains the face track of target object, comprising: Carry out face re-identification to the target video, identify the video frame of the target object as a video frame containing the target object, and determine the face position of the target object in the video frame; For the first video frame and the second video frame containing the target object, when the target object is not identified in at least one third video frame between the first video frame and the second video frame, and the third video frame When the quantity is less than the first quantity threshold, obtain the first face position of the target object in the first video frame and the second face position of the target object in the second video frame; When the distance between the first face position and the second face position is less than a distance threshold, determine that the third video frame contains a target object, and According to the first face position and the second face position, interpolation processing is performed to obtain at least one face position of the target object in the third video frame, so as to generate a face trajectory of the target object. 5. The method of claim 3, wherein the described target video is subjected to face re-identification to obtain the face track of the target object, comprising: Carrying out face re-identification to multiple video frames in the target video, identifying the video frame of the target object as a video frame containing the target object, and determining the face position of the target object in the video frame; For the first video frame and the second video frame containing the target object, when the target object is not identified in at least one third video frame between the first video frame and the second video frame, and the third video frame When the number reaches the second number threshold, based on the first face position of the target object in the first video frame and the second face position of the target object in the second video frame, generate at least two segments of the target object. face trajectory. 6 . The method according to claim 1 , wherein, based on the first feature point set, three-dimensional reconstruction is performed on the target object to obtain the face pose of the target object and the target object. 7 . The three-dimensional coordinates of the facial feature points, including: Perform face pose estimation based on the first feature point set to obtain the face pose of the target object; Based on the facial posture of the target object, three-dimensional reconstruction of the target object is performed through triangulation processing, and the three-dimensional coordinates of the facial feature points of the target object are determined. 7. The method according to claim 6, wherein, performing face pose estimation based on the first feature point set to obtain the face pose of the target object, comprising: Obtain a fundamental matrix based on the positional relationship of each facial feature point in the first feature point set in the plurality of video frames; Normalize the fundamental matrix to obtain an essential matrix; Perform singular value decomposition on the essential matrix to obtain the face pose of the target object. 8. The method of claim 1, wherein the performing three-dimensional reconstruction on the target object based on the first feature point set comprises: Get the in-camera parameters of the target video; performing three-dimensional reconstruction on the target object based on the target camera internal parameters and the first feature point set; The method also includes: In the process of performing 3D reconstruction on the target object, acquiring in-camera parameters of the target video; When the error between the first feature point set and the second feature point set does not satisfy the error condition, based on the acquired in-camera parameters, the target in-camera parameters are updated to Based on the updated internal parameters of the target camera and the first feature point set, three-dimensional reconstruction of the target object is performed. 9. The method according to claim 8, wherein, in the process of performing 3D reconstruction on the target object, acquiring in-camera parameters of the target video comprises: In the process of performing 3D reconstruction on the target object, acquiring a fundamental matrix and an essential matrix; According to the conversion relationship between the fundamental matrix and the essential matrix, the in-camera parameters of the target video are acquired. 10. The method of claim 1, further comprising: When the error between the first feature point set and the second feature point set does not satisfy the error condition, and the number of face feature points in each of the video frames is multiple, the first feature Eliminate the two-dimensional coordinates of at least one face feature point from the point set to obtain a third feature point set; Based on the third feature point set, three-dimensional reconstruction is performed on the target object to obtain the three-dimensional coordinates of the facial feature points of the target object; Projecting the three-dimensional coordinates obtained by performing three-dimensional reconstruction based on the third feature point set on the imaging plane of the camera to obtain a fourth feature point set including the two-dimensional coordinates of the face feature points; The first feature point set is updated based on the third feature point set and the fourth feature point set. 11. The method according to claim 1, wherein the three-dimensional coordinates of the face feature points are projected to obtain a second feature point set comprising the two-dimensional coordinates of the face feature points, comprising: According to the facial posture, the three-dimensional coordinates of the facial feature points are rotated and translated with respect to the imaging plane of the camera to obtain the coordinates of the facial feature points in the camera coordinate system; Obtain the target camera internal parameters corresponding to the target video; According to the internal parameters of the target camera, the coordinates in the camera coordinate system are converted to the coordinates in the image coordinate system to obtain a second feature point set including the two-dimensional coordinates of the facial feature points. 12. The method of claim 1, further comprising: generating a picture corresponding to the information to be displayed based on the information to be displayed; obtaining the original two-dimensional coordinates of the multiple vertices of the picture and the three-dimensional coordinates of the multiple vertices of the image; Based on the target face posture, project the three-dimensional coordinates of the multiple vertices of the picture onto the imaging surface of the camera to obtain the two-dimensional coordinates of the multiple vertices of the picture; Determine a perspective transformation matrix according to the mapping relationship between the original two-dimensional coordinates and the two-dimensional coordinates obtained by projection; Through the perspective transformation matrix, affine transformation is performed on the picture to obtain a transformed target picture; The target picture is superimposed on the video frame to obtain the video frame to which the information to be displayed is added. 13. A processing device for facial gestures in a video, comprising: a detection module, configured to perform facial feature point detection on multiple video frames including the target object in the target video, and obtain a first feature point set including the two-dimensional coordinates of the facial feature points; a reconstruction module, configured to perform three-dimensional reconstruction on the target object based on the first feature point set, to obtain the face pose of the target object and the three-dimensional coordinates of the face feature points of the target object; a projection module for projecting the three-dimensional coordinates of the facial feature points to obtain a second feature point set including the two-dimensional coordinates of the facial feature points; The comparison module is used to compare the second feature point set with the first feature point set, and when the error between the first feature point set and the second feature point set satisfies the error condition, the The face pose obtained by the three-dimensional reconstruction is used as the target face pose of the target object in the target video. 14. A computer equipment, characterized in that, comprising: memory for storing executable instructions; The processor is configured to implement the method for processing a face gesture in a video according to any one of claims 1 to 12 when executing the executable instructions stored in the memory. 15 . A computer-readable storage medium, characterized in that it stores executable instructions for, when executed by a processor, implementing the method for processing a face gesture in a video according to any one of claims 1 to 12 .

Images