CN110874575A - A face image processing method and related equipment - Google Patents

Abstract

The application provides a face image processing method and related equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a full-face feature point set of a face image to be processed; adjusting corresponding feature points in the full-face feature point set according to the face feature adjustment information to obtain an adjusted full-face feature point set; inputting the adjusted full-face feature point set and the face image to be processed into the trained image completion model, and obtaining the full-face image in which the undisplayed face region in the face image to be processed is subjected to completion processing. The method completes the face image to be processed based on the full-face feature point set, realizes more accurate face completion, and realizes more abundant and accurate face image completion by accurately adjusting the full-face feature point set.

Description

A face image processing method and related equipment

technical field

The present application relates to the technical field of artificial intelligence, and in particular, to a facial image processing method and related equipment.

Background technique

With the continuous development of science and technology, image completion technology is very popular. The face image completion technology refers to the completion of the face image containing the incomplete area to obtain a complete face image. Usually, the feature map in the face image containing the incomplete area is extracted, and then the incomplete face image is predicted according to the feature map in the face image containing the incomplete area, and finally the completed face image is obtained. The current face completion method usually extracts the hierarchical feature image of the incomplete face image, and expresses and completes the incomplete face image nonlinearly.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a facial image processing method and related equipment, which are used to provide a facial image processing method based on feature points, which can adjust facial features as needed to obtain a corresponding full-face image.

In a first aspect, a method for processing a face image is provided, including:

obtaining a face image to be processed, where the face image to be processed includes a part of the face that is not displayed;

Inputting the facial image to be processed into a trained facial feature point prediction model to obtain a full-face feature point set of the facial image to be processed;

According to the facial feature adjustment information, adjusting the corresponding feature points in the full-face feature point set to obtain an adjusted full-face feature point set;

Input the adjusted full-face feature point set and the to-be-processed face image into the trained image completion model to obtain a full-face that completes the undisplayed area of the face in the to-be-processed face image image.

In a second aspect, a method for processing a face image is provided, including:

In response to the facial image completion input operation on the facial image completion interface, a completion request is sent to the server, so that the server can respond to the facial image and face to be processed according to the method according to any one of the first aspects. The facial feature adjustment information is processed to obtain a full-face image after the facial image is completed; wherein, the completion request includes the to-be-processed facial image and the facial feature adjustment information;

Acquire and display the full-face image returned by the server.

In a third aspect, a facial image processing apparatus is provided, including:

a transceiver module, configured to obtain a face image to be processed, where the face image to be processed includes a part of the face that is not displayed;

The processing module is used for inputting the face image to be processed into the trained facial feature point prediction model to obtain the full face feature point set of the face image to be processed; The corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set; The model obtains a full-face image obtained by performing complement processing on the undisplayed area of the face in the to-be-processed face image.

In a possible embodiment, the facial feature point prediction model includes a first encoding module, a plurality of second encoding modules and a fully connected module, and the processing module is specifically used for:

Through the first encoding module, feature extraction is performed on the face image to be processed under various scales to obtain a first feature atlas;

Perform convolution pooling processing on the first feature map set through each coding module in the plurality of second coding modules respectively to obtain a second feature value set;

Through the full connection module, a plurality of second feature value sets are spliced to obtain a full face feature point set.

In a possible embodiment, the facial feature adjustment information is an expression classification label, and the processing module is specifically configured to:

Inputting the facial expression classification label and the full-face feature point set into the face adjustment model to obtain the adjusted full-face feature point set;

The face adjustment model is based on a sample facial expression image set, and each sample facial expression image in the sample facial expression image set is marked with a corresponding expression classification label.

In a possible embodiment, the processing module is specifically used for:

Extracting the facial contour feature corresponding to the facial feature adjustment information; wherein, the facial contour feature is used to represent the contour formed by the facial key points;

According to the facial contour feature, the coordinates of the corresponding feature points in the full-face feature point set are adjusted to obtain an adjusted full-face feature point set.

In a possible embodiment, the image completion model includes a third encoding module, a hole convolution module and a decoding module, and the processing module is specifically used for:

converting the adjusted full-face feature point set into a facial feature point map;

Through the third encoding module, the face feature point map and the face image to be processed are subjected to convolution processing at different scales to obtain a third feature atlas; wherein, the third feature atlas Including a feature map set after convolution processing, the third encoding module of the feature map set after convolution processing is to perform convolution processing at different scales on the facial feature point map and the face image to be processed In the process, the final convolution feature map of the output;

Through the residual unit in the hole convolution module, perform hole convolution on the feature atlas after the convolution processing to obtain a fourth feature atlas;

Extracting the fourth feature atlas and the local features of the feature atlas after the convolution processing through the attention unit in the hole convolution module to obtain the fifth feature atlas;

Perform upsampling processing on the fifth feature atlas and the third feature atlas by the decoding module, so as to obtain a full face obtained by performing complement processing on the undisplayed area of the face in the face image to be processed image.

In a possible embodiment, the third feature atlas further includes a first intermediate feature atlas and a second intermediate feature atlas, the first intermediate feature atlas is a map of the facial feature points, and the intermediate convolution results obtained by sequentially performing preset sub-convolution processing on the face images to be processed, and the second intermediate feature atlas is obtained by performing preset sub-convolution processing on the first intermediate feature atlas. Intermediate convolution results, the processing module is specifically used for:

Perform up-sampling processing on the fifth feature atlas by the decoding module to obtain a sixth feature atlas;

Perform weighting processing on the second intermediate feature atlas and the sixth feature atlas by the decoding module to obtain a seventh feature atlas;

Perform weighting and upsampling processing on the seventh feature atlas and the first intermediate feature atlas by the decoding module to obtain an eighth feature atlas;

Perform convolution processing on the eighth feature atlas by the decoding module to obtain a full-face image obtained by performing complement processing on the undisplayed area of the face in the face image to be processed.

In a possible embodiment, the facial feature point prediction model is obtained by training through the following steps:

Obtain a first sample data set; wherein, the first sample data set includes the sample face image set to be processed, and the true-value sample full-face feature point set corresponding to each sample face image to be processed ;

Based on the first sample data set, train a facial feature point prediction model;

Until the prediction loss of the facial feature point prediction model satisfies the first preset condition, the trained facial feature point prediction model is obtained; wherein, the prediction loss is used to represent the full face feature point set of the predicted sample and the full face of the true value sample loss between feature point sets.

In a possible embodiment, the image completion model includes a generative sub-model and a discriminative sub-model, and the loss of the image completion model is weighted according to pixel-wise loss, perceptual loss, style loss, total differential loss and adversarial loss owned;

Wherein, the pixel-by-pixel loss is used to represent the pixel value error of the sample full-face image after completion and the sample face image to be processed; the perceptual loss is used to represent the feature map and sample of the sample full-face image after completion The error between the feature maps of the to-be-processed face image; the style loss is used to represent the completed sample full-face image plus the occluded area image, and the sample to-be-processed face image plus the occlusion area. The information of each channel of the feature map is the sum of the Gramm matrix differences between the channels after quantization; the full differential loss is used to represent the sum of the horizontal and vertical derivatives of the completed sample full face image and the sample face image to be processed. The ratio between the total number of pixel points; the generation loss is used to represent the loss of the discriminant sub-model for the source of the generation result of the generation sub-model and the loss of the real source of the generation result.

In a second aspect, a method for processing a face image is provided, including:

In response to the facial image completion input operation on the facial image completion interface, a completion request is sent to the server, so that the server can respond to the facial image and face to be processed according to the method according to any one of the first aspects. The facial feature adjustment information is processed to obtain a full-face image after the facial image is completed; wherein, the completion request includes the to-be-processed facial image and the facial feature adjustment information;

Acquire and display the full-face image returned by the server.

In a third aspect, a facial image processing apparatus is provided, including:

a transceiver module, configured to obtain a face image to be processed, where the face image to be processed includes a part of the face that is not displayed;

The processing module is used for inputting the face image to be processed into the trained facial feature point prediction model to obtain the full face feature point set of the face image to be processed; The corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set; The model obtains a full-face image obtained by performing complement processing on the undisplayed area of the face in the to-be-processed face image.

In a fourth aspect, a terminal device is provided, including:

A sending module, configured to send a completion request to the server in response to a face image completion input operation on the face image completion interface, so that the server is to be processed according to the method described in any one of the first aspects The facial image and the facial feature adjustment information are processed to obtain a full-face image after the facial image is completed; wherein, the completion request includes the to-be-processed facial image and the facial feature adjustment information; and obtaining the full-face image returned by the server;

The display module is used for displaying the full face image.

In a fifth aspect, a facial image processing device is provided, including:

at least one processor, and

a memory communicatively coupled to the at least one processor;

Wherein, the memory stores instructions executable by the at least one processor, and the at least one processor implements the method according to any one of the first aspect or the second aspect by executing the instructions stored in the memory .

In a sixth aspect, a computer-readable storage medium is provided, the computer-readable storage medium stores computer instructions that, when the computer instructions are executed on a computer, cause the computer to perform any one of the first aspect or the second aspect. method described in item.

Because the embodiment of the present application adopts the above technical solution, it has at least the following technical effects:

In the embodiment of the present application, by obtaining the full-face feature point set of the face image to be processed, a face completion method based on feature points is implemented, and based on the accurate modification of the full-face feature point set, so as to obtain a more accurate adjustment Full face image after. Moreover, since the user does not need to manually adjust the full-face feature point set, the efficiency of adjusting the full-face feature point set can be relatively improved, thereby improving the efficiency of facial image processing. Further, different facial feature adjustment information can be set as required, and then different full-face images can be generated to meet the individual needs of different users and improve the fun of the user's face completion. By adjusting the facial feature points, the pose, expression and facial contour of the image can be modified, and an adjustable image completion scheme is realized.

Description of drawings

FIG. 1 is a schematic structural diagram of a facial image processing device according to an embodiment of the present application;

FIG. 2 is a schematic diagram of an application scenario of a face image processing method provided by an embodiment of the present application;

3 is a schematic diagram of the principle of a facial image processing method provided by an embodiment of the present application;

4 is a schematic diagram of the distribution of each model provided by the embodiment of the present application;

FIG. 5 is a sample face image to be processed generated by two occlusion methods provided by the embodiment of the present application;

6 is an example diagram of a process for adjusting a full-face feature point set provided by an embodiment of the present application;

7 is a schematic diagram of a process for adjusting a full-face feature point set provided by an embodiment of the present application;

FIG. 8 is an interactive schematic diagram of a facial image processing method provided by an embodiment of the present application;

9 is a schematic diagram of a face image completion interface provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of a process of adding occlusion to an original image to generate an image to be processed according to an embodiment of the present application;

FIG. 11 is an example diagram 1 of a process of obtaining a full-face image by complementing provided by an embodiment of the application;

FIG. 12 is an example FIG. 2 in the process of obtaining a full-face image by complementing provided by an embodiment of the present application;

FIG. 13 is an example FIG. 3 in the process of obtaining a full-face image by complementing provided by an embodiment of the present application;

FIG. 14 is an example FIG. 4 in the process of obtaining a full-face image by complementing provided by an embodiment of the present application;

FIG. 15 is an example FIG. 5 in the process of obtaining a full-face image by complementing provided by an embodiment of the present application;

FIG. 16 is an example FIG. 6 in the process of obtaining a full-face image by complementing provided by an embodiment of the present application;

17 is a schematic diagram 1 of an interface for displaying a full-face image provided by an embodiment of the present application;

18 is a schematic diagram 2 of an interface for displaying a full-face image provided by an embodiment of the present application;

FIG. 19 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the application;

FIG. 20 is a schematic structural diagram 1 of a terminal device according to an embodiment of the present application;

FIG. 21 is a second schematic structural diagram of a terminal device according to an embodiment of the application.

Detailed ways

In order to better understand the technical solutions provided by the embodiments of the present application, detailed descriptions will be given below with reference to the accompanying drawings and specific implementation manners.

In order to facilitate those skilled in the art to better understand the technical solutions of the present application, the technical terms involved in the present application are introduced below.

Artificial Intelligence (AI): It is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning.

Computer Vision (CV): Computer vision is a science that studies how to make machines "see". Further, it refers to the use of cameras and computers instead of human eyes to identify, track and measure objects. , and further do graphic processing, so that the computer processing becomes a more suitable image for human eye observation or transmission to the instrument for detection. As a scientific discipline, computer vision studies related theories and technologies, trying to build artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping It also includes common biometric identification technologies such as face recognition and fingerprint recognition.

Face image completion: Image completion refers to the completion of incomplete face images and editing of face attributes. The generated full-face image can be as accurate as the original full-face image, or it can be matched with the complete face image. Consistent in content, so that the completed full-face image looks realistic. Editing facial attributes, for example, edits one or more of facial expressions, facial postures, and facial contours. Facial image completion can be applied to the user's daily photo processing, as well as to the security field.

To-be-processed face image: means that the face image includes a part of the face area that is not displayed, for example, the face has partial defects, or the face has occlusions, and so on. The face referred to in this application includes, but is not limited to, a human face or an animal face.

Facial feature adjustment information: the information used to adjust the coordinates of each feature point in the full-face feature point set, which can be used for the outline, expression, and posture of the face. The format of the facial feature adjustment information may be a coordinate point, an image, a sketch, and the like.

It should be noted that, in this application, a face refers to a human face or an animal face. This application mainly takes a human face as an example for detailed description, and the animal face can also be complemented according to the same principle.

design thinking

In various situations, such as poor lighting conditions in the shooting environment, or the image is damaged by man or nature, the face image may be incomplete. For example, when a camera shoots a criminal at night, a complete face image is not captured. At this time, a face completion method can be used to process the face image.

At present, the facial image method generally expresses and complements the facial image in a non-linear manner, and cannot achieve accurate complementing and accurate modification of the facial image to be processed.

In view of this, the inventor of the present application provides a facial image processing method, which first predicts the full-face feature point set of the face image to be processed, adjusts the full-face feature point set according to the facial feature adjustment information, and obtains the adjusted facial feature point set. The full-face feature point set is based on the adjusted full-face feature point set and the face image to be processed to complete the face image to be processed. Because the whole-face feature point set is extracted in the embodiment of the present application, that is to say, the information for accurately quantifying the face is obtained, which is beneficial to accurately complete the face image, and facilitates the more accurate analysis of the full-face feature point set. After modifying the full-face feature point set, the completed full-face image is also different, which enriches the completed full-face image and meets the different needs of different users for full-face completion.

The inventor of the present application further considers that if the full-face feature point set is adjusted manually, the adjustment efficiency is low, and the accuracy of the full-face image generated by the adjusted full-face feature point set cannot be guaranteed. Therefore, the inventor of the present application considers introducing a face adjustment model, and the user only needs to select the classification label of the corresponding expression, and the face adjustment model can accurately adjust the feature point set of the whole face.

Application Scenario Example

Please refer to FIG. 1 , which shows a schematic structural diagram of a facial image processing device that executes the facial image processing method in the embodiment of the present application. The facial image processing device 100 includes one or more input devices 101 and one or more processors. 102 , one or more memories 103 and one or more output devices 104 .

The input device 101 is used to provide an input interface to obtain the facial image to be processed and the like input by the external device/user. After obtaining the input facial image to be processed, the input device 101 sends the facial image to be processed to the processor 102, and the processor 102 uses the program instructions stored in the memory 103 to implement the completion process of the facial image to be processed, obtaining Completed full face image. The full face image is output through the output device 104 .

The input device 101 may include, but is not limited to, one or more of a physical keyboard, a function key, a trackball, a mouse, a touch screen, an operation stick, and the like. The processor 102 may be a central processing unit (central processing unit, CPU), or a digital processing unit or the like. The memory 103 may be a volatile memory (volatile memory), such as random-access memory (RAM); the memory 103 may also be a non-volatile memory (non-volatile memory), such as read-only memory, flash memory Flash memory, hard disk drive (HDD) or solid-state drive (SSD), or memory 103 is capable of carrying or storing desired program code in the form of instructions or data structures and capable of being used by a computer Access any other medium without limitation. The memory 103 may be a combination of the above-mentioned memories. Output devices 104 are, for example, displays, speakers, and printers, among others.

In a possible embodiment, the facial image processing device 100 may be a client device or a server device. The client device may be a mobile terminal, a stationary terminal or a portable terminal, such as a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, Tablet Computers, Personal Communication System (PCS) Devices, Personal Navigation Devices, Personal Digital Assistants (PDAs), Audio/Video Players, Digital Cameras/Camcorders, Positioning Devices, TV Receivers, Radio Broadcast Receivers, E-book Devices, Games Devices or any combination thereof, including accessories and peripherals for such devices or any combination thereof. It is also envisioned that the facial image processing device 100 can support any type of user-oriented interface (eg, wearable device) and the like. The server device may be a server for various services, a large computing device, and the like. A server can be one or more servers. The server can also be a physical server or a virtual server.

In a possible application scenario, please refer to FIG. 2 , which shows an example of an application scenario. The face image processing device 100 is implemented by the server 220 . The user determines the facial image to be processed to be processed through the client 211 in the terminal device 210, the client 211 generates a completion request, and the server 220 performs the above processing on the facial image to be processed to obtain a full-face image, and returns the full-face image to client 211. The client 211 generally refers to the client 211 that can implement face completion. The client 211 is, for example, a photographing app, an image processing app, etc. The application does not limit the specific type of the client 211 .

working principle

Please refer to FIG. 3 , which shows a schematic diagram of the principle of processing a face image. The image processing process mainly includes three parts: predicting the full-face feature point set part 310, adjusting the full-face feature point set part 320, and complementing the face image to be processed. All part 330.

Specifically, the face image to be processed is I, and the undisplayed area of the face image to be processed can be expressed as M, so the face image to be processed can be expressed as:

where "." represents the Hadamard product of the matrix.

最终获得的全脸图像

可以表示如下：The facial image processing device 100 predicts the full-face feature point set L, adjusts the full-face feature point set L according to the facial feature adjustment information, obtains L', and then inputs L' and the face image I to be processed into the image complement. full model to obtain a full face image

The final full face image

It can be expressed as follows:

表示对未显示区域的补全区域。in,

Indicates the completion area for the undisplayed area.

Next, the prediction full-face feature point set section 310 will be described.

The facial image processing device 100 may obtain a full-face feature point set of the facial image to be processed through the facial feature point prediction model.

In a possible embodiment, please refer to FIG. 4 , the facial feature point prediction model 410 may include a first encoding module 411, a plurality of second encoding modules 412 and a fully connected module 413, wherein:

The first encoding module 411 includes a variety of convolutional layers of different scales. After the image passes through each layer in the first encoding module 411 in sequence, each convolutional layer can obtain corresponding feature maps respectively, thereby obtaining the first feature map set. , that is, the first feature map set includes a plurality of feature maps. It should be noted that the feature map set in this application may include one feature map or multiple feature maps unless it is explicitly defined.

After the first feature atlas is obtained, three second encoding modules 412 are taken as an example in FIG. 4 , the number of which is not actually limited, and each second encoding module 412 includes one or more convolutional layers and one or more convolutional layers. For multiple pooling layers, different second encoding modules 412 can perform different convolution pooling processing on the first feature map set, and each second encoding module 412 outputs a corresponding feature value set, thereby obtaining a second feature value set. Convolution pooling can be performed by convolution first, followed by average pooling on the convolution results, or by convolution first, followed by max pooling on the convolution results.

As an embodiment, there are two encoding modules of different scales in the plurality of second encoding models 412 .

In this embodiment of the present application, a plurality of second encoding modules 412 respectively perform convolution and pooling processing on the first group of feature maps, and different features of the first set of feature maps can be retained in the processing processes of different scales, so that subsequent feature point predictions can be made. more precise.

The second feature value set is spliced through the full connection module 413 to obtain a full face feature point set. The full-face feature point set includes a set of coordinates for key points representing the face. The splicing process can be understood as combining corresponding values in the second feature value set into coordinates according to a preset order, so as to obtain a corresponding full-face feature point set.

The structure of the facial feature point prediction model 410 will be described with a specific example below. A facial feature point prediction model 410 is specifically shown in Table 1 below:

Table 1

In Table 1, c represents the number of output channels, s represents the stride of the convolutional layer, t represents the expansion factor, and n represents the number of repetitions of the corresponding layer.

Among them, in order to facilitate the understanding of the bottleneck layer (bottleneck), the structure of the bottleneck layer involved in Table 1 is exemplified below. The structure of the bottleneck layer with s=2 in Table 1 is that the number of channels is (1x1 of tk) convolution-ReLU6- (the number of channels is tk, the stride is 2, the convolution kernel size is 3x3) deep convolution (each volume The product kernel only convolves one feature map)-ReLU6-(1x1 with k channels) convolution. For the bottleneck layer with s=1, its structure is: (1x1 with channel number tk) convolution-ReLU6- (channel number is tk, stride is 2, convolution kernel size is 3x3) deep convolution-ReLU6-( 1x1) convolution with the number of channels k, and the output result is added to the input of the bottleneck layer to realize the residual connection.

Please refer to Table 1, the first encoding module 411 includes a convolution layer and a plurality of bottleneck layers (1x1 convolution-ReLU6-stride is 2, 3x3 channel-by-channel convolution (Depthwise Convolution)), to obtain the first feature map Set C1, and then pass the first feature atlas C1 through the second coding unit 412. As shown in Table 1, the second coding unit includes a first convolutional layer with a convolution kernel size of 1×1 and a channel number of 1280, and a first convolutional layer with a convolution kernel size of 1×1 Average pooling layer, the second convolutional layer with 1x1 convolution kernel and 128 channels, the second average pooling layer, the third convolutional layer, and the third average pooling layer. Input the first feature atlas C1 into a convolutional layer with a kernel size of 1x1 and a channel number of 1280 to obtain a feature atlas C2. Perform global average pooling on the feature atlas C2 to obtain the feature value set S1. In another branch, the first feature atlas C1 is passed through a convolutional layer with a convolution kernel of 1×1 and a channel number of 128 and global average pooling to obtain a feature value set S2. In the third branch, the feature map set C2 is passed through a convolutional layer with a convolution kernel of 1×1 and a channel number of 128, and global average pooling is performed to obtain the feature value set S3.

Input each feature value set (S1, S2 and S3) into the fully connected unit 413 (that is, the fully connected unit in Table 1), and finally connect S1, S2, S3 and pass through the fully connected layer to obtain 136 values. The 136 values correspond to the x and y coordinates of the 68 face feature points, so as to obtain the final predicted full-face feature point coordinates, that is, the full-face feature point set.

Based on the facial feature point prediction model 410 discussed above, the following describes the training process of the facial feature point prediction model 410, including acquiring the first sample data set part, training facial features based on the first sample data set Part 410 of the point prediction model, two parts are introduced separately below.

Get the first sample dataset:

Specifically, the first sample data set includes a sample face image set to be processed, a true-valued sample full-face feature point set corresponding to each sample face image to be processed, and each sample face image to be processed includes not shown The full-face feature point set of the true value sample refers to the set of real full-face feature points corresponding to the complete face image corresponding to the face image to be processed in the sample. The full face feature point set of the true value sample can be generated by manual annotation, or it can be generated by using data on network resources, or it can be generated by using other network models for annotation. For example, FAN can be used to label the face image of the sample to be processed. The ground-truth sample full-face feature point set corresponding to the corresponding full-face image.

As an example, the sample full-face image set can be input into the FAN network to obtain the ground-truth sample full-face feature point set corresponding to each sample full-face image, and then add occlusion to some or all of the images in the sample full-face image set to Get more datasets.

Specifically, there are various ways to add occlusion, such as adding a occlusion with a random position but the same size of the occlusion area, or adding an occlusion with a random mask. The occlusion of a random mask means that the position and size of the occlusion area are not fixed. The color of the added occlusion may be arbitrary, and the application does not limit the specific color of the occlusion area. But in a training process, the color of the added occlusion is the same, for example, white can be selected. For occlusions of different colors, model training can be performed separately.

For example, please refer to 5, in Fig. 5 (a) represents a sample face image to be processed after occlusion with random positions but the same occlusion area size is added, and (b) in Fig. 5 represents an occlusion with random masks added A sample face image to be processed.

Further, in order to expand the dataset, the occluded images can also be flipped, blurred, and photometric changed. The photometric processing may be, for example, multiplying all pixels of the image by a predetermined factor, and the predetermined factor is in the range of 0.7-1.3, for example.

The facial landmark prediction model 410 is trained based on the first sample data set:

The prediction loss of the facial landmark prediction model 410 can be expressed as follows:

Among them, L _gt represents the ground-truth feature point coordinates, and ‖·‖ ₂ represents the L2 norm.

Input the sample face image to be processed into the face feature point prediction model, and obtain the predicted sample full face feature point set corresponding to the sample face image to be processed.

The loss of the facial feature point prediction model 410 is determined based on the predicted sample full-face feature point set and the ground-truth sample full-face feature point set, for example, the loss can be determined according to formula (3). The parameters are continuously updated according to the loss, for example, the Adam optimizer can be used to update the parameters, wherein the initial values of the parameters in the Adam optimizer can be set as: β ₁ =0, β ₂ =0.9. The initial learning rate of the facial feature point prediction model 410 is set to 10 ⁻⁴ .

When the loss of the facial feature point prediction model 410 satisfies the first preset condition, the trained facial feature point prediction model 410 is obtained. The first preset condition, for example, the loss of the facial feature point prediction model 410 is less than or equal to a certain threshold, or the learning rate of the facial feature point prediction model 410 reaches a preset value.

In a possible embodiment, the facial feature point prediction model 410 can obtain the full face feature point set of the species, but since the distribution of facial feature points of different species may be different, the sample data corresponding to different species can be used The set goes through the above training process in turn to obtain the parameters corresponding to each species. The face image processing apparatus 100 may pre-store various species and their corresponding parameters.

The adjustment full-face feature point set section 320 will be described below.

In a possible embodiment, the facial image processing device 100 may adjust the corresponding feature points in the full-face full-face feature point set based on the facial feature adjustment information through the face adjustment model 420 to obtain the adjusted full-face feature point. face feature point set.

The face adjustment model 420 is exemplified below.

A1:

When the facial feature adjustment information is the facial expression target image, the face adjustment model 420 adopts the facial feature point prediction model 410 discussed above, and obtains the target full range corresponding to the facial expression target image through the facial feature point prediction model. face feature point set, based on the relative positions between the feature points of the target full-face feature point set to adjust the coordinate position of each feature point in the full-face feature point set of the face image to be processed before, so as to obtain the adjusted full-face feature point set. The adjustment method is, for example, adjusting the coordinate position of each feature point in the full-face feature point set of the face image to be processed.

The adjustment method in the embodiment of the present application can not only adjust the expression of the face image, but also adjust the posture and outline of the face image.

For example, please refer to FIG. 6. In FIG. 6, A represents the full-face feature point set corresponding to the facial expression target image, and B in FIG. 6 represents the full-face feature point set of the facial image to be processed. The coordinates of the full-face feature point set (for example, a1, b1, and c1 in Fig. 6) of the face image to be processed (a2, b2, and c1) are adjusted to obtain the full-face feature point set (a2, b2, and c1) of the face image to be processed. The facial feature point set, compared to the graph B, not only the expression has changed, but also the facial contour and posture have also changed correspondingly.

When the face adjustment model 420 adopts the facial feature point prediction model 410 discussed above, the training process of the face adjustment model 420 can refer to the training process of the facial feature point prediction model 410 discussed above, which will not be repeated here. .

A2:

By inputting the expression classification label and the full-face feature point set into the face adjustment model 420, the adjusted full-face feature point set is obtained.

Specifically, the face adjustment model 420 may adjust the coordinates of the feature points in the full-face feature point set according to the expression classification label or the facial expression image, so as to obtain the adjusted full-face feature point set. The structure of the face adjustment model 420 is exemplified below. The expression classification labels can be represented by means of one-hot encoding or the like.

The face adjustment model 420 may include multiple fully connected layers and multiple activation layers, and the number of fully connected layers and activation layers are the same and distributed at intervals. The fully connected layer and the activation layer perform multi-layer operations to realize the migration of the coordinates of the feature points in the full-face feature point set.

Specifically, the face adjustment model 420 includes a first fully connected layer, a first activation layer, a second fully connected layer, a second activation layer, a third fully connected layer, and a third activation layer that are connected in sequence.

The training method of the face adjustment model 420 involved in A2 will be described below.

Referring to FIG. 7 , a sample facial expression image set is obtained, and each sample facial expression image is marked with an expression classification label corresponding to the image, such as smiling, laughing, crying, and the like.

Construct the loss of face adjustment model 420, a specific example is as follows:

为脸部调整模型预测的表情分类标签所对应的全脸特征点集，‖·‖₂表示L2范数。Among them, L _gt is used to represent the ground-truth feature point set corresponding to the sample facial expression image,

The full-face feature point set corresponding to the facial expression classification labels predicted by the face adjustment model, ‖ _· ‖2 represents the L2 norm.

Until the loss of the face adjustment model 420 satisfies the preset condition, the trained face adjustment model 420 is obtained.

In the embodiment of the present application, the face adjustment model 420 pre-learns the full-face feature point sets corresponding to various expression classification labels. When using this model, the corresponding expression classification table label can be selected, and the face adjustment model 420 can Based on the expression classification label, the coordinates of the feature points in the input full-face feature point set are adjusted to obtain the trained face adjustment model 420 . Since the full-face feature point sets corresponding to various expression classification labels are trained in advance, the accuracy in the process of adjusting the full-face feature point set by the face adjustment model 420 can be relatively improved.

A3:

The facial image processing device 100 can extract the facial contour feature corresponding to the facial feature adjustment information, and adjust the coordinates of the corresponding feature points in the full-face feature point set according to the facial contour feature to obtain the adjusted full-face feature point set. .

Specifically, the facial feature adjustment information may be a stick figure drawn by the user's hand, or may be a face image with expressions, no matter which type it is, the facial image processing device 100 can extract the facial feature adjustment information The facial contour features in the face image processing device 100 can also be extracted by a pre-trained contour extraction model to obtain the facial contour features. The face contour feature is used to represent the contour formed by part or all of the key points of the face.

After the facial contour feature is obtained, the coordinates of the corresponding feature points in the full-face feature point set may be offset based on the facial contour to obtain an adjusted full-face feature point set.

In the embodiment of the present application, it is not necessary to pre-train the corresponding model, which can relatively reduce the processing amount of the facial image processing device 100, and can process sketches, etc., so as to improve the accuracy of the user's processing of images.

The complementing part 330 of the face image to be processed will be described below.

The image completion model can be used to complete the to-be-processed face image based on the adjusted full-face feature point set and the to-be-processed face image to obtain a complete image of the undisplayed face in the to-be-processed face image for completion processing. face image.

Please continue to refer to FIG. 4 , the image completion model includes a generating sub-model 430, and the generating sub-model 430 includes a third encoding module 431, a hole convolution module 432 and a decoding module 433, which combine the adjusted full-face feature point set and the face to be processed Part of the image is input into the generation sub-model 430.

Specifically, the adjusted full-face feature point set is first converted into a facial feature point map. The conversion process can be understood as representing each feature point in the form of a feature map. The corresponding position of the adjusted full-face feature point set is set to 1, and the rest are set to 0, so as to obtain the facial feature point map.

The third encoding module 431 may include multiple convolutional layers connected in sequence, and the multiple convolutional layers respectively perform convolution processing at different scales on the face image to be processed and the facial feature point map to obtain the features output by each convolutional layer. to obtain a third feature atlas. The third feature atlas includes a first intermediate feature atlas, a second intermediate feature atlas, and a feature atlas after convolution processing. The first intermediate feature map set is the feature map processed by the convolutional layer in the front part of the third coding module 431, and the second intermediate feature map set is the feature map processed by the convolutional layer in the middle part of the third coding module 431. The processed feature map is the feature map output by the last convolutional layer of the third encoding module 431 .

The feature map after convolution processing is subjected to hole convolution by the residual unit in the hole convolution module 432 to obtain a fourth feature map set, and the fourth feature map set is extracted by the attention unit in the hole convolution module 432, and the volume The local features of the processed feature map are accumulated to obtain the fifth feature map set.

The processing mechanism of the attention unit is introduced below.

Specifically, the attention unit is used to capture the association between long-distance spatial context features, and its specific calculation process is as follows:

之后获得f_d其他分量对f_dj的经过注意力加权后的特征和c_dj，其计算方法为

最终获得第四特征图集的特征部分结果为y_d＝γ_dc_d+f_d。其中γ_d为可训练的参数。For the feature f _d of the fourth feature atlas, Q(f _d ) is first obtained through a convolutional layer with a kernel size of 1x1. Then calculate the inner product s _ij of the f _d i component to the j component, s _ij =[Q(f _d ) ^T ] _i Q(f _d ) _j , and then normalize it by the softmax function to obtain the f _di to f _dj attention value

Then, the attention-weighted feature sum c _dj of other components of f _d on f _dj is obtained, and the calculation method is as follows:

Finally, the result of the feature part of the fourth feature atlas is obtained as y _d =γ _d c _d +f _d . where _γd is a trainable parameter.

For the feature f _e of the feature map after convolution processing, it uses the attention value derived from f _d to perform a weighted summation of the features to obtain c _ej , c _ej =∑ _i β _j,i f _ei . The feature result of the final output feature map set after convolution processing is y _e =γ _e (1-M)c _e + _Mfe . where γ _e is a trainable parameter.

Connecting the feature results of the fifth feature atlas and the feature results of the feature atlas after convolution processing is the fifth feature atlas.

The decoding module 433 performs up-sampling processing on the fifth feature atlas and the third feature atlas, so as to obtain a full-face image in which the undisplayed area of the face in the face image to be processed is complemented.

Specifically, the decoding module 433 may perform up-sampling on the fifth feature map set and all the feature maps in the third feature map set together to obtain a final full-face image.

The decoding module 433 may also perform up-sampling processing on the fifth feature atlas to obtain a sixth feature atlas and perform up-sampling processing to obtain a sixth feature atlas. Then, weighting and up-sampling processing are performed on the second intermediate feature atlas and the sixth feature atlas to obtain a seventh feature atlas. The seventh feature map and the first feature map set are then weighted to obtain an eighth feature map set. The eighth feature map set is then subjected to convolution processing through the decoding module to obtain a completed full-face image.

For example, a generation sub-model 430 is specifically shown in the following table:

Table 2

Among them, k, c, s, and p represent the convolution kernel size, the number of output channels, stride, and padding of the convolutional or deconvolutional layers, respectively. Except for the convolutional layer in the first row in Table 1, which uses Reflection Padding, the rest use Zero Padding. Where IN represents instance normalization (Instance Normalization).

The specific process of generating the sub-model 430 shown in Table 1 will be described below.

Specifically, the facial feature point map and the face image to be processed are first passed through a convolutional layer with a convolution kernel size of 7×7, a channel number of 3, and a filling of 3 to obtain a first intermediate feature map set E1. Then, input the first intermediate feature atlas E1 into a convolutional layer with a kernel size of 4 × 4, a number of channels of 128, a stride of 2, and a padding of 1 to obtain the second intermediate feature atlas E2. After that, E2 is passed through the convolutional layer with the convolution kernel size of 4x4, the number of channels is 256, the stride is 2, and the padding is 1 to obtain the feature atlas E3 after convolution processing.

The fourth feature map set R7 is obtained after E3 undergoes seven first convolution operations as residual blocks of hole convolution. Input R7 and E3 together into the long-term and short-term attention module to obtain the fifth feature map set, and make the fifth feature map set pass through the convolution kernel size of 4x4, the number of channels is 128, the stride is 2, and the padding is 1. Set the convolutional layer for upsampling to obtain the sixth feature atlas D1. E2 and D1 are jointly input to a 1x1 convolutional layer with 256 channels to obtain a weighted feature atlas. After that, the weighted feature map set is input to the convolution kernel with a size of 4x4, the number of channels is 256, the stride is 2, and the transposed convolution layer filled with 1 is up-sampled to obtain the seventh feature map set D2. D2 and E1 are jointly input into a 128-channel 1x1 convolutional layer to obtain a weighted eighth feature map set. The eighth feature map set is input into a convolution kernel with a size of 7, a channel number of 3, and a convolution of 3 padding, and finally the output full-face image is obtained through the tanh layer.

In this embodiment of the present application, the image completion model further includes a discriminant sub-model 440, which is used to determine whether the generation result of the generation sub-module 430 comes from the real source or the generation sub-model 420, and is used to determine whether the generation result is the same as the adjusted true value. Whether the face feature point set matches.

The discriminant sub-model 440 may adopt a simple binary classification model, or may adopt the discriminator structure of Patch-GAN, such as 70×70 Patch-GAN, and the structure of the discriminant sub-model 440 is exemplified below.

table 3

where each row represents a sequence consisting of the listed layers. K, c, s, and p denote the convolution kernel size, number of output channels, stride, and padding of the convolutional or deconvolutional layers, respectively. SN stands for Spectral Normalization, LReLU stands for Leaky ReLU, slope=0.2, relu means that x is greater than or equal to 0 and output x, and less than or equal to 0, the output slope is multiplied by x. The discriminative sub-model 440 in Table 1 includes an attention layer for adaptively processing the corresponding features.

The following is an example of the training process of the image completion model:

Obtain a second sample data set, the second sample data set includes the sample face image set to be processed, the true value sample full face feature point set corresponding to each sample face image to be processed, and the sample face image corresponding to the processing. Sample full face image. For the understanding of the full-face feature point set of the true value sample, refer to the content discussed above, which will not be repeated here.

As an embodiment, a large number of sample full-face images are obtained first, and part or all of the sample full-face images in the sample full-face image set are occluded. For the occlusion method, reference may be made to the content discussed above, which will not be repeated here.

According to the second sample data set, the image completion model is trained until the loss of the image completion model satisfies the second preset condition, and the trained image completion model is obtained.

The loss of the image completion model can be represented in various manners. In this embodiment of the present application, the loss of the image completion model is weighted according to pixel-by-pixel loss, perceptual loss, style loss, full differential loss, and generative loss.

Specifically, the pixel-by-pixel loss is used to represent the pixel value error of the sample full face image after completion and the sample face image to be processed; the perceptual loss is used to represent the feature map of the sample full face image after completion and the sample face to be processed. The error between the feature maps of the partial images; the style loss is used to represent each channel information of the feature map of the sample full face image after completion plus the occlusion area and the sample face image to be processed plus the occlusion area. The sum of the Gramm matrix differences between the vectorized channels; the total differential loss is used to represent the difference between the sum of the horizontal and vertical derivatives of the completed sample full face image and the total number of pixels included in the sample face image to be processed. Ratio; generation loss is used to represent the loss of the discriminative sub-model for the source of the generated results of the generated sub-model and the true source of the generated results.

The specific formulas of each loss are illustrated below.

A formula for calculating pixel-wise loss is as follows:

Among them, ‖· _‖1 represents the L1 norm, and N _m represents the number of occluded pixels.

A formula for calculating the perceptual loss is as follows:

用于表示补全后的样本全脸图像的特征图，φ_p(I)用于表示样本待处理脸部图像的特征图。where φ _p ( ) represents N _p feature maps of size H _p × W _p of the p-th layer extracted from a pre-trained network model (such as VGG19),

It is used to represent the feature map of the sample full face image after completion, and φ _p (I) is used to represent the feature map of the sample face image to be processed.

A formula for calculating style loss is as follows:

表示补全后的样本全脸图像加遮挡区域后的图像，

表示样本待处理脸部图像加遮挡区域后的图像。where G _p (x)=φ _p (x) ^T φ _p (x) represents the Gramm matrix with respect to φ _p (x).

Represents the image after the completed sample full face image plus the occlusion area,

Represents the image of the sample face image to be processed plus the occlusion area.

A formula for calculating the total differential loss is as follows:

表示求一阶导，包括

(水平方向)和

(竖直方向)。where NI represents the total number of pixels of _I ,

Represents the first derivative, including

(horizontal direction) and

(Vertically).

A formula for calculating the generation loss is as follows:

Among them, G represents the generation sub-model 430, D represents the discriminant sub-model 440, and the generation loss _{La adv} is the sum of the formula (9) and the formula (10).

The calculation formula of the loss of the image completion model is expressed as follows:

L _inp =L _pixel +λ _perc L _perc +λ _style L _style +λ _tv L _tv +λ _adv L _adv (11)

Using the second sample data set, the parameters are adjusted based on each training result until the loss of the image completion model satisfies the second preset condition, so as to obtain the parameters of the image completion model.

As an example, hyperparameter settings of λ _perc =0.1, λ _style =250, λ _tv =0.1, λ _adv =0.01.

优化判别子模型440直到模型收敛。As an example, the entire training process alternately uses _Linp to optimize the generator, using

The discriminant submodel 440 is optimized until the model converges.

As an embodiment, the parameters can be adjusted using the Adam optimizer, wherein the parameter values of the Adam optimizer are set: β ₁ =0, β ₂ =0.9, the initial learning rate of the generation sub-model 430 is 10 ⁻⁴ , and the discriminant sub-model 440 The initial learning rate is 10 ^-5 , and the batch size is set to 4. The batch size can be understood as the number of samples used in one training session.

Example process

Based on the application scenario discussed in FIG. 2 , the following describes the facial image processing method, please refer to FIG. 8 , the facial image processing method includes:

S810, the client 211 generates a completion request in response to a face image completion input operation on the face image completion interface.

Specifically, as discussed above, the client 211 supports facial image processing. When the user needs to complete the facial image, he can open the client 211 and perform input operations on the facial image completion interface. The terminal 211 generates a completion request in response to the input operation. The user can input the facial image to be processed and the facial feature adjustment information by custom, or select the facial image to be processed and the facial feature adjustment information recommended by the client 211, for example, the client 211 determines from the user's previous album that the display is not The complete image is recommended to the user for a completion operation. After the user inputs or selects the face image to be processed, the client 211 is equivalent to receiving a completion instruction from the user, thereby generating a completion request. The completion request carries the facial image to be processed and the facial feature adjustment information, or the index corresponding to the facial image to be processed and the index of the facial feature adjustment information.

As an embodiment, the client 211 may receive user-defined facial feature adjustment information, such as user-defined stick figures and the like.

For example, please refer to FIG. 9 , which shows a schematic diagram of a facial image completion interface. After the user selects an image to be processed, the client 211 can recommend facial feature adjustment information, such as “smile” and “normal” in FIG. 9 . and “dissatisfaction”, etc., the user can select the facial feature adjustment information recommended by the client 211 . After the user selects the click operation control 930, it is equivalent to the user performing an input operation. Alternatively, the user may click the custom operation control 910 shown in FIG. 9 to customize and select corresponding facial feature adjustment information. After the user clicks the confirmation control 920, it is equivalent to confirming the input of facial feature adjustment information.

Or, for example, please refer to FIG. 10 , which shows a schematic diagram of a user adding occlusion. After the user selects an image to be processed, the client 211 displays the interface shown in A in FIG. 10 , and the user can also click the add occlusion control 1001 in FIG. 10 . , input self-defined facial feature adjustment information, such as adding the graffiti shown in B in FIG. 10 , after the user clicks the operation control, it is equivalent to the user performing an input operation.

S820, the client 211 sends a completion request to the server 220.

Specifically, after generating the completion request, the client 211 sends the completion request to the server 220 .

S830, the server 220 obtains the face image to be processed.

Specifically, after receiving the completion request, the server 220 parses and obtains the face image to be processed through the information carried in the completion request. At the same time, the server 220 can also obtain facial feature adjustment information.

S840, the server 220 inputs the face image to be processed into the trained face feature point prediction model 410, and obtains a full face feature point set of the face image to be processed.

In a possible embodiment, the facial feature point prediction model 410 includes a first encoding module 411, a plurality of second encoding modules 412 and a full connection module 413, and an example of a specific process for obtaining a full face feature point set is as follows:

Through the first encoding module 411, feature extraction under various scales is performed on the face image to be processed to obtain a first feature map set;

Perform convolution pooling processing on the first feature map set through each coding module in the plurality of second coding modules 412 respectively to obtain a second feature value set;

Through the full connection module 413, a plurality of second feature value sets are spliced to obtain a full face feature point set.

The facial feature point prediction model 410 , the first encoding module 411 , the plurality of second encoding modules 412 and the fully connected module 413 may refer to the content discussed above, and will not be repeated here.

S850: Adjust corresponding feature points in the full-face feature point set according to the face feature adjustment information, to obtain an adjusted full-face feature point set.

Examples of specific adjustment methods are as follows:

method one:

Input the facial feature adjustment information into the facial feature point prediction model 410, obtain a full-face feature point set corresponding to the facial feature adjustment information, and then according to the full-face feature point set corresponding to the facial feature adjustment information to treat the face The full-face feature point set of the image is adjusted to obtain the adjusted full-face feature point set.

For the processing principle of this manner, reference may be made to the content discussed above, which will not be repeated here. This method is suitable for the case where the facial feature adjustment information is an image.

Method two:

The facial feature adjustment information and the full-face feature point set are input into the face adjustment model 420 to obtain an adjusted full-face feature point set.

Specifically, for the face adjustment model 420, reference may be made to the content discussed above, which will not be repeated here. This method is suitable for the case where the facial feature adjustment information is the expression classification label.

Method three:

Extracting the facial contour feature corresponding to the facial feature adjustment information; wherein, the facial contour feature is used to represent the contour formed by the facial key points;

According to the facial contour feature, the coordinates of the corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set.

S860, the server 220 inputs the adjusted full-face feature point set and the to-be-processed face image into the image completion model, and obtains a full-face image that completes the undisplayed area of the face in the to-be-processed face image.

Specifically, for the image completion model, reference may be made to the content discussed above, which will not be repeated here. The image completion model includes a generation sub-model 430 , which includes a third encoding module 431 , a hole convolution module 432 and a decoding module 433 . The following describes how to obtain a full-face image by generating the sub-model 430 .

Converting the adjusted full-face feature point set into a facial feature point map;

Through the third encoding module 431, the facial feature point map and the face image to be processed are subjected to convolution processing at different scales to obtain a third feature map set; wherein, the third feature map set includes the feature maps after convolution processing , the third encoding module of the feature map after convolution processing is the final convolution feature map output during the process of convolution processing at different scales on the facial feature point map and the face image to be processed;

Through the residual unit in the hole convolution module 432, a hole convolution is performed on the feature map after the convolution process to obtain a fourth feature map set;

Through the attention unit in the hole convolution module 432, the fourth feature map set is extracted, and the local features of the feature map after convolution processing are obtained, and the fifth feature map set is obtained;

The decoding module 433 performs up-sampling processing on the fifth feature map set and the third feature map set to obtain a full-face image in which the undisplayed area of the face in the face image to be processed is complemented.

In a possible embodiment, the third feature map set further includes a first intermediate feature map and a second intermediate feature map, and the first intermediate feature map is a sequence of pre-processing the facial feature point map and the facial image to be processed. It is assumed that the intermediate convolution result obtained by the sub-convolution process, and the second intermediate feature map is an intermediate convolution result obtained by performing a preset sub-convolution process on the first intermediate feature map.

The following is an example description of performing up-sampling processing on the fifth feature atlas and the third feature atlas by the decoding module 433 to obtain a full-face image obtained by performing complement processing on the undisplayed area of the face in the face image to be processed. .

Perform upsampling processing on the fifth feature atlas by the decoding module 433 to obtain the sixth feature atlas;

Perform weighted upsampling processing on the second intermediate feature map and the sixth feature map set by the decoding module 433 to obtain the seventh feature map set;

The seventh feature map set and the first intermediate feature map are weighted by the decoding module 433 to obtain the eighth feature map set;

The decoding module 433 performs convolution processing on the eighth feature map set to obtain a full-face image obtained by performing complement processing on the undisplayed area of the face in the face image to be processed.

The generation sub-model 430 includes a third encoding module 431 , and the content of the atrous convolution module 432 and the decoding module 433 can be referred to the content discussed above, which will not be repeated here.

After the server 220 processes the face image to be processed, a full face image can be obtained.

In a possible embodiment, the server 220 stores the correspondence between different species and parameters of different models (including the facial feature point prediction model, the image completion model and the face adjustment model discussed above), the server 220 After obtaining the facial image to be processed, first determine the species classification of the facial image to be processed, and then select parameters corresponding to the species, so as to improve the accuracy of facial image processing.

For example, please refer to FIG. 11 , A, B, C, D, and E in FIG. 11 are respectively the true-valued full-face image, the face image to be processed, and the image corresponding to the predicted face feature point set (in FIG. 11 ). C includes each feature point), the generation sub-model 430 directly completes the face image according to the full-face feature point set, the face image completed according to the full-face feature point set, and the full-face image after splicing the face images to be processed .

For example, please refer to FIG. 12 , A, B, C, D, and E in FIG. 12 respectively represent the ground-truth full-face image, the occluded face, and the image corresponding to the predicted face feature point set (C in FIG. 13 ) (including each feature point), the sub-model 430 generates a face image that is directly completed according to the full-face feature point set, a face image that is completed according to the full-face feature point set, and a full-face image after splicing the face image to be processed.

For example, please refer to FIG. 13 , A, B, C, and D in FIG. 13 respectively represent the occluded face pictures, the full-face image after completion based on A and the expression classification label (usually), based on A and the expression classification label ( Full-face image after completion of dissatisfaction), and full-face image after completion based on A and expression classification labels (laughs).

For example, please refer to FIG. 14 , A, B, C, and D in FIG. 14 represent the occluded face pictures, respectively, based on the full-face image completed by A and the expression classification label (laugh), based on A and the expression classification label ( Normal) full-face image after completion, and full-face image after completion based on A and expression classification label (dissatisfaction).

For example, please refer to Fig. 15, A, B, C, D, and E in Fig. 15 respectively represent the original face image of the true value, the face image after occlusion added manually by the user, and the predicted facial feature point set corresponding to The image, the face image completed according to the full-face feature point set, and the full-face image after splicing the face image to be processed.

For example, please refer to FIG. 16 , A, B, C, D, and E in FIG. 16 respectively represent the original face image of the true value, the face image after occlusion added manually by the user, and the predicted facial feature point set corresponding to The image, the face image completed according to the full-face feature point set, and the full-face image after splicing the face image to be processed.

S870 , the server 220 sends the full-face image to the client 211 .

S880, the client 211 acquires and displays the full-face image.

Specifically, the client 211 displays a full-face image for the user to view. The client 211 can also save the facial image to be processed, the user can click to cancel the completion, and the client 211 displays the facial image to be processed. After obtaining the full-face image, the user can also re-add occlusion and continue to perform operations such as completion.

For example, taking the user selecting the face image to be processed shown in FIG. 9 and selecting “smile” as an example, the server 220 obtains the image shown in FIG. 17 according to the face image to be processed and “smile” shown in FIG. 9 . Full face image.

For example, taking the user selecting the original face image shown in ” to obtain a full-face image as shown in Figure 18.

In a possible embodiment, each model in the server 220 is obtained by training other devices, and the server 220 directly uses each trained model to implement the face image processing method in FIG. 8 .

It should be noted that the image processing device 100 as the server 220 is used as an example for illustration in FIG. 8. In fact, there may be many types of image processing devices 100. For details, please refer to the content discussed above, that is, any image processing device The devices 100 can each implement what is discussed in FIG. 8 .

Based on the same inventive concept, an embodiment of the present application provides an image processing apparatus. Please refer to FIG. 19. The image processing apparatus 1900 is equivalent to being set in the image processing apparatus 100 discussed above, and includes a transceiver module 1901 and a processing module 1902, wherein:

In a possible embodiment, the facial feature point prediction model includes a first encoding module, a plurality of second encoding modules and a fully connected module, and the processing module 1902 is specifically configured to:

Through the first encoding module, the facial image to be processed is subjected to feature extraction under various scales to obtain a first feature atlas;

Perform convolution pooling processing on the first feature atlas through each encoding module in the plurality of second encoding modules to obtain a second feature value set;

Through the full connection module, multiple second feature value sets are spliced to obtain a full face feature point set.

In a possible embodiment, the facial feature adjustment information is an expression classification label, and the processing module 1902 is specifically used for:

Input the expression classification label and the full-face feature point set into the face adjustment model to obtain the adjusted full-face feature point set;

The face adjustment model is based on a sample facial expression image set, and each sample facial expression image in the sample facial expression image set is marked with a corresponding expression classification label.

In a possible embodiment, the processing module 1902 is specifically configured to:

Extracting the facial contour feature corresponding to the facial feature adjustment information; wherein, the facial contour feature is used to represent the contour formed by the facial key points;

According to the facial contour features, the coordinates of the corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set.

In a possible embodiment, the image completion model includes a third encoding module, a hole convolution module and a decoding module, and the processing module 1902 is specifically used for:

Convert the adjusted full-face feature point set into a face feature point map;

Through the third encoding module, the facial feature point map and the face image to be processed are subjected to convolution processing at different scales to obtain a third feature atlas; wherein, the third feature atlas includes the feature atlas after convolution processing. , the third encoding module of the feature map set after convolution processing is the final convolution feature map output during the process of convolution processing at different scales on the facial feature point map and the face image to be processed;

Through the residual unit in the hole convolution module, the hole convolution is performed on the feature atlas after the convolution processing, and the fourth feature atlas is obtained;

Through the attention unit in the atrous convolution module, the fourth feature atlas is extracted, and the local features of the feature atlas after convolution processing are used to obtain the fifth feature atlas;

The decoding module performs up-sampling processing on the fifth feature atlas and the third feature atlas, so as to obtain a full-face image obtained by performing complement processing on the undisplayed area of the face in the face image to be processed.

In a possible embodiment, the third feature atlas further includes a first intermediate feature atlas and a second intermediate feature atlas, and the first intermediate feature atlas is a facial feature point map and a face image to be processed The intermediate convolution results obtained by performing the preset sub-convolution processing in sequence, the second intermediate feature atlas is the intermediate convolution results obtained by performing the preset sub-convolution processing on the first intermediate feature atlas, and the processing module 1902 is specifically used for:

Perform upsampling processing on the fifth feature atlas by the decoding module to obtain the sixth feature atlas;

Perform weighting processing on the second intermediate feature atlas and the sixth feature atlas by the decoding module to obtain the seventh feature atlas;

Perform weighting and upsampling processing on the seventh feature atlas and the first intermediate feature atlas by the decoding module to obtain the eighth feature atlas;

Convolution processing is performed on the eighth feature atlas by the decoding module to obtain a full-face image obtained by performing complement processing on the undisplayed area of the face in the face image to be processed.

In a possible embodiment, the facial feature point prediction model is trained through the following steps:

Acquiring a first sample data set; wherein, the first sample data set includes a sample face image set to be processed, and a true-value sample full-face feature point set corresponding to each sample face image to be processed;

Based on the first sample data set, train a facial feature point prediction model;

Until the prediction loss of the facial feature point prediction model satisfies the first preset condition, the trained facial feature point prediction model is obtained; wherein, the prediction loss is used to represent the full face feature point set of the predicted sample and the full face feature point of the true value sample loss between sets.

In a possible embodiment, the image completion model includes a generation sub-model and a discriminant sub-model, and the loss of the image completion model is weighted according to pixel-by-pixel loss, perceptual loss, style loss, total differential loss and adversarial loss;

Among them, the pixel-by-pixel loss is used to represent the pixel value error of the completed sample full face image and the sample face image to be processed; the perceptual loss is used to represent the feature map of the completed sample full face image and the sample face to be processed. The error between the feature maps of the images; the style loss is used to represent each channel information vector of the feature map of the image after the completion of the sample full face image plus the occlusion area, and the sample face image to be processed plus the occlusion area. The sum of the Gramm matrix differences between the channels after the transformation; the total differential loss is used to represent the ratio between the sum of the horizontal and vertical derivatives of the sample full face image after completion and the total number of pixels included in the sample face image to be processed ;Generation loss is used to represent the loss of the discriminant sub-model to the source of the generated results of the generated sub-model and the real source of the generated results.

As an embodiment, the image processing apparatus 1900 in FIG. 19 can implement the function of the server 220 in FIG. 8 , or implement any one of the facial image processing methods discussed above.

Based on the same inventive concept, an embodiment of the present application provides a terminal device. Please refer to FIG. 20. The terminal device 210 includes a transceiver module 2001 and a display module 2002, wherein:

The transceiver module 2001 is used to respond to the facial image completion input operation on the facial image completion interface, and send a completion request to the server, so that the server can process the facial image to be processed according to the facial image processing method as discussed above. Process with the facial feature adjustment information to obtain a full-face image after the facial image is completed; wherein, the completion request includes the to-be-processed facial image and the facial feature adjustment information; and obtain the full-face image returned by the server;

The display module 2002 is used for displaying a full face image.

It should be noted that the terminal device 210 further includes some other components to assist in implementing corresponding functions, and this application does not list other components.

Based on the same inventive concept, an embodiment of the present application provides a facial image processing device, please continue to refer to FIG. 1 , the facial image processing device 100 includes a processor 102 and a memory 103 , and the processor 102 is used to call a program in the memory 103 instructions to implement any of the facial image processing methods discussed above.

As an embodiment, the processor 102 in FIG. 1 may be used to implement the functions of the transceiver module 1901 and the processing module 1902 in FIG. 19 .

Based on the same inventive concept, an embodiment of the present application provides a terminal device. Please continue to refer to FIG. 21 . The terminal device 210 includes a processor 2101 and a memory 2102. The processor 2101 is used to call program instructions in the memory 2102 to implement the foregoing discussion. Any one of the facial image processing methods, or realize the function of the client 211 in FIG. 8 .

As an embodiment, the processor 2101 in FIG. 21 may be used to implement the functions of the transceiver module 2001 and the display module 2002 in FIG. 20 .

Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when the computer instructions are executed on a computer, the computer executes the facial image discussed above. Approach.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (14)

1. a face image processing method, is characterized in that, comprises: obtaining a face image to be processed, where the face image to be processed includes a part of the face that is not displayed; Inputting the facial image to be processed into a trained facial feature point prediction model to obtain a full-face feature point set of the facial image to be processed; According to the facial feature adjustment information, adjusting the corresponding feature points in the full-face feature point set to obtain an adjusted full-face feature point set; Input the adjusted full-face feature point set and the to-be-processed face image into the trained image completion model to obtain a full-face that completes the undisplayed area of the face in the to-be-processed face image image. 2. The method according to claim 1, wherein the facial feature point prediction model comprises a first encoding module, a plurality of second encoding modules and a fully connected module, and the face image to be processed is input into a previously The trained facial feature point prediction model obtains the full-face feature point set of the to-be-processed facial image, including: Through the first encoding module, feature extraction is performed on the face image to be processed under various scales to obtain a first feature atlas; Perform convolution pooling processing on the first feature map set through each coding module in the plurality of second coding modules respectively to obtain a second feature value set; Through the full connection module, a plurality of second feature value sets are spliced to obtain a full face feature point set. 3. The method of claim 1, wherein the facial feature adjustment information is an expression classification label, and according to the facial feature adjustment information, the corresponding feature points in the full-face feature point set are adjusted to obtain The adjusted full-face feature point set, including: Inputting the facial expression classification label and the full-face feature point set into the face adjustment model to obtain the adjusted full-face feature point set; The face adjustment model is based on a sample facial expression image set, and each sample facial expression image in the sample facial expression image set is marked with a corresponding expression classification label. 4. The method according to claim 1, wherein, according to the facial feature adjustment information, the full-face feature point set is adjusted to obtain an adjusted full-face feature point set, comprising: Extracting the facial contour feature corresponding to the facial feature adjustment information; wherein, the facial contour feature is used to represent the contour formed by the facial key points; According to the facial contour feature, the coordinates of the corresponding feature points in the full-face feature point set are adjusted to obtain an adjusted full-face feature point set. 5. The method of claim 1, wherein the image completion model comprises a third encoding module, a hole convolution module and a decoding module, the adjusted full-face feature point set, and the to-be-processed The face image is input into the trained image completion model to obtain a full-face image that completes the undisplayed area of the face in the face image to be processed, including: converting the adjusted full-face feature point set into a facial feature point map; Through the third encoding module, the face feature point map and the face image to be processed are subjected to convolution processing at different scales to obtain a third feature atlas; wherein, the third feature atlas Including a feature map set after convolution processing, the third encoding module of the feature map set after convolution processing is to perform convolution processing at different scales on the facial feature point map and the face image to be processed In the process, the final convolution feature map of the output; Through the residual unit in the hole convolution module, perform hole convolution on the feature atlas after the convolution processing to obtain a fourth feature atlas; Extracting the fourth feature atlas and the local features of the feature atlas after the convolution processing through the attention unit in the hole convolution module to obtain the fifth feature atlas; Perform upsampling processing on the fifth feature atlas and the third feature atlas by the decoding module, so as to obtain a full face obtained by performing complement processing on the undisplayed area of the face in the face image to be processed image. 6. The method of claim 5, wherein the third feature atlas further comprises a first intermediate feature atlas and a second intermediate feature atlas, the first intermediate The facial feature point map, and the intermediate convolution results obtained by sequentially performing preset sub-convolution processing on the face image to be processed, and the second intermediate feature atlas is preset for the first intermediate feature atlas. The intermediate convolution results obtained by the secondary convolution processing are subjected to up-sampling processing on the fifth feature atlas and the third feature atlas by the up-sampling module to obtain the facial image to be processed. The full-face image with the completion processing of the undisplayed area of the face, including: Perform up-sampling processing on the fifth feature atlas by the decoding module to obtain a sixth feature atlas; Perform weighting processing on the second intermediate feature atlas and the sixth feature atlas by the decoding module to obtain a seventh feature atlas; Perform weighting and upsampling processing on the seventh feature atlas and the first intermediate feature atlas by the decoding module to obtain an eighth feature atlas; Perform convolution processing on the eighth feature atlas by the decoding module to obtain a full-face image obtained by performing complement processing on the undisplayed area of the face in the face image to be processed. 7. The method according to any one of claims 1-6, wherein the facial feature point prediction model is obtained by training through the following steps: Obtain a first sample data set; wherein, the first sample data set includes the sample face image set to be processed, and the true-value sample full-face feature point set corresponding to each sample face image to be processed ; Based on the first sample data set, train a facial feature point prediction model; Until the prediction loss of the facial feature point prediction model satisfies the first preset condition, the trained facial feature point prediction model is obtained; wherein, the prediction loss is used to represent the full face feature point set of the predicted sample and the full face of the true value sample loss between feature point sets. 8. The method according to any one of claims 1-6, wherein the image completion model is obtained by training through the following steps: Obtain a second sample data set; wherein, the second sample data set includes the sample face image set to be processed, the ground-truth sample full-face feature point set corresponding to each sample face image to be processed, and the a sample full-face image corresponding to each sample to-be-processed face image in the sample to-be-processed face image set; training the image completion module based on the second sample data set; Until the value of the loss corresponding to the image completion model satisfies the second preset condition, the trained image completion model is obtained. 9. The method of claim 8, wherein the image completion model comprises a generative sub-model and a discriminative sub-model, and the loss of the image completion model is based on pixel-wise loss, perceptual loss, style loss, full Differential loss and weighted against adversarial loss; Wherein, the pixel-by-pixel loss is used to represent the pixel value error of the sample full-face image after completion and the sample face image to be processed; the perceptual loss is used to represent the feature map and sample of the sample full-face image after completion The error between the feature maps of the to-be-processed face image; the style loss is used to represent the completed sample full-face image plus the occluded area image, and the sample to-be-processed face image plus the occlusion area. The information of each channel of the feature map is the sum of the Gramm matrix differences between the channels after quantization; the full differential loss is used to represent the sum of the horizontal and vertical derivatives of the completed sample full face image and the sample face image to be processed. The ratio between the total number of pixel points; the generation loss is used to represent the loss of the discriminant sub-model for the source of the generation result of the generation sub-model and the loss of the real source of the generation result. 10. A face image processing method, comprising: In response to a face image completion input operation on the face image completion interface, a completion request is sent to the server, so that the server responds to the facial image to be processed according to the method according to any one of claims 1-9. Process with the facial feature adjustment information to obtain a full-face image after the facial image is completed; wherein, the completion request includes the to-be-processed facial image and the facial feature adjustment information; Acquire and display the full-face image returned by the server. 11. A face image processing device, comprising: a transceiver module, configured to obtain a face image to be processed, where the face image to be processed includes a part of the face that is not displayed; The processing module is used for inputting the face image to be processed into the trained facial feature point prediction model to obtain the full face feature point set of the face image to be processed; The corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set; The model obtains a full-face image obtained by performing complement processing on the undisplayed area of the face in the to-be-processed face image. 12. A terminal device, comprising: The sending module is used to send a completion request to the server in response to a face image completion input operation on the face image completion interface, so that the server can respond to the method according to any one of claims 1-9. Process the facial image to be processed and the facial feature adjustment information to obtain a full-face image after the facial image is completed; wherein, the completion request includes the to-be-processed facial image and the facial feature adjustment information; and obtaining the full-face image returned by the server; The display module is used for displaying the full face image. 13. A face image processing device, comprising: at least one processor, and a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions executable by the at least one processor, and the at least one processor implements the method according to any one of claims 1-9 or 10 by executing the instructions stored in the memory . 14. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores computer instructions that, when the computer instructions are executed on a computer, cause the computer to perform any one of claims 1-9 or 10. one of the methods described.

Images