CN111385514A - Portrait processing method and device and terminal - Google Patents

Abstract

The present application provides a portrait processing method and device, and a terminal, which can restore the facial features of the user blocked by the virtual reality device, thereby improving the user experience of the receiving end. The method includes: the terminal acquires a first image of the user, the first image includes the user's face, and a partial area of the user's face is blocked by the virtual reality VR device, and the partial area includes the area where the user's eyes are; the The terminal inputs the first image into the portrait processing model to obtain a second image, where the second image includes the complete face of the user; the terminal sends the second image to the receiving end.

Description

Portrait processing method and device, and terminal

technical field

The present application relates to the field of terminal technology, and more particularly, to a method and apparatus for processing portraits and a terminal in the field of terminal technology.

Background technique

With the continuous development and progress of communication technology, large broadband service anytime and anywhere provides a wider platform for the application and development of virtual reality (virtual reality, VR) technology.

In many VR applications, immersive video calling is a very important business. Immersive video call means that both parties wear VR devices to make a video call. Each caller is equipped with a camera. The camera can capture the real scene of the caller during the video call and transmit it to the other party's VR device for presentation to realize simulation. The call effect of the real-life dialogue with the other party.

However, when the user wears the VR device to make a video call, the VR device will block part of the user's face, for example, the VR glasses will block the user's eyes. Therefore, when the user's call is captured through the camera, the other party cannot see it. To the user's facial features such as eye movements and expressions, the user experience is poor.

SUMMARY OF THE INVENTION

The present application provides a portrait processing method and device, and a terminal, which can restore a user's facial features blocked by a VR device, thereby improving the user experience at the receiving end.

In a first aspect, the present application provides a portrait processing method, the method includes: acquiring a first image of a user, where the first image includes the user's face, and a partial area of the user's face is captured by a virtual reality VR device occlusion, the partial area includes the area where the eyes of the user are located; the first image is input into a portrait processing model to obtain a second image, the second image includes the complete face of the user, wherein the portrait The processing model is obtained by training a sample training data set, and the sample training data set includes a plurality of original images and a plurality of restored images corresponding to the plurality of original images, wherein the plurality of original images are images of at least one original image. Collected from a sample user, the first original image in the multiple original images includes the first sample user, and the partial area of the face of the first sample user is blocked by the VR device, and the multiple original images include the first sample user. The first restored image in the restored image includes the complete face of the first sample user, and the at least one sample user includes the first sample user; and the second image is sent to the receiving end.

By using the portrait processing method provided by the embodiment of the present application, the facial features of the user blocked by the VR device can be restored, thereby improving the user experience of the receiving end.

Optionally, the portrait processing apparatus may also acquire the first image in other manners, which is not limited in this embodiment of the present application.

In a first possible implementation manner, the portrait processing device may receive a third image captured by the first camera device, the third image includes the user, and the portrait processing device obtains the image from the third image to capture the first image.

It should be noted that the first camera device may be the camera device 120 as shown in FIG. 1 . The first camera device and the portrait processing device may be independent devices, respectively, or the first camera device and the portrait processing device may be integrated in the same device, which is not limited in this embodiment of the present application.

It should be noted that, the third image may include at least the user. That is, the third image may further include the environment or scene where the user is located, which is not limited in this embodiment of the present application.

It should be noted that this embodiment of the present application only takes a part of the face of the user as the area where the eyes are located as an example for introduction. It should be understood that this part of the area may also include other areas of the face of the user. The embodiment of the present application This is not limited.

It should also be noted that the first image may include at least the face of the user, wherein a part of the face of the user is blocked by the VR device.

Optionally, the first image may further include at least one of other parts of the user and an environment where the user is located, or the first image may further include other users other than the user, The method for processing the portraits of other users is similar to the method for processing the portraits of the users, and in order to avoid repetition, details are not described here.

It should also be noted that when the scene image also includes other users, and a part of the face of the other user is also blocked by the VR device worn by the other user, the portrait processing device may use a portrait similar to the user. A method similar to the processing method is used to perform portrait processing on other users. To avoid repetition, details are not described here.

It should also be noted that the first image described in this embodiment of the present application may be an independent image, or may be a frame of image in a video stream, which is not limited in this embodiment of the present application.

Optionally, the first image may be an originally captured image, or an image with higher definition obtained after basic image quality processing.

Optionally, the basic image quality processing described in this embodiment of the present application may include at least one image processing step for improving image quality, such as denoising, sharpening, and brightness enhancement, which are not performed in this embodiment of the present application. limited.

Optionally, the portrait processing apparatus may acquire the portrait processing model in various manners, which is not limited in this embodiment of the present application.

In a first possible implementation manner, the portrait processing apparatus may be preconfigured with the portrait processing model at the factory.

In a second possible implementation manner, the portrait processing apparatus may receive the portrait processing model from other devices, that is, the portrait processing model is obtained by training from other devices.

For example, the portrait processing model may be stored in a cloud server, and the portrait processing apparatus may request the portrait processing model from the cloud server through a network.

In a third possible implementation manner, the portrait processing apparatus may train the portrait processing model by itself.

Optionally, taking the area where the user's eyes are located by the VR device as an example, the portrait processing device may acquire a sample training data set, and perform training and learning on the sample training data set to obtain the portrait processing model. The sample training data set includes multiple original images and multiple restored images corresponding to the multiple original images, wherein the multiple original images are collected from at least one sample user, and the multiple original images The first original image in the images includes a first sample user, and the partial area of the face of the first sample user is blocked by the VR device, and the first restored image among the plurality of restored images includes the first A complete face of a sample user, the at least one sample user including the first sample user.

Optionally, the portrait processing apparatus may obtain the portrait processing model by training and learning the sample training data set through various methods, which is not limited in this embodiment of the present application.

In a possible implementation manner, the portrait processing apparatus may obtain the portrait processing model by training and learning the sample training data set through a neural network model.

For example, the neural network model may be a generative adversarial nets (GAN) model, and the GAN may include a conditional generative adversarial network (Conditional GAN), a deep convolutional generative adversarial network (Deep Convolution GAN), and the like.

Using the portrait processing method provided by the embodiment of the present application, the facial features of the user blocked by the VR device in the first image can be restored through the portrait processing model, thereby improving the user experience at the receiving end.

Optionally, the portrait processing apparatus may input the first image and feature reference information into the portrait processing model to obtain the second image. Wherein, the feature reference information includes at least one of an eye feature parameter and a head feature parameter, the eye feature parameter includes at least one of position information, size information, and viewing angle information, and the position information is used for Indicate the position of each of the two eyes of the user, the size information is used to indicate the size of each eye, the angle of view information is used to indicate the eye gaze angle of each eye, the head The facial feature parameters include the three-axis attitude angle and acceleration of the user's head.

Using the portrait processing method provided by the embodiment of the present application, combining the first image and feature reference information, and inputting the portrait processing device, the restoration degree and authenticity of the occluded area can be improved by providing more portrait-related features.

Optionally, the portrait processing apparatus may acquire the feature reference information in various ways, which is not limited in this embodiment of the present application.

(1) How to obtain eye feature parameters

In a first possible implementation manner, the portrait processing device may receive a fourth image captured by a second camera, where the second camera is a built-in camera in the VR device, and the fourth image includes the the two eyes of the user; the portrait processing apparatus may extract the eye feature parameters from the fourth image.

It should be noted that, the second camera device may be a built-in camera device in the VR device, such as a built-in infrared (infrared, IR) camera.

Optionally, the embodiments of the present application only take the second camera as an example of a built-in camera on a VR device, and the second camera may also be a camera located at another location and capable of capturing images of the user being touched. The camera device of the real image of the face area blocked by the VR device is not limited in this embodiment of the present application.

In a second possible implementation manner, the portrait processing apparatus may extract the eye feature parameters from multiple images of the user stored locally, where the multiple images include the eyes of the user.

It should be noted that the plurality of images may be photos including the face taken by the user on a daily basis, such as self-portraits.

In a third possible implementation manner, the portrait processing device may establish a facial feature database of the user through the daily photos taken by the user, and the facial feature database includes feature parameters of various facial organs, such as eye feature parameters, The portrait processing device may retrieve the eye feature parameter from the facial feature database.

In a fourth possible implementation manner, the human image processing device may receive the eye feature parameters from other measuring devices, such as an eye tracking sensor.

Optionally, the multiple images and the facial feature database may also be stored in the cloud, and the portrait processing apparatus may request the cloud through a network, which is not limited in this embodiment of the present application.

It should be noted that, when the occluded content in the first image is restored by the portrait processing model, since the VR device mainly occludes the user's eye area, combined with the user's eye feature parameters, It can improve the restoration and authenticity of the occluded content.

(2) How to obtain head feature parameters

In a possible implementation manner, the portrait processing device may receive the head characteristic parameter measured by the inertial measurement device.

It should be noted that an inertial measurement device, which can also be an inertial measurement unit (Inertial measurement unit, IMU), is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object. Generally, an IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the object in the independent three-axis of the carrier coordinate system, and the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system. Measure the angular velocity and acceleration of an object in three-dimensional space, and use this to calculate the object's attitude.

Optionally, the inertial measurement device may be an independent measurement device fixed on the user's head, or the inertial measurement device may be integrated in the VR device, which is not limited in this embodiment of the present application.

It should be noted that, when the occluded content in the first image is restored by the portrait processing model, since the user's head may be in a motion state, that is, it has a motion speed and a rotation angle, the user's head may be in motion. It can further improve the restoration degree and authenticity of the occluded content.

Optionally, the feature reference information may further include other feature parameters, such as nose feature parameters, face shape feature parameters, hairstyle feature parameters and other parameters that can describe the user's portrait features, which are not limited in this embodiment of the present application.

Optionally, the portrait processing device may send the second image to the receiving end, or the portrait processing device may send a target image to the receiving end, where the target image is a The synthesized or spliced image, or the portrait processing apparatus may send a video stream to the receiving end, where the video stream includes the second image or the target image, which is not limited in this embodiment of the present application.

Correspondingly, the portrait processing apparatus may send the second image, or a target image or video stream including the second image, to the receiving end.

Optionally, the portrait processing apparatus may generate the video stream in various manners, which is not limited in this embodiment of the present application.

In a possible implementation manner, sending the second image by the portrait processing device to the receiving end includes: the portrait processing device splicing the second image and the fifth image to obtain a target image, wherein, The fifth image is an image obtained by cutting the first image from the fourth image; the portrait processing apparatus sends the target image to the receiving end.

In a possible implementation manner, sending the second image by the portrait processing device to the receiving end includes: the portrait processing device synthesizing the second image and the fourth image to obtain a target image, Wherein, the second image is overlaid on the upper layer of the first image in the fourth image; the portrait processing apparatus sends the target image to the receiving end.

Optionally, the portrait processing apparatus may generate the video stream in various manners, which is not limited in this embodiment of the present application.

In a possible implementation manner, the portrait processing apparatus may perform video encoding on the second image to obtain a video image; and obtain the video stream according to the video image, where the video stream includes the video image .

It should be noted that, considering that the restored second image restores the user's eye area blocked by the VR device, and the actual situation is that the user wears the VR device on the face, therefore, the difference between the second image and the actual situation is There are certain differences.

Therefore, the portrait processing apparatus may superimpose an eye mask layer on the user's eye region in the second image, wherein the eye mask layer is subjected to perspective processing with a first transparency to simulate wearing by the user The VR device can improve the authenticity of the restoration.

It should also be noted that the value of the first transparency should be such that it can be seen that the user is wearing the VR device without affecting the display of the occluded eye area under the VR device. The specific value is not limited.

In a second aspect, an embodiment of the present application provides a terminal, including: a processor and a transceiver coupled to the processor,

The processor is configured to control the transceiver to obtain a first image of the user, the first image includes the face of the user, and a part of the face of the user is blocked by a virtual reality VR device; The image is input into a portrait processing model to obtain a second image, where the second image includes the complete face of the user, wherein the portrait processing model is obtained by training a sample training data set, and the sample training data set includes multiple original images and multiple restored images corresponding to the multiple original images, wherein the multiple original images are collected from at least one sample user, and the first original image of the multiple original images includes the first original image. a sample user, and the partial area of the face of the first sample user is blocked by the VR device, the first restored image of the plurality of restored images includes the complete face of the first sample user, the At least one sample user includes the first sample user; and controls the transceiver to send the second image to the receiving end.

In a possible implementation manner, the processor is specifically configured to: input the first image and feature reference information into the portrait processing model to obtain the second image, wherein the feature reference information includes eye at least one of a facial feature parameter and a head feature parameter, the eye feature parameter includes at least one of position information, size information, and viewing angle information, and the position information is used to indicate that the two eyes of the user are in the The position of each eye, the size information is used to indicate the size of each eye, the angle of view information is used to indicate the eye gaze angle of each eye, and the head feature parameter includes the head of the user The three-axis attitude angle and acceleration of the part.

In a possible implementation manner, the feature reference information includes the eye feature parameters, and the processor is further configured to input the first image and the feature reference information into the portrait processing model to obtain Before the second image, the transceiver is controlled to receive a third image of the user captured by a first camera, the third image including the two eyes of the user, wherein the first camera is a built-in camera device of the VR device; the processor is configured to extract the eye feature parameter from the second image.

In a possible implementation manner, the feature reference information includes the head feature parameter, and the processor is further configured to input the first image and the feature reference information into the portrait processing model to obtain Before the second image, the transceiver is controlled to receive the head feature parameter measured by the inertial measurement device.

In a possible implementation manner, the processor is specifically configured to control the transceiver to receive the first image captured by the second camera.

In a possible implementation manner, the processor is specifically configured to: control the transceiver to receive a third image captured by a second camera, where the third image includes the user; intercept the third image from the third image the first image.

In a possible implementation manner, the processor is further configured to: stitch the second image and the fourth image to obtain a target image, wherein the fourth image is captured from the third image the image after the first image; controlling the transceiver to send the target image to the receiving end.

In a possible implementation manner, the processor is further configured to: combine the second image and the third image to obtain a target image, wherein the second image is overlaid on the third image in the upper layer of the first image; controlling the transceiver to send the target image to the receiving end.

In a possible implementation manner, the processor is further configured to superimpose an eye mask layer on the area where the user's eyes are located in the second image, and the eye mask layer is subjected to the first transparency process.

In a possible implementation manner, the portrait processing model is obtained by training the sample training data set through a generative adversarial network model.

In a third aspect, the present application further provides a portrait processing apparatus, which is configured to execute the method in the first aspect or any possible implementation manner of the first aspect. Specifically, the portrait processing apparatus may include a unit for executing the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

In a fourth aspect, an embodiment of the present application further provides a chip device, including: a communication interface and a processor, wherein the communication interface and the processor communicate with each other through an internal connection path, and the processor is configured to implement the above-mentioned first A method in an aspect or any possible implementation thereof.

In a fifth aspect, embodiments of the present application further provide a computer-readable storage medium for storing a computer program, where the computer program includes instructions for implementing the method in the first aspect or any possible implementation manner thereof.

In a sixth aspect, the embodiments of the present application further provide a computer program product, the computer program product includes instructions, when the instructions are run on a computer, the computer can implement the first aspect or any possible implementation manner thereof. Methods.

Description of drawings

FIG. 1 provides a schematic diagram of an application scenario 100 of an embodiment of the present application;

FIG. 2 provides a schematic diagram of another application scenario of an embodiment of the present application;

FIG. 3 provides a schematic flowchart of a portrait processing method 200 according to an embodiment of the present application;

FIG. 4 provides a schematic block diagram of the VR device 110 according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a display interface of a receiving end according to an embodiment of the present application;

FIG. 6 provides a schematic block diagram of a portrait processing apparatus 300 according to an embodiment of the present application;

FIG. 7 provides a schematic block diagram of a mobile phone 400 according to an embodiment of the present application;

FIG. 8 provides a schematic block diagram of a portrait processing system 500 according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

FIG. 1 shows a schematic diagram of an application scenario provided by an embodiment of the present application. As shown in FIG. 1 , a user wears a VR device 110 .

The camera device 120 is used for capturing a scene image of the user, and sending the scene image to the image processing device 130 , wherein the scene image includes the user's face, and a part of the user's face is blocked by the VR device 110 .

The portrait processing device 130 is configured to restore the area of the scene image where the user is blocked by the VR device through the portrait processing method to obtain a target image, wherein the target image includes the complete face of the user.

The portrait processing device 130 is further configured to transmit the target image to the receiving end 140 for display.

It should be noted that the scene image includes at least the face of the user, and FIG. 1 only schematically shows a situation where the scene image includes the face of the user, but the embodiment of the present application is not limited thereto.

Optionally, the scene image may also include at least one of other parts of the user and the environment where the user is located, or the scene image may also include other users other than the user. The embodiment does not limit this. For example: FIG. 2 shows a situation where the scene image includes the whole body of the user.

It should also be noted that when the scene image also includes other users, and a part of the face of the other user is also blocked by the VR device worn by the other user, the portrait processing device 130 may use the same method as that of the user. The portrait processing method is similar to the portrait processing method for other users. To avoid repetition, details are not described here.

It should also be noted that the scene image described in this embodiment of the present application may be a separate image, or may be a frame of image in a video stream, which is not limited in this embodiment of the present application.

It should also be noted that the embodiments of the present application are applicable to any situation in which a scene image of the user needs to be captured.

For example, when the user conducts a video call with the opposite user, the user's scene image is displayed through the display device (receiving end) of the opposite user.

It should be noted that the receiving end described in the embodiments of the present application may be any display device capable of receiving and displaying the target image sent by the terminal, which is not limited in the embodiments of the present application.

For another example: in the case of monitoring the user, a scene image of the user is displayed through a display device (receiving end).

Optionally, the receiving end may also be a terminal having a display screen, such as a VR device, a mobile phone, a smart TV, etc., which is not limited in this embodiment of the present application.

Optionally, the camera device 120 and the portrait processing device 130 in FIG. 1 may be independent devices, respectively, or the camera device 120 and the portrait processing device 130 may be integrated in the same device, which is not limited in this embodiment of the present application.

It should be noted that, regardless of whether the camera 120 and the portrait processing device 130 are integrated into one device or are separate devices, in the following description, the camera device and the portrait processing device will be described.

It should be noted that the VR device described in the embodiments of the present application may be a wearable device capable of realizing a virtual reality function.

It should also be noted that wearable devices, also known as wearable smart devices, are a general term for devices that use wearable technology to intelligently design daily wear and develop wearable devices, such as glasses, helmets, masks, etc. A wearable device is a portable device that is worn directly on the body or integrated into the user's accessories. Wearable device is not only a hardware device, but also realizes powerful functions through software support, data interaction, and cloud interaction. In a broad sense, wearable smart devices include full-featured, large-scale, complete or partial functions without relying on smart phones, such as smart helmets or smart glasses, and only focus on a certain type of application function, which needs to be used in conjunction with other devices such as smart phones. , such as various types of smart bracelets, smart jewelry, patches, etc. that can perform portrait processing and/or image display, which are not limited in the embodiments of the present application.

Optionally, communication between the portrait processing apparatus 130 and the receiving end 140 may be performed in a wired manner or wireless manner, which is not limited in this embodiment of the present application.

It should be noted that, the above-mentioned wired manner may be connected through a data line or through an internal bus connection to realize communication.

It should be noted that the above wireless manner may implement communication through a communication network, and the communication network may be a local area network, or a wide area network switched by a relay device, or includes a local area network and a wide area network. When the communication network is a local area network, exemplarily, the communication network may be a near field communication network such as a wifi hotspot network, a wifi P2P network, a Bluetooth network, a zigbee network, or a near field communication (NFC) network. When the communication network is a wide area network, exemplarily, the communication network may be a 3rd-generation wireless telephone technology (3G) network, a 4th generation mobile communication technology (4G) network , a fifth-generation mobile communication technology (5th-generation mobile communication technology, 5G) network, a future evolved public land mobile network (publicland mobile network, PLMN), or the Internet, etc., which are not limited in this embodiment of the present application.

The application scenarios to which the present application is applicable are described above, and the portrait processing method provided by the embodiments of the present application will be described in detail below.

FIG. 3 shows a schematic flowchart of a portrait processing method 200 provided by an embodiment of the present application. The method 200 may be applied to the application scenario described in FIG. 1 and executed by the portrait processing apparatus 130 .

S210: The first camera device captures a first image of the user, where the first image includes the user's face, and a partial area of the user's face is blocked by the virtual reality VR device, and the partial area includes the user's face the area where the eyes are located;

S220, the first camera device sends the first image of the user to the portrait processing device; correspondingly, the portrait processing device receives the first image sent by the first camera device.

Optionally, in S220, the portrait processing apparatus may also acquire the first image in other manners, which is not limited in this embodiment of the present application.

In a first possible implementation manner, the portrait processing device may receive a third image captured by the first camera device, the third image includes the user, and the portrait processing device obtains the image from the third image to capture the first image.

It should be noted that the first camera device may be the camera device 120 as shown in FIG. 1 . The first camera device and the portrait processing device may be independent devices, for example, the portrait processing device is a terminal, and the first camera device is a camera; or the first camera device and the portrait processing device may be integrated in the same device. A device, such as a terminal, is not limited in this embodiment of the present application.

It should be noted that, the third image may include at least the user. That is, the third image may further include the environment or scene where the user is located, which is not limited in this embodiment of the present application.

It should be noted that this embodiment of the present application only takes a part of the face of the user as the area where the eyes are located as an example for introduction. It should be understood that this part of the area may also include other areas of the face of the user. The embodiment of the present application This is not limited.

It should also be noted that the first image may include at least the face of the user, wherein a part of the face of the user is blocked by the VR device.

Optionally, the first image may further include at least one of other parts of the user and an environment where the user is located, or the first image may further include other users other than the user, The method for processing the portraits of other users is similar to the method for processing the portraits of the users, and in order to avoid repetition, details are not described here.

It should also be noted that when the scene image also includes other users, and a part of the face of the other user is also blocked by the VR device worn by the other user, the portrait processing device may use the portrait of the user. A method similar to the processing method is used to perform portrait processing on other users. To avoid repetition, details are not described here.

It should also be noted that the first image described in this embodiment of the present application may be an independent image, or may be a frame of image in a video stream, which is not limited in this embodiment of the present application.

Optionally, the first image may be an originally captured image, or an image with higher definition obtained after basic image quality processing.

Optionally, the basic image quality processing described in this embodiment of the present application may include at least one image processing step for improving image quality, such as denoising, sharpening, and brightness enhancement, which are not performed in this embodiment of the present application. limited.

S230, the portrait processing device inputs the first image into a portrait processing model to obtain a second image, where the second image includes the complete face of the user, wherein the portrait processing model is obtained by training a data set of samples obtained through training, the sample training data set includes multiple original images and multiple restored images corresponding to the multiple original images, wherein the multiple original images are collected from at least one sample user, and the A first original image of the plurality of original images includes a first sample user, and the partial area of the face of the first sample user is blocked by the VR device, and the first restored image of the plurality of restored images includes all the restored images. the complete face of the first sample user, and the at least one sample user includes the first sample user.

Optionally, before S230, the portrait processing apparatus may first acquire the portrait processing model.

Optionally, the portrait processing apparatus may acquire the portrait processing model in various manners, which is not limited in this embodiment of the present application.

In a first possible implementation manner, the portrait processing apparatus may be preconfigured with the portrait processing model at the factory.

In a second possible implementation manner, the portrait processing apparatus may receive the portrait processing model from other devices, that is, the portrait processing model is obtained by training from other devices.

For example, the portrait processing model may be stored in a cloud server, and the portrait processing apparatus may request the portrait processing model from the cloud server through a network.

In a third possible implementation manner, the portrait processing apparatus may train the portrait processing model by itself.

The following will introduce the training process of the portrait processing model by taking the portrait processing device training the portrait processing model by itself as an example. The process of training the portrait processing model by other equipment is similar to the training process of the portrait processing device itself. Repeat, and will not repeat them here.

Optionally, taking the area where the user's eyes are located by the VR device as an example, the portrait processing device may acquire a sample training data set, and perform training and learning on the sample training data set to obtain the portrait processing model. The sample training data set includes multiple original images and multiple restored images corresponding to the multiple original images, wherein the multiple original images are collected from at least one sample user, and the multiple original images The first original image in the images includes a first sample user, and the partial area of the face of the first sample user is blocked by the VR device, and the first restored image among the plurality of restored images includes the first A complete face of a sample user, the at least one sample user including the first sample user.

Optionally, the portrait processing apparatus may obtain the portrait processing model by training and learning the sample training data set through various methods, which is not limited in this embodiment of the present application.

In a possible implementation manner, the portrait processing apparatus may obtain the portrait processing model by training and learning the sample training data set through a neural network model.

For example, the neural network model may be a generative adversarial nets (GAN) model, and the GAN may include a conditional generative adversarial network (Conditional GAN), a deep convolutional generative adversarial network (Deep Convolution GAN), and the like.

Using the portrait processing method provided by the embodiment of the present application, the facial features of the user blocked by the VR device in the first image can be restored through the portrait processing model, thereby improving the user experience at the receiving end.

Optionally, in S230, the portrait processing apparatus may input the first image and feature reference information into the portrait processing model to obtain the second image. Wherein, the feature reference information includes at least one of an eye feature parameter and a head feature parameter, the eye feature parameter includes at least one of position information, size information, and viewing angle information, and the position information is used for Indicate the position of each of the two eyes of the user, the size information is used to indicate the size of each eye, the angle of view information is used to indicate the eye gaze angle of each eye, the head The facial feature parameters include the three-axis attitude angle and acceleration of the user's head.

That is, the sample training data set used in training the portrait processing model may include multiple original images, multiple restored images corresponding to the multiple original images, and at least one feature reference information, wherein the multiple original images The original image is collected from at least one sample user, the first original image in the plurality of original images includes the first sample user, and the partial area of the face of the first sample user is blocked by the VR device , the first restored image of the plurality of restored images includes the complete face of the first sample user, the at least one sample user includes the first sample user, and the at least one feature reference information includes the at least one sample user. Feature reference information for each sample user in a sample user.

Using the portrait processing method provided by the embodiment of the present application, combining the first image and feature reference information, and inputting the portrait processing device, the restoration degree and authenticity of the occluded area can be improved by providing more portrait-related features.

Optionally, the portrait processing apparatus may acquire the feature reference information in various ways, which is not limited in this embodiment of the present application.

(1) How to obtain eye feature parameters

In a first possible implementation manner, the portrait processing device may receive a fourth image captured by a second camera, where the second camera is a built-in camera in the VR device, and the fourth image includes the the two eyes of the user; the portrait processing apparatus may extract the eye feature parameters from the fourth image.

It should be noted that, the second camera device may be a built-in camera device in the VR device, such as a built-in infrared (infrared, IR) camera.

For example, as shown in FIG. 4 , the second camera device may be the camera device 150 built in the VR device 110 as shown in FIG. 1 .

Optionally, the embodiments of the present application only take the second camera as an example of a built-in camera on a VR device, and the second camera may also be a camera located at another location and capable of capturing images of the user being touched. The camera device of the real image of the face area blocked by the VR device is not limited in this embodiment of the present application.

In a second possible implementation manner, the portrait processing apparatus may extract the eye feature parameters from multiple images of the user stored locally, where the multiple images include the eyes of the user.

It should be noted that the plurality of images may be photos including the face taken by the user on a daily basis, such as self-portraits.

In a third possible implementation manner, the portrait processing device may establish a facial feature database of the user through the daily photos taken by the user, and the facial feature database includes feature parameters of various facial organs, such as eye feature parameters, The portrait processing device may retrieve the eye feature parameter from the facial feature database.

In a fourth possible implementation manner, the human image processing device may receive the eye feature parameters from other measuring devices, such as an eye tracking sensor.

Optionally, the multiple images and the facial feature database may also be stored in the cloud, and the portrait processing apparatus may request the cloud through a network, which is not limited in this embodiment of the present application.

It should be noted that, when the occluded content in the first image is restored by the portrait processing model, since the VR device mainly occludes the user's eye area, combined with the user's eye feature parameters, It can improve the restoration and authenticity of the occluded content.

(2) How to obtain head feature parameters

In a possible implementation manner, the portrait processing device may receive the head characteristic parameter measured by the inertial measurement device.

It should be noted that an inertial measurement device, which can also be an inertial measurement unit (Inertial measurement unit, IMU), is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object. Generally, an IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the object in the independent three-axis of the carrier coordinate system, and the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system. Measure the angular velocity and acceleration of an object in three-dimensional space, and use this to calculate the object's attitude.

Optionally, the inertial measurement device may be an independent measurement device fixed on the user's head, or the inertial measurement device may be integrated in the VR device, which is not limited in this embodiment of the present application.

It should be noted that, when the occluded content in the first image is restored by the portrait processing model, since the user's head may be in a motion state, that is, it has a motion speed and a rotation angle, the user's head may be in motion. It can further improve the restoration degree and authenticity of the occluded content.

Optionally, the feature reference information may further include other feature parameters, such as nose feature parameters, face shape feature parameters, hairstyle feature parameters and other parameters that can describe the user's portrait features, which are not limited in this embodiment of the present application.

S240, the portrait processing apparatus sends the second image to the receiving end; correspondingly, the receiving end receives the second image sent by the portrait processing device.

Optionally, in S240, the portrait processing apparatus may send the second image to the receiving end, or the portrait processing apparatus may send a target image to the receiving end, where the target image is for the first image. An image after synthesizing or splicing two images, or the portrait processing apparatus may send a video stream to the receiving end, and the video stream includes the second image or the target image, which is not implemented in this embodiment of the present application. limited.

Correspondingly, in S240, the portrait processing apparatus may send the second image, or a target image or video stream including the second image, to the receiving end.

Optionally, before S240, the portrait processing apparatus may generate the target image in various manners, which is not limited in this embodiment of the present application.

In a first possible implementation manner, the portrait processing device may stitch the second image and the fifth image to obtain a target image, wherein the fifth image is obtained by cutting out the fourth image. The image after the first image is described.

In a second possible implementation manner, the portrait processing apparatus may synthesize the second image and the fourth image to obtain a target image, wherein the second image is overlaid on the fourth image the upper layer of the first image.

Optionally, before S240, the portrait processing apparatus may generate the video stream in various manners, which is not limited in this embodiment of the present application.

In a possible implementation manner, the portrait processing apparatus may perform video encoding on the second image to obtain a video image; and obtain the video stream according to the video image, where the video stream includes the video image .

It should be noted that, considering that the restored second image restores the user's eye area blocked by the VR device, and the actual situation is that the user wears the VR device on the face, therefore, the difference between the second image and the actual situation is There are certain differences.

In a possible implementation manner, as shown in FIG. 5 , the portrait processing apparatus may superimpose an eye mask layer on the eye region of the user in the second image, wherein the eye mask layer passes through the first A transparent perspective processing.

It should also be noted that the value of the first transparency should be such that it can be seen that the user is wearing the VR device without affecting the display of the occluded eye area under the VR device. The specific value is not limited.

Using the portrait processing method provided by the embodiment of the present application, by superimposing the eye mask layer after the above-mentioned perspective processing on the eye region of the user in the second image, the VR device worn by the user can be simulated, thereby improving the restoration performance. authenticity.

It should be noted that the above-mentioned portrait processing apparatus may be a terminal, and in order to realize the above-mentioned functions, the terminal includes corresponding hardware and/or software modules for executing each function. The present application can be implemented in hardware or in the form of a combination of hardware and computer software in conjunction with the algorithm steps of each example described in conjunction with the embodiments disclosed herein. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functionality for each particular application in conjunction with the embodiments, but such implementations should not be considered beyond the scope of this application.

In this embodiment, the terminal may be divided into functional modules according to the foregoing method example. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that, the division of modules in this embodiment is schematic, and is only a logical function division, and there may be other division manners in actual implementation.

In the case where each functional module is divided according to each function, FIG. 6 shows a possible schematic diagram of the composition of the portrait processing apparatus 300 involved in the above embodiment. As shown in FIG. 6 , the apparatus 300 may include: a transceiver unit 310 and processing unit 320.

The processing unit 320 may control the transceiver unit 310 to implement the methods described in the above embodiments of the method 200, and/or other processes for the techniques described herein.

It should be noted that, all relevant contents of the steps involved in the above method embodiments can be cited in the functional description of the corresponding functional module, which will not be repeated here.

The apparatus 300 provided in this embodiment is configured to execute the above-mentioned portrait processing method, and thus can achieve the same effect as the above-mentioned implementation method.

Where an integrated unit is employed, the apparatus 300 may include a processing module, a storage module, and a communication module. The processing module may be used to control and manage the actions of the apparatus 300, for example, may be used to support the apparatus 200 to perform the steps performed by the above units. The storage module may be used to support the apparatus 300 to execute stored program codes, data, and the like. The communication module can be used for the communication between the apparatus 300 and other devices.

The processing module may be a processor or a controller. It may implement or execute the various exemplary logical blocks, modules and circuits described in connection with this disclosure. The processor may also be a combination that implements computing functions, such as a combination comprising one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and the like. The storage module may be a memory. The communication module may specifically be a device that interacts with other terminals, such as a radio frequency circuit, a Bluetooth chip, and a Wi-Fi chip.

In a possible implementation manner, the apparatus 300 involved in this embodiment of the present application may be a terminal.

It should be noted that the terminal described in the embodiments of the present application may be mobile or fixed, for example, the terminal may be a mobile phone, a camera, a video camera, a tablet personal computer (tablet personal computer), a smart TV, a laptop computer (laptop computer) ), a personal digital assistant (PDA), a personal computer (personal computer), or a wearable device such as a smart watch, etc., which are not limited in this embodiment of the present application.

Taking the terminal being a mobile phone as an example, FIG. 7 shows a schematic structural diagram of the mobile phone 400 . As shown in FIG. 7 , the mobile phone 400 may include a processor 410, an external memory interface 420, an internal memory 421, a universal serial bus (USB) interface 430, a charging management module 440, a power management module 441, a battery 442, an antenna 1, Antenna 2, Mobile Communication Module 450, Wireless Communication Module 460, Audio Module 470, Speaker 470A, Receiver 470B, Microphone 470C, Headphone Interface 470D, Sensor Module 480, Key 490, Motor 491, Indicator 492, Camera 493, Display screen 494, and a subscriber identification module (SIM) card interface 495 and the like.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the mobile phone 400 . In other embodiments of the present application, the mobile phone 400 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 410 may include one or more processing units, for example, the processor 410 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor ( image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent components, or may be integrated in one or more processors. In some embodiments, cell phone 400 may also include one or more processors 410 . The controller can generate an operation control signal according to the instruction operation code and the timing signal, and complete the control of fetching and executing instructions. In some other embodiments, a memory may also be provided in the processor 410 for storing instructions and data. Illustratively, the memory in the processor 410 may be a cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 410 . If the processor 410 needs to use the instruction or data again, it can be called directly from the memory. In this way, repeated access is avoided, and the waiting time of the processor 410 is reduced, thereby improving the efficiency of the mobile phone 400 for processing data or executing instructions.

In some embodiments, processor 410 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuitsound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input/output (general-purpose input/output, GPIO) interface, SIM card interface, and/or USB interface, etc. The USB interface 440 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 440 can be used to connect a charger to charge the mobile phone 400, and can also be used to transmit data between the mobile phone 400 and peripheral devices. The USB interface 440 can also be used to connect an earphone, and play audio through the earphone.

It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration, and does not constitute a structural limitation of the mobile phone 400 . In other embodiments of the present application, the mobile phone 400 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 440 is used to receive charging input from the charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 440 may receive charging input from the wired charger through the USB interface 430 . In some wireless charging embodiments, the charging management module 440 may receive wireless charging input through the wireless charging coil of the mobile phone 400 . While the charging management module 440 charges the battery 442 , it can also supply power to the mobile phone through the power management module 441 .

The power management module 441 is used to connect the battery 442 , the charging management module 440 and the processor 410 . The power management module 441 receives input from the battery 442 and/or the charge management module 440, and supplies power to the processor 410, the internal memory 421, the external memory, the display screen 494, the camera 494, and the wireless communication module 460. The power management module 441 can also be used to monitor parameters such as battery capacity, battery cycle times, battery health status (leakage, impedance). In some other embodiments, the power management module 441 may also be provided in the processor 410 . In other embodiments, the power management module 441 and the charging management module 440 may also be provided in the same device.

The wireless communication function of the mobile phone 400 can be realized by the antenna 1, the antenna 2, the mobile communication module 450, the wireless communication module 460, the modulation and demodulation processor, the baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in handset 400 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, the antenna 4 can be multiplexed into a diversity antenna of the wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 450 may provide a wireless communication solution including 2G/3G/4G/5G, etc. applied on the mobile phone 400 . The mobile communication module 450 may include at least one filter, switch, power amplifier, low noise amplifier (LNA) and the like. The mobile communication module 450 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 450 can also amplify the signal modulated by the modulation and demodulation processor, and then convert it into electromagnetic waves for radiation through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 450 may be provided in the processor 410 . In some embodiments, at least part of the functional modules of the mobile communication module 450 may be provided in the same device as at least part of the modules of the processor 410 .

The wireless communication module 460 can provide applications on the mobile phone 400 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite systems (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.

Optionally, the wireless communication module 460 may be one or more devices integrating at least one communication processing module, wherein one communication processing module may correspond to a network interface, and the network interface may be set in different service function modes, set in the Network interfaces in different modes can establish network connections corresponding to that mode. .

For example, a network connection supporting the P2P function can be established through the network interface in the P2P function mode, a network connection supporting the STA function can be established through the network interface in the STA function mode, and a network supporting the AP function can be established through the network interface in the AP mode. connect.

The wireless communication module 460 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 410 . The wireless communication module 460 can also receive the signal to be sent from the processor 410 , perform frequency modulation on it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2 .

The mobile phone 400 implements a display function through a GPU, a display screen 494, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 494 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 410 may include one or more GPUs that execute program instructions to generate or alter display information.

Display screen 494 is used to display images, video, and the like. Display screen 494 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). , AMOLED), flexible light-emitting diodes (flex light-emitting diodes, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (quantum dotlight emitting diodes, QLED) and so on. In some embodiments, cell phone 400 may include one or more display screens 494 .

In some embodiments of the present application, when the display panel adopts materials such as OLED, AMOLED, FLED, etc., the display screen 494 in FIG. 7 can be bent. Here, the above-mentioned display screen 494 can be bent means that the display screen can be bent to any angle at any position, and can be maintained at this angle, for example, the display screen 494 can be folded from the middle to left and right. It can also be folded up and down from the middle. In this application, a display screen that can be bent is called a foldable display screen. Wherein, the touch display screen may be one screen, or may be a display screen formed by piecing together multiple screens, which is not limited herein.

The display screen 494 of the mobile phone 400 may be a flexible screen. At present, the flexible screen has attracted much attention due to its unique characteristics and great potential. Compared with traditional screens, flexible screens have the characteristics of strong flexibility and bendability, which can provide users with new interactive methods based on the bendable characteristics, and can meet more needs of users for mobile phones. For mobile phones equipped with a foldable display, the foldable display on the mobile phone can be switched between a small screen in a folded state and a large screen in an unfolded state at any time. As a result, users are increasingly using the split-screen feature on phones with foldable displays.

The mobile phone 400 can realize the shooting function through the ISP, the camera 493, the video codec, the GPU, the display screen 494 and the application processor.

The ISP is used to process the data fed back by the camera 494 . For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin tone. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 494 .

Camera 493 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, cell phone 400 may include one or more cameras 493 .

A digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the mobile phone 400 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point, and the like.

Video codecs are used to compress or decompress digital video. Cell phone 400 may support one or more video codecs. In this way, the mobile phone 400 can play or record videos in various encoding formats, such as: Moving Picture Experts Group (moving picture experts group, MPEG) 1, MPEG2, MPEG4, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transfer of business functions between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the mobile phone 400 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 420 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 400 . The external memory card communicates with the processor 410 through the external memory interface 420 to realize the data storage function. For example to save files like music, video etc in external memory card.

Internal memory 421 may be used to store one or more computer programs including instructions. The processor 410 may execute the above-mentioned instructions stored in the internal memory 421, thereby causing the mobile phone 400 to execute the method for off-screen display, various applications and data processing provided in some embodiments of the present application. The internal memory 421 may include a storage program area and a storage data area. Wherein, the stored program area may store the operating system; the stored program area may also store one or more applications (such as gallery, contacts, etc.) and the like. The storage data area can store data (such as photos, contacts, etc.) created during the use of the mobile phone 400 and the like. In addition, the internal memory 421 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic disk storage components, flash memory components, universal flash storage (UFS), and the like. In some embodiments, the processor 410 may execute the instructions stored in the internal memory 421 and/or the instructions stored in the memory provided in the processor 410 to cause the mobile phone 400 to execute the functions provided in the embodiments of the present application. methods for on-screen display, as well as other applications and data processing. The mobile phone 400 can implement audio functions through an audio module 470, a speaker 470A, a receiver 470B, a microphone 470C, an earphone interface 470D, and an application processor. Such as music playback, recording, etc.

The sensor module 480 may include a pressure sensor 480A, a gyro sensor 480B, an air pressure sensor 480C, a magnetic sensor 480D, an acceleration sensor 480E, a distance sensor 480F, a proximity light sensor 480G, a fingerprint sensor 480H, a temperature sensor 480J, a touch sensor 480K, and an ambient light sensor 480L, bone conduction sensor 480M, etc.

The pressure sensor 480A is used to sense pressure signals, and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 480A may be provided on display screen 494 . There are many types of pressure sensors 480A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, and the like. The capacitive pressure sensor may be comprised of at least two parallel plates of conductive material. When a force is applied to pressure sensor 480A, the capacitance between the electrodes changes. The cell phone 400 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 494, the mobile phone 400 detects the intensity of the touch operation according to the pressure sensor 480A. The mobile phone 400 can also calculate the touched position according to the detection signal of the pressure sensor 480A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example, when a touch operation whose intensity is less than the first pressure threshold acts on the short message application icon, the instruction for viewing the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

The gyroscope sensor 480B can be used to determine the motion attitude of the mobile phone 400 . In some embodiments, the angular velocity of cell phone 400 about three axes (ie, X, Y, and Z axes) may be determined by gyro sensor 480B. The gyro sensor 480B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyroscope sensor 480B detects the shaking angle of the mobile phone 400, calculates the distance to be compensated by the lens module according to the angle, and allows the lens to counteract the shaking of the mobile phone 400 through reverse motion to achieve anti-shake. The gyroscope sensor 480B can also be used for navigation and somatosensory game scenarios.

The acceleration sensor 480E can detect the magnitude of the acceleration of the mobile phone 400 in various directions (generally three axes). When the mobile phone 400 is stationary, the magnitude and direction of gravity can be detected. It can also be used to recognize the posture of mobile phones, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.

The ambient light sensor 480L is used to sense ambient light brightness. The mobile phone 400 can adaptively adjust the brightness of the display screen 494 according to the perceived ambient light brightness. The ambient light sensor 480L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 480L can also cooperate with the proximity light sensor 480G to detect whether the mobile phone 400 is in the pocket to prevent accidental touch.

The fingerprint sensor 480H is used to collect fingerprints. The mobile phone 400 can use the collected fingerprint characteristics to unlock the fingerprint, access the application lock, take a picture with the fingerprint, answer the incoming call with the fingerprint, and the like.

The temperature sensor 480J is used to detect the temperature. In some embodiments, the cell phone 400 uses the temperature detected by the temperature sensor 480J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 480J exceeds a threshold value, the mobile phone 400 reduces the performance of the processor located near the temperature sensor 480J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the mobile phone 400 heats the battery 442 to avoid abnormal shutdown of the mobile phone 400 due to low temperature. In some other embodiments, when the temperature is lower than another threshold, the mobile phone 400 boosts the output voltage of the battery 442 to avoid abnormal shutdown caused by low temperature.

Touch sensor 480K, also called "touch panel". The touch sensor 480K may be disposed on the display screen 494, and the touch sensor 480K and the display screen 494 form a touch screen, also called a "touch screen". The touch sensor 480K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to touch operations may be provided through display screen 494 . In other embodiments, the touch sensor 480K may also be disposed on the surface of the mobile phone 400 , which is different from the location where the display screen 494 is located.

The keys 490 include a power-on key, a volume key, and the like. Keys 490 may be mechanical keys. It can also be a touch key. The cell phone 400 can receive key input and generate key signal input related to user settings and function control of the cell phone 400 .

It should be noted that the related functions implemented by the processing unit 320 in FIG. 6 can be implemented by the processor 410 in FIG. 7 , and the related functions implemented by the transceiver unit 310 in FIG. 6 can be implemented by the processor 410 in FIG. 7 . It is realized by controlling the antenna 450 or the antenna 460, or the related functions realized by the transceiver unit 310 in FIG. 6 can be realized by the processor 410 in FIG. 7 controlling other components of the mobile phone 400 through the internal interface.

FIG. 8 shows a portrait processing system 500 provided by an embodiment of the present application. The system 500 includes a mobile phone 510, a VR device 520, and a receiving end 530. A communication interface exists between the mobile phone 510 and the VR device 520, and a communication interface exists between the mobile phone 510 and the receiving end 530. There is a communication interface between them. The mobile phone 510 includes a first camera device 511, a first image processing module 512, a second image processing module 513, a portrait processing module 514, a storage module 515, an image synthesis module 516, and a video encoding module 517, and the VR device 520 includes a display screen 521 , a second camera device 522 , and an inertial measurement device 523 .

Optionally, the inertial measurement device 523 may be integrated in the VR device 520, or may be an independent device, which is not limited in this embodiment of the present application.

Optionally, the first camera device 511 may be a built-in camera device of the mobile phone 510, or may be an independent camera device, which is not limited in this embodiment of the present application.

The display screen 521 is used to play the VR resources acquired from the mobile phone 510 through the interface.

The first camera device 511 is used to capture a third image of the user and send it to the first image processing module 512, wherein the third image includes the user and the scene where the user is located, and a part of the user's face is The VR device 520 blocks, and the partial area includes the area where the user's eyes are located.

The first image processing module 512 is configured to receive the third image sent by the first camera 511; perform basic image processing such as enhancement effects on the third image; intercept the first image from the processed third image, so The first image includes the user's face, and the first image is sent to the portrait processing module 514 .

The second camera device 522 is a built-in camera of the VR device 520, and is used for capturing a fourth image of the user and sending it to the second image processing module 513, wherein the fourth image includes both eyes of the user.

The second image processing module 513 is configured to receive the fourth image sent by the second camera 522 , perform basic image processing such as enhancement effects on the fourth image, and send the processed fourth image to the portrait processing module 514 .

The inertial measurement device 523 is used to acquire the user's head feature parameters, where the head feature parameters include the three-axis posture and acceleration of the user's head, and send them to the portrait processing module 514 .

The portrait processing module 514 is configured to extract the user's eye feature parameters from the received fourth image; input the first image, the eye feature parameters and the head feature parameters into the storage module 515 to obtain a second image; send the second image to the image synthesis module 516, and the second image includes the complete face of the user.

The image synthesis module 516 is used for synthesizing or splicing the second image to obtain a target image, and sending the target to the video encoding module 517 .

The video encoding module 517 is configured to encode the target image to obtain a video stream, and send the video stream to the receiving end 530 .

Optionally, the portrait processing module 514 in FIG. 8 may be an NPU, DSP, GPU, and other computing units described in FIG. 7 , which is not limited in this embodiment of the present application.

It should be noted that the first camera 511 in FIG. 8 may be the camera 494 described in FIG. 7 ; the first image processing module 512 and the second image processing module 513 may belong to the ISP described in FIG. 7 , or The first image processing module 512 and the second image processing module may be different ISPs; the portrait processing module 514 may be computing units such as NPU, DSP, and GPU described in FIG. 7 ; the storage module 515 may be described in FIG. 7 . The internal memory 421, or the memory provided in the processor 410; the image synthesis module 516 may be the GPU described in FIG. 7 ; the video encoding module 517 may be the video encoder described in FIG. 7 .

Optionally, each module in the foregoing mobile phone 500 may also be implemented by other devices capable of implementing the functions implemented by each module in FIG. 7 , which is not limited in this embodiment of the present application.

Optionally, the system 500 provided in this embodiment of the present application is only introduced in one possible implementation manner, which is not limited in this embodiment of the present application, and the system 500 should be able to implement other steps described in the foregoing method embodiment 200 , in order to avoid repetition, it will not be repeated here.

This embodiment also provides a computer storage medium, where computer instructions are stored in the computer storage medium, and when the computer instructions are executed on the electronic device, the electronic device executes the above-mentioned relevant method steps to realize the portrait processing method in the above-mentioned embodiment.

This embodiment also provides a computer program product, which, when the computer program product runs on a computer, causes the computer to execute the above-mentioned relevant steps, so as to realize the portrait processing method in the above-mentioned embodiment.

In addition, the embodiments of the present application also provide an apparatus, which may specifically be a chip, a component or a module, and the apparatus may include a connected processor and a memory; wherein, the memory is used for storing computer execution instructions, and when the apparatus is running, The processor can execute the computer-executable instructions stored in the memory, so that the chip executes the portrait processing methods in the foregoing method embodiments.

Wherein, the server, terminal, computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved may refer to the corresponding provided above. The beneficial effects of the method are not repeated here.

It should be understood that, in various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk and other mediums that can store program codes.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (24)

1. a portrait processing method, is characterized in that, comprises: The terminal acquires a first image of the user, where the first image includes the user's face, and a partial area of the user's face is blocked by the virtual reality VR device, and the partial area includes the area where the user's eyes are located; The terminal inputs the first image into a portrait processing model to obtain a second image, where the second image includes the complete face of the user, wherein the portrait processing model is obtained by training a sample training data set, The sample training data set includes multiple original images and multiple restored images corresponding to the multiple original images, wherein the multiple original images are collected from at least one sample user, and the multiple original images are obtained. The first original image in the plurality of restored images includes a first sample user, and the partial area of the face of the first sample user is blocked by the VR device, and the first restored image of the plurality of restored images includes the first image The complete face of this user, the at least one sample user includes the first sample user; The terminal sends the second image to the receiving end. 2. The method according to claim 1, wherein the terminal inputs the first image into a portrait processing model to obtain a second image, comprising: The terminal inputs the first image and feature reference information into the portrait processing model to obtain the second image, wherein the feature reference information includes at least one of an eye feature parameter and a head feature parameter, The eye feature parameter includes at least one item of position information, size information, and viewing angle information, where the position information is used to indicate the position of each of the two eyes of the user, and the size information is used to indicate the position of each eye of the user. The size of each eye and the viewing angle information are used to indicate the eyeball gaze angle of each eye, and the head feature parameter includes the triaxial attitude angle and acceleration of the user's head. 3 . The method according to claim 2 , wherein the feature reference information includes the eye feature parameters, and the terminal inputs the first image and the feature reference information into the portrait processing model to obtain 3 . Before the second image, the method further includes: The terminal receives a third image of the user captured by a first camera device, where the third image includes the two eyes of the user, wherein the first camera device is a camera built in the VR device device; The terminal extracts the eye feature parameter from the second image. 4 . The method according to claim 2 , wherein the feature reference information includes the head feature parameter, and the terminal inputs the first image and the feature reference information into the portrait processing model to obtain 4 . Before the second image, the method further includes: The terminal receives the head characteristic parameter measured by the inertial measurement device. 5. The method according to any one of claims 1 to 4, wherein the terminal acquiring the first image of the user comprises: The first image captured by the terminal through the second camera. 6. The method according to any one of claims 1 to 4, wherein the terminal acquiring the first image of the user comprises: The terminal shoots a fourth image by using the second camera, and the fourth image includes the user; The terminal captures the first image from the fourth image. 7. The method according to claim 6, wherein the sending of the second image by the terminal to the receiving end comprises: The terminal splices the second image and the fifth image to obtain a target image, wherein the fifth image is an image obtained by intercepting the first image from the fourth image; The terminal sends the target image to the receiving end. 8. The method according to claim 6, wherein the terminal sending the second image to the receiving end comprises: The terminal synthesizes the second image and the fourth image to obtain a target image, wherein the second image is overlaid on the upper layer of the first image in the fourth image; The terminal sends the target image to the receiving end. 9. The method according to any one of claims 1 to 8, wherein the method further comprises: An eye mask layer is superimposed on the area where the user's eyes are located in the second image, and the eye mask layer is subjected to a first transparency process. 10. The method according to any one of claims 1 to 9, wherein the portrait processing model is obtained by training the sample training data set through a generative adversarial network model. 11. A terminal, comprising: a processor and a transceiver coupled with the processor, The processor is configured to control the transceiver to acquire a first image of the user, the first image includes the user's face, and a part of the user's face is blocked by the virtual reality VR device; the first image is The image is input into a portrait processing model to obtain a second image, where the second image includes the complete face of the user, wherein the portrait processing model is obtained by training a sample training data set, and the sample training data set includes multiple original images and multiple restored images corresponding to the multiple original images, wherein the multiple original images are collected from at least one sample user, and the first original image in the multiple original images includes the first original image. a sample user, and the partial area of the face of the first sample user is blocked by the VR device, the first restored image of the plurality of restored images includes the complete face of the first sample user, the At least one sample user includes the first sample user; and controls the transceiver to send the second image to the receiving end. 12. The terminal according to claim 11, wherein the processor is specifically configured to: Inputting the first image and feature reference information into the portrait processing model to obtain the second image, wherein the feature reference information includes at least one of an eye feature parameter and a head feature parameter, the eye feature The external feature parameters include at least one item of position information, size information, and viewing angle information, where the position information is used to indicate the position of each of the two eyes of the user, and the size information is used to indicate the position of each of the two eyes of the user. The size of the eyes and the viewing angle information are used to indicate the eyeball gaze angle of each eye, and the head characteristic parameter includes the triaxial attitude angle and acceleration of the user's head. 13. The terminal according to claim 12, wherein the feature reference information comprises the eye feature parameter, The processor is further configured to control the transceiver to receive the user captured by the first camera device before the first image and feature reference information are input into the portrait processing model to obtain the second image The third image of the third image includes the two eyes of the user, wherein the first camera is a built-in camera of the VR device; The processor is configured to extract the eye feature parameter from the second image. 14. The terminal according to claim 12, wherein the feature reference information comprises the header feature parameter, The processor is further configured to control the transceiver to receive the head measured by an inertial measurement device before the first image and feature reference information are input into the portrait processing model to obtain the second image Characteristic Parameters. The terminal according to any one of claims 11 to 14, wherein the processor is specifically configured to control the transceiver to receive the first image captured by the second camera. 16. The terminal according to any one of claims 11 to 14, wherein the processor is specifically configured to: controlling the transceiver to receive a third image captured by the second camera, the third image including the user; The first image is captured from the third image. 17. The terminal according to claim 16, wherein the processor is further configured to: splicing the second image and the fourth image to obtain a target image, wherein the fourth image is an image obtained by intercepting the first image from the third image; Controlling the transceiver to send the target image to the receiving end. 18. The terminal according to claim 16, wherein the processor is further configured to: synthesizing the second image and the third image to obtain a target image, wherein the second image covers the upper layer of the first image in the third image; Controlling the transceiver to send the target image to the receiving end. 19. The terminal according to any one of claims 11 to 18, wherein the processor is further configured to superimpose an eye mask layer on the area where the user's eyes are located in the second image, and the The eye mask layer is perspectiveized with the first transparency. 20. The terminal according to any one of claims 11 to 19, wherein the portrait processing model is obtained by training the sample training data set through a generative adversarial network model. 21. A portrait processing device, characterized in that the device comprises a unit for implementing the method according to any one of the preceding claims 1 to 10. 22. A chip device, comprising: a communication interface and a processor, wherein the communication interface and the processor communicate with each other through an internal connection path, wherein the processor is used to implement the above claims 1 to 10 The method of any one. 23. A computer-readable storage medium for storing a computer program, wherein the computer program comprises instructions for implementing the method of any one of the preceding claims 1 to 10. 24. A computer program product comprising instructions, characterized in that, when the instructions are executed on a computer, the instructions cause the computer to implement the method according to any one of the preceding claims 1 to 10.

Images