CN109147037B - Special effect processing method, device and electronic device based on 3D model - Google Patents

Abstract

The application provides a special effect processing method and device based on a three-dimensional model and electronic equipment, wherein the method comprises the following steps: acquiring an acquired two-dimensional face image and depth information corresponding to the face image; performing three-dimensional reconstruction on the face according to the depth information and the face image to obtain a three-dimensional model corresponding to the face; recognizing expression categories corresponding to the two-dimensional face images; and fusing the three-dimensional model with the special effect model corresponding to the expression category to obtain the three-dimensional model after special effect processing. The method can realize that different special effect models do not need to be manually switched by a user, improve the automation degree of special effect addition, improve the enjoyment and playability of the user in the special effect addition process, improve the reality of the special effect addition and enable the treated effect to be better and natural.

Description

Special effect processing method, device and electronic device based on 3D model

technical field

The present application relates to the technical field of electronic devices, and in particular, to a three-dimensional model-based special effect processing method, device, and electronic device.

Background technique

With the popularization of electronic devices, more and more users like to take pictures or record their life by using the photographing function of the electronic devices. And in order to make the captured images more interesting, various applications for beautifying the images or adding special effects have been developed. Users can choose their favorite special effects from all the special effects that come with the application to process images according to their own needs, making the images vivid and interesting.

The addition of facial special effects such as tears depends on the user's active selection, resulting in a low degree of automation of adding special effects. In addition, adding special effects to images is performed on a two-dimensional image, so that the special effects cannot be perfectly fitted or matched with the image. As a result, the image processing effect is poor, and the realism added by special effects is not strong.

SUMMARY OF THE INVENTION

The present application proposes a special effect processing method, device and electronic device based on a three-dimensional model, so as to realize that there is no need for a user to manually switch between different special effect models, improve the automation degree of special effect addition, and improve the fun and playability of the user in the special effect addition process It can improve the realism of special effects, and make the processed effects better and more natural. It is used to solve the technical problems of low realism and low automation in the prior art.

An embodiment of the present application proposes a special effect processing method based on a three-dimensional model, including:

acquiring the collected two-dimensional face image and depth information corresponding to the face image;

performing three-dimensional reconstruction on the human face according to the depth information and the human face image to obtain a three-dimensional model corresponding to the human face;

Identify the expression category corresponding to the two-dimensional face image;

The three-dimensional model and the special effect model corresponding to the expression category are fused to obtain a three-dimensional model after special effect processing.

The three-dimensional model-based special effects processing method according to the embodiment of the present application obtains the collected two-dimensional face image and the depth information corresponding to the face image, and then performs three-dimensional reconstruction of the face according to the depth information and the face image. , to obtain the three-dimensional model corresponding to the face, then identify the expression category corresponding to the two-dimensional face image, and finally fuse the three-dimensional model with the special effect model corresponding to the expression category. As a result, the user does not need to manually switch between different special effect models, the automation degree of special effect addition is improved, and the fun and playability of the user in the special effect addition process are improved. In addition, according to the expression made by the user, the corresponding special effect model is determined, so that the special effect model and the three-dimensional model are integrated, which can enhance the realism added by the special effect, and make the processed effect more natural.

Another aspect of the present application provides a three-dimensional model-based special effect processing device, including:

an acquisition module for acquiring the collected two-dimensional face image and depth information corresponding to the face image;

a reconstruction module, configured to perform three-dimensional reconstruction of the human face according to the depth information and the human face image, so as to obtain a three-dimensional model corresponding to the human face;

an identification module for identifying the expression category corresponding to the two-dimensional face image;

The fusion module is used to fuse the three-dimensional model with the special effect model corresponding to the expression category to obtain a three-dimensional model after special effect processing.

The three-dimensional model-based special effect processing device according to the embodiment of the present application obtains the collected two-dimensional face image and the depth information corresponding to the face image, and then performs three-dimensional reconstruction of the face according to the depth information and the face image. , to obtain a three-dimensional model corresponding to the face, then identify the expression category corresponding to the two-dimensional face image, and finally fuse the three-dimensional model with the special effect model corresponding to the expression category to obtain a three-dimensional model after special effect processing. As a result, the user does not need to manually switch between different special effect models, the automation degree of special effect addition is improved, and the fun and playability of the user in the special effect addition process are improved. In addition, according to the expression made by the user, the corresponding special effect model is determined, so that the special effect model and the three-dimensional model are integrated, which can enhance the realism added by the special effect, and make the processed effect more natural.

Another aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and running on the processor, when the processor executes the program, the computer program as described in the present application is implemented. The three-dimensional model-based special effect processing method proposed in the foregoing embodiments.

Another aspect of the present application provides a computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the three-dimensional model-based special effect processing as proposed in the foregoing embodiments of the present application is implemented. method.

Additional aspects and advantages of the present application will be set forth, in part, in the following description, and in part will be apparent from the following description, or learned by practice of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a schematic flowchart of a three-dimensional model-based special effect processing method provided in Embodiment 1 of the present application;

2 is a schematic flowchart of a three-dimensional model-based special effect processing method provided in Embodiment 2 of the present application;

3 is a schematic flowchart of a three-dimensional model-based special effect processing method provided in Embodiment 3 of the present application;

4 is a schematic structural diagram of a three-dimensional model-based special effect processing device provided in Embodiment 4 of the present application;

5 is a schematic structural diagram of a three-dimensional model-based special effect processing device provided in Embodiment 5 of the present application;

6 is a schematic diagram of the internal structure of an electronic device in one embodiment;

7 is a schematic diagram of an image processing circuit as a possible implementation;

FIG. 8 is a schematic diagram of an image processing circuit as another possible implementation.

Detailed ways

The following describes in detail the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to be used to explain the present application, but should not be construed as a limitation to the present application.

This application mainly aims at the technical problems of low realism and low degree of automation added by special effects in the prior art, and proposes a special effect processing method based on a three-dimensional model.

The three-dimensional model-based special effects processing method according to the embodiment of the present application obtains the collected two-dimensional face image and the depth information corresponding to the face image, and then performs three-dimensional reconstruction of the face according to the depth information and the face image. , to obtain a three-dimensional model corresponding to the face, then identify the expression category corresponding to the two-dimensional face image, and finally fuse the three-dimensional model with the special effect model corresponding to the expression category to obtain a three-dimensional model after special effect processing. As a result, the user does not need to manually switch between different special effect models, the automation degree of special effect addition is improved, and the fun and playability of the user in the special effect addition process are improved. In addition, according to the expression made by the user, the corresponding special effect model is determined, so that the special effect model and the three-dimensional model are integrated, which can enhance the realism added by the special effect, and make the processed effect more natural.

The three-dimensional model-based special effect processing method, apparatus, and electronic device according to the embodiments of the present application are described below with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart of a three-dimensional model-based special effect processing method provided by Embodiment 1 of the present application.

As shown in Figure 1, the three-dimensional model-based special effect processing method includes the following steps:

Step 101: Acquire the collected two-dimensional face image and depth information corresponding to the face image.

In this embodiment of the present application, the electronic device may include a visible light image sensor, and a two-dimensional face image may be acquired based on the visible light image sensor in the electronic device. Specifically, the visible light image sensor may include a visible light camera, and the visible light camera may capture the visible light reflected by the face for imaging to obtain a two-dimensional face image.

In this embodiment of the present application, the electronic device may further include a structured light image sensor, and depth information corresponding to the face image may be acquired based on the structured light image sensor in the electronic device. Optionally, the structured light image sensor may include a laser light and a laser camera. Pulse Width Modulation (PWM) can modulate the laser light to emit structured light, and the structured light is irradiated to the face of an adult. The laser camera can capture the structured light reflected by the face for imaging, and obtain the structured light image corresponding to the face. The depth engine can calculate and obtain the depth information corresponding to the face according to the structured light image corresponding to the face, that is, the depth information corresponding to the two-dimensional face image.

Step 102: Perform three-dimensional reconstruction of the human face according to the depth information and the human face image to obtain a three-dimensional model corresponding to the human face.

In this embodiment of the present application, after acquiring the depth information and the face image, three-dimensional reconstruction of the face may be performed according to the depth information and the face image, so as to obtain a three-dimensional model corresponding to the face. In this application, the construction of the three-dimensional model corresponding to the face is obtained by performing three-dimensional reconstruction according to the depth information and the face image, rather than simply acquiring RGB data and depth data.

As a possible implementation manner, the depth information and the color information corresponding to the two-dimensional face image can be fused to obtain a three-dimensional model corresponding to the face. Specifically, based on the face key point detection technology, the key points of the face can be extracted from the depth information, and the key points of the face can be extracted from the color information, and then the key points extracted from the depth information and the color information can be extracted. The key points are registered and the key points are fused, and finally the 3D model corresponding to the face is generated according to the fused key points. Among them, the key points are conspicuous points on the face, or points at key positions, for example, the key points may be the corners of the eyes, the tip of the nose, the corners of the mouth, and the like.

As another possible implementation, based on the face key point detection technology, key points can be identified on the face image to obtain the second key point corresponding to the face image, and then according to the depth information of the second key point and the second key point The position of the key point on the face image is to determine the relative position of the first key point corresponding to the second key point in the three-dimensional model of the face, so that the neighbors can be connected according to the relative position of the first key point in the three-dimensional space. The first key point of , generates a local face 3D frame. The partial human face may include facial parts such as nose, lips, eyes, and cheeks.

After the partial face three-dimensional frame is generated, different partial face three-dimensional frames can be spliced according to the same first key point contained in the different partial face three-dimensional frames to obtain a three-dimensional model corresponding to the face.

Step 103: Identify the expression category corresponding to the two-dimensional face image.

As a possible implementation, the user can pre-record the reference expressions corresponding to different expression categories. For example, the user can pre-record the reference expressions corresponding to the expression categories such as sad, happy, frustrated, angry, and thinking. After the image, the face image and the reference expression can be matched, and the expression category corresponding to the target reference expression in the matching is used as the expression category of the face image.

As another possible implementation manner, at least one frame of face image collected before the current frame may be acquired, and then the expression category may be determined according to the face image of the current frame and at least one frame of face image. For example, according to the face image of the current frame and at least one frame of face image, it can be determined whether the eyebrows are raised or pulled down, the eyes are larger or smaller, the corners of the mouth are raised or pulled down, etc., and then the expression category can be determined. For example, when it is determined that the eyes are getting smaller, the corners of the eyes are raised, and the corners of the mouth are raised according to the face image of the current frame and at least one frame of the face image, the expression category can be determined to be happy.

Step 104 , fuse the three-dimensional model with the special effect model corresponding to the expression category to obtain a three-dimensional model after special effect processing.

In this embodiment of the present application, the correspondence between the expression category and the special effect model may be pre-stored. For example, when the expression category is sad, the special effect model may be tears, when the expression category is anger, the special effect model may be flame, and when the expression category is anger When the category is nervous, the special effect model can be cold sweat and so on.

The special effect model can be stored in the material library of the application program of the electronic device, and different special effect models are stored in the material library, or the application program on the electronic device can also download the new special effect model from the server in real time. The special effect model can be stored in the material library.

Optionally, after the expression category is determined, the above-mentioned correspondence may be queried to obtain a special effect model matching the expression category, and then the three-dimensional model and the special effect model corresponding to the expression category are fused to obtain a three-dimensional model after special effects processing.

As a possible implementation, in order to improve the display effect of the 3D model after special effects processing and enhance the realism of the 3D model after special effects are added, in this embodiment of the present application, the angle of the special effect model relative to the 3D model can be adjusted, so that the 3D model can be The angle of the model and the special effect model are matched, and then the special effect model is rendered and mapped to the 3D model.

Further, after obtaining the 3D model after special effect processing, the 3D model after special effect processing can be displayed on the display interface of the electronic device, so that it is convenient for the user to intuitively know the 3D model after special effect processing.

The three-dimensional model-based special effects processing method according to the embodiment of the present application obtains the collected two-dimensional face image and the depth information corresponding to the face image, and then performs three-dimensional reconstruction of the face according to the depth information and the face image. , to obtain a three-dimensional model corresponding to the face, then identify the expression category corresponding to the two-dimensional face image, and finally fuse the three-dimensional model with the special effect model corresponding to the expression category to obtain a three-dimensional model after special effect processing. As a result, the user does not need to manually switch between different special effect models, the automation degree of special effect addition is improved, and the fun and playability of the user in the special effect addition process are improved. In addition, according to the expression made by the user, the corresponding special effect model is determined, so that the special effect model and the three-dimensional model are integrated, which can enhance the realism added by the special effect, and make the processed effect more natural.

As a possible implementation, in order to improve the efficiency and accuracy of facial expression category recognition, in this application, after obtaining at least one frame of face image collected before the current frame, only after determining at least one frame of face image Only when the difference between the position of each key point and the position of each key point in the face image of the current frame is greater than the threshold, the expression category corresponding to the current frame is identified. The above process will be described in detail below with reference to FIG. 2 .

FIG. 2 is a schematic flowchart of a special effect processing method based on a three-dimensional model provided by Embodiment 2 of the present application.

Referring to FIG. 2, on the basis of the embodiment shown in FIG. 1, step 103 may specifically include the following sub-steps:

Step 201: Identify the position of each key point in the face image of the current frame.

Specifically, the position of each key point in the face image of the current frame can be identified based on the key point identification technology.

Step 202 , for at least one frame of face image collected before the current frame, identify the position of each key point in the at least one frame of face image.

In this embodiment of the present application, at least one frame of face image collected before the current frame may be acquired, and based on the key point recognition technology, the position of each key point in the at least one frame of face image may be determined.

Step 203: Determine if the difference between the positions of the key points in the at least one frame of face image and the positions of the key points in the face image of the current frame is greater than a threshold, if yes, go to Step 204, otherwise, go to Step 205.

The threshold value may be preset in the built-in program of the electronic device, or the threshold value may also be set by the user, which is not limited.

Step 204, identifying the expression category corresponding to the current frame.

In the embodiment of the present application, when the difference between the position of each key point in at least one frame of face image and the position of each key point in the face image of the current frame is greater than the threshold, it indicates that the expressions continuously made by the user have a relatively high value. At this time, the user may want to add special effects. Therefore, the expression category corresponding to the current frame can be further identified, thereby triggering the subsequent steps of adding special effects. For the specific identification process, please refer to the execution process of step 103 in the above embodiment. I won't go into details here.

In step 205, no processing is performed.

In the embodiment of the present application, when the difference between the position of each key point in at least one frame of face image and the position of each key point in the face image of the current frame is not greater than the threshold, it indicates that the continuous expressions made by the user are not If there is a big change, at this time, the user may not want to add special effects, so you can do nothing.

As a possible implementation, referring to FIG. 3 , on the basis of the embodiment shown in FIG. 1 , step 104 may specifically include the following sub-steps:

Step 301: Acquire a corresponding special effect model according to the expression category.

As a possible implementation, the correspondence between the expression category and the special effect model can be stored in advance. After the expression category corresponding to the face image is determined, the above-mentioned correspondence can be queried to obtain the special effect model matching the expression category. The operation is simple. , and is easy to implement.

Step 302: Adjust the angle of the special effect model relative to the three-dimensional model, so that the angles of the three-dimensional model and the special effect model match.

It should be noted that, before mapping the special effect model to the 3D model, the angle of the special effect model relative to the 3D model needs to be adjusted so that the angles of the 3D model and the special effect model match. For example, when the special effect model is tears, when the face in the 3D model is facing the screen or facing the screen sideways, the display effect of tears is different. Therefore, it is necessary to adjust the rotation angle of the special effect model according to the deflection angle of the face, so as to make The angle of the 3D model and the special effect model are matched, thereby improving the effect of subsequent special effects processing.

As a possible implementation manner, angle parameters applicable to different special effect models may be predetermined, wherein the angle parameters may be a fixed value or a value range (for example, [-45°, 45°]), which is not limited . After determining the special effect model corresponding to the expression category, you can query the applicable angle parameters of the special effect model, and then rotate the special effect model, so that the first connection between the preset target key points in the special effect model and the preset reference key in the 3D model The angle between the second line connecting the points conforms to the angle parameter.

Step 303 , according to the special effect model, in the three-dimensional model, query the corresponding key points to be mapped.

It should be understood that different special effect models have different key points to be mapped in the 3D model. For example, when the special effect model is a tear, the tear generally flows from the key point corresponding to the corner of the eye to the key point corresponding to the nose wing, and then flows from the key point corresponding to the nose wing to the key point corresponding to the corner of the mouth. Therefore, you can move the eye corner to the nose wing. , and the corresponding key points from the nose wing to the corner of the mouth as the key points to be mapped. Or, when the special effect model is cold sweat, the cold sweat generally flows from the key point corresponding to the forehead to the key point corresponding to the end of the eyebrow, and then flows from the key point corresponding to the end of the eyebrow to the key point corresponding to the cheek, and then from the key point corresponding to the cheek. The points flow down to the key points corresponding to the chin. Therefore, the key points corresponding to the forehead to the eyebrow end, the eyebrow end to the cheek, and the cheek to the chin can be used as the key points to be mapped.

As a possible implementation, the corresponding relationship between different special effect models and key points to be mapped can be established in advance. After determining the special effect model corresponding to the expression category, the above-mentioned corresponding relationship can be queried to obtain the corresponding relationship in the 3D model. The key point to be mapped corresponding to the special effect model.

Step 304: In the three-dimensional model, the area where the key points to be mapped corresponding to the special effect model are located is used as the area to be mapped.

In the embodiment of the present application, when the special effect models are different, the areas to be mapped are different. When determining the key points to be mapped corresponding to the special effect models in the 3D model, the area where the corresponding key points to be mapped can be used as the area to be mapped.

Step 305 , deform the special effect model according to the to-be-mapped area of the three-dimensional model, so that the deformed special-effect model covers the to-be-mapped area.

In the embodiment of the present application, after determining the area to be mapped in the 3D model, the special effect model may be deformed, so that the deformed special effect model covers the area to be mapped, thereby improving the effect of special effects processing.

Step 306 , after rendering the special effect model, map it to the three-dimensional model.

In order to match the special effect model with the three-dimensional model, thereby ensuring the display effect of the three-dimensional model after special effect processing, in the present application, the special effect model may be rendered and then mapped to the three-dimensional model.

As a possible implementation, the special effect model can be rendered according to the light effect of the three-dimensional model, so that the light effect of the rendered special effect model matches the three-dimensional model, thereby improving the display effect of the three-dimensional model after special effect processing.

In order to realize the above embodiments, the present application also proposes a special effect processing device based on a three-dimensional model.

FIG. 4 is a schematic structural diagram of a special effect processing apparatus based on a three-dimensional model provided by Embodiment 4 of the present application.

As shown in FIG. 4 , the three-dimensional model-based special effect processing apparatus 100 includes: an acquisition module 110 , a reconstruction module 120 , an identification module 130 , and a fusion module 140 . in,

The acquiring module 110 is configured to acquire the collected two-dimensional face image and depth information corresponding to the face image.

The reconstruction module 120 is configured to perform three-dimensional reconstruction of the human face according to the depth information and the human face image, so as to obtain a three-dimensional model corresponding to the human face.

The identification module 130 is used for identifying the expression category corresponding to the two-dimensional face image.

The fusion module 140 is configured to fuse the three-dimensional model with the special effect model corresponding to the expression category to obtain a three-dimensional model after special effect processing.

Further, in a possible implementation manner of the embodiment of the present application, referring to FIG. 5 , on the basis of the embodiment shown in FIG. 4 , the three-dimensional model-based special effect processing apparatus 100 may further include:

As a possible implementation manner, the identification module 130 includes:

The first identification sub-module 131 is used to identify the position of each key point in the face image of the current frame.

The second identification sub-module 132 is configured to identify the position of each key point in the at least one frame of face image collected before the current frame.

The third identification sub-module 133 is configured to identify the expression category corresponding to the current frame if the difference between the positions of the key points in the at least one frame of the face image and the positions of the key points in the face image of the current frame is greater than a threshold .

As a possible implementation manner, the fusion module 140 includes:

The obtaining sub-module 141 is used to obtain the corresponding special effect model according to the expression category.

The adjustment sub-module 142 is used to adjust the angle of the special effect model relative to the three-dimensional model, so that the angles of the three-dimensional model and the special effect model match.

As a possible implementation manner, the adjustment sub-module 142 is specifically used for: querying the applicable angle parameters of the special effect model; Let the included angle between the second connecting lines of the reference key point conform to the angle parameter.

The texture sub-module 143 is used to render the special effect model and then map the texture to the 3D model.

As a possible implementation manner, the texture sub-module 143 is specifically configured to: render the special effect model according to the light effect of the three-dimensional model.

The deformation sub-module 144 is used to deform the special effect model according to the to-be-mapped area of the three-dimensional model after rendering the special-effect model and before mapping the three-dimensional model, so that the deformed special-effect model covers the to-be-mapped area.

The query sub-module 145 is configured to query the corresponding key points to be mapped in the three-dimensional model according to the special effect model before deforming the special effect model according to the to-be-mapped area of the three-dimensional model.

The processing sub-module 146 is configured to, in the three-dimensional model, use the area where the key points to be mapped corresponding to the special effect model are located as the area to be mapped.

It should be noted that, the foregoing explanations on the embodiment of the three-dimensional model-based special effect processing method are also applicable to the three-dimensional model-based special effect processing apparatus 100 of this embodiment, and are not repeated here.

The three-dimensional model-based special effect processing device according to the embodiment of the present application obtains the collected two-dimensional face image and the depth information corresponding to the face image, and then performs three-dimensional reconstruction of the face according to the depth information and the face image. , to obtain a three-dimensional model corresponding to the face, then identify the expression category corresponding to the two-dimensional face image, and finally fuse the three-dimensional model with the special effect model corresponding to the expression category to obtain a three-dimensional model after special effect processing. As a result, the user does not need to manually switch between different special effect models, the automation degree of special effect addition is improved, and the fun and playability of the user in the special effect addition process are improved. In addition, according to the expression made by the user, the corresponding special effect model is determined, so that the special effect model and the three-dimensional model are integrated, which can enhance the realism added by the special effect, and make the processed effect more natural.

In order to realize the above-mentioned embodiments, the present application also proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, the above-mentioned embodiments of the present application provide 3D model-based special effects processing method.

In order to realize the above-mentioned embodiments, the present application also proposes a computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the three-dimensional model-based special effect processing proposed by the foregoing embodiments of the present application is realized. method.

FIG. 6 is a schematic diagram of the internal structure of the electronic device 200 in one embodiment. The electronic device 200 includes a processor 220 , a memory 230 , a display 240 and an input device 250 connected through a system bus 210 . The memory 230 of the electronic device 200 stores an operating system and computer-readable instructions. The computer-readable instructions can be executed by the processor 220 to implement the three-dimensional model-based special effect processing method of the embodiment of the present application. The processor 220 is used to provide computing and control capabilities to support the operation of the entire electronic device 200 . The display 240 of the electronic device 200 may be a liquid crystal display screen or an electronic ink display screen, etc., and the input device 250 may be a touch layer covered on the display 240, or a button, a trackball or a touchpad provided on the casing of the electronic device 200, It can also be an external keyboard, trackpad or mouse, etc. The electronic device 200 may be a mobile phone, a tablet computer, a notebook computer, a personal digital assistant, or a wearable device (eg, a smart bracelet, a smart watch, a smart helmet, and smart glasses).

Those skilled in the art can understand that the structure shown in FIG. 6 is only a schematic diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device 200 to which the solution of the present application is applied. The specific electronic device 200 may include more or fewer components than shown, or combine certain components, or have a different arrangement of components.

In order to clearly illustrate the electronic device provided in this embodiment, please refer to FIG. 7 , which provides an image processing circuit of an embodiment of the present application, and the image processing circuit may be implemented by hardware and/or software components.

It should be noted that FIG. 7 is a schematic diagram of an image processing circuit as a possible implementation manner. For the convenience of description, only various aspects related to the embodiments of the present application are shown.

As shown in FIG. 7 , the image processing circuit specifically includes: an image unit 310 , a depth information unit 320 and a processing unit 330 . in,

The image unit 310 is used for outputting a two-dimensional face image.

The depth information unit 320 is used for outputting depth information.

In this embodiment of the present application, the image unit 310 may be used to obtain a two-dimensional face image, and the depth information unit 320 may be used to obtain depth information corresponding to the face image.

The processing unit 330 is electrically connected to the image unit 310 and the depth information unit 320, respectively, and is used to perform three-dimensional image processing on the face according to the two-dimensional face image obtained by the image unit 310 and the corresponding depth information obtained by the depth information unit 320. Reconstruction is performed to obtain a three-dimensional model corresponding to the face, identify the expression category corresponding to the two-dimensional face image, and fuse the three-dimensional model with the special effect model corresponding to the expression category to obtain a three-dimensional model after special effect processing.

In this embodiment of the present application, the two-dimensional face image obtained by the image unit 310 may be sent to the processing unit 330, and the depth information corresponding to the face image obtained by the depth information unit 320 may be sent to the processing unit 330, and the processing unit 330 may Face image and depth information, perform three-dimensional reconstruction of the face to obtain a three-dimensional model corresponding to the face, identify the expression category corresponding to the two-dimensional face image, and fuse the three-dimensional model with the special effect model corresponding to the expression category. The 3D model after special effects processing is obtained. For the specific implementation process, reference may be made to the explanation of the three-dimensional model-based special effect processing method in the embodiments of FIG. 1 to FIG. 3 , which will not be repeated here.

Further, as a possible implementation manner of the present application, referring to FIG. 8 , on the basis of the embodiment shown in FIG. 7 , the image processing circuit may further include:

As a possible implementation manner, the image unit 310 may specifically include: an electrically connected image sensor 311 and an image signal processing (Image Signal Processing, ISP for short) processor 312 . in,

The image sensor 311 is used to output raw image data.

The ISP processor 312 is configured to output a face image according to the original image data.

In this embodiment of the present application, the raw image data captured by the image sensor 311 is first processed by the ISP processor 312 , and the ISP processor 312 analyzes the raw image data to capture image statistics that can be used to determine one or more control parameters of the image sensor 311 Information, including face images in YUV format or RGB format. The image sensor 311 may include a color filter array (such as a Bayer filter) and a corresponding photosensitive unit. The image sensor 311 may acquire the light intensity and wavelength information captured by each photosensitive unit, and provide information that can be processed by the ISP processor 312. A set of raw image data. After the ISP processor 312 processes the original image data, a face image in YUV format or RGB format is obtained, and sent to the processing unit 330 .

When processing the original image data, the ISP processor 312 can process the original image data pixel by pixel in various formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 312 may perform one or more image processing operations on the raw image data, collecting statistical information about the image data. Among them, the image processing operations can be performed with the same or different bit depth precision.

As a possible implementation manner, the depth information unit 320 includes an electrically connected structured light sensor 321 and a depth map generation chip 322 . in,

The structured light sensor 321 is used to generate an infrared speckle pattern.

The depth map generation chip 322 is configured to output depth information according to the infrared speckle map; the depth information includes a depth map.

In the embodiment of the present application, the structured light sensor 321 projects speckle structured light to the subject, acquires the structured light reflected by the subject, and images the infrared speckle image according to the reflected structured light. The structured light sensor 321 sends the infrared speckle image to the depth map generation chip 322, so that the depth map generation chip 322 determines the morphological change of the structured light according to the infrared speckle image, and then determines the depth of the subject based on this to obtain the depth map (Depth Map), the depth map indicates the depth of each pixel in the infrared speckle map. The depth map generation chip 322 sends the depth map to the processing unit 330 .

As a possible implementation manner, the processing unit 330 includes: a CPU 331 and a GPU (Graphics Processing Unit, graphics processor) 332 that are electrically connected. in,

The CPU 331 is configured to align the face image and the depth map according to the calibration data, and output a three-dimensional model corresponding to the face according to the aligned face image and the depth map.

The GPU 332 is configured to identify the expression category corresponding to the two-dimensional face image, and fuse the three-dimensional model with the special effect model corresponding to the expression category to obtain a three-dimensional model after special effect processing.

In the embodiment of the present application, the CPU 331 obtains the face image from the ISP processor 312, and obtains the depth map from the depth map generation chip 322. Combined with the pre-obtained calibration data, the face image and the depth map can be aligned to determine the person The depth information corresponding to each pixel in the face image. Furthermore, the CPU 331 performs three-dimensional reconstruction of the human face according to the depth information and the human face image, so as to obtain a three-dimensional model corresponding to the human face.

The CPU 331 sends the three-dimensional model corresponding to the human face to the GPU 332, so that the GPU 332 executes the three-dimensional model-based special effect processing method described in the foregoing embodiment according to the three-dimensional model corresponding to the human face, so as to realize the fusion of the three-dimensional model and the special effect model corresponding to the expression category. , and get the 3D model after special effects processing.

Further, the image processing circuit may further include: a display unit 340 .

The display unit 340 is electrically connected to the GPU 332, and is used for displaying according to the three-dimensional model processed by special effects.

Specifically, the special effect processed three-dimensional model obtained by the GPU 332 can be displayed on the display 340 .

Optionally, the image processing circuit may further include: an encoder 350 and a memory 360 .

In the embodiment of the present application, the three-dimensional model after the special effect processing obtained by the GPU 332 may also be encoded by the encoder 350 and stored in the memory 360, where the encoder 350 may be implemented by a coprocessor.

In one embodiment, the memory 360 may be multiple or divided into multiple storage spaces, and the image data processed by the GPU 312 may be stored in a dedicated memory, or may be stored in a dedicated storage space, and may include DMA (Direct Memory Access). access) feature. Memory 360 may be configured to implement one or more frame buffers.

The above process will be described in detail below with reference to FIG. 8 .

As shown in FIG. 8, raw image data captured by image sensor 311 is first processed by ISP processor 312, which analyzes the raw image data to capture image statistics that can be used to determine one or more control parameters of image sensor 311 The information, including the face image in YUV format or RGB format, is sent to CPU331.

As shown in FIG. 8 , the structured light sensor 321 projects speckle structured light to the subject, acquires the structured light reflected by the subject, and forms an infrared speckle image according to the reflected structured light. The structured light sensor 321 sends the infrared speckle image to the depth map generation chip 322, so that the depth map generation chip 322 determines the morphological change of the structured light according to the infrared speckle image, and then determines the depth of the subject based on this to obtain the depth map (Depth Map). The depth map generation chip 322 sends the depth map to the CPU 331 .

The CPU 331 obtains the face image from the ISP processor 312, obtains the depth map from the depth map generation chip 322, and combines the pre-obtained calibration data to align the face image with the depth map, thereby determining each pixel in the face image. corresponding depth information. Furthermore, the CPU 331 performs three-dimensional reconstruction of the human face according to the depth information and the human face image, so as to obtain a three-dimensional model corresponding to the human face.

The CPU331 sends the three-dimensional model corresponding to the human face to the GPU332, so that the GPU332 executes the special effect processing method based on the three-dimensional model as described in the foregoing embodiment according to the three-dimensional model of the human face, so that the three-dimensional model and the special effect model corresponding to the expression category are fused, The 3D model after special effects processing is obtained. The special effect processed three-dimensional model obtained by the GPU 332 may be displayed on the display 340 and/or stored in the memory 360 after being encoded by the encoder 350 .

For example, the following are the steps of implementing the control method using the processor 220 in FIG. 6 or using the image processing circuit (specifically CPU331 and GPU332) in FIG. 8 :

The CPU331 obtains a two-dimensional face image and the depth information corresponding to the face image; The expression category corresponding to the face image; the GPU 332 fuses the three-dimensional model with the special effect model corresponding to the expression category to obtain a three-dimensional model after special effect processing.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present application, "plurality" means at least two, such as two, three, etc., unless expressly and specifically defined otherwise.

Any process or method description in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing custom logical functions or steps of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use with, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or apparatus) or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or apparatus. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wiring (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), Read Only Memory (ROM), Erasable Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, followed by editing, interpretation, or other suitable medium as necessary process to obtain the program electronically and then store it in computer memory.

It should be understood that various parts of this application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one of the following techniques known in the art, or a combination thereof: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Those skilled in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program can be stored in a computer-readable storage medium. When executed, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations to the present application. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (7)

1. A special effect processing method based on a three-dimensional model is characterized by comprising the following steps:

acquiring an acquired two-dimensional face image and depth information corresponding to the face image;

performing three-dimensional reconstruction on a human face according to the depth information and the human face image to obtain a three-dimensional model corresponding to the human face, wherein the key points extracted from the depth information and the key points extracted from the color information corresponding to the human face image are subjected to registration and fusion processing to generate the three-dimensional model;

identifying an expression category corresponding to the two-dimensional face image;

acquiring a corresponding special effect model according to the expression category;

adjusting the angle of the special effect model relative to the three-dimensional model so that the three-dimensional model and the special effect model are angle-matched;

inquiring the corresponding relation between different pre-established special effect models and key points of a to-be-pasted drawing, and acquiring the key points of the to-be-pasted drawing corresponding to the special effect models in the three-dimensional model;

in the three-dimensional model, taking the region where the key point of the to-be-pasted picture corresponding to the special effect model is located as the region of the to-be-pasted picture;

according to the area to be pasted of the three-dimensional model, deforming the special effect model so that the deformed special effect model covers the area to be pasted;

and after rendering the special effect model, pasting a picture to the three-dimensional model.

2. The special effects processing method according to claim 1, wherein the identifying an expression class corresponding to the two-dimensional face image includes:

identifying the position of each key point in the face image of the current frame;

identifying the position of each key point in at least one frame of face image collected before the current frame;

and if the difference between the position of each key point in the at least one frame of face image and the position of each key point in the face image of the current frame is greater than a threshold value, identifying the expression type corresponding to the current frame.

3. The special effects processing method of claim 1, wherein the adjusting the angle of the special effects model relative to the three-dimensional model to angularly match the three-dimensional model and the special effects model comprises:

inquiring the angle parameter applicable to the special effect model;

and rotating the special effect model to enable an included angle between a first connecting line of a preset target key point in the special effect model and a second connecting line of a preset reference key point in the three-dimensional model to accord with the angle parameter.

4. The special effect processing method according to claim 1, wherein the rendering the special effect model includes:

and rendering the special effect model according to the light effect of the three-dimensional model.

5. A three-dimensional model-based special effects processing apparatus, the apparatus comprising:

the acquisition module is used for acquiring the acquired two-dimensional face image and depth information corresponding to the face image;

the reconstruction module is used for performing three-dimensional reconstruction on a human face according to the depth information and the human face image so as to obtain a three-dimensional model corresponding to the human face, wherein the key points extracted from the depth information and the key points extracted from the color information corresponding to the human face image are subjected to registration and fusion processing so as to generate the three-dimensional model;

the recognition module is used for recognizing the expression type corresponding to the two-dimensional face image;

the fusion module is used for acquiring a corresponding special effect model according to the expression category; adjusting the angle of the special effect model relative to the three-dimensional model so that the three-dimensional model and the special effect model are angle-matched; inquiring the corresponding relation between different pre-established special effect models and key points of a to-be-pasted drawing, and acquiring the key points of the to-be-pasted drawing corresponding to the special effect models in the three-dimensional model; in the three-dimensional model, taking the region where the key point of the to-be-pasted picture corresponding to the special effect model is located as the region of the to-be-pasted picture; according to the area to be pasted of the three-dimensional model, deforming the special effect model so that the deformed special effect model covers the area to be pasted; and after rendering the special effect model, pasting a picture to the three-dimensional model.

6. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, which when executed by the processor implements the three-dimensional model based special effects processing method of any of claims 1 to 4.

7. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the three-dimensional model-based special effects processing method according to any one of claims 1 to 4.

Images