CN114155569B - Cosmetic progress detection method, device, equipment and storage medium - Google Patents

Abstract

The present application provides a method, device, device and storage medium for detecting makeup progress. The method includes: acquiring a real-time makeup video of a user currently performing a specific makeup look; and determining whether the user is performing a specific makeup look according to an initial frame image and a current frame image of the real-time makeup video. of the current makeup progress. The present application compares the current frame image of the user's makeup process with the initial frame image to determine the makeup progress. The makeup progress can be detected only through image processing, and the accuracy of makeup progress detection is very high. For makeup processes such as highlighting, contouring, blush, foundation, concealer, eye shadow, eyeliner, and eyebrows, the user's makeup progress can be detected in real time. There is no need to use a deep learning model, the calculation amount is small, the cost is low, the processing pressure of the server is reduced, the efficiency of makeup progress detection is improved, and the real-time requirements of makeup progress detection can be met.

Description

A makeup progress detection method, device, equipment and storage medium

technical field

The present application belongs to the technical field of image processing, and in particular relates to a makeup progress detection method, device, equipment and storage medium.

Background technique

Makeup has become an essential part of many people's daily life. There are many steps in the makeup process. If the makeup progress can be fed back to the user in real time, it will greatly reduce the energy consumption of the user and save the user's makeup time.

At present, there are some related technologies that use deep learning models to provide functions such as virtual makeup testing, skin color detection, and personalized product recommendations. These functions require pre-collecting a large number of face pictures to train deep learning models.

However, face pictures are user privacy data, and it is difficult to collect huge face pictures. Moreover, model training requires a large amount of computing resources and is expensive. The accuracy of the model is inversely proportional to the real-time performance. Make-up progress detection needs to capture the user's face and facial information in real time to determine the user's current make-up progress. The real-time requirements are very high. The deep learning model that can meet the real-time requirements is not accurate enough. high.

Contents of the invention

This application proposes a makeup progress detection method, device, equipment, and storage medium. The current makeup progress is determined through the difference between the initial frame image and the current frame image. The accuracy of makeup progress detection is high, and the calculation amount is small. The cost is low, the processing pressure of the server is reduced, the efficiency of makeup progress detection is improved, and the real-time requirement of makeup progress detection can be met.

The embodiment of the first aspect of the present application proposes a makeup progress detection method, including:

Obtain real-time makeup videos of users currently performing specific makeup looks;

According to the initial frame image and the current frame image of the real-time makeup video, the current makeup progress of the user performing the specific makeup is determined.

In some embodiments of the present application, the specific makeup includes high-gloss makeup or grooming makeup; and according to the initial frame image and the current frame image of the real-time makeup video, it is determined that the current time for the user to perform the specific makeup is Makeup progress, including:

Obtain at least one target makeup area corresponding to the specific makeup;

Acquiring a first target area image corresponding to the specific makeup from the initial frame image according to the target makeup area, and acquiring a second target area image corresponding to the specific makeup from the current frame image;

According to the first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined.

In some embodiments of the present application, the acquiring the first target area image corresponding to the specific makeup from the initial frame image according to the target makeup area includes:

Detecting the first human face key point corresponding to the initial frame image;

Acquiring a face region image corresponding to the initial frame image according to the first face key point;

According to the first human face key point and the target makeup area, a first target area image corresponding to the specific makeup is acquired from the human face area image.

In some embodiments of the present application, the extraction of the first target area image corresponding to the specific makeup from the face area image according to the first human face key point and the target makeup area includes :

Determining one or more target key points located on the contour of the area corresponding to the target makeup area in the human face area image from the first human face key points;

Generate a mask image corresponding to the face area image according to the target key points corresponding to the target makeup area;

An AND operation is performed on the mask image and the face area image to obtain a first target area image corresponding to the specific makeup.

In some embodiments of the present application, the generating the mask image corresponding to the face area image according to the target key points corresponding to the target makeup area includes:

If there are multiple target key points corresponding to the target makeup area, then according to the target key points, determine each edge coordinate of the target makeup area in the face area image; The pixel values of all the pixel points in the area surrounded by two edge coordinates are all modified to preset values, so as to obtain the mask area corresponding to the target makeup area;

If there is one target key point corresponding to the target makeup area, then take the target key point as the center, draw an ellipse area of preset size, and modify the pixel values of all pixels in the ellipse area to A preset value to obtain a mask area corresponding to the target makeup area;

Modifying the pixel values of all pixels outside the mask area to zero to obtain a mask image corresponding to the face area image.

In some embodiments of the present application, the specific makeup includes blush makeup; the determining the current makeup progress of the user performing the specific makeup according to the initial frame image and the current frame image of the real-time makeup video includes :

Obtain at least one target makeup area corresponding to the specific makeup;

Generate a beauty mask map according to the target makeup area;

According to the makeup mask image, the initial frame image and the current frame image, determine the current makeup progress corresponding to the current frame image.

In some embodiments of the present application, the determining the current makeup progress corresponding to the current frame image according to the beauty mask image, the initial frame image and the current frame image includes:

Taking the makeup mask image as a reference, acquiring a makeup-on first target area image from the initial frame image, and acquiring a makeup-on second target area image from the current frame image;

According to the first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined.

In some embodiments of the present application, the specific makeup includes eyeliner makeup; the determining the current makeup progress of the user performing the specific makeup according to the initial frame image and the current frame image of the real-time makeup video includes:

Acquiring the beauty mask corresponding to the initial frame image and the current frame image;

According to the initial frame image, simulate and generate the result image after eyeliner makeup is completed;

According to the makeup mask map, the result image, the initial frame image and the current frame image, determine the current makeup progress corresponding to the current frame image.

In some embodiments of the present application, the determining the current makeup progress corresponding to the current frame image according to the beauty mask image, the result image, the initial frame image and the current frame image includes :

Taking the makeup mask image corresponding to the initial frame image as a reference, acquiring the image of the first target area of makeup from the initial frame image;

Acquiring a makeup-on second target area image from the current frame image according to the makeup mask image corresponding to the current frame image;

Acquiring the third target area image of eyeliner makeup according to the result image;

According to the first target area image, the second target area image and the third target area image, the current makeup progress corresponding to the current frame image is determined.

In some embodiments of the present application, the determining the current makeup progress corresponding to the current frame image according to the first target area image, the second target area image and the third target area image includes:

Converting the first target area image, the second target area image, and the third target area image into images containing a saturation channel in the HLS color space, respectively;

According to the converted first target area image, the second target area image and the third target area image, the current makeup progress corresponding to the current frame image is determined.

In some embodiments of the present application, the current makeup progress corresponding to the current frame image is determined according to the converted first target area image, the second target area image and the third target area image ,include:

respectively calculating a first average pixel value corresponding to the converted first target area image, a second average pixel value corresponding to the second target area image, and a third average pixel value corresponding to the third target area image;

calculating a first difference between a second average pixel value and the first average pixel value, and calculating a second difference between the third average pixel value and the first average pixel value;

Calculate the ratio between the first difference and the second difference to obtain the current makeup progress corresponding to the current frame image.

In some embodiments of the present application, before determining the current makeup progress corresponding to the current frame image according to the first target area image, the second target area image and the third target area image, further include:

performing alignment processing on the first target area image and the second target area image;

Perform alignment processing on the first target area image and the third target area image.

In some embodiments of the present application, the aligning the first target area image and the second target area image includes:

Perform binarization processing on the first target area image and the second target area image respectively to obtain a first binarized mask image corresponding to the first target area image and a corresponding second target area image The second binarized mask image of ;

Perform an AND operation on the first binarized mask image and the second binarized mask image to obtain a second mask corresponding to the intersection area of the first target area image and the second target area image film image.

In some embodiments of the present application, the aligning the first target area image and the second target area image further includes:

Acquiring the face area image corresponding to the initial frame image and the face area image corresponding to the result image;

performing an AND operation on the second mask image and the face area image corresponding to the initial frame image to obtain a new first target area image corresponding to the initial frame image;

An AND operation is performed on the second mask image and the face area image corresponding to the result image to obtain a new second target area image corresponding to the result image.

In some embodiments of the present application, the acquiring the beauty mask corresponding to the initial frame image and the current frame image includes:

Obtain the eyeliner style map selected by the user;

If the user's eye state in the initial frame image is the eye-open state, then obtain the eye-opening pattern map corresponding to the eyeliner pattern map; determine the eye-opening pattern map as the beauty mask corresponding to the initial frame image code map;

If the eye state of the user in the initial frame image is the eye-closed state, then obtain the eye-closed pattern map corresponding to the eyeliner pattern map, and determine the eye-closed pattern map as the beauty makeup corresponding to the initial frame image mask map.

In some embodiments of the present application, the specific makeup includes eye shadow makeup; the determining the current makeup progress of the user performing the specific makeup according to the initial frame image and the current frame image of the real-time makeup video includes:

Obtain the eye shadow mask map;

According to each target makeup area of eye shadow makeup, respectively split the beauty mask map corresponding to each target makeup area from the eye shadow mask map;

The current makeup progress corresponding to the current frame image is determined according to the initial frame image, the current frame image, and the beauty mask map corresponding to each target makeup area.

In some embodiments of the present application, the current makeup corresponding to the current frame image is determined according to the initial frame image, the current frame image, and the beauty mask map corresponding to each of the target makeup areas. progress, including:

Taking the beauty mask image corresponding to each target makeup area as a reference, acquiring a first target area image corresponding to each target makeup area from the initial frame image;

Respectively using the beauty mask image corresponding to each target makeup area as a reference, acquiring a second target area image corresponding to each target makeup area from the current frame image;

According to the first target area image and the second target area image corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is determined.

In some embodiments of the present application, the determining the current makeup progress corresponding to the current frame image according to the first target area image and the second target area image corresponding to each of the target makeup areas includes:

Respectively converting the first target area image and the second target area image corresponding to each target makeup area into images containing preset single-channel components under the HLS color space;

According to the converted first target area image and second target area image corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is determined.

In some embodiments of the present application, the determining the current makeup progress corresponding to the current frame image according to the converted first target area image and the second target area image corresponding to each target makeup area includes :

Respectively calculate the difference absolute value of the preset single-channel component corresponding to the pixels in the same position in the first target area image corresponding to the same target makeup area after conversion and in the second target area image;

Count the number of pixels whose absolute value of the difference corresponding to each target makeup area satisfies the preset makeup completion condition;

Calculating the ratio between the number of pixels corresponding to each target makeup area and the total number of pixels in the corresponding target makeup area to obtain the makeup progress corresponding to each target makeup area;

According to the makeup progress corresponding to each target makeup area and the preset weight corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is calculated.

In some embodiments of the present application, the first target area image is obtained from the initial frame image with reference to the beauty mask image, including:

Detecting the first human face key point corresponding to the initial frame image;

Acquiring a face region image corresponding to the initial frame image according to the first face key point;

Taking the makeup mask image as a reference, the image of the first target area for applying makeup is acquired from the face area image.

In some embodiments of the present application, the acquisition of the makeup-on first target area image from the face area image with reference to the beauty mask image includes:

Converting the beauty mask image and the face area image into a binary image respectively;

Perform an AND operation on the binarized image corresponding to the beauty mask map and the binarized image corresponding to the face area image to obtain the correspondence between the intersection area of the beauty mask map and the face area image The first mask image of ;

An AND operation is performed on the face area image corresponding to the first mask image and the initial frame image to obtain a first target area image.

In some embodiments of the present application, before performing the AND operation on the binarized image corresponding to the beauty mask map and the binarized image corresponding to the face region image, it further includes:

According to the standard human face key points corresponding to the beauty mask map, one or more first positioning points located on the outline of each makeup area in the beauty mask map are determined;

Determining a second anchor point corresponding to each of the first anchor points from the face area image according to the first face key point;

Stretching is performed on the beauty mask image, and each of the first positioning points is stretched to a position corresponding to each of the second positioning points.

In some embodiments of the present application, the acquisition of the makeup-on first target area image from the face area image with reference to the beauty mask image includes:

Splitting the beauty mask image into multiple sub-mask images, each of which includes at least one target makeup area;

converting each of the sub-mask images and the face region image into a binary image;

performing an AND operation on the binarized image corresponding to each of the sub-masks and the binarized image corresponding to the face region image to obtain a respective sub-mask image corresponding to each of the sub-masks;

performing an AND operation on each of the sub-mask images and the face region images corresponding to the initial frame image to obtain a plurality of sub-target region images corresponding to the initial frame image;

Merging the multiple sub-target area images into a first target area image corresponding to the initial frame image.

In some embodiments of the present application, before performing the AND operation on the binarized image corresponding to each of the sub-mask images and the binarized image corresponding to the face region image, it further includes:

According to the standard human face key points corresponding to the beauty mask map, determine one or more first positioning points located on the contour of the target makeup area in the first sub-mask map, the first sub-mask map It is any sub-mask in the plurality of sub-masks;

Determining a second anchor point corresponding to each of the first anchor points from the face area image according to the first face key point;

Stretching is performed on the first sub-mask image, and each of the first positioning points is stretched to a position corresponding to each of the second positioning points.

In some embodiments of the present application, the specific makeup includes eyebrow makeup; the determining the current makeup progress of the user performing the specific makeup according to the initial frame image and the current frame image of the real-time makeup video includes:

Obtaining a first target area image corresponding to the eyebrows from the initial frame image, and acquiring a second target area image corresponding to the eyebrows from the current frame image;

According to the first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined.

In some embodiments of the present application, the obtaining the first target region image corresponding to the eyebrows from the initial frame image includes:

Detecting the first human face key point corresponding to the initial frame image;

Acquiring a face region image corresponding to the initial frame image according to the first face key point;

According to the eyebrow key points included in the first human face key points, a first target area image corresponding to eyebrows is acquired from the human face area image.

In some embodiments of the present application, according to the eyebrow key points included in the first human face key points, intercepting the first target area image corresponding to the eyebrows from the face area image includes:

Interpolating the eyebrow key points between the eyebrows and eyebrow peaks included in the first human face key points to obtain a plurality of interpolation points;

Intercepting all eyebrow key points between the eyebrow head and the eyebrow peak and the closed area formed by the connection of the plurality of interpolation points from the image of the human face area, obtaining a partial eyebrow image between the eyebrow head and the eyebrow peak;

Intercepting the closed area formed by connecting all eyebrow key points between the eyebrow peak and the eyebrow tail from the image of the human face region, obtaining a partial eyebrow image between the eyebrow peak and the eyebrow tail;

The partial eyebrow image between the eyebrow head and the eyebrow peak and the partial eyebrow image between the eyebrow peak and the eyebrow tail are spliced into a first target region image corresponding to the eyebrows.

In some embodiments of the present application, the determining the current makeup progress corresponding to the current frame image according to the first target area image and the second target area image includes:

respectively converting the first target area image and the second target area image into images containing preset single-channel components in HSV color space;

According to the converted first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined.

In some embodiments of the present application, the determining the current makeup progress corresponding to the current frame image according to the converted first target area image and the second target area image includes:

Calculating the absolute value of the difference between the preset single-channel components corresponding to pixels at the same position in the converted first target area image and the second target area image;

Count the number of pixels whose absolute value of the corresponding difference satisfies the preset makeup completion condition;

Calculate the ratio between the counted number of pixels and the total number of pixels in all target makeup areas in the first target area image to obtain the current makeup progress corresponding to the current frame image.

In some embodiments of the present application, before determining the current makeup progress corresponding to the current frame image according to the first target area image and the second target area image, further includes:

Performing binarization processing on the first target area image and the second target area image respectively, to obtain a first binarized mask image corresponding to the first target area image and corresponding to the second target area image The second binarized mask image of ;

Perform an AND operation on the first binarized mask image and the second binarized mask image to obtain a second mask corresponding to the intersection area of the first target area image and the second target area image film image;

Acquiring the face area image corresponding to the initial frame image and the face area image corresponding to the current frame image;

performing an AND operation on the second mask image and the face area image corresponding to the initial frame image to obtain a new first target area image corresponding to the initial frame image;

An AND operation is performed on the second mask image and the face area image corresponding to the current frame image to obtain a new second target area image corresponding to the current frame image.

In some embodiments of the present application, before determining the current makeup progress corresponding to the current frame image, it also includes:

Boundary erosion processing is performed on makeup areas in the first target area image and the second target area image respectively.

In some embodiments of the present application, the specific makeup includes foundation makeup; the determining the current makeup progress of the user performing the specific makeup according to the initial frame image and the current frame image of the real-time makeup video includes:

According to the initial frame image, simulate and generate a result image after completing the specific makeup;

Obtaining the overall image brightness corresponding to the initial frame image, the result image, and the current frame image respectively;

Obtaining the brightness of the face area corresponding to the initial frame image, the result image, and the current frame image respectively;

The current makeup progress corresponding to the current frame image is determined according to the overall image brightness and face area brightness corresponding to the initial frame image, the result image and the current frame image respectively.

In some embodiments of the present application, said obtaining the overall image brightness corresponding to the initial frame image, the result image, and the current frame image respectively includes:

respectively converting the initial frame image, the result image and the current frame image into grayscale images;

Calculating the grayscale average value of the pixels in the grayscale images respectively corresponding to the converted initial frame image, the resultant image, and the current frame image;

The respective grayscale average values corresponding to the initial frame image, the result image and the current frame image are respectively determined as the respective overall image luminances corresponding to the initial frame image, the result image and the current frame image.

In some embodiments of the present application, said obtaining the brightness of the face area corresponding to the initial frame image, the result image and the current frame image respectively includes:

Respectively acquire the face area images corresponding to the initial frame image, the result image and the current frame image;

respectively converting the face area images corresponding to the initial frame image, the result image, and the current frame image into face grayscale images;

Calculate the gray average value of the pixels in the face grayscale images corresponding to the initial frame image, the result image and the current frame image respectively, to obtain the initial frame image, the result image and the current frame image The brightness of the face area corresponding to each frame image.

In some embodiments of the present application, the current makeup corresponding to the current frame image is determined according to the overall image brightness and face area brightness corresponding to the initial frame image, the result image, and the current frame image. progress, including:

Determine the first environmental change brightness corresponding to the current frame image according to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the current frame image;

Determine the second environmental change brightness corresponding to the result image according to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the result image;

According to the brightness of the first environmental change, the brightness of the second environmental change, the brightness of the face area corresponding to the initial frame image, the brightness of the face area corresponding to the current frame image, and the face area corresponding to the result image Brightness, to determine the current makeup progress corresponding to the current frame image.

In some embodiments of the present application, the current frame image is determined according to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the current frame image The brightness of the corresponding first environment changes, including:

Calculating the difference between the overall image brightness corresponding to the initial frame image and the brightness of the corresponding face area to obtain the ambient brightness of the initial frame image;

Calculating the difference between the overall image brightness corresponding to the current frame image and the corresponding face area brightness to obtain the ambient brightness of the current frame image;

Determine the absolute value of the difference between the ambient brightness of the current frame image and the ambient brightness of the initial frame image as the first ambient change brightness corresponding to the current frame image.

In some embodiments of the present application, the changing brightness according to the first environment, the changing brightness according to the second environment, the brightness of the face area corresponding to the initial frame image, and the face area corresponding to the current frame image Brightness, the brightness of the face area corresponding to the result image, determine the current makeup progress corresponding to the current frame image, including:

According to the brightness of the first environmental change, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the current frame image, determine the makeup brightness change value corresponding to the current frame image;

According to the brightness of the second environmental change, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the result image, determine the change value of the makeup brightness corresponding to the result image;

Calculate the ratio of the makeup brightness change value corresponding to the current frame image to the makeup brightness change value corresponding to the result image to obtain the current makeup progress corresponding to the current frame image.

In some embodiments of the present application, the current frame image is determined according to the brightness of the first environmental change, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the current frame image The corresponding brightness change value of makeup includes:

calculating the difference between the brightness of the face area corresponding to the current frame image and the brightness of the face area corresponding to the initial frame image to obtain a total brightness change value corresponding to the current frame image;

Calculate the difference between the total brightness change value and the first environmental change brightness to obtain the makeup brightness change value corresponding to the current frame image.

In some embodiments of the present application, the method also includes:

If the brightness of the first environmental change is greater than a preset threshold, then determine the makeup progress corresponding to the previous frame image as the current makeup progress corresponding to the current frame image;

Sending first prompt information to the user's terminal, where the first prompt information is used to prompt the user to return to the brightness environment corresponding to the initial frame image to apply makeup.

In some embodiments of the present application, the specific makeup includes concealer makeup; the determination of the current makeup progress of the user performing the specific makeup according to the initial frame image and the current frame image of the real-time makeup video includes:

Respectively acquiring facial blemish information corresponding to the initial frame image and the current frame image;

According to the facial blemish information corresponding to the initial frame image and the facial blemish information corresponding to the current frame image, calculate the facial blemish difference value between the current frame image and the initial frame image;

If the facial blemish difference value is greater than a preset threshold, then calculate the current makeup progress corresponding to the current frame image according to the facial blemish difference value and the facial blemish information corresponding to the initial frame image;

If the facial blemish difference value is less than or equal to the preset threshold, then obtain the result image after the user completes the concealment and makeup, and determine according to the initial frame image, the result image and the current frame image The current makeup progress corresponding to the current frame image.

In some embodiments of the present application, the facial blemish information includes blemish categories and corresponding blemish numbers; the facial blemish information corresponding to the initial frame image and the facial blemish information corresponding to the current frame image , calculating the facial blemish difference value between the current frame image and the initial frame image, including:

Calculating the difference between the number of blemishes corresponding to the initial frame image and the number of blemishes corresponding to the current frame image under each blemish category;

Calculate the sum of the differences corresponding to each blemish category, and use the obtained sum as the facial blemish difference value between the current frame image and the initial frame image.

In some embodiments of the present application, the calculating the current makeup progress corresponding to the current frame image according to the facial blemish difference value and the facial blemish information corresponding to the initial frame image includes:

Calculating the sum of the number of blemishes corresponding to each blemish category in the facial blemish information corresponding to the initial frame image to obtain the total number of blemishes;

Calculate the ratio between the facial blemish difference value and the total number of blemishes, and use the ratio as the current makeup progress corresponding to the current frame image.

In some embodiments of the present application, the acquisition of the result image after the user completes the concealment and makeup, and determining the corresponding frame image of the current frame image according to the initial frame image, the result image and the current frame image Current makeup progress, including:

According to the initial frame image, simulate and generate the result image after the user completes the concealer and makeup;

Respectively acquire the face area images corresponding to the initial frame image, the result image and the current frame image;

According to the face area images corresponding to the initial frame image, the result image and the current frame image, the current makeup progress corresponding to the current frame image is determined.

In some embodiments of the present application, the determining the current makeup progress corresponding to the current frame image according to the face area images corresponding to the initial frame image, the result image and the current frame image includes:

Respectively converting the face area images corresponding to the initial frame image, the result image and the current frame image into images containing saturation channels in the HLS color space;

Calculating the smoothing factors corresponding to the respective face area images of the initial frame image, the result image, and the current frame image after conversion through a preset filtering algorithm;

According to the respective smoothing factors corresponding to the initial frame image, the result image and the current frame image, the current makeup progress corresponding to the current frame image is determined.

In some embodiments of the present application, the determining the current makeup progress corresponding to the current frame image according to the respective smoothing factors of the initial frame image, the result image and the current frame image includes:

calculating a first difference between the smoothing factor corresponding to the current frame image and the smoothing factor corresponding to the initial frame image;

calculating a second difference between the smoothing factor corresponding to the result image and the smoothing factor corresponding to the initial frame image;

Calculate the ratio between the first difference and the second difference, and use the ratio as the current makeup progress corresponding to the current frame image.

In some embodiments of the present application, the respectively obtaining the facial blemish information corresponding to the initial frame image and the current frame image includes:

Respectively acquire the face area images corresponding to the initial frame image and the current frame image;

The number of blemishes corresponding to each blemish category in the face area image corresponding to the initial frame image and the current frame image is respectively detected by the preset skin detection model, and the respective corresponding blemishes of the initial frame image and the current frame image are obtained. facial blemish information.

In some embodiments of the present application, the acquiring the face area image corresponding to the initial frame image includes:

Performing rotation correction on the initial frame image and the first human face key point according to the first human face key point corresponding to the initial frame image;

According to the corrected first human face key point, intercepting an image containing a human face area from the corrected initial frame image;

The image containing the face area is scaled to a preset size to obtain a face area image corresponding to the initial frame image.

In some embodiments of the present application, performing rotation correction on the initial frame image and the first facial key point according to the first human face key point includes:

Determine left eye center coordinates and right eye center coordinates respectively according to the left eye key point and right eye key point included in the first human face key point;

Determine the rotation angle and rotation center point coordinates corresponding to the initial frame image according to the left eye center coordinates and the right eye center coordinates;

Rotation correction is performed on the initial frame image and the first human face key point according to the rotation angle and the coordinates of the rotation center point.

In some embodiments of the present application, according to the corrected first human face key point, intercepting an image containing a human face area from the corrected initial frame image includes:

According to the corrected key points of the first human face, image interception is performed on the human face area included in the corrected initial frame image.

In some embodiments of the present application, the image interception of the face area contained in the corrected initial frame image according to the corrected first human face key point includes:

Determining a minimum abscissa value, a minimum ordinate value, a maximum abscissa value, and a maximum ordinate value from the corrected first face key point;

According to the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value, determine the clipping frame corresponding to the face area in the corrected initial frame image;

According to the clipping frame, an image including the face area is clipped from the rectified initial frame image.

In some embodiments of the present application, the method also includes:

Enlarging the interception frame by a preset multiple;

According to the enlarged clipping frame, an image including the face area is clipped from the corrected initial frame image.

In some embodiments of the present application, the method also includes:

According to the size of the image including the face area and the preset size, zooming and translation processing is performed on the corrected first key points of the face.

In some embodiments of the present application, the method also includes:

Detecting whether both the initial frame image and the current frame image only contain the face image of the same user;

If yes, then perform the operation of determining the current makeup progress of the user performing the specific makeup look;

If not, send prompt information to the user's terminal, where the prompt information is used to prompt the user to keep only the face of the same user appearing in the real-time makeup video.

The embodiment of the second aspect of the present application provides a makeup progress detection device, including:

The video acquisition module is used to acquire the real-time makeup video of the user currently performing a specific makeup look;

The makeup progress determination module is configured to determine the current makeup progress of the user performing the specific makeup look according to the initial frame image and the current frame image of the real-time makeup video.

The embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor runs the computer program to Implement the method described in the first aspect above.

The embodiment of the fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

The technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

In the embodiment of the present application, the progress of the makeup is determined by comparing the current frame image of the user's makeup process with the initial frame image. The makeup progress can be detected only through image processing, and the accuracy of makeup progress detection is very high. It can detect the user's makeup progress in real time for the makeup process such as highlights, contouring, blush, foundation, concealer, eye shadow, eyeliner, and eyebrows. There is no need to use a deep learning model, the amount of calculation is small, the cost is low, the processing pressure on the server is reduced, the efficiency of makeup progress detection is improved, and the real-time requirements of makeup progress detection can be met.

Additional aspects and advantages of the present application will be set forth in part in the description which follows, and in part will be obvious from the description which follows, or may be learned by practice of the present application.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the application. Also throughout the drawings, the same reference numerals are used to designate the same components.

In the attached picture:

Fig. 1 shows a flowchart of a makeup progress detection method provided by an embodiment of the present application;

Fig. 2 shows a flow chart of a makeup progress detection method for detecting makeup such as highlights and grooming provided by an embodiment of the present application;

Fig. 3 shows a schematic diagram of a display interface for a user to select a target makeup area displayed by a client provided by an embodiment of the present application;

FIG. 4 shows a schematic diagram of solving the rotation angle of an image provided by an embodiment of the present application;

FIG. 5 shows a schematic diagram of two coordinate system transformations provided by an embodiment of the present application;

Fig. 6 shows a schematic diagram of a module flow chart of a makeup progress detection method for detecting highlights, trimming and other makeup provided by an embodiment of the present application;

Fig. 7 shows a flow chart of a makeup progress detection method for detecting blush and other makeup provided by an embodiment of the present application;

Fig. 8 shows another schematic diagram of the display interface displayed by the client for the user to select a makeup area provided by an embodiment of the present application;

Fig. 9 shows a schematic diagram of a module flow chart of a makeup progress detection method for detecting blush and other makeup provided by an embodiment of the present application;

Fig. 10 shows a flowchart of a makeup progress detection method for detecting eyeliner makeup provided by an embodiment of the present application;

Fig. 11 shows a schematic diagram of a module flow chart of a makeup progress detection method for detecting eyeliner makeup provided by an embodiment of the present application;

Fig. 12 shows a flowchart of a makeup progress detection method for detecting eye shadow makeup provided by an embodiment of the present application;

Fig. 13 shows a schematic diagram of a module flow chart of a makeup progress detection method for detecting eye shadow makeup provided by an embodiment of the present application;

Fig. 14 shows a schematic structural diagram of a makeup progress detection device for detecting eyeshadow makeup provided by an embodiment of the present application;

Fig. 15 shows a flowchart of a makeup progress detection method for detecting eyebrow makeup provided by an embodiment of the present application;

Fig. 16 shows a schematic diagram of a module flow chart of a makeup progress detection method for detecting eyebrow makeup provided by an embodiment of the present application;

Fig. 17 shows a schematic structural diagram of a makeup progress detection device for detecting eyebrow makeup provided by an embodiment of the present application;

Fig. 18 shows a flow chart of a makeup progress detection method for detecting foundation, loose powder and other makeup provided by an embodiment of the present application;

Fig. 19 shows another flow chart of a makeup progress detection method for detecting foundation, loose powder and other makeup provided by an embodiment of the present application;

Fig. 20 shows a schematic diagram of a module flow chart of a makeup progress detection method for detecting foundation, loose powder and other makeup provided by an embodiment of the present application;

Fig. 21 shows a flowchart of a makeup progress detection method for detecting concealer makeup provided by an embodiment of the present application;

Fig. 22 shows a schematic diagram of a module flow chart of a makeup progress detection method for detecting concealer makeup provided by an embodiment of the present application;

Fig. 23 shows a flowchart of a makeup color recognition method provided by an embodiment of the present application;

Fig. 24 shows a schematic structural view of a makeup color recognition device provided by an embodiment of the present application;

Fig. 25 shows a flowchart of an image processing method for removing facial blemishes provided by an embodiment of the present application;

Figure 26(a) shows the user's face image, and Figure 26(b) shows the blemish texture map corresponding to the face image shown in Figure 26(a);

Fig. 27 shows a schematic structural diagram of an image processing device for removing facial blemishes provided by an embodiment of the present application;

Fig. 28 shows a schematic structural diagram of a makeup progress detection device provided by an embodiment of the present application;

Fig. 29 shows a schematic structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 30 shows a schematic diagram of a storage medium provided by an embodiment of the present application.

Detailed ways

Exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided for thorough understanding of the application and to fully convey the scope of the application to those skilled in the art.

It should be noted that, unless otherwise specified, technical terms or scientific terms used in this application shall have the usual meanings understood by those skilled in the art to which this application belongs.

A makeup progress detection method, device, device and storage medium according to the embodiments of the present application will be described below with reference to the accompanying drawings.

At present, there are some virtual makeup trial functions in related technologies, which can be applied to sales counters or mobile phone application software. Face recognition technology is used to provide users with virtual makeup trial services, which can match various makeup looks and display facial fit in real time. In addition, face skin detection services are provided, but these services can only solve the needs of users to choose cosmetics that suit them, or choose a skin care plan that suits them. Based on these services, users can choose highlighting/contour makeup products that suit them, but they cannot display the progress of makeup, and cannot meet the needs of users for real-time makeup. In related technologies, there are still some functions that use deep learning models to provide virtual makeup testing, skin color detection, and personalized product recommendations. These functions require pre-collecting a large number of face pictures to train deep learning models. However, face pictures are user privacy data, and it is difficult to collect huge face pictures. Moreover, model training requires a large amount of computing resources and is expensive. The accuracy of the model is inversely proportional to the real-time performance. Make-up progress detection needs to capture the user's face and facial information in real time to determine the user's current make-up progress. The real-time requirements are very high. The deep learning model that can meet the real-time requirements is not accurate enough. high.

Based on this, an embodiment of the present application provides a makeup progress detection method, which compares the current frame image of the user's makeup process with the initial frame image (ie, the first frame image) to determine the makeup progress. The makeup progress can be detected only through image processing, and the accuracy of makeup progress detection is very high. It can detect the user's makeup progress in real time for the makeup process such as highlights, contouring, blush, foundation, concealer, eye shadow, eyeliner, and eyebrows. There is no need to use a deep learning model, the amount of calculation is small, the cost is low, the processing pressure on the server is reduced, the efficiency of makeup progress detection is improved, and the real-time requirements of makeup progress detection can be met.

Referring to Figure 1, the method specifically includes the following steps:

Step 101: Obtain a real-time makeup video of the user currently performing a specific makeup.

Step 102: According to the initial frame image and the current frame image of the real-time makeup video, determine the current makeup progress of the user performing a specific makeup look.

Specific makeup can be highlight makeup, trimming makeup, blush makeup, foundation makeup, concealer makeup, eye shadow makeup, eyeliner makeup, eyebrow makeup, etc. The process of detecting the makeup progress of different makeup looks will be described in detail below.

Embodiment one

An embodiment of the present application provides a method for detecting makeup progress, which is used to detect the makeup progress corresponding to highlight makeup, trimming makeup or any other makeup that may produce light and dark changes. Referring to Fig. 2, this embodiment specifically comprises the following steps:

Step 201: Obtain at least one target makeup area corresponding to a specific makeup look and a real-time makeup video of the user currently performing the specific makeup look.

The execution subject of this embodiment of the application is a server. A client that is compatible with the makeup progress detection service provided by the server is installed on the terminal such as the user's mobile phone or computer. When the user needs to use the makeup progress detection service, the user opens the client on the terminal, and the client displays all target makeup areas corresponding to the specific makeup. The specific makeup can include highlight makeup or trimming makeup, etc. The target makeup area may include a forehead area, a nose bridge area, a nose tip area, a left cheek area, a right cheek area, a chin area, and the like. The user selects one or more target makeup areas where he needs to apply specific makeup from the multiple displayed target makeup areas.

As an example, the client may display all target makeup areas corresponding to a specific makeup in the form of text selection items. The display interface shown in Figure 3, the display interface includes multiple text selection items corresponding to the target makeup area and the submit button, the user can click to select the text selection item of the target makeup area that he needs, and then click Submit button. After the client detects the submit instruction triggered by the submit button, it acquires one or more target makeup areas selected by the user from the display interface.

As another example, the client may display a face image on which all target makeup areas corresponding to a specific makeup look are identified. The user selects the desired target makeup area from the displayed face image by clicking. The display interface displayed by the client can include a face image and a submit button. Each target makeup area corresponding to a specific makeup can be drawn in the face image by a solid line. After the user clicks the target makeup area that he needs, The outline of the selected target makeup area can be displayed in a preset color (such as red or yellow), or all of the selected target makeup area can be displayed in a preset color. After the user selects the target makeup area he needs, he clicks the submit button. After the client detects the submit command triggered by the submit button, it acquires one or more target makeup areas selected by the user from the displayed face image.

After the user's terminal obtains one or more target makeup areas selected by the user through any of the above methods, it sends the area identification information of the one or more target makeup areas to the server. The area identification information may include the Identification information such as the name or number of each target makeup area. The server receives the area identification information sent by the user's terminal, and determines at least one target makeup application area corresponding to the specific makeup look selected by the user according to the area identification information.

Through the above method, the user can customize the area where he needs to apply a specific makeup look, which can meet the individual needs of different users for specific makeup looks such as highlighting and contouring.

In some other embodiments of the present application, instead of the user selecting the target makeup application area, multiple target makeup application areas corresponding to a specific makeup look may be pre-configured in the server. In this way, when the user needs to detect the makeup progress, there is no need to obtain the target makeup area from the user's terminal, which saves bandwidth, simplifies user operations, and shortens processing time.

The display interface of the above-mentioned client is also provided with a video upload interface. When it is detected that the user clicks on the video upload interface, the camera device of the terminal is called to take a video of the user's makeup. Perform makeup operations for the specific makeup above. The user's terminal transmits the captured makeup video to the server in the form of a video stream. The server receives each frame of the makeup video transmitted by the user's terminal.

In the embodiment of the present application, the server takes the received first frame image as the initial frame image, and uses the initial frame image as a reference to compare the current makeup progress of the specific makeup corresponding to each subsequently received frame image. Since the processing method for each subsequent image frame is the same, the embodiment of the present application uses the current frame image received at the current moment as an example to illustrate the process of makeup progress detection.

After the server obtains at least one target makeup area corresponding to a specific makeup look and the initial frame image and current frame image of the user's makeup video through this step, the server determines the user's current makeup progress through the following steps 202 and 203 .

Step 202: Obtain a first target area image corresponding to the specific makeup from the initial frame image according to the target makeup area, and acquire a second target area image corresponding to the specific makeup from the current frame image.

The server specifically obtains the first target area image corresponding to the initial frame image through the following steps S1-S3, including:

S1: Detect the first face key point corresponding to the initial frame image.

The server is configured with a pre-trained detection model for detecting key points of the face, and the interface service of key point detection of the face is provided through the detection model. After the server obtains the initial frame image of the user's makeup video, it calls the interface service of face key point detection, and uses the detection model to identify all face key points of the user's face in the initial frame image. In order to distinguish the face key points corresponding to the current frame image, all face key points corresponding to the initial frame image are referred to as first human face key points in the embodiment of the present application. All face key points corresponding to the current frame image are referred to as second face key points.

Among them, the recognized key points of the human face include key points on the outline of the user's face and key points of the mouth, nose, eyes, eyebrows and other parts. The number of recognized face key points may be 106.

S2: According to the key points of the first face, obtain the face area image corresponding to the initial frame image.

The server specifically obtains the face area image corresponding to the initial frame image through the following steps S20-S22, including:

S20: According to the key points of the first face, perform rotation correction on the initial frame image and the key points of the first face.

When the user shoots a makeup video through the terminal, the pose angle of the face in each frame of image cannot be guaranteed to be the same. In order to improve the accuracy of the comparison between the current frame image and the initial frame image, it is necessary to Perform rotation correction so that the lines connecting the faces and eyes in each frame of image after correction are on the same horizontal line, so as to ensure that the pose angle of the face in each frame of image is the same, and avoid errors in makeup progress detection due to different pose angles Bigger problem.

Specifically, according to the left eye key point and the right eye key point included in the first human face key point, the left eye center coordinates and the right eye center coordinates are respectively determined. All the left-eye key points in the left-eye area and all the right-eye key points in the right-eye area are determined from the first face key points. Take the mean value of the abscissa of all the determined key points of the left eye, and take the mean of the ordinate of all the key points of the left eye, combine the mean value of the abscissa and the mean value of the ordinate corresponding to the left eye into a coordinate, and determine the coordinate is the coordinates of the center of the left eye. Determine the coordinates of the center of the right eye in the same manner.

Then, according to the left eye center coordinates and the right eye center coordinates, the rotation angle and the rotation center point coordinates corresponding to the initial frame image are determined. As shown in Figure 4, the horizontal difference dx and the vertical difference dy of the two coordinates are calculated according to the left eye center coordinates and the right eye center coordinates, and the line connecting the two eyes between the left eye center coordinates and the right eye center coordinates is calculated length d. According to the length d of the connecting line between the two eyes, the horizontal difference dx and the vertical difference dy, the angle θ between the connecting line between the two eyes and the horizontal direction is calculated, and the angle θ is the corresponding rotation angle of the initial frame image. Then, the center point coordinates of the line connecting the two eyes are calculated according to the left eye center coordinates and the right eye center coordinates, and the midpoint coordinates are the rotation center point coordinates corresponding to the initial frame image.

Perform rotation correction on the initial frame image and the key points of the first face according to the calculated rotation angle and the coordinates of the rotation center point. Specifically, the rotation angle and the coordinates of the rotation center point are input into a preset function used to calculate the rotation matrix of the picture, and the preset function may be the function cv2.getRotationMatrix2D() in OpenCV. Obtain the rotation matrix corresponding to the initial frame image by calling this preset function. Then calculate the product of the initial frame image and the rotation matrix to obtain the rectified initial frame image. The operation of correcting the initial frame image by using the rotation matrix can also be done by calling the function cv2.warpAffine() in OpenCV.

For the key points of the first human face, each key point of the first human face needs to be corrected one by one to correspond to the corrected initial frame image. When correcting the key points of the first face one by one, two transformations of the coordinate system are required. The first time the coordinate system with the origin at the upper left corner of the initial frame image is transformed into the coordinate system with the origin at the lower left corner. The second time Further transform the coordinate system with the lower left corner as the origin into the coordinate system with the above-mentioned rotation center point coordinates as the coordinate origin, as shown in FIG. 5 . After two coordinate system conversions, the conversion of the following formula (1) is performed on each key point of the first human face, and the rotation correction of the key points of the first human face can be completed.

In formula (1), x ₀ and y ₀ are respectively the abscissa and ordinate of the key point of the first face before rotation correction, and x and y are respectively the abscissa and ordinate of the key point of the first face after rotation correction, θ is the above-mentioned rotation angle.

The corrected initial frame image and the key points of the first face are based on the entire image, which not only contains the user’s face information, but also includes other redundant image information, so the corrected image needs to be corrected through the following steps S21 The image is cropped for the face area.

S21: According to the corrected first key point of the human face, intercept an image including the human face area from the corrected initial frame image.

Firstly, the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value are determined from the corrected first face key points. Then according to the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value, determine the clipping frame corresponding to the face area in the corrected initial frame image. Specifically, the minimum abscissa value and the minimum ordinate value form a coordinate point, and the coordinate point is used as the upper left vertex of the interception frame corresponding to the face area. The maximum abscissa value and the maximum ordinate value form another coordinate point, and this coordinate point is used as the vertex of the lower right corner of the interception frame corresponding to the face area. Determine the position of the interception frame in the corrected initial frame image according to the above-mentioned upper left corner vertex and lower right corner vertex, and intercept the image located in the interception frame from the corrected initial frame image, that is, intercept the image containing the face area .

In other embodiments of the present application, in order to ensure that all of the user's face area is intercepted, and to avoid incomplete interception resulting in a large error in the follow-up makeup progress detection, the above interception frame can also be enlarged by a preset multiple, The preset multiple can be 1.15 or 1.25 and so on. The embodiment of the present application does not limit the specific value of the preset multiple, and the preset multiple can be set according to requirements in practical applications. After the aforementioned clipping frame is enlarged by a preset multiple, the image located in the enlarged clipping frame is clipped from the corrected initial frame image, so as to clip an image including the complete face area of the user.

S22: Scale the image including the face area to a preset size to obtain the face area image corresponding to the initial frame image.

After the image including the user's face area is intercepted from the initial frame image in the above manner, the image including the face area is scaled to a preset size to obtain a face area image corresponding to the initial frame image. The preset size may be 390×390 or 400×400 or the like. The embodiment of the present application does not limit the specific value of the preset size, which can be set according to requirements in practical applications.

In order to make the first human face key point adapt to the scaled human face area image, after the captured image containing the human face area is scaled to a preset size, it is also necessary to use the size of the image containing the human face area before scaling and the The preset size is used to zoom and translate the corrected key points of the first human face. Specifically, according to the size of the image containing the face area before zooming and the preset size to which the image needs to be zoomed, determine the translation direction and translation distance of each first face key point, and then according to each first face key point The translation direction and the translation distance corresponding to the point, the translation operation is performed on each first human face key point respectively, and the coordinates of each first human face key point after translation are recorded.

Obtain the face area image from the initial frame image in the above way, and make the first face key point adapt to the obtained face area image through operations such as rotation correction and translation scaling, and then obtain the face area image through the following step S3. The image area corresponding to the target makeup area is extracted from the area image.

In some other embodiments of the present application, before step S3 is performed, Gaussian filtering may be performed on the image of the face area to remove noise in the image of the face area. Specifically, Gaussian filtering is performed on the face area image corresponding to the initial frame image according to a Gaussian kernel of a preset size.

The Gaussian kernel of Gaussian filter is the key parameter of Gaussian filter processing. If the Gaussian kernel is too small, the filtering effect cannot be achieved. If the Gaussian kernel is too large, although the noise information in the image can be filtered out, the image will be smoothed at the same time. Useful information in . In this embodiment of the present application, a Gaussian kernel with a preset size is selected, and the preset size may be 9×9. In addition, another set of parameters of the Gaussian filter function, sigmaX and sigmaY, are set to 0. After Gaussian filtering, the image information will be smoother, thereby improving the accuracy of subsequent acquisition of makeup progress.

The face region image is obtained in the above manner, or after the Gaussian filtering is performed on the face region image, the target region image corresponding to the specific makeup is extracted from the face region image through step S3.

S3: According to the key points of the first face and the target makeup area, extract the first target area image corresponding to the specific makeup from the face area image.

Specific makeup such as highlight makeup or trimming makeup is a makeup method that applies makeup to a fixed area of the face. For example, highlighting makeup or trimming makeup is usually applied to the forehead area, left cheek area, right cheek area, nose bridge area, Highlight or contour specific areas such as the nose tip area or the chin area. Therefore, it is possible to directly extract these specific areas that require specific makeup from the face area image, thereby avoiding the interference of invalid areas on the makeup progress detection of specific makeup, and improving the accuracy of makeup progress detection.

The server specifically obtains the first target area image through the following steps S30-S32, including:

S30: Determine one or more target key points located on the contour of the area corresponding to the target makeup area in the face area image from the first human face key points.

Firstly, the area position corresponding to the target makeup area is determined from the human face area image, and then one or more first human face key points located at the position of the area are determined from the first human face key points. The determined one or more first human face key points are determined as one or more target key points on the contour of the area corresponding to the target makeup area.

For each target makeup area corresponding to the specific makeup acquired in step 201, the target key points corresponding to each target makeup area are respectively determined in the above-mentioned manner.

S31: Generate a mask image corresponding to the face area image according to the target key points corresponding to the target makeup area.

Since one or more target makeup areas corresponding to the specific makeup are obtained in step 201, the specific operations when generating the mask image corresponding to the face area image are different for different target makeup areas. For each target makeup area corresponding to a specific makeup, first determine whether the number of target key points corresponding to the target makeup area is multiple, usually forehead area, left cheek area, right cheek area, nose bridge area, and chin area The number of target key points corresponding to the target makeup area is multiple. For these target makeup areas, each edge coordinate of the target makeup area in the face area image is determined according to each target key point corresponding to the target makeup area. Specifically, the coordinates of the target key points are appropriately shifted to the left, right, or up and down by some pixels according to the preset offset rules, so as to obtain the edge coordinates corresponding to the target makeup area.

In the embodiment of the present application, preset offset rules corresponding to different target makeup areas are pre-configured in the server, and the preset offset rules specify the offset direction and the number of offset pixels of the target key points corresponding to the target makeup area .

For example, for the forehead area, the corresponding target key points include the key points of the face contour on both sides of the forehead and the key points of the face at the hairline. The preset offset rule corresponding to the forehead area can specify that the hairline The target key point at the line is shifted downward, the target key point on the left side of the forehead is shifted to the right, and the target key point on the right side of the forehead is shifted to the left, and the number of pixels to be shifted by the target key point is specified as 4 pixels. Each target key point corresponding to the forehead area is offset according to the preset offset rule, and the coordinate point after the offset of each target key point is the edge coordinate corresponding to the forehead area.

For another example, for the bridge of the nose area, the corresponding target key points may include multiple face key points arranged vertically on the bridge of the nose, and the preset offset rule corresponding to the bridge of the nose area may stipulate that each target key point arranged vertically is to the right Offset and offset to the left by a certain number of pixels, the number of pixels to be offset by the target key point can be specified as 3 pixels or 4 pixels, etc. Each target key point corresponding to the nose bridge area is offset according to the preset offset rule, and the coordinate point after the offset of each target key point is the edge coordinate corresponding to the nose bridge area.

After the edge coordinates corresponding to the target makeup area including multiple target key points are obtained by the above method, the pixel values of all pixels in the area surrounded by each edge coordinate are modified to preset values by the area filling function, and The mask area corresponding to the target makeup area. Wherein, the preset value may be 255.

If it is determined that the number of target key points corresponding to a target makeup area is one, usually the number of target key points corresponding to the target makeup area such as the nose tip area is one. For a target makeup area with only one target key point, draw an elliptical area of preset size with the target key point as the center. The pixel values of all the pixels in the ellipse area are modified to preset values to obtain the mask area corresponding to the target makeup area.

After obtaining the mask area corresponding to each target makeup area corresponding to the specific makeup in the above way, modify the pixel values of all pixels outside the mask area to zero, that is, obtain the mask corresponding to the face area image image.

S32: Perform an AND operation on the mask image and the face area image to obtain a first target area image corresponding to the specific makeup.

After obtaining the mask image of the face region image pair, an AND operation is performed on the mask image and the face region image, that is, an AND operation is performed on the pixel values of the pixel points with the same coordinates in the mask image and the face region image respectively. Since only the pixel values of the pixel points in the target makeup area corresponding to the specific makeup look in the mask image have preset values, the pixel values of other pixel points are all zero. Therefore, after performing the AND operation, only the pixel values at the positions corresponding to the target makeup area in the obtained first target area image are not zero, and the pixel values at other positions are all zero.

In the embodiment of the present application, the pixel value of the pixel points in the target makeup area in the mask image is 255, then after performing an AND operation on the mask image and the face area image, the target makeup area in the first target area image is obtained The pixel value of the pixel in is the pixel value of the pixel in the target makeup area in the face area image. It is equivalent to extracting the image of the target makeup area corresponding to the specific makeup from the face area image.

The color space of the first target area image corresponding to the initial frame image obtained by the above method is the RGB color space. For highlight makeup, it is mainly to brighten the skin color, while for repairing makeup, it is mainly to darken the skin color to form shadows, making the facial features more three-dimensional. For specific makeup looks such as highlight makeup and trimming makeup, it is mainly the change of brightness. The RGB color space is the most commonly used color space in daily life. The linear combination of red, green and blue color components is used to represent the color. The image information obtained in the natural environment is sensitive to brightness. After changing the brightness information of the same picture, RGB All three components will change accordingly. The HSV color space is composed of three components: Hue (hue), Saturation (saturation) and Value (brightness). If the brightness information of the picture changes, only the brightness component will change at this time, while the values of the hue and saturation components will not change. Significant changes occur, compared to the RGB color space, the HSV color space can separate the brightness information.

Therefore, in the embodiment of the present application, after obtaining the first target area image corresponding to the initial frame image, the color space of the first target area image is converted from the RGB color space to the HSV color space. Then, the channel component corresponding to the specific makeup is separated from the converted HSV color space of the first target area image to obtain a single-channel first target area image including only the channel component.

For example, the channel component corresponding to specific makeup such as highlight makeup and trimming makeup is the brightness component, while the channel component corresponding to other specific makeup that changes the color of the face is the hue component, etc.

Convert the image of the first target area to HSV color space conversion, and separate the channel components corresponding to the specific makeup through channel separation, so as to facilitate the subsequent judgment on whether the target makeup area has completed the makeup process of the specific makeup.

For the current frame image, the processing process is the same as that of the above-mentioned initial frame image, the second face key point corresponding to the current frame image is detected, and the face area image corresponding to the current frame image is intercepted according to the second face key point, according to The second face key point image generates a mask image corresponding to the current frame image. According to the mask image, the second target area image corresponding to the specific makeup is extracted from the face area image corresponding to the current frame image. And convert the second target area image to HSV color space, and separate the channel components corresponding to the specific makeup. The details of the processing operation for the current frame image are the same as the above-mentioned processing details for the initial frame image, and will not be repeated here.

After the first target area image corresponding to the initial frame image and the second target area image corresponding to the current frame image are obtained in the above manner, the makeup progress corresponding to the current frame image is determined in the following manner.

Step 203: According to the first target area image and the second target area image, determine the current makeup progress corresponding to the current frame image.

Specifically, the absolute values of the differences of the channel components corresponding to the pixel points in the same position in the first target area image and the second target area image are respectively calculated. For example, if the specific makeup is high-gloss makeup or trimming makeup, the absolute value of the difference in brightness component between pixels with the same coordinates in the first target area image and the second target area image is calculated.

According to the absolute value of the difference corresponding to each pixel point, the area of the area where the specific makeup has been completed is determined. Specifically, the number of pixels whose absolute value of the corresponding difference satisfies the preset makeup completion condition corresponding to the specific makeup look is counted. Wherein, when the specific makeup is a high-gloss makeup, the preset makeup completion condition is that the absolute value of the difference corresponding to the pixel points is greater than the first preset threshold, and the first preset threshold can be 11 or 12, etc. When the specific makeup is cosmetic makeup, the preset makeup completion condition is that the absolute value of the difference corresponding to the pixel points is less than the second preset threshold, and the second preset threshold can be 7 or 8, etc.

Determining the counted number of pixels satisfying the preset makeup completion condition as the area of the area where the specific makeup has been completed. The total number of all pixels in all target makeup areas in the first target area image or the second target area image is counted, and the total number is determined as the total area corresponding to all target makeup areas. Then calculate the ratio between the area area where the specific makeup has been completed and the total area corresponding to the target makeup area, and determine the ratio as the current makeup progress of the specific makeup corresponding to the user.

In some other embodiments of the present application, in order to further improve the accuracy of makeup progress detection, further alignment is performed on the target makeup areas in the first target area image and the second target area image. Specifically, binarization is performed on the first target region image and the second target region image of a single channel that only contain the above-mentioned channel components, that is, the target makeup region in the first target region image and the second target region image The values of the above-mentioned channel components corresponding to the pixels are all modified to 1, and the values of the above-mentioned channel components of the pixels at other positions are all modified to 0. A first binarized mask image corresponding to the first target area image and a second binarized mask image corresponding to the second target area image are obtained through binarization processing.

Then, the AND operation is performed on the first binarized mask image and the second binarized mask image, that is, the pixels at the same position in the first binarized mask image and the second binarized mask image are respectively Perform an AND operation to obtain the mask image of the intersecting area. The area in which the channel component of the pixels in the intersection area mask image is not zero is the overlapped target makeup area in the first target area image and the second target area.

The face region image corresponding to the initial frame image and the face region image corresponding to the current frame image are obtained through the operation of step 202 . Perform an AND operation on the intersecting area mask image and the face area image corresponding to the initial frame image to obtain a new first target area image corresponding to the initial frame image; An AND operation is performed to obtain a new second target area image corresponding to the current frame image.

Since the intersecting area mask image contains the overlapped target makeup area in the initial frame image and the current frame image, the new first target is extracted from the initial frame image and the current frame image respectively through the intersecting area mask image in the above-mentioned manner area image and the new second target area image, so that the position of the target makeup area in the new first target area image and the new second target area image is exactly the same, so that the subsequent comparison of the target makeup in the current frame image The makeup progress is determined by the change of the area and the target makeup area in the initial frame image, which ensures that the compared areas are completely consistent and greatly improves the accuracy of makeup progress detection.

Align the target makeup area in the initial frame image and the current frame image in the above-mentioned manner, and after obtaining the new first target area image and the new second target area image, determine the current frame image again through the operation of the above-mentioned step 203 Corresponding to the current makeup progress.

After the current makeup progress is determined by any of the above methods, the server sends the current makeup progress to the user's terminal. After receiving the current makeup progress, the user's terminal displays the current makeup progress. The current makeup progress can be a ratio or a percentage. The terminal can display the current makeup progress in the form of a progress bar.

During the process of the user's makeup, the makeup progress detection method provided by the embodiment of the present application can detect the makeup progress of each frame image after the first frame image relative to the first frame image in real time, and display the detected makeup progress to the user, It enables users to visually see the progress of their makeup and improve the efficiency of makeup.

In order to facilitate understanding of the method provided by the embodiment of the present application, the following description will be made in conjunction with the accompanying drawings. As shown in Figure 6, the server can mainly include six modules including face correction and cropping, Gaussian filtering, target makeup area extraction, HSV color space conversion, target makeup area alignment, and makeup progress calculation. The face correction and cropping module corrects and crops the face area on the initial frame image and the current frame image acquired by the server, and obtains the face area image corresponding to the initial frame image and the face area image corresponding to the current frame image. The Gaussian filtering module performs Gaussian smoothing on the face area image obtained by the face correction and cropping module, and is used to remove noise in the image. The target makeup area extraction module is based on the first face key point information corresponding to the initial frame image, for the face area image corresponding to the initial frame image after Gaussian filtering, the target makeup area is extracted to obtain the corresponding Image of the first target area. Similarly, the target makeup area extracting module also extracts the second target area image corresponding to the current frame image. The HSV color space conversion module converts the RGB color space of the first target area image and the second target area image obtained by the target makeup area extraction module into the HSV color space. The target makeup area alignment module performs a pixel alignment operation on the single-channel first target area image and the second target area image that only contain the channel components corresponding to the specific makeup after passing through the HSV color space. The makeup progress calculation module calculates the difference between the single-channel first target area image and the second target area image aligned by the target makeup area alignment module to obtain the makeup progress percentage.

In the embodiment of the present application, the user's face area in the video frame is corrected and cropped by using key points of the face, so as to improve the accuracy of recognizing the face area in the video frame. And based on the key points of the face, the target makeup area is determined from the face area image, and the pixel alignment of the target makeup area in the initial frame image and the target makeup area in the current frame image is carried out, which improves the accuracy of the target makeup area. Accuracy of target makeup area recognition on the user's face in each image frame. By adopting Gaussian smoothing and HSV color space conversion, it is possible to clearly obtain the difference between the initial frame image and the target makeup area in the current frame image in the channel component corresponding to the specific makeup, which improves the accuracy of makeup progress detection. Moreover, this application does not adopt the method of deep learning, and does not need to collect a large amount of data in advance. This application captures the real-time picture of the user's make-up, and returns the detection result to the user through server-side calculation. Compared with the model reasoning scheme of deep learning, this application consumes less computing cost in the algorithm processing link, and reduces the processing pressure of the server.

Embodiment two

The embodiment of the present application provides a makeup progress detection method, which is used to detect the makeup progress corresponding to the blush makeup or the colored makeup in special fields such as Beijing opera. Referring to Figure 7, this embodiment specifically includes the following steps:

Step 301: Obtain at least one target makeup area, and obtain an initial frame image and a current frame image in a real-time makeup video in which the user is currently performing a specific makeup look.

The execution subject of this embodiment of the application is a server. A client that is compatible with the makeup progress detection service provided by the server is installed on the terminal such as the user's mobile phone or computer. When the user needs to use the makeup progress detection service, the user opens the client on the terminal, and the client displays multiple target makeup areas corresponding to preset types of makeup, such as displaying multiple target makeup areas corresponding to blush. The displayed target makeup area can be classified according to the face area, for example, it can be divided into the nose area, the cheek area on both sides, the chin area, and the like. Each area category may include contours of multiple target makeup areas with different shapes and/or sizes. The user selects one or more target makeup application areas that he needs to apply makeup from among the displayed multiple target makeup application areas. The client sends the target makeup area selected by the user to the server.

As an example, in the display interface shown in Figure 8, the display interface includes the contours of the target makeup areas corresponding to the nose region, the cheek regions on both sides and the chin region, and the user can select the corresponding contours of each region. Select the face area that you want to apply makeup and the outline of the makeup on the selected face area, and then click the OK button to submit the outline of the target makeup area you selected. The client detects one or more target makeup areas submitted by the user and sends them to the server.

As another example, in the embodiment of the present application, multiple beauty makeup pattern maps may be created based on preset standard human face images, and the beauty makeup pattern maps include outlines of one or more target makeup areas. The preset standard face image is a face image with an unobstructed face, clear facial features, and a line connecting the eyes parallel to the horizontal line. The interface displayed by the client may simultaneously display multiple beauty style diagrams, and the user selects a beauty makeup style diagram from the multiple displayed beauty makeup style diagrams. The client sends the beauty makeup style image selected by the user to the server. The server receives the beauty makeup pattern image sent by the client, and obtains one or more target makeup areas selected by the user from the beauty makeup pattern image.

By any of the above methods, the user can customize and select the target makeup area that he needs to make up, which can meet the personalized makeup requirements of different users for preset types of makeup such as blush.

In other embodiments of the present application, instead of the user selecting the target makeup area, a fixed target makeup area can be preset in the server, and the position and shape of the target makeup area can be set. After the user opens the client, the client prompts the user to apply makeup on the parts corresponding to the target makeup areas set by the server. When the server receives the request from the user to detect the makeup progress, it directly obtains one or more preset target makeup areas from the local configuration file.

The target makeup area is pre-configured in the server. When the user needs to check the progress of makeup, there is no need to obtain the target makeup area from the user's terminal, which saves bandwidth, simplifies user operations, and shortens the processing time.

The display interface of the above-mentioned client is also provided with a video upload interface. When it is detected that the user clicks on the video upload interface, the camera device of the terminal is called to take a video of the user's makeup. Perform preset types of makeup operations such as blush. The user's terminal transmits the captured makeup video to the server in the form of a video stream. The server receives each frame of the makeup video transmitted by the user's terminal.

In other embodiments of the present application, after the server obtains the initial frame image and the current frame image of the user's makeup video, it also detects whether the initial frame image and the current frame image contain only the face image of the same user. First, detect whether the initial frame image and the current frame image contain only one face image, if the initial frame image and/or the current frame image contain multiple face images, or, the initial frame image and/or the current frame image do not contain face image, then send prompt information to the user's terminal. The user's terminal receives and displays the prompt information to prompt the user to keep only the face of the same user appearing in the makeup video. For example, the prompt information may be "please keep only the face of the same person appearing in the camera".

If it is detected that both the initial frame image and the current frame image contain only one face image, it is further determined whether the face image in the initial frame image and the face image in the current frame image belong to the same user. Specifically, the face feature information corresponding to the face image in the initial frame image can be extracted by face recognition technology, and the face feature information corresponding to the face image in the current frame image can be extracted, and the face feature information extracted from the two frame images can be calculated. The similarity of feature information, if the calculated similarity is greater than or equal to the set value, it is determined that the faces in the initial frame image and the current frame image belong to the same user, and the current frame image is determined by the following steps 302 and 303 Corresponding to the current makeup progress. If the calculated similarity is less than the set value, it is determined that the faces in the initial frame image and the current frame image belong to different users, and a prompt message is sent to the user's terminal. The user's terminal receives and displays the prompt information to prompt the user to keep only the face of the same user appearing in the makeup video.

In the embodiment of the present application, the server takes the received first frame image as the initial frame image, and uses the initial frame image as a reference to compare the current makeup progress of the specific makeup corresponding to each subsequently received frame image. Since the processing method for each subsequent image frame is the same, the embodiment of the present application uses the current frame image received at the current moment as an example to illustrate the process of makeup progress detection.

After obtaining at least one target makeup area and the initial frame image and current frame image of the user's makeup video through this step, the server determines the user's current makeup progress through the following steps 302 and 303 .

Step 302: Generate a beauty mask map according to the obtained target makeup area.

Specifically, according to the position and shape of each target makeup area, the outline of each target makeup area is drawn in a preset blank face image. The preset blank face image may be formed by removing pixels in the preset standard face image. After the outline of each target makeup area is drawn in the preset blank face image, pixel filling is performed in each drawn outline, and the outline of the same target makeup area is filled with pixels with the same pixel value, different The pixel values of the filled pixels between target makeup areas are different from each other. The image after the padding operation is used as a beauty mask image.

Step 303: According to the makeup mask image, the initial frame image and the current frame image, determine the current makeup progress corresponding to the current frame image.

Firstly, with the beauty mask image as a reference, the image of the first target area with makeup applied is obtained from the initial frame image, and the image of the second target area with makeup applied is obtained from the current frame image. That is, using the makeup mask image as a mask, the images of the target makeup area where the user needs to make up are intercepted from the initial frame image and the current frame image respectively. Then, according to the intercepted first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined.

Wherein, the process of obtaining the first target area image from the initial frame image is similar to the operation of step 102 in the first embodiment above, and the first human face key point corresponding to the initial frame image is detected through step S1. Through the operation of step S2, the face area image corresponding to the initial frame image is acquired according to the first face key point. For the specific operation process of obtaining the face region image corresponding to the initial frame image, reference may be made to the relevant description in Embodiment 1, and details are not repeated here. The process of acquiring the second target area image from the current frame image is the same as the process of acquiring the first target area image from the initial frame image.

Then, referring to the makeup mask image generated in step 302, the image of the first target area for makeup is obtained from the face area image corresponding to the initial frame image. Makeup such as blush is a way of applying makeup to fixed areas of the face, such as applying blush to specific areas such as the nose area, the cheek area on both sides, and the chin area. Therefore, these specific areas that need to be applied makeup can be directly extracted from the face area image, thereby avoiding the interference of invalid areas on the makeup progress detection, and improving the accuracy of the makeup progress detection.

The server obtains the first target area image specifically through the following steps S40-S42, including:

S40: Convert the beauty mask image and the face region image into binary images respectively.

S41: Perform an AND operation on the binarized image corresponding to the beauty mask map and the binarized image corresponding to the face region image to obtain a first mask corresponding to the intersection area between the beauty makeup mask map and the face region image image.

An AND operation is performed on the pixel values of the pixel points with the same coordinates in the binarized image corresponding to the beauty mask map and the binarized image corresponding to the face region image. Since only the pixel values of the pixels in the target makeup area in the beauty mask image are not zero, the pixels in other areas are all zero. Therefore, the first mask image obtained by the calculation is equivalent to intercepting each target makeup area from the face area image corresponding to the initial frame image.

In other embodiments of the present application, since the beauty mask image is generated based on a preset standard face image, the target makeup area in the beauty mask image may not be the same as the user's actual face image in the initial frame image. The area of makeup completely overlaps, which affects the accuracy of makeup progress detection. Therefore, before performing an AND operation on the binary image corresponding to the beauty mask image and the binary image corresponding to the face area image, it is also possible to compare the target makeup area in the beauty mask image with the initial frame image The corresponding area is aligned.

Specifically, according to the standard human face key points corresponding to the beauty mask image, one or more first positioning points located on the outline of each target makeup area in the beauty mask image are determined. The standard face key points corresponding to the beauty mask image are the standard face key points corresponding to the preset standard face image. For any target makeup area in the beauty mask map, first determine whether the contour of the target makeup area contains standard human face key points, and if so, determine the standard human face key points on the contour as the The first anchor point corresponding to the target makeup area. If not included, then use the standard human face key points around the target makeup area to generate the first positioning point on the contour of the target makeup area by means of linear transformation. Specifically, a translation operation such as moving up, down, left, or right surrounding standard face key points can be used to obtain the first positioning point.

For example, for the nose region, the key point located at the nose can be moved to the left by a certain pixel distance to obtain a point on the left nose. And move the key point of the nose to the right by a certain pixel distance to get a point on the right side of the nose. The key point of the nose, the point on the left alar and the point on the right alar are used as the three first positioning points corresponding to the nose region.

In the embodiment of the present application, the number of first positioning points corresponding to each target makeup area may be a preset number, and the preset number may be 3 or 4, etc.

After obtaining the first positioning point corresponding to each target makeup area in the beauty mask image by the above method, according to the first human face key point corresponding to the initial frame image, determine from the initial frame image the corresponding first positioning point Point corresponds to the second anchor point. Since the standard face key points corresponding to the beauty mask map and the first face key points corresponding to the initial frame image are obtained through the same detection model, the key points at different positions have their own numbers. Therefore, for the first anchor point belonging to the standard human face key point, determine the first human face with the same number as the standard human face key point corresponding to the first anchor point from the first human face key point corresponding to the initial frame image key point, the determined first face key point is used as the second positioning point corresponding to the first positioning point. For the first anchor point obtained by linear transformation using standard face key points, the first face key point corresponding to the first anchor point is determined from the first face key points corresponding to the initial frame image, and the A point obtained by performing the same linear transformation on the first face key point is determined as a second positioning point corresponding to the first positioning point.

After determining the second anchor point corresponding to each first anchor point in the above way, stretch the beauty mask image, and stretch each first anchor point to the position corresponding to each second anchor point , that is, the position of each first anchor point in the stretched beauty mask image is the same as the position of its corresponding second anchor point.

Through the above method, the target makeup area in the beauty mask image can be aligned with the actual makeup area of the user in the initial frame image, so as to ensure that the makeup area can be accurately extracted from the initial frame image through the beauty mask image. The first target area image, thereby improving the accuracy of makeup progress detection.

After aligning the beauty mask image with the initial frame image, the first mask image corresponding to the intersection area between the beauty mask image and the face area image of the initial frame image is obtained through the operation of step S41, and then through step S42 The first target area image corresponding to the initial frame image is deducted in a manner.

S42: Perform an AND operation on the face area image corresponding to the first mask image and the initial frame image to obtain a first target area image corresponding to the initial frame image.

Since the first mask image is a binarized image, an AND operation is performed on the first mask image and the face area image corresponding to the initial frame image, and each color target is intercepted from the face area image corresponding to the initial frame image The image of the makeup area, that is, the first target area image corresponding to the initial frame image is obtained.

In other embodiments of the present application, since each target makeup area in the beauty mask map is incoherent, the beauty mask map can also be split into multiple sub-mask maps, and each sub-mask map The targeted makeup areas included are all different. Then, the split sub-mask image is used to obtain the first target area image from the face area image corresponding to the initial frame image. Specifically, it can be realized through the following steps S43-S47, including:

S43: Split the beauty mask image into multiple sub-mask images, and each sub-mask image includes at least one target makeup area.

The beauty mask image includes multiple incoherent target makeup areas, split the incoherent target makeup areas to obtain multiple sub-mask images, and each sub-mask image can only include one target The makeup area may also include more than one target makeup area. The target makeup areas included in each sub-mask are different from each other, and the pixel values of the pixels in the sub-masks are all zero except the pixel values of the pixels in the target makeup area are not zero.

S44: Convert each sub-mask image and face region image into a binarized image respectively.

S45: Perform an AND operation on the binarized image corresponding to each sub-mask image and the binarized image corresponding to the face region image to obtain a sub-mask image corresponding to each sub-mask image.

For any sub-mask, an AND operation is performed on the pixel values of the pixel points with the same coordinates in the binarized image of the sub-mask and in the binarized image corresponding to the face region image. Since only the pixel values of the pixels in the target makeup area in the sub-makeup mask image are not zero, the pixels in other areas are all zero. Therefore, the calculated sub-mask image is equivalent to intercepting the target makeup area corresponding to the sub-mask image from the face area image corresponding to the initial frame image.

In other embodiments of the present application, since the beauty mask is generated based on a preset standard face image, the sub-mask is split from the beauty mask, so the sub-mask The target makeup area of is probably not completely coincident with the actual makeup area of the user in the initial frame image, thus affecting the accuracy of makeup progress detection. Therefore, before performing an AND operation on the binarized image corresponding to the sub-mask image and the binarized image corresponding to the face area image, the correspondence between the target makeup area in the sub-mask image and the initial frame image can also be first region for alignment.

Specifically, according to the standard face key points corresponding to the beauty mask image, one or more first positioning points located on the outline of the target makeup area in the sub-mask image are determined. According to the first human face key point corresponding to the initial frame image, the second positioning point corresponding to each first positioning point is determined from the initial frame image. Stretch the sub-mask image, and stretch each first anchor point to the position corresponding to each second anchor point, that is, after stretching, each first anchor point in the sub-mask image The positions are all the same as the positions of the corresponding second positioning points.

Through the above method, the target makeup area in the sub-mask image can be aligned with the actual makeup area of the user in the initial frame image, so as to ensure that the first makeup-on area can be accurately extracted from the initial frame image through each sub-mask image. A target image, thereby improving the accuracy of makeup progress detection. By splitting the beauty mask image into multiple sub-mask images, and aligning each sub-mask image with the initial frame image in the above-mentioned manner, compared with directly aligning the beauty mask image with the initial frame image, splitting The accuracy of alignment after the points is higher.

S46: Perform an AND operation on each sub-mask image and the initial frame image to obtain a plurality of sub-target area images corresponding to the initial frame image.

S47: Merge multiple sub-target area images into a first target area image corresponding to the initial frame image.

For the current frame image, the same method can be used to obtain the second target area image corresponding to the current frame image. That is, the face area image corresponding to the current frame image is converted into a binary image, and then the binary image corresponding to the beauty mask image and the binary image corresponding to the face area image of the current frame image are ANDed to obtain A second mask image corresponding to the intersection area between the beauty mask image and the face area image of the current frame image. An AND operation is performed on the face area image corresponding to the second mask image and the current frame image to obtain a second target area image corresponding to the current frame image. Or, perform an AND operation on the binarized image corresponding to the above-mentioned sub-masks and the face area image of the current frame image to obtain the difference between each sub-mask and the face area image of the current frame image Each sub-mask image corresponding to the intersection area between them. An AND operation is performed on each sub-mask image and the face region image corresponding to the current frame image, and the obtained sub-target region images are merged into a second target region image corresponding to the current frame image.

In other embodiments of the present application, considering that the edge of the target makeup area in the actual makeup scene may not have a very clear outline, for example, the closer to the edge in the blush scene, the lighter the color, so that the blush makeup is more natural , without being obtrusive. Therefore, after the first target area image and the second target area image are obtained through the above embodiment, boundary erosion processing is also performed on the target makeup area in the first target area image and the second target area image, so that the target makeup area The boundary is blurred, so that the target makeup area in the first target area image and the second target area image is closer to the real makeup area, thereby improving the accuracy of makeup progress detection.

The color spaces of the first target area image corresponding to the initial frame image and the second target area image corresponding to the current frame image obtained in the above manner are RGB color spaces. In the embodiments of the present application, a large number of experiments are carried out to determine the influence of preset types of makeup such as blush on each channel component of the color space, and it is found that the influence on each color channel in the RGB color space is not much different. The HSV color space is composed of three components: Hue (hue), Saturation (saturation) and Value (brightness). When one of the components changes, the values of the other two components will not change significantly. Compared with the RGB color space , the HSV color space can separate out one of the channel components. And it is determined through experiments that the preset type of makeup has the greatest influence on which channel component among brightness, hue and saturation, and the channel component with the greatest influence is configured in the server as the preset single-channel component corresponding to the preset type of makeup. For preset types of makeup such as blush, the corresponding preset single-channel component may be a brightness component.

After the first target area image corresponding to the initial frame image and the second target area image corresponding to the current frame image are obtained by any of the above methods, both the first target area image and the second target area image are converted from RGB color space to HSV under the color space. And the preset single-channel component is separated from the converted HSV color space of the first target area image to obtain the first target area image including only the preset single-channel component. And, the preset single-channel component is separated from the converted HSV color space of the second target area image to obtain the second target area image including only the preset single-channel component.

Then, according to the converted first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined.

Specifically, the absolute values of the differences of the channel components corresponding to the pixel points in the same position in the first target area image and the second target area image are respectively calculated. For example, if the preset type of makeup is blush, the absolute value of the difference in brightness component between pixels with the same coordinates in the converted first target area image and the second target area image is calculated.

According to the absolute value of the difference corresponding to each pixel point, the area of the area where the specific makeup has been completed is determined. Specifically, count the number of pixels whose absolute value of the corresponding difference satisfies the preset makeup completion condition. Wherein, the preset makeup completion condition is that the absolute value of the difference corresponding to the pixel points is greater than a first preset threshold, and the first preset threshold may be 7 or 8 or the like.

Determining the counted number of pixels satisfying the preset makeup completion condition as the area of the area where the specific makeup has been completed. The total number of all pixels in all target makeup areas in the first target area image or the second target area image is counted, and the total number of pixels is determined as the total area corresponding to all target makeup areas. Then calculate the ratio between the area area where the specific makeup has been completed and the total area corresponding to the target makeup area, and determine the ratio as the current makeup progress of the specific makeup corresponding to the user. That is, the ratio between the counted number of pixels and the total number of pixels in all target makeup areas in the first target area image is calculated to obtain the current makeup progress corresponding to the current frame image.

In some other embodiments of the present application, in order to further improve the accuracy of makeup progress detection, further alignment is performed on the target makeup areas in the first target area image and the second target area image. Specifically, binary processing is performed on the first target area image and the second target area image that only contain the above-mentioned preset single-channel components, that is, the pixels in the first target area image and the second target area image target the makeup area The values of the above-mentioned preset single-channel components corresponding to the points are all modified to 1, and the values of the above-mentioned preset single-channel components of the pixels at other positions are all modified to 0. A first binarized mask image corresponding to the first target area image and a second binarized mask image corresponding to the second target area image are obtained through binarization processing.

Then, the AND operation is performed on the first binarized mask image and the second binarized mask image, that is, the pixels at the same position in the first binarized mask image and the second binarized mask image are respectively An AND operation is performed to obtain a second mask image corresponding to the intersection area of the first target area image and the second target area image. The area in which the preset single-channel component of the pixel in the second mask image is not zero is the overlapping target makeup area in the first target area image and the second target area.

The face area image corresponding to the initial frame image and the face area image corresponding to the current frame image are obtained through the operation of step 303 . Perform an AND operation on the face area image corresponding to the second mask image and the initial frame image to obtain a new first target area image corresponding to the initial frame image; to obtain a new first target area image corresponding to the second mask image and the current frame image An AND operation is performed to obtain a new second target area image corresponding to the current frame image.

Since the second mask image contains the overlapped target makeup area in the initial frame image and the current frame image, the new first target is extracted from the initial frame image and the current frame image respectively through the second mask image in the above-mentioned manner area image and the new second target area image, so that the position of the target makeup area in the new first target area image and the new second target area image is exactly the same, so that the subsequent comparison of the target makeup in the current frame image The makeup progress is determined by the change of the area and the target makeup area in the initial frame image, which ensures that the compared areas are completely consistent and greatly improves the accuracy of makeup progress detection.

Align the target makeup area in the initial frame image and the current frame image in the above-mentioned manner, and after obtaining the new first target area image and the new second target area image, determine the current frame image again through the operation of the above-mentioned step 303 Corresponding to the current makeup progress.

After the current makeup progress is determined by any of the above methods, the server sends the current makeup progress to the user's terminal. After receiving the current makeup progress, the user's terminal displays the current makeup progress. The current makeup progress can be a ratio or a percentage. The terminal can display the current makeup progress in the form of a progress bar.

During the process of the user's makeup, the makeup progress detection method provided by the embodiment of the present application can detect the makeup progress of each frame image after the first frame image relative to the first frame image in real time, and display the detected makeup progress to the user, It enables users to visually see the progress of their makeup and improve the efficiency of makeup.

In order to facilitate understanding of the method provided by the embodiment of the present application, the following description will be made in conjunction with the accompanying drawings. As shown in Figure 9, according to the initial frame image and its corresponding first face key point, as well as the current frame image and its corresponding second face key point, the faces in the initial frame image and the current frame image are respectively Align and crop, and then smooth and denoise the cropped two face region images by Laplacian algorithm. Then the beauty mask image is aligned with the two face area images respectively, and the first target area image and the second target area image are respectively deducted from the two face area images according to the beauty mask image. Boundary erosion processing is performed on the first target area image and the second target area image. Then convert the first target area image and the second target area image into images containing only preset single-channel components in the HSV color space. Alignment is performed on the first target area image and the second target area image again, and then the current makeup progress is calculated based on the two.

In the embodiment of the present application, the user's face area in the video frame is corrected and cropped by using key points of the face, which improves the accuracy of identifying the face area. Based on the key points of the face, the target makeup area is determined from the face area image, and the pixel alignment is performed on the target makeup area in the initial frame image and the current frame image, which improves the accuracy of target makeup area recognition. Align the target makeup area in the initial frame image and the current frame image to reduce the error introduced by the position difference of the target makeup area. When picking out the target makeup area, the incoherent target makeup area can be calculated separately to increase the accuracy of obtaining the target makeup area. The target makeup area in the beauty mask image is also aligned with the target makeup area in the face area image to ensure that the selected target makeup areas are all in the face area image and will not exceed the face boundary. Moreover, this application does not adopt the method of deep learning, and does not need to collect a large amount of data in advance. This application captures the real-time picture of the user's make-up, and returns the detection result to the user through server-side calculation. Compared with the model reasoning scheme of deep learning, this application consumes less computing cost in the algorithm processing link, and reduces the processing pressure of the server.

Embodiment three

An embodiment of the present application provides a method for detecting makeup progress, which is used for makeup progress corresponding to eyeliner makeup. Referring to Figure 10, this embodiment specifically includes the following steps:

Step 401: Obtain an initial frame image and a current frame image in a real-time makeup video in which the user is currently performing a specific makeup look, and obtain a beauty mask image corresponding to the initial frame image and the current frame image.

The execution subject of this embodiment of the application is a server. A client that is compatible with the makeup progress detection service provided by the server is installed on the terminal such as the user's mobile phone or computer. When the user needs to use the makeup progress detection service, the user opens the client on the terminal, and the client displays multiple eyeliner style maps. The eyeliner style maps are made based on preset standard face images, and the preset standard face images are face A face image with no occlusion, clear facial features, and a line connecting the eyes parallel to the horizontal line. Each eyeliner style diagram corresponds to the eyeliner effect of different eyeliner shapes. For example, some eyeliner style diagrams correspond to the eyeliner effect of round eyes, some eyeliner style diagrams correspond to the eyeliner effect of drooping eyes, and some eyeliner style diagrams correspond to the eyeliner effects of round eyes. Picky eyeliner effects, and more. Considering that the user keeps his eyes open most of the time during the eyeliner makeup process, the eye state of the face in the preset standard face image is the eye open state, and the preset standard face image with the eye open state is used to create Eyeliner style chart.

The interface displayed by the client can display multiple eyeliner style diagrams at the same time, and the user selects an eyeliner style diagram from the displayed multiple eyeliner style diagrams. The client sends the eyeliner style diagram selected by the user to the server, and the server receives the eyeliner style diagram sent by the client.

The display interface of the above-mentioned client is also provided with a video upload interface. When it is detected that the user clicks on the video upload interface, the camera device of the terminal is called to take a video of the user's make-up. During the shooting process, the user performs eyeliner makeup on his face. The user's terminal transmits the captured makeup video to the server in the form of a video stream. The server receives each frame of the makeup video transmitted by the user's terminal.

In the embodiment of the present application, the server takes the received first frame image as the initial frame image, and uses the initial frame image as a reference to compare the current makeup progress of the specific makeup corresponding to each subsequently received frame image. Since the processing method for each subsequent image frame is the same, the embodiment of the present application uses the current frame image received at the current moment as an example to illustrate the process of makeup progress detection.

In other embodiments of the present application, after the server obtains the initial frame image and the current frame image of the user's makeup video, it also detects whether the initial frame image and the current frame image contain only the face image of the same user. First, detect whether the initial frame image and the current frame image contain only one face image, if the initial frame image and/or the current frame image contain multiple face images, or, the initial frame image and/or the current frame image do not contain face image, then send prompt information to the user's terminal. The user's terminal receives and displays the prompt information to prompt the user to keep only the face of the same user appearing in the makeup video. For example, the prompt information may be "please keep only the face of the same person appearing in the camera".

If it is detected that both the initial frame image and the current frame image contain only one face image, it is further determined whether the face image in the initial frame image and the face image in the current frame image belong to the same user. Specifically, the face feature information corresponding to the face image in the initial frame image can be extracted by face recognition technology, and the face feature information corresponding to the face image in the current frame image can be extracted, and the face feature information extracted from the two frame images can be calculated. The similarity of feature information, if the calculated similarity is greater than or equal to the set value, it is determined that the faces in the initial frame image and the current frame image belong to the same user. If the calculated similarity is less than the set value, it is determined that the faces in the initial frame image and the current frame image belong to different users, and a prompt message is sent to the user's terminal. The user's terminal receives and displays the prompt information to prompt the user to keep only the face of the same user appearing in the makeup video.

After the server obtains the eyeliner style map selected by the user through the above method, and obtains the initial frame image and the current frame image during the user's make-up process, the server can directly determine the eyeliner style map selected by the user as the beauty image corresponding to the initial frame image and the current frame image. Makeup mask illustration.

Or, in some other embodiments, when making an eyeliner style diagram, for each eyeliner shape eyeliner effect, an eye-opening style diagram corresponding to the eye-opening state and an eye-closing style diagram corresponding to the avoiding state are made , and an eye-opening style map and an eye-closing style map corresponding to each eyeliner style are pre-configured in the server. The eyeliner pattern diagrams displayed on the interface displayed by the client may all be eyes-open pattern diagrams, or all eyeliner pattern diagrams may be eye-closed pattern diagrams. After the user selects the desired eyeliner style diagram from the multiple displayed eyeliner style diagrams, the client sends the eyeliner style diagram selected by the user to the server. According to the eyeliner style map selected by the user and the user's eye state in the initial frame image and the current frame image, the server respectively determines the beauty mask image corresponding to the initial frame image and the beauty mask image corresponding to the current frame image.

For the initial frame image, first analyze the texture features of the eye area of the human face in the initial frame image through image processing, and determine whether the user's eye state in the initial frame image is an eye-open state. If so, according to the eyeliner pattern selected by the user, the eye-opening pattern corresponding to the eyeliner pattern selected by the user is obtained from the pre-configured sets of eye-opening pattern and eye-closing pattern, and the eye-opening pattern It is determined as the beauty mask image corresponding to the initial frame image. If it is determined that the user's eye state in the initial frame image is the eye-closed state, then according to the eyeliner style map selected by the user, the eyeliner style selected by the user is obtained from multiple groups of pre-configured eye-opening style pictures and eye-closed style pictures An eye-closing pattern map corresponding to the figure, and the eye-closing pattern map is determined as a beauty mask map corresponding to the initial frame image.

For the current frame image, the operation is the same as that of the above initial frame image, and the beauty mask image corresponding to the current frame image is determined in the above manner.

According to the user's eye state in the initial frame image and the current frame image, respectively determine the beauty makeup mask image corresponding to the initial frame image and the beauty makeup mask image corresponding to the current frame image. In this way, the eye state corresponding to the initial frame image and its corresponding beauty mask image is consistent, and the current frame image is consistent with the eye state corresponding to the beauty mask image. In turn, the subsequent deduction of the eyeliner makeup area based on the beauty mask image is more accurate, eliminating the error caused by the inconsistency of the eye state in the initial frame image and the current frame image, and improving the accuracy of eyeliner makeup progress detection.

After the server obtains the initial frame image of the user's makeup and its corresponding beauty mask through this step, and obtains the current frame image of the user's makeup and its corresponding beauty mask, it is determined by the following steps 402 and 403 The user's current makeup progress.

Step 402: According to the initial frame image, simulate and generate a result image after eyeliner makeup is completed.

Use 3D rendering technology to render the effect of eyeliner makeup on the initial frame image to obtain the result image.

Step 403: According to the makeup mask image, the result image, the initial frame image and the current frame image, determine the current makeup progress corresponding to the current frame image.

The eyeliner makeup area usually includes the upper eyelid, lower eyelid, and the end of the eye. Therefore, these specific areas that need to be applied makeup can be directly extracted from the image of the face area, thereby avoiding the interference of invalid areas on the progress detection of eyeliner makeup, and improving the accuracy of detection of eyeliner makeup progress.

Firstly, according to the makeup mask image corresponding to the initial frame image, the first target area image of the eyeliner makeup is obtained from the initial frame image. According to the beauty mask map corresponding to the current frame image, the second target area image of the eyeliner makeup is acquired from the current frame image. Since the result image is generated on the basis of the initial frame image, the makeup mask corresponding to the initial frame image can be used to deduct the eyeliner makeup area in the result image. That is, according to the makeup mask image corresponding to the initial frame image, the image of the third target area of eyeliner makeup is obtained from the result image. Then, according to the intercepted first target area image, second target area image and third target area image, the current makeup progress corresponding to the current frame image is determined.

Wherein, the process of obtaining the first target area image from the initial frame image is similar to the operation of step 102 in the first embodiment above, and the first human face key point corresponding to the initial frame image is detected through step S1. Through the operation of step S2, the face area image corresponding to the initial frame image is acquired according to the first face key point. For the specific operation process of obtaining the face region image corresponding to the initial frame image, reference may be made to the relevant description in Embodiment 1, and details are not repeated here. Then, according to the cosmetic mask map corresponding to the initial frame image, the first target area image of the eyeliner makeup is obtained from the face area image corresponding to the initial frame image. The image of the first target area is acquired, or the operations of steps S33-S37 in the second embodiment can also be used to acquire the image of the first target area, which will not be repeated here.

For the result image and the current frame image, the second target area image corresponding to the current frame image and the third target area image corresponding to the result image may be respectively obtained in the same manner as the initial frame.

The first target area image corresponding to the initial frame image, the second target area image corresponding to the current frame image and the third target area image corresponding to the result image obtained in the above manner, the color spaces of these images are RGB color space. In the embodiments of the present application, a large number of tests are carried out in advance to determine the influence of eyeliner makeup on each channel component of the color space, and it is found that the influence on each color channel in the RGB color space is not much different. The HLS color space is composed of three components: Hue (hue), Saturation (saturation) and Light (brightness). Through experiments, it is found that eyeliner makeup can cause significant changes in the saturation component of the HLS color space.

After obtaining the first target area image corresponding to the initial frame image, the second target area image corresponding to the current frame image, and the third target area image corresponding to the result image by any of the above methods, the first target area image, the second target area image Both the area image and the third target area image are converted from the RGB color space to the HLS color space. And the saturation channel is separated from the converted HLS color space of the first target area image to obtain the first target area image including only the saturation channel. The saturation channel is separated from the converted HLS color space of the second target area image to obtain the second target area image including only the saturation channel. And, the saturation channel is separated from the converted HLS color space of the third target area image to obtain the third target area image including only the saturation channel.

Then, according to the converted first target area image, second target area image and third target area image, the current makeup progress corresponding to the current frame image is determined.

Specifically, the first average pixel value corresponding to the converted first target area image, the second average pixel value corresponding to the second target area image, and the third average pixel value corresponding to the third target area image are respectively calculated. Wherein, the average pixel value is the average value of saturation components of all pixels in the eyeliner makeup area in the image.

Calculate the first difference between the second average pixel value and the first average pixel value, the first difference can represent the saturation change of the eyeliner makeup area from the current frame image to the initial frame image, the saturation change is determined by the current The frame image is formed by the current eyeliner makeup operation corresponding to the frame image.

Calculating a second difference between the third average pixel value and the first average pixel value, the second difference can represent the saturation change of the eyeliner makeup area from the result image to the initial frame image, the saturation change is determined by the completed eyeliner formed by applying makeup.

Calculate the ratio between the first difference and the second difference to obtain the current makeup progress corresponding to the current frame image. The ratio between the saturation change caused by the currently performed eyeliner makeup and the saturation change caused by the completed eyeliner makeup is used as the current makeup progress.

In some other embodiments of the present application, in order to further improve the accuracy of eyeliner makeup progress detection, an alignment process is also performed on the first target area image corresponding to the initial frame image and the second target area image corresponding to the current frame image; and, An alignment process is performed on the first target area image corresponding to the initial frame image and the third target area image corresponding to the result image.

Because the operation of aligning the first target area image and the second target area image is the same as the operation of aligning the first target area image and the third target area image. Therefore, the embodiment of the present application is only described in detail by aligning the first target area image and the second target area image.

Specifically, binarization is performed on the first target area image and the second target area image that only contain the saturation channel, that is, the pixel points in the eyeliner makeup area of the first target area image and the second target area image correspond to The values of the saturation components are all modified to 1, and the values of the saturation components of the pixels at other positions are all modified to 0. A first binarized mask image corresponding to the first target area image and a second binarized mask image corresponding to the second target area image are obtained through binarization processing.

Then, the AND operation is performed on the first binarized mask image and the second binarized mask image, that is, the pixels at the same position in the first binarized mask image and the second binarized mask image are respectively An AND operation is performed to obtain a second mask image corresponding to the intersection area of the first target area image and the second target area image. The area where the saturation component of the pixel in the second mask image is not zero is the overlapping eyeliner makeup area in the first target area image and the second target area.

The face area image corresponding to the initial frame image and the face area image corresponding to the current frame image are obtained through the operation of step 403 . Perform an AND operation on the face area image corresponding to the second mask image and the initial frame image to obtain a new first target area image corresponding to the initial frame image; to obtain a new first target area image corresponding to the second mask image and the current frame image An AND operation is performed to obtain a new second target area image corresponding to the current frame image.

Since the second mask image contains the overlapping eyeliner makeup area in the initial frame image and the current frame image, the new first target is extracted from the initial frame image and the current frame image respectively through the second mask image in the above-mentioned manner area image and the new second target area image, so that the positions of the eyeliner makeup area in the new first target area image and the new second target area image are exactly the same, that is, the initial frame image and the result image are aligned The eyeliner makeup area can improve the accuracy of eyeliner makeup progress detection.

Also align the first target area image and the third target area image in the same manner as above, so that the positions of the eyeliner makeup area in the new first target area image and the new third target area image are completely consistent, that is, they are aligned The eyeliner makeup area in the initial frame image and the current frame image can improve the accuracy of eyeliner makeup progress detection.

After the new first target area image, new second target area image and new third target area image are obtained in the above manner, the current makeup progress corresponding to the current frame image is determined again through the operation of the above step 403 .

After the current makeup progress is determined by any of the above methods, the server sends the current makeup progress to the user's terminal. After receiving the current makeup progress, the user's terminal displays the current makeup progress. The current makeup progress can be a ratio or a percentage. The terminal can display the current makeup progress in the form of a progress bar.

During the process of the user's makeup, the makeup progress detection method provided by the embodiment of the present application can detect the makeup progress of each frame image after the first frame image relative to the first frame image in real time, and display the detected makeup progress to the user, It enables users to visually see the progress of their eyeliner makeup and improve makeup efficiency.

In order to facilitate understanding of the method provided by the embodiment of the present application, the following description will be made in conjunction with the accompanying drawings. As shown in Figure 11, according to the initial frame image and its corresponding first face key point, the current frame image and its corresponding second face key point, and the result image and its corresponding third face key point, respectively Correct and crop the faces in the initial frame image, result image and current frame image, and then smooth and denoise the cropped three face region images by Laplacian algorithm. Then align the beauty mask image with the three face area images, respectively deduct the first target area image corresponding to the initial frame image, the second target area image corresponding to the current frame image and the result image according to the beauty mask image The corresponding image of the third target area. Then convert the first target area image, the second target area image and the third target area image into images containing only the saturation channel in the HLS color space. Calculate the first average pixel value, the second average pixel value and the third average pixel value respectively corresponding to the converted first target area image, the second target area image and the third target area image, calculate the second average pixel value and the first Calculate the first difference between the average pixel values and the second difference between the third average pixel value and the first average pixel value, and calculate the ratio between the first difference and the second difference to obtain the current makeup progress.

In the embodiment of the present application, the current frame image and the initial frame image of the user's makeup process are obtained, and the result image of eyeliner makeup is rendered based on the initial frame image. Determine the saturation change value of the eyeliner makeup area from the current frame image to the initial frame image, and determine the saturation change value of the eyeliner makeup area from the result image to the initial frame image, and calculate the saturation change value corresponding to the current frame image The ratio between the saturation change values corresponding to the result image obtains the current makeup progress of the eyeliner makeup. Without using a deep learning model, the eyeliner makeup progress can be accurately detected only through image processing, with a small amount of calculation and low cost, which reduces the processing pressure on the server, improves the efficiency of eyeliner makeup progress detection, and can meet the eyeliner makeup progress detection real-time requirements.

Further, the face key points are used to correct and crop the user's face area in the video frame, which improves the accuracy of recognizing the face area. The eyeliner makeup area in the initial frame image and the current frame image and the initial frame image and the result image are aligned to reduce the error introduced by the position difference of the eyeliner makeup area. When deducting the eyeliner makeup area, the incoherent eyeliner makeup area can be calculated separately to increase the accuracy of obtaining the eyeliner makeup area. The eyeliner makeup area in the beauty mask image is also aligned with the eyeliner makeup area in the face area image to ensure the accuracy of the deducted eyeliner makeup area.

Embodiment four

1. A method for detecting makeup progress, comprising:

Obtain the eye shadow mask map and the initial frame image and current frame image in the real-time makeup video of the user currently performing a specific makeup;

According to each target makeup area of eye shadow makeup, respectively split the beauty mask map corresponding to each target makeup area from the eye shadow mask map;

The current makeup progress corresponding to the current frame image is determined according to the initial frame image, the current frame image, and the beauty mask map corresponding to each target makeup area.

2. According to 1, the determination of the current makeup progress corresponding to the current frame image according to the initial frame image, the current frame image, and the beauty mask map corresponding to each target makeup area includes:

Taking the beauty mask image corresponding to each target makeup area as a reference, acquiring a first target area image corresponding to each target makeup area from the initial frame image;

Respectively using the beauty mask image corresponding to each target makeup area as a reference, acquiring a second target area image corresponding to each target makeup area from the current frame image;

According to the first target area image and the second target area image corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is determined.

3. According to 2, the determination of the current makeup progress corresponding to the current frame image according to the first target area image and the second target area image corresponding to each target makeup area includes:

Respectively converting the first target area image and the second target area image corresponding to each target makeup area into images containing preset single-channel components under the HLS color space;

According to the converted first target area image and second target area image corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is determined.

4. According to 3, according to the converted first target area image and second target area image corresponding to each target makeup area, determining the current makeup progress corresponding to the current frame image includes:

Respectively calculate the difference absolute value of the preset single-channel component corresponding to the pixels in the same position in the first target area image corresponding to the same target makeup area after conversion and in the second target area image;

Count the number of pixels whose absolute value of the difference corresponding to each target makeup area satisfies the preset makeup completion condition;

Calculating the ratio between the number of pixels corresponding to each target makeup area and the total number of pixels in the corresponding target makeup area to obtain the makeup progress corresponding to each target makeup area;

According to the makeup progress corresponding to each target makeup area and the preset weight corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is calculated.

5. According to 2, using the beauty mask image corresponding to each target makeup area as a reference, the first target area image corresponding to each target makeup area is obtained from the initial frame image, including :

Detecting the first human face key point corresponding to the initial frame image;

Acquiring a face region image corresponding to the initial frame image according to the first face key point;

Taking the beauty mask map corresponding to each of the target makeup areas as a reference, the first target area images corresponding to each of the target makeup areas are respectively acquired from the face area images.

6. According to 5, the first makeup mask image corresponding to each target makeup area is respectively obtained from the face area image with reference to the beauty mask map corresponding to each target makeup area. Imagery of the target area, including:

respectively converting the beauty mask map corresponding to the first target makeup area and the face area image into a binary image; the first target makeup area is any one of each of the target makeup areas target makeup area;

Perform an AND operation on the binarized image corresponding to the beauty mask map and the binarized image corresponding to the face area image to obtain the correspondence between the intersection area of the beauty mask map and the face area image The first mask image of ;

An AND operation is performed on the face area image corresponding to the first mask image and the initial frame image to obtain a first target area image.

7. According to 6, before performing the AND operation on the binarized image corresponding to the beauty mask map and the binarized image corresponding to the face region image, it further includes:

According to the standard human face key points corresponding to the beauty mask map, determine one or more first positions located on the outline of the target makeup area included in the beauty mask map point;

Determining a second anchor point corresponding to each of the first anchor points from the face area image according to the first face key point;

Stretching is performed on the beauty mask image, and each of the first positioning points is stretched to a position corresponding to each of the second positioning points.

8. According to 1, before splitting the beauty mask map corresponding to each target makeup area from the eye shadow mask map according to each target makeup area of eye shadow makeup, it also includes: :

According to the standard human face key points corresponding to the eye shadow mask map, determine one or more first positioning points on the outline of each makeup area in the eye shadow mask map;

Determining a second anchor point corresponding to each of the first anchor points from the face area image according to the first face key point;

Stretching is performed on the eye shadow mask image, and each of the first positioning points is stretched to a position corresponding to each of the second positioning points.

9. According to 5, the acquisition of the face area image corresponding to the initial frame image according to the first face key point includes:

Performing rotation correction on the initial frame image and the first human face key point according to the first human face key point corresponding to the initial frame image;

According to the corrected first human face key point, intercepting an image containing a human face area from the corrected initial frame image;

The image containing the face area is scaled to a preset size to obtain a face area image corresponding to the initial frame image.

10. According to 9, performing rotation correction on the initial frame image and the first facial key point according to the first key point of human face, including:

Determine left eye center coordinates and right eye center coordinates respectively according to the left eye key point and right eye key point included in the first human face key point;

Determine the rotation angle and rotation center point coordinates corresponding to the initial frame image according to the left eye center coordinates and the right eye center coordinates;

Rotation correction is performed on the initial frame image and the first human face key point according to the rotation angle and the coordinates of the rotation center point.

11. According to 9, according to the corrected first human face key point, the image containing the human face area is intercepted from the corrected initial frame image, including:

According to the corrected key points of the first human face, image interception is performed on the human face area included in the corrected initial frame image.

12. According to 11, according to the corrected first human face key point, image interception is performed on the human face area contained in the corrected initial frame image, including:

Determining a minimum abscissa value, a minimum ordinate value, a maximum abscissa value, and a maximum ordinate value from the corrected first face key point;

According to the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value, determine the clipping frame corresponding to the face area in the corrected initial frame image;

According to the clipping frame, an image including the face area is clipped from the rectified initial frame image.

13. According to 12, the method further comprising:

Enlarging the interception frame by a preset multiple;

According to the enlarged clipping frame, an image including the face area is clipped from the corrected initial frame image.

14. According to 9, the method further comprises:

According to the size of the image including the face area and the preset size, zooming and translation processing is performed on the corrected first key points of the face.

15. According to 1, the method further comprises:

Detecting whether both the initial frame image and the current frame image only contain the face image of the same user;

If yes, then perform the operation of determining the current makeup progress of the user performing the specific makeup look;

If not, send prompt information to the user's terminal, where the prompt information is used to prompt the user to keep only the face of the same user appearing in the real-time makeup video.

16. A makeup progress detection device, comprising:

The obtaining module is used to obtain the initial frame image and the current frame image in the real-time makeup video of the eye shadow mask map and the user's current specific makeup;

The splitting module is used to split the beauty mask map corresponding to each target makeup area from the eye shadow mask map according to each target makeup area of eye shadow makeup;

The makeup progress determination module is configured to determine the current makeup progress corresponding to the current frame image according to the initial frame image, the current frame image, and the beauty mask map corresponding to each target makeup area.

17. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, the processor runs the computer program to implement any of claims 1-15. one of the methods described.

18. A computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the method according to any one of claims 1-15.

An embodiment of the present application provides a method for detecting makeup progress, which is used for makeup progress corresponding to eye shadow makeup. Referring to Figure 12, this embodiment specifically includes the following steps:

Step 501: Obtain an eye shadow mask image and an initial frame image and a current frame image in a real-time makeup video in which the user is currently performing a specific makeup look.

The execution subject of this embodiment of the application is a server. A client that is compatible with the makeup progress detection service provided by the server is installed on the terminal such as the user's mobile phone or computer. When the user needs to use the makeup progress detection service, the user opens the client on the terminal, and the client displays multiple eye shadow mask images. The eye shadow mask images are made based on preset standard face images, and the preset standard face images It is a face image with an unoccluded face, clear facial features, and a line connecting the eyes parallel to the horizontal line. Each eye shadow mask map corresponds to a different eye shadow makeup effect.

The interface displayed by the client can simultaneously display multiple eye shadow mask diagrams, and the user selects one from the displayed multiple eye shadow mask diagrams. The client sends the eye shadow mask image selected by the user to the server, and the server receives the eye shadow mask image sent by the client.

The display interface of the above-mentioned client is also provided with a video upload interface. When it is detected that the user clicks on the video upload interface, the camera device of the terminal is called to shoot the real-time makeup video of the user. During the shooting process, the user applies eye shadow makeup on his face . The user's terminal transmits the captured real-time makeup video to the server in the form of a video stream. The server receives each frame of the real-time makeup video transmitted by the user's terminal.

In the embodiment of the present application, the server takes the received first frame image as the initial frame image, and uses the initial frame image as a reference to compare the current makeup progress of the specific makeup corresponding to each subsequently received frame image. Since the processing method for each subsequent image frame is the same, the embodiment of the present application uses the current frame image received at the current moment as an example to illustrate the process of makeup progress detection.

In some other embodiments of the present application, after the server obtains the initial frame image and the current frame image of the user's real-time makeup video, it also detects whether the initial frame image and the current frame image contain only the face image of the same user. First, detect whether the initial frame image and the current frame image contain only one face image, if the initial frame image and/or the current frame image contain multiple face images, or, the initial frame image and/or the current frame image do not contain face image, then send prompt information to the user's terminal. The user's terminal receives and displays the prompt information to prompt the user to keep only the face of the same user appearing in the real-time makeup video. For example, the prompt information may be "please keep only the face of the same person appearing in the camera".

If it is detected that both the initial frame image and the current frame image contain only one face image, it is further determined whether the face image in the initial frame image and the face image in the current frame image belong to the same user. Specifically, the face feature information corresponding to the face image in the initial frame image can be extracted by face recognition technology, and the face feature information corresponding to the face image in the current frame image can be extracted, and the face feature information extracted from the two frame images can be calculated. The similarity of the feature information, if the calculated similarity is greater than or equal to the set value, it is determined that the faces in the initial frame image and the current frame image belong to the same user. If the calculated similarity is less than the set value, it is determined that the faces in the initial frame image and the current frame image belong to different users, and a prompt message is sent to the user's terminal. The user's terminal receives and displays the prompt information to prompt the user to keep only the face of the same user appearing in the real-time makeup video.

After the server obtains the initial frame image of the user's make-up, the current frame image and the eye shadow mask through this step, the server determines the current progress of the user's make-up through the following steps 502 and 503 .

Step 502: According to each target makeup area of eye shadow makeup, respectively split the beauty mask map corresponding to each target makeup area from the eye shadow mask map.

Eye shadow makeup mainly involves large-scale smudging of the upper eyelid, brightening the middle part of the upper eyelid, lying on the lower eyelid, and brightening the eyelids. Among them, the large-area smudging of the upper eyelid and the brightening of the middle part of the upper eyelid are both performed on the upper eyelid, and there is an overlapping area. And the eyeshadow application is not complete until all areas have been applied. Therefore, the eye shadow mask map obtained in step 501 needs to be split according to each target makeup area of eye shadow makeup to obtain a beauty mask map corresponding to each target makeup area. For example, splitting out the beauty mask map corresponding to the target makeup area corresponding to the large-area smudged eyelid, the beauty mask map corresponding to the target makeup area corresponding to the middle part of the upper eyelid, and the beauty makeup corresponding to the lying silkworm part of the lower eyelid The mask map and the beauty mask map corresponding to the eye and head parts, etc.

Step 503: Determine the current makeup progress corresponding to the current frame image according to the initial frame image, the current frame image, and the beauty mask map corresponding to each target makeup area.

Firstly, the first target area image corresponding to each target makeup area is obtained from the initial frame image with reference to the beauty mask image corresponding to each target makeup area. And, using the beauty mask image corresponding to each target makeup area as a reference, the second target area image corresponding to each target makeup area is acquired from the current frame image. Then, according to the first target area image and the second target area image corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is determined.

The acquisition process of the first target area image and the second target area image corresponding to each target makeup area is the same. In this embodiment of the present application, a process of acquiring a first target area image corresponding to a target makeup area is taken as an example to describe in detail. The server specifically obtains the first target area image corresponding to the target makeup area through the following steps S1-S3, including:

S1: Detect the first face key point corresponding to the initial frame image.

The server is configured with a pre-trained detection model for detecting key points of the face, and the interface service of key point detection of the face is provided through the detection model. After the server obtains the initial frame image of the user's makeup video, it calls the interface service of face key point detection, and uses the detection model to identify all face key points of the user's face in the initial frame image. In order to distinguish the face key points corresponding to the current frame image, all face key points corresponding to the initial frame image are referred to as first human face key points in the embodiment of the present application. All face key points corresponding to the current frame image are referred to as second face key points.

Among them, the recognized key points of the human face include key points on the outline of the user's face and key points of the mouth, nose, eyes, eyebrows and other parts. The number of recognized face key points may be 106.

S2: According to the key points of the first face, obtain the face area image corresponding to the initial frame image.

The server specifically obtains the face area image corresponding to the initial frame image through the following steps S20-S22, including:

S20: According to the key points of the first face, perform rotation correction on the initial frame image and the key points of the first face.

When the user shoots a makeup video through the terminal, the pose angle of the face in each frame of image cannot be guaranteed to be the same. In order to improve the accuracy of the comparison between the current frame image and the initial frame image, it is necessary to Perform rotation correction so that the lines connecting the faces and eyes in each frame of image after correction are on the same horizontal line, so as to ensure that the pose angle of the face in each frame of image is the same, and avoid errors in makeup progress detection due to different pose angles Bigger problem.

Specifically, according to the left eye key point and the right eye key point included in the first human face key point, the left eye center coordinates and the right eye center coordinates are respectively determined. All the left-eye key points in the left-eye area and all the right-eye key points in the right-eye area are determined from the first face key points. Take the mean value of the abscissa of all the determined key points of the left eye, and take the mean of the ordinate of all the key points of the left eye, combine the mean value of the abscissa and the mean value of the ordinate corresponding to the left eye into a coordinate, and determine the coordinate is the coordinates of the center of the left eye. Determine the coordinates of the center of the right eye in the same manner.

Then, according to the left eye center coordinates and the right eye center coordinates, the rotation angle and the rotation center point coordinates corresponding to the initial frame image are determined. As shown in Figure 4, the horizontal difference dx and the vertical difference dy of the two coordinates are calculated according to the left eye center coordinates and the right eye center coordinates, and the line connecting the two eyes between the left eye center coordinates and the right eye center coordinates is calculated length d. According to the length d of the connecting line between the two eyes, the horizontal difference dx and the vertical difference dy, the angle θ between the connecting line between the two eyes and the horizontal direction is calculated, and the angle θ is the corresponding rotation angle of the initial frame image. Then, the center point coordinates of the line connecting the two eyes are calculated according to the left eye center coordinates and the right eye center coordinates, and the midpoint coordinates are the rotation center point coordinates corresponding to the initial frame image.

Perform rotation correction on the initial frame image and the key points of the first face according to the calculated rotation angle and the coordinates of the rotation center point. Specifically, the rotation angle and the coordinates of the rotation center point are input into a preset function used to calculate the rotation matrix of the picture, and the preset function may be the function cv2.getRotationMatrix2D() in OpenCV. Obtain the rotation matrix corresponding to the initial frame image by calling this preset function. Then calculate the product of the initial frame image and the rotation matrix to obtain the rectified initial frame image. The operation of correcting the initial frame image by using the rotation matrix can also be done by calling the function cv2.warpAffine() in OpenCV.

For the key points of the first human face, each key point of the first human face needs to be corrected one by one to correspond to the corrected initial frame image. When correcting the key points of the first face one by one, two transformations of the coordinate system are required. The first time the coordinate system with the origin at the upper left corner of the initial frame image is transformed into the coordinate system with the origin at the lower left corner. The second time Further transform the coordinate system with the lower left corner as the origin into the coordinate system with the above-mentioned rotation center point coordinates as the coordinate origin, as shown in FIG. 5 . After two coordinate system conversions, the conversion of the following formula (1) is performed on each key point of the first human face, and the rotation correction of the key points of the first human face can be completed.

In formula (1), x ₀ and y ₀ are respectively the abscissa and ordinate of the key point of the first face before rotation correction, and x and y are respectively the abscissa and ordinate of the key point of the first face after rotation correction, θ is the above-mentioned rotation angle.

The corrected initial frame image and the key points of the first face are based on the entire image, which not only contains the user’s face information, but also includes other redundant image information, so the corrected image needs to be corrected through the following steps S21 The image is cropped for the face area.

S21: According to the corrected first key point of the human face, intercept an image including the human face area from the corrected initial frame image.

Firstly, the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value are determined from the corrected first face key points. Then according to the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value, determine the clipping frame corresponding to the face area in the corrected initial frame image. Specifically, the minimum abscissa value and the minimum ordinate value form a coordinate point, and the coordinate point is used as the upper left vertex of the interception frame corresponding to the face area. The maximum abscissa value and the maximum ordinate value form another coordinate point, and this coordinate point is used as the vertex of the lower right corner of the interception frame corresponding to the face area. Determine the position of the interception frame in the corrected initial frame image according to the above-mentioned upper left corner vertex and lower right corner vertex, and intercept the image located in the interception frame from the corrected initial frame image, that is, intercept the image containing the face area .

In other embodiments of the present application, in order to ensure that all of the user's face area is intercepted, and to avoid incomplete interception resulting in a large error in the follow-up makeup progress detection, the above interception frame can also be enlarged by a preset multiple, The preset multiple can be 1.15 or 1.25 and so on. The embodiment of the present application does not limit the specific value of the preset multiple, and the preset multiple can be set according to requirements in practical applications. After the aforementioned clipping frame is enlarged by a preset multiple, the image located in the enlarged clipping frame is clipped from the corrected initial frame image, so as to clip an image including the complete face area of the user.

S22: Scale the image including the face area to a preset size to obtain the face area image corresponding to the initial frame image.

After the image including the user's face area is intercepted from the initial frame image in the above manner, the image including the face area is scaled to a preset size to obtain a face area image corresponding to the initial frame image. The preset size may be 390×390 or 400×400 or the like. The embodiment of the present application does not limit the specific value of the preset size, which can be set according to requirements in practical applications.

In order to make the first human face key point adapt to the scaled human face area image, after the captured image containing the human face area is scaled to a preset size, it is also necessary to use the size of the image containing the human face area before scaling and the The preset size is used to zoom and translate the corrected key points of the first human face. Specifically, according to the size of the image containing the face area before zooming and the preset size to which the image needs to be zoomed, determine the translation direction and translation distance of each first face key point, and then according to each first face key point The translation direction and the translation distance corresponding to the point, the translation operation is performed on each first human face key point respectively, and the coordinates of each first human face key point after translation are recorded.

Obtain the face area image from the initial frame image in the above way, and make the first face key point adapt to the obtained face area image through operations such as rotation correction and translation scaling, and then obtain the face area image through the following step S3. The image area corresponding to the target makeup area is extracted from the area image.

In some other embodiments of the present application, before step S3 is performed, Gaussian filtering may be performed on the image of the face area to remove noise in the image of the face area. Specifically, Gaussian filtering is performed on the face area image corresponding to the initial frame image according to a Gaussian kernel of a preset size.

The Gaussian kernel of Gaussian filter is the key parameter of Gaussian filter processing. If the Gaussian kernel is too small, the filtering effect cannot be achieved. If the Gaussian kernel is too large, although the noise information in the image can be filtered out, the image will be smoothed at the same time. Useful information in . In this embodiment of the present application, a Gaussian kernel with a preset size is selected, and the preset size may be 9×9. In addition, another set of parameters of the Gaussian filter function, sigmaX and sigmaY, are set to 0. After Gaussian filtering, the image information will be smoother, thereby improving the accuracy of subsequent acquisition of makeup progress.

The face area image is obtained in the above manner, or after the Gaussian filtering is performed on the face area image, the first target area image corresponding to the target makeup area is extracted from the face area image by step S3.

S3: Using the beauty mask image corresponding to the target makeup area as a reference, extract a first target area image corresponding to the target makeup area from the face area image.

The first target area image corresponding to the target makeup area is directly extracted from the face area image, which can avoid the interference of other areas on the makeup progress detection of the target makeup area. In particular, mutual interference between overlapping target makeup areas can be avoided, and the accuracy of eye shadow makeup progress detection can be improved.

The server obtains the first target area image specifically through the following steps S40-S42, including:

S40: Transform the beauty mask image corresponding to the target makeup area and the face area image into binary images respectively.

S41: Perform an AND operation on the binarized image corresponding to the beauty mask map and the binarized image corresponding to the face region image to obtain a first mask corresponding to the intersection area between the beauty makeup mask map and the face region image image.

An AND operation is performed on the pixel values of the pixel points with the same coordinates in the binarized image corresponding to the beauty mask map and the binarized image corresponding to the face region image. Since only the pixel values of the pixels in the target makeup area in the beauty mask image are not zero, the pixels in other areas are all zero. Therefore, the first mask image obtained by the calculation is equivalent to intercepting each target makeup area from the face area image corresponding to the initial frame image.

In other embodiments of the present application, since the beauty mask image is generated based on a preset standard face image, the target makeup area in the beauty mask image may not be the same as the user's actual face image in the initial frame image. The area of makeup completely overlaps, which affects the accuracy of makeup progress detection. Therefore, before performing an AND operation on the binary image corresponding to the beauty mask image and the binary image corresponding to the face area image, it is also possible to compare the target makeup area in the beauty mask image with the initial frame image The corresponding area is aligned.

Specifically, according to the standard human face key points corresponding to the beauty mask image, one or more first positioning points located on the outline of the target makeup area in the beauty mask image are determined. The standard face key points corresponding to the beauty mask image are the standard face key points corresponding to the preset standard face image. Firstly, it is determined whether the contour of the target makeup area contains standard human face key points, and if yes, the standard human face key points on the contour are determined as the first positioning point corresponding to the target makeup area. If not included, then use the standard human face key points around the target makeup area to generate the first positioning point on the contour of the target makeup area by means of linear transformation. Specifically, a translation operation such as moving up, down, left, or right surrounding standard face key points can be used to obtain the first positioning point.

In the embodiment of the present application, the number of first positioning points corresponding to the target makeup area may be a preset number, and the preset number may be 3 or 4, etc.

After obtaining the first positioning point corresponding to the target makeup area in the beauty mask image by the above method, according to the first human face key point corresponding to the initial frame image, determine the first positioning point corresponding to each first positioning point from the initial frame image The corresponding second anchor point. Since the standard face key points corresponding to the beauty mask map and the first face key points corresponding to the initial frame image are obtained through the same detection model, the key points at different positions have their own numbers. Therefore, for the first anchor point belonging to the standard human face key point, determine the first human face with the same number as the standard human face key point corresponding to the first anchor point from the first human face key point corresponding to the initial frame image key point, the determined first face key point is used as the second positioning point corresponding to the first positioning point. For the first anchor point obtained by linear transformation using standard face key points, the first face key point corresponding to the first anchor point is determined from the first face key points corresponding to the initial frame image, and the A point obtained by performing the same linear transformation on the first face key point is determined as a second positioning point corresponding to the first positioning point.

After determining the second anchor point corresponding to each first anchor point in the above way, stretch the beauty mask image, and stretch each first anchor point to the position corresponding to each second anchor point , that is, the position of each first anchor point in the stretched beauty mask image is the same as the position of its corresponding second anchor point.

Through the above method, the target makeup area in the beauty mask image can be aligned with the user's actual makeup area in the initial frame image, so as to ensure that the target makeup area can be accurately extracted from the initial frame image through the beauty mask image. The image of the first target area corresponding to the makeup area, thereby improving the accuracy of makeup progress detection.

After aligning the beauty mask map corresponding to the target makeup area with the initial frame image, the first mask corresponding to the intersection area between the beauty mask map and the face area image of the initial frame image is obtained through the operation of step S41. film image, and then deduct the first target area image corresponding to the target makeup area by means of step S42.

S42: Perform an AND operation on the face area image corresponding to the first mask image and the initial frame image to obtain a first target area image corresponding to the target makeup area.

Since the first mask image is a binarized image, an AND operation is performed on the first mask image and the face region image corresponding to the initial frame image, and the target makeup is intercepted from the face region image corresponding to the initial frame image The image of the region, that is, the first target region image corresponding to the target makeup region is obtained.

The first target area image and the second target area image corresponding to each target makeup area can be obtained through the above-mentioned operations of steps S40-S42.

In some other embodiments of the present application, considering that the edge of the target makeup area in the actual makeup scene may not have a very clear outline, for example, the closer to the edge in the eye shadow makeup scene, the lighter the color, thereby making the eye shadow makeup more natural. It won't look obtrusive. Therefore, after the first target area image and the second target area image are obtained through the above embodiment, boundary erosion processing is also performed on the target makeup area in the first target area image and the second target area image, so that the target makeup area The boundary is blurred, so that the target makeup area in the first target area image and the second target area image is closer to the real makeup area, thereby improving the accuracy of makeup progress detection.

The color space of the first target area image and the second target area image corresponding to each target makeup area obtained in the above manner is RGB color space. In the embodiment of the present application, a large number of tests have been conducted in advance to determine the influence of eye shadow makeup on each channel component of the color space, and it is found that the influence on each color channel in the RGB color space is not very different. The HLS color space is composed of three components: Hue (hue), Saturation (saturation) and Light (brightness). Through experiments, it is found that eye shadow makeup can cause significant changes in the brightness component of the HLS color space.

Therefore, after obtaining the first target area image and the second target area image corresponding to each target makeup area by any of the above methods, the first target area image and the second target area image corresponding to each target makeup area are also obtained by The RGB color space is converted to the HLS color space. And separate the preset single-channel component from the HLS color space of each converted first target area image and each second target area image, and obtain only the preset single-channel component under the HLS color space corresponding to each target makeup area. The first target region image and the second target region image of the channel components. The preset single-channel component may be a luma component.

Then, according to the first target area image and the second target area image corresponding to each target makeup area after conversion, the current makeup progress corresponding to the current frame image is determined.

Specifically, the absolute value of the difference between the preset single-channel components corresponding to the pixels at the same position in the first target area image and the second target area image corresponding to the same target makeup area after conversion are respectively calculated. For example, assuming that the target makeup area is a lying silkworm area, calculate the pixels with the same coordinates in the first target area image corresponding to the lying silkworm area in the initial frame image and the second target area image corresponding to the lying silkworm area in the current frame image The absolute value of the difference between the luma components.

Count the number of pixels whose absolute value of the difference corresponding to each target makeup area satisfies the preset makeup completion condition corresponding to the eyeshadow makeup. Wherein, the preset makeup completion condition is that the absolute value of the difference corresponding to the pixel points is greater than the first preset threshold corresponding to the eye shadow makeup, and the first preset threshold can be 11 or 12, etc.

The total number of all pixels in the first target area image or the second target area image corresponding to each target makeup area is counted. Then, for each target makeup area, the ratio of the absolute value of the statistical difference between the number of pixels that meet the preset makeup completion conditions and the total number of pixels in the target makeup area is calculated, respectively, to obtain each target Makeup progress corresponding to the makeup area. Then, according to the makeup progress corresponding to each target makeup area and the preset weight corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is calculated.

Wherein, the sum of the preset weights corresponding to each target makeup area is 1, and the preset weights corresponding to each target makeup area may be the same or different. For example, assuming that there are 4 target makeup areas including the upper eyelid, the middle part of the upper eyelid, the lying silkworm part and the head part, the preset weights corresponding to these 4 target makeup areas can all be 0.25. The embodiment of the present application does not limit the value of the preset weight of each target makeup area, which can be defined according to requirements in practical applications.

In other embodiments of the present application, in order to further improve the accuracy of makeup progress detection, the target makeup area in the first target area image and the second target area image corresponding to the same target makeup area are further aligned .

Specifically, binary processing is performed on the first target area image and the second target area image corresponding to the same target makeup area, which only contain the above-mentioned preset single-channel component, that is, the first target area image and the second target area The values of the preset single-channel components corresponding to the pixels in the target makeup area in the image are all changed to 1, and the values of the preset single-channel components of the pixels at other positions are all changed to 0. A first binarized mask image corresponding to the first target area image and a second binarized mask image corresponding to the second target area image are obtained through binarization processing.

Then perform an AND operation on the first binary mask image and the second binary mask image to obtain a second mask corresponding to the intersection area of the first target area image corresponding to the target makeup area and the second target area image film image. That is, an AND operation is performed on pixels at the same positions in the first binarized mask image and the second binarized mask image to obtain a second mask image in the intersecting area. The area in which the preset single-channel component of the pixel in the second mask image is not zero is the overlapping target makeup area in the first target area image and the second target area.

The face area image corresponding to the initial frame image and the face area image corresponding to the current frame image are obtained through the operations of the foregoing steps. Perform an AND operation on the face area image corresponding to the second mask image and the initial frame image to obtain a new first target area image corresponding to the target makeup area; for the face area corresponding to the second mask image and the current frame image An AND operation is performed on the image to obtain a new second target area image corresponding to the target makeup area.

In some other embodiments of the present application, the above-mentioned second mask image and the first target area image corresponding to the target makeup area after boundary erosion obtained above can also be used to perform an AND operation to obtain the target area corresponding to the target makeup area. New first target area image. And performing an AND operation on the second mask image and the second target area image corresponding to the target makeup area obtained above after boundary erosion, to obtain a new second target area image corresponding to the target makeup area.

Since the second mask image contains the overlapping area of the first target area image corresponding to the target makeup area and the second target area image, the new first target area image and the new target area image are obtained through the second mask image in the above-mentioned manner. The second target area image, so that the positions of the target makeup area in the new first target area image and the new second target area image are completely consistent, so that the follow-up is based on the completely aligned target makeup area on the preset single-channel component. Changes are used to determine the makeup progress, ensuring that the areas for comparison are completely consistent, and greatly improving the accuracy of makeup progress detection.

Align the target makeup area in the initial frame image and the current frame image by any of the above methods, and after obtaining the new first target area image and the new second target area image, determine the target again through the operation of the above step 503 Makeup progress corresponding to the makeup area. For each other target makeup area, the initial frame image and each other target makeup area in the current frame image are aligned according to the above method, and the makeup progress corresponding to each other target makeup area is respectively obtained. Then, according to the obtained makeup progress of each target makeup area, the current makeup progress corresponding to the current frame image is calculated.

After the current makeup progress is determined by any of the above methods, the server sends the current makeup progress to the user's terminal. After receiving the current makeup progress, the user's terminal displays the current makeup progress. The current makeup progress can be a ratio or a percentage. The terminal can display the current makeup progress in the form of a progress bar.

During the process of the user's makeup, the makeup progress detection method provided by the embodiment of the present application can detect the makeup progress of each frame image after the first frame image relative to the first frame image in real time, and display the detected makeup progress to the user, It enables users to visually see the progress of their makeup and improve the efficiency of makeup.

In order to facilitate understanding of the method provided by the embodiment of the present application, the following description will be made in conjunction with the accompanying drawings. As shown in Figure 13, according to the initial frame image and its corresponding first face key point, the current frame image and its corresponding second face key point, the face in the initial frame image and the current frame image are corrected respectively and cropping. The eye shadow mask map is split into beauty mask maps corresponding to each target makeup area. Then, the beauty mask image corresponding to each target makeup area is aligned with the face area images corresponding to the initial frame image and the current frame image respectively. Deduct the first target area image corresponding to each target makeup area from the initial frame image with reference to the beauty mask image corresponding to each target makeup area, and use the beauty mask image corresponding to each target makeup area The code map refers to the second target area image corresponding to each target makeup area deducted from the current frame image. Then, the first target area image and the second target area image corresponding to each target makeup area are converted into images containing only preset single-channel components in the HLS color space. Calculate the absolute value of the difference between the preset single-channel components corresponding to the pixels in the same position in the first target area image corresponding to the same target makeup area and the second target area image after conversion. Count the number of pixels whose absolute value of the difference corresponding to each target makeup area satisfies the preset makeup completion condition. Calculate the ratio between the number of pixels corresponding to each target makeup area and the total number of pixels in the corresponding target makeup area to obtain the makeup progress corresponding to each target makeup area. According to the makeup progress corresponding to each target makeup area and the preset weight corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is calculated.

In the embodiment of the present application, the user's face area in the video frame is corrected and cropped by using key points of the face, which improves the accuracy of identifying the face area. According to the overlapping features of eye shadow makeup areas, the eye shadow mask map is split into beauty mask maps corresponding to each target makeup area, and the makeup progress is detected for each target makeup area, which improves the makeup progress. detection accuracy. Align the target makeup area in each beauty mask image with the target makeup area in the face area image in the video frame to ensure that each beauty mask image corresponds to the initial frame image and the current frame image The position of the target makeup area is consistent. The first target area image and the second target area image corresponding to each target makeup area are respectively deducted from the initial frame image and the current frame image through the aligned beauty mask image. Furthermore, the first target area image and the second target area image corresponding to the same target makeup area are also aligned again, which reduces the error introduced by position information during comparison. Moreover, this application does not use deep learning methods, and does not need to collect a large amount of data in advance. By capturing real-time images of the user's make-up and calculating on the server side, the detection results are returned to the user. Compared with the model reasoning scheme of deep learning, this application consumes less computing cost in the algorithm processing link, and reduces the processing pressure of the server.

An embodiment of the present application also provides a makeup progress detection device, which is used to implement the above makeup progress detection method to detect the progress of eye shadow makeup in real time. Referring to Figure 14, the device specifically includes:

The obtaining module 1401 is used to obtain the initial frame image and the current frame image in the eye shadow mask map and the real-time makeup video in which the user is currently performing a specific makeup look;

Splitting module 1402, is used for according to each target makeup area of eye shadow makeup, respectively splits the beauty makeup mask map corresponding to each target makeup area from the eye shadow mask map;

The makeup progress determination module 1403 is configured to determine the current makeup progress corresponding to the current frame image according to the initial frame image, the current frame image, and the beauty mask map corresponding to each target makeup area.

The makeup progress determination module 1403 is used to obtain the first target area image corresponding to each target makeup area from the initial frame image with reference to the beauty mask map corresponding to each target makeup area; The beauty mask image corresponding to the target makeup area is used as a reference, and the second target area image corresponding to each target makeup area is obtained from the current frame image; according to the first target area image and the second target area image corresponding to each target makeup area 2. Target area image, determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 1403 is used to convert the first target area image and the second target area image corresponding to each target makeup area into images containing preset single-channel components in the HLS color space; The first target area image and the second target area image corresponding to the target makeup area determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 1403 is used to calculate the absolute value of the difference between the preset single-channel components corresponding to the pixels in the same position in the first target area image and the second target area image corresponding to the same target makeup area after conversion; statistics The absolute value of the difference corresponding to each target makeup area satisfies the number of pixels that meet the preset makeup completion conditions; calculate the difference between the number of pixels corresponding to each target makeup area and the total number of pixels in the corresponding target makeup area to obtain the makeup progress corresponding to each target makeup area; according to the makeup progress corresponding to each target makeup area and the preset weight corresponding to each target makeup area, calculate the current makeup progress corresponding to the current frame image.

The makeup progress determination module 1403 is used to detect the first human face key point corresponding to the initial frame image; according to the first human face key point, obtain the human face area image corresponding to the initial frame image; The makeup mask image is used as a reference, and the first target area image corresponding to each target makeup area is respectively obtained from the face area image.

The makeup progress determination module 1403 is used to respectively convert the beauty mask map corresponding to the first target makeup area and the face area image into a binary image; the first target makeup area is each target makeup area Any target makeup area; perform an AND operation on the binary image corresponding to the beauty mask image and the binary image corresponding to the face area image to obtain the intersection area corresponding to the beauty mask image and the face area image The first mask image: performing an AND operation on the face area image corresponding to the first mask image and the initial frame image to obtain a first target area image.

The makeup progress determination module 1403 is further configured to determine one or more facial key points located in the beauty mask image on the outline of the target makeup area included in the beauty mask image according to the standard face key points corresponding to the beauty mask image. The first positioning point; according to the first key point of the face, determine the second positioning point corresponding to each first positioning point from the face area image; stretch the beauty mask image, and convert each second positioning point A positioning point is stretched to the position corresponding to each second positioning point.

The makeup progress determination module 1403 is also used to determine one or more first positioning points on the outline of each makeup area in the eyeshadow mask image according to the standard human face key points corresponding to the eye shadow mask image; according to the first Face key points, determine the second positioning point corresponding to each first positioning point from the face area image; stretch the eye shadow mask map, stretch each first positioning point to the corresponding each at the position corresponding to the second positioning point.

The makeup progress determination module 1403 is used to perform rotation correction on the initial frame image and the first human face key point according to the first human face key point corresponding to the initial frame image; The image containing the face area is intercepted from the initial frame image; the image containing the face area is scaled to a preset size to obtain the face area image corresponding to the initial frame image.

The makeup progress determination module 1403 is used to determine the left eye center coordinates and right eye center coordinates respectively according to the left eye key points and right eye key points included in the first human face key points; according to the left eye center coordinates and right eye center coordinates, Determine the rotation angle and the coordinates of the rotation center point corresponding to the initial frame image; perform rotation correction on the initial frame image and the key points of the first face according to the rotation angle and the rotation center point coordinates.

The makeup progress determination module 1403 is configured to, according to the corrected first key points of the human face, perform image interception on the human face area included in the corrected initial frame image.

The makeup progress determination module 1403 is used to determine the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value from the corrected first face key point; according to the minimum abscissa value, the minimum ordinate value , the maximum abscissa value and the maximum ordinate value, determine the interception frame corresponding to the face area in the corrected initial frame image; according to the interception frame, intercept the image containing the face area from the corrected initial frame image.

The makeup progress determination module 1403 is further configured to enlarge the clipping frame by a preset multiple; and according to the enlarged clipping frame, clip an image containing a human face area from the corrected initial frame image.

The makeup progress determination module 1403 is further configured to perform scaling and translation processing on the corrected first human face key points according to the size of the image including the human face area and the preset size.

The device also includes: a face detection module, which is used to detect whether the initial frame image and the current frame image only contain the face image of the same user; if so, perform the operation of determining the current makeup progress of the user for a specific makeup look; If not, send prompt information to the user's terminal, and the prompt information is used to prompt the user to keep only the face of the same user appearing in the real-time makeup video.

The makeup progress detection device provided in the above embodiments of the present application is based on the same inventive concept as the makeup progress detection method provided in the embodiments of the present application, and has the same beneficial effects as the methods adopted, run or realized by its stored application programs.

Embodiment five

1. A method for detecting makeup progress, comprising:

Obtain the initial frame image and the current frame image in the real-time makeup video of the user currently performing a specific makeup look;

Obtaining a first target area image corresponding to the eyebrows from the initial frame image, and acquiring a second target area image corresponding to the eyebrows from the current frame image;

According to the first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined.

2. According to 1, the acquisition of the first target region image corresponding to the eyebrows from the initial frame image includes:

Detecting the first human face key point corresponding to the initial frame image;

Acquiring a face region image corresponding to the initial frame image according to the first face key point;

According to the eyebrow key points included in the first human face key points, a first target area image corresponding to eyebrows is acquired from the human face area image.

3. According to 2, according to the eyebrow key points included in the first human face key points, the first target area image corresponding to the eyebrows is intercepted from the face area image, including:

Interpolating the eyebrow key points between the eyebrows and eyebrow peaks included in the first human face key points to obtain a plurality of interpolation points;

Intercepting all eyebrow key points between the eyebrow head and the eyebrow peak and the closed area formed by the connection of the plurality of interpolation points from the image of the human face area, obtaining a partial eyebrow image between the eyebrow head and the eyebrow peak;

Intercepting the closed area formed by connecting all eyebrow key points between the eyebrow peak and the eyebrow tail from the image of the human face region, obtaining a partial eyebrow image between the eyebrow peak and the eyebrow tail;

The partial eyebrow image between the eyebrow head and the eyebrow peak and the partial eyebrow image between the eyebrow peak and the eyebrow tail are spliced into a first target region image corresponding to the eyebrows.

4. According to 1, the determining the current makeup progress corresponding to the current frame image according to the first target area image and the second target area image includes:

respectively converting the first target area image and the second target area image into images containing preset single-channel components in HSV color space;

According to the converted first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined.

5. According to 4, the determination of the current makeup progress corresponding to the current frame image according to the converted first target area image and the second target area image includes:

Calculating the absolute value of the difference between the preset single-channel components corresponding to pixels at the same position in the converted first target area image and the second target area image;

Count the number of pixels whose absolute value of the corresponding difference satisfies the preset makeup completion condition;

Calculate the ratio between the counted number of pixels and the total number of pixels in all target makeup areas in the first target area image to obtain the current makeup progress corresponding to the current frame image.

6. According to 1, before determining the current makeup progress corresponding to the current frame image according to the first target area image and the second target area image, further comprising:

Performing binarization processing on the first target area image and the second target area image respectively, to obtain a first binarized mask image corresponding to the first target area image and corresponding to the second target area image The second binarized mask image of ;

Perform an AND operation on the first binarized mask image and the second binarized mask image to obtain a second mask corresponding to the intersection area of the first target area image and the second target area image film image;

Acquiring the face area image corresponding to the initial frame image and the face area image corresponding to the current frame image;

performing an AND operation on the second mask image and the face area image corresponding to the initial frame image to obtain a new first target area image corresponding to the initial frame image;

An AND operation is performed on the second mask image and the face area image corresponding to the current frame image to obtain a new second target area image corresponding to the current frame image.

7. According to 1, before the determination of the current makeup progress corresponding to the current frame image, it further includes:

Boundary erosion processing is performed on makeup areas in the first target area image and the second target area image respectively.

8. According to 2, the acquisition of the face area image corresponding to the initial frame image according to the first face key point includes:

Performing rotation correction on the initial frame image and the first human face key point according to the first human face key point corresponding to the initial frame image;

According to the corrected first human face key point, intercepting an image containing a human face area from the corrected initial frame image;

The image containing the face area is scaled to a preset size to obtain a face area image corresponding to the initial frame image.

9. According to 8, performing rotation correction on the initial frame image and the first facial key point according to the first key point of human face, including:

Determine left eye center coordinates and right eye center coordinates respectively according to the left eye key point and right eye key point included in the first human face key point;

Determine the rotation angle and rotation center point coordinates corresponding to the initial frame image according to the left eye center coordinates and the right eye center coordinates;

Rotation correction is performed on the initial frame image and the first human face key point according to the rotation angle and the coordinates of the rotation center point.

10. According to 8, according to the corrected first human face key point, the image containing the human face area is intercepted from the corrected initial frame image, including:

According to the corrected key points of the first human face, image interception is performed on the human face area included in the corrected initial frame image.

11. According to 10, according to the corrected first human face key point, image interception is performed on the human face area contained in the corrected initial frame image, including:

Determining a minimum abscissa value, a minimum ordinate value, a maximum abscissa value, and a maximum ordinate value from the corrected first face key point;

According to the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value, determine the clipping frame corresponding to the face area in the corrected initial frame image;

According to the clipping frame, an image including the face area is clipped from the rectified initial frame image.

12. According to 11, the method further comprises:

Enlarging the interception frame by a preset multiple;

According to the enlarged clipping frame, an image including the face area is clipped from the corrected initial frame image.

13. According to 8, the method further comprises:

According to the size of the image including the face area and the preset size, zooming and translation processing is performed on the corrected first key points of the face.

14. According to any one of 1-13, the method further comprises:

Detecting whether both the initial frame image and the current frame image only contain the face image of the same user;

If yes, then perform the operation of determining the current makeup progress of the user performing the specific makeup look;

If not, send prompt information to the user's terminal, where the prompt information is used to prompt the user to keep only the face of the same user appearing in the real-time makeup video.

15. A makeup progress detection device, comprising:

The video acquisition module is used to acquire the initial frame image and the current frame image in the real-time makeup video of the user currently performing a specific makeup look;

A target area acquiring module, configured to acquire a first target area image corresponding to the eyebrows from the initial frame image, and acquire a second target area image corresponding to the eyebrows from the current frame image;

A progress determination module, configured to determine the current makeup progress corresponding to the current frame image according to the first target area image and the second target area image.

16. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, the processor runs the computer program to implement any of claims 1-14. one of the methods described.

17. A computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the method according to any one of claims 1-14.

An embodiment of the present application provides a method for detecting makeup progress, which is used for makeup progress corresponding to eyebrow makeup. Referring to Figure 15, this embodiment specifically includes the following steps:

Step 601: Obtain an initial frame image and a current frame image in a real-time makeup video in which the user is currently performing a specific makeup look.

The execution subject of this embodiment of the application is a server. A client that is compatible with the makeup progress detection service provided by the server is installed on the terminal such as the user's mobile phone or computer. When the user needs to use the makeup progress detection service, the user opens the client on the terminal, and the display interface of the client is provided with a video upload interface. In the video, the user performs eyebrow makeup on his face during the shooting process. The user's terminal transmits the captured makeup video to the server in the form of a video stream. The server receives each frame of the makeup video transmitted by the user's terminal.

In the embodiment of the present application, the server takes the received first frame image as the initial frame image, and uses the initial frame image as a reference to compare the current makeup progress of the specific makeup corresponding to each subsequently received frame image. Since the processing method for each subsequent image frame is the same, the embodiment of the present application uses the current frame image received at the current moment as an example to illustrate the process of makeup progress detection.

In other embodiments of the present application, after the server obtains the initial frame image and the current frame image of the user's makeup video, it also detects whether the initial frame image and the current frame image contain only the face image of the same user. First, detect whether the initial frame image and the current frame image contain only one face image, if the initial frame image and/or the current frame image contain multiple face images, or, the initial frame image and/or the current frame image do not contain face image, then send prompt information to the user's terminal. The user's terminal receives and displays the prompt information to prompt the user to keep only the face of the same user appearing in the makeup video. For example, the prompt information may be "please keep only the face of the same person appearing in the camera".

If it is detected that both the initial frame image and the current frame image contain only one face image, it is further determined whether the face image in the initial frame image and the face image in the current frame image belong to the same user. Specifically, the face feature information corresponding to the face image in the initial frame image can be extracted by face recognition technology, and the face feature information corresponding to the face image in the current frame image can be extracted, and the face feature information extracted from the two frame images can be calculated. The similarity of feature information, if the calculated similarity is greater than or equal to the set value, it is determined that the faces in the initial frame image and the current frame image belong to the same user. If the calculated similarity is less than the set value, it is determined that the faces in the initial frame image and the current frame image belong to different users, and a prompt message is sent to the user's terminal. The user's terminal receives and displays the prompt information to prompt the user to keep only the face of the same user appearing in the makeup video.

After obtaining the initial frame image and the current frame image of the user's make-up through this step, the server determines the current make-up progress of the user through the following steps 602 and 603 .

Step 602: Obtain a first target area image corresponding to the eyebrows from the initial frame image, and acquire a second target area image corresponding to the eyebrows from the current frame image.

Wherein, the acquisition process of the first target area image is the same as the acquisition process of the second target area image. In this embodiment of the present application, the process of acquiring the image of the first target area is taken as an example to describe in detail. Specifically, the server obtains the first target area image from the initial frame image through the following steps S5-S7.

S5: Detect the key points of the first human face corresponding to the initial frame image.

The server is configured with a pre-trained detection model for detecting key points of the face, and the interface service of key point detection of the face is provided through the detection model. After the server obtains the initial frame image of the user's makeup video, it calls the interface service of face key point detection, and uses the detection model to identify all face key points of the user's face in the initial frame image. In order to distinguish the face key points corresponding to the current frame image, all face key points corresponding to the initial frame image are referred to as first human face key points in the embodiment of the present application. All face key points corresponding to the current frame image are referred to as second face key points.

Among them, the recognized key points of the human face include key points on the outline of the user's face and key points of the mouth, nose, eyes, eyebrows and other parts. The number of recognized face key points may be 106.

S6: According to the key points of the first face, acquire the face area image corresponding to the initial frame image.

The server specifically obtains the face area image corresponding to the initial frame image through the following steps S60-S62, including:

S60: Perform rotation correction on the initial frame image and the first human face key points according to the first human face key points.

When the user shoots a makeup video through the terminal, the pose angle of the face in each frame of image cannot be guaranteed to be the same. In order to improve the accuracy of the comparison between the current frame image and the initial frame image, it is necessary to Perform rotation correction so that the lines connecting the faces and eyes in each frame of image after correction are on the same horizontal line, so as to ensure that the pose angle of the face in each frame of image is the same, and avoid errors in makeup progress detection due to different pose angles Bigger problem.

Specifically, according to the left eye key point and the right eye key point included in the first human face key point, the left eye center coordinates and the right eye center coordinates are respectively determined. All the left-eye key points in the left-eye area and all the right-eye key points in the right-eye area are determined from the first face key points. Take the mean value of the abscissa of all the determined key points of the left eye, and take the mean of the ordinate of all the key points of the left eye, combine the mean value of the abscissa and the mean value of the ordinate corresponding to the left eye into a coordinate, and determine the coordinate is the coordinates of the center of the left eye. Determine the coordinates of the center of the right eye in the same manner.

Then, according to the left eye center coordinates and the right eye center coordinates, the rotation angle and the rotation center point coordinates corresponding to the initial frame image are determined. As shown in Figure 4, the horizontal difference dx and the vertical difference dy of the two coordinates are calculated according to the left eye center coordinates and the right eye center coordinates, and the line connecting the two eyes between the left eye center coordinates and the right eye center coordinates is calculated length d. According to the length d of the connecting line between the two eyes, the horizontal difference dx and the vertical difference dy, the angle θ between the connecting line between the two eyes and the horizontal direction is calculated, and the angle θ is the corresponding rotation angle of the initial frame image. Then, the center point coordinates of the line connecting the two eyes are calculated according to the left eye center coordinates and the right eye center coordinates, and the midpoint coordinates are the rotation center point coordinates corresponding to the initial frame image.

Perform rotation correction on the initial frame image and the key points of the first face according to the calculated rotation angle and the coordinates of the rotation center point. Specifically, the rotation angle and the coordinates of the rotation center point are input into a preset function used to calculate the rotation matrix of the picture, and the preset function may be the function cv2.getRotationMatrix2D() in OpenCV. Obtain the rotation matrix corresponding to the initial frame image by calling this preset function. Then calculate the product of the initial frame image and the rotation matrix to obtain the rectified initial frame image. The operation of correcting the initial frame image by using the rotation matrix can also be done by calling the function cv2.warpAffine() in OpenCV.

For the key points of the first human face, each key point of the first human face needs to be corrected one by one to correspond to the corrected initial frame image. When correcting the key points of the first face one by one, two transformations of the coordinate system are required. The first time the coordinate system with the origin at the upper left corner of the initial frame image is transformed into the coordinate system with the origin at the lower left corner. The second time Further transform the coordinate system with the lower left corner as the origin into the coordinate system with the above-mentioned rotation center point coordinates as the coordinate origin, as shown in FIG. 5 . After two coordinate system conversions, the conversion of the following formula (1) is performed on each key point of the first human face, and the rotation correction of the key points of the first human face can be completed.

In formula (1), x ₀ and y ₀ are respectively the abscissa and ordinate of the key point of the first face before rotation correction, and x and y are respectively the abscissa and ordinate of the key point of the first face after rotation correction, θ is the above-mentioned rotation angle.

The corrected initial frame image and the key points of the first face are based on the entire image, which not only contains the user’s face information, but also includes other redundant image information, so the corrected image needs to be corrected in the following steps S61 The image is cropped for the face area.

S61: According to the corrected first key point of the human face, intercept an image including the human face area from the corrected initial frame image.

Firstly, the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value are determined from the corrected first face key points. Then according to the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value, determine the clipping frame corresponding to the face area in the corrected initial frame image. Specifically, the minimum abscissa value and the minimum ordinate value form a coordinate point, and the coordinate point is used as the upper left vertex of the interception frame corresponding to the face area. The maximum abscissa value and the maximum ordinate value form another coordinate point, and this coordinate point is used as the vertex of the lower right corner of the interception frame corresponding to the face area. Determine the position of the interception frame in the corrected initial frame image according to the above-mentioned upper left corner vertex and lower right corner vertex, and intercept the image located in the interception frame from the corrected initial frame image, that is, intercept the image containing the face area .

In other embodiments of the present application, in order to ensure that all of the user's face area is intercepted, and to avoid incomplete interception resulting in a large error in the follow-up makeup progress detection, the above interception frame can also be enlarged by a preset multiple, The preset multiple can be 1.15 or 1.25 and so on. The embodiment of the present application does not limit the specific value of the preset multiple, and the preset multiple can be set according to requirements in practical applications. After the aforementioned clipping frame is enlarged by a preset multiple, the image located in the enlarged clipping frame is clipped from the corrected initial frame image, so as to clip an image including the complete face area of the user.

S62: Scale the image including the face area to a preset size to obtain a face area image corresponding to the initial frame image.

After the image including the user's face area is intercepted from the initial frame image in the above manner, the image including the face area is scaled to a preset size to obtain a face area image corresponding to the initial frame image. The preset size may be 390×390 or 400×400 or the like. The embodiment of the present application does not limit the specific value of the preset size, which can be set according to requirements in practical applications.

In order to make the first human face key point adapt to the scaled human face area image, after the captured image containing the human face area is scaled to a preset size, it is also necessary to use the size of the image containing the human face area before scaling and the The preset size is used to zoom and translate the corrected key points of the first human face. Specifically, according to the size of the image containing the face area before zooming and the preset size to which the image needs to be zoomed, determine the translation direction and translation distance of each first face key point, and then according to each first face key point The translation direction and the translation distance corresponding to the point, the translation operation is performed on each first human face key point respectively, and the coordinates of each first human face key point after translation are recorded.

Obtain the face area image from the initial frame image in the above manner, and make the first key point of the face adapt to the obtained face area image through operations such as rotation correction and translation scaling, and then obtain the face area image through the following step S7. A first target region image corresponding to the eyebrows is extracted from the region image.

In some other embodiments of the present application, before step S7 is performed, Gaussian filtering may be performed on the image of the face area to remove noise in the image of the face area. Specifically, Gaussian filtering is performed on the face area image corresponding to the initial frame image according to a Gaussian kernel of a preset size.

The Gaussian kernel of Gaussian filter is the key parameter of Gaussian filter processing. If the Gaussian kernel is too small, the filtering effect cannot be achieved. If the Gaussian kernel is too large, although the noise information in the image can be filtered out, the image will be smoothed at the same time. Useful information in . In this embodiment of the present application, a Gaussian kernel with a preset size is selected, and the preset size may be 9×9. In addition, another set of parameters of the Gaussian filter function, sigmaX and sigmaY, are set to 0. After Gaussian filtering, the image information will be smoother, thereby improving the accuracy of subsequent acquisition of makeup progress.

The face region image is obtained in the above manner, or after Gaussian filtering is performed on the face region image, the first target region image corresponding to the eyebrows is extracted from the face region image corresponding to the initial frame image through step S7.

S7: According to the eyebrow key points included in the first face key points, extract the first target area image corresponding to the eyebrows from the face area image corresponding to the initial frame image.

When detecting the progress of eyebrow makeup, it is necessary to deduct the image of the area where the eyebrows are located to avoid the influence of other areas on the progress detection of eyebrow makeup, and to deduct the eyebrow area, and then only perform operations on the eyebrow area, reducing the amount of calculation. Accuracy is also improved.

The first human face key points obtained above include multiple eyebrow key points, for example, 18 eyebrow key points may be included. These multiple eyebrow key points are distributed at different positions from the head of the eyebrow to the end of the eyebrow on the contour of the eyebrow. In order to improve the accuracy of extracting the image of the first target region corresponding to the eyebrows, the embodiment of the present application uses linear interpolation to obtain more points on the outline of the eyebrows, so as to extract images based on more points. Since the tail of eyebrows is pointed, it is not convenient for linear interpolation. Therefore, in the embodiment of the present application, the process of extracting the image of the first target region corresponding to the eyebrows is divided into two parts. A part is from the eyebrow head to the eyebrow peak, and more points are obtained by linear difference to deduct the image. The other part is the section from the peak of the eyebrow to the end of the eyebrow, and the image is extracted by using the currently obtained key points of the eyebrows from the peak of the eyebrow to the end of the eyebrow.

Specifically, a linear interpolation is performed on the eyebrow key points included in the first human face key points between the head of the eyebrow and the peak of the eyebrow to obtain a plurality of interpolation points. Connect all the eyebrow key points between the eyebrow head and the eyebrow peak in the face area image corresponding to the initial frame image and the obtained multiple interpolation points sequentially along the eyebrow contour line to obtain a closed area, which circles the eyebrow head to the eyebrow peak section part of the eyebrow area. The image of the closed area is intercepted from the face area image corresponding to the initial frame image to obtain a partial eyebrow image between the head of the eyebrow and the peak of the eyebrow.

All eyebrow key points between the eyebrow peak and the eyebrow tail in the face area image corresponding to the initial frame image are connected sequentially along the eyebrow contour line to obtain a closed area, which circles part of the eyebrow area from the eyebrow peak to the eyebrow tail . The image of the closed area is intercepted from the face area image corresponding to the initial frame image to obtain a part of the eyebrow image between the eyebrow peak and the eyebrow tail.

The part of the eyebrow image between the eyebrow head and the eyebrow peak and the part of the eyebrow image between the eyebrow peak and the eyebrow tail are stitched into the first target area image corresponding to the eyebrow.

For the current frame image, the second target area image corresponding to the eyebrows is also obtained from the current frame image according to the above steps S5-S7.

In some other embodiments of the present application, considering that the edge of the eyebrow makeup in the actual makeup scene may not have a very clear outline, the boundary is usually blurred so as not to appear abrupt. Therefore, after the first target area image and the second target area image are obtained through the above-mentioned embodiment, boundary erosion processing is also performed on the eyebrow area in the first target area image and the second target area image respectively, so that the target with eyebrow makeup is put on makeup. The boundary of the area is blurred, which in turn improves the accuracy of makeup progress detection.

Step 603: According to the first target area image and the second target area image, determine the current makeup progress corresponding to the current frame image.

The color spaces of the first target region image corresponding to the eyebrows in the initial frame image and the second target region image corresponding to the eyebrows in the current frame image obtained in the above manner are RGB color spaces. In the embodiment of the present application, a large number of experiments are carried out to determine the influence of eyebrow makeup on each channel component of the color space in advance, and it is found that the influence on each color channel in the RGB color space is not very different. The HSV color space is composed of three components: Hue (hue), Saturation (saturation) and Value (brightness). When one of the components changes, the values of the other two components will not change significantly. Compared with the RGB color space , the HSV color space can separate out one of the channel components. And through experiments, it is determined which channel component the eyebrow makeup has the greatest influence on brightness, hue and saturation, and the channel component with the greatest influence is configured in the server as the preset single-channel component corresponding to the preset type of makeup. For eyebrow makeup, the corresponding preset single-channel component may be a brightness component.

After the first target region image corresponding to the eyebrows in the initial frame image and the second target region image corresponding to the eyebrows in the current frame image are obtained by the above method, both the first target region image and the second target region image are converted by RGB color space For the HSV color space. And separating the preset single-channel component from the converted HSV color space of the first target area image to obtain the first target area image including only the preset single-channel component. And, the preset single-channel component is separated from the converted HSV color space of the second target area image to obtain the second target area image including only the preset single-channel component.

Then, according to the converted first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined.

Specifically, the absolute values of the differences of the above-mentioned channel components corresponding to the pixel points in the same position in the converted first target area image and the second target area image are respectively calculated. For example, the absolute value of the difference in brightness component between pixels with the same coordinates in the converted first target area image and the second target area image is calculated. Count the number of pixels whose absolute value of the corresponding difference satisfies the preset makeup completion condition. Wherein, the preset makeup completion condition is that the absolute value of the difference corresponding to the pixel points is greater than a first preset threshold, and the first preset threshold may be 7 or 8 or the like.

The total number of all pixels in the eyebrow area in the first target area image or the second target area image is counted. Then calculate the ratio between the number of pixels meeting the preset makeup completion condition and the total number of pixels in the eyebrow region, and determine the ratio as the current makeup progress.

In some other embodiments of the present application, in order to further improve the accuracy of makeup progress detection, the eyebrow regions in the above-mentioned first target area image and the second target area image are further aligned. Specifically, binary processing is performed on the first target area image and the second target area image that only contain the above-mentioned preset single-channel components, that is, the pixels in the first target area image and the second target area image target the makeup area The values of the above-mentioned preset single-channel components corresponding to the points are all modified to 1, and the values of the above-mentioned preset single-channel components of the pixels at other positions are all modified to 0. A first binarized mask image corresponding to the first target area image and a second binarized mask image corresponding to the second target area image are obtained through binarization processing.

Then, the AND operation is performed on the first binarized mask image and the second binarized mask image, that is, the pixels at the same position in the first binarized mask image and the second binarized mask image are respectively An AND operation is performed to obtain a second mask image corresponding to the intersection area of the first target area image and the second target area image. The area in which the preset single-channel component of the pixel in the second mask image is not zero is the overlapping target makeup area in the first target area image and the second target area.

The face area image corresponding to the initial frame image and the face area image corresponding to the current frame image are obtained through the operations of the foregoing steps. Perform an AND operation on the face area image corresponding to the second mask image and the initial frame image to obtain a new first target area image corresponding to the eyebrows in the initial frame image; for the face area corresponding to the second mask image and the current frame image An AND operation is performed on the image to obtain a new second target area image corresponding to the eyebrow in the current frame image.

In some other embodiments of the present application, an AND operation may be performed on the second mask image and the first target region image corresponding to the eyebrow after boundary erosion processing to obtain a new first target region image corresponding to the eyebrow. And, an AND operation is performed on the second mask image and the second target region image corresponding to the eyebrow after the boundary erosion processing to obtain a new second target region image corresponding to the eyebrow.

Since the second mask image contains the overlapped target makeup area in the initial frame image and the current frame image, the new first target area image and the new second target area image are extracted respectively through the second mask image in the above-mentioned manner, Make the position of the target makeup area in the new first target area image and the new second target area image completely consistent, so that the subsequent comparison of the target makeup area in the current frame image and the change of the target makeup area in the initial frame image Determining the makeup progress ensures that the areas for comparison are completely consistent, which greatly improves the accuracy of makeup progress detection.

Align the target makeup area in the initial frame image and the current frame image in the above-mentioned manner, and after obtaining the new first target area image and the new second target area image, determine the corresponding area of the current frame image through the operation of the above step 603 again. Current makeup progress.

After the current makeup progress is determined by any of the above methods, the server sends the current makeup progress to the user's terminal. After receiving the current makeup progress, the user's terminal displays the current makeup progress. The current makeup progress can be a ratio or a percentage. The terminal can display the current makeup progress in the form of a progress bar.

During the process of the user's makeup, the makeup progress detection method provided by the embodiment of the present application can detect the makeup progress of each frame image after the first frame image relative to the first frame image in real time, and display the detected makeup progress to the user, It enables users to visually see the progress of their makeup and improve the efficiency of makeup.

In order to facilitate understanding of the method provided by the embodiment of the present application, the following description will be made in conjunction with the accompanying drawings. As shown in Figure 16, according to the initial frame image and its corresponding first face key point, and the current frame image and its corresponding second face key point, the faces in the initial frame image and the current frame image are respectively Align and crop, and then smooth and denoise the cropped two face region images by Laplacian algorithm. Then, the first target region image and the second target region image corresponding to the eyebrows are respectively deducted from the two face region images. Boundary erosion processing is performed on the first target area image and the second target area image. Then convert the first target area image and the second target area image into images containing only preset single-channel components in the HSV color space. Alignment is performed on the first target area image and the second target area image again, and then the current makeup progress is calculated based on the two.

In the embodiment of the present application, the user's face area in the video frame is corrected and cropped by using key points of the face, which improves the accuracy of identifying the face area. Based on the key points of the face, the target area image corresponding to the eyebrow is deducted from the face area image, and the pixel alignment is performed on the target area image corresponding to the initial frame image and the current frame image, which improves the accuracy of the target area image corresponding to the eyebrow . Align the target makeup area in the initial frame image and the current frame image to reduce the error introduced by the position difference of the target makeup area. A segmented interpolation algorithm is introduced when extracting the eyebrow area to make the extracted eyebrow area more coherent and accurate. Moreover, this application does not adopt the method of deep learning, and does not need to collect a large amount of data in advance. This application captures the real-time picture of the user's make-up, and returns the detection result to the user through server-side calculation. Compared with the model reasoning scheme of deep learning, this application consumes less computing cost in the algorithm processing link, and reduces the processing pressure of the server.

An embodiment of the present application further provides a makeup progress detection device, which is used to implement the above-mentioned makeup progress detection method for detecting eyebrow makeup progress. Referring to Figure 17, the device specifically includes:

The video acquisition module 1601 is used to acquire the initial frame image and the current frame image in the real-time makeup video in which the user is currently performing a specific makeup look;

The target area acquisition module 1602 is used to acquire the first target area image corresponding to the eyebrows from the initial frame image, and acquire the second target area image corresponding to the eyebrows from the current frame image;

The progress determination module 1603 is configured to determine the current makeup progress corresponding to the current frame image according to the first target area image and the second target area image.

The target area acquisition module 1602 is used to detect the first human face key point corresponding to the initial frame image; according to the first human face key point, obtain the corresponding human face area image of the initial frame image; according to the first human face key point included Eyebrow key points, obtain the first target area image corresponding to the eyebrows from the face area image.

The target area acquisition module 1602 is used to interpolate the eyebrow key points between the eyebrows and the eyebrow peaks included in the first human face key points to obtain a plurality of interpolation points; intercept all eyebrows between the eyebrows and the eyebrow peaks from the face area image A closed area formed by connecting key points and multiple interpolation points to obtain a part of the eyebrow image between the head of the eyebrow and the peak of the eyebrow; a closed area formed by connecting all eyebrow key points between the peak of the eyebrow and the end of the eyebrow is intercepted from the image of the face area , to obtain a part of the eyebrow image between the eyebrow peak and the eyebrow tail; the part of the eyebrow image between the eyebrow head and the eyebrow peak and the part of the eyebrow image between the eyebrow peak and the eyebrow tail are stitched into the first target area image corresponding to the eyebrows.

A progress determination module 1603, configured to respectively convert the first target area image and the second target area image into images containing preset single-channel components in the HSV color space; according to the converted first target area image and the second target area image , to determine the current makeup progress corresponding to the current frame image.

The progress determination module 1603 is used to calculate the absolute value of the difference of the preset single-channel component corresponding to the pixels in the same position in the converted first target area image and the second target area image respectively; the absolute value of the corresponding statistical difference satisfies Preset the number of pixels for makeup completion conditions; calculate the ratio between the statistical number of pixels and the total number of pixels in all target makeup areas in the first target area image to obtain the current makeup progress corresponding to the current frame image.

The progress determination module 1603 is further configured to perform binarization processing on the first target area image and the second target area image respectively to obtain the first binarized mask image corresponding to the first target area image and the second target area image corresponding to The second binarized mask image; the AND operation is performed on the first binarized mask image and the second binarized mask image to obtain the first target area image corresponding to the intersection area of the second target area image Two mask images; obtain the face area image corresponding to the initial frame image and the face area image corresponding to the current frame image; perform an AND operation on the face area image corresponding to the second mask image and the initial frame image to obtain the initial frame image The corresponding new first target area image; performing an AND operation on the second mask image and the face area image corresponding to the current frame image to obtain a new second target area image corresponding to the current frame image.

The device also includes: a boundary erosion module, which is used to respectively perform boundary erosion processing on the makeup area in the first target area image and the second target area image.

The target area acquisition module 1602 is used to perform rotation correction on the initial frame image and the first human face key point according to the first human face key point corresponding to the initial frame image; The image containing the face area is intercepted from the initial frame image; the image containing the face area is scaled to a preset size to obtain the face area image corresponding to the initial frame image.

The target area acquisition module 1602 is used to determine the left eye center coordinates and right eye center coordinates respectively according to the left eye key points and right eye key points included in the first human face key points; according to the left eye center coordinates and right eye center coordinates, Determine the rotation angle and the coordinates of the rotation center point corresponding to the initial frame image; perform rotation correction on the initial frame image and the key points of the first face according to the rotation angle and the rotation center point coordinates.

The target area acquiring module 1602 is configured to perform image capture on the face area included in the corrected initial frame image according to the corrected first key point of the human face.

The target area acquisition module 1602 is used to determine the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value from the corrected first face key point; according to the minimum abscissa value, the minimum ordinate value , the maximum abscissa value and the maximum ordinate value, determine the interception frame corresponding to the face area in the corrected initial frame image; according to the interception frame, intercept the image containing the face area from the corrected initial frame image.

The target area acquisition module 1602 is also configured to enlarge the clipping frame by a preset multiple; and according to the enlarged clipping frame, clip an image containing the face area from the corrected initial frame image.

The target area acquisition module 1602 is further configured to perform scaling and translation processing on the corrected first key points of the human face according to the size of the image including the human face area and the preset size.

The device also includes: a face detection module, which is used to detect whether the initial frame image and the current frame image only contain the face image of the same user; if so, perform the operation of determining the current makeup progress of the user for a specific makeup look; If not, send prompt information to the user's terminal, and the prompt information is used to prompt the user to keep only the face of the same user appearing in the real-time makeup video.

The makeup progress detection device provided in the above embodiments of the present application is based on the same inventive concept as the makeup progress detection method provided in the embodiments of the present application, and has the same beneficial effects as the methods adopted, run or realized by its stored application programs.

Embodiment six

An embodiment of the present application provides a method for detecting makeup progress, which is used for makeup progress corresponding to makeup looks that can be applied to the entire face, such as foundation and loose powder. Referring to Figure 18, this embodiment specifically includes the following steps:

Step 701: Obtain an initial frame image and a current frame image in a real-time makeup video in which the user is currently performing a specific makeup look.

The operation of step 701 is the same as the operation of step 601 in the fifth embodiment above, and will not be repeated here.

Step 702: According to the initial frame image, simulate and generate a result image after completing a specific makeup.

Use 3D rendering technology to render the effect of specific makeup on the initial frame image to obtain the result image. For example, a resultant image after applying foundation makeup is rendered on the initial frame image by using a 3D rendering technology.

Step 703: According to the initial frame image, the result image and the current frame image, determine the current makeup progress corresponding to the current frame image.

The server specifically determines the current makeup progress through the following steps S8-S10, including:

S8: Obtain the overall image brightness corresponding to the initial frame image, the result image, and the current frame image respectively.

For the initial frame image, first convert the initial frame image into a grayscale image, then calculate the grayscale average value of all pixels in the grayscale image corresponding to the initial frame image after conversion, and determine the calculated grayscale average value as the initial frame image The corresponding overall image brightness.

For the result image and the current frame image, the average gray value of all pixels in the gray image corresponding to the result image is calculated in the same manner to obtain the overall image brightness corresponding to the result image. And calculating the average gray value of all pixels in the gray image corresponding to the current frame image to obtain the overall image brightness corresponding to the current frame image.

Wherein, the overall image brightness includes the brightness of the face area in the image and the ambient brightness of the background where the face is located.

S9: Respectively acquire the brightness of the face area corresponding to the initial frame image, the result image and the current frame image.

The server specifically obtains the brightness of the face area corresponding to each image through the following steps S91-S93, specifically including:

S91: Respectively acquire face region images corresponding to the initial frame image, the result image and the current frame image.

The specific operation process of obtaining the corresponding face area images of the initial frame image, the result image and the current frame image is the same, and can refer to the process of steps S5-S6 in the fifth embodiment above, and will not be repeated here.

S92: Convert the face region images respectively corresponding to the initial frame image, the result image and the current frame image into face grayscale images.

S93: Calculating gray averages of pixels in the face gray images corresponding to the initial frame image, the result image, and the current frame image respectively, to obtain the brightness of the face area corresponding to the initial frame image, the result image, and the current frame image.

Convert the face area image corresponding to the initial frame image into a grayscale image of the face, calculate the average gray value of all pixels in the gray image of the face, and determine the average gray value as the face area corresponding to the initial frame image brightness. In the same manner, the brightness of the human face area corresponding to the result image and the brightness of the human face area corresponding to the current frame image are calculated respectively.

The above steps S8 and S9 may be executed in parallel or serially, and this embodiment of the present application does not limit the execution sequence of steps S8 and S9. Through the above steps S8 and S9, respectively obtain the overall image brightness and the face area brightness corresponding to the initial frame image, the overall image brightness and the face area brightness corresponding to the result image, and the overall image brightness and the face area brightness corresponding to the current frame image, and then The current makeup progress is determined through the operation of the following step S10.

S10: Determine the current makeup progress corresponding to the current frame image according to the overall image brightness and face area brightness corresponding to the initial frame image, the result image, and the current frame image.

Since the user applies makeup of the target type under a certain brightness environment, the brightness of the user’s face is superimposed on the influence of both the makeup effect and the ambient light. Therefore, it is necessary to eliminate the influence of the ambient light when determining the current makeup progress, and only consider the human body. Brightness changes in the face image due to makeup effects to ensure the accuracy of the current makeup progress obtained.

For the environmental change between the current frame image and the initial frame image, according to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the current frame image, determine the current frame image. - Change the brightness of the environment. Specifically, the difference between the brightness of the overall image corresponding to the initial frame image and the brightness of the face area corresponding to the initial frame image is calculated. This difference is the brightness of all background parts in the initial frame image, which is called the environment of the initial frame image brightness. And calculate the difference between the overall image brightness corresponding to the current frame image and the face area brightness corresponding to the current frame image, the difference is the brightness of all background parts in the current frame image, called the ambient brightness of the current frame image. Calculate the absolute value of the difference between the ambient brightness of the current frame image and the ambient brightness of the initial frame image, and determine the absolute value of the difference as the first ambient change brightness corresponding to the current frame image. The first environment change brightness reflects the change of environment brightness between the current frame image and the initial frame image.

Since the result image is obtained by rendering the makeup effect of the target type on the basis of the initial frame image, it may be affected by the rendering operation, making the brightness of the background part in the result image inconsistent with the brightness of the background part in the initial frame image, that is There may also be changes in ambient brightness between the resulting image and the original frame image. Therefore, the embodiment of the present application also determines the second environmental change brightness corresponding to the result image according to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the result image. Specifically, the difference between the brightness of the overall image corresponding to the result image and the brightness of the face area corresponding to the current frame image is calculated, and the difference is the brightness of all background parts in the result image, which is called the ambient brightness of the result image. Calculate the absolute value of the difference between the ambient brightness of the result image and the ambient brightness of the initial frame image, and determine the absolute value of the difference as the second ambient change brightness corresponding to the result image. The second environment change brightness reflects the change of environment brightness between the result image and the initial frame image.

After obtaining the first environment change brightness and the second environment change brightness in the above manner, according to the first environment change brightness, the second environment change brightness, the face area brightness corresponding to the initial frame image, the face area brightness corresponding to the current frame image, The brightness of the face area corresponding to the result image determines the current makeup progress corresponding to the current frame image.

First, according to the brightness of the first environmental change, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the current frame image, determine the makeup brightness change value corresponding to the current frame image. Specifically, calculate the difference between the brightness of the face area corresponding to the current frame image and the brightness of the face area corresponding to the initial frame image, and obtain the total brightness change value corresponding to the current frame image, which includes the makeup effect Changes in brightness caused by changes in ambient light and changes in brightness caused by changes in ambient light. The difference between the total brightness change value and the first environmental change brightness is calculated to obtain the makeup brightness change value corresponding to the current frame image.

The total brightness change value from the current frame image to the reference frame image face area is calculated by the above method, and the environmental brightness change from the current frame image to the reference frame image is deducted from the total brightness change value, and the obtained makeup brightness change value is even better. It is close to the actual brightness change value formed by the makeup operation of the target type, and the accuracy is high.

Then, according to the brightness of the second environmental change, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the result image, the makeup brightness change value corresponding to the result image is determined. Specifically, the difference between the brightness of the face area corresponding to the result image and the brightness of the face area corresponding to the initial frame image is calculated to obtain the total brightness change value corresponding to the result image, which includes the upper and lower values of the completed target type The brightness changes caused by makeup and the brightness changes caused by the rendering operation used to generate the resulting image on the background part. The difference between the total brightness change value and the second environmental change brightness is calculated to obtain the makeup brightness change value corresponding to the result image.

Calculate the total brightness change value from the result image to the face area of the reference frame image by the above method, and deduct the environmental brightness change caused by the rendering operation on the background part of the result image from the total brightness change value, and obtain the makeup brightness change value It is closer to the actual brightness change value formed by completing the makeup operation of the target type, and the accuracy is very high.

Finally, the ratio of the makeup brightness change value corresponding to the current frame image to the makeup brightness change value corresponding to the result image is calculated to obtain the current makeup progress corresponding to the current frame image.

The current makeup progress corresponding to the current frame image can be accurately obtained by calculating the ratio between the makeup brightness change value caused by makeup application in the current frame image and the makeup brightness change value caused by makeup completion in the result image. In this process, the influence of environmental brightness changes is eliminated, which greatly improves the accuracy of detecting the progress of makeup.

In some other embodiments of the present application, the environment brightness may change greatly during the process of applying makeup by the user, such as sudden brightening or sudden darkening. When the ambient brightness changes greatly, it has a great influence on the brightness change of the face area, which reduces the accuracy of makeup progress detection. Therefore, the embodiment of the present application obtains the first environmental change brightness from the current frame image to the reference frame image through the above method, and then compares the first environmental change brightness with a preset threshold, which can be 50 or 60, etc., in this application The embodiment does not limit the specific value of the preset threshold, which can be set according to requirements in practical applications.

If it is compared that the brightness of the first environmental change is less than or equal to the preset threshold, it is determined that the environmental brightness change between the current frame image and the reference frame image is not large, and the current makeup progress corresponding to the current frame image is determined in the above manner.

If it is compared that the brightness of the first environmental change is greater than the preset threshold, it indicates that the environmental brightness changes greatly between the current frame image and the reference frame image, and the makeup progress corresponding to the previous frame image is directly determined as the current makeup progress corresponding to the current frame image. schedule. And send the first prompt information to the user's terminal, and the user's terminal receives and displays the first prompt information to remind the user to return to the brightness environment corresponding to the initial frame image to apply makeup.

In this way, when the ambient brightness changes greatly, the makeup progress corresponding to the previous frame image can be directly used as the current makeup progress, saving computing resources. And avoid the situation that the detected makeup progress suddenly increases or decreases greatly, and improves the stability and accuracy of the makeup progress detection.

After the current makeup progress is determined by any of the above methods, the server sends the current makeup progress to the user's terminal. After receiving the current makeup progress, the user's terminal displays the current makeup progress. The current makeup progress can be a ratio or a percentage. The terminal can display the current makeup progress in the form of a progress bar.

During the process of the user's makeup, the makeup progress detection method provided by the embodiment of the present application can detect the makeup progress of each frame image after the first frame image relative to the first frame image in real time, and display the detected makeup progress to the user, It enables users to visually see the progress of their makeup and improve the efficiency of makeup.

In order to facilitate understanding of the method provided by the embodiment of the present application, the following description will be made in conjunction with the accompanying drawings. As shown in Figure 19, A1: Acquire the initial frame image and the current frame image in the user's real-time makeup video. A2: Render the makeup effect of the target type on the initial frame image to obtain the result image. A3: Obtain the overall image brightness corresponding to the initial frame image, the result image and the current frame image respectively, and then execute steps A6 and A7 in parallel. A4: Obtain the face region images corresponding to the initial frame image, the result image and the current frame image respectively. A5: Obtain the face area luminance of the face area images corresponding to the initial frame image, the result image and the current frame image respectively, and then execute steps A6 and A7 in parallel. A6: According to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the current frame image, determine the first environment change brightness corresponding to the current frame image, and then perform steps A8 and A9 in parallel . A7: According to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the result image, determine the second environment change brightness corresponding to the result image, and then execute steps A8 and A9 in parallel. A8: According to the brightness of the first environmental change, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the current frame image, determine the makeup brightness change value corresponding to the current frame image, and then perform step A10. A9: According to the brightness of the second environment change, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the result image, determine the change value of the makeup brightness corresponding to the result image. A10: Calculate the ratio of the makeup brightness change value corresponding to the current frame image to the makeup brightness change value corresponding to the result image to obtain the current makeup progress corresponding to the current frame image.

As shown in Figure 20, according to the reference frame image and its corresponding first face key point, the current frame image and its corresponding second face key point, and the result image and its corresponding third face key point, calculate The overall image brightness corresponding to each of the three images is displayed. Then, according to the face key point information corresponding to each image, the faces in the three images are corrected and cropped respectively, and the face area images corresponding to the reference frame image, the current frame image and the result image are respectively obtained. The face area luminance corresponding to the cropped three face area images is calculated respectively. According to the overall image brightness and face area brightness corresponding to each image, the first environment change brightness corresponding to the current frame image is calculated. If the brightness of the first environmental change is greater than the preset threshold, the user is prompted to return to the brightness environment corresponding to the initial frame image to apply makeup. If the brightness of the first environmental change is less than or equal to the preset threshold, then calculate the brightness difference between the brightness of the face area of the current frame image and the brightness of the face area of the initial frame image, and calculate the difference between the brightness difference and the brightness of the first environmental change difference. And calculate the brightness difference between the brightness of the face area of the result image and the brightness of the face area of the initial frame image, and calculate the difference between the brightness difference and the brightness of the second environment change. Calculate the ratio between the above-mentioned difference corresponding to the current frame image and the above-mentioned difference corresponding to the result image to obtain the current makeup progress.

In the embodiment of the present application, on the basis of the initial frame image of the user's makeup process, the result image after the makeup is completed is simulated and generated. The current makeup progress is determined according to the brightness change of the face area in the current frame image, the initial frame image, and the resulting image, and the influence of the environment brightness change is deducted from the brightness change of the face area, which greatly improves the accuracy of makeup progress detection It is flexible, and does not use a deep learning model, does not need to collect a large amount of data in advance, has a small amount of calculation, low cost, reduces the processing pressure on the server, improves the efficiency of makeup progress detection, and can meet the real-time requirements of makeup progress detection.

Embodiment seven

An embodiment of the present application provides a makeup progress detection method, which is used for the makeup progress corresponding to the concealer makeup. Referring to Figure 21, this embodiment specifically includes the following steps:

Step 801: Obtain an initial frame image and a current frame image in a real-time makeup video in which the user is currently performing a specific makeup look.

The operation of step 801 is the same as the operation of step 601 in the fifth embodiment above, and will not be repeated here.

Step 802: Respectively acquire facial blemish information corresponding to the initial frame image and the current frame image.

Firstly, the face area images corresponding to the initial frame image and the current frame image are acquired respectively. The specific operation process of obtaining the face area images corresponding to the initial frame image and the current frame image is the same, and can refer to the process of steps S5-S6 in the fifth embodiment above, and will not be repeated here.

Then use the preset skin detection model to detect the number of blemishes corresponding to each blemish category in the face area image corresponding to the initial frame image, and use each detected blemish category and its corresponding number of blemishes as the face blemish corresponding to the initial frame image information. Similarly, the number of blemishes corresponding to each blemish category in the face area image corresponding to the current frame image is detected through the preset skin detection model, and the facial blemish information corresponding to the current frame image is obtained.

Among them, the preset skin detection model is obtained by training the neural network model through a large number of face images in advance, and can identify and classify blemishes such as acne, spots, and wrinkles on the face in the face image. The above blemish categories include one or more of blemishes such as acne, spots, and wrinkles. The number of pimples, the number of spots, the number of wrinkles, etc. in the face area images corresponding to the initial frame image and the current frame image are identified through the preset skin detection model.

Step 803: Determine the current makeup progress corresponding to the current frame image according to the initial frame image and the current frame image and their corresponding facial blemish information.

The server specifically determines the current makeup progress through the following steps B1-B3, including:

B1: According to the facial blemish information corresponding to the initial frame image and the facial blemish information corresponding to the current frame image, calculate the facial blemish difference value between the current frame image and the initial frame image.

The facial blemish information includes a blemish category and a corresponding blemish number, and the blemish category includes one or more of blemishes, spots, wrinkles, and the like. Calculate the difference between the number of defects corresponding to the initial frame image under each defect category and the number of defects corresponding to the current frame image, and then calculate the sum of the differences corresponding to each defect category, and use the obtained sum as the current frame image The face blemish difference value from the original frame image.

For example, assuming that the blemish category includes acne and blemishes, the facial blemish information corresponding to the initial frame image includes 5 blemishes and 4 blemishes. The number of pimples included in the facial blemish information corresponding to the current frame image is 3, and the number of spots is 1. Then the difference of the acne between the initial frame image and the current frame image is 2, and the difference of the spots is 3. Calculate the sum of the difference between the acne difference and the spot to be 5, and then the face blemish difference between the current frame image and the initial frame image is 5.

The facial blemish difference value can reflect the difference in smoothness and detail between the current frame image and the initial frame image, which is caused by various factors such as concealer makeup, light changes, and shooting angle changes. The embodiment of the present application has measured through a large number of experiments that when the difference value of facial blemishes is greater than a certain value, it can be determined that the difference is mainly caused by concealer makeup, and the certain value is configured in the server as a preset threshold value. The preset threshold value can be 4 or 5 etc.

After the facial blemish difference value is calculated by the above method, compare the facial blemish difference value with the preset threshold, if the facial blemish difference value is greater than the preset threshold value, it indicates that the current facial change caused by concealer makeup is relatively obvious, Subsequently, the current makeup progress is determined through the operation of step B2. If the facial blemish difference value is less than or equal to the preset threshold, it indicates that the current facial change caused by concealer makeup is very small, and the current makeup progress is determined through the operation of step B3.

B2: If the facial blemish difference value is greater than the preset threshold, calculate the current makeup progress corresponding to the current frame image according to the facial blemish difference value and the facial blemish information corresponding to the initial frame image.

The sum of the number of blemishes corresponding to each blemish category in the facial blemish information corresponding to the initial frame image is calculated to obtain the total number of blemishes. Calculate the ratio between the facial blemish difference value and the total number of blemishes, and use the ratio as the current makeup progress corresponding to the current frame image.

For example, it is assumed that the facial blemish information corresponding to the initial frame image includes 5 acnes and 4 spots. The number of pimples included in the facial blemish information corresponding to the current frame image is 3, and the number of spots is 1. Then the facial blemish difference value between the current frame image and the initial frame image is 5, the total number of blemishes corresponding to the initial frame image is 9, and the current makeup progress corresponding to the current frame image is 5/9.

When the face scene difference value between the current frame image and the initial frame image is greater than the preset threshold, that is, when the face difference is mainly caused by concealer makeup, directly calculate the difference between the face defect difference value and the total number of defects corresponding to the initial frame image The ratio of is used as the current makeup progress, the makeup progress determination process is simple, the amount of calculation is small, the current makeup progress can be quickly determined, the efficiency is high, and it can meet the real-time requirements of the concealer makeup progress detection.

After the current makeup progress is determined in the above manner, the server sends the current makeup progress to the user's terminal. After receiving the current makeup progress, the user's terminal displays the current makeup progress. The current makeup progress can be a ratio or a percentage. The terminal can display the current makeup progress in the form of a progress bar.

B3: If the facial blemish difference value is less than or equal to the preset threshold, obtain the result image after the user completes the concealment and makeup, and determine the current makeup progress corresponding to the current frame image according to the initial frame image, result image and current frame image.

When it is judged that the above facial blemish difference value is less than or equal to the preset threshold, it is considered that the difference between the current frame image and the initial frame image is small, if the face blemish difference value and the total number of blemishes in the initial frame image are used directly The current makeup progress, there will be a big error. Therefore, in this case, the embodiment of the present application does not use the difference value of facial blemishes and the total number of blemishes in the initial frame image to determine the current makeup progress, but first obtains the result image after the user completes the concealment and makeup application, and then uses the initial frame image to determine the current makeup progress. , the result image and the current frame image, and determine the current makeup progress corresponding to the current frame image.

First, use 3D rendering technology to render the effect of concealer and makeup on the initial frame image to obtain the result image. Before generating the result image in the embodiment of this application, the initial frame image can be rotated and corrected so that the line connecting the two eyes in the image is parallel to the horizontal line, and then the completed concealer can be rendered on the basis of the rotated and corrected initial frame image. The result image after makeup, so that the line connecting the two eyes of the face in the result image is also parallel to the horizontal line, so that there is no need to perform rotation correction on the result image, which saves the amount of calculation.

After obtaining the result image after the concealer and makeup are completed, the third face key point corresponding to the result image is detected by the detection model used to detect the key point of the face, and according to the third face key point, through step S5 in the fifth embodiment The method of - S6 intercepts the face area image from the result image, and the specific interception process will not be repeated here.

Obtain the face area image corresponding to the initial frame image, the face area image corresponding to the current frame image, and the face area image corresponding to the result image, and then determine the current makeup progress according to the face area images corresponding to the three.

The color space of the face area image corresponding to the initial frame image obtained above, the face area image corresponding to the current frame image, and the result image are RGB color space. In the embodiments of the present application, a large number of tests have been conducted in advance to determine the influence of concealer and makeup on each channel component of the color space, and it is found that the influence on each color channel in the RGB color space is not much different. The HLS color space is composed of three components: Hue (hue), Saturation (saturation) and Light (brightness). Through experiments, it is found that concealer and makeup can cause significant changes in the saturation component of the HLS color space. Therefore, firstly, the face region images corresponding to the initial frame image, the result image and the current frame image are converted from the RGB color space to the HLS color space. Then the saturation channel is separated from the HLS color space of the converted image, and the images of the face regions corresponding to the initial frame image, the result image and the current frame image respectively contain only the saturation channel images in the HLS color space.

After the above conversion, the smoothing factors corresponding to the face area images of the converted initial frame image, the result image and the current frame image are respectively calculated through a preset filtering algorithm. Wherein, the preset filtering algorithm may be a Laplacian algorithm or a Gaussian filtering algorithm. Taking the Gaussian filtering algorithm as an example, the smoothing factor corresponding to each face area image can be calculated according to a Gaussian kernel of a preset size. The preset size may be 7*7, or other values. The embodiment of the present application does not limit the specific value of the preset size.

After the smoothing factors corresponding to the initial frame image, the result image and the current frame image are obtained, the current makeup progress corresponding to the current frame image is determined accordingly. Specifically, calculate the first difference between the smoothing factor corresponding to the current frame image and the smoothing factor corresponding to the initial frame image, and calculate the second difference between the smoothing factor corresponding to the result image and the smoothing factor corresponding to the initial frame image. Calculate the ratio between the first difference and the second difference, and use the ratio as the current makeup progress corresponding to the current frame image.

After the current makeup progress is determined in the above manner, the server sends the current makeup progress to the user's terminal. After receiving the current makeup progress, the user's terminal displays the current makeup progress. The current makeup progress can be a ratio or a percentage. The terminal can display the current makeup progress in the form of a progress bar.

During the process of the user's makeup, the makeup progress detection method provided by the embodiment of the present application can detect in real time the makeup progress of each frame image after the initial frame image relative to the initial frame image, and display the detected makeup progress to the user, so that the user You can intuitively see the progress of your concealer makeup and improve your makeup efficiency.

In order to facilitate understanding of the method provided by the embodiment of the present application, the following description will be made in conjunction with the accompanying drawings. As shown in Figure 22, according to the initial frame image and its corresponding first face key point, the current frame image and its corresponding second face key point, the face in the initial frame image and the current frame image are corrected respectively and cropping, and then call the skin detection model to detect the facial defect information corresponding to the initial frame image and the current frame image respectively. Based on the facial blemish information of the two, the facial blemish difference value between the current frame image and the initial frame image is calculated. If the facial blemish difference value is greater than the preset threshold, the ratio between the facial blemish difference value and the total number of blemishes corresponding to the initial frame image is calculated to obtain the current makeup progress. If the facial blemish difference value is less than or equal to the preset threshold, the result image after concealment and makeup is generated by rendering on the initial frame image, and the third face key point corresponding to the result image is detected, and according to the third face key point Rectify and crop the faces in the resulting image. Convert the corresponding face area images of the initial frame image, current frame image and result image into images containing only the saturation channel in the HLS color space. The smoothing factors corresponding to the converted three human face region images are respectively calculated through a preset filtering algorithm. Calculate a first difference between the smoothing factor corresponding to the current frame image and the smoothing factor corresponding to the initial frame image. A second difference between the smoothing factor corresponding to the result image and the smoothing factor corresponding to the initial frame image is calculated. Calculate the ratio between the first difference and the second difference, and use the ratio as the current makeup progress corresponding to the current frame image.

In the embodiment of the present application, the current frame image and the initial frame image of the user's makeup process are acquired, and the difference value of facial blemishes from the current frame image to the initial frame image is determined. When the facial blemish difference value is greater than the preset threshold, calculate The ratio between the facial blemish difference value and the total number of blemishes corresponding to the initial frame image is the current makeup progress of concealer makeup. When the facial blemish difference value is less than or equal to the preset threshold, the result image of the concealed makeup is simulated on the basis of the initial frame image, and the corresponding smoothing factors of the initial frame image, the result image and the current frame image are respectively determined, Calculate the difference between the smoothing factor between the current frame image and the initial frame image, and calculate the difference between the smoothing factor between the result image and the initial frame image, and the ratio between the two differences is the current makeup progress.

In this way, when the facial blemish difference between the current frame image and the initial frame image is greater than the preset threshold, the process of determining the makeup progress is simple, the amount of calculation is small, and the current makeup progress can be quickly determined, which is very efficient and can meet the needs of concealer makeup. Real-time requirements for progress detection. When the facial blemish difference value is less than or equal to the preset threshold, a filtering algorithm is introduced to calculate the smoothing factor corresponding to each image, and the makeup progress is determined according to the change of the smoothing factor, which can perform more accurate calculations for more subtle changes, avoiding This prevents the short-term sudden increase in makeup progress.

Further, in the embodiment of the present application, the face key points are used to correct and crop the user's face area in the video frame, so as to improve the accuracy of recognizing the face area.

The embodiment of the present application can accurately detect the concealer makeup progress only through image processing, with a small amount of calculation and low cost, which reduces the processing pressure on the server, improves the efficiency of concealer makeup progress detection, and can meet the real-time detection of concealer makeup progress. Performance requirements, reducing the algorithm's dependence on hardware resources and human input costs.

Embodiment eight

1. A makeup color recognition method, the method comprising:

Get the user's makeup image;

Identifying the hue category to which the color of the preset part of the user's face in the makeup image belongs;

According to the tone category corresponding to the preset part, the user's makeup tone is determined.

2. According to 1, identifying the hue category to which the color of the preset part of the user's face in the makeup image belongs, including:

Detecting the key points of the face corresponding to the makeup image;

Acquiring a face area image corresponding to the makeup image according to the key points of the face;

Identifying the hue category to which the color of the preset part of the user's face in the face area image belongs.

3. According to 2, identify the hue category to which the color of the preset part of the user's face in the face area image belongs, including:

According to the key points of the human face, intercepting the image of the target area corresponding to the preset part from the image of the human face area;

Acquiring the pixel dominant color of the image of the target area;

According to the main color of the pixel, the hue category to which the color of the preset part belongs is determined.

4. According to 3, the preset part includes facial skin; according to the key points of the face, the target area image corresponding to the preset part is intercepted from the face area image, including:

According to the key points of the human face, a facial image is intercepted from the facial area image;

According to the key points of the human face, the eyebrow area, the eye area and the mouth area are cut out from the facial image to obtain an image of the target area corresponding to the facial skin.

5. According to 3, the preset part includes an eye shadow part; according to the key points of the face, the target area image corresponding to the preset part is intercepted from the face area image, including:

According to the eye key points included in the human face key points, an eye image is intercepted from the human face region image;

performing image expansion processing on the eye image for a preset number of times;

According to the eye key points included in the eye key points, the eye area is cut out from the dilated eye area to obtain a target area image corresponding to the eye shadow part.

6. According to 3, the preset part includes the mouth; according to the key points of the face, the target area image corresponding to the preset part is intercepted from the face area image, including:

Respectively interpolate the upper lip upper edge key point, the upper lip lower edge key point, the lower lip upper lip key point and the lower lip lower edge key point to obtain the upper lip upper edge interpolation point, the upper lip upper edge key point, and the upper lip edge key point. Lower edge interpolation point, lower lip upper edge interpolation point and lower lip lower edge interpolation point;

According to the key point of the upper edge of the upper lip, the interpolation point of the upper edge of the upper lip, the key point of the lower edge of the upper lip and the interpolation point of the lower edge of the upper lip, the upper lip image is intercepted from the image of the human face region ;

According to the key point of the upper edge of the lower lip, the interpolation point of the upper edge of the lower lip, the key point of the lower edge of the lower lip and the interpolation point of the lower edge of the lower lip, the lower lip image is intercepted from the image of the human face region ;

Stitching the upper lip image and the lower lip image into a target area image corresponding to the mouth.

7. According to 3, according to the main color of the pixel, determine the hue category to which the color of the preset part belongs, including:

converting the color space of the main color of the pixel to the preset color space corresponding to the preset part;

According to each color channel value of the main color of the pixel in the preset color space, determine the hue interval to which each color channel value belongs;

The hue category corresponding to the hue interval is determined as the hue category to which the color of the preset part belongs.

8. According to 7, the preset parts include at least one of facial skin, eye shadow parts, and mouth; the preset color space corresponding to facial skin includes LAB color space, and the preset color space corresponding to eye shadow parts and mouth includes HSV color space.

9. According to any one of 1-8, the preset parts include one or more parts of facial skin, eye shadow parts and mouth, and the makeup of the user is determined according to the color tone category corresponding to the preset parts shades, including:

Determine the color coefficient corresponding to the hue category of each preset part;

calculating the makeup color coefficient of the user according to the preset weight corresponding to each preset part and the color coefficient;

The tone category corresponding to the makeup color coefficient is determined as the user's makeup tone.

10. According to any one of 2-8, according to the key points of the human face, the facial area image corresponding to the makeup image is acquired, including:

performing rotation correction on the makeup image and the key points of the human face according to the key points of the human face;

According to the corrected key points of the human face, an image containing a human face area is intercepted from the corrected makeup image;

The image containing the face area is scaled to a preset size to obtain a face area image corresponding to the makeup image.

11. According to 10, performing rotation correction on the makeup image and the key points of the face according to the key points of the face, including:

Determine left eye center coordinates and right eye center coordinates respectively according to the left eye key points and right eye key points included in the face key points;

According to the left eye center coordinates and the right eye center coordinates, determine the rotation angle and rotation center point coordinates corresponding to the makeup image;

Perform rotation correction on the makeup image and the key points of the human face according to the rotation angle and the coordinates of the rotation center point.

12. According to 10, according to the corrected key points of the human face, the image containing the human face area is intercepted from the corrected makeup image, including:

According to the corrected key points of the human face, image interception is performed on the human face area contained in the corrected makeup image.

13. According to 12, according to the corrected key points of the human face, image interception is performed on the human face area contained in the corrected makeup image, including:

Determine the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value from the corrected described face key points;

According to the minimum abscissa value, the minimum ordinate value, the maximum abscissa value, and the maximum ordinate value, determine a clipping frame corresponding to the face area in the corrected makeup image;

According to the clipping frame, an image including the face area is clipped from the rectified makeup image.

14. According to 13, also includes:

Enlarging the interception frame by a preset multiple;

According to the enlarged clipping frame, an image including the face area is clipped from the corrected makeup image.

15. According to 10, also includes:

According to the size of the image including the human face area and the preset size, zooming and translation processing is performed on the corrected key points of the human face.

16. According to any one of 1-8, it also includes:

Detecting whether at least one complete human face image is included in the makeup image;

If yes, then perform the operation of identifying the tone category to which the color of the preset part of the user's face in the makeup image belongs;

If not, sending prompt information to the user's terminal, where the prompt information is used to prompt the user to provide a makeup image including at least one complete human face image.

17. A makeup color recognition device, comprising:

An acquisition module, configured to acquire the makeup image of the user;

An identification module, configured to identify the tone category to which the color of the preset part of the user's face in the makeup image belongs;

A determining module, configured to determine the user's makeup tone according to the tone category corresponding to the preset part.

18. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor runs the computer program to implement any one of 1-16. described method.

19. A computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the method described in any one of 1-16.

An embodiment of the present application provides a makeup color recognition method. The method recognizes the hue category to which the color of the user's preset part in the makeup image belongs, and then determines the user's makeup tone according to the hue category corresponding to the preset part. The user's makeup tone can be accurately identified only through image processing without using a deep learning model for processing. The amount of calculation is small, the cost is low, and the processing pressure on the server is reduced.

Referring to Figure 23, the method specifically includes the following steps:

Step 901: Obtain a makeup image of a user.

The user's makeup image includes at least one face image. The makeup image can be a single image, or any video frame in the user's makeup video. The execution subject of this embodiment of the application is a server. A client that matches the service of identifying makeup color provided by the server is installed on a user terminal such as a user's mobile phone or computer. The client is provided with an interface for submitting makeup images. When the user needs to identify the makeup color corresponding to a certain makeup, the user submits the makeup image through this interface.

Specifically, when the client detects that the user clicks on the interface, a makeup color recognition interface is displayed, and the interface may include a shooting button and/or a local file upload interface. If the client detects that the user clicks the shooting button, the camera on the user terminal is invoked to take a makeup image or a makeup video. Send the captured makeup image or makeup video to the server. If the client detects that the user clicks on the local file upload interface, a list of local folders is displayed so that the user can select a desired makeup image or makeup video therefrom. The client sends the makeup image or makeup video selected by the user to the server.

The server receives the makeup image or makeup video sent by the user terminal. If the server receives the makeup video sent by the user terminal, the currently received current frame image is used as the makeup image to be recognized.

In other embodiments of the present application, after the server obtains the makeup image of the user, it also detects whether the makeup image contains at least one complete human face image. If the makeup image includes one or more complete human face images, the makeup tone corresponding to each complete human face image in the makeup image is identified according to the method provided in this embodiment. If it is detected that the makeup image does not contain a face image, or all the face images included are incomplete, a prompt message is sent to the user's terminal. The user's terminal receives and displays the prompt information to prompt the user to provide a makeup image including at least one complete face image. For example, the prompt information may be "please keep at least one complete human face in the makeup image".

After the server obtains the makeup image of the user through this step, it determines and recognizes the makeup tone of the user's face in the makeup image through the operations of the following steps 902 and 903 .

Step 902: Identify the tone category to which the color of the preset part of the user's face in the makeup image belongs.

The makeup image includes at least one complete human face image. Since the process of identifying the makeup tone of each complete human face image is the same, the embodiment of the present application uses the process of identifying the makeup tone of a complete human face image as an example for illustration. For at least one complete human face image included in the makeup image, any one of the complete human face images is selected as the user's face currently to be processed.

The server specifically recognizes the hue category to which the color of the preset part of the user's face belongs through the following steps S1-S3, including:

S1: Detect the key points of the face corresponding to the makeup image.

The server is configured with a pre-trained detection model for detecting key points of the face, and the interface service of key point detection of the face is provided through the detection model. After the server acquires the user's makeup image, it calls the interface service of facial key point detection, and identifies all the key points of the user's face in the makeup image through the detection model.

Among them, the recognized key points of the human face include key points on the outline of the user's face and key points of the mouth, nose, eyes, eyebrows and other parts. The number of recognized face key points may be 106.

S2: Acquire the face area image corresponding to the makeup image according to the key points of the face.

The server specifically obtains the face area image corresponding to the current user's face through the following steps S20-S22, including:

S20: Perform rotation correction on the makeup image and the key points of the face according to the key points of the face.

Specifically, according to the left eye key point and the right eye key point included in the face key points corresponding to the user's face, the left eye center coordinates and the right eye center coordinates are respectively determined. All the left-eye key points in the left-eye area and all the right-eye key points in the right-eye area are determined from the face key points. Take the mean value of the abscissa of all the determined key points of the left eye, and take the mean of the ordinate of all the key points of the left eye, combine the mean value of the abscissa and the mean value of the ordinate corresponding to the left eye into a coordinate, and determine the coordinate is the coordinates of the center of the left eye. Determine the coordinates of the center of the right eye in the same manner.

Then, according to the center coordinates of the left eye and the center coordinates of the right eye, the corresponding rotation angle and the coordinates of the rotation center point of the makeup image are determined. As shown in Figure 4, the horizontal difference dx and the vertical difference dy of the two coordinates are calculated according to the left eye center coordinates and the right eye center coordinates, and the line connecting the two eyes between the left eye center coordinates and the right eye center coordinates is calculated length d. According to the length d of the connecting line between the two eyes, the horizontal difference dx and the vertical difference dy, the angle θ between the connecting line between the two eyes and the horizontal direction is calculated, and the included angle θ is the corresponding rotation angle of the makeup image. Then calculate the coordinates of the center point of the line connecting the two eyes according to the center coordinates of the left eye and the center coordinates of the right eye, and the coordinates of the midpoint point are the coordinates of the rotation center point corresponding to the makeup image.

According to the calculated rotation angle and the coordinates of the rotation center point, the makeup image and the key points of the face are corrected for rotation. Specifically, the rotation angle and the coordinates of the rotation center point are input into a preset function used to calculate the rotation matrix of the picture, and the preset function may be the function cv2.getRotationMatrix2D() in OpenCV. The rotation matrix corresponding to the makeup image is obtained by calling this preset function. Then calculate the product of the makeup image and the rotation matrix to obtain the corrected makeup image. Correcting the makeup image using the rotation matrix can also be done by calling the function cv2.warpAffine() in OpenCV.

For face key points, each face key point needs to be corrected one by one to correspond to the corrected makeup image. When correcting the key points of the face one by one, two coordinate system conversions are required. The first time the coordinate system with the upper left corner of the makeup image as the origin is transformed into the coordinate system with the lower left corner as the origin, and the second time is further transformed with The coordinate system with the lower left corner as the origin is transformed into a coordinate system with the above-mentioned rotation center point coordinates as the coordinate origin, as shown in FIG. 5 . After two coordinate system transformations, the transformation of the following formula (1) is performed on each key point of the face, and the rotation correction of the key points of the face can be completed.

In formula (1), x ₀ and y ₀ are the abscissa and ordinate of the key points of the face before rotation correction, respectively, x and y are the abscissa and ordinate of the key points of the face after rotation correction, and θ is the above-mentioned rotation angle.

The corrected makeup image and face key points are based on the entire image. The entire image not only contains the user’s face information, but also includes other redundant image information. Therefore, it is necessary to perform artificial intelligence on the corrected image through the following step S21 Cropping of the face region.

S21: According to the corrected key points of the human face, an image including the human face area is intercepted from the corrected makeup image.

According to the key points of the corrected face, image interception is performed on the area containing the user's face in the corrected makeup image. Firstly, the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value are determined from the corrected face key points. Then according to the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value, determine the clipping frame corresponding to the face area in the corrected makeup image. Specifically, the minimum abscissa value and the minimum ordinate value form a coordinate point, and the coordinate point is used as the upper left vertex of the interception frame corresponding to the face area. The maximum abscissa value and the maximum ordinate value form another coordinate point, and this coordinate point is used as the vertex of the lower right corner of the interception frame corresponding to the face area. Determine the position of the clipping frame in the corrected makeup image according to the above-mentioned upper left corner vertex and lower right corner vertex, and cut out the image located in the clipping frame from the corrected makeup image, that is, cut out the image containing the user's face.

In other embodiments of the present application, in order to ensure that all of the user's face area is intercepted, and to avoid incomplete interception resulting in a large error in the follow-up makeup progress detection, the above interception frame can also be enlarged by a preset multiple, The preset multiple can be 1.15 or 1.25 and so on. The embodiment of the present application does not limit the specific value of the preset multiple, and the preset multiple can be set according to requirements in practical applications. After the aforementioned clipping frame is enlarged by a preset multiple, the image located in the enlarged clipping frame is clipped from the corrected makeup image, so as to clip an image including the complete face area of the user.

S22: Scale the image containing the face area to a preset size to obtain an image of the face area corresponding to the makeup image.

After the image including the face area of the user is intercepted from the makeup image in the above manner, the image including the face area is scaled to a preset size to obtain the face area image corresponding to the makeup image. The preset size may be 390×390 or 400×400 or the like. The embodiment of the present application does not limit the specific value of the preset size, which can be set according to requirements in practical applications.

In order to make the key points of the face adapt to the scaled face area image, after scaling the captured image containing the face area to the preset size, it is also necessary to Size, scale and translate the above-mentioned corrected face key points. Specifically, according to the size of the image containing the face area before scaling and the preset size to which the image needs to be scaled, determine the translation direction and translation distance of each key point of the face, and then according to the translation direction and translation distance corresponding to each key point of the face The translation distance is to perform a translation operation on each key point of the face, and record the coordinates of each key point of the face after translation.

The face area image is obtained from the makeup image in the above way, and the key points of the face are adapted to the obtained face area image through operations such as rotation correction and translation scaling. Afterwards, the tone category to which the color of the preset part of the user's face belongs is identified through the operation of step S3.

S3: Identify the tone category to which the color of the preset part of the user's face in the face area image belongs.

Specifically, through the following steps S30-S32, the color tone category corresponding to the preset position is identified, including:

S30: According to the key points of the face, intercept the target area image corresponding to the preset part from the face area image.

In the embodiment of the present application, the preset parts may include one or more of facial skin, eye shadow parts, mouth and other parts.

For the facial skin, firstly, according to all the key points of the user's face, the face image is cut out from the face area image to remove the hair and background area. Then, according to all the eyebrow key points included in the face key points, the eyebrow area is removed from the face image. According to all the eye key points included in the face key points, the eye area is cut out from the face image. And according to all the key points of the mouth included in the key points of the face, the mouth area is removed from the face image. Thus, the image of the target area including only the facial skin is obtained, which facilitates subsequent identification of the facial skin tone, eliminates the interference of other parts of the color on the identification of the facial skin tone, and improves the accuracy of identifying the facial skin tone.

For the eye shadow part, first, according to the eye key points included in the face key points, the eye image is intercepted from the face area image, and the eye image includes the upper and lower eyelid areas and the eye area. Eye shadow makeup is located in the upper and lower eyelid areas, and the color of the whites and eyeballs in the eye area will interfere with the color recognition of the eye shadow area. Therefore the eye area needs to be removed. Firstly, image expansion processing is performed on the obtained eye image for a preset number of times, so as to ensure that the eye image with eye makeup includes the eye shadow part after the expansion processing. The preset number of times may be 3 times or 4 times, etc. Then, according to the eye key points included in the eye key points, the eye area is removed from the expanded eye area to obtain the target area image corresponding to the eye shadow. Such follow-up is convenient for identifying the color tone of the eye shadow, eliminates the interference of eye color on the identification of the eye shadow color tone, and improves the accuracy of identifying the eye shadow color tone.

For the mouth, first determine the key points of the upper edge of the upper lip, the key points of the lower edge of the upper lip, the key points of the upper edge of the lower lip and the key points of the lower edge of the user's face from the key points of the face corresponding to the user's face. Perform linear interpolation on key points along the upper edge of the upper lip to obtain multiple interpolation points on the upper edge of the upper lip. Interpolate the key points on the lower edge of the upper lip to obtain multiple interpolation points on the lower edge of the upper lip. According to the upper lip upper edge key point, upper lip upper edge interpolation point, upper lip lower edge key point and upper lip lower edge interpolation point, the upper lip image is intercepted from the face area image.

Specifically, a smoother curve can be obtained by sequentially connecting a plurality of key points on the upper edge of the upper lip with a plurality of interpolation points on the upper edge of the upper lip, and the curve is the boundary line of the upper edge of the upper lip. A smoother curve can be obtained by sequentially connecting multiple key points on the lower edge of the upper lip with multiple interpolation points on the lower edge of the upper lip, which is the boundary line of the lower edge of the upper lip. The area enclosed by the boundary line of the upper edge of the upper lip and the lower edge of the upper lip is the area of the upper lip. The upper lip area is cut out from the face area image to obtain an upper lip image.

Perform linear interpolation on key points on the upper edge of the lower lip to obtain multiple interpolation points on the upper edge of the lower lip. Perform linear interpolation on the key points along the lower edge of the lower lip to obtain multiple interpolation points on the lower edge of the lower lip. According to the key point of the upper edge of the lower lip, the interpolation point of the upper edge of the lower lip, the key point of the lower edge of the lower lip and the interpolation point of the lower edge of the lower lip, the lower lip image is cut out from the image of the face region.

Specifically, a smoother curve can be obtained by sequentially connecting a plurality of key points on the upper edge of the lower lip with a plurality of interpolation points on the upper edge of the lower lip, and the curve is the boundary line of the upper edge of the lower lip. A smoother curve can be obtained by sequentially connecting multiple key points on the lower edge of the lower lip with multiple interpolation points on the lower edge of the lower lip, which is the boundary line of the lower edge of the lower lip. The area enclosed by the boundary line of the upper edge of the lower lip and the lower edge of the lower lip is the area of the lower lip. The lower lip area is cut out from the face area image to obtain the lower lip image.

The obtained upper lip image and lower lip image are stitched into the target area image corresponding to the mouth. In this way, the area between the upper lip and the lower lip can be removed, especially when the mouth is opened, the image of the inner area of the mouth can be removed, so as to avoid the area between the upper lip and the lower lip from affecting the recognition of the color of the mouth, and improve the recognition of lipstick in the mouth Tonal accuracy.

The embodiment of the present application does not specifically limit the preset position, and the preset position can also be other parts, and according to the key points of the face, the target area image corresponding to the preset position is deducted from the face area image, so that the target area image can be obtained from the target area. The color of the preset part is recognized in the image.

S31: Obtain the dominant color of the pixels of the image of the target area.

After the target area image corresponding to the preset location is obtained through the above step S30, the main color tone is extracted from the target area image to determine the pixel main color of the target area image. Specifically, convert the target area image read by opencv from the BGRA color space to the RGBA color space that conforms to the PIL reading format, then call the get_palette(image) function to create a palette, and use the median cut algorithm to gather similar colors , to obtain the pixel dominant color of the target area image.

S32: Determine the hue category to which the color of the preset part belongs according to the main color of the pixels in the image of the target area.

The main color of pixels in the image of the target area includes the values of the three color channels R, G, and B. The makeup tones of different preset parts have their own characteristics. For example, the makeup tones of facial skin may include pink, fair, natural, wheaten, ancient For different tones such as copper and dark, it is difficult to judge the hue category corresponding to the main color of the pixel through the RGB color space for lighter tones such as pink, white, and natural. For the lipstick tone of the mouth, it may include pink nude, red, orange, reddish brown and other shades. It is difficult to distinguish pink, red and other shades with small differences through the RGB color space. Therefore, in the embodiment of the present application, preset color spaces corresponding to different preset parts are pre-configured, and different makeup tones of the preset parts can be more easily distinguished through the preset color spaces corresponding to the preset parts. Moreover, the embodiment of the present application also pre-configures a plurality of hue categories corresponding to different preset positions, and configures a hue range corresponding to each hue category. The hue interval corresponding to the hue category includes the value interval of each color channel of the color belonging to the hue category in the corresponding preset color space.

After obtaining the dominant pixel color of the image of the target area through step S31, the color space of the dominant color of the pixel of the image of the target area is converted from RGB to the preset color space corresponding to the preset location. Then, according to each color channel value of the main color of the pixel in the preset color space, determine the hue interval to which each color channel value belongs. The tone category corresponding to the tone interval is determined as the tone category to which the color of the preset part belongs.

In the embodiment of the present application, the preset part includes at least one of facial skin, eye shadow part, and mouth; the preset color space corresponding to the facial skin may include the LAB color space, and the LAB color space is composed of one brightness channel and two Composed of color channels, each color is represented by three values of L, A, and B. Among them, L represents brightness, A represents the component from green to red, and B represents the component from blue to yellow. Lighter colors in the LAB color space can also be accurately distinguished. The preset color space corresponding to the eye shadow part and the mouth part may include the HSV color space, and the colors with a small hue difference can also be accurately distinguished under the HSV color space.

Therefore, in this step, for the facial skin part, the pixel dominant color of the target area image corresponding to the facial skin is converted from the RGB color space to the LAB color space. According to the L, A, and B three channel values of the main color of the pixel corresponding to the facial skin under the LAB color space, determine the hue interval to which the three channel values of L, A, and B belong. The hue category corresponding to the hue interval is determined as the hue category to which the skin color of the user's face belongs.

For the eye shadow part, the pixel main color of the target area image corresponding to the eye shadow part is converted from the RGB color space to the HSV color space. According to the three channel values of H, S and V of the main color of the pixel corresponding to the eye shadow part in the HSV color space, determine the hue interval to which the three channel values of H, S and V belong. The hue category corresponding to the hue interval is determined as the hue category to which the eye shadow color of the user's face belongs.

For the mouth, the pixel main color of the target area image corresponding to the mouth is converted from the RGB color space to the HSV color space. According to the H, S, and V channel values of the main color of the pixel corresponding to the mouth in the HSV color space, determine the hue interval to which the H, S, and V channel values belong. The hue category corresponding to the hue interval is determined as the hue category to which the lipstick color of the user's face belongs.

Step 903: Determine the user's makeup tone according to the tone category corresponding to the preset part.

In the embodiment of the present application, the preset part is one or more parts, and the color coefficient corresponding to each hue category is pre-configured. For example, the color coefficient corresponding to the fair tone can be 0.2, and the color coefficient corresponding to the brown-orange hue can be 0.3, etc. Wait. After the hue category corresponding to each preset part is identified through the above steps, the color coefficient corresponding to the hue category of each preset part is respectively determined. Calculate the user's makeup color coefficient according to the preset weight and color coefficient corresponding to each preset part. Specifically, the product of the preset weight corresponding to each preset part and the color coefficient is calculated respectively, and then the products corresponding to each preset part are summed, and the obtained sum value is used as the makeup color coefficient of the user.

The embodiment of the present application also pre-configures the color coefficient intervals corresponding to the hue categories of different makeup looks. After the user's makeup color coefficient is determined in the above manner, the color coefficient interval to which the makeup color coefficient belongs is determined, and the hue category corresponding to the color coefficient interval is determined as the user's makeup hue. The makeup color can be considered as the main color of the user's facial makeup.

As an example, the preset parts include facial skin, eye shadow parts and mouth. After identifying the skin color tone of the facial skin, the eye shadow tone of the eye shadow part, and the lipstick tone of the mouth, respectively determine the color coefficient corresponding to the skin color tone, the color coefficient corresponding to the eye shadow tone, and the color coefficient corresponding to the lipstick tone. Assume that the color coefficient corresponding to the skin color tone is "fair" is 0.2, the color coefficient corresponding to the eye shadow tone "brown orange" is 0.3, the color coefficient corresponding to the lipstick tone "nude pink" is 0.5, and the facial skin, eye shadow parts and mouth The preset weights are all 0.33, then calculate 0.33*0.2+0.33*0.3+0.33*0.5=0.33, that is, calculate the makeup color coefficient of the user to be 0.33. Assuming that the color coefficient range corresponding to the nude pink tone is 0-10, it is determined that the user's makeup tone is the nude pink tone.

As some other examples, the preset parts in this embodiment of the present application may only include facial skin and mouth, and the user's makeup tone is identified in combination with facial skin color and lipstick color. Alternatively, the preset parts may only include facial skin and eye shadow parts, and the user's makeup tone can be identified in combination with facial skin color and eye shadow color. Alternatively, the preset parts may only include the eye shadow part and the mouth part, and the user's makeup color can be identified by combining the lipstick color and the eye shadow color. Alternatively, the preset part may only include facial skin, and the user's makeup tone is identified through the facial skin color. Alternatively, the preset parts may only include the eye shadow part, and the user's makeup tone is identified through the color of the eye shadow. Alternatively, the preset part may only include the mouth, and the user's makeup tone is identified through the lipstick color.

In the embodiment of the present application, the tone category of the color of the user's preset part in the makeup image is identified, and then the user's makeup tone is determined according to the tone category corresponding to the preset part. Extract the tone of at least one part of the user's face, and use the extracted tone to automatically identify the tone of the user's makeup. Furthermore, by combining one or more dimensions of makeup such as skin color, eye shadow, lipstick, etc., only the color information of parts that have a greater influence on the makeup color is extracted, and the influence of other makeup effects on the makeup tone is excluded, increasing the accuracy of the makeup tone. Uniformity and accuracy of recognition. And only through image processing can accurately identify the user's makeup color without using deep learning model for processing, the amount of calculation is small, the cost is low, the processing pressure of the server is reduced, and the algorithm's dependence on hardware resources and human input are reduced cost.

An embodiment of the present application further provides a makeup color recognition device, which is used to implement the makeup color recognition method provided in any one of the above embodiments. As shown in Figure 24, the device includes:

An acquisition module 100, configured to acquire the makeup image of the user;

An identification module 200, configured to identify the hue category to which the color of the preset part of the user's face in the makeup image belongs;

The determination module 300 is configured to determine the user's makeup tone according to the tone category corresponding to the preset part.

The identification module 200 is used to detect the key points of the face corresponding to the makeup image; obtain the face area image corresponding to the makeup image according to the key points of the face; and identify the hue category to which the color of the preset part of the user's face in the face area image belongs.

The identification module 200 is used to intercept the target area image corresponding to the preset part from the face area image according to the key points of the face; obtain the main color of the pixel of the target area image; determine the color of the preset part according to the main color of the pixel. category of hues.

The preset parts include facial skin; the identification module 200 is used to intercept the facial image from the facial area image according to the key points of the human face; according to the key points of the human face, the eyebrow area, the eye area and the mouth are removed from the facial image region to obtain the image of the target region corresponding to the facial skin.

The preset parts include eye shadow parts; the identification module 200 is used to intercept the eye image from the face area image according to the eye key points included in the face key points; perform image expansion processing on the eye image for a preset number of times; According to the eye key points included in the eye key points, the eye area is cut out from the expanded eye area to obtain a target area image corresponding to the eye shadow.

The preset parts include the mouth; the identification module 200 is used to interpolate the key points on the upper edge of the upper lip, the lower edge of the upper lip, the upper edge of the lower lip, and the lower edge of the lower lip. , get the upper lip interpolation point, the upper lip lower edge interpolation point, the lower lip upper edge interpolation point and the lower lip lower edge interpolation point; point and the interpolation point along the lower edge of the upper lip, and the upper lip image is intercepted from the image of the face area; according to the key point on the upper edge of the lower lip, the interpolation point on the upper edge of the lower lip, the key point on the lower edge of the lower lip and the interpolation point on the lower edge of the lower lip, The lower lip image is cut out from the face area image; the upper lip image and the lower lip image are stitched into a target area image corresponding to the mouth.

The determining module 300 is used to convert the color space of the main color of the pixel to the preset color space corresponding to the preset position; according to the color channel values of the main color of the pixel in the preset color space, determine the hue interval to which each color channel value belongs; The tone category corresponding to the tone interval is determined as the tone category to which the color of the preset part belongs.

The preset part includes at least one of facial skin, eye shadow part, and mouth; the preset color space corresponding to the facial skin includes LAB color space, and the preset color space corresponding to the eye shadow part and mouth includes HSV color space.

The preset parts include one or more parts in facial skin, eye shadow parts and mouth, and the determination module 300 is used to determine the color coefficient corresponding to the hue category of each preset part; The weight and color coefficient are used to calculate the user's makeup color coefficient; and the hue category corresponding to the makeup color coefficient is determined as the user's makeup hue.

The identification module 200 is used to rotate and correct the makeup image and the key points of the face according to the key points of the face; according to the key points of the face after correction, the image containing the face area is intercepted from the corrected makeup image; The image of the face area is scaled to a preset size to obtain the image of the face area corresponding to the makeup image.

The identification module 200 is configured to determine the left eye center coordinates and right eye center coordinates respectively according to the left eye key points and right eye key points included in the face key points; determine the makeup image corresponding The rotation angle and the coordinates of the rotation center point; according to the rotation angle and the coordinates of the rotation center point, the makeup image and the key points of the face are rotated and corrected.

The recognition module 200 is configured to perform image capture on the face area contained in the corrected makeup image according to the key points of the corrected face.

The identification module 200 is used to determine the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value from the corrected face key points; according to the minimum abscissa value, minimum ordinate value, maximum abscissa value value and the maximum ordinate value, determine the clipping frame corresponding to the face area in the corrected makeup image; according to the clipping frame, cut out the image containing the face area from the corrected makeup image.

The recognition module 200 is further configured to enlarge the clipping frame by a preset multiple; and according to the zoomed-in clipping frame, clip an image containing a human face area from the corrected makeup image.

The identification module 200 is further configured to perform scaling and translation processing on the corrected key points of the face according to the size of the image including the face area and the preset size.

The device also includes: a complete human face detection module, which is used to detect whether at least one complete human face image is included in the makeup image; if so, perform the operation of identifying the hue category to which the color of the preset part of the user's face in the makeup image belongs; if If not, a prompt message is sent to the user's terminal, where the prompt message is used to prompt the user to provide a makeup image including at least one complete face image.

The makeup color recognition device provided by the above embodiments of the present application is based on the same inventive concept as the makeup color recognition method provided by the embodiments of the present application, and has the same beneficial effect as the method adopted, run or realized by its stored application program.

Embodiments of the present application further provide an electronic device to implement the above method for recognizing makeup color. It shows a schematic diagram of an electronic device provided by some embodiments of the present application. The electronic equipment includes: a processor, a memory, a bus and a communication interface, the processor, the communication interface and the memory are connected through a bus; a computer program that can run on the processor is stored in the memory, and the processor runs The computer program executes the makeup color recognition method provided in any one of the foregoing embodiments of the present application.

Wherein, the memory may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the device network element and at least one other network element is realized through at least one communication interface (which may be wired or wireless), and the Internet, wide area network, local network, metropolitan area network, etc. can be used.

The bus can be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into address bus, data bus, control bus and so on. Wherein, the memory is used to store a program, and the processor executes the program after receiving the execution instruction, and the makeup color recognition method disclosed in any one of the above-mentioned embodiments of the present application can be applied to the processor, or by Processor implementation.

A processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The above-mentioned processor can be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit ( ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, and the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware.

The electronic device provided in the embodiment of the present application is based on the same inventive concept as the makeup color recognition method provided in the embodiment of the present application, and has the same beneficial effect as the method adopted, operated or realized.

Embodiments of the present application also provide a computer-readable storage medium corresponding to the makeup color recognition method provided in the foregoing embodiments, where the computer-readable storage medium shown is an optical disc on which a computer program (that is, a program product) is stored. , when the computer program is run by the processor, it will execute the makeup color recognition method provided in any of the foregoing implementation manners.

It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random Access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other optical and magnetic storage media will not be repeated here.

The computer-readable storage medium provided by the above-mentioned embodiments of the present application is based on the same inventive concept as the makeup color recognition method provided by the embodiments of the present application, and has the same beneficial effect as the method adopted, executed or realized by the stored application program.

Embodiment nine

1. An image processing method, comprising:

Detect key point coordinates corresponding to facial blemishes in the user's face image;

According to the key point coordinates corresponding to the facial blemish and the preset material image, a blemish texture map is generated;

Generate a concealed human face image corresponding to the human face image according to the human face image and the blemish texture map.

2. According to 1, the defect texture map is generated according to the key point coordinates corresponding to the facial defects and the preset material image, including:

Obtain a blank texture image corresponding to the face image;

Locating a position corresponding to the facial blemish on the blank texture image according to the key point coordinates corresponding to the facial blemish;

Mapping the preset material image at the position corresponding to the facial blemish on the blank texture image to obtain a blemish texture map.

3. According to 1, according to the face image and the blemish texture map, generating a blemish-covered face image corresponding to the face image includes:

Performing blurring processing on the face image to obtain a dermabrasion image corresponding to the face image;

According to the blemish texture map, image fusion is performed on the blemish area in the face image and the blemish area in the skin care image at the same position to obtain a blemish-covered face image corresponding to the face image.

4. According to 3, according to the blemish texture map, image fusion is performed on the blemish area in the face image and the blemish area in the same position in the skin care image, including:

Obtain the transparency A value of the pixel at the first key point coordinates from the blemish texture map; the first key point coordinates are any key point coordinates in the key point coordinates corresponding to the facial blemish;

According to the transparency A value, fusion processing is performed on the pixel point at the coordinate of the first key point in the face image and the pixel point at the coordinate of the first key point in the skin care image.

5. According to 4, according to the transparency A value, the pixel point at the coordinate of the first key point in the face image and the pixel point at the coordinate of the first key point in the skin care image are compared Perform fusion processing, including:

Acquiring the first RGB color value of the pixel at the first key point coordinates from the face image, and acquiring the second RGB of the pixel at the first key point coordinates from the skin care image color value;

According to the first RGB color value, the second RGB color value and the transparency A value, calculate the difference between the pixel at the first key point coordinates in the face image and the pixel in the dermabrasion image The fused pixel value after the pixel point fusion at the first key point coordinates;

and resetting the current pixel value of the pixel at the coordinate of the first key point in the face image to the fused pixel value.

6. According to 5, according to the first RGB color value, the second RGB color value and the transparency A value, calculate the relationship between the pixel point at the first key point coordinate in the face image and the grinding The fused pixel value after the fusion of the pixels at the first key point coordinates in the skin image, including:

calculating a first product between the first RGB color value and the transparency A value and a second product between the second RGB color value and the transparency A value; calculating the first product and the the average of the second product;

The average value is determined as a fused pixel value after fusing the pixel point at the coordinate of the first key point in the face image and the pixel point at the coordinate of the first key point in the dermabrasion image.

7. According to any one of 1-6, the preset material image is an image with gradual transparency, and the transparency of pixels decreases from the edge of the preset material image to the center of the preset material image.

8. According to any one of 1-6, it also includes:

Detecting whether a face area is included in the face image;

If yes, then perform the operation of detecting the key point coordinates corresponding to facial blemishes in the user's face image;

If not, send prompt information to the terminal of the user, where the prompt information is used to prompt the user that the face image needs to include a face area.

9. An image processing device, comprising:

A detection module, configured to detect key point coordinates corresponding to facial blemishes in the user's face image;

The first generation module is used to generate a texture map of defects according to the key point coordinates corresponding to the facial defects and the preset material image;

The second generating module is configured to generate a concealed human face image corresponding to the human face image according to the human face image and the blemish texture map.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor runs the computer program to realize any of items 1-8. described method.

11. A computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the method described in any one of 1-8.

At present, people have more and more demands for photo editing functions. For example, for blemishes such as acne and color spots on the face, users usually want to use the photo editing function to remove these blemishes after taking pictures. In the related art, the user is usually required to blur the defect area in the face image for each pixel, so the user needs to determine the position of the defect in the image and manually trigger the blur processing, which has poor accuracy and low efficiency.

Based on this, the embodiment of the present application provides an image processing method, which automatically identifies facial blemishes such as pimples and spots in the face image, and performs material mapping for facial blemishes in the blank texture image corresponding to the face image to obtain a blemish texture map . Based on the blemish texture map, a blemish-covered human face image with a blemish-covering effect is automatically generated. In this way, there is no need for the user to manually optimize each pixel, which greatly improves the processing performance and can obtain the effect of concealing the face image in real time.

Referring to Figure 25, the method specifically includes the following steps:

Step 1001: Detect key point coordinates corresponding to facial blemishes in the user's face image.

The execution subject of the embodiment of the present application is a server or any other terminal capable of providing the function of concealing and retouching images. The embodiment of the present application takes the execution subject as the server as an example for illustration.

When the user needs to use the function of concealing and retouching, the user sends the face image to be retouched to the server. Specifically, a user terminal such as a user's mobile phone or a computer may be provided with a client that matches the concealment and retouching function provided by the server, and the client may be provided with an interface for submitting an image to be processed. When the client detects that the user clicks on the interface, a picture editing interface is displayed, and the picture editing interface may include a shooting button and/or a local file upload interface. When the client detects that the user clicks the capture button, it invokes the camera on the user terminal to capture the user's face image. When the client detects that the user clicks on the local file upload interface, the local folder directory is displayed so that the user can select the face image to be uploaded.

The client captures the user's face image through the camera, or receives the user's uploaded face image, and then sends the face image to the server. The server receives the face image of the user.

In some other embodiments of the present application, after the server obtains the user's face image, it also detects whether the face image contains a face area. If the face image includes a face area, the facial blemishes existing in the face area are concealed according to the method provided in this embodiment. If it is detected that the face image does not contain a face area, a prompt message is sent to the user's terminal. The user's terminal receives and displays the prompt information to remind the user that the face image needs to include a face area. For example, the prompt information may be "please provide a face image including a face area".

After obtaining the face image that includes the face area by the above method, detect whether there are facial blemishes such as acne or spots in the face image through the preset skin detection model, if it is detected that the face image does not contain facial blemishes, Then directly return the face image to the user terminal, or return information for prompting that there is no facial blemish to the user terminal.

If it is detected by the preset skin detection model that at least one facial blemish is included in the face image, the key point coordinates corresponding to each facial blemish are identified and recorded, and the key point coordinates corresponding to the facial blemish include the vertex coordinates of the facial blemish and texture coordinates.

Step 1002: Generate a blemish texture map according to the key point coordinates corresponding to the facial blemish and the preset material image.

Firstly, obtain the blank texture image corresponding to the user's face image. A blank texture image can be obtained by modifying the colored texture in the face image to a blank texture. That is, the position corresponding to the face in the blank texture map is blank.

Then, according to the key point coordinates corresponding to the detected facial blemishes, the position corresponding to the facial blemishes is located on the blank texture image. Using GPU (Graphics Processing Unit, Graphics Processing Unit) rendering technology, the preset material image is textured at the position corresponding to the facial blemish on the blank texture image to obtain a blemish texture map.

Wherein, the color of the preset material image may be any color such as red, yellow, or green. The preset material image is an image with a gradient of transparency, and the transparency of pixels decreases from the edge of the preset material image to the center of the preset material image. The preset material images are made according to the characteristics of blemishes such as acne and pigmentation. Most facial blemishes are round or oval in shape, with lighter edges and darker colors towards the center. Therefore, in the embodiment of the present application, the preset material image is set as a circular or elliptical image, and the transparency decreases from the edge to the center, that is, the closer to the edge, the more transparent, and the closer to the center, the more opaque. In this way, the preset material image is more suitable for the characteristics of real facial blemishes, and the preset material image is subsequently used to cover the facial blemishes in the face image, with higher accuracy and better concealment effect.

Paste the preset material image on the location of the key point coordinates corresponding to the user's facial blemishes in the blank texture map to obtain a blemish texture map. Then, the blemish texture map includes not only the location information of the facial blemish, but also the transparency distribution information of the preset material image.

As shown in Figure 26, (a) is the face image of the user, and (b) is the blemish texture map corresponding to the face image shown in (a). (b) The circular spot image in the figure is the preset material image. It can be seen from the figure that the edge color of the preset material image is lighter and the transparency is higher, and the color is darker and the transparency is lower toward the center.

Step 1003: Generate a concealed face image corresponding to the face image according to the face image and the blemish texture map.

The server specifically generates the concealed human face image through the following steps S1 and S2, specifically including:

S1: Blurring the face image to obtain a dermabrasion image corresponding to the face image.

The face image is blurred by a preset blur algorithm, and the preset blur algorithm may be a mean blur algorithm, a Gaussian blur algorithm, or the like. In the embodiment of the present application, only the face area in the face image can be blurred. First, the face area is identified from the face image, and then the face area is subjected to a full-face skin smoothing process through a preset blur algorithm to obtain Corresponding microdermabrasion images.

S2: According to the blemish texture map, perform image fusion on the blemish area in the face image and the blemish area in the same position in the microdermabrasion image to obtain a blemish-covered face image corresponding to the face image.

The embodiment of the present application only focuses on the blemish area where the facial blemish detected in the face image is located, performs image fusion on the blemish area in the face image and the blemish area in the same position in the skin care image, and performs image fusion on the non-blemish area in the face image The area is not processed, and the original image of the non-defective area in the face image is kept unchanged. And when performing fusion on the blemish area, the image fusion is performed based on the position of the facial blemish provided in the blemish texture map and the transparency distribution of the preset material image.

Since the fusion process corresponding to each pixel in the defect area is the same, this embodiment of the present application only uses one pixel as an example for detailed description.

Specifically, any one of the key point coordinates corresponding to facial blemishes is referred to as a first key point coordinate. In the embodiment of the present application, the color space of each image is RGBA, where A is also called Alpha, which is a transparency parameter. Obtain the transparency A value of the pixel at the coordinate of the first key point from the blemish texture map. Then, according to the transparency A value, fusion processing is performed on the pixel point at the coordinate of the first key point in the face image and the pixel point at the coordinate of the first key point in the microdermabrasion image.

Specifically, the first RGB color value of the pixel at the first key point coordinate is obtained from the face image, and the second RGB color value of the pixel at the first key point coordinate is obtained from the skin care image. According to the first RGB color value, the second RGB color value and the transparency A value, calculate the fusion pixel after the pixel at the first key point coordinate in the face image is fused with the pixel at the first key point coordinate in the microdermabrasion image value. Reset the current pixel value of the pixel at the coordinate of the first key point in the face image to the fused pixel value.

In the process of calculating the fused pixel value corresponding to the first key point coordinates, calculate the first product between the first RGB color value and the transparency A value and the second product between the second RGB color value and the transparency A value; calculate The average of the first product and the second product. The average value is determined as the fused pixel value after the pixel at the coordinate of the first key point in the face image is fused with the pixel at the coordinate of the first key point in the skin care image.

For each key point coordinate corresponding to the detected facial blemish, the fused pixel value after the pixel point in the face image corresponding to each key point coordinate is fused with the pixel point in the microdermabrasion image is calculated in the above manner. Then replace the current pixel value of the pixel point at the coordinates of each key point in the face image with the corresponding fusion pixel value, thereby completing the blemish area in the face image and the corresponding position in the skin care image in the face image Blending between blemish areas.

In other embodiments of the present application, the face image may include the face regions of multiple users, and the concealment processing may be performed on each face region in parallel or serially through the concealment processing method provided in the embodiment of the present application.

After performing concealment processing on the face area in the face image in the above manner to obtain a concealed face image corresponding to the face image, the concealed face image may also be returned to the user terminal. The user terminal receives and displays the concealed face image.

In the embodiment of the present application, facial blemishes in the face image are automatically identified, and material mapping is performed on the facial blemishes in the blank texture image corresponding to the face image to obtain a blemish texture map. The whole face microdermabrasion is performed on the face image to obtain the microdermabrasion image. Based on the blemish texture map, image fusion is performed between the blemish area in the face image and the blemish area in the same position in the microdermabrasion image to automatically generate a blemish-covering face image with a blemish effect. There is no need for the user to manually optimize each pixel, and paste the preset material image at the position corresponding to the facial blemish in the blank texture map corresponding to the face image. The transparency of the preset material image decreases from the edge to the center, which is in line with acne The actual characteristics of facial blemishes such as spots and spots. Based on the blemish texture map, the face image and the blemish area in the microdermabrasion image are fused to obtain a more natural blemish removal effect. Moreover, the whole process is carried out automatically without manual intervention, which greatly improves the processing performance and can obtain the effect of concealing blemishes on human face images in real time.

An embodiment of the present application further provides an image processing device, which is configured to execute the image processing method provided in any one of the foregoing embodiments. As shown in Figure 27, the device includes:

A detection module 400, configured to detect key point coordinates corresponding to facial blemishes in the user's face image;

The first generation module 500 is used to generate a texture map of defects according to the key point coordinates corresponding to the facial defects and the preset material image;

The second generating module 600 is configured to generate a concealed human face image corresponding to the human face image according to the human face image and the blemish texture map.

The first generation module 500 is used to obtain a blank texture image corresponding to the face image; locate the corresponding position of the facial defect on the blank texture image according to the key point coordinates corresponding to the facial defect; locate the corresponding position of the facial defect on the blank texture image Texture the preset material image to obtain the defect texture map.

The second generation module 600 is used to blur the face image to obtain a skin-removal image corresponding to the face image; according to the blemish texture map, perform a blurring process on the blemish area in the face image and the blemish area in the blemish image The images are fused to obtain a concealed face image corresponding to the face image.

The second generating module 600 is used to obtain the transparency A value of the pixel at the first key point coordinates from the blemish texture map; the first key point coordinates are any key point coordinates in the key point coordinates corresponding to facial blemishes; according to The transparency A value is used to fuse the pixel at the coordinate of the first key point in the face image and the pixel at the coordinate of the first key point in the microdermabrasion image.

The second generation module 600 is used to obtain the first RGB color value of the pixel at the first key point coordinates from the face image, and obtain the second RGB of the pixel at the first key point coordinates from the dermabrasion image Color value; according to the first RGB color value, the second RGB color value and the transparency A value, calculate the pixel point at the first key point coordinates in the face image and the pixel point at the first key point coordinates in the microdermabrasion image after fusion The fused pixel value; the current pixel value of the pixel point at the coordinates of the first key point in the face image is reset to the fused pixel value.

The second generation module 600 is used to calculate the first product between the first RGB color value and the transparency A value and the second product between the second RGB color value and the transparency A value; calculate the first product and the second product Average value; the average value is determined as the fused pixel value after the pixel at the first key point coordinate in the face image is fused with the pixel at the first key point coordinate in the skin care image.

The preset material image is an image with a gradient of transparency, and the transparency of pixels decreases from the edge of the preset material image to the center of the preset material image.

The device includes: a face detection module, which is used to detect whether the face image contains a face area; if yes, perform the operation of detecting the key point coordinates corresponding to the facial blemishes in the user's face image; if not, send a prompt The information is sent to the user's terminal, and the prompt information is used to prompt the user that the face image needs to include a face area.

The image processing device provided in the above embodiments of the present application is based on the same inventive concept as the image processing method provided in the embodiments of the present application, and has the same beneficial effect as the method adopted, executed or realized by the stored application program.

Embodiments of the present application further provide an electronic device to execute the above image processing method. It shows a schematic diagram of an electronic device provided by some embodiments of the present application. The electronic equipment includes: a processor, a memory, a bus and a communication interface, the processor, the communication interface and the memory are connected through a bus; a computer program that can run on the processor is stored in the memory, and the processor runs The computer program executes the image processing method provided in any one of the foregoing embodiments of the present application.

Wherein, the memory may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the device network element and at least one other network element is realized through at least one communication interface (which may be wired or wireless), and the Internet, wide area network, local network, metropolitan area network, etc. can be used.

The bus can be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into address bus, data bus, control bus and so on. Wherein, the memory is used to store the program, and the processor executes the program after receiving the execution instruction, and the image processing method disclosed in any implementation mode of the above-mentioned embodiments of the present application can be applied to the processor, or can be processed by the processor implement.

A processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The above-mentioned processor can be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit ( ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, and the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware.

The electronic device provided in the embodiment of the present application is based on the same inventive concept as the image processing method provided in the embodiment of the present application, and has the same beneficial effect as the method adopted, operated or implemented.

Embodiments of the present application also provide a computer-readable storage medium corresponding to the image processing method provided in the foregoing embodiments, where the computer-readable storage medium shown is an optical disc on which a computer program (ie, a program product) is stored. When the computer program is run by the processor, it will execute the image processing method provided in any of the foregoing implementation manners.

It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random Access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other optical and magnetic storage media will not be repeated here.

The computer-readable storage medium provided by the above embodiments of the present application is based on the same inventive concept as the image processing method provided by the embodiments of the present application, and has the same beneficial effect as the method adopted, run or realized by the stored application program.

Embodiment ten

An embodiment of the present application further provides a makeup progress detection device, which is used to implement the makeup progress detection method provided in any one of the above embodiments. As shown in Figure 28, the device includes:

A video acquisition module 700, configured to acquire a real-time makeup video of the user currently performing a specific makeup look;

The makeup progress determination module 800 is configured to determine the current makeup progress of the user performing a specific makeup look according to the initial frame image and the current frame image of the real-time makeup video.

The specific makeup includes highlight makeup or grooming makeup; the makeup progress determination module 800 is configured to obtain at least one target makeup area corresponding to the specific makeup; according to the target makeup area, obtain the first target corresponding to the specific makeup from the initial frame image The area image, and the second target area image corresponding to the specific makeup is obtained from the current frame image; according to the first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined.

The makeup progress determination module 800 is used to detect the first human face key point corresponding to the initial frame image; according to the first human face key point, obtain the human face area image corresponding to the initial frame image; The makeup area is used to obtain the first target area image corresponding to the specific makeup from the face area image.

The makeup progress determination module 800 is used to determine one or more target key points located on the contour of the area corresponding to the target makeup area in the face area image from the first human face key points; The key point is to generate a mask image corresponding to the face area image; perform an AND operation on the mask image and the face area image to obtain the first target area image corresponding to the specific makeup.

The makeup progress determination module 800 is used to determine each edge coordinate of the target makeup area in the face area image according to the target key points if there are multiple target key points corresponding to the target makeup area; The pixel values of all pixels in the area surrounded by edge coordinates are modified to preset values, and the mask area corresponding to the target makeup area is obtained; if there is one target key point corresponding to the target makeup area, then With the target key point as the center, draw an ellipse area with a preset size, modify the pixel values of all pixels in the ellipse area to the preset value, and obtain the mask area corresponding to the target makeup area; The pixel values of all the pixels in are modified to zero, and the mask image corresponding to the face area image is obtained.

The specific makeup includes blush makeup; the makeup progress determination module 800 is used to obtain at least one target makeup area corresponding to the specific makeup; generate a beauty mask map according to the target makeup area; and the current frame image to determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to use the makeup mask image as a reference to obtain the first target area image of makeup from the initial frame image, and obtain the second target area image of makeup from the current frame image; according to the A target area image and a second target area image are used to determine the current makeup progress corresponding to the current frame image.

The specific makeup includes eyeliner makeup; the makeup progress determination module 800 is used to obtain the beauty mask map corresponding to the initial frame image and the current frame image; according to the initial frame image, simulate the result image after the eyeliner makeup is completed; code image, result image, initial frame image, and current frame image to determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is configured to use the makeup mask corresponding to the initial frame image as a reference to obtain the image of the first target area for makeup application from the initial frame image; according to the makeup mask corresponding to the current frame image, from Obtain the second target area image of makeup in the current frame image; acquire the third target area image of eyeliner makeup according to the result image; determine the current frame according to the first target area image, the second target area image and the third target area image The current makeup progress corresponding to the image.

Makeup progress determination module 800, configured to respectively convert the first target area image, the second target area image and the third target area image into images containing saturation channels in the HLS color space; according to the converted first target area image, The second target area image and the third target area image determine the current makeup progress corresponding to the current frame image.

Makeup progress determination module 800, configured to calculate respectively the first average pixel value corresponding to the converted first target area image, the second average pixel value corresponding to the second target area image, and the third average pixel value corresponding to the third target area image value; calculate the first difference between the second average pixel value and the first average pixel value, and calculate the second difference between the third average pixel value and the first average pixel value; calculate the first difference and the first difference The ratio between the two difference values is used to obtain the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is configured to perform alignment processing on the first target area image and the second target area image; and perform alignment processing on the first target area image and the third target area image.

Makeup progress determination module 800, configured to perform binarization processing on the first target area image and the second target area image respectively, to obtain the first binarized mask image corresponding to the first target area image and the second target area image corresponding to The second binarized mask image; the AND operation is performed on the first binarized mask image and the second binarized mask image to obtain the first target area image corresponding to the intersection area of the second target area image Two mask images.

The makeup progress determination module 800 is used to obtain the face area image corresponding to the initial frame image and the face area image corresponding to the result image; perform an AND operation on the face area image corresponding to the second mask image and the initial frame image to obtain the initial A new first target area image corresponding to the frame image; an AND operation is performed on the second mask image and the face area image corresponding to the result image to obtain a new second target area image corresponding to the result image.

The makeup progress determination module 800 is used to obtain the eyeliner style diagram selected by the user; if the user's eye state in the initial frame image is an eye-opening state, then obtain the eye-opening style diagram corresponding to the eyeliner style diagram; determine the eye-opening style diagram as The beauty mask map corresponding to the initial frame image; if the user's eye state in the initial frame image is the closed eye state, then obtain the closed eye pattern map corresponding to the eyeliner pattern map, and determine the closed eye pattern map as the initial frame image corresponding beauty mask illustration.

The specific makeup includes eye shadow makeup; the makeup progress determination module 800 is used to obtain the eye shadow mask map; according to each target makeup area of the eye shadow makeup, split the target makeup area corresponding to each target makeup area from the eye shadow mask map. Makeup mask map; determine the current makeup progress corresponding to the current frame image according to the initial frame image, the current frame image, and the beauty mask map corresponding to each target makeup area.

The makeup progress determination module 800 is used to obtain the first target area image corresponding to each target makeup area from the initial frame image with reference to the beauty mask map corresponding to each target makeup area; The beauty mask image corresponding to the target makeup area is used as a reference, and the second target area image corresponding to each target makeup area is obtained from the current frame image; according to the first target area image and the second target area image corresponding to each target makeup area 2. Target area image, determine the current makeup progress corresponding to the current frame image.

Makeup progress determination module 800, configured to respectively convert the first target area image and the second target area image corresponding to each target makeup area into images containing preset single-channel components in the HLS color space; The first target area image and the second target area image corresponding to the target makeup area determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to calculate the absolute value of the difference between the preset single-channel components corresponding to the pixels in the same position in the first target area image and the second target area image corresponding to the same target makeup area after conversion; statistics The absolute value of the difference corresponding to each target makeup area satisfies the number of pixels that meet the preset makeup completion conditions; calculate the difference between the number of pixels corresponding to each target makeup area and the total number of pixels in the corresponding target makeup area to obtain the makeup progress corresponding to each target makeup area; according to the makeup progress corresponding to each target makeup area and the preset weight corresponding to each target makeup area, calculate the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to detect the first human face key point corresponding to the initial frame image; according to the first human face key point, obtain the corresponding human face area image of the initial frame image; The image of the first target area for applying makeup is acquired from the face area image.

The makeup progress determination module 800 is used to convert the beauty mask image and the face area image into binary images respectively; the binary image corresponding to the beauty mask image and the binary image corresponding to the face area image Perform an AND operation to obtain the first mask image corresponding to the intersection area of the beauty mask image and the face area image; perform an AND operation on the face area image corresponding to the first mask image and the initial frame image to obtain the first target area image.

The makeup progress determination module 800 is used to determine one or more first positioning points on the outline of each makeup area in the beauty mask image according to the standard face key points corresponding to the beauty mask image; according to the A face key point, determine the second positioning point corresponding to each first positioning point from the face area image; stretch the beauty mask map, and stretch each first positioning point to the corresponding at the position corresponding to each second positioning point of .

The makeup progress determination module 800 is used to split the beauty mask image into multiple sub-mask images, each sub-mask image includes at least one target makeup area; convert each sub-mask image and the face area image respectively is a binarized image; respectively perform an AND operation on the binarized image corresponding to each sub-mask image and the binarized image corresponding to the face area image to obtain the respective sub-mask image corresponding to each sub-mask image; Perform an AND operation on each sub-mask image with the face area image corresponding to the initial frame image to obtain multiple sub-target area images corresponding to the initial frame image; merge the multiple sub-target area images into a first target area image corresponding to the initial frame image.

The makeup progress determination module 800 is configured to determine one or more first positioning points located on the outline of the target makeup area in the first sub-mask image according to the standard face key points corresponding to the beauty mask image, the first The sub-mask is any sub-mask in a plurality of sub-masks; according to the first key point of people's face, the second positioning point corresponding to each first positioning point is determined from the face area image; Stretching is performed on the first sub-mask image, and each first positioning point is stretched to a corresponding position corresponding to each second positioning point.

The specific makeup includes eyebrow makeup; the makeup progress determination module 800 is used to obtain the first target area image corresponding to the eyebrows from the initial frame image, and obtain the second target area image corresponding to the eyebrows from the current frame image; according to the first target area image and the second target area image to determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to detect the first human face key point corresponding to the initial frame image; according to the first human face key point, obtain the human face area image corresponding to the initial frame image; according to the first human face key point included Eyebrow key points, obtain the first target area image corresponding to the eyebrows from the face area image.

The makeup progress determination module 800 is used to interpolate the eyebrow key points between the eyebrow head and the eyebrow peak included in the first human face key point to obtain multiple interpolation points; intercept all eyebrows between the eyebrow head and the eyebrow peak from the face area image A closed area formed by connecting key points and multiple interpolation points to obtain a part of the eyebrow image between the head of the eyebrow and the peak of the eyebrow; a closed area formed by connecting all eyebrow key points between the peak of the eyebrow and the end of the eyebrow is intercepted from the image of the face area , to obtain a part of the eyebrow image between the eyebrow peak and the eyebrow tail; the part of the eyebrow image between the eyebrow head and the eyebrow peak and the part of the eyebrow image between the eyebrow peak and the eyebrow tail are stitched into the first target area image corresponding to the eyebrows.

Makeup progress determination module 800, configured to respectively convert the first target area image and the second target area image into images containing preset single-channel components in the HSV color space; according to the converted first target area image and the second target area image Image, determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to calculate the absolute value of the difference of the preset single-channel component corresponding to the pixels in the same position in the converted first target area image and the second target area image respectively; count the corresponding absolute value of the difference The number of pixels that meet the preset makeup completion conditions; calculate the ratio between the statistical number of pixels and the total number of pixels in all target makeup areas in the first target area image, and obtain the current makeup progress corresponding to the current frame image.

Makeup progress determination module 800, configured to perform binarization processing on the first target area image and the second target area image respectively, to obtain the first binarized mask image corresponding to the first target area image and the corresponding second target area image The second binarized mask image; the AND operation is performed on the first binarized mask image and the second binarized mask image to obtain the first target area image corresponding to the intersection area of the second target area image Two mask images; obtain the face area image corresponding to the initial frame image and the face area image corresponding to the current frame image; perform an AND operation on the face area image corresponding to the second mask image and the initial frame image to obtain the initial frame image The corresponding new first target area image; performing an AND operation on the second mask image and the face area image corresponding to the current frame image to obtain a new second target area image corresponding to the current frame image.

The makeup progress determination module 800 is configured to perform boundary erosion processing on the makeup areas in the first target area image and the second target area image respectively.

The specific makeup includes foundation makeup; the makeup progress determination module 800 is used to simulate and generate the result image after the specific makeup is completed according to the initial frame image; respectively obtain the overall image brightness corresponding to the initial frame image, the result image and the current frame image; respectively obtain the initial The brightness of the face area corresponding to the frame image, the result image, and the current frame image; according to the overall image brightness and the brightness of the face area corresponding to the initial frame image, the result image, and the current frame image, determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to convert the initial frame image, the result image and the current frame image into grayscale images respectively; Gray-scale average value: determine the respective gray-scale average values corresponding to the initial frame image, the result image and the current frame image as respective overall image luminances corresponding to the initial frame image, the result image and the current frame image.

The makeup progress determination module 800 is used to respectively acquire the corresponding human face area images of the initial frame image, the result image and the current frame image; respectively convert the corresponding human face area images of the initial frame image, the result image and the current frame image into human faces Grayscale image: Calculate the grayscale average of the pixels in the grayscale image of the face corresponding to the initial frame image, the result image and the current frame image respectively, and obtain the corresponding face areas of the initial frame image, the result image and the current frame image brightness.

The makeup progress determination module 800 is used to determine the first environment change brightness corresponding to the current frame image according to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the current frame image; The overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the result image determine the second environment change brightness corresponding to the result image; according to the first environment change brightness and the second environment change brightness , the brightness of the face area corresponding to the initial frame image, the brightness of the face area corresponding to the current frame image, and the brightness of the face area corresponding to the result image, to determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to calculate the difference between the brightness of the overall image corresponding to the initial frame image and the brightness of the corresponding face area to obtain the ambient brightness of the initial frame image; calculate the brightness of the overall image corresponding to the current frame image and its corresponding The difference between the brightness of the face area obtains the ambient brightness of the current frame image; the absolute value of the difference between the ambient brightness of the current frame image and the ambient brightness of the initial frame image is determined as the first environmental change corresponding to the current frame image brightness.

The makeup progress determination module 800 is used to determine the makeup brightness change value corresponding to the current frame image according to the brightness of the first environmental change, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the current frame image; Change the brightness of the environment, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the result image, determine the change value of the makeup brightness corresponding to the result image; calculate the change value of the makeup brightness corresponding to the current frame image and the result image The ratio of the brightness change value of the makeup application to obtain the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to calculate the difference between the brightness of the face area corresponding to the current frame image and the brightness of the face area corresponding to the initial frame image to obtain the total brightness change value corresponding to the current frame image; calculate the total brightness change value The difference between the brightness of the first environmental change and the brightness of the first environment change is used to obtain the change value of the makeup brightness corresponding to the current frame image.

The makeup progress determination module 800 is configured to determine the makeup progress corresponding to the previous frame image as the current makeup progress corresponding to the current frame image if the brightness of the first environmental change is greater than the preset threshold; send the first prompt information to the user's terminal, The first prompt information is used to prompt the user to return to the brightness environment corresponding to the initial frame image to apply makeup.

The specific makeup includes concealer makeup; the makeup progress determination module 800 is used to obtain the facial blemish information corresponding to the initial frame image and the current frame image respectively; according to the facial blemish information corresponding to the initial frame image and the facial blemish corresponding to the current frame image information to calculate the facial blemish difference value between the current frame image and the initial frame image; if the facial blemish difference value is greater than the preset threshold, then calculate the current facial blemish difference value and the facial blemish information corresponding to the initial frame image. The current makeup progress corresponding to the frame image; if the facial blemish difference value is less than or equal to the preset threshold, the result image after the user completes the concealer and makeup is obtained, and the current frame image is determined according to the initial frame image, the result image and the current frame image. current make-up progress.

The makeup progress determination module 800 is used to calculate the difference between the number of blemishes corresponding to the initial frame image and the number of blemishes corresponding to the current frame image under each blemish category; calculate the sum of the differences corresponding to each blemish category, and you will get The sum value of is used as the face blemish difference value between the current frame image and the initial frame image.

The makeup progress determination module 800 is used to calculate the sum of the number of blemishes corresponding to each blemish category in the facial blemish information corresponding to the initial frame image to obtain the total number of blemishes; to calculate the ratio between the difference value of facial blemishes and the total number of blemishes, and The ratio is used as the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to simulate and generate the result image after the user completes the concealer and makeup according to the initial frame image; respectively obtain the corresponding face area images of the initial frame image, the result image and the current frame image; according to the initial frame image, The face area images corresponding to the result image and the current frame image respectively determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to respectively convert the corresponding face area images of the initial frame image, the result image and the current frame image into images containing saturation channels in the HLS color space; The smoothing factors corresponding to the face area images of the frame image, the result image and the current frame image; according to the smoothing factors corresponding to the initial frame image, the result image and the current frame image, determine the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to calculate the first difference between the smoothing factor corresponding to the current frame image and the smoothing factor corresponding to the initial frame image; the first difference between the smoothing factor corresponding to the calculation result image and the smoothing factor corresponding to the initial frame image The second difference: calculate the ratio between the first difference and the second difference, and use the ratio as the current makeup progress corresponding to the current frame image.

The makeup progress determination module 800 is used to respectively acquire the face area images corresponding to the initial frame image and the current frame image; respectively detect each of the face area images corresponding to the initial frame image and the current frame image through the preset skin detection model. The number of blemishes corresponding to the blemish category is used to obtain the facial blemish information corresponding to the initial frame image and the current frame image respectively.

The makeup progress determination module 800 is used to perform rotation correction on the initial frame image and the first human face key point according to the first human face key point corresponding to the initial frame image; The image containing the face area is intercepted from the initial frame image; the image containing the face area is scaled to a preset size to obtain the face area image corresponding to the initial frame image.

The makeup progress determination module 800 is used to determine the left eye center coordinates and right eye center coordinates respectively according to the left eye key points and right eye key points included in the first human face key points; according to the left eye center coordinates and right eye center coordinates, Determine the rotation angle and the coordinates of the rotation center point corresponding to the initial frame image; perform rotation correction on the initial frame image and the key points of the first face according to the rotation angle and the rotation center point coordinates.

The makeup progress determination module 800 is configured to, according to the corrected first key points of the human face, perform image interception on the human face area included in the corrected initial frame image.

The makeup progress determination module 800 is used to determine the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value from the corrected first human face key point; according to the minimum abscissa value, the minimum ordinate value , the maximum abscissa value and the maximum ordinate value, determine the interception frame corresponding to the face area in the corrected initial frame image; according to the interception frame, intercept the image containing the face area from the corrected initial frame image.

The makeup progress determination module 800 is further configured to enlarge the clipping frame by a preset multiple; and according to the zoomed-in clipping frame, clip an image containing a face area from the corrected initial frame image.

The makeup progress determination module 800 is further configured to perform scaling and translation processing on the corrected first key points of the human face according to the size of the image including the human face area and the preset size.

The device also includes: a face detection module, which is used to detect whether the initial frame image and the current frame image only contain the face image of the same user; if so, perform the operation of determining the current makeup progress of the user for a specific makeup look; If not, send prompt information to the user's terminal, and the prompt information is used to prompt the user to keep only the face of the same user appearing in the real-time makeup video.

The makeup progress detection device provided in the above embodiments of the present application is based on the same inventive concept as the makeup progress detection method provided in the embodiments of the present application, and has the same beneficial effects as the methods adopted, run or realized by its stored application programs.

The embodiment of the present application also provides an electronic device to implement the above method for detecting the progress of makeup. Please refer to FIG. 29 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. As shown in Figure 29, the electronic device 11 includes: a processor 1100, a memory 1101, a bus 1102 and a communication interface 1103, the processor 1100, the communication interface 1103 and the memory 1101 are connected through the bus 1102; A computer program running on the processor 1100, when the processor 1100 runs the computer program, executes the makeup progress detection method provided in any one of the foregoing embodiments of the present application.

Wherein, the memory 1101 may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 1103 (which may be wired or wireless), and the Internet, wide area network, local network, metropolitan area network, etc. can be used.

The bus 1102 can be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into address bus, data bus, control bus and so on. Wherein, the memory 1101 is used to store a program, and the processor 1100 executes the program after receiving an execution instruction, and the makeup progress detection method disclosed in any implementation manner of the above-mentioned embodiments of the present application can be applied to the processor 1100 , or implemented by the processor 1100.

The processor 1100 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method may be implemented by an integrated logic circuit of hardware in the processor 1100 or instructions in the form of software. The above-mentioned processor 1100 can be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, and the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory 1101, and the processor 1100 reads the information in the memory 1101, and completes the steps of the above method in combination with its hardware.

The electronic device provided in the embodiment of the present application is based on the same inventive concept as the method for detecting the progress of makeup provided in the embodiment of the present application, and has the same beneficial effect as the method adopted, operated or realized.

The embodiment of the present application also provides a computer-readable storage medium corresponding to the makeup progress detection method provided in the above-mentioned embodiment. Please refer to FIG. A program (that is, a program product), when the computer program is run by a processor, it will execute the makeup progress detection method provided in any of the foregoing implementation manners.

It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random Access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other optical and magnetic storage media will not be repeated here.

The computer-readable storage medium provided by the above embodiments of the present application is based on the same inventive concept as the makeup progress detection method provided by the embodiments of the present application, and has the same beneficial effect as the method adopted, run or implemented by the stored application program.

It should be noted:

In the description provided herein, numerous specific details are set forth. However, it is understood that the embodiments of the application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be understood that in the above description of exemplary embodiments of the application, in order to streamline the application and to facilitate understanding of one or more of the various inventive aspects, various features of the application are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting a schematic representation that the claimed application requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following this Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this application.

In addition, those skilled in the art will appreciate that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the present application. and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The above is only a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person familiar with the technical field can easily conceive of changes or changes within the technical scope disclosed in this application Replacement should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (55)

1. A makeup progress detection method, characterized in that, comprising: Obtain real-time makeup videos of users currently performing specific makeup looks; According to the initial frame image and the current frame image of the real-time makeup video, determine the current makeup progress of the user performing the specific makeup; Wherein, according to the initial frame image and the current frame image of the real-time makeup video, determining the current makeup progress of the user performing the specific makeup includes: The specific makeup includes at least one of highlight makeup, trimming makeup, blush makeup, eyeliner makeup, eye shadow makeup or eyebrow makeup, and the first target area image corresponding to the specific makeup is obtained from the initial frame image, and acquiring a second target area image corresponding to the specific makeup from the current frame image; acquiring a single-channel image of the first target area and the second target area that only contain channel components corresponding to the specific makeup image; perform binarization processing on the first target area image and the second target area image of a single channel that only contain the channel components, to obtain a first binarization mask corresponding to the first target area image A second binarized mask image corresponding to the image and the second target area image; an AND operation is performed on the first binarized mask image and the second binarized mask image to obtain an intersection area mask image; based on the The intersecting area mask image generates a new first target area image and a new second target area image respectively; according to the new first target area image and the new second target area image, it is determined that the user performs the specific makeup current make-up progress. 2. The method according to claim 1, wherein the specific makeup includes high-gloss makeup or grooming makeup; and according to the initial frame image and current frame image of the real-time makeup video, it is determined that the user performs The current make-up progress for the specific look in question, including: Obtain at least one target makeup area corresponding to the specific makeup; Acquiring a first target area image corresponding to the specific makeup from the initial frame image according to the target makeup area, and acquiring a second target area image corresponding to the specific makeup from the current frame image; According to the first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined. 3. The method according to claim 2, wherein said acquiring the first target area image corresponding to the specific makeup from the initial frame image according to the target makeup area comprises: Detecting the first human face key point corresponding to the initial frame image; Acquiring a face region image corresponding to the initial frame image according to the first face key point; According to the first human face key point and the target makeup area, a first target area image corresponding to the specific makeup is acquired from the human face area image. 4. The method according to claim 3, characterized in that, according to the first human face key point and the target makeup area, extracting the first corresponding to the specific makeup from the human face area image An image of the target area, including: Determining one or more target key points located on the contour of the area corresponding to the target makeup area in the human face area image from the first human face key points; Generate a mask image corresponding to the face area image according to the target key points corresponding to the target makeup area; An AND operation is performed on the mask image and the face area image to obtain a first target area image corresponding to the specific makeup. 5. The method according to claim 4, wherein the generating a mask image corresponding to the face area image according to the target key points corresponding to the target makeup area comprises: If there are multiple target key points corresponding to the target makeup area, then according to the target key points, determine each edge coordinate of the target makeup area in the face area image; The pixel values of all the pixel points in the area surrounded by two edge coordinates are all modified to preset values, so as to obtain the mask area corresponding to the target makeup area; If there is one target key point corresponding to the target makeup area, then take the target key point as the center, draw an ellipse area of preset size, and modify the pixel values of all pixels in the ellipse area to A preset value to obtain a mask area corresponding to the target makeup area; Modifying the pixel values of all pixels outside the mask area to zero to obtain a mask image corresponding to the face area image. 6. The method according to claim 1, wherein the specific makeup includes blush makeup; and according to the initial frame image and the current frame image of the real-time makeup video, it is determined that the user performs the specific makeup Current makeup progress, including: Obtain at least one target makeup area corresponding to the specific makeup; Generate a beauty mask map according to the target makeup area; According to the makeup mask image, the initial frame image and the current frame image, determine the current makeup progress corresponding to the current frame image. 7. The method according to claim 6, wherein the current makeup progress corresponding to the current frame image is determined according to the beauty mask image, the initial frame image and the current frame image, include: Taking the makeup mask image as a reference, acquiring a makeup-on first target area image from the initial frame image, and acquiring a makeup-on second target area image from the current frame image; According to the first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined. 8. The method according to claim 1, wherein the specific makeup includes eyeliner makeup; and according to the initial frame image and the current frame image of the real-time makeup video, it is determined that the user performs the specific makeup. Current makeup progress, including: Acquiring the beauty mask corresponding to the initial frame image and the current frame image; According to the initial frame image, simulate and generate the result image after eyeliner makeup is completed; According to the makeup mask map, the result image, the initial frame image and the current frame image, determine the current makeup progress corresponding to the current frame image. 9. The method according to claim 8, characterized in that, according to the cosmetic mask map, the result image, the initial frame image and the current frame image, it is determined that the current frame image corresponds to Current makeup progress, including: Taking the makeup mask image corresponding to the initial frame image as a reference, acquiring the image of the first target area of makeup from the initial frame image; Acquiring a makeup-on second target area image from the current frame image according to the makeup mask image corresponding to the current frame image; Acquiring the third target area image of eyeliner makeup according to the result image; According to the first target area image, the second target area image and the third target area image, the current makeup progress corresponding to the current frame image is determined. 10. The method according to claim 9, characterized in that, according to the first target area image, the second target area image and the third target area image, determine the corresponding Current makeup progress, including: Converting the first target area image, the second target area image, and the third target area image into images containing a saturation channel in the HLS color space, respectively; According to the converted first target area image, the second target area image and the third target area image, the current makeup progress corresponding to the current frame image is determined. 11. The method according to claim 10, wherein the current frame is determined according to the converted first target area image, the second target area image and the third target area image The current makeup progress corresponding to the image, including: respectively calculating a first average pixel value corresponding to the converted first target area image, a second average pixel value corresponding to the second target area image, and a third average pixel value corresponding to the third target area image; calculating a first difference between a second average pixel value and the first average pixel value, and calculating a second difference between the third average pixel value and the first average pixel value; Calculate the ratio between the first difference and the second difference to obtain the current makeup progress corresponding to the current frame image. 12. The method according to any one of claims 9-11, characterized in that, according to the first target area image, the second target area image and the third target area image, the Before the current makeup progress corresponding to the current frame image, it also includes: performing alignment processing on the first target area image and the second target area image; Perform alignment processing on the first target area image and the third target area image. 13. The method according to claim 12, wherein the aligning the first target area image and the second target area image comprises: Perform binarization processing on the first target area image and the second target area image respectively to obtain a first binarized mask image corresponding to the first target area image and a corresponding second target area image The second binarized mask image of ; Perform an AND operation on the first binarized mask image and the second binarized mask image to obtain a second mask corresponding to the intersection area of the first target area image and the second target area image film image. 14. The method according to claim 13, wherein the aligning the first target area image and the second target area image further comprises: Acquiring the face area image corresponding to the initial frame image and the face area image corresponding to the result image; performing an AND operation on the second mask image and the face area image corresponding to the initial frame image to obtain a new first target area image corresponding to the initial frame image; An AND operation is performed on the second mask image and the face area image corresponding to the current frame image to obtain a new second target area image corresponding to the result image. 15. The method according to any one of claims 8-11, wherein the acquiring the beauty mask corresponding to the initial frame image and the current frame image comprises: Obtain the eyeliner style map selected by the user; If the user's eye state in the initial frame image is the eye-open state, then obtain the eye-opening pattern map corresponding to the eyeliner pattern map; determine the eye-opening pattern map as the beauty mask corresponding to the initial frame image code map; If the eye state of the user in the initial frame image is the eye-closed state, then obtain the eye-closed pattern map corresponding to the eyeliner pattern map, and determine the eye-closed pattern map as the beauty makeup corresponding to the initial frame image mask map. 16. The method according to claim 1, wherein the specific makeup includes eye shadow makeup; and according to the initial frame image and the current frame image of the real-time makeup video, it is determined that the user performs the specific makeup. Current makeup progress, including: Obtain the eye shadow mask map; According to each target makeup area of eye shadow makeup, respectively split the beauty mask map corresponding to each target makeup area from the eye shadow mask map; The current makeup progress corresponding to the current frame image is determined according to the initial frame image, the current frame image, and the beauty mask map corresponding to each target makeup area. 17. The method according to claim 16, wherein the current frame image is determined according to the initial frame image, the current frame image, and the beauty mask map corresponding to each target makeup area. The current makeup progress corresponding to the frame image, including: Taking the beauty mask image corresponding to each target makeup area as a reference, acquiring a first target area image corresponding to each target makeup area from the initial frame image; Respectively using the beauty mask image corresponding to each target makeup area as a reference, acquiring a second target area image corresponding to each target makeup area from the current frame image; According to the first target area image and the second target area image corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is determined. 18. The method according to claim 17, wherein the current makeup corresponding to the current frame image is determined according to the first target area image and the second target area image corresponding to each of the target makeup areas. progress, including: Respectively converting the first target area image and the second target area image corresponding to each target makeup area into images containing preset single-channel components under the HLS color space; According to the converted first target area image and second target area image corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is determined. 19. The method according to claim 18, wherein, according to the first target area image and the second target area image corresponding to each target makeup area after conversion, it is determined that the current frame image corresponds to Current makeup progress, including: Respectively calculate the difference absolute value of the preset single-channel component corresponding to the pixels in the same position in the first target area image corresponding to the same target makeup area after conversion and in the second target area image; Count the number of pixels whose absolute value of the difference corresponding to each target makeup area satisfies the preset makeup completion condition; Calculating the ratio between the number of pixels corresponding to each target makeup area and the total number of pixels in the corresponding target makeup area to obtain the makeup progress corresponding to each target makeup area; According to the makeup progress corresponding to each target makeup area and the preset weight corresponding to each target makeup area, the current makeup progress corresponding to the current frame image is calculated. 20. The method according to any one of claims 7, 9, and 17, wherein the first target area image is obtained from the initial frame image with reference to the beauty mask image, comprising: Detecting the first human face key point corresponding to the initial frame image; Acquiring a face region image corresponding to the initial frame image according to the first face key point; Taking the makeup mask image as a reference, the image of the first target area for applying makeup is acquired from the face area image. 21. The method according to claim 20, characterized in that, using the makeup mask image as a reference, obtaining the first target area image for makeup from the face area image comprises: Converting the beauty mask image and the face area image into a binary image respectively; Perform an AND operation on the binarized image corresponding to the beauty mask map and the binarized image corresponding to the face area image to obtain the correspondence between the intersection area of the beauty mask map and the face area image The first mask image of ; An AND operation is performed on the face area image corresponding to the first mask image and the initial frame image to obtain a first target area image. 22. The method according to claim 21, wherein, before performing AND operation on the binarized image corresponding to the beauty mask map and the binarized image corresponding to the face region image, further include: According to the standard human face key points corresponding to the beauty mask map, one or more first positioning points located on the outline of each makeup area in the beauty mask map are determined; Determining a second anchor point corresponding to each of the first anchor points from the face area image according to the first face key point; Stretching is performed on the beauty mask image, and each of the first positioning points is stretched to a position corresponding to each of the second positioning points. 23. The method according to claim 20, characterized in that, using the makeup mask image as a reference, obtaining the first target area image for makeup from the face area image comprises: Splitting the beauty mask image into multiple sub-mask images, each of which includes at least one target makeup area; converting each of the sub-mask images and the face region image into a binary image; performing an AND operation on the binarized image corresponding to each of the sub-masks and the binarized image corresponding to the face region image to obtain a respective sub-mask image corresponding to each of the sub-masks; performing an AND operation on each of the sub-mask images and the face region images corresponding to the initial frame image to obtain a plurality of sub-target region images corresponding to the initial frame image; Merging the multiple sub-target area images into a first target area image corresponding to the initial frame image. 24. The method according to claim 23, wherein, before performing the AND operation on the binarized image corresponding to each of the sub-mask images and the binarized image corresponding to the face region image ,Also includes: According to the standard human face key points corresponding to the beauty mask map, determine one or more first positioning points located on the contour of the target makeup area in the first sub-mask map, the first sub-mask map It is any sub-mask in the plurality of sub-masks; Determining a second anchor point corresponding to each of the first anchor points from the face area image according to the first face key point; Stretching is performed on the first sub-mask image, and each of the first positioning points is stretched to a position corresponding to each of the second positioning points. 25. The method according to claim 1, wherein the specific makeup includes eyebrow makeup; and according to the initial frame image and the current frame image of the real-time makeup video, it is determined that the user performs the specific makeup. Current makeup progress, including: Obtaining a first target area image corresponding to the eyebrows from the initial frame image, and acquiring a second target area image corresponding to the eyebrows from the current frame image; According to the first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined. 26. The method according to claim 25, wherein said obtaining the first target region image corresponding to the eyebrows from the initial frame image comprises: Detecting the first human face key point corresponding to the initial frame image; Acquiring a face region image corresponding to the initial frame image according to the first face key point; According to the eyebrow key points included in the first human face key points, a first target area image corresponding to eyebrows is acquired from the human face area image. 27. The method according to claim 26, characterized in that, according to the eyebrow key points included in the first human face key points, the first target area image corresponding to the eyebrows is intercepted from the face area image ,include: Interpolating the eyebrow key points between the eyebrows and eyebrow peaks included in the first human face key points to obtain a plurality of interpolation points; Intercepting all eyebrow key points between the eyebrow head and the eyebrow peak and the closed area formed by the connection of the plurality of interpolation points from the image of the human face area, obtaining a partial eyebrow image between the eyebrow head and the eyebrow peak; Intercepting the closed area formed by connecting all eyebrow key points between the eyebrow peak and the eyebrow tail from the image of the human face region, obtaining a partial eyebrow image between the eyebrow peak and the eyebrow tail; The partial eyebrow image between the eyebrow head and the eyebrow peak and the partial eyebrow image between the eyebrow peak and the eyebrow tail are spliced into a first target region image corresponding to the eyebrows. 28. The method according to any one of claims 2, 7, and 25, wherein the current frame corresponding to the current frame image is determined according to the first target area image and the second target area image. Makeup progress, including: respectively converting the first target area image and the second target area image into images containing preset single-channel components in HSV color space; According to the converted first target area image and the second target area image, the current makeup progress corresponding to the current frame image is determined. 29. The method according to claim 28, wherein the determining the current makeup progress corresponding to the current frame image according to the converted first target area image and the second target area image comprises : Calculating the absolute value of the difference between the preset single-channel components corresponding to pixels at the same position in the converted first target area image and the second target area image; Count the number of pixels whose absolute value of the corresponding difference satisfies the preset makeup completion condition; Calculate the ratio between the counted number of pixels and the total number of pixels in all target makeup areas in the first target area image to obtain the current makeup progress corresponding to the current frame image. 30. The method according to any one of claims 2, 7, and 25, wherein the current frame image corresponding to the current frame image is determined according to the first target area image and the second target area image. Before makeup progress, also include: Performing binarization processing on the first target area image and the second target area image respectively, to obtain a first binarized mask image corresponding to the first target area image and corresponding to the second target area image The second binarized mask image of ; Perform an AND operation on the first binarized mask image and the second binarized mask image to obtain a second mask corresponding to the intersection area of the first target area image and the second target area image film image; Acquiring the face area image corresponding to the initial frame image and the face area image corresponding to the current frame image; performing an AND operation on the second mask image and the face area image corresponding to the initial frame image to obtain a new first target area image corresponding to the initial frame image; An AND operation is performed on the second mask image and the face area image corresponding to the current frame image to obtain a new second target area image corresponding to the current frame image. 31. The method according to any one of claims 7, 17, 25, characterized in that before determining the current makeup progress corresponding to the current frame image, further comprising: Boundary erosion processing is performed on makeup areas in the first target area image and the second target area image respectively. 32. The method according to claim 1, wherein the specific makeup includes foundation makeup; and according to the initial frame image and the current frame image of the real-time makeup video, it is determined that the user performs the specific makeup. Current makeup progress, including: According to the initial frame image, simulate and generate a result image after completing the specific makeup; Obtaining the overall image brightness corresponding to the initial frame image, the result image, and the current frame image respectively; Obtaining the brightness of the face area corresponding to the initial frame image, the result image, and the current frame image respectively; The current makeup progress corresponding to the current frame image is determined according to the overall image brightness and face area brightness corresponding to the initial frame image, the result image and the current frame image respectively. 33. The method according to claim 32, wherein said acquiring the overall image brightness corresponding to said initial frame image, said result image and said current frame image respectively comprises: respectively converting the initial frame image, the result image and the current frame image into grayscale images; Calculating the grayscale average value of the pixels in the grayscale images respectively corresponding to the converted initial frame image, the resultant image, and the current frame image; The respective grayscale average values corresponding to the initial frame image, the result image and the current frame image are respectively determined as the respective overall image luminances corresponding to the initial frame image, the result image and the current frame image. 34. The method according to claim 32, wherein the acquiring the brightness of the face area corresponding to the initial frame image, the result image and the current frame image respectively comprises: Respectively acquire the face area images corresponding to the initial frame image, the result image and the current frame image; respectively converting the face area images corresponding to the initial frame image, the result image, and the current frame image into face grayscale images; Calculate the gray average value of the pixels in the face grayscale images corresponding to the initial frame image, the result image and the current frame image respectively, to obtain the initial frame image, the result image and the current frame image The brightness of the face area corresponding to each frame image. 35. The method according to any one of claims 32-34, characterized in that, according to the respective overall image brightness and face area brightness corresponding to the initial frame image, the result image and the current frame image , determining the current makeup progress corresponding to the current frame image, including: Determine the first environmental change brightness corresponding to the current frame image according to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the current frame image; Determine the second environmental change brightness corresponding to the result image according to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the result image; According to the brightness of the first environmental change, the brightness of the second environmental change, the brightness of the face area corresponding to the initial frame image, the brightness of the face area corresponding to the current frame image, and the face area corresponding to the result image Brightness, to determine the current makeup progress corresponding to the current frame image. 36. The method according to claim 35, wherein, according to the overall image brightness and face area brightness corresponding to the initial frame image and the overall image brightness and face area brightness corresponding to the current frame image, Determining the brightness of the first environmental change corresponding to the current frame image includes: Calculating the difference between the overall image brightness corresponding to the initial frame image and the brightness of the corresponding face area to obtain the ambient brightness of the initial frame image; Calculating the difference between the overall image brightness corresponding to the current frame image and the corresponding face area brightness to obtain the ambient brightness of the current frame image; Determine the absolute value of the difference between the ambient brightness of the current frame image and the ambient brightness of the initial frame image as the first ambient change brightness corresponding to the current frame image. 37. The method according to claim 35, wherein the changing brightness according to the first environment, the changing brightness of the second environment, the brightness of the face area corresponding to the initial frame image, the brightness of the current frame The brightness of the face area corresponding to the image and the brightness of the face area corresponding to the result image determine the current makeup progress corresponding to the current frame image, including: According to the brightness of the first environmental change, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the current frame image, determine the makeup brightness change value corresponding to the current frame image; According to the brightness of the second environmental change, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the result image, determine the change value of the makeup brightness corresponding to the result image; Calculate the ratio of the makeup brightness change value corresponding to the current frame image to the makeup brightness change value corresponding to the result image to obtain the current makeup progress corresponding to the current frame image. 38. The method according to claim 37, wherein the changing brightness according to the first environment, the brightness of the face area corresponding to the initial frame image, and the brightness of the face area corresponding to the current frame image, Determine the makeup brightness change value corresponding to the current frame image, including: calculating the difference between the brightness of the face area corresponding to the current frame image and the brightness of the face area corresponding to the initial frame image to obtain a total brightness change value corresponding to the current frame image; Calculate the difference between the total brightness change value and the first environmental change brightness to obtain the makeup brightness change value corresponding to the current frame image. 39. The method of claim 35, further comprising: If the brightness of the first environmental change is greater than a preset threshold, then determine the makeup progress corresponding to the previous frame image as the current makeup progress corresponding to the current frame image; Sending first prompt information to the user's terminal, where the first prompt information is used to prompt the user to return to the brightness environment corresponding to the initial frame image to apply makeup. 40. The method according to claim 1, wherein the specific makeup includes concealer makeup; and according to the initial frame image and the current frame image of the real-time makeup video, it is determined that the user performs the specific makeup. Current makeup progress, including: Respectively acquiring facial blemish information corresponding to the initial frame image and the current frame image; According to the facial blemish information corresponding to the initial frame image and the facial blemish information corresponding to the current frame image, calculate the facial blemish difference value between the current frame image and the initial frame image; If the facial blemish difference value is greater than a preset threshold, then calculate the current makeup progress corresponding to the current frame image according to the facial blemish difference value and the facial blemish information corresponding to the initial frame image; If the facial blemish difference value is less than or equal to the preset threshold, then obtain the result image after the user completes the concealment and makeup, and determine according to the initial frame image, the result image and the current frame image The current makeup progress corresponding to the current frame image. 41. The method according to claim 40, wherein the facial blemish information includes blemish categories and corresponding blemish numbers; the facial blemish information corresponding to the initial frame image and the current frame image For the corresponding facial blemish information, calculate the facial blemish difference value between the current frame image and the initial frame image, including: Calculating the difference between the number of blemishes corresponding to the initial frame image and the number of blemishes corresponding to the current frame image under each blemish category; Calculate the sum of the differences corresponding to each blemish category, and use the obtained sum as the facial blemish difference value between the current frame image and the initial frame image. 42. The method according to claim 40, wherein the current makeup progress corresponding to the current frame image is calculated according to the facial blemish difference value and the facial blemish information corresponding to the initial frame image, include: Calculating the sum of the number of blemishes corresponding to each blemish category in the facial blemish information corresponding to the initial frame image to obtain the total number of blemishes; Calculate the ratio between the facial blemish difference value and the total number of blemishes, and use the ratio as the current makeup progress corresponding to the current frame image. 43. The method according to claim 40, wherein the acquisition of the result image after the user completes the concealment and makeup is determined according to the initial frame image, the result image and the current frame image. Describe the current makeup progress corresponding to the current frame image, including: According to the initial frame image, simulate and generate the result image after the user completes the concealer and makeup; Respectively acquire the face area images corresponding to the initial frame image, the result image and the current frame image; According to the face area images corresponding to the initial frame image, the result image and the current frame image, the current makeup progress corresponding to the current frame image is determined. 44. The method according to claim 43, characterized in that, according to the face area images corresponding to the initial frame image, the result image and the current frame image, determine the corresponding face area of the current frame image Current makeup progress, including: Respectively converting the face area images corresponding to the initial frame image, the result image and the current frame image into images containing saturation channels in the HLS color space; Calculating the smoothing factors corresponding to the respective face area images of the initial frame image, the result image, and the current frame image after conversion through a preset filtering algorithm; According to the respective smoothing factors corresponding to the initial frame image, the result image and the current frame image, the current makeup progress corresponding to the current frame image is determined. 45. The method according to claim 44, wherein the current makeup corresponding to the current frame image is determined according to the respective smoothing factors of the initial frame image, the resultant image, and the current frame image. progress, including: calculating a first difference between the smoothing factor corresponding to the current frame image and the smoothing factor corresponding to the initial frame image; calculating a second difference between the smoothing factor corresponding to the result image and the smoothing factor corresponding to the initial frame image; Calculate the ratio between the first difference and the second difference, and use the ratio as the current makeup progress corresponding to the current frame image. 46. The method according to claim 40, wherein said obtaining the respective facial blemish information corresponding to said initial frame image and said current frame image respectively comprises: Respectively acquire the face area images corresponding to the initial frame image and the current frame image; The number of blemishes corresponding to each blemish category in the face area image corresponding to the initial frame image and the current frame image is respectively detected by the preset skin detection model, and the respective corresponding blemishes of the initial frame image and the current frame image are obtained. facial blemish information. 47. The method according to any one of claims 3, 26, 34, 43, 46, wherein said acquiring the face area image corresponding to said initial frame image comprises: Performing rotation correction on the initial frame image and the first human face key point according to the first human face key point corresponding to the initial frame image; According to the corrected first human face key point, intercepting an image containing a human face area from the corrected initial frame image; The image containing the face area is scaled to a preset size to obtain a face area image corresponding to the initial frame image. 48. The method according to claim 47, wherein, according to the first human face key point corresponding to the initial frame image, the initial frame image and the first human face key point are rotated and corrected ,include: Determine left eye center coordinates and right eye center coordinates respectively according to the left eye key point and right eye key point included in the first human face key point; Determine the rotation angle and rotation center point coordinates corresponding to the initial frame image according to the left eye center coordinates and the right eye center coordinates; Rotation correction is performed on the initial frame image and the first human face key point according to the rotation angle and the coordinates of the rotation center point. 49. The method according to claim 47, wherein, according to the corrected first human face key point, intercepting an image containing a human face area from the corrected initial frame image comprises: Determining a minimum abscissa value, a minimum ordinate value, a maximum abscissa value, and a maximum ordinate value from the corrected first face key point; According to the minimum abscissa value, the minimum ordinate value, the maximum abscissa value and the maximum ordinate value, determine the clipping frame corresponding to the face area in the corrected initial frame image; According to the clipping frame, an image including the face area is clipped from the rectified initial frame image. 50. The method of claim 49, further comprising: Enlarging the interception frame by a preset multiple; According to the enlarged clipping frame, an image including the face area is clipped from the corrected initial frame image. 51. The method of claim 47, further comprising: According to the size of the image including the face area and the preset size, zooming and translation processing is performed on the corrected first key points of the face. 52. The method of claim 1, further comprising: Detecting whether both the initial frame image and the current frame image only contain the face image of the same user; If yes, then perform the operation of determining the current makeup progress of the user performing the specific makeup look; If not, send prompt information to the user's terminal, where the prompt information is used to prompt the user to keep only the face of the same user appearing in the real-time makeup video. 53. A makeup progress detection device, comprising: The video acquisition module is used to acquire the real-time makeup video of the user currently performing a specific makeup look; A makeup progress determination module, configured to determine the current makeup progress of the user performing the specific makeup look according to the initial frame image and the current frame image of the real-time makeup video; The makeup progress determination module is specifically used for the specific makeup including at least one of highlight makeup, trimming makeup, blush makeup, eyeliner makeup, eye shadow makeup or eyebrow makeup, and obtains the A first target area image corresponding to the specific makeup, and acquiring a second target area image corresponding to the specific makeup from the current frame image; acquiring a single-channel image of the first target area that only includes channel components corresponding to the specific makeup The target area image and the second target area image; respectively performing binarization processing on the first target area image and the second target area image of a single channel containing only the channel components to obtain a first target area The first binarized mask image corresponding to the image and the second binarized mask image corresponding to the second target area image; performing an AND operation on the first binarized mask image and the second binarized mask image, Obtaining an intersecting area mask image; generating a new first target area image and a new second target area image based on the intersecting area mask image; according to the new first target area image and the new second target area image , to determine the current makeup progress of the user performing the specific makeup. 54. An electronic device comprising a memory, a processor and a computer program stored on the memory and operable on the processor, wherein the processor runs the computer program to realize the The method of any one of 1-52. 55. A computer-readable storage medium, on which a computer program is stored, wherein the program is executed by a processor to implement the method according to any one of claims 1-52.

Images