JP6710440B1 - Makeup evaluation device, makeup evaluation system, and makeup evaluation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a makeup evaluation apparatus, a makeup evaluation system, and a makeup evaluation method which can be used when a visually impaired person applies makeup to his or her face.
An evaluation unit 43 evaluates the quality of makeup applied to a user based on reference image data 61, pre-makeup image data 63, and comparison image data 65. The evaluation unit 43 mainly includes an extraction unit 43a, a first calculation unit 43b, a presence/absence determination unit 43c, a second calculation unit 43d, and a finish determination unit 43e.
[Selection diagram] Figure 1

Description

The present invention relates to a makeup evaluation device, a makeup evaluation system, and a makeup evaluation method.

BACKGROUND ART Conventionally, a makeup assisting apparatus that presents a user with a makeup method based on a first image (user image) and a second image (model image) is known (for example, Patent Document 1). ..

Japanese Patent No. 5991536

However, if the person applying the makeup is a visually impaired person, the makeup cannot be correctly applied to his/her face even if the makeup assisting device of Patent Document 1 is used.

Therefore, an object of the present invention is to provide a makeup evaluation apparatus, a makeup evaluation system, and a makeup evaluation method that can be used when a visually handicapped person applies makeup to his or her face.

In order to solve the above problems, the invention of claim 1 is a makeup evaluation apparatus, comprising an imaging unit capable of capturing a face image of a user, and at least face image data of the user, that is, the use. Reference image data in which the user's face is previously imaged in a state where the face of the user is applied with reference makeup, and pre-makeup image data in which the user's face before makeup is imaged by the imaging unit. And face image data captured by the image capturing unit subsequent to capturing of the pre-makeup image data, where the user's face is in a state where makeup is applied to at least a part of the user's face. The appearance of the makeup applied to the user based on the storage unit capable of storing the captured comparison image data, the reference image data, the pre-makeup image data, and the comparison image data. And a notifying unit for notifying the user of desired information, and among the facial elements included in the user's face, makeup that satisfies the desired finish is not applied. image one or more of the face elements, as the element of interest, the evaluation unit, the reference image data, the decorative image data before, and from each of the comparison image data, corresponding to the existing area of each face element After extracting the extraction unit capable of extracting data and the existing regions corresponding to the element of interest from the pre-makeup image data and the comparison image data as the first and second regions, respectively, by the extraction unit. , A first calculation unit that calculates a first index value for each face element included in the attention element by using image data included in each of the first and second regions, and each included in the attention element. A presence/absence determining unit configured to execute a determination as to whether or not makeup is applied to the face element based on a corresponding one of the first index values calculated by the first calculating unit; For the makeup-finished element determined by the presence/absence determining unit that the makeup is applied among the face elements included in the attention element, the existence area corresponding to the makeup-finished element from the comparison image data and the reference image data is set, By using the image data included in each of the third and fourth regions after being extracted by the extraction unit as the third and fourth regions, respectively, The second calculation unit that calculates two index values and the determination of the workability of each face element included in the makeup element are based on the corresponding one of the second index values calculated by the second calculation unit. Configured to run The finishing determination unit is provided, and the notification unit notifies the user of the determination result of the finishing determination unit.

Further, the invention of claim 2 is the makeup evaluation apparatus according to claim 1, wherein each of the reference image data, the pre-makeup image data, and the comparison image data is obtained by horizontally reversing a captured image. It is characterized by being.

The invention of claim 3 is the makeup evaluation apparatus according to claim 1 or 2, wherein the face image captured by the image capturing unit is newly stored in the storage unit as the comparison image data. In addition, the evaluation unit evaluates the quality of makeup applied to the user.

Further, in the invention of claim 4, in the makeup evaluation apparatus according to claim 1 or 2, the comparison image data stored in the storage unit is updated by the face image captured by the imaging unit. At this time, the evaluation unit evaluates the quality of the makeup applied to the user.

According to a fifth aspect of the present invention, in the makeup evaluation apparatus according to any one of the first to fourth aspects, the evaluation unit determines that the makeup is completed for at least one face element included in the attention element. In the case of the face element, a registration unit that registers completion information indicating that makeup is completed for the face element in the first database stored in the storage unit, and the first database corresponds to each face element. Each record included in the first database including a record includes at least sequence information indicating a makeup order of the corresponding face element, face element information for identifying the corresponding face element, and corresponding face element. And the completion information indicating whether or not makeup is completed, and the notification unit is based on the sequence information, face element information, and completion information of each record registered in the first database. The next feature is to inform the user of a facial element to be applied next.

Further, the invention of claim 6 is the makeup evaluation apparatus according to claim 5, further comprising an input unit capable of receiving an instruction from the user after the result of the determination by the finish determination unit is notified, The comparison image data stored in the storage unit is updated with a face image captured by the image capturing unit in response to an instruction from the input unit.

Further, in the invention of claim 7, in the makeup evaluation apparatus according to claim 5, when the judgment result of the finishing judgment part is within a desired range, the registration part stores the corresponding record in the first database in the corresponding record. It is characterized by registering completion information.

Further, the invention of claim 8 is a makeup evaluation system, comprising a terminal and an information processing device, wherein the terminal provides an image pickup unit capable of picking up a face image of the user, and And a notification unit for notifying desired information, wherein the information processing apparatus uses at least the face image data of the user, that is, the face with a reference makeup applied to the user. Image data of the person's face captured, pre-makeup image data of the user's face before makeup captured by the image capturing unit, and image capturing by the image capturing unit subsequent to capturing the pre-makeup image. A storage unit capable of storing face image data, and comparative image data in which at least a part of the face of the user is applied with makeup, and the reference The image data, the pre-makeup image data, and the comparison image data, and based on the comparison image data, has an evaluation unit for evaluating the quality of the makeup applied to the user, each included in the face of the user of the face elements, when one or a plurality of faces elements cosmetic that satisfies the desired workmanship has not been subjected to, are noted element, the evaluation unit of the information processing apparatus, the reference image data, the decorative front image An extraction unit capable of extracting image data corresponding to the existence area of each face element from each of the data and the comparison image data, and an existence corresponding to the attention element from the pre-makeup image data and the comparison image data. By extracting the regions as the first and second regions by the extraction unit and using the image data included in each of the first and second regions, each face element included in the attention element is obtained. With respect to the first index value calculated by the first calculation section, and a determination as to whether or not makeup has been applied to each face element included in the attention element. Of these, the presence/absence determination unit configured to be executed based on the corresponding one, and the makeup element determined to have makeup applied to the facial elements included in the attention element by the presence/absence determination unit, The presence areas corresponding to the makeup-finished element are extracted from the comparison image data and the reference image data as the third and fourth areas, respectively, by the extraction unit, and then extracted into the third and fourth areas, respectively. By using the included image data, the second calculation unit that calculates the second index value for each face element included in the makeup element, and the determination of the workability of each face element included in the makeup element are performed as described above. The second calculated by the second calculation unit Of the two index values, a finish judging unit configured to execute based on the corresponding one, the notification unit of the terminal having received the judgment result of the finish judging unit It is characterized by notifying the user.

According to the invention of claim 9, in the makeup evaluation system according to claim 8, each of the reference image data, the pre-makeup image data, and the comparison image data is obtained by horizontally reversing a captured image. It is characterized by

The invention of claim 10 is the makeup evaluation system according to claim 8 or 9, wherein the information processing device determines that the makeup is completed for at least one face element included in the attention element. When the determination is made according to the above, a registration unit that registers completion information indicating that makeup is completed for the face element in the first database stored in the storage unit is further provided, and the first database stores each face element. Each record included in the first database including a record corresponding to, at least sequence information indicating the order of applying makeup to the corresponding face element, face element information for identifying the corresponding face element, and Completion information indicating whether or not makeup of the face element has been completed can be stored, and the notification unit includes sequence information of each record registered in the first database, face element information, and completion information. It is characterized in that the user is informed of the facial element to be applied next based on the above.

Further, the invention of claim 11 is the makeup evaluation system according to claim 10, wherein the terminal is an input unit capable of accepting an instruction from the user after the determination result by the finish determination unit is notified. The comparison image data stored in the storage unit is updated by the face image captured by the image capturing unit in response to an instruction from the input unit.

A twelfth aspect of the present invention is a makeup evaluation method, wherein the appearance of makeup applied to a user is evaluated based on (a) reference image data, pre-makeup image data, and comparative image data. And (b) informing the user of desired information, wherein one of the facial elements included in the face of the user is not covered with makeup that satisfies the desired finish. or more face elements, as the element of interest, the reference image data in a state where the decorative is applied as a reference face of the user, the face of the user in the face image data captured Yes, the pre-makeup image data is face image data in which the face of the user before makeup is imaged by the imaging unit, and the comparison image data is captured by the imaging unit subsequent to the imaging of the pre-makeup image. Face image data, wherein at least a part of the face of the user is applied makeup, the face of the user is imaged, and the step (a) includes (a-1 ) From each of the reference image data, the pre-makeup image data, and the comparative image data, a step of extracting image data corresponding to the existence region of each face element, (a-2) the pre-makeup image data and the Image data included in each of the first and second regions after extracting existing regions corresponding to the element of interest from the comparison image data as the first and second regions respectively in the step (a-1) By calculating the first index value for each face element included in the target element, and (a-3) determining whether or not makeup is applied to each face element included in the target element Of the first index values calculated in the step (a-2) based on the corresponding one, and (a-4) the face element included in the attention element by the presence/absence determining unit. With respect to the makeup-applied elements determined to have been applied with makeup, the presence areas corresponding to the makeup-applied elements from the comparison image data and the reference image data are defined as the third and fourth areas, respectively, and the step (a- Calculating the second index value for each face element included in the makeup-finished element by using the image data included in each of the third and fourth regions after being extracted in (1); -5) a step of performing a workability determination of each face element included in the makeup element based on a corresponding one of the second index values calculated in the step (a-4), In the step (b), the notification unit uses the judgment result of the step (a-5) in the use. It is characterized by notifying the user.

According to the first to twelfth aspects of the present invention, by using the reference image data, the pre-makeup image data, and the comparison image data, the appearance of makeup applied to the user can be evaluated without using the visual sense. can do. Therefore, even a visually handicapped person can apply makeup well to his or her face.

In particular, in the inventions described in claims 2 and 9, the reference image data, the pre-makeup image data, and the comparison image data are each obtained by horizontally reversing the captured image. That is, each face element included in each image data has the same arrangement as the face image reflected in the mirror. Therefore, the user can apply makeup with the same feeling as when the face is reflected in the mirror.

It is a block diagram showing an example of composition of a makeup evaluation device in a 1st embodiment of the present invention. It is a figure for explaining reference image data. It is a figure for demonstrating the image data before makeup. It is a figure for demonstrating comparative image data. It is a figure for demonstrating each face image data used for makeup evaluation. It is a figure for demonstrating each face image data used for makeup evaluation. It is a figure for demonstrating each face image data used for makeup evaluation. It is a figure for demonstrating an example of the data structure of a 1st database. It is a flow chart which shows a makeup evaluation procedure. It is a figure for demonstrating an example of the data structure of a 1st database. It is a block diagram for demonstrating the makeup|decoration evaluation system in the 2nd Embodiment of this invention. It is a block diagram for demonstrating the terminal in the 2nd Embodiment of this invention. It is a block diagram for explaining an information processor in a 2nd embodiment of the present invention. It is a block diagram for demonstrating the makeup|decoration evaluation system in the 3rd Embodiment of this invention. It is a block diagram for demonstrating the terminal in the 3rd Embodiment of this invention. It is a block diagram for demonstrating the information processing apparatus in the 3rd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<1. First Embodiment>
<1.1. Structure of makeup evaluation device>
FIG. 1 is a block diagram showing an example of the configuration of a makeup evaluation apparatus 40 according to the first embodiment of the present invention. 2 to 4 are diagrams for explaining the reference image data 61, the pre-makeup image data 63, and the comparison image data 65, respectively.

Here, the makeup evaluation device 40 is a device for evaluating makeup applied to the face of a user (including a visually impaired person). The makeup evaluation apparatus 40 is composed of an electronic computer including a so-called personal computer or a workstation, or a mobile terminal such as a so-called smartphone (Smartphone) or a tablet terminal.

As shown in FIG. 1, the makeup evaluation apparatus 40 mainly includes a CPU 41, a large-capacity storage unit 51, a RAM 52, a ROM 53, a display unit 54, a communication processing unit 55, an input unit 56, and an imaging unit 57. And have. Further, the CPU 41, the mass storage unit 51, the RAM 52, the ROM 53, the communication processing unit 55, the display processing unit 58, and the input/output processing unit 59 are electrically connected via the bus 41a.

The imaging unit 57 is configured by a camera having a so-called CCD (Charge Coupled Device) sensor or CMOS (Complementary MOS) sensor as an imaging element. The image capturing unit 57 is capable of capturing a face image of the user.

The large-capacity storage unit 51 is a storage unit configured by an element having a larger storage capacity than the RAM 52, such as a silicon disk drive or a hard disk drive, and exchanges data with the RAM 52 as necessary. ..

The RAM 52 includes a so-called volatile storage unit. The RAM 52 is capable of storing data (at least reference image data 61 which is face image data of the user, pre-makeup image data 63, comparison image data 65, the first database 70, etc.) used in the calculation of the CPU 41. Has been done.

Here, the reference image data 61 is a state in which the user's face is applied with a makeup that serves as a reference (for example, a beauty company member of a cosmetics company applies makeup to the user's face). This is face image data in which a face is imaged in advance (see FIG. 2). The pre-makeup image data 63 is face image data in which the face of the user before makeup is captured by the image capturing unit 57 (see FIG. 3 ).

Further, the comparison image data 65 is face image data captured by the image capturing unit 57 after capturing the pre-makeup image data 63 (see FIG. 4 ). In this face image data, the face of the user is imaged with at least a part of the face of the user having makeup applied.

Further, as the reference image data 61, the pre-makeup image data 63, and the comparison image data 65, those obtained by horizontally reversing the image captured by the image capturing unit 57 may be used. That is, each face element included in each image data has the same arrangement as the face image reflected in the mirror. For example, in the case of FIG. 2, the face element 91 corresponds to the left eyebrow, and the face element 92 corresponds to the right eyebrow.

The ROM 53 is composed of a so-called non-volatile memory. A program 53a is stored in the ROM 53, for example. As the RAM 52 and the ROM 53, a flash memory which is a readable/writable non-volatile memory may be used.

The CPU 41 executes operation control and data processing according to the program 53a of the ROM 53. Further, the CPU 41 implements the arithmetic function corresponding to the blocks described in the CPU 41 in FIG. 1 (reference numerals 43 (43a to 43e), 45, and 46 are given respectively). The arithmetic functions corresponding to the blocks 42 to 46 will be described later.

The input unit 56 is an input device such as a keypad, a keyboard, or a mouse. Thereby, the user of the makeup evaluation apparatus 40 can perform the input work based on the display content of the display unit 54 (for example, receive an instruction from the user).

The display unit 54 is a display device including a so-called liquid crystal display, a plasma display, or an organic EL (Electro Luminescence) display, and displays characters and images on the screen. The display processing unit 58 is composed of a video controller, and is electrically connected to the display unit 54 via a signal line 58a. Therefore, when the drawing processing is executed by the display processing unit 58, characters and figures are displayed on the screen of the display unit 54.

The communication processing unit 55 executes data transmission/reception with an external device connected via the network 9 in accordance with a command from the CPU 41. The input/output processing unit 59 is used for exchanging data between the makeup evaluation apparatus 40 and an external device connected to the makeup evaluation apparatus 40. As shown in FIG. 1, the input/output processing unit 59 is electrically connected to the input unit 56 via the signal line 59a. Thereby, the CPU 41 of the makeup evaluation apparatus 40 can receive the input information from the input unit 56.

<1.2. Calculation function realized by CPU of makeup evaluation device>
Each of FIG. 5 to FIG. 7 is a diagram for explaining each face image data 61, 63, 65 used for makeup evaluation. FIG. 8 is a diagram for explaining an example of the data structure of the first database 70. Here, the functions corresponding to the evaluation unit 43, the notification unit 45, and the registration unit 46 will be described with reference to FIGS. 1 and 5 to 8.

The evaluation unit 43 evaluates the quality of makeup applied to the user based on the reference image data 61, the pre-makeup image data 63, and the comparison image data 65. As shown in FIG. 1, the evaluation unit 43 mainly includes an extraction unit 43a, a first calculation unit 43b, a presence determination unit 43c, a second calculation unit 43d, and a finish determination unit 43e.

Note that, for convenience of description, in the present embodiment, in the reference image data 61, the pre-makeup image data 63, and the comparison image data 65, only the face elements 91 to 95 are subjected to makeup workability evaluation.

The extraction unit 43a is capable of extracting image data corresponding to the presence regions of the face elements 91 to 95 from each of the reference image data 61, the pre-makeup image data 63, and the comparison image data 65.

For example, when the (one) face element 91 (left eyebrow) is the target element, the image data corresponding to the existing area 91a from the reference image data 61 and the existing area 91b from the pre-makeup image data 63. Image data corresponding to the existing area 91c is extracted from the comparison image data 65.

Further, when the face element 92 (right eyebrow) is the attention element, the image data corresponding to the existing area 92a from the reference image data 61, the image data corresponding to the existing area 92b from the pre-makeup image data 63, Image data corresponding to the existing area 92c is extracted from the comparison image data 65.

Further, when the face element 95 (near the lips) is the attention element, the image data corresponding to the existing area 95a from the reference image data 61, the image data corresponding to the existing area 95b from the pre-makeup image data 63, Image data corresponding to the existing area 95c is extracted from the comparison image data 65.

Further, when a plurality of face elements 93 (near the left eyelid where the eyeline is drawn) and a face element 94 (near the right eyelid where the eyeline is drawn) are the attention elements, the existence area 93a from the reference image data 61, The image data corresponding to 94a is extracted from the pre-makeup image data 63, the image data corresponding to the existing regions 93b and 94b, and the comparative image data 65 is extracted from the image data corresponding to the existing regions 93c and 94c.

The first calculation unit 43b causes the extraction unit 43a to extract the existing area corresponding to the element of interest from the pre-makeup image data 63 and the comparison image data 65. In addition, the first calculation unit 43b uses the image data included in each of the extracted regions (first and second regions) to determine the first index value for each face element included in the attention element. Is calculated.

For example, when the (one) face element 91 is set as the element of interest, the first calculation unit 43b converts the image data corresponding to the existing area 91b (first area) from the pre-makeup image data 63 into the comparison image data 65. The image data corresponding to the existing area 91c (second area) is extracted by the extraction unit 43a.

Then, the first calculation unit 43b sums up the differences in the density values of the corresponding pixels for the image data corresponding to the existing area 91b (first area) and the image data corresponding to the existing area 91c (second area). The sum (difference summed value) is calculated as the first index value of the face element 91.

When a plurality of face elements 93 and 94 are set as the attention element, the first calculation unit 43b includes the image data corresponding to the existing areas 93b and 94b in the pre-makeup image data 63 and the existence in the comparison image data 65. The image data corresponding to the areas 93c and 94c are extracted by the extraction unit 43a.

Then, the first calculation unit 43b sums up the differences in the density values of the corresponding pixels for the image data corresponding to the existing area 93b (first area) and the image data corresponding to the existing area 93c (second area). The calculated result is calculated as the first index value of the face element 93.

Similarly, the first calculation unit 43b causes the difference in the density values of the corresponding pixels between the image data corresponding to the existing area 94b (first area) and the image data corresponding to the existing area 94c (second area). Is calculated as the first index value of the face element 94.

The presence/absence determining unit 43c determines whether or not makeup is applied to each face element included in the target element based on the corresponding one of the first index values calculated by the first calculating unit 43b. Is configured to.

For example, when the (one) face element 91 is the target element, the presence/absence determining unit 43c determines that the face element 91 is present when the first index value (difference summed value) corresponding to the face element 91 is equal to or greater than a predetermined value. It is judged that the makeup is applied to.

In addition, when a plurality of face elements 93 and 94 are set as attention elements, the presence/absence determination unit 43c determines whether or not makeup is applied to each of the face elements 93 and 94. That is, when the first index value (difference summed value) corresponding to the face element 93 is equal to or greater than the predetermined value, the presence/absence determining unit 43c determines that the face element 93 is being makeupd. Similarly, when the first index value (difference summed value) corresponding to the face element 94 is greater than or equal to a predetermined value, the presence/absence determining unit 43c determines that the face element 94 is being makeupd.

The second calculation unit 43d determines, from the comparison image data 65 and the reference image data 61, the makeup applied to the makeup-completed element determined by the presence/absence determination unit 43c among the facial elements included in the attention element, from the comparison image data 65. The extraction unit 43a extracts the existing area corresponding to the element.

Further, the second calculator 43d calculates the second index value for each face element included in the makeup element by using the image data included in each of the extracted areas (third and fourth areas). To do.

For example, when the (one) face element 91 is the makeup-applied element, the second calculation unit 43d uses the comparison image data 65 to convert the image data corresponding to the existing area 91c (third area) into the reference image data 61. The image data corresponding to the existing area 91a (fourth area) is extracted by the extraction unit 43a.

Then, the second calculation unit 43d determines the correlation coefficient (covariance of each image with respect to the image data corresponding to the existing area 91c (third area) and the image data corresponding to the existing area 91a (fourth area). A value obtained by dividing the data by the standard deviation of the data) is calculated as the second index value of the face element 91.

When the plurality of face elements 93 and 94 are made-up elements, the second calculation unit 43d
The extraction unit 43a causes the comparison image data 65 to extract image data corresponding to the existing regions 93c and 94c, and the reference image data 61 to extract image data corresponding to the existing regions 93a and 94a.

Then, the second calculation unit 43d calculates the correlation coefficient of the image data corresponding to the existing area 93c (third area) and the image data corresponding to the existing area 93a (fourth area) of the face element 93. It is calculated as the second index value. Similarly, the second calculation unit 43d determines the correlation coefficient between the image data corresponding to the existing area 94c (third area) and the image data corresponding to the existing area 94a (fourth area) as the face element. It is calculated as the second index value of 94.

The finish determination unit 43e is configured to perform the determination of the workability of each face element included in the makeup element based on the corresponding one of the second index values calculated by the second calculation unit 43d. There is.

For example, when the (one) face element 91 is set as a makeup element, the finish determination unit 43e determines that the face element 91 has a second index value (correlation coefficient) equal to or larger than a predetermined value when the face element 91 is a makeup element. It is judged that the makeup for 91 is applied with the desired finish.

Further, when the plurality of face elements 93, 94 are made-up elements, the finish determination unit 43e executes a workability determination for each face element 93, 94. That is, when the second index value (correlation coefficient) corresponding to the face element 93 is equal to or larger than the predetermined value, the finish determination unit 43e determines that the face element 93 is covered with makeup that satisfies the desired finish. Similarly, when the second index value (correlation coefficient) corresponding to the face element 94 is greater than or equal to a predetermined value, the finish determination unit 43e determines that the face element 93 is covered with makeup that satisfies the desired finish. To do.

Here, the first database 70 will be described. The first database 70 is a data group that can store sequence information indicating the order in which makeup is applied to the face elements 91 to 95 in association with the corresponding face elements 91 to 95, and is stored in the RAM 52 (see FIG. 1). It is memorable. As shown in FIG. 8, the first database 70 mainly includes fields (columns) of "identification ID 71", "face element information 72", and "completion flag 73".

A value for uniquely identifying each record (each row) registered in the first database 70 is stored in the “identification ID 71” field. A character string for identifying each face element is stored in the "face element information 72".

Here, when the records are sorted so that the value of the "identification ID 71" is in ascending order, the arrangement of the "face element information 72" field is the makeup order of each face element. For example, in the case of the present embodiment, as shown in FIG. 8, a face element 91 (left eyebrow), a face element 92 (right eyebrow), a face element 93 (near the left eyelid where the eyeline is drawn), a face element 94 ( The vicinity of the right eyelid where the eyeline is drawn) and the face element 95 (near the lips) are the makeup order set in advance.

The “completion flag 73” is a flag indicating whether or not makeup corresponding to a desired finish is applied to the corresponding face element. In the present embodiment, at the start of makeup, the “FALSE” value is set in the “completion flag 73” of all records. In addition, when the finish determination unit 43e determines that the corresponding face element is covered with makeup that satisfies the desired appearance, the "TRUE" value is set in the "completion flag 73" of the corresponding record. In this way, the “completion flag 73” can be used as completion information indicating whether or not makeup on each face element is completed.

As described above, the first database 70 includes records corresponding to each face element. Further, each record included in the first database 70 is at least
(1) Identification ID 71 (sequence information) indicating the order of applying makeup to the corresponding face element,
(2) Face element information 72 for identifying the corresponding face element,
(3) A completion flag 73 (completion information) indicating whether or not makeup of the corresponding face element is completed,
Can be stored.

The notification unit 45 notifies the user of desired information. For example, the notification unit 45 may notify the user of the determination result of the finish determination unit 43e by using voice, vibration, and/or display on the display unit 54.

That is, when it is determined that the makeup that satisfies the desired workmanship is applied to some of the attention elements, the notification unit 45 outputs, for example, the name of the face element for which the workability is determined to be good and the workability. The user is informed of the name of the face element that has not reached the desired level.

When the evaluation unit 43 determines that makeup has been completed for at least one face element included in the element of interest (more specifically, a makeup element), the registration unit 46 has completed makeup for this face element. The completion information indicating that is registered in the first database 70.

For example, the registration unit 46 sets the “completion flag 73” of the corresponding record included in the first database 70 when the determination result of the finish determination unit 43e is in the desired range (for example, the second index value is equal to or larger than the predetermined value). Change the "FALSE" value to the "TRUE" value.

As a result, the notification unit 45 notifies the user of the next facial element to apply makeup based on the identification ID 71, the facial element information 72, and the completion flag 73 of each record registered in the first database 70. be able to.

For example, when the value of each record in the first database 70 is as shown in FIG. 8, the notification unit 45 determines (1) “completion flag 73”=“FALSE” from the first database 70 and (2 ) A record having the smallest “identification ID 71” value is extracted. Then, the notification unit 45 notifies the user that the face element 92 should be put on next by reading the face element information 72 of the extracted record.

<1.3. Makeup evaluation procedure>
FIG. 9 is a flowchart showing a makeup evaluation procedure. FIG. 10 is a diagram for explaining an example of the data structure of the first database 70. Here, a makeup evaluation procedure using the makeup evaluation apparatus 40 will be described with reference to FIGS. 5 to 10.

First, in this determination procedure, the storage status of the comparison image data 65 stored in the RAM 52 (storage unit) is monitored (S101). And
(1) When the face image captured by the image capturing unit 57 is newly stored in the RAM 52 as the comparison image data 65, or
(2) When the comparison image data 65 stored in the RAM 52 is updated by the face image captured by the image capturing unit 57,
The evaluation unit 43 evaluates the quality of the makeup applied to the user according to steps S102 to S106. For example, when the face image is captured by the image capturing unit 57 and the comparison image data 65 in the RAM 52 is updated to that shown in FIG. 7, the process proceeds to step S102.

In the extraction process of step S102, the “completion flag 73” of each record included in the first database 70 is referred to, and the element of interest is extracted based on the “completion flag 73” of each referred record ( S102).

For example, in the case of FIG. 8, a record with “completion flag 73”=“FALSE” (a record with “identification ID 71”=“2” to “5”) is extracted. Then, of each face element, the one corresponding to the extracted record (“face element information 72”=“face element 92” to “face element 95”) is extracted as the element of interest.

Subsequently, the first calculation unit 43b calculates the first index value of each face element included in the target element (S103), and the presence/absence determination unit 43c applies makeup to each face element included in the target element. It is determined whether or not it has been performed (S104).

Here, when it is determined by the presence/absence determining unit 43c that no makeup is applied to any face element included in the target element (for example, when the first index value is a sum of differences, all calculated values are calculated). When the first index value is less than the predetermined value: S104), it is determined that none of the face elements included in the target element has makeup applied, and the process proceeds to step S108.

On the other hand, if it is determined that makeup is applied to at least one face element of the attention elements (S104), the second calculation unit 43d determines whether the face elements included in the makeup elements are the second face elements. While the index value is calculated (S105), the finish determination unit 43e determines whether makeup of each face element included in the makeup-finished element is completed (S106).

Here, when the finish determination unit 43e determines that the makeup is not completed for any face element included in the makeup-completed element (for example, when the second index value is a correlation coefficient, all the calculated When the second index value of is less than the predetermined value: S106), it is determined that makeup has not been completed on any face element included in the makeup-finished element, and the process proceeds to step S108.

On the other hand, if it is determined that the makeup applied to at least one face element of the makeup-applied elements is good (S106), the correspondence among the records included in the first database 70 is determined. After the completion information is registered in the item to be processed (S107), the process proceeds to step S108.

For example, in the case of FIG. 7 and FIG. 8, when it is determined that makeup of the face element 93 is newly completed in a state where makeup of the face element 91 has already been completed, the registration unit 46 causes the first Of the records included in the database 70, the record with “identification ID 71”=“3” is extracted, and the “completion flag 73” of the extracted record is changed from the “FALSE” value to the “TRUE” value (FIG. 10).

In step S108, the notification unit 45 notifies the makeup status of each face element included in the target element of step S104 or each face element included in the makeup-finished element of step S106.

For example, in step S104, when it is determined that makeup has not been applied to any face element included in the attention element, the notification unit 45
(1) Each face element (face element 92 to face element 95 in FIG. 8) corresponding to “completion flag 73”=“FALSE” is not applied with makeup, and
(2) The face element to be applied next (face element 92 in FIG. 8),
Is notified to the user.

In addition, in step S106, when it is determined that the makeup is not completed for any face element included in the makeup-completed element (the makeup that satisfies the desired finish is not applied), the notification unit 45
(1) Removing makeup applied to all face elements included in the makeup element, and
(2) The face element to be applied next (face element 92 in FIG. 8),
Is notified to the user.

Furthermore, in step S106, when it is determined that the makeup of at least one element of the face elements included in the makeup element is not completed, the notification unit 45
(1) Removing makeup applied to at least one element (hereinafter, referred to as “unfinished element”), and
(2) Face element to be applied next,
Is notified to the user.

In addition, in step S106, when it is determined that the makeup of at least one of the face elements included in the makeup element is not completed, the face element to be makeup next is updated by the registration unit 46. The selected first database 70 is selected.

Then, in step S109, when it is determined that the makeup is not completed for a part of the face elements (that is, the “completion flag 73” of some records included in the first database 70 is “FALSE”. If yes), the process returns to step S101. In this case, the comparison image data 65 stored in the RAM 52 may be updated with the face image captured by the image capturing unit 57 in accordance with an instruction from the input unit 56.

On the other hand, in step S109, when it is determined that the makeup of all the face elements is completed (that is, the “completion flag 73” of all the records included in the first database 70 becomes “TRUE”. If so), the makeup evaluation is completed.

<1.4. Advantages of the makeup evaluation apparatus according to the first embodiment>
As described above, in the makeup evaluation apparatus 40 according to the first embodiment, the reference image data 61, the pre-makeup image data 63, and the comparison image data 65 are used to apply to the user without using the visual sense. You can evaluate the quality of makeup. Therefore, even a visually handicapped person can apply makeup well to his or her face.

Further, in the makeup evaluation apparatus 40 according to the first embodiment, each of the reference image data 61, the pre-makeup image data 63, and the comparison image data 65 can be obtained by horizontally reversing the captured image. That is, each face element included in each image data has the same arrangement as the face image reflected in the mirror. Therefore, the user can apply makeup with the same feeling as when the user's face is reflected in the mirror.

<2. Second Embodiment>
Next, a second embodiment of the present invention will be described. The makeup evaluation apparatus 40 and the makeup evaluation system 100 according to the first and second embodiments are common in that they realize similar functions. Further, the functions realized by makeup evaluation apparatus 40 are different in that they are realized by either terminal 110 or information processing apparatus 140. Therefore, the difference will be mainly described below.

The same components in the makeup evaluation apparatus 40 and makeup evaluation system 100 are designated by the same reference numerals, and the components designated by the same reference numerals have already been described in the first embodiment. Therefore, the description is omitted in this embodiment.

<2.1. Structure of makeup evaluation system>
11 to 13 are block diagrams each showing an example of a configuration of the makeup evaluation system 100, the terminal 110, and the information processing device 140 according to the second embodiment of the present invention. A user of the makeup evaluation system 100 (including a visually impaired person) can evaluate makeup applied to his/her face by using this system 100. As illustrated in FIG. 11, the makeup evaluation system 100 mainly includes one or a plurality of terminals 110 (110a and 110b) and an information processing device 140.

The terminal 110 is configured by a mobile terminal such as a so-called smartphone (Smartphone) or a tablet terminal. As shown in FIG. 11, the terminal 110 is connected to the information processing apparatus 140 via the network 9. The detailed configuration of the terminal 110 will be described later.

The information processing device 140 is an electronic computer configured by a so-called personal computer or workstation, and executes desired processing according to the image data transmitted from the terminal 110. The detailed configuration of the information processing device 140 will be described later.

<2.2. Device configuration>
FIG. 12 is a diagram showing an example of the configuration of the terminal 110. As shown in FIG. 12, the terminal 110 mainly includes a CPU 111, a RAM 122, a ROM 123, a communication processing unit 125, a touch panel unit 126, and an imaging unit 127. The CPU 111, the RAM 122, the ROM 123, the communication processing unit 125, the display processing unit 128, and the input/output processing unit 129 are electrically connected via the bus 111a.

The RAM 122 includes a so-called volatile storage unit. The RAM 122 can store data used in the calculation of the CPU 111. The ROM 123 is composed of a so-called non-volatile memory. A program 123a is stored in the ROM 123, for example. As the RAM 122 and the ROM 123, a flash memory which is a readable/writable non-volatile memory may be used.

The CPU 111 executes operation control and data processing according to the program 123a of the ROM 123. Further, the notification unit 114 described in the CPU 111 in FIG. 12 has the same calculation function as the notification unit 45, and is realized by the CPU 111.

The communication processing unit 125 executes data transmission/reception with the information processing device 140 connected via the network 9 in accordance with a command from the CPU 111. The touch panel unit 126 has, for example, a function as a “display device” such as a liquid crystal display and a function as a “touch panel” that can specify a position on the screen by touching the screen with a finger or a dedicated pen. ing.

The image capturing unit 127, like the image capturing unit 57, is configured by a camera having a so-called CCD sensor or CMOS sensor as an image capturing element. As a result, the image capturing unit 127 can capture a face image of the user.

Similar to the display processing unit 38, the display processing unit 128 includes a video controller and is electrically connected to the touch panel unit 126 via a signal line 128a. Therefore, when the drawing processing is executed by the display processing unit 128, characters and figures are displayed on the screen of the touch panel unit 126.

The input/output processing unit 129 is used for exchanging data between the terminal 110 and an external device of the terminal 110. As shown in FIG. 12, the input/output processing unit 129 is electrically connected to the touch panel unit 126 and the imaging unit 127 via the signal line 129a. As a result, the CPU 111 of the terminal 110 can acquire position information on the screen of the touch panel unit 126.

<2.3. Configuration of information processing device>
As shown in FIG. 13, the information processing apparatus 140 mainly includes a CPU 141, a large-capacity storage unit 51, a RAM 52, a ROM 53, a display unit 54, a communication processing unit 55, an input unit 56, and an imaging unit 57. And have. The CPU 141, the mass storage unit 51, the RAM 52, the ROM 53, the communication processing unit 55, the display processing unit 58, and the input/output processing unit 59 are electrically connected via the bus 41a.

The CPU 141 executes operation control and data processing according to the program 123a of the ROM 123. The CPU 141 implements the arithmetic function corresponding to the blocks described in the CPU 141 in FIG. 13 (reference numerals 43 (43a to 43e), 45, and 46 are given respectively).

<2.4. Advantages of Makeup Evaluation System of Second Embodiment>
As described above, in the makeup evaluation system 100 of the second embodiment, similar to the makeup evaluation apparatus 40 of the first embodiment, the reference image data 61, the pre-makeup image data 63, and the comparison image data 65 are acquired. By using it, it is possible to evaluate the quality of makeup applied to the user without using the visual sense. Therefore, even a visually handicapped person can apply makeup well to his or her face.

<3. Third Embodiment>
Next, a third embodiment of the present invention will be described. The makeup evaluation systems 100 and 200 of the second and third embodiments have the same configuration except that the data stored in the corresponding RAMs 52 and 252 and the RAMs 122 and 222 are different. Therefore, the difference will be mainly described below.

The same components in makeup evaluation systems 100 and 200 are designated by the same reference numerals, and the components designated by the same reference numerals have already been described in the first and second embodiments. Therefore, the description is omitted in this embodiment.

14 to 16 are block diagrams each showing an example of the configuration of makeup evaluation system 200, terminal 210, and information processing device 240 according to the third embodiment of the present invention. As shown in FIG. 14, the makeup evaluation system 200 mainly includes one or a plurality of terminals 210 (210a, 210b) and an information processing device 240.

FIG. 15 is a block diagram showing an example of the configuration of the terminal 210. As shown in FIG. 15, the terminal 210 mainly includes a CPU 111, a RAM 222, a ROM 123, a communication processing unit 125, a touch panel unit 126, and an imaging unit 127. Further, the CPU 111, the RAM 222, the ROM 123, the communication processing unit 125, the display processing unit 128, and the input/output processing unit 129 are electrically connected via the bus 111a.

The RAM 222 includes a so-called volatile storage unit, like the RAM 122. The RAM 222 is capable of storing data (at least the pre-makeup image data 63 and the comparison image data 65, which are the face image data of the user, the first database 70, etc.) used in the calculation of the CPU 111.

FIG. 16 is a block diagram showing an example of the configuration of the information processing device 240. As illustrated in FIG. 16, the information processing device 240 mainly includes the CPU 141, the large-capacity storage unit 51, the RAM 252, the ROM 53, the display unit 54, the communication processing unit 55, the input unit 56, and the imaging unit 57. And have. Further, the CPU 141, the mass storage unit 51, the RAM 252, the ROM 53, the communication processing unit 55, the display processing unit 58, and the input/output processing unit 59 are electrically connected via the bus 41a.

The RAM 252 includes a so-called volatile storage unit, like the RAM 52. The RAM 252 is capable of storing data (at least the reference image data 61, which is the user's face image data) used in the calculation of the CPU 141.

As described above, in the makeup evaluation system 200 according to the third embodiment, similar to the makeup evaluation system 100 according to the second embodiment, the reference image data 61, the pre-makeup image data 63, and the comparative image data 65 are acquired. By using it, it is possible to evaluate the quality of makeup applied to the user without using the visual sense. Therefore, even a visually handicapped person can apply makeup well to his or her face.

<4. Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments and various modifications can be made.

(1) The evaluation unit 43 in the first to third modes has been described as being realized by software by the CPU 41 (see FIG. 1) and the CPU 141 (see FIG. 13 and FIG. 16), but is not limited thereto. Not a thing. For example, the function of the evaluation unit 43 may be realized by hardware such as an electronic circuit.

40 makeup evaluation device 100, 200 makeup evaluation system 110 terminal 140 information processing device

Claims (12)

A makeup evaluation device,
(a) an image pickup unit capable of picking up a face image of the user,
(b) At least the face image data of the user, that is,
(b-1) in a state where the reference makeup is applied to the user's face, and reference image data in which the user's face is imaged in advance,
(b-2) pre-makeup image data in which the face of the user before makeup is imaged by the imaging unit,
(b-3) face image data captured by the image capturing unit after capturing the pre-makeup image data, wherein the user is in a state where makeup is applied to at least a part of the user's face. Comparative image data of the face of
A storage unit capable of storing
(c) based on the reference image data, the pre-makeup image data, and the comparison image data, an evaluation unit that evaluates the quality of makeup applied to the user,
(d) a notification unit that notifies the user of desired information,
Equipped with
The case of each face element in the face of the user, one or a plurality of faces elements cosmetic that satisfies the desired workmanship has not been subjected to, are noted element,
The evaluation unit is
(c-1) From each of the reference image data, the pre-makeup image data, and the comparison image data, an extraction unit capable of extracting image data corresponding to the existence region of each face element,
(c-2) After the presence area corresponding to the element of interest is extracted as the first and second areas from the pre-makeup image data and the comparison image data by the extraction unit, respectively,
A first calculation unit that calculates a first index value for each face element included in the target element by using image data included in each of the first and second regions;
(c-3) Whether or not makeup is applied to each face element included in the element of interest is determined based on the corresponding one of the first index values calculated by the first calculator. A presence determination unit configured to
(c-4) With respect to the makeup-finished element which is determined by the presence/absence determining unit that the makeup is applied among the face elements included in the attention element,
After extracting the existing regions corresponding to the makeup-finished element from the comparison image data and the reference image data as the third and fourth regions, respectively, by the extraction unit,
A second calculation unit that calculates a second index value for each face element included in the makeup-finished element by using image data included in each of the third and fourth areas;
(c-5) It is configured to execute the determination of the workability of each face element included in the makeup element based on the corresponding one of the second index values calculated by the second calculation unit. A finishing judgment section,
Have
The makeup evaluation device, wherein the notification unit notifies the user of the determination result of the finish determination unit. The makeup evaluation apparatus according to claim 1,
The makeup evaluation apparatus, wherein each of the reference image data, the pre-makeup image data, and the comparison image data is obtained by horizontally reversing a captured image. The makeup evaluation apparatus according to claim 1 or 2,
When the face image captured by the image capturing unit is newly stored in the storage unit as the comparison image data, the evaluation unit evaluates the quality of makeup applied to the user. Makeup evaluation device. The makeup evaluation apparatus according to claim 1 or 2,
When the comparison image data stored in the storage unit is updated by the face image captured by the image capturing unit, the evaluation unit evaluates the quality of makeup applied to the user. Makeup evaluation device. The makeup evaluation apparatus according to any one of claims 1 to 4,
(e) When the evaluation unit determines that makeup has been completed for at least one face element included in the attention element, completion information indicating that makeup has been completed for the face element is stored in the storage unit. Registration unit to register in the first database,
Further equipped with,
The first database includes a record corresponding to each face element,
Each record included in the first database is at least
Sequence information indicating the order of applying makeup to the corresponding face element,
Face element information for identifying the corresponding face element,
Completion information indicating whether makeup of the corresponding face element is completed,
It is possible to store
The notifying unit notifies the user of a facial element to be applied next based on the sequence information, the facial element information, and the completion information of each record registered in the first database. Makeup evaluation device. The makeup evaluation apparatus according to claim 5,
(f) an input unit capable of accepting an instruction from the user after being notified of the determination result by the finish determination unit,
Further equipped with,
The makeup evaluation apparatus, wherein the comparison image data stored in the storage unit is updated by a face image captured by the image capturing unit according to an instruction from the input unit. The makeup evaluation apparatus according to claim 5,
The makeup|decoration evaluation apparatus characterized by the said registration part registering the said completion information in the corresponding record of the said 1st database, when the judgment result of the said finish judgment part is a desired range. A makeup evaluation system,
(a) a terminal,
(b) an information processing device,
Equipped with
The terminal is
(a-1) an image pickup unit capable of picking up a face image of the user,
(a-2) an informing unit for informing the user of desired information,
Have
The information processing device,
(b-1) at least the face image data of the user, that is,
Reference image data in which the face of the user is imaged in a state where the reference makeup is applied to the face of the user,
Before-makeup image data in which the face of the user before makeup is imaged by the imaging unit,
Face image data captured by the image capturing unit subsequent to capturing of the pre-makeup image, wherein the user's face is captured with makeup applied to at least a part of the user's face. Comparison image data,
A storage unit capable of storing
(b-2) based on the reference image data, the pre-makeup image data, and the comparison image data, an evaluation unit for evaluating the quality of makeup applied to the user,
Have
The case of each face element in the face of the user, one or a plurality of faces elements cosmetic that satisfies the desired workmanship has not been subjected to, are noted element,
The evaluation unit of the information processing device,
From each of the reference image data, the pre-makeup image data, and the comparison image data, an extraction unit capable of extracting image data corresponding to the existence region of each face element,
After extracting the existing regions corresponding to the element of interest from the pre-makeup image data and the comparison image data as the first and second regions, respectively, by the extraction unit,
A first calculation unit that calculates a first index value for each face element included in the target element by using image data included in each of the first and second regions;
It is configured to perform a determination as to whether or not makeup is applied to each face element included in the attention element, based on a corresponding one of the first index values calculated by the first calculation unit. The presence/absence determination unit,
With respect to the makeup-finished element which is determined by the presence/absence determining unit that the makeup is applied among the face elements included in the attention element,
After extracting the existing regions corresponding to the makeup-finished element from the comparison image data and the reference image data as the third and fourth regions, respectively, by the extraction unit,
A second calculation unit that calculates a second index value for each face element included in the makeup-finished element by using image data included in each of the third and fourth areas;
A finish determination unit configured to execute the determination of the workability of each face element included in the makeup element based on the corresponding one of the second index values calculated by the second calculation unit;
Have
The makeup evaluation system, wherein the notification unit of the terminal that receives the determination result of the finish determination unit notifies the user of the determination result. The makeup evaluation system according to claim 8,
The makeup evaluation system, wherein each of the reference image data, the pre-makeup image data, and the comparison image data is obtained by horizontally reversing a captured image. In the makeup evaluation system according to claim 8 or 9,
The information processing device,
(b-3) When the evaluation unit determines that makeup has been completed for at least one face element included in the attention element, completion information indicating that makeup has been completed for the face element is stored in the storage unit. A registration unit for registering in the stored first database,
Further equipped with,
The first database includes a record corresponding to each face element,
Each record included in the first database is at least
Sequence information indicating the order of applying makeup to the corresponding face element,
Face element information for identifying the corresponding face element,
Completion information indicating whether makeup of the corresponding face element is completed,
It is possible to store
The notifying unit notifies the user of a facial element to be applied next based on the sequence information, the facial element information, and the completion information of each record registered in the first database. Makeup evaluation system. The makeup evaluation system according to claim 10,
The terminal is
(a-3) an input unit capable of receiving an instruction from the user after the determination result by the finish determination unit is notified,
Further equipped with,
The makeup evaluation system, wherein the comparison image data stored in the storage unit is updated by a face image captured by the image capturing unit in response to an instruction from the input unit. A makeup evaluation method,
(a) a step of evaluating the quality of makeup applied to the user based on the reference image data, the pre-makeup image data, and the comparison image data,
(b) informing the user of desired information,
Equipped with
The case of each face element in the face of the user, one or a plurality of faces elements cosmetic that satisfies the desired workmanship has not been subjected to, are noted element,
The reference image data is face image data in which the face of the user is imaged in a state where the reference makeup is applied to the face of the user,
The pre-makeup image data is face image data in which the face of the user before makeup is imaged by an imaging unit,
The comparison image data is face image data captured by the image capturing unit after capturing the pre-makeup image, and in a state where makeup is applied to at least a part of the face of the user, The face was captured,
The step (a) is
(a-1) From each of the reference image data, the pre-makeup image data, and the comparison image data, a step of extracting image data corresponding to the existence region of each face element,
(a-2) After extracting the existing areas corresponding to the element of interest from the pre-makeup image data and the comparison image data as the first and second areas, respectively, in the step (a-1),
Calculating a first index value for each face element included in the target element by using image data included in each of the first and second regions;
(a-3) Based on the corresponding one of the first index values calculated in the step (a-2), it is determined whether or not makeup is applied to each face element included in the attention element. The steps to perform,
(a-4) With respect to the makeup-finished element which is determined by the presence/absence determining unit that the makeup is applied among the face elements included in the attention element,
After extracting existing regions corresponding to the makeup-finished element from the comparative image data and the reference image data as the third and fourth regions, respectively, by the step (a-1),
Calculating the second index value for each face element included in the makeup element by using the image data included in each of the third and fourth regions;
(a-5) a step of performing a workability determination of each face element included in the makeup element based on a corresponding one of the second index values calculated in the step (a-4),
Have
In the step (b), the notifying unit notifies the user of the determination result of the step (a-5).

Images