JP2018116589A - State identification apparatus, program, and method using changed image group of target images - Google Patents

Abstract

An object of the present invention is to provide an apparatus capable of identifying the state of an identification target with higher accuracy even if the image of the identification target appears in various ways under various circumstances. Kind Code: A1 A state identification device that uses an image of an identification target and identifies a state of the identification target by a discrimination model is provided for one input image of the identification target. image group generating means for performing processing for changing the direction and maintaining the state, and generating an image group including the image generated by this processing; identification processing means for using the generated image group when identifying Here, it is also preferable that the image group generating means at least change at least one type related to the appearance of the one image in a direction opposite to the direction of the one image. [Selection drawing] Fig. 1

Description

  The present invention relates to a technique for identifying a state of a predetermined target based on an image related to the predetermined target.

  Conventionally, various techniques have been devised for identifying a state of a predetermined object, for example, a human facial expression using an image related to the predetermined object, for example, a photographic image obtained by photographing a face.

  In particular, in the field of human facial expression recognition, many studies have been conducted using positive, negative, and neutral three classification models and Paul Ekman's seven classification models (neutral, joy, disgust, anger, surprise, sadness, and fear). Are working on improving facial expression recognition technology.

  As an example of such an approach, Patent Literature 1 discloses a technique for learning a feature amount of a large amount of face image data based on the above classification model and identifying a facial expression based on the feature amount. In particular, this technique aims to efficiently collect learning data of natural facial expressions, not intentionally created faces, and create a discriminator with high recognition accuracy.

JP 2011-150381 A

  However, with the conventional technology as described in Patent Document 1, in order to achieve high identification accuracy that can withstand an actual facial expression recognition environment, a large number of facial images taken in various situations and in various shooting conditions However, it becomes necessary as learning data.

  If this learning data is not available in large quantities and the discriminator cannot sufficiently learn the facial images of various shooting conditions in various real situations, there are many cases where the facial expression is erroneously determined. As a result, the discrimination accuracy is lowered, and the discriminator becomes unusable or less versatile.

  For example, in facial expression identification in a face image facing directly in front, even in a discriminator that has reached a certain discrimination accuracy, a part of the face may protrude from the image range (image frame) in the face image to be identified. , The face is facing up, down, left or right, the head is tilted, parts such as hats or sunglasses are attached, and the illuminance (lightness) of the face part of the image is too low If it is too high, the identification accuracy often decreases significantly.

  On the other hand, in reality, it is extremely difficult to prepare a large amount of face images of various shooting conditions in various situations as learning data. As a result, in the identification of facial expressions in a face image, depending on the situation of the face image and how it was photographed, there are many situations in which facial expressions are erroneously identified by processing with low facial expression identification accuracy. Resulting in.

  Therefore, the present invention provides an apparatus, a program, and a method capable of identifying the state of an identification target with higher accuracy even for an image of the identification target that is viewed in various ways under various circumstances. The purpose is to provide.

According to the present invention, a state identification device for identifying a state of an identification object by an identification model using an image related to the identification object,
An image group including an image generated by the processing is performed on the input image related to the identification target by performing a process of changing the appearance of the identification target in the one image and maintaining the state. Image group generation means for generating;
When constructing the identification model and / or identifying the state using the identification model, a state identification device is provided that includes an identification processing unit that uses the generated image group.

  As an embodiment of the state identification device according to the present invention, the image group generation means, for the one image, for at least one type related to the appearance, in a direction opposite to the direction in the one image. It is also preferable to make at least the change.

  Further, as another embodiment of the state identification device according to the present invention, the image group generation means, for the one image, for at least one type related to the appearance, a direction set in advance for the at least one type It is also preferable to change at least a direction suitable for identification.

  Furthermore, as still another embodiment of the state identification device according to the present invention, the image group generation unit may change the one image in accordance with a plurality of types of the appearance. preferable.

As still another embodiment of the state identification device according to the present invention, the image group generation means is preset with respect to the at least one type for at least one type related to the appearance for the one image. To generate a set of images to build the identification model characterized by the direction,
The identification processing means preferably constructs the identification model using the generated image group.

Furthermore, as still another embodiment of the state identification device according to the present invention, the image group generation means may take all of the at least one type related to the appearance with respect to the one image. To generate an image group for constructing the identification model in which the variation in the identification result for each direction for the at least one type is suppressed,
The identification processing means preferably constructs the identification model using the generated image group.

  Further, as still another embodiment of the state identification device according to the present invention, the image group generation means, for one image associated with one state as a correct answer, for at least one type related to the appearance, It is also preferable to make a change by a preset number of changes for the one state to generate the image group of a predetermined number of samples.

  Here, in the embodiment according to the above-described change number setting, the image group generation unit is configured such that the number of samples of the image associated with the one state is smaller than the number of samples of the image associated with the other state. In this case, it is also preferable to change the image associated with the one state by a larger number of changes than the image associated with the other state.

Furthermore, as still another embodiment of the state identification device according to the present invention, the identification processing means generates the image generated when the identification model is constructed and when the state is identified using the identification model. Use groups,
When generating an image group used for identifying the state, the image group generating means constructs the identification model for any type related to the appearance of the identification target with respect to one input image. The change was made in the same direction as the change for the type made when generating the image group used at the time, while the change for the type was not made when generating the image group In some cases it is also preferred not to make any changes.

  As still another embodiment of the state identification device according to the present invention, the identification processing means determines the state for each of the images included in the generated image group using the identification model, and the image The weighted average of the state determined by the weight set in advance according to whether or not is the original image before the change and / or the weight set in advance according to the changed type in the image, It is also preferable to use the weighted average result as the identification result of the state.

  Furthermore, as still another embodiment of the state identification device according to the present invention, it is preferable that the image group generation unit generates the image group including one image related to the identification target.

As still another embodiment of the state identification device according to the present invention, the state identification device extracts an image region related to the identification target for one image related to the input identification target, and the image region And a degree determination means for determining the presence and / or degree of occurrence of the type in at least one type related to the appearance based on
It is also preferable that the image group generation unit changes the at least one type by changing the determined presence state and / or the degree.

  Furthermore, in the state identification device according to the present invention, specifically, the image group generation means displays the degree of the identification target in the one image and the appearance of the identification target as a plurality of types related to the appearance. Change at least one of the orientation or inclination, presence / absence or type of the object to be identified, brightness of the object to be identified, hue of the object to be identified, and degree of contrast between the object to be identified and the background Is also preferable.

In the state identification device according to the present invention, specifically, the identification target is a face, and the state is a facial expression,
The image group generation means performs a process of changing the appearance of the face in the one image and maintaining the facial expression for the one image related to the input identification target,
It is also preferable that the identification processing means identifies facial expressions in one input image using the identification model.

According to the present invention, there is also a state identification system for identifying a state of the identification target by an identification model using an image related to the identification target,
An image group including an image generated by the processing is performed on the input image related to the identification target by performing a process of changing the appearance of the identification target in the one image and maintaining the state. Image group generation means for generating;
An identification model construction means for constructing the identification model by the generated image group;
There is provided a state identification system having identification processing means for identifying the state by the identification model using the generated image group.

According to the present invention, there is further provided a state identification program for causing a computer mounted on an apparatus for identifying a state of the identification target using an identification model using an image related to the identification target,
An image group including an image generated by the processing is performed on the input image related to the identification target by performing a process of changing the appearance of the identification target in the one image and maintaining the state. Image group generation means for generating;
When the identification model is constructed and / or when the identification model is used to identify the state, a state identification program is provided that causes a computer to function as identification processing means that uses the generated image group.

According to the present invention, there is further provided a state identification method implemented in a computer mounted on an apparatus for identifying a state of the identification target using an identification model using an image related to the identification target,
An image group including an image generated by the processing is performed on the input image related to the identification target by performing a process of changing the appearance of the identification target in the one image and maintaining the state. Generating step;
A state identification method is provided that includes using the generated image group when constructing the identification model and / or identifying the state using the identification model.

  According to the state identification device, the program, and the method of the present invention, it is possible to identify the state of the identification target with higher accuracy even for the image of the identification target that is displayed in various ways under various circumstances. it can.

It is a functional block diagram which shows the function structure in one Embodiment of the state identification apparatus by this invention. It is a flowchart which shows the outline of one Embodiment of the state identification method by this invention. It is face image data for demonstrating the example of the appearance type which concerns on the determination in the appearance degree determination part. It is a flowchart which shows one Embodiment of the change image preparation / image group production | generation process which concerns on this invention. It is a schematic diagram which shows one Embodiment of the identification model constructed | assembled by the identification model construction part. It is a schematic diagram which shows one Embodiment of the state identification system by this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[State identification device]
FIG. 1 is a functional block diagram showing a functional configuration in an embodiment of a state identification device according to the present invention.

  According to FIG. 1, the smartphone 1 as the state identification device of the present embodiment has a built-in camera 102 having a known configuration, and for example, the user's face is photographed by using the camera 102. A photographic image is generated, the facial expression of the user reflected in the generated photographic image is identified, and the identification result can be displayed on, for example, the touch panel display (TP / DP) 106. Naturally, it is also possible to obtain a photographic image of the face as a facial expression identification target from the outside via the communication interface 101 via the communication network and perform facial expression identification processing.

  Furthermore, as one application example, the application 121 of the smartphone 1, for example, the conversation AI application, uses the facial expression identification result, for example, understands the emotion (utterance intention) of the user who is interacting with, and displays the response content. It is also possible to make adjustments and personalize the dialogue with the user.

  Furthermore, in this embodiment, the smartphone 1 acquires a learning (human face photograph) image in the facial expression identification processing unit 114 for facial expression identification from the image management server 2 via the communication interface unit 101. Is also preferable. At this time, the smartphone 1 uses the image group generation unit 113 to generate a learning image group used for learning based on the acquired learning image. As a result, it is possible to prepare a sufficient amount of learning images without taking in a large amount of image samples from the outside as conventionally.

A smartphone 1 as such a state identification device according to the present invention is a device that uses an image related to an identification target (for example, a human face) and identifies the state of the identification target (for example, facial expression) by an identification model,
(A) A process for changing the appearance of the identification target (for example, a face) in the one image and maintaining the state (for example, a facial expression) is performed on one input image related to the identification target. Image group generation means (image group generation unit 113) for generating an image group including the generated image;
(B) Identification processing means (expression identification processing unit) that uses the image group generated in (A) above when constructing an identification model and / or identifying a state (for example, a facial expression) using the identification model 114).

  Here, when constructing an identification model, a large amount of images related to an identification target is usually required as learning data in order to achieve a practically accurate identification accuracy. Moreover, it is preferable that a large number of images that differ from each other in the appearance of the identification target are included. However, in reality, it is extremely difficult to prepare a large number of identification target images having various appearances. In addition, when identifying a state using an identification model, depending on how the identification target appears in the identification target image, the state identification result by the identification model is affected, and there are many cases where the state is identified incorrectly. Was the actual situation.

  On the other hand, according to the smart phone 1 provided with the said structure (A) and (B), the image group containing the image produced | generated by the process which changes the appearance of an identification target and maintains the state is produced | generated. Thereby, for example, it is also possible to reliably prepare a learning and / or identification image group that is a larger number of images related to identification objects that are viewed in various ways. As a result, the state of the identification target can be identified with higher accuracy even in the case of an image of the identification target that is viewed in various ways under various circumstances.

  Of course, the state identification device according to the present invention embodied in the above-described smartphone 1 is not applied only when the target state to be identified is a facial expression of a human face. According to the present invention, even a state of an identification target whose state identification result is affected by the appearance in an image representing one state can be identified with higher accuracy. For example, it is possible to identify the sex, age, race (type), clothes, etc. of a human or animal. Furthermore, the present invention can also be applied to recognition of various objects such as furniture, daily necessities, and automobiles. Conventionally, the result of such state identification has included a large error or wrong depending on how it is seen in the image, but according to the present invention, such a state can be identified more accurately. It is.

  Furthermore, naturally the state identification apparatus by this invention embodied in the smart phone 1 is not limited to a smart phone. For example, a tablet computer, a notebook computer, a personal computer (PC), a set top box (STB), a robot, a digital signage, or the like can be employed as the state identification device. For example, in these devices (terminals) with a built-in camera, even in various real environments, by reading the user's facial expression, a response corresponding to the information related to the read facial expression is made, or information relating to the read facial expression It is also possible to evaluate an action or the like for the user that has been performed first.

[State identification method]
FIG. 2 is a flowchart showing an outline of an embodiment of the state identification method according to the present invention. Here, in this embodiment, the identification target is a human face, and the state of the identification target is the facial expression of this face. 2A shows a flow of facial expression identification model construction processing (learning processing) using the learning image group according to the present invention, and FIG. 2B shows the determination image group according to the present invention. The flow of used facial expression identification processing is shown. Hereinafter, first, an outline of the facial expression identification model construction process will be described with reference to FIG.

(S101) A learning image is acquired. This learning image becomes an original image for generating a learning image group and increasing the number of image samples during the learning process.

(S102) A face image region is extracted from the acquired learning image.
(S103) In the extracted face image area, the presence / absence and / or degree of occurrence of the appearance type is determined for each preset appearance type.

  Here, the types of appearance include, for example, one state in which the face is facing the front and four states in which the face is facing upward, downward, left and right when viewed from the front. Can be set in advance. Further, the degree of the appearance type may be an angle formed by the direction in which the face is facing with reference to the front direction (zero degree). This appearance type will be described later in detail with reference to FIG.

(S104, S105) In the acquired learning image, a changed image is created based on the determination result of the presence type and / or degree of appearance type for each appearance type, and a learning image group including the changed image is created. Generate. These modified image creation processing and learning image group generation processing will be described later in detail with reference to FIG.
(S106) A facial expression identification model is constructed using the generated learning image group. This completes the facial expression identification model construction process.

Next, an outline of facial expression identification processing will be described with reference to FIG.
(S201) An identification target image is acquired. This identification target image is a state identification target, and thereafter becomes an original image for generating a determination image group and increasing the number of image samples during the state identification processing.

(S202) A face image area is extracted from the acquired identification target image.
(S203) In the extracted face image area, the presence / absence and / or degree of occurrence of the appearance type is determined for each preset appearance type. The appearance type can be the same as that described in step S103. In any case, this type of appearance will be described later in detail with reference to FIG.

(S204, S205) In the acquired identification target image, a changed image is created based on the determination result of the presence type and / or degree of appearance type for each appearance type, and a determination image group including the changed image is created. Generate. These modified image creation processing and learning image group generation processing can be the same as the processing in steps S104 and S105, or can be linked processing associated with each other. In any case, these processes will be described in detail later with reference to FIG.

(S206) For each of the generated determination image groups, the facial expression is determined using the generated facial expression identification model. The facial expression identification model used here may be constructed in step S106. Alternatively, for example, a facial expression identification model generated in advance by a known method can be used.

(S207) The determination results in step S206 are combined to identify facial expressions in the acquired identification target image. This completes the facial expression identification process.

[Device configuration]
Next, referring back to FIG. 1, a specific configuration of the state identification device according to the present invention will be described below. As shown in the functional block diagram of FIG. 1, the smartphone 1 of the present embodiment, which is a state identification device (facial expression identification device), includes a communication interface unit 101, a camera 102, an image database 103, and a changed image storage unit 104. The determination result storage unit 105, the touch panel display (TP / DP) 106, and the processor memory are included. Here, the processor memory realizes a state identification function (facial expression identification function) by executing a program that causes the computer of the smartphone 1 to function.

  Furthermore, the processor memory includes an image management unit 111, an appearance degree determination unit 112, an image change unit 113a, a learning image group generation unit 113b, and a determination image group generation unit 113c as functional configuration units. Unit 113, facial expression identification processing unit (state identification processing unit) 114 including identification model construction unit 114a and facial expression determination unit 114b, application 121, and input / output control unit 122. Here, the flow of processing shown by connecting the functional components of the smartphone 1 in FIG. 1 with arrows is understood as an embodiment of the state identification method (expression identification method) according to the present invention.

The communication interface unit 101
(A) A learning image that is an original image for generating a learning image group, a large amount of learning data in the facial expression identification processing unit 114,
(B) The identification target image that is the original image for generation of the determination image group and the data of the identification target image that is directly subjected to the identification processing by the facial expression identification processing unit 114 are transferred from the external image management server 2 via a communication network such as the Internet And output to the image management unit 111. It is also possible to download the facial expression identification program (application) according to the present invention or an application program that can provide a service using the facial expression identification result, such as a conversation AI application, via the communication interface unit 101. It has become.

  For example, the camera 102 can photograph a user's face and output face image data as an identification target image to the image management unit 111.

  The image management unit 111 receives the above-described image data from the communication interface unit 101 or the camera 105, and stores and manages the image data in the image database 103. Further, in response to an instruction from the user or the like, a request from in-device processing, or the like, the learning image or the identification target image that is the original image for generating the image group is output to the appearance degree determination unit 112.

  Similarly, in FIG. 1, the appearance degree determination unit 112 determines an image region related to the identification target (for example, the face) for one image (learning image or identification target image) related to the input identification target (for example, the face). Based on this image area, the presence / absence and / or degree of occurrence of the type in at least one type of appearance is determined. Here, it is also possible to interpret that occurrence of a type is (degree) = 1 and that no type is generated (degree) = 0. Incidentally, after this, the image group generation unit 113 changes the original image by changing the state and / or degree of presence / absence determined here for the type, and creates a changed image.

  The extraction of these image areas and the determination relating to the type of appearance use, for example, a known method of scanning an image and extracting an edge whose pixel value changes abruptly to detect an identification target (for example, a face). Can be implemented. Moreover, you may utilize the open source which enables such extraction. Furthermore, it is also possible to use an extraction method using machine learning such as publicly known deep learning.

  FIG. 3 is face image data for explaining an example of the appearance type related to the determination by the appearance degree determination unit 112.

  As an example of the appearance type, FIG. 3A shows an image in which the face protrudes from the screen. In this way, as the appearance type, for example, a type relating to the protrusion of the face including one state where the face does not protrude from the screen and a state where the face protrudes from each of the upper, lower, left and right sides of the screen is set. Can do. Here, as the degree of the appearance type, for example, the area or the number of pixels of the face portion that protrudes outside the image ((the total area of the face or the number of pixels) − ( It may be the ratio of the area of the face portion or the number of pixels)) remaining in the image.

  As another example of the appearance type, FIG. 3B shows an image with a large face orientation / tilt angle. In this way, the types of appearance include, for example, one state where the face is facing the front, and a state where the face is facing toward each of the top, bottom, left and right when viewed from the front. A type related to the inclination may be set. Here, the degree of the appearance type can be, for example, an angle formed by a direction in which the face is facing with reference to the front direction (zero degree) (and an inclination angle representing the degree of inclination of the face).

  The determination of the appearance type related to the orientation of the face is performed by, for example, generating an identification model in which patterns of these four orientations are learned in advance from an image of a face facing in the up, down, left, and right directions. This method can be implemented by using a method of determining the face orientation using. Similarly, for the determination of the appearance type related to the tilt of the face, a method for generating an identification model in which a plurality of patterns having different tilts is similarly learned and using the method for determining the tilt of the face is used. Can be implemented.

  Furthermore, as yet another example of the appearance type, FIG. 3C shows an image in which a wearing object such as sunglasses, a hat, or a mask is attached to the face or the vicinity of the face. In this way, for example, as a type of appearance, for example, one state where no attachment is attached to the face or the vicinity of the face, and glasses, sunglasses, goggles, a hat, cheek cheeks, a scarf, a mask, and other attachments on the face or the vicinity of the face It is possible to set the type related to the addition of the wearing object including the state where each of the objects is attached. Here, as the degree of the appearance type, for example, the area or the number of pixels of the face part shielded by the wearing object ((the total area of the face or the number of pixels) − (the total number of the faces or the number of pixels) − ( It may be the ratio of the area or the number of pixels of the exposed face part that is not shielded)).

  In addition, for the determination of the appearance type related to the addition of the wearing object, for example, an identification model in which the presence / absence of the wearing object is learned in advance is generated from the image of the face to which the predetermined wearing object is attached. This can be implemented by using a method for determining the presence / absence of a wearing object using the.

  As yet another example of the appearance type, FIG. 3D shows an image in which the illuminance (brightness) of the face is too high or too low. Thus, for example, the appearance type is, for example, one state in which the average brightness (illuminance) of all pixels in the face image area is within a predetermined reference range, and the average brightness (illuminance) of all pixels in the face image area. You may set the type which concerns on the illumination intensity (brightness) of a face including the state which is too high or too low exceeding a predetermined reference range. Here, the degree of the appearance type can be, for example, the difference between the average value of the illuminance (lightness) and the illuminance (lightness) reference value.

  Of course, the appearance types related to the determination by the appearance degree determination unit 112 are not limited to the above four. One, two, or three of them may be included, and other types of appearance may be included. Here, as other types of appearance, for example, the type of appearance related to the average of colors (saturation) in all pixels of the face image region, the brightness / saturation contrast between the face image region and the background region in the face image And the size of the face part (for example, the ratio of the area of the face part or the number of pixels to the entire image).

  That is, examples of appearance types are summarized as follows: (a) the degree of identification target appearing in one image, (b) the direction or inclination of the identification target appearing, and (c) presence / absence of something blocking the identification target or At least one of the type, (d) the brightness of the identification target, (e) the color of the identification target, (f) the degree of contrast between the identification target and the background, and the like can be adopted as the appearance type. As described above, in the image, if the identification target can cause a change that can be discriminated while maintaining the state to be the identification target, a series of appearances related to the change may be set as the appearance type. Is possible.

  Returning to FIG. 1, the image changing unit 113 a of the image group generating unit 113 identifies one image (learning image or identification target image) related to the input identification target (for example, a face) in this one image. A process of changing the appearance of the target (for example, the face) and maintaining the state is performed.

  Next, the learning image group generation unit 113b and the determination image group generation unit 113c of the image group generation unit 113 respectively generate a learning image group and a determination image group including the changed image created by the processing in the image change unit 113a. It is also preferable that the generated image group (and the original image) is stored in the modified image storage unit 104 and is appropriately read and used.

  Here, to each image included in the learning image group, the “correct information” assigned to the learning image that is the original image is added as it is. When a facial expression is identified as a state to be identified, values representing three categories (for example, 1, 0, and −1, respectively) relating to facial expressions of positive, neutral, and negative can be used as the correct answer information. Naturally, other facial expression classification models, such as the emotion classification model based on Paul Ekman's seven categories (neutral, joy, disgust, anger, surprise, sadness, fear), and more detailed emotion classification models (eg, Paul A value that represents the categories in the Ekman model's seven categories, plus fun, contempt, satisfaction, embarrassment, excitement, guilt, pride based on achievement, security, persuasion, joy, and shame. Can be used as correct answer information.

  Note that the generated image group preferably includes one input image (original image). Of course, an image group that does not include the original image may be generated, but by using the image group that includes the original image, an identification model with higher identification accuracy can be constructed, or the identification accuracy by the identification model can be improved. It becomes possible to do. Here, various embodiments that can be taken by the image change / image group generation processing in the image group generation unit 113 described above will be described below.

(Mode a) At least one change in the direction opposite to the direction in the one image is performed for at least one appearance type for one input image (learning image or identification target image).

  In this form a, for example, when one input image is a left-facing face image with respect to the appearance type related to the face orientation / tilt (described in FIG. 3B), this left-facing face is displayed. It is possible to create a modified image that has been subjected to a process of changing to the right (for example, to invert), and to generate an image group including at least the modified image. For example, an image group (a learning image group or a determination image group) including a left-facing face image (original image) and a right-facing face image (changed image) may be generated. By the way, by such processing, it becomes possible to improve the versatility of the identification model to be constructed (robustness with respect to the appearance in the image) or to improve the identification accuracy corresponding to the deviation of the identification model. .

(Mode b) For at least one appearance type with respect to one input image (learning image or identification target image), the direction is set in advance with respect to the appearance type and is suitable for identification. Make at least changes.

  In this form b, for example, for the appearance type related to the illuminance (brightness) of the face (described in FIG. 3D), the brightness (illuminance) of all the pixels of the face image area in one input image is displayed. If the average value is too high beyond the predetermined reference lightness (illuminance) range, the lightness (illuminance) at each pixel is changed so that the average value of the lightness (illuminance) becomes a value in the reference lightness (illuminance) range. A modified image that has been processed can be created, and an image group that includes at least the modified image can be generated. For example, an image group including an image (original image) whose average value of brightness (illuminance) is too high and an image (changed image) in which the average value of brightness (illuminance) is within a predetermined reference brightness (illuminance) range ( Learning image group or determination image group) may be generated. Incidentally, it is possible to construct a discrimination model with higher discrimination performance and improve the accuracy of the discrimination result by the discrimination model by such processing.

(Mode c) The input one image (the learning image or the identification target image) is changed in accordance with a plurality of preset appearance types.

  In this form c, for example, the appearance type related to the face orientation / tilt (described in FIG. 3B) and the appearance type related to the illuminance (brightness) of the face (described in FIG. 3D). For both types, it is possible to create a modified image that has been modified at the same time. Specifically, for example, an image of a face facing left and an average value of brightness (illuminance) in the face image area that is too high (original image), an image of a face facing right and brightness (illuminance). ) May be created and included in the image group (learning image group or determination image group). Incidentally, by such processing, it is possible to improve the versatility of the identification model to be constructed (robustness against various changes in appearance) and improve the accuracy of the identification result by the identification model.

(Mode d) For one input image (learning image or identification target image), at least one appearance type is changed in a direction set in advance with respect to the appearance type, and features are determined depending on the direction. An image group for constructing an identification model to be attached is generated.

  In this form d, for example, with regard to the appearance type related to the orientation / tilt of the face (described in FIG. 3B), when one input image is a face image that is not downward, this face is faced downward. It is possible to create a modified image that has been subjected to the process of changing to, and generate an image group including at least the modified image. That is, in this case, it is possible to generate an image group (a learning image group or a determination image group) that always includes an image of a downward face (or frequently). By the way, by such processing, an identification model characterized by a predetermined direction (for example, an identification model that can particularly accurately identify a facial expression of a downward face, for example) is constructed, or the identification model characterized by a predetermined direction It is also possible to improve the identification accuracy by inputting an image according to the above.

(Mode e) For one input image (learning image or identification target image), at least one appearance type is changed in all possible directions for the appearance type, and the appearance type An image group for constructing an identification model in which variations in identification results for each direction are suppressed is generated.

  In this form e, for example, when one input image is a left-facing face image with respect to the appearance type related to the face orientation / tilt (described in FIG. 3B), this left-facing face is selected. Then, it is possible to create a modified image that has been processed to change to the right, upward, and downward, and generate an image group (learning image group or determination image group) that includes the original image and these modified images. By the way, by this kind of processing, an identification model with higher identification accuracy is constructed, and the versatility of the identification model (robustness with respect to the appearance in the image) is improved, or the accuracy of identification by the identification model is improved. Is also possible.

(Mode f) For one image (learning image) associated with one state as a correct answer, at least one appearance type is changed by the number of changes set in advance for the one state. A learning image group having a predetermined number of samples is generated.

  According to this form f, for example, it is possible to prepare as many image samples as necessary for learning that are associated with one state (as a correct answer). Here, the change by the set number of changes is, for example, a step that changes the “degree” of the appearance in the appearance type step by step and has a plurality of (only the number of changes) different degrees of the appearance type. This is accomplished by creating an image.

  Here, when the number of samples of the image associated with one state is smaller than the number of samples of the image associated with the other state, the other state is determined for the image associated with the one state. It is also preferable to make the change by the number of changes larger than the image associated with. For example, when constructing a facial expression identification model that identifies whether a facial expression is positive, neutral, or negative, a facial image sample of a facial expression that is negative as a correct answer is compared to other states. It is relatively difficult to obtain. In this case, the original image in the negative state is compared with the original image in the other state, for example, in the face image area in the appearance type related to the illuminance (brightness) of the face (described in FIG. 3D). The average value of the brightness (illuminance) can be changed more finely, and more changed images can be created. As a result, it is possible to cause the facial expression identification model to perform good learning with less bias.

(Mode g) In the case of using an image group both when constructing an identification model and when identifying a state using this identification model, when generating a determination image group used for state identification, For any one type of input image (identification target image) related to the appearance of the identification target, the type used when generating the learning image group used in constructing the identification model The change is performed in the same direction as the change. On the other hand, when the learning image group is generated, the change is not performed if the change is not performed on the type.

  According to the form g, it is possible to prepare an image that has been subjected to the change process in the same direction for any type of appearance in the learning image group and the determination image group. That is, it is possible to align the tendency of change in appearance between the learning image group and the determination image group. As a result, for example, an identification model characterized by a predetermined direction (for example, an identification model that can particularly accurately identify a facial expression of a downward face, for example) is constructed, and an image input according to the feature of the identification model is performed. As a result, the identification accuracy can be improved.

  FIG. 4 is a flowchart showing an embodiment of the modified image creation / image group generation processing according to the present invention. The flow shown in the figure corresponds to the processing flow of steps S104 and S204 and steps S105 and S205 shown in FIG.

(S301) For the change process for the original image, the appearance type to be changed, and the direction and degree of change for the type are set.
(S302) Here, it is determined for which appearance type the change process is to be performed, and the process is sequentially performed for the set appearance type.

  In addition, it is also preferable that the set appearance type and the direction and degree (attribute) in the change of the type are registered in the appearance type list database and referred to as appropriate. By the way, in the present embodiment, four types of appearances are set: a type related to the protrusion of the face, a type related to the orientation / tilt of the face, a type related to the attachment addition, and a type related to the illuminance (brightness) of the face, Processing is sequentially performed for these appearance types.

(S303a, S304a) Whether the face (image area) extracted from the input image (original image) protrudes from the screen is determined for the appearance type related to the protrusion of the face. Here, when the true determination (determination that it is protruding) is performed, the change process is not performed for the appearance type. On the other hand, when a false determination is made, a segmentation process is performed for causing the face (image area) to protrude from the screen by a predetermined amount in a predetermined direction with respect to the input image (original image) where the face does not protrude.

(S303b, S304b, S304c) Whether or not the face (image area) extracted from the input image (original image) is oriented in any of the top, bottom, left, and right directions for the type of appearance related to the orientation / tilt of the face, And it is determined whether or not the face is tilted. Here, when a false determination (determination that it is facing front and not tilted) is made in any case, a process of tilting the face by a predetermined angle in a predetermined direction (for example, rotating the face image area) is performed. . In this case, the change process may not be performed for the appearance type. On the other hand, if a determination other than false is made for either of them, a process of changing (for example, reversing) the orientation or inclination of the face is performed, or the face is tilted by a predetermined angle in a predetermined direction (for example, rotating the face image area) Process).

(S303c, S304d) Determine whether or not a predetermined wearing object (for example, glasses or a hat) exists in or near the face image area extracted from the input image (original image) for the type related to the wearing object addition. . Here, when the true determination (determination that the predetermined wearing object exists) is performed, the change process is not performed on the appearance type. On the other hand, when a false determination is made, a process of adding a predetermined wearing object to a predetermined position in or near the face image area is performed on an input image (original image) without a predetermined wearing object. .

(S303d, S304e, S304f, S304g) For the type relating to the illuminance (brightness) of the face, the average value of the luminosity (illuminance) at all the pixels in the face image area extracted from the input image (original image) is a predetermined reference lightness. Whether it is within the (illuminance) range, beyond the range, too high, or below the range is too low. Here, when it is determined that it is within the range, the change process is not performed for the appearance type. Alternatively, in this case, a predetermined brightness (illuminance) changing process may be performed. On the other hand, if it is determined that the lightness (illuminance) average value exceeds the range and is too high or low and too low, the brightness (illuminance) of the input image (original image) with too high illuminance (lightness) The illuminance (brightness) changing process is performed so that the average value falls within a predetermined reference lightness (illuminance) range.

(S305) It is determined whether or not the change process has been determined for all the set appearance types or whether or not the change process has been performed. If a false determination is made here, the process proceeds to step S302 in order to perform the process for the appearance type for which the change process (determination) is not performed. Note that it is also possible to perform a change process for a plurality of appearance types for one input image (original image). In this case, in this step S305, it is determined that the input image that has already been subjected to the change process for one appearance type is subjected to the change process for the remaining appearance type.

(S306) On the other hand, when a true determination is made in step S305, a learning image group or a determination image group including the created changed image is generated. This completes the modified image creation / image group generation processing.

  Naturally, the modified image creation / image group generation processing according to the present invention is not limited to the embodiment described above. For example, it is possible to set various processing modes by applying at least one of the above-described forms a to g or combining a plurality thereof. In any case, according to the modified image creation / image group generation processing according to the present invention, it is possible to prepare a sufficient number of image samples with fewer original images as compared with the related art. Alternatively, it is possible to prepare a larger amount of image samples than in the past.

  Returning to FIG. 1, the facial expression identification processing unit 114 includes an identification model construction unit 114a and a facial expression determination unit 114b in the present embodiment. Among these, the identification model construction unit 114a performs learning using the generated learning image group to construct a (facial expression) identification model. At this time, correct information (originally possessed by the original image) is added to each image sample included in the learning image group. The identification model construction unit 114a refers to this correct answer information to collate and proceeds with learning.

  Here, the constructed identification model can be, for example, a classifier including a convolutional neural network (CNN) which is a kind of deep learning. Naturally, the discrimination model constructed by the learning image group according to the present invention is not limited to the discriminator including CNN, and may be a discriminator according to another machine learning method capable of state discrimination. Further, for example, as an identification model for determining facial expressions, feature amounts of parts such as eyes, nose, mouth, and eyebrows in the face image region are generated, and facial expressions are determined based on the similarity to the feature amount of the reference image region. It is also possible to adopt a method that does this.

  On the other hand, the facial expression determination unit 114b inputs the generated determination image group to the constructed identification model, and displays the changed image included in the determination image group and the state (expression) of the identification target in the identification target image (original image). A determination state (for example, 1 if the facial expression is positive, 0 if neutral, and -1 if negative) and a determination value (score) are output. In either one of the identification model construction unit 114a and the facial expression determination unit 114b, it is possible to perform processing using a large number of images for normal learning and identification target images without using the generated image group. is there.

  FIG. 5 is a schematic diagram showing an embodiment of an identification model constructed by the identification model construction unit 114a.

  As shown in FIG. 5, in this embodiment, the identification model constructed by the identification model construction unit 114a is configured based on a CNN that is a kind of forward propagation type. This CNN contains multiple convolution layers, which mimic the function of simple cells in the visual cortex of animals, and generate a feature map by sliding a kernel (weighted matrix filter) against the image. This is a layer for executing the convolution process. With this convolution process, it is possible to extract basic features such as edges and gradients while gradually reducing the resolution of the image, and obtain information on local correlation patterns.

  In this CNN, it is also preferable that each convolution layer is paired with a pooling layer (subsampling layer), and the convolution process and the pooling process are repeatedly performed. Here, the pooling process is a process that mimics the function of complex cells in the visual cortex of animals. The feature map output from the convolution layer (convolution filter response in a certain area) is expressed as a maximum value or an average value. In summary, it is a process of ensuring local translational invariance while reducing adjustment parameters. Thereby, it is possible to absorb a difference in appearance due to a slight shift in the image and acquire an appropriate feature amount that captures the original feature.

  The learning image group generation unit 113b outputs the generated learning image group to the identification model construction unit 114a so that the CNN can perform learning. Specifically, as a multi-layer network by inputting several image samples included in a large number of learning image groups generated from a large number of original images, and excluding the last layer among a plurality of layers in the CNN. The response is output as a feature amount, and this output is collated with a correct answer, and learning is performed by generating / updating neuron connection weights, network configuration parameters, and the like.

  Usually, such learning of CNN is performed using, for example, a large-scale image data set composed of a large number of identification target images stored in a general image database. On the other hand, in this embodiment, since a learning image group is generated from the original image and input to the CNN is performed, the number of images (original) is smaller than the number of learning image samples that are normally required. A large number of image samples required for the number of images) can be prepared.

  Returning to FIG. 1, the facial expression determination unit 114 b of the facial expression identification processing unit 114 inputs the generated determination image group to the constructed identification model, and the modified image included in the determination image group and the identification target image (original image) ) To determine the state (facial expression) of the identification target. Here, the constructed facial expression identification model includes, as correct answer information added to the learning image group, for example, values corresponding to three categories related to facial expressions such as positive, neutral and negative (for example, 1, 0 and −1). The case where it is learned will be described. In this case, the determination state for each image in the facial expression determination unit 114b (for example, 1 if the facial expression is positive, 0 if neutral, and -1 if negative), the corresponding determination value (score) among these three categories. ) Is determined to be the highest category state. The determination value (score) is an index indicating the confidence (reliability) of the corresponding category, and it is preferable that the determination value (score) is normalized so that the total value is 1.

  The facial expression identification processing unit 114 determines a state (facial expression) for each of the images included in the generated determination image group using the constructed identification model, and determines a determination state (for example, 1 if the expression is positive, neutral. If it is negative, -1) if negative, and the determination value corresponding to each category are output, and the final state identification is performed based on these determined determination values (scores). For example, an average value of determination values (scores) determined for each image in these determination image groups is preferably used as the state identification result.

Moreover, as a change aspect,
(A) Depending on whether the image whose state has been determined is the original image before the change,
Different “weights” may be set in advance, and using the weights, a weighted average of determination values (scores) determined for each image in the determination image group may be performed, and the weighted average result may be used as a state identification result. For example, by performing the weighted average process with the weight for the determination value of the original image being larger than the weight for the other image (changed image), the determination result of the original image is more strongly reflected in the final state identification result. It is also possible to make it.

As a further modification,
(B) According to the changed type in the image,
Different “weights” may be set in advance, and using the weights, a weighted average of determination values (scores) determined for each image in the determination image group may be performed, and the weighted average result may be used as a state identification result. For example, if it is found that the determination result of the facial expression is most strongly influenced by the orientation / tilt of the face, the weight for the determination value of the changed image changed for the appearance type related to the orientation / tilt of the face, You may perform a weight average process as a value larger than the weight with respect to the determination value of the change image changed about another appearance type. As a result, it is possible to obtain a state identification result in consideration of the strong influence of the face direction and inclination.

  Furthermore, it is also preferable to set a “weight” based on both (a) and (b) and derive the state identification result by this weight. It is also possible to determine the final state identification result by performing arithmetic processing other than (weight) averaging, for example, majority decision, on the determination value of each image.

  Here, the state identification result (and the state determination result) described above may be directly output to the application 121 and used, for example, or stored in the determination result storage unit 105, and thereafter appropriately, for example, by the application 121. It is also preferable to read and use. The application 121 can use the acquired state identification result (and the state determination result) as, for example, facial expression determination data in a predetermined application program. Further, the use result (processing result) and the original state identification result (and the state determination result) may be displayed on the touch panel display 106 via the input / output control unit 122, and may be displayed externally through the communication interface unit 101. It may be transmitted to the information processing apparatus.

[State identification system]
FIG. 6 is a schematic diagram showing an embodiment of a state identification system according to the present invention.

  According to FIG. 6, the state identification system of this embodiment has the facial expression identification preparation apparatus 3 as a server, and the smart phone 5 as a state identification apparatus by this invention.

  Among these, the facial expression identification preparation device 3 includes an appearance degree determination unit 312, an image group generation unit 313, and a facial expression identification processing unit 314 including an identification model construction unit 314 a. These functional components have the same functions and configurations as the functional components of the same name in the smartphone 1 shown in FIG. Therefore, the facial expression identification preparation device 3 generates a learning image group using, for example, a learning image acquired from the external image management server 2 as an original image, constructs an identification model using the generated learning image group, and constructs the constructed identification. The model (information thereof) can be transmitted to the smartphone 5, for example. Incidentally, the facial expression identification preparation apparatus 3 of the present embodiment having such a configuration generates a learning image group for learning by an identification model, and can be understood as a state identification apparatus according to the present invention.

  On the other hand, the smartphone 5 has an expression including a communication interface unit 501, a camera 502, a touch panel display 503, an image management unit 511, an appearance degree determination unit 512, an image group generation unit 513, and an expression determination unit 514 b. An identification processing unit 514, an application 521, and an input / output control unit 522 are included. These functional components have the same functions and configurations as the functional components of the same name in the smartphone 1 shown in FIG. Therefore, the smartphone 5 receives, for example, the identification model (information thereof) constructed from the facial expression identification preparation device 3, and generates and generates a determination image group using, for example, the identification target image captured by the camera 502 as an original image. Depending on the determination image group, the state can be identified using the obtained identification model, and the state identification result can be used by the application 521.

  As described above, in this embodiment, the identification model is constructed (learned) by the facial expression identification preparation device 3. Therefore, the smartphone 5 does not have to acquire a large amount of image samples from the image management server 2 or the like, and does not need to perform an identification model construction process with a large processing load. As a result, in the smartphone 5, the information processing amount to be executed in the device and the information amount to be taken in the device can be significantly reduced. In other words, the smartphone 5 can realize suitable facial expression identification with the size and processing capability at the portable terminal level.

  As yet another embodiment, the smartphone 5 does not include any of the appearance degree determination unit 512, the image group generation unit 513, and the facial expression identification processing unit 514, and the facial expression identification preparation device 3 performs these functions. All the components are provided, and the smartphone 5 may acquire and use the state identification result (information thereof) from the facial expression identification preparation device 3. In such an embodiment, the facial expression identification preparation device 3 functions as a state identification device according to the present invention.

  Incidentally, the terminal that receives the facial expression identification result output from the server (facial expression identification preparation device 3) as described above is not limited to a smartphone. For example, it may be a thin client terminal as a device suitable for use in a tablet computer, notebook computer, PC, or IOT (Internet Of Things) environment, and further, STB, signage, robot It is possible to employ various types of terminals.

  As described above in detail, according to the present invention, an image group including images generated by the process of changing the appearance of the identification target and maintaining the state is generated. As a result, it is possible to reliably prepare more images to be identified that have various appearances as compared to the conventional case. As a result, even for an image of an identification target that looks differently under various circumstances, the influence of the situation and the appearance is adjusted, and the state of the identification target can be identified with higher accuracy. Can do it.

  Specifically, according to the present invention, for example, in contrast to the conventional problem that the identification model identification accuracy remains low due to a small number of learning image samples, the number of image samples for learning (and also image patterns). By increasing the number, it is possible to construct an identification model having high identification accuracy (and high robustness to the situation and appearance). In addition, it is possible to increase the number of input images (and the number of image patterns) and improve the identification accuracy (and improve the robustness with respect to the situation and appearance) during state identification using the identification model. It is.

  Incidentally, an application program capable of providing various services can be developed by using the highly accurate facial expression identification result realized based on the present invention. As such an application, for example, by using this facial expression identification result, it is possible to understand the emotion (utterance intention) of the terminal user who is interacting, adjust the response content, or personalize the content of the interaction with the user. An interactive AI application that can be used.

  Various changes, modifications, and omissions of the various embodiments of the present invention described above within the scope of the technical idea and the viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be any limitation. The present invention is limited only by the claims and the equivalents thereof.

1, 5 Smartphone (state identification device)
101, 501 Communication interface unit 102, 502 Camera 103 Image database 104 Changed image storage unit 105 Judgment result storage unit 106, 503 Touch panel display (TP / DP)
111,511 Image management unit 112,312,512 Appearance degree determination unit 113,313,513 Image group generation unit 113a Image change unit 113b Learning image group generation unit 113c Determination image group generation unit 114,314,514 Expression expression identification processing unit 114a, 314a Identification model construction unit 114b, 514b Expression determination unit 121, 521 Application 122, 522 Input / output control unit 2 Image management server 3 Expression identification preparation device

Claims (17)

A state identification device that uses an image related to an identification object and identifies the state of the identification object by an identification model,
An image group including an image generated by the processing is performed on the input image related to the identification target by performing a process of changing the appearance of the identification target in the one image and maintaining the state. Image group generation means for generating;
A state identification apparatus comprising: an identification processing unit that uses the generated image group when the identification model is constructed and / or when the state is identified using the identification model.   2. The image group generation unit performs at least a change in a direction opposite to a direction in the one image with respect to the one image with respect to at least one type related to the appearance. The state identification device described in 1.   The image group generation means at least changes at least one type related to the appearance with respect to the one image in a direction that is preset with respect to the at least one type and that is suitable for identification. The state identification device according to claim 1 or 2.   The said image group production | generation means changes according to the some of several types which concern on the said appearance with respect to the said one image, The any one of Claim 1 to 3 characterized by the above-mentioned. State identification device. The image group generation means performs, for the one image, at least one type relating to the appearance, changing the at least one type in a direction set in advance, and the identification characterized by the direction Generate images to build a model,
5. The state identification device according to claim 1, wherein the identification processing unit constructs the identification model using the generated image group. 6. The image group generation means performs, for the one image, for at least one type related to the appearance, changes in all possible directions for the at least one type, and the direction for the at least one type. Generate a group of images for constructing the identification model in which the variation in the identification results for each is suppressed,
5. The state identification device according to claim 1, wherein the identification processing unit constructs the identification model using the generated image group. 6.   The image group generation means performs, for one image associated with one state as a correct answer, for at least one type related to the appearance, as many changes as the number of changes set in advance for the one state. The state identification apparatus according to claim 1, wherein the image group is generated with a predetermined number of samples.   When the number of samples of the image associated with the one state is smaller than the number of samples of the image associated with the other state, the image group generation unit performs the processing for the image associated with the one state. The state identification device according to claim 7, wherein the number of changes is larger than that of an image associated with the other state. The identification processing means uses the generated image group when constructing the identification model and identifying the state using the identification model,
When the image group generation means generates an image group to be used for identifying the state, the identification model for any type related to the appearance of the identification target for one input image. Change in the same direction as the change for the type made when generating the image group used in the construction, while the change for the type is not made when generating the image group The state identification device according to any one of claims 1 to 8, wherein no change is made in the event of a failure. The identification processing means determines the state of each of the images included in the generated image group using the identification model, and is set in advance depending on whether the image is the original image before the change. The weighted average of the state is determined by the weight and / or the weight set in advance according to the type changed in the image, and the weighted average result is used as the identification result of the state. The state identification device according to any one of claims 1 to 9.   The state identification device according to claim 1, wherein the image group generation unit generates the image group including one image related to the identification target. The state identification device extracts an image area related to the identification target for one input image related to the identification target, and generates the type in at least one type related to the appearance based on the image area A degree determination means for determining the presence and / or degree of
The said image group production | generation means changes by changing the state and / or the said degree of the said presence / absence determined about the said at least 1 classification. State identification device.   The image group generation means includes, as a plurality of types related to the appearance, the degree of the identification object appearing in the one image, the direction or inclination of the identification object appearing, and whether or not there is something that blocks the identification object Alternatively, at least one of the type, the brightness of the identification target, the hue of the identification target, and the degree of contrast between the identification target and the background is changed. The state identification device according to item. The identification target is a face, and the state is a facial expression,
The image group generation means performs a process of changing the appearance of the face in the one image and maintaining the facial expression on the input one image related to the identification target,
The state identification device according to claim 1, wherein the identification processing unit identifies facial expressions in one input image using the identification model. A state identification system for identifying a state of the identification target by an identification model using an image related to the identification target,
An image group including an image generated by the processing is performed on the input image related to the identification target by performing a process of changing the appearance of the identification target in the one image and maintaining the state. Image group generation means for generating;
An identification model construction means for constructing the identification model by the generated image group;
A state identification system comprising: identification processing means for identifying the state by the identification model using the generated image group. A state identification program for causing a computer mounted on a device for identifying a state of the identification target by using an identification model using an image related to the identification target,
An image group including an image generated by the processing is performed on the input image related to the identification target by performing a process of changing the appearance of the identification target in the one image and maintaining the state. Image group generation means for generating;
A state identification program for causing a computer to function as an identification processing unit that uses the generated image group when constructing the identification model and / or identifying the state using the identification model . A state identification method implemented in a computer mounted on an apparatus for identifying a state of the identification target by using an identification model using an image related to the identification target,
An image group including an image generated by the processing is performed on the input image related to the identification target by performing a process of changing the appearance of the identification target in the one image and maintaining the state. Generating step;
Using the generated image group when the identification model is constructed and / or when the identification model is used to identify the state.