JP6710817B2 - Smile infection determination system, smile infection determination method and program - Google Patents

Description

The present invention relates to a technique for determining that a human smile is contagious, and is used in the field of IoT (internet of things) and the like.

In recent years, the effects of human smiles have been drawing attention, and various technologies related to smiles have been utilized. For example, a system has been proposed that automatically activates various target devices on the condition that an operator's smile is detected (see Patent Document 1).

JP, 2017-14806, A

The technique described in Patent Document 1 only detects whether or not a certain person has a smiling face, and does not analyze the influence of a smiling face between humans. Therefore, an object of the present invention is to provide a technique capable of knowing whether or not a smiling face has been transmitted to a third party.

Means for solving problems

The present invention relates to an acquisition unit that acquires an image captured by a camera, and a smile determination unit that analyzes the image and determines whether the first person and the second person shown in the camera are smiling faces. And when the second person smiles after the line of sight of the first person is directed to the second person, or the second person with respect to the first person who is smiling When the second person smiles after the line of sight of the person is directed, a contagion judging means for judging that a smile is transmitted from the first person to the second person is provided, and the image is A first image captured by a first camera and a second image captured by a second camera provided at a position different from the first camera are included, and the acquisition unit includes: The first image and the second image, and the installation position, the imaging direction, and the imaging range of each camera are acquired, and the infection determination unit determines the installation position, the imaging direction, and the imaging range of each camera, and The positional relationship between the first person and the second person is determined based on the position of the first person in the first image and the position of the second person in the second image. A smile infection determination system that identifies and determines that a smile has been transmitted from the first person to the second person based on the positional relationship.
The present invention comprises the steps of obtaining an image captured by a camera, analyzing the image, and determining whether the first person and the second person shown in the camera are smiling faces, When the second person smiles after the first person's line of sight is directed to the second person, or the second person's line of sight is changed with respect to the first person who is smiling. Determining that the smile has been transmitted from the first person to the second person when the second person smiles after being directed. The first image captured by the camera and the second image captured by the second camera provided at a position different from the first camera are included, and in the step of acquiring, the first image Image, the second image, and the installation position, the imaging direction, and the imaging range for each camera, and determining that the infection has occurred, the installation position, the imaging direction, and the imaging range of each camera, and The positional relationship between the first person and the second person is determined based on the position of the first person in the first image and the position of the second person in the second image. A smile infection determination method for identifying and determining that a smile has been transmitted from the first person to the second person based on the positional relationship.
According to the present invention, a computer obtains an image captured by a camera, analyzes the image, and determines whether the first person and the second person shown in the camera are smiling. And when the second person smiles after the line of sight of the first person is directed to the second person, or when the second person is smiling with respect to the first person When the second person smiles after the line of sight of the person is directed, a step of determining that a smile is transmitted from the first person to the second person is executed, and the image includes In the step of acquiring, the first image captured by the first camera and the second image captured by the second camera provided at a position different from the first camera are included, In the step of acquiring the first image and the second image, the installation position, the imaging direction, and the imaging range for each camera, and determining that the infection has occurred, the installation position, the imaging direction, and the imaging range of each camera Of the first person and the second person based on the position of the first person in the first image and the position of the second person in the second image. A program for identifying a positional relationship and determining that a smile has been transmitted from the first person to the second person based on the positional relationship.
The present invention, an acquisition unit that acquires an image of a person, a determination unit that analyzes the image to determine whether the person is a smile, and determines whether the smile has been transmitted to a third party Provided is a smile infection determination system including an infection determination means.

The infection determination means may determine that the person has been infected in consideration of a time relationship in which the certain person and the third person smile.

The infection determination means may determine that the infection has occurred in consideration of the positional relationship between the certain person and the third party.

The infection determination means may determine that the infection has occurred in consideration of the behavior of the certain person or the third party.

If it is determined that the infection has occurred, a specifying unit that specifies the certain person as a smiling influencer may be provided.

The present invention also includes an acquisition step of acquiring an image of a certain person, a determination step of analyzing the image to determine whether the certain person has a smile, and whether the smile has been transmitted to a third party. There is provided a smile infection determination method comprising a determination infection determination step.

Further, the present invention provides a computer with an acquisition step of acquiring an image of a person, a determination step of analyzing the image to determine whether the person is a smile, and the smile transmitted to a third party. Provided is a program for executing an infection determination step of determining whether or not.

According to the present invention, it is possible to know whether or not a smile has been transmitted to a third party.

The figure which shows the outline of the smile infection determination system 1. The hardware block diagram of the user terminal 300. 3 is a hardware block diagram of the server device 200. FIG. 3 is a functional block diagram of the server device 200. FIG. The figure which shows the operation example of the smile infection determination system 1.

100-1 to 100-3... Camera, 200... Server device, 201... CPU, 202... ROM, 203... RAM, 204... Auxiliary storage device, 205... Communication IF, 21... Acquisition unit, 22... Smile determination unit, 23... Infection determination unit, 24... Specification unit, 300... User terminal, 301... CPU, 302... ROM , 303... RAM, 304... Auxiliary storage device, 305... Communication IF, 306... Display unit, 307... Operation unit, 900... Network

[Constitution]
FIG. 1 is a block diagram showing an outline of a smile infection determination system 1. The smiling face infection determination system 1 includes a camera 100, a server device 200, and a user terminal 300. The camera 100, the server device 200, and the user terminal 300 are communicably connected via a network 900.

The camera 100 includes a camera 100-1, a camera 100-2, and a camera 100-3. Each camera 100 is a plurality of network cameras provided at different positions, and has a function of transmitting image data indicating an image pickup position, an image pickup date and time, and an image picked up to the server device 200. The installation position of the camera 100 is, for example, a station, a commercial facility, a road, or a public facility. The number of cameras 100 shown in the figure is an example, and may be one or more. The cameras 100 may or may not have the same use (security purpose, fixed-point observation purpose, public, commercial), function, and performance (image quality, zoom function, pan function, data transfer capability, etc.). Good. The camera 100 does not have to be fixed, and may be mounted on a movable terminal. For example, when the user terminal 200 is a device with a camera function, the user terminal 200 may have the function of the camera 100. Imaging and transmission of image data are preferably performed at fixed time intervals or at preset timings. Alternatively, when a predetermined event (for example, a request from the server device 200 or a sensor included in the camera 100 detects predetermined information), imaging and/or transmission of image data is executed. Good.

FIG. 2 is a block diagram illustrating a hardware configuration of the user terminal 300. The user terminal 300 may be, for example, a portable computer such as a smartphone, a tablet, or a camera type, as well as a wearable computer such as a glass type, a head mounted display type or a contact lens type, or a desktop type or a notebook type personal computer. .. The user terminal 300 includes a CPU 301 (Central Processing Unit), a ROM (Read Only Memory) 303, a RAM (Random Access Memory) 303, an auxiliary storage device 304, a communication IF 305, a display unit 306, and an operation unit 307.

The CPU 301 is a processor that performs various calculations. The ROM 302 is, for example, a non-volatile memory that stores programs and data used to boot the user terminal 30. The RAM 303 is a volatile memory that functions as a work area when the CPU 301 executes a program. The auxiliary storage device 304 is a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores programs and data used in the user terminal 300. The communication IF 305 is an interface for performing communication via the network 900 according to a predetermined communication standard. This communication standard may be a wireless communication standard or a wired communication standard. In the case of wireless communication, the communication IF 305 includes an antenna and a communication circuit that operate according to a communication standard such as LTE (Long Term Evolution) or Wi-Fi (registered trademark). The display unit 306 includes, for example, a liquid crystal panel and a liquid crystal drive circuit, and displays an image according to image data. The operation unit 307 includes, for example, operators such as keys and touch sensors, receives a user operation, and supplies a signal corresponding to the operation to the CPU 301. The user terminal 300 may include other configurations such as a GPS (Global Positioning System) unit and a voice input/output unit in addition to the configuration illustrated in FIG.

FIG. 3 is a block diagram illustrating a hardware configuration of the server device 200 as the smile infection determination device. The server device 200 is a computer device having a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, and a communication IF 205. The CPU 201 is a processor that performs various calculations. The ROM 202 is a non-volatile memory that stores, for example, a program and data used to start the server device 200. The RAM 203 is a volatile memory that functions as a work area when the CPU 201 executes a program. The auxiliary storage device 204 is a non-volatile storage device such as an HDD or SSD, and stores programs and data used in the server device 200. The functions shown in FIG. 4 described later are realized by the CPU 201 executing this program. The communication IF 205 is an interface for performing communication via the network 900 according to a predetermined communication standard. The server device 200 may include other configurations such as a display unit and an operation unit in addition to the configuration illustrated in FIG.

FIG. 4 is a block diagram illustrating the functional configuration of the server device 200. The server device 200 includes an acquisition unit 21, a smile determination unit 22, an infection determination unit 23, and an identification unit 24. The acquisition unit 21 has a function of acquiring a captured image including a person, and acquires an image of a certain person.

The smile determination unit 22 has a function of analyzing a captured image to determine a smiling person, and determines whether the person who analyzes the image has a smile. This image may be a still image or a moving image.

The infection determination unit 23 has a function of determining whether or not the smile of the first person has affected the smile of the second person when the smiles of the plurality of persons are detected. Determine if it was transmitted to a third party. Here, the infection determination unit 23 makes the above determination based on the relationship between the times when the first person and the second person smile, that is, the time relationship when a person and a third person smile. To do. Further, the infection determination unit 23 makes the above determination based on the relationship between the positions where the first person and the second person are smiling, that is, the positional relationship between the certain person and a third person.

The identifying unit 24 has a function of identifying the first person who has influenced the smile of the second person as a smile influencer and storing information about the influencer. If it is determined that the smile is contagious, identify a person as the smile influencer.

[motion]
FIG. 5 shows an operation example of the smile infection determination system 1. Each camera 100 transmits image data indicating a captured image (here, a moving image) to the server device 200. The acquisition unit 21 acquires these image data (step S1).

Next, the smile determination unit 22 performs image analysis on the image data of any one of the cameras 100 and compares the image with a predetermined feature point of the face of the person, so that a plurality of images included in the image are included. The person's face is recognized (step S2).

Next, the smile determination unit 22 performs time-series image analysis of the image data and determines whether or not a person (first person) has a smile among a plurality of persons included in the image ( Step S3). This determination is performed by collating the temporal change of the captured image with a predetermined temporal change of the feature points when the face of the person becomes a smile.

After determining that a certain person (first person) is smiling (step S3; YES), the smile determining unit 22 performs image analysis on the image data, and selects one of a plurality of persons included in the image. The (first person) determines whether or not a different third person (second person) smiles (step S4). When it is not determined that the third person (second person) is smiling before a predetermined period elapses after it is determined that a person (first person) is smiling (step S8; YES). ), the processing for the camera 100 that has been the processing target ends, and the processing shifts to the processing for the image data of the next camera 100.

On the other hand, if it is determined that a third person (second person) is smiling by a predetermined period after it is determined that a person (first person) is smiling (step (S4; YES), the infection determination unit 23 determines the relationship between the time and position when a person (first person) smiles and the time and position when a third person (second person) smiles ( By determining whether or not the time condition and the position condition) satisfy a predetermined smile contagion condition (step S5), the smile of one person (first person) affects a third person (second person). It is then determined that the third person (second person) has a smile, that is, that a smile has been transmitted (step S6).

The time condition used here is a condition that the difference between the time when a person (first person) smiles and the time when a third person (second person) smiles is less than a threshold value. .. Here, the time when a person (first person) smiles may be the beginning of a smile when a person (first person) begins to smile, or a person (first person) may smile. ) May be an arbitrary period of a smiling face, or may be the end of a smiling face when a person (first person) has finished smiling. On the other hand, the time when the third person (second person) smiles is preferably the beginning of the smile when the third person (second person) begins to smile.

The position condition used here is that the distance between the position where a person (first person) smiles and the position where a third person (second person) smiles is less than a threshold value. Is. Furthermore, in consideration of the face orientations of a person (first person) and a third person (second person), the third person (second person) smiles at the position where the third person (second person) smiles. A position condition that the person (first person) can see a smile from the second person) may be used. Even if the person (first person) cannot see the smile from the third person (second person) at the position where the third person (second person) smiles, At a position where a laughing voice can be heard when a person (first person) smiles, the smile of the person (first person) affects a third person (second person), It is also possible to determine that the three persons (second person) have smiled.

When it is determined that the smile of a certain person (first person) is transmitted to a third party (second person) (step S6), the identifying unit 24 makes the person (first person) smile. It is identified as an influencer, and information regarding the influencer is stored (step S7). The influencer information stored here includes, for example, the shooting date and time of the camera 100 and the installation position of the camera 100. Furthermore, when a person can be identified by face recognition technology, the identification information (name and ID) of a person (first person) may be included in the influencer information, and each person is determined to be an influencer. The number of times performed and the period may be accumulated and stored. These pieces of information are transmitted from the server device 200 to the user terminal 300 in response to a request from the user terminal 300, and are displayed on the display unit 306. This makes it possible to confirm whether or not the smile is transmitted, and the frequency of the transmission. Further, when the person can be identified by the face recognition technology, it is possible to know who is determined to be the smiling influencer.

By performing the above processing on the images captured by all the cameras 100 by the server device 200, it is possible to determine whether or not the smile has been transmitted to a third party.

[Modification]
The present invention is not limited to the above-mentioned embodiment, and various modifications can be made. Hereinafter, some modified examples will be described. Two or more of the following modifications may be used in combination.

[Modification 1]
The infection determination unit 23 may determine that the smiling face has been transmitted based on the action of a person (first person) or a third person (second person). For example, the infection determination unit 23 identifies the line of sight of a person (first person) or a third person (second person) by using image analysis related to eye gaze detection, and a person who smiles (first person). When the third person (second person) smiles after the line of sight of the person) turns to the third person (second person), or a person who smiles (first person) On the other hand, if the third person (second person) smiles after the third person's (second person)'s line of sight turns, it may be determined that the smile is contagious. In addition, using the infection determination unit 23, a technique of identifying a person's behavior by image analysis, a gesture of a person (first person) or a third person (second person) (for example, looking into the other person's face, , An action of pointing a finger to the other party, a gesture that attracts the other person's attention, etc.) and a gesture of a person (first person) who smiles is performed to a third party (second person). After a third person (second person) smiles, or after a third person (second person) gestures to a smiling person (first person) When a third person (second person) smiles, it may be determined that the smile is contagious.

[Modification 2]
In the embodiment, the server device 200 determines whether or not a smile is transmitted between a plurality of persons included in an image captured by one camera 100. It may be possible to determine whether or not a smile is transmitted. In this case, each camera 100, when transmitting the image data indicating the captured image to the server device 200, adds data indicating the image capturing direction and the image capturing range in addition to its own position (coordinates) and transmits the image data. The acquisition unit 21 acquires these image data and additional data. Note that the data indicating the position (coordinates), the imaging direction, and the imaging range of each camera 100 is held in advance by the server device 200 in association with the camera ID, and each camera 100 converts the image data indicating the captured image into image data. It is also possible to add the camera ID of itself and transmit it. Even in this case, the server device 200 can specify the image capturing direction and the image capturing range as well as the position (coordinates) of the camera 100 capturing the image data.

Next, the smile determination unit 22 analyzes the image data, and determines whether the person included in the image is a smile. At this time, the smile determination unit 22 determines the actual position of the smiling person based on the correlation between the data indicating the position of each camera 100, the imaging direction, and the imaging range and the position of the person in the captured image ( Coordinates). The smile determination unit 22 uses the actual position (coordinates) of each person as described above and then determines that the person (first person) smiles, and then determines the predetermined value of the person (first person). It is determined whether or not a third party (second person) smiles within the range of the distance. The infection determination unit 23 determines the relationship between the time and position at which a person (first person) smiles and the time and position at which a third person (second person) smiles (time condition and position condition). ) Satisfies a predetermined smile contagion condition, it is determined that a smile has been transmitted from a person (first person) to a third person (second person).

[Modification 3]
Relationship between the time and position when a person (first person) smiles and the time and position when a third person (second person) smiles, which is used for judging the transmission of a smile (time condition and position condition) ) May vary depending on some condition. For example, the time condition or the position condition may be relaxed when a certain person (first person) has a high degree of smile (when the face is greatly distorted and laughed). Specifically, when the smile level of a person (first person) is greater than or equal to a reference value, the time when a person (first person) smiles and when The threshold value of the difference from the time when the three persons (second person) smiled may be increased. When the smile degree of a person (first person) is greater than or equal to a reference value, the position at which the person (first person) smiles and a third person The threshold value of the distance from the position where the second person) smiles may be increased.

[Modification 4]
The timing at which the server device 200 acquires an image from the camera 100 and the timing at which the server device 200 executes image analysis are arbitrary. For example, each camera 100 may not voluntarily transmit image data to the server device 200, and the server device 200 may request the image data from the camera 100 in response to a request from the user terminal 300. Alternatively, the server device 200 may execute the image analysis according to a predetermined schedule regardless of the timing of the request from the user. The acquisition of the captured image, the image analysis, the smile determination, and the infection determination may be performed by a plurality of nodes or may be performed by one node. For example, the user terminal 300 may have the function of the server device 200.

The present invention may be provided as an information processing method including the steps of processing performed in the server device 200 or the smile infection determination system 1. That is, the present invention includes an acquisition step of acquiring an image of a person, a smile determination step of analyzing the image to determine whether the person is a smile, and whether the smile has been transmitted to a third party. And a contagion judging step for judging whether the person is smiling or not. Further, the present invention may be provided as a program executed in the server device 200 or the user terminal 320. Such a program may be provided in the form of being recorded in a recording medium such as an optical disk, or may be provided in such a form that it is downloaded to a computer via a network such as the Internet and is installed and usable. It is possible. Note that the order of the processing procedures described in this specification may be changed as long as there is no contradiction. For example, the methods described herein present elements of the various steps in a sample order, and are not limited to the presented order of acquisitions.

It should be noted that any reference to elements using the designations "first", "second", etc., as used herein, does not generally limit the amount or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be employed there or that the first element must precede the second element in any way.

Claims (4)

An acquisition means for acquiring an image taken by the camera,
Smile judging means for judging whether the first person and the second person shown in the camera are smiling by analyzing the image;
When the second person smiles after the first person's line of sight is directed to the second person, or the second person's line of sight with respect to the first person who is smiling And a contagion judging means for judging that a smile is transmitted from the first person to the second person when the second person smiles after being pointed at,
The image includes a first image captured by a first camera and a second image captured by a second camera provided at a position different from the first camera,
The acquisition unit acquires the first image and the second image, and the installation position, the imaging direction, and the imaging range for each camera,
The infection determination means includes an installation position of each camera, an imaging direction and an imaging range, a position of the first person in the first image, and a position of the second person in the second image. Based on the positional relationship between the first person and the second person, and it is determined that a smile is transmitted from the first person to the second person based on the positional relationship. Judgment system. The smile contagion according to claim 1, further comprising a specifying unit that specifies the first person as a smile influencer when it is determined that a smile has been transmitted from the first person to the second person. Judgment system. A step of acquiring an image taken by a camera,
Analyzing the image to determine whether the first person and the second person in the camera are smiling;
When the second person smiles after the first person's line of sight is directed to the second person, or the second person's line of sight with respect to the first person who is smiling When the second person smiles after being pointed at, the step of determining that a smile is transmitted from the first person to the second person,
The image includes a first image captured by a first camera and a second image captured by a second camera provided at a position different from the first camera,
In the step of acquiring, the installation position, the imaging direction, and the imaging range are acquired for each of the first image and the second image, and each camera,
In the step of determining that the infection has occurred, the installation position of each camera, the imaging direction and the imaging range, the position of the first person in the first image, and the second person in the second image. Based on the position, the positional relationship between the first person and the second person is specified, and it is determined that a smile is transmitted from the first person to the second person based on the positional relationship. Yes Smile contagion judgment method. On the computer,
A step of acquiring an image taken by a camera,
Analyzing the image to determine whether the first person and the second person in the camera are smiling;
When the second person smiles after the first person's line of sight is directed to the second person, or the second person's line of sight with respect to the first person who is smiling And a step of determining that a smile has been transmitted from the first person to the second person when the second person smiles after being pointed at
The image includes a first image captured by a first camera and a second image captured by a second camera provided at a position different from the first camera,
In the step of acquiring, the installation position, the imaging direction, and the imaging range are acquired for each of the first image and the second image, and each camera,
In the step of determining that the infection has occurred, the installation position of each camera, the imaging direction and the imaging range, the position of the first person in the first image, and the second person in the second image. Based on the position, the positional relationship between the first person and the second person is specified, and it is determined that a smile is transmitted from the first person to the second person based on the positional relationship. Program to do.

Images