JP7649934B2 - Behavior estimation device, behavior estimation method, program, and behavior estimation system - Google Patents

Description

The present invention relates to a behavior estimation device, a behavior estimation method, a program, and a behavior estimation system for estimating human behavior.

There is a known behavior estimation device that detects the joints of a person captured in a time-series image and estimates the subject's behavior based on changes in the positions of the detected joints (see, for example, Patent Document 1).

Patent No. 6525181

Conventional behavior estimation devices estimate behavior based solely on the movement of a subject, which makes it difficult to distinguish between behaviors with similar movements.

Therefore, the present invention has been made in consideration of these points, and aims to improve the accuracy of estimating human behavior.

The behavior estimation device according to the first aspect of the present invention includes an image acquisition unit that acquires captured image data showing a captured image, a person detection unit that detects the movement of a person included in the captured image, a subject detection unit that detects the type of subject included in the captured image, a relationship identification unit that identifies a relative relationship between the person and the subject associated with the movement of the person, and an estimation unit that estimates the behavior of the person based on a combination of the type of the subject and the relative relationship.

The relationship identification unit may identify the distance between the person and the subject as the relative relationship, and the estimation unit may estimate the activity content based on a result of comparing a reference distance associated with the type of the subject detected by the subject detection unit with the distance between the person and the subject identified by the relationship identification unit.

The relationship identification unit may identify the relationship between the direction of the person's gaze and the position of the subject as the relative relationship, and the estimation unit may estimate the behavior content based on a result of comparing a reference gaze relationship, which is the relationship between the direction of the gaze and the position of the subject associated with the type of the subject detected by the subject detection unit, with the relationship between the direction of the person's gaze and the position of the subject identified by the relationship identification unit.

The relationship identification unit may identify the relationship between the person's body orientation and the position of the subject as the relative relationship, and the estimation unit may estimate the behavior content based on a result of comparing a reference body orientation relationship, which is the relationship between the body orientation and the position of the subject associated with the type of the subject detected by the subject detection unit, with the relationship between the direction of the person's line of sight identified by the relationship identification unit and the position of the subject.

The image acquisition unit may acquire multiple captured image data created in time series, the relationship identification unit may identify a change in the relative relationship, and the estimation unit may estimate the behavior of the person based on a combination of the type of the subject and the change in the relative relationship.

The estimation unit may estimate the behavioral content further based on a result of determining whether the subject was included in any of the plurality of captured image data before the person detection unit detected the person.

The person detection unit may further detect attributes of the person, and the estimation unit may estimate the activity content based on the attributes of the person detected by the person detection unit.

The subject detection unit detects a first subject and a second subject included in the captured image, the relationship identification unit identifies a first relative relationship that is a relationship between the person and the first subject associated with the person's movement, a second relative relationship that is a relationship between the person and the second subject associated with the person's movement, and a third relative relationship that is a relationship between the first subject and the second subject associated with the person's movement, and the estimation unit may estimate the behavior of the person based on a combination of the type of the subject and at least two of the relative relationships, the first relative relationship, the second relative relationship, or the third relative relationship.

The apparatus may further include a setting reception unit that receives settings of attributes of the subject included in the captured image, and the estimation unit may estimate the action content based on the attributes of the subject.

The relationship identification unit may further identify a positional relationship between the subject detected by the subject detection unit and the subject for which the setting reception unit has accepted an attribute setting, and the estimation unit may further estimate the action content based on the positional relationship identified by the relationship identification unit and the attribute of the subject for which the setting reception unit has accepted an attribute setting.

The device may further include a location identification unit that identifies a location shown in the captured image, and an output unit that outputs notification information when the activity content estimated by the estimation unit and the location identified by the location identification unit satisfy a predetermined condition.

The output unit may output the notification information of a type corresponding to a combination of the location identified by the location identification unit and the action content.

The device may further include a color identification unit that identifies at least one of the color changes of the person or the color changes of the subject, and the estimation unit may estimate the behavioral content based on at least one of the color changes of the person or the color changes of the subject identified by the color identification unit.

The image acquisition unit may acquire a plurality of the captured image data, the relationship identification unit may identify a plurality of the relative relationships, and the estimation unit may estimate the behavior of the person based on a combination of each of the plurality of relative relationships and the type of the subject.

The behavior estimation method according to the second aspect of the present invention includes the steps of acquiring captured image data representing a captured image, executed by a computer, detecting the movement of a person included in the captured image, detecting the type of subject included in the captured image, identifying a relative relationship between the person and the subject associated with the movement of the person, and estimating the behavior of the person based on a combination of the type of subject and the relative relationship.

The program according to the third aspect of the present invention causes a computer to execute the steps of acquiring captured image data representing a captured image, detecting the movement of a person included in the captured image, detecting the type of subject included in the captured image, identifying a relative relationship between the person and the subject associated with the person's movement, and estimating the behavior of the person based on a combination of the type of subject and the relative relationship.

The behavior estimation system according to the fourth aspect of the present invention includes a behavior estimation device that estimates a person's behavior based on an image generated by an imaging device, and an information terminal that displays the person's behavior estimated by the behavior estimation device. The behavior estimation device includes an image acquisition unit that acquires captured image data showing the captured image, a person detection unit that detects the movement of a person included in the captured image, a subject detection unit that detects the type of subject included in the captured image, a relationship identification unit that identifies a relative relationship between the person and the subject associated with the movement of the person, an estimation unit that estimates the behavior of the person based on a combination of the type of subject and the relative relationship, and a transmission unit that transmits behavior content information indicating the behavior estimated by the estimation unit to the information terminal. The information terminal includes an information receiving unit that receives the behavior content information, and a display unit that displays the behavior based on the behavior content information received by the information receiving unit.

The present invention has the effect of improving the accuracy of estimating human behavior.

FIG. 1 is a diagram for explaining an overview of a behavior inference system S according to an embodiment of the present invention. FIG. 2 is a diagram for explaining the configuration of a behavior inference system S. 2 is a diagram for explaining a configuration of an information terminal 2. FIG. FIG. 1 is a diagram for explaining a configuration of a behavior inference device 10. FIG. 4 is a diagram illustrating an example of an action content database. 11 is a flowchart showing an example of an operation of the behavior estimation device 10 to estimate the content of a person's behavior. FIG. 1 is a diagram showing a configuration of a behavior inference device 10 that infers human behavior content based on a change in color of a person or a subject.

[Outline of behavior estimation system S]
Fig. 1 is a diagram for explaining an overview of a behavior estimation system S according to this embodiment. Fig. 2 is a diagram showing the configuration of the behavior estimation system S. The behavior estimation system S includes an imaging device 1, an information terminal 2, and a behavior estimation device 10. As shown in Fig. 2, the imaging device 1, the information terminal 2, and the behavior estimation device 10 are connected via a network N. The network N is, for example, an intranet or the Internet. The network N may include a wireless network.

Figure 1(a) is a diagram of space R seen from above, and Figure 1(b) is a diagram of space R seen from the side. Space R is a space in which objects are placed, such as a store sales floor, a store warehouse, a factory warehouse, or a facility equipment storage area. As shown in Figure 1, multiple shelves T (T1 to T4) are installed in space R. As shown in Figure 1(b), multiple objects P (P1 to P3) are placed on each of the multiple shelves T (T1 to T3). As shown in Figure 1, a person M is staying in space R. Person M is carrying a bag B that can hold object P.

In FIG. 1, the behavior estimation system S includes a behavior estimation device 10 that estimates the behavior of a person M based on captured images generated by one or more imaging devices 1 installed in a space R. The imaging device 1 is, for example, a surveillance camera, and captures an area in which the behavior of the person is to be estimated. The imaging device 1 transmits captured image data generated by capturing the image to the behavior estimation device 10. The imaging device 1 may generate video data as the captured image data. The video data is data encoded using a compression standard such as H.264 or H.265.

The behavior estimation device 10 estimates the behavior of person M based on the captured image generated by the imaging device 1, and transmits behavior content information indicating the estimated behavior to the information terminal 2. The behavior estimation device 10 is an information processing device, such as a computer, that estimates the behavior of person M based on the relative relationship between person M and a subject included in the captured image. The subject is, for example, multiple shelves T, multiple objects P, a bag B, or another person. The behavior estimation device 10 may be a single information processing device, or may be composed of multiple information processing devices, such as a cloud-type server.

The behavior estimation device 10 identifies a relative relationship between person M and a subject, and estimates the behavior of person M based on a combination of the identified relative relationship and the type of subject. The relative relationship is, for example, the distance between person M and object P, the relationship between the direction of person M's line of sight and the position of object P, or the relationship between the orientation of person M's body and the position of object P. The type of subject is represented by information that can identify the subject based on at least one of its shape or color, such as a canned drink, a knife, or a shopping basket. In this embodiment, for ease of explanation, the type of subject may be represented by a product name. The subject is an object or a person.

When person M picks up object P2 and then puts object P2 into bag B, the behavior estimation device 10 determines the relative relationship as "person M's hand, which is in contact with object P2, is in close proximity to bag B" based on the distance between person M's hand and the subject. Next, when the type of detected object P2 is a "canned drink" and the type of detected bag B is a "handbag," the behavior estimation system S determines that "person M put the canned drink into the handbag" based on the combination of the determined relative relationship and the subject types of canned drink and handbag, thereby inferring that person M's behavior is shoplifting.

On the other hand, even if the identified relative relationship is the same, if the type of the detected object P2 is a "canned drink" and the type of the detected bag B is a "shopping basket," the behavior estimation device 10 will infer that the behavior of person M is shopping by identifying that "person M put a canned drink into the shopping basket" based on the combination of the relative relationship and the subject types of canned drink and shopping basket.

The behavior estimation device 10 transmits the estimation result to the information terminal 2. The information terminal 2 is, for example, a smartphone, a tablet, or a personal computer. The information terminal 2 is, for example, a terminal used by a sales clerk who monitors a sales floor in a store or an employee who manages a warehouse in a factory (hereinafter referred to as the "manager").

The information terminal 2 displays on a display the behavioral content of the person estimated by the behavior estimation device 10. The information terminal 2 may, for example, acquire behavioral content information including the behavioral content of the person estimated by the behavior estimation device 10 and a captured image in which the person appears, and display the captured image included in the behavioral content information in association with the behavioral content of the person included in the captured image.

In this way, the behavior estimation device 10 can accurately distinguish behaviors with similar movements by estimating the behavioral content of person M based on a combination of the relative relationship between person M and the subject and the type of subject. Then, the behavior estimation device 10 notifies the information terminal 2 that a preset behavior (e.g., shoplifting) has been detected, allowing the administrator using the information terminal 2 to accurately grasp that a preset behavior has been performed.

The administrator of the information terminal 2 can set various pieces of information that the behavior estimation device 10 uses to estimate the behavior of a person. In response to the administrator's operation, the information terminal 2 transmits setting information, which is information used to estimate the behavior of a person, to the behavior estimation device 10.

The setting information includes, for example, at least one of subject attribute information or subject type information. Subject attribute information is information that associates the subject's position or shape with the subject's attributes. The subject's attributes are information that indicates the subject's characteristics, such as the subject's owner or the subject's function. For example, the attribute information of shelf T1 shown in FIG. 1 is information that associates the shelf's position or shape with "store", which indicates the attribute of shelf T1.

The subject type information is information that associates the shape of the subject with the type of the subject. For example, if the subject is a canned drink, the subject type information is information that associates "canned drink," which indicates the type of the subject, with the shape of the canned drink.

[Configuration of information terminal 2]
3 is a diagram for explaining the configuration of the information terminal 2. The information terminal 2 has a communication unit 21, a storage unit 22, a control unit 23, a display unit 24, and an operation unit 25. The control unit 23 has an information transmission unit 231 and an information reception unit 232.

The communication unit 21 includes a communication device for transmitting and receiving information via the network N. The communication device is, for example, a LAN (Local Area Network) controller or a wireless LAN controller.

The storage unit 22 has storage media such as a read only memory (ROM), a random access memory (RAM), and a solid state drive (SSD). The storage unit 22 stores the programs executed by the control unit 23.

The control unit 23 is, for example, a CPU (Central Processing Unit). The control unit 23 functions as an information transmission unit 231 and an information reception unit 232 by executing a program stored in the storage unit 22. The information transmission unit 231 transmits setting information of the behavior estimation device 10 created by an administrator who uses the information terminal 2 to the behavior estimation device 10 via the communication unit 21. The information reception unit 232 receives behavior content information transmitted by the behavior estimation device 10 via the communication unit 21.

The display unit 24 has a display device such as a liquid crystal display or an organic EL (Electro Luminescence). The display unit 24 displays the activity content based on the activity content information received by the information receiving unit 232. The display unit 24 extracts, for example, the activity content of a person included in the activity content information received by the information receiving unit 232 and a captured image in which the person appears. The display unit 24 displays the extracted captured image on the display and also displays the activity content of the person appearing in the captured image in association with the captured image.

The operation unit 25 is a device that accepts operations by the administrator who uses the information terminal 2, and is a keyboard, mouse, or touch panel. The operation unit 25 notifies the information transmission unit 231 of information indicating the content of the administrator's operations.

[Configuration of behavior inference device 10]
4 is a diagram for explaining the configuration of the behavior inference device 10. The behavior inference device 10 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has an image acquisition unit 131, a person detection unit 132, a subject detection unit 133, a setting reception unit 134, a relationship identification unit 135, an estimation unit 136, a location identification unit 137, and an output unit 138.

The communication unit 11 includes a communication device for transmitting and receiving information via the network N. The communication device is, for example, a LAN controller or a wireless LAN controller. The storage unit 12 has storage media such as ROM, RAM, and SSD. The storage unit 12 stores a program executed by the control unit 13. The storage unit 12 stores setting information acquired from the information terminal 2.

The memory unit 12 stores various information in a database that is used by the estimation unit 136 to estimate the content of a person's activity. The memory unit 12 stores, for example, an activity content database in which the content of a person's activity, the movement of the person, the type of subject, and the relative relationship between the person and the subject are associated with each other. The subject is an object or a person.

Figure 5 is a diagram showing an example of an activity content database. In Figure 5, categories of human activity content, activity content, and multiple requirements for identifying the activity content are associated with each other. The multiple requirements are the attributes of the person whose activity is to be estimated, the movement of the person, multiple types of subjects ("Subject 1" and "Subject 2" shown in Figure 5), and the relative relationship between the person and the subjects. The attributes of a person indicate the position or role of a person in the space in which the human activity is to be estimated, such as a shopper or an employee.

For example, the activity content "looking" is associated with the requirements that the person's attribute is "shopper", the person's movement is "gazing", the subject 1 is "product", and the relative relationship is "the distance between the person and subject 1 is less than distance A1" and "the directional difference between the person's line of sight and subject 1 is less than directional difference A2". Distance A1 is, for example, the maximum distance at which a person can come into contact with the subject. Direction difference A2 is, for example, the direction of the person's line of sight that is thought to be gazing at the subject, which is the farthest from the subject. Details will be described later, but the estimation unit 136 identifies the activity content of the subject included in the captured image by identifying the activity content that corresponds to the combination of requirements included in the activity content database.

Returning to FIG. 4, the control unit 13 is, for example, a CPU. The control unit 13 executes the programs stored in the memory unit 12 to function as an image acquisition unit 131, a person detection unit 132, a subject detection unit 133, a setting reception unit 134, a relationship identification unit 135, an estimation unit 136, a location identification unit 137, and an output unit 138.

The image acquisition unit 131 acquires captured image data indicating a captured image. The image acquisition unit 131 acquires captured image data generated by the imaging device 1 via the communication unit 11. The image acquisition unit 131 may acquire multiple captured image data created in chronological order.

The person detection unit 132 detects the movement of a person included in the captured image. The movement of the person is, for example, the movement as described in the "Personal Movement" column shown in FIG. 5. The person detection unit 132 detects the position of a person included in the captured image, and further detects the movement of the person whose position has been detected. The person detection unit 132 detects the movement of a person included in the captured image, for example, by identifying the positions of the joints of the person included in the captured image and changes in the direction of the person's line of sight. The person detection unit 132 outputs person detection information indicating the detected movement of the person to the estimation unit 136.

The person detection unit 132 may further detect attributes of a person. For example, the person detection unit 132 detects attributes of a person by determining whether or not the person is an employee based on the pattern or color of the work clothes or uniform worn by the employee. The person detection unit 132 may detect attributes of a person by determining whether or not the person is an employee based on the results of comparing the features of the employee, such as the contour and shape of the employee's face, stored in the memory unit 12 with the features of the person included in the captured image.

The subject detection unit 133 detects the type of subject included in the captured image. The subject detection unit 133 detects the position of the subject included in the captured image, and further detects the type of the subject whose position has been detected. The subject detection unit 133 detects the subject included in the captured image based on the shape of the subject included in the setting information stored in the storage unit 12, for example, and detects the type of subject included in the captured image based on the type of subject associated with the shape of the subject. The subject detection unit 133 outputs subject detection information indicating the position of the detected subject and the type of the subject to the estimation unit 136.

The subject detection unit 133 may detect multiple subjects included in the captured image. The subject detection unit 133 may detect a first subject and a second subject included in the captured image. The first subject and the second subject are, for example, items of the types listed in the "Subject 1" and "Subject 2" columns shown in FIG. 5. When the subject detection unit 133 detects multiple subjects, the estimation unit 136 estimates the behavior of a person based on the type of one or more of the detected subjects.

The setting reception unit 134 receives settings for the attributes of the subject included in the captured image. The setting reception unit 134 receives setting information including subject attribute information and subject type information transmitted by the information terminal 2, for example, to receive settings for the attributes of the subject included in the captured image. The setting reception unit 134 may store the setting information including the received settings for the subject attributes in the storage unit 12, or may output the setting information to the subject detection unit 133.

The relationship specification unit 135 specifies a relative relationship, which is a relationship between a person and a subject associated with the person's movement. The relationship specification unit 135 specifies the relative relationship based on the person detection information output by the person detection unit 132 and the subject detection information output by the subject detection unit 133. For example, the relationship specification unit 135 specifies the relative relationship between a part corresponding to the person's movement detected by the person detection unit 132 and the subject. For example, if the person's movement detected by the person detection unit 132 is "gazing", the relationship specification unit 135 specifies the relationship between the person's eyes and the subject, and if the person's movement is "taking", the relationship specification unit 135 specifies the relationship between the person's hands and the subject. The relationship specification unit 135 notifies the estimation unit 136 of the specified relative relationship. The relative relationship is used by the estimation unit 136 to specify the content of the person's action. For example, the relationship specification unit 135 specifies the distance between the person and the subject as the relative relationship. For example, the relationship specification unit 135 specifies the distance between the person and the subject based on the difference between the position of the person's joints included in the person detection information and the position of the subject included in the subject detection information.

The relationship identifying unit 135 may identify the relationship between the direction of a person's gaze and the position of a subject as a relative relationship. For example, when a person's movement is a person's eye movement such as "gazing" or "looking," the relationship identifying unit 135 identifies the relationship between the direction of the person's gaze included in the person detection information and the position of the subject included in the subject detection information as a relative relationship. For example, the relationship identifying unit 135 identifies the difference between the direction of gaze when a person looks directly at a subject and the direction of gaze included in the person detection information as a relative relationship.

The relationship identifying unit 135 may identify the relationship between the orientation of the person's body and the position of the subject as a relative relationship. For example, when the person's movement is "talking", the relationship identifying unit 135 identifies the relationship between the orientation of the person's body identified based on the positions of the person's joints included in the person detection information and the position of the subject included in the subject detection information as a relative relationship. For example, the relationship identifying unit 135 identifies the difference between the orientation of the person's body when facing the subject and the orientation of the body identified based on the positions of the person's joints as a relative relationship.

The relationship identification unit 135 may identify a change in the relative relationship based on the identified multiple relative relationships. The relationship identification unit 135 identifies multiple relative relationships based on multiple movements of a person and a subject detected based on multiple captured images created in time series, for example, and identifies a change in the relative relationship based on each of the identified multiple relative relationships.

For example, if at time T the relationship identification unit 135 identifies the relative relationship as "the distance between the person and the subject is 10 cm," and at time T+1 the relationship identification unit 135 identifies the relative relationship as "the distance between the person and the subject is 50 cm," then based on the relative relationship identified at time T and time T+1, the relationship identification unit 135 identifies that "the person is moving in a direction away from the subject." By identifying the change in the relative relationship in this way, the relationship identification unit 135 can identify the relative relationship with high accuracy.

The relationship identification unit 135 may identify multiple relative relationships when the person whose behavior is to be estimated and multiple subjects are included in the captured image. The relationship identification unit 135 may identify, for example, a first relative relationship that is a relationship between the person and a first subject associated with the movement of the person, a second relative relationship that is a relationship between the person and a second subject associated with the movement of the person, and a third relative relationship that is a relationship between the first subject and the second subject associated with the movement of the person. The first relative relationship is, for example, a relative relationship between the person and subject 1 shown in FIG. 5, the second relative relationship is a relative relationship between the person and subject 2 shown in FIG. 5, and the third relative relationship is a relative relationship between subject 1 and subject 2.

For example, if a customer staying in a sales floor of a store puts a product in a bag, the customer, who is the person estimating the behavior, is acting using the product, which is the first subject, and the bag, which is the second subject. Therefore, the relationship identification unit 135 identifies a first relative relationship, which is the relationship between the customer and the product, a second relative relationship, which is the relationship between the customer and the bag, and a third relative relationship, which is the relationship between the product and the bag. By the relationship identification unit 135 notifying the estimation unit 136 of multiple such relative relationships, the estimation unit 136 can improve the accuracy of identifying the content of a person's behavior.

The relationship identification unit 135 may identify the positional relationship between a subject for which an attribute is set and other subjects among multiple subjects included in the captured image. For example, the relationship identification unit 135 further identifies the positional relationship between a subject detected by the subject detection unit 133 and a subject for which the setting reception unit 134 has received an attribute setting.

For example, an image captured of a sales floor in a store includes images of fixedly installed product shelves and images of products placed on the product shelves. When the purpose is to detect shoplifting, the relationship between the position of the product shelves and the position of the products is effective in inferring that a person is shoplifting. Thus, the relationship identification unit 135 identifies the change in distance between the position of the product, which is the subject detected by the subject detection unit 133, and the position of the product shelf, which is the subject for which the attribute settings have been accepted by the setting acceptance unit 134.

The estimation unit 136 estimates the activity content of a person. The estimation unit 136 estimates the activity content of a person based on a combination of the type of subject and the relative relationship. The estimation unit 136 searches, for example, whether or not a combination of the type of subject and the relative relationship is included in the activity content database stored in the memory unit 12. If the activity content database includes a similar or matching combination, the estimation unit 136 estimates that the activity content associated with the similar or matching combination is the activity content of a person.

For example, when a person in a store's sales area places a product in a container, the subject detection unit 133 detects the product, which is the first subject, and the container, which is the second subject. The relationship identification unit 135 identifies the relative relationship based on the distance between the person's hand and the first subject, and the distance between the first subject and the second subject. The relationship identification unit 135 identifies the relative relationship as "the distance between the person's hand and the first subject, and the distance between the first subject and the second subject are each less than a predetermined distance (e.g., distance D1 and distance D2 shown in FIG. 5)."

If the type of the second subject is a "bag," the estimation unit 136 estimates the person's behavior as shoplifting, and if the type of the second subject is a "shopping basket," the estimation unit 136 estimates the person's behavior as shopping. In this way, even if the people's behaviors are similar, the estimation unit 136 can estimate the person's behavior with high accuracy by estimating the person's behavior using the type of subject.

Furthermore, the estimation unit 136 may estimate the person's behavior based on the fact that the relationship identification unit 135 has identified the change in the relative relationship as "the distance between the first subject and the second subject becomes shorter." By operating in this manner, the estimation unit 136 can estimate the person's behavior with even higher accuracy.

When the relationship identification unit 135 identifies the distance between the person and the subject as a relative relationship, the estimation unit 136 may estimate the activity content based on the result of comparing a reference distance associated with the type of subject detected by the subject detection unit 133 with the distance between the person and the subject identified by the relationship identification unit 135. The reference distance is, for example, a distance that corresponds to the characteristics of the person's activity content, such as "less than distance A1" or "less than distance B1" shown in FIG. 5. The reference distance may be a distance that differs depending on the type of subject.

For example, when a person staying in a store is gazing at a product that is the subject, the relationship identification unit 135 identifies the relative relationship between the person and the subject as "the distance between the person and the subject is distance Y1." In this case, the estimation unit 136 compares the distance Y1 identified by the relationship identification unit 135 with the reference distance "less than distance A1" shown in FIG. 5. If the distance Y1 is less than distance A1, the estimation unit 136 determines that there is a possibility that the person is looking at the product, since this matches "the distance between the person and subject 1 is less than distance A1," which is one of the relative relationships associated with the behavior content "looking at a product."

When the relationship identifying unit 135 identifies the relationship between the direction of the person's gaze and the position of the subject as a relative relationship, the estimation unit 136 may estimate the activity content based on the result of comparing a reference gaze relationship, which is the relationship between the direction of the gaze and the position of the subject associated with the type of subject detected by the subject detection unit 133, with the relationship between the direction of the person's gaze and the position of the subject identified by the relationship identifying unit 135. The relationship between the direction of the gaze and the position of the subject is represented by the difference between the direction of the person's gaze and the direction of the gaze when the person looks directly at the subject. The reference gaze relationship is a directional difference corresponding to the characteristic of the activity content, for example, "less than directional difference A2" as shown in FIG. 5.

For example, when a person staying in a store is looking at a product that is the subject, the relationship identifying unit 135 identifies the relative relationship between the person and the subject as "the relationship between the direction of the person's gaze and the position of the subject is a directional difference Y2." In this case, the estimation unit 136 compares the directional difference Y2 identified by the relationship identifying unit 135 with the reference gaze relationship shown in FIG. 5, "less than directional difference A2." If the directional difference Y2 is a direction closer to the subject than the directional difference A2, the estimation unit 136 determines that there is a possibility that the person is looking at the product, since this matches one of the relative relationships associated with the behavior "looking at a product," "the directional difference between the person's gaze and subject 1 is less than directional difference A2."

The estimation unit 136 may estimate that the person's behavior is "looking at a product" when both the distance between the person and the subject and the directional difference between the direction of the person's line of sight and the position of the subject meet criteria. The estimation unit 136 estimates the person's behavior based on multiple relative relationships, thereby further improving the accuracy of estimating the behavior.

When the relationship identifying unit 135 identifies the relationship between the person's body orientation and the subject's position as a relative relationship, the estimation unit 136 may estimate the activity content based on the result of comparing a reference body orientation relationship, which is the relationship between the body orientation and the subject's position associated with the type of subject detected by the subject detection unit 133, with the relationship between the person's line of sight identified by the relationship identifying unit 135 and the subject's position. The relationship between the body orientation and the subject's position is represented by the difference between the body orientation and the body orientation when the person faces the subject. The reference body orientation relationship is the difference in orientation corresponding to the characteristics of the activity content, and is, for example, the orientation "less than angle difference B2" shown in FIG. 5.

For example, when a person (a shopper) staying in a store is talking to an employee who is the subject, the relationship identification unit 135 identifies the relative relationship between the person and the subject as "the relationship between the person's body orientation and the subject's position is an angle difference of X1." In this case, the estimation unit 136 compares the angle difference X1 identified by the relationship identification unit 135 with the reference body orientation relationship "less than angle difference B2" shown in FIG. 5. If the angle difference X1 is facing in a direction closer to the subject than the angle difference B2, the estimation unit 136 determines that there is a possibility that the person is talking to an employee, since it matches one of the relative relationships associated with the activity "talking to an employee," "the angle difference between the person's body orientation and subject 1 is less than angle difference B2."

The estimation unit 136 may estimate that the person's behavior is "talking to an employee" if both the distance between the person and the subject and the angle difference between the orientation of the person's body and the position of the subject meet the criteria. The estimation unit 136 estimates the person's behavior content based on multiple relative relationships, thereby further improving the accuracy of estimating the behavior content.

When the relationship identification unit 135 identifies a change in the relative relationship using multiple relative relationships, the estimation unit 136 may estimate the person's behavior based on a combination of the type of subject and the change in the relative relationship. For example, when a person staying in a store picks up a product and puts it into a bag, the relationship identification unit 135 identifies the distance between the product, which is the first subject, and the bag, which is the second subject, at each of multiple times as the relative relationship. The relationship identification unit 135 identifies the change in the relative relationship as "the distance between the first subject and the second subject becomes shorter."

The estimation unit 136 determines that the action of a person putting a product into a bag based on the change in the relative relationship identified by the relationship identification unit 135 and the combination of the subject type, which is a product and a bag, and estimates the person's behavior as "shoplifting." In this way, by using the change in the relative relationship, the estimation unit 136 can improve the accuracy of estimating the person's behavior.

When the estimation unit 136 estimates the behavior of a person based on a combination of the type of subject and a change in the relative relationship, the estimation unit 136 may estimate the behavior based on the result of determining whether or not the subject was included in any of the multiple captured image data before the person was detected by the person detection unit 132.

For example, if a person in a store's sales floor picks up an object placed on a shelf, it is necessary to determine whether the type of object is a product. In this case, the estimation unit 136 determines whether an object shown in an image shown by captured image data created, for example, before the store opens matches an object shown in an image shown by captured image data created at the time when the person detection unit 132 detected a person who picked up an object placed on a shelf. If it is determined that there is a match, the estimation unit 136 identifies the type of object as a product. If the person puts an object in a bag and the object is a product, the estimation unit 136 estimates that the person's behavior is "shoplifting."

By operating the estimation unit 136 in this manner, for example, if a customer who comes to a store leaves a personal item on a shelf in the store and moves away, and then returns to retrieve the personal item after a certain amount of time has passed, the estimation unit 136 can distinguish between the personal item that the customer took from the shelf and the product based on the image captured before the store opens. As a result, the estimation unit 136 can be prevented from erroneously estimating that a behavior that is not shoplifting is "shoplifting."

When the relationship identification unit 135 identifies multiple relative relationships based on the captured image, the estimation unit 136 may estimate the content of a person's behavior based on a combination of each of the multiple relative relationships and the type of subject.

For example, if a person in a store's sales floor puts a product placed on a shelf into a bag, the relationship identification unit 135 identifies the distance between the person and the shelf, which is an example of a first relative relationship, and the distance between the person and the product, which is an example of a second relative relationship. If the distance indicated by the first relative relationship and the distance indicated by the second relative relationship are each less than a reference distance, the estimation unit 136 estimates that the person's behavior is shoplifting based on the combination of the first relative relationship, the second relative relationship, and the types of subjects, which are the product and the bag.

Furthermore, when the relationship identification unit 135 identifies a change in the second relative relationship based on multiple second relative relationships identified at multiple times, the estimation unit 136 may estimate the person's behavior based on a combination of the change in the first relative relationship and the second relative relationship, and the type of subject. For example, the estimation unit 136 may estimate the person's behavior as shoplifting based on a combination of the first relative relationship, "the distance between the person and the shelf is less than a reference distance," the change in the second relative relationship, "the distance between the person and the product becomes shorter," and the types of subjects, which are the shelf and the product.

When the relationship identification unit 135 identifies multiple relative relationships based on a person and multiple subjects, the estimation unit 136 may estimate the behavior of the person based on a combination of the multiple relative relationships. For example, the estimation unit 136 may estimate the behavior of the person based on a combination of the type of subject and at least two relative relationships, the first relative relationship, the second relative relationship, or the third relative relationship.

For example, when a person puts an item into a bag on the sales floor of a store, the subject detection unit 133 detects the item and the bag as the subject. The relationship identification unit 135 identifies the relationship between the person and the item as a first relative relationship, the relationship between the person and the bag as a second relative relationship, and the relationship between the item and the bag as a third relative relationship. The estimation unit 136 estimates the person's behavior as "shoplifting" based on the fact that the type of subject is a bag and item, that the distance between the person's hand and the bag is less than D1, and that the distance between the bag and the item is less than D2.

Note that, although an example has been given here in which the estimation unit 136 estimates the person's behavior as "shoplifting" using the second relative relationship and the third relative relationship, the estimation unit 136 may also estimate the person's behavior by further using the first relative relationship, "a person is holding a product."

The estimation unit 136 may estimate the behavioral content further based on the attributes of the person detected by the person detection unit 132. For example, if a person puts an item into a bag on a sales floor in a store, if the person's attributes are a customer, the estimation unit 136 estimates the behavioral content of the person as "shoplifting," and if the person's attributes are an employee, the estimation unit 136 estimates the behavioral content of the person as "organizing items." By operating in this manner, the estimation unit 136 can correctly estimate the behavioral content even when the movements of the person are similar, such as "shoplifting," which is the action of picking up an item, and "organizing items."

The estimation unit 136 may estimate the behavioral content based on the attributes of the subject received by the setting reception unit 134. The attributes of the subject are information indicating the characteristics of the subject, such as the subject's owner and the subject's function. For example, if the setting reception unit 134 receives that the attribute of an object placed on a shelf installed in a store's warehouse or the like is "store property," the estimation unit 136 estimates that the behavioral content of the employee making the movement of picking up the object is "organizing merchandise or equipment." On the other hand, if the setting reception unit 134 determines that the attribute of the object is not "store property," the estimation unit 136 estimates that the behavioral content of the employee picking up the object is, for example, "moving personal belongings."

Furthermore, when the relationship identification unit 135 identifies the positional relationship between the subject detected by the subject detection unit 133 and the subject for which the setting reception unit 134 has accepted the attribute setting, the estimation unit 136 may further estimate the behavior based on the positional relationship identified by the relationship identification unit 135 and the attribute of the subject for which the setting reception unit 134 has accepted the setting. For example, when the subject identified by the relationship identification unit 135 is a product shelf and the subject for which the setting reception unit 134 has accepted the setting is an object placed on the product shelf, the estimation unit 136 determines that a person other than an employee is making the movement of putting an object placed on the product shelf into a bag, based on the positional relationship identified by the relationship identification unit 135. Next, when the attribute of the object is "store property," the estimation unit 136 estimates that the behavior of the person other than an employee is shoplifting.

By estimating the behavioral content based on the attributes of the subject in this way, the estimation unit 136 can distinguish between the products and the objects other than the products and estimate the behavioral content of the person even when multiple products and multiple objects other than the products (e.g. personal belongings) are placed in the same place.

The location identification unit 137 identifies a location that appears in the captured image. For example, the location identification unit 137 identifies a location that appears in the captured image based on a subject that appears in the captured image. For example, the location identification unit 137 may identify a location that appears in the captured image based on a GPS (Global Positioning System) signal. The location identification unit 137 notifies the output unit 138 of information indicating the identified location.

The output unit 138 functions as a transmission unit that transmits activity content information indicating the activity content estimated by the estimation unit 136 to the information terminal 2. The output unit 138 may output notification information when the activity content estimated by the estimation unit 136 and the location identified by the location identification unit 137 satisfy a predetermined condition. The predetermined condition is that the user of the information terminal 2 should receive the notification information and take some kind of action. For example, if the imaging device is installed in a store, the user of the information terminal 2 is an employee or security guard of the store.

Specifically, if the behavior estimated by the estimation unit 136 is "shoplifting" and the location identified by the location identification unit 137 is a "store," an employee or security guard at the store must apprehend the shoplifter. Therefore, the output unit 138 determines that the specified condition is met and outputs notification information.

On the other hand, if the activity content estimated by the estimation unit 136 is "organizing merchandise" and the location identified by the location identification unit 137 is "store," the estimated activity content is a normal employee activity, and therefore the output unit 138 determines that the predetermined condition is not satisfied and does not output notification information. By operating in this manner, the output unit 138 can output information required by the user of the information terminal 2.

The output unit 138 may output notification information of a type corresponding to the combination of the location and behavior content identified by the location identification unit 137. For example, if the location identified by the location identification unit 137 is a "sales floor of a store" and the behavior content of the person estimated by the estimation unit 136 is "shoplifting," the output unit 138 outputs notification information indicating that shoplifting has occurred on the sales floor of the store.

[Flowchart of the behavior inference device 10]
Fig. 6 is a flowchart showing an example of an operation of the behavior inference device 10 inferring the behavioral details of a person. Fig. 6 illustrates an operation of the behavior inference device 10 inferring the behavioral details of a person by specifying a change in a relative relationship based on the relative relationship at time T1 and the relative relationship at time T2 and using the specified change in the relative relationship.

The image acquisition unit 131 acquires the first captured image data generated by the imaging device 1 at time T1 (S11). The location identification unit 137 identifies the location depicted in the captured image indicated by the first captured image data acquired by the image acquisition unit 131. The person detection unit 132 detects the position of a person included in the captured image indicated by the first captured image data (S12). The subject detection unit 133 detects the position of a subject and the type of subject depicted in the captured image indicated by the first captured image data (S13). The relationship identification unit 135 identifies a relative relationship between the position of the person, the position of the subject, and the type of subject at time T1 (S14).

The image acquisition unit 131 acquires the second captured image data generated by the imaging device 1 at time T2 (S15). The person detection unit 132 detects the position of a person included in the captured image represented by the second captured image data (S16). The subject detection unit 133 detects the position of a subject and the type of subject appearing in the captured image represented by the second captured image data (S17). The relationship identification unit 135 identifies a relative relationship between the position of the person, the position of the subject, and the type of subject at time T2 (S18).

The relationship identification unit 135 identifies a change in the relative relationship based on the relative relationship at time T1 identified in S14 and the relative relationship at time T2 identified in S18 (S19). The estimation unit 136 estimates the person's behavior based on a combination of the change in the relative relationship identified by the relationship identification unit 135 and the type of subject detected by the subject detection unit 133 (S20).

If an operation to end the process of estimating the person's behavioral content has not been performed (NO in S21), the behavior estimation device 10 repeats the processes from S11 to S19. If an operation to end the process of estimating the person's behavioral content has been performed (YES in S21), the behavior estimation device 10 ends the process.

[Other application examples]
In the above description, an example was given of estimating the behavior of a customer or employee in a store. However, the behavior estimation device 10 may estimate various types of behavior of people based on captured images in other locations.

(Color-based estimation)
Fig. 7 is a diagram showing a configuration of the behavior estimation device 10 that estimates the behavior of a person based on a change in color of the person or the subject. The behavior estimation device 10 shown in Fig. 7 differs from the behavior estimation device 10 shown in Fig. 4 in that it has a color identification unit 139, but is the same in other respects. The color identification unit 139 identifies at least one of a change in color of the person or a change in color of the subject. The color identification unit 139 identifies a change in color of the person or the subject included in the captured image acquired by the image acquisition unit 131, based on the position of the person detected by the person detection unit 132 or the position of the subject detected by the subject detection unit 133.

The estimation unit 136 may estimate the behavioral content based on at least one of the color change of the person or the color change of the subject identified by the color identification unit 139. For example, if a person is using a knife, the subject detection unit 133 detects the knife, which is a type of subject. The relationship identification unit 135 identifies the relative relationship, which is the relationship between the person and the knife, which is the subject, as "the distance between the person and the subject is less than a reference distance." The color identification unit 139 identifies that the person's hand and the knife have changed to red based on the position of the person and the position of the knife. The estimation unit 136 estimates that the person is injured based on the change of the person's hand and the knife to red.

In addition, when other people are around the person using the knife, the relationship identification unit 135 identifies the change in the relative relationship between the subject knife and the other people as "the distance between the knife and the other people changes significantly from time to time," and the estimation unit 136 estimates that the person's behavior is dangerous behavior. Then, the color identification unit 139 identifies that the color of the knife or the other people has changed to red, and the estimation unit 136 estimates that the other people are injured.

In this way, the color identification unit 139 identifies the change in color of the person or subject, and the estimation unit 136 identifies the person's behavior based on the change in color, so that the behavior estimation device 10 can estimate the content of an action that requires an emergency response, such as bleeding. Furthermore, the behavior estimation device 10 can estimate the content of an action, such as spilling paint, by detecting a change to a color other than red, for example.

(Detection of dangerous behavior)
The behavior inference device 10 may detect a dangerous behavior occurring outdoors, for example, based on an image captured by a surveillance camera installed outdoors. The dangerous behavior is, for example, an act of destroying property such as a bench installed in a park. When a person performing karate movements and a bench are detected based on the image captured by the surveillance camera, the relationship specification unit 135 specifies the relative relationship between the person and the bench, which is the subject of the image.

When the relationship identification unit 135 identifies the relative relationship as "the distance between the person and the subject is equal to or greater than the threshold", the estimation unit 136 identifies the person's behavior as "practicing karate". The threshold is, for example, a distance indicating the range of movement of a person when practicing a martial art such as karate. On the other hand, when the relationship identification unit 135 identifies the relative relationship as "the distance between the person and the subject is less than the threshold", the estimation unit 136 estimates the person's behavior as "destroying a bench". By operating in this manner, the behavior estimation device 10 can monitor objects installed outdoors and the behavior of people around the objects.

(Detection of intrusion into prohibited area)
The behavior inference device 10 may infer a behavior of a person entering a no-entry area based on an image from a surveillance camera. For example, if a fence is installed at the boundary of the no-entry area, the subject detection unit 133 detects the fence, which is the subject. If a person climbs over the fence to enter, the person detection unit 132 detects the movement of the person climbing over the fence. The relationship identification unit 135 identifies the relative relationship as "a person climbing over the subject." The estimation unit 136 infers the content of the person's behavior as "climbing over a fence" based on a combination of the fence, which is the type of subject, and the relative relationship.

If no fence is installed in the no-entry area, the subject detection unit 133 may detect the no-entry area as the type of subject. If a person enters the no-entry area, the person detection unit 132 detects that the person is moving. The relationship identification unit 135 identifies the relative relationship as "a person is moving inside the subject" based on the position information indicating the no-entry area that is the subject and the position to which the person has moved. The estimation unit 136 estimates the content of the person's behavior as "entering a no-entry area."

By operating in this manner, the behavior estimation device 10 can detect when a person enters a forbidden area. The behavior estimation device 10 notifies the information terminal 2 that a person has entered the forbidden area, so that the administrator of the forbidden area can learn of the person's intrusion and issue a warning to the person.

[Effects of the behavior inference device 10]
As described above, the behavior inference device 10 includes the person detection unit 132 that detects the movement of a person included in a captured image, the subject detection unit 133 that detects the type of subject included in the captured image, and the relationship specification unit 135 that specifies the relative relationship between the person and the subject associated with the person's movement. The estimation unit 136 infers the behavior of the person based on the type of subject detected by the subject detection unit 133 and the relative relationship specified by the relationship specification unit 135.

The behavior estimation device 10 estimates the behavior of a person using the type of subject and the relative relationship between the person and the subject in this way, and is therefore able to accurately estimate the behavior of a person even if the movements of the person are similar. Furthermore, according to the behavior estimation system S, the information terminal 2 receives behavior content information indicating the behavior of a person estimated by the behavior estimation device 10, and thus, for example, a store employee who is the user of the information terminal 2 can be informed of abnormal behavior of a person with high accuracy. As a result, the user of the information terminal 2 can respond to the person's behavior accurately.

Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

REFERENCE SIGNS LIST 1 Imaging device 2 Information terminal 10 Behavior estimation device 11 Communication unit 12 Memory unit 13 Control unit 21 Communication unit 22 Memory unit 23 Control unit 24 Display unit 25 Operation unit 131 Image acquisition unit 132 Person detection unit 133 Subject detection unit 134 Setting reception unit 135 Relationship identification unit 136 Estimation unit 137 Location identification unit 138 Output unit 139 Color identification unit 231 Information transmission unit 232 Information reception unit

Claims (11)

an image acquisition unit that acquires captured image data representing a captured image;
a person detection unit that detects a movement of a person and an attribute of the person included in the captured image;
a subject detection unit that detects a type of subject other than the person included in the captured image;
a relationship specification unit that specifies a relative relationship between the person and the subject in association with the person's movement;
an estimation unit that estimates whether the behavioral content of the person is fraudulent behavior or not based on a combination of the attribute of the person detected by the person detection unit, the type of the subject detected by the subject detection unit, and the relative relationship identified by the relationship identification unit by referring to behavioral content data in which an attribute of the person, whether the person is an employee or not, a type of the subject, a relative relationship between the person and the subject, and information indicating whether the behavioral content of the person is fraudulent behavior or not ;
A behavior estimation device having the above configuration. The estimation unit estimates whether the behavior of the person is unauthorized behavior based on a subject attribute of whether the subject, which is different from the person, is owned by a facility.
The behavior inference device according to claim 1 . The type of the subject is represented by any one of a product, equipment, person, or an item other than a product or equipment .
The behavior inference device according to claim 1 . the subject detection unit detects a no-entry area as a type of subject included in the captured image,
the estimation unit estimates the behavioral content of the person that has entered the no-entry area based on a combination of the type of the subject detected by the subject detection unit and the relative relationship by referring to the behavioral content data.
The behavior inference device according to claim 1 . the subject detection unit detects a fence as a boundary of the no-entry area,
the estimation unit estimates that the person has climbed over the fence based on the relative relationship that the position of the person detected by the person detection unit is higher than the fence.
The behavior inference device according to claim 4 . the relationship specification unit specifies a distance between the person and the subject as the relative relationship;
the estimation unit estimates the activity content further based on a result of comparing a reference distance associated with the type of the subject detected by the subject detection unit with a distance between the person and the subject identified by the relationship identification unit.
The behavior inference device according to claim 1 . the image acquisition unit acquires a plurality of pieces of captured image data created in chronological order;
The relationship specification unit specifies a change in the relative relationship,
The estimation unit estimates the action content of the person based on a combination of the type of the subject and the change in the relative relationship.
The behavior inference device according to claim 1 . the subject detection unit detects a first subject and a second subject included in the captured image;
the relationship specifying unit specifies a first relative relationship that is a relationship between the person and the first subject associated with a movement of the person, a second relative relationship that is a relationship between the person and the second subject associated with a movement of the person, and a third relative relationship that is a relationship between the first subject and the second subject associated with a movement of the person,
the estimation unit estimates the action content of the person based on a combination of the type of the subject and at least two of the first relative relationship, the second relative relationship, or the third relative relationship.
The behavior inference device according to claim 1 . The computer executes
acquiring captured image data indicative of a captured image;
detecting a movement of a person and an attribute of the person included in the captured image;
detecting a type of a subject other than the person included in the captured image;
Identifying a relative relationship between the person and the subject in association with the person's movement;
a step of referring to behavior content data in which an attribute of the person, whether or not the person is an employee, a type of the subject, a relative relationship between the person and the subject, and information indicating whether or not the behavior content of the person is fraudulent behavior are associated with each other, and estimating whether or not the behavior content of the person is fraudulent behavior based on a combination of the detected attribute of the person, the detected type of the subject, and the specified relative relationship;
The behavior estimation method has the following features. On the computer,
acquiring captured image data indicative of a captured image;
detecting a movement of a person and an attribute of the person included in the captured image;
detecting a type of a subject other than the person included in the captured image;
Identifying a relative relationship between the person and the subject in association with the person's movement;
a step of referring to behavior content data in which an attribute of the person, whether or not the person is an employee, a type of the subject, a relative relationship between the person and the subject, and information indicating whether or not the behavior content of the person is fraudulent behavior are associated with each other, and estimating whether or not the behavior content of the person is fraudulent behavior based on a combination of the detected attribute of the person, the detected type of the subject, and the specified relative relationship;
A program for executing. A behavior estimation device that estimates a behavior of a person based on a captured image generated by an imaging device, and an information terminal that displays the behavior of the person estimated by the behavior estimation device,
The behavior estimation device includes:
an image acquisition unit that acquires captured image data representing a captured image;
a person detection unit that detects a movement of a person and an attribute of the person included in the captured image;
a subject detection unit that detects a type of subject other than the person included in the captured image;
a relationship specification unit that specifies a relative relationship between the person and the subject in association with the person's movement;
an estimation unit that estimates whether the behavioral content of the person is fraudulent behavior or not based on a combination of the attribute of the person detected by the person detection unit, the type of the subject detected by the subject detection unit, and the relative relationship identified by the relationship identification unit by referring to behavioral content data in which an attribute of the person, whether the person is an employee or not, a type of the subject, a relative relationship between the person and the subject, and information indicating whether the behavioral content of the person is fraudulent behavior or not ;
a transmission unit that transmits, to the information terminal, activity content information indicating the activity content estimated by the estimation unit;
having
The information terminal includes:
an information receiving unit for receiving the action content information;
a display unit that displays the activity content based on the activity content information received by the information receiving unit;
A behavior estimation system having the above configuration.

Images