JP2019012328A - Person action estimation system - Google Patents

Abstract

To solve the problem in which operation recognition of a person with video analysis often has difficulty in accurate action estimation of the person due to a dead zone frequently generated in a video.SOLUTION: A person action estimation system 1 for distinguishing actions carried out by persons using tools comprises: a person action distinction unit 15 which outputs person action candidates for an action captured in a video from a predetermined person action definition 12; a tool operation distinction unit 25 which outputs, on the basis of sensor information from a tool 101, tool operation candidates for the tool from which the sensor information is acquired from a predetermined tool operation definition 22; and a comprehensive person action distinction unit 2 which estimates the action captured in the video on the basis of the person action candidates and the tool operation candidates.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a human behavior estimation system.

  Patent Document 1 states that “a person action determination device calculates a feature amount for each person area by means of person area detection means for detecting one or more person areas included in a video by machine learning at a predetermined frame interval, A human trajectory generating means for determining a human area having a similar feature amount of the human area in a plurality of frame images as a human area of the same person, and generating a human trajectory by connecting the gravity center positions of the human areas of the same person; The feature amount is calculated every time, and it is determined whether or not the feature amount of the person locus satisfies the action condition. When the feature amount of the person locus satisfies the action condition, the person performs an action corresponding to the action condition. And a human action determination unit that determines that the user is present. "

JP 2011-1000017 A

  Human motion recognition by video analysis is based on the premise that a person region is cut out from video, the outline of the person is accurately acquired, and the person can be extracted accurately. However, even if a person can be extracted, a blind spot occurs in the human action due to the position of the camera, the position of the hand, the position of the body, etc., and it is difficult to accurately estimate the action of the person.

  Patent Document 1 describes a person behavior determination device that determines a person's behavior based on whether or not the feature amount of the person trajectory satisfies a behavior condition set in advance for each person's behavior. However, it is difficult to set an action condition for an action hidden in the blind spot, and in such a case, it is considered that information is insufficient to accurately determine the action of the person.

  For this purpose, the configuration described in the claims is adopted. For example, according to the present invention, a human behavior estimation system that discriminates a behavior performed by a person using a tool is determined in advance based on a video acquisition unit that acquires a video of the action and a video from the video acquisition unit. From a human action definition, a human action discriminating section that outputs human action candidates for actions taken in the video, a tool data acquiring section that acquires sensor information from a sensor attached to the tool, and a tool data acquiring section Based on the sensor information of the tool, a tool motion discrimination unit that outputs tool motion candidates for the tool for which sensor information has been acquired from a predetermined tool motion definition, and a human behavior candidate output from the human behavior discrimination unit and And a general human action discriminating unit for estimating an action photographed in the video from the video acquiring unit based on the tool motion candidate output from the tool motion discriminating unit.

  The action of the person to be the action estimation target can be realized with high accuracy by using the action estimation from the video of photographing the person and the motion estimation of the tool used by the person.

It is a figure which shows the processing flow of a person action estimation system. It is an example of area division of surveillance space. It is a figure which shows the definition (learning data) of the human action information of an image | video. It is a figure which shows the definition (learning data) of the tool operation information of tool (sensor) data. It is a figure which shows the flowchart of an integrated person action determination part. It is an example of an image of a surveillance space. It is another example of a picture of surveillance space. It is another example of a picture of surveillance space. It is a mapping figure of a person action estimation result and a tool operation estimation result. It is a figure which shows the hardware constitutions which implement | achieve a person action estimation system.

  FIG. 1 shows a processing flow of the human behavior estimation system 1. The human behavior estimation system 1 may be realized in a local environment such as a general PC (Personal Computer) or may be realized via a network such as a cloud.

  FIG. 8 shows a hardware configuration example for realizing the human behavior estimation system 1. The computer 800 includes a processor 801, a main memory 802, an auxiliary memory 803, an input / output interface 804, a display interface 805, and a network interface 806, which are coupled by a bus 807. The input / output interface 804 is connected to an input device 809 such as a keyboard and a mouse to provide a user interface. The display interface 805 is connected to the display 808. A network interface 806 is an interface for connecting the computer 800 and an external network (not shown).

  The auxiliary storage 803 is generally composed of a nonvolatile memory such as an HDD or a flash memory, and stores a program executed by the computer 800, data to be processed by the program, and the like. The main memory 802 is composed of a RAM, and temporarily stores a program, data necessary for executing the program, and the like according to instructions from the processor 801. The processor 801 executes a program loaded from the auxiliary memory 803 to the main memory 802.

  Each processing block of the human behavior estimation system 1 is stored in the auxiliary memory 803 as a program, loaded from the auxiliary memory 803 to the main memory 802 and executed by the processor 801. 1 are also stored in the auxiliary memory 803 or called from the auxiliary memory 803 to the main memory 802 for processing. Hereinafter, as an application example, a case where the human behavior estimation system 1 is applied to a monitoring system that monitors whether a worker on a production line is correctly mounting or processing a part will be described.

  The camera device 200 photographs a person who is an object of behavior estimation (in this example, a worker on the production line). The video acquisition unit 10 acquires a video (moving image) captured by the camera device 200 and stores it in the video information storage unit 11. In the system, an action of a person to be discriminated in advance is defined as a human action definition 12. For example, the table 400 of FIG. 3 shows an example in which human behavior is defined for three categories of “person”, “region”, and “motion”.

  The “person” category defines who is the target of behavior estimation. In this example, “person A” and “person B” are defined. The “region” category defines in which region of the video where the person who is the target of behavior estimation is photographed (working). This is because what work is performed by the worker on the production line is roughly determined, and where the person who is the object of the action estimation is acting is important information for action estimation. For this reason, as shown in FIG. 2, the video space photographed by the camera device 200 is divided (in this example, divided into 9 areas of 3 × 3), and the position where the person whose behavior is to be estimated is defined. . For simplicity, the camera apparatus 200 is fixed, and the present embodiment is described as an example in which areas are divided by video space. However, if the camera apparatus 200 is a stereo camera, it may be defined in a three-dimensional space. Further, the real space region may be divided and the real space region where the person whose behavior is to be estimated is determined from the video by a known video processing technique. The “motion” category defines what action (work) the person who is the object of action estimation is performing. In this example, operations such as “tighten a screw”, “drill a hole”, and “loosen a screw” are defined. This definition may be determined according to the content to be discriminated from the video (moving image) captured by the camera device 200. The “operation” category can be determined by identifying work to be discriminated in the monitoring system, which is performed by the worker on the production line by a work manual or the like. Note that it is not required to define all of these categories, and it is not necessary to define the “person” category if it is not necessary to specify who the person whose behavior is to be estimated. Or you may provide another definition category as needed like an operator's clothes and equipment, for example.

  On the other hand, sensors 103a and 103b such as a vibration sensor and an acceleration sensor are attached to a driver 101a and a cone 101b which are tools used by a person who is a target of behavior estimation. The type of sensor is not particularly limited, and may be different for each tool, or a plurality of sensors may be provided. The tool data acquisition unit 20 acquires position information indicating a position where the tool 101 is used, sensor data from the sensor 103 attached to the tool 101, information obtained by processing the sensor data, or tool output information. Store in the data storage unit 21. For example, when the acceleration sensor 103a is attached to the driver 101a, the acceleration data acquired during the screwing operation and the processed trajectory data are accumulated. Further, the position information where the tool 101 is used may be acquired from the tool or detected from the video acquired by the video acquisition unit 10. In the system, a tool motion associated with a human action to be discriminated in advance is defined as a tool motion definition 22. For example, the table 500 of FIG. 4 shows an example in which human behavior is defined for three categories of “tool”, “region”, and “motion”.

  The “tool” category defines a tool used by a person whose behavior is to be estimated. In this example, “driver” and “cone” are defined. The “region” category defines the region in which the tool is used. This is due to the fact that where the workers on the production line perform what operations and where, the places where the tools are used are limited accordingly. The area can be defined in the same manner as the human action definition, and the same area definition as the human action definition or a different area definition may be used. In the example of FIG. 4, the same area definition is made. The “Action” category defines actions performed on the tool. In this example, an operation such as “tighten a screw”, “loosen a screw” for a driver, and “open a hole” for a cone is defined.

  The human behavior learning unit 13 first defines the video acquired from the camera device 200 and stored in the video information storage unit 11 based on the human behavior definition 12. This becomes learning data. A table 400 shown in FIG. 3 is an example of learning data. For example, a record 401 is an image “a.mpeg” in which “person A” performs an operation of “screwing” in an area “X1Y1 and X1Y2”. It is defined as being. A model for discriminating human behavior from the learning data 400 is created. For example, a model for discriminating human behavior from a video acquired from the camera device 200 is created using machine learning such as deep learning. A model that is a human behavior learning result is stored in the human behavior learning result accumulation unit 14.

  The tool motion learning unit 23 first defines the tool data acquired from the sensor 103 attached to the tool 101 and stored in the tool data storage unit 21 based on the tool motion definition 22. This becomes learning data. A table 500 illustrated in FIG. 4 is an example of learning data. For example, a record 501 includes tool data “a.csv” that is tool data in which a “driver” performs a “screw tightening” operation in an area “X1Y3”. It is defined as being. If the sensor attached to the tool is an acceleration sensor, the tool data “a.csv” is time-series data of detected acceleration and feature data (text data file) obtained by processing it. A model for discriminating tool movement is created from the learning data 500. For example, using machine learning such as deep learning, a model for discriminating the tool operation from the tool data acquired from the sensor 103 attached to the tool is created. The model that is the tool motion learning result is stored in the tool motion learning result storage unit 24.

  The human behavior determination unit 15 applies the model stored in the human behavior learning result accumulation unit 14 to the video from the video acquisition unit 10, estimates the human behavior, and outputs a human behavior candidate. Here, it is allowed to output a plurality of action candidates. Similarly, the tool motion determination unit 25 applies the model stored in the tool motion learning result storage unit 24 to the tool data from the tool data acquisition unit 20, estimates the tool motion, and outputs a tool motion candidate. . Here, it is allowed to output a plurality of motion candidates.

  FIG. 5 shows a flowchart of the general human behavior determination unit 2. First, the presence / absence of a result of the tool motion determination unit 25 is determined (S51). When there is no tool motion candidate from the tool motion determination unit 25, the human behavior candidate of the human behavior determination unit 15 is output as a result. When there is a tool motion candidate of the tool motion determining unit 25, the human behavior candidate of the human behavior determining unit 15 and the tool motion candidate of the tool motion determining unit 25 are compared to determine a match (S52). When the human action candidate from the human action determining unit 15 and the tool action candidate from the tool operation determining unit 25 match, the match is output as a human action determination result. If they do not match, the human behavior is estimated from the human behavior candidate and the tool motion candidate, and the estimation result is output (S53).

  The general human action discriminating unit 2 increases the judgment accuracy based on the operation information from the tool because it is often difficult to judge the action of the person to be estimated from the video alone. For example, FIG. 6A is a scene of an image showing a situation where “person A” “screws” in region “X1Y1, X1Y2” with “driver 101a”. Similarly, FIG. 6B is a scene of an image showing a situation where “person A” “drills” in the area “X2Y1, X2Y2” in “cone 101b”. Similarly, FIG. 6C is a scene of an image showing the situation where “person B” “looses the screw” in the area “X2Y1, X2Y2” with “driver 101a”. These are difficult to determine because they are similar images even when a person visually determines an action. In addition, it is difficult to accurately discriminate these similar behaviors in action determination by image processing, and furthermore, the tool that is the basis for important action determination is hidden in the shadows of people and can always be confirmed from the image. difficult.

  For example, it is assumed that the person A is performing an operation of screwing with a driver in the lower left half region of the entire monitoring region. In this embodiment, first, the video acquisition unit 10 displays a video image of the situation in the tool data for the situation. Is acquired by the tool data acquisition unit 20, and the human behavior determination unit 15 and the tool motion determination unit 25 respectively determine using the model of the learning result storage unit. Here, it is assumed that the human action definition is as shown in FIG. 3 and the tool motion definition is as shown in FIG.

  At this time, the human action discriminating unit 15 causes the human action candidate 401 that “person A” “screws” in the area “X1Y1, X1Y2” and “person A” “screws" in the area “X2Y1, X2Y2”. "Personal action candidate 402", "Person B" says "Perform a hole" in the area "X2Y1, X2Y2", and "Person A" say "Unscrew" in the area "X2Y1, X2Y2" The human action candidate 405 may be output as a human action candidate.

  In addition, when a person moves while moving or the work position crosses the section of the monitoring area in the human action definition, the human action discriminating unit 15 may select a plurality of human action candidates even if it is a series of work. May be output. For example, in the above example, both the human action candidate 401 and the human action candidate 402 are output. In such a case, in the case of continuous operation, either one (for example, a region when a candidate is estimated) may be output.

  On the other hand, the tool motion determination unit 25 outputs an operation candidate 502 that “driver” tightens a screw in the region “X2Y1” and an operation candidate 503 that “driver” tightens a screw in the region “X2Y2”. there's a possibility that. Note that if the tool position information is misaligned or the position of the tool straddles the section of the monitoring area in the tool motion definition, the tool motion determination unit 25 may use a plurality of tools even if it is a series of operations. There is a possibility of outputting motion candidates. In such a case, as in the case of the human action candidate, in the case of continuous motion, either one (for example, the region when the candidate is estimated) may be output.

  First, the comprehensive human action discriminating unit 2 collates the human action candidates output by the human action discriminating unit 15 with the tool motion candidates output by the tool operation discriminating unit 25 and removes contradictions. For example, the human action candidate 403 “punching” and the human action candidate 405 “loosen screw” are excluded from the output tool motion candidates. Further, as shown in FIG. 7, the human behavior candidate estimated by the human behavior determination unit 15 and the tool motion candidate estimated by the tool motion determination unit 25 are mapped and output as a human behavior estimation candidate based on the overlap. In this case, in the region “X2Y1, X2Y2”, “person A” uses “driver 101a” to “tighten” as a human action estimation candidate. Note that the general human action discriminating unit 2 may also learn a mapping situation as shown in FIG. 7 as a classification problem by machine learning such as deep learning, and estimate human actions.

  In addition, since the tool use area is defined in this embodiment, for example, if the tool motion definition is as shown in FIG. 4, the cone 101b is defined only for use in the areas “X2Y2” and “X3Y1”. Therefore, it is possible to issue a warning if the tool movement discriminating unit 25 or the comprehensive human behavior discriminating unit 2 can acquire the usage status outside the areas “X2Y2” and “X3Y1”, thereby preventing erroneous use of the tool and misoperation. It is also possible to plan. Specifically, the tool movement determination unit 25 issues a warning when it is determined that there is no corresponding tool movement definition, or when the general human action determination unit 2 estimates that there is no corresponding human action definition.

  As described above, the case where the present invention is applied to the monitoring system in the production line has been described as an example. However, the present invention is not limited to the described embodiment, and includes various modifications. For example, the tool can be extended to a general tool or thing used by a person whose behavior is to be estimated. The above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

1: human behavior estimation system, 2: comprehensive human behavior determination unit, 10: video acquisition unit, 11: video information storage unit, 12: human behavior definition, 13: human behavior learning unit, 14: human behavior learning result storage unit, 15: Human action determination unit, 20: Tool data acquisition unit, 21: Tool data storage unit, 22: Tool motion definition, 23: Tool motion learning unit, 24: Tool motion learning result storage unit, 25: Tool motion determination unit, 101: Tool, 103: Sensor, 200: Camera device.

Claims (6)

A human behavior estimation system that discriminates a human action using a tool,
A video acquisition unit that acquires a video of the behavior;
Based on the video from the video acquisition unit, from a predetermined human behavior definition, a human behavior determination unit that outputs human behavior candidates for the behavior photographed in the video,
A tool data acquisition unit for acquiring sensor information from a sensor attached to the tool;
Based on the sensor information from the tool data acquisition unit, a tool operation determination unit that outputs a tool operation candidate for the tool from which the sensor information has been acquired from a predetermined tool operation definition;
Based on the human action candidate output from the human action determination unit and the tool action candidate output from the tool action determination unit, a comprehensive human action that estimates the action taken in the video from the video acquisition unit A human behavior estimation system having a discrimination unit. In claim 1,
The human action definition includes first area information indicating an area where the action is performed,
The tool action definition includes second area information indicating an area where the tool is used,
The comprehensive human action determination unit performs mapping based on the first area information of the human action candidate and the second area information of the tool action candidate, thereby performing the action taken on the video from the video acquisition unit. Human behavior estimation system that estimates In claim 2,
The first area information is a human behavior estimation system defined by dividing a video space of the video. In claim 2,
When the tool motion determination unit determines that there is no corresponding tool motion definition for the tool from which the sensor information is acquired, or the comprehensive human behavior determination unit responds to the action photographed in the video A human behavior estimation system that issues a warning when it is estimated that there is no corresponding human behavior definition. In claim 1,
The human behavior determination unit is configured to detect the behavior captured in the video using a learning model obtained by machine learning using learning data in which the human behavior definition and the video from the video acquisition unit are associated with each other. A human behavior estimation system for outputting all human behavior candidates. In claim 1,
The tool motion determination unit has acquired the sensor information using a learning model obtained by machine learning using learning data in which the tool motion definition and the sensor information from the tool data acquisition unit are associated with each other. A human behavior estimation system that outputs the tool motion candidates for the tool.

Images