JP2022113042A - Task estimation device, control method thereof, information processing program, and recording medium - Google Patents

Abstract

An object of the present invention is to estimate, with sufficient accuracy, the details of a work performed by a worker on a work object from a captured image.
An information processing device (10) verifies the content of prediction using a work prediction model (171), which is a learned model, by the content predicted without using the work prediction model (171).
[Selection drawing] Fig. 1

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a work estimating device and the like for estimating the content of work performed by a worker in a workplace.

2. Description of the Related Art Conventionally, there have been known various attempts to use imaging data obtained by imaging the implementation status of a work process at a production site such as a factory to improve the work process. For example, Patent Literature 1 listed below discloses a safety management system that uses analysis results of image data to determine worker behavior. In addition, research is also progressing on HOI detection for detecting an action that a person is performing with respect to an object from a captured image as an interaction (HOI) between a person and an object captured in the captured image. It is

JP 2018-173957 A

However, there is a problem that the prediction of work content based on HOI detection alone cannot be said to have sufficient accuracy.

An object of one aspect of the present invention is to realize a work estimation device or the like capable of estimating the content of work performed by a worker on a work target from a captured image with sufficient accuracy.

In order to solve the above problems, a work estimation device according to one aspect of the present invention is a work estimation device for estimating the content of a work performed by a worker in a workplace, the workplace image capturing the workplace, or The feature information generated from the workplace image is input, and the content of the work performed by the worker captured in the workplace image on the work object captured in the workplace image is learned. A prediction unit that uses a work prediction model, which is a model, to predict the content of the work that the worker is performing on the work object from the workplace image or the feature information, and a prediction that does not use the work prediction model. a judgment unit for judging whether or not the prediction by the prediction unit is correct using at least one of the content of the work performed and a judgment result as to whether or not the worker is present in the workplace; an estimating unit for estimating the content of the work predicted by the prediction unit as the content of the work performed by the worker in the workplace when the prediction by the prediction unit is determined to be correct.

According to the above configuration, the work estimating device determines whether the content of the work predicted using the work prediction model is correct or not, the content of the work predicted without using the work prediction model, and the work location. It is determined using at least one of the determination results of the presence or absence of the worker in the above. Then, when it is determined that the prediction using the work prediction model is correct, the work estimation device predicts the content of the work predicted using the work prediction model to the work performed by the worker in the work place. Presumed to be the content of the work.

That is, the work estimating device determines whether the content of the work predicted using the work prediction model is correct or not, the content of the work predicted without using the work prediction model, and the worker at the work place. Verification is performed using at least one of the determination results of existence or non-existence.

Therefore, the work estimating device has the effect of being able to estimate with high accuracy the content of the work performed by the worker in the work place from the work place image.

A work estimating device according to an aspect of the present invention is a work estimating device, based on detection information of a sensor that detects at least one state of a tool used in the work, the work target, and the work place, and the work performed by the worker in the work place. a work detection unit that predicts the content of the work that is being performed, and the determination unit detects the content of the work predicted by the prediction unit and the content of the work predicted by the work detection unit; match, it may be determined that the prediction of the prediction unit is correct.

According to the above configuration, the work estimation device determines whether or not the work content predicted from the detection information of the sensor matches the work content predicted using the work prediction model. . Then, when it is determined that the two match, the work estimation device estimates that the content of the work predicted using the work prediction model is the content of the work performed by the worker in the workplace.

That is, the work estimating device verifies whether the work content predicted using the work prediction model is correct or not based on the work content predicted from the detection information of the sensor.

Therefore, the work estimating device has the effect of being able to estimate with high accuracy the content of the work performed by the worker in the work place from the work place image.

A work estimating device according to an aspect of the present invention includes (A) an overall image obtained by imaging a plurality of workplaces including the workplace, or (B) the workplace image, or (C) from the overall image or the workplace image. Using a presence/absence model, which is a learned model whose input is the generated feature information and whose output is whether or not the worker is present in the workplace, the entire image, the workplace image, or the entire A presence/absence prediction unit that predicts whether or not the worker is present in the workplace from an image or feature information generated from the workplace image, and the determination unit determines whether the worker is present in the workplace by the presence/absence prediction unit. If it is predicted that the worker will be present at , it may be determined that the prediction by the prediction unit is correct.

According to the above configuration, the work estimation device uses the presence/absence model to predict whether or not the worker is present in the workplace. Then, when predicting that the worker is present in the workplace, the work estimation device converts the content of the work predicted using the work prediction model to the content of the work performed by the worker in the workplace. Assume there is.

In other words, the work estimation device verifies the correctness of the content of the work predicted using the work prediction model based on the presence or absence of the worker in the workplace predicted using the presence/absence model.

Therefore, the work estimating device has the effect of being able to estimate with high accuracy the content of the work performed by the worker in the work place from the work place image.

A work estimating device according to an aspect of the present invention is a work estimating device, based on detection information of a sensor that detects at least one state of a tool used in the work, the work target, and the work place, and the work performed by the worker in the work place. (A) an overall image captured of a plurality of workplaces including the workplace, or (B) the workplace image, or (C) the overall image or the workplace The entire image, the workplace image, or a presence/absence prediction unit that predicts whether or not the worker is present in the workplace based on feature information generated from the overall image or the workplace image; When it is predicted by the unit that the worker is present in the workplace, and when the content of the work predicted by the prediction unit matches the content of the work predicted by the work detection unit, the prediction unit It may be determined that the prediction is correct.

According to the above configuration, the work estimating device predicts the content of the work predicted from the detection information of the sensor, and the presence or absence of the worker in the workplace. Then, the work estimation device predicts that the worker is present in the workplace, and when the content predicted from the detection information of the sensor matches the content predicted using the work prediction model, the work prediction It determines that the content predicted using the model is correct.

In other words, the work estimation device predicts the correctness of the work content predicted using the work prediction model, the work content predicted from the detection information of the sensor, and the work content predicted using the presence/absence model. The presence or absence of the worker is verified.

Therefore, the work estimating device has the effect of being able to estimate with high accuracy the content of the work performed by the worker in the work place from the work place image.

In the work estimating device according to one aspect of the present invention, the work prediction model is predicted from detection information of a sensor that detects at least one state of a tool used in the work, the work object, and the work place. , the content of the work performed by the worker in the workplace is constructed by machine learning using a learning data set in which the workplace image or feature information generated from the workplace image is labeled. may be

According to the above configuration, the work estimation device uses a learning data set labeled with "the content of the work performed by the worker in the workplace" predicted from the detection information of the sensor. Build the work prediction model by machine learning.

It is known that supervised learning requires a large amount of teacher data (learning data set). The labels (metadata) that are present are also called “annotations”.

For example, even if the training data learned when building the work prediction model does not require setting of a bounding box for detecting the worker, the following 5 one annotation is required. That is, it is necessary to set the X and Y coordinates of the lower left point and the upper right point of the four points indicating the range of the bounding box for detecting the work object with respect to the workplace image. , which means that four annotations are required. Furthermore, information (label) indicating "contents of work performed by the worker imaged in the workplace image on the work target imaged in the workplace image" is added to the workplace image. is required, which means one annotation is required.

Then, for each of the large number of workplace images, let a human determine "the details of the work that the worker imaged in the workplace image is doing with respect to the work target imaged in the workplace image". If so, the cost required for it is extremely high.

Therefore, when generating the training data to be learned when constructing the work prediction model, the "work content" predicted from the detection information of the sensor is used instead of the "work content" determined by a person from the workplace image. , is given as a label indicating the "work content". In other words, in the training data learned when building the work prediction model, the "work content" predicted from the detection information of the sensor is not the "work content" determined by a person from the workplace image. It is affixed as a label indicating "work content".

Therefore, the cost required to construct the work prediction model (in particular, the annotation cost required to generate the learning data set) is the label indicating the work content, which is determined by a human from the workplace image. Compared to the case of using "content", it can be greatly suppressed. For example, one of the above-mentioned five annotations required for the workplace image is changed from "contents determined by humans" to "contents predicted by a device such as the work estimation device from the detection information of the sensor". Since it is changed, the annotation cost can be suppressed by 20%.

Therefore, according to the above configuration, the work estimating device uses the work prediction model whose annotation cost is significantly reduced, and from the work place image, the content of the work performed by the worker in the work place. can be estimated with high precision.

In order to solve the above problems, a control method according to one aspect of the present invention is a control method for a work estimating device for estimating the content of work performed by a worker in a workplace, the workplace capturing an image of the workplace. An image or feature information generated from the workplace image is input, and the content of work performed by the worker captured in the workplace image on the work object captured in the workplace image is output. a prediction step of predicting the content of the work that the worker is performing on the work object from the workplace image or the feature information using a work prediction model that is a learned model; and a determination step of determining whether the prediction in the prediction step is correct or not using at least one of the content of the work predicted in the prediction step and the determination result of whether or not the worker is present in the workplace; an estimation step of estimating the content of the work predicted in the prediction step as the content of the work performed by the worker in the workplace when the prediction in the prediction step is determined to be correct in the step; and including.

According to the above configuration, the control method determines whether the content of the work predicted using the work prediction model is correct, the content of the work predicted without using the work prediction model, and the content of the work predicted without using the work prediction model. It is determined using at least one of the determination results of the presence or absence of the worker. Then, in the control method, when the prediction using the work prediction model is determined to be correct, the content of the work predicted using the work prediction model is changed to the work performed by the worker in the work place. presumed to be the contents of

That is, the control method determines whether the content of the work predicted using the work prediction model is correct or not, the content of the work predicted without using the work prediction model, and the presence of the worker in the work place. Verification is performed using at least one of the determination results as to whether or not to

Therefore, the control method has the effect of being able to estimate with high accuracy the content of the work performed by the worker in the workplace from the workplace image.

Advantageous Effects of Invention According to one aspect of the present invention, it is possible to estimate, with sufficient accuracy, the content of work performed by a worker on a work target from a captured image.

1 is a block diagram showing the main configuration of an information processing apparatus according to Embodiment 1 of the present invention; FIG. 1. It is a figure which shows the whole outline|summaries, such as a control system containing the information processing apparatus of FIG. FIG. 2 is a flow diagram illustrating an overall outline of processing executed by the information processing apparatus in FIG. 1 when verifying a first prediction using only a work prediction among the second predictions; FIG. 3 is a flow diagram illustrating an overall outline of processing executed by the information processing apparatus in FIG. 1 when verifying a first prediction using only presence/absence prediction out of second predictions; 2 is a flow diagram illustrating an overview of processing executed by the information processing apparatus of FIG. 1 when verifying a first prediction using both presence/absence prediction and work prediction; FIG. FIG. 10 is a diagram illustrating an example of annotations necessary for generating training data; FIG.

[Embodiment 1]
Hereinafter, an embodiment (hereinafter also referred to as "this embodiment") according to one aspect of the present invention will be described based on the drawings. The same or corresponding parts in the drawings are denoted by the same reference numerals, and the description thereof will not be repeated. In the present embodiment, the information processing device 10 will be described as a typical example of "a work estimation device for estimating the content of the work Op performed by the worker Hu in the workplace Ar".

In the following description, "n", "m", "p", "q", "x", and "y" each represent an integer of "1" or more, and "p" and "q" are integers different from each other, and "x" and "y" are integers different from each other.

§1. Application Example To facilitate understanding of the information processing apparatus 10 (work estimation apparatus) according to an aspect of the present invention, first, an environment to which the information processing apparatus 10 is applied will be described. The information processing apparatus 10 generates information (support information) that contributes to, for example, efficiency improvement of the work Op performed by the worker Hu on the work object Ob by using, for example, the analysis result of the workplace image Ic that captures the workplace Ar. . Therefore, first, the worker Hu, the work object Ob, the work Op, the workshop Ar, and the factory Fa including one or more workshops Ar will be described.

(About factory and work process)
In the factory Fa, which is a production site, for example, various products are produced through one or more work processes Pr(1), Pr(2), Pr(3), . . . Pr(n). Each of the plurality of work processes Pr(1), Pr(2), Pr(3), . , the “assembly of the main work into the main body” process, and the “inspection” process.

When it is necessary to distinguish a plurality of factories Fa from each other, a subscript such as "(A)", "(B)", "(C)", ..., "(Z)" is attached to the code. to distinguish. For example, "factory Fa (A)", "factory Fa (B)", "factory Fa (X)", ..., "factory Fa (Z)" are used for distinction. When there is no particular need to distinguish each of the plurality of factories Fa, they are simply referred to as "factory Fa".

When it is necessary to distinguish between a plurality of work processes Pr, the symbols "(1)", "(2)", "(3)", . . . , "( n)" or the like to distinguish them. For example, "work process Pr(1)", "work process Pr(2)", "work process Pr(3)", ..., "work process Pr(n)" are described for distinction. When there is no particular need to distinguish each of the plurality of work processes Pr, they are simply referred to as "work process Pr".

(About work)
In the work process Pr, the worker Hu performs one or more works Op(1), Op(2), Op(3), . . . Op(n). For example, in work process Pr(x), worker Hu performs tasks Op(x1), Op(x2), and Op(x3). For example, if the work process Pr(m) is the "painting" process, the work Op(x1) may be "grasping" the work object Ob, and the work Op(x2) may be the work object Ob. The work Op(x3) may be "painting" of the object Ob, and the work Op(x3) may be "carrying" the work object Ob.

When it is necessary to distinguish each of a plurality of work Ops, the codes are "(1)", "(2)", "(3)", ..., "(n)", etc. , and are simply referred to as "operation Op" when there is no particular need to distinguish them.

(About workshop)
Factory Fa includes one or more workplaces Ar(1), Ar(2), Ar(3), . . . Ar(n). A workplace Ar is also called a "cell", ie a factory Fa contains one or more cells. Each of the one or more workplaces Ar(1), Ar(2), Ar(3), . ), . . . Pr(n). That is, the work area Ar(m) is an area where the worker Hu performs the work Op(m) included in the work process Pr(m).

When it is necessary to distinguish between a plurality of workplaces Ar, the symbols "(1)", "(2)", "(3)", ..., "(n)", etc. and are simply referred to as "Workplace Ar" when there is no particular need to distinguish them.

(Regarding work objects)
One or more work objects Ob, which are objects of work Op performed by the worker Hu, are arranged in the work place Ar. Ob(x3), . . . , Ob(xn) are arranged.

The work object Ob is an entity other than the worker Hu, such as a so-called "work", or a tool used by the worker Hu to perform the work Op, for example.

When it is necessary to distinguish between a plurality of work objects Ob, the symbols "(1)", "(2)", "(3)", ..., "( n)", etc., to distinguish them, and when there is no particular need to distinguish them, they are simply referred to as "work objects Ob".

(About workers)
In the factory Fa, there are one or more workers Hu who execute the work Op included in the work process Pr. For example, workers Hu(1), Hu(2), Hu(3), (n) exists. The worker Hu may be identified by, for example, a worker ID attached to the top of the cap worn by the worker Hu. Specifically, worker Hu(1) and worker Hu(2) in factory Fa are identified by worker ID(1) attached to the cap worn by worker Hu(1) and worker Hu(2). Each may be identified by the worker ID (2) attached to the cap worn by (2). Also, a certain worker Hu(1) may be regarded as a work object Ob for another worker Hu(2).

Regarding the worker Hu, when it is necessary to distinguish each of a plurality of workers Hu, the symbols "(1)", "(2)", "(3)", ..., "(n)" , etc., and are simply referred to as "worker Hu" when there is no particular need to distinguish them.

(Equipment and operation)
Factory Fa comprises one or more pieces of equipment 40(1), 40(2), 40(3), . . . 40(n). Equipment 40 includes an input device 41 such as a sensor and an output device 42 such as an actuator.

The output device 42 is a device whose output is controlled by the PLC 20 . The worker Hu may use the output of the output device 42 when performing the work Op. Specifically, the worker Hu uses the action Ac executed by the output device 42 to perform the work Op. you can go That is, the worker Hu may use the output device 42 as a tool when performing the work Op. In other words, the output device 42 is used by the worker Hu when the worker Hu performs the work Op. It can be a tool. For example, when the worker Hu performs the work Op(x) included in the work process Pr(x), the output device 42(x) may be used. The operator Hu may perform the operation Op(x) using the operation Ac(x) performed by .

The input device 41 is a sensor that detects at least one state of a tool used in the work Op (for example, the output device 42 and a tool 90 other than the output device 42 to be described later), the work object Ob, and the work place Ar ( detection device). For example, the input device 41 can be used for power consumption of a tool, grasping of the tool/work object Ob by the worker Hu, pressing of a button provided by the tool/workplace Ar/workpiece Ob, etc. by the worker Hu,・Detect opening and closing of an opening/closing mechanism provided on the work object Ob and the like.

Specifically, the input device 41 is, for example, a current detection sensor, and the input device 41 is used when the worker Hu performs "work Op(x): screw tightening work". Current may be sensed. The input device 41 may be, for example, a gripping sensor, and may detect that the operator Hu has gripped the "instrument 90(x): electric screwdriver". Furthermore, the input device 41 is, for example, a start/end button, and may detect pressing by the operator Hu. Further, the input device 41 is, for example, a photoelectric sensor, and may detect opening and closing of an opening/closing mechanism included in the tool, work area Ar, work object Ob, and the like.

The input device 41 periodically outputs information indicating the detected state (hereinafter also referred to as “detection information Id”) to the PLC 20 via the control network 50 . In particular, the input device 41 includes "the time (detection time Td) at which at least one state of the tool used in the work Op, the work object Ob, and the work place Ar is detected" in the detection information Id, and outputs the information to the PLC 20. Output periodically.

Regarding the equipment 40, when it is necessary to distinguish each of the plurality of equipment 40, the symbols "(1)", "(2)", "(3)", ..., "(n)", etc. and are simply referred to as "equipment 40" when there is no particular need to distinguish between them. Similarly, for the input devices 41, when it is necessary to distinguish between the plurality of input devices 41, the symbols "(1)", "(2)", "(3)", ..., " (n)” and the like, and are simply referred to as “input device 41” when there is no particular need to distinguish them. In addition, when it is necessary to distinguish between the plurality of output devices 42, the symbols "(1)", "(2)", "(3)", . . . , "( n)” or the like to distinguish them, and when there is no particular need to distinguish them, they are simply referred to as “output devices 42”.

Furthermore, when it is necessary to distinguish between a plurality of actions Ac, the symbols "(1)", "(2)", "(3)", . . . , "(n)" , etc., and are simply referred to as "action Ac" when there is no particular need to distinguish between them.

Also, "at least one state of the tool (eg, output device 42(x) or instrument 90(x)) used in the operation Op(x), the work object Ob(x), and the workplace Ar(x)" The detection information Id indicating is sometimes expressed as "detection information Id(x)".

(About tools)
In addition to the output device 42, the factory Fa may be provided with fixtures 90 (not shown) other than the output device 42. The worker Hu may use the tool 90 when performing the work Op. Specifically, the worker Hu may use the action Ac performed by the tool 90 to perform the work Op. For example, when the worker Hu performs the work Op(x) included in the work process Pr(x), the tool 90(x) may be used. The operator Hu may perform the operation Op(x) using the operation Ac(x). For example, the worker Hu uses the “operation Ac(x): rotating operation” executed by the “instrument 90(x): electric screwdriver” to perform “operation Op(x): screw tightening operation”. you can go

As described above, the state of the instrument 90 (for example, the state of being used by the worker Hu) is detected by the input device 41, and the input device 41 transmits the detected "state of the instrument 90" to the control network 50. periodically output to the PLC 20 via. That is, when the input device 41 detects the "state of the instrument 90", the input device 41 generates detection information Id including the detected "state of the instrument 90" and the time (detection time Td) at which the "state of the instrument 90" is detected. , to the PLC 20 periodically.

(About the workshop image)
The workplace image Ic is a captured image of the workplace Ar, and each of the plurality of workplace images Ic is a captured image of each of the plurality of workplaces Ar. For example, the work place image Ic is a captured image of the worker Hu who is performing the work Op in the work place Ar from the front, and the work place image Ic is a captured image generated for each work place Ar.

A plurality of workplace images Ic may be distinguished by the workplace Ar captured in each. For example, each of the plurality of workplace images Ic may be distinguished by associating each of the plurality of workplace images Ic with the workplace Ar captured in each workplace image Ic.

That is, the workplace image Ic obtained by imaging the workplace Ar(1) may be referred to as the workplace image Ic(1), and the workplace image Ic obtained by imaging the workplace Ar(2) may be referred to as the workplace image Ic(2). Similarly, the workplace image Ic obtained by imaging the workplace Ar(n) may be described as the workplace image Ic(n).

A plurality of workplace images Ic are simply referred to as "workplace images Ic" when it is not necessary to distinguish between the plurality of workplace images Ic by the workplace Ar captured in each.

(About the whole image)
While the workplace image Ic is a captured image of each workplace Ar, the overall image Ia is a captured image in which a plurality of workplaces Ar are captured. is a captured image.

First, an overview of the control system 1 including the information processing device 10 will be described below with reference to FIG.

(Overview of system including information processing device)
FIG. 2 is a diagram showing an overall overview of the control system 1 and the like including the information processing device 10. As shown in FIG. In the environment described so far, the information processing device 10 highly accurately estimates the content of the work Op performed by the worker Hu captured in the workplace image Ic from the workplace image Ic that captures the workplace Ar.

The cell camera 30 is installed in the workplace Ar (that is, cell) and generates a workplace image Ic. For example, a cell camera 30 is installed for each workplace Ar, and each of the plurality of cell cameras 30 is installed at a position where it is possible to capture an image of a worker Hu who is performing a work Op in the workplace Ar from substantially the front for each workplace Ar. be. The cell camera 30 outputs the generated workplace image Ic to the information processing device 10, and in particular, outputs the workplace image Ic to the information processing device 10 together with the imaging time Tp that is the time when the workplace Ar is imaged.

When distinguishing the cell cameras 30 installed for each work area Ar in the factory Fa, each of the plurality of cell cameras 30 is associated with the work area Ar in which each cell camera 30 is installed. 30 may be distinguished. For example, the cell camera 30 installed at work place Ar(1) is described as cell camera 30(1), and the cell camera 30 installed at work place Ar(2) is described as cell camera 30(2). good too. Similarly, the cell camera 30 installed in the workplace Ar(n) may be referred to as cell camera 30(n).

The ceiling camera 80 is installed, for example, on the ceiling of the factory Fa, and generates an overall image Ia that obliquely looks down on a plurality of workplaces Ar. Generate. However, it is not essential that one ceiling camera 80 is installed on the ceiling of the factory Fa, and each of the plurality of ceiling cameras 80 installed on the ceiling of the factory Fa captures a plurality of workplaces Ar in the factory Fa. A full image Ia may be generated. In other words, the ceiling camera 80 only needs to be able to generate an overall image Ia of a plurality of workplaces Ar.

In the following description, one ceiling camera 80, which is a wide-area imaging camera, is installed on the ceiling of the factory Fa, and the one ceiling camera 80 generates an overall image Ia of the entire factory Fa. An example is given. The ceiling camera 80 outputs the generated overall image Ia to the information processing device 10, and in particular, together with the overall time Ta that is the time when the entire factory Fa was captured (that is, the time when the overall image Ia was generated) The image Ia is output to the information processing device 10 .

(Prediction of work content by image analysis of workplace images)
The information processing device 10 is included in the control system 1 and is communicably connected to the cell camera 30 via, for example, a USB (Universal Serial Bus) cable. The information processing apparatus 10 acquires a workplace image Ic of the workplace Ar from the cell camera 30, performs image analysis on the acquired workplace image Ic, and determines whether the worker Hu captured in the workplace image Ic is working. The content of the work Op performed on the object Ob is predicted. The information processing apparatus 10 uses a HOI (Human Object Interaction) detection algorithm to extract from the workplace image Ic "the details of the work Op performed by the worker Hu captured in the workplace image Ic with respect to the work object Ob". Predict. In particular, the information processing apparatus 10 acquires the workplace image Ic and the imaging time Tp, which is the time when the workplace image Ic was generated, from the cell camera 30, and predicts the following from the acquired workplace image Ic and the imaging time Tp: to run. That is, the information processing apparatus 10 predicts "the details of the work Op that the worker Hu imaged in the workplace image Ic is performing on the work object Ob in the workplace Ar at the imaging time Tp". The "contents of the work Op performed by the worker Hu on the work object Ob" predicted from the workplace image Ic is particularly described as "at the imaging time Tp, the worker Hu is doing on the work object Ob in the workplace Ar. The content of the work Op that is being performed" is also referred to as the first prediction Pf.

For example, the information processing apparatus 10 performs image analysis on a workplace image Ic(x) that is an image of the workplace Ar(x). ), the content of the operation Op(x) being performed is predicted. In particular, the information processing apparatus 10 determines from the workplace image Ic(x) that "at the imaging time Tp at which the workplace image Ic(x) was captured, the worker Hu(x) was the work object Ob in the workplace Ar(x). The content of the work Op(x) performed for (x) is predicted. Predicted from the workplace image Ic(x), "at the imaging time Tp at which the workplace image Ic(x) was captured, the worker Hu(x) performed the work object Ob(x) in the workplace Ar(x). The content of the work Op(x) currently being executed may be particularly referred to as a "first prediction Pf(x)".

(Prediction of worker presence/absence by image analysis of the entire image)
The information processing device 10 is included in the control system 1 and is communicably connected to the ceiling camera 80 via, for example, a USB (Universal Serial Bus) cable. The information processing device 10 acquires an overall image Ia of the entire factory Fa from the ceiling camera 80, performs image analysis on the acquired overall image Ia, and determines whether the worker Hu is in the workplace Ar. Predict what The information processing apparatus 10 uses a flow line detection algorithm (object detection algorithm) such as YOLO, for example, from the overall image Ia to determine whether there is a worker Hu in the workplace Ar captured in the overall image Ia. Predict. In particular, the information processing apparatus 10 acquires the overall image Ia and the overall time Ta, which is the time when the overall image Ia was generated, from the ceiling camera 80, and from the acquired overall image Ia and the overall time Ta, the following predictions are made: to run. That is, the information processing apparatus 10 predicts "whether the worker Hu is in the workplace Ar (in particular, the workplace Ar captured in the workplace image Ic) at the overall time Ta".

Therefore, the information processing apparatus 10 predicts "whether or not the worker Hu is present in the workplace Ar (in particular, the workplace Ar captured in the workplace image Ic) at the overall time Ta corresponding to the imaging time Tp". can be done.

The presence or absence of the worker Hu in the workplace Ar predicted from the overall image Ia, in particular, the prediction of the presence or absence of the worker Hu in the workplace Ar at the overall time Ta corresponding to the imaging time Tp. Also referred to as predicted Pa.

For example, the information processing device 10 detects "whether there is a worker Hu(x) in the workplace Ar(x) imaged in the workplace image Ic(x) generated by the cell camera 30(x)." is predicted by image analysis for the entire image Ia generated by . In particular, the information processing apparatus 10 determines from the overall image Ia that, at the overall time Ta corresponding to "the imaging time Tp at which the workplace image Ic(x) was captured", the worker Hu(x) Predict whether you will stay in Prediction of "Whether worker Hu(x) is in workplace Ar(x) at overall time Ta corresponding to 'imaging time Tp at which workplace image Ic(x) was captured'" predicted from overall image Ia is particularly referred to as "presence/absence prediction Pa(x)".

(Prediction of work content using detection information)
In addition, the information processing apparatus 10 acquires, from a PLC (Programmable Logic Controller) 20, process information Ip generated from an operation result La indicating the content and result of the operation Ac executed by the device 40 whose operation Ac is controlled by the PLC 20. do. In the process information Ip, for example, at least one of “the time when the device 40 performed the action Ac” and “the time when the input device 41 or the like detected the execution of the action Ac by the device 40” is the detection time Td. included as

The information processing apparatus 10 uses the process information Ip acquired from the PLC 20 to grasp the state of the output device 42 (for example, the operation state including the contents and results of the operation Ac), and from the grasped state, the worker Hu determines the output device 42 is used to predict the contents of the work Op being performed. In particular, the information processing apparatus 10 grasps "the state of the output device 42 at the 'detection time Td corresponding to the imaging time Tp'" from the process information Ip including the detection time Td. Then, the information processing apparatus 10 changes the status of the output device 42 at 'detection time Td corresponding to the imaging time Tp' to the grasped 'state of the output device 42 at 'detection time Td corresponding to the imaging time Tp'. The content of the work Op performed using the device 42 is predicted.

However, the PLC 20 may grasp the state of the output device 42 from the output of the device 40 or the like, and predict the content of the work Op performed by the worker Hu using the output device 42 from the grasped state. Then, the PLC 20 may output the process information Ip including the content of the predicted work Op to the information processing device 10 . That is, it may be the PLC 20 or the information processing apparatus 10 that predicts the content of the work Op performed by the worker Hu using the output device 42 from the state of the output device 42 . The information processing apparatus 10 only needs to be able to acquire or generate “the details of the work Op performed by the worker Hu using the output device 42 ” predicted from the state of the output device 42 .

Here, the process information Ip is information indicating the state of the output device 42 and the like generated from "the output of the device 40 controlled by the PLC 20 and the control details of the output device 42 by the PLC 20". Therefore, the PLC 20 can also be regarded as a detection device that detects the state of the output device 42 from "the output of the output device 42 obtained as feedback from the output device 42 and the control details for the output device 42". Further, the process information Ip is "detection information" which is information indicating "at least one state of the tools (eg, the output device 42 and the instrument 90), the work object Ob, and the workplace Ar used in the work Op". It can also be regarded as a kind of "Id".

Furthermore, the information processing apparatus 10 receives from the PLC 20 the "at least one of the tools (for example, the output device 42 and the instrument 90) used for the work Op, the work object Ob, and the work place Ar detected by the input device 41. Acquire the detection information Id indicating "one state".

The information processing apparatus 10 may acquire the detection information Id indicating the state detected by the input device 41 directly from the input device 41 without going through the PLC 20 .

The detection information Id includes the detection time Td, which is the time when the input device 41 detects "at least one state of the tool used in the work Op, the work object Ob, and the workplace Ar".

The information processing apparatus 10 uses the detection information Id acquired from the PLC 20 to grasp the state of at least one of the tool used in the work Op, the work object Ob, and the work place Ar, and from the grasped state, the operator Hu performs. Predict the content of the work Op that is being performed.

However, based on the detection information Id acquired from the input device 41, the state of at least one of the tool, the work object Ob, and the workplace Ar is grasped, and the content of the work Op performed by the worker Hu is predicted from the grasped state. It may be the PLC 20 that does. Then, the PLC 20 may output the process information Ip including the content of the predicted work Op to the information processing device 10 . That is, it is the PLC 20 or the information processing device 10 that predicts the contents of the work Op performed by the worker Hu from at least one state of the tool, the work object Ob, and the workplace Ar. good. The information processing apparatus 10 only needs to be able to acquire or generate the "contents of the work Op performed by the worker Hu" predicted from at least one state of the tool, work object Ob, and workplace Ar.

That is, the information processing apparatus 10 acquires the detection information Id (including the process information Ip) indicating "at least one state of the tool used in the operation Op, the work object Ob, and the work place Ar" acquired from the PLC 20. do. The information processing apparatus 10 predicts “the content of the work Op performed by the worker Hu” from the acquired detection information Id, and particularly, “at the “detection time Td corresponding to the imaging time Tp”, the worker Hu The content of the work Op being performed" is predicted. However, the information processing apparatus 10 acquires from the PLC 20 the “contents of the work Op performed by the worker Hu” predicted by the PLC 20 from the detection information Id, together with the detection information Id or instead of the detection information Id. good too. For example, the information processing apparatus 10, along with the detection information Id, or instead of the detection information Id, "contents of the work Op performed by the worker Hu at 'detection time Td corresponding to the imaging time Tp'" You may acquire from PLC20.

``Details of the work Op performed by the worker Hu'' predicted from the detection information Id (including the process information Ip) are The content of the work Op that is being performed" is also referred to as a work prediction Po.

For example, the information processing apparatus 10 detects the contents of the work Op(x) performed by the worker Hu(x) in the workplace Ar(x), including detection information Id(x) (including process information Ip(x) below). ). In other words, the information processing apparatus 10 "at least one state of the tools provided in the workplace Ar(x), the work object Ob(x) arranged in the workplace Ar(x), and the workplace Ar(x). , the content of the task Op(x) is predicted from the detection information Id(x) indicating ". "A tool (for example, at least one of the output device 42(x) and the instrument 90(x)) provided in the workplace Ar(x)" is used when the worker Hu(x) performs the work Op(x). It is a tool used for In particular, the information processing apparatus 10 determines from the process information Ip(x) that "at the 'detection time Td corresponding to the image capturing time Tp', the worker Hu(x) detects the work object Ob(x) at the workplace Ar(x). ), the content of the work Op(x) performed for ) is predicted. of the work Op(x) performed by the worker Hu(x) at the "detection time Td corresponding to the imaging time Tp" predicted from the detection information Id(x) (including the process information Ip(x)). content" may be referred to as "work prediction Po(x)".

(Verification of predicted work content and output of estimation results)
The information processing apparatus 10 verifies "the content of the work Op performed by the worker Hu on the work object Ob" predicted from the workplace image Ic using at least one of the following two predictions. That is, the information processing apparatus 10 predicts the content predicted from the workplace image Ic, the prediction of "presence or absence of worker Hu" predicted from the entire image Ia, and the prediction of "worker Hu is doing" predicted from the detection information Id. It verifies using at least one of "the content of the work Op". That is, the information processing apparatus 10 verifies whether or not the first prediction Pf is correct using at least one of the presence/absence prediction Pa and the work prediction Po. ) and the work prediction Po(x).

The first prediction Pf(x) is the work performed by the worker Hu(x) on the work object Ob(x) at the work place Ar(x), which is predicted by the information processing apparatus 10 from the work place image Ic(x). The content of Op(x)” is predicted. The "first prediction Pf(x)" is, for example, predicted from the workplace image Ic(x). The contents of the work Op(x) performed on the work object Ob(x) at (x)".

The presence/absence prediction Pa(x) is a prediction of “whether or not there is a worker Hu in the workplace Ar(x)” predicted by the information processing apparatus 10 from the entire image Ia. Presence/absence prediction Pa(x) is, for example, predicted from overall image Ia when worker Hu(x) is in the workplace at overall time Ta corresponding to ``imaging time Tp at which workplace image Ic(x) was captured. Are you at Ar(x)?"

The work prediction Po(x) is a prediction of "the details of the work Op(x) performed by the worker Hu(x) at the workplace Ar(x)" predicted by the information processing apparatus 10 from the detection information Id(x). is. The work prediction Po(x) is, for example, "at the detection time Td corresponding to the imaging time Tp" predicted from the detection information Id(x). The content of the operation Op(x) that is present”.

For example, the information processing device 10 (particularly, the determination unit 150 to be described later) predicts, in the presence/absence prediction Pa(x), that ``the worker Hu is present in the workplace Ar(x)'', the ``first prediction Pf(x) is correct." More specifically, the information processing apparatus 10 predicts that the worker Hu(x) is in the workplace Ar(x) at 'the total time Ta corresponding to the imaging time Tp' as the presence/absence prediction Pa(x). If it is predicted, it is determined that "the first prediction Pf(x) is correct". Then, the information processing device 10 (in particular, the estimation unit 160 described later) outputs the first prediction Pf(x) determined to be correct by the determination unit 150 to the outside as an estimation result.

For example, when the work Op(x) in the work prediction Po(x) and the work Op(x) in the first prediction Pf(x) match, the information processing device 10 (in particular, the determination unit 150 described later) determines, One prediction Pf(x) is correct." More specifically, the information processing apparatus 10 calculates "contents of work Op(x) at 'detection time Td corresponding to imaging time Tp'" in presence/absence prediction Pa(x), first prediction Pf(x) is matched with the "contents of the work Op(x) at the imaging time Tp". After confirming that the two match, the information processing apparatus 10 determines that "the first prediction Pf(x) is correct." Then, the information processing device 10 (in particular, the estimation unit 160 described later) outputs the first prediction Pf(x) determined to be correct by the determination unit 150 to the outside as an estimation result.

For example, when the presence/absence prediction Pa(x) predicts that “the worker Hu is present in the workplace Ar(x),” the information processing apparatus 10 (in particular, the determination unit 150 described later) determines the work prediction Po(x ) matches the first prediction Pf(x). More specifically, the information processing apparatus 10 predicts that the worker Hu(x) is in the workplace Ar(x) at 'the total time Ta corresponding to the imaging time Tp' as the presence/absence prediction Pa(x). If so, confirm agreement between the work prediction Po(x) and the first prediction Pf(x). The information processing apparatus 10 calculates “contents of the work Op(x) at the detection time Td corresponding to the imaging time Tp” in the presence/absence prediction Pa(x) and “work at the imaging time Tp in the first prediction Pf(x). Op(x) content", it is determined that the first prediction Pf(x) is correct. Then, the information processing device 10 (in particular, the estimation unit 160 described later) outputs the first prediction Pf(x) determined to be correct by the determination unit 150 to the outside as an estimation result.

(About master-slave control system)
In the control system 1, the equipment 40 used for carrying out the work process Pr in the factory Fa is controlled by the PLC 20 as a line controller. That is, the control system 1 is constructed as a master-slave control system in which the PLC 20 is a master and each of the plurality of devices 40 is a slave, and each of the plurality of devices 40 can communicate with the PLC 20 via a network (control network 50). It is connected to the. PLC 20 is called a “master” in the sense that it manages data transmission over control network 50 . "Master" and "slave" are defined by focusing on the control function of data transmission on the control network 50, and what kind of information is transmitted and received between each device is not particularly limited.

The PLC 20 is a control device (controller) that controls the entire control system 1 and is connected to each of the plurality of devices 40 so as to be communicable. The PLC 20 acquires information from each of the plurality of devices 40 as input devices 41 (measurement devices) as input data. PLC20 performs the arithmetic processing using the acquired input data according to the user program incorporated previously. The PLC 20 executes the arithmetic processing to determine the control details for the control system 1, and determines the control details for each of the plurality of devices 40 as output devices 42 such as actuators, for example. Then, the PLC 20 outputs control data corresponding to the determined control content to each of the plurality of devices 40 (in particular, the output device 42). The PLC 20 acquires input data from each of the plurality of devices 40 (specifically, the input device 41) and obtains control data from each of the plurality of devices 40 (specifically, the output device 42). is repeatedly executed at a predetermined cycle (control cycle). The PLC 20 may be connected to, for example, a display unit and an operation unit (not shown). The display section is composed of a liquid crystal panel or the like capable of displaying an image, and the operation section is typically composed of a touch panel, a keyboard, a mouse, or the like.

The device 40 is a slave in the control system 1 as a master-slave control system with the PLC 20 as a master. The device 40 is an input device 41 that repeatedly transmits input data to the PLC 20 in each predetermined control cycle, or receives repeated control data from the PLC 20 in each predetermined control cycle and operates according to the received control data. It is the output device 42 .

The device 40 (in particular, the input device 41) is, for example, a sensor that transmits detection results and the like to the PLC 20 as input data. For example, the input device 41 may be a photoelectric sensor, a barcode reader that transmits read results, or an inspection machine (tester) that transmits inspection results.

The equipment 40 (specifically, the output device 42) may be a robot or the like as an output device that performs screw tightening, picking, or the like, or may be various actuators.

Also, the device 40 may be a PT (Programmable Terminal) to which a plurality of input devices 41 are connected.

The control network 50 transmits various data received by the PLC 20 or transmitted by the PLC 20. Typically, various industrial Ethernet (registered trademark) can be used, and is sometimes referred to as a field network. . Industrial Ethernet (registered trademark) includes, for example, EtherCAT (registered trademark), Profinet IRT, MECHATROLINK (registered trademark)-III, Powerlink, SERCOS (registered trademark)-III, and CIP Motion. You can use any one of them. Furthermore, a field network other than Industrial Ethernet (registered trademark) may be used. For example, if motion control is not performed, DeviceNet, CompoNet/IP (registered trademark), etc. may be used.

In this embodiment, the control system 1 in which data is transmitted and received between the PLC 20 (master) and the device 40 (slave) by sequentially transferring data frames on the control network 50 will be described. That is, data frames are sequentially transferred on the control network 50 in a predetermined control cycle, so that data is repeatedly transmitted and received between the PLC 20 and the device 40 in each control cycle. By sequentially transferring data frames on the control network 50, data may be transmitted and received between multiple devices 40, that is, between multiple slaves.

The information processing device 10 is included in the control system 1 , but the information processing device 10 may not be a slave in the control system 1 .

In the control system 1, which is a master-slave control system in which the device 40 is a slave, the PLC 20, which is the master, repeatedly receives the operation result La and the detection information Id from the device 40, which is the slave, at predetermined control cycles, for example.

(About process information)
In the control system 1, which is a master-slave control system in which the device 40 is a slave, the PLC 20, which is the master, repeatedly receives the operation result La from the device 40, which is the slave, at predetermined control cycles, for example. The action result La is information indicating "the content and result of the action Ac executed by the device 40". That is, the device 40 repeats, at a predetermined cycle, an action result La indicating the content and result of the action Ac that was actually executed when performing the work process Pr, and transmits the action result La to the PLC 20 . For example, the device 40(x) repeats the action result La(x) indicating the content and result of the action Ac(x) executed when the work process Pr(x) is performed at the control cycle, and transmits it to the PLC 20. do.

The PLC 20 acquires, as the operation result La of the device 40, the measurement result, which is the result of the measurement operation performed by the device 40 as the input device 41 (measurement device), for example. In addition, when the device 40 is an inspection machine, the PLC 20 displays the result of the inspection operation performed by the device 40, for example, the inspection result such as "the inspection standard was met or not met", and the operation result of the device 40. Obtained as La. Furthermore, the PLC 20 acquires, for example, the result of the output operation performed by the device 40 as the output device 42 as the operation result La of the device 40 . If the device 40 is a robot that performs screw tightening, picking, etc., the PLC 20 acquires operation results La such as screw tightening times and picking results (picking success or picking error) as the operation results La of the device 40 .

The PLC 20 repeatedly receives from the device 40, at predetermined intervals, an action result La indicating the content and result of the action Ac actually executed by the device 40 in performing the work process Pr, and stores the received action result La as process information Ip. is transmitted to the information processing apparatus 10, that is, transferred. In addition, the PLC 20 transmits information generated using the operation result La repeatedly received from the device 40 at a predetermined cycle to the information processing apparatus 10 as the process information Ip.

Furthermore, the PLC 20 may transmit the operation result La repeatedly received from the device 40 at a predetermined cycle to the outside of the control system 1 as the process information Ip. For example, the PLC 20 receives the operation result La repeatedly from the device 40 at a predetermined cycle as the process information Ip, and connects it to an MES (Manufacturing Execution System) or the like. Network).

The information processing apparatus 10 identifies the operation start time Tms, the operation completion time Tme, and the operation period Da of the operation Ac executed by the equipment 40 when performing the work process Pr from the process information Ip (operation result La) acquired from the PLC 20. . Then, the information processing apparatus 10 performs various determinations on the action Ac using the specified action start time Tms, action completion time Tme, action period Da, and predetermined action reference Sa corresponding to the action Ac.

However, the PLC 20 may perform the above-described determination on the operation result La (particularly, the operation Ac) included in the process information Ip, and the PLC 20 includes the above-described determination result in the process information Ip, or It may be transmitted to the information processing device 10 instead of the process information Ip.

In the above description, the action start time Tms is the point in time when the equipment 40 used in the work process Pr starts executing the action Ac in performing the work process Pr, and the action completion time Tme is the time when the action Ac is performed. It is the point at which the execution is completed. The operation period Da is a period from the operation start time Tms to the operation completion time Tme.

As described above, the process information Ip can also be regarded as information indicating "the state of the tool (in particular, the output device 42) used in the work Op". The process information Ip can also be regarded as information indicating, for example, "the state of the tool (in particular, the output device 42) used in the work Op at the 'detection time Td corresponding to the imaging time Tp'". Therefore, the process information Ip is a kind of detection information Id indicating "at least one state of the tool (for example, the output device 42 and the instrument 90) used in the work Op, the work object Ob, and the work place Ar." I can understand.

(About detection information)
In the control system 1, the PLC 20 receives the detection information Id from the device 40 (particularly, the input device 41), which is a slave, repeatedly, for example, every predetermined control period. The detection information Id includes at least one of the “tools used in the work Op (for example, the output device 42 and instruments 90 other than the output device 42 described later), the work object Ob, and the work place Ar detected by the input device 41. It is information indicating "one state". That is, the device 40 (particularly, the input device 41) repeats "at least one state of the tool used for the work Op by the worker Hu, the work object Ob, and the work place Ar" at a predetermined cycle, and sends to the PLC 20 and send. The detection information Id includes information indicating the time when the input device 41 detected "at least one state of the tool used in the operation Op, the work object Ob, and the workplace Ar" (that is, the detection time Td). .

The PLC 20, for example, receives information indicating the magnitude of current flowing from the device 40(y) as the input device 41(y) to the device 90(x) (such as the power consumption of the device 90(x)), and outputs the information to the device 90( Acquired as detection information Id(x) for x).

The PLC 20 receives, for example, information indicating whether or not the instrument 90(x) has been gripped by the worker Hu from the device 40(y) as the input device 41(y) as detection information Id for the instrument 90(x). (x) is obtained.

For example, the PLC 20 receives the information indicating the opening/closing of the opening/closing mechanism of the work object Ob(x) from the device 40(y) as the input device 41(y). (x) is obtained.

For example, the PLC 20 transmits information indicating whether or not a button provided to the work object Ob(x) has been pressed from the device 40(y) as the input device 41(y) to the work object Ob(x). Acquired as detection information Id(x).

For example, the PLC 20 receives information indicating opening/closing of an opening/closing mechanism provided in the workplace Ar(x) from the device 40(y) as the input device 41(y) as detection information Id(x) for the workplace Ar(x). ,get.

For example, the PLC 20 receives information indicating whether or not a button provided in the workplace Ar(x) has been pressed from the device 40(y) as the input device 41(y) as detection information Id(x) for the workplace Ar(x). x).

The PLC 20 repeatedly receives the detection information Id from the input device 41 at a predetermined cycle, and transmits, that is, transfers the received detection information Id to the information processing apparatus 10 .

Furthermore, the PLC 20 may transmit the detection information Id repeatedly received from the input device 41 at predetermined intervals to the outside of the control system 1, like the process information Ip. For example, the PLC 20 transmits the detection information Id repeatedly received from the input device 41 at a predetermined cycle to an in-house LAN (Local Area Network) shown in FIG. 2 connected to an MES (Manufacturing Execution System) or the like. may

From the detection information Id acquired from the PLC 20, the information processing apparatus 10 determines "at least one state of the tool used for the work Op by the worker Hu, the work object Ob, and the work place Ar", and the state of the input device 41. detection time Td, which is the time when the is detected. Then, the information processing apparatus 10 determines the work at the detection time Td based on the specified detection time Td and "at least one state of the tool, the work object Ob, and the work place Ar used by the worker Hu in the work Op". The content of the work Op performed by the person Hu is predicted. That is, the information processing apparatus 10 predicts the content of the work Op performed by the worker Hu at the detection time Td from the detection information Id acquired from the PLC 20 .

However, the PLC 20 may execute the process of predicting the content of the work Op performed by the worker Hu at the detection time Td from the detection information Id. In that case, the PLC 20 may include the prediction result in the detection information Id or may transmit it to the information processing apparatus 10 instead of the detection information Id.

(For systems and devices other than master-slave control systems)
FIG. 2 shows an in-house LAN system, other network systems, etc. in addition to the control system 1 as a master-slave control system. The in-house LAN is connected to a process information DB (Database), which is also called MES. In the process information DB, information indicating "standard actions to be performed by the equipment 40 used for carrying out the work process Pr" is stored as the action standard Sa.

In the example shown in FIG. 2, an event management device 60 that monitors and manages various events occurring in the factory Fa is connected to the process information DB as the MES via an in-house LAN. However, it is not essential that the event management device 60 is connected to the process information DB via the in-house LAN, and the event management device 60 may not be provided.

Furthermore, the PLC 20 is connected to the process information DB via the in-house LAN. Although not shown, the process information DB and the information processing device 10 may be connected. In addition to the MES, an ERP (Enterprise Resources Planning), a WMS (Warehouse Management System), etc. (not shown) may be connected to the in-house LAN.

In FIG. 2, the process information DB is connected to a moving image storage server or the like via an "other network" different from the control network 50 and the in-house LAN. The information processing device 10 is connected to the video storage server or the like via another network, and the workplace image Ic and the overall image Ia transmitted from the information processing device 10, the contents of the work Op estimated by the information processing device 10, In addition, support information and the like are stored in a video storage server or the like. An external device 70 implemented by a PC (Personal Computer) or the like is connected to the moving image storage server or the like. The external device 70 displays, for example, the workplace image Ic, the overall image Ia, the content of the work Op estimated by the information processing device 10, support information, and the like, and visualizes the process information Ip and the like. That is, the external device 70 displays a list of information necessary for improving the work process Pr, and displays information indicating the bottleneck work process Pr, the date and time of an error that occurred in the work process Pr, and the like in the corresponding work place image. It is displayed in association with Ic.

As described above, the cell camera 30 captures the workplace Ar to generate the workplace image Ic, and transmits the generated workplace image Ic to the information processing apparatus 10 via a communication cable such as a USB (Universal Serial Bus) cable, for example. Send to

In addition, the ceiling camera 80 captures a bird's-eye view of the entire factory Fa to generate an overall image Ia, and transmits the generated overall image Ia to the information processing apparatus 10 via a communication cable such as a USB cable, for example. Send.

The information processing device 10 is realized by, for example, a PC or the like, and combines process information Ip acquired from the PLC 20 and a workplace image Ic acquired from the cell camera 30, and is a data extraction device that enables efficient use of both. may be The information processing apparatus 10 generates process information Ip including "operation results La indicating the contents and results of the actual operations Ac of the plurality of devices 40 in the factory Fa" based on the "workplace image Ic" and the "work estimated from the workplace image Ic". The work process Pr is visualized by combining with "Contents of Op". The information processing apparatus 10 visualizes the work process Pr by combining the process information Ip acquired from the PLC 20, the workplace image Ic acquired from the cell camera 30, and the content of the work Op estimated from the workplace image Ic. , to efficiently and precisely analyze the work process Pr.

For example, in addition to visualizing the process information Ip, the information processing apparatus 10 extracts the work process Pr that is a bottleneck, and creates a workplace image Ic that captures the implementation status of the extracted work process Pr, and the extracted work process Pr. is combined with the process information Ip of Therefore, for example, the user can easily identify the cause of the bottleneck and the cause of the defect in the work process Pr that is a bottleneck, and can efficiently perform the improvement work of the work process Pr. become able to. In addition, the information processing apparatus 10 can also be used for traceability when a defect occurs.

Further, the information processing apparatus 10 combines the workplace image Ic and the process information Ip of each of the plurality of devices 40, thereby improving the analysis accuracy of both the analysis of the workplace image Ic and the analysis of the process information Ip. can be done. For example, the information processing apparatus 10 uses the process information Ip (operation result La) acquired from the PLC 20 to specify the operation start time Tms at which the device 40 started the operation Ac and the operation completion time Tme at which the operation Ac was completed. do. The information processing apparatus 10 analyzes the workplace image Ic captured during the operation period Da, which is the period from the operation start time Tms to the operation completion time Tme. The work Op of the person Hu can be associated with high accuracy.

Further, the information processing device 10 acquires from the process information DB an operation standard Sa indicating "a standard operation that the equipment 40 used to perform the work process Pr should perform". Then, the information processing apparatus 10 uses the acquired action standard Sa to perform a determination on the "actual action Ac performed by the device 40 used to perform the work process Pr". The information processing apparatus 10 may analyze the workplace image Ic that captures the implementation status of the work process Pr corresponding to the action Ac determined to be different from the standard action indicated by the action standard Sa.

(About estimation of work content)
The information processing device 10 analyzes the workplace image Ic and predicts the content of the work Op performed by the worker Hu on the work object Ob. In particular, the information processing apparatus 10 detects, from the workplace image Ic(x) obtained by imaging the workplace Ar(x), "At the imaging time Tp at which the workplace image Ic(x) was imaged, the worker Hu(x) The content of the work Op(x) performed on the work object Ob(x) at (x) is predicted. The work Op performed by the worker Hu in the workplace Ar is generally the work Op performed on the work object Ob. Therefore, the work Op can be regarded as an interaction between the worker Hu and the work object Ob for a combination (pair) of the worker Hu and the work object Ob.

Here, conventionally, a method of detecting HOI (Human Object Interaction), which is an interaction between a person imaged in the workplace image Ic and an object imaged in the workplace image Ic, from the workplace image Ic has been researched. It is For example, HOI detection techniques are known that localize both humans and objects and identify interactions between them.

The information processing apparatus 10 can use the HOI detection algorithm to estimate the content of the work Op performed by the worker Hu on the work object Ob. That is, the information processing apparatus 10 converts the content of the work Op performed by the worker Hu on the work object Ob to the HOI for the "pair of the worker Hu and the work object Ob" captured in the workplace image Ic. can be predicted using a HOI detection algorithm that detects .

However, it is difficult to say that the accuracy of prediction using the HOI detection algorithm is sufficiently high. That is, from the workplace image Ic, the prediction accuracy of the HOI detection algorithm that predicts the content of the work Op performed by the worker Hu captured in the workplace image Ic with respect to the work object Ob captured in the workplace image Ic is , not high enough. In particular, it is known that when the following phenomenon occurs in the workplace image Ic, the accuracy of HOI detection for that workplace image Ic generally tends to decrease.

That is, the accuracy of HOI detection for a workplace image Ic in which "occlusion" occurs, in which an obstacle or the like in front completely or partially hides at least one of the worker Hu and the work object Ob, generally decreases. known to have a tendency

In addition, when the size (scale) of at least one of the worker Hu and the work object Ob imaged in the workplace image Ic is (extremely) small or large, the accuracy of HOI detection for that workplace image Ic is generally known to decline.

The accuracy of HOI detection for a workplace image Ic in which all or part of at least one of the worker Hu and the work object Ob has occurred, for example, jumping out of the imaging range and thus not being captured, generally decreases. known to have a tendency

It is known that when at least one of the worker Hu and the work object Ob is wholly or partly transparent, the accuracy of HOI detection for the workplace image Ic generally tends to decrease.

Further, it is known that the accuracy of HOI detection for the workplace image Ic, in which the work object Ob is imaged at an angle, size, etc. that is not normally imaged, generally tends to decrease. For example, when the angle (rotation), size, etc. of the work object Ob imaged in the workplace image Ic are of a rare type, it is known that the accuracy of HOI detection for that workplace image Ic generally tends to decrease. It is

It is also known that the accuracy of HOI detection for the workplace image Ic generally tends to decrease when the lighting at the time of imaging is too bright or too dark.

In the following description, for HOI detection (that is, prediction of the content of the work Op), the information processing apparatus 10 uses a bounding box to place the worker Hu and the work object Ob into the work place image Ic An example of detecting from will be described. However, it is not essential for the information processing device 10 to use the bounding box to detect the worker Hu and the work object Ob. The information processing apparatus 10 may use “Mask R-CNN” to perform a classification task (segmentation) of the worker Hu and the work object Ob for the workplace image Ic. Further, the information processing apparatus 10 may detect the worker Hu and the work object Ob from the workplace image Ic by using the Attention mechanism used in deep learning at the same time as "Mask R-CNN". The information processing apparatus 10 may improve the prediction accuracy of HOI detection using "Mask R-CNN", the Attention mechanism, or the like.

(Verification of HOI detection results)
As mentioned above, the accuracy of prediction using the HOI detection algorithm is far from high enough. Therefore, the information processing apparatus 10 converts the contents of the HOI (that is, the work Op) predicted using the HOI detection algorithm (that is, the first prediction Pf(x)) into the presence/absence prediction Pa(x) and the work prediction Po(x). ) to verify using at least one of

The information processing apparatus 10 uses the HOI detection algorithm in combination with the "flow line detection algorithm such as YOLO" and the "detection information Id from the input device 41 or the like", so that the estimation of the "work Op content" is robust. improve accuracy and estimation accuracy. That is, the information processing apparatus 10 uses at least one of the presence/absence prediction Pa by the flow line detection algorithm and the work prediction Po predicted from the detection information Id as a noise filter for the first prediction Pf (HOI detection result) by the HOI detection algorithm. use.

The information processing device 10 is a work estimation device that estimates the "contents of the work Op" using at least one of the "flow line detection algorithm" and the "detection information Id from the input device 41 or the like" in addition to the HOI detection algorithm. is.

(Reduction of annotation cost)
The work prediction model 171 as a learned model used in the HOI detection algorithm is, for example, a set of teacher data Dt1 labeled below for the workplace image Ic or its feature information (learning data set DS1). Is built by learning. That is, a set of teacher data Dt1 labeled with "contents of the work Op performed by the worker Hu imaged in the workplace image Ic on the work object Ob imaged in the workplace image Ic" is used. The work prediction model 171 is constructed through machine learning. The “label” attached to the workplace image Ic or its feature information in the training data Dt1 is also called “annotation” or “metadata”.

In the present embodiment, the “contents of the work Op performed by the worker Hu imaged in the workplace image Ic on the work object Ob imaged in the workplace image Ic” attached as a label is the detection information Id is the work prediction Po predicted from That is, supervised learning for a set of teacher data Dt1 in which ``the content of the work Op performed by the worker Hu'' predicted from the detection information Id is attached as a label to the workplace image Ic or its feature information. The work prediction model 171 is constructed by

Generally, it is known that generation of a large amount of teacher data Dt1 required for constructing a trained model is very time-consuming, and so-called annotation cost is known to be extremely high. For example, for each of a large number of workplace images Ic, a human is provided with "the details of the work Op that the worker Hu imaged in the workplace image Ic is doing with respect to the work object Ob imaged in the workplace image Ic". If you want to make a judgment, the cost required for it is extremely high.

Therefore, in the present embodiment, the "content of the work Op performed by the worker Hu" predicted from the detection information Id is used as the "label" attached to the workplace image Ic or its characteristic information. As a result, the cost (annotation cost) required to generate a large amount of teaching data Dt1 can be suppressed, that is, the cost required for constructing the task prediction model 171 can be suppressed. Therefore, the information processing apparatus 10 can realize highly accurate and robust estimation of the “contents of the work Op” using the work prediction model 171 constructed at low cost.

(Arrangement of the information processing apparatus according to the present embodiment)
The contents described so far with reference to FIG. 2 can be organized as follows. That is, the information processing device 10 is a work estimation device that estimates the content of the work Op performed by the worker Hu in the workplace Ar, and includes a first prediction unit 120 (prediction unit), a determination unit 150, and an estimation unit 160 and.

Using the work prediction model 171, the first prediction unit 120 uses the work site image Ic or the feature information generated from the work site image Ic to determine the work Op performed by the worker Hu on the work object Ob. Predict "content". The work prediction model 171 is a trained model that receives as input "the workplace image Ic that captures the workplace Ar" or "feature information generated from the workplace image Ic" and outputs the following information. That is, the work prediction model 171 is a learned model that outputs "the details of the work Op performed by the worker Hu imaged in the workplace image Ic on the work object Ob imaged in the workplace image Ic". is.

The determination unit 150 determines the first It determines whether the prediction by the one prediction unit 120 is correct. That is, the determination unit 150 uses at least one of the work prediction Po and the presence/absence prediction Pa to determine whether the first prediction Pf is correct.

When the determination unit 150 determines that the prediction of the first prediction unit 120 is correct, the estimation unit 160 changes the content of the work Op predicted by the first prediction unit 120 to "Worker Hu is doing at work place Ar. Contents of the work Op". That is, when it is verified that the first prediction Pf is correct using at least one of the work prediction Po and the presence/absence prediction Pa, the estimating unit 160 sets the first prediction Pf to "Worker Hu is doing in work place Ar. It is output as an estimation result for "the content of the work Op".

According to the above configuration, the information processing apparatus 10 uses at least one of the following two predictions to determine whether the content of the work Op predicted using the work prediction model 171 (that is, the first prediction Pf) is correct. do. That is, the information processing apparatus 10 predicts at least one of the content of the work Op predicted without using the work prediction model 171 (work prediction Po) and the determination result of the presence or absence of the worker Hu in the workplace Ar (presence prediction Pa). is used to determine whether the first prediction Pf is correct.

When the information processing apparatus 10 determines that the prediction using the work prediction model 171 is correct, the information processing apparatus 10 changes the content of the work Op predicted using the work prediction model 171 to "the work Op performed by the worker Hu in the work place Ar. It is assumed that the contents of

That is, the information processing apparatus 10 determines whether the content of the work Op predicted using the work prediction model 171 is correct or not, the content of the work Op predicted without using the work prediction model 171, and the worker Hu in the work place Ar. Verification is performed using at least one of the determination results of existence or non-existence.

Therefore, the information processing apparatus 10 has the effect of being able to estimate "the details of the work Op performed by the worker Hu in the workplace Ar" from the workplace image Ic with high accuracy.

The information processing apparatus 10 further includes a work prediction unit 142 (work detection unit) that predicts "the content of the work Op performed by the worker Hu in the workplace Ar" from the detection information Id. The detection information Id is information indicating "at least one state of the tool used in the work Op, the work object Ob, and the work place Ar" detected by the input device 41 (sensor).

The determination unit 150 determines that the content of the work Op predicted by the first prediction unit 120 (that is, the first prediction Pf) matches the content of the work Op that is predicted by the work prediction unit 142 (that is, the work prediction Po). Then, it is determined that the prediction of the first prediction unit 120 is correct.

According to the above configuration, the information processing apparatus 10 determines whether or not the content of the work Op predicted from the detection information Id of the input device 41 matches the content of the work Op predicted using the work prediction model 171. judge. When the information processing apparatus 10 determines that the two match, the information processing apparatus 10 estimates that the content of the work Op predicted using the work prediction model 171 is "the content of the work Op performed by the worker Hu in the work place Ar". do.

That is, the information processing apparatus 10 verifies whether the content of the work Op predicted using the work prediction model 171 is correct or not based on the content of the work Op predicted from the detection information Id of the input device 41 .

Therefore, the information processing apparatus 10 has the effect of being able to estimate "the details of the work Op performed by the worker Hu in the workplace Ar" from the workplace image Ic with high accuracy.

The information processing apparatus 10 further includes a presence/absence prediction unit 141, and when the presence/absence prediction unit 141 predicts that "the worker Hu is present in the workplace Ar," the determination unit 150 determines that the prediction of the first prediction unit 120 is correct. I judge. The presence/absence prediction unit 141 uses the presence/absence model 172 to determine, from the overall image Ia, the workplace image Ic, or the feature information generated from the overall image Ia or the workplace image Ic, "worker Hu exists in workplace Ar. Whether or not” is predicted. The presence/absence model 172 is a trained model, and its input is (A) an entire image Ia captured of a plurality of workplaces Ar including the workplace Ar, or (B) an entire workplace image Ic, or (C) an entire image Ia. Alternatively, it is feature information generated from the workplace image Ic. Moreover, the output of the presence/absence model 172 is information indicating "whether or not the worker Hu exists in the workplace Ar".

According to the above configuration, the information processing apparatus 10 uses the presence/absence model 172 to predict "whether or not the worker Hu is present in the workplace Ar". When the information processing apparatus 10 predicts that "the worker Hu exists in the workplace Ar", the content of the work Op predicted using the work prediction model 171 is changed to "the work performed by the worker Hu in the workplace Ar". Contents of Op".

In other words, the information processing apparatus 10 verifies the correctness of the contents of the work Op predicted using the work prediction model 171 by “presence/absence of the worker Hu in the workplace Ar” predicted using the presence/absence model 172 .

Therefore, the information processing apparatus 10 has the effect of being able to estimate "the details of the work Op performed by the worker Hu in the workplace Ar" from the workplace image Ic with high accuracy.

The information processing apparatus 10 may include both the work prediction unit 142 and the presence/absence prediction unit 141 . If the presence/absence prediction unit 141 predicts that the worker Hu is present in the workplace Ar and the following conditions are satisfied, the determination unit 150 determines that the prediction of the first prediction unit 120 is correct. That is, when the content of the work Op predicted by the first prediction unit 120 and the content of the work Op predicted by the work prediction unit 142 match, the determination unit 150 determines that the prediction of the first prediction unit 120 is correct. do.

According to the above configuration, the information processing apparatus 10 predicts the content of the work Op predicted from the detection information Id of the input device 41 and the presence or absence of the worker Hu in the work area Ar. Then, when the information processing apparatus 10 predicts that the worker Hu is present in the workplace Ar and the following conditions are satisfied, the prediction of the first prediction unit 120 is determined to be correct. That is, when the content predicted from the detection information Id of the input device 41 matches the content predicted using the work prediction model 171, the information processing apparatus 10 determines that the content predicted using the work prediction model 171 is correct. do.

That is, the information processing apparatus 10 determines whether the content of the work Op predicted using the work prediction model 171 is correct or not, the content of the work Op predicted from the detection information Id of the input device 41, and the work location predicted using the presence/absence model 172. It is verified by the presence or absence of the worker Hu in Ar.

Therefore, the information processing apparatus 10 has the effect of being able to estimate "the details of the work Op performed by the worker Hu in the workplace Ar" from the workplace image Ic with high accuracy.

§2. Configuration Example Next, details of the information processing apparatus 10 whose outline has been described so far will be described with reference to FIG. 1 .

FIG. 1 is a block diagram showing the main configuration of an information processing apparatus 10. As shown in FIG. The information processing apparatus 10 illustrated in FIG. 1 includes functional blocks such as a first acquisition unit 110, a first prediction unit 120, a second acquisition unit 130, a second prediction unit 140, a determination unit 150, an estimation unit 160, and , and a storage unit 170 .

The information processing apparatus 10 may include the following units in addition to the functional blocks described above. That is, the information processing apparatus 10 controls the storage (storage) of the workplace image Ic in a predetermined storage area (storage area), and the time (staying time) during which the worker Hu stays in the workplace Ar. A measurement unit that measures, an analysis unit that associates the process information Ip with the residence time, and the like may be provided. In order to ensure the simplicity of the description, the configuration of the information processing apparatus 10 that is not directly related to the present embodiment is omitted from the description and block diagrams. However, the information processing apparatus 10 may include these omitted configurations in accordance with the actual situation of implementation.

The functional blocks of the information processing apparatus 10 are stored in a storage device (storage unit 170) realized by, for example, a ROM (read only memory), an NVRAM (non-volatile random access memory), or the like. This can be realized by reading out a program stored in a RAM (random access memory) or the like (not shown) and executing the program. Devices that can be used as arithmetic units include, for example, CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating point number Processing Unit), PPU (Physics Processing Unit), a microcontroller, or a combination thereof.

First, the details of each of the first acquisition unit 110, the first prediction unit 120, the second acquisition unit 130, the second prediction unit 140, the determination unit 150, and the estimation unit 160 will be described below.

(Regarding functional blocks other than memory)
The first acquisition unit 110 acquires the workplace image Ic of the workplace Ar captured by the cell camera 30 from the cell camera 30 and outputs the acquired workplace image Ic to the first prediction unit 120 .

The first prediction unit 120 performs image analysis on the workplace image Ic captured by the cell camera 30, and determines whether the worker Hu imaged in the workplace image Ic is the work object Ob imaged in the workplace image Ic The content of the work Op being performed" is predicted. For example, the first prediction unit 120 performs model-based image analysis on the workplace image Ic, and determines that "the worker Hu captured in the workplace image Ic is the work object Ob captured in the workplace image Ic. Predict "the content of the work Op being performed for".

The first prediction unit 120, which performs model-based image analysis, is implemented as, for example, a neural network, and detects an operation Op as an HOI by executing a HOI detection algorithm. Specifically, the first prediction unit 120 refers to the storage unit 170 and acquires the work prediction model 171 . The work prediction model 171 receives the workplace image Ic as an input, and outputs "the details of the work Op performed by the worker Hu imaged in the workplace image Ic on the work object Ob imaged in the workplace image Ic". is a trained model. The first prediction unit 120 uses the work prediction model 171 acquired with reference to the storage unit 170 to determine, from the workplace image Ic, "the worker Hu captured in the workplace image Ic is captured in the workplace image Ic. "Contents of the work Op being performed for the work object Ob present" is predicted. Hereinafter, "neural network" may be abbreviated as "NN".

For example, the first prediction unit 120 uses the work prediction model 171 from the workplace image Ic(x) to generate the following first prediction Pf(x). That is, the first prediction unit 120 predicts that "the work Op performed by the worker Hu imaged in the workplace image Ic(x) on the work object Ob(x) imaged in the workplace image Ic(x) Generate a first prediction Pf(x), which is a prediction about "the content of (x)".

In particular, the first prediction unit 120 uses the imaging time Tp, which is “the time when the cell camera 30 images the workplace Ar(x) and generates the workplace image Ic(x)”, from the workplace image Ic(x) to , performs the following predictions: That is, the first prediction unit 120 predicts, from the workplace image Ic(x), "At the imaging time Tp, the worker Hu(x) is performing work Op on the work object Ob(x) in the workplace Ar(x). The content of (x)” is predicted.

The first prediction unit 120 outputs the content of the work Op predicted from the workplace image Ic using the work prediction model 171 to the determination unit 150 as the first prediction Pf. For example, the first prediction unit 120 outputs the first prediction Pf(x) generated from the workplace image Ic(x) to the determination unit 150 .

Second acquisition section 130 acquires various types of information for generating second prediction Ps from the outside, and outputs the acquired various types of information to second prediction section 140 . The second prediction Ps is a prediction for verifying (correctness of) the first prediction Pf, and includes at least one of the existence prediction Pa and the work prediction Po. The second acquisition section 130 illustrated in FIG. 1 includes a whole image acquisition section 131 and a detection information acquisition section 132 .

The overall image acquisition unit 131 acquires the overall image Ia captured by the ceiling camera 80 from a bird's-eye view of the entire factory Fa from the ceiling camera 80, and transmits the acquired overall image Ia to the second prediction unit 140. Output to the prediction unit 141 .

The detection information acquisition unit 132 acquires, from the PLC 20, detection information Id (including process information Ip) indicating "at least one state of the tool used in the operation Op, the work object Ob, and the workplace Ar." As described above, the detection information Id is, for example, "at least one state of the tool used in the work Op, the work object Ob, and the work place Ar" detected by the input device 41 as a detection device (measuring device). This is the information shown. The detection information acquisition unit 132 outputs the acquired detection information Id to the first prediction unit 120 .

The second prediction unit 140 generates a second prediction Ps including at least one of the presence/absence prediction Pa and the work prediction Po, and outputs the generated second prediction Ps to the determination unit 150 . Second prediction unit 140 includes presence/absence prediction unit 141 and work prediction unit 142 .

The presence/absence prediction unit 141 performs image analysis on the entire image Ia acquired from the second acquisition unit 130 (particularly, the entire image acquisition unit 131), and determines whether the worker Hu in the workplace Ar captured in the workplace image Ic. Presence/absence prediction Pa is generated. The presence/absence prediction unit 141 performs, for example, a model-based image analysis on the entire image Ia, and predicts "whether there is a worker Hu in the workplace Ar captured in the workplace image Ic".

The presence/absence prediction unit 141 that performs model-based image analysis is implemented as, for example, an NN, and predicts the presence/absence of worker Hu in workplace Ar by executing a flow line detection algorithm (object detection algorithm) such as YOLO. . Specifically, the presence/absence prediction unit 141 refers to the storage unit 170 and acquires the presence/absence model 172 . The presence/absence model 172 is a learned model that receives the entire image Ia and outputs "whether there is a worker Hu in the workplace Ar (more precisely, each of the plurality of workplaces Ar in the factory Fa)" as an output. The presence/absence prediction unit 141 uses the presence/absence model 172 acquired by referring to the storage unit 170 to predict “whether there is a worker Hu in the workplace Ar captured in the workplace image Ic” from the overall image Ia. For example, the presence/absence prediction unit 141 uses the presence/absence model 172 from the entire image Ia to predict whether there is a worker Hu in the workplace Ar(x) captured in the workplace image Ic(x). A prediction Pa(x) is generated.

In particular, the presence/absence prediction unit 141 determines "the time at which the ceiling camera 80 captures images of a plurality of workplaces Ar including 'the workplace Ar(x) imaged in the workplace image Ic(x)' to generate the entire image Ia". The following prediction is performed from the entire image Ia using the overall time Ta which is: That is, the presence/absence prediction unit 141 determines from the overall image Ia that the worker Hu(x) is in the workplace Ar(x) at the overall time Ta corresponding to the “imaging time Tp at which the workplace image Ic(x) was captured.” Predict whether there will be

The presence/absence prediction unit 141 uses the presence/absence model 172 from the entire image Ia for a predetermined period (for example, 5 seconds) to determine “whether there is a worker Hu in the workplace Ar captured in the workplace image Ic”. "Whether there is a worker Hu in the workplace Ar at the imaging time Tp" may be predicted. The presence/absence prediction of the worker Hu using the presence/absence model 172 by the presence/absence prediction unit 141 may fail such as false detection and non-detection. Therefore, even if the presence/absence prediction unit 141 predicts, for example, that "worker Hu is present" for only one second out of five seconds, if it predicts that "worker Hu is not present" for the remaining four seconds, The prediction that "worker Hu is present" may be regarded as an erroneous detection (that is, an erroneous prediction). Even if it is predicted that “the worker Hu is present” for only one second, if it is predicted that “the worker Hu is not present” for the remaining four seconds, the presence/absence prediction unit 141 predicts “ It may be predicted that the operator Hu is absent. In other words, the presence/absence prediction unit 141 makes a presence/absence prediction Pa that "there was no worker Hu in the 'workplace Ar captured in the workplace image Ic' during the 5 seconds" based on the entire image Ia for the 5 seconds. may be generated.

To summarize, the presence/absence prediction unit 141 only predicts that “the worker Hu was present” for a period of a predetermined percentage (eg, 50%) or more of the predetermined period from the entire image Ia of the predetermined period, Presence/absence prediction Pa indicating that “worker Hu was present” may be generated for the entire predetermined period. For example, the presence/absence prediction unit 141 predicts, for the entire image Ia for 5 seconds, only when the period during which ``the worker Hu was present'' is 2.5 seconds or more, and determines that ``the worker Hu was present during the 5 seconds. It is also possible to generate the presence/absence prediction Pa of "has".

The flow line detection algorithm used by the presence/absence prediction unit 141 to predict the presence/absence of the worker Hu is, for example, YOLO. flow line detection algorithm. The presence/absence prediction unit 141 should be able to predict "whether there is a worker Hu in the workplace Ar captured in the workplace image Ic" from the entire image Ia by the flow line detection algorithm.

The presence/absence prediction unit 141 predicts presence/absence prediction Pa (for example, presence/absence Prediction Pa(x)) is output to determination unit 150 .

The work prediction unit 142 uses the detection information Id (including the process information Ip) acquired from the second acquisition unit 130 (particularly, the detection information acquisition unit 132) to determine the “contents of the work Op performed by the worker Hu”. is predicted, that is, the work prediction Po is generated. As described above, the detection information Id is information indicating "at least one state of the tool (for example, the output device 42 and the instrument 90) used in the work Op, the work object Ob, and the work place Ar." For example, the work prediction unit 142 generates the following work prediction Po(x) from the detection information Id(x). That is, the work prediction unit 142 predicts "the details of the work Op(x) performed by the worker Hu(x) in the work place Ar(x)" from the detection information Id(x). Generate (x). Here, as described above, the detection information Id(x) includes "the tool (eg, the output device 42(x) or the instrument 90(x)) used for the work Op(x), the work object Ob(x), and , at least one state of the workplace Ar(x).

For example, the work prediction unit 142 determines that “the operation Ac(x) will be executed and the work Op(x) corresponding to the operation Ac(x) will worker Hu performed at workplace Ar(x)".

For example, the work prediction unit 142 determines, based on the "power consumption of the tool," that "the operation Ac(x) is executed and the work Op(x) corresponding to the operation Ac(x) is performed by the worker Hu at the workplace Ar(x). I went," he predicts.

For example, the work prediction unit 142 predicts that "work Op(x) was performed by worker Hu at workplace Ar(x)" because "a tool or work object Ob(x) was gripped". Predict.

For example, the work prediction unit 142 determines that “the work Op(x) is the work place Ar ( x) was performed by worker Hu".

For example, the work prediction unit 142 determines that "the work Op(x) is the work place Ar(x ) was performed by worker Hu".

As described above, the work prediction unit 142 predicts "the content of the work Op performed by the worker Hu" from the detection information Id. The content of the work Op(x) performed by the worker Hu(x) is predicted.

In particular, the work prediction unit 142 predicts "at least one state of the tool used for the work Op(x), the work object Ob(x), and the work place Ar(x)" included in the detection information Id(x). The following prediction is performed using the detection time Td, which is the time when is detected. That is, the work prediction unit 142 determines from the detection information Id(x) that the worker Hu(x) is performing Predict "contents of operation Op(x)".

The work prediction unit 142 outputs the “contents of the work Op performed by the worker Hu” predicted from the detection information Id to the determination unit 150 as the work prediction Po. For example, the work prediction unit 142 predicts the work prediction Po(x), which is "the content of the work Op(x) performed by the worker Hu(x) at the workplace Ar(x)" predicted from the detection information Id(x). is output to the determination unit 150 .

The determination unit 150 verifies (correctness of) the first prediction Pf using the second prediction Ps, that is, verifies (correctness of) the first prediction Pf using at least one of the existence prediction Pa and the work prediction Po. verify. Specifically, the determination unit 150 verifies (correctness of) the first prediction Pf(x) using at least one of the existence prediction Pa(x) and the work prediction Po(x).

As described above, the first prediction Pf(x) is predicted from the workplace image Ic(x), for example, "At the imaging time Tp, the worker Hu(x) is in the workplace Ar(x) and the work object Ob The contents of the work Op(x) performed for (x)". Presence/absence prediction Pa(x) is, for example, predicted from the entire image Ia, at the overall time Ta corresponding to the "imaging time Tp at which the workplace image Ic(x) was captured". Are you at Ar(x)? The work prediction Po(x) is, for example, the worker Hu(x ) is the content of the operation Op(x) being performed by ).

For example, if the presence/absence prediction Pa(x) predicts that “the worker Hu is present in the workplace Ar(x)”, the determination unit 150 determines that “the first prediction Pf(x) is correct”. In particular, the determining unit 150 predicts that "the worker Hu is present in the workplace Ar(x) at the imaging time Tp, which is the time when the cell camera 30 generates the workplace image Ic(x)." The first prediction Pf(x) is correct."

For example, when the work Op(x) in the work prediction Po(x) matches the work Op(x) in the first prediction Pf(x), the determination unit 150 determines that "the first prediction Pf(x) is correct." judge. In particular, the determination unit 150 selects “work Op(x) at the imaging time Tp(x) in the work prediction Po(x)” and “work at the imaging time Tp(x) in the first prediction Pf(x) Op(x)”, it is determined that “the first prediction Pf(x) is correct”.

For example, the determining unit 150 predicts that “the worker Hu is present in the workplace Ar(x)”, and the work Op(x) in the work prediction Po(x) and the work Op(x) in the first prediction Pf(x) (x), it is determined that "the first prediction Pf(x) is correct". In particular, the determination unit 150 determines that "the first prediction Pf(x) is correct" when both of the following two conditions are satisfied. That is, in the work prediction Po(x), the first condition is that "at the imaging time Tp, which is the time when the cell camera 30 generated the workplace image Ic(x), the worker Hu is present in the workplace Ar(x). It is predicted that The second condition is "the work Op(x) at the imaging time Tp(x) in the predicted work Po(x)" and "the work Op(x) at the imaging time Tp(x) in the first prediction Pf(x)". (x)".

The determination unit 150 notifies the estimation unit 160 of the first prediction Pf together with the determination result (verification result). The determination unit 150 may further notify the estimation unit 160 of the second prediction Ps (at least one of the presence/absence prediction Pa and the work prediction Po).

When the determination unit 150 determines that "the first prediction Pf(x) is correct", the estimation unit 160 sets the first prediction Pf(x) to "the content of the work Op performed by the worker Hu at the workplace Ar". and outputs the first prediction Pf(x) to the outside as an estimation result.

When the determining unit 150 determines that the first prediction Pf(x) is incorrect, the estimating unit 160 determines that there is no worker Hu at the workplace Ar(x), or the desired There is no content of the work Op(x) that can be estimated with accuracy." is output to the outside. The estimation unit 160 may output the estimation result described above to the moving image storage server or the like in FIG. 2 together with the workplace image Ic.

(Regarding memory)
The storage unit 170 is a storage device that stores various data used by the information processing device 10 . Note that the storage unit 170 stores (1) a control program executed by the information processing apparatus 10, (2) an OS program, (3) an application program for executing various functions of the information processing apparatus 10, and (4) Various data read when executing the application program may be stored non-temporarily. The above data (1) to (4) are stored in, for example, ROM (read only memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (registered trademark) (Electrically EPROM), HDD (Hard Disc Drive), and the like. Stored in non-volatile storage. The information processing device 10 may include a temporary storage unit (not shown). The temporary storage unit is a so-called working memory that temporarily stores data used for calculation, calculation results, etc. in the course of various processes executed by the information processing apparatus 10, and is a volatile memory such as a RAM (random access memory). Consists of equipment. Which data is to be stored in which storage device is appropriately determined based on the purpose of use, convenience, cost, physical restrictions, or the like of the information processing device 10 . The storage unit 170 further stores a work prediction model 171 and a presence/absence model 172 .

In the work prediction model 171, the first prediction unit 120 predicts from the work place image Ic that "the work Op performed by the worker Hu imaged in the work place image Ic on the work object Ob imaged in the work place image Ic is It is a model for predicting "content". For example, the work prediction model 171 receives the work place image Ic as an input, and "contents of the work Op performed by the worker Hu imaged in the work place image Ic on the work object Ob imaged in the work place image Ic". It is a trained model that outputs information indicating

The presence/absence model 172 is a model for the presence/absence prediction unit 141 to predict "whether there is a worker Hu in the workplace Ar captured in the workplace image Ic" from the overall image Ia. The presence/absence model 172 is a learned model that receives the entire image Ia and outputs "whether there is a worker Hu in the workplace Ar (more precisely, each of the plurality of workplaces Ar in the factory Fa)" as an output.

§3. Operation Example FIGS. 3, 4, and 5 are flow diagrams for explaining the overall outline of the processing executed by the information processing apparatus 10, respectively. The information processing apparatus 10 can highly accurately estimate "the content of the work Op performed by the worker Hu" by verifying (correct or incorrect) the first prediction Pf using the second prediction Ps.

(Processing example when verification is performed using only work prediction Po)
FIG. 3 is a flowchart for explaining the overall outline of the processing performed by the information processing apparatus 10 when verifying (correctness of) the first prediction Pf using only the work prediction Po among the second predictions Ps. As illustrated in FIG. 3, the information processing apparatus 10 (particularly, the determination unit 150) may verify (correctness of) the first prediction Pf using only the work prediction Po.

As shown in FIG. 3 , the determining unit 150 first determines whether or not “the content of the work Op performed by the worker Hu” is predicted in the work prediction Po, that is, whether or not “some work Op is performed based on the detection information Id”. (S110).

When confirming that "it was predicted that some kind of work Op was being performed from the detection information Id" (YES in S110), the determination unit 150 further adds in the first prediction Pf that "the work Op being performed by the worker Hu 'contents' is predicted. That is, the determination unit 150 confirms "whether or not it is predicted that some work Op is being performed using the work prediction model 171" (S120).

When confirming that "it was predicted that some work Op was being performed using the work prediction model 171" (YES in S120), the determination unit 150 performs the following determination. That is, the determination unit 150 determines “whether or not the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171” (S130). That is, the determination unit 150 determines whether or not the “work Op in the work prediction Po” and the “work Op(x) in the first prediction Pf” match.

When the determination unit 150 determines that "the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171" (YES in S130), the estimation The unit 160 outputs the following estimation results. That is, the estimating unit 160 outputs the content of the work Op predicted by the first acquiring unit 110 from the work place image Ic using the work prediction model 171 as an estimation result (S140), that is, the first prediction Pf(x) is output as the estimation result.

When it is confirmed that "it was not predicted that any work Op is being performed from the detection information Id" (NO in S110), that is, the work prediction unit 142 cannot predict the content of the work Op from the detection information Id. If not, the estimation unit 160 outputs the following estimation result. That is, the estimating unit 160 outputs the estimation result that "there is no content of the work Op that could be estimated with the desired accuracy" (S150).

When it is confirmed that "it was not predicted that any work Op is being performed using the work prediction model 171" (NO in S120), the estimation unit 160 outputs the following estimation result. That is, the estimating unit 160 outputs the estimation result that "there is no content of the work Op that could be estimated with the desired accuracy" (S150). That is, when the first prediction unit 120 cannot predict the content of the work Op from the workplace image Ic, the estimation unit 160 outputs the estimation result that "there is no content of the work Op that could be estimated with the desired accuracy." .

When it is determined that "the content of the work Op predicted from the detection information Id does not match the content of the work Op predicted using the work prediction model 171" (NO in S130), the estimation unit 160 performs the following: output the estimation result of That is, the estimating unit 160 outputs the estimation result that "there is no content of the work Op that could be estimated with the desired accuracy" (S150).

The information processing apparatus 10 verifies (correctness of) the first prediction Pf using the work prediction Po, that is, uses the HOI detection algorithm in combination with the "detection information Id from the input device 41 or the like". The information processing apparatus 10 includes the first prediction unit 120 that executes the HOI detection algorithm, and the work prediction unit 142 that generates the work prediction Po from the detection information Id, thereby preventing erroneous detection (prediction) by the first prediction unit 120. error). Therefore, the information processing apparatus 10 can cancel erroneous detection (prediction error) of the HOI detection algorithm and realize highly accurate estimation.

The work prediction unit 142 predicts "the content of the work Op performed by the worker Hu" from the detection information Id from the input device 41 or the like, that is, generates the work prediction Po. Then, the work prediction unit 142 using the detection information Id predicts “the content of the work Op performed by the worker Hu” rather than the prediction of “the content of the work Op performed by the worker Hu” by the first prediction unit 120 that executes the HOI detection algorithm. The prediction accuracy of "the content of the work Op that is present" is higher. That is, the prediction accuracy of the work prediction Po is higher than the prediction accuracy of the first prediction Pf.

Therefore, by using the first prediction Pf and the work prediction Po, the information processing apparatus 10 can estimate "the content of the work Op performed by the worker Hu" with high accuracy. The type of “work Op performed by worker Hu” that can be predicted by the first prediction unit 120 by executing the HOI detection algorithm is the type of “work Op performed by worker Hu” that can be predicted by the work prediction unit 142 from the detection information Id. It may be the same as the type of "Work Op" that is present, or the former may be more than the latter.

In general, HOI detection by the HOI detection algorithm (that is, prediction of "work Op content") fails if at least one of the worker Hu and the work object Ob cannot be detected (object detection) from the workplace image Ic. do. Therefore, the failure of HOI detection, that is, ``it was not predicted that some work Op was being performed using the work prediction model 171'', is referred to as ``failure to detect an object (for example, worker Hu) ( In other words, it can be regarded as "absent".

(Example of processing when verification is performed using only presence/absence prediction Pa)
FIG. 4 is a flowchart for explaining the overall outline of the processing performed by the information processing apparatus 10 when verifying (correctness of) the first prediction Pf using only the presence/absence prediction Pa of the second predictions Ps. As illustrated in FIG. 4, the information processing device 10 (in particular, the determination unit 150) may verify (correctness of) the first prediction Pf using only the presence/absence prediction Pa.

As shown in FIG. 4 , the determining unit 150 first determines whether or not the presence/absence prediction Pa predicts that “the worker Hu is present in the workplace Ar”, that is, “using the presence/absence model 172, the worker Hu is (S210).

As described above, the presence/absence prediction unit 141 predicts "whether there is a worker Hu in the workplace Ar captured in the workplace image Ic during the predetermined period" from the entire image Ia for a predetermined period (for example, 5 seconds). You may The presence/absence prediction unit 141 only predicts that "the worker Hu is present" for a period of a predetermined ratio (for example, 50%) of the predetermined period. It may be predicted that the worker Hu was in the workplace Ar where

When confirming that "it was predicted that the worker Hu was present in the workplace Ar using the presence/absence model 172" (YES in S210), the determination unit 150 further determines in the first prediction Pf that "the worker Hu is It is confirmed whether or not the content of the work Op is predicted. That is, the determination unit 150 confirms "whether or not it is predicted that some work Op is being performed using the work prediction model 171" (S220).

When determination unit 150 confirms that "it was predicted that some work Op was being performed using work prediction model 171" (YES in S220), estimation unit 160 outputs the following estimation result. do. That is, the estimation unit 160 outputs, as an estimation result, the content of the work Op predicted by the first acquisition unit 110 from the work place image Ic using the work prediction model 171 (S230), that is, the first prediction Pf(x) is output as the estimation result.

When confirming that "it was predicted that the worker Hu did not exist in the workplace Ar using the presence/absence model 172" (NO in S210), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "there is no worker Hu in the workplace Ar" (S250). That is, when the presence/absence prediction unit 141 predicts from the entire image Ia that "the worker Hu is absent in the workplace Ar captured in the workplace image Ic", the estimation unit 160 predicts that "the worker Hu is absent in the workplace Ar". and output the estimation results.

When it is confirmed that "it was not predicted that any work Op was being performed using the work prediction model 171" (NO in S220), the estimation unit 160 outputs the following estimation results. That is, the estimating unit 160 outputs the estimation result that "there is no work Op that could be estimated with desired accuracy" (S240). That is, when the first prediction unit 120 cannot predict the content of the work Op from the workplace image Ic, the estimation unit 160 outputs the estimation result that "there is no content of the work Op that could be estimated with the desired accuracy." .

The information processing apparatus 10 verifies (correctness of) the first prediction Pf using the presence/absence prediction Pa, that is, uses the HOI detection algorithm in combination with the "flow line detection algorithm such as YOLO". That is, the information processing apparatus 10 includes the first prediction unit 120 that executes the HOI detection algorithm and the presence/absence prediction unit 141 that executes the flow line detection algorithm. ) can be canceled. The flow line detection algorithm can predict (detect) only the presence or absence of the worker Hu, but the prediction accuracy of the flow line detection algorithm is higher than the prediction accuracy of the HOI detection algorithm. Therefore, by using the first prediction Pf and the presence/absence prediction Pa, the information processing apparatus 10 can estimate "the details of the work Op performed by the worker Hu" with high accuracy.

(Example of processing when verification is performed using both presence/absence prediction Pa and work prediction Po)
FIG. 5 is a flow diagram illustrating the overall outline of the process executed by the information processing apparatus 10 when verifying (correctness of) the first prediction Pf using both the presence/absence prediction Pa and the work prediction Po. As illustrated in FIG. 5, the information processing device 10 (in particular, the determination unit 150) may verify (correctness of) the first prediction Pf using both the existence prediction Pa and the work prediction Po.

As shown in FIG. 5 , the determining unit 150 first determines whether or not the presence/absence prediction Pa predicts that “the worker Hu is present in the workplace Ar”, that is, “using the presence/absence model 172, the worker Hu (S310).

As described above, the presence/absence prediction unit 141 may predict "whether the worker Hu is present in the workplace Ar captured in the workplace image Ic during the predetermined period" from the entire image Ia of the predetermined period. The presence/absence prediction unit 141 predicts that "the worker Hu will be present" for a period of a predetermined proportion or more of the predetermined period, and only when it predicts that "the worker Hu will be present" during the predetermined period, the presence/absence prediction unit 141 predicts that "the worker Hu will be present in the workplace Ar captured in the workplace image Ic during the predetermined period. It may be predicted that there was a person Hu.

After confirming that "worker Hu was predicted to be present in workplace Ar using the presence/absence model 172" (YES in S310), determination unit 150 next determines in work prediction Po that "worker Hu is It is confirmed whether or not the "contents of the work Op that is being performed" is predicted. That is, the determination unit 150 confirms "whether or not it was predicted from the detection information Id that some work Op was being performed" (S320).

When confirming that "it was predicted that some work Op was being performed from the detection information Id" (YES in S320), the determination unit 150 determines in the first prediction Pf that "the work Op performed by the worker Hu Check whether "content" is expected. That is, the determination unit 150 confirms "whether or not it is predicted that some work Op is being performed using the work prediction model 171" (S330).

After confirming that "it was predicted that some work Op was being performed using the work prediction model 171" (YES in S330), the determination unit 150 performs the following determination. That is, the determination unit 150 determines “whether or not the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171” (S340). That is, the determination unit 150 determines whether or not the “work Op in the work prediction Po” and the “work Op(x) in the first prediction Pf” match.

When the determination unit 150 determines that "the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171" (YES in S340), the estimation The unit 160 outputs the following estimation results. That is, the estimating unit 160 outputs the contents of the work Op predicted by the first acquiring unit 110 from the work place image Ic using the work prediction model 171 as an estimation result (S350), that is, the first prediction Pf(x) is output as the estimation result.

When confirming that "it was predicted that the worker Hu did not exist in the workplace Ar using the presence/absence model 172" (NO in S310), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "there is no worker Hu in the workplace Ar" (S360). That is, when the presence/absence prediction unit 141 predicts from the entire image Ia that "the worker Hu is absent in the workplace Ar captured in the workplace image Ic", the estimation unit 160 predicts that "the worker Hu is absent in the workplace Ar". and output the estimation results.

When it is confirmed that "it was not predicted that any work Op is being performed from the detection information Id" (NO in S320), that is, the work prediction unit 142 cannot predict the content of the work Op from the detection information Id. If not, the estimation unit 160 outputs the following estimation result. That is, the estimating unit 160 outputs the estimation result that "there is no work Op content that could be estimated with desired accuracy" (S370).

When it is confirmed that "it was not predicted that any work Op is being performed using the work prediction model 171" (NO in S330), the estimation unit 160 outputs the following estimation result. That is, the estimating unit 160 outputs the estimation result that "there is no work Op content that could be estimated with desired accuracy" (S370). That is, when the first prediction unit 120 cannot predict the content of the work Op from the workplace image Ic, the estimation unit 160 outputs the estimation result that "there is no content of the work Op that could be estimated with the desired accuracy." .

When it is determined that "the content of the work Op predicted from the detection information Id does not match the content of the work Op predicted using the work prediction model 171" (NO in S340), the estimation unit 160 performs the following: output the estimation result of That is, the estimating unit 160 outputs the estimation result that "there is no work Op content that could be estimated with desired accuracy" (S370).

As described above, events that can be predicted from the detection information Id (including the process information Ip) are different from events that can be predicted by the flow line detection algorithm. Specifically, the work prediction Po generated by the work prediction unit 142 from the detection information Id is a prediction of "the content of the work Op performed by the worker Hu". On the other hand, the presence/absence prediction Pa generated by the presence/absence prediction unit 141 using the flow line detection algorithm is a prediction about "whether the worker Hu is present in the workplace Ar captured in the workplace image Ic".

Therefore, the information processing apparatus 10 (in particular, the determination unit 150) can use at least one of the work prediction Po and the presence/absence prediction Pa to verify (correctness of) the first prediction Pf. Specifically, as shown in FIG. 3, the determination unit 150 may verify (correctness of) the first prediction Pf using only the work prediction Po, or as shown in FIG. You may verify (correctness of) 1st prediction Pf only using.

Further, the determination unit 150 may verify (correct or not) the first prediction Pf using both the presence/absence prediction Pa and the work prediction Po. In particular, as shown in FIG. 5, the determination unit 150 first filters the presence or absence of the worker Hu by the presence/absence prediction Pa, and when it is predicted that the worker Hu exists, further uses the work prediction Po to perform the first Prediction Pf (correctness) may be verified.

The processing executed by the information processing apparatus 10, which has been described with reference to FIGS. 3, 4, and 5, can be organized as follows. That is, the control method executed by the information processing device 10 is a control method for a work estimation device that estimates "the details of the work Op performed by the worker Hu in the work place Ar". The control method executed by the information processing apparatus 10 includes prediction steps (S120, S220, S330), determination steps (S130, S210, S310, S340), and estimation steps (S140, S230, S350).

In the prediction step, using the work prediction model 171, "contents of the work Op performed by the worker Hu on the work object Ob" is determined from "the work place image Ic or the feature information generated from the work place image Ic". Predict. The work prediction model 171 is a trained model that receives as input "the workplace image Ic that captures the workplace Ar" or "feature information generated from the workplace image Ic" and outputs the following information. That is, the work prediction model 171 is a learned model that outputs "the details of the work Op performed by the worker Hu imaged in the workplace image Ic on the work object Ob imaged in the workplace image Ic". is.

The judgment step uses at least one of the content of the work Op predicted without using the work prediction model 171 and the judgment result as to whether or not the worker Hu is present in the work place Ar to determine the prediction in the prediction step. Judge right or wrong. That is, the determination step uses at least one of the work prediction Po and the presence/absence prediction Pa to determine whether the first prediction Pf is correct.

In the estimation step, when it is determined that "the prediction in the prediction step is correct" in the determination step, the content of the work Op predicted in the prediction step is changed to "the content of the work Op performed by the worker Hu in the workplace Ar. content”. That is, in the estimation step, when at least one of the work prediction Po and the presence/absence prediction Pa is used to verify that the first prediction Pf is correct, the first prediction Pf is regarded as "the work performed by the worker Hu at the workplace Ar. The content of Op" is output as an estimation result.

According to the above configuration, the control method determines whether the content of the work Op predicted using the work prediction model 171 is correct, the content of the work Op predicted without using the work prediction model 171, and the content of the work Op predicted without using the work prediction model 171. At least one of the determination results of whether or not the worker Hu is present is used for determination. When the prediction using the work prediction model 171 is determined to be correct, the control method changes the content of the work Op predicted using the work prediction model 171 to "the work performed by the worker Hu in the work place Ar. Contents of Op".

That is, the control method determines whether the content of the work Op predicted using the work prediction model 171 is correct or not, the content of the work Op predicted without using the work prediction model 171, and the presence of the worker Hu in the work place Ar. Verification is performed using at least one of the determination results as to whether or not to

Therefore, the control method has the effect of being able to estimate with high accuracy "the content of the work Op performed by the worker Hu in the workplace Ar" from the workplace image Ic.

(Example of the first prediction unit and presence/absence prediction unit realized as a neural network (NN))
As described above, at least one of the first prediction unit 120 and the presence/absence prediction unit 141 may be implemented as an NN. In particular, at least one of the first prediction unit 120 and the presence/absence prediction unit 141 may be realized as a CNN (Convolutional Neural Network) or a DNN (Deep Neural Network).

For example, the first prediction unit 120 (work prediction model 171) predicts from the workplace image Ic that "the worker Hu imaged in the workplace image Ic is performing the work object Ob imaged in the workplace image Ic. Output the first prediction Pf indicating "the content of the work Op". Further, for example, the presence/absence prediction unit 141 (presence/absence model 172) outputs a presence/absence prediction Pa indicating "whether there is a worker Hu in the workplace Ar captured in the workplace image Ic" from the entire image Ia.

Next, learning processing for constructing the work prediction model 171 and presence/absence model 172, which are learned models, will be described below.

(learning process)
(About the work prediction model)
The first prediction unit 120 implemented as a neural network constructs a work prediction model 171, which is a trained model, by supervised learning for a learning data set DS1, which is a set of teacher data Dt1, for example. The teacher data Dt1 is information indicating at least "the details of the work Op performed by the worker Hu imaged in the workplace image Ic on the work object Ob imaged in the workplace image Ic" for the workplace image Ic. is the labeled data. “Teaching data” is also called “learning data”. Further, the process in which "the first prediction unit 120 constructs the work prediction model 171 by supervised learning on the learning data set DS1" is also referred to as "learning process SS1".

In the learning process SS1, the first prediction unit 120 implemented as an NN learns the work prediction model 171 that outputs the following information in response to the input of the workplace image Ic by supervised learning on the learning data set DS1. built as a finished model. That is, the work prediction model 171 receives the workplace image Ic as an input, and "contents of the work Op performed by the worker Hu imaged in the workplace image Ic with respect to the work object Ob imaged in the workplace image Ic". is a trained model whose output is

That is, in the workplace image Ic in the training data Dt1, information indicating "the details of the work Op performed by the worker Hu imaged in the workplace image Ic on the work object Ob imaged in the workplace image Ic". is labeled. Information attached as a label to the workplace image Ic in the teacher data Dt1 is output by the work prediction model 171 constructed by machine learning for the learning data set DS1, which is a set of the teacher data Dt1, in response to the input of the workplace image Ic. It is similar to the information that

The first prediction unit 120 realized as a NN stores the work prediction model 171 constructed by the learning process SS1 in the storage unit 170. FIG.

(Preparing teacher data)
FIG. 6 is a diagram illustrating an example of annotations necessary for generating teacher data Dt1. In the training data Dt1 illustrated in FIG. 6, the following labels (annotations) are attached to the workplace image Ic. That is, the workplace image Ic is labeled with a dashed-dotted line indicating the "range of the bounding box for detecting the worker Hu". Also, the workplace image Ic is labeled with a two-dot chain line indicating the "range of the bounding box for detecting the work object Ob". Furthermore, in the work place image Ic, a straight line with arrows at both ends indicates the content of the work Op performed by the worker Hu imaged in the work place image Ic on the work object Ob imaged in the work place image Ic. ” is attached as a label. A straight line with arrows at both ends connects approximately the center of the "range of the bounding box for detecting the worker Hu" and approximately the center of the "range of the bounding box for detecting the work object Ob".

To prepare training data Dt1 as exemplified in FIG. 6, even if it is not necessary to set a bounding box for detecting worker Hu, the following five annotations are required for workplace image Ic. . That is, it is necessary to set the X and Y coordinates of each of the lower left point and the upper right point among the four points indicating the range of the bounding box for detecting the work object Ob for the workplace image Ic. , which means that four annotations are required. Furthermore, for the workplace image Ic, there is one annotation indicating "the content of the work Op performed by the worker Hu captured in the workplace image Ic with respect to the work object Ob captured in the workplace image Ic". necessary.

Here, supervised learning generally requires a large amount of teacher data Dt1, and in order to prepare a large amount of teacher data Dt1, the above five annotations are performed for each of the large number of workplace images Ic. is required. For this reason, for each of a large number of workplace images Ic, "the content of the work Op performed by the worker Hu imaged in the workplace image Ic on the work object Ob imaged in the workplace image Ic" is determined by a human being. The cost of doing so is extremely high.

Therefore, in the present embodiment, when preparing the training data Dt1, the “contents of the work Op” predicted from the detection information Id are used instead of the “contents of the work Op” determined by a human from the workplace image Ic. is given to the workplace image Ic as a label indicating "contents of". That is, in the training data Dt1, the "contents of the work Op" predicted by the work prediction unit 142 from the detection information Id is not the "contents of the work Op" determined by a human from the workplace image Ic. It is attached as a label indicating

Therefore, the cost required for constructing the work prediction model 171 can be greatly reduced compared to the case of using the "content determined by a human from the workplace image Ic" as the label indicating the "content of the work Op". In particular, the annotation cost required to generate the learning data set DS1 can be greatly reduced compared to the case of using "content determined by a human from the workplace image Ic" as a label indicating "content of the work Op". For example, one of the above-mentioned five annotations required for the workplace image Ic is changed from "determination by a human" to "predicted by the work prediction unit 142 from the detection information Id". Cost can be reduced by 20%.

For example, the following "contents of work Op(x)" is attached as a label to the workplace image Ic(x) generated by imaging the workplace Ar(x) with the cell camera 30 at a certain imaging time Tp. generates teacher data Dt1(x). That is, the work predicting unit 142 predicts from the detection information Id(x) the “contents of the work Op(x) performed by the worker Hu(x) in the workplace Ar(x) at the imaging time Tp”. Teacher data Dt1(x) labeled as an image Ic(x) is generated. The detection information Id(x) is the “tool (for example, the output device 42(x) or the instrument 90(x)), the work object Ob(x), and the work place used for the work Op(x) at the imaging time Tp. information indicating at least one state of Ar(x).

Specifically, the work prediction unit 142 calculates, for example, "at the imaging time Tp, the instrument 90(x): the magnitude of the electric current flowing through the electric driver", "at the imaging time Tp, the worker Hu Operation Op(x) at (x): Screw tightening operation is performed." The teacher data Dt1(x) is generated by adding this predicted content (that is, "content of operation Op(x): screw tightening operation") to the workplace image Ic(x).

As described above, in the information processing apparatus 10, the work prediction model 171 is a learned model constructed by machine learning using the learning data set DS1. In the learning data set DS1, the “workplace image Ic or feature information generated from the workplace image Ic” is labeled with the following information. That is, "the worker Hu is doing in the workplace Ar" predicted from the detection information Id of the input device 41 that detects "at least one state of the tool used in the work Op, the work object Ob, and the workplace Ar" The content of the current work Op" is attached as a label.

According to the above configuration, the information processing apparatus 10 creates a learning data set labeled with "the details of the work Op performed by the worker Hu in the workplace Ar" predicted from the detection information Id of the input device 41. The work prediction model 171 is constructed by the machine learning used.

It is known that supervised learning requires a large amount of teacher data (learning data set). The labels (metadata) that are present are also called “annotations”.

For example, even if the teacher data Dt1 learned when constructing the work prediction model 171 does not require the setting of a bounding box for detecting the worker Hu, the following for the workplace image Ic: 5 annotations are required. That is, it is necessary to set the X and Y coordinates of each of the lower left point and the upper right point among the four points indicating the range of the bounding box for detecting the work object Ob for the workplace image Ic. , which means that four annotations are required. Further, for the workplace image Ic, information (label) indicating "contents of the work Op performed by the worker Hu imaged in the workplace image Ic on the work object Ob imaged in the workplace image Ic". is required, that is, one annotation is required.

Then, for each of a large number of workplace images Ic, "the content of the work Op performed by the worker Hu imaged in the workplace image Ic on the work object Ob imaged in the workplace image Ic" is determined by a human. The cost of doing so is extremely high.

Therefore, when generating the training data Dt1, the "contents of the work Op" predicted from the detection information Id of the input device 41 is used instead of the "contents of the work Op" determined by a human from the workplace image Ic. It is given as a label indicating "content". That is, in the training data Dt1, the "contents of the work Op" predicted from the detection information Id of the input device 41 is not the "contents of the work Op" determined by a human from the workplace image Ic. It is attached as a label indicating

Therefore, the cost required for constructing the work prediction model 171 can be greatly reduced compared to the case of using the "content determined by a human from the workplace image Ic" as the label indicating the "content of the work Op". In particular, the annotation cost required to generate the learning data set DS1 can be greatly reduced compared to the case of using "content determined by a human from the workplace image Ic" as a label indicating "content of the work Op". For example, one of the five annotations necessary for the workplace image Ic is changed from "determination by a human" to "predicted by a device such as the information processing device 10 from the detection information Id". Therefore, the annotation cost can be reduced by 20%.

Therefore, according to the above-described configuration, the information processing apparatus 10 uses the work prediction model 171 whose annotation cost is greatly reduced to predict "the content of the work Op performed by the worker Hu" from the workplace image Ic. , can be estimated with high accuracy.

(About presence/absence model)
The presence/absence prediction unit 141 implemented as an NN constructs a presence/absence model 172, which is a trained model, by supervised learning for a data set DS2, which is a set of teacher data Dt2, for example. The training data Dt2 is data in which the entire image Ia is labeled with information indicating "whether there is a worker Hu in the workplace Ar (more precisely, each of the plurality of workplaces Ar in the factory Fa)". . The process in which the presence/absence prediction unit 141 constructs the presence/absence model 172 by supervised learning for the data set DS2 is also referred to as the "learning process SS2".

In the learning process SS2, the presence/absence prediction unit 141 realized as an NN constructs a presence/absence model 172 that outputs the following information in response to the input of the entire image Ia as a learned model by supervised learning on the data set DS2. do. That is, the presence/absence model 172 is a learned model that takes the entire image Ia as an input and outputs "whether there is a worker Hu in the workplace Ar (more precisely, each of the plurality of workplaces Ar in the factory Fa)". .

That is, in the training data Dt2, the entire image Ia is labeled with information indicating "whether there is a worker Hu in the workplace Ar (more precisely, each of the plurality of workplaces Ar in the factory Fa)". The information labeled as the entire image Ia in the teacher data Dt2 is the information output by the presence/absence model 172 constructed by machine learning for the data set DS2, which is a collection of the teacher data Dt2, in response to the input of the entire image Ia. It is the same.

The presence/absence prediction unit 141 implemented as an NN stores the presence/absence model 172 constructed by the learning process SS2 in the storage unit 170 .

However, it is not essential for the presence/absence prediction unit 141 to construct the presence/absence model 172 by the learning process SS2. The presence/absence prediction unit 141 uses a general learned model used in the flow line detection algorithm as the presence/absence model 172, and from the entire image Ia, "Workplace Ar (more precisely, each of the plurality of workplaces Ar in the factory Fa) is there a worker Hu at the location?" may be predicted.

§4. Modified example (about machine learning)
So far, an example has been described in which the work prediction model 171 and the presence/absence model 172 are constructed as learned data by machine learning on supervised data. However, the machine learning for constructing the task prediction model 171 and the presence/absence model 172 is not limited to machine learning for supervised data. Machine learning for constructing the task prediction model 171 and presence/absence model 172 is not limited to supervised learning, and may be unsupervised learning or other machine learning such as reinforcement learning.

(About the whole image)
So far, an example has been described in which the presence/absence prediction unit 141 predicts "whether or not there is a worker Hu in the workplace Ar captured in the workplace image Ic" from the overall image Ia. However, the presence/absence prediction unit 141 may predict "whether the worker Hu is present in the workplace Ar captured in the workplace image Ic" from the workplace image Ic. That is, it is not essential for the information processing apparatus 10 to acquire the entire image Ia separately from the workplace image Ic. The information processing device 10 (in particular, the determination unit 150) can verify whether the first prediction Pf generated from the workplace image Ic is correct or not based on the presence/absence prediction Pa generated from at least one of the overall image Ia and the workplace image Ic. . For example, the determination unit 150 determines that the presence/absence prediction Pa(x) generated from at least one of the entire image Ia and the workplace image Ic(x) is a prediction that "the worker Hu is present in the workplace Ar(x)". , it is determined that "the first prediction Pf(x) is correct".

(About work forecast)
So far, an example has been described in which the information processing apparatus 10 is the entity that predicts "the content of the work Op performed by the worker Hu" from the detection information Id. That is, the work prediction unit 142 grasps "at least one state of the tool used in the work Op, the work object Ob, and the work place Ar" from the detection information Id, and generates the work prediction Po from the grasped state. I have explained an example to do.

However, from the detection information Id, "at least one state of the tool used in the work Op, the work object Ob, and the work place Ar" is grasped, and from the grasped state, "the content of the work Op performed by the worker Hu ” may not be the information processing apparatus 10 . The information processing apparatus 10 only needs to be able to acquire or generate the "contents of the work Op performed by the worker Hu" predicted from at least one state of the tool, work object Ob, and workplace Ar.

For example, the PLC 20 may generate the work prediction Po from the detection information Id. In that case, the information processing device 10 (in particular, the detection information acquisition unit 132) acquires the result of the prediction executed by the PLC 20, that is, the work prediction Po. Then, using the work prediction Po generated by the PLC 20, the determination unit 150 verifies (correctness of) the first prediction Pf.

That is, when the information processing device 10 (especially, the determination unit 150) uses the work prediction Po to verify (correctness of) the first prediction Pf, the entity that generates the work prediction Po from the detection information Id is the information processing device 10 or may not be the information processing device 10 .

(About the second prediction)
So far, an example in which the information processing apparatus 10 includes both the presence/absence prediction unit 141 and the work prediction unit 142 has been described. do not have. The determination unit 150 only needs to verify the (correctness of) the first prediction Pf using the second prediction Ps. It would be good if it could be verified by Therefore, when the presence/absence prediction Pa is not used for verifying (correctness of) the first prediction Pf, the information processing apparatus 10 does not need to include the presence/absence prediction unit 141 . Further, when the work prediction Po is not used for verification of (correctness of) the first prediction Pf, the information processing apparatus 10 does not need to include the work prediction unit 142 .

[Example of realization by software]
The functional blocks of the information processing device 10 (specifically, the first acquisition unit 110, the first prediction unit 120, the second acquisition unit 130, the second prediction unit 140, the determination unit 150, and the estimation unit 160) are integrated circuits. It may be realized by a logic circuit (hardware) formed in (IC chip) or the like. Also, these functional blocks may be implemented by software using a CPU, GPU, DSP, or the like.

In the latter case, the information processing apparatus 10 includes a CPU, a GPU, a DSP, etc. that execute instructions of a program, which is software that implements each function, and a computer (or CPU) that records the programs and various data so that they can be read. A ROM or storage device (these are referred to as a "recording medium"), and a RAM or the like for developing the above program. The object of the present invention is achieved by a computer (or CPU, GPU, DSP, etc.) reading the program from the recording medium and executing it. As the recording medium, a "non-temporary tangible medium" such as a tape, disk, card, semiconductor memory, programmable logic circuit, or the like can be used. Also, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. It should be noted that the present invention can also be implemented in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention.

10 information processing device (work estimation device)
41 Input device (sensor)
42 Output device (tool)
90 instruments (tools)
120 first prediction unit (prediction unit)
141 presence/absence prediction unit 142 work prediction unit (work detection unit)
150 determination unit 160 estimation unit 171 work prediction model 172 presence/absence model Ar work place DS1 learning data set Hu worker Ia whole image Ic work place image Id detection information Ob work object Op work S120 (prediction step)
S130 (Determination step)
S140 (estimation step)
S210 (Determination step)
S220 (prediction step)
S230 (estimation step)
S310 (Determination step)
S330 (prediction step)
S340 (Determination step)
S350 (estimation step)

Claims (8)

A work estimation device for estimating the content of work performed by a worker in a workplace,
A work performed by the worker imaged in the work place image with respect to the work target imaged in the work place image, using as an input a work place image obtained by imaging the work place or feature information generated from the work place image. a prediction unit that predicts the content of the work performed by the worker on the work object from the workplace image or the feature information using a work prediction model that is a learned model that outputs the content of;
Determining whether the prediction by the prediction unit is correct or not using at least one of the content of the work predicted without using the work prediction model and a determination result as to whether or not the worker is present in the workplace. a determination unit;
an estimating unit for estimating, when the determining unit determines that the prediction by the predicting unit is correct, the content of the work predicted by the predicting unit as the content of the work performed by the worker in the workplace; ,
A work estimating device comprising: A work detection unit that predicts the content of the work performed by the worker in the work area from detection information from a sensor that detects at least one state of the tool, the work object, and the work area used in the work. further comprising
2. The determination unit according to claim 1, wherein when the content of the work predicted by the prediction unit matches the content of the work predicted by the work detection unit, the determination unit determines that the prediction by the prediction unit is correct. work estimator. (A) an overall image obtained by imaging a plurality of workplaces including the workplace, or (B) the workplace image, or (C) feature information generated from the overall image or the workplace image is input, and the workplace Using a presence/absence model that is a learned model that outputs whether or not the worker exists, from the overall image, the workplace image, or the feature information generated from the overall image or the workplace image , further comprising a presence/absence prediction unit that predicts whether or not the worker exists in the workplace,
The work estimating device according to claim 1, wherein the determining unit determines that the prediction by the predicting unit is correct when the presence/absence predicting unit predicts that the worker is present in the workplace. A work detection unit that predicts the content of the work performed by the worker in the work area from detection information from a sensor that detects at least one state of the tool, the work object, and the work area used in the work. When,
(A) an overall image obtained by imaging a plurality of workplaces including the workplace, or (B) the workplace image, or (C) feature information generated from the overall image or the workplace image is input, and the workplace Using a presence/absence model that is a learned model that outputs whether or not the worker exists, from the overall image, the workplace image, or the feature information generated from the overall image or the workplace image , a presence/absence prediction unit that predicts whether or not the worker is present in the workplace;
further comprising
The determination unit predicts that the worker will be present in the workplace by the presence/absence prediction unit, and determines the content of the work predicted by the prediction unit and the content of the work predicted by the work detection unit. 2. The work estimating device according to claim 1, wherein the prediction of the prediction unit is determined to be correct when the . The work prediction model predicts the tool used in the work, the work object, and the information detected by a sensor that detects at least one state of the work place, and the work performed by the worker in the work place. 5. Any one of claims 1 to 4, wherein the work content is constructed by machine learning using a learning data set labeled with the workplace image or feature information generated from the workplace image. Work estimating device according to. A control method for a work estimation device for estimating the content of work performed by a worker in a workplace, comprising:
A work performed by the worker imaged in the work place image with respect to the work target imaged in the work place image, using as an input a work place image obtained by imaging the work place or feature information generated from the work place image. a prediction step of predicting the content of the work performed by the worker on the work object from the workplace image or the feature information using a work prediction model that is a learned model that outputs the content of;
Determining whether the prediction in the prediction step is correct or not using at least one of the content of the work predicted without using the work prediction model and the determination result of whether or not the worker is present in the work place. a determination step to
When the determination step determines that the prediction in the prediction step is correct, the content of the work predicted in the prediction step is estimated as the content of the work performed by the worker in the workplace. an estimation step;
Control method including. An information processing program for causing a computer to function as the work estimation device according to any one of claims 1 to 5, the information processing program for causing the computer to function as each of the units. A computer-readable recording medium recording the information processing program according to claim 7 .

Images