JP7665986B2 - Work level conversion device, work level conversion method, and work level conversion program - Google Patents

Description

The present invention relates to a task level conversion device, a task level conversion method, and a task level conversion program.

At production sites such as factories, the movements of workers are analyzed to recognize the work performed by the worker and evaluate the content of that work. For example, the following Patent Document 1 discloses a work recognition device that associates an object with a part of the body of the worker who used the object, recognizes the work performed by the worker, and evaluates the content of the work based on the recognition results.

Patent No. 6444573

Meanwhile, the work carried out at a production site can be divided into work levels with different granularity. For example, the work of packing a box can be divided into the work of fixing the box to a jig and the work of storing the product in the box. In this case, the work of packing a box is a higher-level work, and the work of fixing the box to a jig and the work of storing the product in the box are lower-level work. The work of fixing a box to a jig, which has been divided into this lower-level work, can be further divided into the work of gripping the box, the work of transporting the box, and the work of adjusting the box. In this case, the work of fixing the box to the jig is a higher-level work, and the work of gripping the box, the work of transporting the box, and the work of adjusting the box are lower-level work.

When evaluating the content of work, the content of the evaluation may be set for each level of work. Therefore, when recognizing work, being able to recognize different levels of work allows for more effective evaluation.

However, the technology in Patent Document 1 is limited to recognizing tasks at the same level and cannot convert the recognized tasks into tasks at a different level.

Here, Japanese Patent Application No. 2019-200826 filed by the applicant of the present application describes a method for converting a higher-level task into multiple lower-level tasks by dividing a task based on a target motion into tasks on a component-by-component basis. However, there is no description of a method for converting multiple lower-level tasks into higher-level tasks.

The present invention provides a task level conversion device, a task level conversion method, and a task level conversion program that can convert multiple lower-level tasks into higher-level tasks.

The activity level conversion device according to one aspect of the present disclosure includes a generation unit that generates sequence data of target activity information that associates a target activity detected from time-series information related to a worker's activity with incidental information associated with the target activity, a converter that converts lower-level activity sequence data identified by a combination of the target activity and the incidental information into higher-level activity sequence data composed of a plurality of lower-level activities, and a conversion output unit that converts the sequence data of the target activity information into higher-level activity sequence data based on the sequence data of the target activity information and outputs the converted sequence data.

According to this aspect, series data of target action information is generated based on the target action and incidental information detected from the time-series information on the worker's actions, and the series data of the target action information is converted to higher-level work series data based on a converter that converts lower-level work series data identified by a combination of the target action and incidental information into higher-level work series data and the series data of the target action information, and can be output.

In the above aspect, the converter may be configured with a trained model that uses task information indicating the correspondence between lower-level task sequence data and higher-level task sequence data as training data, and is trained to convert the target action information sequence data into higher-level task sequence data and output it.

This makes it possible to obtain higher-level task sequence data by inputting sequence data of target action information into the trained model.

In the above aspect, the conversion output unit may further output, for each target action, start timing information indicating whether or not a higher-level task has been started by the target action.

This allows you to determine when higher level work begins.

In the above aspect, the supplementary information may include information that can be recognized from the time-series information and information that is annotated to the time-series information.

This can improve the accuracy of identifying the target action.

A task level conversion method according to another aspect of the present disclosure is a method executed by a processor, and includes generating sequence data of target task information that associates a target task detected from time-series information related to a worker's task with accompanying information accompanying the target task, a converter that converts lower-level task sequence data identified by a combination of the target task and the accompanying information into higher-level task sequence data composed of a plurality of lower-level tasks, and converting the sequence data of the target task information into higher-level task sequence data based on the sequence data of the target task information and outputting the converted sequence data.

According to this aspect, series data of target action information is generated based on the target action and incidental information detected from the time-series information on the worker's actions, and the series data of the target action information is converted to higher-level work series data based on a converter that converts lower-level work series data identified by a combination of the target action and incidental information into higher-level work series data and the series data of the target action information, and can be output.

A task level conversion program according to another aspect of the present disclosure causes a computer to function as a generator that generates sequence data of target task information that associates a target task detected from time-series information related to a worker's task with accompanying information associated with the target task, a converter that converts lower-level task sequence data identified by a combination of the target task and the accompanying information into higher-level task sequence data composed of multiple lower-level tasks, and a conversion output unit that converts the sequence data of the target task information into higher-level task sequence data based on the sequence data of the target task information and outputs the converted data.

According to this aspect, series data of target action information is generated based on the target action and incidental information detected from the time-series information on the worker's actions, and the series data of the target action information is converted to higher-level work series data based on a converter that converts lower-level work series data identified by a combination of the target action and incidental information into higher-level work series data and the series data of the target action information, and can be output.

The present invention provides a task level conversion device, a task level conversion method, and a task level conversion program that can convert multiple lower-level tasks into higher-level tasks.

1 is a diagram illustrating an example of an overview of a system including an operation level conversion device according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of a functional configuration of a system including an operation level conversion device. 10 is a diagram illustrating an example of target action information stored in the activity level conversion device. 10 is a diagram illustrating an example of task information stored in the task level conversion device. FIG. 2 is a diagram illustrating a hardware configuration of an operation level conversion device. 11 is a flowchart illustrating an example of an operation level conversion process executed by the operation level conversion device.

Below, an embodiment of one aspect of the present invention (hereinafter, referred to as "this embodiment") will be described with reference to the drawings. Note that in each drawing, parts with the same reference numerals have the same or similar configurations.

§1 Application Example First, an example of a situation to which the present invention is applied will be described with reference to Fig. 1. In a system 100 including an activity level conversion device 10 according to this embodiment, the image sensors 20a, 20b, and 20c capture the actions of a worker A performed in a certain activity area R, and the activity level conversion device 10, which acquires the captured video, outputs a higher-level activity based on a target action corresponding to a lower-level activity detected from the video. The higher and lower activity levels will be described below.

When work levels with different granularity are set, the work set at the coarser (higher) work level will be the higher level work, and the work set at the finer (lower) work level will be the lower level work.

For example, if the "packing task" consists of the "task of fixing the box to the jig" and the "task of storing the product in the box," then the "packing task" is a higher-level task, and the "task of fixing the box to the jig" and the "task of storing the product in the box" are lower-level tasks.

Also, if the lower level "task of fixing the box to the jig" is composed of "task of gripping the box", "task of transporting the box", and "task of adjusting the box", then "task of fixing the box to the jig" will be the higher level task, and "task of gripping the box", "task of transporting the box", and "task of adjusting the box" will be the lower level tasks.

Here, grasping is defined as the action of moving the hand to the work object and grabbing it, for example, the action of going to grab a part or tool. Transport is defined as the action of moving the work object to the target location, for example, the action of carrying a part or tool toward the product being assembled. Adjustment is defined as the action of moving the work to the target state, for example, the action of assembling parts.

The present invention involves training a model to learn the correspondence between lower-level task sequence data and higher-level task sequence data, generating a trained model for task level conversion, inputting task sequence data detected from a video into the trained model, and outputting the converted higher-level task sequence data from the trained model.

This makes it possible to convert task sequence data consisting of multiple tasks detected from a video into higher-level task sequence data.

In other words, the task level conversion device 10 according to this embodiment can convert multiple tasks at a lower level into tasks at a higher level.

§2 Configuration example [Functional configuration]
Next, an example of the functional configuration of a system 100 including an activity level conversion device 10 according to the present embodiment will be described with reference to FIG. 2. The system 100 includes three image sensors 20a, 20b, and 20c, and an activity level conversion device 10. In the following, when there is no need to distinguish between the three image sensors 20a, 20b, and 20c, they will be referred to as image sensors 20. The activity level conversion device 10 has, as its functional configuration, for example, an acquisition unit 11, a detection unit 12, a generation unit 13, a conversion output unit 14, and a storage unit 19. The storage unit 19 stores, for example, a video 19a, target action information 19b, activity information 19c, and a learned model 19d.

The details of each functional configuration of the system 100, including the task level conversion device 10, are explained below.

<Image sensor>
The image sensor 20 is, for example, a general-purpose camera, and captures a video including a scene in which the worker A is performing an action in the working area R. The image sensor 20 has, for example, a detection unit as a functional configuration. The detection unit detects the action of the worker A, and outputs a video showing the action as time-series information.

Each image sensor 20a, 20b, 20c is positioned so that it can capture the entire work area R and the entire body of worker A. In this case, for example, each image sensor 20a, 20b, 20c may be positioned so that it can capture the entire work area R and the entire body of worker A, or each image sensor 20a, 20b, 20c may be positioned so that it can capture a portion of the work area R and worker A and cover the entire work area R and the entire body of worker A by combining the respective videos. Also, each image sensor 20a, 20b, 20c may capture the work area R and worker A at different magnifications. There is no need to provide three image sensors 20; at least one or more may be provided.

<Acquisition Department>
The acquisition unit 11 acquires time-series information (video in this embodiment) related to the actions performed by the worker A from the image sensor 20. The time-series information acquired by the acquisition unit 11 is transmitted to the storage unit 19 and stored as a video 19a. The acquisition unit 11 also acquires the video 19a stored in the storage unit 19.

<Detection unit>
The detection unit 12 recognizes the position on the image where the hand of the worker A is present from within the video 19a, and detects the target motion based on the motion at the recognized position. For example, grasping, transporting, and adjustment can be set as the target motion. In addition, for example, a motion of grabbing a box or part can be set as a motion corresponding to grasping, for example, a motion of carrying a box or part can be set as a motion corresponding to transporting, and for example, a motion of fixing or assembling a box or part can be set as a motion corresponding to adjustment. The target motion and the motion corresponding to the target motion can be set as appropriate according to the work content, and it is preferable to store the correspondence between them in the storage unit 19 in advance.

Specifically, the detection unit 12 recognizes the position on the video 19a where the hand of the worker A is present, and if it determines that the worker A is making a movement to grasp a part, it detects grasping as the target motion.

<Generation section>
The generating unit 13 generates object motion information 19b in which the object motion detected by the detecting unit 12 is associated with the accompanying information associated with the object motion. The object motion information 19b will be described with reference to FIG.

The target action information 19b is sequence data having, for example, a NO item, a target action item, and an additional information item as data items. The NO item stores the order in which the target action occurred. Instead of the order in which the target action occurred, the time at which the target action occurred or the elapsed time from the start of the work may be stored. The target action item stores the content of the target action. The additional information item stores additional information that is associated with the target action. The additional information may include, for example, information that can be recognized from the video 19a and information that is annotated to the video 19a. Specifically, this includes objects such as parts and boxes, the contact/separation state between the parts or boxes and the hands, the worker, the work process, and information that is known at the time the video was shot.

For example, in the target action information with a NO of "1" in FIG. 3, "grasp" is stored as the target action, and "box" is stored as the additional information. This target action information allows specification that the worker is gripping a box at the time of NO "1". Furthermore, in the target action information with a NO of "4", "grasp" is stored as the target action, and "part 1" is stored as the additional information. This target action information allows specification that the worker is gripping part 1 at the time of NO "4". Furthermore, in the target action information with a NO of "6", "transport" is stored as the target action, and "part 1" is stored as the additional information. This target action information allows specification that the worker is gripping a box at the time of NO "6".

<Conversion output section>
Returning to the description of Fig. 2, the conversion output unit 14 inputs the sequence data of the target action information 19b generated by the generation unit 13 to the trained model 19d, and causes the trained model 19d to output the converted higher-level task sequence data.

The trained model 19d is a trained model for task level conversion, and is generated, for example, by having a model learn the correspondence between lower-level task series data and higher-level task series data. As the trained model, for example, a deep learning model used in the field of natural language processing (NLP), such as Transformer, can be used.

In this embodiment, for example, task information 19c indicating the correspondence between lower-level task sequence data identified by a combination of target actions and accompanying information and higher-level task sequence data is used as training data to generate a trained model 19d.

Work information 19c will be described with reference to FIG. 4. In FIG. 4, "Level 1", "Level 2", and "Level 3" are set as work levels by way of example. "Level 1" is the lowest level, "Level 3" is the highest level, and "Level 2" is the middle level.

At "Level 1," lower-level task sequence data is set that chronologically arranges the contents of the target action, which is the smallest unit of work, and the associated information that accompanies that target action.

In "Level 2", higher level work sequence data corresponding to each time series data constituting the lower level work sequence data is set. In the "Level 2" work sequence data, start timing information is set that indicates the timing when each work in "Level 2" started. For example, "1" is set to data when a work has started, and "0" is set to other data.

For example, the three data in the top row of level 2 are common to the task of "fixing the box to the jig," and the start timing information of the data in the top row is set to "1." Therefore, by identifying the task whose start timing information is set to "1," it is possible to identify the timing at which the task of "fixing the box to the jig" began.

The five data in the lower row of level 2 alternate between the tasks of "putting part 1 in a box" and "putting part 2 in a box." For example, this type of task sequence data is set when the task of "putting part 1 in a box" is performed with the right hand and the task of "putting part 2 in a box" is performed in parallel with the left hand. In this type of task, "1" is set to the start timing information corresponding to the timing when the task of "putting part 1 in a box" is started with the right hand, and the timing when the task of "putting part 2 in a box" is started with the left hand.

Note that if each work item includes "Level 1" additional information, such as "Level 2," the setting of the additional information can be omitted.

At "Level 3", the highest level of task sequence data corresponding to each of the time series data constituting the task sequence data at "Level 1" and "Level 2" is set. Note that when one task is set as the highest level, such as at "Level 3", the settings of the additional information and start timing information can be omitted.

Here, when outputting the converted higher-level task sequence data, the conversion output unit 14 may further output start timing information for each task (target action).

[Hardware configuration]
Next, an example of the hardware configuration of the activity level conversion device 10 according to this embodiment will be described with reference to Fig. 5. The activity level conversion device 10 has a CPU (Central Processing Unit) 10a corresponding to a calculation device, a RAM (Random Access Memory) 10b and a ROM (Read Only Memory) 10c corresponding to a storage unit 19, a communication device 10d, an input device 10e, and a display device 10f. These components are connected to each other via a bus so that they can transmit and receive data to each other. In this embodiment, the activity level conversion device 10 is configured by one computer, but the activity level conversion device 10 may be realized by using multiple computers.

The CPU 10a executes programs stored in the RAM 10b or ROM 10c and functions as a control unit that performs calculations and processing of data. The CPU 10a receives various input data from the input device 10e or communication device 10d, and displays the results of calculations on the input data on the display device 10f or stores them in the RAM 10b or ROM 10c.

RAM 10b is composed of, for example, a semiconductor memory element, and stores rewritable data. ROM 10c is composed of, for example, a semiconductor memory element, and stores readable but non-rewritable data.

The communication device 10d is an interface that connects the operation level conversion device 10 to an external device. The communication device 10d is connected to the image sensor 20 via a communication network such as a LAN (Local Area Network) or the Internet, for example, and receives video from the image sensor 20.

The input device 10e is an interface that accepts data input from a user and may include, for example, a keyboard, a mouse, and a touch panel.

The display device 10f is an interface that visually displays the results of calculations performed by the CPU 10a, and can be configured, for example, with an LCD (Liquid Crystal Display).

The program may be provided by being stored in a computer-readable storage medium such as RAM 10b or ROM 10c, or may be provided via a communication network connected by communication device 10d. In the work level conversion device 10, the CPU 10a executes the program to perform the operations of the acquisition unit 11, detection unit 12, generation unit 13, and conversion output unit 14 shown in FIG. 2. Note that these physical configurations are examples and do not necessarily have to be independent configurations. For example, the work level conversion device 10 may be equipped with an LSI (Large-Scale Integration) that integrates the CPU 10a with the RAM 10b and ROM 10c.

§3 Operation Example FIG. 6 is a flow chart showing an example of the task level conversion process executed by the task level conversion device 10 according to this embodiment.

First, the acquisition unit 11 of the task level conversion device 10 acquires a video, which is time-series information on the actions of the worker A, from the image sensor 20 (step S101). This acquired video is stored in the storage unit 19 as video 19a.

Next, the detection unit 12 of the work level conversion device 10 recognizes the position on the image where the hand of the worker A is located based on the video 19a, and detects the target action (step S102).

Then, the generating unit 13 of the activity level conversion device 10 generates target action information 19b based on the target action detected in step S102 and the associated information associated with the target action (step S103). This generated target action information 19b is stored in the storage unit 19.

Then, the conversion output unit 14 of the task level conversion device 10 inputs the sequence data of the target action information 19b generated in step S103 to the trained model 19d, and outputs the converted higher-level task sequence data from the trained model 19d (step S104). Then, the task level conversion process is terminated.

As described above, the task level conversion device 10 according to this embodiment generates sequence data of target task information 19b based on the target task and associated information detected from the video 19a relating to the task of worker A, and converts the sequence data of target task information 19b into higher-level task sequence data based on the trained model 19d that converts lower-level task sequence data identified by a combination of the target task and associated information into higher-level task sequence data and the sequence data of target task information 19b, and outputs the converted sequence data.

Therefore, the task level conversion device 10 according to this embodiment can convert multiple tasks at a lower level into tasks at a higher level.

§4 Modifications The present invention is not limited to the above-described embodiment, and can be embodied in various other forms without departing from the spirit of the present invention. Therefore, the above-described embodiment is merely illustrative in all respects and should not be interpreted as being restrictive.

For example, in the above-described embodiment, a trained model (converter) is used when converting lower-level work sequence data into higher-level work sequence data, but this is not limited to this. For example, a work information table (converter) such as that shown in FIG. 4 may be used to convert lower-level work sequence data into higher-level work sequence data.

In the above embodiment, the time series information is a video, but the time series information is not limited to a video. Specifically, the time series information may be information on coordinate values indicating the motion of worker A measured by a motion capture device provided in place of the image sensor 20, or information indicating the motion of worker A measured by having worker A wear an acceleration sensor or a gyro sensor in place of the image sensor 20. The time series information may be information indicating the change in pressure value measured by a pressure sensor placed in the work area R in place of the image sensor 20, or information indicating a change in an event detected by a photoelectric sensor placed in the work area R in place of the image sensor 20. Furthermore, the time series information is not limited to using each of the above information individually, and may be a combination of two or more of the above information.

The embodiment of the present invention may also be described as in the following appendices. However, the embodiment of the present invention is not limited to the form described in the appendices below. The embodiment of the present invention may also be in a form in which the descriptions between the appendices are substituted or combined.

[Appendix 1]
a generation unit (13) that generates sequence data of target action information (19b) in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action;
a converter for converting lower-level task sequence data specified by a combination of the target motion and the incidental information into higher-level task sequence data composed of a plurality of the lower-level tasks, and a conversion output unit (14) for converting the sequence data of the target motion information (19b) into the higher-level task sequence data based on the sequence data of the target motion information (19b) and outputting the converted sequence data;
A working level conversion device (10) comprising:

[Appendix 2]
The converter is configured by a trained model (19d) that uses task information (19c) indicating a correspondence relationship between the lower level task series data and the higher level task series data as teacher data, and trains the trained model (19d) to convert the sequence data of the target action information (19b) into the higher level task series data and output the converted data.
2. The working level conversion device (10) of claim 1.

[Appendix 3]
The conversion output unit (14) further outputs, for each of the target movements, start timing information indicating whether or not a higher-level task has been started by the target movement.
3. The working level conversion device (10) according to claim 2.

[Appendix 4]
The supplementary information includes information that can be recognized from the time-series information and information that is annotated to the time-series information.
3. The working level conversion device (10) according to claim 2.

[Appendix 5]
1. A processor-implemented method, comprising:
generating sequence data of target action information (19b) in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action;
a converter for converting lower-level task sequence data specified by a combination of the target motion and the incidental information into higher-level task sequence data composed of a plurality of the lower-level tasks, and converting the sequence data of the target motion information (19b) into the higher-level task sequence data based on the sequence data of the target motion information (19b) and outputting the converted sequence data;
A working level conversion method including:

[Appendix 6]
Computer,
a generation unit (13) for generating sequence data of target action information (19b) in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action;
a converter for converting lower-level task sequence data specified by a combination of the target motion and the incidental information into higher-level task sequence data composed of a plurality of the lower-level tasks, and a conversion output unit (14) for converting the sequence data of the target motion information (19b) into the higher-level task sequence data based on the sequence data of the target motion information (19b) and outputting the converted sequence data;
A working level conversion program that acts as a.

10... Activity level conversion device, 10a... CPU, 10b... RAM, 10c... ROM, 10d... communication device, 10e... input device, 10f... display device, 11... acquisition unit, 12... detection unit, 13... generation unit, 14... conversion output unit, 19... storage unit, 19a... video, 19b... target action information, 19c... activity information, 19d... trained model, 20... image sensor, 100... system, A... worker, R... work area

Claims (4)

a generation unit that generates sequence data of target action information in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action;
a trained model that uses task information that associates lower-level task series data specified by a combination of the target task and the incidental information with higher-level task series data composed of a plurality of the lower- level tasks as teacher data, inputs sequence data of the target task information, and trains the trained model to convert the input sequence data of the target task information into the higher-level task series data and output the converted sequence data;
Equipped with
The trained model outputs the higher-level work sequence data, with start timing information indicating whether or not a higher-level work included in the higher-level work sequence data is started, set for each target action.
Working level conversion device. The supplementary information includes information that can be recognized from the time-series information and information that is annotated to the time-series information.
2. The working level conversion device according to claim 1 . 1. A processor-implemented method, comprising:
generating sequence data of target action information in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action;
using task information that associates lower-level task series data identified by a combination of the target task and the incidental information with higher-level task series data composed of a plurality of tasks at the lower level as teacher data , inputting the sequence data of the target task information, and converting the input sequence data of the target task information into the higher-level task series data and outputting the converted sequence data, thereby generating a trained model;
Including,
The trained model outputs the higher-level work sequence data, with start timing information indicating whether or not a higher-level work included in the higher-level work sequence data is started, set for each target action.
How to convert work levels. Computer,
a generation unit that generates sequence data of target action information in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action;
a trained model that uses task information that associates lower-level task series data specified by a combination of the target task and the incidental information with higher-level task series data composed of a plurality of the lower- level tasks as teacher data, inputs the sequence data of the target task information, and trains the trained model to convert the input sequence data of the target task information into the higher-level task series data and output the converted sequence data;
Function as a
The trained model outputs the higher-level work sequence data, with start timing information indicating whether or not a higher-level work included in the higher-level work sequence data is started, set for each target action.
Work level conversion program.