JP2024180058A - Display system and display method for work machine - Google Patents

Abstract

To easily compare operation performed by an operator with another operation in training for the operation of a work machine.SOLUTION: A display system receives input of first operation data from a first operator. The display system simulates a change in the posture of a first work machine according to the first operation data. The display system reproduces a change in the posture of a second work machine on the basis of model data generated on the basis of second operation data input from a second operator in the past. The display system simultaneously displays a first dynamic image showing the simulated change in the posture of the first work machine and a second dynamic image showing the reproduced change in the posture of the second work machine. The second dynamic image is an image in which a work tool of the second work machine is drawn.SELECTED DRAWING: Figure 8

Description

This disclosure relates to a display system and a display method for a work machine.

Patent Document 1 discloses a technology in which an operator simulates the operation of a work machine while referring to the operation of the work machine by himself or another operator. The technology described in Patent Document 1 plays back video images of past work performed by a remotely operated work machine, and simulates the work in a work environment related to the playback position when the video image is stopped.

JP 2021-103193 A

In training simulations, it is necessary to check previous operation of the work machine at the same time as simulating the operation. For example, a novice trainee can efficiently learn the operation by following the operation of an experienced trainer to operate the work machine. In addition, an intermediate trainee can notice small differences in operation by recognizing the differences between his operation and that of an experienced trainer in real time.

The objective of the present disclosure is to provide a display system and display method for a work machine that allows an operator to easily compare operations with other operations during training on the operation of the work machine.

According to a first aspect of the present invention, a work machine display system receives input of first operation data by a first operator, simulates a change in posture of a first work machine according to the first operation data, reproduces the change in posture of a second work machine based on model data generated based on second operation data previously input by a second operator, and simultaneously displays a first moving image representing the simulated change in posture of the first work machine and a second moving image representing the reproduced change in posture of the second work machine. The second moving image is an image depicting a working implement of the second work machine.

According to the above aspect, when training on the operation of a work machine, the operation by the operator can be easily compared with other operations.

1 is a schematic diagram showing a configuration of a training system according to a first embodiment. 2 is a block diagram showing a software configuration of the training simulator according to the first embodiment. FIG. 4 is a flowchart showing a method for creating model data according to the first embodiment. FIG. 4 is a diagram showing an example of an input screen for setting data according to the first embodiment; 1 is a flowchart (part 1) showing a method for implementing training according to the first embodiment. 11 is a flowchart (part 2) showing a method for implementing training according to the first embodiment. 11 is a flowchart (part 3) showing a method for implementing training according to the first embodiment. 4 is an example of image data when rendering the entire ghost machine according to the first embodiment. 11 is an example of image data when rendering only a bucket of a ghost machine according to the first embodiment. 11 is an example of image data when displaying a trajectory of a blade tip of a bucket of a ghost machine according to the first embodiment. 13 is an example of image data when displaying a lever operation of a trainer according to the first embodiment. 13 is an example of image data when displaying the line of sight of a trainer according to the first embodiment. 4 is an example of image data when a side view is displayed according to the first embodiment. FIG. 4 is a diagram illustrating an example of a trajectory model according to the first embodiment. 4 is a flowchart showing a method for evaluating work data by a trainer according to the first embodiment. 5 is a flowchart showing a method for a trainee to view an evaluation result according to the first embodiment. FIG. 1 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment.

First Embodiment
Configuration of Training System 1
Hereinafter, the embodiments will be described in detail with reference to the drawings.
FIG. 1 is a schematic diagram showing the configuration of a training system 1 according to the first embodiment.
The training system 1 according to the first embodiment is a system that allows a trainee, who is an operator unfamiliar with operating the work machine 100, to simulate the operation of the work machine 100 while referring to the operation of a trainer, who is an operator skilled in operating the work machine 100. The trainee can receive evaluation of his or her own operation by the training system 1 and the trainer. Both the trainer and the trainee are operators of the work machine 100. In the first embodiment, the training system 1 is used for training in the operation of a hydraulic excavator, which is the work machine 100. It should be noted that the type of work machine 100 is not limited to a hydraulic excavator, and may be other work machines such as a bulldozer, a wheel loader, or a forklift.

The training system 1 includes a data server 10 and one or more training simulators 30.

The data server 10 stores model data and work data used for training. The model data represents the operation of the work machine 100 by the trainer. The trainee performs training with reference to the model data. The work data represents the operation of the work machine 100 by the trainee. The training system 1 and the trainer evaluate the trainee's operation based on the work data. The model data and work data include a time series of the operation of the operating lever, a time series of the posture of the work machine 100 (joint angles and turning angles of the work machine), and a time series of the line of sight. Hereinafter, data that represents a time series of the operation and behavior of the work machine 100, such as the model data and work data, is also referred to as behavior record data.

Specifically, the data server 10 includes a model data table T1 for storing model data, a work data table T2 for storing work data, and a user table T3 for storing authentication data.
The model data table T1 stores model data, a model ID which is an ID (identification information) of the model data, and summary data which indicates the work content reproduced by the model data, in association with each other.
The work data table T2 stores, in association with each other, work data, a work ID which is the ID (identification information) of the work data, a model ID of the corresponding model data, evaluation data indicating the evaluation by the training system 1, and comment data indicating the evaluation by the trainer.
The user table T3 stores a user ID, a user category, and authentication information in association with each other. The value of the user category is either trainer or trainee. The authentication information may be, for example, a password.

The training simulator 30 accepts operation input by the operator and simulates the behavior of the work machine 100 in response to the operation input. The training simulator 30 generates an image representing the behavior of the work machine 100 and presents it to the operator. The training simulator 30 also reproduces the behavior of the work machine 100 based on the model data stored in the data server 10, generates an image representing the behavior of the work machine 100, and presents it to the operator.

The training simulator 30 includes a calculation device 31, an operation device 33, and a head-mounted display 35. The operation device 33 is an input interface for operating the work machine 100. The operation device 33 according to the first embodiment is two operation levers (a right operation lever and a left operation lever). The operation device 33 may differ depending on the type of work machine 100 to be simulated. The head-mounted display 35 displays an image calculated by the calculation device 31. The head-mounted display 35 includes an eye tracker that detects the line of sight of the wearer and an IMU (Inertial Measurement Unit) that detects the attitude of the head-mounted display 35. The calculation device 31 performs a simulation of the work machine 100 in the virtual space V based on the input of the operation device 33. The calculation device 31 determines the direction of the line of sight in the virtual space V based on the attitude of the head-mounted display 35, and renders the simulation results. As a result, the virtual space V is displayed on the head-mounted display 35 in conjunction with the attitude of the head-mounted display 35.

Configuration of the work machine 100
The work machine 100 includes a running body 110, a rotating body 120, and a work implement 130.
The traveling body 110 supports the work machine 100 so that the work machine 100 can travel.
The rotating body 120 is supported by the running body 110 so as to be rotatable about a rotation center. A cab 121 is provided at the front of the rotating body 120. A rendering camera for rendering the virtual space V by the computing device 31 is provided in the cab 121.
The work machine 130 is supported at the front of the revolving body 120 so as to be drivable in the vertical direction.
The work machine 130 includes a boom 131, an arm 132, and a bucket 133 as a work tool. For example, the work tool has a cutting edge extending in the width direction. Other examples of the work tool include end attachments such as a clam bucket, a tilt bucket, a tilt rotate bucket, a breaker, and a grappler.

The base end of the boom 131 is rotatably attached to the revolving body 120 via a boom pin. In the work machine 100 shown in Fig. 1, the boom 131 is provided at the center of the front of the revolving body 120, but the present invention is not limited to this and the boom 131 may be attached offset in the left-right direction. In this case, the center of rotation of the revolving body 120 is not located on the operating plane of the work implement 130.
The arm 132 connects the boom 131 and the bucket 133. A base end of the arm 132 is rotatably attached to a tip end of the boom 131 via an arm pin.
The bucket 133 is rotatably attached via a pin to the tip of the arm 132. The bucket 133 functions as a container for storing excavated soil and sand.

Configuration of training simulator 30
FIG. 2 is a block diagram showing the software configuration of the arithmetic device 31 included in the training simulator 30 according to the first embodiment.
The calculation device 31 provided in the training simulator 30 includes an input unit 311, an acquisition unit 312, a reproduction unit 313, a simulator 314, a rendering unit 315, a display control unit 316, a generation unit 317, an evaluation unit 318, a transmission unit 319, a setting memory unit 320, a comment unit 321, and an authentication unit 322.

The input unit 311 acquires operation data from the operating device 33, posture data measured by the IMU of the head mounted display 35, and gaze data measured by the eye tracker of the head mounted display 35. The gaze data is represented by a direction based on the display surface of the head mounted display 35. It should be noted that the absolute gaze direction can be determined by combining the posture data and the gaze data. Hereinafter, the gaze data measured by the eye tracker is referred to as primary gaze data, and the gaze data indicating the absolute gaze direction determined from the posture data and gaze data is referred to as secondary gaze data.

The acquisition unit 312 acquires behavior record data (model data and work data) from the data server 10 .
The reproduction unit 313 reproduces the behavior of the work machine 100 based on the behavior record data acquired by the acquisition unit 312. Hereinafter, the work machine 100 reproduced in the virtual space V by the reproduction unit 313 will be referred to as a ghost machine 100G. The reproduction unit 313 reproduces the behavior of the work machine 100 by arranging the ghost machine 100G in accordance with the time series of the joint angles of the work implement 130 and the swing angle of the swing body 120 included in the behavior record data. Note that the reproduction unit 313 according to other embodiments may, for example, simulate the behavior of the work machine 100 based on time series data of the movement of the operation lever included in the behavior record data, and arrange the ghost machine 100G.

The simulator 314 simulates the behavior of the work machine 100 based on the operation data input to the input unit 311. Hereinafter, the work machine 100 simulated by the simulator 314 is referred to as the avatar machine 100A. The simulator 314 simulates the behavior of the avatar machine 100A by calculating the angular velocities of the revolving body 120 and the work machine 130 according to the operation amount indicated by the operation data.

The rendering unit 315 renders the ghost machine 100G and the avatar machine 100A, and generates image data. The rendering camera in the virtual space is located in the driver's cab 121 of the avatar machine 100A. The rendering camera is directed toward the front of the head-mounted display 35 indicated by the posture data input to the input unit 311, based on the front of the avatar machine 100A. Rendering by the rendering unit 315 is performed at a predetermined frame rate. Therefore, the image data generated by the rendering unit 315 is treated as frame images of a moving image.

The display control unit 316 outputs the image data rendered by the rendering unit 315 to the head-mounted display 35.

The generation unit 317 generates behavior record data based on the operation data, posture data, and primary gaze data input to the input unit 311, and the posture of the avatar machine 100A simulated by the simulator 314. Specifically, the generation unit 317 obtains the positions and postures of the running body 110, the revolving body 120, the boom 131, the arm 132, and the bucket 133 of the avatar machine 100A in the virtual space based on the posture of the avatar machine 100A simulated by the simulator 314. Each position and posture is represented in a virtual space coordinate system, which is a three-dimensional Cartesian coordinate system that defines the virtual space. The generation unit 317 generates secondary gaze data indicating the direction of the gaze in the virtual space coordinate system based on the positions and postures of the running body 110, the revolving body 120, the boom 131, the arm 132, and the bucket 133 in the virtual space, as well as the posture data and the primary gaze data.

The evaluation unit 318 evaluates the operation by the trainee based on the difference between the model data acquired by the acquisition unit 312 and the work data generated by the generation unit 317. For example, the evaluation unit 318 calculates the position and attitude of the revolving unit 120 in the virtual space indicated by the model data and the work data, as well as the distance between the attitude data and the secondary gaze data. The evaluation unit 318 calculates the distance between the model data and the work data for each of the time series of the positions and attitudes of the traveling unit 110, the revolving unit 120, the boom 131, the arm 132, and the bucket 133, the time series of the attitude data, and the time series of the secondary gaze data, and produces a weighted sum of the distances as an evaluation value. In this case, the closer the evaluation value is to zero, the higher the evaluation. The distance in the time series may be calculated using the DTW method.

The transmission unit 319 transmits the behavior record data generated by the generation unit 317 to the data server 10.

The setting storage unit 320 stores the setting data of the training simulator 30. The setting data stores the playback speed of the ghost machine 100G, whether or not to display the ghost machine 100G, whether or not to display the trajectory of the cutting edge of the tool of the ghost machine 100G, whether or not to display a side view, whether or not to display the trainer's lever operation, and whether or not to display the trainer's line of sight. The setting data can be updated by the trainee's operation. The side view is an image that displays the virtual space from the side of the avatar machine 100A. The side view is drawn in the driver's cab.

The comment section 321 accepts input of comments for the work data. Comments can be associated with playback timing. This allows the comment section 321 to be set so that the comment is displayed at a specified playback timing when the work data is played back.

The authentication unit 322 authenticates the operator of the training simulator 30 based on the authentication information stored in the data server 10.

<<Processing of Training System 1>>
When the training simulator 30 is started, the authentication unit 322 displays a login screen on the head mounted display 35. The login screen displays, for example, an input form for a user ID and authentication information. When the operator inputs the user ID and the value of the authentication information on the login screen, the authentication unit 322 obtains the authentication information associated with the input user ID from the user table T3 of the data server 10, and verifies the authentication information.

If the authentication unit 322 succeeds in the matching, the display control unit 316 causes the head mounted display 35 to display a menu screen.
When the user category of the operator is a trainer, a trainer menu is displayed on the menu screen for allowing the operator to select either the creation of model data or the evaluation of work data.
If the user category of the operator is a trainee, a trainee menu is displayed on the menu screen for allowing the operator to select from among carrying out training, viewing evaluation results, and changing settings.

<<Generation of model data>>
FIG. 3 is a flowchart showing a method for creating model data according to the first embodiment.
When the trainer logs in to the training simulator 30 and selects creation of model data on the menu screen, the simulator 314 places the avatar machine 100A in an initial posture and a field in the virtual space (step S101).

The input unit 311 acquires operation data from the operating device 33, posture data measured by the IMU of the head-mounted display 35, and primary gaze data measured by the eye tracker of the head-mounted display 35 (step S102).

Based on the operation data input in step S102, the simulator 314 simulates the behavior of the avatar machine 100A after a predetermined frame time (step S103). At this time, the simulator 314 calculates the positions and attitudes of the running body 110, the rotating body 120, the boom 131, the arm 132, and the bucket 133.

The simulator 314 determines the position and orientation of the rendering camera in the virtual space based on the position and orientation of the rotating body 120 and the orientation data input in step S102 (step S104). The rendering unit 315 renders the virtual space in which the avatar machine 100A is placed from the rendering camera determined in step S104 (step S105). The display control unit 316 outputs the image data rendered in step S105 to the head-mounted display 35 (step S106).

The generation unit 317 generates secondary gaze data indicating the direction of the trainer's gaze in the virtual space based on the position and orientation of the rendering camera calculated in step S104 and the primary gaze data acquired in step S102 (step S107). The generation unit 317 generates one frame of behavior record data based on the operation data acquired in step S102, the orientation of the avatar machine 100A calculated in step S104, and the secondary gaze data generated in step S107 (step S108).

The generation unit 317 determines whether the trainer's operation has ended (step S109). When the trainer determines that the operation is to end, the trainer instructs the trainer to end the operation by performing a predetermined operation on the training simulator 30, such as moving his/her gaze to an end button on the screen. If the trainer's operation has not ended (step S109: NO), the process returns to step S102, and the next frame is simulated.

On the other hand, when the trainer's operation is completed (step S109: YES), the generation unit 317 compiles the generated behavior record data of multiple frames as model data (step S110). The generation unit 317 also accepts input of summary data for the model data from the trainer (step S111). The transmission unit 319 transmits the model data and summary data to the data server 10 (step S112). As a result, the data server 10 assigns a model ID to the received model data, and records the model ID, model data, and summary data in association with each other in the model data table T1.

Change settings
When the trainee logs in to the training simulator 30 and selects to change the settings on the menu screen, the display control unit 316 reads out the setting data stored in the setting storage unit 320 and outputs an input screen for the setting data to the head mounted display 35. FIG. 4 is a diagram showing an example of an input screen for the setting data according to the first embodiment. The input screen includes input forms for the playback speed of the ghost machine 100G, whether or not to display the ghost machine 100G, whether or not to display the trajectory of the blade tip of the ghost machine 100G, whether or not to display a side view, whether or not to display the trainer's lever operation, and whether or not to display the trainer's line of sight, and a decision button. The setting data can be updated by the trainee's operation. When the display of the ghost machine 100G is ON in the input form, an input form for whether or not to display only the bucket 133 of the ghost machine 100G is displayed on the input screen. When the decision button is operated by the trainee's operation, the setting storage unit 320 updates the setting data according to the value input to the input screen.

《Training Implementation》
Fig. 5 is a flowchart (part 1) showing a method for implementing training according to the first embodiment. Fig. 6 is a flowchart (part 2) showing a method for implementing training according to the first embodiment. Fig. 7 is a flowchart (part 3) showing a method for implementing training according to the first embodiment.
When the trainee logs in to the training simulator 30 and selects to carry out training on the menu screen, the acquisition unit 312 accesses the data server 10 and acquires a list of model data recorded in the model data table T1 (step S201). The display control unit 316 outputs a selection screen for selecting one model data from the acquired list to the head mounted display 35 (step S202). At this time, the selection screen displays a list of model data and summary data indicating the contents of the model data. The trainee reads the summary data and selects one model data.

The acquisition unit 312 accesses the data server 10 and acquires the model data selected by the trainee from the model data table T1 (step S203). The simulator 314 places the avatar machine 100A, the ghost machine 100G, and a field in the virtual space (step S204). The avatar machine 100A and the ghost machine 100G are placed overlappingly at the same position.

The input unit 311 acquires operation data from the operating device 33, posture data measured by the IMU of the head-mounted display 35, and primary gaze data measured by the eye tracker of the head-mounted display 35 (step S205).

The simulator 314 simulates the behavior of the avatar machine 100A after a predetermined frame time based on the operation data input in step S205 (step S206). At this time, the simulator 314 calculates the positions and orientations of the running body 110, the rotating body 120, the boom 131, the arm 132, and the bucket 133. The simulator 314 calculates the position and orientation of the rendering camera in the virtual space based on the position and orientation of the rotating body 120 and the orientation data input in step S205 (step S207).

The reproduction unit 313 reproduces the behavior of the ghost machine 100G by calculating the posture of the ghost machine 100G based on the time series of the joint angles of the work machine 130 and the rotation angle of the rotating body 120 contained in the model data acquired in step S203 and the playback speed of the ghost machine 100G indicated by the setting data (step S208).

The reproducing unit 313 judges whether or not the setting data indicates that the ghost machine 100G should be displayed (step S209). If the setting data indicates that the ghost machine 100G should be displayed (step S209: YES), the reproducing unit 313 judges whether or not the setting data indicates that only the bucket 133 of the ghost machine 100G should be displayed (step S210). If the setting data indicates that only the bucket 133 should be displayed (step S210: NO), the reproducing unit 313 determines that all the components of the ghost machine 100G are to be rendered (step S211). In this case, the image data is drawn as shown in FIG. 8. FIG. 8 is an example of image data when rendering the entire ghost machine 100G according to the first embodiment.
If it is set that only the bucket 133 should be displayed (step S210: YES), the reproduction unit 313 determines the bucket 133 of the ghost machine 100G as the rendering target, and excludes components other than the bucket 133 of the ghost machine 100G from the rendering target (step S212). In this case, the image data is drawn as shown in Fig. 9. Fig. 9 is an example of image data when rendering only the bucket 133 of the ghost machine 100G according to the first embodiment.
If it is not set that the ghost machine 100G should be displayed (step S209: NO), the reproducing unit 313 excludes the ghost machine 100G from the rendering target (step S213).

Next, the reproduction unit 313 judges whether or not the setting data indicates that the trajectory of the blade tip of the ghost machine 100G should be displayed (step S214). If the setting indicates that the trajectory of the blade tip of the ghost machine 100G should be displayed (step S214: YES), the reproduction unit 313 generates a blade tip object Obj1 indicating the trajectory of the blade tip, and places the blade tip object Obj1 at both ends and the center of the blade tip of the bucket 133 of the ghost machine 100G calculated in step S208 (step S215). The blade tip object Obj1 may be, for example, a small sphere. The blade tip object Obj1 may correspond to one or more points on the blade tip, for example, and each blade tip object Obj1 represents the trajectory of each point. The reproduction unit 313 increases the transparency of the blade tip object Obj1 generated in the past by a predetermined value (step S216). As a result, the multiple cutting edge objects Obj1 represent the trajectory of the cutting edge position of the bucket 133 at the most recent predetermined time, and the more recent the position, the more transparent the cutting edge object Obj1 is displayed. In another embodiment, the reproduction unit 313 may change the size of the cutting edge object Obj1 instead of the transparency. The cutting edge object Obj1 whose transparency exceeds a predetermined threshold may be removed from the virtual space. For example, the cutting edge object Obj1 whose transparency exceeds a predetermined threshold may not be displayed. In this case, the image data is rendered as shown in FIG. 10. FIG. 10 is an example of image data when displaying the trajectory of the cutting edge of the bucket 133 of the ghost machine 100G according to the first embodiment. The multiple cutting edge objects Obj1 are rendered as curves representing the trajectory of the cutting edge.
If the setting is not made to display the trajectory of the blade tip of the ghost machine 100G (step S214: NO), the reproduction unit 313 does not place the blade tip object Obj1 in the virtual space. The blade tip object Obj1 may have different colors at the left end, center, and right end of the blade tip. For example, the blade tip object Obj1 may be drawn at the left end, center, or right end of the blade tip. The blade tip object Obj1 may also be represented as a band-like trajectory along which a line segment runs from the left end to the right end of the blade tip.

Next, the reproduction unit 313 determines whether or not the setting data indicates that the trainer's lever operation should be displayed (step S217). If the setting indicates that the lever operation should be displayed (step S217: YES), the reproduction unit 313 places the lever object Obj2 at a position a predetermined distance away from the rendering camera position determined in step S207 in the direction the rendering camera faces (step S218).

Figure 11 is an example of image data when displaying the trainer's lever operation according to the first embodiment. The lever object Obj2 includes two reference circles Obj21 representing the left and right operating levers, two operation points Obj22 representing the input positions of the left and right operating levers by the trainee, and multiple plots Obj23 indicating the input positions of the levers by the trainer. The two reference circles Obj21 are arranged with a predetermined distance between them in the left and right direction. The lever object Obj2 is arranged so that the reference circle faces the rendering camera.

Based on the operation data included in the model data, the reproduction unit 313 places the plot Obj23 at a position away from the center of the reference circle Obj21 by the operation amount at the time of reproduction in step S208, and moves the operation point Obj22 to a position away from the center of the reference circle Obj21 by the operation amount indicated by the operation data acquired in step S205 (step S219). In other words, the lever object Obj2 represents the change in the position of the operation lever when the operation lever is viewed in plan from above the operation plane of the operation lever. The reproduction unit 313 reduces the size of the plot Obj23 placed in the past by a predetermined value (step S220). As a result, the multiple plots Obj23 represent the trajectory of the lever operation at the most recent predetermined time. Note that in another embodiment, the reproduction unit 313 may change the transparency instead of the size of the plot Obj23. The plot Obj23 whose size is below a predetermined threshold may be removed from the virtual space.
If it is not set that the lever operation should be displayed (step S217: NO), the reproducing unit 313 does not place the lever object Obj2.

Next, the reproduction unit 313 judges whether or not the setting data indicates that the trainer's line of sight should be displayed (step S221). If the setting indicates that the line of sight should be displayed (step S221: YES), the reproduction unit 313 places the line of sight object Obj3 in the virtual space (step S222). The line of sight object Obj3 is placed at a position a predetermined distance away from the position of the rendering camera determined in step S207 in the direction indicated by the secondary line of sight data. The line of sight object Obj3 may be, for example, a small sphere. The reproduction unit 313 increases the transparency of the line of sight object Obj3 placed in the past by a predetermined value (step S223). As a result, the multiple line of sight objects Obj3 represent the trajectory of the trainer's line of sight at the most recent predetermined time. In another embodiment, the reproduction unit 313 may change the size or hue of the plot Obj23 instead of the transparency. Line of sight objects Obj3 whose transparency exceeds a predetermined threshold value may be removed from the virtual space. For example, when the transparency of the gaze object Obj3 exceeds a predetermined threshold, the reproduction unit 313 may not display the gaze object Obj3. Fig. 12 is an example of image data when displaying the gaze of the trainer according to the first embodiment.
If it is not set that the line of sight should be displayed (step S221: NO), the reproduction unit 313 does not place the line of sight object Obj3 in the virtual space.

The rendering unit 315 determines whether or not the setting data indicates that a side view should be displayed (step S224). If the setting indicates that a side view should be displayed (step S224: YES), the rendering unit 315 positions the rendering camera to the side of the avatar machine 100A so that the camera faces the avatar machine 100A (step S225). The rendering unit 315 generates a side view image Obj4 by drawing a virtual space in which the avatar machine 100A and the ghost machine 100G are positioned (step S226). The simulator 314 positions the side view image Obj4 at a predetermined position inside the cab 121 (step S227). FIG. 13 is an example of image data when a side view according to the first embodiment is displayed.

Next, the rendering unit 315 renders a virtual space in which the avatar machine 100A and the ghost machine 100G are placed from the rendering camera with the position and orientation determined in step S207 (step S228). The display control unit 316 outputs the image data rendered in step S228 to the head-mounted display 35 (step S229). Since the rendering unit 315 generates image data for each frame time, the image data is treated as a frame image of a moving image. Moreover, by rendering the virtual space in which the avatar machine 100A and the ghost machine 100G are placed, the rendering unit 315 can be said to have generated moving image data that simultaneously displays a first moving image in which the avatar machine 100A appears and a second moving image in which the ghost machine 100G appears. Moreover, the rendering unit 315 can be said to have generated moving image data that displays the first moving image in which the avatar machine 100A appears and the second moving image in which the ghost machine 100G appears in the same virtual space in an overlapping manner.

The generation unit 317 generates secondary gaze data indicating the direction of the trainee's gaze in the virtual space based on the position and orientation of the rendering camera calculated in step S207 and the primary gaze data acquired in step S205 (step S230). The generation unit 317 generates one frame of behavior record data based on the operation data acquired in step S205, the orientation of the avatar machine 100A calculated in step S206, and the secondary gaze data generated in step S230 (step S231).

The generation unit 317 determines whether the training has ended (step S232). For example, the generation unit 317 may determine that the training has ended when a certain amount of time has elapsed since the playback position of the model data reached the end of the time series. If the training has not ended (step S232: NO), the process returns to step S205, and the next frame is simulated.

On the other hand, when the training is completed (step S232: YES), the generating unit 317 compiles the generated behavior record data of the multiple frames as work data (step S233). The evaluating unit 318 calculates an evaluation value for evaluating the operation by the trainee based on the difference between the work data generated in step S233 and the model data acquired in step S203 (step S234). The transmitting unit 319 transmits the work data generated in step S233, the model ID of the model data acquired in step S203, and the evaluation value calculated in step S234 to the data server 10 (step S235). As a result, the data server 10 assigns a work ID to the received work data, and records the work ID, work data, model ID, and evaluation value in the work data table T2 in association with each other.
Furthermore, the display control unit 316 outputs an evaluation screen displaying the evaluation values calculated in step S234 to the head mounted display 35 (step S236). The evaluation screen may display evaluation values for each of the turning operation, the operation of the work machine 130, and the line of sight.

The evaluation unit 318 generates a trajectory model, which is a three-dimensional model in which the movement trajectory of the blade tip from the start to the end of the training by the trainee's operation, which is indicated by the work data, and the movement trajectory of the blade tip from the start to the end of the training by the trainer's operation, which is indicated by the model data, are arranged in a three-dimensional space. FIG. 14 is a diagram showing an example of a trajectory model according to the first embodiment. The display control unit 316 may rotatably display the trajectory model generated by the evaluation unit 318 on the head-mounted display 35. This allows the display control unit 316 to simultaneously display the movement trajectory of the blade tip from the trainee's operation and the movement trajectory of the blade tip from the trainer's operation. The rotation of the trajectory model may be performed by changing the attitude of the head-mounted display 35, or may be performed by operating the operation device 33.

<Work data evaluation>
FIG. 15 is a flowchart showing a method for evaluating work data by a trainer according to the first embodiment.
When the trainer logs in to the training simulator 30 and selects to evaluate work data on the menu screen, the acquisition unit 312 accesses the data server 10 and acquires a list of work data recorded in the work data table T2 (step S301). The display control unit 316 outputs a selection screen for selecting one piece of work data from the acquired list to the head mounted display 35 (step S302). The trainer selects one piece of work data.

The acquisition unit 312 accesses the data server 10 and acquires the work data selected by the trainer from the work data table T2 (step S303). The reproduction unit 313 places the ghost machine 100G and the field in the virtual space (step S304).

The reproduction unit 313 reproduces the behavior of the ghost machine 100G by calculating the attitude of the ghost machine 100G based on the time series of the joint angles of the work machine 130 and the rotation angle of the rotating body 120 included in the work data acquired in step S303 (step S305). The reproduction unit 313 calculates the position and attitude of the rendering camera in the virtual space based on the position and attitude of the rotating body 120 of the ghost machine 100G and the attitude data input in step S305 (step S306).

The reproduction unit 313 places the lever object Obj2 at a position a predetermined distance away from the rendering camera position determined in step S306 in the direction in which the rendering camera faces (step S307). This allows the trainer to check the lever operation of the trainee.
The reproduction unit 313 places the line-of-sight object Obj3 in the virtual space (step S308). The line-of-sight object Obj3 is placed at a position a predetermined distance away from the position of the rendering camera determined in step S306 in the direction indicated by the secondary line-of-sight data.

The rendering unit 315 renders the virtual space in which the ghost machine 100G is placed from the rendering camera with the position and orientation determined in step S306 (step S309). The display control unit 316 outputs the image data rendered in step S309 to the head-mounted display 35 (step S310).

The comment unit 321 determines whether or not a comment has been input from the trainer at the current playback timing (step S311). If a comment has been input, comment data is generated that associates the input comment with the playback timing of the comment (step S312).

The reproduction unit 313 determines whether the playback of the work data has ended (step S313). If the playback has not ended (step S313: NO), the process returns to step S305 and the simulation of the next frame is performed.

On the other hand, if playback has ended (step S313: YES), the transmission unit 319 transmits the work ID of the work data acquired in step S303 and the comment data generated in step S312 to the data server 10 (step S314). As a result, the data server 10 adds the received comment data to the data string associated with the received work ID in the work data table T2.

Viewing the evaluation results
FIG. 16 is a flowchart showing a method for a trainee to view an evaluation result according to the first embodiment.
When the trainee logs in to the training simulator 30 and selects to view the evaluation results on the menu screen, the acquisition unit 312 accesses the data server 10 and acquires a list of work data recorded in the work data table T2 (step S401). The list acquired at this time may be a list of work data created by the logged-in trainee. The display control unit 316 outputs a selection screen for selecting one work data from the acquired list to the head-mounted display 35 (step S402). The trainee selects one work data. The acquisition unit 312 accesses the data server 10 and acquires the work data selected by the trainee from the work data table T2 (step S403). The reproduction unit 313 places the ghost machine 100G and the field in the virtual space (step S404).

The reproduction unit 313 reproduces the behavior of the ghost machine 100G by calculating the attitude of the ghost machine 100G based on the time series of the joint angles of the work machine 130 and the rotation angle of the rotating body 120 included in the work data acquired in step S403 (step S405). The reproduction unit 313 determines the position and attitude of the rendering camera in the virtual space based on the position and attitude of the rotating body 120 of the ghost machine 100G and the attitude data input in step S405 (step S406).

The reproduction unit 313 places the lever object Obj2 at a position a predetermined distance away from the rendering camera position determined in step S406 in the direction in which the rendering camera faces (step S407). The reproduction unit 313 places the line of sight object Obj3 in the virtual space (step S408).

The rendering unit 315 renders a virtual space in which the ghost machine 100G is placed from the rendering camera with the position and orientation determined in step S406 (step S409). The comment unit 321 determines whether or not a comment associated with the current playback timing exists in the comment data included in the work data (step S410). If a comment associated with the current playback timing exists (step S410: YES), the comment is superimposed on the rendered image data (step S411). The display control unit 316 outputs the image data rendered in step S409 or step S411 to the head-mounted display 35 (step S412).

The reproducing unit 313 judges whether the reproduction of the work data is completed (step S413). If the reproduction is not completed (step S413: NO), the process returns to step S405 and the simulation of the next frame is performed. On the other hand, if the reproduction is completed (step S413: YES), the training simulator 30 ends the process.
The training system 1 can also play back work data to which no evaluation has been given by a trainer, in the procedure shown in Fig. 16. In this case, since there are no comments, the comment superimposition process in step S411 is skipped entirely.

<Action and Effects>
In this way, the training system 1 according to the first embodiment executes the following processing. The training system 1 accepts input of first operation data by a trainee who is a first operator. The training system 1 simulates a change in posture of the avatar machine 100A which is a first work machine according to the first operation data. The training system 1 reproduces the behavior of the ghost machine 100G which is a second work machine based on model data generated based on second operation data previously input by a trainer who is a second operator. The training system 1 simultaneously displays a first moving image showing a change in posture of the avatar machine 100A and a second moving image showing the behavior of the ghost machine 100G.
In this way, the training system 1 allows the trainee to train while comparing his/her own operation with the operation performed by the trainer. In other words, the first operator and the second operator may be the same operator. By comparing past operation data with current operation data, the same operator can check the degree of improvement in operation or look back on past operations.

The model data for the training system 1 according to the first embodiment includes a time series of the posture of the work machine 100 simulated according to the second operation data. This means that the training system 1 does not need to perform a simulation using the simulator 314 to reproduce the behavior of the ghost machine 100G. In other words, the training system 1 can reproduce the behavior of the ghost machine 100G by simply changing the posture of the ghost machine 100G according to the posture indicated by the time series. Note that in other embodiments, the model data may not include a time series of the posture of the work machine 100, and may be represented by a time series of operation data. In this case, the training system 1 can reproduce the behavior of the ghost machine 100G by performing a simulation according to the time series of the operation data.

The model data of the training system 1 according to the first embodiment includes a time series of the trainer's gaze when the second operation data is input. This allows the training system 1 to present the trainee with not only the method of operating the work machine 100, but also points that require attention. For example, points that require attention are the gaze points that require attention when operating the work machine 100.

The training system 1 according to the first embodiment records work data and comment data from a trainer in association with each other. The training system 1 simultaneously displays a moving image that reproduces a change in the posture of the work machine 100 based on the work data, and the comment data associated with the work data. This allows the trainee to receive feedback from others on his or her operation.
In particular, the training system 1 according to the first embodiment displays the comment data when the playback timing of the moving image coincides with the playback timing related to the comment data, thereby enabling the trainee to recognize which operation, among a series of operations of the work machine 100, requires attention.

The training system 1 according to the first embodiment evaluates the trainee's operation based on the difference between the first operation data related to the work data and the second operation data related to the model data. This allows the trainee to objectively recognize the magnitude of the deviation of the operation from the model data.

The training system 1 according to the first embodiment accepts an input of the playback speed of the second moving image related to the model data from the trainee, and simultaneously displays the first moving image showing the avatar machine 100A and the second moving image showing the ghost machine 100G and played at the specified playback speed. This allows the trainee to train according to his/her own skill level. That is, when the trainee is not familiar with the operation, he/she can train while carefully checking the trainer's operation by moving the ghost machine 100G at a low playback speed. On the other hand, when the trainee becomes familiar with the operation, he/she can train to be closer to the trainer's actual operation by moving the ghost machine 100G at a playback speed close to 1x. In addition, the trainee can train under more difficult conditions than the actual machine by moving the ghost machine 100G at a playback speed faster than 1x. In this case, the trainee can operate the actual machine carefully by feeling the time it takes to operate it longer.

The training system 1 according to the first embodiment executes the following process. The training system 1 accepts input of first operation data by a trainee who is a first operator. The training system 1 simulates a change in posture of an avatar machine 100A which is a first work machine according to the first operation data. The training system 1 reproduces the operation of the trainer according to second operation data previously input by a trainer who is a second operator. The training system 1 simultaneously displays a first moving image which represents a change in posture of the avatar machine 100A and a lever object Obj2 which represents the operation of the trainer. This allows the trainee to simulate the work machine 100 while visually checking the operation by the trainer. In addition, according to the first embodiment, an operation point Obj22 which represents the trainee's lever operation and a plot Obj23 which represents the trainer's lever operation are simultaneously displayed as the lever object Obj2. This allows the trainee to simulate the work machine 100 while visually checking the operation by the trainer.

The lever object Obj2 according to the first embodiment includes a reference circle Obj21, which is a reference position image showing the reference position of the control lever, and a plot Obj23 showing the position of the control lever after operation. This allows the trainee to easily recognize the input position of the control lever based on the positional relationship between the reference circle Obj21 and the plot Obj23.

The lever object Obj2 according to the first embodiment includes a number of plots Obj23 that indicate past positions of the control lever. The older the plots Obj23 are, the smaller or more transparent they are. This allows the trainee to recognize the trajectory of the control lever and prevents confusion between the current input position of the control lever and the past input position.

The lever object Obj2 in the first embodiment is located forward in the line of sight direction on the head-mounted display 35. This allows the trainee to visually recognize the operation by the trainer regardless of the direction he or she is facing.

The training system 1 according to the first embodiment executes the following process. The training system 1 accepts input of first operation data by a trainee who is a first operator. The training system 1 simulates a change in posture of the avatar machine 100A which is a first work machine according to the first operation data. The training system 1 reproduces a change in posture of the ghost machine 100G which is a second work machine based on model data generated based on second operation data input by a second operator in the past. The training system 1 simultaneously displays a first moving image representing a change in posture of the avatar machine 100A and a second moving image representing a change in posture of the ghost machine 100G. At this time, the initial positions of the avatar machine 100A and the ghost machine 100G are the same. In other words, the training system 1 displays the avatar machine 100A and the ghost machine 100G in a superimposed manner. This allows the trainee to easily recognize the difference between his or her operation and the trainer's operation. For example, it is easier to recognize the difference in behavior compared to when the avatar machine 100A and the ghost machine 100G are displayed side-by-side.

The moving image of the avatar machine 100A in the training system 1 according to the first embodiment is an image in which the avatar machine 100A is rendered from a viewpoint located in the driver's cab 121 of the avatar machine 100A. The moving image of the ghost machine 100G in the training system 1 according to the first embodiment is an image in which the ghost machine 100G is rendered from a viewpoint located in the driver's cab 121 of the avatar machine 100A. In this way, the training system 1 allows the trainee to easily recognize the difference between his or her own operation and the trainer's operation from the viewpoint from the driver's cab 121 of the avatar machine 100A.

The training system 1 according to the first embodiment can render only the bucket 133, which is the working tool of the ghost machine 100G. This can prevent the rotating body 120, boom 131, etc. of the ghost machine 100G from interfering with the avatar machine 100A and reducing visibility. On the other hand, the training system 1 renders the entire ghost machine 100G, allowing the trainee to more clearly recognize the difference in posture between the avatar machine 100A and the ghost machine 100G.

The ghost machine 100G according to the first embodiment has a higher transparency than the avatar machine 100A. This allows the training system 1 to prevent the ghost machine 100G from interfering with the simulation. The ghost machine 100G according to other embodiments may have a higher transparency than the avatar machine 100A and a different hue from the avatar machine 100A. This makes it easier to distinguish the ghost machine 100G from the avatar machine 100A. Furthermore, the ghost machine 100G according to other embodiments may be displayed with only an outline.

The training system 1 according to the first embodiment executes the following process. The training system 1 accepts input of first operation data by a trainee who is a first operator. The training system 1 simulates a change in posture of an avatar machine 100A which is a first work machine according to the first operation data. The training system 1 reproduces a change in posture of a ghost machine 100G which is a second work machine based on model data generated based on second operation data previously input by a second operator. The training system 1 simultaneously displays the avatar machine 100A and a cutting edge object Obj1 which represents the trajectory of a bucket 133 which is a work tool of the ghost machine 100G. This allows the trainee to easily recognize the difference between his/her own operation and the trainer's operation.

The training system 1 according to the first embodiment places cutting edge objects Obj1 corresponding to each of the multiple points on the cutting edge. Each cutting edge object Obj1 represents the trajectory of the respective point. By providing cutting edge objects Obj1 for the multiple points on the cutting edge, the trainee can easily recognize the position of the cutting edge of the bucket 133 in the depth direction.

In the first embodiment, the color of the blade tip object Obj1 varies depending on each of the multiple points. This reduces the possibility of confusing and recognizing multiple blade tip objects Obj1.

The training system 1 according to the first embodiment executes the following processing. The training system 1 accepts input of operation data by an operator. The training system 1 detects the operator's line of sight when inputting the operation data. The training system 1 simulates a change in the posture of the work machine 100 in accordance with the operation data. The training system 1 simultaneously displays a posture image representing the simulated change in posture of the work machine 100 and a gaze object Obj3 representing the change in gaze. This allows the training system 1 to display the relationship between the operation of the work machine 100 by the operator and the gaze direction. This allows the trainee to recognize the points to focus on during operation from the trainer's gaze. This also allows the trainer to evaluate the driving based on the trainee's gaze.

In the first embodiment, the cutting edge object Obj1, the plot Obj23 of the lever object Obj2, and the line of sight object Obj3 are multiple points that represent a trajectory. On the other hand, in other embodiments, some or all of these objects may be curves that represent a trajectory.

Second Embodiment
In the training system 1 according to the first embodiment, the behavior of the ghost machine 100G is continuously reproduced at a reproduction speed specified in the setting data. On the other hand, when the trainee is a beginner, there is a possibility that the operation cannot keep up with the behavior of the ghost machine 100G. In such a case, if the ghost machine 100G is continuously reproduced and the postures of the avatar machine 100A and the ghost machine 100G differ greatly, the trainee may lose sight of the indicators of the operation. The training system 1 according to the second embodiment enables appropriate training to be carried out even when the trainee's operation is slow.

As shown in FIG. 5, the training system 1 according to the first embodiment reproduces the behavior of the ghost machine 100G for each frame time in step S208. In contrast, in the second embodiment, the following process is executed instead of step S208.

The reproduction unit 313 calculates the distance between the posture of the avatar machine 100A calculated in step S206 and the previous posture of the ghost machine 100G. The distance between the postures can be calculated, for example, by taking the square root of the sum of the squares of the difference in the rotation angle, the difference in the boom angle, the difference in the arm angle, and the difference in the bucket angle. If the calculated distance exceeds a threshold value previously set in the setting data, the reproduction unit 313 maintains the posture of the ghost machine 100G. On the other hand, if the calculated distance does not exceed a threshold value previously set in the setting data, the reproduction unit 313 reproduces the behavior of the ghost machine 100G by calculating the posture of the ghost machine 100G based on the time series of the joint angle of the work machine 130 and the rotation angle of the rotating body 120 included in the model data and the playback speed of the ghost machine 100G indicated by the setting data.

Thus, according to the second embodiment, the training system 1 suspends reproduction of the behavior of the ghost machine 100G when the difference between the posture of the simulated avatar machine 100A and the posture of the ghost machine 100G exceeds a predetermined threshold. For example, when suspending reproduction of the behavior of the ghost machine 100G, the training system 1 may store an image of the moment of suspension. In this way, the training system 1 according to the second embodiment enables appropriate training to be carried out even when the trainee's operations are slow.

Other Embodiments
Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes are possible. That is, in other embodiments, the order of the above-mentioned processes may be changed as appropriate. Also, some of the processes may be executed in parallel.

The training system 1 according to the embodiment described above includes a data server 10 and a training simulator 30, but is not limited to this. For example, the training system 1 according to another embodiment may be configured with the training simulator 30 alone. In this case, the training simulator 30 can carry out training based on model data recorded in advance. Furthermore, the training system 1 according to another embodiment may carry out training while referring to work data operated by the trainee in the past. Furthermore, in another embodiment, a part of the training simulator 30 may be provided in an external computer. For example, in another embodiment, the training simulator 30 may only be responsible for inputting operation data and displaying calculation results, and the simulation calculations may be performed by an external device.

The training system 1 according to the embodiment described above uses the head-mounted display 35 as a display device, but is not limited to this. For example, in another embodiment, a large display used for remote operation of the work machine 100 may be used as a display device. At this time, the input unit 311 may receive designation of the position and attitude of the rendering camera from the operator. Examples of designation of the position and attitude of the rendering camera include a subjective viewpoint (driver's viewpoint), an objective viewpoint (diagonally behind the driver), an overhead viewpoint (to the side of the work machine 100), and a free viewpoint. The rendering unit 315 renders the virtual space according to the designated position and attitude of the rendering camera. In addition, the display control unit 316 may receive instructions to replay (start over) or pause when playing back the work data. In addition, the display control unit 316 may display a seek bar indicating the playback position and receive an instruction to cue to an arbitrary point by operating the seek bar.

Computer Configuration
FIG. 17 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment.
The computer 90 comprises a processor 91 , a main memory 92 , a storage 93 , and an interface 94 .
The above-mentioned training simulator 30 is implemented in a computer 90. The operations of the above-mentioned processing units are stored in the storage 93 in the form of a program. The processor 91 reads the program from the storage 93, loads it in the main memory 92, and executes the above-mentioned processing in accordance with the program. The processor 91 also secures storage areas in the main memory 92 corresponding to the above-mentioned storage units in accordance with the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a microprocessor.

The program may be for implementing part of the functions to be performed by the computer 90. For example, the program may be implemented by combining it with other programs already stored in the storage or with other programs implemented in other devices. In another embodiment, the computer 90 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, part or all of the functions implemented by the processor 91 may be implemented by the integrated circuit. Such an integrated circuit is also included as an example of a processor. In another embodiment, the computer 90 may be virtualized on one or more computers.

Examples of storage 93 include a magnetic disk, a magneto-optical disk, an optical disk, and a semiconductor memory. Storage 93 may be an internal medium directly connected to the bus of computer 90, or an external medium connected to computer 90 via interface 94 or a communication line. In addition, when this program is distributed to computer 90 via a communication line, computer 90 that receives the program may expand the program into main memory 92 and execute the above-mentioned processing. In at least one embodiment, storage 93 is a non-transitory tangible storage medium.

The program may be for realizing some of the functions described above. Furthermore, the program may be a so-called differential file (differential program) that realizes the functions described above in combination with another program already stored in storage 93.

1...Training system 10...Data server 100...Working machine 100A...Avatar machine 100G...Ghost machine 110...Traveling body 120...Rotating body 121...Driver's cab 130...Working machine 131...Boom 132...Arm 133...Bucket 30...Training simulator 31...Calculation device 311...Input unit 312...Acquisition unit 313...Reproduction unit 314...Simulator 315...Rendering unit 316...Display control unit 317...Generation unit 318...Evaluation unit 319...Transmission unit 320...Settings storage unit 321...Comment unit 322...Authentication unit 33...Operation device 35...Head-mounted display 90...Computer 91...Processor 92...Main memory 93...Storage 94...Interface Obj1...Blade tip object Obj2...Lever object Obj21...Reference circle Obj22...Operation point Obj23...Plot Obj3...Gaze object Obj4...Side view image T1...Model data table T2...Work data table T3...User table V...Virtual space

Claims (7)

A display system for a work machine, comprising:
accepting input of first operation data by a first operator;
simulating a change in attitude of a first work machine in accordance with the first operation data;
reproducing a change in the posture of the second working machine based on model data generated based on second operation data previously input by the second operator;
simultaneously displaying a first moving image representing a simulated change in posture of the first work machine and a second moving image representing a reproduced change in posture of the second work machine;
The second moving image is an image depicting a work implement of the second work machine. The display system according to claim 1 , wherein the first moving image and the second moving image are displayed in an overlapping manner. the first moving image is an image in which the first work machine is rendered from a viewpoint located within a cab of the first work machine,
3. The display system according to claim 1 or 2, wherein the second moving image is an image in which the second work machine is rendered from a viewpoint located within a cab of the first work machine. The display system according to claim 3 , wherein the initial attitudes of the first work machine and the second work machine are the same. The display system according to claim 4 , wherein the second moving image is an image in which a rotating body of the second work machine is not depicted. The display system of claim 5 , wherein the second moving image has a higher transparency than the first moving image. A computer of a training system for providing training in operation of a work machine receives an input of first operation data by a first operator,
simulating a change in attitude of a first work machine in accordance with the first operation data;
reproducing a change in the posture of the second working machine based on model data generated based on second operation data previously input by the second operator;
simultaneously displaying a first moving image representing a simulated change in posture of the first work machine and a second moving image representing a reproduced change in posture of the second work machine;
The second moving image is an image depicting a work implement of the second work machine.

Images