CN118556150A - Support device, construction machine, and program - Google Patents

Abstract

The present invention provides a technology that enables a construction machine to perform actions more accurately. A controller (30) according to one embodiment of the present invention comprises: a work object shape acquisition unit (302B) that acquires data related to the shape of a work object around an excavator (100); an inference unit (302C) that uses a learning model (LM) that has been machine-learned based on training data related to the actions of the excavator (100) based on operations of a relatively skilled operator that establishes a correspondence with the shape of the work object, and infers an action that matches the shape of the work object around the excavator (100) from a plurality of candidate actions of the excavator (100) in a specified operation based on the data related to the shape of the work object around the excavator (100); and a suggestion unit (302D) that suggests one or more of the plurality of candidate actions to a user based on the inference result of the inference unit (302C).

Description

Support equipment, construction machinery and procedures

Technical Field

The present invention relates to a supporting device for a construction machine, etc.

Background Art

Construction machines such as excavators are known (for example, refer to Patent Document 1).

Previous technical literature

Patent Literature

Patent Document 1: Japanese Patent Application Publication No. 2020-029769

Summary of the invention

Technical issues to be solved by the invention

When a construction machine is used to perform a certain operation, multiple actions need to be used separately depending on the state of the operation object, such as the terrain shape. For example, when an excavator performs ground leveling operations, an action of sweeping sand forward with the back of the bucket, a horizontal pull-in action, a rolling action, etc. are used. Therefore, for example, in the case of an inexperienced operator, it is difficult to select an appropriate action from multiple candidate actions, which may result in a decrease in work efficiency.

Therefore, an object of the present invention is to provide a technology that enables a construction machine to operate more accurately.

Means for solving technical problems

In order to achieve the above-mentioned purpose, in one embodiment of the present invention, a support device is provided, which comprises: an acquisition unit, which acquires data related to the shape of a work object around a construction machine; and a suggestion unit, which suggests to a user an action from a plurality of candidate actions of the construction machine in a specified operation based on the data acquired by the acquisition unit.

Furthermore, in another embodiment of the present invention, a construction machine is provided, comprising: an acquisition unit for acquiring data related to the shape of a work object around the construction machine; and a suggestion unit for suggesting to a user an action from a plurality of candidate actions of the construction machine in a specified work based on the data acquired by the acquisition unit.

Furthermore, in another embodiment of the present invention, a program is provided, which enables the support device to perform the following steps: an acquisition step of acquiring data related to the shape of a work object around the construction machine, and a suggestion step of suggesting to the user an action from a plurality of candidate actions of the construction machine in a specified work based on the data acquired in the acquisition step.

Effects of the Invention

According to the above-described embodiment, the construction machine can be operated more accurately.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an example of a shovel operation support system.

FIG. 2 is a plan view showing an example of a shovel.

FIG. 3 is a diagram showing an example of a configuration related to remote control of a shovel.

FIG. 4 is a block diagram showing an example of the hardware configuration of the shovel.

FIG. 5 is a diagram showing an example of the hardware configuration of the information processing device.

FIG. 6 is a functional block diagram showing a first example of a functional configuration related to the operation suggestion function of the shovel operation support system.

FIG. 7 is a flowchart schematically showing a first example of processing related to the operation suggestion function of the shovel.

FIG. 8 is a functional block diagram showing a second example of a functional configuration related to the operation suggestion function of the shovel operation support system.

FIG. 9 is a flowchart schematically showing a second example of processing related to the operation suggestion function of the shovel.

FIG. 10 is a diagram showing a first example of display content of the display device related to the operation suggestion function of the shovel.

FIG. 11 is a diagram showing a second example of the display content of the display device related to the operation suggestion function of the shovel.

FIG. 12 is a diagram showing a third example of the display content of the display device related to the operation suggestion function of the shovel.

FIG. 13 is a diagram showing a third example of the display content of the display device related to the operation suggestion function of the shovel.

FIG. 14 is a diagram showing a fourth example of the display content of the display device related to the operation suggestion function of the shovel.

FIG. 15 is a diagram showing a fifth example of the display content of the display device related to the operation suggestion function of the shovel.

FIG. 16 is a functional block diagram showing an example of a functional configuration related to generation of a target trajectory of a working area of a shovel.

FIG. 17 is a diagram showing an example of a screen related to generation of a target trajectory of a working area of a shovel.

FIG. 18 is a diagram showing another example of a screen related to the generation of a target trajectory of the working area of the shovel.

FIG. 19 is a diagram showing still another example of a screen related to generation of a target trajectory of a working area of a shovel.

FIG. 20 is a flowchart schematically showing an example of processing related to generation of a target trajectory of a working area of a shovel.

DETAILED DESCRIPTION

Hereinafter, embodiments will be described with reference to the drawings.

[Overview of Operation Support System]

First, with reference to FIGS. 1 to 3 , an overview of an operation support system SYS according to the present embodiment will be described.

FIG. 1 is a diagram showing an example of an operation support system SYS. FIG. 1 shows a left side view of an excavator 100. FIG. 2 is a top view showing an example of an excavator 100. FIG. 3 is a diagram showing an example of a structure related to remote operation of an excavator. In the following, the direction in which the attachment AT extends when the excavator 100 is viewed from above (the upper direction in FIG. 2 ) is sometimes defined as “front” to describe the direction in the excavator 100 or the direction viewed from the excavator 100.

As shown in FIG. 1 , the operation support system SYS includes a shovel 100 and an information processing device 200 .

The operation support system SYS cooperates with the shovel 100 using the information processing device 200 , and performs support related to the operation of the shovel 100 .

The shovel 100 included in the operation support system SYS may be one or a plurality of shovels.

In the operation support system SYS, the shovel 100 is a construction machine that is a target of support related to operation.

As shown in FIGS. 1 and 2 , the shovel 100 includes a lower traveling body 1 , an upper swing body 3 , an attachment AT including a boom 4 , an arm 5 , and a bucket 6 , and a cab 10 .

The lower traveling body 1 uses crawler tracks 1C to allow the excavator 100 to travel. The crawler tracks 1C include a left crawler track 1CL and a right crawler track 1CR. The crawler tracks 1CL are hydraulically driven by a traveling hydraulic motor 1ML. Similarly, the crawler tracks 1CL are hydraulically driven by a traveling hydraulic motor 1MR. Thus, the lower traveling body 1 can move on its own.

The upper revolving body 3 is mounted on the lower traveling body 1 so as to be revolvable via the revolving mechanism 2. For example, the upper revolving body 3 is revolved relative to the lower traveling body 1 by hydraulically driving the revolving mechanism 2 with a revolving hydraulic motor 2M.

The boom 4 is mounted at the front center of the upper revolving body 3 so as to be able to pitch about a rotation axis along the left-right direction. The arm 5 is mounted at the front end of the boom 4 so as to be able to rotate about a rotation axis along the left-right direction. The bucket 6 is mounted at the front end of the arm 5 so as to be able to rotate about a rotation axis along the left-right direction.

The bucket 6 is an example of an end attachment and is used for excavation work, for example.

The bucket 6 is mounted on the front end of the arm 5 in a manner that allows for appropriate replacement according to the work content of the excavator 100. That is, a bucket of a different type from the bucket 6 may be mounted on the front end of the arm 5 in place of the bucket 6, for example, a relatively large bucket, a bucket for slopes, a bucket for dredging, etc. Also, a terminal attachment other than a bucket may be mounted on the front end of the arm 5, for example, a mixer, a crusher, a crusher, etc. Also, a preliminary attachment such as a quick coupling, a tilt rotator, etc. may be provided between the arm 5 and the terminal attachment.

The boom 4, the arm 5 and the bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8 and a bucket cylinder 9, respectively.

The cab 10 is a control room for an operator to get on and operate the shovel 100. The cab 10 is mounted on the front left side of the upper swing body 3, for example.

For example, the shovel 100 operates driven elements such as the lower traveling body 1 (ie, a pair of left and right crawler tracks 1CL and 1CR), the upper swing body 3, the boom 4, the arm 5, and the bucket 6 according to operations of an operator riding in the cab 10.

Furthermore, the shovel 100 is configured to be operated by an operator riding in the cab 10, but may also be configured to be remotely operated from the outside of the shovel 100 instead of or in addition thereto. When the shovel 100 is remotely operated, the interior of the cab 10 may be unmanned. The following description is based on the premise that the operator's operation includes at least one of the operation of the operating device 26 by the operator in the cab 10 and the remote operation by an external operator.

For example, as shown in FIG. 3 , the remote operation includes a method of operating the shovel 100 by inputting operation related to the actuator of the shovel 100 in the remote operation support device 300 .

The remote operation support device 300 is, for example, installed in a management center that externally manages the operation of the shovel 100. Furthermore, the remote operation support device 300 may be a portable operation terminal, in which case the operator can remotely operate the shovel 100 while directly checking the operation status of the shovel 100 from the periphery of the shovel 100.

The shovel 100 can transmit an image (hereinafter referred to as a "peripheral image") representing the state of the periphery including the front of the shovel 100 based on the camera image output by the camera device 40 described later to the remote operation support device 300, for example, via the communication device 60 described later. Furthermore, the remote operation support device 300 can display the image (peripheral image) received from the shovel 100 on the display device. Similarly, various information images (information screens) displayed on the output device 50 (display device 50A) inside the cab 10 of the shovel 100 can also be displayed on the display device of the remote operation support device 300. Thus, the operator using the remote operation support device 300 can remotely operate the shovel 100 while confirming the display content such as the image or information screen representing the state of the periphery of the shovel 100 displayed on the display device. Furthermore, the excavator 100 can operate the actuator according to the remote operation signal indicating the content of the remote operation received by the communication device 60 from the remote operation support device 300, thereby driving the driven elements such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5 and the bucket 6.

Furthermore, in the remote operation, for example, a method of operating the shovel 100 by a person (e.g., a worker) around the shovel 100 inputting voice or gestures from outside to the shovel 100 may be included. Specifically, the shovel 100 recognizes voices made by workers around the shovel 100 or gestures made by workers through a voice input device (e.g., a microphone) or a gesture input device (e.g., a camera) mounted on the shovel 100. Furthermore, the shovel 100 may also operate the actuator according to the content of the recognized voice or gesture, thereby driving the driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper swing body 3, the boom 4, the arm 5, and the bucket 6.

Furthermore, the operation of the excavator 100 can also be monitored remotely. In this case, a remote monitoring support device having the same function as the remote operation support device 300 can also be provided. The remote monitoring support device is, for example, the information processing device 200. Thus, a monitor who is a user of the remote monitoring support device can monitor the operation status of the excavator 100 while confirming the peripheral image displayed on the display device of the remote monitoring support device. Furthermore, for example, when it is determined that monitoring is necessary from the perspective of safety, the monitor can intervene in the operation of the operator of the excavator 100 by making a predetermined input using the input device of the remote monitoring support device and make it stop urgently.

The information processing device 200 communicates with the shovel 100 to cooperate with each other and provides support related to the operation of the shovel 100 .

The information processing device 200 is, for example, a server or management terminal device in a management office set up in the construction site of the excavator 100 or a management center located at a location different from the construction site of the excavator 100 for managing the operating status of the excavator 100. The management terminal device may be, for example, a fixed terminal device such as a desktop PC (Personal Computer), or a portable terminal device (mobile terminal) such as a tablet terminal, a smart phone, or a laptop PC. In the latter case, workers at the construction site, supervisors who supervise the work, managers who manage the construction site, etc. can carry the portable information processing device 200 and move around in the construction site. In the latter case, the operator can, for example, bring the portable information processing device 200 into the cockpit of the excavator 100. In addition, the information processing device 200 may be provided in multiple numbers according to the purpose of remote monitoring, processing related to the suggestion function of suggesting the action of the excavator 100 to the operator described later, etc.

The information processing device 200 acquires data related to the operating state from the shovel 100, for example. Thus, the information processing device 200 can grasp the operating state of the shovel 100 and monitor whether the shovel 100 has abnormalities, etc. Furthermore, the information processing device 200 can display the data related to the operating state of the shovel 100 on the display device 208 described later and allow the user to confirm.

The information processing device 200 transmits various data such as programs and reference data used in the processing of the controller 30 of the shovel 100 to the shovel 100. The shovel 100 can thereby perform various processes related to the operation of the shovel 100 using various data downloaded from the information processing device 200.

The information processing device 200 also performs processing for supporting a function related to suggesting an operation of the shovel 100 to an operator described later (hereinafter referred to as an “operation suggestion function”) (see FIG. 6 ), for example. The details will be described later.

[Hardware structure of the operation support system]

Next, the hardware configuration of the operation support system SYS will be described with reference to FIG. 4 and FIG. 5 in addition to FIG. 1 to FIG. 3 .

＜Hardware structure of excavator＞

FIG. 4 is a block diagram showing an example of the hardware configuration of the shovel 100 .

4 , a double line represents a path for transmitting mechanical power, a solid line represents a path for high-pressure hydraulic oil to drive a hydraulic actuator, a dotted line represents a path for transmitting pilot pressure, and a dotted line represents a path for transmitting electrical signals.

The excavator 100 includes various components such as a hydraulic drive system related to hydraulic drive of driven elements, an operating system related to operation of driven elements, a user interface system related to information exchange with users, a communication system related to communication with the outside, and a control system related to various controls.

Hydraulic drive system

As shown in Fig. 4, the hydraulic drive system of the excavator 100 includes a hydraulic actuator HA that hydraulically drives the driven elements such as the lower traveling body 1 (left and right crawler tracks 1C), the upper swing body 3 and the attachment AT, etc., as described above. In addition, the hydraulic drive system of the excavator 100 involved in this embodiment includes an engine 11, a regulator 13, a main pump 14 and a control valve 17.

The hydraulic actuator HA includes travel hydraulic motors 1ML and 1MR, a swing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.

In addition, in the shovel 100, a part or all of the hydraulic actuator HA may be replaced with an electric actuator. That is, the shovel 100 may be a hybrid shovel or an electric shovel.

The engine 11 is a prime mover of the excavator 100 and a main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses diesel as fuel. The engine 11 is, for example, mounted on the rear of the upper slewing body 3. The engine 11 rotates constantly at a preset target speed under direct or indirect control by a controller 30 described later, and drives the main pump 14 and the pilot pump 15.

In addition, other prime movers (for example, electric motors) or the like may be mounted on the shovel 100 instead of the engine 11 or in addition thereto.

The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as "tilt angle") according to a control command from the controller 30.

The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, at the rear of the upper slewing body 3, similarly to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, the stroke length of the piston is adjusted by adjusting the deflection angle of the swash plate by the regulator 13 under the control of the controller 30 to control the discharge flow rate or discharge pressure.

The control valve 17 drives the hydraulic actuator HA according to the operator's operation of the operating device 26, the content of the remote operation, or the operation instruction corresponding to the automatic driving function. The control valve 17 is mounted, for example, on the central part of the upper rotating body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic pipeline, and the working oil supplied from the main pump 14 is selectively supplied to each hydraulic actuator according to the operator's operation or the operation instruction corresponding to the automatic driving function. Specifically, the control valve 17 includes a plurality of control valves (also called "direction switching valves"), which control the flow rate and flow direction of the working oil supplied from the main pump 14 to each hydraulic actuator HA.

"operating system"

As shown in FIG. 4 , the operating system of the shovel 100 includes a pilot pump 15 , an operating device 26 , a hydraulic control valve 31 , a shuttle valve 32 , and a hydraulic control valve 33 .

The pilot pump 15 supplies pilot pressure to various hydraulic devices via the pilot line 25. Like the engine 11, the pilot pump 15 is mounted, for example, at the rear of the upper swing body 3. The pilot pump 15 is, for example, a fixed displacement hydraulic pump and is driven by the engine 11 as described above.

In addition, the pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic oil discharged from the main pump 14 can be supplied to various hydraulic devices as pilot pressure after being reduced in pressure by a predetermined pressure reducing valve to a relatively low-pressure hydraulic oil.

The operating device 26 is provided near the driver's seat of the cab 10 and is used by the operator to operate various driven elements. Specifically, the operating device 26 is used by the operator to operate the hydraulic actuator HA that drives each driven element, and as a result, the operator can operate the driven element that is the driving target of the hydraulic actuator HA. The operating device 26 includes a pedal device or a joystick device for operating each driven element (hydraulic actuator HA).

For example, as shown in FIG. 4 , the operating device 26 is a hydraulic pilot type. Specifically, the operating device 26 uses the working oil supplied from the pilot pump 15 through the pilot line 25 and the pilot line 25A branched from it to output the pilot pressure corresponding to the operation content to the pilot line 27A on the secondary side. The pilot line 27A is connected to an inlet port of the reciprocating valve 32, and is connected to the control valve 17 via the pilot line 27 connected to the outlet port of the reciprocating valve 32. As a result, the pilot pressure corresponding to the operation content related to various driven elements (hydraulic actuator HA) in the operating device 26 can be input to the control valve 17 via the reciprocating valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26 by the operator or the like.

Furthermore, the operating device 26 may be an electrical type. In this case, the pilot line 27A, the reciprocating valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electrical signal (hereinafter referred to as an "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. Then, the controller 30 outputs a control instruction corresponding to the content of the operation signal to the hydraulic control valve 31, that is, outputs a control signal corresponding to the operation content of the operating device 26. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

Furthermore, the control valve (directional switching valve) for driving each hydraulic actuator HA built into the control valve 17 may be of electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 may be directly input to the control valve 17, that is, the electromagnetic solenoid type control valve.

Furthermore, as described above, a part or all of the hydraulic actuator HA may be replaced with an electric actuator. In this case, the controller 30 may output a control instruction corresponding to the operation content of the operating device 26 or the content of the remote operation defined by the remote operation signal to the electric actuator or a driver driving the electric actuator, etc. Furthermore, when the shovel 100 is remotely operated, the operating device 26 may be omitted.

The hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) of the operating object of the operating device 26 and for each driving direction of the driven element (hydraulic actuator HA) (for example, the lifting direction and the lowering direction of the boom 4). That is, two hydraulic control valves 31 are provided for each hydraulic actuator HA of the double-acting type. The hydraulic control valve 31 can be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and is configured to be able to change its flow area (that is, the cross-sectional area through which the working oil can flow). Thus, the hydraulic control valve 31 can output a prescribed pilot pressure to the pilot line 27B on the secondary side using the working oil of the pilot pump 15 supplied through the pilot line 25B. Therefore, as shown in FIG. 4 , the hydraulic control valve 31 can indirectly act on the control valve 17 with a prescribed pilot pressure corresponding to a control signal from the controller 30 through the reciprocating valve 32 between the pilot line 27B and the pilot line 27. Therefore, the controller 30 supplies the pilot pressure corresponding to the operation content of the operating device 26 from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the shovel 100 based on the operator's operation. In addition, the controller 30 supplies the pilot pressure corresponding to the operation command corresponding to the automatic driving function from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the shovel 100 based on the automatic driving function.

Furthermore, the controller 30 can control the hydraulic control valve 31, for example, and realize the remote operation of the shovel 100. Specifically, the controller 30 outputs a control signal corresponding to the content of the remote operation specified by the remote operation signal received from the remote operation support device 300 to the hydraulic control valve 31 through the communication device 60. As a result, the controller 30 supplies the pilot pressure corresponding to the remote operation content from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the shovel 100 based on the remote operation of the operator.

The shuttle valve 32 has two inlet ports and one outlet port, and outputs the working oil having the higher pilot pressure of the pilot pressure input to the two inlet ports to the outlet port. The shuttle valve 32 is provided for each driven element (hydraulic actuator HA) of the operating object of the operating device 26 and for each driving direction of the driven element (hydraulic actuator HA). One of the two inlet ports of the shuttle valve 32 is connected to the pilot line 27A on the secondary side of the operating device 26 (specifically, the above-mentioned joystick device or pedal device included in the operating device 26), and the other is connected to the pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 through the pilot line 27. The corresponding control valve is a control valve that drives the hydraulic actuator that is the operating object of the above-mentioned joystick device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, these shuttle valves 32 can respectively allow the higher pilot pressure of the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure of the pilot line 27B on the secondary side of the hydraulic control valve 31 to act on the pilot port of the corresponding control valve. That is, the controller 30 can control the corresponding control valve independently of the operator's operation of the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operation of the driven elements (lower traveling body 1, upper swing body 3, attachment AT) independently of the operator's operation state of the operating device 26, thereby realizing the remote operation function.

The hydraulic control valve 33 is provided in the pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured to be able to change its flow path area, for example. The hydraulic control valve 33 operates according to the control signal input from the controller 30. Thus, when the operating device 26 is operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, even when the operating device 26 is operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. And, for example, even when the operating device 26 is operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to be lower than the pilot pressure output from the hydraulic control valve 31. Therefore, the controller 30 can reliably act on the pilot port of the control valve in the control valve 17 by controlling the hydraulic control valve 31 and the hydraulic control valve 33, for example, regardless of the operation content of the operating device 26. Therefore, the controller 30 can more appropriately implement the remote control function or the automatic driving function of the shovel 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31 , for example.

User Interface System

As shown in FIG. 4 , the user interface system of the excavator 100 includes an operating device 26 , an output device 50 , and an input device 52 .

The output device 50 outputs various information to a user of the shovel 100 (for example, an operator of the cab 10 or an external remote operator) or a person around the shovel 100 (for example, a worker or a driver of a construction vehicle).

For example, the output device 50 includes a lighting device or a display device 50A (refer to FIG6 ) that outputs various information by a visual method. The lighting device is, for example, a warning light (indicator light). The display device 50A is, for example, a liquid crystal display or an organic EL (Electroluminescence) display. For example, as shown in FIG2 , the lighting device or the display device 50A can be disposed inside the cockpit 10, and various information can be output to an operator inside the cockpit 10 by a visual method. Furthermore, the lighting device or the display device 50A can also be disposed on the side of the upper rotating body 3, and various information can be output to workers around the excavator 100 by a visual method.

Furthermore, for example, the output device 50 includes a sound output device 50B (see FIG. 6 ) that outputs various information in an auditory manner. The sound output device 50B includes, for example, a buzzer or a speaker. The sound output device 50B may be provided, for example, at least one of the inside and outside of the cab 10, and outputs various information in an auditory manner to an operator in the cab 10 or to people (staff, etc.) around the shovel 100.

Furthermore, for example, the output device 50 may also include a device that outputs various information by a tactile method such as vibration of the driver's seat.

The input device 52 receives various inputs from the user of the shovel 100, and inputs signals corresponding to the received inputs to the controller 30. The input device 52 is provided, for example, inside the cab 10, and receives inputs from the operator or the like inside the cab 10. Furthermore, the input device 52 may be provided, for example, on the side of the upper swing body 3, and receive inputs from the staff or the like around the shovel 100.

For example, the input device 52 includes an operation input device that receives operation input. The operation input device may include a touch panel mounted on the display device, a touch pad provided around the display device, a button switch, a joystick, a switch key, a knob switch provided on the operation device 26 (joystick device), and the like.

Furthermore, for example, the input device 52 may include a voice input device for receiving a user's voice input. The voice input device may include, for example, a microphone.

Furthermore, for example, the input device 52 may include a gesture input device that receives a gesture input by the user. The gesture input device includes, for example, an imaging device that captures a state of a gesture performed by the user.

Furthermore, for example, the input device 52 may include a biometric input device for receiving a user's biometric input, which may include, for example, input of biometric information such as a user's fingerprint or iris.

Communication Systems

As shown in FIG. 4 , the communication system of the shovel 100 according to the present embodiment includes a communication device 60 .

The communication device 60 is connected to an external communication line and communicates with a device that is separately installed from the excavator 100. In addition to the device located outside the excavator 100, the device that is separately installed from the excavator 100 may also include a portable terminal device (mobile terminal) brought into the cockpit 10 by the user of the excavator 100. The communication device 60 may, for example, include a mobile communication module that complies with specifications such as 4G ( ^4th Generation) or 5G ( ^5th Generation). In addition, the communication device 60 may, for example, include a satellite communication module. In addition, the communication device 60 may also, for example, include a WiFi communication module or a Bluetooth (registered trademark) communication module. In addition, the communication device 60 may also include a plurality of communication devices corresponding to the communication line of the connection object.

For example, the communication device 60 communicates with an external device such as the information processing device 200 or the remote operation support device 300 in the construction site through a local communication line built at the construction site. The local communication line is, for example, a mobile communication line based on a local 5G (so-called local 5G) built at the construction site or a local network (LAN: Local Area Network) based on WiFi6.

Furthermore, for example, the communication device 60 communicates with the information processing device 200 or the remote operation support device 300 located outside the construction site via a wide area communication line including the construction site, i.e., a wide area network (WAN). The wide area network includes, for example, a wide area mobile communication network, a satellite communication network, the Internet, and the like.

Control Systems

As shown in Fig. 4 , the control system of the shovel 100 includes a controller 30. Furthermore, the control system of the shovel 100 according to the present embodiment includes an operation pressure sensor 29, an imaging device 40, and sensors S1 to S5.

The controller 30 performs various controls related to the shovel 100 .

The functions of the controller 30 can be realized by any hardware or any combination of hardware and software. For example, as shown in FIG4 , the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C and an interface device 30D connected by a bus B1.

The auxiliary storage device 30A is a nonvolatile storage means, and stores programs to be installed, as well as necessary files and data, etc. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory.

For example, when there is an instruction to start a program, the memory device 30B loads the program in the auxiliary storage device 30A so that the CPU 30C can read it. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

For example, the CPU 30C executes a program loaded into the memory device 30B, and realizes various functions of the controller 30 according to the commands of the program.

The interface device 30D functions as, for example, a communication interface for connecting to a communication line inside the shovel 100. The interface device 30D may include a plurality of different types of communication interfaces depending on the type of communication line to be connected.

Furthermore, the interface device 30D functions as an external interface for reading data from a recording medium or writing data to a recording medium. The recording medium is, for example, a dedicated tool connected to a connector provided inside the cab 10 via a detachable cable. Furthermore, the recording medium may be, for example, a general-purpose recording medium such as an SD memory card or a USB (Universal Serial Bus) memory. Thus, a program for realizing various functions of the controller 30 may be provided, for example, by a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. Furthermore, the program may be downloaded from another computer outside the shovel 100 via the communication device 60 and installed in the auxiliary storage device 30A.

In addition, part of the functions of the controller 30 may be realized by other controllers (control devices). In other words, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers.

The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot operating device 26, that is, detects the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. The detection signal of the pilot pressure corresponding to the operating state related to each driven element (hydraulic actuator HA) in the operating device 26 detected by the operating pressure sensor 29 is input to the controller 30.

When the operating device 26 is an electrical type, the operating pressure sensor 29 can be omitted. This is because the controller 30 can grasp the operating state of each driven element of the operating device 26 based on the operating signal input from the operating device 26 .

The camera device 40 acquires an image of the vicinity of the shovel 100. Furthermore, the camera device 40 may acquire (generate) three-dimensional data (hereinafter referred to as "three-dimensional data of the object") representing the position and shape of an object around the shovel 100 within the camera range (angle of view) based on the acquired image and data related to the distance described later. The three-dimensional data of the object around the shovel 100 is, for example, data representing coordinate information of a point group on the surface of the object or distance image data.

For example, as shown in FIG. 2 , the camera device 40 includes a camera 40F for photographing the front of the upper swing body 3, a camera 40B for photographing the rear of the upper swing body 3, a camera 40L for photographing the left side of the upper swing body 3, and a camera 40R for photographing the right side of the upper swing body 3. Thus, the camera device 40 can photograph the entire circumference of the shovel 100 as the center when looking down at the shovel 100, that is, a range of 360 degrees in the angular direction. In addition, the operator can visually recognize the camera images of the cameras 40B, 40L, and 40R or the peripheral images such as the processed images generated based on the camera images through the output device 50 (display device) or the remote operation display device to confirm the status of the left, right, and rear of the upper swing body 3. In addition, the operator can visually recognize the camera images of the camera 40F or the peripheral images such as the processed images generated based on the camera images through the remote operation display device, and remotely operate the shovel 100 while confirming the movement of the attachment AT including the bucket 6. Hereinafter, the cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as “camera 40X”.

The camera 40X is, for example, a monocular camera and may be capable of acquiring data related to distance (depth) in addition to two-dimensional images, such as a stereo camera or a TOF (Time Of Flight) camera (hereinafter collectively referred to as a "3D camera").

The output data of the camera device 40 (camera 40X) (for example, image data or three-dimensional data of objects around the shovel 100, etc.) is input to the controller 30 through a one-to-one communication line or an in-vehicle network. Thus, for example, the controller 30 can monitor objects around the shovel 100 based on the output data of the camera 40X. And, for example, the controller 30 can determine the surrounding environment of the shovel 100 based on the output data of the camera 40X. And, for example, the controller 30 can determine the posture state of the attachment AT shown in the camera image based on the output data of the camera 40X (camera 40F). And, for example, the controller 30 can determine the posture state of the body (upper swing body 3) of the shovel 100 based on the objects around the shovel 100 based on the output data of the camera 40X.

In addition, part of the cameras 40F, 40B, 40L, and 40R may be omitted. For example, when the excavator 100 is not remotely operated, the cameras 40F and 40L may be omitted. This is because it is easier to confirm the state of the front or left side of the excavator 100 from the operator of the cab 10. Furthermore, a distance sensor may be provided on the upper rotating body 3 instead of the camera device 40 (camera 40X) or in addition thereto. The distance sensor is, for example, installed on the upper part of the upper rotating body 3 to obtain data related to the distance and direction of the surrounding objects based on the excavator 100. Furthermore, the distance sensor may also obtain (generate) three-dimensional data (for example, data of coordinate information of a point group) of objects around the excavator 100 within the sensing range based on the obtained data. The distance sensor is, for example, LIDAR (Light Detection and Ranging). Furthermore, for example, the distance sensor may be a millimeter wave radar, an ultrasonic sensor, an infrared sensor, or the like.

The sensor S1 is mounted on the boom 4, and detects a posture angle (hereinafter referred to as the "boom angle") centered on the rotation axis at the base end of the connection portion of the boom 4 and the upper rotating body 3. The sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), and the like. The same may be said of sensors S2 to S4 hereinafter. Furthermore, the sensor S1 may also include a cylinder sensor for detecting the telescopic position of the boom cylinder 7. The same may be said of sensors S2 and S3 hereinafter. The detection signal of the boom angle of the sensor S1 is input to the controller 30. Thus, the controller 30 can grasp the posture state of the boom 4.

The sensor S2 is mounted on the boom 5, and detects a posture angle (hereinafter referred to as "the boom angle") centered on the rotation axis at the base end of the connection portion between the boom 5 and the boom 4. The detection signal of the boom angle detected by the sensor S2 is input to the controller 30. Thus, the controller 30 can grasp the posture state of the boom 5.

The sensor S3 is mounted on the bucket 6 and detects a posture angle (hereinafter referred to as "arm angle") about the rotation axis of the base end of the connection portion between the bucket 6 and the boom 5. The detection signal of the boom angle detected by the sensor S3 is input to the controller 30. Thus, the controller 30 can grasp the posture state of the bucket 6.

The sensor S4 detects the tilt state of the machine body (for example, the upper rotating body 3) relative to a predetermined reference plane (for example, a horizontal plane). The sensor S4 is installed on the upper rotating body 3, for example, and detects the tilt angle (hereinafter referred to as "front-rear tilt angle" and "left-right tilt angle") of the excavator 100 (that is, the upper rotating body 3) centered on two axes in the front-rear direction and the left-right direction. The detection signal corresponding to the tilt angle (front-rear tilt angle and left-right tilt angle) detected by the sensor S4 is input to the controller 30. As a result, the controller 30 can grasp the tilt state of the machine body (upper rotating body 3).

The sensor S5 is mounted on the upper rotating body 3, and outputs detection information related to the rotation state of the upper rotating body 3. The sensor S5 detects, for example, the rotation angular velocity or the rotation angle of the upper rotating body 3. The sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, etc. The detection information related to the rotation state detected by the sensor S5 is input to the controller 30. Thus, the controller 30 can grasp the rotation state such as the rotation angle of the upper rotating body 3.

In addition, when the sensor S4 includes a gyro sensor, a 6-axis sensor, an IMU, etc. that can detect angular velocity around three axes, the rotation state (for example, rotation angular velocity) of the upper rotating body 3 can be detected based on the detection signal of the sensor S4. In this case, the sensor S5 can be omitted. Furthermore, when the posture state of the upper rotating body 3 or the attachment AT can be grasped based on the output of the camera device 40 or the distance sensor, at least a part of the sensors S1 to S5 can also be omitted.

＜Hardware structure of information processing device＞

FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device 200 .

The functions of the information processing device 200 are implemented by arbitrary hardware or arbitrary combination of hardware and software, etc. For example, as shown in FIG5 , the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed computing device 205, a communication interface 206, an input device 207, and a display device 208 connected by a bus B2.

The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. The recording medium 201A includes, for example, a floppy disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, and a USB memory. Thus, the information processing device 200 can read various data used in the processing through the recording medium 201A, store it in the auxiliary storage device 202, or install programs that realize various functions.

Furthermore, the information processing device 200 may acquire various data or programs used in processing from an external device via the communication interface 206 .

The auxiliary storage device 202 stores various installed programs, and stores files and data required for various processes, etc. The auxiliary storage device 202 includes, for example, a HDD (Hard Disc Drive), an SSD (Solid State Disc), a flash memory, and the like.

When there is an instruction to start the program, the memory device 203 reads and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, a DRAM (Dynamic Random Access Memory) or an SRAM.

The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203 , and realizes various functions related to the information processing device 200 according to the programs.

The high-speed computing device 205 performs computing at a relatively high speed in conjunction with the CPU 204. The high-speed computing device 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array).

In addition, the high-speed computing device 205 may be omitted depending on the required computing processing speed.

The communication interface 206 is used as an interface for connecting to external devices so as to be communicable. Thus, the information processing device 200 can communicate with external devices such as the shovel 100 through the communication interface 206. The communication interface 206 may have multiple communication interfaces depending on the communication method with the connected device.

The input device 207 receives various inputs from the user.

The input device 207 includes, for example, an operation input device that receives mechanical operation input from the user. The operation input device includes, for example, a button, a switch key, a joystick, etc. In addition, the operation input device includes, for example, a touch panel installed on the display device 208, a touch pad provided separately from the display device 208, etc.

Furthermore, the input device 207 includes, for example, a voice input device capable of receiving voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

Furthermore, the input device 207 includes, for example, a gesture input device capable of receiving gesture input from the user. The gesture input device includes, for example, a camera capable of capturing the gesture state of the user.

Furthermore, the input device 207 includes, for example, a biometric input device capable of receiving a biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data containing information related to a user's fingerprint or iris.

The display device 208 displays an information screen or an operation screen to the user. For example, the display device 208 includes the above-mentioned remote operation display device. The display device 208 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.

In addition, the remote operation support device 300 can be implemented by any hardware or any combination of hardware and software, and can adopt the same hardware structure as the information processing device 200. For example, the remote operation support device 300 is the same as the information processing device 200 (Figure 5), and is composed of a computer including a CPU, a memory device, an auxiliary storage device, an interface device, an input device, and a display device as the center. The memory device is, for example, an SRAM or a DRAM. The auxiliary storage device is, for example, an HDD, an SSD, an EEPROM, or a flash memory. The interface device includes an external interface for connecting to an external recording medium or a communication interface for communicating with the outside of the excavator 100. The input device includes, for example, a joystick-type operation input device. Thus, the operator can use the operation input device to perform operation input related to the actuator of the excavator 100, and the remote operation support device 300 can use the communication interface to send a signal corresponding to the operation input to the excavator 100. Therefore, the operator can remotely operate the excavator 100 using the remote operation support device.

[Example 1 of the action suggestion function]

Next, a suggestion function (action suggestion function) of the user (operator) regarding the action of the shovel 100 will be described with reference to FIGS. 6 and 7 in addition to FIGS. 1 to 5 .

＜Functional structure＞

FIG. 6 is a functional block diagram showing a first example of a functional configuration related to the operation suggestion function of the operation support system SYS.

The shovel 100 includes a support device 150. The support device 150 supports the operation of the shovel 100 by the operator.

As shown in FIG. 6 , the support device 150 includes a controller 30 , a camera device 40 , an output device 50 , and a communication device 60 .

The controller 30 includes an operation log providing unit 301 and a work supporting unit 302 as functional units.

In addition, when there are a plurality of shovels 100 included in the operation support system SYS, there may be a shovel 100 in which the controller 30 includes only the former of the operation log providing unit 301 and the operation support unit 302, and a shovel 100 in which the controller 30 includes only the latter. In this case, the former shovel 100 only has a function of acquiring the operation log of the shovel 100 used for the operation support function (operation suggestion function) of the operator in the latter shovel 100 and providing it to the information processing device 200. The same may be applied to the second example (FIG. 8) of the operation suggestion function described later.

The information processing device 200 includes an action log acquisition unit 2001 , an action log storage unit 2002 , a training data generation unit 2003 , a machine learning unit 2004 , a learned model storage unit 2005 , and a transmission unit 2006 as functional units.

The action log providing unit 301 is a functional unit for acquiring the action log of the shovel 100, which is the raw data for realizing the action suggestion function, and providing it to the information processing device 200. Specifically, the action log when an operator with a long driving experience and relatively experience in the shovel 100 (hereinafter referred to as a "skilled operator" for convenience) operates the shovel 100 is acquired and provided to the information processing device 200.

The operation log of the shovel 100 includes data related to the shape of the work object around the shovel 100 and data related to the operation of the shovel 100 performed on the shape of the work object. The data related to the shape of the work object around the shovel 100 is, for example, data related to the topographic shape of the ground at the construction site which is the work object of the shovel 100. The data related to the shape of the work object of the shovel 100 is, for example, image data of the camera 40 or three-dimensional data of the work object obtained from the image data. The data related to the operation of the shovel 100 is, for example, data indicating the operation content of the operator. The data indicating the operation content of the operator is, for example, output data of the operation pressure sensor 29 in the case of a hydraulic pilot type operating device 26 or output data (operation signal data) of the operating device 26 in the case of an electric type operating device 26. In addition, the data related to the operation of the shovel 100 may be data indicating the operation state of the shovel 100 actually performed according to the operation of the operator. The data indicating the operation state of the shovel 100 is, for example, output data of the sensors S1 to S5 or data related to the posture state of the shovel 100 acquired based on the output data of the sensors S1 to S5 .

The operation log providing unit 301 includes an operation log recording unit 301A, an operation log storage unit 301B, and an operation log transmitting unit 301C.

The action log recording unit 301A obtains the action log of the shovel 100 and records it in the action log storage unit 301B. For example, each time the shovel 100 performs an action, the action log recording unit 301A records data related to the shape of the work object around the shovel 100 at the start of the action or immediately before the action and data related to the action of the shovel 100 in the action log storage unit 301B.

The action log storage unit 301B stores the action logs of the shovel 100 in an accumulated form. For example, the action log storage unit 301B stores data related to the shape of the work object around the shovel 100 and data related to the action of the shovel 100 in an associated form for each action of the shovel 100. Specifically, the action log storage unit 301B can store record data indicating the correspondence between the data related to the shape of the work object around the shovel 100 and the data related to the action of the shovel 100 for each action of the shovel 100, so as to construct a database of the action logs.

Furthermore, the operation logs of the operation log storage unit 301B that have been transmitted to the information processing device 200 by the operation log transmission unit 301C described later may be deleted afterwards.

The action log sending unit 301C sends the action log of the shovel 100 stored in the action log storage unit 301B to the information processing device 200 through the communication device 60. In addition, the action log sending unit 301C may send record data of the correspondence between the shape of the work object around the shovel 100 and the data related to the action of the shovel 100 for each action of the shovel 100 to the information processing device 200.

For example, the operation log transmission unit 301C transmits the operation log of the shovel 100 which has not been transmitted and is stored in the operation log storage unit 301B to the information processing device 200 in response to a signal (hereinafter referred to as a "transmission request signal") which is received from the information processing device 200 requesting transmission of the operation log of the shovel 100. Furthermore, the operation log transmission unit 301C may automatically transmit the operation log of the shovel 100 which has not been transmitted and is stored in the operation log storage unit 301B to the information processing device 200 at a predetermined time. The predetermined time is, for example, when the operation of the shovel 100 is stopped (the key switch is turned "off") or when the operation is started (the key switch is turned "on").

The operation log acquisition unit 2001 acquires the operation log of the shovel 100 received from the shovel 100 .

The operation log acquisition unit 2001 automatically transmits a transmission request signal to the shovel 100 according to the operation of the user of the information processing device 200 or at a predetermined time, thereby acquiring the operation log of the shovel 100. In addition, the operation log acquisition unit 2001 may also acquire the operation log of the shovel 100 transmitted from the shovel 100 at a predetermined time.

The action log of the shovel 100 acquired by the action log acquisition unit 2001 is stored in an accumulated form in the action log storage unit 2002. For example, similarly to the case of the action log storage unit 301B, the action log storage unit 2002 stores data related to the shape of the work object around the shovel 100 and data related to the action of the shovel 100 in an associated form for each action of the shovel 100.

The training data generation unit 2003 generates training data for machine learning based on the operation log of the shovel 100 in the operation log storage unit 2002. The training data generation unit 2003 may automatically generate the training data by batch processing, or may generate the training data based on the input from the user of the information processing device 200. The training data is a combination of data related to the shape of the work object around the shovel 100 as input data and data representing the operation of the shovel 100 corresponding to the shape of the work object corresponding to the input data (hereinafter referred to as "correct answer data") as correct answer output data.

The correct answer data includes, for example, data indicating the type of action selected from a plurality of candidate actions that can be performed in a specified operation. The plurality of candidate actions include, for example, a sweeping action, a horizontal pull-in action, a rolling action, and a broom action during the ground leveling operation of the ground at the construction site. The sweeping action is, for example, an action of pushing the bucket 6 forward along the ground by operating the attachment AT and sweeping the sand and soil forward with the back of the bucket 6. In the sweeping action, for example, the attachment AT performs a lowering action of the boom 4 and an opening action of the dipper arm 5. The horizontal pull-in action is, for example, an action of leveling the unevenness of the ground (surface of the terrain) by operating the attachment AT and moving the tip of the bucket 6 in a manner of pulling it approximately horizontally toward the front side along the ground. In the horizontal pull-in action, for example, the attachment AT performs a lifting action of the boom 4 and a closing action of the dipper arm 5. The rolling action is, for example, an action of compacting the ground with the back of the bucket 6 by operating the attachment AT. Furthermore, the rolling action may be an action in which the bucket 6 is pushed forward along the ground and the back of the bucket 6 is used to sweep the sand and soil to a predetermined position in front, and then the back of the bucket 6 is used to compact the ground at a predetermined position. In the rolling action, for example, the attachment AT performs a lowering action of the boom 4 when compacting the ground. The sweeping action, for example, is an action in which the upper rotating body 3 is operated and the bucket 6 is rotated left and right along the ground. Furthermore, the sweeping action, for example, is an action in which the attachment AT and the upper rotating body 3 are operated and the bucket 6 is rotated left and right alternately along the ground while the bucket 6 is pushed forward. In the sweeping action, for example, the upper rotating body 3 repeats the left and right rotation action alternately. Furthermore, in the sweeping action, for example, in addition to the left and right rotation action of the upper rotating body 3, the attachment AT may also perform a lowering action of the boom 4 and an opening action of the dipper arm 5 in the same manner as in the case of the sweeping action. Furthermore, the correct answer data may include, for example, data indicating the trajectory of the bucket 6 when the shovel 100 is operating.

The machine learning unit 2004 generates a learned model LM by performing machine learning on the basic learning model based on the training data group generated by the training data generating unit 2003. The learned model LM (basic learning model) includes, for example, a neural network such as DNN (Deep Neural Network).

The learned model LM, for example, takes data related to the shape of the working object around the excavator 100 as an input condition, and outputs the predicted probability of each of the multiple candidate actions to be performed in the specified operation. The predicted probability represents the reliability of the candidate action. This is because, as described above, the action log when the excavator 100 is operated by a skilled person is reflected in the learned model LM, and it is believed that the higher the predicted probability, the higher the reliability of selecting the candidate action. In addition, the predicted probability represents the degree of matching relative to the shape of the working object around the excavator 100 as an input condition. This is because it is believed that the higher the predicted probability, the higher the possibility that the skilled person judges that the candidate action matches the shape of the working object. In addition, the learned model LM can also take data related to the shape of the working object around the excavator 100 as an input condition, and output data representing the trajectory of the bucket 6 (hereinafter referred to as "target trajectory") for each of the multiple candidate actions. Furthermore, the learned model may also take data related to the shape of the working object around the excavator 100 as an input condition, output a plurality of data representing the target trajectory of the bucket 6 for each of a plurality of candidate actions, and output a prediction probability of each of the target trajectories of the plurality of buckets 6. As in the case of the prediction probability of the candidate actions, the prediction probability represents the reliability of the target trajectory of the object or the degree of matching with the shape of the working object around the excavator 100 as an input condition. Furthermore, the learned model LM may be generated for each of a plurality of different operations. For example, the learned model LM is generated for each operation such as ground leveling operation, slope construction operation, and earth filling operation.

The learned model LM output by the machine learning unit 2004 is stored in the learned model storage unit 2005 .

The transmitting unit 2006 transmits the learned model LM to the shovel 100 .

For example, when a learned model LM is generated by the machine learning unit 2004, the transmission unit 2006 transmits the most recently generated learned model LM to the shovel 100. Furthermore, the transmission unit 2006 may transmit the latest learned model LM in the learned model storage unit 2005 to the shovel 100 in response to a signal received from the shovel 100 requesting the transmission of the learned model LM.

The work support unit 302 is a functional unit for supporting the work of the shovel 100 based on the operation of the operator.

The work support unit 302 includes a learned model storage unit 302A, a work object shape acquisition unit 302B, an estimation unit 302C, and a suggestion unit 302D.

The learned model LM transmitted from the information processing device 200 and received via the communication device 60 is stored in the learned model storage unit 302A.

The work object shape acquisition unit 302B acquires data on the shape (topographic shape) of the work object around the shovel 100 based on the output of the imaging device 40 or the distance sensor.

The inference unit 302C infers, from among a plurality of candidate actions that can be performed in a predetermined operation, an action that is relatively reliable or highly matched to the shape of the work object around the shovel 100 based on the data related to the shape of the work object around the shovel 100. Furthermore, the inference unit 302C may infer, for each of the plurality of candidate actions, one or more target trajectories of the bucket 6 that are relatively highly reliable or highly matched based on the data related to the shape of the work object around the shovel 100.

Specifically, the inference unit 302C may use the learned model LM and take the data related to the shape of the work object around the shovel 100 as an input condition to infer a relatively reliable or matching action for the shape of the work object around the shovel 100. Furthermore, the inference unit 302C may use the learned model LM and take the data related to the shape of the work object around the shovel 100 as an input condition to infer the target trajectory of one or more buckets 6 with relatively high reliability or matching.

Based on the inference result of the inference unit 302C, the suggestion unit 302D suggests to the operator of the cockpit 10 an action of the excavator 100 with relatively high reliability or matching degree with respect to the shape of the working object around the excavator 100 through the output device 50 such as the display device 50A. As a result, even an inexperienced operator can select a more appropriate action corresponding to the shape of the current working object around the excavator 100. Therefore, the convenience of the operator can be improved, and the working efficiency of the excavator 100 can be further improved. The action suggested to the operator can be one or more. For example, the suggestion unit 302D notifies the numerical value of the matching degree (reliability) of the shape of the working object around the excavator 100 with respect to all or part of a plurality of candidate actions, thereby suggesting an action with relatively high matching degree (refer to FIG. 10 described later).

Furthermore, the suggestion unit 302D may also suggest, through the output device 50, a target trajectory of the bucket 6 in the motion of the suggestion object that has a relatively high reliability or matching degree with respect to the shape of the work object around the shovel 100 based on the inference result of the inference unit 302C. Thus, even an inexperienced operator can grasp a more appropriate target trajectory of the bucket 6 corresponding to the shape of the current work object around the shovel 100, and can operate the shovel 100 to achieve the target trajectory. Therefore, the convenience of the operator can be further improved, and the working efficiency of the shovel 100 can be further improved.

Furthermore, the suggestion unit 302D may also suggest a plurality of target trajectories of the bucket 6 in the action of the suggestion object having a relatively high reliability or matching degree with respect to the shape of the working object around the excavator 100 through the output device 50 of the action object. Thus, the operator can grasp a plurality of more appropriate target trajectories of the bucket 6 corresponding to the shape of the current working object around the excavator 100, and can operate the excavator 100 to achieve a target trajectory selected by himself. Therefore, the working efficiency of the excavator 100 can be improved in a form reflecting the operator's intention. For example, the suggestion unit 302D notifies the numerical value of the matching degree (reliability) with respect to the shape of the working object around the excavator 100 for each of the plurality of target trajectories of the action of the suggestion object, thereby suggesting a target trajectory of the bucket 6 having a relatively high matching degree (refer to FIGS. 11 and 13 described later).

Furthermore, when the shovel 100 is remotely operated, the suggestion unit 302D may also suggest an action with relatively high reliability or matching degree or a target trajectory of the bucket 6 to the operator using the remote operation support device 300 through the communication device 60. At this time, the suggestion unit 302D transmits data indicating the suggestion content to the remote operation support device 300 through the communication device 60. Thus, the remote operation support device 300 can suggest an action with relatively high reliability or matching degree or a target trajectory of the bucket 6 to the operator using the remote operation support device 300 using a display device or a sound output device.

＜Processing＞

FIG. 7 is a flowchart schematically showing a first example of processing related to the operation suggestion function of the shovel 100 .

The flowchart of Fig. 7 is started when a predetermined input for starting the action suggestion function is received through the input device 52 or the input device of the remote operation support device 300. Hereinafter, the same may be applied to the flowchart of Fig. 9 described later.

As shown in FIG. 7 , in step S102 (an example of an acquisition step), the work object shape acquisition unit 302B acquires data on the shape of the work object around the shovel 100 based on the output of the imaging device 40 .

When the process of step S102 is completed, the controller 30 proceeds to step S104 .

In step S104 , the estimation unit 302C estimates a motion with a relatively high degree of matching (reliability) with respect to the shape of the current work object around the shovel 100 , based on the data acquired in step S102 .

When the process of step S104 is completed, the controller 30 proceeds to step S106 .

In step S106 (an example of a suggestion step), the suggestion unit 302D displays the action to be suggested or the target trajectory of the action among the plurality of candidate actions on the display device 50A based on the estimation result of step S104 .

When the process of step S106 is completed, the controller 30 proceeds to step S108 .

In step S108, the controller 30 determines whether the driven element (actuator) is operated. If the driven element is not operated, the controller 30 proceeds to step S110, and if the driven element is operated, the controller 30 proceeds to step S112.

In step S110, the controller 30 determines whether the end condition is satisfied. The end condition is, for example, receiving a prescribed input indicating the end of the action suggestion function from the operator through the input device 52 or the input device of the remote operation support device 300. In addition, the end condition may be receiving a prescribed input indicating the end of the operation from the operator through the input device 52 or the input device of the remote operation support device 300. In addition, the end condition may be that the controller 30 determines the end of the operation based on the camera image of the camera device 40. When the end condition is satisfied, the controller 30 ends the processing of this flowchart, and when the end condition is not satisfied, it returns to step S108.

On the other hand, in step S112, the controller 30 determines whether the operation of the driven element corresponding to one movement of the shovel 100 is completed based on the operation state of the operating device 26 or the operation state of the shovel 100. The controller 30 can grasp the operation state of the operating device 26 or the operation state of the shovel 100 based on the output of the operating pressure sensor 29, the operation signal output from the operating device 26, the output of the sensors S1 to S5, etc. When the operation of the driven element corresponding to one movement of the shovel 100 is completed, the controller 30 proceeds to step S114. If it is not completed, the controller 30 waits until it is completed (repeating the process of step S112).

In step S114, the controller 30 determines whether the end condition is satisfied. If the end condition is satisfied, the controller 30 ends the processing of this flowchart, and if the end condition is not satisfied, the controller 30 returns to step S102.

Thus, in this example, the support device 150 can suggest to the operator, via the display device 50A or the remote operation support device 300 , an operation of the shovel 100 or a target trajectory of the bucket 6 that is highly compatible (reliable) with the shape of the work object around the shovel 100 .

[Example 2 of the action suggestion function]

Next, a second example of the operation suggestion function of the shovel 100 to the user (operator) will be described with reference to FIGS. 8 and 9 in addition to FIGS. 1 to 5 .

In the following, the same reference numerals are used for the same or corresponding structures as those in the first example, and the description will be mainly focused on the parts different from those in the first example, and the description of the same or corresponding contents as those in the first example may be simplified or omitted.

＜Functional structure＞

FIG. 8 is a functional block diagram showing a first example of a functional configuration related to the operation suggestion function of the operation support system SYS.

As shown in FIG. 8 , the support device 150 of the shovel 100 includes a controller 30 , a hydraulic control valve 31 , an imaging device 40 , an output device 50 (display device 50A), an input device 52 , and a communication device 60 .

Similar to the first example described above, the controller 30 includes an operation log providing unit 301 and a work supporting unit 302 as functional units.

The work support unit 302 includes a learned model storage unit 302A, a work object shape acquisition unit 302B, an estimation unit 302C, a suggestion unit 302D, and a motion control unit 302E.

The motion control unit 302E controls the hydraulic control valve 31 based on the input of the instruction from the operator received through the input device 52 or the communication device 60, and automatically performs the motion of the shovel 100 suggested to the operator by the suggestion unit 302D. Thus, the support device 150 can automatically perform the motion of the suggested object corresponding to the shape of the current work object around the shovel 100 based on the input of the instruction from the operator. Therefore, even an inexperienced operator can more accurately perform the motion of the shovel 100 corresponding to the shape of the current work object around the shovel 100 by only inputting the instruction. Therefore, the convenience of the operator can be further improved, and the working efficiency of the shovel 100 can be further improved.

For example, when the motion of the suggested object is one, the motion control unit 302E controls the hydraulic control valve 31 according to the input of the instruction from the operator, and automatically executes the motion of the shovel 100 of the suggested object suggested by the suggestion unit 302D. Also, for example, when there are multiple motions of the suggested object, the motion control unit 302E automatically executes one motion selected by the input of the instruction from the operator among the multiple motions of the suggested object. Also, for example, when one target trajectory of the bucket 6 is suggested by the suggestion unit 302D, the motion control unit 302E automatically executes the motion of the shovel 100 of the suggested object so that the bucket 6 moves along the target trajectory of the suggested object. Also, for example, when multiple target trajectories of the bucket 6 are suggested by the suggestion unit 302D, the motion control unit 302E automatically executes the motion of the suggested object so that the bucket 6 moves along one target trajectory selected by the input of the instruction from the operator among the multiple target trajectories.

The action log recording unit 301A records the action log including the data related to the shape of the work object acquired by the work object shape acquisition unit 302B and the data indicating the action or target trajectory performed by the action control unit 302E in the action log storage unit 301B. Thus, the action log sending unit 301C can store the action log including the data related to the shape of the work object and the data indicating the action or target trajectory of the shovel 100 actually performed by the operator based on the shape of the work object in the action log storage unit 301B. Moreover, the action log sending unit 301C can upload the accumulated action log to the information processing device 200. Therefore, the machine learning unit 2004 can relearn or additionally learn the learned model LM using the action log to update the learned model LM.

Furthermore, the machine learning unit 2004 compares the relearned or additionally learned learned model LM with the current learned model LM using predetermined evaluation data, and when the evaluation result of the former is higher, the learned model LM in the learned model storage unit 2005 may be updated.

＜Processing＞

FIG. 9 is a flowchart schematically showing a second example of processing related to the operation suggestion function of the shovel 100 .

As shown in FIG. 9 , the processing of steps S202 to S206 is the same as that of steps S102 to S106 in FIG. 7 , and thus the description thereof is omitted.

If the process of step S206 is completed, the controller 30 proceeds to step S208 .

In step S208, the controller 30 determines whether an input (hereinafter referred to as "input of execution instruction") from the operator is received through the input device 52 or the communication device 60. If the input of execution instruction is not received, the controller 30 proceeds to step S210, and if the input of execution instruction is received, the controller 30 proceeds to step S212.

In step S210, the controller 30 determines whether the end condition is satisfied. If the end condition is satisfied, the controller 30 ends the processing of this flowchart, and if the end condition is not satisfied, the controller 30 returns to step S208.

On the other hand, in step S212, the motion control unit 302E controls the hydraulic control valve 31 and automatically executes the motion specified by the execution instruction input. And, when the target track is specified by the execution instruction input, the motion control unit 302E executes the motion of the shovel 100 specified by the execution instruction input so that the bucket 6 moves along the target track specified by the execution instruction input.

If the process of step S212 is completed, the controller 30 proceeds to step S214.

In step S214 , the motion log recording unit 301A records the motion log including data on the shape of the work object acquired by the work object shape acquisition unit 302B and data indicating the motion or target trajectory performed by the motion control unit 302E in the motion log storage unit 301B.

If the process of step S214 is completed, the controller 30 proceeds to step S216.

In step S216, the controller 30 determines whether the end condition is satisfied. If the end condition is satisfied, the controller 30 ends the processing of this flowchart, and if the end condition is not satisfied, the controller 30 returns to step S202.

In addition, in this example, the work object shape acquisition unit 302B may acquire data on the shape of the work object by predicting a change in the shape of the work object due to the operation of the shovel 100 in the previous process of step S212 .

As described above, in this example, the support device 150 can automatically execute the operation of the shovel 100 or the target trajectory of the bucket 6 that has a high degree of matching (reliability) to the shape of the work object around the shovel 100 according to the instruction of the operator.

In this example, the support device 150 can store motion logs including data on the shape of work objects around the shovel 100 and data on the automatically performed motions or target trajectories of the shovel 100. Therefore, the support device 150 can update the learned model LM using the stored motion logs.

[Specific example of display content related to the action suggestion function of the shovel]

Next, a specific example of the display content of the display device 50A related to the operation suggestion function of the shovel 100 will be described with reference to FIGS. 10 to 15 .

In addition, the display contents of FIGS. 10 to 15 may also be displayed on the display device of the remote operation support device 300 .

＜Case 1＞

FIG. 10 is a diagram showing a first example (screen 1000 ) of display content of the display device 50A related to the operation suggestion function of the shovel 100 .

Screen 1000 includes images 1001 to 1006 .

Image 1001 is an image showing a work object around the excavator 100. In this example, image 1001 is an image showing a work object (the ground surface of the construction site) around the excavator 100 when viewed from a predetermined viewpoint around the excavator 100, generated based on the output (image data) of the camera 40 using a known image processing technique.

The image 1002 schematically shows the shovel 100. In this example, the image 1002 schematically shows the shovel 100 when viewed from the same viewpoint as the image 1001, and is displayed superimposed on the image 1001.

Image 1003 is an image showing the actions of the recommended object suggested by the suggestion unit 302D from among the plurality of candidate actions in the form of a catalog. In this example, in image 1003, the actions of the recommended object include images 1003A to 1003D showing the rows of the sweeping action, the horizontal zooming action, the rolling action, and the broom action from among the plurality of candidate actions. In addition, in this example, the reliability (matching degree) of the brooming action, the sweeping action, the horizontal zooming action, and the rolling action is expressed in images 1003A to 1003D. Thus, the operator can select an action to be performed by the excavator 100 by his own operation or automatically from among the actions of the recommended object in consideration of the reliability (matching degree).

In addition, only the action with the highest reliability (matching degree) (in this example, the sweeping action) may be described in image 1003. That is, the suggestion unit 302D may also suggest to the operator only the action with the highest reliability (matching degree) among multiple candidate actions in the specified operation through image 1003. In addition, only the actions (for example, the sweeping action and the horizontal zooming action) whose reliability (matching degree) is above a specified benchmark (for example, 30%) among multiple candidate actions may be described in image 1003. That is, the suggestion unit 302D may also suggest to the operator only the actions whose reliability (matching degree) is above a specified benchmark among multiple candidate actions.

Image 1004 is an image showing a target trajectory for each action of the recommended object described in image 1003. Image 1004 is displayed superimposed on image 1001 around image 1002. Thus, the operator can easily understand the target trajectory for each action of the recommended object by comparing image 1001 showing the state of the ground at the construction site around the shovel 100 and image 1002 showing the shovel 100. Image 1004 includes images 1004A to 1004D.

Image 1004A is an image showing a target trajectory of a sweeping motion.

Image 1004B is an image showing a target trajectory of the horizontal zoom-in operation.

Image 1004C is an image showing a target track for the rolling action.

Image 1004D is an image showing a target trajectory of the broom action.

In addition, the images 1004A to 1004D may be expressed in a manner that allows for differentiation between the portion of the target track that contacts the work object (ground) and the portion other than the target track. For example, the images 1004A to 1004D may have different colors for the portion of the target track that contacts the work object and the portion other than the target track. This can assist in providing a sense of perspective on the image 1001 of the images 1004A to 1004D corresponding to the target track.

In this example, a cursor with a pear-skin pattern is represented in image 1003A corresponding to the sweeping action. Also, in this example, image 1004A corresponding to the sweeping action of image 1004 is represented by a thicker line than images 1004B to 1004D corresponding to other actions. Thus, in this example, a state in which the sweeping action is selected is represented. For example, the operator can select any one of the sweeping action, the horizontal zooming action, the rolling action, and the broom action by specifying any one of images 1003A to 1003D using the input device 52. Similarly, for example, the operator can select any one of the sweeping action, the horizontal zooming action, the rolling action, and the broom action by specifying any one of images 1004A to 1004D using the input device 52.

Image 1005 is an icon for confirming execution of an action selected by the user (operator) from among actions to be suggested.

For example, the operator can automatically cause the shovel 100 to perform the selected operation by operating the image 1005 using the input device 52 .

The image 1006 is an icon for ending the operation suggestion function of the shovel 100. The same applies to images 1106, 1206, 1306, 1406, and 1506 described below.

Thus, in this example, the controller 30 displays a plurality of actions from a plurality of candidate actions in the ground leveling operation on the display device 50A together with their respective reliability (matching degree) with respect to the shape (terrain shape) of the current work object around the shovel 100. Thus, the controller 30 can suggest to the operator an action with a relatively high reliability (matching degree) with respect to the shape (terrain shape) of the current work object around the shovel 100.

＜Example 2＞

FIG. 11 is a diagram showing a second example (screen 1100 ) of display content of the display device 50A related to the operation suggestion function of the shovel 100 .

The following description will focus on the parts that are different from the first example, and the description of the same or corresponding contents as the first example may be simplified or omitted.

Screen 1100 includes images 1101 to 1106 .

Similar to the image 1001 in FIG. 10 , the image 1101 is an image showing work objects around the shovel 100 .

Similar to the diagram 1002 in FIG. 10 , the diagram 1102 is an image schematically showing the shovel 100 .

Image 1103 is an image showing a plurality of target trajectories of bucket 6 in a directory format with respect to an action of a suggested object suggested by suggestion unit 302D among a plurality of candidate actions. In this example, image 1103 includes images 1103A to 1103D showing rows of four target trajectories ("Sweep I" to "Sweep IV") of bucket 6, respectively, with respect to a sweep action as an action of a suggested object. Furthermore, in this example, the reliability (matching degree) of the four target trajectories of bucket 6 is shown in images 1103A to 1103D. Thus, the operator can select a target trajectory of bucket 6 to be executed by the excavator 100 through his own action or automatically from among the four target trajectories of bucket 6 with respect to an action of a suggested object (sweep action) in consideration of reliability.

In addition, only the target trajectory of the bucket 6 with the highest reliability (matching degree) (in this example, "Sweep I") may be expressed in the image 1103. That is, the suggestion unit 302D may also suggest to the operator only the action with the highest reliability (matching degree) among multiple target trajectories of an action of the suggestion object in the specified operation through the image 1103. Furthermore, only the target trajectories (for example, "Sweep I" and "Sweep II") whose reliability (matching degree) is above a specified benchmark (for example, 30%) among multiple target trajectories of the bucket 6 may be expressed in the image 1103. That is, the suggestion unit 302D may also suggest to the operator only the target trajectories whose reliability (matching degree) is above a specified benchmark among multiple target trajectories of the bucket 6 for an action of the suggestion object. The same may be true for the image 1203A described later.

Image 1104 is an image showing four target trajectories for one action of the suggested object described in image 1103. Image 1104 is displayed on image 1001 in an overlapping manner around image 1002, similarly to image 1004. Thus, the operator can easily grasp a plurality of target trajectories for one action (sweeping action) of the suggested object while comparing image 1101 showing the state of the ground at the construction site around the shovel 100 and image 1102 showing the shovel 100. Image 1104 includes images 1104A to 1104D.

Image 1104A is an image showing a target trajectory of a swipe action corresponding to image 1103A (“Swipe I”).

Image 1104B is an image showing a target trajectory of the sweep action corresponding to image 1103B (“Sweep II”).

Image 1104C is an image showing a target trajectory of the sweep action corresponding to image 1103C (“Sweep III”).

Image 1104D is an image showing a target trajectory of the sweep action corresponding to image 1103D (“Sweep IV”).

In this example, a cursor with a pear-skin pattern is represented in image 1103A corresponding to "Sweep Action I". Furthermore, in this example, image 1104A corresponding to the target track of "Sweep Action I" in image 1104 is represented by a thicker line than images 1104B to 1104D corresponding to other target tracks. Thus, in this example, a state in which the target track of "Sweep I" is selected among the four target tracks is represented. For example, the operator can select any one of the four target tracks by specifying any one of images 1103A to 1103D using the input device 52. Similarly, for example, the operator can select any one of the four target tracks of the bucket 6 by specifying any one of images 1104A to 1104D using the input device 52.

The image 1105 is an icon for executing one action (sweep-out action) of the recommended object so that the bucket 6 moves along a target track selected by the user (operator) from among a plurality of target tracks.

For example, the operator can automatically cause the shovel 100 to perform one action of the recommended object so that the bucket 6 moves along the selected target track by operating the image 1105 using the input device 52 .

Thus, in this example, the controller 30 displays on the display device 50A a plurality of target trajectories of the bucket 6 for one action related to the ground leveling work and the respective reliability (matching degree) with respect to the shape (terrain shape) of the current work object around the shovel 100. Thus, the controller 30 can suggest to the operator a plurality of target trajectories of the bucket 6 having a relatively high reliability (matching degree) with respect to the shape (terrain shape) of the current work object around the shovel 100.

＜Case 3＞

FIG. 12 and FIG. 13 are diagrams showing a third example (screens 1200 and 1300 ) of the display content of the display device 50A related to the operation suggestion function of the shovel 100 .

The following description will focus on the parts that are different from the first and second examples described above, and the description of the same or corresponding contents as the first and second examples described above may be simplified or omitted.

Screen 1200 includes images 1201 to 1206 .

Similar to the image 1001 in FIG. 10 , the image 1201 is an image showing work objects around the shovel 100 .

Similar to the image 1002 in FIG. 10 , the image 1202 schematically shows the shovel 100 .

Similar to the image 1003 in FIG. 10 , the image 1203 is an image showing the actions of the suggested objects suggested by the suggestion unit 302D from among the plurality of candidate actions in the form of a catalog. In this example, the actions of the suggested objects in the image 1203 include images 1203A to 1203D showing the rows of the sweeping action, the horizontal zooming action, the rolling action, and the broom action from among the plurality of candidate actions. In addition, in this example, the reliability (matching degree) for the brooming action, the sweeping action, the horizontal zooming action, and the rolling action is expressed in the images 1203A to 1203D.

Also, similar to the image 1103 of FIG. 11 , FIG. 1203A is an image showing multiple target tracks of the bucket 6 in a catalog format with respect to one action (sweep action) of the recommended object. In this example, in the image 1203A, with respect to the sweep action having the highest reliability (matching degree) among multiple candidate actions, there are images 1203A1 to 1203A4 of rows showing the four target tracks (“sweep I” to “sweep IV”) of the bucket 6. Also, in this example, the reliability (matching degree) of the four target tracks of the bucket 6 is shown in FIGS. 1203A1 to 1203A4.

Similar to the image 1004 in Fig. 10 , the image 1204 is an image showing a target trajectory for each action of the recommended object described in the image 1203. The image 1204 includes images 1204A to 1204D.

Image 1204A is an image showing a target trajectory of the sweeping operation, and more specifically, is an image showing a target trajectory (“Sweep I”) with the highest reliability among the target trajectories (“Sweep I” to “Sweep IV”) of the bucket 6 corresponding to the sweeping operation.

Images 1204B to 1204D are the same as images 1004B to 1004D in FIG. 10 , and thus description thereof will be omitted.

In this example, a cursor with a pear-skin pattern is depicted in image 1003A corresponding to the swipe action. Also, in this example, image 1004A corresponding to the swipe action of image 1004 is depicted with a thicker line than images 1004B to 1004D corresponding to other actions. Thus, in this example, the state in which the swipe action is selected is depicted.

The image 1205 is an icon for confirming the execution of the action selected by the user (operator) from the actions to be suggested.

In addition, image 1205 is an icon for transitioning to screen 1300 for selecting four target tracks of bucket 6 corresponding to the sweeping action when the sweeping action with the highest reliability among the recommended actions is selected. That is, in the state of screen 1200, if image 1205 is operated by input device 52, transition to screen 1300 occurs.

Screen 1300 includes images 1301 - 1306 .

Image 1303 is an image showing suggested actions suggested by suggestion unit 302D from among the plurality of candidate actions in the form of a list, similar to image 1203 in FIG12. Specifically, image 1303 includes images 1203A to 1203D showing rows of swiping action, horizontal zooming action, rolling action, and broom action, respectively, from among the plurality of candidate actions as suggested actions.

Also, similar to the image 1203A of FIG12 , FIG1303A is an image showing multiple target trajectories of the bucket 6 in a catalog format with respect to one action (sweep action) of the recommended object. Specifically, in the image 1203A, with respect to the sweep action having the highest reliability (matching degree) among the multiple candidate actions, there are images 1303A1 to 1303A4 of rows showing four target trajectories of the bucket 6 ("sweep I" to "sweep IV"), respectively.

Similar to the image 1104 of Fig. 11 , the image 1304 is an image showing four target trajectories for one action of the object of advice expressed in the image 1103. Specifically, the image 1304 includes images 1304A to 1304D.

Images 1304A to 1304D are respectively the same as images 1104A to 1104D in FIG. 11 , and thus description thereof is omitted.

In this example, a cursor with a pear skin pattern is depicted in image 1303A1 corresponding to "Sweep Action I". In addition, in this example, image 1304A corresponding to the target track of "Sweep Action I" in image 1304 is depicted with a thicker line than images 1304B to 1304D corresponding to other target tracks. Thus, in this example, the state in which the target track of "Sweep Action I" is selected among the four target tracks is depicted.

The image 1305 is an icon for executing one action (sweep-out action) of the recommended object so that the bucket 6 moves along a target track selected by the user (operator) from among a plurality of target tracks.

Thus, in this example, the controller 30 displays multiple actions among multiple candidate actions related to the ground leveling operation together with the reliability on the display device 50A, and displays multiple target trajectories for the action with the highest reliability on the display device 50A. Thus, the controller 30 can suggest to the operator multiple actions with relatively high reliability relative to the shape (terrain shape) of the current work object around the excavator 100 and multiple target trajectories of the bucket 6 for the action with the highest reliability.

＜Case 4＞

FIG. 14 is a diagram showing a fourth example (screen 1400 ) of display content of the display device 50A related to the operation suggestion function of the shovel 100 .

The following description will focus on the parts that are different from the first to third examples described above, and the description of the same or corresponding contents as the first to third examples described above may be simplified or omitted.

Screen 1400 includes images 1401 to 1406 .

Similar to the image 1001 in FIG. 10 , the image 1401 is an image showing work objects around the shovel 100 .

Similar to the image 1002 in FIG. 10 , the image 1402 schematically shows the shovel 100 .

Similar to the image 1003 in Fig. 10, the image 1403 is an image showing the actions of the suggestion target suggested by the suggestion unit 302D from among the plurality of candidate actions in the form of a list. Specifically, the actions of the suggestion target in the image 1403 include images 1403A to 1403D showing rows of the swiping action, the horizontal zooming action, the rolling action, and the broom action from among the plurality of candidate actions.

Similar to the image 1004 in Fig. 10 , the image 1404 is an image showing a target trajectory for each action of the recommended object expressed in the image 1403. Specifically, the image 1404 includes images 1404A to 1404D.

Images 1404A to 1404D are respectively the same as images 1004A to 1004D in FIG. 10 , and thus description thereof is omitted.

In this example, the worker W is shown in the image region where the image 1403A of the image 1401 is displayed superimposed. Therefore, if the most reliable action (sweep action) is selected, the accessory AT may come too close to the worker W or the accessory AT may come into contact with the worker W.

In contrast, in this example, the image 1403B corresponding to the horizontal zoom-in action is represented by a cursor with a pear-skin pattern, and the image 1404B corresponding to the horizontal zoom-in action is represented by a thicker line than the images 1404A, 1404C, and 1404D corresponding to the other actions. That is, in this example, the operator selects an action (horizontal zoom-in action) different from the sweeping action through the input device 52, and wants the shovel 100 to execute it. This can prevent the attachment AT from getting too close to the worker W or coming into contact with the worker W.

Image 1405 is the same as image 1005 in FIG. 10 , and therefore description thereof will be omitted.

Thus, in this example, by superimposing the target trajectory of the action of the recommended object on the image 1401 representing the state around the shovel 100, the operator can understand the relationship between the target trajectory and obstacles such as the worker W at the construction site. Therefore, the convenience of the operator or the working efficiency of the shovel 100 can be improved, and the safety of the shovel 100 can be improved.

＜Case 5＞

FIG. 15 is a diagram showing a fifth example (screen 1500 ) of display content of the display device 50A related to the operation suggestion function of the shovel 100 .

The following description will focus on the parts that are different from the first to fourth examples described above, and the description of the same or corresponding contents as the first to fourth examples described above may be simplified or omitted.

Screen 1500 includes images 1501 to 1506 .

Similar to the image 1001 in FIG. 10 , the image 1501 is an image showing a work object around the shovel 100 .

Similar to the image 1002 in FIG. 10 , the image 1502 is an image schematically showing the shovel 100 .

Similar to the image 1003 in Fig. 10, the image 1503 is an image showing the actions of the suggestion target suggested by the suggestion unit 302D from among the plurality of candidate actions in the form of a list. Specifically, the actions of the suggestion target in the image 1503 include images 1503A to 1503D showing rows of the horizontal zoom action, the sweep action, the rolling action, and the broom action from among the plurality of candidate actions.

Similar to the image 1004 in Fig. 10 , the image 1504 is an image showing a target trajectory for each action of the recommended object expressed in the image 1503. Specifically, the image 1504 includes images 1504A to 1504D.

Images 1404A to 1404D are respectively the same as images 1004A to 1004D in FIG. 10 , and thus description thereof is omitted.

Image 1004A is an image showing a target trajectory of a sweeping motion.

Image 1004B is an image showing a target trajectory of the horizontal zoom-in operation.

Image 1004C is an image showing a target track for the rolling action.

Image 1004D is an image showing a target trajectory of the broom action.

In this example, the image area where images 1504A, 1504C, and 1504D of image 1501 are displayed overlappingly is included in area 1501A where the ground leveling work has been completed. Therefore, if the action with the highest reliability (the horizontal zoom-in action corresponding to image 1504A) is selected, not only will useless work be performed on the area where the ground leveling work has been completed, but the area will also be affected by the useless work and need to be restored to its original state. As a result, the working efficiency of the excavator 100 may be reduced or the progress of the work at the construction site may be delayed.

In contrast, in this example, a cursor with a pear-skin pattern is depicted in the image 1503B corresponding to the sweeping action, and the image 1504B corresponding to the sweeping action is depicted with a thicker line than the images 1504A, 1504C, and 1504D corresponding to the other actions. That is, in this example, the operator selects an action (horizontal zooming action) different from the sweeping action and wants the excavator 100 to perform it. This can prevent the excavator 100 from performing work on an area where work has already been completed. Therefore, it is possible to prevent the working efficiency of the excavator 100 from being reduced or the progress of work at the construction site from being delayed.

Furthermore, in this example, an image indicating an area where the ground leveling work has been completed is superimposed on the area 1501A (for example, an image where the area 1501A is covered with a diagonal line, etc.). Thus, the operator can more reliably understand that the area 1501A is an area where the ground leveling work has been completed. At this time, information related to the area where the work has been completed in the construction site is transmitted from, for example, the information processing device 200 to the shovel 100.

Image 1505 is the same as image 1005 in FIG. 10 , and therefore description thereof will be omitted.

Thus, in this example, by superimposing the target trajectory of the action of the recommended object on the image 1401 representing the state around the shovel 100, the operator can understand the relationship between the target trajectory and the completion status of the work at the construction site, etc. Therefore, the operator can select a more appropriate action or target trajectory according to the completion status of the work at the construction site.

[Another example of the action suggestion function]

Next, another example of the action suggestion function will be described.

The first example or the second example of the above-mentioned action suggestion function may be combined in its content as appropriate, or may be modified or altered.

For example, in the first or second example of the above-mentioned action suggestion function, the suggestion unit 302D may suggest only a suggestion target action among a plurality of candidate actions executable in a predetermined task and omit suggestion of a target trajectory corresponding to the suggestion target action.

Furthermore, in the first example or the second example or the modified example of the above-mentioned action suggestion function, the suggestion unit 302D can not only suggest a target track with a relatively high reliability (matching degree) relative to the shape of the work object around the excavator 100, but also deliberately suggest an action or target track with a relatively low reliability (matching degree). For example, an action with a relatively low reliability (matching degree) is an action of the excavator 100 or a target track of the bucket 6 whose reliability (matching degree) is lower than a predetermined reference calculated based on the learned model LM. Furthermore, an action with a relatively low reliability (matching degree) can also be an action of the excavator 100 or a target track of the bucket 6 calculated based on a learned model different from the learned model LM, that is, a learned model that has been machine-learned using a training data set including inappropriate actions or target tracks. Thus, the operator's reliance on the action suggestion function can be suppressed, and the operator can be urged to use the action suggestion function according to his own appropriate judgment. At this time, if an action with a relatively low reliability or matching degree is selected, the situation can also be recorded as a log in the auxiliary storage device 30A of the controller 30. Furthermore, the log may also include identification information of the operator. Thus, for example, the support device 150 (controller 30) can label an operator who has a high probability of selecting an action of the shovel 100 or a target track of the bucket 6 with relatively low reliability (matching degree). Therefore, the manager of the construction site can manage the use of the action suggestion function for each operator by checking the log or label afterward.

Furthermore, in the first example or the second example or the modified example of the above-mentioned action suggestion function, part or all of the functions of the support device 150 may be transferred to the remote operation support device 300. For example, the function of the suggestion unit 302D is transferred to the remote operation support device 300. Furthermore, in addition to the suggestion unit 302D, the function of the inference unit 302C may be transferred to the remote operation support device 300. Furthermore, in addition to the suggestion unit 302D and the inference unit 302C, the function of the work object shape acquisition unit 302B may be transferred to the remote operation support device 300. At this time, the image data of the camera 40 is transmitted from the shovel 100 to the remote operation support device 300.

Furthermore, in the first example or the second example or the modified example of the above-mentioned action suggestion function, part or all of the functions of the support device 150 may be transferred to the information processing device 200. For example, the work object shape acquisition unit 302B is transferred to the information processing device 200. At this time, the image data of the camera 40 is sent from the shovel 100 to the information processing device 200. Furthermore, in addition to the work object shape acquisition unit 302B, the function of the inference unit 302C may be transferred to the information processing device 200. Furthermore, in addition to the work object shape acquisition unit 302B and the inference unit 302C, the function of the suggestion unit 302D may be transferred to the information processing device 200. Thus, for example, the portable information processing device 200 brought into the cockpit 10 can suggest one or more actions to the operator from a plurality of candidate actions of the shovel 100 to be performed in a predetermined operation, or suggest the trajectory of the action.

Furthermore, the support device 150 involved in the first example or the second example or the modified example of the above-mentioned action suggestion function may also suggest one or more actions among a plurality of candidate actions in a predetermined operation of another construction machine different from the excavator 100 to the operator. For example, the other construction machine is a forestry machine having a felling device. In this case, the support device 150 may suggest an action for one or more trees based on the action of the felling device from among the actions for a plurality of candidate trees existing on site.

[Role of support equipment (1)]

Next, the operation of the support device according to this embodiment will be described.

In the present embodiment, the support device includes an acquisition unit and a suggestion unit. The support device is, for example, the support device 150. The construction machine is, for example, the excavator 100. The acquisition unit is, for example, the work object shape acquisition unit 302B. The suggestion unit is, for example, the suggestion unit 302D. Specifically, the acquisition unit acquires data related to the shape of the work object (for example, the shape of the terrain) around the construction machine. Moreover, the suggestion unit suggests to the user an action from among a plurality of candidate actions of the construction machine in a specified operation based on the data related to the shape of the work object around the construction machine acquired by the acquisition unit.

Furthermore, in the present embodiment, the construction machine may include the above-mentioned assist device.

Thus, the support device can, for example, suggest to the operator of the construction machine an action that better matches the shape of the work object around the construction machine from among a plurality of candidate actions that can be performed in a predetermined operation. Therefore, the construction machine can be made to perform actions more accurately. Therefore, the working efficiency of the construction machine can be improved.

Furthermore, in the present embodiment, the support device includes an inference unit. The inference unit is, for example, the inference unit 302C. Specifically, the inference unit uses a learned model that has been machine-learned based on training data related to the actions of the construction machinery based on the operation of a relatively proficient operator that establishes a correspondence with the shape of the work object, and infers an action that matches the shape of the work object around the construction machinery from a plurality of candidate actions based on data related to the shape of the work object around the construction machinery acquired by the acquisition unit. The learned model is, for example, the learned model LM. Furthermore, the suggestion unit may also suggest an action from a plurality of candidate actions based on the inference result of the inference unit.

As a result, the support device can use the learned model to suggest an action that better matches the shape of the work object around the construction machine from a plurality of candidate actions that can be performed in a predetermined work.

Furthermore, in the present embodiment, the suggestion unit may suggest a plurality of actions from among a plurality of candidate actions to the user based on the data on the shape of the work object around the construction machine acquired by the acquisition unit.

Thus, the support device can provide options to the operator, and urge the operator to make a judgment according to his/her own will. Therefore, by reflecting the judgment of the operator, the construction machine can be operated more accurately.

Furthermore, in the present embodiment, the plurality of actions to be suggested may include an action that has a relatively low degree of matching with the shape of the work object around the construction machine among the plurality of candidate actions.

Thus, the support device can urge the operator to make a judgment according to his/her will. Therefore, by reflecting the judgment of the operator, the construction machine can be operated more accurately.

Furthermore, in this embodiment, the suggestion unit may suggest an action from among a plurality of candidate actions and a degree of matching of the action with respect to the shape of the work object around the construction machine based on the data related to the shape of the work object around the construction machine acquired by the acquisition unit.

Thus, the support device can provide the operator with materials for determining whether to execute the action. Therefore, the support device can urge the operator to make a more appropriate judgment, thereby causing the construction machine to operate more accurately.

Furthermore, in this embodiment, the suggestion unit may also suggest multiple actions among multiple candidate actions and the degree of matching of each of the multiple actions with respect to the shape of the work object around the construction machine based on data related to the shape of the work object around the construction machine acquired by the acquisition unit.

Thus, the support device can provide the operator with a plurality of options related to the operation of the construction machine and can provide judgment materials for the selections. Therefore, the support device can urge the operator to make a more appropriate judgment, thereby causing the construction machine to operate more accurately.

Furthermore, in this embodiment, the suggestion unit may suggest an action among a plurality of candidate actions and a trajectory of a working part of the construction machine based on the action based on the data on the shape of the working object around the construction machine acquired by the acquisition unit.

Thus, the support device can provide the operator with materials for determining whether to execute the action. Therefore, the support device can urge the operator to make a more appropriate judgment, thereby causing the construction machine to operate more accurately.

Furthermore, in the present embodiment, the suggestion unit may also suggest actions among the plurality of candidate actions and a plurality of trajectories of the work site based on the actions based on the data on the shape of the work object around the construction machine acquired by the acquisition unit.

Thus, the support device can provide the operator with a plurality of options related to the trajectory of the work site corresponding to the proposed target action. Therefore, the support device can urge the operator to make a more appropriate judgment, thereby allowing the construction machine to operate more accurately.

Furthermore, in the present embodiment, the suggestion unit may also suggest an action from a plurality of candidate actions and a plurality of tracks of a working part based on the action, and a degree of matching of each of the plurality of tracks with respect to the shape of the working object surrounding the construction machine, based on data related to the shape of the working object surrounding the construction machine acquired by the acquisition unit.

Thus, the support device can provide the operator with a plurality of options of the work site and can provide judgment materials for the selection. Therefore, the support device can urge the operator to make a more appropriate judgment, thereby allowing the construction machine to operate more accurately.

Furthermore, in this embodiment, the support device may also include a display unit. The display unit is, for example, the display device 50A. Furthermore, the suggestion unit may superimpose the trajectory of the working part based on the action of the suggestion object among the plurality of candidate actions on the image showing the state of the surroundings of the construction machine and display it on the display unit.

As a result, the operator can more easily understand the relationship between the track of the working area and the shape of the working object around the construction machine. Therefore, the support device can urge the operator to make a more appropriate judgment, so that the construction machine can operate more accurately.

Furthermore, in the present embodiment, the suggestion unit may display on the display unit in different forms a track portion that contacts the work object and other track portions among tracks based on the action to be suggested among the plurality of candidate actions.

Thus, the support device can assist the operator in gaining a sense of perspective on an image showing the state of the periphery of the construction machine, thereby enabling the operator to more accurately grasp the trajectory.

Furthermore, in the present embodiment, the support device includes a control unit. The control unit is, for example, the action control unit 302E. Specifically, the control unit can cause the construction machine to automatically execute the action suggested by the suggestion unit based on the input of an instruction from the user.

This can further improve the convenience for the user (operator) and enable the construction machine to operate more accurately.

Furthermore, in the present embodiment, the acquisition unit may acquire data related to the shape of the work object around the construction machine by predicting the shape of the work object around the construction machine after the operation automatically executed by the control unit is performed.

Thus, when the support device supports the work of the construction machine while repeatedly providing support for the operation of the construction machine, it is possible to provide support for the operation of the construction machine more flexibly. Therefore, the work efficiency of the construction machine can be further improved.

[Function related to the generation of tracks of the working part of the excavator]

Next, a function related to the generation of a trajectory of a working portion of the shovel 100 (hereinafter referred to as a “target trajectory”) will be described with reference to FIGS. 16 to 19 in addition to FIGS. 1 to 5 .

In the following, the same reference numerals are used for structures identical or corresponding to the structures related to the above-mentioned action suggestion function, and the description will be centered around parts different from the above-mentioned action suggestion function, and the description of the contents identical or corresponding to the above-mentioned action suggestion function may be simplified or omitted.

＜Functional structure＞

FIG. 16 is a block diagram showing a first example of a functional structure related to the generation of a target trajectory of the working part of the shovel 100. FIG. 17 is a diagram showing an example (screen 700) of a screen related to the generation of a target trajectory of the working part of the shovel 100 displayed on the display device 50A. FIG. 18 is a diagram showing another example (screen 800) of a screen related to the generation of a target trajectory of the working part of the shovel 100 displayed on the display device 50A. FIG. 19 is a diagram showing still another example (screen 900) of a screen related to the generation of a target trajectory of the working part of the shovel 100 displayed on the display device 50A.

When the shovel 100 is remotely operated, the same screen as that in FIG. 17 and FIG. 18 is displayed on the remote operation support device 300 (display device).

The working part of the shovel 100 is, for example, the blade tip or the back surface of the bucket 6 .

The shovel 100 includes a support device 150 . The support device 150 performs support related to the work of the shovel 100 .

16 , the support device 150 includes an operation device 26, a controller 30, a camera device 40, and an output device 50. Furthermore, when the shovel 100 is remotely operated, the support device 150 may include a communication device 60.

The controller 30 includes an operation log providing unit 301 and a work supporting unit 302 as functional units.

In addition, when there are a plurality of shovels 100 included in the operation support system SYS, there may be a shovel 100 in which the controller 30 includes only the former of the operation log providing unit 301 and the operation support unit 302, and a shovel 100 in which the controller 30 includes only the latter. In this case, the former shovel 100 has only a function of acquiring the operation log of the shovel 100 used for the operation support function (function related to the generation of the trajectory of the work site) of the operator in the latter shovel 100 and providing it to the information processing device 200.

The information processing device 200 includes an action log acquisition unit 2001 , an action log storage unit 2002 , a training data generation unit 2003 , a machine learning unit 2004 , a learned model storage unit 2005 , and a transmission unit 2006 as functional units.

The action log providing unit 301 is a functional unit that obtains the action log of the shovel 100, which is the original data for realizing the function of generating the target trajectory of the working part of the shovel 100, and provides it to the information processing device 200. Specifically, the action log when an operator with a long driving experience and relatively experience of the shovel 100 (hereinafter referred to as "experienced operator" for convenience) operates the shovel 100 is obtained and provided to the information processing device 200.

The operation log of the shovel 100 includes data related to the shape of the work object around the shovel 100 and data related to the operation of the shovel 100 performed on the shape of the work object. The data related to the shape of the work object around the shovel 100 is, for example, data related to the topographic shape of the ground at the construction site which is the work object of the shovel 100. The data related to the shape of the work object of the shovel 100 is, for example, image data of the camera 40 or three-dimensional data of the work object obtained from the image data. The data related to the operation of the shovel 100 is, for example, data indicating the operation content of the operator. The data indicating the operation content of the operator is, for example, output data of the operation pressure sensor 29 in the case of a hydraulic pilot type operating device 26 or output data (operation signal data) of the operating device 26 in the case of an electric type operating device 26. In addition, the data related to the operation of the shovel 100 may be data indicating the operation state of the shovel 100 actually performed according to the operation of the operator. The data indicating the operation state of the shovel 100 is, for example, output data of the sensors S1 to S5 or data related to the posture state of the shovel 100 acquired based on the output data of the sensors S1 to S5 .

The operation log providing unit 301 includes an operation log recording unit 301A, an operation log storage unit 301B, and an operation log transmitting unit 301C.

The action log recording unit 301A obtains the action log of the shovel 100 and records it in the action log storage unit 301B. For example, each time the shovel 100 performs an action, the action log recording unit 301A records data related to the shape of the work object around the shovel 100 at the start of the action or immediately before the action and data related to the action of the shovel 100 in the action log storage unit 301B.

The action log storage unit 301B stores the action logs of the shovel 100 in an accumulated form. For example, the action log storage unit 301B stores data related to the shape of the work object around the shovel 100 and data related to the action of the shovel 100 in an associated form for each action of the shovel 100. Specifically, the action log storage unit 301B can store record data indicating the correspondence between the data related to the shape of the work object around the shovel 100 and the data related to the action of the shovel 100 for each action of the shovel 100, so as to construct a database of the action logs.

Furthermore, the operation logs of the operation log storage unit 301B that have been transmitted to the information processing device 200 by the operation log transmission unit 301C described later may be deleted afterwards.

The action log sending unit 301C sends the action log of the shovel 100 stored in the action log storage unit 301B to the information processing device 200 through the communication device 60. In addition, the action log sending unit 301C may send record data of the correspondence between the shape of the work object around the shovel 100 and the data related to the action of the shovel 100 for each action of the shovel 100 to the information processing device 200.

For example, the operation log transmission unit 301C transmits the operation log of the shovel 100 which has not been transmitted and is stored in the operation log storage unit 301B to the information processing device 200 in response to a signal (hereinafter referred to as a "transmission request signal") which is received from the information processing device 200 requesting transmission of the operation log of the shovel 100. Furthermore, the operation log transmission unit 301C may automatically transmit the operation log of the shovel 100 which has not been transmitted and is stored in the operation log storage unit 301B to the information processing device 200 at a predetermined time. The predetermined time is, for example, when the operation of the shovel 100 is stopped (the key switch is turned "off") or when the operation is started (the key switch is turned "on").

The operation log acquisition unit 2001 acquires the operation log of the shovel 100 received from the shovel 100 .

The operation log acquisition unit 2001 automatically transmits a transmission request signal to the shovel 100 according to the operation of the user of the information processing device 200 or at a predetermined time, thereby acquiring the operation log of the shovel 100. In addition, the operation log acquisition unit 2001 may also acquire the operation log of the shovel 100 transmitted from the shovel 100 at a predetermined time.

The action log of the shovel 100 acquired by the action log acquisition unit 2001 is stored in an accumulated form in the action log storage unit 2002. For example, similarly to the case of the action log storage unit 301B, the action log storage unit 2002 stores data related to the shape of the work object around the shovel 100 and data related to the action of the shovel 100 in an associated form for each action of the shovel 100.

The training data generation unit 2003 generates training data for machine learning based on the operation log of the shovel 100 in the operation log storage unit 2002. The training data generation unit 2003 may automatically generate the training data by batch processing, or may generate the training data based on the input from the user of the information processing device 200. The training data is a combination of data related to the shape of the work object around the shovel 100 as input data and data indicating the track (track) of the work part of the shovel 100 corresponding to the input data as correct answer output data (hereinafter referred to as "correct answer data").

The data indicating the trajectory of the working part of the shovel 100 is generated based on the output data of the sensors S1 to S5 included in the data related to the operation of the shovel 100 , for example.

The machine learning unit 2004 generates a learned model LM by performing machine learning on the basic learning model based on the training data group generated by the training data generating unit 2003. The learned model LM (basic learning model) includes, for example, a neural network such as DNN (Deep Neural Network).

The learned model LM, for example, takes the type of motion of the shovel 100 and data related to the shape of the work object around the shovel 100 as input conditions, and outputs data indicating the target trajectory of the working part of the shovel 100 and the predicted probability. In addition, the data indicating the target trajectory of the working part of the shovel 100 and the type of motion of the shovel 100 of the learned model LM include, for example, excavation motion, sweeping motion, horizontal pull-in motion, rolling motion, and broom motion. The sweeping motion is, for example, a motion in which the bucket 6 is pushed forward along the ground by operating the attachment AT and the back of the bucket 6 is used to sweep the sand and soil forward. In the sweeping motion, for example, the attachment AT performs a lowering motion of the boom 4 and an opening motion of the dipper arm 5. The horizontal pull-in motion is, for example, a motion in which the tip of the bucket 6 is moved to be pulled toward the front side substantially horizontally along the ground to smooth the unevenness of the ground surface by operating the attachment AT. In the horizontal pull-in motion, for example, the attachment AT performs a lifting motion of the boom 4 and a closing motion of the dipper arm 5. The rolling action is, for example, an action of operating the attachment AT to compact the ground with the back of the bucket 6. Furthermore, the rolling action may be an action of pushing the bucket 6 forward along the ground, sweeping the sand and soil with the back of the bucket 6 to a predetermined position in front, and then compacting the ground at a predetermined position with the back of the bucket 6. In the rolling action, for example, the attachment AT performs a lowering action of the boom 4 while compacting the ground. The sweeping action is, for example, an action of operating the upper rotating body 3 to rotate the bucket 6 left and right in a state along the ground. Furthermore, the sweeping action may be, for example, an action of operating the attachment AT and the upper rotating body 3 to rotate the bucket 6 left and right alternately in a state along the ground and pushing the bucket 6 forward. In the sweeping action, for example, the upper rotating body 3 repeats the left and right rotation action alternately. Furthermore, in the sweeping action, for example, in addition to the left and right rotation action of the upper rotating body 3, the lowering action of the boom 4 and the opening action of the dipper arm 5 may be performed in the same manner as in the case of the sweeping action. The prediction probability indicates the reliability of the target trajectory of the working part. This is because, as described above, the learned model LM reflects the action log when the excavator 100 is operated by a skilled person, and it is believed that the higher the prediction probability, the higher the reliability of the target trajectory of the working part. In addition, the prediction probability indicates the matching degree of the target trajectory of the working part relative to the shape of the working object around the excavator 100 as an input condition. This is because it is believed that the higher the prediction probability, the higher the possibility that the skilled person judges that the candidate action matches the shape of the working object. For example, the learned model LM is generated for each operation such as ground leveling operation, slope construction operation, and filling operation.

The learned model LM output by the machine learning unit 2004 is stored in the learned model storage unit 2005 .

The transmitting unit 2006 transmits the learned model LM to the shovel 100 .

For example, when a learned model LM is generated by the machine learning unit 2004, the transmission unit 2006 transmits the most recently generated learned model LM to the shovel 100. Furthermore, the transmission unit 2006 may transmit the latest learned model LM in the learned model storage unit 2005 to the shovel 100 in response to a signal received from the shovel 100 requesting the transmission of the learned model LM.

The work support unit 302 is a functional unit for supporting the work of the shovel 100 based on the operation of the operator.

The work support unit 302 includes a learned model storage unit 302A, a work object shape acquisition unit 302B, an action selection unit 302F, a condition setting unit 302G, a trajectory generation unit 302H, a display processing unit 302I, and an action control unit 302E.

The learned model LM transmitted from the information processing device 200 and received via the communication device 60 is stored in the learned model storage unit 302A.

The work object shape acquisition unit 302B acquires data on the shape (topographic shape) of the work object around the shovel 100 based on the output of the imaging device 40 or the distance sensor.

The action selection unit 302F selects an action (type) of the shovel 100 from a plurality of action candidates based on an input from a user (operator) received through the input device 52. Furthermore, when the shovel 100 is remotely operated, the action selection unit 302F may select an action of the shovel 100 from a plurality of action candidates based on an input from a user (operator) using the remote operation support device 300 received through the communication device 60.

The condition setting unit 302G sets the preconditions related to the generation of the target trajectory of the working part of the excavator 100 based on the input from the user (operator) received through the input device 52. When the excavator 100 is remotely operated, the condition setting unit 302G may also set the preconditions related to the target trajectory of the excavator 100 based on the input from the user (operator) using the remote operation support device 300 received through the communication device 60. In addition, the condition setting unit 302G may also automatically set the preconditions without relying on the input from the user. For example, the condition setting unit 302G may also automatically set the preconditions based on a learning model generated by using the history of the data related to the shape of the working object and the data combined with the preconditions set for the shape of the working object as a training data group. At this time, the condition setting unit 302G may also automatically correct the set preconditions based on the input from the user.

The prerequisite is, for example, a point in the terrain shape around the excavator 100 that becomes a target when the excavator 100 operates (hereinafter referred to as a "target point"). The target point includes, for example, a target point through which the working part passes when the excavator 100 operates or a point corresponding to a position where sand and soil are discharged from the bucket 6 when the excavator 100 operates. In addition, the prerequisite may also include the posture state of the bucket 6 at the target point (the posture angle of the bucket 6).

The track generation unit 302H generates a target track of the working part of the shovel 100 based on the data acquired by the work object shape acquisition unit 302B, the target shape of the work object, the action selected by the action selection unit 302F, and the precondition set by the condition setting unit 302G. The target shape of the work object is, for example, a target construction surface representing a plane or a curved surface as a construction object formed by working on the work object (the ground at the construction site). The target shape of the work object is set, for example, based on a parameter representing a plane or a curved surface input by a user through the input device 52 or the remote operation support device 300 (input device). In addition, the target shape of the work object can also be transmitted to the shovel 100 from an external device such as the information processing device 200. The track generation unit 302H uses the data acquired by the work object shape acquisition unit 302B, the target shape of the work object, the action selected by the action selection unit 302F, and the precondition set by the condition setting unit 302G as input data, and applies the learned model LM. Furthermore, the trajectory generation unit 302H may output the target trajectory of the working part from the learning model LM by using the target shape of the working object, the action selected by the action selection unit 302F, and the data acquired by the working object shape acquisition unit 302B as input data. Furthermore, the trajectory generation unit 302H may generate the target trajectory of the working part by optimizing the output target trajectory of the working part according to the precondition set by the condition setting unit 302G.

The display processing unit 302I displays a screen related to the generation of the target trajectory of the working part of the shovel 100 on the display device 50A (refer to Figures 17 and 18). The screen related to the generation of the target trajectory of the working part of the shovel 100 includes, for example, an operation screen for the user (operator) to perform operation input related to the action of the shovel 100 selected by the action selection unit 302F or the precondition set by the condition setting unit 302G. In addition, the screen related to the generation of the target trajectory of the working part of the shovel 100 includes a display screen showing the target trajectory of the working part of the shovel 100 generated by the trajectory generation unit 302H. In addition, when the shovel 100 is remotely operated, the display processing unit 302I may also send data related to the screen related to the generation of the target trajectory of the working part of the shovel 100 to the remote operation support device 300 through the communication device 60. Thereby, the display processing unit 302I can display a screen related to the generation of the target trajectory of the working part of the shovel 100 on the remote operation support device 300 (display device).

For example, as shown in FIG. 17 and FIG. 18 , the display processing unit 302I displays screen images 700 and 800 on the display device 50A.

As shown in FIG. 17 , a screen 700 includes images TG, CG, SB, and PB1.

The image TG is an image showing the terrain shape around the shovel 100. The image TG is generated based on the data acquired by the work object shape acquisition unit 302B. In this example, the image TG is an image showing the terrain shape around the shovel 100 observed from a predetermined viewpoint outside the shovel 100. The predetermined viewpoint can be changed, for example, based on an input from a user (operator) through the input device 52 or the remote operation support device 300 (input device).

The image CG is an image showing the shovel 100 .

The positional relationship between the images TG and CG is set to be the same as the actual positional relationship between the terrain shape around the shovel 100 and the shovel 100 .

The image SB is an image showing candidate actions selectable by the action selection unit 302F. In this example, the image SB includes images SB1 to SB5 showing candidate actions of the shovel 100 that can be performed in the ground leveling work.

The image SB1 is an operation icon for the user to select a combination of the excavation operation and the soil displacement operation of the shovel 100 .

The image SB2 is an operation icon for the user to select the sweeping operation of the shovel 100 .

The image SB3 is an operation icon for the user to select the horizontal zoom-in operation of the shovel 100 .

The image SB4 is an operation icon for the user to select the broom operation of the shovel 100 .

The image SB5 is an operation icon for the user to select the rolling operation of the shovel 100 .

The user can designate any one of the images SB1 to SB5 through the input device 52 or the remote operation support device 300 (input device), and select the action of the shovel 100 through the action selection unit 302F. In this example, the cursor (the pear skin pattern in the figure) is located on the image SB1, and the state of the action combination of the excavation action and the soil displacement action of the shovel 100 is selected.

In addition, in the image SB, in addition to the images SB1 to SB5, operation icons for the user to select other actions different from the actions corresponding to the images SB1 to SB5 may be displayed. In addition, in the image SB, operation icons for the user to select other actions different from the images SB1 to SB5 may be displayed instead of at least one of the images SB1 to SB5.

In this example, the image TG includes image regions TG1 and TG2.

The image region TG1 represents a convex portion on the ground surface located around (in front of) the shovel 100 .

The image region TG2 represents a concave portion on the ground surface located around (in front of) the shovel 100 .

Furthermore, in this example, the screen 700 includes images P1 and P2 corresponding to target points.

The image P1 is displayed superimposed on the image region TG1.

The image P2 is displayed superimposed on the image region TG2.

For example, the user can specify the image areas TG1 and TG2 through the input device 52 or the remote operation support device 300 (input device), and thus set the target points corresponding to the images P1 and P2 through the condition setting unit 302G. The user can set the target point within the entire range of the image TG through the input device 52 or the remote operation support device 300 (input device), or can set the target point within the entire range of the image TG limited to the range that the working part of the bucket 6 can reach. In the former case, if the target point is set within the range that the working part of the bucket 6 can reach within the entire range of the image TG, the display content indicating an error (warning) can also be displayed on the screen 700. In the latter case, an image indicating the range that the working part of the bucket 6 can reach can also be displayed superimposed on the image TG. In addition, the user can also delete the set target point through the input device 52 or the remote operation support device 300 (input device).

Furthermore, in this example, images RC1 and RC2 are displayed on screen 700 so as to be accompanied by images P1 and P2, respectively.

Image RC1 is an image showing a prerequisite condition of the posture angle of bucket 6 corresponding to the target point corresponding to image P1.

Image RC2 is an image showing a prerequisite condition of the posture angle of bucket 6 corresponding to the target point corresponding to image P2.

For example, the user can designate the images P1 and P2 via the input device 52 or the remote operation support device 300 (input device), and the condition setting unit 302G can set the preconditions of the posture angle of the bucket 6 corresponding to the images RC1 and RC2.

The image PB1 is an operation icon for causing the trajectory generation unit 302H to generate a trajectory of the working area of the bucket 6 according to the action selected on the screen 800 and the preconditions set on the screen 800 .

For example, the user can operate the image PB1 via the input device 52 or the remote operation support device 300 (input device), and the target trajectory of the bucket 6 can be generated by the trajectory generation unit 302H.

When the image PB1 is operated, the display content of the display device 50A transitions from the screen 700 to the screen 800 .

Screen 800 includes images TG, CG, and SB similarly to screen 700. Also, screen 800 includes images P1 and P2 similarly to screen 700. Also, screen 800 includes images OG, CG1, and PB2.

The image OG is an image showing the target track.

The image CG1 is an image showing the bucket 6 and is displayed with an image OG corresponding to a target trajectory.

In this example, the image OG represents a target track for realizing an action of digging out soil corresponding to the target point of the image P1 and discharging it to the target point of the image P2. The image OG may also be expressed so as to distinguish between the track portion where the working part of the bucket 6 contacts the soil and the track portion other than the soil. For example, the image OG displays the track portion where the working part of the bucket 6 contacts the soil and the track portion other than the soil in different colors.

The image PB2 is an operation icon for reproducing, on the screen 800 , an action of moving the working area of the bucket 6 along a target trajectory corresponding to the image OG in a dynamic image (animation).

For example, the user can operate the image PB2 via the input device 52 or the remote operation support device 300 (input device) to display a dynamic image of the image CG1 corresponding to the bucket 6 moving along the image OG corresponding to the target trajectory on the screen 800. Therefore, the user can determine whether the target trajectory is correct by checking the dynamic image.

Furthermore, the shape of the work object (terrain shape) after the shovel 100 moves the bucket 6 along the target track may be displayed in the dynamic image. That is, the shape of the work object (terrain shape) around the shovel 100 predicted after the shovel 100 moves the bucket 6 along the target track corresponding to the image OG may be displayed in the screen 800. Thus, the user can more accurately determine whether the target track is correct by checking the dynamic image and the changes in the predicted terrain shape.

When the image PB1 is operated, the display content of the display device 50A transitions from the screen 800 to the screen 900 .

Screen 900 includes images TG, CG, and SB similarly to screen 800. Also, screen 900 includes images P1 and P2 similarly to screen 800. Also, screen 900 includes images OG and CG1. Also, screen 900 includes image PB3.

The image PB3 is an operation icon for automatically operating the shovel 100 so that the working part of the bucket 6 moves along the target trajectory corresponding to the image OG.

For example, the user can operate the image PB3 via the input device 52 or the remote operation support device 300 (input device) to automatically operate the shovel 100 via the operation control unit 302E so that the bucket 6 moves along a target trajectory corresponding to the image OG.

In addition, the shovel 100 can be automatically operated so that the bucket 6 moves along the target trajectory corresponding to the image OG without the user confirming the above-mentioned dynamic image. In this case, on the screen 800, an operation icon corresponding to the image PB3 is displayed instead of the image PB2.

17 , the motion control unit 302E operates the shovel 100 so that the working part of the bucket 6 moves along the target track generated by the track generation unit 302H according to the input from the user (operator) received through the input device 52. Specifically, the motion control unit 302E can control the hydraulic control valve 31 while grasping the position of the working part of the bucket 6 from the output of the sensors S1 to S5, so that the shovel 100 operates so that the working part of the bucket 6 moves along the target track.

For example, the motion control unit 302E operates the shovel 100 so that the working part of the bucket 6 moves along the target trajectory generated by the trajectory generation unit 302H, based on the input of the motion execution instruction from the user.

Furthermore, the operation control unit 302E may operate the shovel 100 to assist the operator's operation based on the operation of the operating device 26 or a remote operation signal so that the working part of the bucket 6 moves along the target trajectory generated by the trajectory generation unit 302H.

＜Processing＞

Next, with reference to FIG. 20 , a description will be given of a process related to generation of a target trajectory of a working area of the shovel 100 .

FIG. 20 is a flowchart schematically showing an example of processing related to generation of a target trajectory of a working area of the shovel 100 .

20 is repeatedly executed, for example, during activation of a function related to generation of a target trajectory of a working portion of the shovel 100. The function related to generation of a target trajectory of a working portion of the shovel 100 is activated (started) by input of an instruction from a user received by the input device 52 or the remote operation support device 300 (input device).

As shown in FIG. 20 , in step S302 (an example of an acquisition step), the work object shape acquisition unit 302B acquires data on the shape of the work object around the shovel 100 from the imaging device 40 .

If the process of step S302 is completed, the controller 30 proceeds to step S304.

In step S304 (an example of a display step), the display processing unit 302I displays a setting screen (eg, screen 700 ) including an image representing the terrain shape on the display device 50A or the remote operation support device 300 (display device) based on the data acquired in step S302 .

If the process of step S304 is completed, the controller 30 proceeds to step S306 .

In step S306 , the action selection unit 302F selects one action from a plurality of candidate actions of the shovel 100 based on the input from the user.

If the process of step S306 is completed, the controller 30 proceeds to step S308 .

In step S308 , the condition setting unit 302G (an example of a setting step) sets a precondition related to the generation of a target trajectory of the working site of the shovel 100 based on an input from the user.

If the process of step S308 is completed, the controller 30 proceeds to step S310 .

In addition, the order of steps S306 and S308 may be changed according to the input from the user.

In step S310 (an example of a generation step), the trajectory generation unit 302H generates a target trajectory of the working area of the shovel 100 under the premise set in step S308 with respect to the operation selected in step S306 .

If the process of step S310 is completed, the controller 30 proceeds to step S312 .

In step S312 , the display processing unit 302I displays the image showing the target trajectory generated in step S310 on the display device 50A or the remote operation support device 300 (display device).

If the process of step S312 is completed, the controller 30 proceeds to step S314.

In step S314, the controller 30 determines whether an operation input instructing execution of the action of the shovel 100 for moving the working part of the bucket 6 along the target trajectory generated in step S312 is received. When an operation input instructing execution of the action of the shovel 100 is received, the controller 30 proceeds to step S316, and when other operations are received, specifically, when an operation for regenerating the target trajectory is received, the controller 30 returns to step S306.

In step S316 , the operation control unit 302E controls the hydraulic control valve 31 to automatically operate the shovel 100 so that the working part of the bucket 6 moves along the target trajectory generated in the most recent process of step S310 .

When the process of step S316 is completed, the controller 30 ends the process of this flowchart.

When the process of step S316 is completed, the shovel 100 (attachment AT) may be in a state where the working part of the bucket 6 is located at the end of the target track. The shovel 100 (attachment AT) may return to the posture state before the process of step S314 is started.

As described above, in this example, the support device 150 (controller 30) can generate a target trajectory that matches the shape of the work object. Therefore, the working efficiency of the shovel 100 can be improved.

Furthermore, in this example, the support device 150 can generate a target trajectory that matches the prerequisites such as the target point and the posture angle of the bucket 6. Therefore, a more appropriate target trajectory can be generated by reflecting the judgment or intention of the user based on the shape of the work object.

Furthermore, in this example, the support device 150 can automatically operate the shovel 100 so that the working part of the bucket 6 moves along the generated target trajectory. Therefore, even an inexperienced operator can operate the shovel 100 appropriately, and as a result, the working efficiency of the shovel 100 can be further improved.

[Another example of the function related to the generation of the trajectory of the working part]

Next, another example of the function related to the generation of the trajectory of the work site will be described.

The above-mentioned embodiments of the function of generating a trajectory of a work site may be combined in their contents as appropriate, or may be modified or altered.

For example, in the above-mentioned embodiment, the trajectory generation unit 302H may generate the target trajectory without using the learned model LM. For example, a trajectory that serves as a reference for the working part is predefined for each of a plurality of candidate actions, and the trajectory generation unit 302H may generate the target trajectory for the working part by optimizing the trajectory that serves as a reference for the action selected by the action selection unit 302F according to data related to the shape (terrain shape) of the working object around the excavator 100 and preconditions.

Furthermore, in the above-mentioned embodiment or its modified example, data related to the shape of the work object around the shovel 100 may be acquired based on data from a camera or a distance sensor or the like provided outside the shovel 100. For example, the data from the camera or the distance sensor provided at the construction site is received by the shovel 100 via the communication device 60, and the work object shape acquisition unit 302B can acquire data related to the shape of the work object around the shovel 100. Furthermore, for example, data from the camera or the distance sensor carried by a drone flying over the construction site is received by the shovel 100 via the communication device 60, and the work object shape acquisition unit 302B can acquire data related to the shape of the work object around the shovel 100.

Furthermore, the support device 150 according to the above-described embodiment or its modified example may generate a trajectory of a working location of a construction machine other than the shovel 100 .

Furthermore, in the above-described embodiment or its modified example, the support device 150 may have the above-described construction machine movement suggestion function in addition to the function related to the generation of the trajectory of the working location of the construction machine.

Furthermore, in the above-described embodiment or its modified example, part or all of the functions of the support device 150 may be transferred to the remote operation support device 300 .

Furthermore, in the above-described embodiment or its modified example, part or all of the functions of the support device 150 may be transferred to the information processing device 200 .

[Role of support equipment (2)]

Next, the operation of the support device according to this embodiment will be described.

For example, there is known a technique for setting teaching points and generating a trajectory of a working location of a construction machine based on the teaching points (for example, refer to Japanese Patent Application Laid-Open No. 2021-50576).

However, for example, in Patent Document 1, it is necessary to actually operate the shovel to operate the construction machine and set the teaching points. As a result, it may take a lot of time and effort to generate a target trajectory of the work site.

In contrast, in the present embodiment, the support device includes an acquisition unit, a display unit, a setting unit, and a generation unit. The support device is, for example, the support device 150. The acquisition unit is, for example, the work object shape acquisition unit 302B. The display unit is, for example, the display device 50A. The setting unit is, for example, the condition setting unit 302G. The generation unit is, for example, the track generation unit 302H. Specifically, the acquisition unit acquires data related to the terrain shape of the construction object around the construction machine. The construction machine is, for example, the excavator 100. And, the display unit displays an image representing the terrain shape of the construction object based on the data acquired by the acquisition unit. And, the setting unit sets a point (target point) in the terrain shape of the construction object that becomes a target when the construction machine performs an action. And, the generation unit generates a track of the work part of the construction machine based on the data acquired by the acquisition unit, the target shape of the construction object, and the point set by the setting unit.

Thus, it is possible to more easily generate a track of the working part of the construction machine. Furthermore, since the target point is set, it is possible to generate a more appropriate track of the working part that reflects the judgment or intention of the user who visually recognizes the shape of the working object around the construction machine, for example. Therefore, the working efficiency of the construction machine can be improved.

Furthermore, in this embodiment, the support device may also include a selection unit. Specifically, the selection unit may also select an action from a plurality of candidate actions of the construction machine based on an input from a user. Furthermore, the generation unit may also generate a trajectory of a working part based on an action of the construction machine based on the data acquired by the acquisition unit and the points set by the setting unit.

Thus, the assist device can limit the movement of the construction machine and generate a trajectory of the work site. Therefore, the work efficiency of the construction machine can be further improved.

Furthermore, in the present embodiment, the setting unit may set a point (target point) in the terrain shape around the construction machine that becomes a target when the construction machine operates and the posture of the working part corresponding to the point based on input from the user.

As a result, the support device can generate a more appropriate trajectory of the work site that reflects the user's judgment or intention regarding the posture of the work site, thereby further improving the work efficiency of the construction machine.

Furthermore, in the present embodiment, the display unit may display an image showing the track generated by the generation unit in a superimposed manner on the image showing the topographic shape around the construction machine.

Thus, the user can visually check the generated image. Furthermore, the user can more accurately judge the rationality of the generated track by visually recognizing the terrain shape around the construction machine and the generated track at the same time.

Furthermore, in the present embodiment, the display unit may superimpose on the image showing the topographic shape around the construction machine a dynamic image showing the working area moving along the track generated by the generation unit.

This allows the user to more accurately determine the rationality of the generated trajectory by checking the moving image.

Furthermore, in the present embodiment, the display unit may display an image showing the shape of the work object around the construction machine predicted after the work site moves along the trajectory generated by the generation unit.

Thus, by confirming the shape of the work object after the work site moves along the generated trajectory, the rationality of the generated trajectory can be determined more accurately.

Furthermore, in the present embodiment, the support device may include a control unit that automatically operates the construction machine according to the trajectory generated by the generation unit based on the user's input. The control unit is, for example, the operation control unit 302E.

Thus, even a relatively inexperienced operator can move the working part along the target track, thereby improving work efficiency and user convenience.

Furthermore, in the present embodiment, the construction machine may include the above-mentioned operation support device.

Thereby, the construction machine can more easily generate a target trajectory and can improve work efficiency.

As mentioned above, although embodiment is described in detail, this invention is not limited to this specific embodiment, Various deformation|transformation and change are possible within the range of the summary described in a claim.

This application claims priority based on Japanese Patent Application No. 2022-058984 filed on March 31, 2022 and Japanese Patent Application No. 2022-060273 filed on March 31, 2022, the entire contents of which are incorporated herein by reference.

Explanation of symbols

1-lower walking body, 1C, 1CL, 1CR-crawler, 1ML, 1MR-walking hydraulic motor, 2-slewing mechanism, 2M-slewing hydraulic motor, 3-upper slewing body, 4-boom, 5-arm, 6-bucket, 7-boom cylinder, 8-arm cylinder, 9-bucket cylinder, 10-cabin, 11-engine, 13-regulator, 14-main pump, 15-pilot pump, 17-control valve, 26-operating device, 29-operating pressure sensor, 30-controller, 31-hydraulic control valve, 32-reciprocating valve, 33-hydraulic control valve, 40-camera device, 50-output device, 50A-display device, 50B-sound output device, 52-input device, 60-communication device, 100-excavator, 150-support device, 200-information processing device, 300-remote operation support device, 301 -action log providing unit, 301A-action log recording unit, 301B-action log storage unit, 301C-action log sending unit, 302-work support unit, 302A-learned model storage unit, 302B-work object shape acquisition unit, 302C-inference unit, 302D-suggestion unit, 302E-action control unit, 302F-action selection unit, 302G-condition setting unit, 302H-track generation unit, 302I-display processing unit, 2001-action log acquisition unit, 2002-action log storage unit, 2003-training data generation unit, 2004-machine learning unit, 2005-learned model storage unit, 2006-transmission unit, AT-attachment, HA-hydraulic actuator, LM-learned model, S1~S5-sensors, SYS-operation support system.

Claims (as amended under Article 19)

1. (After correction) A support device, characterized in that:

Have:

An acquisition unit, which acquires data related to the shape of the work object around the construction machine; and

The suggestion unit suggests an action to the user from a plurality of candidate actions of different types regarding a prescribed operation of the construction machinery based on the data acquired by the acquisition unit.

2. (After correction) The support device according to claim 1, wherein:

When the prescribed operation is the leveling of the ground at the construction site, the multiple candidate actions include at least two of the sweeping action, the horizontal pulling action, the rolling action and the brooming action.

3. (After correction) The support device according to claim 1 or 2, comprising:

The inference unit uses a machine-learned learning model based on training data related to the actions of the construction machinery based on the operation of a relatively skilled operator, which is associated with the shape of the work object, and infers the action that matches the shape of the work object around the construction machinery from the plurality of candidate actions based on the data acquired by the acquisition unit.

The suggestion unit suggests an action from among the multiple candidate actions based on the inference result of the inference unit.

4. (After correction) A support device according to claim 1 or 2, wherein:

The suggestion unit suggests multiple actions to the user from the multiple candidate actions based on the data acquired by the acquisition unit.

5. (After correction) The support device according to claim 4, wherein:

The multiple actions of the suggested object include actions that have a relatively low degree of matching with the shape of the construction object around the construction machine among the multiple candidate actions.

6. (After correction) A support device according to claim 1 or 2, wherein:

When the suggestion unit suggests one or more actions from the plurality of candidate actions based on the data acquired by the acquisition unit, the suggestion unit also suggests the degree to which the action matches the shape of the work object around the construction machine.

7. (After correction) A support device according to claim 1 or 2, wherein:

When the suggestion unit suggests an action from the plurality of candidate actions based on the data acquired by the acquisition unit, the suggestion unit also suggests one or more tracks of the working part of the construction machine based on the action.

8. (After correction) The support device according to claim 7, wherein:

When suggesting an action from the plurality of candidate actions based on the data acquired by the acquisition unit, the suggestion unit also suggests a plurality of tracks of the working part based on the action and the degree to which each of the plurality of tracks matches the shape of the working object around the construction machine.

9. (After correction) The support device according to claim 7, which has a display unit,

The suggestion unit superimposes the trajectory of the work site based on the action of the suggested object suggested from the plurality of candidate actions on an image representing the state of the surroundings of the construction machine and displays it on the display unit.

10. (After correction) The support device according to claim 1 or 2, comprising:

The control unit, when the action of the construction machine is suggested by the suggestion unit, causes the construction machine to automatically execute the action suggested by the suggestion unit based on the input of instructions from the user.

11. (After correction) The support device according to claim 1 or 2, comprising:

A display unit, which displays an image representing the shape of the work object based on the data acquired by the acquisition unit;

A setting unit, which sets a point in the shape of the work object that becomes a target when the construction machinery performs an action; and

The generation unit generates a trajectory of the working part of the construction machine based on the data acquired by the acquisition unit, the target shape of the working object and the points set by the setting unit.

12. (After correction) A construction machine having:

An acquisition unit, which acquires data related to the shape of the work object around the construction machine; and

The suggestion unit suggests an action to the user from a plurality of candidate actions of different types regarding a prescribed operation of the construction machinery based on the data acquired by the acquisition unit.

13. (Corrected) A program that causes a support device to perform the following steps:

Acquisition step, acquiring data related to the shape of the work object around the construction machinery; and

A suggestion step, based on the data acquired in the acquisition step, suggests an action to the user from a plurality of candidate actions of different types regarding a prescribed operation of the construction machinery.

14.(Deleted)

15.(Deleted)

16.(Deleted)

Claims (16)

1. A support device is provided with:

An acquisition unit that acquires data relating to the shape of an object to be worked around the construction machine; and

And a suggestion unit that suggests an action among a plurality of candidate actions of the construction machine in a predetermined operation to a user based on the data acquired by the acquisition unit.

2. The support apparatus according to claim 1, comprising:

An estimating unit that uses a learning model in which machine learning is performed based on training data related to the operation of the construction machine based on the operation of the operator with relatively high proficiency, which is associated with the shape of the work object, and that calculates an operation matching the shape of the work object around the construction machine from the plurality of candidate operations based on the data acquired by the acquiring unit,

The suggesting unit suggests an action from among the plurality of candidate actions based on the estimation result of the estimating unit.

3. The support device according to claim 1 or 2, wherein,

The suggesting section suggests a plurality of actions among the plurality of candidate actions to a user based on the data acquired by the acquiring section.

4. The support apparatus according to claim 3, wherein,

The plurality of actions of the recommended object include actions of the plurality of candidate actions that have a relatively low degree of matching with respect to the shape of the work object around the construction machine.

5. The support device according to any one of claims 1 to 4, wherein,

The suggesting unit suggests, based on the data acquired by the acquiring unit, a degree of matching between an action among the plurality of candidate actions and a shape of the action with respect to a work object around the construction machine.

6. The support apparatus according to claim 5, wherein,

The suggestion unit suggests, based on the data acquired by the acquisition unit, a degree of matching between a plurality of actions among the plurality of candidate actions and a shape of each of the plurality of actions with respect to a work object around the construction machine.

7. The support device according to any one of claims 1 to 6, wherein,

The suggestion unit suggests, based on the data acquired by the acquisition unit, an operation of the plurality of candidate operations and a trajectory of a work site of the construction machine based on the operation together.

8. The support apparatus according to claim 7, wherein,

The suggesting unit suggests, based on the data acquired by the acquiring unit, an action among the plurality of candidate actions and a plurality of trajectories of the work site based on the action together.

9. The support apparatus according to claim 8, wherein,

The suggestion unit suggests, based on the data acquired by the acquisition unit, an action among the plurality of candidate actions, a plurality of trajectories of the work site based on the action, and a degree of matching of each trajectory of the plurality of trajectories with respect to a shape of a work object around the construction machine.

10. The support device according to any one of claims 7 to 9, comprising a display unit,

The advice unit superimposes an orbit of the work site based on the action of the advice target among the plurality of candidate actions on an image showing a state around the construction machine, and displays the superimposed orbit on the display unit.

11. The support apparatus according to claim 10, wherein,

The advice unit displays, on the display unit, a track portion in contact with the work object and other track portions in a track based on the action of the advice object among the plurality of candidate actions in a different manner.

12. The support apparatus according to any one of claims 1 to 11, comprising:

And a control unit that causes the construction machine to automatically execute the action recommended by the recommendation unit, in response to an input of an instruction from a user.

13. The support apparatus according to claim 12, wherein,

The acquisition unit predicts the shape of the work object around the construction machine after the operation automatically performed by the control unit, and acquires data on the shape of the work object around the construction machine.

14. The support apparatus according to any one of claims 1 to 13, comprising:

A display unit configured to display an image indicating a shape of the work object based on the data acquired by the acquisition unit;

A setting unit that sets a point that is a target when the construction machine is operating in the shape of the work object; and

And a generation unit configured to generate a track of a work site of the construction machine based on the data acquired by the acquisition unit, the target shape of the work object, and the point set by the setting unit.

15. A construction machine is provided with:

An acquisition unit that acquires data relating to the shape of an object to be worked around the construction machine; and

And a suggestion unit that suggests an action among a plurality of candidate actions of the construction machine in a predetermined operation to a user based on the data acquired by the acquisition unit.

16. A program for causing a support apparatus to execute the steps of:

an acquisition step of acquiring data related to a shape of an object around the construction machine; and

And a suggestion step of suggesting, to a user, an action among a plurality of candidate actions of the construction machine in the prescribed work, based on the data acquired in the acquisition step.