CN118525121A - Operation support device, construction machine, remote operation support device, and program - Google Patents

Abstract

The present invention provides a technology that enables a construction machine to operate more accurately by receiving instructions from outside the construction machine. An operation support device (150) according to one embodiment of the present invention comprises: an operation instruction receiving unit (301) that recognizes a gesture indicating an instruction related to the operation of the excavator (100) from an indicator around the excavator (100) based on sensor information indicating the state around the excavator (100) and receives instructions related to the operation of the excavator (100); and an operation instruction notification unit (302) that notifies a user of the instruction related to the operation of the excavator (100) received by the operation instruction receiving unit (301).

Description

Operation support device, construction machine, remote operation support device and program

Technical Field

The present invention relates to an operation support device for a construction machine, etc.

Background Art

There is known a technique for operating a construction machine based on an instruction from outside the construction machine (for example, refer to Patent Document 1).

Patent Document 1 discloses that a worker's action (gesture) is recognized based on time-series image information (dynamic image information) around a construction machine (shovel), and the construction machine is operated based on the action.

Previous technical literature

Patent Literature

Patent Document 1: Japanese International Publication No. 2020/032267

Summary of the invention

Technical issues to be solved by the invention

However, for example, when an instruction is issued from outside the construction machine and the operation is performed directly, there is a blind spot such as the position of the rear of the construction machine around the construction machine depending on the position of the person issuing the instruction, and as a result, it may not be possible to perform appropriate operations corresponding to the conditions around the construction machine. Also, for example, when an instruction is issued from outside the construction machine through human actions such as gestures and the operation is performed directly, it may not be possible to accurately perform the operation on the construction machine depending on the recognition accuracy.

Therefore, an object of the present invention is to provide a technology capable of causing a construction machine to operate more accurately by an instruction from outside the construction machine.

Means for solving technical problems

In order to achieve the above object, in one embodiment of the present invention, there is provided an operation support device, which comprises:

a receiving unit that receives instructions related to the operation of the construction machine from outside the construction machine; and

The notifying unit notifies a user of the instruction related to the operation of the construction machine received by the receiving unit.

Furthermore, in another embodiment of the present invention, a construction machine including the above-mentioned operation support device is provided.

Furthermore, in another embodiment of the present invention, there is provided a remote operation support device, which includes the above-mentioned operation support device,

The operation unit receives an operation input for remote operation of the construction machine.

Furthermore, in another embodiment of the present invention, a program is provided, which causes an information processing device to execute the following steps:

an acquisition step of acquiring information of instructions related to the operation of the construction machine from outside the construction machine; and

A notification step of notifying a user of instructions related to the operation of the construction machine based on the information acquired in the acquisition step.

Effects of the Invention

According to the above-described embodiment, the construction machine can be operated more accurately by an instruction from outside the construction machine.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an example of an operation support system.

FIG. 2 is a plan view showing an example of a shovel.

FIG. 3 is a diagram showing an example of a configuration related to remote control of a shovel.

FIG. 4 is a block diagram showing an example of the hardware configuration of the shovel.

FIG. 5 is a diagram showing an example of the hardware configuration of the information processing device.

FIG. 6 is a functional block diagram showing a first example of the functional configuration of the operation support device.

FIG. 7 is a diagram showing an example of an operation instruction notification screen displayed on the display device.

FIG. 8 is a functional block diagram showing a second example of the functional configuration of the operation support device.

FIG. 9 is a diagram showing another example of a construction machine.

FIG. 10 is a diagram showing an example of a remote operation system for a shovel.

FIG. 11 is a block diagram showing an example of the hardware configuration of the remote operation support device.

FIG. 12 is a functional block diagram showing an example of a functional configuration related to a periphery monitoring function of a shovel.

FIG. 13 is a flowchart schematically showing an example of an authentication process of a person (remote operator) around the shovel.

FIG. 14 is a diagram showing an example of a safety system for a shovel.

FIG. 15 is a block diagram showing another example of the hardware configuration of the shovel.

FIG. 16 is a functional block diagram showing an example of a configuration related to a safety function of a shovel.

DETAILED DESCRIPTION

Hereinafter, embodiments will be described with reference to the drawings.

[Overview of Operation Support System]

First, with reference to FIGS. 1 to 3 , an overview of an operation support system SYS according to the present embodiment will be described.

FIG. 1 is a diagram showing an example of an operation support system SYS. FIG. 1 shows a left side view of an excavator 100. FIG. 2 is a top view showing an example of an excavator 100. FIG. 3 is a diagram showing an example of a structure related to remote operation of an excavator. In the following, the direction in which the attachment AT extends when the excavator 100 is viewed from above (the upper direction in FIG. 2 ) is sometimes defined as “front” to describe the direction in the excavator 100 or the direction viewed from the excavator 100.

As shown in FIG. 1 , the operation support system SYS includes a shovel 100 and an information processing device 200 .

The operation support system SYS cooperates with the shovel 100 using the information processing device 200 , and performs support related to the operation of the shovel 100 .

The shovel 100 included in the operation support system SYS may be one or a plurality of shovels.

In the operation support system SYS, the shovel 100 is a construction machine that is a target of support related to operation.

As shown in FIGS. 1 and 2 , the shovel 100 includes a lower traveling body 1 , an upper swing body 3 , an attachment AT including a boom 4 , an arm 5 , and a bucket 6 , and a cab 10 .

The lower traveling body 1 uses crawler tracks 1C to allow the excavator 100 to travel. The crawler tracks 1C include a left crawler track 1CL and a right crawler track 1CR. The crawler tracks 1CL are hydraulically driven by a traveling hydraulic motor 1ML. Similarly, the crawler tracks 1CL are hydraulically driven by a traveling hydraulic motor 1MR. Thus, the lower traveling body 1 can move on its own.

The upper revolving body 3 is mounted on the lower traveling body 1 so as to be revolvable via the revolving mechanism 2. For example, the upper revolving body 3 is revolved relative to the lower traveling body 1 by hydraulically driving the revolving mechanism 2 with a revolving hydraulic motor 2M.

The boom 4 is mounted at the front center of the upper revolving body 3 so as to be able to pitch about a rotation axis along the left-right direction. The arm 5 is mounted at the front end of the boom 4 so as to be able to rotate about a rotation axis along the left-right direction. The bucket 6 is mounted at the front end of the arm 5 so as to be able to rotate about a rotation axis along the left-right direction.

The bucket 6 is an example of an end attachment and is used for excavation work, for example.

Furthermore, a lifting hook HK is mounted on the bucket 6. Thus, the excavator 100 can suspend the load on the hook HK to operate at least one of the lower traveling body 1, the upper swing body 3, and the attachment AT, thereby performing a lifting operation (lifting operation) of transporting the load to a predetermined transport destination.

The base end of the hook HK is rotatably connected to a bucket pin connecting the arm 5 and the bucket 6. Thus, when performing work other than lifting work such as excavation, the hook HK can be accommodated in the space formed between the two bucket links.

The bucket 6 is mounted on the front end of the arm 5 in a manner that allows for appropriate replacement according to the work content of the excavator 100. That is, a bucket of a different type from the bucket 6 may be mounted on the front end of the arm 5 in place of the bucket 6, for example, a relatively large bucket, a bucket for slopes, a bucket for dredging, etc. Also, a terminal attachment other than a bucket may be mounted on the front end of the arm 5, for example, a mixer, a crusher, a crusher, etc. Also, a preliminary attachment such as a quick coupling, a tilt rotator, etc. may be provided between the arm 5 and the terminal attachment.

The boom 4, the arm 5 and the bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8 and a bucket cylinder 9, respectively.

The cab 10 is a control room for an operator to get on and operate the shovel 100. The cab 10 is mounted on the front left side of the upper swing body 3, for example.

For example, the shovel 100 operates driven elements such as the lower traveling body 1 (ie, a pair of left and right crawler tracks 1CL and 1CR), the upper swing body 3, the boom 4, the arm 5, and the bucket 6 according to operations of an operator riding in the cab 10.

Furthermore, the shovel 100 is configured to be operated by an operator riding in the cab 10, but may also be configured to be remotely operated from the outside of the shovel 100 instead of or in addition thereto. When the shovel 100 is remotely operated, the interior of the cab 10 may be unmanned. The following description is based on the premise that the operator's operation includes at least one of the operation of the operating device 26 by the operator in the cab 10 and the remote operation by an external operator.

For example, as shown in FIG. 3 , the remote operation includes a method of operating the shovel 100 by inputting operation related to the actuator of the shovel 100 in the remote operation support device 300 .

The remote operation support device 300 is, for example, installed in a management center that externally manages the operation of the shovel 100. Furthermore, the remote operation support device 300 may be a portable operation terminal, in which case the operator can remotely operate the shovel 100 while directly checking the operation status of the shovel 100 from the periphery of the shovel 100.

The shovel 100 can, for example, use the communication device 60 described later and transmit an image (hereinafter referred to as a "peripheral image") representing the state of the periphery of the shovel 100 including the front, based on the camera image output by the camera device 40 described later, to the remote operation support device 300 through the communication line NW. In addition, the remote operation support device 300 can display the image (peripheral image) received from the shovel 100 on the display device. In addition, similarly, various information images (information screens) displayed on the output device 50 (display device) inside the cab 10 of the shovel 100 can also be displayed on the display device of the remote operation support device 300. As a result, the operator using the remote operation support device 300 can remotely operate the shovel 100 while checking the display content such as the image or information screen representing the state of the periphery of the shovel 100 displayed on the display device. Furthermore, the excavator 100 can operate the actuator according to the remote operation signal indicating the content of the remote operation received by the communication device 60 from the remote operation support device 300, thereby driving the driven elements such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5 and the bucket 6.

Furthermore, in the remote operation, for example, a method of operating the shovel 100 by a person (e.g., a worker) around the shovel 100 inputting voice or gestures from outside to the shovel 100 may be included. Specifically, the shovel 100 recognizes voices made by workers around the shovel 100 or gestures made by workers through a voice input device (e.g., a microphone) or a gesture input device (e.g., a camera) mounted on the shovel 100. Furthermore, the shovel 100 may also operate the actuator according to the content of the recognized voice or gesture, thereby driving the driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper swing body 3, the boom 4, the arm 5, and the bucket 6.

Furthermore, the operation of the excavator 100 can also be monitored remotely. In this case, a remote monitoring support device having the same function as the remote operation support device 300 can also be provided. The remote monitoring support device is, for example, the information processing device 200. Thus, a monitor who is a user of the remote monitoring support device can monitor the operation status of the excavator 100 while confirming the peripheral image displayed on the display device of the remote monitoring support device. Furthermore, for example, when it is determined that monitoring is necessary from the perspective of safety, the monitor can intervene in the operation performed by the operator of the excavator 100 and make it stop urgently by making a predetermined input using the input device of the remote monitoring support device.

The information processing device 200 communicates with the shovel 100 via the communication line NW to cooperate with each other and provides support related to the operation of the shovel 100 .

The information processing device 200 is, for example, a server or a management terminal device in a management office set up in the construction site of the excavator 100 or a management center located at a location different from the construction site of the excavator 100 for managing the operating status of the excavator 100. The management terminal device can be, for example, a fixed terminal device such as a desktop PC (Personal Computer), or a portable terminal device (mobile terminal) such as a tablet terminal, a smart phone, or a laptop PC. In the latter case, workers at the construction site, supervisors who supervise the work, managers who manage the construction site, etc. can carry the portable information processing device 200 and move around the construction site. And, in the latter case, the operator can, for example, bring the portable information processing device 200 into the cockpit of the excavator 100.

The information processing device 200 acquires data related to the operating state from the shovel 100, for example. Thus, the information processing device 200 can grasp the operating state of the shovel 100 and monitor whether the shovel 100 has abnormalities, etc. Furthermore, the information processing device 200 can display the data related to the operating state of the shovel 100 on the display device 208 described later and allow the user to confirm.

The information processing device 200 transmits various data such as programs and reference data used in the processing of the controller 30 of the shovel 100 to the shovel 100. The shovel 100 can thereby perform various processes related to the operation of the shovel 100 using various data downloaded from the information processing device 200.

[Hardware structure of the operation support system]

Next, the hardware configuration of the operation support system SYS will be described with reference to FIG. 4 and FIG. 5 in addition to FIG. 1 to FIG. 3 .

＜Hardware structure of excavator＞

FIG. 4 is a block diagram showing an example of the hardware configuration of the shovel 100 .

4 , a double line represents a path for transmitting mechanical power, a solid line represents a path for high-pressure hydraulic oil to drive a hydraulic actuator, a dotted line represents a path for transmitting pilot pressure, and a dotted line represents a path for transmitting electrical signals.

The excavator 100 includes various components such as a hydraulic drive system related to hydraulic drive of driven elements, an operating system related to operation of driven elements, a user interface system related to information exchange with users, a communication system related to communication with the outside, and a control system related to various controls.

Hydraulic drive system

As shown in Fig. 4, the hydraulic drive system of the excavator 100 includes a hydraulic actuator HA that hydraulically drives the driven elements such as the lower traveling body 1 (left and right crawler tracks 1C), the upper swing body 3 and the attachment AT, etc., as described above. In addition, the hydraulic drive system of the excavator 100 involved in this embodiment includes an engine 11, a regulator 13, a main pump 14 and a control valve 17.

The hydraulic actuator HA includes travel hydraulic motors 1ML and 1MR, a swing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.

In addition, in the shovel 100, a part or all of the hydraulic actuator HA may be replaced with an electric actuator. That is, the shovel 100 may be a hybrid shovel or an electric shovel.

The engine 11 is a prime mover of the excavator 100 and a main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses diesel as fuel. The engine 11 is, for example, mounted on the rear of the upper slewing body 3. The engine 11 rotates constantly at a preset target speed under direct or indirect control by a controller 30 described later, and drives the main pump 14 and the pilot pump 15.

In addition, other prime movers (for example, electric motors) or the like may be mounted on the shovel 100 instead of the engine 11 or in addition thereto.

The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as "tilt angle") according to a control command from the controller 30.

The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, at the rear of the upper slewing body 3, similarly to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, the stroke length of the piston is adjusted by adjusting the deflection angle of the swash plate by the regulator 13 under the control of the controller 30 to control the discharge flow rate or discharge pressure.

The control valve 17 drives the hydraulic actuator HA according to the operator's operation of the operating device 26 or the content of the remote operation. The control valve 17 is mounted, for example, on the central part of the upper swing body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic pipeline, and the working oil supplied from the main pump 14 is selectively supplied to each hydraulic actuator according to the operator's operation. Specifically, the control valve 17 includes a plurality of control valves (also called "direction switching valves"), which control the flow rate and flow direction of the working oil supplied from the main pump 14 to each hydraulic actuator HA.

"operating system"

As shown in FIG. 4 , the operating system of the shovel 100 includes a pilot pump 15 , an operating device 26 , a hydraulic control valve 31 , a shuttle valve 32 , and a hydraulic control valve 33 .

The pilot pump 15 supplies pilot pressure to various hydraulic devices via the pilot line 25. Like the engine 11, the pilot pump 15 is mounted, for example, at the rear of the upper swing body 3. The pilot pump 15 is, for example, a fixed displacement hydraulic pump and is driven by the engine 11 as described above.

In addition, the pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic oil discharged from the main pump 14 can be supplied to various hydraulic devices as pilot pressure after being reduced in pressure by a predetermined pressure reducing valve to a relatively low-pressure hydraulic oil.

The operating device 26 is provided near the driver's seat of the cab 10 and is used by the operator to operate various driven elements. Specifically, the operating device 26 is used by the operator to operate the hydraulic actuator HA that drives each driven element, and as a result, the operator can operate the driven element that is the driving target of the hydraulic actuator HA. The operating device 26 includes a pedal device or a joystick device for operating each driven element (hydraulic actuator HA).

For example, as shown in FIG. 4 , the operating device 26 is a hydraulic pilot type. Specifically, the operating device 26 uses the working oil supplied from the pilot pump 15 through the pilot line 25 and the pilot line 25A branched from it to output the pilot pressure corresponding to the operation content to the pilot line 27A on the secondary side. The pilot line 27A is connected to an inlet port of the reciprocating valve 32, and is connected to the control valve 17 via the pilot line 27 connected to the outlet port of the reciprocating valve 32. As a result, the pilot pressure corresponding to the operation content related to various driven elements (hydraulic actuator HA) in the operating device 26 can be input to the control valve 17 via the reciprocating valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26 by the operator or the like.

Furthermore, the operating device 26 may be an electrical type. In this case, the pilot line 27A, the reciprocating valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electrical signal (hereinafter referred to as an "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. Then, the controller 30 outputs a control instruction corresponding to the content of the operation signal to the hydraulic control valve 31, that is, a control signal corresponding to the operation content of the operating device 26. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

Furthermore, the control valve (directional switching valve) for driving each hydraulic actuator HA built into the control valve 17 may be of electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 may be directly input to the control valve 17, that is, the electromagnetic solenoid type control valve.

Furthermore, as described above, a part or all of the hydraulic actuator HA may be replaced with an electric actuator. In this case, the controller 30 may output a control instruction corresponding to the operation content of the operating device 26 or the content of the remote operation defined by the remote operation signal to the electric actuator or a driver driving the electric actuator, etc. Furthermore, when the shovel 100 is remotely operated, the operating device 26 may be omitted.

The hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) of the operating object of the operating device 26 and for each driving direction of the driven element (hydraulic actuator HA) (for example, the lifting direction and the lowering direction of the boom 4). That is, two hydraulic control valves 31 are provided for each passive hydraulic actuator HA. The hydraulic control valve 31 can be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and is configured to be able to change its flow area (that is, the cross-sectional area through which the working oil can flow). Thus, the hydraulic control valve 31 can output a prescribed pilot pressure to the pilot line 27B on the secondary side using the working oil of the pilot pump 15 supplied through the pilot line 25B. Therefore, the hydraulic control valve 31 can indirectly act on the control valve 17 with a prescribed pilot pressure corresponding to a control signal from the controller 30 through the reciprocating valve 32 between the pilot line 27B and the pilot line 27.

The controller 30 controls the hydraulic control valve 31, for example, to realize the remote operation of the shovel 100. Specifically, the controller 30 outputs a control signal corresponding to the content of the remote operation specified by the remote operation signal received from the remote operation support device 300 to the hydraulic control valve 31 through the communication device 60. As a result, the controller 30 supplies the pilot pressure corresponding to the remote operation content from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the shovel 100 based on the remote operation of the operator.

When the operating device 26 is an electric type, the controller 30 supplies a pilot pressure corresponding to the operation content (operation signal) of the operating device 26 directly from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the shovel 100 based on the operator's operation.

The shuttle valve 32 has two inlet ports and one outlet port, and outputs the working oil having the higher pilot pressure of the pilot pressure input to the two inlet ports to the outlet port. The shuttle valve 32 is provided for each driven element (hydraulic actuator HA) of the operating object of the operating device 26 and for each driving direction of the driven element (hydraulic actuator HA). One of the two inlet ports of the shuttle valve 32 is connected to the pilot line 27A on the secondary side of the operating device 26 (specifically, the above-mentioned joystick device or pedal device included in the operating device 26), and the other is connected to the pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 through the pilot line 27. The corresponding control valve is a control valve that drives the hydraulic actuator that is the operating object of the above-mentioned joystick device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, these shuttle valves 32 can respectively allow the higher pilot pressure of the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure of the pilot line 27B on the secondary side of the hydraulic control valve 31 to act on the pilot port of the corresponding control valve. That is, the controller 30 can control the corresponding control valve independently of the operator's operation of the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operation of the driven elements (lower traveling body 1, upper swing body 3, attachment AT) independently of the operator's operation state of the operating device 26, thereby realizing the remote operation function.

The hydraulic control valve 33 is provided in the pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured to be able to change its flow path area, for example. The hydraulic control valve 33 operates according to the control signal input from the controller 30. Thus, when the operating device 26 is operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, even when the operating device 26 is operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. And, for example, even when the operating device 26 is operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to be lower than the pilot pressure output from the hydraulic control valve 31. Therefore, the controller 30 can reliably act on the pilot port of the control valve in the control valve 17 by controlling the hydraulic control valve 31 and the hydraulic control valve 33, for example, regardless of the operation content of the operating device 26. Therefore, the controller 30 can more accurately realize the remote control function of the shovel 100 or the like by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31 , for example.

User Interface System

As shown in FIG. 4 , the user interface system of the excavator 100 includes an operating device 26 , an output device 50 , and an input device 52 .

The output device 50 outputs various information to a user of the shovel 100 (for example, an operator of the cab 10 or an external remote operator) or a person around the shovel 100 (for example, a worker or a driver of a construction vehicle).

For example, the output device 50 includes a lighting device or a display device 50A (refer to FIG6 ) that outputs various information by a visual method. The lighting device is, for example, a warning light (indicator light). The display device 50A is, for example, a liquid crystal display or an organic EL (Electroluminescence) display. For example, as shown in FIG2 , the lighting device or the display device 50A can be disposed inside the cockpit 10, and various information can be output to an operator inside the cockpit 10 by a visual method. Furthermore, the lighting device or the display device 50A can also be disposed on the side of the upper rotating body 3, and various information can be output to workers around the excavator 100 by a visual method.

Furthermore, for example, the output device 50 includes a sound output device 50B (see FIG. 6 ) that outputs various information in an auditory manner. The sound output device 50B includes, for example, a buzzer or a speaker. The sound output device 50B may be provided, for example, at least one of the inside and outside of the cab 10, and outputs various information in an auditory manner to an operator in the cab 10 or to people (staff, etc.) around the shovel 100.

Furthermore, for example, the output device 50 may also include a device that outputs various information by a tactile method such as vibration of the driver's seat.

The input device 52 receives various inputs from the user of the shovel 100, and inputs signals corresponding to the received inputs to the controller 30. The input device 52 is provided, for example, inside the cab 10, and receives inputs from the operator or the like inside the cab 10. Furthermore, the input device 52 may be provided, for example, on the side of the upper swing body 3, and receive inputs from the staff or the like around the shovel 100.

For example, the input device 52 includes an operation input device that receives input from the user based on mechanical operation. The operation input device may include a touch panel installed on the display device, a touch pad provided around the display device, a button switch, a joystick, a switch key, a knob switch provided on the operation device 26 (joystick device), etc.

Furthermore, for example, the input device 52 may include a voice input device for receiving a user's voice input. The voice input device may include, for example, a microphone.

Furthermore, for example, the input device 52 may include a gesture input device that receives a gesture input by the user. The gesture input device includes, for example, an imaging device that captures a state of a gesture performed by the user.

Furthermore, for example, the input device 52 may include a biometric input device for receiving a user's biometric input, which may include, for example, input of biometric information such as a user's fingerprint or iris.

Communication Systems

As shown in FIG. 4 , the communication system of the shovel 100 according to the present embodiment includes a communication device 60 .

The communication device 60 is connected to an external communication line NW, and communicates with a device that is separately installed from the excavator 100. In addition to the device located outside the excavator 100, the device that is separately installed from the excavator 100 may also include a portable terminal device (mobile terminal) brought into the cockpit 10 by the user of the excavator 100. The communication device 60 may, for example, include a mobile communication module that complies with specifications such as 4G ( ^4th Generation) or 5G ( ^5th Generation). In addition, the communication device 60 may, for example, include a satellite communication module. In addition, the communication device 60 may also include, for example, a WiFi communication module or a Bluetooth (registered trademark) communication module. In addition, the communication device 60 may also include a plurality of communication devices corresponding to the communication line NW of the connection object.

For example, the communication device 60 communicates with an external device such as the information processing device 200 or the remote operation support device 300 in the construction site through a local communication line NW built at the construction site. The local communication line NW is, for example, a mobile communication line based on a local 5G (so-called local 5G) built at the construction site or a local network (LAN: Local Area Network) based on WiFi6.

Furthermore, for example, the communication device 60 communicates with the information processing device 200 or the remote operation support device 300 located outside the construction site via a wide area communication line NW including the construction site, i.e., a wide area network (WAN). The wide area network includes, for example, a wide area mobile communication network, a satellite communication network, the Internet, and the like.

Control Systems

4 , the control system of the shovel 100 includes a controller 30 . Furthermore, the control system of the shovel 100 according to the present embodiment includes an operation pressure sensor 29 and an imaging device 40 .

The controller 30 performs various controls related to the shovel 100 .

The functions of the controller 30 can be realized by any hardware or any combination of hardware and software. For example, as shown in FIG4 , the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C and an interface device 30D connected by a bus B1.

The auxiliary storage device 30A is a nonvolatile storage means, and stores programs to be installed, as well as necessary files and data, etc. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory.

For example, when there is an instruction to start a program, the memory device 30B loads the program in the auxiliary storage device 30A so that the CPU 30C can read it. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

For example, the CPU 30C executes a program loaded into the memory device 30B, and realizes various functions of the controller 30 according to the commands of the program.

The interface device 30D functions as, for example, a communication interface for connecting to a communication line inside the shovel 100. The interface device 30D may include a plurality of different types of communication interfaces depending on the type of communication line to be connected.

Furthermore, the interface device 30D functions as an external interface for reading data from a recording medium or writing data to a recording medium. The recording medium is, for example, a dedicated tool connected to a connector provided inside the cab 10 via a detachable cable. Furthermore, the recording medium may be, for example, a general-purpose recording medium such as an SD memory card or a USB (Universal Serial Bus) memory. Thus, a program for realizing various functions of the controller 30 may be provided, for example, by a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. Furthermore, the program may be downloaded from another computer outside the shovel 100 via the communication device 60 and installed in the auxiliary storage device 30A.

In addition, part of the functions of the controller 30 may be realized by other controllers (control devices). In other words, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers.

The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot operating device 26, that is, detects the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. The detection signal of the pilot pressure corresponding to the operating state related to each driven element (hydraulic actuator HA) in the operating device 26 detected by the operating pressure sensor 29 is input to the controller 30.

When the operating device 26 is an electrical type, the operating pressure sensor 29 can be omitted. This is because the controller 30 can grasp the operating state of each driven element of the operating device 26 based on the operating signal input from the operating device 26 .

The camera device 40 acquires an image of the vicinity of the shovel 100. Furthermore, the camera device 40 may acquire (generate) three-dimensional data (hereinafter referred to as "three-dimensional data of the object") representing the position and shape of an object around the shovel 100 within the camera range (angle of view) based on the acquired image and data related to the distance described later. The three-dimensional data of the object around the shovel 100 is, for example, data representing coordinate information of a point group on the surface of the object or distance image data.

For example, as shown in FIG2 , the camera device 40 includes a camera 40F for photographing the front of the upper swing body 3, a camera 40B for photographing the rear of the upper swing body 3, a camera 40L for photographing the left side of the upper swing body 3, and a camera 40R for photographing the right side of the upper swing body 3. Thus, the camera device 40 can photograph the entire circumference of the shovel 100 as the center, that is, the range of 360 degrees in the angular direction when looking down at the shovel 100. In addition, the operator can visually recognize the peripheral images such as the camera images of the cameras 40B, 40L, and 40R or the processed images generated based on the camera images through the output device 50 (display device) or the remote operation support device 300, and confirm the status of the left, right, and rear of the upper swing body 3. Furthermore, the operator can remotely operate the shovel 100 while visually recognizing the camera image of the camera 40F or the peripheral image such as the processed image generated based on the camera image through the remote operation support device 300, and confirming the movement of the attachment AT including the bucket 6. Hereinafter, the cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X".

The camera 40X is, for example, a monocular camera and may be capable of acquiring data related to distance (depth) in addition to two-dimensional images, such as a stereo camera or a TOF (Time Of Flight) camera (hereinafter collectively referred to as a "3D camera").

The output data of the camera device 40 (camera 40X) (for example, image data or three-dimensional data of objects around the shovel 100, etc.) is input to the controller 30 through a one-to-one communication line or an in-vehicle network. Thus, for example, the controller 30 can monitor objects around the shovel 100 based on the output data of the camera 40X. And, for example, the controller 30 can determine the surrounding environment of the shovel 100 based on the output data of the camera 40X. And, for example, the controller 30 can determine the posture state of the attachment AT shown in the camera image based on the output data of the camera 40X (camera 40F). And, for example, the controller 30 can determine the posture state of the body (upper swing body 3) of the shovel 100 based on the objects around the shovel 100 based on the output data of the camera 40X.

In addition, part of the cameras 40F, 40B, 40L, and 40R may be omitted. For example, when the excavator 100 is not remotely operated, the cameras 40F and 40L may be omitted. This is because it is easier to confirm the state of the front or left side of the excavator 100 from the operator of the cab 10. Furthermore, a distance sensor may be provided on the upper rotating body 3 instead of the camera device 40 (camera 40X) or in addition thereto. The distance sensor is, for example, installed on the upper part of the upper rotating body 3 to obtain data related to the distance and direction of the surrounding objects based on the excavator 100. Furthermore, the distance sensor may also obtain (generate) three-dimensional data (for example, data of coordinate information of a point group) of objects around the excavator 100 within the sensing range based on the obtained data. The distance sensor is, for example, LIDAR (Light Detection and Ranging). Furthermore, for example, the distance sensor may be a millimeter wave radar, an ultrasonic sensor, an infrared sensor, or the like.

＜Hardware structure of information processing device＞

FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device 200 .

The functions of the information processing device 200 are implemented by arbitrary hardware or arbitrary combination of hardware and software, etc. For example, as shown in FIG5 , the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed computing device 205, a communication interface 206, an input device 207, and a display device 208 connected by a bus B2.

The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. The recording medium 201A includes, for example, a floppy disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, and a USB memory. Thus, the information processing device 200 can read various data used in the processing through the recording medium 201A, store it in the auxiliary storage device 202, or install programs that realize various functions.

Furthermore, the information processing device 200 may acquire various data or programs used in processing from an external device via the communication interface 206 .

The auxiliary storage device 202 stores various installed programs, and stores files and data required for various processes, etc. The auxiliary storage device 202 includes, for example, a HDD (Hard Disc Drive), an SSD (Solid State Disc), a flash memory, and the like.

When there is an instruction to start the program, the memory device 203 reads and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, a DRAM (Dynamic Random Access Memory) or an SRAM.

The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203 , and realizes various functions related to the information processing device 200 according to the programs.

The high-speed computing device 205 performs computing at a relatively high speed in conjunction with the CPU 204. The high-speed computing device 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array).

In addition, the high-speed computing device 205 may be omitted depending on the required computing processing speed.

The communication interface 206 is used as an interface for connecting to external devices so as to be communicable. Thus, the information processing device 200 can communicate with external devices such as the shovel 100 through the communication interface 206. The communication interface 206 may have multiple communication interfaces depending on the communication method with the connected device.

The input device 207 receives various inputs from the user.

The input device 207 includes, for example, an operation input device that receives mechanical operation input from the user. The operation input device includes, for example, a button, a switch key, a joystick, etc. In addition, the operation input device includes, for example, a touch panel installed on the display device 208, a touch pad provided separately from the display device 208, etc.

Furthermore, the input device 207 includes, for example, a voice input device capable of receiving voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

Furthermore, the input device 207 includes, for example, a gesture input device capable of receiving gesture input from the user. The gesture input device includes, for example, a camera capable of capturing the gesture state of the user.

Furthermore, the input device 207 includes, for example, a biometric input device capable of receiving a biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data containing information related to a user's fingerprint or iris.

The display device 208 displays an information screen or an operation screen to the user. For example, the display device 208 includes the above-mentioned remote operation display device. The display device 208 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.

In addition, the remote operation support device 300 can be implemented by any hardware or any combination of hardware and software, and can adopt the same hardware structure as the information processing device 200. For example, the remote operation support device 300 is the same as the information processing device 200 (Figure 5), and is composed of a computer including a CPU, a memory device, an auxiliary storage device, an interface device, an input device, and a display device as the center. The memory device is, for example, an SRAM or a DRAM. The auxiliary storage device is, for example, an HDD, an SSD, an EEPROM, or a flash memory. The interface device includes an external interface for connecting to an external recording medium or a communication interface for communicating with the outside of the excavator 100. The input device includes, for example, a joystick-type operation input device. Thus, the operator can use the operation input device to perform operation input related to the actuator of the excavator 100, and the remote operation support device 300 can use the communication interface to send a signal corresponding to the operation input to the excavator 100. Therefore, the operator can remotely operate the excavator 100 using the remote operation support device.

[Functional structure of operation support device]

Next, the functional configuration of the operation support device 150 mounted on the shovel 100 will be described with reference to FIGS. 6 to 8 in addition to FIGS. 1 to 5 .

＜Case 1＞

Fig. 6 is a block diagram showing a first example of the functional configuration of the operation support device 150. Fig. 7 is a diagram showing an example (operation instruction notification screen 800) of a screen (hereinafter referred to as "operation instruction notification screen") displayed on the display device 50A and notifying instructions related to the operation of the shovel 100.

The operation support device 150 supports the operation of the shovel 100 by the operator OP riding in the cabin 10 .

As shown in FIG. 6 , the operation support device 150 includes an operation device 26 , a controller 30 , an imaging device 40 , an output device 50 , and a communication device 60 .

The controller 30 includes an operation instruction accepting unit 3001 and an operation instruction notifying unit 3002 as functional units.

The operation instruction receiving unit 3001 receives instructions related to the operation of the excavator 100 from the outside of the excavator 100. The instructions related to the operation of the excavator 100 include, for example, instructions to specify a driven element of the operation object and operate the driven element in a specific direction (hereinafter, for convenience, referred to as "operation instructions to specify driven elements"). The operation instructions to specify driven elements include, for example, operation instructions to raise or lower the boom 4, operation instructions to rotate the upper rotating body 3 to the left or right, etc. In addition, the instructions related to the operation of the excavator 100 include operation instructions to specify a prescribed action of the excavator 100 included in a prescribed operation and implement the action (hereinafter, for convenience, referred to as "operation instructions to specify actions"). The prescribed operation includes, for example, an operation that is achieved by a combination of a series of actions of a plurality of excavators 100, such as a lifting operation. The lifting operation is composed of, for example, a lifting action of a hanging object, an action of horizontally moving the hanging object, an action of lowering the hanging object, and an action of stopping the movement of the hanging object. The action of horizontal movement of the suspended object includes the action of horizontal movement of the suspended object caused by the walking of the lower walking body 1 and the action of horizontal movement of the suspended object caused by the rotation of the upper rotating body 3. In addition, the instructions related to the operation of the excavator include instructions for adjusting the operation speed of the driven element accompanying the current operation or the movement speed of the suspended object (hereinafter, for convenience, referred to as "speed adjustment instructions"). The speed adjustment instructions include, for example, instructions for increasing the speed and instructions for reducing the speed. In addition, the instructions for increasing the speed and the instructions for reducing the speed may also include instructions for the increase or decrease of the specified speed, respectively. In addition, the instructions related to the operation of the excavator 100 include instructions related to the steps (preparation) of the operation performed by operating the excavator 100 (hereinafter, referred to as "operation step instructions"). The operation step instructions include operations such as operations whose order of operations has not yet been determined, instructions indicating the order of operations, or instructions indicating the order of operations when multiple operations are performed continuously.

For example, the operation instruction receiving unit 3001 receives instructions related to the operation of the shovel 100 from people around the shovel 100 (hereinafter, for convenience, referred to as "instructors") The instructors are, for example, workers who collaborate in the same operation, supervisors who supervise the operation of the shovel 100, or managers who manage the construction site of the shovel 100.

Specifically, the operation instruction receiving unit 3001 can recognize a gesture indicating an instruction related to the operation of the shovel 100 performed by an indicator around the shovel 100 based on the camera image (dynamic image) of the camera device 40 by using a known image recognition technology, etc. Thus, the operation instruction receiving unit 3001 can receive an instruction related to the operation of the shovel 100 corresponding to the recognized gesture.

The gestures representing instructions related to the operation of the excavator 100 are predefined for each instruction content of the object, and information related to the gestures of each instruction content is pre-registered in the auxiliary storage device 30A, etc. For example, in the case of an operation instruction for specifying a driven element, different gestures are predefined for each instruction content such as the operation of the lifting direction of the boom 4, the operation of the lowering direction of the boom 4, the operation of the left rotation direction of the upper swing body 3, and the operation of the right rotation direction of the upper swing body 3. And, for example, in the case of an action instruction related to a lifting operation, a gesture is predefined for each instruction content such as the action of lifting a hanging object, the action of moving a hanging object horizontally, the action of dropping a hanging object, the action of stopping the movement of a hanging object, and the action of slightly moving a hanging object. The gesture for specifying the action of slightly moving a hanging object may also include an instruction content specifying the direction or amount of the action. And, for example, in the case of a speed adjustment instruction, a gesture is predefined for each instruction content such as an instruction to increase the speed, an instruction to reduce the speed, etc. The gesture for specifying an instruction to increase the speed or an instruction to reduce the speed may also include a gesture for specifying the increase or decrease of the speed. Furthermore, for example, in the case of work procedure instructions, a gesture is defined in advance for each of a plurality of actions or for each of a plurality of tasks, and the work procedure instructions are implemented by combining the gestures.

Furthermore, the operation instruction receiving unit 3001 may recognize a gesture of an instructor around the shovel 100 based on the output of the distance sensor.

Furthermore, the operation instruction receiving unit 3001 may receive instructions related to the operation of the shovel 100 based on a signal including instructions related to the operation of the shovel 100 (hereinafter referred to as "instruction signal") received from the information processing device 200 carried by the indicator around the shovel 100. At this time, the information processing device 200 transmits the instruction signal including the contents of the instruction related to the operation of the shovel 100 specified by the input of the indicator through the input device 207, and the shovel 100 receives the instruction signal through the communication device 60. Thus, the indicator around the shovel 100 can issue instructions related to the operation of the shovel 100 to the shovel 100 (operator OP) using the information processing device 200.

Furthermore, the operation instruction receiving unit 3001 receives instructions related to the operation of the shovel 100 from a supervisor who remotely monitors the operation of the shovel 100 through the information processing device 200. For example, in the case of a lifting operation, workers at the construction site and the like need to be separated from the crane by a certain distance, so there may be a situation where the supervisor can more accurately grasp the status of the hanging object and issue instructions related to the operation of the shovel 100.

Specifically, the operation instruction receiving unit 3001 can receive instructions related to the operation of the shovel 100 based on a signal (instruction signal) including instructions related to the operation of the shovel 100 received from the information processing device 200 used by the supervisor. At this time, the information processing device 200 transmits an instruction signal including the contents of the instruction related to the operation of the shovel 100 specified by the input of the instruction person through the input device 207, and the shovel 100 receives the instruction signal through the communication device 60. Thus, the instruction person around the shovel 100 can issue instructions related to the operation of the shovel 100 to the shovel 100 (operator OP) using the information processing device 200.

The operation instruction notification unit 3002 notifies the operator OP of the shovel 100 of the instruction related to the operation of the shovel 100 received by the operation instruction reception unit 3001 through the output device 50. The operation instruction notification unit 3002 may perform the notification visually through the display device 50A or the like, may perform the notification auditorily through the sound output device 50B, or may perform the notification by both.

For example, the operation instruction notification unit 3002 notifies the content of the instruction related to the operation of the shovel 100 directly through the output device 50 .

Furthermore, the operation instruction notification unit 3002 may notify the operation method of the operation device 26 corresponding to the content of the instruction related to the operation of the shovel 100 through the output device 50 (see FIG. 7 ). Thus, the specific operation method of the operation device 26 is notified, so that, for example, even when the operator OP who actually operates the shovel 100 relatively little is operating, the content of the instruction related to the operation of the shovel 100 can be easily grasped.

As shown in FIG7 , an image 801 simulating a joystick device included in the operating device 26 is displayed in the operation instruction notification screen 800. The image 801 includes an image 801L simulating an operating lever on the left side of the driver's seat and an image 801R simulating an operating lever on the right side of the driver's seat. The operating lever on the left side of the driver's seat is used, for example, to operate the boom 5 by operating in the front-to-back direction and to perform a rotating operation on the upper rotating body 3 by opening and closing in the left-to-right direction. The operating lever on the right side of the driver's seat is used, for example, to perform a lifting and lowering operation on the boom 4 by operating in the front-to-back direction and to perform an opening and closing operation on the bucket 6 by operating in the left-to-right direction.

In this example, a mark image 802 indicating an operation target is displayed superimposed on an image 801R of an operation lever on the right side of the driver's seat. This allows the operator OP to easily recognize that the operation lever on the right side is an operation target.

In this example, a downward arrow image 803 indicating the operation direction is displayed along with the image 801R of the operation lever of the driver's seat. Thus, the operator OP can easily recognize that an instruction to operate the right operation lever downward (i.e., an instruction to lower the boom 4) has been issued.

In addition, the operation instruction notification unit 3002 may also use a lighting device (indicator lamp) mounted on the operating device 26 instead of or in addition to the display device 50A to notify the operating method of the operating device 26 corresponding to the content of the instruction related to the operation of the shovel 100. For example, an indicator lamp indicating the operation object and an indicator lamp indicating the operation direction are mounted on the operating lever on the left side and the operating lever on the right side of the driver's seat.

Thus, in this example, the controller 30 can notify the operator OP of instructions related to the operation of the shovel 100 from outside the shovel 100 through the output device 50. Therefore, the operator OP can operate the shovel 100 more accurately after confirming the instructions from outside the shovel 100.

Furthermore, in this example, the instruction related to the operation of the shovel 100 is received by the shovel 100 (controller 30), and the operator OP performs the operation after recognizing the received instruction related to the operation of the shovel 100. Therefore, the shovel 100 is operated after the shovel 100 and the operator OP double-check the instruction from the outside of the shovel 100, so the shovel 100 can be operated more accurately.

＜Example 2＞

FIG. 8 is a functional block diagram showing a second example of the functional configuration of the operation support device 150 .

The description will be mainly focused on the parts different from the above-mentioned first example ( FIG. 6 ), and the description of the contents identical or corresponding to the above-mentioned first example may be omitted.

8 , for convenience, a communication device 60 representing a receiving function of receiving a signal from the information processing device 200 and a communication device 60 representing a transmitting function of transmitting a signal to the information processing device 200 are depicted separately.

As shown in FIG. 8 , the operation support device 150 includes a controller 30 , an imaging device 40 , an output device 50 , and a hydraulic control valve 33 .

The controller 30 includes an operation instruction receiving unit 3001 , an operation instruction notifying unit 3002 , an operation content evaluating unit 3003 , a storage unit 3004 , and a transmitting unit 3005 as functional units.

The operation content evaluation unit 3003 evaluates the actual operation content of the operator with respect to the content of the instruction related to the operation of the shovel 100 which is received by the operation instruction reception unit 3001 and notified by the operation instruction notification unit 3002 .

For example, the operation content evaluation unit 3003 evaluates whether there is a difference between the content of the instruction related to the operation of the shovel 100 and the content of the actual operation input to the operation device 26 by the operator OP after the notification by the operation instruction notification unit 3002. Furthermore, when there is a difference between the content of the instruction related to the operation of the shovel 100 and the content of the actual operation input to the operation device 26 by the operator OP, the operation content evaluation unit 3003 notifies the user of the difference through the output device 50. The operation content evaluation unit 3003 may notify the user visually through the display device 50A or the like, may notify the user auditorily through the sound output device 50B, or may notify the user through both. Thus, for example, regarding the operation performed by the operator OP, it is possible to warn the operator OP that the operation is different from the instruction related to the operation of the shovel 100 from the outside of the shovel 100.

Furthermore, when there is a difference between the content of the instruction related to the operation of the shovel 100 and the content of the actual operation input of the operator OP to the operating device 26, the operation content evaluation unit 3003 may control the hydraulic control valve 33 to limit the operation of the shovel 100. The limitation of the operation of the shovel 100 includes, for example, stopping the operation of the shovel 100 in which the driven element is stopped and maintained in a stopped state without depending on the operation of the operator. Furthermore, the limitation of the operation of the shovel 100 may include, for example, decelerating the operation of the shovel 100 in which the operation of the driven element is slower than usual with respect to the operation of the operator. The driven elements for limiting the operation of the shovel 100 may be all the driven elements or only a part of the driven elements including the objects that are operated by the actual operation input of the operator OP to the operating device 26. Thus, even when the operator OP performs an operation different from the content of the instruction related to the operation of the shovel 100, the operation of the shovel 100 can be limited, and the safety of the shovel 100 can be improved.

Furthermore, when there is a difference between the content of the instruction regarding the operation of the shovel 100 and the content of the actual operation input by the operator OP to the operating device 26 , the operation content evaluation unit 3003 may record a log indicating this fact in the storage unit 3004 .

The log includes, for example, information indicating the date and time when a difference occurs between the content of the instruction related to the operation of the shovel 100 and the content indicating the actual operation input of the operator OP to the operating device 26, and information indicating the fact. Furthermore, the log may include the content of the instruction related to the operation of the shovel 100 and the content of the actual operation input of the operator OP to the operating device 26. Furthermore, the log may include data related to the operating state of the shovel 100 when a difference occurs between the content of the instruction related to the operation of the shovel 100 and the content of the actual operation input of the operator OP to the operating device 26. The data related to the operating state of the shovel 100 includes, for example, image data of the camera 40. Thus, for example, the logs accumulated in the storage unit 3004 can be analyzed, and evaluation related to the instruction of the instruction person, evaluation related to the operation of the operator, or evaluation related to the work of the shovel 100 can be facilitated.

The storage unit 3004 stores the above-mentioned log data.

In addition, the log data in the storage unit 3004 may be deleted at a predetermined time after being uploaded to the information processing device 200 via the transmission unit 3005 described later.

The transmission unit 3005 transmits (uploads) the log data in the storage unit 3004 to the information processing device 200 .

For example, the sending unit 3005 automatically sends the unsent part of the log of the storage unit 3004 to the information processing device 200 at a predetermined time. The predetermined time is, for example, the time when the operation of the shovel 100 is stopped (i.e., the time when the key switch is turned "off") or the time when the operation of the shovel 100 is started (i.e., the time when the key switch is turned "on"), etc.

Furthermore, the transmission unit 3005 may transmit the untransmitted log data in the storage unit 3004 to the information processing device 200 in response to a signal for requesting transmission of log data received from the information processing device 200 via the communication device 60 .

Thus, in this example, the controller 30 can evaluate the difference between the content of the instruction related to the operation of the shovel 100 and the content of the actual operation input by the operator OP. Therefore, when the operation of the operator OP is different from the instruction, a warning is issued to the operator OP, or the action of the shovel 100 is restricted, thereby improving the safety of the shovel 100.

[Another example of operation support system]

Next, another example of the operation support system SYS will be described with reference to FIG. 9 .

Fig. 9 is a diagram showing another example of a construction machine, more specifically, a side view showing an example of a crawler crane 500.

The above-described embodiments related to the operation support system SYS may be combined in their contents as appropriate, or may be modified or altered.

For example, in the above-mentioned embodiment, the operation support device 150 may also use the remote operation support device 300 to support the operation of the operator who remotely operates the shovel 100. In this case, the operation instruction notification unit 3002 of the operation support device 150 transmits data related to the notification of instructions related to the operation of the shovel 100 to the remote operation support device 300 via the communication device 60. Thus, the remote operation support device 300 can notify the remote operator of the instructions related to the operation of the shovel 100 using the display device or the sound output device based on the data received from the shovel 100.

Furthermore, in the above-mentioned embodiment or its modified example, the same function as that of the operation support device 150 may be provided in the remote operation support device 300 (hereinafter, for convenience, referred to as "remote operation operation support device"). That is, part or all of the functions of the controller 30 related to the operation support device 150 may be transferred to the remote operation support device 300. In this case, the remote operation operation support device includes a remote operation operation device instead of the operation device 26, and the remote operation operation device is included in the input device of the remote operation support device 300.

Specifically, the function of the operation instruction notification unit 3002 can be transferred to the remote operation support device 300. As a result, the remote operation support device 300 can notify the remote operator of the instruction related to the operation of the shovel 100 through the display device or the like. Therefore, the remote operator can confirm the notification and perform operation input to the input device (remote operation operation device) of the remote operation support device 300 to operate the shovel 100. At this time, the data indicating the instruction related to the operation of the shovel 100 from the indicator around the shovel 100 received by the operation instruction receiving unit 3001 is sent from the shovel 100 to the information processing device 200 via the communication device 60.

Furthermore, in addition to the function of the operation instruction notification unit 3002 of the operation support device 150, the function of the operation instruction receiving unit 3001 may be transferred to the remote operation support device 300. At this time, the data of the camera device 40 is transmitted (uploaded) from the shovel 100 to the remote operation support device 300 via the communication device 60. Thus, the remote operation support device can recognize the gesture of the pointer around the shovel 100 and receive instructions related to the operation of the shovel 100. In addition, at this time, the instruction signal may be transmitted from the information processing device 200 used by the pointer to the remote operation support device 300 via the shovel 100, or may be directly transmitted from the information processing device 200 to the remote operation support device 300.

Similarly, in addition to the function of the operation instruction notification unit 3002 of the operation support apparatus 150 , the functions of the operation content evaluation unit 3003 , the storage unit 3004 , or the transmission unit 3005 may be transferred to the remote operation support apparatus 300 .

Furthermore, in the above-mentioned embodiment or its modified example, the same functions as those of the operation support device 150 may be provided in the portable information processing device 200 that the operator brings into the cab 10 of the excavator 100 (hereinafter, referred to as "portable operation support device" for convenience). That is, part or all of the functions of the controller 30 related to the operation support device 150 may also be transferred to the information processing device 200.

Specifically, the function of the operation instruction notification unit 3002 can be transferred to the portable information processing device 200. As a result, the portable information processing device 200 can notify the remote operator of the instruction related to the operation of the shovel 100 through the display device or the like. Therefore, the remote operator can confirm the notification and perform operation input to the input device (remote operation operation device) of the portable information processing device 200 to operate the shovel 100. At this time, the data indicating the instruction related to the operation of the shovel 100 from the indicator around the shovel 100 received by the operation instruction receiving unit 3001 is sent from the shovel 100 to the information processing device 200 through the communication device 60.

Furthermore, in addition to the function of the operation instruction notification unit 3002 of the operation support device 150, the function of the operation instruction receiving unit 3001 may be transferred to the portable information processing device 200. At this time, the data of the camera device 40 is transmitted (uploaded) from the shovel 100 to the portable information processing device 200 via the communication device 60. Thus, the remote operation support device can recognize the gesture of the pointer around the shovel 100 and receive instructions related to the operation of the shovel 100. In addition, at this time, the instruction signal may be transmitted from the information processing device 200 used by the pointer to the portable information processing device 200 via the shovel 100, or may be directly transmitted from the information processing device 200 to the portable information processing device 200.

Similarly, in addition to the function of the operation instruction notification unit 3002 of the operation support device 150 , the functions of the operation content evaluation unit 3003 , the storage unit 3004 , or the transmission unit 3005 may be transferred to the portable information processing device 200 .

Furthermore, the operation support device 150 according to the above-mentioned embodiment or its modified example may also receive instructions related to the operation of other construction machines different from the excavator 100, and notify the user, that is, the operator of the construction machine, of the instructions related to the operation of the construction machine. For example, as shown in FIG. 9 , the other construction machine is a crawler crane 500. In this case, the operation support system SYS includes other construction machines such as the crawler crane 500 instead of or in addition to the excavator 100.

[effect]

Next, the operation of the operation support system SYS (operation support device) according to the present embodiment will be described.

For example, when an instruction is issued from outside the construction machine and the construction machine is directly operated, there may be a blind spot such as the rear of the construction machine around the construction machine depending on the position of the person issuing the instruction, and as a result, appropriate operations corresponding to the conditions around the construction machine may not be performed. Also, when instructions are issued from outside the construction machine by human actions such as gestures and the construction machine is directly operated, the actions of the construction machine may not be accurately operated depending on the recognition accuracy.

In contrast, in the present embodiment, the operation support device includes a receiving unit that receives instructions related to the operation of the construction machine from outside the construction machine, and a notification unit that notifies the user of the instructions related to the operation of the construction machine received by the receiving unit. The operation support device is, for example, the operation support device 150, or an operation support device for remote operation, or a portable operation support device. Furthermore, the construction machine is, for example, the excavator 100 or the crawler crane 500. Furthermore, the receiving unit is, for example, the operation instruction receiving unit 3001. Furthermore, the notification unit is, for example, the operation instruction notification unit 3002.

Furthermore, the operation support device may include an operation unit that receives an operation input to the construction machine by the user. The operation unit is, for example, the operation device 26 of the shovel 100 or an operation device for remote operation in the remote operation support device 300 .

Thus, the contents of instructions from outside the construction machine are recognized by the operation support device, and the contents of the instructions are notified to the user (operator), the user recognizes the contents of the instructions, and the user who recognizes the contents of the instructions operates the construction machine. Therefore, the contents of the instructions related to the operation of the construction machine are double-checked by the operation support device and the operator. As a result, for example, in the case where there is a problem in the recognition of the instructions based on the operation support device 150, the operator can be prevented from performing the operation according to the contents of the instructions. And, for example, in the case where there are obstacles around the excavator 100 that the indicator cannot grasp and the operation according to the contents of the instructions is inappropriate, the operation according to the contents of the instructions can be prevented from being performed. Therefore, the operation support device can make the construction machine operate more accurately through instructions from outside the construction machine.

Furthermore, in the present embodiment, the instructions related to the operation of the construction machinery may include a first instruction for specifying a driven element of the construction machinery and operating the driven element in a specific direction, a second instruction for specifying an action constituting an operation and performing an operation to implement the action, a third instruction for performing an operation to adjust the operation speed of the construction machinery, and at least one of a fourth instruction related to multiple actions or steps of multiple operations.

Thus, the operation support device can instruct the user to operate a specific driven element in a specific direction. Furthermore, the operation support device can instruct the user to perform an operation that constitutes a specific action of the operation. Furthermore, the operation support device can instruct the user to adjust the action speed of the construction machine. Furthermore, the operation support device can instruct the user to perform multiple actions or multiple steps of the operation.

Furthermore, in the present embodiment, the receiving unit may receive instructions regarding the operation of the construction machine from an instructor around the construction machine.

This allows the user to operate the construction machine while confirming instructions from an instructor around the construction machine.

Furthermore, in the present embodiment, the receiving unit may recognize a gesture indicating an instruction related to the operation of the construction machine from the instruction person based on the sensor information indicating the state of the periphery of the construction machine, and receive the instruction related to the operation of the construction machine.

This allows the instructor to issue instructions to the user who operates the construction machine through hand gestures.

Furthermore, in this embodiment, a notification unit may be provided for notifying the user when there is a difference between the content of the instruction related to the operation of the construction machine notified by the notification unit and the content of the subsequent actual operation input to the construction machine. The notification unit is, for example, the operation content evaluation unit 3003.

Thus, the operation support device can warn the user, for example, that there is a difference between the content of the actual user's operation input to the construction machine and the instruction related to the operation of the construction machine.

Furthermore, in this embodiment, a recording unit may be provided, which, when there is a difference between the content of the instruction related to the operation of the construction machine notified by the notification unit and the content of the subsequent actual operation input to the construction machine, records a log indicating the situation in a predetermined storage unit. The recording unit is, for example, the operation content evaluation unit 3003.

Thereby, the operation support device can analyze the log, for example, and perform evaluation related to the instruction of the instructor, evaluation related to the operation of the operator, evaluation related to the work of the construction machine, and the like.

Furthermore, in the present embodiment, the construction machine may include the above-mentioned operation support device.

Thereby, the construction machine can notify users such as an operator riding on the construction machine or an operator performing remote operation of instructions from outside the construction machine.

Furthermore, in the present embodiment, the remote operation support device may also include the above-mentioned operation support device. Furthermore, the operation unit may also receive operation input for remote operation of the construction machine. The remote operation support device is, for example, the remote operation support device 300. The operation unit is, for example, an operation device for remote operation.

Thereby, the remote operation support device can notify the remote operator of instructions from outside the construction machine.

[Overview of remote operation system]

Next, with reference to FIG. 10 , an overview of the remote operation system SYS1 of the shovel 100 according to the present embodiment will be described.

Hereinafter, the shovel 100 will be described mainly around portions that are different from the above-described operation support system SYS, and description of portions that are the same as or correspond to those of the operation support system SYS may be omitted.

Fig. 10 is a diagram showing an example of a remote operation system SYS1 of the shovel 100. Fig. 10 shows a left side view of the shovel 100.

In addition, the top view of the shovel 100 may be the same as the above-mentioned Fig. 2. Therefore, in the description of the remote operation system SYS1, the top view of the shovel 100 is omitted, and the above-mentioned Fig. 2 is used.

As shown in FIG. 10 , the remote operation system SYS1 includes a shovel 100 and a remote operation support device 300 .

The remote operation system SYS1 implements remote operation of the shovel 100 by the user using the remote operation support device 300. Hereinafter, for convenience, a user who remotely operates the shovel 100 using the remote operation support device 300 may be referred to as a "remote operation operator".

The shovel 100 is a construction machine that is subject to remote control.

In addition, the object of remote operation in the remote operation system SYS1 may be another construction machine different from the excavator 100. For example, the remote operation system SYS1 may include a crawler crane 500 as the object of remote operation (see FIG. 9 above). The construction machine of the object of ensuring safety in the safety system SYS2 described later may also be the same.

As shown in FIGS. 2 and 10 , the shovel 100 includes a lower traveling body 1 , an upper swing body 3 , an attachment AT including a boom 4 , an arm 5 , and a bucket 6 , and a cab 10 .

The lower traveling body 1 includes, for example, a left crawler track 1CL and a right crawler track 1CR. The crawler tracks 1CL and 1CR are hydraulically driven by corresponding traveling hydraulic motors 1M to move themselves.

The upper revolving body 3 is mounted on the lower traveling body 1 so as to be revolvable via the revolving mechanism 2. For example, the upper revolving body 3 is revolved relative to the lower traveling body 1 by hydraulically driving the revolving mechanism 2 with a revolving hydraulic motor 2M.

The boom 4 is mounted at the front center of the upper revolving body 3 so as to be able to pitch about a rotation axis in the left-right direction. The arm 5 is mounted at the front end of the boom 4 so as to be able to rotate about a rotation axis in the left-right direction. The bucket 6 is mounted at the front end of the arm 5 so as to be able to rotate about a rotation axis in the left-right direction.

The bucket 6 is an example of an end attachment, and is mounted on the front end of the arm 5 in a manner that allows for appropriate replacement according to the work content of the excavator 100. That is, a bucket of a different type from the bucket 6 can be mounted on the front end of the arm 5 instead of the bucket 6, for example, a relatively large bucket, a bucket for slopes, a bucket for dredging, etc. Also, an end attachment other than a bucket can be mounted on the front end of the arm 5, for example, a mixer, a crusher, a crusher, etc. can be mounted. Also, a preliminary attachment such as a quick coupling, a tilt rotator, etc. can be provided between the arm 5 and the end attachment.

The boom 4, the arm 5 and the bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8 and a bucket cylinder 9, respectively.

The cab 10 is a control room in which an operator sits and operates the shovel 100 , and is mounted on the front left side of the upper swing body 3 , for example.

As described above, the shovel 100 is configured to be remotely operated from the remote operation support device 300. Specifically, the shovel 100 operates driven elements such as the lower traveling body 1 (i.e., a pair of left and right crawlers 1C), the upper swing body 3, the boom 4, the arm 5, and the bucket 6 according to the operation content of the remote operator received from the remote operation support device 300. When the shovel 100 is remotely operated, the interior of the cab 10 may be unmanned.

Furthermore, the excavator 100 may be configured to operate driven elements such as the lower traveling body 1 (i.e., a pair of left and right crawlers 1C), the upper swing body 3, the boom 4, the arm 5, and the bucket 6 according to the operation of the operator riding in the cab 10. Hereinafter, the operation of the remote operator and the operation of the operator in the cab 10 may be collectively referred to as "operator's operation".

In addition, when the shovel 100 can be operated only by remote control, the cab 10 may be omitted.

The remote operation support device 300 receives input related to the operation of the shovel 100 from a user (remote operator) and transmits a signal indicating the operation content (hereinafter referred to as a “remote operation signal”) to the shovel 100 , thereby supporting the remote operation of the shovel 100 .

The remote operation support device 300 is, for example, a portable terminal device (i.e., a mobile terminal) that can be carried by a user. Thus, at the construction site of the shovel 100, the user can visually recognize the position of the shovel 100 and the working object of the shovel 100 while using the remote operation support device 300 to remotely operate the shovel 100.

The remote operation support device 300 is a mobile terminal dedicated to remote operation of the shovel 100, and may be a general-purpose mobile terminal such as a smartphone or a tablet terminal. In the latter case, a dedicated application program that can operate in cooperation with the shovel 100 may be pre-installed.

[Hardware structure of remote operating system]

Next, the hardware structure of the remote operation system SYS1 will be described with reference to FIG. 11 in addition to FIG. 10 .

In addition, the hardware configuration of the shovel 100 may be the same as that of Fig. 4 described above. Therefore, illustration and description of the hardware configuration of the shovel 100 in the remote operation system SYS1 are omitted.

＜Hardware structure of remote operation support device＞

FIG. 11 is a diagram showing an example of the hardware configuration of the remote operation support device 300 .

The functions of the remote operation support device 300 are implemented by arbitrary hardware or a combination of arbitrary hardware and software, etc. For example, as shown in FIG11 , the remote operation support device 300 includes an external interface 301, an auxiliary storage device 302, a memory device 303, a CPU 304, a communication interface 306, an input device 307, and a display device 208 connected via a bus B2.

The external interface 301 functions as an interface for reading data from or writing data to the recording medium 301A. The recording medium 301A includes, for example, a floppy disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, and a USB memory. Thus, the remote operation support device 300 can read various data used in the process through the recording medium 301A, store it in the auxiliary storage device 302, or install programs that realize various functions.

Furthermore, the remote operation support device 300 may acquire various data or programs from an external device via the communication interface 306 .

The auxiliary storage device 302 stores various installed programs, and stores files and data required for various processes, etc. The auxiliary storage device 302 includes, for example, a HDD (Hard Disc Drive), an SSD (Solid State Drive), a flash memory, and the like.

When there is an instruction to start the program, the memory device 303 reads and stores the program from the auxiliary storage device 302. The memory device 303 includes, for example, a DRAM (Dynamic Random Access Memory) or an SRAM.

The CPU 304 executes various programs loaded from the auxiliary storage device 302 to the memory device 303 , and realizes various functions related to the remote operation support device 300 according to the programs.

The communication interface 306 is used as an interface for connecting to an external device so as to be communicable. Thus, the remote operation support device 300 can communicate with an external device such as the shovel 100 through the communication interface 306. The communication interface 306 may have a plurality of communication interfaces depending on the communication method with the connected device.

The input device 307 receives various inputs from the user. For example, the input device 307 includes an input device for an operator to perform remote operation, that is, an operation device for remote operation.

The input device 307 includes, for example, an operation input device for receiving mechanical operation input from the user. The operation input device includes, for example, a button, a switch key, a joystick, etc. In addition, the operation input device includes, for example, a touch panel installed on the display device 208, a touch pad provided separately from the display device 208, etc.

Furthermore, the input device 307 includes, for example, a voice input device capable of receiving voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

Furthermore, the input device 307 includes, for example, a gesture input device capable of receiving gesture input from the user. The gesture input device includes, for example, a camera capable of capturing the gesture state of the user.

Furthermore, the input device 307 includes, for example, a biometric input device capable of receiving a biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data containing information related to a user's fingerprint or iris.

The display device 208 displays an information screen or an operation screen to the user. For example, the display device 208 includes the above-mentioned remote operation display device. The display device 208 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.

[Functional structure related to the peripheral monitoring function of the excavator]

Next, a functional configuration related to the periphery monitoring function of the shovel 100 will be described with reference to FIG. 12 .

FIG. 12 is a block diagram showing an example of a functional configuration related to the periphery monitoring function of the shovel 100 .

The controller 30 includes an object detection unit 3011 , a position estimation unit 3012 , a gesture recognition unit 3013 , an authentication unit 3014 , a tracking unit 3015 , and a security control unit 3016 .

The object detection unit 3011 detects obstacles (hereinafter referred to as "monitored objects") that are monitored objects around the excavator 100. Specifically, the object detection unit 3011 can perform processing for detecting monitored objects around the excavator 100 in each specified processing cycle. Monitored objects include, for example, workers and others. In addition, monitored objects may also include, for example, other construction machinery or construction vehicles. Monitored objects may also include, for example, specific fixed objects at the construction site, such as utility poles, fences, and triangular cones. In addition, monitored objects may also include, for example, specific terrain shapes at the construction site, such as trenches and pits.

The object detection unit 3011 detects the monitoring object around the shovel 100 using a known method, for example, based on the output data (camera image) of the camera 40 (an example of the second sensor) or the distance sensor (an example of the second sensor) of the shovel 100. In addition, when both the camera 40 and the distance sensor are mounted on the shovel 100, the object detection unit 3011 may also detect the object around the shovel 100 by applying the sensor fusion technology to the output data of the camera 40 and the distance sensor.

The position estimation unit 3012 estimates the actual position of the monitored object detected by the object detection unit 3011 as viewed from the shovel 100 (hereinafter referred to as the "actual position"). Specifically, if the monitored object is detected by the object detection unit 3011, the position estimation unit 3012 can perform a process of estimating the actual position of the monitored object. Furthermore, when multiple monitored objects are detected by the object detection unit 3011, the position estimation unit 3012 can estimate the actual positions of each of the multiple monitored objects.

For example, the position estimation unit 3012 determines the actual existing position of the detected monitoring object based on the position and size of the partial area of the monitored object detected in the image corresponding to the camera 40X. In addition, when the camera 40X is a 3D camera, the position estimation unit 3012 may also estimate the actual existing position of the detected monitoring object based on the coordinate information of the partial area (pixel group) of the monitored object detected in the image corresponding to the camera 40X, which is based on the shovel 100. In addition, when both the camera 40X and the distance sensor are installed in the shovel 100, the position estimation unit 3012 may also estimate the actual existing position of the detected monitoring object in the same way as in the case of the 3D camera by applying the sensor fusion technology to the output data of the camera 40X and the distance sensor. In addition, the position estimation unit 3012 may also estimate the actual existing position of the detected monitoring object based on the coordinate information of the part of the monitored object detected in the three-dimensional data of the object around the shovel generated from the output data of the distance sensor.

The gesture recognition unit 3013 (an example of the recognition unit) recognizes a predetermined gesture performed by a person around the shovel 100 based on the output data of the camera device 40 (an example of the first sensor). In addition, the gesture recognition unit 3013 may recognize a predetermined gesture performed by a person around the shovel 100 based on the output data (for example, point group data) of the distance sensor (an example of the first sensor). The predetermined gesture includes, for example, a predetermined posture or waving. In addition, the predetermined gesture may also include all actions of the person such as the movement of the person.

The authentication unit 3014 authenticates a specific person around the shovel 100. The specific person is, for example, an operator who uses the remote operation support device 300 to remotely operate the shovel 100 around the shovel 100, or a manager or supervisor of the construction site of the shovel 100. Thus, the controller 30 can distinguish between the authenticated specific person and other people around the shovel 100 to perform control related to the surrounding monitoring function (refer to FIG. 13).

Specifically, when the gesture recognition unit 3013 recognizes a predetermined gesture, the authentication unit 3014 authenticates the person who performs the predetermined gesture. The authentication unit 3014 estimates the actual location of the person who performs the predetermined gesture in the same manner as the location estimation unit 3012, and outputs the location information of the authenticated person. The details will be described later (see FIG. 13 ).

The tracking unit 3015 tracks the actual location of the person authenticated by the authentication unit 3014. Thus, the controller 30 can distinguish the person authenticated by the authentication unit 3014 from the persons detected by the object detection unit 3011.

For example, if the authentication of a specific person is completed by the authentication unit 3014, the tracking unit 3015 compares the position information of the person authenticated by the authentication unit 3014 with the position information of the objects around the shovel 100 and the types of the objects estimated by the position estimation unit 3012. Based on the comparison result, the tracking unit 3015 identifies the authenticated person from the objects detected by the object detection unit 3011. Then, the tracking unit 3015 tracks the actual location of the person authenticated by the authentication unit 3014 based on the detection result of the object detection unit 3011 and the estimation result of the position estimation unit 3012 in each predetermined processing cycle.

The safety control unit 3016 (an example of the first control unit) performs control related to the functional safety of the shovel 100 .

For example, when the object detection unit 3011 detects a monitoring object within a predetermined range around the shovel 100 (machine), the safety control unit 3016 activates the safety function. Specifically, when the actual existence position of the monitoring object determined by the position estimation unit 3012 is within a predetermined range around the shovel 100, the safety control unit 3016 may activate the safety function.

The safety function may include, for example, a notification function that outputs an alarm or the like to at least one of the inside of the cab 10, the outside of the cab 10, and the remote operator of the shovel 100 and notifies that a monitoring object has been detected. Thus, it is possible to urge the operator inside the cab 10, the staff around the shovel 100, and the operator who remotely operates the shovel 100 to pay attention to the presence of the monitoring object in the monitoring area around the shovel 100. Hereinafter, the notification function to the inside of the cab 10 (the operator, etc.) is sometimes referred to as an "internal notification function", the notification function to the outside of the shovel 100 (the staff, etc.) is sometimes referred to as an "external notification function", and the notification function to the operator who remotely operates the shovel 100 is sometimes referred to as a "remote notification function" for distinction.

Furthermore, the safety function may include, for example, a motion limiting function that limits the motion of the shovel 100 relative to the operation of the operating device 26 or the remote operation. As a result, the motion of the shovel 100 can be forcibly limited, thereby reducing the possibility of the shovel 100 approaching or contacting with surrounding objects. The motion limiting function may include a motion deceleration function that makes the motion speed of the shovel 100 relative to the operation of the operating device 26 or the remote operation slower than usual. Furthermore, the motion limiting function may also include a motion stopping function that stops the motion of the shovel 100 and maintains the stopped state regardless of the presence or absence of the operation of the operating device 26 or the remote operation.

For example, when the object detection unit 3011 detects a monitored object within a predetermined range (hereinafter referred to as a "notification range") around the excavator 100, the safety control unit 3016 starts the notification function. The notification range is, for example, a range in which the distance D from a predetermined portion of the excavator 100 is less than the threshold value Dth1. The predetermined portion of the excavator 100 is, for example, the upper swing body 3. Furthermore, the predetermined portion of the excavator 100 may also be, for example, the bucket 6 or the hook at the front end of the attachment AT. The threshold value Dth1 may be constant regardless of the direction from which the predetermined portion of the excavator 100 is viewed, or may vary depending on the direction from which the predetermined portion of the excavator 100 is viewed.

The safety control unit 3016, for example, controls the output device 50 (sound output device) to start an internal notification function or an external notification function based on sound (i.e., an auditory method) to at least one of the inside and the outside of the cockpit 10. At this time, the safety control unit 3016 may also make the pitch, sound pressure, timbre, sound emission cycle, sound content, etc. of the sound periodically emitted different according to various conditions.

Furthermore, the safety control unit 3016, for example, starts an internal notification function based on a visual method. Specifically, the safety control unit 3016 can control the output device 50 (display device) inside the cockpit 10 to display the surrounding image and the image indicating the detected surveillance object together in the output device 50. Furthermore, the safety control unit 3016 can also emphasize the surveillance object reflected in the surrounding image of the output device 50 (display device) displayed inside the cockpit 10 or the position on the surrounding image corresponding to the detected surveillance object. More specifically, the safety control unit 3016 can overlap and display a frame surrounding the detected surveillance object on the surrounding image of the output device 50 displayed inside the cockpit 10, or overlap and display a mark at a position on the surrounding image corresponding to the actual existence position of the detected surveillance object. Thus, the output device 50 can realize the visual notification function for the operator. Furthermore, the safety control unit 3016 can also use a warning light or lighting device inside the cockpit 10 to notify the operator inside the cockpit 10 of the detection of the surveillance object.

Furthermore, the safety control unit 3016 may also start the external notification function based on the visual method by, for example, controlling the output device 50 (for example, a lighting device such as a headlight or a display device) provided on the side of the storage part of the upper rotating body 3. Furthermore, the safety control unit 3016 may also start the external notification function by, for example, sending a command signal indicating the start of the notification function to a terminal device (mobile terminal) carried by a person around the excavator 100, such as a worker, supervisor, or manager at the construction site. The terminal device carried by the worker, supervisor, or manager at the construction site may be, for example, a general mobile terminal such as a smart phone or a tablet terminal. Furthermore, the terminal device carried by the worker, supervisor, or manager at the construction site may be a wearable terminal. The wearable terminal may be, for example, smart glasses. Furthermore, the safety control unit 3016 may also start the internal notification function by a tactile method by, for example, controlling a vibration generating device that vibrates the driver's seat where the operator is seated. Thus, the controller 30 can make the operator or the workers and supervisors around the excavator 100 aware that there is a monitored object (e.g., a worker) at a relatively close position around the excavator 100. Therefore, the controller 30 can urge the operator to confirm the safety status around the excavator 100, or urge the workers in the monitored area to evacuate from the monitored area.

Furthermore, the safety control unit 3016 may also transmit a command signal indicating the activation of the notification function to the remote operation support device 300 through the communication device 60, thereby activating the remote notification function. At this time, the remote operation support device 300 may output an alarm based on a visual method or an auditory method when receiving the command signal from the shovel 100. Thus, the operator who remotely operates the shovel 100 can grasp that the monitored object has entered the notification range around the shovel 100 based on an alarm based on the remote notification function through the remote operation support device 300 in addition to the alarm based on the external notification function.

In addition, the remote notification function of the safety control unit 3016 may also be transferred to the remote operation support device 300. At this time, the remote operation support device 300 receives information related to the detection status of the monitored object based on the object detection unit 3011 or the specific result of the actual existence position of the monitored object based on the position estimation unit 3012 from the shovel 100. Then, the remote operation support device 300 determines whether the monitored object enters the notification range based on the received information, and when the monitored object exists in the notification range, the remote notification function is activated.

Furthermore, the safety control unit 3016 may make the notification form (ie, notification method) different according to the positional relationship between the monitored object detected within the notification range and the upper rotating body 3 .

For example, when the monitored object detected by the object detection unit 3011 in the notification range exists at a relatively far position relative to the specified part of the shovel 100, the safety control unit 3016 may output an alarm with a relatively low urgency (hereinafter referred to as an "alarm of the attention level") to urge attention to the monitored object. Hereinafter, for convenience, the range of the notification range that is relatively far from the specified part of the shovel 100, that is, the range corresponding to the alarm of the attention level, is sometimes referred to as the "attention notification range". On the other hand, when the monitored object detected by the object detection unit 3011 in the notification range exists at a relatively close position relative to the specified part of the shovel 100, the safety control unit 3016 may output an alarm with a relatively high urgency (hereinafter referred to as an "alarm of the warning level") to notify that the monitored object is approaching the specified part of the shovel 100 and the danger level is gradually increasing. Hereinafter, the range of the notification range that is relatively close to the specified part of the shovel 100, that is, the range corresponding to the alarm of the warning level, is sometimes referred to as the "alarm notification range".

At this time, the safety control unit 3016 can make the pitch, sound pressure, timbre, sound emission cycle, etc. of the sound output from the output device 50 (sound output device) different between the warning level alarm and the alert level alarm. In addition, the safety control unit 3016 can make the color, shape, size, presence or absence of flashing, flashing cycle, etc. of the image indicating that the monitoring object displayed on the output device 50 (display device) is detected, and the image (for example, a frame or mark, etc.) emphasizing the monitoring object or the position of the monitoring object displayed on the peripheral image of the output device 50 different between the warning level alarm and the alert level alarm. In this way, the controller 30 can make the operator etc. understand the urgency, in other words, the proximity of the monitoring object to the predetermined part of the shovel 100, based on the difference in the notification sound (alarm sound) output from the output device 50 or the notification image displayed on the output device 50.

After the notification function is started, if the monitored object detected by the object detection unit 3011 is no longer detected within the notification range, the security control unit 3016 can stop the notification function. Also, after the notification function is started, if a predetermined input for canceling the activation of the notification function is received through the input device 52, the security control unit 3016 can stop the notification function.

Furthermore, for example, when the object detection unit 3011 detects that a monitored object is detected within a specified range (hereinafter referred to as the "action restriction range") around the excavator 100, the safety control unit 3016 starts the action restriction function. The action restriction range is set to be the same as the above-mentioned notification range, for example. Furthermore, the action restriction range can be set to be a range whose outer edge is closer to a specified part of the excavator 100 than the notification range, for example. Thus, for example, if the monitored object enters the notification range from the outside, the safety control unit 3016 first starts the notification function, and then, if the monitored object enters the inner action restriction range, the action restriction function can also be started. Therefore, the controller 30 can start the notification function and the action restriction function in stages in response to the movement of the monitored object to the inside within the monitoring area.

Specifically, if the monitored object is detected within the action restriction range within the threshold value Dth2 (≤Dth1) at a distance D from the predetermined part of the shovel 100, the safety control unit 3016 can activate the action restriction function. The threshold value Dth2 can be constant regardless of the direction observed from the predetermined part of the shovel 100, or it can change according to the direction observed from the predetermined part of the shovel 100.

Furthermore, the motion restriction range includes at least one of a motion deceleration range in which the motion speed of the shovel 100 relative to the operation of the operating device 26 or the remote operation is made slower than usual, and a motion stop range in which the motion of the shovel 100 is stopped and maintained in a stopped state regardless of the presence or absence of the operation of the operating device 26 or the remote operation. For example, when the motion restriction range includes both the motion deceleration range and the motion stop range, the motion stop range is a range in the motion restriction range close to a predetermined portion of the shovel 100. Furthermore, the motion deceleration range is a range set outside the motion stop range in the motion restriction range.

The safety control unit 3016 starts the action limiting function that limits the action of the excavator 100 by controlling the hydraulic control valve 31. At this time, the safety control unit 3016 can limit the action of all driven elements (i.e., the corresponding hydraulic actuators), or can limit the action of a part of the driven elements (hydraulic actuators). Thus, when there is a monitored object around the excavator 100, the controller 30 can slow down or stop the action of the excavator 100. Therefore, the controller 30 can suppress the contact between the monitored object around the excavator 100 and the excavator 100 or the hanging object. In addition, the safety control unit 3016 can also start the action limiting function (action stopping function) by controlling the most upstream electromagnetic switching valve 25V (refer to Figure 16) of the pilot line 25 and cutting off the pilot line 25.

Furthermore, when the monitoring object detected by the object detection unit 3011 is no longer detected within the action restriction range after the action restriction function is started, the safety control unit 3016 may stop the action restriction function. Furthermore, when a predetermined input for cancelling the activation of the action restriction function is received through the input device 52 after the action restriction function is started, the safety control unit 3016 may stop the action restriction function. The content of the input for cancelling the activation of the notification function to the input device 52 may be the same as or different from the content of the input for cancelling the action restriction function.

Furthermore, the security control unit 3016 may make the control related to activation of the security function different between a person authenticated by the authentication unit 3014 and other objects among the objects detected by the object detection unit 3011 .

For example, in the case of a person who has been authenticated by the authentication unit 3014, unlike other objects, the safety control unit 3016 prohibits (cancels) activation of the safety function. Thus, for example, a remote operator who has been authenticated by the authentication unit 3014 can approach the shovel 100 and check the condition of the shovel 100 or the condition of the ground on which the shovel 100 is working, without considering activation of the notification function or the action restriction function.

If the remote operation is suspended, the shovel 100 will basically not start. Therefore, even if the remote operator approaches the shovel 100 to a certain extent, the safety of the shovel 100 will be limited.

Furthermore, the security control unit 3016 may prohibit (cancel) only a part of the security functions.

For example, the safety control unit 3016 prohibits (cancels) the notification function, while continuing the action restriction function. Thus, the action restriction function can slow down the action of the shovel 100 based on remote operation, prohibit remote operation, and stop the action of the shovel 100, thereby further improving the safety of the shovel 100.

Furthermore, the safety control unit 3016 may only prohibit (cancel) a part of the motion restriction function and continue the other functions. Specifically, the safety control unit 3016 may only prohibit the motion restriction function of a part of the motion restriction functions of the plurality of driven elements and continue the motion restriction function of the other driven elements. For example, the safety control unit 3016 determines whether to prohibit or continue the motion restriction function of each driven element according to the position (direction) of the person authenticated by the authentication unit 3014 relative to the excavator 100. Specifically, in the case of a person authenticated by the authentication unit 3014, the safety control unit 3016 may continue the motion restriction function of the upper swing body 3 and determine whether to prohibit the motion restriction function of the lower traveling body 1 or the attachment AT (boom 4, arm 5 and bucket 6) according to the position of the person. This is because if the upper swing body 3 rotates, a person in any direction may approach from the excavator 100, while the motion direction of the lower traveling body 1 or the attachment AT is determined, and even if the person is made to move according to the position of the person, the reduction in safety is limited. Thus, for example, for an authenticated person approaching from the side of the upper swing body 3 , the movement restriction function of the upper swing body 3 is continued to maintain the safety of the shovel 100 , and the movement restriction of the attachment AT is prohibited, thereby suppressing a decrease in the working efficiency of the shovel 100 .

Furthermore, the security control unit 3016 may also prohibit (cancel) only a part of the notification function and continue the rest of the functions. Specifically, the security control unit 3016 may also prohibit the notification function for the authenticated person and continue the notification function for other people. For example, the notification function of the mobile terminal carried by the authenticated person (e.g., a worker, supervisor, manager, etc. at the construction site) is prohibited (canceled), and the alarm sound is muted, while the notification function of the mobile terminal carried by other people continues.

Furthermore, in the case of a person who has been authenticated by the authentication unit 3014, the safety control unit 3016 may also suspend (limit) the activation of the safety function of the excavator 100 compared to the case of other objects. For example, when the auditory notification function is activated for a person who has been authenticated by the authentication unit 3014, the safety control unit 3016 suspends the activation of the safety function of the excavator 100 in the direction of lowering the level of the attention reminder of the output alarm sound. For example, the level of the attention reminder can be lowered by lowering the pitch or the sound pressure. Furthermore, for example, when the action restriction function is activated for a person who has been authenticated by the authentication unit 3014, the safety control unit 3016 suspends the degree of restriction on the action of the excavator 100. For example, generally, when the action stop function is activated, the degree of restriction on the action of the excavator 100 can be suspended by activating the action deceleration function or reducing the deceleration degree of the action deceleration function. Thus, for example, even when the remote operator authenticated by the authentication unit 3014 needs to closely check the condition of the shovel 100 or the condition of the ground surface on which the shovel 100 is working, a decrease in the working efficiency of the shovel 100 can be suppressed.

Furthermore, similar to the case of prohibiting (releasing) the safety function, the safety control unit 3016 may temporarily suspend activation of only a part of the safety function.

Furthermore, in the case of a person who has been authenticated by the authentication unit 3014, the safety control unit 3016 may also suspend the start condition of the shovel 100 compared to the case of other objects. Specifically, in the case of a person who has been authenticated by the authentication unit 3014, the safety control unit 3016 may suspend the start condition of the shovel 100 in a direction closer to the outer edge of the notification range or the action restriction range than in the case of other objects. Thus, for example, even if a person who has been authenticated by the authentication unit 3014 approaches the shovel 100 to a certain extent, it is possible to make it difficult for the safety function to be activated. Therefore, for example, even if a remote operator who has been authenticated by the authentication unit 3014 needs to confirm the condition of the shovel 100 or the condition of the ground of the working object of the shovel 100 at a close distance, it is possible to suppress the reduction in the working efficiency of the shovel 100.

Furthermore, similar to the case of prohibiting (releasing) the safety function, the safety control unit 3016 may temporarily suspend only a part of the activation conditions of the safety function.

Furthermore, when there is a person who has been authenticated by the authentication unit 3014, the safety control unit 3016 may also notify the information related to the authenticated person through the output device 50. Thus, the controller 30 can notify the operator of the cockpit 10 of the existence of the authenticated person through the output device 50. Therefore, for example, the safety control unit 3016 displays the existence of the person who has been authenticated by the authentication unit 3014 or the position information of the person on the output device 50 (display device). Furthermore, the safety control unit 3016 may also display information related to the release status or suspension status of the safety function for the authenticated person on the output device 50.

Similarly, when there is a person who has been authenticated by the authentication unit 3014, the security control unit 3016 may also notify the remote operator of the information related to the authenticated person through the display device 208 or the sound output device of the remote operation support device 300. Furthermore, when the remote operator himself has been authenticated, the remote operator may be notified so that the information related to the remote operator and the information related to other authenticated persons can be distinguished.

Similarly, the safety control unit 3016 may also notify the construction site staff, supervisors, managers, etc. of the information related to the authenticated person through the display device or sound output device of the terminal device carried by the construction site staff, supervisors, managers, etc. In addition, in the case where the person carrying the terminal device is authenticated, the person carrying the terminal device (the construction site staff, supervisors, managers, etc.) may also be notified so as to distinguish the information related to the carrier of the authenticated terminal device and other authenticated persons.

Furthermore, the safety control unit 3016 may switch its function between “ON” (valid) and “OFF” (invalid) according to a predetermined input by an operator or the like to the input device 52 or the input device 307 .

[Authentication process for people around excavator]

Next, the authentication process of the people around the shovel 100, especially the remote operator, will be described.

FIG. 13 is a flowchart schematically showing an example of an authentication process of a person (remote operator) around the shovel 100 .

The flowchart of Fig. 13 is repeatedly executed at a predetermined processing interval during the operation of the shovel 100 from the start to the stop of the shovel 100. Furthermore, the flowchart of Fig. 13 may be repeatedly executed at a predetermined processing interval during the operation of the shovel 100 when the function of the safety control unit 3016 is "ON" (valid).

As shown in FIG. 13 , in step S102 , the gesture recognition unit 3013 performs a process of recognizing a predetermined gesture of a person around the shovel 100 based on the output (captured image) of the imaging device 40 .

When the process of step S102 is completed, the controller 30 proceeds to step S104 .

In step S104 , the authentication unit 3014 determines whether the gesture indicating the authentication request (an example of the first gesture) has been recognized by the gesture recognition unit 3013 .

The gesture indicating the authentication request is, for example, predetermined and pre-registered in the auxiliary storage device 30A of the controller 30. Furthermore, the gesture indicating the authentication request may include an action of approaching the shovel 100. Thus, even if a worker or the like working around the shovel 100 accidentally performs an action similar to the gesture indicating the authentication request, the possibility of the action being recognized as the gesture indicating the authentication request can be reduced. For example, the gesture recognition unit 3013 can determine whether the target object has performed an action of approaching the shovel 100 based on the history of the actual existing position estimated by the position estimation unit 3012.

When the gesture recognition unit 3013 recognizes a gesture indicating an authentication request, the authentication unit 3014 proceeds to step S106 . Otherwise, the process of this flowchart is terminated.

In step S106 , the authentication unit 3014 transmits a request signal for requesting execution of the authentication gesture (an example of the second gesture) to a predetermined terminal device around the shovel 100 via the communication device 60 .

The prescribed terminal device is a terminal device that can communicate with the excavator 100 through a prescribed communication line. The prescribed terminal device includes, for example, the remote operation support device 300 or a terminal device carried by a worker, supervisor, manager, etc. at the construction site of the excavator 100. For example, the controller 30 uses the communication device 60 to pre-pair with the prescribed terminal based on a communication standard for short-distance communication such as WiFi or Bluetooth (registered trademark), so that a request signal can be sent to the prescribed terminal device.

For example, if the specified terminal device receives the request signal, it notifies the user of the authentication gesture by visual means such as screen display or auditory means such as sound output. Thus, for example, the remote operator can recognize the first stage gesture (gesture indicating the authentication request) on the excavator 100 side and perform the second stage gesture (authentication gesture) according to the notification.

When the process of step S106 is completed, the controller 30 proceeds to step S108 .

In this case, the controller 30 may notify the user that the authentication request gesture has been recognized or urge the user to perform the authentication gesture through the output device 50 (eg, a display device or a lighting device) provided on the side surface of the upper swing body 3 .

In step S108 , the gesture recognition unit 3013 performs a process of recognizing a predetermined gesture of a person around the shovel 100 based on the output (captured image) of the imaging device 40 .

When the process of step S108 is completed, the controller 30 proceeds to step S110 .

In step S110 , the authentication unit 3014 determines whether the gesture indicating the authentication request and the authentication gesture are recognized by the gesture recognition unit 3013 as being performed by the same person.

The authentication gesture is, for example, an arbitrary gesture that is different from the motion (gesture) performed by a person around the shovel 100. Furthermore, the authentication gesture may be predetermined and registered in advance in the auxiliary storage device 30A of the controller 30 or the like.

For example, the gesture recognition unit 3013 can determine whether the recognized gestures are performed by the same person based on the history of the detection results of the object detection unit 3011 and the inference results of the position inference unit 3012 and the position on the camera image where the authentication gesture is recognized.

When the gesture recognition unit 3013 does not recognize that the gesture indicating the authentication request and the authentication gesture are performed by the same person, the authentication unit 3014 proceeds to step S112 . When they are recognized, the authentication unit 3014 proceeds to step S114 .

In step S112, the authentication unit 3014 determines whether a time equivalent to the timeout has passed since the start point set after the moment when the gesture of the authentication request is recognized by the gesture recognition unit 3013. The start point may be, for example, when the process of step S104 is completed, or when the process of step S106 is completed. If the time equivalent to the timeout has not passed, the authentication unit 3014 returns to step S108, and if the time equivalent to the timeout has passed, the authentication unit 3014 ends the process of this flowchart.

Meanwhile, in step S114, the authentication unit 3014 completes the authentication of the person who has performed the authentication gesture recognized by the gesture recognition unit 3013, and outputs location information indicating the actual location of the authenticated person. Thus, the tracking unit 3015 can track the actual location of the authenticated person.

When the process of step S114 is completed, the controller 30 ends the process of this flowchart.

In this way, the controller 30 can authenticate the specific person by recognizing the two-stage gesture performed by the specific person around the shovel 100 .

Thus, for example, a specific person can be authenticated by a process with a lower load than when a specific person is recognized and authenticated by an image, etc. Furthermore, for example, compared with a case where a specific person is authenticated by only a single-stage gesture, a situation where a person is authenticated by an accidental gesture and the safety of the shovel 100 is reduced can be suppressed.

[Overview of the safety system of the excavator]

Next, the outline of the safety system SYS2 of the shovel 100 will be described with reference to FIG. 14 .

Hereinafter, the shovel 100 will be described mainly around the parts that are different from the operation support system SYS or the remote operation system SYS1 described above, and the description of the same or corresponding contents as those in the remote operation system SYS1 may be omitted.

FIG. 14 is a diagram showing an example of the safety system SYS2 of the shovel 100 .

The safety system SYS2 includes the shovel 100 and the terminal device 400 .

The safety system SYS2 ensures the safety of the excavator 100. A person carrying a terminal device 400 capable of communicating with the excavator 100 is authenticated, and the person is allowed to approach the excavator 100 or perform operations related to the excavator 100 in the authenticated state. For example, approaching the excavator 100 includes approaching the cab 10 of the excavator 100 or approaching the inside of the upper rotating body 3. For example, approaching the cab 10 includes unlocking or opening the door of the cab 10. Approaching the inside of the upper rotating body 3 includes unlocking or opening the door provided on the side or upper surface of the upper rotating body 3 in order to approach the engine room, cooling room, pump room, etc. In addition, approaching the excavator 100 includes, for example, electrical contact with the controller 30 of the excavator 100 using a terminal specified by the terminal device 400. Operations related to the shovel 100 include, for example, operations for starting the shovel 100, such as starting operations of the prime mover (engine 11), operations of driven elements (actuators) of the shovel 100 (operations of the operating device 26), etc. In addition, the safety system SYS2 may distinguish between authenticated persons and other persons and allow authenticated persons to approach the shovel 100 or operate the shovel 100. On the other hand, the safety system SYS2 operates in a manner that suppresses the approach to the shovel 100 or the operation of the shovel 100 by persons who are not authenticated. In addition, the safety system SYS2 may operate in a manner that suppresses the approach to the shovel 100 or the operation of the shovel 100 by persons who are not authenticated.

The shovel 100 is a construction machine whose safety is ensured by the safety system SYS2 .

The shovel 100 operates driven elements such as the lower traveling body 1 (ie, a pair of left and right crawler tracks 1C), the upper swing body 3, the boom 4, the arm 5, and the bucket 6 according to operations of an operator riding in the cab 10.

Furthermore, similarly to the case of the remote operation system SYS1 described above, the shovel 100 may be configured to be remotely operated from outside the shovel 100 instead of or in addition to being operable by an operator riding in the cab 10 .

As in the case of the remote operation system SYS1 described above, the remote operation includes, for example, a method of operating the shovel 100 by inputting operation related to the actuator of the shovel 100 by the remote operation support device 300 (see FIG. 10 ). However, unlike the case of the remote operation system SYS1 described above, the remote operation support device 300 may be provided at a location separated from the shovel 100 to a certain extent, such as a management center that manages the operation of the shovel 100 from outside the construction site.

Specifically, the shovel 100 can transmit an image (peripheral image) showing the state of the surroundings including the front of the shovel 100 based on the camera image output by the camera device 40 to the remote operation support device 300, for example, through the communication device 60. In addition, the remote operation support device 300 can display the image (peripheral image) received from the shovel 100 on the display device 208. In addition, similarly, various information images (information screens) displayed on the output device 50 (display device) inside the cab 10 of the shovel 100 can also be displayed on the display device 208 of the remote operation support device 300. As a result, the operator using the remote operation support device 300 can confirm the display content such as the image showing the state of the surroundings of the shovel 100 displayed on the display device 208. Therefore, the operator using the remote operation support device 300 can remotely operate the shovel 100 from a position where the operation of the shovel 100 cannot be directly visually recognized.

Furthermore, the excavator 100 can also automatically operate the actuator without depending on the operator's operation content. Thus, the excavator 100 realizes the function of automatically operating at least a part of the driven elements such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5 and the bucket 6, which is the "autopilot function" or "MC (Machine Control) function".

The automatic driving function includes, for example, a function of automatically operating a driven element (actuator) other than the driven element (actuator) of the operation target according to the operation of the operation device 26 by the operator or the remote operation ("semi-automatic driving function" or "operation support type MC function"). In addition, the automatic driving function may include a function of automatically operating at least a part of the plurality of driven elements (actuators) without the operation of the operation device 26 by the operator or the remote operation ("full automatic driving function" or "full automatic type MC function"). In the shovel 100, when the full automatic driving function is effective, the interior of the cockpit 10 may be unmanned. In addition, the semi-automatic driving function or the full automatic driving function may include a mode in which the operation content of the driven element (actuator) of the automatic driving target is automatically determined according to a predetermined rule. In addition, the semi-automatic driving function or the full automatic driving function may also include a mode in which the shovel 100 autonomously performs various judgments and autonomously determines the operation content of the driven element (actuator) of the automatic driving target according to the judgment result (so-called "autonomous driving function").

Furthermore, when the shovel 100 has an automatic driving function, the operation of the shovel 100 can also be remotely monitored. In this case, a remote monitoring support device having the same function as the remote operation support device 300 can also be provided. Thus, the user of the remote monitoring support device, i.e., the monitor, can monitor the status of the operation of the shovel 100 based on the automatic driving function while confirming the surrounding image displayed on the display device of the remote monitoring support device. Furthermore, for example, when the monitor determines that it is necessary from the perspective of safety, the monitor can intervene in the automatic driving function of the shovel 100 by making a predetermined input using the input device of the remote monitoring support device to make an emergency stop.

The terminal device 400 is a mobile terminal which is a portable terminal device carried by the user of the shovel 100 .

The terminal device 400 may be a dedicated mobile terminal for receiving the above authentication, or may be a general-purpose mobile terminal such as a smartphone or a tablet terminal. In the latter case, a dedicated application program that can cooperate with the shovel 100 and operate may be pre-installed.

[Hardware structure of the safety system]

Next, the hardware configuration of the security system SYS2 will be described with reference to FIG. 15 in addition to FIG. 14 .

The hardware configuration of the terminal device 400 may be the same as that of the remote operation support device 300 ( FIG. 11 ). Therefore, illustration and description of the hardware configuration of the terminal device 400 are omitted.

FIG. 15 is a block diagram showing another example of the hardware configuration of the shovel 100 .

As shown in FIG15 , the control system of the shovel 100 includes a controller 30 . Also, the control system of the shovel 100 includes an operation pressure sensor 29 , a camera 40 , and a camera 70 .

The camera device 70 is provided inside the cockpit 10 , and acquires an image showing the internal state of the cockpit 10 .

For example, the camera device 70 is a single-lens camera, or a 3D camera.

In addition, at least one of the imaging device 40 and the imaging device 70 may be omitted.

[Functional structure related to safety functions of excavators]

Next, with reference to FIG. 16 , a functional configuration related to the safety function of the shovel 100 will be described.

FIG. 16 is a block diagram showing an example of a functional configuration of a safety function of the shovel 100 .

As shown in FIG. 16 , the controller 30 includes a gesture recognition unit 3013 , an authentication unit 3014 , and a security control unit 3017 .

The gesture recognition unit 3013 recognizes a predetermined gesture performed by a person around the shovel 100 based on the output data of the camera 40. Alternatively, the gesture recognition unit 3013 may recognize a predetermined gesture performed by a person inside the shovel 100 based on the camera 70 (an example of the first sensor).

As in the case of the perimeter monitoring function, the authentication unit 3014 authenticates a specific person around the shovel 100. Furthermore, the authentication unit 3014 may authenticate a person inside the cab 10 of the shovel 100. The specific person is a regular user of the shovel 100. The regular user of the shovel 100 is, for example, the owner of the shovel 100, the operator of the shovel 100, the maintenance personnel who maintain the shovel 100, and the like.

Specifically, as described above, the authentication unit 3014 authenticates the person who performs the predetermined gesture when the gesture recognition unit 3013 recognizes the predetermined gesture. More specifically, the authentication unit 3014 can work in conjunction with the gesture recognition unit 3013 and authenticate a specific person around the shovel 100 by performing the same authentication process as that of FIG. 13 described above. At this time, the “predetermined terminal device” output in the process of step S106 includes the terminal device 400.

Furthermore, the authentication unit 3014 authenticates the person who performs the predetermined gesture by recognizing the predetermined gesture performed by the person inside the cab 10 of the shovel 100 based on the output data (camera image) of the camera device 70. More specifically, the authentication unit 3014 can work in conjunction with the gesture recognition unit 3013 and authenticate a specific person inside the shovel 100 by performing the same authentication process as that of FIG. 13 described above.

Furthermore, the authentication unit 3014 may perform authentication based on other methods in addition to authentication based on a predetermined gesture. In this case, for convenience, authentication based on a predetermined gesture is sometimes referred to as "first authentication", and authentication based on other methods is sometimes referred to as "second authentication".

The safety control unit 3017 (an example of the second control unit) performs control related to ensuring the safety of the shovel 100. Specifically, if the shovel 100 is approached or an operation related to the shovel 100 is performed without being authenticated by the authentication unit 3014, the safety control unit 3017 activates a safety function.

The safety function includes, for example, a notification function of notifying, through the output device 50, that the safety of the vicinity of the shovel 100 is reduced, that is, the possibility of the shovel 100 being stolen, etc. The safety function may also include a notification function of notifying, through the communication device 60, a predetermined external device of the possibility of the safety of the vicinity of the shovel 100 being reduced, that is, the possibility of the shovel 100 being stolen, etc. The predetermined external device is, for example, a terminal device carried by a user of the shovel 100 or a server device of a management center that manages the shovel 100 from the outside.

Furthermore, for example, the safety function includes a function of disabling the start operation of the shovel 100 and making the shovel 100 unable to start regardless of the presence or absence of various operations (hereinafter referred to as a “start disabling function”).

Furthermore, for example, the safety function includes an operation stopping function of forcibly maintaining the electromagnetic switching valve 25V at the most upstream of the pilot line 25 in a shutoff state and maintaining the shovel 100 in an operation stopping state regardless of the presence or absence of various operations.

Furthermore, the safety control unit 3017 prohibits activation of the safety function and permits approach to the shovel 100 or operations related to the shovel 100 when the authentication is passed by the authentication unit 3014 .

Furthermore, when the first authentication and the second authentication are performed by the authentication unit 3014, the safety control unit 3017 may also temporarily suspend the activation of the safety function in a state where the first authentication is completed by the authentication unit 3014. Furthermore, the authentication unit 3014 prohibits the activation of the safety function in a state where the second authentication is further completed, and allows access to the shovel 100 or all operations related to the shovel 100. For example, the safety control unit 3017 prohibits the notification function in a state where the first authentication is completed by the authentication unit 3014, and temporarily suspends the activation of the safety function by activating the activation invalidation function or the operation stop function.

Furthermore, when the first authentication and the second authentication are performed by the authentication unit 3014, the safety control unit 3017 may also suspend the activation condition of the safety function in a state where the first authentication is completed by the authentication unit 3014. Then, in a state where the authentication unit 3014 further completes the second authentication, for example, the safety control unit 3017 prohibits the activation of the safety function and allows access to the shovel 100 or all operations related to the shovel 100. For example, in a state where the first authentication is completed by the authentication unit 3014, the safety control unit 3017 allows access to the cab 10 of the shovel 100, etc., and on the other hand, suspends the activation condition of the safety function in a manner that prohibits operations related to the shovel 100.

In this way, similarly to the case of the remote operation system SYS1 , the controller 30 can authenticate a specific person by recognizing a two-stage gesture performed by a specific person around the shovel 100 .

Thus, for example, a specific person can be authenticated by a process with a lower load than when a specific person is recognized and authenticated by an image, etc. Furthermore, for example, compared with a case where a specific person is authenticated by only recognizing a gesture at one stage, a situation where a person is authenticated by an accidental gesture and the safety of the shovel 100 is reduced can be suppressed.

[Another example of a remote operation system, security system]

Next, another example of the remote operation system SYS1 or the security system SYS2 will be described.

The above-mentioned embodiments related to the remote operation system SYS1 or the security system SYS2 may be combined appropriately, or modifications or changes may be added thereto.

For example, the same hardware as the camera device 40, 70 or the distance sensor of the shovel 100 in the above embodiment and the same function as the authentication unit 3014 of the shovel 100 can also be installed in various devices that need to authenticate the user. The various devices include electronic devices, industrial machinery, automobiles, etc. Electronic devices include information processing devices such as terminal devices and server devices.

[effect]

Next, the role of the construction machine (shovel 100 ) in the remote operation system SYS1 or the safety system SYS2 according to the present embodiment will be described.

In the present embodiment, the construction machine (e.g., excavator 100) includes a first sensor (e.g., camera 40 or distance sensor or camera 70), a recognition unit (e.g., gesture recognition unit 3013), and an authentication unit (e.g., authentication unit 3014). Specifically, the first sensor acquires information related to objects around the construction machine or inside a cab (e.g., cab 10). The recognition unit recognizes gestures of a person around the construction machine or inside the cab based on the output of the first sensor. Furthermore, when a first gesture of a person around the construction machine or inside the cab 10 is recognized by the recognition unit, and a second gesture of the same person is subsequently recognized by the recognition unit, the authentication unit authenticates the person.

Conventionally, there is known a technology that can operate various devices through gestures from an outside person (for example, refer to International Publication No. 2020/032267).

The above document discloses a shovel that recognizes surrounding workers and can be operated by a predetermined gesture from the worker. This can improve user convenience.

However, when the construction machine can be operated by a predetermined gesture, if multiple people are recognized by the device, it is possible that the predetermined gesture of a person different from the specific operator is recognized by the construction machine and the device is started regardless of the person's intention. In addition, for example, a malicious third party who knows that the operation can be performed by a predetermined gesture may abuse the gesture-based operation function.

In contrast, in the present embodiment, the construction machine (e.g., the controller 30) can perform gesture-based human authentication in two stages. Therefore, for example, even if the first gesture is accidentally recognized, since the second gesture is also required, it is possible to suppress a situation that would cause the operation to be started regardless of the intention of the person performing the first gesture. Furthermore, for example, even if information of either the first gesture or the second gesture is leaked, since the other gesture is required, it is possible to suppress a situation such as abuse of the gesture-based operating function. Therefore, the security of the construction machine can be ensured more accurately.

Furthermore, in the present embodiment, the second gesture may be a gesture different from the gesture performed by other persons present around the construction machine or inside the cab 10 .

This can prevent a situation where multiple users are authenticated at the same time. Furthermore, for example, a user of a construction machine who wishes to be authenticated only needs to memorize a predetermined first gesture, thereby improving user convenience.

Furthermore, in the present embodiment, the first gesture may include an action of a person around the construction machine approaching the construction machine.

Thus, for example, even in a situation where there are a plurality of workers working around a construction machine, it is possible to reliably recognize the first gesture of a person who intends to receive authentication.

Furthermore, in the present embodiment, the construction machine may also include a second sensor (e.g., a camera 40 or a distance sensor), a detection unit (e.g., an object detection unit 3011), and a first control unit (e.g., a safety control unit 3016). Specifically, the second sensor may acquire information related to objects around the construction machine. When the first sensor acquires information related to objects around the construction machine, the second sensor may be the same as or different from the first sensor. Furthermore, the detection unit may detect a person around the construction machine based on the output of the second sensor. Furthermore, when a person is detected within a specified range around the construction machine by the detection unit, the first control unit may activate a specified safety function. Furthermore, the first control unit may prohibit the activation of a specified safety function when a person who has been authenticated by the authentication unit is detected by the detection unit. Furthermore, the first control unit may also suspend the activation of a specified safety function when a person who has been authenticated by the authentication unit is detected by the detection unit. Furthermore, the first control unit may also suspend the activation of a specified safety function when a person who has been authenticated by the authentication unit is detected by the detection unit. Furthermore, the first control unit may also suspend the activation of a specified safety function when a person who has been authenticated by the authentication unit is detected by the detection unit.

Thus, for example, an operator or supervisor of a remote operation around a construction machine can suppress a decrease in work efficiency caused by activation of a safety function of the construction machine by receiving gesture-based authentication, and can confirm the working status near the construction machine. Therefore, the convenience of an operator or supervisor of a remote operation around a construction machine can be improved.

Furthermore, in the present embodiment, the prescribed safety function may also include a notification function for notifying that a person is detected within a prescribed range of the construction machine. Furthermore, the first control unit may also prohibit a portion of the notification function, suspend the start, or suspend the start condition when the detection unit detects a person authenticated by the authentication unit.

This makes it possible to more accurately achieve a balance between the convenience of an operator or supervisor who remotely operates the periphery of the construction machine and the safety of the construction machine.

Furthermore, in this embodiment, the notification function may include a first notification function for notifying at least one of the operator of the construction machine and the surroundings of the construction machine, and a second notification function for notifying a person detected by the detection unit. Furthermore, the first control unit may prohibit only the second notification function, the deferred start, or the deferred start condition of the first notification function and the second notification function when the detection unit detects a person authenticated by the authentication unit.

Thus, by prohibiting the second notification function, etc., the convenience of authenticated persons such as remote operators or supervisors operating around the construction machine is improved, and by continuing the first notification function, the safety of the construction machine can be ensured more accurately.

Furthermore, in this embodiment, the prescribed safety function may also include an action restriction function for restricting the action of the construction machine. Furthermore, the first control unit may also prohibit a part of the action restriction function, suspend the start, or suspend the start condition when the detection unit detects a person authenticated by the authentication unit.

This makes it possible to more accurately achieve a balance between the working efficiency of the construction machine and the safety of the construction machine.

Furthermore, in the present embodiment, the construction machine may also include a lower traveling body (e.g., lower traveling body 1), an upper rotating body (e.g., upper rotating body 3) that is rotatably mounted on the lower traveling body, and an attachment (e.g., attachment AT) mounted on the upper rotating body. Furthermore, the motion limiting function may also include a first motion limiting function that limits the motion of the upper rotating body and a second motion limiting function that limits the motion of the attachment. Furthermore, the first control unit may only prohibit the second motion limiting function, the suspended start, or the suspended start condition of the first motion limiting function and the second motion limiting function when the detection unit detects a person who has been authenticated by the authentication unit.

Thus, by prohibiting the second motion limiting function, etc., the operation of the construction machine is continued, thereby suppressing the reduction in the working efficiency of the construction machine, and by continuing the first motion limiting function, it is possible to more accurately ensure the safety of the construction machine.

Furthermore, in this embodiment, the construction machine may also be provided with a communication device (e.g., the communication device 60). Specifically, the communication device may also be configured to be able to communicate with a terminal device (e.g., the remote operation support device 300 or the terminal device 400) carried by a user of the construction machine. Furthermore, when the first gesture is recognized, the recognition unit may also send a signal requesting the second gesture to the terminal device via the communication device.

This allows the user of the construction machine to more accurately grasp the execution timing of the second gesture.

Furthermore, in the present embodiment, the operation of the construction machine may be remotely controlled based on a signal from a terminal device (for example, the remote operation support device 300 ).

Thus, in the vicinity of the construction machine, an operator who uses a terminal device to remotely operate the construction machine can suppress the reduction in work efficiency caused by the activation of the safety function of the construction machine, and can get close to the construction machine. Therefore, the operating status of the construction machine or the status of the construction machine can be confirmed more accurately. Therefore, the remote operator can remotely operate the construction machine more accurately.

Furthermore, in the present embodiment, the construction machine may include a second control unit (e.g., safety control unit 3017). Specifically, the second control unit may also activate a prescribed safety function for approaching or operating the construction machine. Furthermore, the second control unit may prohibit the activation of a prescribed safety function for approaching or operating the construction machine by a person who has been authenticated by the authentication unit. Furthermore, the second control unit may also suspend the activation conditions of a prescribed safety function for approaching or operating the construction machine by a person who has been authenticated by the authentication unit.

Thus, for example, the user of the construction machine can release or suspend the safety function through the mobile terminal based on gestures. Therefore, the safety of the construction machine can be ensured and the convenience of the user of the construction machine can be improved.

As mentioned above, although embodiment is described in detail, this invention is not limited to this specific embodiment, Various deformation|transformation and change are possible within the range of the summary described in a claim.

For example, the shovel 100 may include all components of the operation support system SYS, the remote operation system SYS1 , and the safety system SYS2 , any two components thereof, or any one component thereof.

This application claims priority based on Japanese Patent Application No. 2022-057576 filed on March 31, 2022 and Japanese Patent Application No. 2022-036078 filed on March 9, 2022, the entire contents of these Japanese patent applications are incorporated herein by reference.

Explanation of symbols

1-lower walking body, 1C, 1CL, 1CR-crawler, 2-slewing mechanism, 3-upper slewing body, 4-boom, 5-arm, 6-bucket, 10-cabin, 26-operating device, 29-operating pressure sensor, 30-controller, 31-hydraulic control valve, 32-reciprocating valve, 33-hydraulic control valve, 40-camera, 40B, 40F, 40L, 40R-camera, 50-output device, 50A-display device, 50B-sound output device, 52-input device, 60-communication device, 70-camera, 100-excavator, 150-operation support device, 200-information processing device device, 300-remote operation support device, 3001-operation instruction receiving unit, 3002-operation instruction notification unit, 3003-operation content evaluation unit, 3004-storage unit, 3005-sending unit, 3011-object detection unit, 3012-position inference unit, 3013-gesture recognition unit, 3014-authentication unit, 3015-tracking unit, 3016-safety control unit, 3017-safety control unit, 400-terminal device, 500-crawler crane, AT-attachment, HK-hook, OP-operator, SYS-operation support system, SYS1-remote operation system, SYS2-safety system.

Claims (as amended under Article 19)

1. (After correction) A construction machine having:

The operation unit accepts the user's input on the operation of the construction machinery;

A receiving unit that receives instructions related to the operation of the construction machine from outside the construction machine; and

The notification unit notifies the user of instructions related to the operation of the construction machinery received by the receiving unit.

2. (After correction) The construction machine according to claim 1, wherein:

Instructions related to the operation of the construction machinery include a first instruction for specifying a driven element of the construction machinery and operating the driven element in a specific direction, a second instruction for specifying an action constituting a job and performing an operation to implement the action, a third instruction for performing an operation to adjust the speed of the construction machinery, and at least one of a fourth instruction related to multiple actions or steps of multiple jobs.

3. (After correction) The construction machine according to claim 1 or 2, wherein:

The receiving unit receives instructions related to the operation of the construction machine from an instructor around the construction machine.

4. (After correction) The construction machine according to claim 3, wherein:

The receiving unit recognizes a gesture indicating an instruction related to the operation of the construction machine from the indicator based on the sensor information indicating the state of the surroundings of the construction machine, and receives the instruction related to the operation of the construction machine.

5. (After correction) The construction machine according to claim 1 or 2, comprising:

A notification unit notifies a user when there is a difference between the content of the instructions related to the operation of the construction machine notified by the notification unit and the content of the subsequent actual operation input to the construction machine.

6. (After correction) The construction machine according to claim 1 or 2, comprising:

The recording unit records a log indicating the situation in a prescribed storage unit when there is a difference between the content of the instructions related to the operation of the construction machine notified by the notification unit and the content of the subsequent actual operation input to the construction machine.

7.(Deleted)

8.(Deleted)

9.(Deleted)

10. (After correction) The construction machine according to claim 1 or 2, comprising:

The first sensor acquires information related to objects around the construction machine or inside the cockpit;

A recognition unit that recognizes gestures of people around the construction machine or inside the cockpit based on the output of the first sensor; and

The authentication unit authenticates a person when the first gesture of a person around the construction machine is recognized by the recognition unit and the second gesture of the same person is subsequently recognized by the recognition unit.

11. The construction machine according to claim 10, wherein:

The second gesture is a gesture different from the actions performed by other people around the construction machine or inside the cockpit.

12. The construction machine according to claim 10, wherein:

The first gesture includes the action of people around the construction machine approaching the construction machine.

13. The construction machine according to claim 10, comprising:

The second sensor acquires information related to objects around the construction machinery;

A detection unit detects people around the construction machine based on the output of the second sensor; and

The first control unit starts a specified safety function when a person is detected within a specified range around the construction machine by the detection unit.

The first control unit prohibits the start of the specified security function when the detection unit detects a person who has been authenticated by the authentication unit, suspends the start of the specified security function when the detection unit detects a person who has been authenticated by the authentication unit, or suspends the start condition of the specified security function when the detection unit detects a person who has been authenticated by the authentication unit.

14. The construction machine according to claim 13, wherein:

The specified safety functions include a notification function that notifies when a person is detected within the specified range,

When the detection unit detects a person who has been authenticated by the authentication unit, the first control unit prohibits part of the notification function, suspends the start or suspends the start condition.

15. The construction machine according to claim 14, wherein:

The prescribed safety functions include a first notification function for notifying the operator of the construction machine and at least one of the surroundings of the construction machine and a second notification function for notifying the person detected by the detection unit.

When the detection unit detects a person who has been authenticated by the authentication unit, the first control unit only prohibits the first notification function and the second notification function, suspends the start, or suspends the start condition.

16. The construction machine according to claim 10, comprising:

Communication device, capable of communicating with the terminal device carried by the user of the construction machinery,

When the first gesture is recognized, the recognition unit sends a signal requesting the second gesture to the terminal device through the communication device.

17. The construction machine according to claim 16 is remotely controlled based on the signal from the terminal device.

18. The construction machine according to claim 10, comprising:

The second control unit is responsible for the safety functions required for approaching or operating construction machinery.

The second control unit prohibits the activation of the specified safety function when a person who has been authenticated by the authentication unit approaches or operates the construction machine, suspends the activation of the specified safety function when a person who has been authenticated by the authentication unit approaches or operates the construction machine, or suspends the activation conditions of the specified safety function when a person who has been authenticated by the authentication unit approaches or operates the construction machine.

19.(Deleted)

Claims (19)

1. An operation support device is provided with:

A receiving unit that receives an instruction relating to an operation of a construction machine from outside the construction machine; and

And a notification unit configured to notify a user of an instruction regarding the operation of the construction machine received by the receiving unit.

2. The operation support device according to claim 1, wherein,

The instructions related to the operation of the construction machine include at least one of a1 st instruction for designating a driven element of the construction machine and operating the driven element in a specific direction, a2 nd instruction for designating an operation constituting a job and performing the operation, a3 rd instruction for performing an operation of adjusting an operation speed of the construction machine, and a 4 th instruction related to a plurality of operations or steps of a plurality of jobs.

3. The operation support device according to claim 1 or 2, wherein,

The receiving unit receives an instruction concerning an operation of the construction machine from an instruction person around the construction machine.

4. The operation support device according to claim 3, wherein,

The receiving unit recognizes a gesture indicating an instruction concerning the operation of the construction machine from the indicator based on the sensor information indicating the state around the construction machine, and receives the instruction concerning the operation of the construction machine.

5. The operation support device according to claim 1 or 2, comprising:

And a notification unit configured to notify a user when there is a difference between the content of the instruction related to the operation of the construction machine notified by the notification unit and the content of the actual operation input to the construction machine.

6. The operation support device according to claim 1 or 2, comprising:

And a recording unit that, when there is a difference between the content of the instruction related to the operation of the construction machine notified by the notification unit and the content of the actual operation input to the construction machine thereafter, records a log indicating the difference in the content in a predetermined storage unit.

7. The operation support device according to claim 1 or 2, comprising:

and an operation unit that receives an operation input from a user to the construction machine.

8. A remote operation support device according to claim 7,

The operation unit receives an operation input for remote operation of the construction machine.

9. A construction machine provided with the operation support device according to claim 1 or 2.

10. The construction machine according to claim 9, comprising:

A1 st sensor for acquiring information related to objects around the construction machine or inside the cockpit;

An identification unit that identifies a gesture of a person around the construction machine or in the cockpit, based on the output of the 1 st sensor; and

And an authentication unit that authenticates a person around the construction machine when the recognition unit recognizes the 1 st gesture of the person and then the recognition unit recognizes the 2 nd gesture of the same person.

11. The construction machine according to claim 10, wherein,

The 2 nd gesture is a gesture different from a gesture performed by another person existing around the construction machine or inside the cockpit.

12. The construction machine according to claim 10, wherein,

The 1 st gesture includes an action of a person near the construction machine approaching the construction machine.

13. The construction machine according to claim 10, comprising:

a2 nd sensor for acquiring information related to objects around the construction machine;

a detection unit that detects a person around the construction machine from the output of the 2 nd sensor; and

A 1 st control part for starting a prescribed safety function when the detection part detects the person in a prescribed range around the construction machine,

The 1 st control unit prohibits the activation of the predetermined security function when the person who passes the authentication by the authentication unit is detected by the detection unit, suspends the activation of the predetermined security function when the person who passes the authentication by the authentication unit is detected by the detection unit, or suspends the activation condition of the predetermined security function when the person who passes the authentication by the authentication unit is detected by the detection unit.

14. The construction machine according to claim 13, wherein,

The prescribed security function includes a notification function that notifies that a person is detected within the prescribed range,

The 1 st control unit prohibits a part of the notification functions, the suspension start, or the suspension start condition when the detection unit detects the person authenticated by the authentication unit.

15. The construction machine according to claim 14, wherein,

The prescribed safety functions include a1 st notifying function for notifying at least one of an operator of the construction machine and the periphery of the construction machine, and a2 nd notifying function for notifying a person detected by the detecting section,

The 1 st control unit prohibits only the 2 nd notification function, the suspend start, or the suspend start condition among the 1 st notification function and the 2 nd notification function when the detection unit detects the person authenticated by the authentication unit.

16. The construction machine according to claim 10, comprising:

a communication device capable of communicating with a terminal device carried by a user of the construction machine,

When the 1 st gesture is recognized, the recognition part transmits a signal requesting the 2 nd gesture to a terminal device through the communication device.

17. The construction machine according to claim 16, which remotely operates an operation thereof according to a signal from the terminal device.

18. The construction machine according to claim 10, comprising:

A2 nd control unit for starting a predetermined safety function for approaching or operating the construction machine,

The 2 nd control unit prohibits the activation of the predetermined safety function for the approach or operation of the construction machine by the person authenticated by the authentication unit, suspends the activation of the predetermined safety function for the approach or operation of the construction machine by the person authenticated by the authentication unit, or suspends the activation condition of the predetermined safety function for the approach or operation of the construction machine by the person authenticated by the authentication unit.

19. A program that causes an information processing apparatus to execute the steps of:

A receiving step of receiving an instruction relating to an operation of a construction machine from outside the construction machine; and

And a notification step of notifying a user of the instruction regarding the operation of the construction machine received in the reception step.