JP7376440B2 - Remote control device and remote control system - Google Patents

Description

The present invention relates to a device for remotely controlling a work machine.

In a remote control system in which a working machine equipped with a working part is remotely controlled wirelessly by a controller installed at the operator's seat, a technology has been proposed to avoid a decrease in work efficiency due to image delay in monitor display. (For example, see Patent Document 1). Specifically, a line drawing of the working part is created at a position corresponding to the working part in the image taken by a camera installed on the working machine, and the line drawing is transmitted wirelessly and superimposed on the image taken by the camera to be displayed on a monitor. Is displayed. For this reason, a line drawing with a small time lag due to transmission is used, and the work unit is displayed at a position close to the actual position. Thereby, even if an image delay occurs in the display on the monitor, the operator can operate the work unit while supplementing the image delay on the monitor based on the line drawing displayed on the monitor.

Patent No. 5919022

However, since the means for creating line drawings is installed in the work machine, if a problem occurs in the transmission of line drawings, the line drawings superimposed on the camera footage may be temporarily lost, or the actual work part There is a possibility that the deviation between the position of the line drawing and the position of the line drawing becomes large.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a remote control device and a remote control system that can reduce or eliminate the discrepancy between the position of a marking image of a working part such as a line drawing in an image with respect to the position of a working part in real space. shall be.

The remote control device of the present invention includes:
A remote control device for remotely controlling a working machine having a working mechanism,
a remote wireless communication device having a communication function with an actual wireless communication device installed in the work machine;
a remote image output device;
a remote control mechanism;
a remote control element that recognizes an operation mode of the remote control mechanism by an operator and causes the remote wireless communication device to transmit a command signal according to the operation mode to the actual wireless communication device;
A working environment image including a working part of the working mechanism, which is acquired through an imaging device mounted on the working machine, transmitted from the actual wireless communication device, and received by the remote wireless communication device, is sent to the remote image output device. a first output control element to output;
estimating the space occupation mode of the working section at a second time point after the first time point based on the operation mode of the remote control mechanism at the first time point recognized by the remote control element; A sign image representing the space occupation mode of the working area at a second time point without impairing the visibility of the working area is added to the working environment image at a third time point included in the period from the first time point to the second time point. a second output control element superimposed on the image output device and outputting the image to the remote image output device.

The remote control system of the present invention includes the remote control device of the present invention and the working machine.

FIG. 1 is an explanatory diagram regarding the configuration of a remote control system as an embodiment of the present invention. FIG. 2 is an explanatory diagram regarding the configuration of a remote control device. An explanatory diagram regarding the configuration of a working machine. An explanatory diagram regarding the functions of the remote control system. FIG. 2 is an explanatory diagram regarding a first example of a method for estimating the position and orientation of a working part. FIG. 7 is an explanatory diagram regarding a second example of a method for estimating the position and orientation of a working part. An explanatory diagram regarding a work environment image. FIG. 4 is an explanatory diagram regarding a first output form of a marker image. FIG. 7 is an explanatory diagram regarding a second output form of a marker image.

(Remote control system configuration)
A remote control system as an embodiment of the present invention shown in FIG. 1 includes a remote control device 20 and a working machine 40 that is a target for remote control by the remote control device 20. The remote control device 20 and the work machine 40 are configured to be able to communicate with each other via network. The server 10 may intervene in mutual communication between the remote control device 20 and the working machine 40, and the server 10 may constitute a part of the remote control system.

(Configuration of remote control device)
The remote control device 20 includes a remote input interface 210, a remote output interface 220, and a remote control device 24. The remote input interface 210 includes a remote control mechanism 211. The remote output interface 220 includes an image output device 221 and a remote wireless communication device 222. The remote control device 24 includes a remote control element 240, a first output control element 241, and a second output control element 242. Each element is composed of an arithmetic processing unit (single-core processor or multi-core processor, or processor cores that constitute it), reads necessary data and software from a storage device such as memory, and processes the data according to the software. Execute calculation processing.

The remote control element 240 recognizes the operation mode of the remote control mechanism 211 by the operator, and causes the remote wireless communication device 222 to transmit a command signal corresponding to the operation mode to the actual wireless communication device 422. The first output control element 241 is a work attachment 44 (=work mechanism) that is acquired through an actual image pickup device 412 mounted on the work machine 40, transmitted from the actual wireless communication device 422, and received by the remote wireless communication device 222. The image output device 221 is caused to output the work environment image Img1 including the bucket 445 (=work part). The second output control element 242 controls the operation of the bucket 445 at a second time point t=t2 after the first time point based on the operation mode of the remote control mechanism 211 at the first time point t=t1 recognized by the remote control element 240. Estimate the space occupation mode. The second output control element 242 outputs a sign image Img2 (t2) that represents the estimated space occupation mode of the bucket 445 without impairing the visibility of the work section (bucket 445) at a third time point t=t3. t3) and output to the image output device 221.

The remote control mechanism 211 includes a travel operating device, a turning operating device, a boom operating device, an arm operating device, and a bucket operating device. Each operating device has an operating lever that is rotated. The operating lever (traveling lever) of the traveling operating device is operated to move the lower traveling body 41 of the working machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal may be provided that is fixed to the base or lower end of the travel lever. The operating lever (swing lever) of the swing operating device is operated to move a hydraulic swing motor that constitutes the swing mechanism 43 of the work machine 40 . The operating lever (boom lever) of the boom operating device is operated to move the boom cylinder 442 of the work machine 40. The operating lever (arm lever) of the arm operating device is operated to move the arm cylinder 444 of the work machine 40. The operating lever (bucket lever) of the bucket operating device is operated to move the bucket cylinder 446 of the work machine 40. The remote control mechanism 211 has the same configuration as the actual machine operating mechanism 411 installed in the working machine 40, but has a different configuration from the actual machine operating mechanism 411, such as a touch panel type or button type operating mechanism through which operation commands are input. It may be an operating mechanism.

For example, as shown in FIG. 2, the operating levers constituting the remote control mechanism 211 are arranged around a seat St on which an operator is seated. The seat St is in the form of a high-back chair with an armrest, but may be in any form in which the remote operator can sit, such as a low-back chair without a headrest, or a chair without a backrest. .

A pair of left and right travel levers 2110 corresponding to left and right crawlers are arranged side by side in front of the seat St. One operating lever may also serve as multiple operating levers. For example, when the right operating lever 2111 provided in front of the right frame of the seat St shown in FIG. 3 functions as a boom lever when operated in the front-back direction, and when operated in the left-right direction It may also function as a bucket lever. Similarly, the left operating lever 2112 provided at the front of the left frame of the seat St shown in FIG. In some cases, it may function as a pivot lever. The lever pattern may be arbitrarily changed according to an operator's operation instruction.

For example, as shown in FIG. 2, the image output devices 221 include a right diagonal front image output device 2211, a front image output device 2212, and a left diagonal front image output device 2211, a front image output device 2212, and a left diagonal front image output device disposed at the right diagonal front, the front, and the left diagonal front of the seat St, respectively, as shown in FIG. It is composed of a diagonal front image output device 2213. The image output devices 2211 to 2213 may further include a speaker (sound output device).

(Composition of working machine)
The work machine 40 includes a real machine control device 400, a real machine input interface 410, a real machine output interface 420, and a work attachment 44. The actual machine control device 400 is composed of an arithmetic processing device (a single-core processor or a multi-core processor, or a processor core constituting this processor), reads necessary data and software from a storage device such as a memory, and processes the data into the software. Execute the arithmetic processing accordingly.

The work machine 40 is, for example, a crawler excavator (construction machine), and as shown in FIG. The upper revolving body 42 is provided with an upper revolving body 42. The turning mechanism 43 corresponds to one of the working mechanisms. A cab (driver's cab) 424 is provided at the front left side of the upper revolving body 42. A work attachment 44 is provided at the front central portion of the upper revolving body 220.

The real machine input interface 410 includes a real machine operating mechanism 411, a real machine imaging device 412, and a state sensor group 414. The actual machine operating mechanism 411 includes a plurality of operating levers arranged around a seat arranged inside the cab 424 in the same way as the remote operating mechanism 211 . The cab 424 is provided with a drive mechanism or a robot that receives a signal corresponding to the operating mode of the remote control lever and moves the actual control lever based on the received signal. Note that the operating mode of the work machine 40 may be controlled by controlling the opening degree of a valve constituting the hydraulic circuit, etc., instead of moving the actual machine operating lever. The actual imaging device 412 is installed inside the cab 424, for example, and images the environment including at least a portion of the work attachment 44 through the front window of the cab 424. The status sensor group 414 measures the relative attitude angle of the upper rotating body 42 with respect to the lower traveling body 41, the relative attitude angle of the boom 441 with respect to the upper rotating body 42, the relative attitude angle of the arm 443 with respect to the boom 441, and the relative attitude angle of the bucket 445 with respect to the arm 443. It is composed of various sensors that output signals representing the operating mode or operating state of the working mechanism, such as the relative posture angle.

The real device output interface 420 includes a real device wireless communication device 422.

The work attachment 44 as one of the work mechanisms includes a boom 441 mounted on the upper revolving body 42 so as to be able to raise and lower, an arm 443 rotatably connected to the tip of the boom 441, and a boom 443 attached to the tip of the arm 443. A bucket 445 as a working unit is rotatably connected. The work attachment 44 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446, which are configured by extendable hydraulic cylinders.

The boom cylinder 442 is interposed between the boom 441 and the upper revolving body 42 so as to extend and contract when supplied with hydraulic oil and rotate the boom 441 in the up-and-down direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to expand and contract when supplied with hydraulic oil and rotate the arm 443 around a horizontal axis with respect to the boom 441 . The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to expand and contract when supplied with hydraulic oil and rotate the bucket 445 about a horizontal axis relative to the arm 443 .

(function)
The functions of the remote control system configured as described above will be explained using the flowchart shown in FIG. In the flowchart, the block "C●" is used to simplify the description, and means the transmission and/or reception of data, and the processing in the branch direction is executed on the condition that the data is transmitted and/or received. It means conditional branching.

In the remote control device 20, the presence or absence of a specified operation by the operator through the remote input interface 210 is determined (FIG. 4/STEP 200). The “designated operation” is, for example, a touch operation such as a tap, a swipe, a flick, or a pinch out/in on the touch panel that constitutes the remote input interface 210. If the determination result is negative (FIG. 4/STEP 200...NO), the series of processes ends. On the other hand, if the determination result is positive (FIG. 4/STEP 200...YES), the first output control element 241 transmits a work environment image request through the remote wireless communication device 222 (FIG. 4/STEP 202). The work environment image request includes at least one of the identifier of the remote control device 20 and the identifier of the operator.

In the work machine 40, when a work environment image request is received through the real machine wireless communication device 422 (FIG. 4/C41), the real machine control device 400 acquires a captured image through the real machine imaging device 412 (FIG. 4/STEP 402). The actual device control device 400 transmits captured image data representing the captured image to the remote control device 10 via the actual device wireless communication device 422 (FIG. 4/STEP 404). The captured image may be acquired through an imaging device installed at the work site or outside the work machine 40.

When the remote control device 20 receives captured image data through the remote wireless communication device 222 (FIG. 4/C20), a working environment image corresponding to the captured image data is output to the image output device 221 (FIG. 4/C20). STEP 204). The work environment image is a simulated work environment image generated based on all or part of the captured image itself. As a result, as shown in FIG. 7, for example, a work environment image Img1 in which a boom 441, an arm 443, and a bucket 445 that are part of the work attachment 44 as a work mechanism are reflected is displayed on the image output device 221. Displayed in series.

In the remote control device 20, the operation mode of the remote control mechanism 211 is recognized by the remote control element 240 (FIG. 4/STEP 206), and a remote control command signal corresponding to the operation mode is sent to the remote control via the remote wireless communication device 222. It is transmitted to the server 10 (FIG. 4/STEP 208).

In the work machine 40, when the actual machine control device 400 receives an operation command through the real machine wireless communication device 422 (FIG. 4/C42), the operation of the work attachment 44, etc. is controlled (FIG. 4/STEP 406). For example, the soil in front of the working machine 40 is scooped up using the bucket 445, the upper revolving body 410 is rotated, and then the soil is dropped from the bucket 445.

In the remote control device 20, based on the operation mode of the remote control mechanism 211 at the first time point recognized by the remote control element 240, the second output control element 242 controls the operation of the working part (bucket 445) of the work attachment 44 at the first time point. The space occupation mode at t=t1 or a second time point t=t2 after this is estimated (or predicted), and a marker image Img2 representing the space occupation mode of the working part is generated (FIG. 4/STEP 210). Specifically, based on the position or inclination angle of the operating lever constituting the operating mechanism 211, the relative attitude angle of the upper rotating body 42 with respect to the lower traveling body 41, the relative attitude angle of the boom 441 with respect to the upper rotating body 42, and the relative attitude angle with respect to the boom 441. The relative attitude angle of the arm 443 and the relative attitude angle of the bucket 445 with respect to the arm 443 are estimated as the actual machine operation mode, and based on the estimation results, the actual machine coordinate system (with respect to the work machine 40) of the bucket 445 (work part) is estimated. (a coordinate system in which the position and orientation are fixed) is estimated. Then, according to a coordinate transformation matrix (rotation matrix or a combination of a rotation matrix and a translation matrix) corresponding to the position and orientation of the actual machine imaging device 412 in the actual machine coordinate system, coordinate values representing the space occupation mode of the working part in the actual machine coordinate system are determined. , coordinates are transformed into coordinate values representing the estimated space occupation mode of the work unit or the estimated extension mode of the image area where the bucket 445 exists in the work environment image coordinate system.

In addition to or in place of the bucket 445, space-occupying features such as at least one of the arm 443 and the boom 441 may be recognized as the space-occupying feature of the working section.

When estimating the space occupation mode of the work unit at the second time point t=t2, some or all of the first communication delay time TTD1, the first response delay time TRD1, the second communication delay time TTD2, and the second response delay time TRD2 are determined. will be taken into consideration. “First communication delay time TTD1” is a communication delay time from when a command signal corresponding to the operation mode of the remote control mechanism 211 is transmitted from the remote wireless communication device 222 until it is received by the actual wireless communication device 422. . The "first response delay time TRD1" is the time when the operation of the work machine 40 is controlled by the actual machine control device 400 according to the command signal after the command signal is received by the actual machine wireless communication device 422 (the command signal is 40) is the response delay time. “Second communication delay time TTD2” is a communication delay time from when the captured image data (environmental image data) is transmitted from the actual wireless communication device 422 until it is received by the remote wireless communication device 222. “Second response delay time TRD2” is a response delay time from when the remote wireless communication device 222 receives environmental image data until the environmental image is output to the image output device 221 according to the environmental image data. .

In a situation where the operator is operating the remote control mechanism 211, the position and orientation of the work unit displayed on the image output device 221 at a certain time t=t is determined by the second delay time (the second communication delay time TTD2 and the second communication delay time TTD2). It represents the real space position and orientation of the working unit at time t=t−(TTD2+TRD2), which is the total time of response delay time TRD2) before. The real space position and orientation of the work unit at the previous time point t=t-(TTD2+TRD2) is further changed by the first delay time (total time of the first communication delay time TTD1 and the first response delay time TRD1) to the previous time point t= This corresponds to the operation mode of the remote control mechanism 211 in t-(TTD2+TRD2)-(TTD1+TRD1). Therefore, based on the operation mode of the remote control mechanism 211 at the first time point t=t1, the space occupancy mode of the working section at the second time point t=t2, which is a time point after the first time point by TTD1+TRD1+TTD2+TRD2, is estimated or predicted. be done.

However, if the delay time is negligibly small, the space occupation mode of the work unit at the second time point t=t2 may be estimated or predicted without taking the delay time into consideration. For example, if the first communication delay time TTD1 and the second communication delay time TTD2 are negligibly small due to further increase in communication speed, the second time point is a time point that is after the first time point t=t1 by TRD1 or TRD1+TRD2. The space occupation mode of the working part at t=t2 may be estimated or predicted. In addition, if the first response delay time TRD1 and the second response delay time TRD2 are so small that they can be ignored, at the second time point t=t2, which is the same as the first time point t=t1 or almost the same time later than this, The space occupation aspect of the work station may be estimated or predicted.

The marker image Img2 is, for example, a semi-transparent image. An edge point group is detected in the work environment image Img1(t), and by shape pattern matching of the image area surrounded by the edge point group, an image area corresponding to the work part (bucket 445) is recognized, and the size and size are added to the image area. The shapes are matched and a marker image Img2 is generated. In addition to a semi-transparent image, the marker image Img2 includes a line drawing representing the outline of the image area or an arrow-shaped image whose tip is a point on the outline of the image area, etc., to improve the visibility of the bucket 445 in the work environment image Img1. Without detracting, the images may have various combinations of shapes, sizes, and colors that represent the space occupancy or position and orientation of the bucket 445.

Then, the second output control element 242 outputs the sign image Img2 at the second time point t=t2 to the image output device 221 by superimposing it on the work environment image Img1 at the third time point t=t3 (FIG. 4/STEP 212 ).

The third time point t=t3 may be the same time point as the second time point t=t2. The third time t=t3 may be an intermediate time t=t1+(1−α)t2 (0<α<1) between the first time t=t1 and the second time t=t2. The third time point t=t3 is the total time of a part of the first communication delay time TTD1, first response delay time TRD1, second communication delay time TTD2, and second response delay time TRD2 than the first time point t=t1. It may be at a later point in time. For example, at the third time point t=t3, the weighted sum β1・TTD1+β2・TRD1 (0≦β1≦1, 0≦ β2≦1 (excluding the case where β1=β2=0) may be a later time point t=t1+β1·TTD1+β2·TRD1. When β1=0 and β2=1, the third time point t=t3 is a time point t=t1+TRD1 which is after the first time point t=t1 by the first response delay time TRD1.

FIG. 5 conceptually shows how the coordinate values of the position and orientation (one or both of the position and orientation) of the work unit (bucket 445) change in the coordinate system of the work environment image Img1 output by the image output device 221. is shown. The broken line in FIG. 5 represents the manner in which the coordinate values of the position and orientation of the working unit change, which are determined according to the received captured image data (see FIG. 4/C20). The solid line in FIG. 5 represents the manner in which the coordinate values of the position and orientation of the work unit are expected to change depending on the operation manner of the remote control mechanism 211.

Here, a period from when the operation of the remote control mechanism 211 is started at time t=t0 to when the operation of the remote control mechanism 211 is stopped at time t=tn will be considered. The position of the working part in the working environment image system (u, v) is represented by a two-dimensional vector, and the posture of the working part is represented by a two-dimensional vector directed from the rear end (the attachment point to the arm 443) to the tip. .

As shown by the solid line in FIG. 5, the position and orientation coordinate values of the working part at any second time point are determined by the solid line according to the operation mode of the remote control mechanism 211 at the first time point before the second time point. It is predicted as shown in .

As shown by the broken line in FIG. 5, in periods other than the period t=t11 to t12 and the period t=t21 to t22, the position and orientation coordinate values of the working part determined according to the captured image data and the remote control mechanism The position and orientation coordinate values of the working part predicted according to the operation mode of 211 are almost the same. In this case, for example, as shown in FIG. 8, a translucent sign image Img2(τ1) imitating the bucket 445 is placed in the work environment image Img(τ1) so as to substantially overlap the image area representing the bucket 445. and is output to the image output device 221.

On the other hand, before and after the period t=t11 to t12 and the period t=t21 to t22, due to a communication failure between the remote control device 20 and the working machine 40, etc., the work is performed according to the captured image data in the work environment image. The position and orientation coordinate values of the working part are changing discontinuously and deviate from the position and orientation coordinate values of the working part predicted according to the operation mode of the remote control mechanism 211. In this case, for example, as shown in FIG. 9, a translucent marker image Img2 imitating the bucket 445 is placed at a position that is predicted to exist originally, although it is shifted from the image area representing the bucket 445. (τ2) is superimposed on the work environment image Img(τ2) and output to the image output device 221. At the beginning of each period t=t11-t12 and period t=t21-t22, the first time point is outside the period, but after that, the first time point and the second time point are changed so that the first time point is included in the period. A time interval of points in time may be set.

(effect)
According to the remote control system with this configuration, the sign image Img2 is generated in the remote control device 20, and is superimposed on the work environment image Img1 and output to the image output device 221 (FIG. 4/STEP210→STEP212, FIG. 8 and Figure 9). Therefore, there is no problem in transmitting the marker image Img2 such as a line drawing.

Due to a transmission delay in the captured image data, the position and orientation coordinate values of the working part according to the captured image data in the work environment image may change discontinuously due to communication failure between the remote control device 20 and the working machine 40, etc. Therefore, the position and orientation coordinate values of the working part may deviate from the predicted position and orientation coordinate values depending on the operation mode of the remote control mechanism 211. In this case, a translucent sign image Img2 (τ2) imitating the bucket 445 is superimposed on the work environment image Img (τ2) at a position that is predicted to originally exist, at a position shifted from the image area representing the bucket 445. and output to the image output device 221 (see FIG. 9).

Therefore, it is possible to reduce or eliminate the discrepancy between the position of the bucket 445 (work unit) in the real space and the position of the marker image Img2 of the bucket 445 in the work environment image Img1.

(Other embodiments of the present invention)
The remote control element 240 detects the working mechanism (swivel mechanism 43 and working attachment 44 ) through the status sensor 414 mounted on the working machine 40 , is transmitted from the actual machine wireless communication device 422 , and is received by the remote wireless communication device 222 . The second output control element 242 recognizes the operation mode of the working mechanism at the second time point based on the deviation between the operation mode of the remote control mechanism 211 at the first time point recognized by the remote control element 240 and the operation mode of the working mechanism. The space occupancy of the working unit (bucket 445) may be predicted.

Similar to FIG. 5, FIG. 6 conceptually shows how the coordinate values of the position and orientation of the work unit (bucket 445) change in the coordinate system of the work environment image Img1 output by the image output device 221. There is. The broken line in FIG. 6 represents the manner in which the coordinate values of the position and orientation of the working unit change, which are determined according to the received captured image data (see FIG. 4/C20).

When the remote control mechanism 211 (for example, the turning lever) is operated, a response delay may occur until the working mechanism (for example, the turning mechanism 43) operates in response to the operation. In this case, for example, the coordinate values of the position and orientation of the work unit predicted according to the operation mode of the remote control mechanism 211 shown by the dashed-dotted line in FIG. A change occurs before the coordinate values of the position and orientation of the working part, which are estimated according to the operation mode of the mechanism 40.

Therefore, for example, depending on this response delay (deviation between the operation mode of the remote control mechanism 211 and the operation mode of the working mechanism 40), the time interval Δt=t2− between the first time point t=t1 and the second time point t=t2 t1 is set. As a result, the manner in which the coordinate values of the position and orientation of the working part change, which are determined according to the captured image data indicated by the broken lines in FIG. The manner in which the coordinate values of the position and orientation of the working part change as shown by solid lines in FIG. 6 is predicted. Therefore, by reducing the influence of the response delay, it is possible to further reduce or eliminate the deviation between the position of the bucket 445 (work unit) in the real space and the position of the marker image Img2 of the bucket 445 in the work environment image Img1.

When the remote control element 240 recognizes that the first specified operation (for example, a turning operation using a turning lever) has stopped as the operation mode of the remote control mechanism 211, the second output control element 242 simultaneously outputs the stop. Alternatively, the output of the marker image Img2 in the image output device 221 may be stopped after a certain period of time (eg, response delay time). As a result, even though the first designation operation has been stopped, the work part can be visually recognized in the work environment image Img1 that is output to the image output device 221 due to the remaining index image Img2 that is displaced in accordance with the first designation operation. Deterioration of sexuality is avoided.

When the remote control element 240 recognizes the start of the second designated operation as the operation mode of the remote control mechanism 211, the second output control element 242 may cause the image output device 221 to output the marker image Img2 for a limited time. . As a result, for a certain period of time after the start of the second designation operation, the index image Img2 that is displaced in accordance with the second designation operation is superimposed on the work environment image Img1 and output to the image output device 221. The second specifying operation may be the same type of operation as the first specifying operation, or may be a different type of operation. The information necessary for the operator is to know whether the work machine 40 is responding to the operation input from the remote control mechanism 211. Therefore, by outputting the marker image Img2 to the image output device 221 for a certain period from the start of the operation, the bucket is It is possible to improve the efficiency of remote operation by the operator while omitting the calculation load for predicting the space occupation mode of 445.

DESCRIPTION OF SYMBOLS 10... Remote control server, 20... Remote control device, 24... Remote control device, 40... Working machine, 210... Remote input interface, 211... Remote control mechanism, 220... Remote output interface, 221... Image output device, 222... Remote Wireless communication equipment, 240, remote control element, 241..first output control element, 242..second output control element, 410..actual device input interface, 412..actual device imaging device, 414..state sensor group, 420..actual device output interface, 440... Work attachment (work mechanism), 445... Bucket (work part), Img1... Work environment image, Img2... Sign image.

Claims (10)

A remote control device for remotely controlling a working machine having a working mechanism,
a remote wireless communication device having a communication function with an actual wireless communication device installed in the work machine;
a remote image output device;
a remote control mechanism;
a remote control element that recognizes an operation mode of the remote control mechanism by an operator and causes the remote wireless communication device to transmit a command signal according to the operation mode to the actual wireless communication device;
A working environment image including a working part of the working mechanism, which is acquired through an imaging device mounted on the working machine, transmitted from the actual wireless communication device, and received by the remote wireless communication device, is sent to the remote image output device. a first output control element to output;
estimating the space occupation mode of the working section at a second time point after the first time point based on the operation mode of the remote control mechanism at the first time point recognized by the remote control element; A sign image representing the space occupation mode of the working area at a second time point without impairing the visibility of the working area is added to the working environment image at a third time point included in the period from the first time point to the second time point. a second output control element superimposed on the image output device and outputting the image to the remote image output device. The remote control device according to claim 1,
The second output control element causes the remote image output device to output the sign image superimposed on the work environment image at the third time point that is an intermediate time point between the first time point and the second time point. Features a remote control device. The remote control device according to claim 2,
The second output control element is
a first communication delay time that is a communication delay time from when a command signal corresponding to the operation mode of the remote control mechanism is transmitted from the remote wireless communication device until it is received by the actual wireless communication device; A first delay that is at least one of a first response delay time that is a response delay time from when a command signal is received by the actual wireless communication device until the operation of the work machine is controlled according to the command signal. time and
a second communication delay time that is a communication delay time from when the working environment image is transmitted from the actual wireless communication device to when it is received by the remote wireless communication device; a second delay time that is at least one of the second response delay times that are the response delay time from when the image is received by the remote image output device until the image is output to the remote image output device;
estimating the space occupancy mode of the work unit by setting a time point later than the first time point as the second time point by a total delay time that is the sum of;
The remote control device is characterized in that the sign image is superimposed on the work environment image and outputted to the remote image output device at the third time point that is after the first time point by the first delay time. . The remote control device according to claim 3,
The second output control element superimposes the sign image on the work environment image at the third time point that is after the first time point by the first response delay time as the first delay time. A remote control device configured to output to the remote image output device. The remote control device according to claim 1,
The remote control is characterized in that the second output control element causes the remote image output device to output the sign image superimposed on the work environment image at the third time point, which is the same time point as the second time point. Device. The remote control device according to claim 5,
The second output control element is
a first communication delay time that is a communication delay time from when a command signal corresponding to the operation mode of the remote control mechanism is transmitted from the remote wireless communication device until it is received by the actual wireless communication device; A first delay that is at least one of a first response delay time that is a response delay time from when a command signal is received by the actual wireless communication device until the operation of the work machine is controlled according to the command signal. time and
a second communication delay time that is a communication delay time from when the working environment image is transmitted from the actual wireless communication device to when it is received by the remote wireless communication device; a second delay time that is at least one of the second response delay times that are the response delay time from when the image is received by the remote image output device until the image is output to the remote image output device;
The remote control device is characterized in that the space occupation mode of the working unit is estimated by setting a time point later than the first time point by a total delay time that is the sum of the above as the second time point. The remote control device according to any one of claims 1 to 6,
the remote control element recognizes an operating mode of the working mechanism that is detected through a state sensor mounted on the working machine, transmitted from the actual wireless communication device, and received by the remote wireless communication device;
The second output control element controls the operation of the working unit at the second time point based on the deviation between the operation mode of the remote control mechanism at the first time point and the operation mode of the working mechanism, which is recognized by the remote control element. A remote control device characterized by estimating a space occupation mode. The remote control device according to any one of claims 1 to 7,
The second output control element is characterized in that when the remote control element recognizes that the first designated operation is stopped as the operation mode of the remote control mechanism, the second output control element stops outputting the marker image on the remote image output device. remote control device. The remote control device according to any one of claims 1 to 8,
The second output control element causes the remote image output device to output the marker image in a time-limited manner when the remote control element recognizes that the start of the second designated operation is an operation mode of the remote operation mechanism. Features a remote control device. A remote control system comprising the remote control device according to any one of claims 1 to 9 and the working machine.

Images