JP7122863B2 - Image display device, image display method and remote control system - Google Patents

Description

The present invention provides an image display that displays image data for a person to refer to when remotely controlling a lifting mechanism (for example, a mechanism using a robot arm) that is provided in a mobile robot and holds and lifts an object. It relates to an apparatus and an image display method. The present invention also relates to a remote control system including the image display device, a mobile robot, and an operating device operated by a person to remotely control a lifting mechanism of the mobile robot.

If the object is placed on the ground or elsewhere, the mobile robot can be used to pick up the object and carry it to another location as follows. A mobile robot moves to a place where an object is placed. A camera provided on the mobile robot captures an image of an area including an object, and the image data thus obtained is transmitted to a device on the remote control side. The device on the remote control side displays the transmitted image data on the display, and a person looks at the displayed image data and operates an operation part (for example, an operation lever or button) to control the mobile robot. Remote control the upper mechanism (for example, a robot arm). According to this maneuver, the lifting mechanism grasps and lifts the object, and in this state the mobile robot moves to another location. It should be noted that the mobile robot can move, for example, by being remotely controlled by a person or by autonomous movement.

The objects mentioned above are, for example, dangerous goods. If the hazardous material is placed in a high traffic area, it can be transported by mobile robot to another location where there is no human presence as described above. Dangerous goods include, for example, Improvised Explosive Devices (IEDs) or unexploded ordnance.

Patent Document 1 below describes the removal of unexploded ordnance using a mobile robot, which is a crawler-type traveling body, as follows. When the unexploded ordnance is picked up by a work machine provided on the mobile robot, the camera provided on the mobile robot takes an image of the unexploded ordnance, and the obtained image data is transmitted to the device on the remote control side. This image data is displayed on the display of the device on the remote control side. A person can remotely control the working machine by viewing the displayed image data.

Japanese Patent Application Laid-Open No. 2005-121343

When an object is placed on a slope that is inclined with respect to the horizontal plane, and the object is held and lifted by the above-described lifting mechanism, in the displayed image data, a person can move in the horizontal direction or the vertical direction. difficult to recognize. Therefore, it may take a long time to cause the lifting mechanism to properly grasp and lift the object by remote control by a person. For example, in order to properly grasp an object with a robot hand provided at the tip of a lifting mechanism, if the robot hand should be oriented vertically, for example, it is difficult for humans to recognize the vertical direction. Therefore, it may take time to orient the robot hand vertically.

In addition, if a person cannot recognize the horizontal or vertical direction by looking at the displayed image data, or if he or she incorrectly recognizes the horizontal or vertical direction, he or she can be lifted by remote control by a person. Even if the mechanism grips the object, there is a possibility that the object cannot be lifted in the vertical direction and the object is dropped. For example, when the above-mentioned robot hand is supposed to hold an object and lift it in the vertical direction, there is a possibility that the object may fall out of the robot hand if the object is lifted in a direction greatly tilted from the vertical direction. be.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image data that captures an area including an object so that a person can easily determine the direction of the horizontal plane or the vertical direction even if the object is placed on a slope that is inclined with respect to the horizontal plane. to make it recognizable.

In order to achieve the above object, according to the present invention, there is provided an image display device for displaying image data of an area including an object when a lifting mechanism provided in a mobile robot for gripping and lifting an object is remotely controlled, comprising:
a communication unit that receives image data of an area including an object captured by a camera provided on the mobile robot;
a display for displaying image data received by the communication unit;
One or both of a horizontal plane index indicating the direction of the horizontal plane and a vertical index indicating the vertical direction based on the orientation information detected by the orientation sensor provided on the mobile robot, using information that can specify the orientation of the camera with respect to the horizontal plane as orientation information. is displayed on a display together with image data.

Further, according to the present invention, there is provided an image display method for displaying image data of an area including an object during remote control of a lifting mechanism provided in a mobile robot for gripping and lifting an object, the method comprising:
(A) receiving image data of an area including an object captured by a camera provided on a mobile robot;
(B) A horizontal plane index indicating the direction of the horizontal plane and a vertical index indicating the vertical direction are determined based on the orientation information detected by the orientation sensor provided in the mobile robot, using information that can specify the orientation of the camera with respect to the horizontal plane as orientation information. An image display method is provided that causes one or both to be displayed on a display along with the image data.

Further, according to the present invention, there is provided a remote control system comprising a mobile robot provided with a lifting mechanism for gripping and lifting an object, and a remote control device for remotely controlling the lifting mechanism,
The remote control device is
the image display device described above;
an operation device that is operated to remotely control the lifting mechanism and that generates an operation command according to the operation performed;
mobile robots are
A remote control system is provided that includes a motion controller that controls the operation of a lifting mechanism according to a motion command sent from a remote control device.

According to the above-described present invention, the horizontal plane index indicating the direction of the horizontal plane and the vertical direction are determined based on the posture information detected by the posture sensor provided in the mobile robot, using the information that can specify the posture of the camera with respect to the horizontal plane as the posture information. One or both of the indicated vertical indicators are caused to appear on the display along with the image data. Therefore, when remotely controlling the lifting mechanism, a person can easily recognize the horizontal plane direction or the vertical direction in the image data by looking at one or both of the displayed horizontal plane index and vertical plane index.

1 is a block diagram illustrating a remote control system according to an embodiment of the invention; FIG. An example of a mobile robot is shown. 4 shows an example of camera image data displayed on a display screen. An example of image data of a camera, a horizontal plane index, and a vertical index displayed on a display screen is shown. FIG. 5 is an explanatory diagram of the index distance in FIG. 4 and is a conceptual diagram viewed from the horizontal direction. 4 is a flow chart illustrating an image display method according to an embodiment of the present invention;

An embodiment of the present invention will be described based on the drawings. In addition, the same code|symbol is attached|subjected to the part which is common in each figure, and the overlapping description is abbreviate|omitted.

FIG. 1 is a block diagram illustrating a remote piloting system 100 according to an embodiment of the invention. A remote control system 100 includes a mobile robot 10 and a remote control device 20 .

<Configuration of mobile robot>
An example of mobile robot 10 is shown in FIG. A mobile robot 10 is used to lift and carry an object 1 placed on the ground surface 2 or elsewhere. In other words, the mobile robot 10 can move to a place where the object 1 is placed, pick up the object 1 at the place, and move to another place to carry the object 1 . In FIG. 2, the object 1 is placed on a ground surface 2 inclined with respect to the horizontal. Object 1 is, for example, a dangerous object. Dangerous goods are, for example, improvised explosive devices or unexploded ordnance. If the object 1 is a dangerous object, the mobile robot 10 is used to remove the dangerous object, for example by lifting and carrying the dangerous object placed on the ground surface 2 where a person passes.

The mobile robot 10 includes a main body 3 provided with a mobile mechanism 5 for moving on the ground surface 2 . The moving mechanism 5 may be a rotating body 5a that moves the main body 3 by being rotationally driven while in contact with the ground surface 2. The rotating body 5a is a crawler belt (that is, an endless belt) in the example of FIG. but it could also be a wheel.

The movement of the mobile robot 10 may be performed by remote control or autonomous movement control by a person. In the former case, for example, a person can move the mobile robot 10 to a desired location by operating a mobile control device (not shown). In the latter case, for example, by transmitting information indicating a desired location to a movement control unit provided in the mobile robot 10, the movement control unit moves the mobile robot 10 to the location indicated by the information. Control the movement of the robot 10 . Since the remote control of the movement of the mobile robot 10 and the autonomous movement of the mobile robot 10 can be performed, for example, by a known technique, detailed description thereof will be omitted.

Configuration of Lifting Mechanism A lifting mechanism 7 is provided on the main body 3 of the mobile robot 10 . The lifting mechanism 7 operates to lift an object 1 placed on the ground surface 2 or elsewhere. The lifting mechanism 7 includes a proximal end portion 7a connected to the main body 3, a distal end portion 7b, a gripping device 7c attached to the distal end portion 7b and configured to contact and grip the object 1, and a proximal end portion. It includes an intermediate portion 7d connecting 7a and tip portion 7b. The gripping device 7c may have, for example, a pair of gripping members 7c1 (see FIG. 3 described later) that sandwich and grip the object 1 . The intermediate portion 7d moves the distal end portion 7b relative to the proximal end portion 7a and the main body 3 by operating relative to the proximal end portion 7a. The intermediate portion 7d may include, for example, a plurality of arms rotatably connected to each other as shown in FIG. 2, and the proximal end portion 7a, the distal end portion 7b, and the intermediate portion 7d may constitute a robot arm.

The base end portion 7a may be connected to the main body 3 so as to be rotatable around the axis C. Axis C is oriented in a direction fixed to body 3 . The axis C may be oriented in a direction that intersects (for example, orthogonally) both the left-right direction and the front-rear direction of the main body 3 . The left-right direction and the front-rear direction of the main body 3 are horizontal when the main body 3 is in a horizontal posture. Also, the distal end portion 7b may be rotatably connected to the intermediate portion 7d. The intermediate portion 7d may be rotatably connected to the base end portion 7a.

With the mobile robot 10 positioned within the upper limit distance from the object 1, the lifting mechanism 7 can operate as follows. By operating the lifting mechanism 7, the gripping device 7c (in the example of FIG. 2, a pair of gripping members 7c1 constituting a robot hand) of the tip portion 7b is moved to a position where the object 1 can be gripped. In the example of FIG. 2, a pair of open gripping members 7c1 are moved to a position such that the object 1 is positioned between them. The gripping device 7c is then operated to grip the object 1 . In the example of FIG. 2, the pair of gripping members 7c1 are operated so as to close, so that the object 1 is sandwiched and gripped by the pair of gripping members 7c1. Next, the target object 1 is lifted from the ground surface 2 by operating the lifting mechanism 7 while the target object 1 is gripped by the gripping device 7c.

The mobile robot 10 is provided with an operation control section 9 for controlling the operation of the lifting mechanism 7, as shown in FIG. The motion control unit 9 receives a motion command generated by an operation device 20B, which will be described later, via the communication unit 19 on the control side and the communication unit 15 on the mobile robot side. The motion control unit 9 controls the lifting mechanism 7 so that the lifting mechanism 7 performs the motion indicated by the received motion command. At this time, the operation control section 9 controls the operation of the lifting mechanism 7 by controlling the driving device 11 that drives the lifting mechanism 7 . In the example of FIG. 2, the driving device 11 includes an actuator provided at a connection portion between the arms forming the intermediate portion 7d and driving the arms to rotate with respect to each other, , an actuator that rotationally drives the distal end portion 7b with respect to the intermediate portion 7d, and an actuator that opens and closes the pair of gripping members 7c1. Each actuator may be, for example, a servomotor, but may also be another device.

- Configuration for Acquiring Data Used in Image Display Device The mobile robot 10 is provided with a camera 13 . The camera 13 captures an image of an area including the object 1 and generates image data of the area. The camera 13 is provided on the main body 3 of the mobile robot 10 in the example of FIG. may Also, a plurality of cameras may be provided on the mobile robot 10, and the camera to be used may be selected from among these by operating the operation device 20B. In this case, camera 13 is the selected camera in the following. The image data generated by the camera 13 is transmitted to the image display device 20A from the mobile robot side communication unit 15 provided in the mobile robot 10 as shown in FIG. Transmission by the communication unit 15 may be performed using wireless communication.

The mobile robot 10 is provided with a posture sensor 17 as shown in FIG. The orientation sensor 17 detects information that can specify the orientation of the camera 13 with respect to the horizontal plane (hereinafter referred to as orientation information). The orientation information detected by the orientation sensor 17 is transmitted from the communication unit 15 on the mobile robot side to the image display device 20A.

When the camera 13 is provided on the main body 3 , the orientation sensor 17 may be one using a gyroscope 17 a provided on the main body 3 or the camera 13 .

When the camera 13 is provided at the tip portion 7 b of the lifting mechanism 7 , the attitude sensor 17 detects, for example, the gyroscope 17 a provided on the main body 3 and the movement amount of each part of the lifting mechanism 7 with respect to the main body 3 . includes an operation amount detection unit 17b for detecting the In this case, the posture information includes the posture of the main body 3 with respect to the horizontal plane detected using the gyroscope 17a, and the movement amount of each part of the lifting mechanism 7 detected by the movement amount detection section 17b. In the example of FIG. 2, the movement amount of each part of the lifting mechanism 7 is the rotation angle of the plurality of arms forming the intermediate part 7d with respect to each other, the rotation angle of the base end part 7a around the axis C, and the tip end relative to the intermediate part 7d. It includes the rotation angle of the portion 7b. These rotation angles may mean rotation angles from predetermined reference rotation positions of the arm, the base end portion 7a, and the tip end portion 7b, respectively.

<Configuration of remote control device>
As shown in FIG. 1, the remote control device 20 includes an image display device 20A and an operation device 20B. The image display device 20A includes a communication section 19, a display 21, and a display control device 23 on the control side.

The control-side communication unit 19 receives image data generated by the camera 13 of the mobile robot 10 and transmitted from the mobile robot-side communication unit 15 . This image data is input from the communication section 19 on the control side to the display control device 23 (display control section 23b). The control-side communication unit 19 receives posture information detected by the posture sensor 17 of the mobile robot 10 and transmitted from the mobile robot-side communication unit 15 . This posture information is input from the communication unit 19 on the control side to the display control device 23 (that is, the direction calculation unit 23a and the display control unit 23b, which will be described later).

The display 21 is for displaying image data received by the communication unit 19 on the control side. This image data is displayed for reference by a person when remotely controlling the lifting mechanism 7 of the mobile robot 10 . FIG. 3A shows image data displayed on the screen 21a of the display 21. FIG.

The display control device 23 causes the display 21 to display the horizontal plane index Mh indicating the horizontal plane and the vertical index Mv indicating the vertical direction together with the input image data based on the inputted posture information. The display control device 23 includes a direction calculator 23a and a display controller 23b. The horizontal plane index Mh indicates a plurality of horizontal directions that intersect with each other.

The direction calculation unit 23a obtains one or both of the horizontal plane direction and the vertical direction in the image data described above based on the input attitude information. FIG. 3(B) shows the directions (two horizontal directions perpendicular to each other) Dh1 and Dh2 of the horizontal plane H obtained by the direction calculation unit 23a and the vertical direction Dv in the image data of FIG. 3(A). .

In one example, the direction calculator 23a obtains the directions Dh1 and Dh2 of the horizontal plane H and the vertical direction Dv at the set position Ps (set pixel) in the image data, as shown in FIG. 3B, based on the orientation information. However, the set position Ps is a predetermined pixel position in the above-described displayed image data, and the position in the real world corresponding to this position is a set distance away from the camera 13 when the image data is acquired. Suppose there is That is, assuming that the above image data captures the object 1 and a certain position in the real world, the certain position in the image data becomes the set position Ps, and the position and the camera 13 at the time of the image pickup is the set distance. Such set positions and set distances may be determined in advance and set in the direction calculator 23a.

Note that the horizontal directions Dh1 and Dh2, the vertical direction Dv, the set position Ps, and the horizontal plane H (in FIG. 3B, the broken-line horizontal frame H included in the horizontal plane) in FIG. and may not actually be displayed on the display 21 .

When the camera 13 is provided on the main body 3, the direction calculation unit 23a calculates the direction of the horizontal plane H at the set position Ps in the image data based on the orientation information and the known orientation of the camera 13 with respect to the main body 3. One or both of Dh1, Dh2 and the vertical direction Dv are obtained. When the camera 13 is provided at the distal end 7b of the lifting mechanism 7, the direction calculation unit 23a includes posture information (including the posture of the main body 3 with respect to the horizontal plane and the amount of movement of each part of the lifting mechanism 7), The known orientation of the camera 13 with respect to the distal end 7b, the known position of the camera 13 at the distal end 7b, and the known dimensions of each part of the lifting mechanism 7 (in the example shown, the length of each arm, the proximal end 7a ), one or both of the directions Dh1 and Dh2 of the horizontal plane H at the set position Ps and the vertical direction Dv are obtained.

Based on one or both of the directions Dh1 and Dh2 of the horizontal plane H at the set position Ps and the vertical direction Dv obtained by the direction calculator 23a, the display control unit 23b calculates a horizontal plane index Mh representing a horizontal plane including the set position Ps, and , and the vertical index Mv are displayed on the display 21 together with the above image data. For example, the display control unit 23b causes the horizontal surface index Mh and the vertical index Mv to be superimposed and displayed on the image data described above. The vertical index Mv may be displayed at the set position Ps described above. FIG. 4 shows an example of image data, a horizontal plane index Mh, a vertical index Mv, etc. displayed on the display 21 by the display control unit 23b.

The horizontal plane index Mh may be a horizontal plane image displayed in the image data. In this case, the horizontal plane index Mh may be, for example, an image of a grid three-dimensionally displayed so as to be positioned on the horizontal plane, as shown in FIG. For example, as shown in FIG. 4, this grid includes a plurality of lines extending in the horizontal direction of the screen 21a and arranged at grid intervals in the depth direction of the image data, and a plurality of lines extending in the depth direction of the image data and the screen 21a. may include a plurality of lines arranged at grid intervals in a direction along the horizontal direction of the . Note that the display control unit 23b may display a horizontal plane image (for example, the grid image) as the horizontal plane index Mh over a range set in the image data. The set range is a range corresponding to a specified range of distance from the camera 13 (for example, a range of 1 m to 2 m) when the image data is acquired.

The display control unit 23b causes the display 21 to display one or both of the dimension of the next horizontal surface index Mh and the distance from the main body 3 in addition to the image data, the horizontal surface index Mh, and the vertical index Mv as described above. good too.

Display of Dimension of Horizontal Plane Index The display control unit 23b causes the display 21 to display the horizontal plane index Mh overlaid on the image data, and causes the display 21 to display the dimension of the horizontal plane index Mh in association with the horizontal plane index Mh. The dimension of the horizontal surface index Mh may be the dimension of a part of the horizontal surface index Mh, or may be the dimension of the entire horizontal surface index Mh. In the former case, the display control unit 23b adjusts the dimension of a part of the horizontal plane index Mh (in the example of FIG. 4, the grid spacing along the left-right direction of the screen 21a and the grid spacing along the depth direction of the image data). In the latter case, the display 21 may display the overall dimension of the horizontal plane index Mh in association with the entirety. Note that the dimension of the horizontal plane index Mh means the corresponding dimension in the real world. That is, assuming that the horizontal plane index Mh exists in the real world, and assuming that the above image data is obtained by imaging the object 1 and the horizontal plane index Mh, the dimension of the horizontal plane index Mh in the real world (one (partial or overall dimensions) are the dimensions shown on the display 21 .

In the example of FIG. 4, the dimension of the grid spacing, which is part of the horizontal surface index Mh, is indicated by arrows A1 and A2 representing the grid spacing and numerical values P and Q attached to the arrows A1 and A2, respectively. The arrow A1 represents the grid spacing along the horizontal direction of the image data, and the arrow A2 represents the grid spacing along the depth direction of the image data. In FIG. 4, the arrows A1 and A2 are associated with the grid spacing dimension by their indicated positions.

The display control unit 23b calculates the dimension of the horizontal plane index Mh based on, for example, the set position Ps, the set distance, and the characteristics of the camera 13, and causes the display 21 to display the dimension as described above.

Display of distance from main body The display control unit 23b displays the distance (simply referred to as the index distance) from the set point on the main body 3 (for example, the front end of the main body 3) to the displayed horizontal surface index Mh or vertical index Mv. It is displayed on the display 21 in association with the horizontal plane index Mh or the vertical index Mv (for example, set points on the indices Mh and Mv). The index distance may be the actual distance between the set point on the body 3 and the position in the real world corresponding to the set point on the horizontal index Mh or the vertical index Mv in the image data. The index distance may be a horizontal distance. The horizontal distance between two positions (index distance or tip distance described later) means that the two positions (for example, a set point on the main body 3 and a set point P1 or P2 described later) are vertically aligned on one horizontal plane. When projected, it is the distance between the two projected positions (the same applies hereinafter). In the example of FIG. 4A or 4B, the index distance is the horizontal distance from the set point on the body 3 to the set point P1 or P2 on the horizontal plane index Mh. FIG. 5 is a conceptual diagram of FIG. 4A or 4B viewed from the horizontal direction (horizontal direction orthogonal to the vertical plane including the front-rear direction of the main body 3).

In FIG. 4A, the set point P1 is the point at the middle position in the horizontal plane index Mh (for example, the horizontal plane index Mh of the grid image). This intermediate position may be a position in the center of the horizontal plane index Mh (for example, the center of the horizontal plane index Mh), but is not limited to this. In FIG. 4A, the index distance is associated with the set point P1 on the horizontal plane index Mh by the display position of the arrow A3, and displayed by the numerical value R1 attached to the arrow A3.

In FIG. 4B, the set point P2 is the closest point in the horizontal plane index Mh (for example, the horizontal plane index Mh of the grid image). Here, the “front side” means the side closer to the main body 3 . That is, the “point on the frontmost side” may be the point that corresponds to the point closest to the set point of the main body 3 among the points in the real world to which the points on the horizontal surface index Mh correspond. In FIG. 4B, the index distance is associated with the set point P2 on the horizontal plane index Mh by the display position of the arrow A3, and displayed by the numerical value R2 attached to the arrow A3.

The index distance may be obtained by the display control unit 23b based on the characteristics of the camera 13 and the like. When the camera 13 is provided on the main body 3, the display control unit 23b controls the known position of the set point on the main body 3, the known mounting position and mounting orientation of the camera 13 with respect to the main body 3, and the main body with respect to the horizontal plane. 3, the set position Ps and the set distance, the position of the set point (the set point P1 or P2 in FIGS. 4A and 4B) on the horizontal surface index Mh or the vertical index Mv, and the characteristics of the camera 13. The index distance may be determined based on When the camera 13 is provided at the tip 7b of the lifting mechanism 7, the display control unit 23b controls the known position of the set point on the body 3, the known mounting position of the camera 13 with respect to the tip 7b, and the The mounting posture, the posture of the main body 3 with respect to the horizontal plane, the set position Ps and the set distance, and the set point (the set point P1 or P2 in FIGS. 4A and 4B) on the horizontal surface index Mh or the vertical index Mv. The index distance may be determined based on the position, the characteristics of the camera 13, the amount of movement of each part of the lifting mechanism 7, and the known dimensions of each part. Here, the "posture of the main body 3 with respect to the horizontal plane and the amount of movement of each part of the lifting mechanism 7" used to obtain the index distance is determined by the detection unit of the mobile robot (for example, the above-mentioned It may be detected by the gyroscope 17a or the motion amount detection unit 17b), and may be included in the above-described posture information input to the display control device 23. FIG. If the index distance is not a horizontal distance but a three-dimensional distance, it is not necessary to use the orientation of the main body 3 with respect to the horizontal plane to obtain the index distance.

The display control unit 23b may cause the display 21 to display the distance from the set point on the main body 3 to the tip portion 7b of the lifting mechanism 7 (referred to as the tip portion distance) together with the index distance. In this case, the display control unit 23b controls the input operation amount of each part of the lifting mechanism 7, known dimensions of each part of the lifting mechanism 7, known positions of the set points on the main body 3, and relative to the horizontal plane. Based on the posture of the main body 3, the tip distance from the set point on the main body 3 to the tip 7b of the lifting mechanism 7 is calculated. For example, as shown in FIG. 4, Scm may be displayed at a predetermined position (for example, lower right of the image data) as the tip distance. Here, "the posture of the main body 3 with respect to the horizontal plane and the amount of movement of each part of the lifting mechanism 7" used for calculating the tip distance is determined by the detection unit of the mobile robot (for example, the above-mentioned It may be detected by the gyroscope 17a or the movement amount detection unit 17b), and may be included in the above-mentioned posture information input to the display control device 23. FIG. When the above index distance is a horizontal distance, the tip distance is a horizontal distance, and when the above index distance is a three-dimensional distance, the tip distance is A three-dimensional distance is preferable. If the tip distance is not a horizontal distance but a three-dimensional distance, it is not necessary to use the attitude of the main body 3 with respect to the horizontal plane to obtain the tip distance.

<Configuration of operation device>
The operating device 20B is for remote control of the lifting mechanism 7 by a person. In FIG. 1, the operation device 20B includes an operation unit 25 operated by a person, and a command generation unit 27 that generates an action command according to the operation performed on the operation unit 25. FIG.

The control-side communication unit 19 wirelessly transmits the action command generated by the command generation unit 27 to the mobile robot 10 . The communication unit 15 on the mobile robot side receives the motion command transmitted from the communication unit 19 on the control side and inputs it to the motion control unit 9 .

As the operation portion 25, a movement operation portion 25a for moving the tip portion 7b of the lifting mechanism 7 with respect to the main body 3 and a gripping operation portion 25b for causing the gripping device 7c to perform a gripping operation are provided. good.

The action command generated by the command generation unit 27 by operating the movement operation unit 25a may indicate, for example, the movement of the tip part 7b in the set direction (hereinafter simply referred to as the set direction) with respect to the main body 3 of the mobile robot 10. FIG. That is, the movement operation unit 25a can be operated to command movement in the set direction, and the command generation unit 27 generates an operation command indicating movement in the set direction by the operation.

There may be a plurality of setting directions, and the movement operation section 25a may be configured so that a person can perform an operation to instruct the movement operation section 25a to move in each of the setting directions. In this case, the movement operation unit 25a may be configured so that a person can simultaneously perform operations to instruct the movement operation unit 25a to move in two or more set directions.

The movement operation part 25a may include, for example, one or more levers. In this case, each setting direction may correspond to any direction in which the lever is operated (for example, the direction in which the lever is tilted from the reference posture). When the lever is operated in a direction corresponding to one of the set directions, the command generator 27 generates an operation command indicating movement in the set direction. Note that when the lever is operated in a direction having two directional components corresponding to the two set directions, the command generation unit 27 simultaneously generates an operation command indicating movement in each of the two set directions. you can Note that the command generation unit 27 may generate an operation command indicating movement in the set direction only while the lever is being operated in the direction corresponding to the set direction. As a result, the distal end portion 7b moves in the set direction for the specified time.

In one example, the multiple setting directions may be the following (1) to (3).
(1) the lateral direction of the main body 3; (2) the longitudinal direction of the main body 3;

In this case, the movement operation part 25a may include, for example, first and second levers. When the first lever is tilted to the right as seen from the operator, the command generator 27 generates an operation command indicating movement of the distal end portion 7b in the right direction of the left and right directions in (1) above. When the first lever is tilted to the left as seen from the operator, the command generation unit 27 generates an operation command indicating movement of the tip portion 7b in the left direction of the left and right directions in (1) above. Similarly, when the first lever is tilted forward as seen from the operator, the command generation unit 27 generates an operation command indicating movement of the distal end portion 7b in the forward direction of the front-rear direction in (2) above. When the lever 1 is tilted rearward as seen from the operator, the command generator 27 generates an operation command indicating movement of the distal end portion 7b in the rearward direction of the front-rear direction in (2) above. Similarly, when the second lever is tilted forward as viewed from the operator, an operation command indicating movement of the tip portion 7b in one of the above directions (3) (two directions opposite to each other) is issued. is generated by the command generation unit 27, and when the second lever is tilted rearward as seen from the operator, an operation command indicating movement of the distal end portion 7b in the other direction of the direction (3) is generated. generated by the unit 27; However, the movement operation unit 25a may have other configurations.

In another example, the multiple setting directions may be the following (4) to (6).
(4) the direction around the axis C of the base end portion 7a of the lifting mechanism 7;
(5) the direction in which the lifting mechanism 7 is facing; Here, the direction in which the lifting mechanism 7 faces is the direction from the proximal end portion 7a to the distal end portion 7b of the lifting mechanism 7, and is a plane including the left-right direction of the main body 3 and the front-rear direction of the main body 3. parallel direction.
(6) A direction intersecting (for example, perpendicular to) both the left-right direction of the main body 3 and the front-rear direction of the main body 3 .

In this case as well, the movement operation section 25a is configured so that a person can operate the movement operation section 25a to command movement in each of the set directions (4) to (6).

Further, the gripping operation section 25b may be configured to be able to perform an operation for instructing the motion of gripping the object 1 by the gripping device 7c. For example, in a state in which the grasping device 7c of the distal end portion 7b is moved to a position where the object 1 can be grasped by the operation of the movement operation portion 25a, the grasping operation portion 25b (for example, a button) is operated to execute grasping. By doing so, the gripping device 7c performs an operation of gripping the object 1 . In the example of FIGS. 2 and 3, the operation may be an operation of gripping the object 1 by closing the pair of gripping members 7c1 that are open.

<Image display method>
FIG. 6 is a flow chart showing an image display method performed in the remote control system 100 described above.

In step S1, the mobile robot 10 moves to a place where the object 1 is placed by remote control or autonomous movement, and stops at a position where the object 1 can be grasped and lifted by the lifting mechanism 7. do. In the case of autonomous movement, position information indicating the location is given to a movement control section (not shown) of the mobile robot 10 .

In step S2, the camera 13 of the mobile robot 10 takes an image of an area including the object 1 and generates image data of this area.

In step S3, the attitude sensor 17 of the mobile robot 10 detects the above-described attitude information when performing step S2. Steps S2 and S3 may be performed simultaneously.

In step S4, the image data and orientation information obtained in steps S2 and S3 are transmitted from the communication unit 15 of the mobile robot to the remote control device 20. FIG.

In step S<b>5 , the image data and attitude information transmitted in step S<b>4 are received by the communication unit 19 on the control side and input to the display control device 23 .

In step S6, the direction calculation unit 23a of the display control device 23 calculates the direction of the horizontal plane at the set position Ps in the image data input in step S5 based on the orientation information input in step S5, as described above. and/or the vertical direction.

In step S7, the display control unit 23b displays the image data input in step S5 on the display 21, and based on one or both of the horizontal direction and the vertical direction at the set position Ps obtained in step S6, The horizontal surface index Mh and the vertical index Mv are displayed on the display 21 . At this time, the display control unit 23b may display the horizontal surface index Mh and the vertical index Mv at the set position Ps in the image data.

Further, in step S7, the display control unit 23b of the display control device 23 calculates the dimension of the horizontal plane index Mh based on the characteristics of the camera 13, etc., as described above, and compares the dimension of the horizontal plane index Mh with the horizontal plane index Mh. They may be displayed on the display 21 in association with each other.

Further, in step S7, the display control unit 23b of the display control device 23 controls the distance from the set point on the main body 3 to the set point on the horizontal plane index Mh or the vertical index Mv based on the characteristics of the camera 13, etc., as described above. (index distance) may be obtained, and the index distance may be displayed on the display 21 in association with a set point on the horizontal surface index Mh or the vertical index Mv.
Further, as described above, the display control unit 23b may cause the display 21 to display the distance (tip distance) from the set point on the main body 3 to the tip 7b of the lifting mechanism 7 along with the index distance. .

Steps S2 to S7 described above are repeatedly performed. While viewing the image data, the horizontal index Mh, the vertical index Mv, the dimensions of the horizontal index Mh, and the index distance displayed on the display 21 , the operator operates the operation unit 25 to cause the lifting mechanism 7 to move the object. Grab and lift object 1.

<Effects of Embodiment>
In the embodiment described above, for example, the following effects are obtained.

The posture sensor 17 provided in the mobile robot 10 detects information that can specify the posture of the camera 13 with respect to the horizontal plane as posture information. Based on this posture information, one or both of a horizontal plane index Mh indicating the direction of the horizontal plane and a vertical index Mv indicating the vertical direction are displayed on the display 21 together with image data obtained by imaging the area of the object 1 . Therefore, when remotely controlling the lifting mechanism 7, a person can easily recognize the horizontal plane direction or the vertical direction in the image data by looking at one or both of the displayed horizontal plane index Mh and vertical index Mv. Therefore, even when the object 1 is placed on a slope inclined with respect to the horizontal plane, the lifting mechanism 7 of the mobile robot 10 can be easily remotely controlled so as to grasp and lift the object 1. Become. For example, the operator can easily remotely control the lifting mechanism 7 so that the lifting mechanism 7 lifts the object 1 vertically.

Based on the posture information detected by the posture sensor 17 when the camera 13 captures an image of the area including the object 1, the direction calculator 23a obtains one or both of the horizontal plane direction and the vertical direction in the image data. Based on one or both of the horizontal plane direction and the vertical direction thus obtained, the display control unit 23b can cause the display 21 to display the horizontal plane index Mh and the vertical plane index Mv.

When the horizontal plane index Mh is an image of a grid displayed three-dimensionally so as to be positioned on the horizontal plane in the image data, the horizontal plane extends by looking at the direction in which such a grid extends. Makes it easier to recognize the direction you are facing.

When the display control unit 23b causes the display 21 to display the horizontal plane index Mh superimposed on the image data, and displays the dimensions of the horizontal plane index Mh or the vertical index Mv on the display 21, the person can refer to these dimensions. Therefore, by comparing the size of the object 1 with the horizontal surface index Mh or the vertical index Mv in the image data, the size of the object 1 can be easily recognized.

When the display control unit 23b causes the display 21 to display the distance from the set point on the main body 3 to the set point on the displayed horizontal surface index Mh or vertical index Mv, a person can refer to this distance. , the distance from the main body 3 to the object 1 can be easily recognized by comparing the position of the horizontal surface index Mh or the vertical index Mv with the position of the object 1 in the image data.

When the display control unit 23b causes the display 21 to display the distance from the set point on the main body 3 to the tip portion 7b of the lifting mechanism 7 (tip portion distance) together with the index distance, the person can display the index distance and the tip portion 7b. By referring to the tip portion distance, the difference in distance between the horizontal surface index Mh or the vertical index Mv and the tip portion 7b (gripping device 7c) can be confirmed. In this case, a person further recognizes the positional relationship between the object 1 and the tip portion 7b (grasping device 7c) by looking at the positional relationship between the horizontal surface index Mh or the vertical index Mv in the image data and the object 1. easier. As a result, it becomes easier to remotely control the lifting mechanism 7 so as to grip a desired portion of the object 1 with the gripping device 7c.

<Change example>
It goes without saying that the present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the technical idea of the present invention. For example, any one of Modifications 1 to 3 below may be adopted, or a plurality of Modifications 1 to 3 may be arbitrarily combined and adopted. In this case, points not described below may be the same as those described above.

(Modification 1)
The direction calculator 23 a may be provided in the mobile robot 10 . In this case, one or both of the horizontal direction and the vertical direction in the image data are determined by the direction calculation unit 23a and input to the display control unit 23b via the communication unit 15 on the mobile robot side and the communication unit 19 on the control side. be done.

(Modification 2)
The display control unit 23b may cause the display 21 to display only one of the horizontal surface index Mh and the vertical surface index Mv (horizontal surface index Mh or vertical surface index Mv).

(Modification 3)
A laser sensor (for example, a laser range finder LRF) may be used instead of the camera 13 described above. In this case, in the above description, the image data is distance image data generated by the measurement of the laser sensor.

1 object, 2 ground surface, 3 main body, 5 moving mechanism, 5a rotating body, 7 lifting mechanism, 7a proximal end, 7b distal end, 7c gripping device, 7c1 gripping member, 7d intermediate portion, 9 motion control section, 10 Mobile robot 11 Driving device 13 Camera 15 Mobile robot side communication unit 17 Attitude sensor 17a Gyroscope 17b Movement amount detection unit 19 Control side communication unit 20 Remote control side device 20A Image display device , 20B operation device, 21 display, 21a screen, 23 display control device, 23a direction calculation unit, 23b display control unit, 25 operation unit, 25a movement operation unit, 25b grip operation unit, 27 command generation unit, 100 remote control system, C axis, Dh1, Dh2 direction of horizontal plane (horizontal direction), Dv vertical direction, H horizontal plane (horizontal frame), Ps setting position, Mh horizontal plane index, Mv vertical index

Claims (8)

An image display device for displaying image data of an area including an object when a lifting mechanism provided in a mobile robot for gripping and lifting an object is remotely controlled,
a communication unit that receives image data of an area including the object captured by a camera provided on the mobile robot;
a display that displays the image data received by the communication unit;
Using information that can specify the orientation of the camera with respect to the horizontal plane as orientation information, and based on the orientation information detected by the orientation sensor provided in the mobile robot, a horizontal plane index representing the horizontal plane and a vertical index indicating the vertical direction are determined, A display control device for displaying on the display together with the image data ,
The image display device , wherein the horizontal plane index is an image of a grid that is three-dimensionally displayed so as to be positioned on the horizontal plane in the image data, and is displayed superimposed on the image data . The display control device is
a direction calculation unit that calculates a horizontal plane direction and a vertical direction in the image data based on the posture information;
2. The image according to claim 1, further comprising a display control unit for displaying said horizontal surface index and said vertical surface index together with said image data on said display based on the horizontal direction and vertical direction obtained by said direction calculation unit. display device. 3. The image display device according to claim 1, wherein said vertical indicator is an arrow . An image display device for displaying image data of an area including an object when a lifting mechanism provided in a mobile robot for gripping and lifting an object is remotely controlled,
a communication unit that receives image data of an area including the object captured by a camera provided on the mobile robot;
a display that displays the image data received by the communication unit;
Using information that can specify the orientation of the camera with respect to the horizontal plane as orientation information, and based on the orientation information detected by an orientation sensor provided in the mobile robot, a horizontal plane index indicating the direction of the horizontal plane is displayed together with the image data on the display. and a display control device for displaying on
The display control device is
a direction calculation unit that calculates a direction of a horizontal plane in the image data based on the posture information;
a display control unit for displaying the horizontal surface index together with the image data on the display based on the direction of the horizontal surface obtained by the direction calculation unit;
The display control unit causes the display to display the horizontal plane index superimposed on the image data, and causes the display to display a dimension of the horizontal plane index, the dimension being a corresponding real-world dimension. display device. An image display device for displaying image data of an area including an object when a lifting mechanism provided in a mobile robot for gripping and lifting an object is remotely controlled,
a communication unit that receives image data of an area including the object captured by a camera provided on the mobile robot;
a display that displays the image data received by the communication unit;
A horizontal plane index indicating the direction of the horizontal plane and a vertical index indicating the vertical direction are determined based on the orientation information detected by the orientation sensor provided in the mobile robot, using information that can specify the orientation of the camera with respect to the horizontal plane as orientation information. a display control device for displaying one or both on the display together with the image data;
The display control device is
a direction calculation unit that calculates one or both of a horizontal plane direction and a vertical direction in the image data based on the posture information;
a display control unit for displaying one or both of the horizontal plane index and the vertical direction along with the image data on the display based on one or both of the horizontal plane direction and the vertical direction obtained by the direction calculation unit;
The display control unit causes the display to display one or both of the horizontal plane index and the vertical index together with the image data, and displays the displayed horizontal plane index or the vertical index from a set point on the main body of the mobile robot. An image display device that causes the display to display a distance to an index as an index distance. a motion amount of each part of the lifting mechanism is detected by a detection part of the mobile robot and input to the display control part;
The display control unit
Based on the amount of movement, the known dimensions of each part of the lifting mechanism, and the known location of the set point on the body, the distance from the set point on the body to the distal end of the lifting mechanism. Calculate the distance as the tip distance,
6. The image display device according to claim 5, wherein said tip distance, said index distance, and one or both of said horizontal plane index and said vertical index are displayed on said display together with said image data. An image display method for displaying image data of an area including an object when a lifting mechanism provided in a mobile robot for gripping and lifting an object is remotely operated, the method comprising:
(A) receiving image data of an area including the object captured by a camera provided on the mobile robot;
(B) A horizontal plane index indicating a horizontal plane and a vertical index indicating a vertical direction based on the orientation information detected by the orientation sensor provided in the mobile robot, using information that can specify the orientation of the camera with respect to the horizontal plane as orientation information. is displayed on the display together with the image data ,
The image display method , wherein the horizontal plane index is a grid image that is three-dimensionally displayed so as to be positioned on the horizontal plane in the image data, and is superimposed on the image data in (B) above . A remote control system comprising a mobile robot provided with a lifting mechanism for gripping and lifting an object, and a remote control device for remotely controlling the lifting mechanism,
The remote control device,
The image display device according to any one of claims 1 to 6,
an operation device that is operated to remotely control the lifting mechanism and that generates an operation command according to the operation performed;
The mobile robot is
A remote control system comprising an operation control unit that controls the operation of the lifting mechanism according to the operation command transmitted from the remote control device.

Images