DE102017007908A1 - Method for controlling the movement of a mobile robot - Google Patents

Abstract

The invention relates to a method for controlling the movement of a mobile robot (R) by a gesture of a gesture generator (G), wherein the gesture generator (G) is detected by means of at least one camera (K) and from the acquired camera data movement data for the robot ( R), which are transmitted to the robot (R) and converted by it into a movement, wherein in a the gesture (G) associated first coordinate system, in particular that has its origin in the gesture generator (G), from the posture of The coordinates of a body posture changing gesture point (GP) are determined and from the coordinates of the gesture point (GP) and the coordinates of a fixed in the first coordinate system reference point (RP1) a connection vector (V) between gesture point (GP ) and reference point (RP1) is calculated and assigned to the robot (R) and the robot (R) mitbew second coordinate system, in particular that has its origin in the robot (R), the robot (R) relative to a reference point (RP2) of the second coordinate system performs a movement whose direction starting from its reference point (RP2) the direction of the connection vector (V) corresponds and their speed depends on the amount of the connection vector (V).

Description

The invention relates to a method for controlling the movement of a mobile robot by a gesture of a gesture transmitter, wherein the gesture sensor is detected by at least one camera data technology and from the captured camera data motion data for the robot are formed, which are transmitted to the robot and from there into a Movement be implemented.

In the prior art, for example, it is known to control mobile robots by means of a gesture. The control is not done for the purpose of controlling a movement of the robot, but to trigger a special action of the robot with the gesture. For example, For example, a flight drone equipped with a camera may detect a particular gesture of a gesture by image processing software and, upon detection, trigger the creation of a photograph of the gesture generator. To do this, the gesture maker must make a very specific gesture, as only this is detected by the image evaluation of the robot.

It is also known to control the movement of stationary moving robot arms by means of gestures of a gesture generator. In this motion control, the robotic arm typically accurately tracks the motion that the gesture generator detects, e.g. pretending with his arm.

In mobile robot motion control, hitherto, remote controls of various types, e.g. using a smartphone. Such remote controls have input means to affect the movement of the robot over this. For example, The movement of a mobile robot can be controlled by different inclinations of the remote control.

The disadvantage here is that by using a remote control is often not very intuitive and a user of such a remote control must first learn how to handle this. Furthermore, the need for a remote control system is complex and expensive.

Thus, it is an object of the invention to provide a method of controlling the movement of a mobile robot that allows intuitive control even for inexperienced persons and without the need to handle a remote control.

This object is achieved in that, in a method of the generic type mentioned in the first coordinate system associated with the gesture from the camera data representing the posture of the gesture the coordinates of a changing with the posture gesture point are determined and from the coordinates of the gesture point and the coordinates of a reference point fixed in the first coordinate system, a connection vector between the gesture point and the reference point is calculated and, in a second coordinate system assigned to the robot and moving with the robot, the robot makes a movement relative to a reference point of the second coordinate system, whose direction starts from its reference point of direction of the connection vector and whose speed depends on the magnitude of the connection vector.

For the invention, it is essential to get along completely without a remote control to be operated. Rather, it is essential in the invention that the control of the mobile robot results from the posture of the gesture generator.

The posture is in this case detected by at least one camera data, which is understood that the gesture is imaged by a lens of at least one camera on the camera chip and the optical image of the posture is converted into camera data, as known in the art is. With the help of this camera data, the coordinates of the gesture point are determined. This can be done with software in the camera or externally to the camera with software that runs in an evaluation unit to which the camera data is transmitted. By using more than one camera, it can be ensured that at least one of the multiple cameras completely captures the gesture. It may be provided to utilize only those camera data from that camera with respect to the determination of the coordinates of the gesture point, which fully captures the gesture. The data of other camera may e.g. be discarded.

In the invention, there is no need to recognize one or more specific gestures in the captured camera data. Rather, the invention is based on the determination of a gesture point in the first coordinate system of the gesture generator for each arbitrarily adopted posture of the gesture, the coordinates of which are determined in the first coordinate system. This gesture point or its coordinates thus change when the gesture changes its posture. A gesture within the meaning of the invention is thus any possible posture and not one or more specific. A gesture controller thus controls a robot according to the invention as a function of its assumed posture.

Furthermore, a reference point with its coordinates is firmly defined in the first coordinate system. It is now possible from the coordinates of the variable gesture point and the fixed reference point in the first coordinate system to calculate a connection vector between these points. Such a vector is defined by a direction and a length or an amount. According to the invention, in the movement of the mobile robot, the direction of the connection vector is then interpreted as the direction of movement of the robot, and the amount or the length of the connection vector specifies a speed with which the robot is to move in said direction. The speed can be calculated, for example, by means of a stored formula relationship from the amount or the length of the connection vector. Such a relationship may represent a proportionality between length / amount and speed.

The direction and the speed or data from which these are calculated are, according to the invention, communicated to the robot by an evaluation unit, which has received the camera data, insofar as it does not itself comprise the evaluation unit.

The robot performs the movement in said direction and the determined speed starting from a fixed reference point in a second coordinate system that is assigned to the robot.

Thus, in the invention, the desired direction and speed in the first coordinate system of the gesture sensor is detected or determined and interpreted by the robot in its own second coordinate system. The two coordinate systems are thus different from each other.

The coordinate system of the gesture is preferably moved along with the gesture. If this thus changes its position in space, ie in the earth-tracking system, the first coordinate system follows the gesture generator. A specific posture will thus always lead to the determination of the same coordinates of the gesture point, no matter where the gesture is in the grounding system. This is preferably particularly easy to realize when the origin of the first coordinate system is in the gesture generator.

In the same way, preferably the second coordinate system of the robot is moved with the robot, in particular here the origin of the second coordinate system is placed in the robot.

Each of the two reference points can also be given by the respective origin of the respective coordinate system. However, this is not mandatory.

Already hereby, the control according to the invention is very intuitive, because the gesture generator does not have to move into the robot.

The invention may provide that the coordinates of the gesture point in the first coordinate system are determined from the coordinates of at least one body extremity of the gesture sensor represented in the camera data. Thereby, e.g. the coordinates of the gesture point directly be the coordinates of a predetermined body extremity of the gesture generator.

The invention may e.g. provide at one or more body extremities of the gesture sensor to place optical markers that are detected with the at least one camera. The coordinates of such a marker, e.g. The coordinates of the gesture point can form on the hand. The coordinates of the gesture point can also be calculated from the coordinates of at least two markers.

The concrete determination of such coordinates of one or more markers from the camera data of at least one camera is known to the person skilled in the art and forms no essential subject of the invention.

Preferably, the invention provides that the coordinates of the gesture point in the first coordinate system are formed by the coordinates of the intersection of a direction vector defined by the posture of the gesture with the ground plane, preferably of the forearm direction of the gesture direction vector representing the ground plane on which the gesture in particular, wherein the reference point in the first coordinate system of the gesture transmitter is arranged in the same ground plane, preferably ventrally and in the sagittal plane of the gesture transmitter.

Such an embodiment is generally possible in the control of any mobile robot, but is particularly advantageous and intuitive in the control of ground moving robots, e.g. Cleaning robots, such as vacuum robots.

By the point of intersection of preferably the forearm direction with the ground plane, a gesture point in the ground plane is in fact arranged on the ground, to which the gesture giver points with his hand or his fingers. Thus, the direction of an index finger extended away from the hand at least substantially corresponds to the forearm direction, but due to the significantly longer Education of the forearm compared to the finger significantly easier or at least with higher accuracy in the camera data can be determined.

For example, It can provide the invention that the direction vector, in particular of the forearm direction is determined depending on the coordinates and / or angle of at least two body joints of the gesture transmitter. For this purpose, the coordinates / angles of the at least two body joints can be determined from the camera data. For example, the directional vector of the forearm direction can be determined as a function of the coordinates of the elbow joint and of a joint of the hand extremity, in particular of the wrist.

In principle, the invention can provide for the direction vector, in particular the forearm direction, to be determined by means of any evaluation of camera data known in the prior art.

On the other hand, the invention particularly preferably provides that the respective joint coordinates and / or joint angles of joint positions of the gesture transmitter and in particular of the elbow joint and the joint of the hand extremity are determined by the method of skeletal tracking on the basis of the data of at least one depth camera.

Such depth cameras are commercially available and provide as camera data directly a data stream which provides the necessary positions and / or angles of said joints and other joints of the body of the gesture, so that only the desired data of the joints used for determining the coordinate of the gesture point separates from the data stream and be used. As a result, the invention is particularly cost feasible.

Such a depth camera can e.g. comprise a projector, in particular infrared projector, is emitted with the pulsating light, wherein the sensor of the camera receives the reflected light from the gesture and based on the duration of the light for each pixel of the camera calculates a depth information to which an RGB image can be assigned , This allows a distance image of the gesture generator to be determined to the camera. From the distance data, the coordinates and / or angles of the joints of the skeleton of the gesture generator can be determined. By calculating this data in the camera infrastructure, external computers are advantageously relieved.

For example, such a depth camera or several thereof need only be positioned in space so that at least one depth camera detects the gesture and supplies the required joint data (coordinates and / or angles). Particularly preferably, at least one depth camera is arranged on the mobile robot. For example, For example, several cameras can be arranged at the same angular distance from each other on the peripheral periphery of a robot, in particular three cameras. This ensures that the gesture is always in the "field of vision" of at least one camera of the robot.

In general, the invention can provide for the evaluation of the camera data in an evaluation unit, e.g. one is external to the robot. The direction and speed data may be communicated to the robot to be controlled by communication, e.g. a radio communication. In the case of an arrangement of at least one camera on the robot, the entire evaluation and control of the robot can be carried out by an evaluation unit of the robot itself. In particular, a robot can carry at least its depth camera, so that the electronics of the robot only needs to evaluate their provided data stream.

The invention may further provide that the speed determined from the magnitude of the connection vector between the gesture point and the reference point is limited to a predetermined maximum value. Thus, a restriction to such maximum values can be made, which allow a control of the robot by the gesture with the desired accuracy.

Similarly, the speed determined from the magnitude of the connection vector can be increased to a predetermined minimum value. Thus, the robot - if it moves - has a minimum speed.

The invention will be explained in more detail with reference to FIG.

The 1 shows a gesture generator G who is on a ground level B gets up. On this example, also the mobile robot R drive, which is shown here for better clarity in supervision. By means of a depth camera K becomes the gesture G optically recorded. The data stream of the depth camera K is sent to an evaluation unit A transmitted. The data stream includes the coordinates and / or joint angles of the joints 1 to 20 of the gesture G , To determine this data of the data stream, a manufacturer-side software is running in the camera, which is not the subject of the invention.

In the evaluation unit is made of the coordinates and or joint angles of the joints 6 and 7 So here's the elbow joint and wrist a forearm direction vector UV certainly, the direction of the forearm between these joints 6 and 7 represents and points down to the ground level. The forearm direction vector UV thus cuts the bottom of life.

The calculation is carried out by an implemented software in the evaluation unit A , From the vector data of the forearm direction vector UV and stored ground level data will continue to be the coordinates of the gesture point through the software GP calculated, thus in the ground level B lies. This calculation is done with reference to a first coordinate system that the gesture provider G assigned.

The first coordinate system is still a reference point RP1 firmly defined with its coordinates, in particular thus stored in the evaluation unit. Also this reference point RP1 here lies preferably in the ground level B , The reference point RP1 can eg be ventral to the gesture generator G , that is to say defined in the direction of sight, eg in the sagittal plane, ie in the middle of the body in front of the gesture giver.

Deviating from the figure shows the arrangement of the reference point RP1 of the first coordinate system from the sagittal plane in the direction of the gesture giving forearm laterally offset. For example, the reference point RP1 also in the ground level B vertically below the shoulder joint 5 be arranged, which opens up the advantage that a vertically depending arm, as explained below, does not trigger movement of the robot.

To control the movement of the robot R now becomes from the known coordinates of the reference point RP1 and the variable determined by evaluation coordinates of the gesture point GP a connection vector V calculated, which is drawn here only for clarity separated, but practically in the ground plane B is at the reference point RP1 begins and in the gesture point GP ends.

The direction of the connection vector V now forms the direction in which the robot R in its own second coordinate system relative to its reference point defined therein RP2 to move. The reference point RP2 is placed here, for example, in the center of the robot R.

The drive motors of the robot R are controlled so that the robot R is moved at a speed, in particular, drives, the amount of the vector V depends. With increasing distance between the gesture point GP and the reference point RP1 in the first coordinate system of the gesture thus increases the speed.

In a vertically hanging forearm, there is no coordinate difference between the gesture point GP and reference point RP1 if this is below the shoulder joint 5 lies, so that the robot R not moved.

The control takes place here from those in the evaluation A calculated data for "direction" and "speed" in that this data through a radio link F to the robot R be transmitted and this implements the data in the movement. Such a communication path can be omitted if the at least one camera K from the robot R itself is worn.

By merely "pointing" to a point on the ground next to the reference point RP1 can thus be a gesture generator G cause the controlled robot to move in its own coordinate system toward a point that is the reference point RP2 in the same direction as the gesture point shown G to the reference point RP1 , The speed of the movement is dependent on the distance between the gesture point GP and reference point RP1 ,

This results in a very intuitive control of a mobile robot even by untrained people who act as a gesture.

In a preferred embodiment and with general validity for all embodiments, a trajectory of the robot which drives it can be defined by subsequently differently assumed postures, in particular forearm directions.

For example, in one possible embodiment, a program loop may be executed in a control software of the robot, which is a movement of the robot in the calculated direction relative to its reference point RP2 and at the calculated speed.

This loop is given new values for "direction" and "speed" by the evaluation unit with each pass. New values do not necessarily have to be changed values.

This causes a trajectory of the robot to be composed of the locations that the robot has approached with the given values of direction and velocity with each loop pass of its control software.

The spatial distance between such locations of a trajectory is variable because of the different speeds traveled between the locations, whereas the time interval between the locations is identical, provided that looping through this control software always requires the same time.

The work was made possible with the support of a scholarship as part of the DAAD's FIT worldwide program.

Claims (10)

Method for controlling the movement of a mobile robot (R) by a gesture of a gesture generator (G), wherein the gesture generator (G) is detected by means of at least one camera (K) and from the acquired camera data movement data for the robot (R) are formed , which are transmitted to the robot (R) and converted by it into a movement, characterized in that in a the gesture generator (G) associated first coordinate system, in particular that has its origin in the gesture generator (G), from the posture of the gesture (G) representative camera data, the coordinates of a body posture changing gesture point (GP) are determined and from the coordinates of the gesture point (GP) and the coordinates of a first coordinate system fixed reference point (RP1) a connection vector (V) between the gesture point (GP) and reference point (RP1) is calculated and moved in one of the robot (R) and with the robot (R) moves second coordinate system, in particular that has its origin in the robot (R), the robot (R) relative to a reference point (RP2) of the second coordinate system performs a movement whose direction starting from its reference point (RP2) the direction of the connection vector (V) corresponds and their speed depends on the amount of the connection vector (V). Method according to Claim 1 , characterized in that the coordinates of the gesture point (GP) in the first coordinate system are determined from the coordinates of at least one body extremity of the gesture transmitter (G) represented in the camera data. Method according to Claim 2 , characterized in that the coordinates of the gesture point (GP) are directly the coordinates of a predetermined body extremity. Method according to Claim 2 characterized in that the coordinates of the gesture point (GP) in the first coordinate system are formed by the coordinates of the intersection of a directional vector defined by the body posture of the gesture generator (G) with the ground plane (B), preferably the forearm direction of the gesture generator (G) Directional vector (UV) with the ground plane (B) on which the gesture generator (G) is, in particular wherein the reference point (RP1) in the first coordinate system of the gesture transmitter (G) in the same ground plane (B) is arranged, preferably ventral and in the sagittal plane of the gesture (G). Method according to Claim 4 , characterized in that the directional vector (UV) is determined as a function of the coordinates and / or angles of at least two body joints (6, 7) of the gesture transmitter (G). Method according to Claim 5 , characterized in that the directional vector (UV) of the forearm direction is determined as a function of the coordinates of the elbow joint (6) and a joint of the hand extremity (7, 8), in particular of the wrist (7). Method according to Claim 5 or 6 characterized in that the respective joint coordinates and / or joint angles are determined by the method of skeletal tracking based on the data of at least one depth camera (K). Method according to one of the preceding claims, characterized in that the speed determined from the magnitude of the connection vector (V) is limited to a predetermined maximum value. Method according to one of the preceding claims, characterized in that the speed determined from the magnitude of the connection vector (V) is increased to a predetermined minimum value. Procedure according to one of the previous Claims 4 to 9 , characterized in that it is used for controlling a floor-traveling robot (R), in particular a floor cleaning robot (R).

Images