CN115469576A - A Teleoperation System Based on Hybrid Mapping of Human-Robot Arm Heterogeneous Motion Space - Google Patents
A Teleoperation System Based on Hybrid Mapping of Human-Robot Arm Heterogeneous Motion Space Download PDFInfo
- Publication number
- CN115469576A CN115469576A CN202211059660.8A CN202211059660A CN115469576A CN 115469576 A CN115469576 A CN 115469576A CN 202211059660 A CN202211059660 A CN 202211059660A CN 115469576 A CN115469576 A CN 115469576A
- Authority
- CN
- China
- Prior art keywords
- human
- arm
- mapping
- mechanical arm
- joint
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B19/00—Programme-control systems
- G05B19/02—Programme-control systems electric
- G05B19/04—Programme control other than numerical control, i.e. in sequence controllers or logic controllers
- G05B19/042—Programme control other than numerical control, i.e. in sequence controllers or logic controllers using digital processors
- G05B19/0423—Input/output
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/20—Pc systems
- G05B2219/25—Pc structure of the system
- G05B2219/25257—Microcontroller
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Automation & Control Theory (AREA)
- Manipulator (AREA)
Abstract
本发明公开一种基于人‑机械臂异构运动空间混合映射的机械臂遥操作系统,使用相机识别操作人员的关节位置信息并进行数据处理,得到操作人员的部分关节角度以及手臂末端坐标位姿,将人手臂末端位姿映射到机械臂末端,并对机械臂进行逆运动学求解获得机械臂的期望关节角度,针对多解和解不存在的情况,参考操作人员的关节角度进行调整,并根据期望关节角对机械臂进行运动规划,控制真实机械臂运动,使机械臂尽可能地模仿人手臂姿态运动。
The invention discloses a manipulator teleoperating system based on hybrid mapping of human-manipulator heterogeneous motion space, which uses a camera to identify the joint position information of the operator and performs data processing to obtain part of the joint angles of the operator and the coordinate pose of the end of the arm , map the pose of the end of the human arm to the end of the manipulator, and solve the inverse kinematics of the manipulator to obtain the expected joint angle of the manipulator. For the situation where there are multiple solutions or no solution exists, adjust the joint angle of the operator with reference to the joint angle of the operator, and according to The joint angle is expected to plan the motion of the robotic arm, control the motion of the real robotic arm, and make the robotic arm mimic the posture of the human arm as much as possible.
Description
技术领域technical field
本发明属于人机交互领域,是一种基于光学姿态捕捉和姿态映射远程控制机器人的方法。The invention belongs to the field of human-computer interaction, and relates to a method for remotely controlling a robot based on optical attitude capture and attitude mapping.
背景技术Background technique
受限于当前机器人智能水平,相当长一段时间内,研制出完全自主的机器人仍是遥不可及的目标,特别是在核辐射、空间、深海等危险、远距离或非结构环境下。将人类的高级决策及灵活操作能力与机器人进行有机结合,发展人在回路的精细遥操作技术是未来相当长一段时间内的必然选择。Limited by the current level of robot intelligence, the development of a fully autonomous robot is still an unattainable goal for a long time, especially in dangerous, long-distance or unstructured environments such as nuclear radiation, space, and deep sea. Organically combining human's advanced decision-making and flexible operation capabilities with robots, and developing human-in-the-loop fine teleoperation technology will be an inevitable choice for a long time in the future.
虽然遥操作机器人领域已经取得了巨大进展,甚至其中一些已经商业化,得以应用于实际,但操作人员指挥从端机器人的方式并没有太大的改变。一般来说,操作者在对从端机器人在远程环境中的行为进行视觉监控的同时,直接向从机器人发出特定的指令,称之为直接遥操作,它经常给操作者带来沉重的认知负担。除此之外,操作者与遥控机器人之间通信的时延和带宽限制一直是制约遥操作系统性能的主要因素;在使用传统的遥操作系统时,操作者的2种重要感知(视觉和力觉)被剥夺,严重影响操作者完成遥操作任务的能力。尤其是面对非结构环境、突发的事故时,人类操作者不能够及时处理问题,会降低任务成功率,甚至给用户带来人身安全隐患,也会使得操作者处于高度紧张状态,易产生精神疲劳,降低工作效率。Although the field of teleoperated robots has made great progress, and some of them have even been commercialized and put into practical use, the way operators command slave robots has not changed much. Generally speaking, the operator directly sends specific instructions to the slave robot while visually monitoring the behavior of the slave robot in the remote environment, which is called direct teleoperation, which often brings heavy cognitive burden to the operator. burden. In addition, the delay and bandwidth limitation of the communication between the operator and the remote robot have been the main factors restricting the performance of the teleoperation system; when using the traditional teleoperation system, the operator's two important perceptions (vision and force) sense) is deprived, seriously affecting the operator's ability to complete teleoperation tasks. Especially in the face of unstructured environments and unexpected accidents, human operators cannot deal with problems in time, which will reduce the success rate of tasks, and even bring hidden dangers to users' personal safety, and will also make operators in a state of high tension and prone to accidents. Mental fatigue reduces work efficiency.
为了解决上述问题,科研人员进行了许多研究,大致可以归为2类:其一,结合基于视觉、触觉、肌肉神经等感知信号的人机交互技术的特殊性能,能提高遥操作机器人系统的工作效率和控制精度;其二,针对性地研究开发人机遥操作控制模式,充分发挥人类操作者与机器人各自的优势,在缓解人类操作者疲倦的同时提高系统的灵敏度、精确度,实现人机任务合理配置。In order to solve the above problems, researchers have conducted a lot of research, which can be roughly classified into two categories: First, combining the special performance of human-computer interaction technology based on visual, tactile, muscle nerve and other sensory signals can improve the work of teleoperated robot systems. Efficiency and control accuracy; Second, research and develop man-machine remote operation control mode in a targeted manner, give full play to the respective advantages of human operators and robots, improve the sensitivity and accuracy of the system while alleviating the fatigue of human operators, and realize man-machine Tasks are properly configured.
要对机器人进行遥操作,需要生成运动指令,基于光学的姿态捕捉技术是所有解决方案中最不受物理约束的技术。商用的基于标记的产品,如OptiTrack,旨在通过跟踪反射标记来捕捉姿势[Spanlang B,Navarro X,Normand J M,Kishore S,Pizarro R,SlaterM.Real time whole body motion mapping for avatars and robots[A].In:the 19thACM Symposium on Virtual Reality Software and Technology[C].Singapore:Association for Computing Machinery,2013:175-178.]。然而,这种方法由耗时的标记过程组成,操作符要正确地使用所有标记,这使得它的应用没有那么广泛。例如,无标记动作捕捉系统Capture Live承诺提供更好的便利性,但设备和许可证的较高价格也极大地限制了它们的普及。此外,这些商业方法都要求操作者在被摄像机阵列包围的有限室内工作空间进行操作,导致环境灵活性较低。如今,便携式深度检测设备如著名的微软Kinect、Intel RealSense和LeapMotion变得越来越可行,研究人员已经使得减少动作捕捉摄像机的数量(甚至只有一台)可以实现,并极大地简化了动作捕捉过程。To teleoperate a robot, motion commands need to be generated, and optical-based pose capture is the least physically constrained of all solutions. Commercial marker-based products, such as OptiTrack, aim to capture poses by tracking reflective markers [Spanlang B, Navarro X, Normand J M, Kishore S, Pizarro R, Slater M. Real time whole body motion mapping for avatars and robots [A] .In:the 19thACM Symposium on Virtual Reality Software and Technology[C].Singapore:Association for Computing Machinery,2013:175-178.]. However, this approach consists of a time-consuming tagging process for operators to use all tags correctly, which makes it less widely applicable. For example, the markerless motion capture system Capture Live promises better convenience, but the high price of equipment and licenses has also greatly limited their popularity. In addition, these commercial methods all require operators to operate in a limited indoor workspace surrounded by camera arrays, resulting in less environmental flexibility. Today, portable depth detection devices such as the famous Microsoft Kinect, Intel RealSense, and LeapMotion are becoming more and more feasible. Researchers have made it possible to reduce the number of motion capture cameras (or even just one) and greatly simplify the motion capture process. .
人-机器人姿态映射,即将人的姿态转换为机器人的姿态,弥合了人-机器人运动之间的最后差距。Luo等人[Luo R C,Shih B H,Lin T W.Real time human motionimitation of anthropomorphic dual arm robot based on Cartesian impedancecontrol[A].In:IEEE International Symposium on Robotic and SensorsEnvironments[C].Piscataway:IEEE,2013:25-30.]使用避免逆运动学计算过程的笛卡儿空间阻抗控制实现了对6自由度机械臂的在线示教。然而,笛卡儿空间的运动映射只关注人机末端位姿的同步,机械臂肩部和肘部关节的运动轨迹另需人为规划。Nguyen等人[NguyenV V,Lee J H,et al.Full-body imitation of human motions with Kinect andheterogeneous kinematic structure of humanoid robot[A].In:IEEE/SICEInternational Symposium on System Integration[C].Piscataway:IEEE,2012:93-98.]首先建立人臂与3自由度机械臂的关节空间映射关系,然后使用空间向量法实现了对人臂关节角度的求解并传递给机械臂作为控制指令实现实时的运动模仿。吴伟国等人[吴伟国,栗华,高力扬.人体步行捕捉下的双足机器人跟随步行与实验[J].哈尔滨工业大学学报,2017,49(1):21-29.]对人类下肢进行6自由度逆运动学计算后建立步行样本,然后使用运动相似性指标进行样本拼接来驱动机器人实现步行跟随。Human-robot pose mapping, i.e. converting human poses to robot poses, bridges the last gap between human-robot motion. Luo et al [Luo R C, Shih B H, Lin T W.Real time human motionimitation of anthropomorphic dual arm robot based on Cartesian impedance control[A].In:IEEE International Symposium on Robotic and SensorsEnvironments[C].Piscataway:IEEE,2013: 25-30.] Online teaching of a 6-DOF manipulator is realized using Cartesian spatial impedance control that avoids the inverse kinematics calculation process. However, the motion mapping in Cartesian space only focuses on the synchronization of the end pose of the man-machine, and the motion trajectories of the shoulder and elbow joints of the robotic arm need to be planned manually. Nguyen et al. [NguyenV V, Lee J H, et al.Full-body imitation of human motions with Kinect and heterogeneous kinematic structure of humanoid robot[A].In:IEEE/SICEInternational Symposium on System Integration[C].Piscataway:IEEE,2012 :93-98.] First establish the joint space mapping relationship between the human arm and the 3-DOF robotic arm, and then use the space vector method to solve the joint angle of the human arm and pass it to the robotic arm as a control command to realize real-time motion simulation. Wu Weiguo et al [Wu Weiguo, Li Hua, Gao Liyang. Biped robot following walking and experiment captured by human walking[J]. Journal of Harbin Institute of Technology, 2017,49(1):21-29.] Performed six-freedom human lower limbs After the degree inverse kinematics calculation, the walking samples are established, and then the motion similarity index is used for sample splicing to drive the robot to achieve walking follow.
在实际的基于视觉的遥操作过程中,由于摄像机噪声、人体关节识别的局限性以及人体骨骼识别的误差,使用单一的映射方法很难保证非拟人机器人能够很好地模仿人体手臂的姿态。末端位姿映射可以使机器人末端很好地跟随人的手腕,但存在多个逆运动学解和无解的问题。关节角度映射可以使机器人更好地模仿人臂的姿态,但是人臂的关节数量与机器人的关节数量并不相等,而且人臂的一些关节角度很难测量。In the actual vision-based teleoperation process, due to camera noise, limitations in human joint recognition, and errors in human bone recognition, it is difficult to ensure that non-anthropomorphic robots can well imitate human arm poses using a single mapping method. The end pose mapping can make the end of the robot follow the human wrist well, but there are many inverse kinematics solutions and no solutions. Joint angle mapping can enable the robot to better imitate the posture of the human arm, but the number of joints in the human arm is not equal to the number of joints in the robot, and some joint angles of the human arm are difficult to measure.
发明内容Contents of the invention
针对上述问题,本发明提出基于人-机械臂异构运动空间混合映射的遥操作系统,对人体关节进行识别并将其映射到机械臂,减少了对人体运动的限制,并使机械臂可以做出与人手臂相似的姿态,在遥操作中实现了较好的效果。In view of the above problems, the present invention proposes a teleoperation system based on the hybrid mapping of human-robot heterogeneous motion space, which can identify the joints of the human body and map them to the robotic arm, which reduces the restrictions on human motion and enables the robotic arm to do A gesture similar to that of a human arm has been achieved, and better results have been achieved in teleoperation.
本发明基于人-机械臂异构运动空间混合映射的遥操作系统,分为两大模块,分别为人-机械臂运动映射模块与机械臂运动规划和控制系统。The present invention is based on the teleoperation system of human-manipulator arm heterogeneous motion space hybrid mapping, which is divided into two modules, namely, the human-manipulator arm motion mapping module and the manipulator arm motion planning and control system.
其中人-机械臂运动映射模块构,建了人体关节角度及位置姿态识别系统,采用Kinect相机的人体关节识别系统获取人体关节位置数据并绘制人体骨架,对人体关节位置数据进行卡尔曼滤波,再计算输出人手臂末端位姿以及关节角度。Among them, the human-robot motion mapping module is constructed, and the human joint angle, position and posture recognition system is built. The human joint position recognition system of the Kinect camera is used to obtain the human joint position data and draw the human skeleton, and the Kalman filter is performed on the human joint position data, and then Calculate and output the pose and joint angle of the end of the human arm.
并采用人机运动空间混合映射方法,先将人手臂末端位姿映射到机械臂的末端,对机械臂进行逆运动学求解;进一步加入关节映射,获得机械臂关节角度最优解。And using the human-machine motion space hybrid mapping method, first map the end pose of the human arm to the end of the robotic arm, and then solve the inverse kinematics of the robotic arm; further add joint mapping to obtain the optimal solution for the joint angle of the robotic arm.
机械臂运动规划和控制系统使用socket建立了Windows与ROS的通信传输,将期望关节角度数据发布到ROS节点,并使用Moveit订阅并进行运动规划,最终将控制信息传输给真实机械臂,控制真实机械臂进行运动。The robot arm motion planning and control system uses socket to establish the communication transmission between Windows and ROS, publishes the expected joint angle data to the ROS node, and uses Moveit to subscribe and perform motion planning, and finally transmits the control information to the real robot arm to control the real machine arm movement.
本发明的优点:Advantages of the present invention:
1、本发明基于人-机械臂异构运动空间混合映射的遥操作系统,采用无标记动作识别,对操作人员的运动约束较小,且采集人体关节数据仅需要一台Kinect相机,硬件配置要求低。1. The present invention is based on the teleoperation system based on the hybrid mapping of human-robot heterogeneous motion space, which adopts unmarked motion recognition, which has less movement constraints on the operator, and only needs a Kinect camera to collect human joint data, and the hardware configuration requirements Low.
2、本发明基于人-机械臂异构运动空间混合映射的遥操作系统,解决了末端姿态映射导致机械臂逆运动学多解和解不存在的问题。2. The present invention is based on the teleoperation system of human-manipulator arm heterogeneous motion space hybrid mapping, which solves the problem of multiple solutions and non-existence of the inverse kinematics of the manipulator caused by the terminal attitude mapping.
3、本发明基于人-机械臂异构运动空间混合映射的遥操作系统,仅根据Kinect相机采集人体关节数据,无法计算出人手臂的所有关节角度,采用混合映射方法避免了这一问题,仅需测量三个关节角度。3. The present invention is based on the teleoperation system of human-manipulator arm heterogeneous motion space hybrid mapping. Only according to the Kinect camera to collect human joint data, it is impossible to calculate all the joint angles of the human arm. The hybrid mapping method is used to avoid this problem. Three joint angles need to be measured.
4、本发明基于人-机械臂异构运动空间混合映射的遥操作系统,机械臂可以较好地模仿人手臂的姿态,做出与人手臂姿态类似的动作,操作人员更容易控制其完成操作任务。4. The present invention is based on the teleoperation system based on the hybrid mapping of human-robot heterogeneous motion space. The robot arm can better imitate the posture of the human arm and make actions similar to the posture of the human arm. It is easier for the operator to control it to complete the operation Task.
附图说明Description of drawings
图1为本发明基于人-机械臂异构运动空间混合映射的遥操作系统整体框图;Fig. 1 is the overall block diagram of the teleoperation system based on the mixed mapping of human-robot heterogeneous motion space in the present invention;
图2为人类手臂的运动学模型;Fig. 2 is the kinematics model of human arm;
图3为ur5机械臂的运动学模型;Figure 3 is the kinematics model of the ur5 robotic arm;
图4为连杆之间的参数变换图;Fig. 4 is a parameter conversion diagram between connecting rods;
图5为25个相对于摄像机坐标的三维关节位置;Figure 5 shows 25 three-dimensional joint positions relative to the camera coordinates;
图6为人体右臂模型。Figure 6 is a human right arm model.
具体实施方式detailed description
下面结合附图对本发明做进一步详细说明。The present invention will be described in further detail below in conjunction with the accompanying drawings.
本发明基于人-机械臂异构运动空间混合映射的遥操作系统,如图1所示,分为两大模块,分别为人-机械臂运动映射模块与机械臂运动规划。The present invention is based on the teleoperation system of human-robot heterogeneous motion space hybrid mapping, as shown in Figure 1, which is divided into two modules, namely, the human-robot motion mapping module and the robot motion planning.
其中,人-机械臂运动映射模块采用Kinect相机获取人体关节信息,并绘制人体骨架;使用任意的六自由度机械臂作为遥操作控制的对象,将人手臂末端的位姿以及人手臂各个关节角度分别映射机械臂末端及机械臂各个关节,两者的映射组合到一起,形成混合映射方法,最终得到机械臂期望关节角度。Among them, the human-robot motion mapping module uses the Kinect camera to obtain human joint information, and draws the human skeleton; uses any six-degree-of-freedom robotic arm as the object of teleoperation control, and maps the pose of the end of the human arm and the angles of each joint of the human arm The end of the manipulator and each joint of the manipulator are mapped separately, and the mappings of the two are combined to form a hybrid mapping method, and finally the desired joint angle of the manipulator is obtained.
机械臂运动规划和控制模块通过ROS机器人控制系统来实现,根据机械臂期望关节角度,在Moveit中对其进行运动规划,并在Gazebo中仿真,并控制真实机械臂运动。The motion planning and control module of the manipulator is realized by the ROS robot control system. According to the expected joint angle of the manipulator, it is planned in Moveit, simulated in Gazebo, and controls the real manipulator movement.
所述人-机械臂运动映射模块中建立运动学模型,该运动学模型为手臂末端位姿和关节角度映射基础,具体建立方法采用D-H法,在机器人运动学建模的基础上,建立了5-DOF手臂模型和UR5机器人模型两个运动学模型;其中,5-DOF手臂模型包括肩部3-DOF和肘部2-DOF;Ur5机器人模型为6-DOF机械臂;可见两个模型间为异构模型,如图2和图3所示。The kinematics model is established in the human-robot motion mapping module. The kinematics model is the basis for mapping the pose and joint angle of the end of the arm. The specific establishment method adopts the D-H method. On the basis of the kinematic modeling of the robot, 5 Two kinematic models of -DOF arm model and UR5 robot model; among them, the 5-DOF arm model includes shoulder 3-DOF and elbow 2-DOF; Ur5 robot model is a 6-DOF mechanical arm; it can be seen that the two models are Heterogeneous models, as shown in Figure 2 and Figure 3.
D-H参数定义如图4所示,图4中zi与zi-1分别与机械臂第i与第i-1个关节转轴重合,xi-1沿第i-1个关节轴和第i个关节轴的公垂线ai-1方向;xi沿第i个关节轴和第i个关节轴的公垂线ai方向;αi-1为zi-1和zi之间的夹角,ai-1沿着xi-1测量的从zi-1到zi的连接长度,其中xi-1垂直于zi-1和zi;di为xi-1到xi沿zi测量的链接偏移量;θi为xi-1和xi之间的夹角。The definition of DH parameters is shown in Figure 4. In Figure 4, z i and z i-1 coincide with the rotation axes of the i-th and i-1 joints of the manipulator respectively, and x i-1 is along the i-1 joint axis and the i-th joint axis. The direction of the common vertical line a i-1 of the joint axis; x i is along the i-th joint axis and the direction of the common vertical line a i of the i-th joint axis; α i-1 is the distance between z i-1 and z i Included angle, a i-1 is the connection length from z i-1 to z i measured along x i-1 , where x i-1 is perpendicular to z i-1 and z i ; d i is x i-1 to z i The link offset of xi measured along zi ; θi is the angle between xi -1 and xi .
根据D-H法,从机械臂第i-1个关节坐标系到第i个关节坐标系的坐标变换描述如式(1)所示。According to the D-H method, the coordinate transformation description from the i-1 joint coordinate system of the manipulator to the i joint coordinate system is shown in formula (1).
进一步可由式(1)推导得到人体手臂从基本坐标系到终端坐标系的坐标变换矩阵 Further, the coordinate transformation matrix of the human arm from the basic coordinate system to the terminal coordinate system can be derived from formula (1)
同理,得到式(3)中的6自由度机械臂的变换矩阵为 Similarly, the transformation matrix of the 6-DOF manipulator in formula (3) is obtained as
上式中,[nx,ny,nz;ox,oy,oz;ax,ay,az]与[px py pz]分别表示末端执行器的姿态和位置。In the above formula, [n x , n y , n z ; o x , o y , o z ; a x , a y , a z ] and [p x p y p z ] denote the posture and position of the end effector, respectively .
所述人-机械臂运动映射模中Kinect相机获取人体关节数据并进行滤波处理的方法为:The Kinect camera in the described human-mechanical arm motion mapping model acquires the human body joint data and performs filtering processing as follows:
通过Kinect直接获取人体关节数据,其中包含25个相对于摄像机坐标的三维关节位置,如图5所示,包括:脊柱根基、脊柱中段、脖子、头部、右肩、右肘、右手腕、右手掌、左肩、左肘、左手腕、左手掌、右髋、右膝盖、右脚踝、右脚、左髋、左膝盖、左脚踝、左脚、脊柱与肩的交点、右手尖、右拇指、左手尖、左拇指。Human body joint data is obtained directly through Kinect, which contains 25 three-dimensional joint positions relative to the camera coordinates, as shown in Figure 5, including: spine foundation, mid-spine, neck, head, right shoulder, right elbow, right wrist, right Palm, left shoulder, left elbow, left wrist, left palm, right hip, right knee, right ankle, right foot, left hip, left knee, left ankle, left foot, intersection of spine and shoulder, right tip, right thumb, left hand Pointer, left thumb.
当人体保持静止时,Kinect采集的关节点的位置不是恒定的,而是在实际值附近波动;其有两个原因,一是静止的人体并不是真正静止的;二是Kinect和身体跟踪方法收集的数据有误差。同时凸起的关节点的坐标会影响对人腕部坐标的描述。为了提高系统的准确性,在使用数据前需要对采集到的数据通过卡尔曼滤波进行数据过滤,如下:When the human body remains still, the position of the joint points collected by Kinect is not constant, but fluctuates around the actual value; there are two reasons for this, one is that the static human body is not really static; the other is that Kinect and body tracking methods collect data has errors. At the same time, the coordinates of the raised joint points will affect the description of the coordinates of the human wrist. In order to improve the accuracy of the system, it is necessary to filter the collected data through Kalman filtering before using the data, as follows:
xk=Akxk-1+Bkuk+wk zk=Hkxk+vk (4)x k =A k x k-1 +B k u k +w k z k =H k x k +v k (4)
xk是状态向量,uk是状态控制向量,zk是测量向量。Ak是状态转移矩阵,Bk是控制输入矩阵,Hk是状态观测矩阵。wk代表过程噪声,vk代表测量噪声。wk和vk服从高斯分布,其协方差矩阵分别为Q和R,即wk~N(0,Q),vk~N(0,R)。x k is the state vector, u k is the state control vector, z k is the measurement vector. A k is the state transition matrix, B k is the control input matrix, and H k is the state observation matrix. w k represents the process noise and v k represents the measurement noise. w k and v k obey Gaussian distribution, and their covariance matrices are Q and R respectively, that is, w k ~N(0,Q), v k ~N(0,R).
所述人-机械臂运动映射模中在采集到人体关节数据后,对人手臂末端位姿及关节角度进行计算,方法如下:In the human-mechanical arm motion mapping module, after collecting the data of human joints, the pose and joint angle of the end of the human arm are calculated, and the method is as follows:
当人体与地面不垂直时,人体手臂的基础坐标会出错。为了解决这一问题,本发明中建立了人体右臂的几何模型,如图5所示。该模型有三个坐标系统,分别为肩部坐标系统手腕坐标系统和Kinect坐标 需要左肩、右肩和脊柱的平面来构建坐标系,图中A点为左肩,G点为脊柱中段,O点为右肩,C点为右手腕,D点、E点、F点分别为右拇指、右手与右手指尖;上述各点均为kinect相机直接测量获得,与相机识别的25个点相对应。When the human body is not perpendicular to the ground, the basic coordinates of the human arm will be wrong. In order to solve this problem, the geometric model of the right arm of the human body is established in the present invention, as shown in FIG. 5 . The model has three coordinate systems, namely the shoulder coordinate system Wrist Coordinate System and Kinect coordinates The planes of the left shoulder, right shoulder and spine are needed to construct the coordinate system. In the figure, point A is the left shoulder, point G is the middle of the spine, point O is the right shoulder, point C is the right wrist, and points D, E, and F are the right Thumb, right hand and right fingertips; the above points are directly measured by the kinect camera, corresponding to the 25 points recognized by the camera.
令上述点O为基础坐标的原点,根据上述建立的人体右臂的几何模型,则:Let the above point O be the origin of the basic coordinates, and according to the geometric model of the right arm of the human body established above, then:
①人手臂末端位姿计算方法为:① The calculation method of the end pose of the human arm is:
与向量方向相同,与平面AGO的法向量相同。因此,是由右手坐标系的规定得到的,如下: with vector same direction, Same as normal vector for planar AGO. therefore, It is obtained by the regulation of the right-handed coordinate system, as follows:
点C为末端执行器(即右手腕)坐标的原点。因此,腕关节的位置为:Point C is the origin of the coordinates of the end effector (ie, the right wrist). Therefore, the position of the wrist joint is:
用表示平面DEF的法向量,与向量方向相同。和都是由右手得到的,如下:use represents the normal vector of the plane DEF, with vector same direction. with are obtained by the right hand, as follows:
②人手臂关节角度计算方法② Calculation method of human arm joint angle
如图6所示,大臂与YOZ平面的夹角θ1是通过计算向量在YOZ平面上的投影与的夹角获得的,大臂与小臂的夹角θ2是向量与的夹角,手腕的夹角θ3用与的夹角表示。As shown in Figure 6, the angle θ 1 between the boom and the YOZ plane is obtained by calculating the vector The projection on the YOZ plane is the same as The included angle is obtained, and the included angle θ 2 between the arm and the forearm is a vector and The included angle, the included angle of the wrist θ 3 is used and angle representation.
通过上述得到人手臂末端位姿及关节角度后,即可将人手臂末端的位姿以及人手臂各个关节角度分别映射机械臂末端及机械臂各个关节,具体映射过程如下:After obtaining the pose and joint angle of the end of the human arm through the above, the pose of the end of the human arm and the angles of each joint of the human arm can be mapped to the end of the manipulator and each joint of the manipulator. The specific mapping process is as follows:
Ⅰ、进行末端姿态映射Ⅰ. Perform terminal attitude mapping
采用点对点方法将人手臂末端位置映射到机器人的末端执行器位置,结合式(2)与式(3)得到的两个变换矩阵,通过逆运动学求解获得机械臂各个关节角度。映射公式如下:The point-to-point method is used to map the position of the end of the human arm to the position of the end effector of the robot. Combining the two transformation matrices obtained by formula (2) and formula (3), the angles of each joint of the manipulator are obtained by solving the inverse kinematics. The mapping formula is as follows:
其中,[xr,yr,zr]T是机器人末端执行器的位置;[xh,yh,zh]T是人类手臂末端执行器位置;kx、ky、kz是比例系数;lx、ly、lz是坐标的平移距离;kx、ky、kz、lx、ly、lz根据人手臂和机械臂的实际尺寸计算得到。where [x r , y r , z r ] T is the robot end effector position; [x h , y h , z h ] T is the human arm end effector position; k x , ky , k z are the ratio Coefficients; l x , ly , l z are translational distances of the coordinates; k x , ky , k z , l x , ly , l z are calculated according to the actual size of the human arm and the mechanical arm.
一般情况下,机械臂逆运动学求解会存在多个解,本发明针对最优解选择提出最优解应满足:In general, there will be multiple solutions for the inverse kinematics solution of the manipulator. The present invention proposes that the optimal solution should satisfy the following requirements for optimal solution selection:
1)为了保证机械臂运动的连续性和稳定性,当前时刻的第1个关节角度值(即机械臂基座旋转角度)选择与上一时刻差值最小的;1) In order to ensure the continuity and stability of the movement of the manipulator, the angle value of the first joint at the current moment (that is, the rotation angle of the manipulator base) is selected to have the smallest difference from the previous moment;
2)为了使机械臂做出与人手臂相似的姿态,在1)的基础上,选择与θ1差值最小的第2个关节角度值。按照以上方式最终可确定一组最优解。2) In order to make the robot arm make a posture similar to that of the human arm, on the basis of 1 ), select the second joint angle value with the smallest difference from θ1. According to the above method, a group of optimal solutions can be finally determined.
Ⅱ、加入关节角度映射Ⅱ. Add joint angle mapping
由于Kinect相机识别精度有限,存在两个问题:1)逆运动学解不存在;2)机械臂的姿态与期望的姿态存在一定的差距。Due to the limited recognition accuracy of the Kinect camera, there are two problems: 1) the inverse kinematics solution does not exist; 2) there is a certain gap between the pose of the robot arm and the expected pose.
针对逆运动学解不存在的情况,机械臂的第2~4个关节角度值无法求解,可将人手臂的关节角度θ1、θ2、θ3直接映射到机械臂的第2~4个关节。In the case that the inverse kinematics solution does not exist, the angle values of the 2nd to 4th joints of the manipulator cannot be solved, and the joint angles θ 1 , θ 2 , θ 3 of the human arm can be directly mapped to the 2nd to 4th joint angles of the manipulator. joint.
针对姿态不一致的问题,加入关节角度映射,将前述得到的最优解中的第2、3个关节角度值改为θ1、θ2。Aiming at the problem of inconsistency in attitude, joint angle mapping is added, and the second and third joint angle values in the optimal solution obtained above are changed to θ 1 and θ 2 .
添加关节角度映射之后的控制效果明显优于单纯的末端位姿映射,由此实现了人-机械臂异构运动空间的混合映射。The control effect after adding the joint angle mapping is significantly better than the simple end pose mapping, thus realizing the hybrid mapping of the heterogeneous motion space of the human-robot arm.
所述机械臂运动规划和控制模块采用socket实现Windows端与Linux端ROS系统的通信,将通过前述方法的到的最终期望关节角度值发布到ROS节点上,在ROS端订阅话题消息,为了减小时延,每次都订阅话题节点上最新的消息。Moveit是ROS中一系列移动操作的功能包的组成,主要包含运动规划,碰撞检测,运动学,3D感知,操作控制等功能,可以与真实机械臂建立通信。因此用Moveit对机械臂进行正向运动学轨迹规划,最终实现对真实机械臂的控制。Described manipulator motion planning and control module adopts socket to realize the communication of Windows end and Linux end ROS system, publishes the final expected joint angle value obtained by the aforementioned method to the ROS node, subscribes topic message at ROS end, in order to reduce time Delay, subscribe to the latest news on the topic node every time. Moveit is a function package of a series of mobile operations in ROS, mainly including motion planning, collision detection, kinematics, 3D perception, operation control and other functions, which can establish communication with real robotic arms. Therefore, Moveit is used to plan the forward kinematics trajectory of the manipulator, and finally realize the control of the real manipulator.
基于上述遥操作系统,本发明提出遥操作控制具体流程如下:Based on the above-mentioned teleoperation system, the present invention proposes the specific flow of teleoperation control as follows:
A、获取由Kinect摄像头采集的人体关节坐标;A, obtain the human body joint coordinates collected by the Kinect camera;
B、对人体关节数据进行滤波,并计算人手臂末端位姿及关节角度;B. Filter the human joint data, and calculate the pose and joint angle of the end of the human arm;
C、根据人手臂和机械臂的真实长度计算映射参数kx、ky、kz、lx、ly、lz,并进行末端位姿映射;C. Calculate the mapping parameters k x , ky , k z , l x , ly , l z according to the real length of the human arm and the robotic arm, and perform terminal pose mapping;
D、根据机械臂的D-H参数构建逆运动学求解函数;D. Construct an inverse kinematics solution function according to the D-H parameters of the mechanical arm;
E、根据最优解选择原则选出最优解;E. Select the optimal solution according to the optimal solution selection principle;
F、添加关节角度映射;F. Add joint angle mapping;
G、实现通信,将期望关节角度发布到ROS节点上;G. Realize communication and publish the desired joint angle to the ROS node;
H、订阅话题消息并进行处理,用Moveit进行运动规划,并在Gazebo中进行仿真;H. Subscribe and process topic messages, use Moveit for motion planning, and simulate in Gazebo;
G、操作者正对Kinect相机做出动作,机械臂跟随操作者右手臂的运动做出相应的姿态,实时交互完成遥操作任务。G. The operator is making an action on the Kinect camera, and the robotic arm follows the movement of the operator's right arm to make a corresponding gesture, and interacts in real time to complete the teleoperation task.
Claims (7)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211059660.8A CN115469576B (en) | 2022-09-01 | 2022-09-01 | A teleoperation system based on hybrid mapping of human-robot heterogeneous motion space |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211059660.8A CN115469576B (en) | 2022-09-01 | 2022-09-01 | A teleoperation system based on hybrid mapping of human-robot heterogeneous motion space |
Publications (2)
Publication Number | Publication Date |
---|---|
CN115469576A true CN115469576A (en) | 2022-12-13 |
CN115469576B CN115469576B (en) | 2024-06-18 |
Family
ID=84369347
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202211059660.8A Active CN115469576B (en) | 2022-09-01 | 2022-09-01 | A teleoperation system based on hybrid mapping of human-robot heterogeneous motion space |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN115469576B (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116214549A (en) * | 2023-01-19 | 2023-06-06 | 中国科学院微小卫星创新研究院 | Teleoperation system and teleoperation method for space robot |
CN116602772A (en) * | 2023-05-19 | 2023-08-18 | 哈尔滨思哲睿智能医疗设备股份有限公司 | Surgical robot, control method, device and medium for slave-end surgical arm of surgical robot |
CN116619422A (en) * | 2023-06-21 | 2023-08-22 | 哈尔滨理工大学 | Motion capture and wrist joint angle estimation mechanical arm mapping method based on human body characteristic signals |
CN116979419A (en) * | 2023-09-12 | 2023-10-31 | 国网江苏省电力有限公司常州供电分公司 | Bypass cable traction system and method |
CN117921677A (en) * | 2024-03-06 | 2024-04-26 | 睿尔曼智能科技(北京)有限公司 | Mechanical arm posture following control method and device |
CN118046394A (en) * | 2024-04-15 | 2024-05-17 | 帕西尼感知科技(张家港)有限公司 | A remote motion control method, device, system and storage medium |
WO2025043853A1 (en) * | 2023-08-28 | 2025-03-06 | 东南大学 | Safety shared control system based on performance of teleoperator |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108638069A (en) * | 2018-05-18 | 2018-10-12 | 南昌大学 | A kind of mechanical arm tail end precise motion control method |
CN108828996A (en) * | 2018-05-31 | 2018-11-16 | 四川文理学院 | A kind of the mechanical arm remote control system and method for view-based access control model information |
US20210197384A1 (en) * | 2019-12-26 | 2021-07-01 | Ubtech Robotics Corp Ltd | Robot control method and apparatus and robot using the same |
CN113110142A (en) * | 2021-03-29 | 2021-07-13 | 航天时代电子技术股份有限公司 | Follow-up remote control operation table and remote control method thereof |
US20220203526A1 (en) * | 2020-12-30 | 2022-06-30 | Ubtech Robotics Corp Ltd | Humanoid robot control method, humanoid robot using the same, and computer readable storage medium |
-
2022
- 2022-09-01 CN CN202211059660.8A patent/CN115469576B/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108638069A (en) * | 2018-05-18 | 2018-10-12 | 南昌大学 | A kind of mechanical arm tail end precise motion control method |
CN108828996A (en) * | 2018-05-31 | 2018-11-16 | 四川文理学院 | A kind of the mechanical arm remote control system and method for view-based access control model information |
US20210197384A1 (en) * | 2019-12-26 | 2021-07-01 | Ubtech Robotics Corp Ltd | Robot control method and apparatus and robot using the same |
US20220203526A1 (en) * | 2020-12-30 | 2022-06-30 | Ubtech Robotics Corp Ltd | Humanoid robot control method, humanoid robot using the same, and computer readable storage medium |
CN113110142A (en) * | 2021-03-29 | 2021-07-13 | 航天时代电子技术股份有限公司 | Follow-up remote control operation table and remote control method thereof |
Non-Patent Citations (1)
Title |
---|
赵京;王鑫;张自强;龚世秋;李晓辉;: "基于肘部自运动的主从异构7自由度机械臂运动映射及其几何逆解", 机械工程学报, no. 15, 5 August 2020 (2020-08-05), pages 195 - 204 * |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116214549A (en) * | 2023-01-19 | 2023-06-06 | 中国科学院微小卫星创新研究院 | Teleoperation system and teleoperation method for space robot |
CN116214549B (en) * | 2023-01-19 | 2024-03-01 | 中国科学院微小卫星创新研究院 | Teleoperation system and teleoperation method for space robot |
CN116602772A (en) * | 2023-05-19 | 2023-08-18 | 哈尔滨思哲睿智能医疗设备股份有限公司 | Surgical robot, control method, device and medium for slave-end surgical arm of surgical robot |
CN116619422A (en) * | 2023-06-21 | 2023-08-22 | 哈尔滨理工大学 | Motion capture and wrist joint angle estimation mechanical arm mapping method based on human body characteristic signals |
WO2025043853A1 (en) * | 2023-08-28 | 2025-03-06 | 东南大学 | Safety shared control system based on performance of teleoperator |
CN116979419A (en) * | 2023-09-12 | 2023-10-31 | 国网江苏省电力有限公司常州供电分公司 | Bypass cable traction system and method |
CN117921677A (en) * | 2024-03-06 | 2024-04-26 | 睿尔曼智能科技(北京)有限公司 | Mechanical arm posture following control method and device |
CN118046394A (en) * | 2024-04-15 | 2024-05-17 | 帕西尼感知科技(张家港)有限公司 | A remote motion control method, device, system and storage medium |
CN118046394B (en) * | 2024-04-15 | 2024-06-21 | 帕西尼感知科技(张家港)有限公司 | A remote motion control method, device, system and storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN115469576B (en) | 2024-06-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN115469576B (en) | A teleoperation system based on hybrid mapping of human-robot heterogeneous motion space | |
CN107943283B (en) | Robotic arm pose control system based on gesture recognition | |
CN107363813B (en) | A desktop industrial robot teaching system and method based on wearable device | |
CN105291138B (en) | It is a kind of to strengthen the visual feedback platform of virtual reality immersion sense | |
CN109262609A (en) | Mechanical arm tele-control system and method based on virtual reality technology | |
CN106313049B (en) | A kind of apery mechanical arm motion sensing control system and control method | |
CN113829343B (en) | Real-time multitasking and multi-man-machine interaction system based on environment perception | |
CN110815189B (en) | Robot rapid teaching system and method based on mixed reality | |
CN107856014B (en) | Mechanical arm pose control method based on gesture recognition | |
Luo et al. | Real time human motion imitation of anthropomorphic dual arm robot based on Cartesian impedance control | |
CN106346485A (en) | Non-contact control method of bionic manipulator based on learning of hand motion gestures | |
CN102350700A (en) | Method for controlling robot based on visual sense | |
CN110045740A (en) | A kind of Mobile Robot Real-time Motion planing method based on human behavior simulation | |
CN105014677A (en) | Visual mechanical arm control device and method based on Camshift visual tracking and D-H modeling algorithms | |
CN115157261B (en) | Flexible robotic arm remote operation human-computer interaction device and method based on mixed reality | |
CN118322199B (en) | A teleoperation method for a five-finger dexterous arm-hand robot based on data gloves | |
CN111300408B (en) | Motion planning control method for humanoid double-arm robot combining shape similarity and look similarity | |
CN116197899A (en) | Active robot teleoperation system based on VR | |
Xiong et al. | Predictive display and interaction of telerobots based on augmented reality | |
CN111702787A (en) | Human-machine cooperative control system and control method | |
CN115488880A (en) | Simulation test system for mechanical arm visual servo grabbing in space environment | |
Chen et al. | Application of haptic virtual fixtures on hot-line work robot-assisted manipulation | |
CN117301056A (en) | A robot remote control system and method based on real-time capture of arm postures | |
Li et al. | Intelligent robotic arm for human pose recognition based on teleoperation system | |
Xu et al. | Design of a human-robot interaction system for robot teleoperation based on digital twinning |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |