CN112667823B - A method, system and computer-readable medium for semantic parsing of robotic arm task execution sequence - Google Patents

Abstract

The invention discloses a semantic analysis method, a semantic analysis system and a computer readable medium for a mechanical arm task execution sequence, wherein a mechanical arm task execution point sequence and a state sequence of all objects in a scene corresponding to an execution point are obtained, and a task subprocess sequence is divided; determining the action type and the acting object of the mechanical arm gripper in the task subprocess, and analyzing the space semantic relationship of the acting object; and integrating semantic analysis results of the task subprocess, constructing a task-based semantic analysis graph, and obtaining a mechanical arm task action sequence. According to the invention, the complex task demonstration data in the unstructured environment is converted into the structured semantic analysis chart which can be understood and executed by the mechanical arm by analyzing the task execution point sequence of the mechanical arm and the change of all object state data in the task, and the mechanical arm can be guided to execute the task consistent with the task semantic expression in different initial scenes, so that the mechanical arm has the capability of learning and analyzing the task.

Description

Semantic parsing method, system and computer for manipulator task execution sequence read media

technical field

The invention belongs to the technical field of manipulators, and relates to a semantic analysis method, system and computer-readable medium of a task execution sequence of a manipulator.

Background technique

The robotic arm is an automated mechanical device that has been widely used in industrial manufacturing, medical treatment, national defense technology, entertainment services, etc. It can accept operating instructions and accurately locate a certain point in three-dimensional space to perform operations, that is, to imitate the execution of human arms. Tasks, instead of humans to complete some heavy, precise and dangerous tasks autonomously. Compared with the execution of human arms, the execution of tasks by robotic arms has the characteristics of high precision, fast, and ultra-safe tasks.

When faced with tasks with complex semantics, at present, multiple robotic arms are usually used to complete them step by step, and each robotic arm only focuses on performing one action in the task. Therefore, when facing different tasks, tasks need to be manually disassembled. points, issue different instructions to multiple robotic arms to complete the entire task step by step. People expect that the robotic arm has the ability to analyze tasks independently, and can analyze tasks into a series of executable action sequences based on the execution data of different tasks, and then distribute the actions to multiple robotic arms to complete complex tasks together. At present, there are few analysis methods for manipulator tasks. One of them is based on task description text analysis, and the syntax analysis tree is obtained by syntactic analysis and grammatical analysis of natural language, but this method lacks the spatial semantic relationship of the operating object. express.

The analysis method proposed by the present invention is to convert complex task demonstration data in an unstructured environment into structured data that can be understood and executed by the manipulator by analyzing the sequence of manipulator task execution points and the changes in the state data of all objects in the task. The semantic analysis diagram of the task, which includes the task action sequence, the action object and the spatial semantic relationship of the object, it can guide the manipulator to perform tasks consistent with the task semantics in different initial scenes, which makes the manipulator have the ability to learn the task Analytical ability greatly expands the range of use of the robotic arm and improves its performance.

Contents of the invention

In order to achieve the above object, the present invention provides a semantic analysis method, system and computer-readable medium of a task execution sequence of a manipulator. By analyzing the sequence of task execution points of a manipulator and the changes of all object state data in the task, non- The complex task demonstration data in a structured environment is converted into a structured semantic analysis graph that the manipulator can understand and execute, including task action sequences, action objects, and spatial semantic relationships of objects, which can guide the manipulator in different initial stages. In the scene, the tasks consistent with the semantic performance of the tasks are performed, which makes the manipulator have the ability to learn and analyze the tasks, greatly expands the scope of use of the manipulator, improves its performance, and solves the problem of operating objects in the prior art. The resolution of spatial-semantic relationships is a poorly studied problem.

The technical solution adopted in the present invention is a semantic analysis method of a manipulator task execution sequence, comprising the following steps:

Obtain the task execution point sequence of the manipulator and the state sequence of all objects in the scene corresponding to the execution point, and determine the split point of the task execution point sequence by performing difference processing on the opening and closing states in the state sequence of the manipulator gripper with a task length of N , each segmentation point divides the state sequence of the gripper of the manipulator into n consecutive task sub-process sequences;

解析任务子过程s_j中包含的所有物体的运动特征、获取任务子过程s_j中发生变化的物体列表和与机械臂抓手直接接触的物体列表，确定任务子过程s_j机械臂抓手的动作类型及作用的物体，并根据任务子过程s_j结束时所有物体的状态特征，解析作用物体的空间语义关系；Determine the type of manipulation of the robotic arm gripper for the task subprocess _sj

Analyze the motion characteristics of all objects contained in the task sub-process s _j , obtain the list of objects that have changed in the task sub-process _{s j} and the list of objects that are in direct contact with the gripper of the manipulator, and determine the The type of action and the object to be acted on, and according to the state characteristics of all objects at the end of the task sub-process s _j , analyze the spatial semantic relationship of the acted object;

Integrate the semantic analysis results of the task sub-process s _j , build a semantic analysis graph based on the task execution sequence of the manipulator, and obtain the task action sequence of the manipulator.

Further, the task execution point sequence of the manipulator and the state sequence of all objects in the scene corresponding to the execution point are obtained, the task execution point sequence of the manipulator is specifically the state sequence of the gripper of the manipulator, and the state sequence of all objects is determined by each Composition of the state sequence of a single object, the task execution point sequence segmentation point is determined by performing difference processing on the opening and closing state in the state sequence of the robotic arm gripper with a task length of N, and each segmentation point divides the state sequence of the robotic arm gripper Divided into n consecutive task sub-process sequences, specifically:

The state sequence of the gripper of the robotic arm is {GS ₁ , GS ₂ , GS ₃ ..., GS _i , ..., GS _N }, where the i-th state data of the gripper of the robotic arm GS _i =(pos _i , qua _i , open _i ), pos _i = (x _i , y _i , z _i ) represents the position coordinates of the i-th state of the gripper of the robotic arm, x _i represents the x-axis coordinate of the i-th state of the gripper of the robotic arm, and y _i Indicates the y-axis coordinate of the i-th state of the robotic arm gripper, z _i represents the z-axis coordinate of the i-th state of the robotic arm gripper; qua _i = (q _x , q _y , q _z , q _w ) represents the The direction of the i-th state of the hand, (q _x , q _y , q _z ) represents the rotation axis of the i-th state of the robotic arm gripper, q _x is the x-axis component of the rotation axis, q _y represents the y-axis component of the rotation axis, q _z represents the z-axis component of the rotation axis, q _w represents the angle component of the i-th state of the gripper of the robotic arm rotating around the rotation axis, open _i = {0, 1} represents the opening and closing state of the i-th state of the gripper of the robotic arm, open _i =0 is close, open _i =1 is open;

单个物体a的第i个状态数据

name表示单个物体a的名称，pos_i＝(x_i，y_i，z_i)表示单个物体a的第i个状态的位置坐标；qua_i＝(q_x，q_y，q_z，q_w)表示单个物体a的第i个状态的方向，color_i＝(c_r，c_g，c_b)表示单个物体a的第i个状态的颜色，c_r表示单个物体a的第i个状态的颜色在R通道上的颜色分量，c_g表示单个物体a的第i个状态的颜色在G通道上的颜色分量，c_b表示单个物体a的第i个状态的颜色在B通道上的颜色分量；bbox_i＝(x_min，x_max，y_min，y_max，z_min，z_max)表示单个物体a的第i个状态的大小，以物体a中心为原点坐标轴，(x_min，x_max)表示物体a大小在x轴上的范围，(y_min，y_max)表示物体a大小在y轴上的范围，(z_min，z_max)表示物体a大小在z轴上的范围；rel＝{None，sub}表示单个物体a是某个物体不可脱离的组件，rel＝None表示单个物体a不是任何物体不可脱离的组件，rel＝sup表示单个物体a是物体sup不可脱离的组件，sub表示物体；R表示以单个物体a中心为原点的坐标系A相对于世界坐标系W的旋转矩阵，R的具体表达如下式所示：The state sequence of a single object is

The i-th state data of a single object a

name represents the name of a single object a, pos _i = ( _xi , y _i , _zi ) represents the position coordinates of the i-th state of a single object a; qua _i = (q _x , q _y , q _z , q _w ) Indicates the direction of the i-th state of a single object a, color _i = (c _r , c _g , c _b ) represents the color of the i-th state of a single object a, and c _r represents the color of the i-th state of a single object a The color component on the R channel, c _g represents the color component of the color of the i-th state of a single object a on the G channel, and c _b represents the color component of the color of the i-th state of a single object a on the B channel; bbox _i = (x _min , x _max , y _min , y _max , z _min , z _max ) represents the size of the i-th state of a single object a, with the center of object a as the origin coordinate axis, (x _min , x _max ) Indicates the range of the size of object a on the x-axis, (y _min , y _max ) represents the range of the size of object a on the y-axis, (z _min , z _max ) represents the range of the size of object a on the z-axis; rel={ None, sub} indicates that a single object a is an inseparable component of an object, rel=None indicates that a single object a is not an inseparable component of any object, rel=sup indicates that a single object a is an inseparable component of an object sup, and sub indicates an object ; R represents the rotation matrix of the coordinate system A with the center of the single object a as the origin relative to the world coordinate system W, and the specific expression of R is shown in the following formula:

Among them, the rotation matrix R consists of four columns, the first column (R ₀ , R ₄ , R ₈ ) is the direction component of the coordinate system A relative to the world coordinate system W on the x-axis, and the second column (R ₁ , R ₅ , R ₉ ) is the direction component of the coordinate system A relative to the world coordinate system W on the y-axis, and the third column (R ₂ , R ₆ , R ₁₀ ) is the direction component of the coordinate system A relative to the world coordinate system W on the z-axis The direction component, the fourth column (R ₃ , R ₇ , R ₁₁ ) is the translation component of the coordinate system A relative to the world coordinate system W on the x, y, and z axes respectively;

即

Let the opening and closing state of the i-th state of the robotic arm gripper be open _i , and the opening and closing state of the next robotic arm gripper state be open _i+1 , 1≤i<N, and the two adjacent The opening and closing state in the state of the gripper of the robotic arm is subjected to difference processing to obtain the opening and closing state difference

which is

按照下述公式确定任务序列中的分割点p_j，具体为：According to the opening and closing state difference

Determine the split point p _j in the task sequence according to the following formula, specifically:

Among them, j>1, 0<i<N, p ₁ =0, p _n-1 =N-1, p ₁ =0 means that the first split point is the starting point of the task sequence, p _n-1 =N-1 Indicates that the last split point is the end point of the task sequence;

The segmentation points obtained in the state sequence of the gripper of the manipulator are formed into a segmentation point sequence {p ₁ , p ₂ , p ₃ , ..., p _j , ..., p _n-1 }, p _j <N, each The segmentation point divides the state sequence of the gripper of the robotic arm into n consecutive sub-process sequences {s ₁ , s ₂ , s ₃ , .., s _j , ..., s _n }, where s _j represents the segmentation point p The state sequence of the manipulator gripper from _j to the split point p _j+1 is recorded as the task sub-process s _j (p _j ,p _j+1 ).

的确定过程，具体为：根据任务子过程s_j中机械臂抓手的位置变化、方向变化、开合状态确定任务子过程s_j的机械臂抓手的操作类型

按时序得到n个任务子过程的机械臂抓手的操作序列：Further, the operation type of the manipulator gripper of the task sub-process s _j

The determination process of , specifically: according to the position change, direction change, and opening and closing state of the manipulator gripper in the task subprocess s _j , determine the operation type of the manipulator gripper in the task subprocess s _j

Obtain the operation sequence of the manipulator gripper of n task sub-processes in time sequence:

机械臂抓手起始位置

终止位置

起始方向

终止方向

开合状态计算得到任务子过程机械臂抓手的运动特征

According to the task sub-process

The starting position of the gripper of the robotic arm

end position

starting direction

end direction

The motion characteristics of the robot arm gripper in the task sub-process are obtained by calculating the opening and closing state

的计算公式为

Motion characteristics of the gripper of the robotic arm

The calculation formula is

是任务子过程s_j中机械臂抓手的位移；

是任务子过程s_j中机械臂抓手的旋转角度；

是任务子过程s_j中机械臂抓手的开合状态；In the formula,

is the displacement of the gripper of the manipulator in the task sub-process s _j ;

is the rotation angle of the gripper of the manipulator in the task sub-process s _j ;

is the opening and closing state of the gripper of the manipulator in the task sub-process s _j ;

in,

表示机械臂抓手在旋转到子过程终止p_j+1时绕转轴旋转的角度分量，

表示机械臂抓手在旋转到子过程起始p_j时绕转轴旋转的角度分量；In the formula,

Indicates the angle component of the rotation around the axis of rotation of the gripper of the manipulator when it rotates to the end of the sub-process p _j+1 ,

Indicates the angular component of the rotation around the axis of rotation of the gripper of the manipulator when it rotates to the start of the subprocess p _j ;

包括三个，分别为：reach表示到达、move表示移动、rotate表示旋转，

的计算公式具体为：The operation type of the robot arm gripper of the task subprocess s _j

Including three, namely: reach means arrival, move means movement, rotate means rotation,

The specific calculation formula is:

表示任务子过程s_j中机械臂抓手的位移，threshold_dis是判定机械臂抓手产生位移的阈值；

表示任务子过程s_j中机械臂抓手的旋转角度，threshold_θ是判断机械臂抓手发生旋转变化的阈值。In the formula,

Indicates the displacement of the gripper of the robotic arm in the task sub-process s _j , and threshold _dis is the threshold for determining the displacement of the gripper of the robotic arm;

Indicates the rotation angle of the gripper of the manipulator in the task sub-process s _j , threshold _θ is the threshold for judging the rotation change of the gripper of the manipulator.

终止位置

起始方向

终止方向

以及起始颜色

终止颜色

获取单个物体a在任务子过程s_j的运动特征

进而获取任务子过程s_j中包含的所有物体的运动特征；Further, the analysis process of the motion characteristics of all objects contained in the task sub-process s _j is specifically: according to the starting position of a single object a contained in the task sub-process s _j

end position

starting direction

end direction

and the starting color

end color

Obtain the motion characteristics of a single object a in the task sub-process s _j

Then obtain the motion characteristics of all objects contained in the task sub-process s _j ;

的计算公式为：Among them, the motion characteristics of a single object a in the task sub-process s _j

The calculation formula is:

其中：

in:

是任务子过程s_j中物体a的位移；

是任务子过程s_j中物体a的旋转角度；

是任务子过程s_j中物体a的颜色变化；

表示在任务子过程s_j中物体a是否消失，

表示物体a在经过任务子过程s_j后消失，

表示物体a在经过任务子过程s_j后依然存在；

分别是任务子过程s_j终止时物体a的位置坐标，

分别是任务子过程s_j开始时物体a的位置坐标；

表示物体a在旋转到任务子过程s_j终止时的角度分量，

是物体a在旋转到任务子过程s_j起始p_j时的角度分量；

分别表示任务子过程s_j开始时物体a的颜色通道，

分别表示任务子过程s_j终止时物体a的颜色通道；

代表在任务子过程s_j起始执行点对应的物体中含物体a；

代表在任务子过程s_j终止执行点对应的物体中不含物体a。in,

is the displacement of object a in task sub-process s _j ;

is the rotation angle of object a in task sub-process s _j ;

is the color change of object a in the task sub-process s _j ;

Indicates whether the object a disappears in the task sub-process s _j ,

Indicates that the object a disappears after passing through the task sub-process s _j ,

Indicates that the object a still exists after passing through the task sub-process s _j ;

are the position coordinates of the object a when the task sub-process s _j terminates, respectively,

are the position coordinates of the object a when the task sub-process s _j starts;

Indicates the angle component of the object a when it is rotated to the end of the task sub-process s _j ,

is the angle component of the object a when it rotates to the start p _j of the task sub-process s _j ;

Respectively represent the color channels of the object a when the task sub-process s _j starts,

Respectively represent the color channels of object a when the task sub-process s _j terminates;

Indicates that object a is included in the objects corresponding to the start execution point of the task sub-process s _j ;

It means that object a is not included in the objects corresponding to the termination execution point of the task sub-process s _j .

Further, the acquisition process of the list of objects changed in the task sub-process s _j and the list of objects directly in contact with the gripper of the manipulator is as follows:

列表、

列表、

列表、

列表；Obtain the motion characteristics of the objects that have displacement, rotation, color change, and disappearance in the task sub-process s _j respectively, and store the names of the objects that change in the task sub-process s _j into

list,

list,

list,

list;

According to the state data of the gripper of the manipulator and the state data of all objects, obtain the object list t_objects in the task sub-process s _j that is in direct contact with the gripper of the manipulator.

中获取，其中状态数据中的序号i等于任务子过程中起始p_j和终止p_j+1的序号；Further, the task sub-process s _j is the determination process of the action type of the gripper of the manipulator and the object it acts on, specifically: the task sub-process s _j (p _j , p _j+1 ) starts p _j and ends p _j+1 properties of an object from a single object a state data

, where the serial number i in the state data is equal to the serial numbers of the start p _j and the end p _j+1 in the task sub-process;

The specific formula for determining the action type of the task sub-process s _j robotic arm gripper is as follows:

是任务子过程s_j中机械臂抓手的操作类型；

是任务子过程s_j中产生位移的物体个数；

是任务子过程s_j中发生旋转的物体个数；

是任务子过程s_j中发生颜色变化的物体个数；

是任务子过程s_j中机械臂抓手直接接触的物体；

是任务子过程s_j中机械臂抓手直接接触物体的rel值，表示该物体是某个物体不可脱离的组件，rel＝None表示该物体不是任何物体不可脱离的组件，rel＝sup表示该物体是物体sup不可脱离的组件；

是任务子过程s_j中消失的物体个数；当机械臂抓手的动作类型是“move”并且机械臂抓手直接接触物体是作为物体sup不可分割的组件时，任务子过程s_j的动作类型具体是“push”还是“pull”，要根据物体与sup对象的距离变化进一步区分，具体的公式如下：in,

is the operation type of the robotic arm gripper in the task sub-process s _j ;

is the number of displaced objects in the task sub-process s _j ;

is the number of objects rotated in the task sub-process s _j ;

is the number of objects whose color changes in the task sub-process s _j ;

is the object directly contacted by the gripper of the robotic arm in the task sub-process _sj ;

is the rel value of the object in direct contact with the gripper of the robotic arm in the task sub-process s _j , indicating that the object is an inseparable component of an object, rel=None indicates that the object is not an inseparable component of any object, rel=sup indicates that the object is an inseparable component of the object sup;

is the number of objects that disappear in the task sub-process s _j ; when the action type of the robotic arm gripper is "move" and the direct contact of the robotic arm gripper with the object is an inseparable component of the object sup, the action of the task sub-process s _j Whether the type is "push" or "pull" should be further distinguished according to the distance between the object and the sup object. The specific formula is as follows:

是位移变化物体在任务子过程起始p_j时的位置坐标；

是位移变化物体的sup对象在任务子过程起始p_j时的位置坐标；distance是计算两个坐标点间的距离公式；in,

is the position coordinate of the displacement-changing object at the start p _j of the task sub-process;

is the position coordinate of the sup object of the displacement changing object at the start p _j of the task sub-process; distance is the formula for calculating the distance between two coordinate points;

Define the relationship between the action type of the above-mentioned robotic arm gripper and the object it acts on, as follows:

物体a是通过在t_objects中查找在子过程s_j序列之后且离它最近子过程s_k对应的物体，即(a，s_k)，j＜k；Arrive at object a, denoted as "arrived":

Object a is searched in t_objects for the object corresponding to the sub-process s _k which is after the sequence of sub-process s _j and is closest to it, namely (a, s _k ), j<k;

物体a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)；Rotate object a, expressed as "rotation":

The object a is to find the object corresponding to s _j by searching t_objects, namely (a, s _j );

物体中a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)，a^*是物体a最终到达的物体，即为

的中发生位置改变的物体；Moving object a, denoted as "moving":

In the object, a is to find the object corresponding to s _j by searching t_objects, that is, (a, s _j ), and a ^* is the object that object a finally arrives at, that is,

Objects whose positions have changed in ;

物体a为

中位置发生改变的物体，物体b为

中颜色发生改变的物体；Pressing object a makes object b change color, denoted as "press":

object a is

The object whose position changes in the middle, the object b is

Objects that change color in

物体中a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)，物体b为

中不是物体a的另一个物体，b^*是物体b最终到达的物体；Will hold object a and push object b, which means "indirect push": for

In the object a is to find the object corresponding to s _j by searching t_objects, that is, (a, s _j ), the object b is

in another object that is not object a, b ^* is the object that object b ends up at;

物体a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)，如果查找失败，物体a为空；物体b是

中位置发生改变的物体；Hold object a and push object b, expressed as "push":

Object a is to find the object corresponding to s _j by searching t_objects, namely (a, s _j ), if the search fails, object a is empty; object b is

Objects whose positions have changed in the

物体a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)；物体b是

中位置发生改变的物体；Grab object a and pull object b, denoted as "pull":

Object a is found by looking up the object corresponding to s _j in t_objects, namely (a, s _j ); object b is

Objects whose positions have changed in the

物体a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)；物体b是

中位置发生改变的物体。Take object a and clear object b, denoted as "clear":

Object a is found by looking up the object corresponding to s _j in t_objects, namely (a, s _j ); object b is

Objects whose positions have changed.

type表述两个物体的关系类型；Further, according to the state characteristics of all objects at the end of the task sub-process s _j , analyze the spatial semantic relationship of the acting objects, specifically: according to the state characteristics of all objects at the end of the task sub-process s _j , analyze the acting object a and other objects object The spatial semantic relationship of is expressed as

type expresses the relationship type of two objects;

转换到世界坐标系W上的坐标

确定物体a在世界坐标系W下的边界点坐标，计算公式如下：Transform the bbox boundaries of any two objects into the same coordinate system, establish a coordinate system A with the center of object a as the origin, and point on the coordinate system A

Convert to coordinates on the world coordinate system W

To determine the boundary point coordinates of object a in the world coordinate system W, the calculation formula is as follows:

Among them, R is the rotation matrix of the coordinate system A relative to the world coordinate system W, the rotation matrix R is composed of four columns, the first column (R _a0 , R _a4 , R _a8 ) is the coordinate system A relative to the world coordinate system W on the x-axis , the second column (R _a1 , R _a5 , R _a9 ) is the direction component of the coordinate system A relative to the world coordinate system W on the y-axis, and the third column (R _a2 , R _a6 , R _a10 ) is The direction component of the coordinate system A relative to the world coordinate system W on the z axis, the fourth column (R _a3 , R _a7 , R _a11 ) is the coordinate system A relative to the world coordinate system W, respectively on the x, y, and z axes The translation component of

Establish a coordinate system with the center of the object b as the origin, and determine the boundary point coordinates of the object b under the world coordinate system W;

Determine the type of the spatial semantic relationship between object a and object b.

Further, the judgment method of the spatial semantic relationship type between object a and object b is specifically: through the relationship between the center point distance between object a and object b and the maximum boundary of the bbox of the two objects; if the following conditions are met, proceed to S1, otherwise the object a is irrelevant to object b, type=separated, the judgment conditions are as follows:

是任务子过程终止(p_i+1)时物体a的位置坐标；

是任务子过程终止时物体b的位置坐标；distance是计算两个坐标点之间的距离公式；

是任务子过程终止(p_i+1)时物体a分别在x、y、z轴上最大的边界值；

是任务子过程终止(p_i+1)时物体b分别在x、y、z轴上最大的边界值；in,

is the position coordinate of object a when the task sub-process terminates (p _i+1 );

is the position coordinate of object b when the task sub-process is terminated; distance is the formula for calculating the distance between two coordinate points;

is the maximum boundary value of object a on the x, y, and z axes respectively when the task sub-process terminates (p _i+1 );

are the maximum boundary values of the object b on the x, y, and z axes respectively when the task sub-process terminates (p _i+1 );

S1. Determine the type of spatial semantic relationship through the relationship between object a and object b on the x, y, and z axes of the world coordinate system W; if the following conditions are not met, proceed to S2. The specific formula is as follows:

是指物体a在世界坐标系W的x轴上的边界范围；

是指物体b在世界坐标系W的x轴上的边界范围；

是指物体a在世界坐标系W的y轴上的边界范围；

是指物体b在世界坐标系W的y轴上的边界范围；

是指物体a在世界坐标系W的z轴上的边界范围；

是指物体b在世界坐标系W的z轴上的边界范围；In the formula,

Refers to the boundary range of object a on the x-axis of the world coordinate system W;

Refers to the boundary range of object b on the x-axis of the world coordinate system W;

Refers to the boundary range of object a on the y-axis of the world coordinate system W;

Refers to the boundary range of object b on the y-axis of the world coordinate system W;

Refers to the boundary range of object a on the z-axis of the world coordinate system W;

Refers to the boundary range of object b on the z-axis of the world coordinate system W;

If the above conditions are met, the following judgments are made to determine the type of spatial semantic relationship. The specific formula is as follows:

的关系，确定空间语义关系类型，具体的公式如下：S2. Through the center coordinates (x _a , y _a , z _a ), (x _b , y _b , z _b ) of object a and object b in the world coordinate system and the two objects

relationship, to determine the type of spatial semantic relationship, the specific formula is as follows:

Another object of the present invention is to provide a semantic analysis system for manipulator task execution sequence, including:

memory for storing instructions executable by the processor;

a processor, configured to execute the instructions to implement the above method.

Yet another object of the present invention is to provide a computer-readable medium storing computer program codes, the computer program codes implementing the above method when executed by a processor.

The beneficial effects of the present invention are:

(1) The present invention converts complex task demonstration data in an unstructured environment into structured semantics that the manipulator can understand and execute by analyzing the sequence of manipulator task execution points and the changes in the state data of all objects in the task Analytical diagrams, including task action sequences, action objects, and spatial semantic relationships of objects, which can guide the manipulator to perform tasks consistent with the task semantics in different initial scenarios, which makes the manipulator have the ability to learn and analyze tasks , greatly expanding the range of use of the robotic arm and improving its performance.

(2) The semantic analysis diagram obtained by the present invention can be used for the visual motion planning of the manipulator to do complex tasks, and predict the next execution action of the current visual observation according to the visual observation at a certain moment of the task video demonstration and the previous execution action, which is It is of great help to the splitting of complex tasks and motion planning of the robotic arm in an unstructured environment.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a working flow chart of the semantic analysis method of the manipulator task execution sequence of the present invention.

Fig. 2 is an example diagram of the spatial semantic relationship between objects in the present invention.

Fig. 3 is a semantic analysis diagram obtained by the present invention.

Fig. 4 is a semantic analysis diagram further expanded in the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

As shown in Figures 1 to 4, a semantic analysis method for a task execution sequence of a manipulator includes the following steps:

Obtain the task execution point sequence of the manipulator and the state sequence of all objects in the scene corresponding to the execution point, and determine the segmentation point by performing difference processing on the opening and closing states in the state sequence of the robotic arm gripper with a task length of N. Each segmentation point Divide the state sequence of the robotic arm gripper into n consecutive task sub-process sequences:

Among them, the start and end of the task execution point sequence of the manipulator is the beginning and end of the execution of the entire manipulator task; the task execution point sequence of the manipulator is specifically the state sequence of the manipulator gripper; the state sequence of all objects is determined by each A sequence of states for a single object consists of;

Among them, the state sequence of the gripper of the robotic arm includes the position and direction of the gripper and the opening and closing state of the gripper; if the length of the state sequence of the gripper of the robotic arm is N, then the state sequence of the gripper of the robotic arm is {GS ₁ , GS ₂ , GS ₃ ..., GS _i , ..., GS _N }, where the i-th state data of the manipulator gripper GS _i =(pos _i , qua _i , open _i ), pos _i =(x _i , y _i , z _i ) represent the position coordinates of the i-th state of the robotic arm gripper, x _i represent the x-axis coordinates of the i-th state of the robotic arm gripper, and y _i represent the y of the i-th state of the robotic arm gripper axis coordinates, z _i represents the z-axis coordinates of the i-th state of the gripper of the robotic arm; qua _i = (q _x , q _y , q _z , q _w ) represents the direction of the i-th state of the gripper of the robotic arm, (q _x , q _y , q _z ) represent the rotating shaft of the i-th state of the gripper of the manipulator, q _x is the x-axis component of the rotating shaft, q _y represents the y-axis component of the rotating shaft, q _z represents the z-axis component of the rotating shaft, q _w represents the angular component of the i-th state of the robotic arm gripper rotating around the axis, open _i = {0, 1} represents the opening and closing state of the i-th state of the robotic arm gripper (open _i = 0 is closed, open _i = 1 is on).

单个物体a的第i个状态数据

name表示单个物体a的名称，pos_i＝(x_i，y_i，z_i)表示单个物体a的第i个状态的位置坐标；qua_i＝(q_x，q_y，q_z，q_w)表示单个物体a的第i个状态的方向，color_i＝(c_r，c_g，c_b)表示单个物体a的第i个状态的颜色，c_r表示单个物体a的第i个状态的颜色在R通道上的颜色分量，c_g表示单个物体a的第i个状态的颜色在G通道上的颜色分量，c_b表示单个物体a的第i个状态的颜色在B通道上的颜色分量；bbox_i＝(x_min，x_max，y_min，y_max，z_min，z_max)表示单个物体a的第i个状态的大小，以物体a中心为原点坐标轴，(x_min，x_max)表示物体a大小在x轴上的范围，(y_min，y_max)表示物体a大小在y轴上的范围，(z_min，z_max)表示物体a大小在z轴上的范围；rel＝{None，sub}表示单个物体a是某个物体不可脱离的组件，rel＝None表示单个物体a不是任何物体不可脱离的组件，rel＝sup表示单个物体a是物体sup不可脱离的组件，sub表示物体；R表示以单个物体a中心为原点的坐标系A相对于世界坐标系W的旋转矩阵，R的具体表达如下式所示：Among them, the state sequence of a single object includes the name, position, direction, color, size of a single object, whether there is an object to which it belongs (“whether there is an object to which it belongs” means whether the single object is an inseparable component of an object), and The rotation matrix of the coordinate system A with the center of a single object as the origin relative to the world coordinate system W; the state sequence of a single object a is

The i-th state data of a single object a

name represents the name of a single object a, pos _i = ( _xi , y _i , _zi ) represents the position coordinates of the i-th state of a single object a; qua _i = (q _x , q _y , q _z , q _w ) Indicates the direction of the i-th state of a single object a, color _i = (c _r , c _g , c _b ) represents the color of the i-th state of a single object a, and c _r represents the color of the i-th state of a single object a The color component on the R channel, c _g represents the color component of the color of the i-th state of a single object a on the G channel, and c _b represents the color component of the color of the i-th state of a single object a on the B channel; bbox _i = (x _min , x _max , y _min , y _max , z _min , z _max ) represents the size of the i-th state of a single object a, with the center of object a as the origin coordinate axis, (x _min , x _max ) Indicates the range of the size of object a on the x-axis, (y _min , y _max ) represents the range of the size of object a on the y-axis, (z _min , z _max ) represents the range of the size of object a on the z-axis; rel={ None, sub} indicates that a single object a is an inseparable component of an object, rel=None indicates that a single object a is not an inseparable component of any object, rel=sup indicates that a single object a is an inseparable component of an object sup, and sub indicates an object ; R represents the rotation matrix of the coordinate system A with the center of the single object a as the origin relative to the world coordinate system W, and the specific expression of R is shown in the following formula:

Among them, the rotation matrix R consists of four columns, the first column (R ₀ , R ₄ , R ₈ ) is the direction component of the coordinate system A relative to the world coordinate system W on the x-axis, and the second column (R ₁ , R ₅ , R ₉ ) is the direction component of the coordinate system A relative to the world coordinate system W on the y-axis, and the third column (R ₂ , R ₆ , R ₁₀ ) is the direction component of the coordinate system A relative to the world coordinate system W on the z-axis The direction component, the fourth column (R ₃ , R ₇ , R ₁₁ ) is the translation component of the coordinate system A relative to the world coordinate system W on the x, y, and z axes respectively.

Among them, the segmentation point is determined by performing difference processing on the opening and closing states in the state sequence of the robotic arm gripper with a task length of N, and each segmentation point divides the state sequence of the robotic arm gripper into n consecutive task sub-processes sequence, specifically:

即Let the opening and closing state of the i-th state of the robotic arm gripper be open _i , and the opening and closing state of the next robotic arm gripper state be open _i+1 , 1≤i<N, and the two adjacent The opening and closing state in the state of the gripper of the robotic arm is subjected to difference processing to obtain the opening and closing state difference

which is

按照下述公式确定任务序列中的分割点p_j，具体为：

According to the above opening and closing state difference

Determine the split point p _j in the task sequence according to the following formula, specifically:

Among them, j>1, 0<i<N, p ₁ =0, p _n-1 =N-1, p ₁ =0 means that the first split point is the starting point of the task sequence, p _n-1 =N-1 Indicates that the last split point is the end point of the task sequence;

Form the segmentation point sequence {p ₁ , p ₂ , p ₃ , ..., p _j , ..., p _n-1 }, p _j < N, each segmentation point divides the state sequence of the gripper of the robotic arm into n continuous sub-process sequences {s ₁ , s ₂ , s ₃ , .., s _j , ..., s _n }, where s _j represents The state sequence of the manipulator gripper from the split point p _j to the split point p _j+1 is recorded as the task sub-process s _j (p _j , p _j+1 )

解析任务子过程s_j中包含的所有物体的运动特征，分析任务子过程s_j的机械臂抓手的操作类型

任务子过程s_j中包含的所有物体的运动特征、任务子过程s_j中发生变化的物体列表和与机械臂抓手直接接触的物体列表，确定任务子过程s_j机械臂抓手的动作类型及作用的物体，并根据任务子过程s_j结束时所有物体的状态特征，解析作用物体的空间语义关系：Determine the type of manipulation of the robotic arm gripper for the task subprocess _sj

Analyze the motion characteristics of all objects contained in the task sub-process s _j , and analyze the operation type of the manipulator gripper of the task sub-process s _j

The motion characteristics of all objects contained in the task sub-process _sj , the list of objects changed in the task sub-process _sj and the list of objects in direct contact with the gripper of the manipulator determine the action type of the gripper of the manipulator in the task sub-process _sj and the acting objects, and according to the state characteristics of all objects at the end of the task sub-process s _j , analyze the spatial semantic relationship of the acting objects:

的确定过程，具体为：根据任务子过程s_j中机械臂抓手的位置变化、方向变化、开合状态确定任务子过程s_j的机械臂抓手的操作类型

按时序得到n个任务子过程的机械臂抓手的操作序列；The operation type of the robot arm gripper of the task subprocess s _j

The determination process of , specifically: according to the position change, direction change, and opening and closing state of the manipulator gripper in the task subprocess s _j , determine the operation type of the manipulator gripper in the task subprocess s _j

Obtain the operation sequence of the manipulator gripper of n task sub-processes in time sequence;

机械臂抓手起始位置

终止位置

起始方向

终止方向

开合状态，计算得到任务子过程机械臂抓手的运动特征

According to the task sub-process

The starting position of the gripper of the robotic arm

end position

starting direction

end direction

In the open and close state, the motion characteristics of the robot arm gripper in the task sub-process are calculated

Motion characteristics of the gripper of the robotic arm

是任务子过程s_j中机械臂抓手的位移；

是任务子过程s_j中机械臂抓手的旋转角度；

是任务子过程s_j中机械臂抓手的开合状态；In the formula,

is the displacement of the gripper of the manipulator in the task sub-process s _j ;

is the rotation angle of the gripper of the manipulator in the task sub-process s _j ;

is the opening and closing state of the gripper of the manipulator in the task sub-process s _j ;

in,

表示机械臂抓手在旋转到子过程终止p_j+1时绕转轴旋转的角度分量，

是机械臂抓手在旋转到子过程起始p_j时绕转轴

旋转的角度分量；In the formula,

Indicates the angle component of the rotation around the axis of rotation of the gripper of the manipulator when it rotates to the end of the sub-process p _j+1 ,

is the gripper of the manipulator revolves around the rotation axis when it rotates to the start p _j of the sub-process

the angular component of the rotation;

表示任务子过程s_j中机械臂抓手的开合状态就是任务子过程起始p_j时机械臂抓手的开合状态。

Indicates that the opening and closing state of the gripper of the manipulator in the task sub-process s _j is the opening and closing state of the gripper of the manipulator at the beginning of the task sub-process p _j .

包括三个，分别为：reach表示到达、move表示移动、rotate表示旋转，

的计算公式具体为：Among them, the operation type of the manipulator gripper of the task subprocess s _j

Including three, namely: reach means arrival, move means movement, rotate means rotation,

The specific calculation formula is:

表示任务子过程s_j中机械臂抓手的位移，threshold_dis是判定机械臂抓手产生位移的阈值；

表示任务子过程s_j中机械臂抓手的旋转角度，threshold_θ是判断机械臂抓手发生旋转变化的阈值；In the formula,

Indicates the displacement of the gripper of the robotic arm in the task sub-process s _j , and threshold _dis is the threshold for determining the displacement of the gripper of the robotic arm;

Indicates the rotation angle of the gripper of the robotic arm in the task sub-process s _j , threshold _θ is the threshold for judging the rotation change of the gripper of the robotic arm;

为reach时，其限制条件为：机械臂抓手产生位移的阈值小于等于任务子过程s_j机械臂抓手的移动距离，并且机械臂抓手开合状态为开；operation type

When it is reach, its restriction condition is: the threshold value of the displacement generated by the gripper of the robotic arm is less than or equal to the moving distance of the gripper of the robotic arm in the task sub-process s _j , and the opening and closing state of the gripper of the robotic arm is open;

为move时，其限制条件为：机械臂抓手产生位移的阈值小于等于任务子过程s_j机械臂抓手的移动距离，并且机械臂抓手开合状态为合；operation type

When it is move, its restriction condition is: the threshold value of the displacement generated by the gripper of the robotic arm is less than or equal to the moving distance of the gripper of the robotic arm in the task sub-process s _j , and the opening and closing state of the gripper of the robotic arm is closed;

为rotate时，其限制条件为：机械臂抓手产生位移的阈值大于任务子过程s_j机械臂抓手的移动距离，并且机械臂抓手发生旋转变化的阈值小于等于任务子过程s_j中机械臂抓手的旋转角度，并且机械臂抓手开合状态为合。operation type

When it is rotate, the restrictive conditions are: the threshold of the displacement of the gripper of the robotic arm is greater than the moving distance of the gripper of the robotic arm in the task sub-process s _j , and the threshold of the rotation change of the gripper of the robotic arm is less than or equal to the mechanical arm gripper in the task sub-process s _j The rotation angle of the arm gripper, and the opening and closing state of the arm gripper is Closed.

终止位置

起始方向

终止方向

以及起始颜色

终止颜色

获取单个物体a在任务子过程s_j的运动特征

进而获取任务子过程s_j中包含的所有物体的运动特征；Among them, the analysis process of the motion characteristics of all objects contained in the task sub-process s _j is specifically: according to the starting position of a single object a contained in the task sub-process s _j

end position

starting direction

end direction

and the starting color

end color

Obtain the motion characteristics of a single object a in the task sub-process s _j

Then obtain the motion characteristics of all objects contained in the task sub-process s _j ;

的计算公式为：The motion characteristics of a single object a in the task sub-process s _j

The calculation formula is:

其中：

in:

是任务子过程s_j中物体a的位移；

是任务子过程s_j中物体a的旋转角度；

是任务子过程s_j中物体a的颜色变化；

表示在任务子过程s_j中物体a是否消失，

表示物体a在经过任务子过程s_j后消失，

表示物体a在经过任务子过程s_j后依然存在；

分别是任务子过程s_j终止时物体a的位置坐标，

分别是任务子过程s_j开始时物体a的位置坐标；

表示物体a在旋转到任务子过程s_j终止时的角度分量，

是物体a在旋转到任务子过程s_j起始p_j时的角度分量；

分别表示任务子过程s_j开始时物体a的颜色通道，

分别表示任务子过程s_j终止时物体a的颜色通道；

代表在任务子过程s_j起始执行点对应的物体中含物体a；

代表在任务子过程s_j终止执行点对应的物体中不含物体a。in,

is the displacement of object a in task sub-process s _j ;

is the rotation angle of object a in task sub-process s _j ;

is the color change of object a in the task sub-process s _j ;

Indicates whether the object a disappears in the task sub-process s _j ,

Indicates that the object a disappears after passing through the task sub-process s _j ,

Indicates that the object a still exists after passing through the task sub-process s _j ;

are the position coordinates of the object a when the task sub-process s _j terminates, respectively,

are the position coordinates of the object a when the task sub-process s _j starts;

Indicates the angle component of the object a when it is rotated to the end of the task sub-process s _j ,

is the angle component of the object a when it rotates to the start p _j of the task sub-process s _j ;

Respectively represent the color channels of the object a when the task sub-process s _j starts,

Respectively represent the color channels of object a when the task sub-process s _j terminates;

Indicates that object a is included in the objects corresponding to the start execution point of the task sub-process s _j ;

It means that object a is not included in the objects corresponding to the termination execution point of the task sub-process s _j .

Among them, the acquisition process of the object list that changes in the task sub-process s _j and the object list that is in direct contact with the gripper of the manipulator is as follows:

列表、

列表、

列表、

列表，具体如下述：Obtain the movement characteristics of objects that have displacement, rotation, color change, and disappearance in the task sub-process s _j respectively, and store the names of the objects that change in the task sub-process s _j into

list,

list,

list,

list, as follows:

列表；Store the name of the object whose moving distance is greater than the displacement change threshold in the task sub-process s _j into

list;

其中，threshold_{o_dis}是判定物体产生位移的阈值，rel^a≠None标志物体a为某一物体sup不可脱离的组件，

是sup在子过程s_j中产生的位移；Assuming that object a generates displacement in subprocess s _j , then:

Among them, threshold _{o_dis} is the threshold for determining the displacement of an object, rel ^a ≠ None indicates that object a is an inseparable component of an object sup,

is the displacement generated by sup in the subprocess s _j ;

Determining an object a as an object that produces displacement, the restrictive conditions are: the moving distance of object a in the task sub-process s _j is greater than or equal to the threshold value of displacement, and object a is not an inseparable component of any object or object a is inseparable The moving distance of the component object is less than the displacement threshold;

列表：假设物体a在任务子过程s_j中只发生旋转则有：

Store the name of the object that only rotates in the task sub-process s _j into

List: Assuming that object a only rotates in task sub-process s _j , then:

Among them, threshold _{o_dis} is the threshold for determining the displacement of the object, and threshold _{o_θ} is the threshold for determining the rotation of the object;

列表：假设物体a在任务子过程s_j中发生颜色变化则有：

Store the name of the object whose color changes in the task sub-process s _j into

List: Assuming that object a changes color in task sub-process s _j , then:

列表：假设物体a在任务子过程s_j中消失则有：

Store the name of the object that disappeared in the task subprocess s _j into

List: Suppose object a disappears in task sub-process s _j then:

According to the state data of the above-mentioned robotic arm gripper and the state data of all objects, obtain the object list t_objects in the task sub-process s _j that is in direct contact with the robotic arm gripper;

Suppose t_objects={(a_name, 1), (b_name, 4)}, where a_name represents the name of object a, 1 represents the first task sub-process, (a_name, 1) represents the gripper of the robotic arm in sub-process s ₁ Direct contact with object a, (b_name, ₄ ) means that the gripper of the robotic arm directly contacts object b in sub-process s4; object a satisfies the following relationship:

表示第p₁个执行点对应物体α的位置坐标

表示第p₁个执行点中机械臂抓手的位置坐标；distance是计算两个坐标点之间的距离公式；min_dis是第p₁个执行点对应时刻所有物体与机械臂抓手距离的最小值；

表示子过程s₁中机械臂抓手的开合状态；

表示物体a在子过程s₁中的位移。

表示在子过程s₁起始p₁和终止p₂时刻，物体a与机械臂抓手之间的距离没有发生改变。in,

Indicates the position coordinates of the pth _1st execution point corresponding to the object α

Indicates the position coordinates of the gripper of the robotic arm at the _p1th execution point; distance is the formula for calculating the distance between two coordinate points; min_dis is the minimum distance between all objects and the gripper of the robotic arm at the moment corresponding to the _p1th execution point ;

Indicates the opening and closing state of the gripper of the robotic arm in sub-process _s1 ;

Indicates the displacement of object a in subprocess _s1 .

Indicates that the distance between the object a and the gripper of the manipulator does not change at the start p ₁ and end p ₂ moments of the sub-process s ₁ .

中获取，其中状态数据中的序号i等于任务子过程中起始p_j和终止p_j+1的序号；Among them, the task sub-process s _j is the determination process of the action type of the manipulator arm gripper and the object to be acted on, specifically: when the task sub-process s _j (p _j , p _j+1 ) starts at p _j and ends at p _j+1 Object properties from individual object a state data

, where the serial number i in the state data is equal to the serial numbers of the start p _j and the end p _j+1 in the task sub-process;

The specific formula for determining the action type of the task sub-process s _j robotic arm gripper is as follows:

是任务子过程s_j中机械臂抓手的操作类型；

是任务子过程s_j中产生位移的物体个数；

是任务子过程s_j中发生旋转的物体个数；

是任务子过程s_j中发生颜色变化的物体个数；

是任务子过程s_j中机械臂抓手直接接触的物体；

是任务子过程s_j中机械臂抓手直接接触物体的rel值，表示该物体是某个物体不可脱离的组件，rel＝None表示该物体不是任何物体不可脱离的组件，rel＝sup表示该物体是物体sup不可脱离的组件；

是任务子过程s_j中消失的物体个数；当机械臂抓手的动作类型是“move”并且机械臂抓手直接接触物体是作为物体sup不可分割的组件时，任务子过程s_j的动作类型具体是“push”还是“pull”，要根据物体与sup对象的距离变化进一步区分，具体的公式如下：in,

is the operation type of the robotic arm gripper in the task sub-process s _j ;

is the number of displaced objects in the task sub-process s _j ;

is the number of objects rotated in the task sub-process s _j ;

is the number of objects whose color changes in the task sub-process s _j ;

is the object directly contacted by the gripper of the robotic arm in the task sub-process _sj ;

is the rel value of the object in direct contact with the gripper of the robotic arm in the task sub-process s _j , indicating that the object is an inseparable component of an object, rel=None indicates that the object is not an inseparable component of any object, rel=sup indicates that the object is an inseparable component of the object sup;

is the number of objects that disappear in the task sub-process s _j ; when the action type of the robotic arm gripper is "move" and the direct contact of the robotic arm gripper with the object is an inseparable component of the object sup, the action of the task sub-process s _j Whether the type is "push" or "pull" should be further distinguished according to the distance between the object and the sup object. The specific formula is as follows:

是位移变化物体在任务子过程起始p_j时的位置坐标；

是位移变化物体的sup对象在任务子过程起始p_j时的位置坐标；distance是计算两个坐标点间的距离公式。in,

is the position coordinate of the displacement-changing object at the start p _j of the task sub-process;

is the position coordinate of the sup object of the displacement-changing object at the start p _j of the task sub-process; distance is the formula for calculating the distance between two coordinate points.

Define the relationship between the action type of the above-mentioned robotic arm gripper and the object it acts on, as follows:

物体a是通过在t_objects中查找在子过程s_j序列之后且离它最近子过程s_k对应的物体，即(a，s_k)，j＜k。Arrive at object a, denoted as "arrived":

Object a is searched in t_objects for the object corresponding to the sub-process s _k that follows the sub-process s _j sequence and is closest to it, ie (a, s _k ), j<k.

物体a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)。Rotate object a, expressed as "rotation":

The object a is found by looking up the object corresponding to s _j in t_objects, namely (a, s _j ).

物体中a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)，a^*是物体a最终到达的物体，即为

的中发生位置改变的物体。Moving object a, denoted as "moving":

In the object, a is to find the object corresponding to s _j by searching t_objects, that is, (a, s _j ), and a ^* is the object that object a finally arrives at, that is,

Objects that change position in .

物体a为

中位置发生改变的物体，物体b为

中颜色发生改变的物体。Pressing object a makes object b change color, denoted as "press":

object a is

The object whose position changes in the middle, the object b is

Objects that change color.

物体中a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)，物体b为

中不是物体a的另一个物体，b^*是物体b最终到达的物体。Will hold object a and push object b, which means "indirect push": for

In the object a is to find the object corresponding to s _j by searching t_objects, that is, (a, s _j ), the object b is

is another object that is not object a, and b ^* is the object that object b eventually arrives at.

物体a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)，如果查找失败，物体a为空；物体b是

中位置发生改变的物体。Hold object a and push object b, expressed as "push":

Object a is to find the object corresponding to s _j by searching t_objects, namely (a, s _j ), if the search fails, object a is empty; object b is

Objects whose positions have changed.

物体a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)；物体b是

中位置发生改变的物体。Grab object a and pull object b, denoted as "pull":

Object a is found by looking up the object corresponding to s _j in t_objects, namely (a, s _j ); object b is

Objects whose positions have changed.

物体a是通过查找t_objects中查找与s_j对应的物体，即(a，s_j)；物体b是

中位置发生改变的物体，可能不止一个。Take object a and clear object b, denoted as "clear":

Object a is found by looking up the object corresponding to s _j in t_objects, namely (a, s _j ); object b is

There may be more than one object whose position has changed in .

type表述两个物体的关系类型，这里定义四种关系类型，有“包含”、“相交”、“相邻”和“接触”，如图2所示，物体B₁与物体B₂的空间语义关系为“包含”，即物体B₁在物体B₂中，表示为sem(B₁，B₂，contain)；物体P₁与物体P₂的关系是“相交”，两个物体之间部分相交，表示为sem(P₁，P₂，cross)；物体O₁与周围物体O₂～O₄的关系和物体C₁与周围物体C₂、C₃的关系是“相邻”，存在上下关系、左右关系和前后关系，例如，物体O₁在物体O₂的下方，表示为sem(O₁，O₂，below)；物体O₁在物体O₅的关系是“相离”，即两个物体之间没有接触，表示为sem(O₁，O₂，separated)。According to the state characteristics of all objects at the end of the task sub-process s _j , analyze the spatial semantic relationship of the acting object, specifically: according to the state characteristics of all objects at the end of the task sub-process s _j , analyze the spatial semantics of the acting object a and other objects object relationship, expressed as

type expresses the relationship type of two objects. Four types of relationship are defined here, including "contains", "intersects", "adjacent" and "contact". As shown in Figure 2, the spatial semantics of object B ₁ and object B ₂ The relationship is "contains", that is, object B ₁ is in object B ₂ , expressed as sem(B ₁ , B ₂ , contain); the relationship between object P ₁ and object P ₂ is "intersect", and the two objects partially intersect , expressed as sem(P ₁ , P ₂ , cross); the relationship between object O ₁ and surrounding objects O ₂ ～ O ₄ and the relationship between object C ₁ and surrounding objects C ₂ and C ₃ are "adjacent", and there is an upper-lower relationship , left-right relationship and front-back relationship, for example, object O ₁ is below object O ₂ , expressed as sem(O ₁ , O ₂ , below); the relationship between object O ₁ and object O ₅ is "separation", that is, two There is no contact between objects, expressed as sem(O ₁ , O ₂ , separated).

转换到世界坐标系W上的坐标

计算公式如下：Each object in the scene has its own coordinate system to determine the direction of the object. Before judging the relationship between the object a and object b corresponding to the execution point when the task sub-process s _j ends p _j+1 , it is necessary to first obtain the coordinates of the two objects The boundary range of the bbox is transformed into the same coordinate system, assuming that the coordinate system A is the coordinate system with the center of the object a as the origin, and the point on the coordinate system A

Convert to coordinates on the world coordinate system W

Calculated as follows:

Among them, R is the rotation matrix of the coordinate system A relative to the world coordinate system W, the rotation matrix R is composed of four columns, the first column (R _a0 , R _a4 , R _a8 ) is the coordinate system A relative to the world coordinate system W on the x-axis , the second column (R _a1 , R _a5 , R _a9 ) is the direction component of the coordinate system A relative to the world coordinate system W on the y-axis, and the third column (R _a2 , R _a6 , R _a10 ) is The direction component of the coordinate system A relative to the world coordinate system W on the z axis, the fourth column (R _a3 , R _a7 , R _a11 ) is the coordinate system A relative to the world coordinate system W, respectively on the x, y, and z axes translation component of .

Based on the above coordinate system conversion method, the coordinates of the boundary points of object a and object b in the world coordinate system W are obtained, and the range of the boundary points of object a and object b on the x, y, and z axes of the world coordinate system W is determined.

The method for judging the type of the spatial semantic relationship between object a and object b, specifically:

Through the relationship between the distance between the center point of object a and object b and the maximum boundary of the bbox of the two objects; if the following conditions are met, then proceed to S1, otherwise object a and object b are irrelevant, type=separated, the judgment conditions are as follows:

是任务子过程终止(p_i+1)时物体a的位置坐标；

是任务子过程终止时物体b的位置坐标；distance是计算两个坐标点之间的距离公式；

是任务子过程终止(p_i+1)时物体a分别在x、y、z轴上最大的边界值；

是任务子过程终止(p_i+1)时物体b分别在x、y、z轴上最大的边界值；in,

is the position coordinate of object a when the task sub-process terminates (p _i+1 );

is the position coordinate of object b when the task sub-process is terminated; distance is the formula for calculating the distance between two coordinate points;

is the maximum boundary value of object a on the x, y, and z axes respectively when the task sub-process terminates (p _i+1 );

are the maximum boundary values of the object b on the x, y, and z axes respectively when the task sub-process terminates (p _i+1 );

S1. Determine the type of spatial semantic relationship through the relationship between object a and object b on the x, y, and z axes of the world coordinate system W; if the following conditions are not met, proceed to S2. The specific formula is as follows:

是指物体a在世界坐标系W的x轴上的边界范围；

是指物体b在世界坐标系W的x轴上的边界范围；

是指物体a在世界坐标系W的y轴上的边界范围；

是指物体b在世界坐标系W的y轴上的边界范围；

是指物体a在世界坐标系W的z轴上的边界范围；

是指物体b在世界坐标系W的z轴上的边界范围。in,

Refers to the boundary range of object a on the x-axis of the world coordinate system W;

Refers to the boundary range of object b on the x-axis of the world coordinate system W;

Refers to the boundary range of object a on the y-axis of the world coordinate system W;

Refers to the boundary range of object b on the y-axis of the world coordinate system W;

Refers to the boundary range of object a on the z-axis of the world coordinate system W;

It refers to the boundary range of the object b on the z-axis of the world coordinate system W.

If the above conditions are met, the following judgments are made to determine the type of spatial semantic relationship. The specific formula is as follows:

的关系，确定空间语义关系类型，具体的公式如下：S2. Through the center coordinates (x _a , y _a , z _a ), (x _b , y _b , z _b ) of object a and object b in the world coordinate system and the two objects

relationship, to determine the type of spatial semantic relationship, the specific formula is as follows:

Integrate the semantic analysis results of the task sub-process s _j , build a task-based semantic analysis graph, and obtain the action sequence of the manipulator task, specifically: integrate the analysis module results of the task sub-process s _j , that is, the action sequence of the task sub-process s _j , The action object and the spatial semantic relationship of the action object, construct a task-based semantic analysis map, and obtain the action sequence of the manipulator task

其中

是子过程s_j的动作类型

是任务子过程s_j动作作用的物体列表。

in

is the action type of subprocess s _j

is the list of objects that the action of the task subprocess s _j acts on.

An embodiment of the present invention also provides a semantic parsing system for a task execution sequence of a manipulator, including: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the method as described above .

A semantic analysis system for manipulator task execution sequence, as shown in Figure 1, consists of three modules, including task sequence segmentation module, task sub-process analysis module, and semantic analysis graph construction module.

The task sequence segmentation module is responsible for obtaining the task execution point sequence of the manipulator and the state sequence of all objects in the scene corresponding to the execution point, which is determined by performing difference processing on the opening and closing states in the state sequence of the manipulator gripper with a task length of N Task execution point sequence segmentation point, each segmentation point divides the state sequence of the gripper of the robotic arm into n consecutive task sub-process sequences;

解析的任务子过程s_j中包含的所有物体的运动特征、获取的任务子过程s_j中发生变化的物体列表和与机械臂抓手直接接触的物体列表，分析每个任务子过程s_j中机械臂抓手的操作类型、状态发生变化的物体属性、机械臂抓手直接接触的物体之间的关系，确定任务子过程s_j机械臂抓手的动作类型及作用的物体，并根据任务子过程s_j结束时所有物体的状态特征，解析作用物体的空间语义关系；The task subprocess analysis module is responsible for the operation type of the manipulator arm gripper according to the determined task subprocess s _j

The motion characteristics of all the objects contained in the parsed task sub-process s _j , the list of objects changed in the acquired task sub-process s _j and the list of objects that are in direct contact with the gripper of the manipulator, are analyzed in each task sub-process s _j The operation type of the gripper of the manipulator, the properties of the objects whose state changes, and the relationship between the objects directly contacted by the _gripper of the manipulator determine the action type of the gripper of the manipulator and the objects it acts on, and according to the task The state characteristics of all objects at the end of the process _sj , and analyze the spatial semantic relationship of the acting objects;

The semantic analysis graph building module is responsible for integrating the semantic analysis results of the task sub-process s _j , constructing a task-based semantic analysis graph, and obtaining the task action sequence of the manipulator.

The above-mentioned semantic analysis system for the task execution sequence of the manipulator can be implemented as a computer program, stored in a hard disk, and recorded in a processor for execution, so as to implement the method of the embodiment of the present invention.

The embodiment of the present invention also provides a computer-readable medium storing computer program code, and the computer program code implements the above-mentioned method for semantic analysis of a task execution sequence of a manipulator when executed by a processor.

When a semantic analysis method for a task execution sequence of a manipulator is implemented as a computer program, it can also be stored in a computer-readable storage medium as a product. For example, computer-readable storage media may include, but are not limited to, magnetic storage devices (e.g., hard disk, floppy disk, magnetic stripe), optical disks (e.g., compact disk (CD), digital versatile disk (DVD)), smart cards, and flash memory devices ( For example, Electrically Erasable Programmable Read Only Memory (EPROM), card, stick, key drive). Additionally, various storage media described in embodiments of the present invention can represent one or more devices and/or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and/or storage media) capable of storing, containing and/or carrying code and/or instructions and/or data.

It should be understood that the above-described embodiments are illustrative only. Embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processing unit can be implemented in one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays ( FPGA), processor, controller, microcontroller, microprocessor and/or other electronic units designed to perform the functions described in the present invention, or a combination thereof.

The semantic analysis graph obtained by the above method for several simple tasks with different characteristics is shown in Figure 3. T1 represents the semantic analysis graph of "opening a certain door"; T2 represents "closing a certain door" T3 represents the semantic analysis diagram of "pressing a certain button"; T4 represents the semantic analysis diagram of "using one object to push another object"; T5 represents the semantic analysis diagram of "moving an object" Semantic analysis diagram; T6 represents the semantic analysis diagram of "clearing an object"; among them, the dotted line is an optional action, which is determined according to the learning task.

For the execution flow chart of complex tasks, for the convenience of display, the above simple tasks are combined and represented. When analyzing complex tasks, the analysis is still performed according to the above steps, and there is no distinction between simple and complex tasks. Here are the execution flow charts of several complex tasks, as shown in Figure 3. For example, T7 can be "put the object into the drawer", which can be decomposed into "pull the drawer open", "move the object into the drawer". "; T8 represents moving different objects multiple times, such as "arranging multiple cubes into a pyramid", "putting a complete tableware (plate, fork, cup, etc.)"; T9 can be "turning on or off the TV with the remote control machine", which can be decomposed into "point the remote control at the TV", "press the power button of the remote control"; T10 can be "turn on the TV with the remote control placed in the drawer", which can be decomposed into "turn the drawer Pull it away", "point the remote control at the TV", "press the power button of the remote control".

In the following, the complex task T10 in Figure 3 is specifically taken as an example "turn on the TV with the remote control placed in the drawer" to further expand the semantic analysis diagram, as shown in Figure 4 . The semantic analysis diagram of the complex task T10 is a set of continuous action sequences composed of three simple task combinations and the objects on which the actions act, expressed as

{(reach, drawer handle), (pull, drawer handle ^* , drawer), (reach, remote), (move, remote ^* ), (reach, remote button), (press, remote button ^* )} , where * represents the changed object and its attributes.

The semantic analysis diagram of T10 also includes objects related to the space of the acting object, which can be the sup object that the object belongs to and cannot be separated from, or the object with the closest semantic relationship in the space, and the priority of the relationship tightness is "include ">"Intersect">"Adjacency", if the acting object does not have the three types of relationship objects of "Contain", "Intersect", and "Adjacent", select the object with the closest distance in its "Distance" relationship as the related object object. This enables the robotic arm to further determine the target position of the action by means of the spatial semantic relationship of the object in addition to locating the target position through the object attributes.

It should be noted that in this application, relative terms such as first, second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations Any such actual relationship or order exists between. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Each embodiment in this specification is described in a related manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims (5)

1. A semantic analysis method of a manipulator task execution sequence, characterized in that, comprising the following steps: Obtain the task execution point sequence of the manipulator and the state sequence of all objects in the scene corresponding to the execution point. The task execution point sequence of the manipulator is specifically the state sequence of the gripper of the manipulator. Composition of the state sequence, by performing difference processing on the opening and closing state in the state sequence of the manipulator gripper with a task length of N to determine the task execution point sequence segmentation point, each segmentation point divides the state sequence of the robotic arm gripper into n a continuous sequence of task sub-processes; Determine the type of manipulation of the robotic arm gripper for the task subprocess _sj

Analyze the motion characteristics of all objects contained in the task sub-process s _j , obtain the list of objects that have changed in the task sub-process _{s j} and the list of objects that are in direct contact with the gripper of the manipulator, and determine the The type of action and the object to be acted on, and according to the state characteristics of all objects at the end of the task sub-process s _j , analyze the spatial semantic relationship of the acted object; Integrate the semantic analysis results of the task sub-process s _j , construct a semantic analysis graph based on the task execution sequence of the manipulator, and obtain the action sequence of the manipulator task; The acquisition of the task execution point sequence of the manipulator and the state sequence of all objects in the scene corresponding to the execution point, the task execution point sequence of the manipulator is specifically the state sequence of the gripper of the manipulator, and the state sequence of all objects is determined by each individual in the scene The state sequence composition of the object is determined by performing difference processing on the opening and closing states in the state sequence of the robot arm gripper with a task length of N to determine the sequence segmentation points of the task execution point, and each segmentation point divides the state sequence of the robotic arm gripper It is a sequence of n consecutive task sub-processes, specifically: The state sequence of the manipulator gripper is {GS ₁ , GS ₂ , GS ₃ ...,GS _i ,...,GS _N }, where the i-th state data of the manipulator gripper GS _i =(pos _i ,qua _i , open _i ), pos _i = (x _i , y _i , z _i ) represents the position coordinates of the i-th state of the gripper of the robotic arm, x _i represents the x-axis coordinate of the i-th state of the gripper of the robotic arm, and y _i represents the mechanical The y-axis coordinate of the i-th state of the arm gripper, z _i represents the z-axis coordinate of the i-th state of the robotic arm gripper; qua _i = (q _x ,q _y ,q _z ,q _w ) represents the first The direction of the i state, (q _x , q _y , q _z ) represents the rotation axis of the i-th state of the gripper of the robotic arm, q _x is the x-axis component of the rotation axis, q _y represents the y-axis component of the rotation shaft, and q _z represents The z-axis component of the rotation axis, q _w represents the angle component of the i-th state of the robotic arm gripper rotating around the rotation axis, open _i = {0,1} represents the opening and closing state of the i-th state of the robotic arm gripper, open _i = 0 is close, open _i = 1 is open; The state sequence of the single object is

The i-th state data of a single object a

name indicates the name of a single object a, pos _i = ( _xi , y _i , z _i ) indicates the position coordinates of the i-th state of a single object a; qua _i = (q _x ,q _y ,q _z ,q _w ) Indicates the direction of the i-th state of a single object a, color _i = (c _r , c _g , c _b ) represents the color of the i-th state of a single object a, and c _r represents the color of the i-th state of a single object a The color component on the R channel, c _g represents the color component of the color of the i-th state of a single object a on the G channel, and c _b represents the color component of the color of the i-th state of a single object a on the B channel; bbox _i ＝(x _min ,x _max ,y _min ,y _max ,z _min ,z _max ) indicates the size of the i-th state of a single object a, with the center of object a as the origin coordinate axis, (x _min ,x _max ) Indicates the range of the size of object a on the x-axis, (y _min , y _max ) indicates the range of the size of object a on the y-axis, (z _min , z _max ) indicates the range of the size of object a on the z-axis; rel={ None,sub} indicates whether a single object a is an inseparable component of an object, rel=None indicates that a single object a is not an inseparable component of any object, rel=sub indicates that a single object a is an inseparable component of an object sub, and sub indicates Object; R represents the rotation matrix of the coordinate system A with the center of a single object a as the origin relative to the world coordinate system W, and the specific expression of R is shown in the following formula:

Among them, the rotation matrix R consists of four columns, the first column (R ₀ , R ₄ , R ₈ ) is the direction component of the coordinate system A relative to the world coordinate system W on the x-axis, and the second column (R ₁ , R ₅ , R ₉ ) is the direction component of the coordinate system A relative to the world coordinate system W on the y-axis, and the third column (R ₂ , R ₆ , R ₁₀ ) is the direction component of the coordinate system A relative to the world coordinate system W on the z-axis The direction component, the fourth column (R ₃ , R ₇ , R ₁₁ ) is the translation component of the coordinate system A relative to the world coordinate system W on the x, y, and z axes respectively; Let the opening and closing state of the i-th state of the robotic arm gripper be open _i , and the opening and closing state of the next robotic arm gripper state be open _i+1 , 1≤i<N, and the two adjacent The opening and closing state in the state of the gripper of the robotic arm is subjected to difference processing to obtain the opening and closing state difference

which is

According to the opening and closing state difference

Determine the split point p _j in the task sequence according to the following formula, specifically:

Among them, j>1, 0<i<N, p ₁ =0, p _n-1 =N-1, p ₁ =0 means that the first split point is the starting point of the task sequence, p _n-1 =N-1 Indicates that the last split point is the end point of the task sequence; Form the segmentation point sequence {p ₁ ,p ₂ ,p ₃ ,…,p _j ,…,p _n-1 } from the segmentation points obtained in the state sequence of the gripper of the robotic arm, where p _j <N, each segmentation point will be The state sequence of the arm gripper is divided into n continuous sub-process sequences {s ₁ , s ₂ , s ₃ ,...,s _j ,...,s _n }, where s _j represents the split point p _j to the split point p _j The state sequence of the gripper of the manipulator with ₊₁ is recorded as the task sub-process s _j (p _j ,p _j+1 ); The analysis process of the motion characteristics of all objects contained in the task sub-process s _j is specifically: according to the starting position of a certain single object a contained in the task sub-process s _j

end position

starting direction

end direction

and the starting color

end color

Obtain the motion characteristics of a single object a in the task sub-process s _j

Then obtain the motion characteristics of all objects contained in the task sub-process s _j ; Among them, the motion characteristics of a single object a in the task sub-process s _j

The calculation formula is:

in:

in,

is the displacement of object a in task sub-process s _j ;

is the rotation angle of object a in task sub-process s _j j;

is the color change of object a in the task sub-process s _j ;

Indicates whether the object a disappears in the task sub-process s _j ,

Indicates that the object a disappears after passing through the task sub-process s _j ,

Indicates that the object a still exists after passing through the task sub-process s _j ;

are the position coordinates of the object a when the task sub-process s _j terminates, respectively,

are the position coordinates of the object a when the task sub-process s _j starts;

Indicates the rotation angle of the object a when it is rotated to the end of the task sub-process s _j ,

is the angle component of the object a when it rotates to the start p _j of the task sub-process s _j ;

Respectively represent the color channels of the object a when the task sub-process s _j starts,

Respectively represent the color channels of object a when the task sub-process s _j terminates;

Indicates that object a is included in the objects corresponding to the start execution point of the task sub-process s _j ;

It means that the object a is not included in the objects corresponding to the termination execution point of the task sub-process s _j ; According to the action type of the task sub-process _sj and the object of the action of the mechanical arm gripper, specifically: the properties of the object when the task sub-process sj(pj, pj+1) starts pj and terminates pj+1 from a single object a status data

, where the serial number i in the status data is equal to the serial numbers of the start pj and the end pj+1 in the task sub-process; The specific formula for determining the action type of the task sub-process sj robotic arm gripper is as follows:

Among them, opt _sj is the operation type of the gripper of the robotic arm in the task sub-process s _j ; len(pos_object _sj ) is the number of displaced objects in the task sub-process s _j ;

is the number of objects that rotate in the task sub-process _{s j} ; len( _{color_object sj} ) is the number of objects that change color in the task sub-process s _j ; object;

is the rel value of the object in direct contact with the gripper of the robotic arm in the task sub-process s _j , indicating that the object is an inseparable component of an object, rel=None indicates that the object is not an inseparable component of any object, rel=sup indicates that the object is an inseparable component of the object sup; len(clear_object _sj ) is the number of objects that disappear in the task sub-process s _j ; when the action type of the gripper of the manipulator is "move" and the gripper of the manipulator directly touches the object, it is regarded as an object sup When the components are inseparable, whether the action type of the task sub-process s _j is "push" or "pull", it should be further distinguished according to the distance between the object and the sup object. The specific formula is as follows:

in,

is the position coordinate of the displacement-changing object at the start p _j of the task sub-process;

is the position coordinate of the sup object of the displacement changing object at the start p _j of the task sub-process; distance is the formula for calculating the distance between two coordinate points; Define the relationship between the action type of the above-mentioned robotic arm gripper and the object it acts on, as follows: Arrive at object a, denoted as "arrived":

Object a is searched in t_objects for the object corresponding to the sub-process s _k which is after the sequence of sub-process s _j and is closest to it, namely (a, s _k ), j<k; Rotate object a, expressed as "rotation":

The object a is to find the object corresponding to s _j by searching t_objects, namely (a, s _j ); Moving object a, denoted as "moving":

In the object, a is to find the object corresponding to s _j by searching t_objects, that is, (a, s _j ), and a* is the object that object a finally reaches, that is, the object whose position has changed in pos_object _sj ; Pressing object a makes object b change color, denoted as "press":

Object a is the object whose position changes in pos_object _sj , and object b is the object whose color changes in color_object _sj ; Will hold object a and push object b, which means "indirect push": for

In the object, a is to find the object corresponding to s _j by searching t_objects, that is, (a, s _j ), object b is another object in pos_object _sj that is not object a, and b* is the object that object b finally arrives at; Hold object a and push object b, expressed as "push":

Object a is to find the object corresponding to s _j by searching t_objects, namely (a, s _j ), if the search fails, object a is empty; object b is the object whose position in pos_object _sj has changed; Grab object a and pull object b, denoted as "pull":

Object a is to find the object corresponding to s _j by searching t_objects, namely (a, s _j ); object b is the object whose position has changed in pos_object _sj ; Take object a and clear object b, denoted as "clear":

Object a is to find the object corresponding to s _j by searching t_objects, namely (a, s _j ); object b is the object whose position has changed in pos_object _sj ; According to the state characteristics of all objects at the end of the task sub-process _sj , analyze the spatial semantic relationship of the acting object, specifically: according to the state characteristics of all objects at the end of the task sub-process _sj , analyze the relationship between the acting object a and other objects object The spatial semantic relationship, expressed as

type expresses the relationship type of two objects; Transform the bbox boundaries of any two objects into the same coordinate system, establish a coordinate system A with the center of the object a as the origin, and point P on the coordinate system A

Coordinates transformed to the world coordinate system W

To determine the boundary point coordinates of object a in the world coordinate system W, the calculation formula is as follows:

Among them, R is the rotation matrix of the coordinate system A relative to the world coordinate system W, the rotation matrix R is composed of four columns, the first column (R _a0 , R _a4 , R _a8 ) is the coordinate system A relative to the world coordinate system W on the x-axis , the second column (R _a1 , R _a5 , R _a9 ) is the direction component of the coordinate system A relative to the world coordinate system W on the y-axis, and the third column (R _a2 , R _a6 , R _a10 ) is The direction component of coordinate system A relative to the world coordinate system W on the z axis, the last column (R _a3 , R _a7 , R _a11 ) is the coordinate system A relative to the world coordinate system W, respectively on the x, y, z axes translation component; Establish a coordinate system with the center of the object b as the origin, and determine the boundary point coordinates of the object b under the world coordinate system W; Determine the type of spatial semantic relationship between object a and object b; The method for judging the type of the spatial semantic relationship between object a and object b is specifically: through the relationship between the center point distance between object a and object b and the maximum boundary of the bbox of the two objects; if the following conditions are met, then proceed to S1, otherwise object a It is irrelevant to object b, type=separated, and the judgment conditions are as follows:

in,

is the position coordinate of object a when the task sub-process terminates (p _i+1 );

is the position coordinate of object b when the task sub-process is terminated; distance is the formula for calculating the distance between two coordinate points;

is the maximum boundary value of object a on the x, y, and z axes respectively when the task sub-process terminates (p _i+1 );

are the maximum boundary values of the object b on the x, y, and z axes respectively when the task sub-process terminates (p _i+1 ); S1. Determine the type of spatial semantic relationship through the relationship between object a and object b on the x, y, and z axes of the world coordinate system W; if the following conditions are not met, proceed to S2. The specific formula is as follows:

In the formula,

Refers to the boundary range of object a on the x-axis of the world coordinate system W;

Refers to the boundary range of object b on the x-axis of the world coordinate system W;

Refers to the boundary range of object a on the y-axis of the world coordinate system W;

Refers to the boundary range of object b on the y-axis of the world coordinate system W;

Refers to the boundary range of object a on the z-axis of the world coordinate system W;

Refers to the boundary range of object b on the z-axis of the world coordinate system W; If the above conditions are met, the following judgments are made to determine the type of spatial semantic relationship. The specific formula is as follows:

S2. Through the center coordinates (x _a , y _a , z _a ) and (x _b , y _b , z _b ) of object a and object b in the world coordinate system and the two objects

relationship, to determine the type of spatial semantic relationship, the specific formula is as follows:

2. the semantic analysis method of a kind of manipulator task execution sequence according to claim 1, it is characterized in that, the determination process of the operation type of the manipulator gripper of described task subprocess _sj , specifically: according to task subprocess The position change, direction change, and opening and closing state of the manipulator gripper in the process s _j determine the operation type opt _sj of the manipulator gripper in the task sub-process s _j , and obtain the operations of the manipulator gripper in n task sub-processes in time sequence sequence: According to the task sub-process s _j (p _j ,p _j+1 ) the initial position of the gripper of the manipulator

end position

starting direction

end direction

The motion characteristics of the robot arm gripper in the task sub-process are obtained by calculating the opening and closing state

Motion characteristics of the gripper of the robotic arm

The calculation formula is

In the formula,

is the displacement of the gripper of the manipulator in the task sub-process s _j ;

is the rotation angle of the gripper of the manipulator in the task sub-process s _j ;

is the opening and closing state of the gripper of the manipulator in the task sub-process s _j ; in,

In the formula,

Indicates the angle component of the rotation of the gripper of the manipulator around the rotation axis when it rotates to the end of the sub-process pj+1,

Indicates the angular component of the rotation around the axis of rotation of the gripper of the manipulator when it rotates to the start of the subprocess p _j ; The operation type of the robot arm gripper of the task subprocess s _j

Including three, namely: reach means arrival, move means movement, rotate means rotation,

The specific calculation formula is:

In the formula,

Indicates the displacement of the gripper of the robotic arm in the task sub-process s _j , and threshold _dis is the threshold for determining the displacement of the gripper of the robotic arm;

Indicates the rotation angle of the gripper of the manipulator in the task sub-process s _j , threshold _θ is the threshold for judging the rotation change of the gripper of the manipulator. 3. the semantic analysis method of a kind of manipulator task execution sequence according to claim 1, is characterized in that, the object list that changes in the described task sub-process s _j and the object list that directly contacts with manipulator gripper The acquisition process is as follows: Obtain the movement characteristics of the objects that have displacement, rotation, color change, and disappearance in the task sub-process s _j respectively, and store them in the state data when the task sub-process s _j terminates

list,

list,

list,

list; According to the state data of the gripper of the manipulator and the state data of all objects, obtain the object list t_objects in the task sub-process s _j that is in direct contact with the gripper of the manipulator. 4. A semantic analysis system of a manipulator task execution sequence, characterized in that it comprises: memory for storing instructions executable by the processor; A processor, configured to execute the instructions to implement the method according to any one of claims 1-3. 5. A computer-readable medium, characterized in that computer program code is stored, and the computer program code realizes the method according to any one of claims 1-3 when executed by a processor.

Images