CN120328089A - A method and device for identifying and grabbing empty luggage basket - Google Patents

Abstract

The invention discloses a luggage empty basket identification method and a device, the method comprises the steps of collecting images of a luggage conveyer belt and the movement conditions of surrounding personnel, predicting the movement track and the behavior of the surrounding personnel according to the movement conditions of the surrounding personnel, the method comprises the steps of calculating the motion trail of the grabbing mechanical arm based on a prediction result, obtaining an optimal motion trail based on the establishment of multi-target trail planning, determining grabbing sequences of a plurality of grabbing targets and performing algorithm optimization. The invention realizes efficient luggage empty basket grabbing, can simultaneously recover a plurality of empty baskets, predicts the state of the luggage basket in advance based on the motion trail and the motion of passengers, ensures the running stability and the safety of the empty basket recovery mechanical arm under the condition of considering the optimal time and the energy consumption, and can make path planning more quickly and simply for the existing path modes in the historical data.

Description

Luggage empty basket recognition and grabbing method and device

Technical Field

The invention relates to the field of luggage empty basket recognition and grabbing, in particular to a luggage empty basket recognition and grabbing method and device.

Background

As the volume of aviation passenger continues to rise, the scale of modern hub airport terminal continues to expand, and the high-efficiency guarantee capability of a luggage system faces serious challenges. For non-standardized luggage such as soft packages, special-shaped backpacks and the like, the luggage is normally loaded in a luggage basket during luggage delivery. The operation not only ensures the passing stability of the baggage in the high-speed conveying line, but also can effectively avoid the winding of the flexible package in the conveying process.

In the prior art, an automatic luggage basket grabbing system is provided, an empty basket image grabbing is completed through a visual camera, a cooperative robot is transmitted in real time, and the grabbing of the empty basket is achieved through the robot, but the grabbing robot in the prior art only grabs one basket at a time, so that the efficiency is low, intelligent planning and obstacle avoidance cannot be achieved, the luggage extraction action of passengers cannot be predicted in advance, the overall efficiency is low, and the performance is poor.

Disclosure of Invention

(One) solving the technical problems

In order to solve the technical problems, the invention provides a luggage empty basket identification grabbing method and device.

(II) technical scheme

In order to solve the technical problems and achieve the aim of the invention, the invention is realized by the following technical scheme:

a luggage empty basket recognition and grabbing method comprises the following steps:

s1, image acquisition and preprocessing, including acquiring a baggage conveyor belt image and surrounding personnel movement conditions through a camera;

S2, luggage basket identification and position tracking;

identifying the position of the basket in the image based on the specific shape of the basket, and tracking the position of the basket in combination with the running speed of the conveyor belt;

S3, predicting the motion trail and behaviors of surrounding personnel according to the motion condition of the surrounding personnel, and if the personnel are identified to pick up luggage, acquiring the future luggage basket emptying time and the moving position according to the motion speed and the moving speed of the conveyor belt;

s4, calculating a motion trail of the grabbing mechanical arm based on a prediction result, wherein the method comprises the steps of obtaining an optimal motion trail based on the establishment of multi-target trail planning, and determining grabbing sequences of a plurality of grabbing targets;

s5, optimizing a motion trail planning algorithm;

Further, the step S3 further includes processing the image acquired by the camera based on YoloV target detection algorithm, extracting the pedestrian from the background area of the image, adding a1×1 convolution module between YoloV and CSPBlock of the recognition model, and in the spatial pyramid pooling module, parallel adopting maximum pooling and average pooling and adding the results of the two.

Further, the motion trail prediction for surrounding people comprises prediction based on a pedestrian history trail sequence and pedestrian directions.

Furthermore, in the behavior prediction, the action recognition model based on the graph neural network realizes action data modeling and classification by constructing a dynamic relationship topological structure.

Further, the objective function of the multi-objective trajectory planning is as follows:

L=w₁L₁+w₂L₂+w₃L₃+ε

L ₁ is a time objective function, L ₂ is an energy consumption objective function, L ₃ is a risk level objective function, epsilon is a penalty factor, and w ₁、w₂、w₃ is a weight factor of the three respectively.

Further, the L ₃ risk degree objective function is determined based on the height and the distance from surrounding personnel and the movement errors when the mechanical arm moves after clamping the empty basket and the mechanical arm rotates;

r ₁ is a distance risk weight during linear motion, dis ₁ is a minimum distance between the mechanical arm and surrounding personnel during linear motion, sigma ₁ is a distance sensitivity coefficient during linear motion, h ₁ is a mechanical arm end height during linear motion, delta ₁ is a height adjustment coefficient during linear motion, r ₂ is a distance risk weight during rotational motion, dis ₂ is a minimum distance between the mechanical arm and surrounding personnel during rotational motion, sigma ₂ is a distance sensitivity coefficient during rotational motion, h ₂ is a mechanical arm end height during rotation, delta ₂ is a height adjustment coefficient during rotation, r ₃ is a motion error risk weight, x _ac is an actual position, and x _de is a set position.

Furthermore, epsilon comprises acceleration and jerk penalty, and the mechanical vibration and abrasion are caused by the speed and the change of the acceleration in the movement process of the mechanical arm, so that the acceleration and jerk are introduced;

Wherein k _a,k_j is a parameter adjusted according to the rigidity of the mechanical arm and the task dynamics respectively; In order for the acceleration to be a function of the acceleration, For jerk, n ₁ and n ₂ are the acceleration and the number of jerk changes, respectively.

Further, the step S5 comprises constructing a path mode, matching the similarity and selecting the path mode, wherein the step comprises the steps of acquiring a group of data of the position of an empty basket to be processed with a complete motion track and the predicted free time in the historical data, clustering the number of the empty baskets collected according to the complete motion track and the relative positions according to the complete motion track from the empty position of a mechanical arm to the position of a recovery empty basket returned to the stacking position, and carrying out similarity matching on the data to be processed, and planning by adopting the existing motion track in a historical database according to the matching situation.

The invention also provides a luggage empty basket identification grabbing device, which comprises an image acquisition and preprocessing module, a control module and a control module, wherein the image acquisition and preprocessing module is used for acquiring images of a luggage conveyor belt and the movement condition of surrounding personnel through a camera;

A basket identification and position tracking module for identifying the position of the basket in the image based on the basket specific shape and tracking the basket position in combination with the running speed of the conveyor belt;

the surrounding personnel movement condition prediction module is used for predicting movement tracks and behaviors of surrounding personnel;

The grabbing mechanical arm motion track calculation module is used for obtaining an optimal motion track based on the establishment of multi-target track planning and determining grabbing sequences of a plurality of grabbing targets;

and the motion trail planning algorithm optimization module is used for optimizing the motion trail planning algorithm in a lightweight way based on the path pattern recognition.

In addition, to achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon program instructions of a baggage basket identification grabbing method, the program instructions of the baggage basket identification grabbing being executable by one or more processors to implement the steps of the baggage basket identification grabbing method as described above.

(III) beneficial effects

Compared with the prior art, the invention has the beneficial effects that:

(1) According to the invention, efficient luggage empty basket grabbing is realized, a plurality of empty baskets are recovered while starting from an origin point and returning to the origin point at one time, and the state of the luggage basket is predicted in advance based on the motion trail and the motion of a passenger, so that the grabbing mechanical arm can reach a designated position in advance, and the efficiency is improved;

(2) The invention intelligently plans the grabbing motion track and the action of the mechanical arm based on an improved path planning method, combines the planning of time, energy consumption and risk degree mechanical energy track, combines punishment factors, and ensures the running stability and safety of the empty basket recovery mechanical arm under the condition of considering the optimal time and energy consumption.

(3) The invention optimizes the motion trail planning algorithm, selects the existing planning trail by matching the constructed path modes, reduces the complexity of path operation, and can make path planning more quickly and simply for the existing path modes in the historical data.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

fig. 1 is a schematic flow chart of a baggage empty basket recognition grabbing method according to an embodiment of the application.

Detailed Description

Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

Other advantages and effects of the present disclosure will become readily apparent to those skilled in the art from the following disclosure, which describes embodiments of the present disclosure by way of specific examples. It will be apparent that the described embodiments are merely some, but not all embodiments of the present disclosure. The disclosure may be embodied or practiced in other different specific embodiments, and details within the subject specification may be modified or changed from various points of view and applications without departing from the spirit of the disclosure. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict. All other embodiments, which can be made by one of ordinary skill in the art without inventive effort, based on the embodiments in this disclosure are intended to be within the scope of this disclosure.

It should also be noted that the illustrations provided in the following embodiments merely illustrate the basic concepts of the disclosure by way of illustration, and only the components related to the disclosure are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.

Referring to fig. 1, a baggage empty basket recognition grasping method includes the steps of:

s1, image acquisition and preprocessing, including acquiring a baggage conveyor belt image and surrounding personnel movement conditions through a camera;

S2, luggage basket identification and position tracking;

identifying the position of the basket in the image based on the specific shape of the basket, and tracking the position of the basket in combination with the running speed of the conveyor belt;

s3, predicting the motion trail and the behavior of surrounding personnel according to the motion condition of the surrounding personnel, wherein the method comprises the following steps:

s31, extracting surrounding personnel, processing an image acquired by a camera based on YoloV target detection algorithm, and extracting pedestrians from an image background area;

Because the complex environment area is identified in an airport, people can be interfered by surrounding environment during identification, especially, some humanoid billboards and the like can be used for carrying out false identification on the identification, in order to further improve the identification of surrounding people, a 1X1 convolution module is added between YoloV and CSPBlock of an identification model, and in a space pyramid pooling module, a strategy of maximum pooling and average pooling and adding results of the maximum pooling and the average pooling is adopted in parallel, so that the traditional single pooling operation is replaced. The design can better keep the characteristic information of different layers and enhance the expression capability of the characteristics. Then, through the improved feature pyramid structure, the feature image output by CSPBlock is added with the feature image output by the feature pyramid, so that the loss of shallow semantic information is effectively avoided, and the overall performance of the model is improved.

S32 motion trail prediction

Let the foot position of the ith person at time t be expressed as

Wherein, the Coordinates of foot positions, the foot position sequence is

In the area close to the luggage conveyor where there is a basket, i.e. indicating that he is about to pick up the luggage or not. The invention adds a pedestrian direction as input based on a history track sequence of people, namely learning distribution p (FP _f|FP_obs,OP_obs), wherein FP _f is a predicted pedestrian motion track and OR _obs is a pedestrian direction.

Alternatively, the present invention obtains the trajectory of the person's movement at each future time based on the LSTM network.

S33, predicting the behavior, namely predicting whether surrounding personnel pick up luggage of a luggage basket or not, wherein the method is realized by the following steps:

The action recognition algorithm based on the multi-feature fusion graph neural network can model and classify and predict the action data of surrounding people by fusing various feature information and utilizing the graph neural network model. Assume that there are N samples and M action categories, each sample having K features including joint angle, body posture, movement speed, etc.

And the action recognition model based on the graph neural network realizes data modeling and classification by constructing a dynamic relationship topological structure. The architecture adopts a dynamic relationship modeling and characteristic propagation mechanism between nodes, and captures the time-space association characteristic in the action sequence by using the message transmission paradigm of the graph structure. In the feature extraction stage, the layering graph rolling operation respectively learns the local joint movement mode and the global limb cooperation rule, and the expression capability of key action features is enhanced through self-adaptive edge weight adjustment. The network depth merges node attributes (coordinates and speeds) and edge attributes (joint distance and motion phase), and multi-modal feature interaction is achieved by adopting a multi-head attention mechanism. And finally, mapping the high-order graph representation to an action tag space through a full-connection classification layer to finish the end-to-end identification task. The framework fully mines topological relevance in action data through hierarchical feature aggregation and context sensing mechanisms of the graph structure, and excellent classification robustness and generalization capability are shown in a complex scene.

S34, predicting corresponding luggage basket vacation time and moving position according to personnel picking luggage intention.

If the person is identified to pick up the luggage, the luggage basket emptying time and the moving position at the future moment are obtained according to the action speed and the moving speed of the conveyor belt.

S4, calculating motion trail of the grabbing mechanical arm based on prediction results

According to the invention, the process of starting from the origin point and returning to the origin point at one time is realized, a plurality of empty baskets are recovered simultaneously, an optimal motion track is obtained based on the establishment of a multi-target track plan, the grabbing sequence of a plurality of grabbing targets is determined, and the mechanical arm is rotated according to the position and the angle of the next empty basket to be grabbed, and the grabbed empty basket and the next empty basket to be grabbed are overlapped and grabbed together at the same angle. The method comprises the following steps:

S41, constructing a mechanical arm motion model

The method comprises the steps of establishing a mechanical arm motion model based on an improved D-H modeling method, setting an origin of a connecting rod coordinate system at a joint of the joint and a previous connecting rod joint, setting a Z axis of the coordinate system as a joint line of the joint and a next connecting rod joint, obtaining a homogeneous transformation matrix between adjacent coordinate systems according to an improved D-H parameter method, and establishing the origin of the coordinate at the head end of the connecting rod by the improved D-H parameter coordinate system, so that parameter interpretation ambiguity caused by coordinate system drift is eliminated, and the derivation process of the homogeneous transformation matrix between the adjacent coordinate systems is remarkably simplified.

S42, establishing a multi-target track planning model

S421, establishing a model

The invention performs track planning based on an improved Diikstra algorithm, combines the planning of time, energy consumption and risk degree mechanical energy tracks based on future luggage basket vacation time and moving position obtained in the step S3, combines punishment factors, and ensures the operation stability and safety of the empty basket recovery mechanical arm under the condition of considering optimal time and energy consumption.

S422, determining an objective function

L=w₁L₁+w₂L₂+w₃L₃+ε

L ₁ is a time objective function, L ₂ is an energy consumption objective function, L ₃ is a risk degree objective function, epsilon is a penalty factor, and w ₁、w₂、w₃ is a weight factor of the three respectively;

wherein, the

T _i is the time taken by the ith sub-path to move, and n is the number of sub-paths contained in the whole movement path;

P _i is the power of the i-th discretization sub-path linear motion, the motions 1 to n ₁ are idle, the motions n ₁ +1 to n are the loads with empty basket, k is the number of the mechanical arm rotation and the end execution action, and P _j is the power consumption of the corresponding mechanical arm rotation and the end execution action.

The L ₃ risk degree objective function is determined based on the height and the distance from surrounding personnel and the movement errors when the mechanical arm moves after clamping the empty basket and the mechanical arm rotates;

r ₁ is a distance risk weight during linear motion, dis ₁ is a minimum distance between the mechanical arm and surrounding personnel during linear motion, sigma ₁ is a distance sensitivity coefficient during linear motion, h ₁ is a mechanical arm end height during linear motion, delta ₁ is a height adjustment coefficient during linear motion, r ₂ is a distance risk weight during rotational motion, dis ₂ is a minimum distance between the mechanical arm and surrounding personnel during rotational motion, sigma ₂ is a distance sensitivity coefficient during rotational motion, h ₂ is a mechanical arm end height during rotation, delta ₂ is a height adjustment coefficient during rotation, r ₃ is a motion error risk weight, x _ac is an actual position, and x _de is a set position.

Wherein the risk weight in the case of rotational movement is higher than in the case of linear movement;

Epsilon comprises acceleration and jerk penalty, and the mechanical vibration and abrasion are caused by the speed and the change of the acceleration in the motion process of the mechanical arm, so that the acceleration and jerk are introduced;

Wherein k _a,k_j is a parameter adjusted according to the rigidity of the mechanical arm and the task dynamics respectively; In order for the acceleration to be a function of the acceleration, For jerk, n ₁ and n ₂ are the acceleration and the number of jerk changes, respectively.

S423, performing dynamic track planning;

and carrying out motion track planning on the mechanical arm by adopting a dynamic planning method, specifically, predicting the motion track and the behavior of surrounding motion personnel according to real-time video, matching the planned track of the mechanical arm, and if collision between the planned track of the mechanical arm and the behavior of the surrounding personnel is detected to possibly happen, carrying out motion track planning on the mechanical arm again.

S5, optimizing a motion trail planning algorithm;

Because the motion trail planning algorithm has higher time complexity and larger required calculation resource, and the environment and personnel motion modes at the luggage extraction place are relatively fixed, the invention optimizes the motion trail planning algorithm in a light-weight way based on path mode identification, and specifically comprises the steps of constructing a path mode, matching the similarity and selecting the path mode;

S51, constructing a path mode, namely acquiring a group of data of the position of an empty basket to be processed and the predicted empty time in the historical data by taking a complete motion track as a unit, starting from a stacking position according to empty load of a mechanical arm to returning the empty basket to the stacking position as a complete motion track, and clustering according to the quantity and the relative positions of the empty baskets collected by the complete motion track to construct R path modes;

Optionally, the clustering method includes clustering by using a random forest.

Matching the number and the relative positions of empty baskets of the data to be processed with the R path modes in the previous step, and if the matching degree is greater than a set threshold value, considering that the two path modes belong to the same path mode;

And S53, planning by adopting the existing motion trail in the historical database according to the matching situation.

Therefore, the complexity of path operation is reduced, and path planning can be more quickly and simply performed on the path modes existing in the historical data.

According to the method, efficient luggage empty basket grabbing is achieved, the state of the luggage basket is predicted in advance based on the motion trail and the motion of a passenger, the grabbing mechanical arm can reach a designated position in advance, efficiency is improved, intelligent planning is conducted on the grabbing motion trail and the motion of the mechanical arm based on an improved path planning method, planning of time, energy consumption and risk degree mechanical energy trail is combined, punishment factors are combined, and running stability and safety of the empty basket recovery mechanical arm are guaranteed under the condition that optimal time and energy consumption are considered. The motion trail planning algorithm is optimized, and the existing planning trail is selected by matching the constructed path modes, so that the complexity of path operation is reduced, and the path planning can be more quickly and simply performed on the existing path modes in the historical data.

The embodiment of the invention also provides a luggage empty basket identification grabbing device, which comprises:

the image acquisition and preprocessing module is used for acquiring images of the luggage conveyor belt and the movement condition of surrounding personnel through the camera;

A basket identification and position tracking module for identifying the position of the basket in the image based on the basket specific shape and tracking the basket position in combination with the running speed of the conveyor belt;

the surrounding personnel movement condition prediction module is used for predicting movement tracks and behaviors of surrounding personnel;

The grabbing mechanical arm motion track calculation module is used for obtaining an optimal motion track based on the establishment of multi-target track planning and determining grabbing sequences of a plurality of grabbing targets;

and the motion trail planning algorithm optimization module is used for optimizing the motion trail planning algorithm in a lightweight way based on the path pattern recognition.

In addition, the embodiment of the invention also provides a computer readable storage medium, and the computer readable storage medium stores program instructions of the luggage basket identification grabbing method, and the program instructions of the luggage basket identification grabbing method can be executed by one or more processors to realize the steps of the luggage basket identification grabbing method.

The above examples are only illustrative of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention, and various modifications and improvements made by those skilled in the art to the technical solution of the present invention should fall within the scope of protection defined by the claims of the present invention without departing from the spirit of the present invention.

Claims (10)

1. A luggage empty basket identification method, characterized by comprising the following steps:

s1, image acquisition and preprocessing, including acquiring a baggage conveyor belt image and surrounding personnel movement conditions through a camera;

S2, luggage basket identification and position tracking;

identifying the position of the basket in the image based on the specific shape of the basket, and tracking the position of the basket in combination with the running speed of the conveyor belt;

S3, predicting the motion trail and behaviors of surrounding personnel according to the motion condition of the surrounding personnel, and if the personnel are identified to pick up luggage, acquiring the future luggage basket emptying time and the moving position according to the motion speed and the moving speed of the conveyor belt;

s4, calculating a motion trail of the grabbing mechanical arm based on a prediction result, wherein the method comprises the steps of obtaining an optimal motion trail based on the establishment of multi-target trail planning, and determining grabbing sequences of a plurality of grabbing targets;

and S5, optimizing a motion trail planning algorithm.

2. A baggage claim 1 wherein step S3 further comprises processing the image captured by the camera based on YoloV target detection algorithm, extracting pedestrians from the background area of the image, adding a 1 x 1 convolution module between YoloV and CSPBlock of the identification model, and in the spatial pyramid pooling module, using the maximum pool and average pool in parallel and adding the results of the two.

3. A luggage void identification method as claimed in claim 2, wherein the prediction of the motion profile of surrounding personnel comprises a prediction based on a historical sequence of pedestrian profiles and pedestrian direction.

4. A luggage void basket recognition method according to claim 2, wherein the action recognition model based on the graph neural network in the action prediction realizes the modeling and classification of the action data by constructing a dynamic relationship topological structure.

5. A baggage empty basket recognition method according to claim 1, wherein the objective function of the multi-objective trajectory planning is as follows:

L=w₁L₁+w₂L₂+w₃L₃+ε

L ₁ is a time objective function, L ₂ is an energy consumption objective function, L ₃ is a risk level objective function, epsilon is a penalty factor, and w ₁、w₂、w₃ is a weight factor of the three respectively.

6. A luggage void basket recognition method according to claim 5, wherein the L ₃ risk level objective function is determined based on the height and distance from surrounding personnel and movement errors when the robotic arm moves after gripping the void basket and the robotic arm rotates;

r ₁ is a distance risk weight during linear motion, dis ₁ is a minimum distance between the mechanical arm and surrounding personnel during linear motion, sigma ₁ is a distance sensitivity coefficient during linear motion, h ₁ is a mechanical arm end height during linear motion, delta ₁ is a height adjustment coefficient during linear motion, r ₂ is a distance risk weight during rotational motion, dis ₂ is a minimum distance between the mechanical arm and surrounding personnel during rotational motion, sigma ₂ is a distance sensitivity coefficient during rotational motion, h ₂ is a mechanical arm end height during rotation, delta ₂ is a height adjustment coefficient during rotation, r ₃ is a motion error risk weight, x _ac is an actual position, and x _de is a set position.

7. A method of identifying a luggage void as claimed in claim 5 wherein epsilon includes acceleration and jerk penalties, the speed and acceleration variations during the movement of the mechanical arm causing mechanical vibration and wear, the invention introducing accelerations and jerks;

Wherein k _a,k_j is a parameter adjusted according to the rigidity of the mechanical arm and the task dynamics respectively; In order for the acceleration to be a function of the acceleration, For jerk, n ₁ and n ₂ are the acceleration and the number of jerk changes, respectively.

8. The method for identifying empty basket of luggage as claimed in claim 1, wherein the step S5 comprises constructing path modes, matching similarity and selecting path modes, wherein the method comprises obtaining a group of data of the position of an empty basket to be processed with a complete motion track and predicted free time in historical data, starting from a stacking position to a position of a recovered empty basket returning to the stacking position according to empty load of a mechanical arm as a complete motion track, clustering according to the number and relative positions of the empty baskets collected by the complete motion track, constructing a plurality of path modes, matching the similarity of the data to be processed, and planning by adopting the existing motion track in a historical database according to the situation of matching.

9. An apparatus based on a baggage air basket identification method according to any one of claims 1 to 8, comprising:

the image acquisition and preprocessing module is used for acquiring images of the luggage conveyor belt and the movement condition of surrounding personnel through the camera;

A basket identification and position tracking module for identifying the position of the basket in the image based on the basket specific shape and tracking the basket position in combination with the running speed of the conveyor belt;

the surrounding personnel movement condition prediction module is used for predicting movement tracks and behaviors of surrounding personnel;

The grabbing mechanical arm motion track calculation module is used for obtaining an optimal motion track based on the establishment of multi-target track planning and determining grabbing sequences of a plurality of grabbing targets;

and the motion trail planning algorithm optimization module is used for optimizing the motion trail planning algorithm in a lightweight way based on the path pattern recognition.

10. A computer readable storage medium having stored thereon program instructions of a baggage basket identification method, the baggage basket identification program instructions being executable by one or more processors to implement the steps of the baggage basket identification method according to any one of claims 1 to 8.