CN118544361A

CN118544361A - Luggage handling system based on computer vision and mechanical arm

Info

Publication number: CN118544361A
Application number: CN202411002096.5A
Authority: CN
Inventors: 唐红武; 李睿; 申震云; 吴林; 张家辉; 吴坤; 侯远哲; 刘森林
Original assignee: China Travelsky Mobile Technology Co Ltd
Current assignee: China Travelsky Mobile Technology Co Ltd
Priority date: 2024-07-25
Filing date: 2024-07-25
Publication date: 2024-08-27
Anticipated expiration: 2044-07-25
Also published as: CN118544361B

Abstract

The invention provides a luggage handling system based on computer vision and a mechanical arm, which relates to the technical field of luggage handling, and comprises: the mechanical arm and the vision device are respectively connected with the upper computer in a communication way; the end part of the mechanical arm is provided with a first conveyor belt, so that when the luggage falls off from the luggage conveying device, the luggage is connected through the first conveyor belt; determining current residual stacking space information in the luggage van; determining pose information and size information of the current luggage; determining the stacking pose information of the current luggage on the luggage truck; determining a target orientation of the first conveyor belt when the first conveyor belt is docked with the current luggage; controlling the first conveyor to dock the current baggage dropped from the baggage conveyor in a target orientation; the current baggage on the first conveyor belt is piled up to a corresponding piling position of the piling pose information on the baggage car; the carrying system of the invention can not cause any damage to the luggage when the luggage with different sizes, different weights and different degrees of hardness is connected.

Description

Luggage handling system based on computer vision and mechanical arm

Technical Field

The invention relates to the technical field of baggage handling, in particular to a baggage handling system based on computer vision and a mechanical arm.

Background

In the field of civil aviation, baggage handling is a significant challenge for the modern aviation industry, and as the passenger flow at airports is increasing, the number of baggage for passengers is also increasing; for the processing of the baggage, the conventional baggage handling mode mostly adopts manual handling, but the efficiency of manual handling of the baggage is lower; in order to improve the carrying efficiency of luggage, the common modes are: the luggage is clamped by a mechanical arm with a mechanical clamping jaw at the front end, so that the luggage is carried; but the size, the weight and the hardness degree of the luggage are different, and when the luggage is clamped by using the mechanical clamping jaw, the luggage is easy to fall off, so that the luggage is damaged.

Disclosure of Invention

The invention provides a luggage carrying system based on computer vision and a mechanical arm, aiming at the technical problems that the luggage carrying efficiency and the stable operation of airport luggage transportation are affected and the probability of luggage damage is increased when the luggage is carried in a manual carrying mode.

Based on the technical problems, the application provides a luggage handling system based on computer vision and a mechanical arm, which comprises: the system comprises a mechanical arm, an upper computer and a vision device; the mechanical arm and the vision device are respectively in communication connection with the upper computer; the vision device is used for collecting a luggage car image and a luggage image; the end part of the mechanical arm is provided with a first conveyor belt, so that when the luggage falls off from the luggage conveying device, the luggage is connected through the first conveyor belt;

The upper computer is used for executing the following steps:

S100, acquiring current residual stacking space information and pose information and size information of current baggage in a baggage car in response to the arrival of the baggage at a designated position of a baggage conveying device; the method comprises the steps that current residual stacking space information in a luggage car is obtained according to a current luggage car image, and pose information and size information of current luggage are obtained according to the current luggage image;

S200, determining stacking pose information and position information of the current luggage on the luggage truck according to the current residual stacking space information and the size information of the current luggage in the luggage truck;

s300, determining the target orientation of the first conveyor belt when the current luggage is connected according to the stacking pose information of the current luggage on the luggage truck and the pose information of the current luggage; when the first conveyor belt is used for connecting the current luggage with the target direction, the length direction of the current luggage is parallel or perpendicular to the conveying direction of the first conveyor belt;

S400, controlling the first conveyor belt to dock the current baggage falling from the baggage conveyor with the target direction;

S500, controlling the first conveyor belt to move to a position corresponding to the position information of the current luggage on the luggage car, and controlling the first conveyor belt to rotate forward, so that the current luggage drops to the position corresponding to the position information of the current luggage on the luggage car in a posture corresponding to the stacking posture information.

Further, the vision device comprises a first image acquisition device and a second image acquisition device; the first image acquisition equipment is used for acquiring a luggage image; the second image acquisition device is used for acquiring images of the luggage van.

Further, the first image acquisition device comprises a laser radar and/or an RGB-D camera; the second image acquisition device comprises a lidar and/or an RGB-D camera.

Further, step S300 includes the steps of:

S310, determining a first included angle between the central line of the current luggage and the central line of the luggage car according to the stacking pose information of the current luggage on the luggage car; the central line of the current luggage is parallel to the length direction of the current luggage, and the central line of the luggage car is parallel to the length direction of the luggage car;

S320, determining a second included angle between the central line of the current luggage and the conveying direction of the luggage conveying device according to the pose information of the current luggage;

S330, if the first included angle is within a first preset included angle range, determining the direction parallel to the central line of the first conveyor belt and the central line of the current luggage as a target direction;

S340, if the first included angle is within a second preset included angle range, determining the direction perpendicular to the central line of the first conveyor belt and the central line of the current luggage as a target direction; wherein, any included angle in the first preset included angle range is smaller than the minimum included angle in the second preset included angle range.

Further, the first included angle ranges from 0 degrees to 20 degrees, and the second included angle ranges from 70 degrees to 90 degrees.

Further, after step S300 and before step S400, the upper computer is further configured to perform the following steps:

S301, if the length direction of the current baggage is not parallel or perpendicular to the conveying direction of the first conveyor belt, controlling the first conveyor belt to reversely rotate.

Further, the system further comprises: a controller; the controller is respectively in communication connection with the upper computer and the mechanical arm; a read-only space, a write-only space and a readable and writable space are preset in the controller;

The upper computer is preset with a control unit, and the control unit comprises an interface calling layer and a control module layer; the control unit layer is in communication connection with the control module layer;

the interface calling layer is provided with a controller cache, and a read-only space, a write-only space and a readable and writable space are preset in the controller cache; and the addresses in the read-only space, the write-only space and the readable and writable space in the controller cache are mapped with the addresses in the read-only space, the write-only space and the readable and writable space in the controller one by one respectively.

Further, the upper computer is preset with a synchronization process, and the synchronization process is used for executing the following steps:

S610, acquiring a value of an updated address in a read-only space of the controller and a value of the updated address in a write-only space of the controller cache every preset time interval;

S611, writing the value of each address with update in the read-only space of the controller into the corresponding address in the read-only space of the controller cache, and writing the value of each address with update in the write-only space of the controller into the corresponding address in the read-only space of the controller.

Further, the synchronization process is further configured to perform the following steps:

s620, acquiring a current value of a target address in a readable and writable space cached by the controller and a current value of a designated address in the readable and writable space of the controller at each preset time interval; the target address is any address in a readable and writable space cached by the controller; the target address and the appointed address have a mapping relation;

S621, if the current value of the target address is different from the value of the target address at a preset time, and the current value of the appointed address is the same as the value of the appointed address at the preset time, replacing the current value of the appointed address with the current value of the target address; the last preset time is a time which is a preset time interval before the current time;

S622, if the current value of the target address is the same as the value of the target address at a preset time and the current value of the designated address is different from the value of the designated address at the preset time, replacing the current value of the target address with the current value of the designated address.

s623, if the current value of the target address is the same as the value of the target address at a preset time, and the current value of the appointed address is the same as the value of the appointed address at the preset time, not operating the target address and the appointed address;

S624, if the current value of the target address is different from the value of the target address at a preset time, and the current value of the appointed address is different from the value of the appointed address at the preset time, generating a data abnormity prompt.

Further, the baggage conveyor includes a second conveyor belt which is fixed by the lifting device.

Further, before step S100, the upper computer is further configured to perform the following steps:

s010, in response to the luggage carrying system sending a luggage waiting and carrying instruction, controlling the lifting device to lift to a first preset height, so that the luggage falls onto the first conveyor belt from the luggage conveying device.

S020, responding to manual operation Li Zhiling sent by the luggage carrying system, and controlling the lifting device to be lowered to a second preset height so as to carry the luggage manually; wherein the first preset height is greater than the second preset height.

Further, the duration of the preset time interval ranges from 20ms to 1000ms.

The invention has at least the following beneficial effects:

According to the luggage carrying system based on the computer vision and the mechanical arm, the mechanical arm and the vision device are respectively in communication connection with an upper computer; the vision device is arranged at a preset position above the baggage conveying device; the vision device is used for collecting the luggage car image and the luggage image; the end part of the mechanical arm is provided with a first conveyor belt, so that when the luggage falls off from the luggage conveying device, the luggage is connected through the first conveyor belt; the luggage is connected in a conveyor belt mode, so that the luggage slides onto the first conveyor belt from the luggage conveying device, and the luggage is connected; therefore, when the luggage with different sizes, different weights and different degrees of hardness is connected, no damage is caused to the luggage.

Further, the upper computer determines the stacking pose information of the current luggage on the luggage car according to the current residual stacking space information and the size information of the current luggage in the luggage car; determining the target orientation of the first conveyor belt when the current luggage is connected according to the stacking pose information of the current luggage on the luggage truck and the pose information of the current luggage; controlling the first conveyor to dock the current baggage dropped from the baggage conveyor in a target orientation; the current baggage on the first conveyor belt is piled up to a corresponding piling position of the piling pose information on the baggage car, so that automatic piling of the baggage is realized; and when the first conveyor belt is used for connecting the luggage, the target orientation of the first conveyor belt when the first conveyor belt is used for connecting the current luggage can be determined according to the stacking pose information of the current luggage on the luggage truck and the pose information of the current luggage, so that the luggage can fall to the corresponding stacking position more easily when being stacked.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic hardware structure diagram of a baggage handling system based on computer vision and a mechanical arm according to an embodiment of the present invention;

FIG. 2 is a system level schematic of a computer vision and robotic arm based baggage handling system according to an embodiment of the present invention;

fig. 3 is a flowchart of steps executed by an upper computer of a baggage handling system based on computer vision and a mechanical arm according to an embodiment of the present invention.

Detailed Description

The following description of the technical solutions in the embodiments of the present application will be clear and complete, and it is obvious that the described embodiments are only some embodiments of the present application, but not all embodiments. All other embodiments, based on the embodiments of the application, which are obtained by a person skilled in the art without making any inventive effort, are within the scope of the application.

It is noted that in the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It should also be understood that the term "and/or" if used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

In the description of the present application and the appended claims, the term "if used may be interpreted as" in the context of..or "when..or" upon "or" in response to a determination "or" in response to detection "depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".

In the description of the present specification and the appended claims, the terms "first," "second," "third," and the like, if used, are used merely to distinguish between descriptions, and are not to be construed as indicating or implying a relative importance, nor are they used to describe a particular order or precedence.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise.

It should be understood that, the sequence numbers of the steps in the following embodiments do not mean the sequence of execution, and the execution sequence of each process should be determined by the functions and the internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application;

Exemplary, as in the present embodiment, "S100, … …; s200, … …; s300, … …; "and the like, which are exemplary only, provide one possible order of execution and not the order of execution that must be assumed. For example, the above description only indicates that the steps S100, S200 and S300 may be sequentially performed, but are not necessarily sequentially performed, and those skilled in the art may determine an actual and feasible order of execution according to the functions and inherent logic of each step. For example, step S200, step S300 and step S100 may be sequentially performed, or step S100 may be performed followed by step S200 and step S300 being performed in parallel, or other possible execution sequences; meanwhile, "S100, … …; s200, … …; s300, … …; "and does not limit the complete method to only 3 steps, and a person skilled in the art can determine whether to add other reasonable steps before and after any step according to actual requirements.

A baggage handling system based on computer vision and robot will be described with reference to a hardware configuration diagram of the baggage handling system based on computer vision and robot shown in fig. 1.

The baggage handling system based on computer vision and robotic arms comprises: the system comprises a mechanical arm, an upper computer and a vision device; the mechanical arm and the vision device are respectively in communication connection with the upper computer; the vision device is arranged at a preset position above the baggage conveying device; the vision device is used for collecting a luggage car image and a luggage image; the end of the mechanical arm is provided with a first conveyor belt to receive baggage by the first conveyor belt when the baggage falls from the baggage conveyor.

In this embodiment, the system further includes: a controller; the controller is respectively in communication connection with the upper computer and the mechanical arm; the vision device comprises a first image acquisition device and a second image acquisition device; the first image acquisition equipment is used for acquiring a luggage image; the second image acquisition device is used for acquiring images of the luggage van.

In this embodiment, the controller may be a programmable logic controller (Programmable Logic Controller, PLC); the upper computer can be an industrial personal computer or a movable computer and the like; the first image acquisition device may be a lidar or an RGB-D camera; the second image acquisition device may be a lidar or an RGB-D camera; when the laser radar is adopted, three-dimensional point cloud data of the luggage or the luggage car can be directly obtained, and when the RGB-D camera is adopted, a depth image of the luggage or the luggage car can be obtained, and then the depth image is converted into corresponding three-dimensional point cloud data; the kilomega optical fiber can be adopted for communication between the mechanical arm and the controller, between the controller and the upper computer and between the vision device and the upper computer; the second image capturing device may be installed at a position above a center point of a parking area of the luggage car to entirely cover the parking area of the luggage car, so that the second image capturing device can completely capture a spatial area image in the luggage car.

Further, the baggage conveyor comprises a second conveyor belt which is fixed by a lifting device; the lifting device is lifted to a preset height so that the baggage falls from the baggage conveyor onto the first conveyor.

In this embodiment, when the baggage handling system is used for baggage handling, the second conveyor belt may be raised to a predetermined height, and baggage may slide down onto the first conveyor belt via the second conveyor belt; when the baggage handling system is not in use, the second conveyor may then be lowered for other handling.

The upper computer is used for executing the following steps:

S100, acquiring current residual stacking space information and pose information and size information of current baggage in a baggage car in response to the arrival of the baggage at a designated position of a baggage conveying device; the current residual stacking space information in the luggage car is obtained according to the current luggage car image, and the pose information and the size information of the current luggage are obtained according to the current luggage image.

In this embodiment, if the current baggage is the first baggage to be carried at this time, before executing step S100, the method further includes the following steps:

s001, judging whether each preset subarea in the luggage car parking area is shielded; wherein the preset sub-area is part of a luggage cart parking area.

In this embodiment, a plurality of preset sub-areas are provided in the luggage car parking area, for example: the sub-areas corresponding to the four vertexes of the rectangular luggage van parking area are preset sub-areas; judging whether each preset subarea in the luggage car parking area is blocked or not by an image recognition method; for example: and judging whether each preset subarea in the parking area of the luggage van is blocked or not by judging whether the difference value between the current RGB value of the subarea image corresponding to each preset subarea and the RGB value at the last moment is in the preset difference value range or not.

S002, if each sub-region is blocked, entering S100; otherwise, S001 is entered.

In this embodiment, the preset baggage car parking area is only used for parking baggage cars, and if each sub-area is blocked, it indicates that a baggage car has been parked in the baggage car parking area, then S100 may be entered; otherwise, continuing to judge whether each preset subarea in the luggage car parking area is blocked or not so as to determine whether the luggage car is parked in the luggage car parking area or not.

If the baggage being carried at this time is piled up on the baggage, the second image device immediately shoots an image of the baggage car to calculate the piling pose information of the next line Li Duiying.

In this embodiment, the installation position of the second image capturing device is fixed and known, and after the luggage car is parked in the parking area of the luggage car, the spatial position coordinates of the luggage car can also be obtained; the initial stacking space information when the luggage truck does not stack luggage is known, the space information occupied by the luggage which is stacked on the current luggage truck can be identified according to the luggage truck image, and the current residual stacking space information in the luggage truck can be obtained by subtracting the space information occupied by the luggage which is stacked on the current luggage truck from the initial stacking space information; it should be noted that, according to actual needs, a person skilled in the art can determine the initial stacking space information of the luggage cart and the space information occupied by the luggage stacked on the luggage cart by using the existing image processing method, which is not described herein.

Further, as shown in fig. 2, a plurality of sensors are installed on the baggage conveyor: for example: the device comprises a proximity switch, an infrared pair tube and a weighing sensor; the sensor can determine whether the luggage reaches the preset position of the luggage conveying device or not, and can acquire the weight of the luggage; in this embodiment, before the first image capturing device captures an image of the baggage, it is necessary to determine whether the baggage reaches the preset position of the baggage conveyor by the above-mentioned sensor, and if the baggage reaches the preset position of the baggage conveyor, the first image capturing device captures an image of the baggage.

Further, the baggage handling system may further comprise a safety monitoring device comprising a safety apparatus, such as: the safety equipment comprises a safety grating, a guardrail switch, human body induction and the like; the safety monitoring device detects personnel and the environment through the safety equipment, and performs start-stop control of the system according to the detection result; for example, if a person or an environment detection abnormality exists in a preset working area, the baggage handling system is controlled to stop working so as to avoid accidents.

S200, determining stacking pose information and position information of the current luggage on the luggage car according to the current residual stacking space information and the size information of the current luggage in the luggage car.

In this embodiment, the installation position of the first image capturing device is fixed and known, the installation position of the baggage conveyor is also fixed and known, and the spatial coordinates between the first image capturing device and the baggage conveyor may be calibrated in advance; accordingly, pose information and size information of the current baggage can be determined by the image of the current baggage photographed by the first image pickup device; those skilled in the art can determine pose information and size information of the baggage according to actual needs by using an existing image processing method, and details are not repeated here.

In this embodiment, after pose information of the baggage is determined, the mechanical arm needs to move from a stationary initial position to a position where the baggage is located, coordinate information of the stationary initial position of the mechanical arm can be obtained, position information of the baggage is also obtained, and first motion trail information of the mechanical arm can be generated according to two position coordinates; it should be noted that, a person skilled in the art can determine the first motion trajectory information of the mechanical arm by using the existing motion trajectory determination method according to actual needs, which is not described herein.

As shown in fig. 2, the information management, the motion planning, the scheduling strategy, the visual identification and the stacking strategy of each mechanism of the system are all calculated on an upper computer, and the calculation result is sent to a controller so as to carry out logic control of hardware through the controller.

In this embodiment, after the remaining stacking space information of the luggage cart and the size information of the luggage are obtained, the stacking position of the luggage on the luggage cart may be calculated on the upper computer through a preset stacking strategy; the palletizing strategy may be to preferentially palletize baggage to a side of the truck remote from the robotic arm and proximate to the side wall of the truck.

S300, determining the target orientation of the first conveyor belt when the current luggage is connected according to the stacking pose information of the current luggage on the luggage truck and the pose information of the current luggage; when the first conveyor belt is used for connecting the current luggage with the target direction, the length direction of the current luggage is parallel or perpendicular to the conveying direction of the first conveyor belt.

Further, step S300 may include the steps of:

S310, determining a first included angle between the central line of the current luggage and the central line of the luggage car according to the stacking pose information of the current luggage on the luggage car; the central line of the current luggage is parallel to the length direction of the current luggage, and the central line of the luggage car is parallel to the length direction of the luggage car.

S320, determining a second included angle between the central line of the current luggage and the conveying direction of the luggage conveying device according to the pose information of the current luggage.

And S330, if the first included angle is within a first preset included angle range, determining the direction parallel to the central line of the first conveyor belt and the central line of the current luggage as the target direction.

In this embodiment, the first preset included angle may range from 0 ° to 20 °; when the first included angle is within a first preset included angle range, the central line of the luggage is judged to be parallel to the central line of the luggage car, so that the judging redundancy is improved.

S340, if the first included angle is within a second preset included angle range, determining the direction perpendicular to the central line of the first conveyor belt and the central line of the current luggage as a target direction; wherein, any included angle of the first preset included angle range is smaller than the minimum included angle of the second preset included angle range.

In this embodiment, the second preset included angle may range from 70 ° to 90 °; when the second included angle is within a second preset included angle range, the center line of the luggage is perpendicular to the center line of the luggage car, so that the judging redundancy is improved.

It should be noted that, one end of the width of the luggage cart is close to the mechanical arm for parking, i.e. the center line of the luggage cart passes through the center of the mechanical arm; when the baggage is transferred on the baggage transferring device, there are two cases, one is that the central line of the baggage is the same as the transferring direction of the baggage transferring device, and the other is that the central line of the baggage is different from the transferring direction of the baggage transferring device; when the luggage is put on the luggage barrow, the central line of the luggage is parallel and vertical to the central line of the luggage barrow; thus, to accommodate stacking of the first conveyor, it is necessary to determine the target orientation of the first conveyor when the first conveyor is docked with the baggage; the determination of the target orientation is based on the calculated position and orientation of the baggage on the baggage car, so that the baggage is connected in the target orientation, and the baggage can be tidily placed in the line Li Ma during subsequent stacking, so that the space utilization rate of the baggage car is improved.

S400, controlling the first conveyor to dock the current baggage dropped from the baggage conveyor with the target orientation.

In this embodiment, after receiving the first motion track information and the target orientation, the controller performs time sequence and logic calculation on each actuator of the mechanical arm, and sends a corresponding calculation result to the mechanical arm controller at a corresponding time sequence, where the mechanical arm controller controls each actuator of the mechanical arm to perform a corresponding action; the actuator comprises: the device comprises a tail end conveyor belt, an auxiliary supporting mechanism, a mechanical arm six-axis motor, a ground rail servo motor and the like.

In this embodiment, the controller performs time sequence and logic calculation on each actuator of the mechanical arm, and sends a corresponding calculation result to the mechanical arm controller at a corresponding time sequence, where the mechanical arm controller controls each actuator of the mechanical arm to execute a corresponding action; when the first conveying belt reaches the position corresponding to the stacking pose information on the luggage truck, the first conveying belt is rotated forward, so that the luggage slides to the corresponding position, and stacking of the luggage is completed.

Further, the luggage handling system based on the computer vision and the mechanical arm further comprises a demonstrator, wherein the demonstrator is in communication connection with the controller; the demonstrator is used for debugging and parameter setting of the controller, and simple debugging and parameter setting of the controller can be directly carried out on the demonstrator without connecting a computer, so that the controller is more convenient.

Further, a read-only space, a write-only space and a readable and writable space are preset in the controller; the upper computer is preset with a control unit, and the control unit comprises an interface calling layer and a control module layer; the control unit layer is in communication connection with the control module layer; the interface calling layer is provided with a controller cache, and a read-only space, a write-only space and a readable and writable space are preset in the controller cache; and the addresses in the read-only space, the write-only space and the readable and writable space in the controller cache are mapped with the addresses in the read-only space, the write-only space and the readable and writable space in the controller one by one respectively. For example, the read-only space of the controller is preset with an address a ', and then the read-only space of the upper computer is also preset with an address a' having a mapping relation with the address a.

Further, the control module layer comprises an incoming material control module, a luggage connection module and a luggage stacking module; the incoming material control module is used for determining whether the luggage connection device of the mechanical arm is connected with luggage or not and acquiring ruler information of the luggage.

In this embodiment, after a preset position of the luggage conveyer belt is located at Li Daoda, the incoming material control module first determines the number of currently connected luggage or the current remaining stacking space information of the luggage truck, so as to determine whether the luggage can be further connected, and if the luggage can be further connected, the size information of the luggage is obtained; otherwise, generating an instruction that the luggage cannot be connected continuously.

The luggage connection module is used for determining first movement track information and stacking position information according to the size information of the luggage, and sending the first movement track information and the stacking position information to the controller cache so as to update the address value in the controller through the controller cache.

The luggage connection module can call a preset stacking strategy, a motion planning algorithm and the like to determine the stacking position information of the first motion trail information; and updating the value of the corresponding address in the controller by the information.

The Li Ma stacking module is used for acquiring current residual stacking space information on the luggage truck after the luggage stacking is finished; the stacking space information on the luggage truck acquired by the luggage handling system is always up to date, so that the stacking position information of the subsequent luggage can be calculated conveniently.

Further, the upper computer is preset with a synchronization process, and the synchronization process is further used for executing the following steps:

s620, acquiring a current value of a target address in a readable and writable space cached by the controller and a current value of a designated address in the readable and writable space of the controller at each preset time interval; the target address is any address in a readable and writable space cached by the controller; the target address and the designated address have a mapping relationship.

In this embodiment, the synchronization process is configured to perform data synchronization on each address in the controller cache and a corresponding address in the controller; the time interval of the adjacent two times of data synchronization is a preset time interval; the duration range of the preset time interval is 20ms-1000ms; for example: the preset time interval is 200ms; the synchronization process performs a synchronization update of the variables on the upper computer with the variables on the controller once every 200 ms.

S621, if the current value of the target address is different from the value of the target address at a preset time, and the current value of the appointed address is the same as the value of the appointed address at the preset time, replacing the current value of the appointed address with the current value of the target address; the last preset time is a time that is a preset time interval before the current time.

In this embodiment, the address in the read-only space can only read the value corresponding to the address, and the address in the write-only space can only write the corresponding value to the address, and the readable and writable address indicates that the value corresponding to the address can be written and read.

S623, if the current value of the target address is the same as the value of the target address at a preset time, and the current value of the designated address is the same as the value of the designated address at a preset time, the target address and the designated address are not operated.

In this embodiment, under normal conditions, since the data processing timing memory of the upper layer service logic has already been determined, the values of the two addresses having the mapping relationship are not updated simultaneously in the same period, and if the current value of the target address is different from the value of the target address at a preset time, and the current value of the designated address is different from the value of the designated address at a preset time, this indicates that the system data is abnormal.

It should be noted that, the task on the upper computer performs the read-write operation on the variable on the upper computer, and the task on the upper computer does not directly perform the read-write operation on the variable on the controller; the updating of the values of the variables on the controller is controlled by a synchronization process, not in real time, but at preset time intervals; by adopting the data synchronization strategy, when a plurality of tasks perform read-write operation on the same variable, the controller does not need to be accessed by a plurality of tasks, the controller is prevented from being paralyzed, and the stability of data read-write is improved.

In addition, when the service task of the luggage handling system needs the value of a certain variable, the corresponding variable value is only needed to be read on the upper computer, the controller is not needed to be accessed frequently, the communication pressure between the upper computer and the controller is reduced, and the stability of data transmission is improved.

In this embodiment, the multitasking is performed on the read-write operation of the same variable with a certain time sequence, where the time sequence is preset and determined by the processing logic of the service end; that is, the problem of system breakdown caused by simultaneous read-write operation of multiple tasks on the same variable is avoided.

Before step S100, the upper computer is further configured to perform the following steps:

In this embodiment, the second conveyor belt is fixed by the lifting device, and after the handling system is ready, an instruction for waiting for handling the baggage is generated, which indicates that the handling system can execute the handling action at this time, and the lifting device is lifted to the first preset height, so that the baggage can slide from the second conveyor belt onto the first conveyor belt; however, the transporting system also fails, and when the transporting system fails, the lifting device is lowered to a second preset height so as to manually transport the baggage; thereby avoiding the situation that the luggage can not be carried when the carrying system is in fault.

In the luggage handling system based on computer vision and the mechanical arm, the mechanical arm and the vision device are respectively connected with an upper computer in a communication way; the vision device is arranged at a preset position above the baggage conveying device; the vision device is used for collecting the luggage car image and the luggage image; the end part of the mechanical arm is provided with a first conveyor belt, so that when the luggage falls off from the luggage conveying device, the luggage is connected through the first conveyor belt; the luggage is connected in a conveyor belt mode, so that the luggage slides onto the first conveyor belt from the luggage conveying device, and the luggage is connected; therefore, when the luggage with different sizes, different weights and different degrees of hardness is connected, no damage is caused to the luggage.

In addition, the upper computer determines the current residual stacking space information in the luggage car according to the current luggage car image; determining pose information and size information of the current luggage according to the current luggage image; determining stacking pose information of the current luggage on the luggage vehicle according to the current residual stacking space information and the size information of the current luggage in the luggage vehicle; determining the target orientation of the first conveyor belt when the current luggage is connected according to the stacking pose information of the current luggage on the luggage truck and the pose information of the current luggage; controlling the first conveyor to dock the current baggage dropped from the baggage conveyor in a target orientation; the current baggage on the first conveyor belt is piled up to a corresponding piling position of the piling pose information on the baggage car, so that automatic piling of the baggage is realized; and when the first conveyor belt is used for connecting the luggage, the target orientation of the first conveyor belt when the first conveyor belt is used for connecting the current luggage can be determined according to the stacking pose information of the current luggage on the luggage truck and the pose information of the current luggage, so that the luggage can fall to the corresponding stacking position more easily when being stacked.

Furthermore, although the steps of the methods in the present disclosure are depicted in a particular order in the drawings, this does not require or imply that the steps must be performed in that particular order, or that all illustrated steps be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform, etc.

While certain specific embodiments of the invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the invention. Those skilled in the art will also appreciate that many modifications may be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A computer vision and robotic arm based baggage handling system, the system comprising: the system comprises a mechanical arm, an upper computer and a vision device; the mechanical arm and the vision device are respectively in communication connection with the upper computer; the vision device is used for collecting a luggage car image and a luggage image; the end part of the mechanical arm is provided with a first conveyor belt, so that when the luggage falls off from the luggage conveying device, the luggage is connected through the first conveyor belt;

The upper computer is used for executing the following steps:

2. The computer vision and robotic arm based baggage handling system of claim 1, wherein the vision device comprises a first image acquisition device and a second image acquisition device; the first image acquisition equipment is used for acquiring a luggage image; the second image acquisition device is used for acquiring images of the luggage van.

3. The computer vision and robotic arm based baggage handling system of claim 2, wherein the first image acquisition device comprises a lidar and/or an RGB-D camera; the second image acquisition device comprises a lidar and/or an RGB-D camera.

4. The computer vision and robotic arm based baggage handling system of claim 1, wherein step S300 comprises the steps of:

5. The computer vision and robotic arm based baggage handling system of claim 4, wherein said first included angle ranges from 0 ° to 20 ° and said second included angle ranges from 70 ° to 90 °.

6. The computer vision and robotic arm based baggage handling system of claim 1, wherein after step S300 and before step S400, said upper computer is further configured to perform the steps of:

7. The computer vision and robotic arm based baggage handling system of claim 1, wherein the system further comprises: a controller; the controller is respectively in communication connection with the upper computer and the mechanical arm; a read-only space, a write-only space and a readable and writable space are preset in the controller;

The upper computer is preset with a control unit, and the control unit comprises an interface calling layer and a control module layer; the control unit is in communication connection with the control module layer;

8. The computer vision and robotic arm based baggage handling system of claim 7, wherein the upper computer is pre-configured with a synchronization process for performing the steps of:

9. The computer vision and robotic arm based baggage handling system of claim 8, wherein the synchronization process is further configured to perform the steps of:

10. The computer vision and robotic arm based baggage handling system of claim 9, wherein the synchronization process is further configured to perform the steps of:

11. The computer vision and robotic arm based baggage handling system of claim 1, wherein the baggage conveyor comprises a second conveyor belt, the second conveyor belt being secured by a lifting device.

12. The computer vision and robotic arm based baggage handling system of claim 11, wherein prior to step S100, said upper computer is further configured to perform the steps of:

13. The computer vision and robotic arm based baggage handling system of claim 12, wherein prior to step S100, said upper computer is further configured to perform the steps of:

14. The computer vision and robotic arm based baggage handling system of claim 9, wherein the predetermined time interval has a duration ranging from 20ms to 1000ms.