CN118833757A - Forklift robot and operation control method and controller thereof - Google Patents

Abstract

The present invention discloses a forklift robot, which is provided with a pan-tilt platform on the top of the forklift robot, and a camera arranged on the pan-tilt platform. The forklift robot has a lifting rod, and the top of the lifting rod is fixedly connected to the pan-tilt platform. The operation control method of the forklift robot includes: obstacle avoidance detection during the movement of the forklift robot, control of the picking process and control of the placing process. The forklift robot picking process control includes: when the forklift robot arrives at the picking point, adjusting the camera angle of view through the pan-tilt platform to identify whether there is cargo, cargo stacking detection, pallet or tray identification, and vehicle body cargo detection. The forklift robot placing process control includes: when the forklift robot arrives at the placing point, adjusting the camera angle of view through the pan-tilt platform to identify whether there is cargo, cargo stacking detection before and after placing, and cargo height identification.

Description

A forklift robot and its operation control method and controller

Technical Field

The invention belongs to the technical field of robots, and in particular relates to a forklift robot and an operation control method and a controller thereof.

Background Art

With the widespread application of forklift mobile robots in the fields of industry, logistics, etc., improving the robot's ability to avoid obstacles, pick up and place goods, and stack has become an urgent problem to be solved. Since a single fixed camera is limited by the insufficient field of view and has a blind spot, it is difficult to complete the robot's tasks. Therefore, in the prior art, the commonly used methods are to increase the number of cameras, add multiple sensors, and increase the camera's degree of freedom through mechanical structure design and electronic control methods. However, these solutions have the following shortcomings:

Patent document CN216764205U proposes a method of mounting multiple fixed cameras on a forklift. Four fixed first cameras are mounted on the top of the forklift, facing the front, back, left, and right directions of the unmanned forklift; a second fixed camera is mounted between the two front legs near the base; although these five fixed cameras can achieve obstacle avoidance and pick-up and release, the number of cameras is redundant, and they can only identify pallets/trays with fixed heights, and cannot adaptively adjust the camera height, which increases costs.

In patent document CN220033927U, a method of installing a forward fixed camera and multiple laser radars on a forklift is proposed. An obstacle avoidance camera is fixed on the head of the forklift to identify obstacles in front; multiple obstacle avoidance radars are fixed at the bottom edge of the forklift to detect obstacles on both sides and the rear of the forklift; multiple photoelectric sensors are fixed on the fork arm of the unmanned forklift to detect obstacles in front of the fork arm. Because this method uses a fixed laser radar for obstacle avoidance, it can only identify obstacles at the same height, and cannot identify obstacles below the radar height or above the radar height or irregular obstacles, which increases the risk of collision and also increases costs.

Patent document CN213595798U discloses a method of installing a lifting and rotating mechanism on top of a forklift and setting a camera on the top. The lifting mechanism drives the camera to rotate. The forklift camera cannot perform pitch motion, which limits the fork body detection and cargo observation range, increasing the risk of collision.

Patent document CN220056232U discloses a method for driving a camera to move with a fork. The forklift is equipped with two first cameras on both sides of the fork, and three second cameras on the telescopic mechanism below the lifting assembly. The first and second cameras move with the movement of the fork, and maintain a standard recognition distance and recognition direction when picking up and placing goods. In the forklift cargo delivery scenario, the first camera is installed on both sides of the fork, so that the lateral field of view will be blocked by the vehicle body, and the second camera is installed under the fork, so that the upper field of view will be blocked by the cargo, resulting in a blind spot in the field of view, and the camera utilization rate is not high.

Patent document CN115092855A discloses a method of combining an electric lifting rod with a pan/tilt in a forklift and an electric slide rod driving a laser sensor. The camera can be raised and lowered, rotated 360 degrees, and the camera height and rotation angle can be adjusted according to the height of the cargo for imaging; the electric slide rod can drive the laser sensor to rise and fall, solving the problem of inaccurate laser ranging caused by too many holes in the pallet. However, this method adds an electric slide rod and a laser sensor to identify the pallet, which increases the cost.

Summary of the invention

One embodiment of the present invention is a forklift robot operation control method based on a jacking pan-tilt camera, based on forklift robot visual recognition. The forklift robot is provided with a pan-tilt on the top, and a camera arranged on the pan-tilt. The forklift robot has a jacking rod, and the top of the jacking rod is fixedly connected to the pan-tilt. The forklift robot operation control method includes: obstacle avoidance detection, cargo picking process control and cargo placing process control during the forklift robot's movement.

The embodiments of the present invention not only realize the pitch and yaw movement of the pan-tilt camera provided on the forklift robot, but also can realize automatic lifting by using the method of lifting the pan-tilt with forks, thereby comprehensively improving the robot's perception and operation capabilities in complex environments while reducing costs, thereby improving the robot's operating efficiency and safety.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

FIG1 is a perspective schematic diagram of a forklift robot according to one embodiment of the present invention.

FIG. 2 is a side view of a forklift robot according to one embodiment of the present invention.

FIG. 3 is a front view of a forklift robot according to one embodiment of the present invention.

FIG. 4 is a top view of a forklift robot according to one embodiment of the present invention.

FIG. 5 is a schematic diagram of cargo/pallet identification before cargo pickup by a forklift robot according to one embodiment of the present invention.

FIG6 is a schematic diagram of obstacle/cargo identification after a forklift robot picks up goods according to one embodiment of the present invention.

FIG. 7 is a schematic diagram of single-layer delivery point identification before delivery of a forklift robot according to one embodiment of the present invention.

FIG8 is a schematic diagram of multi-layer delivery point identification before delivery of a forklift robot according to one embodiment of the present invention.

FIG. 9 is a schematic diagram of a forklift robot driving a camera to rise according to one embodiment of the present invention.

FIG. 10 is a schematic diagram of motion obstacle detection for a forklift robot according to one embodiment of the present invention.

FIG. 11 is a flow chart of motion obstacle detection for a forklift robot according to one embodiment of the present invention.

FIG. 12 is a flowchart of picking up goods by a forklift robot according to one embodiment of the present invention.

FIG. 13 is a flowchart of placing goods by a forklift robot according to one embodiment of the present invention.

FIG. 14 is a three-dimensional schematic diagram of a forklift robot according to one embodiment of the present invention.

FIG. 15 is a schematic diagram of a lifting rod according to one embodiment of the present invention.

FIG. 16 is a side view of a forklift robot according to one embodiment of the present invention.

FIG. 17 is a front view of a forklift robot according to one embodiment of the present invention.

FIG. 18 is a schematic diagram of calculating the risk of camera occlusion of a forklift robot according to one embodiment of the present invention.

FIG19 is a schematic diagram of calculating the risk of camera occlusion of a forklift robot according to one embodiment of the present invention.

Among them, 1 – camera,

2——Gimbal, 21——Pitch motor, 22——Yaw motor,

3——Lifting rod,

4 – Forks,

5 – Pallets,

6——Goods.

DETAILED DESCRIPTION

In order to address the defects of existing forklift robots, the present invention proposes a forklift robot with a lifting pan-tilt camera. By mounting a pan-tilt camera and a lifting rod on the forklift robot, a comprehensive camera movement solution is designed for forklift tasks. By automatically adjusting the camera viewing angle and height at each task stage, the robot has comprehensive perception capabilities during movement, picking up and placing goods, thereby improving work efficiency and safety.

The objectives of this disclosure include:

1. Improve the robot's obstacle avoidance and detection capabilities during movement. Equip the camera with a pan/tilt and a lifting mechanism to achieve omnidirectional observation, forward vision, rearward vision, left/right scanning, up/down scanning, etc., to ensure that the robot has comprehensive perception capabilities during starting, straight driving, and turning, etc., and to solve the problems in the existing technology that a single fixed camera cannot perceive the environment in all directions, a single or multiple fixed sensors block the line of sight, and the pan/tilt lifting camera mechanism is complex to control.

2. Improve the accuracy and efficiency of the picking process. Through the steering of the camera and the lifting of the fork lifting rod, the camera can be raised and lowered to accurately identify the goods and pallets/pallets, ensure that the forks enter the appropriate position, and reduce the risk of uneven placement of goods.

3. Improve the accuracy and efficiency of the cargo placement process. Through the camera's forward, horizontal, and downward vision and automatic adjustment of the lifting rod, the destination and cargo can be accurately identified to ensure that the cargo is placed correctly, solving the problem of incorrect cargo placement or limited field of view in the stacking scene.

According to one or more embodiments, a forklift robot includes a main controller, a camera, a pan/tilt and a lifting rod. The main controller is responsible for controlling the forklift's movement, detection, obstacle avoidance, picking up goods, unloading and other actions. The control modules loaded on the main controller of the forklift robot include: a motion obstacle avoidance detection module, a picking up goods module and a placing down goods module.

The top of the forklift is equipped with a gimbal, a camera and a lifting rod. The camera is set on the gimbal, and the camera lens moves as the gimbal rotates. The lifting rod is in the shape of an inverted "T". The head of the lifting rod is fixedly connected to the bottom of the gimbal, and the bottom of the lifting rod is located above the fork. The lifting rod is located at a preset height by default. When the fork rises and reaches this preset height, the fork contacts the bottom of the lifting rod and rises with the lifting rod to avoid the problem of the cargo on the fork blocking the camera lens. When the fork drops, the lifting rod will also drop due to gravity. When the fork drops to the preset height of the lifting rod, the lifting rod stops at the default height, disengages from the fork, and the fork continues to drop. Therefore, the rise of the lifting rod depends on the lifting force of the fork, and the descent of the lifting rod depends on gravity. Under the control of the forklift robot controller, the lifting rod combines the action of the gimbal to achieve precise control of the camera movement. When the forklift fork is lifted to the bottom of the lifting rod, the fork can drive the lifting rod to move. This design realizes the linkage in height between the forklift fork and the gimbal and camera set at the head of the lifting rod. In this way, the linkage of the camera height does not require the main controller to perform specific detection or control, nor does it require additional driving force for the camera to adjust its height. It greatly simplifies the design of the structure and control system, saves the energy consumption of the forklift, and simplifies the design of the control logic. The existing lifting method for the camera gimbal usually requires the addition of an additional control motor, which drives the camera gimbal to lift. This existing method will significantly increase the equipment cost and maintenance cost. Therefore, the design of the lifting rod in the embodiment of the present disclosure solves this problem well.

The motion obstacle avoidance detection module controls the forklift in the following ways: before and after the forklift picks up the goods, the camera rotates to observe the surrounding environment and detect obstacles to ensure the safe movement of the forklift. Before the forklift starts, the camera looks around; when the forklift is moving straight, the camera looks forward; before the forklift passes an intersection, the camera scans left and right; before the forklift turns, the camera scans left or right.

The control of the forklift by the pickup module includes: when navigating to the pickup point, it will identify the presence of goods at the pickup point, detect the stacking of goods before picking up, identify the pallet, and detect the cargo on the vehicle body to ensure accurate forking of goods and reduce the risk of falling. When the forklift navigates to the pickup point, the camera turns and looks straight or downward to identify whether there is any goods and whether the goods are stacked stably; before the fork is inserted, the camera automatically adjusts the height and pitch angle to identify the pallet/pallet; after the fork is raised, the camera returns to the initial position to identify the placement of cargo on the vehicle body; before starting to deliver goods, the camera looks around. When the forklift is set to travel in the positive direction during navigation to the pickup point, the camera needs to turn from the positive direction to the reverse direction when picking up goods, and observe the rear when the forklift is backed out.

The control of the forklift by the cargo placement module includes: when navigating to the cargo placement point, identifying whether there is cargo at the cargo placement point, detecting cargo stacking before and after cargo placement, and identifying cargo height. When the forklift navigates to the cargo placement point, the camera turns to look straight or downward to identify whether there is cargo or other obstacles on the ground. Before the fork is inserted, the camera determines the optimal fork height by identifying the height of the cargo placement point; after placing the cargo, the camera turns to look forward, and if there is no obstacle, the forklift moves forward. The fork drives the camera to restore the default height, and the camera looks up and down to identify whether there is a risk of falling when the cargo is placed. After completing the cargo placement task, the camera looks around the surrounding environment and waits for the scheduling system to issue the next task. Here, the optimal fork height refers to the ideal height that the fork needs to be adjusted to when the forklift is performing a task of picking up or placing cargo in order to ensure work efficiency, safety and cargo stability. This height allows the fork to be smoothly and accurately inserted or placed at the preset position below or above the pallet/pallet.

Therefore, the embodiments of the present disclosure have the following beneficial effects:

(1) Improved the perception ability of the forklift robot. By equipping the robot with a pan-tilt camera and a lifting rod, the robot has comprehensive perception capabilities during movement, picking up and placing goods. The design of the camera's pitch and yaw movement and the lifting movement enables the robot to observe a wider range of surrounding environments, detect more obstacles and cargo information, and thus more accurately judge its own movement direction and operation mode.

(2) Improve the operating efficiency of forklift robots. By automatically adjusting the camera's viewing angle and height, the robot can be more accurate and faster in operations such as picking up and placing goods. For example, when picking up goods, the camera identifies the presence of goods, the stacking status, the pallets and the cargo on the vehicle body, etc., helping the robot to quickly find the goods and accurately pick them up, reducing the time spent searching for goods and misoperation. When placing goods, the camera identifies the ground conditions and cargo information, allowing the robot to accurately place the goods in the designated location, avoiding damage to the goods and wasting space. In high-altitude working scenarios such as platforms or stacking, the fork rises to a certain height and drives the camera upward through lifting, making it safer and more efficient to identify goods, pallets, etc.

(3) Enhanced safety. Through obstacle avoidance detection and omnidirectional perception capabilities, the robot is safer during movement. The camera observes the surrounding environment in all directions, allowing the robot to promptly detect obstacles in front and pedestrians around, so as to avoid and prevent collisions in time. At the same time, through precise picking and placing operations, damage to goods and casualties are reduced, and the safety of operations is improved.

(4) Improved economic benefits. By improving operational efficiency and safety, the waste of manpower and material resources is reduced. For example, through automated picking and placing operations, labor costs and the risk of cargo damage can be reduced. At the same time, through precise perception capabilities and obstacle avoidance detection technology, the cost of robot maintenance and replacement can be reduced, thereby improving economic benefits.

According to one or more embodiments, a forklift mobile robot, as shown in FIG1, is equipped with a pan-tilt camera and a lifting rod on the top of the forklift, the lifting rod is in an inverted "T" shape, and the head of the lifting rod is fixedly connected to the bottom of the pan-tilt. The lifting rod is under the control of the forklift robot controller and combines the movement of the pan-tilt to achieve precise control of the camera movement. When the forklift fork is lifted to the bottom of the lifting rod, the fork can drive the lifting rod to move. In this way, the height setting of the pan-tilt camera and the up and down movement of the fork are achieved to achieve a follow-up synchronization effect, which simplifies the additional control of the height of the pan-tilt camera by the forklift robot control system, and the control strategy is simpler and more reasonable. Other views of the forklift robot are shown in Figures 2, 3 and 4.

As shown in Figure 10. During the movement of the forklift, the control system of the forklift mobile robot automatically adjusts the viewing angle and height of the camera according to the movement state and navigation information of the forklift to perform obstacle avoidance detection. Before starting, the camera looks around to detect the surrounding environment to ensure a safe start. When going straight, the camera looks forward (the angle with the X-axis is 0° in the mobile robot coordinate system) to detect obstacles in front. The forward direction here is the direction on the other side relative to the fork direction. In order to meet the control requirements for the camera gimbal in the embodiment of the present disclosure, a DC stepper motor can be used to drive the gimbal for rotation control to improve the pitch and yaw motion control mode of the gimbal. The DC stepper motor has a maximum rotation speed of 40 to 50°/second horizontally and 10 to 24°/second vertically to adapt to the forklift operation in the embodiment of the present disclosure.

In the disclosed embodiment, when the pan-tilt camera of the forklift robot predicts that it will pass through an intersection or turn ahead, it will start scanning the pan-tilt camera to the left or right at a preset distance in advance, with the purpose of adjusting the camera angle in advance so that the pan-tilt camera will not miss important situations at intersections or turns. This is actually a problem caused by the fact that the field of view of the pan-tilt camera itself is not wide enough, and the control system needs to provide remedial measures. As for how much distance in advance is better and what the advance steering angle is, this needs to be set according to the actual travel speed of the forklift robot and the specific width of the route. For example, in an actual example, at least 12m before passing through an intersection or turning, the camera scans to the left (turn left 120°) or to the right (turn right 120°) to prevent a collision. Here, considering the horizontal rotation speed of the camera pan-tilt, the maximum horizontal rotation speed of the pan-tilt can reach 40-50°/second, which can be calculated according to 45°. The driving speed of the forklift is ≤1m/s in public areas or environments with other logistics mobile robots; the rated speed is ≤1.5m/s in restricted areas or environments with other logistics mobile robots. If calculated at 1.5m/s, when the camera gimbal turns 120°, 2.6 seconds have passed and the forklift has traveled 3.9m. When turning from 120° to 0°, 2.6 seconds have passed. Considering that the forklift will slow down when passing through the intersection, calculated at 1m/s, the forklift has traveled 2.6m. In addition to turning to the other side and returning to the front, the turn at 3.9+2.6+2.6+2.6=11.7m can ensure that the camera returns to the front after turning. Therefore, 12m is considered here. Or,

8m before the turn, the camera scans to the left (120° left) or right (120° right) to prevent collision. As mentioned above, here, the horizontal rotation speed of the camera gimbal is considered and calculated as 45°. The forklift speed is calculated as 1.5m/s. When turning 120°, 2.6 seconds have passed and 3.9m have traveled. When turning from 120° to 0°, 2.6 seconds have passed. Considering that the forklift will slow down before turning, the forklift has traveled 2.6m according to 1m/s. Therefore, the gimbal turns at 3.9+2.6=6.5m to ensure that the camera returns to the front after turning. Therefore, 8m is considered here.

When the forklift is in motion, the camera detects static or dynamic obstacles at any time and performs static or dynamic obstacle avoidance. After avoiding the obstacles, the path is replanned based on whether the forklift is off course.

According to one or more embodiments, as shown in FIG11 , the forklift robot control system in the disclosed embodiment includes a motion obstacle detection module, wherein the control actions for the gimbal camera include: the gimbal camera looks around when the forklift starts; the gimbal camera maintains a forward-looking state when the forklift is moving straight; starts to look left and right before reaching the intersection; the gimbal camera scans to the left and rear before moving to the left/turning to the left/turning to the left; and the gimbal camera scans to the right and rear before moving to the right/turning to the right/turning to the right. When the motion obstacle detection module finds an obstacle on the route, it issues an obstacle avoidance action command and passes through the obstacle area, synchronously combining the above-mentioned gimbal camera control.

As shown in FIG12 , the picking process of the forklift robot includes identifying the goods through the pan-tilt camera, and the pan-tilt camera is in a state of rear-view, horizontal and/or downward viewing. If the placement of the goods is unstable, an alarm is issued. When identifying the height of the goods, the pan-tilt camera is in a state of rear-view, downward and/or upward viewing. When the optimal fork height of the fork is greater than the height of the lifting rod (i.e., the lifting rod of the pan-tilt camera), the fork rises and the pan-tilt camera rises with it; if the optimal fork height is less than the lifting rod, the pan-tilt camera identifies the cargo pallet by rear-view, horizontal or downward viewing. Then, the forklift moves forward and the fork is inserted; the fork is raised to contact the pallet, the fork receives the goods, and then the pan-tilt camera looks forward and the forklift moves backward. At this time, the pan-tilt camera looks backward again and judges the optimal fork height again. If the optimal fork height is greater than the height of the lifting rod, the fork is lowered to the height of normal navigation. After the forklift obtains the goods, it identifies the goods and determines whether the goods on the fork are at risk of falling. At this time, the gimbal camera is in a rear-view, horizontal or top-view state. Since the height of the lifting rod is a fixed value. Since the forklift robot is moving, the fork must be lowered to the height of normal navigation. If the optimal fork height is greater than the lifting rod height (which means that the lifting rod is raised at this time), the fork will drive the lifting rod and the camera to the default height when it descends; if the optimal fork height is less than or equal to the lifting rod height (which means that the lifting rod is not raised at this time and is always at the default height), the fork will be lowered to the default height.

As shown in Figure 13, the cargo placement process of the forklift robot includes identifying the environment of the cargo placement point, and the pan-tilt camera is in a rear-view, front-view or top-view state. If there are obstacles or other cargo blocking it, an alarm will be issued. At this time, the pan-tilt camera identifies the cargo height, and the pan-tilt camera is in a rear-view, top-view or top-view state. When the optimal fork height is greater than the height of the lifting rod, the fork rises, and the pan-tilt camera also rises. At this time, the forklift moves forward, the fork enters the cargo placement space area, puts down the cargo, and the forklift moves backward. At this time, if the optimal fork height is higher than the height of the lifting rod, the fork is lowered to the navigation travel height, and the pan-tilt camera is lowered to the default navigation travel height. Finally, the pan-tilt camera looks backward, front-view or top-view to identify and detect the status of the cargo placed, and if there is a corner risk, an alarm message is issued.

In the disclosed embodiment, after the forklift mobile robot arrives at the pickup point, the pan-tilt camera turns to rear view, horizontal view or downward view (the pitch angle is dynamically adjusted according to the height of the pickup point) as needed to identify whether there is any cargo on the ground and whether the cargo is placed stably. If there is no cargo on the ground or the cargo is placed unstably and there is a risk of falling, an alarm is triggered and the dispatching system is informed to handle it. The camera determines the optimal fork height by identifying the height of the pallet. When the optimal fork height is greater than the bottom height of the lifting rod, the fork rises to the optimal fork height, driving the lifting rod and the camera to move, ensuring that the camera can see the cargo and the pallet at the same time above the fork. Before the fork enters the cargo, the camera automatically adjusts the pitch angle to identify the pallet/pallet, and automatically adjusts the forklift position and fork height according to the identified pallet position until it is aligned with the pallet/pallet. After the forklift moves forward and drives the fork into the cargo, the camera turns to look forward before the forklift moves backward. If there is no obstacle, the forklift moves backward. The camera turns to the rear view. When the optimal fork height is greater than the bottom height of the lifting rod, the fork descends, driving the lifting rod and the camera to the default height. Then the fork returns to the navigation height. At this time, the camera looks down at 45 degrees to identify the placement of the cargo on the vehicle body and whether there is a risk of falling. If there is a risk of falling, an alarm will be issued and the dispatching system will be informed to handle it. After confirmation, the camera returns to the initial angle, the forklift completes the pickup, and the camera turns to the front view to start transporting the cargo. As shown in Figures 5 and 6. Here, for the risk of falling of the cargo placement, the camera obtains images for prediction and judgment, in which a deep learning model can be used to quickly judge through a lightweight network. Since the camera is in a state of looking down at the cargo at this time, the algorithm for judging the risk of falling cargo through images will be different from the usual front view angle.

When arriving at the delivery point, the camera turns to rear view, horizontal or downward view (the pitch angle is dynamically adjusted according to the height of the pickup point) to identify whether there are already goods or other obstacles at the delivery point. If there are already goods at the delivery point or there are obstacles blocking the delivery point, an alarm will be issued and the dispatching system will be informed to handle it. At the delivery point, the camera determines the optimal fork height by identifying the height of the pallet. When the optimal fork height is greater than the bottom height of the lifting rod, the fork rises to the optimal fork height, driving the lifting rod and the camera to move, ensuring that the camera can see the goods and pallets above the fork at the same time. After the fork places the goods, the camera turns to forward view. If there are no obstacles, the forklift moves forward. The camera turns to rear view. When the optimal fork height is greater than the bottom height of the lifting rod, the fork descends, driving the lifting rod and the camera to the default height. Then the fork returns to the navigation height. At this time, the camera pitches and identifies the placement of the goods at the delivery point to see if there is a risk of falling. If there is a risk of falling, an alarm will be issued and the dispatching system will be informed to handle it; after confirmation, the forklift completes the delivery, and the camera turns to forward view to wait for the dispatching system to arrange. As shown in Figures 7 and 8. The lifting rod in Figure 8 is the lifting rod mentioned above, which is used to drive the PTZ camera to move up and down. Here, because the goods are stacked in layers, the forklift needs to handle it accordingly, including:

1. The dispatching system sends tasks to forklifts, including cargo height and number of cargo layers;

2. Identify whether there are obstacles at the delivery point. When the forklift arrives at the delivery point, it can adjust the pitch angle according to the cargo height given by the dispatching system and the actual identified height to ensure that the delivery point can be identified (whether single-layer or multi-layer);

3. Determine the best fork height before placing goods. When the forklift is picking up and placing goods, the fork needs to be precisely controlled. The specific principle is as follows:

When picking up goods, the fork must be precisely aligned with the hole of the pallet (pallet) or the appropriate position below it to ensure that the fork can be smoothly inserted into the pallet. Ensure that the goods will not slip due to too large or too small insertion angle. If the fork deviates too much from this optimal height, it may cause the entire cargo to fall due to collision with the pallet or pallet, or the fork may not reach the pallet or pallet, causing the picking identification. Therefore, by identifying the height of the delivery point to determine the optimal fork height, unnecessary lifting times are reduced, reducing the risk of cargo damage.

When releasing the goods, the fork needs to be lowered in advance to a position close to but slightly higher than the release point (the distance can be adjusted). It is convenient to control the speed and accuracy of the goods descent, avoid impact and vibration caused by rapid descent, and protect the goods from damage. If the height of the fork is too far above the release point, it takes extra time to lower the height, and it is unsafe to lift it too high; if the height of the fork is below the goods at the release point, direct placement will cause the fork to collide with the existing goods, which will cause damage to the goods and cause accidents. Therefore, there is an optimal fork height when releasing the goods, which is usually not far above the release point. The embodiment of the present disclosure adopts this height of 0.1m. It can be seen that the optimal fork height designed in the embodiment of the present disclosure can not only improve work efficiency, but also ensure work safety. As shown in Figures 5 to 9, the schematic diagram of the principle of the forklift robot pan-tilt camera for cargo identification, cargo height identification, pallet identification, release point identification and fork rise during the process of picking up and releasing goods is shown. Among them, "1" in Figure 5 indicates the position of the pan-tilt camera, and "4" is the fork.

Therefore, in actual operation, if the optimal fork height stage is less than the bottom height of the lifting rod (usually single-layer at this time), the forklift arrives at the cargo release point and the fork rises to the optimal fork height, and the cargo is placed directly; if the optimal fork height stage is greater than the bottom height of the lifting rod (usually multi-layer at this time), the forklift arrives at the cargo release point and the fork rises to the optimal fork height, and the fork rises together with the lifting rod. For stacked cargo, the multi-layer and single-layer here refer to the number of layers of cargo.

Another variation of the disclosed embodiment is a forklift robot as shown in FIGS. 14, 15, 16, and 17. In the aforementioned embodiment, when the cargo enters the space between the lifting rod and the fork, or when the fork rises, it is actually the cargo on the fork that lifts the lifting rod, which makes it difficult for the robot to accurately determine the actual lifting height of the fork and cannot implement the preset control strategy. Therefore, in this embodiment, a baffle is provided on the inner side of the forklift fork so that the cargo on the fork cannot directly contact the lifting rod. At the same time, the design of the baffle can ensure that the baffle can move synchronously with the fork so as not to interfere with the lifting effect of the fork on the lifting rod.

In order to eliminate possible risks, the forklift robot of the disclosed embodiment shall perform restrictive calculations on factors such as cargo before operation, including limiting the size of cargo so that the cargo does not exceed the rated safety standard, ensuring the normal and safe operation of the forklift. Before and after the cargo picking task, the placement of the cargo will be checked to ensure that the cargo does not exceed the range of the pallet or tray. If there is a risk, an alarm will be issued and the dispatching system will be informed to handle it. The restrictive calculations of the operations described here are described in detail below.

When the forklift robot is performing a picking task, there is no problem of blocking the camera because there is no cargo on the fork. When the forklift robot is performing a placing task, whether it is ground handling or stacking, it needs to determine whether the size of the cargo is within the limit before placing the cargo to avoid the cargo blocking the camera during operation. Therefore, the occurrence of the occlusion problem discussed here can usually be divided into two situations when the forklift performs the placing task, ground handling and stacking, and two different fork lifting strategies can be used for stacking tasks.

When a forklift robot performs a cargo stacking task, there are two strategies for adjusting the forks:

1. Adjust the forks to the optimal height Strategy 1 - Stop the forklift at the leading point and adjust the forks. The specific process is: the forklift stops when it reaches the leading point, and after the forks are raised to the optimal height, keep the forks at the same height and move to the loading point. The leading point mentioned here is set at the starting point of the fork length.

The advantage of this operation is that the forklift lifts the forks in a stationary state, avoiding the unstable factors that may be caused by dynamic operation, ensuring the safe and reliable picking process. The disadvantage is that it needs to pause at the front point and wait for the forks to be lifted into place, which increases the overall operation time. However, in order to ensure safety, the increased operation time of this operation strategy is worth it.

2. Adjust the fork to the optimal height Strategy 2 - Start adjusting the fork while moving after the leading point. The specific process is: before the forklift robot reaches the leading point, the fork is first raised to a certain height, and then the fork height is continuously adjusted to the optimal state while approaching the goods.

The advantage of this operation is that the forklift can lift the forks while moving without stopping, which achieves seamless operation and improves work efficiency. The problem with this operation is that since the forks have been lifted to a certain height at the front point, there may be a problem of the cargo on the forks blocking the camera; before releasing the cargo, it is necessary to ensure that the forks can be raised to the optimal fork height.

Through practical observation, it is found that the problem of the forklift camera being blocked in the forklift robot in the embodiment of the present disclosure will occur in ground transportation and stacking transportation using strategy 2 of adjusting the fork to the optimal height.

For the risk of camera being blocked during ground handling operations of forklift robots, it is necessary to perform restrictive calculations and judgments on the width and height of the cargo to avoid possible risks. Here, set the following parameters:

H _c : Maximum height of the gimbal camera;

d: The depth range of the PTZ camera at the cargo placement point;

w _g : cargo width;

_hg : cargo height;

x: The horizontal distance between the camera and the delivery point at the leading point.

At the leading point, the critical situation where the cargo on the fork does not block the camera is shown in Figure 18.

According to the triangle formula:

According to similar triangles:

From (1) and (2), we can get

Therefore, for a given forklift and camera parameters H _c , d,

When h _g = h ₀ , the maximum cargo width is w ₀ , so the cargo width should satisfy

When w _g = w ₀ , the maximum cargo height is h ₀ , so the cargo height should satisfy

If the cargo width or cargo height restriction is not met, it is determined that there is a risk of the camera being blocked.

When the forklift robot is performing a cargo stacking task, in order to avoid the PTZ camera's line of sight being blocked, it is necessary to make restrictive judgments and calculations on the cargo width and height. When the stacking handling strategy 2 of adjusting the forks to the optimal height is adopted, the following parameters are involved:

H _c : Maximum height of the gimbal camera;

d: The depth range of the PTZ camera at the cargo placement point;

w _g : cargo width;

_hg : cargo height;

x: The horizontal distance between the camera and the cargo placement point at the leading point;

n: cargo layer height at the cargo release point;

h _up1 : The height of the fork lift when reaching the front point;

h _up2 : The height from the front point to the height of the fork when releasing the cargo;

v _fl : moving speed of the forklift;

v _f : moving speed of the fork;

The critical situation where the cargo on the fork at the leading point does not block the camera is shown in FIG19 .

According to the triangle formula:

Assume that the time from the lead point to the delivery point is t, then:

h _up2 = t*v _f (5)

h _up1 +h _up2 =n*h _g (6)

From equations (3), (4), (5), and (6), we can get:

According to similar triangles:

Substituting (7) into (8) yields the expression including the forklift and camera parameters H _c , n, v _fl , v _f , d,

When h _g ＝h ₀ , the maximum cargo width is w ₀ , so the cargo width should satisfy w _g ＜w ₀ ;

When w _g = w ₀ , the maximum cargo height is h ₀ , so the cargo height should satisfy h _g <h ₀ . If the cargo width or cargo height restriction is not met, it is determined that there is a risk of camera being blocked.

In addition, in the disclosed embodiment, in addition to detecting motion obstacle avoidance and dynamically adjusting the pan/tilt parameters according to a given reference range and strategy, the forklift camera also needs to solve the pan/tilt control strategy in the image detection of the forklift during picking up and placing goods. At this time, the pan/tilt angle adjustment can be divided into two stages:

Phase 1: Static fuzzy adjustment. The dispatch system will give the height of the target object and other parameters (the parameters are often given by the user). Before reaching the target point, the gimbal camera adjusts the viewing angle according to the parameters given by the dispatch system so that the target point can be seen. Since these parameters are given, there will always be a certain error in the actual situation, so this is a fuzzy adjustment.

Stage 2: Dynamic real-time adjustment. After arriving at the pickup or delivery point, the camera can already see the target point and can adjust the height and other parameters of the target point object based on the real-time captured image, so it is a dynamic real-time adjustment.

The disclosed embodiment realizes the improvement of all-round perception and operation capability of the surrounding environment and goods by installing a gantry camera on the upper part of the forklift mobile robot. The operation efficiency and safety are improved by automatically adjusting the camera angle and height at each task stage.

In the disclosed embodiment, the camera carried by the gimbal can be a depth camera, or a 3D laser radar, a 2D camera, or a binocular camera can be used instead of a depth camera to achieve more accurate environmental perception and obstacle recognition. The laser radar can provide accurate three-dimensional information of the surrounding environment; the 2D camera + depth estimation algorithm can obtain high-definition image information and pseudo laser radar data, and combined with the navigation system and control system, achieve accurate obstacle avoidance and path planning.

The lifting rod can also be replaced by an adjustable telescopic rod or track system, an electric slide rod or a robotic arm design to achieve the functions of camera height adjustment and viewing angle change, providing the mobile robot with a wider range of motion and more flexible operability.

In summary, the forklift mobile robot proposed in this disclosure has the following technical features:

1. Gimbal camera with integrated pitch, yaw and lift motion.

By integrating a jacking gimbal camera on top of the forklift, the pitch and yaw movements of the camera are realized. When the fork rises to the jacking rod, it drives the jacking rod and the camera to do lifting and lowering movements. Since the jacking rod does not need to be driven by itself but relies on the thrust of the rising fork, this structural design improves the robot's perception and operation capabilities in complex environments without providing additional power.

2. Comprehensive camera rotation solution.

A comprehensive camera rotation solution is designed for forklift tasks, including motion obstacle avoidance detection, picking up and placing goods, etc. The camera angle and height are automatically adjusted at each task stage, so that the robot has comprehensive perception capabilities and improves work efficiency and safety.

3. Camera rotation obstacle avoidance detection solution.

During the movement of the forklift, the camera rotates to observe the surrounding environment and detect obstacles. Before starting, when driving straight, before passing an intersection, and before turning, the camera looks around, looks forward, scans left and right, and scans left and right to ensure the safe movement of the forklift.

4. Camera rotation pickup plan.

When navigating to the pickup point, the camera identifies the presence of goods, the stacking status of goods, and information such as the pallet and body cargo, helping the robot to quickly find the goods and accurately pick them up, reducing the risk of falling.

5. Camera rotation and cargo release plan.

When navigating to the delivery point, the camera identifies the ground conditions and cargo information, enabling the robot to accurately place the cargo at the designated location, avoiding damage to the cargo and waste of space.

6. Improvement of operating efficiency and economic benefits.

By improving the accuracy of robot operations and reducing the number of times human intervention is required, it can improve operational efficiency and economic benefits.

It should be understood that in the embodiments of the present invention, the term "and/or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character "/" in this article generally indicates that the associated objects before and after are in an "or" relationship.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims (14)

1. A forklift robot, characterized in that a pan-tilt platform is provided on the top of the forklift robot, and a camera is arranged on the pan-tilt platform, The forklift robot has a lifting rod, the top of which is fixedly connected to the pan/tilt head, so that the camera moves with the up and down movement of the lifting rod. The bottom of the lifting rod is located above the forklift fork. When the fork rises and contacts the bottom of the jacking rod, the fork rises with the jacking rod. When the fork descends, the jacking rod descends together under the action of gravity. When the fork is lowered to the preset height of the lifting rod, the lifting rod stops at the preset height, the fork continues to be lowered, and the lifting rod is out of contact with the fork. 2. A forklift robot operation control method, based on the forklift robot as claimed in claim 1, characterized in that the operation control method includes: obstacle avoidance detection, cargo picking process control and cargo placing process control during the movement of the forklift robot. 3. The method according to claim 2, characterized in that the obstacle avoidance detection step of the forklift robot during movement comprises: Before the forklift robot starts, the camera looks around; When the forklift robot moves straight, the camera looks forward; Before the forklift robot passes through the intersection, the camera scans left and right; Before the forklift robot turns, the camera scans left or right. 4. The method according to claim 2, characterized in that the forklift robot picking process control comprises: When the forklift robot arrives at the pickup point, it adjusts the camera's viewing angle through the gimbal to identify the presence of goods, detect cargo stacking, identify pallets or trays, and detect cargo on the vehicle body. 5. The method according to claim 2, characterized in that the forklift robot cargo release process control comprises: When the forklift robot arrives at the delivery point, it adjusts the camera angle through the gimbal to identify the presence of goods, detect the stacking of goods before and after delivery, and identify the height of goods. The method according to claim 2 , wherein the camera is a depth camera. 7. The method as claimed in claim 2 is characterized in that, during the process of picking up goods by the forklift, the forklift robot determines the placement status of the goods on the fork through a camera's overhead view. 8. The method according to claim 2, characterized in that, during the process of placing the goods by the forklift robot, Before placing the goods, determine the optimal fork height. If the optimal fork height is lower than the bottom height of the lifting rod, after the forklift arrives at the placing point, the fork rises to the optimal fork height and is placed directly; If the optimum fork height is higher than the bottom height of the lifting rod, when the forklift reaches the cargo release point and the fork rises to the optimum fork height, the fork rises together with the lifting rod. 9. The method according to claim 6, characterized in that a 3D laser radar, a 2D camera, or a binocular camera is used instead of the depth camera to achieve environment perception and obstacle recognition. 10. The method according to claim 6, characterized in that any one of 3D laser radar, 2D camera, or binocular camera or a combination thereof is used in combination with the camera to achieve environment perception and obstacle recognition. 11. The forklift robot according to claim 1, characterized in that the lifting rod uses an adjustable telescopic rod structure. 12. A forklift robot controller, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 2 to 10. 13. A storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 2 to 10 is implemented. 14. A computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method according to any one of claims 2 to 10.