KR102772458B1 - Method for preventing collision between human and robot in realtime and system for preventing collision in realtime using the same - Google Patents

Abstract

The present invention is a technology developed through the Seoul Business Agency's 2023 Robot Technology Commercialization Support Project (RO230017) of the Seoul Metropolitan Government, "Development of a lightweight gripper suitable for service sites and a standardized/non-standardized object recognition system with a gripping success rate of 90% or higher."
According to the present invention, a real-time collision avoidance system for preventing collision between a person and a robot can be provided, comprising: a human skeleton extraction unit configured to extract human skeleton information; a robot skeleton extraction unit configured to extract robot skeleton information; a motion vector generation unit configured to generate a motion vector of the person and a motion vector of the robot using the human skeleton information and the robot skeleton information, respectively; and a collision risk determination unit configured to determine a collision risk between the person and the robot based on the human motion vector and the robot motion vector, wherein the human skeleton information and the robot skeleton information each include a plurality of three-dimensional coordinate information for a plurality of points.

Description

{METHOD FOR PREVENTING COLLISION BETWEEN HUMAN AND ROBOT IN REALTIME AND SYSTEM FOR PREVENTING COLLISION IN REALTIME USING THE SAME}

The present invention relates to a real-time collision avoidance method for preventing collisions between humans and robots and a real-time collision avoidance system using the same. More specifically, the present invention belongs to the field of robotics and human-robot interaction, and is a technology for improving safety during physical interaction between humans and robots, and particularly relates to a method and system for preventing real-time collisions by controlling the movement of a robot using three-dimensional skeleton data of a human and a robot.

The present invention is a technology developed through the Seoul Business Agency's 2023 Robot Technology Commercialization Support Project (RO230017) of the Seoul Metropolitan Government, "Development of a lightweight gripper suitable for service sites and a standardized/non-standardized object recognition system with a gripping success rate of 90% or higher."

Recently, with the development of robot technology, the use of robots in industrial sites and service fields has increased, and cases of people and robots working in the same space are increasing. Accordingly, as the physical interaction between robots and humans increases, the risk of collision between people and robots exists, and securing safety is emerging as an important issue.

Existing robot collision avoidance systems mainly rely on simple distance sensors or 2D vision systems, and have limitations in accuracy and response methods. In particular, these methods have problems in reducing the work efficiency of robots or making it difficult to accurately predict collisions in complex work environments. Therefore, more sophisticated and effective collision avoidance and stability enhancement technologies are required. In particular, technologies that track human movements in real time and adjust robot movements are essential.

(Patent Document 0001) Republic of Korea Registered Patent No. 10-2369062

The purpose of the present invention is to provide a method and system capable of preventing collision between a person and a robot by analyzing the movement vectors of the person and the robot to determine the risk of collision and controlling the movement of the robot.

In addition, the present invention aims to provide a method and system capable of more efficiently determining the risk of collision between a person and a robot and controlling the movement of a robot by using skeleton data of a person and a robot.

In addition, the present invention aims to provide a more sophisticated and effective collision prevention and safety enhancement system by adjusting the motion vector of a robot that can reduce the risk of collision while tracking human movements in real time.

In addition, the present invention aims to provide a method and system capable of improving safety in work and service environments of robots and humans and maintaining work efficiency of robots.

The problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

According to one embodiment of the present invention, a real-time collision avoidance system for preventing collision between a person and a robot can be provided, comprising: a human skeleton extraction unit configured to extract skeleton information of a person; a robot skeleton extraction unit configured to extract skeleton information of a robot; a motion vector generation unit configured to generate a motion vector of the person and a motion vector of the robot, respectively, using the skeleton information of the person and the skeleton information of the robot; and a collision risk determination unit configured to determine a collision risk between the person and the robot based on the motion vector of the person and the motion vector of the robot, wherein the skeleton information of the person and the skeleton information of the robot each include a plurality of pieces of three-dimensional coordinate information for a plurality of points.

In addition, the collision risk determination unit may be configured to extract the shortest distance between the person and the robot by using a plurality of three-dimensional coordinate information included in the skeleton information of the person and a plurality of three-dimensional coordinate information included in the skeleton information of the robot, and determine the collision risk based on the size of the shortest distance.

In addition, the motion vector generation unit may be configured to generate the motion vector of the person and the motion vector of the robot by using the difference between the previous frame and the current frame using three-dimensional coordinate information for a plurality of points stored according to a preset time interval.

In addition, the device may further include a vector filtering unit configured to filter at least one specific motion vector among a plurality of motion vectors generated by the motion vector generating unit, and the collision risk determination unit may be configured to determine the collision risk between the person and the robot using the motion vector filtered by the vector filtering unit.

Additionally, the vector filtering unit may be configured to filter a motion vector for a point in a preset specific area among a plurality of points.

Additionally, the vector filtering unit may be configured to filter the motion vector based on at least one of the distance between the point of the person and the robot, the size of the vector, and area information.

Additionally, the region information is information related to whether it is a hand or face region, and can be configured to calculate the motion vector by applying a weight based on the region information.

In addition, the collision risk determination unit may be configured to determine the collision risk using at least one of the distance, speed, and direction of the motion vector of the person and the motion vector of the robot.

In addition, the robot movement control unit may further be configured to change the movement path of the robot to reduce the collision risk if the collision risk exceeds a predetermined standard.

Additionally, the robot movement control unit may be configured to reduce the movement speed of the robot when the collision risk is at the first level, and may be configured to change both the movement speed and direction of the robot when the collision risk is at a second level higher than the first level.

Additionally, the robot movement control unit may be configured to change the movement path of the robot based on a motion vector of the robot adjusted using the motion vector of the human.

In addition, the robot skeleton extraction unit may be configured to obtain three-dimensional skeleton data of the robot by using transformation of the robot coordinate system and the camera coordinate system based on the joint angle of the robot.

According to another embodiment of the present invention, a real-time collision avoidance method for preventing collision between a person and a robot can be provided, comprising: a step of extracting skeleton information of a person by a human skeleton extraction unit; a step of extracting skeleton information of a robot by a robot skeleton extraction unit; a step of generating a motion vector of the person and a motion vector of the robot, respectively, by a motion vector generation unit using the skeleton information of the person and the skeleton information of the robot; and a step of determining a collision risk between the person and the robot based on the motion vector of the person and the motion vector of the robot, wherein the skeleton information of the person and the skeleton information of the robot each include a plurality of three-dimensional coordinate information for a plurality of points.

According to the present invention, a method and system capable of preventing collision between a person and a robot can be provided by analyzing the movement vectors of a person and a robot to determine the risk of collision and controlling the movement of the robot.

In addition, according to the present invention, a method and system can be provided that can more efficiently determine the risk of collision between a person and a robot and control the movement of a robot by using skeleton data of a person and a robot.

In addition, according to the present invention, a more sophisticated and effective collision prevention and safety enhancement system can be provided by adjusting the motion vector of a robot to reduce the risk of collision while tracking human movement in real time.

In addition, according to the present invention, a method and system can be provided that can improve safety in the work and service environment of robots and humans and maintain the work efficiency of the robot.

The problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

FIG. 1 is a block diagram illustrating the configuration of a real-time collision avoidance system between a person and a robot according to one embodiment of the present invention.
FIGS. 2A to 2C are exemplary diagrams for explaining a process for extracting a human skeleton according to one embodiment of the present invention.
FIGS. 3A to 3D are exemplary diagrams for explaining a process for extracting a skeleton of a robot according to one embodiment of the present invention.
Figure 4 is an exemplary diagram for explaining a vector generation process according to one embodiment of the present invention.
FIG. 5 is a flowchart illustrating a real-time collision avoidance method for preventing collision between a person and a robot according to one embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the embodiments of the present invention in the drawings, parts that are not related to the description are omitted.

The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular expression may include the plural expression unless the context clearly indicates otherwise.

In this specification, the terms “include,” “have,” or “comprise” are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not excluding in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In addition, the components shown in the embodiments of the present invention are independently depicted to indicate different characteristic functions, and do not mean that each component is formed as a separate hardware or software configuration unit. That is, each component is described by listing each component for convenience of explanation, and at least two components among each component may be combined to form one component, or one component may be divided into multiple components to perform a function. Such integrated embodiments and separated embodiments of each component are also included in the scope of the rights of the present invention as long as they do not deviate from the essence of the present invention.

In addition, the following examples are provided to more clearly explain to a person having average knowledge in the art, and the shapes and sizes of elements in the drawings may be exaggerated for more clear explanation.

Hereinafter, with reference to the attached drawings, a preferred embodiment according to the present invention will be described.

FIG. 1 is a block diagram illustrating the configuration of a real-time collision avoidance system between a person and a robot according to one embodiment of the present invention.

Referring to FIG. 1, a real-time collision avoidance system (100) for preventing collision between a person and a robot may include a communication unit (110), a camera unit (120), and a processing unit (130), and some components may be omitted or additional components may be added as needed.

Here, the communication unit (110) is configured to perform communication between each component in the real-time collision avoidance system (100) through a network, and is also configured to receive necessary information from an external server or an external device through the network, or transmit acquired information to the external server or the external device. Here, the network may be a network connected wired or wirelessly. In addition, the connection network may be a network in which an external device and a mobile robot are directly connected, and may also be a private network created by a repeater. When the network is a wireless communication network, it may include cellular communication or short-range communication. For example, the cellular communication may include at least one of LTE (Long-Term Evolution), LTE-A (LTE Advanced), 5G (5th Generation), 6G (6th Generation), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), UMTS (Universal Mobile Telecommunications System), WiBro (Wireless Broadband), or GSM (Global System for Mobile Communications). Additionally, the short-range communication may include at least one of Wi-Fi (Wireless Fidelity), Bluetooth, Zigbee, NFC (Near Field Communication), or RFID (Radio Frequency Identification). However, the communication method is not limited thereto, and future wireless communication technologies may also be included.

The camera unit (120) can be configured to photograph a person or robot in a work space in real time, acquire an image, and transmit it to a processing unit (130) to perform image processing. For example, it can be configured to acquire an RGB image and a depth image of each object, and can be configured with a plurality of cameras as needed.

The processing unit (130) is configured to extract three-dimensional skeleton data of a human and a robot, analyze a motion vector over time to determine a collision risk, and adjust and control the current motion vector of the robot accordingly, thereby performing a collision risk determination and prevention process between a human and a robot. The processing unit may include a central processing unit (CPU), an application processor (AP), etc., and may be installed in the order product delivery system or configured in a separate device or server to be connected via a communication unit (110).

The processing unit (130) may include, for example, a human skeleton extraction unit (131), a robot skeleton extraction unit (132), a motion vector generation unit (133), a vector filtering unit (134), a collision risk determination unit (135), and a robot movement adjustment unit (136), and the programs or program modules included in each of these processing units may be configured in the form of an operating system, an application program, or a program, and may be physically stored on various types of widely used storage devices. Such a program or program module may include one or more routines, subroutines, programs, objects, components, instructions, data structures, and various forms for performing a specific task or executing a specific data type, but is not limited to these forms. Hereinafter, the operation of each component of the processing unit (130) will be examined in order.

Human Skeleton Extraction Unit (131)

The human skeleton extraction unit (131) is configured to extract human skeleton information using an image acquired through the camera unit (120), and performs the role of detecting major human joint points and estimating their positions. There may be various methods for estimating a human skeleton, such as a method using sensor data (mainly images or depth information), and the present invention is not limited to a specific method, and various skeleton estimation methods can be applied.

As an example of a human skeleton estimation method, a pose estimation vision artificial intelligence (AI) model such as YOLOv8 and depth information from a depth camera can be combined to estimate 3D joint coordinates.

First, in the data collection stage, RGB images and depth information are acquired through the camera unit (120), and, for example, a color image of a person is collected in real time as in Fig. 2a, and depth information can be acquired by using a depth camera to obtain depth data for each pixel. This depth information indicates the distance from the camera to the object, and the RGB image and depth information are synchronized so that they are captured at the same time.

Next, 2D (two-dimensional) pose estimation can be performed using an AI model such as YOLOv8. YOLOv8 is a state-of-the-art deep learning model for real-time object detection and pose estimation, which can detect people by processing the input RGB image and estimate 2D joint positions. Such an AI model outputs 2D coordinates (u_i, v_i) for each joint i, and the major joints include the head, shoulder, elbow, wrist, hip, knee, and ankle. As shown in Fig. 2b, a total of N = 17 joints are estimated, and there are 17 keypoints in the default setting of such a 2D pose model, and each keypoint represents a specific part of the human body. For example, the mapping of each index to its corresponding body joint is: 0 is the nose, 1 is the left eye, 2 is the right eye, 3 is the left ear, 4 is the right ear, 5 is the left shoulder, 6 is the right shoulder, 7 is the left elbow, 8 is the right elbow, 9 is the left wrist, 10 is the right wrist, 11 is the left hip, 12 is the right hip, 13 is the left knee, 14 is the right knee, 15 is the left ankle, and 16 is the right ankle.

Finally, conversion to 3D coordinates can be performed. The depth value d_i corresponding to each 2D joint position (u_i, v_i) is extracted from the depth image, and this depth value indicates how far the corresponding joint is from the camera. In this way, by combining the 2D coordinates acquired through the AI model and the depth information extracted from the depth image, the 3D coordinates for each point are finally obtained, thereby obtaining 3D skeleton data of a person.

Robot skeleton extraction unit (132)

The robot skeleton extraction unit (132) is configured to extract skeleton information of the robot, and detects major joint points of the robot composed of multiple joints and estimates the positions. The robot skeleton estimation can be performed similarly to the human skeleton extraction unit (131) through a camera image, but more accurately, a method using the robot's API (application programming interface) can be used. The present invention is not limited to a specific robot or method, and various robot skeleton estimation methods can be applied. The sensors or libraries used can be changed according to the system requirements and the type of robot.

As an example, the RTDE (Real-Time Data Exchange) library of the Universal Robots (UR) robot can be used to obtain joint-related data such as joint angles, and then convert them into 3D coordinates of each joint using the Denavit-Hartenberg (DH) parameters as shown in Fig. 3a. In addition, for the coordinates of the fingertips of the robot arm of the robot, the Tool Center Point (TCP) can be obtained directly from the API and used. In addition, the robot coordinate system can be converted into the camera coordinate system in order to compare it with the 3D skeleton of a human.

More specifically, the joint angle of a robot is a value representing how much each joint of the robot rotates or moves, and the Denavit-Hartenberg (DH) parameter is a method to mathematically express the relationship between each link and joint of the robot, and is used to build a kinematic model of the robot. Forward kinematics is a method to calculate the position and posture of each part of the robot when the joint angle of the robot is known, and calibration is the process of finding the relationship between different coordinate systems (e.g., robot coordinate system and camera coordinate system) so that the coordinates can be converted.

First, in the data collection stage, the joint angle is obtained, and the angle value θ_j of each joint can be collected in real time using the robot's real-time library. The robot generally has multiple joints (e.g., 6), and each joint performs a rotational motion.

Next, the transformation to 3D coordinates is performed. Using the Denavit-Hartenberg (DH) parameters, the geometric relationship between each joint and link of the robot can be expressed, and forward kinematics can be performed, and the DH parameters (a _j , α _j , d _j , θ _j ) can be defined for each link. Here, a _j is the link length, α _j is the twist angle of the link, d _j is the link offset, and θ _j is the joint angle (a value measured in real time). The TCP (Tool Center Point) coordinate of the robot arm can be calculated using the transformation matrix to the last joint, or the TCP (Tool Center Point) coordinate obtained directly using the RTDE library of the UR robot can be used. Through this process, the 3D position of each joint can be obtained with respect to the robot base coordinate system, as shown in Fig. 3b.

Next, coordinate system transformation is performed. In order to compare with the human skeleton data, the robot's coordinate system is transformed into the camera coordinate system. To this end, as in Fig. 3c, eye-to-hand calibration can be performed to obtain a transformation matrix between the two coordinate systems.

Finally, by obtaining the three-dimensional coordinates for each joint of the robot based on the camera coordinate system, the robot's skeleton data can be extracted, and this data can be used in a collision avoidance algorithm by comparing it with human skeleton data.

Motion vector generation unit (133)

The motion vector generation unit (133) is configured to generate a motion vector of a person and a motion vector of a robot, respectively, using the skeleton information of a person and the skeleton information of a robot, and may be configured to generate a motion vector of a person and a motion vector of a robot, using the difference between a previous frame and a current frame, using three-dimensional coordinate information for a plurality of points stored according to a preset time interval.

The motion vector generation unit (133) stores three-dimensional skeleton data of a person and a robot over time, and tracks the motion vector of each joint point based on this. It stores skeleton data input in real time in frame units, and can maintain data for a certain period of time in the past to be used for motion analysis.

For example, looking at the data storage structure, the time window can store skeleton data for the past N frames based on the current frame. For example, when N = 10, a total of 10 frame data including the current frame can be held as shown in FIG. 4. Here, the data structure consists of person skeleton data Pi(t) and robot skeleton data Rj(t). Here, Pi(t) is the 3D coordinate of the ith joint point at time t, and is expressed as Pi(t) = (x _i (t), y _i (t), z _i (t)). Similarly, Rj(t) is the 3D coordinate of the jth joint point at time t, and is expressed as Rj(t) = (x _j (t), y _j (t), z _j (t)).

At this time, the method of generating a motion vector is as follows. First, select two points t_1 and t_2 to be compared to generate a vector by setting a time interval. Generally, the current frame time t and a specific time t-Δt in the past are used. For example, the current frame time t and the past frame time t'=t-k Δt are compared, where k is the frame interval and Δt is the time interval between frames.

[Mathematical formula: Correlation between each frame and time]

Skeleton motion vector generation consists of computing the vector V_i(t) for each joint point i. The vector computing formula is Vi(t) = Pi(t) - Pi(t'), where Vi(t) is the motion vector of the ith joint from time t' to t, and Pi(t) and Pi(t') are the positions of the ith joint at times t and t', respectively.

[Mathematical formula: Calculating skeleton movement vector]

An example of its use is comparing data from 5 frames ago with current data. If the frame rate is 30 fps, k = 5 and Δt = 1/30 seconds, and the past time is t' = t - 5 1/30 = t - 1/6 seconds. Through this, the motion vector V_i(t) for each joint of a person can be calculated to determine the movement for the last 1/6 seconds.

[Math: Frame Time Example]

The operation of the motion vector generation unit (133) first stores the 3D skeleton data of a person and a robot input in real time in the data collection and storage stage in units of frames. Up to N frame data are maintained, and when new data is input, the oldest data is deleted. Next, in the vector generation stage, a motion vector is generated using past and present skeleton data according to a set time interval. The vector is calculated and stored for each joint point. Finally, in the data output stage, the generated vector is used in risk assessment or robot vector adjustment and control in the subsequent stage, and the vector data can be utilized for attribute analysis such as the size and direction of movement.

In this way, the direction and speed of movement can be identified by expressing the change in position of joint points over time in vector form through the motion vector generated by skeleton extraction, which can be used for collision risk prediction, movement pattern analysis, robot motion planning, etc.

Vector filtering unit (134)

The vector filtering unit (134) may be configured to filter or apply weights to at least one specific motion vector determined to be important among a plurality of motion vectors generated by the motion vector generating unit. The collision risk determining unit (135) may be configured to determine the collision risk between a person and a robot by using the motion vector filtered by the vector filtering unit (134) or by using a vector value to which a weight is applied. Here, the vector filtering unit may be configured to filter motion vectors for points of a specific area preset among a plurality of points, and for example, the specific area may be preset as an area with a high possibility of collision or a dangerous area in the event of a collision, such as a human hand or face area, a robot's fingertips, or an area with a large turning radius.

Additionally, the vector filtering unit (134) may be configured to filter a motion vector based on at least one of the distance between a person and a robot, the size of the vector, and area information, or to apply a weight to a motion vector based on at least one of the distance between a person and a robot, the size of the vector, and area information.

In this way, the vector filtering unit (134) selects representative vectors that are deemed important from the skeleton data of humans and robots, thereby more efficiently determining the risk of collision and performing operations to improve safety.

The present invention describes three methods for filtering a specific vector, and these methods can be used in combination.

As a first example, the shortest distance-based selection method is intended to identify the most dangerous point in terms of the distance between a human and a robot. The Euclidean distance D_ij(t) between the human skeleton point P_i(t) and the robot skeleton point R_j(t) is calculated using the equation below, and the point pair (i ^* , j ^* ) with the minimum distance is selected, so that the motion vectors of the points with the shortest distance can be filtered. In addition, the point pairs with distances within a predetermined criterion can be filtered, so that the corresponding motion vectors can be filtered.

[Mathematical formula: Selection formula based on shortest distance calculation based on Euclidean distance]

As a second example, the momentum-based selection method is intended to identify collision risks in the most active areas of movement. The magnitude (velocity) of each skeleton vector is calculated using the formula below, and the joint points with the largest vector magnitude are selected, thereby filtering out vectors with the largest vector magnitudes. In addition, the corresponding motion vectors with vector magnitudes greater than a predetermined criterion can be filtered.

[Mathematical formula: Selection formula based on vector momentum]

As a third example, the weight-based selection method reflects the importance of specific areas with high risk in the event of a collision, such as a human hand, face area, or a robot's fingertips, by increasing the importance of these areas in the risk assessment. The importance is adjusted by assigning weights to each joint point corresponding to a specific area, such as a hand or face area, and the vector size S _i (t) to which the weight is applied is calculated. The corresponding mathematical formula is as follows.

[Mathematical formula: Weight-based selection formula]

Collision risk assessment unit (135)

The collision risk determination unit (135) may be configured to determine the collision risk and potential risk level between a person and a robot based on the motion vector of the person and the motion vector of the robot by using the vectors selected from the vector filtering unit (134) or the entire motion vectors. For example, the collision risk may be determined by using at least one of the distance, speed, and direction of the motion vector of the person and the motion vector of the robot.

The collision risk assessment unit (135) comprehensively analyzes movement and location information of people and robots to evaluate the risk level to predict the possibility of collision, and can then adjust the robot's operation or generate an alarm according to the risk level.

For example, the risk of collision can be assessed by analyzing the interaction between the human motion vector V_i(t) and the robot motion vector U_j(t). Various algorithms can be applied to calculate the risk by considering the direction, speed, distance, etc. of the motion vector.

As an example, we describe a method for evaluating the likelihood that movements will collide by calculating the inner product between motion vectors and applying a formula to numerically express the risk.

First, the inner product between the human motion vector V _i (t) and the robot motion vector U _j (t) is calculated to determine whether the movements are in a direction that collides with each other. The mathematical formulas for calculating the inner product and cosine angle are as follows.

[Mathematical formula: inner product and cosine angle between vectors]

Here, the closer the value of cosθ is to 1, the more it moves in the same direction, and the closer it is to -1, the more it moves in the opposite direction.

Additionally, the risk R(t) can be calculated by synthesizing distance, speed, and direction, and can be expressed as the following mathematical formula.

[Mathematical formula: Risk R(t) calculation formula]

Here, f is a function that calculates the risk, and various forms of functions can be applied. For example, it can be defined as a mathematical formula in the following form.

[Mathematical formula: Example of risk calculation function formula]

Here, w _{i and} w _j are the weights of the human and robot joints, D _ij (t) is the distance between the corresponding points, and cosθ is the directionality between the motion vectors.

Here, the risk level is classified according to the value of the generated risk R(t), and corresponding countermeasures are taken. The classification of risk levels and the response methods may vary depending on the system requirements and settings. For example, the risk levels can be classified into 'Safe', 'Caution', and 'Danger', and the robot's operation control or notification method can be applied according to each level.

For example, the boundary value of the risk level R(t) can be set as a first boundary value for distinguishing between the safety and caution levels and a second boundary value for distinguishing between the caution and danger levels, and the risk level can be classified into danger, caution, and safety levels based on R(t).

For example, at the safety level, the robot continues to perform the previously planned actions. At the caution level, for example, the robot's speed can be reduced, or at the same time, a visual or auditory warning can be given to the user. Also, at the danger level, the robot's movement path can be changed to move in a safe direction, and a strong alarm can be generated to notify the user of the danger.

In addition to these exemplary methods, this type of risk assessment method can apply various safety assessment techniques such as machine learning algorithms, statistical modeling, and rule-based systems. Depending on the selected method, the risk can be calculated, and the robot's actions can be controlled or a notification can be provided to the user based on the results.

Robot movement control unit (136)

The robot movement control unit (136) may be configured to change the movement of the robot to reduce the collision risk if the collision risk is higher than a predetermined standard. For example, the robot movement control unit (136) may be configured to reduce the movement speed of the robot if the collision risk is at a first level, which is a caution level, and may be configured to change both the movement speed and direction of the robot if the collision risk is at a second level, which is a higher risk level than the first level.

In addition, the robot movement control unit (136) can be configured to change the existing movement path by adjusting the current movement vector of the robot based on the movement vector of the robot adjusted using the movement vector of the person to more effectively avoid collisions between the person and the robot. Safety can be secured by controlling the movement of the robot in this way in real time.

There are various methods for adjusting the motion vector of a robot, and it is not limited to a specific method. As an example, we will explain how to perform vector operations on the current motion vector of a robot using a human motion vector, and control the robot using the adjusted motion vector.

First, the motion vector related to the original movement of the robot is called Urobot(t), and the motion vector of the human estimated from the human movement is called Vhuman(t). In the risk assessment module, if the risk is above a certain level, the motion vector of the robot can be adjusted as in the following mathematical formula.

[Mathematical formula: Robot motion vector adjustment formula]

Here, U'robot(t) is the motion vector of the adjusted robot, and α is the adjustment coefficient. This equation can effectively prevent collisions by adjusting the motion vector of the robot in a direction that increases the possibility of avoiding collisions by considering the motion vector of the human.

The robot movement control unit (136) controls the actual movement of the robot based on the adjusted movement vector U'robot(t) of the robot, and at this time, the speed, direction, etc. of the robot can be adjusted in real time based on the adjusted vector. Through this method, the robot can adjust the movement path to minimize the risk by reflecting the movement of a person in a dangerous situation, and prevent collisions.

FIG. 5 is a flowchart illustrating a real-time collision avoidance method for preventing collision between a person and a robot according to one embodiment of the present invention.

First, human skeleton information can be extracted using an image obtained through the camera unit (120) and an artificial intelligence model, etc. by the human skeleton extraction unit (131), and 3D coordinates for each point of the skeleton can be obtained. (S510)

In addition, by the robot skeleton extraction unit (132), real-time joint information of the robot can be collected and skeleton information of the robot can be extracted by using the transformation of the robot coordinate system and the camera coordinate system, thereby obtaining the three-dimensional coordinates of each joint of the robot (S520).

Next, by the motion vector generation unit (133), the motion vector of the person and the motion vector of the robot can be generated using the skeleton information of the person and the skeleton information of the robot, respectively (S530). For example, the generation of the motion vector of the skeleton can be utilized to analyze the properties of the movement, such as the size and direction, by calculating and storing the vector value for each point of the skeleton.

In addition, by the vector filtering unit (134), specific motion vectors with high importance can be filtered (S540). For example, vectors with high importance among the motion vectors of humans and robots can be selected and filtered, or weights can be applied, and a method such as a selection method based on the shortest distance between points, a selection method based on the size of a vector, or a method of applying weights to a specific area can be used.

Next, the collision risk assessment unit (135) can assess the collision risk between a person and the robot based on the movement vector of the person and the movement vector of the robot. (S550) At this time, the movement and position information of the person and the robot are comprehensively analyzed to evaluate the risk level for predicting the possibility of collision, and the robot's operation control or notification control can be performed according to the risk level.

Next, the robot movement adjustment unit (136) can change the movement of the robot to reduce the risk of collision (S560). For example, by adding a value obtained by applying an adjustment coefficient to the human movement vector to the current robot movement vector and applying it, the robot movement can be adjusted in real time by considering the human movement direction.

The various embodiments described herein may be implemented in hardware, middleware, microcode, software, and/or combinations thereof. For example, the various embodiments may be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions described herein, or a combination thereof.

Also, for example, various embodiments may be embodied or encoded in a computer-readable medium containing instructions. The instructions embodied or encoded in the computer-readable medium may cause a programmable processor or other processor to perform a method when the instructions are executed, for example. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. The storage media may be any available media that can be accessed by a computer. For example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage media, magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of computer-accessible instructions or data structures.

Such hardware, software, firmware, etc. may be implemented within the same device or within separate devices to support the various operations and functions described herein. Additionally, components, units, modules, components, etc. described as “~units” in the present invention may be implemented together or individually as separate but interoperable logic devices. The depiction of different features for modules, units, etc. is intended to emphasize different functional embodiments and does not necessarily imply that they must be realized by separate hardware or software components. Rather, the functionality associated with one or more modules or units may be performed by separate hardware or software components, or integrated into common or separate hardware or software components.

Although the operations are depicted in the drawings in a particular order, it should not be understood that these operations need to be performed in the particular order depicted, or in any sequential order, or that all of the depicted operations need to be performed to achieve a desired result. In some circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various components in the embodiments described above should not be understood to require such separation in all embodiments, and it should be understood that the components depicted may generally be integrated together in a single software product or packaged into multiple software products.

Although the present invention has been described with reference to the embodiments shown in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be determined by the technical idea of the appended claims.

110: Communications Department
120: Camera section
130: Processing Unit

Claims (13)

In a real-time collision avoidance system to prevent collisions between people and robots,
A human skeleton extraction unit configured to extract human skeleton information;
A robot skeleton extraction unit configured to extract skeleton information of a robot;
A motion vector generation unit configured to generate a motion vector of the person and a motion vector of the robot, respectively, using the skeleton information of the person and the skeleton information of the robot; and
A collision risk determination unit configured to determine the collision risk between the person and the robot based on the motion vector of the person and the motion vector of the robot.
Including,
The skeleton information of the above person and the skeleton information of the above robot each include multiple three-dimensional coordinate information for multiple points,
Further comprising a vector filtering unit configured to filter at least one specific motion vector among a plurality of motion vectors generated by the motion vector generating unit,
The above collision risk determination unit is configured to determine the collision risk between the person and the robot using the motion vector filtered by the vector filtering unit.
A real-time collision avoidance system, wherein the vector filtering unit is configured to filter a motion vector based on at least one of the distance between the points of the person and the robot, the size of the vector, and area information. A real-time collision avoidance system in the first paragraph, wherein the collision risk determination unit extracts the shortest distance between the person and the robot by using a plurality of three-dimensional coordinate information included in the skeleton information of the person and a plurality of three-dimensional coordinate information included in the skeleton information of the robot, and determines the collision risk based on the size of the shortest distance. A real-time collision avoidance system in claim 1, wherein the motion vector generation unit is configured to generate the motion vector of the person and the motion vector of the robot by using the difference between the previous frame and the current frame using three-dimensional coordinate information for a plurality of points stored according to a preset time interval. delete A real-time collision avoidance system in claim 1, wherein the vector filtering unit is configured to filter a motion vector for a point in a preset specific area among a plurality of points. delete A real-time collision avoidance system, wherein in the first paragraph, the region information is information related to whether it is a hand or face region, and is configured to calculate the motion vector by applying a weight based on the region information. A real-time collision avoidance system in the first paragraph, wherein the collision risk determination unit is configured to determine the collision risk by using at least one of the distance, speed, and direction of the motion vector of the person and the motion vector of the robot. A real-time collision avoidance system further comprising a robot movement adjustment unit configured to change the movement path of the robot to reduce the collision risk when the collision risk in the first paragraph is higher than a predetermined standard. A real-time collision avoidance system in claim 9, wherein the robot movement control unit is configured to reduce the movement speed of the robot when the collision risk is at a first level, and is configured to change both the movement speed and direction of the robot when the collision risk is at a second level higher than the first level. A real-time collision avoidance system in claim 9, wherein the robot movement control unit is configured to change the movement path of the robot based on a movement vector of the robot controlled using a movement vector of the person. A real-time collision avoidance system in claim 1, wherein the robot skeleton extraction unit is configured to obtain three-dimensional skeleton data of the robot by using transformation between the robot coordinate system and the camera coordinate system based on the joint angle of the robot. In a real-time collision avoidance method for preventing collisions between humans and robots,
A step of extracting human skeleton information by a human skeleton extraction unit;
A step of extracting skeleton information of a robot by a robot skeleton extraction unit;
A step of generating a motion vector of the person and a motion vector of the robot, respectively, by using the skeleton information of the person and the skeleton information of the robot by a motion vector generating unit; and
A step of determining the risk of collision between the person and the robot based on the motion vector of the person and the motion vector of the robot by a collision risk determination unit.
, and the skeleton information of the person and the skeleton information of the robot each include a plurality of three-dimensional coordinate information for a plurality of points,
A step of filtering at least one specific motion vector among a plurality of motion vectors generated by the motion vector generation unit is further included by the vector filtering unit,
The above collision risk determination unit further includes a step of determining the collision risk between the person and the robot using the motion vector filtered by the vector filtering unit,
A real-time collision avoidance method, wherein the vector filtering unit is configured to filter a motion vector based on at least one of the distance between the points of the person and the robot, the size of the vector, and area information.