CN111531581B - Industrial robot fault action detection method and system based on vision - Google Patents

Abstract

The invention provides a method and a system for detecting fault actions of an industrial robot based on vision, wherein the method for detecting the fault actions of the industrial robot based on the vision comprises the following steps of S1: collecting standard operation videos of the industrial robot, and establishing a standard operation mode video frame sequence of the industrial robot; s2: acquiring an industrial robot operation image in real time, and acquiring an industrial robot real-time action image; s3: matching the real-time action image of the industrial robot with the standard operation mode video frame sequence of the industrial robot, judging whether an image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S2, and if not, executing S4; s4: and controlling the industrial robot to suddenly stop. The invention has the advantages that the sudden failure of the industrial robot body is discovered in a non-contact mode, the safety accident that the robot hurts people is avoided when the robot is cooperated with the human machine, and the detection process is simple and accurate.

Description

A vision-based industrial robot fault action detection method and system

technical field

The invention relates to the technical field of intelligent manufacturing, in particular to a vision-based detection method and system for industrial robot fault actions.

Background technique

Industrial robot is a complex system composed of automation, machinery, embedded, hydraulic, electrical and other hardware and its control software. It can replace workers in some dangerous and complicated repetitive labor. Industrial robots have been widely used in the manufacturing industry due to their high precision and no need for rest. However, with the extensive application of industrial robots, incidents of industrial robots hurting people happen from time to time. The main causes of industrial robot safety accidents are human factors and the failure of the robot itself. Among them, the safety accidents caused by the robot's own misoperation accounted for more than half of the proportion. Human factors can be controlled through enhanced management and training, while safety issues caused by robot misoperations need to be resolved through technical means. Due to various reasons such as signal interference, device aging, and metal fatigue, there are a large number of robot misoperations during the robot operation process. Misoperation of the robot can cause the robot to move out of balance, lead to extrusion and collision accidents, and seriously threaten the lives of nearby personnel. Especially in the context of human-robot collaboration, the issue of robot safety is of paramount importance.

The Chinese patent with the authorized announcement number CN106625724B discloses a method for controlling the safety of an industrial robot on a cloud control platform. First, download the corresponding level of security protection logic from the cloud control platform to the security protection module according to the site conditions of the industrial robot; secondly, Calculate and analyze the real-time state information of each axis and end of the industrial robot through the safety protection logic, send an alarm message and control the robot to stop moving when an abnormal state occurs; finally, use the safety protection logic to analyze the control commands issued by the cloud control platform, Judging whether it will make the position and posture of the industrial robot exceed the safety protection range, and finally make a judgment of isolation or execution of control instructions.

The Chinese patent whose publication number is CN101509839 discloses a method for fault diagnosis of cluster industrial robots based on outlier mining, including the following steps:

1) Using a multi-input channel data acquisition card to obtain the operating state data of the cluster industrial robot; the operating state data includes: total power consumption, base vibration, power and operating current of each motor, angular velocity of the rotary joint, and task execution results;

2) Organize and classify the obtained running status data in a unified format, distinguish the data source and data type by adding data identifiers, and then transfer to the system database for storage;

3) Carry out cluster analysis on the running status data of clustered industrial robots, use the outlier point mining method to calculate the outlier factor of each industrial robot to obtain its outlier degree, and separate outlier points according to the outlier degree, and further determine Whether the individual industrial robot represented by the outlier point is faulty, and the specific part of the robot fault is judged by the type of abnormal operating parameters, and the fault diagnosis result is obtained;

4) Store information including operating status data and fault diagnosis results of industrial robots in the system database, and directly display the data through a dedicated display port as a basis for managing, maintaining and updating industrial robots.

In the prior art, it is necessary to use multiple data acquisition devices to collect the status information of the industrial robot, and process the status information collected by the multiple data acquisition devices to determine whether the status of the industrial robot is abnormal. The detection process is relatively complicated and the cost is high.

Contents of the invention

The purpose of the present invention is to overcome the shortcomings in the prior art, to provide a vision-based industrial robot fault action detection method and system, which can detect sudden faults of the industrial robot body in a non-contact manner, and avoid occurrence of faults during human-machine cooperation. The advantages of safety accidents caused by robots hurting people, and the detection process are simple and accurate.

The purpose of the present invention is achieved by the following technical solutions: a vision-based industrial robot failure action detection method, comprising the following steps,

S1: Collect the standard operation video of the industrial robot, and establish the video frame sequence of the standard operation mode of the industrial robot;

S2: Real-time collection of industrial robot operation images, to obtain real-time action images of industrial robots;

S3: Match the real-time action image of the industrial robot with the video frame sequence of the standard operating mode of the industrial robot, and judge whether there is an image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operating mode of the industrial robot. If yes, execute S2, if not , execute S4;

S4: Control the emergency stop of the industrial robot.

The beneficial effect of the present invention is that the method has the advantages of adopting a non-contact mode to collect the operation images of the industrial robot in real time, matching the real-time action image of the industrial robot with the video frame sequence of the standard operation mode of the industrial robot, and judging the video frame sequence of the standard operation mode of the industrial robot Whether there is an image that approximately matches the real-time action image of the industrial robot, if so, judge that the working posture of the industrial robot is normal; if not, judge that the working posture of the industrial robot is abnormal and control the emergency stop of the industrial robot. Real-time status information of real-time status information or operating status data of industrial robots, the detection process is simple and accurate and the cost is low.

Further, the S1 specifically includes the following steps,

S11: collect standard operation videos of industrial robots;

S12: Extract T video frames from the standard operation video of the industrial robot to form a video frame sequence;

S13: Extracting the frame of the video frame sequence containing a cycle of the action image of the industrial robot, and establishing the video frame sequence of the industrial robot operation mode;

S14: Carry out image segmentation on the video frame sequence of the industrial robot operation mode, separate the industrial robot image, and establish a video frame sequence of the industrial robot standard operation mode.

The beneficial effect of adopting the above further solution is that the standard operation video of the industrial robot is collected, and T video frame extraction is performed on the standard operation video of the industrial robot to form a sequence of video frames. In order to facilitate the collection of video, the video frame sequence does not only include an image of an industrial robot operating periodically, so it is necessary to cut the video frame sequence, extract the frame of the video frame sequence containing a cycle of the industrial robot's action image, and establish an industrial robot operation Pattern video frame sequence. In order to increase the accuracy of industrial robot motion detection, it is necessary to segment the video frame sequence of the industrial robot operation mode, separate the industrial robot image, and establish the video frame sequence of the industrial robot standard operation mode.

Further, said S13 specifically includes the following steps,

S131: The video frame sequence is <I ₁ , I ₂ ,…I _n >, I _k , k∈N are image frames, each frame of image contains an operation action of the industrial robot, and marks the starting frame of a working cycle of the working robot I _s and end frame I _e ;

S132: Extract image frames of one working cycle of the working robot, and generate a sequence of video frames <I _s , I _s+1 , . . . I _e > in the working mode of the industrial robot.

The beneficial effect of adopting the above-mentioned further solution is that, in order to facilitate the collection of video, when collecting the standard operation video of industrial robots, it is not strictly required to collect from the beginning of a cycle. Therefore, the video frame sequence does not only include a cycle of operation. Therefore, it is necessary to cut the video frame sequence, extract the frames containing the action image of one cycle of the industrial robot in the video frame sequence, and establish the video frame sequence of the industrial robot operation mode. By marking the start frame I _s and the end frame I _e of a working cycle of the working robot to extract the image frames of a working cycle of the working robot, and generate the video frame sequence of the industrial robot working mode <I _s , I _s+1 ,…I _e > .

Further, the image segmentation in S14 specifically includes,

S141: Determine the color C _r of the industrial robot;

S142: I is an operation image including an industrial robot, P is any pixel in I, judge whether the color value of P is in the δ field centered on C _r , if yes, execute S143, if not, execute S144;

S143: Set the color value of P to black;

S144: Set the color value of P to white.

The beneficial effect of adopting the above-mentioned further solution is that in order to avoid misjudgment caused by the background image when the real-time action image of the industrial robot is matched with the video frame sequence of the standard operating mode of the industrial robot, it is necessary to convert each frame in the video frame sequence of the standard operating mode of the industrial robot to The industrial robot image is extracted and matched with the industrial robot image in the real-time action image of the industrial robot to improve the accuracy of the fault action detection of the industrial robot.

Further, the S3 specifically includes the following steps.

S31: Initialize the serial number variable q ₀ of the real-time action image =-1;

S32: Find whether there is an image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operation mode of the industrial robot, if yes, execute S33, if not, execute S4;

S33: Record the serial number q ₁ of the image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operation mode of the industrial robot;

S34: If q ₁ =-1Vq ₁ =q ₀ +1, set q ₀ =q ₁ , where V is an operation symbol, representing an OR operation, and execute S2.

Further, said S32 specifically includes,

S321: Initialize the image frame sequence number q ₁ in the video frame sequence in the standard operation mode of the industrial robot, q ₁ =s;

S322: Calculate the difference between the real-time action image of the industrial robot and the image frame Iq1 in the video frame sequence of the standard operating mode of the industrial robot, the image difference calculation method is:

Among them, d(I ₁ , I ₂ ) represents the difference between image I ₁ and image I ₂ , m×n represents the resolution of the image, and I ₁ (i, j) represents the i _- th row and the The color value of the j-column pixel, I ₂ (i, j) represents the color value of the i-th row and j-th column pixel of the image I ₂ ;

S323: Determine whether the difference is smaller than the threshold D, if yes, execute S325, if not, execute S324;

S324: Let q ₁ =q ₁ +1, execute S322;

S325: Determine that the image with the sequence number q ₁ in the video frame sequence of the standard operation mode of the industrial robot is an approximate matching image of the real-time action image of the industrial robot.

A vision-based industrial robot fault action detection system, including,

The image acquisition device is used to collect the standard operation video of the industrial robot, and is also used to collect the real-time action image of the industrial robot in real time;

The fault detection device is used to receive the standard operation video of the industrial robot collected by the image acquisition device to establish the video frame sequence of the standard operation mode of the industrial robot, and is also used to receive the real-time action image of the industrial robot collected by the image acquisition device in real time, and combine the real-time action image of the industrial robot with the The video frame sequence of the standard operating mode of the industrial robot is matched to determine whether there is an image that approximately matches the real-time action image of the industrial robot in the video frame sequence of the standard operating mode of the industrial robot. The emergency stop control signal is sent after the action image approximately matches the image;

The controller is used for the emergency stop control signal sent by the fault detection device and controls the industrial robot to stop working.

The beneficial effect of adopting the above-mentioned further scheme is that the system can collect the operation images of the industrial robots in real time in a non-contact manner, and match the real-time action images of the industrial robots with the video frame sequence of the standard operation mode of the industrial robots through the fault detection device to judge the industrial robot. Whether there is an image that approximately matches the real-time action image of the industrial robot in the video frame sequence of the standard operation mode, if so, judge that the working posture of the industrial robot is normal; if not, judge that the working posture of the industrial robot is abnormal, and control the industrial robot to emergency stop through the controller The data acquisition device collects the real-time state information of each axis and end of the industrial robot or the operating state data of the industrial robot. The detection process is simple and accurate and the cost is low.

Further, the fault detection device includes an industrial robot standard operation video establishment unit, an image segmentation unit and an image matching unit,

The standard operation video building unit of the industrial robot is used to extract T video frames from the standard operation video of the industrial robot to form a video frame sequence, and is also used to extract frames containing a cycle of the action image of the industrial robot in the video frame sequence to establish an industrial robot Operation mode video frame sequence, each frame in the video frame sequence of the industrial robot operation mode contains an operation action of the industrial robot;

The image segmentation unit is used to extract the industrial robot image in each frame of the video frame sequence of the machine industrial robot operation mode, and send it to the industrial robot standard operation video establishment unit;

The industrial robot standard operation video establishment unit is used to receive the industrial robot image of each frame extracted by the image segmentation unit, and generate the industrial robot standard operation video;

The image segmentation unit is also used to extract the industrial robot image in the real-time action image of the industrial robot;

The image matching unit is used to match the real-time motion image of the industrial robot with the image in the video frame sequence of the standard operation mode of the industrial robot, and judge whether there is an image approximately matching the real-time motion image of the industrial robot in the video frame sequence of the standard operation mode of the industrial robot , and determine that there is no image that approximately matches the real-time action image of the industrial robot in the video frame sequence of the standard operating mode of the industrial robot, and then send an emergency stop control signal.

The beneficial effect of adopting the above further solution is that the image acquisition device collects the standard operation video of the industrial robot, and the standard operation video establishment unit of the industrial robot extracts T video frames from the standard operation video of the industrial robot to form a sequence of video frames. In order to facilitate the collection of video, the video frame sequence does not only include an image of an industrial robot operating periodically, so it is necessary to cut the video frame sequence, extract the frame of the video frame sequence containing a cycle of the industrial robot's action image, and establish an industrial robot operation Pattern video frame sequence. In order to increase the accuracy of industrial robot motion detection, the image segmentation unit is required to segment the video frame sequence of the industrial robot’s operation mode to separate the industrial robot image, and the industrial robot standard operation video establishment unit establishes the industrial robot based on the image processed by the image segmentation unit Standard operating mode video frame sequence.

Further, the image segmentation unit extracting the industrial robot image includes the following steps,

S141: Determine the color C _r of the industrial robot;

S142: I is an operation image including an industrial robot, P is any pixel in I, judge whether the color value of P is in the δ field centered on C _r , if yes, execute S143, if not, execute S144;

S143: Set the color value of P to black;

S144: Set the color value of P to white.

The beneficial effect of adopting the above-mentioned further scheme is that, in order to prevent the background image from causing misjudgment when the image matching unit matches the real-time motion image of the industrial robot with the video frame sequence of the standard operating mode of the industrial robot, it is necessary to include the video frame sequence of the standard operating mode of the industrial robot. The industrial robot image of each frame is extracted and matched with the industrial robot image in the real-time action image of the industrial robot to improve the accuracy of the industrial robot fault action detection.

Further, the image matching unit is also used to determine that there is an image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operating mode of the industrial robot, and then record the video frame sequence of the standard operating mode of the industrial robot to approximately match the real-time action image of the industrial robot The sequence number q ₁ of the image.

The beneficial effect of adopting the above-mentioned further solution is that the image matching unit is also used to judge that there is an image that approximately matches the real-time action image of the industrial robot in the video frame sequence of the standard operation mode of the industrial robot, and then record the image matching the video frame sequence of the standard operation mode of the industrial robot with the industrial robot. The sequence number q ₁ of the image that the real-time action image approximately matches is convenient for the operator to judge the accuracy of the image matching unit according to the sequence number q ₁ recorded in the image and the real-time working status of the industrial robot.

Description of drawings

Fig. 1 is the schematic diagram of a kind of vision-based industrial robot malfunction detection system of embodiment 1 of the present invention;

Fig. 2 is the schematic flow chart that the present invention is used to show industrial robot fault motion detection;

Fig. 3 is the schematic diagram that the present invention is used for showing the video frame sequence process of establishing the industrial robot standard operating mode;

FIG. 4 is a schematic diagram of the present invention for displaying an approximate matching process of real-time action images of industrial robots.

Detailed ways

The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

Example 1

Referring to Figures 1 and 2, a vision-based industrial robot fault action detection system includes,

The image acquisition device is used to collect the standard operation video of the industrial robot, and is also used to collect real-time action images of the industrial robot in real time; it is worth noting that, in this embodiment, the image acquisition device is a high-definition camera;

The fault detection device is used to receive the standard operation video of the industrial robot collected by the image acquisition device to establish the video frame sequence of the standard operation mode of the industrial robot, and is also used to receive the real-time action image of the industrial robot collected by the image acquisition device in real time, and combine the real-time action image of the industrial robot with the The video frame sequence of the standard operating mode of the industrial robot is matched to determine whether there is an image that approximately matches the real-time action image of the industrial robot in the video frame sequence of the standard operating mode of the industrial robot. The emergency stop control signal is sent after the action image approximately matches the image;

The controller is used for the emergency stop control signal sent by the fault detection device and controls the industrial robot to stop working.

Specifically, this system adopts a non-contact method to collect the operation images of industrial robots in real time, matches the real-time action images of industrial robots with the video frame sequence of the standard operation mode of the industrial robot through the fault detection device, and judges the video frame sequence of the standard operation mode of the industrial robot. Whether there is an image that approximately matches the real-time action image of the industrial robot, if yes, judge that the working posture of the industrial robot is normal; if not, judge that the working posture of the industrial robot is abnormal, and control the emergency stop of the industrial robot through the controller. The real-time status information of each axis and end or the operating status data of industrial robots, the detection process is simple and accurate and the cost is low.

Referring to Fig. 1, it is worth noting that the fault detection device includes an industrial robot standard operation video establishment unit, an image segmentation unit and an image matching unit. The three units are described in turn below.

Referring to Figure 3, the industrial robot standard operation video establishment unit is used to perform T video frame extraction on the industrial robot standard operation video. It is worth noting that the T video frame extraction refers to the interval between two adjacent frames in the video frame sequence The time is T, forming a sequence of video frames, <I ₁ , I ₂ ,…I _n >, I _k , k∈N are image frames. The video building unit for the standard operation of the industrial robot is also used to extract frames containing a cycle of motion images of the industrial robot in the video frame sequence, and establish a video frame sequence of the industrial robot operation mode <I _s , I _s+1 ,...I _e >, wherein, I _s and I _e are the start frame I _s and the end frame I _{e of a working cycle of the working robot, respectively,} and each frame in the video frame sequence of the industrial robot operation mode contains an operation action of the industrial robot.

It is worth noting that, in this embodiment, the starting frame I _s and the ending frame I _e of a working cycle of the working robot are determined by manual marking. In another implementation, another method can be adopted to determine the start frame I _s and the end frame I _e of a working cycle of the working robot, for example, first determine the starting image of a working cycle of the working robot as the starting frame I _s , and determine The image that is separated from the start frame I _s by a cycle N time is the end frame I _e . In another embodiment, another method can be adopted to determine the start frame I _s and the end frame I _e of a working cycle of the working robot, for example, first determine the starting image of a working cycle of the working robot as the starting frame I _s , looking for an image whose time interval from the start frame I _s is greater than the time threshold and whose image similarity with the start frame I _s is greater than the similarity threshold is the end frame I _e .

The image segmentation unit is used to extract the industrial robot image in each frame image of the machine industrial robot operation mode video frame sequence, and send it to the industrial robot standard operation video establishment unit.

It is worth noting that the image segmentation unit extracting the industrial robot image includes the following steps,

S141: Determine the color C _r of the industrial robot;

S142: I is an operation image including an industrial robot, P is any pixel in I, judge whether the color value of P is in the δ field centered on C _r , if yes, execute S143, if not, execute S144;

S143: Set the color value of P to black;

S144: Set the color value of P to white.

In this way, the binarization processing of the operation image of the industrial robot is realized, the industrial robot is separated from the background, and the accuracy of the fault action detection of the industrial robot is improved.

The industrial robot standard operation video building unit is used to receive the industrial robot image of each frame extracted by the image segmentation unit, and generate the industrial robot standard operation video.

The image segmentation unit is also used to extract the industrial robot image in the real-time action image of the industrial robot.

Referring to Figure 4, the image matching unit is used to match the real-time action images of the industrial robot with the images in the video frame sequence of the standard operating mode of the industrial robot, and judge whether there is an approximate match between the video frame sequence of the standard operating mode of the industrial robot and the real-time action image of the industrial robot After judging that there is no image that approximately matches the real-time action image of the industrial robot in the video frame sequence of the standard operating mode of the industrial robot, an emergency stop control signal is sent. The image matching unit is also used to judge that there is an image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operating mode of the industrial robot, and then record the sequence of the image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operating mode of the industrial robot No. q ₁ .

It is worth noting that the image matching unit matches the real-time action images of the industrial robot with the images in the video frame sequence of the standard operating mode of the industrial robot, specifically including the following steps,

S31: Initialize the serial number variable q ₀ of the real-time action image =-1;

S32: Find whether there is an image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operation mode of the industrial robot, if yes, execute S33, if not, send an emergency stop control signal to the controller;

S33: Record the serial number q ₁ of the image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operation mode of the industrial robot;

S34: If q ₁ =-1Vq ₁ =q ₀ +1, then set q ₀ =q ₁ , where V is an operation symbol, representing an OR operation, and matching the real-time action image of the next frame of industrial robot.

It is also worth noting that S32 specifically includes,

S321: Initialize the image frame sequence number q ₁ in the video frame sequence in the standard operation mode of the industrial robot, q ₁ =s;

S322: Calculate the difference between the real-time action image of the industrial robot and the image frame Iq1 in the video frame sequence of the standard operating mode of the industrial robot, the image difference calculation method is:

Among them, d(I ₁ , I ₂ ) represents the difference between image I ₁ and image I ₂ , m×n represents the resolution of the image, and I ₁ (i, j) represents the i _- th row and the The color value of the j-column pixel, I ₂ (i, j) represents the color value of the i-th row and j-th column pixel of the image I ₂ ;

S323: Determine whether the difference is smaller than the threshold D, if yes, execute S325, if not, execute S324;

S324: Let q ₁ =q ₁ +1, execute S322;

S325: Determine that the image with the sequence number q ₁ in the video frame sequence of the standard operation mode of the industrial robot is an approximate matching image of the real-time action image of the industrial robot, and execute S33.

It is also worth noting that, in order to facilitate the operator to judge the accuracy of the image matching unit according to the serial number q ₁ recorded in the image and the real-time working status of the industrial robot, the image matching unit records the real-time information in the video frame sequence of the standard operating mode of the industrial robot and the industrial robot. The sequence number q ₁ of the image that the action image approximately matches, the operator can query the sequence number q ₁ through an external human-computer interaction device (such as a display screen, mouse, and keyboard), and make a judgment in combination with the real-time operation action of the industrial robot.

It is worth noting that, in this embodiment, the controller communicates with the industrial robot control cabinet, and the controller stops the industrial robot in an emergency; in another embodiment, the controller can be directly connected to the electric control switch in the power line of the industrial robot , by controlling the disconnection of the electric control switch, the emergency stop of the industrial robot is controlled.

It is also worth noting that in this embodiment, the hardware device of the fault detection device may include a central processing unit (Central Processing Unit, CPU), and may also include other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), dedicated Integrated Circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The hardware device of the fault detection device also includes a memory. The memory may be an internal storage unit of the processor, such as a hard disk or internal memory of the processor. The memory can also be an external storage device of the processor, such as a plug-in hard disk equipped on the processor, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card), etc. . Memory may also include both internal storage units of the processor and external storage devices. Memory is used to store computer programs and other programs and data required by the processor. The memory can also be used to temporarily store data that has been output or will be output.

It is also worth noting that in this embodiment, the hardware device of the controller may include a central processing unit (Central Processing Unit, CPU), and may also include other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The hardware device of the controller also includes a memory. The memory may be an internal storage unit of the processor, such as a hard disk or internal memory of the processor. The memory can also be an external storage device of the processor, such as a plug-in hard disk equipped on the processor, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card), etc. . Memory may also include both internal storage units of the processor and external storage devices. Memory is used to store computer programs and other programs and data required by the processor. The memory can also be used to temporarily store data that has been output or will be output.

Example 2

Referring to Fig. 2, a kind of vision-based industrial robot malfunction detection method comprises the following steps,

S1: Collect the standard operation video of the industrial robot, and establish the video frame sequence of the standard operation mode of the industrial robot;

S2: Real-time collection of industrial robot operation images, to obtain real-time action images of industrial robots;

S3: Match the real-time action image of the industrial robot with the video frame sequence of the standard operating mode of the industrial robot, and judge whether there is an image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operating mode of the industrial robot. If yes, execute S2, if not , execute S4;

S4: Control the emergency stop of the industrial robot.

Specifically, the method has the steps of collecting the operation image of the industrial robot in real time in a non-contact manner, matching the real-time action image of the industrial robot with the video frame sequence of the standard operation mode of the industrial robot, and judging whether there is any difference in the video frame sequence of the standard operation mode of the industrial robot. The image of the real-time action image of the industrial robot is approximately matched. If yes, it is judged that the working posture of the industrial robot is normal; Or the operating status data of industrial robots, the detection process is simple and accurate and the cost is low.

Each step is described in detail below in turn.

Referring to Figure 3, S1 specifically includes the following steps,

S11: collect standard operation videos of industrial robots;

S12: Perform T video frame extraction on the standard operation video of the industrial robot to form a video frame sequence; where it is worth noting that the T video frame extraction means that the interval between two adjacent frames in the video frame sequence is T;

S13: Extracting the frame of the motion image of one cycle of the industrial robot in the video frame sequence, and establishing the video frame sequence of the industrial robot operation mode;

S14: Carry out image segmentation on the video frame sequence of the industrial robot operation mode, separate the industrial robot image, and establish the video frame sequence of the industrial robot standard operation mode.

Specifically, the standard operation video of the industrial robot is collected, and T video frame extraction is performed on the standard operation video of the industrial robot to form a sequence of video frames. In order to facilitate the collection of video, the video frame sequence does not only include an image of an industrial robot operating periodically, so it is necessary to cut the video frame sequence, extract the frame of the video frame sequence containing a cycle of the industrial robot's action image, and establish an industrial robot operation Pattern video frame sequence. In order to increase the accuracy of industrial robot motion detection, it is necessary to segment the video frame sequence of the industrial robot operation mode, separate the industrial robot image, and establish the video frame sequence of the industrial robot standard operation mode.

It is worth noting that S13 specifically includes the following steps,

S131: The video frame sequence is <I ₁ , I ₂ ,…I _n >, I _k , k∈N are image frames, each frame of image contains an operation action of the industrial robot, and marks the starting frame of a working cycle of the working robot I _s and end frame I _e ;

S132: Extract image frames of one working cycle of the working robot, and generate a sequence of video frames <I _s , I _s+1 , . . . I _e > in the working mode of the industrial robot.

It is also worth noting that in this embodiment, the starting frame I _s and the ending frame I _e of a working cycle of the working robot are determined by manual marking. In another implementation, another method can be adopted to determine the start frame I _s and the end frame I _e of a working cycle of the working robot, for example, first determine the starting image of a working cycle of the working robot as the starting frame I _s , and determine The image that is separated from the start frame I _s by a cycle N time is the end frame I _e . In another embodiment, another method can be adopted to determine the start frame I _s and the end frame I _e of a working cycle of the working robot, for example, first determine the starting image of a working cycle of the working robot as the starting frame I _s , looking for an image whose time interval from the start frame I _s is greater than the time threshold and whose image similarity with the start frame I _s is greater than the similarity threshold is the end frame I _e .

It is also worth noting that the image segmentation in S14 specifically includes,

S141: Determine the color C _r of the industrial robot;

S142: I is an operation image including an industrial robot, P is any pixel in I, judge whether the color value of P is in the δ field centered on C _r , if yes, execute S143, if not, execute S144;

S143: Set the color value of P to black;

S144: Set the color value of P to white.

In this way, the binarization processing of the operation image of the industrial robot is realized, and the industrial robot is separated from the background.

S2 specifically includes,

Collect the operation images of industrial robots in real time, perform image segmentation on the operation images of industrial robots, and obtain real-time action images of industrial robots.

It is worth noting that image segmentation in S2 includes the following steps,

S21: Determine the color C _r of the industrial robot;

S22: I is an operation image including an industrial robot, P is any pixel in I, judge whether the color value of P is in the δ field centered on C _r , if yes, execute S143, if not, execute S144;

S23: set the color value of P to black;

S24: Set the color value of P to white.

Through the image segmentation step in step S14 and the image segmentation step in S2, first, the influence of the background image in the image matching process in S3 is reduced; second, the color of the industrial robot itself and the color of the industrial robot standard operation video are reduced. The influence of the difference between the colors of the industrial robot itself improves the accuracy of image matching in S3, while improving the generality of the method.

Referring to Fig. 4, it is worth noting that S3 specifically includes the following steps.

S31: Initialize the serial number variable q ₀ of the real-time action image =-1;

S32: Find whether there is an image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operation mode of the industrial robot, if yes, execute S33, if not, execute S4;

S33: Record the serial number q ₁ of the image approximately matching the real-time action image of the industrial robot in the video frame sequence of the standard operation mode of the industrial robot;

S34: If q ₁ =-1Vq ₁ =q ₀ +1, set q ₀ =q ₁ , where V is an operation symbol, representing an OR operation, and execute S2.

It is also worth noting that S32 specifically includes,

S321: Initialize the image frame sequence number q ₁ in the video frame sequence in the standard operation mode of the industrial robot, q ₁ =s;

S322: Calculate the difference between the real-time action image of the industrial robot and the image frame Iq1 in the video frame sequence of the standard operating mode of the industrial robot, the image difference calculation method is:

Among them, d(I ₁ , I ₂ ) represents the difference between image I ₁ and image I ₂ , m×n represents the resolution of the image, and I ₁ (i, j) represents the i _- th row and the The color value of the j-column pixel, I ₂ (i, j) represents the color value of the i-th row and j-th column pixel of the image I ₂ ;

S323: Determine whether the difference is smaller than the threshold D, if yes, execute S325, if not, execute S324;

S324: Let q ₁ =q ₁ +1, execute S322;

S325: Determine that the image with the sequence number q ₁ in the video frame sequence of the standard operation mode of the industrial robot is an approximate matching image of the real-time action image of the industrial robot, and execute S33.

Specifically, in order to facilitate operators to judge the accuracy of the method according to the serial number q ₁ recorded in the image and the real-time working status of the industrial robot, record the images in the video frame sequence of the standard operation mode of the industrial robot that approximately match the real-time action image of the industrial robot The serial number q ₁ , the operator can judge by querying the serial number q ₁ and combining the real-time operation actions of the industrial robot.

The above are only preferred embodiments of the present invention, and it should be understood that the present invention is not limited to the forms disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be used in It is within the contemplation of this document with modification from the above teachings or skill or knowledge in the relevant art. The modifications and changes made by those skilled in the art without departing from the spirit and scope of the present invention should be protected by the appended claims of the present invention.

Claims (8)

1. A method for detecting fault actions of an industrial robot based on vision is characterized by comprising the following steps,

s1: collecting standard operation videos of the industrial robot, and establishing a standard operation mode video frame sequence of the industrial robot;

s2: acquiring an industrial robot operation image in real time, and acquiring an industrial robot real-time action image;

s3: matching the real-time action image of the industrial robot with the standard operation mode video frame sequence of the industrial robot, judging whether an image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S2, and if not, executing S4;

the S3 specifically comprises the following steps:

s31: sequence number variable for initializing real-time motion imageq ₀ =-1;

S32: searching whether an image approximately matched with the real-time motion image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S33, and if not, executing S4;

s33: recording serial numbers of images approximately matched with real-time motion images of industrial robot in standard operation mode video frame sequence of industrial robotq ₁ ；

S34: if it is

Then make an order

Wherein, V is an operation symbol, represents an OR operation, and executes S2;

the S32 specifically includes the following steps:

s321: image frame number in video frame sequence in standard operation mode of initial industrial robotq ₁ ，q ₁ =s；

S322: calculating the difference value of the real-time motion image of the industrial robot and an image frame Iq1 in a standard operation mode video frame sequence of the industrial robot, wherein the difference value calculating method comprises the following steps:

wherein d (I) ₁ ,I ₂ ) Representing an image I ₁ And image I ₂ Difference between, m × n represents resolution of image, I ₁ (I, j) represents an image I ₁ Of the ith row and the jth column of pixels, I ₂ (I, j) represents an image I ₂ The color value of the ith row and jth column of pixels;

s323: judging whether the difference value is smaller than a threshold value D, if so, executing S325, and if not, executing S324;

s324: order toq ₁ =q ₁ +1, go to S322;

s325: judging the serial number in the video frame sequence of the standard working mode of the industrial robot to beq ₁ The image (2) is an approximate matching image of the real-time action image of the industrial robot, and S33 is executed;

s4: and controlling the industrial robot to suddenly stop.

2. The vision based industrial robot malfunction motion detection method according to claim 1, wherein said S1 specifically comprises the steps of,

s11: collecting standard operation video of an industrial robot;

s12: performing T video frame extraction on the standard operation video of the industrial robot to form a video frame sequence;

s13: extracting frames containing motion images of one period of the industrial robot in the video frame sequence, and establishing an industrial robot operation mode video frame sequence;

s14: and carrying out image segmentation on the industrial robot operation mode video frame sequence, separating the industrial robot image, and establishing the industrial robot standard operation mode video frame sequence.

3. The vision based industrial robot malfunction motion detection method according to claim 2, wherein said S13 specifically comprises the steps of,

s131: the sequence of video frames is<I ₁ , I ₂ , … I _n >，

For image frames, each image frame contains a working movement of the industrial robot, marking the start frame of a working cycle of the working robotI _s And an end frameI _e ；

S132: extracting image frames of a working cycle of the working robot to generate a video frame sequence of an industrial robot working mode<I _s , I _s+1 , … I _e >。

4. The vision-based industrial robot malfunction detection method according to claim 2, wherein the image segmentation in S14 specifically includes,

s141: determining color of industrial robotC _r ；

S142：IIn order to contain the working image of an industrial robot,Pis composed ofIAny pixel in (2), judgingPWhether or not the color value of (A) isC _r Is centered

If yes, executing S143, otherwise executing S144;

s143: will be provided withPIs set to black;

s144: will be provided withPIs set to white.

5. A vision-based industrial robot malfunction detection system, characterized in that the vision-based industrial robot malfunction detection method of claim 1 is adopted, comprising,

the image acquisition device is used for acquiring standard operation videos of the industrial robot and acquiring real-time action images of the industrial robot in real time;

the fault detection device is used for receiving the standard operation video of the industrial robot acquired by the image acquisition device to establish a standard operation mode video frame sequence of the industrial robot, receiving the real-time action image of the industrial robot acquired by the image acquisition device in real time, matching the real-time action image of the industrial robot with the standard operation mode video frame sequence of the industrial robot, judging whether an image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot or not, and sending an emergency stop control signal after judging that the image approximately matched with the real-time action image of the industrial robot does not exist in the standard operation mode video frame sequence of the industrial robot;

and the controller is used for sending an emergency stop control signal to the fault detection device and controlling the industrial robot to stop working.

6. The vision-based industrial robot malfunction detection system according to claim 5, wherein said malfunction detection means comprises an industrial robot standard job video creation unit, an image segmentation unit, and an image matching unit,

the industrial robot standard operation video establishing unit is used for extracting T video frames of an industrial robot standard operation video to form a video frame sequence, extracting frames containing motion images of one period of an industrial robot in the video frame sequence and establishing an industrial robot operation mode video frame sequence, wherein each frame in the industrial robot operation mode video frame sequence contains one operation motion of the industrial robot;

the image segmentation unit is used for extracting an industrial robot image in each frame of image of the robot industrial robot working mode video frame sequence and sending the image to the industrial robot standard working video establishing unit;

the industrial robot standard operation video establishing unit is used for receiving the industrial robot image of each frame extracted by the image segmentation unit and generating an industrial robot standard operation video;

the image segmentation unit is also used for extracting an industrial robot image in the real-time action image of the industrial robot;

the image matching unit is used for matching the real-time action image of the industrial robot with the image in the standard operation mode video frame sequence of the industrial robot, judging whether the image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot or not, and sending an emergency stop control signal after judging that the image approximately matched with the real-time action image of the industrial robot does not exist in the standard operation mode video frame sequence of the industrial robot.

7. The vision based industrial robot malfunction detection system of claim 6, wherein the image segmentation unit extracts an industrial robot image comprising the steps of,

s141: determining color of industrial robotC _r ；

S142：IIn order to contain the working image of an industrial robot,Pis composed ofIAny pixel in (2), judgingPWhether or not the color value of (A) isC _r Is centered

If yes, executing S143, otherwise executing S144;

s143: will be provided withPIs set to black;

s144: will be provided withPIs set to white.

8. The vision-based industrial robot malfunction detection system according to claim 6, wherein the image matching unit is further configured to record a serial number of an image approximately matching the real-time motion image of the industrial robot in the standard work pattern video frame sequence of the industrial robot after determining that the image approximately matching the real-time motion image of the industrial robot exists in the standard work pattern video frame sequence of the industrial robotq ₁ 。

Images