CN115111970B - A firework molding detection device and detection method that integrates 2D and 3D visual perception - Google Patents

Abstract

The invention discloses a firework forming detection device that integrates 2D and 3D visual perception. An RGBD camera is fixedly installed on the light source camera adjustment seat. The RGBD camera is located at the opening of the surface light source; the light source camera adjustment seat is installed on the detection device bracket; The industrial control computer is connected to the RGBD camera and the granulator through wires. The invention also discloses a detection method of the above-mentioned fireworks forming detection device. Realize intelligent detection of the forming process of fireworks beads to achieve the purpose of separating human and drug, eliminate safety hazards in the production process of fireworks beads, and improve the quality of finished fireworks beads products.

Description

A firework molding detection device and detection method that integrates 2D and 3D visual perception

Technical field

The invention belongs to the field of automatic detection technology, and relates to a firework forming detection device and a detection method that integrates 2D and 3D visual perception.

Background technique

At present, my country is the world's largest producer, exporter and consumer of fireworks. Fireworks production accounts for 90% of global production and approximately 80% of world trade volume. However, fireworks production is a typical labor-intensive and high-risk manufacturing industry. During the production process, pyrotechnic powder is easily excited by unexpected energy, causing explosion accidents and causing heavy losses to people's lives and property. In recent years, with the efforts of the government and enterprises, the mechanization of the fireworks production industry has been vigorously promoted to achieve the separation of humans and drugs, effectively reducing the occurrence of vicious safety accidents involving casualties. However, there are still many problems in the fireworks production process, such as lack of online quality detection and control, poor product consistency, and low automation. There is an urgent need for safe, efficient, and reliable online detection and control technology.

The molding of fireworks beads is a link in the fireworks production process that involves a large amount of chemicals, a high risk factor, and frequent safety accidents. At present, the specific process of forming fireworks beads is to send the gunpowder powder mixed according to a specific ratio in the silo through the discharge port into a granulator that is tilted (the tilt angle is 40 to 60 degrees) and rotates at high speed around the central axis. In the machine (diameter 1~1.5 meters, rotation speed 15~30 rpm), loose gunpowder powder rotates together under the combined action of gravity, centrifugal force and friction in the inclined and rotating granulator. When it reaches a certain height, the powder The material slides down due to gravity. During this process, the spraying device intermittently sprays mist-like adhesive (mainly alcohol) onto the surface of the gunpowder powder. The gunpowder powder sticks to each other and rolls under the action of the adhesive, and gradually grows into a spherical bright bead. When the particle size of the bright beads reaches a certain requirement, the powder feeding and spraying are stopped, but the bright beads continue to roll in the granulator for a certain period of time and are polished to improve their mechanical strength and roundness. Finally, the bright beads with qualified particle size are sent to the drying process by the conveyor belt.

At present, the working condition identification and control of the fireworks bead forming process mainly rely on the observation of on-site operators, who judge the state of the beads (bead particle size, bead growth rate, surface smoothness, etc.) based on experience and manually adjust the bead process. Parameters (shotting volume, powder feeding speed, polishing time, etc.) ensure that the quality of the fireworks beads produced meets the requirements. However, this production method, which relies on manual operation and control of the fireworks bead forming process, is highly labor-intensive, subjective, error-prone, and low-efficiency. It is difficult to achieve objective evaluation and unified understanding of the working conditions, resulting in the production process of the beads being difficult to achieve. The situation fluctuates greatly, the quality and output of bright beads cannot meet production requirements, and workers are directly exposed to gunpowder production, which can easily lead to major production safety accidents due to improper operation. To sum up, there is an urgent need for an automatic detection method and device for the formation of fireworks beads to solve these problems.

Contents of the invention

In order to achieve the above objectives, the present invention provides a firework forming detection device and a detection method that integrates 2D and 3D visual perception to realize intelligent detection of the fireworks bead forming process, achieve the purpose of human and drug separation, and eliminate the production of fireworks beading potential safety hazards in the process and improve the quality of finished fireworks beads.

The technical solution adopted in the embodiment of the present invention is a firework forming detection device that integrates 2D and 3D visual perception, including: an RGBD camera, a surface light source, a light source camera adjustment seat, a detection device bracket and an industrial control computer; the RGBD camera It is fixedly installed on the light source camera adjustment seat, and the RGBD camera is located at the opening of the surface light source; the light source camera adjustment seat is installed on the detection device bracket; the industrial control computer is connected to the RGBD camera and the granulator through wires.

Another technical solution adopted in the embodiment of the present invention is a detection method using a fireworks forming detection device that integrates 2D and 3D visual perception, including the following steps:

Step S1, 2D and 3D image collection of the fireworks bead forming process: Use the RGBD camera (101) to collect images of the bright bead particles in the granulator (106) during the fireworks bead forming process and send them to the industrial control computer (105);

Step S2. Production of training samples and test samples for edge contours of bright bead particle images: use the collected 2D fireworks bright bead particle images to create training samples and test samples;

Step S3. Bright bead particle image edge contour extraction network training and testing: Use the prepared bright bead particle image training samples to train the Transformer-UNet network model. After the network model training is completed, the 2D bright bead particles in the test sample are The image is input into the network model to extract its edge contour;

Step S4. Identification and determination of occlusion of bright bead particles: On the basis of extracting the edge contour of the image of bright bead particles, use the collected 3D image information to identify and judge the occlusion between bright bead particles, and remove the blocked bright bead particles for detection. interference with results;

Step S5. Bright bead particle image segmentation and feature extraction: After edge contour extraction and occlusion recognition are performed on the bright bead particle image, the extracted edge contour information is used to segment a single bright bead particle image, and its shape is obtained from the 2D image. Feature and grayscale feature information are extracted separately;

Step S6: Based on the statistics of bright bead particle image characteristic information, determine the working conditions of the bright bead forming process and adjust parameters in a timely manner.

The beneficial effects of the present invention are: through the combination of 2D and 3D, 3D technology is introduced into the formation of fireworks beads, thereby realizing fast and intelligent detection of the fireworks beads forming process, replacing the existing manual detection method, and enabling the detection of fireworks beads. Provide quick feedback on changes in working conditions during the molding process to improve the quality of finished fireworks beads. The detection method and device can realize the separation of human and drug during the bright bead forming process in the fireworks production process, eliminate safety hazards in the production process of fireworks bright beads, and curb the occurrence of major production accidents.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of a detection device according to an embodiment of the present invention.

Figure 2 is a schematic structural diagram of the Transformer-UNet network according to the embodiment of the present invention.

Figure 3 is a schematic front view of the embodiment of the present invention.

Figure 4 is a flow chart of the detection method according to the embodiment of the present invention.

Figure 5 is a diagram showing the results of merging 2D and 3D information to identify and determine the occlusion of bright bead particles implemented in the present invention (the occlusion of the bright bead particles is the outline area).

Figure 6 is a diagram showing the comparative results of detecting the particle size distribution of bright beads using the detection method implemented in the present invention and the manual measurement method.

Figure 7 is the comparison result of detecting the particle size distribution of bright beads using only 2D image detection method and manual measurement method.

In Figure 1, 101. RGBD camera, 102. Light source camera adjustment seat, 103. Surface light source, 104. Detection device bracket, 105. Industrial control computer, 106. Granulator.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

As shown in Figure 1, the present invention provides a fireworks forming detection device that integrates 2D and 3D visual perception, including: RGBD camera 101, surface light source 103, light source camera adjustment seat 102, detection device bracket 104 and industrial control computer 105. Among them, the RGBD camera 101 collects images of the bright bead particles in the granulator 106 and sends the collected image information to the industrial control computer 105. The collected images include 2D and 3D information images of the bright bead particles; the surface light source 103 is used to improve the collection The grayscale intensity of the bright bead particle image overcomes the interference of external ambient light and ensures the stability of collecting the bright bead particle image. In addition, a hole is made in the center of the surface light source 103 and the RGBD camera 101 is installed in the hole to avoid the RGBD camera 101 is blocked so as to produce shadows in the image; the light source camera adjustment seat 102 is used to adjust the position, height and tilt angle of the light source 103 and the RGBD camera 101 to match the fireworks granulator 106 of different heights and to make the RGBD camera shooting direction consistent with The horizontal plane of the granulator 106 is vertical; the detection device bracket 104 is mainly used to fix the light source camera adjustment seat 102; the industrial control computer 105 mainly processes the collected bright bead particle image information and forms decision-making information based on the processing results, which is sent to the granulator 106 The control system makes timely adjustments to the working conditions of the bright bead forming process.

The invention also provides a detection method using a fireworks forming detection device that integrates 2D and 3D visual perception, including the following steps:

S1. 2D and 3D image collection of fireworks bright bead forming process: collect images of bright bead particles in the granulator 106 during the fireworks bright bead forming process;

S2. Production of training samples and test samples for edge contours of bright bead particle images: use the collected 2D fireworks bright bead particle images to create training samples and test samples;

S3. Bright bead particle image edge contour extraction network training and testing: Use the prepared bright bead particle image training samples to train the Transformer-UNet network model. After the network model training is completed, the 2D bright bead particle image in the test sample is By inputting it into the network model, its edge contour can be obtained;

S4. Identification and determination of bright bead particle occlusion: On the basis of extracting the edge contour of the bright bead particle image, use the collected 3D image information to identify and judge the occlusion between bright bead particles, and remove the blocked bright bead particles to improve the detection results. interference;

S5. Bright bead particle image segmentation and feature extraction: After edge contour extraction and occlusion recognition are performed on the bright bead particle image, the extracted edge contour information can be used to segment a single bright bead particle image, and its shape can be obtained from the 2D image. Feature and grayscale feature information are extracted separately;

S6. Based on the statistics of bright bead particle image characteristic information, determine the working conditions of the bright bead forming process and adjust parameters in a timely manner.

In some embodiments, in step S1, the RGBD camera 101 is used to simultaneously collect RGB images (2D) and depth images (3D) of bright bead particles, laying the foundation for the subsequent fusion of 2D and 3D visual perception information to extract the characteristics of bright bead particles. Base.

In some embodiments, in step S2, the training sample is a 2D bright bead particle edge contour image. During the production process, the bright bead particle edge contour pixels are marked with a grayscale value of 255, and the remaining pixels are marked with a grayscale value of 0. mark.

Furthermore, in the step S3, due to phenomena such as occlusion and extrusion of the bright bead particles during the formation of fireworks beads, the edges between the particles are blurred, dark, or even missing. Traditional image processing algorithms based on edge detection or threshold segmentation It is difficult to accurately extract the weak edges of such grains, and it is easy to cause over-segmentation and under-segmentation. The present invention provides an edge contour point extraction method based on a deep learning network: Transformer-UNet. The network is shown in Figure 2 and mainly consists of two parts. The dotted box on the right is the UNet network structure, and the network is the entire Transformer-UNet. The skeleton structure of the UNet network; the dotted box on the left is the feature extraction Transformer. This part has a total of 3 layers consisting of 6 Transformer layers, where each layer is composed of two Transformer layers connected in sequence.

The advantage of the Transformer-UNet network is that the skeleton structure UNet network uses a fully convolutional network instead of a fully connected layer. The network can meet the requirements of small sample training, and the training and testing processes are less time-consuming. However, because Unet cannot model the contextual relationship of distant features in the image, it can extract the global feature information in the image. Therefore, the present invention uses the powerful self-attention mechanism of Transformer to extract global features in the image, that is, the convolutional layer (Convolutional layer) and Transformer layer of the UNet network are used as Encoder to extract the local feature information and global feature information of the image respectively. Extract, and fuse the extracted local feature information and global feature information in an additive manner (Fusion) of the image features, and input them into the Decoder layer of the UNet skeleton network to restore the image feature information, and finally obtain the bright bead particles. Output image of edge contour points.

Furthermore, in step S4, based on the extraction of the edge contours of bright bead particles in S3, in view of the phenomenon that occlusion between particles causes errors in subsequent feature extraction, it is proposed to use depth images (3D) to identify and judge particle occlusion. As shown in Figure 3, in the target bright bead particles Extract a fixed number of edge points on the edge contour to edge points For example, take it as the tangent point and point to the target bright bead particle along the edge direction. The direction of the normal vector is a circle with a radius of 5 pixels. ,in Target bright bead particles and intersection area. Also with edge points is the tangent point, and points to the adjacent bright bead particles along the edge direction. The direction of the normal vector is a circle with a radius of 5 pixels. ,in for adjacent bright bead particles and intersection area. Using the collected depth image (3D) information, the depth information is converted into the height information of the bright bead particles, and the intersection areas are analyzed respectively. and Calculate the average height value of the middle pixel and ,in for The average height value of the middle pixels, for The average height value of pixels in the middle; if It means that the target bright bead particle is not blocked by the adjacent particle. By searching and calculating the edge contour points of all target bright bead particles, all of them satisfy Then it is judged that the target bright bead particles are not blocked, otherwise if This means that the target bright bead particle is blocked by the adjacent particle.

Further, in step S5, based on the edge contour extraction and occlusion recognition of the bright bead particle image in S4, the blocked bright bead particles are removed to eliminate the errors they bring to subsequent statistics, and the unoccluded bright bead particles are extracted. For its edge contour points, firstly, based on the prior knowledge that the bright bead particles are circular, perform least square circle fitting on the edge contour points, calculate their radius, and then calculate the gray level co-occurrence matrix of the connected area surrounded by the edge contour points, and finally The radius distribution values (shape features) and gray-level co-occurrence matrix averages (gray-level features) of all unobstructed bright bead particles are counted to complete the statistical extraction of image feature information.

Further, in step S6, based on the bright bead particle image characteristic information collected in S5, the bright bead particle image characteristic information is sent to the control decision-making system of the granulator 106 to determine the bright bead forming process working conditions and make timely adjustments. Parameters to make the bright beads particles meet the quality requirements.

Example 1

As shown in Figure 4, follow these steps:

S1. Preparation work before detection includes: 1. RGBD camera 101 parameter adjustment, physical and pixel coordinate calibration; 2. Adjustment of the height and angle of the surface light source 103 so that the surface light source and the surface of the bright bead particles remain parallel, and there is no gap between the bright bead particles. Generate shadows; 3. Adjust the brightness of the surface light source 103 to increase the grayscale intensity of the bright bead particle image, overcome the interference of external ambient light, and ensure the stability of the bright bead particle image collection; 4. Collect the 2D image of the bright bead particle and create edges Contour training sample images, use the training samples to train the Transformer-UNet network model, and complete the training of the edge contour extraction network.

S2. Power on the detection device.

S3, RGBD camera 101 collects 2D and 3D image information of bright bead particles.

S4. Use the trained Transformer-UNet network to extract edge contour information of the 2D bright bead particle image.

S5. Use the collected 3D bright bead particle image information combined with the 2D edge contour point information to identify the occluded bright bead particles.

S6. After removing the occluded bright bead particles, use the least squares circle fitting method to calculate the radius of the edge contour of the unobstructed bright bead particles, and calculate the gray level co-occurrence matrix surrounding the connected area.

S7. Statistics of bright bead particle image information and sends it to the granulator control decision-making system to adjust working condition parameters;

S8. End the testing and prepare to enter the next batch of testing.

Figure 5 shows the result of using the method of fusing 2D and 3D image information provided in the present invention to detect occluded bright bead particles, in which the outline area is determined to be occluded bright bead particles. Figures 6 and 7 show the comparison results of the detection method implemented in the present invention, the manual detection method, and the comparison results of the detection of bright bead particle size distribution using only the 2D image detection method. It can be seen that after using the detection method in the present invention to identify and determine the blocked bright bead particles, the obtained particle size distribution percentage results and particle size distribution cumulative percentage results are better than the measurement results obtained by only using the 2D image detection method. The results are close to the manual detection method.

The above-mentioned embodiment has the characteristics of non-contact detection, high detection accuracy, and fast detection speed. It can replace the existing manual detection method, quickly detect the fireworks bead forming process, and provide timely feedback on changes in working conditions, so as to The granulator control system adjusts and controls the parameters of the bright bead forming process to improve the quality of the finished fireworks bead products, and can realize the separation of human and drug during the fireworks bead production process, eliminate safety hazards in the fireworks bead production process, and avoid The occurrence of major safety accidents.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims (7)

1. The detection method of the firework forming detection device integrating 2D and 3D visual perception is characterized by comprising the following steps of:

step S1, acquiring 2D and 3D images in the firework bright bead forming process: an RGBD camera (101) is used for collecting bright bead particle images in a granulator (106) in the process of forming the bright beads of fireworks and sending the bright bead particle images to an industrial control computer (105);

s2, manufacturing a bright bead particle image edge profile training sample and a test sample: manufacturing a training sample and a test sample by using the collected 2D firework bright bead particle images;

step S3, extracting network training and testing edge contours of bright bead particle images: training a transducer-UNet network model by using the prepared bright bead particle image training sample, and inputting a 2D bright bead particle image in the test sample into the network model to extract the edge contour of the 2D bright bead particle image after the training of the network model is completed;

s4, bright bead particle shielding identification judgment: on the basis of extracting the edge contour of the bright bead particle image, the collected 3D image information is utilized to identify and judge the shielding existing among the bright bead particles, and the interference of the shielded bright bead particles on the detection result is removed;

step S5, bright bead particle image segmentation and feature extraction: after edge contour extraction and shielding identification judgment are carried out on the bright bead particle images, the extracted edge contour information is utilized to segment the single bright bead particle images, and shape characteristics and gray characteristic information of the single bright bead particle images are respectively extracted from the 2D images;

s6, judging the working condition of the bright bead forming process according to statistics of the bright bead particle image characteristic information, and timely adjusting parameters;

the method for identifying and judging the shielding existing between the bright bead particles in the step S4 comprises the following steps: at the target bright bead particles S ₁ Extracting a fixed number of edge points from the edge contour, taking the edge points as tangent points, and pointing to the target bright bead particles S along the edge direction ₁ Is a circle R in the normal vector direction _g Wherein R is ₁ ＝S ₁ ∩R _g To target bright bead particles S ₁ And R is R _g Is a cross region of (2); also take the edge point as the tangential point and point to the adjacent bright bead particles S along the edge direction ₂ Radius in the normal vector direction of (d) and circle R _g Circle R with same radius _s Wherein R is ₂ ＝S ₂ ∩R _s Is adjacent bright bead particles S ₂ And R is R _s Is a cross region of (2); converting depth information into height information of bright bead particles by using acquired 3D image information, and respectively aiming at the crossing region R ₁ And R is R ₂ The average height value H of the middle pixel point is calculated ₁ And H ₂ If H ₁ ≥H ₂ Then it is indicated that the target bright bead particles are not blocked by the adjacent particles, and all the edge contour points of the target bright bead particles are searched and calculated to meet H ₁ ≥H ₂ Judging that the target bright bead particles are not shielded, otherwise if H ₁ ＜H ₂ It is indicated that the target bright bead particle is occluded by the adjacent particle.

2. The method of claim 1, wherein the bright bead image in step S1 comprises a 2D image and a 3D image.

3. The method for detecting the firework forming detection device by utilizing the fusion of 2D and 3D visual perception according to claim 1, wherein the training sample in the step S2 is a 2D bright bead particle edge contour image, the bright bead particle edge contour pixel points are marked with gray values 255 in the manufacturing process, and the rest pixel points are marked with gray values 0.

4. The method for detecting the firework forming detection device by using the 2D and 3D visual perception according to claim 1, wherein the transformation-UNet network model in the step S3 comprises a transformation network and UNet network structure, UNet is a skeleton structure of the whole transformation-UNet network; the transducer network has 3 layers, and each layer is formed by sequentially connecting two transducer layers; the UNet network adopts a full convolution network to replace a full connection layer.

5. The method for detecting a firework molding detection device by using 2D and 3D visual perception according to claim 1 or 4, wherein the method for extracting the edge profile in step S3 is as follows: and the convolution layer of the UNet network is used as an Encoder to extract the local feature information of the image, the transform layer is used as the Encoder to extract the global feature information of the image, the extracted local feature information and the global feature information are fused in an adding mode, the fusion of the image features is input into the Decode layer of the UNet skeleton network, the image feature information is restored, and finally the output image of the edge contour point of the bright bead particle is obtained.

6. The method for detecting the firework molding detection device by using the 2D and 3D visual perception fusion method according to claim 1, wherein the method for extracting the shape feature and gray feature information in the step S5 is as follows: removing the blocked bright bead particles, extracting edge contour points of the non-blocked bright bead particles, firstly carrying out least square circle fitting on the edge contour points based on priori knowledge that the bright bead particles are in a circular shape, calculating the radius of the edge contour points, then calculating gray level co-occurrence matrixes of connected areas surrounded by the edge contour points, and finally counting radius distribution values and gray level co-occurrence matrix average values of all the non-blocked bright bead particles to finish the statistical extraction of image characteristic information.

7. The method for detecting the firework forming detection device by utilizing the 2D and 3D visual perception according to claim 1, wherein the step S6 is specifically: on the basis of the image characteristic information of the bright bead particles counted in the step S5, the image characteristic information of the bright bead particles is sent to a control decision system of a granulator (106) through an industrial control computer (105), and the working conditions of the bright bead forming process are judged and parameters are timely adjusted, so that the bright bead particles meet the quality requirements.