CN110059676A - A kind of aviation plug hole location recognition methods based on deep learning Yu multiple target distribution sorting - Google Patents

Abstract

The invention discloses an aviation plug hole position identification method based on deep learning and multi-target distribution sorting, comprising the following steps: starting a camera and initializing, capturing an aviation plug image, training a plug positioning network model to locate the aviation plug target in the industrial scene image , judge whether the candidate target meets the size constraints, crop the plug hole area from the image, train the hole position recognition network model to identify the hole position and error-proof pin array in the plug hole area, and number the aviation plug hole position array according to the multi-object sorting principle , determine the installation status of the air plug hole position, match the database and output the identification result. The aviation plug hole position identification method based on deep learning and multi-object distribution sorting can visually identify the aviation plug and the plug hole position by taking the image of the aviation plug, sort and number the plug hole position, and determine the installation result of the aviation plug, which saves the cost of checking the aviation plug. The labor cost of the plug hole installation state is high and the efficiency is high.

Description

A kind of aviation plug hole position recognition based on deep learning and multi-objective distribution sorting method

technical field

The invention relates to the field of aircraft manufacturing intelligent assembly, in particular to an aviation plug hole position identification method based on deep learning and multi-objective distribution sorting.

Background technique

In aircraft manufacturing, aircraft assembly cost and assembly workload account for about half of the entire aircraft cost and workload, and wire plug assembly occupies a large proportion in aircraft assembly. Aviation plugs are mainly used to connect electrical communication between different devices. There are many types and quantities of aviation plugs on the plane. Each type of aviation plug has its own independent installation status. The wires are installed from the back of the aviation plug to the hole of the aviation plug. , It is necessary to verify the installation result of the aviation plug. This process relies heavily on labor. The worker needs to distinguish the hole position of the installed wire on the front of the aviation plug and the hole position of the uninstalled wire, and search the installation status results stored in the database for the plug. By comparing whether the installation results are consistent, it is judged whether there are holes for missing wires in the plug, or holes for wrong wires. This process is cumbersome and inefficient, and it is difficult to adapt to the rapid production needs of aircraft batches.

SUMMARY OF THE INVENTION

The invention provides an air plug hole position identification method based on deep learning and multi-target distribution sorting.

In order to achieve the above object, the present invention has adopted the following workflow:

S1. Start the camera and initialize it, set the focus mode of the camera, and adjust the brightness of the camera flash.

S2. After the camera is focused, the front image of the aviation plug is captured in the on-site industrial scene.

S3. Send the captured image of the aviation plug into the trained plug positioning network model, and output the category of the aviation plug and the plug coordinates in the image coordinate system as the candidate target of the aviation plug.

S4. Determine whether the candidate target of the aviation plug satisfies the size constraint. If not, repeat steps S2 and S3 to recapture the image of the aviation plug.

S5. If the candidate target of the aviation plug satisfies the size constraints in the S4, crop the plug hole area from the image according to the plug coordinates output by the plug positioning network model in the S3.

S6. Send the plug hole position area into the trained hole position identification network model, output the plug hole position coordinates and the error-proof pin coordinates in the plug hole position area, and complete the connection between the plug hole position and the error-proof pin in the plug hole position area. Class recognition and location detection.

S7. According to the coordinates of the plug hole positions and the coordinates of the error-proofing pins, perform array multi-object sorting on the plug hole positions and the error-proofing pins, and number the plug hole positions.

S8. Determine the installation state of each plug hole position, determine whether a wire is installed in the plug hole position, and then know the installation result of the entire aviation plug.

S9. Match the installation result of the aviation plug with the installation result in the database, output the installation inspection result of the aviation plug, and determine whether the installation result of the aviation plug is consistent with the database.

Preferably, the plug positioning network model training method in the step S3 includes:

S31. Collect a large number of frontal images of different types of aviation plugs in the scene, mark the type of aviation plugs in each image and the plug coordinates in the image coordinate system, and create an image data set D1 of aviation plugs.

S32 , performing an image enhancement operation on each image in the aviation plug image data set D1 to obtain an enhanced aviation plug image data set D2.

S33 , constructing a plug positioning deep neural network, establishing a training target of the plug positioning network model, returning the plug coordinates in the air plug image, and classifying the air plug type in the image.

S34, use the aviation plug image data set D1 and the enhanced aviation plug image data set D2 to train a plug positioning neural network, and fit the parameters of the network model.

S35. After the training is completed, save the plug positioning network model.

Preferably, the method for training the hole position identification network model in the step S6 includes:

S61. Cut out the aviation plug in each image as a new image according to the plug coordinates marked in the aviation plug image data set D1, create a plug hole area data set D3, and mark each image in the plug hole area data set D3 The coordinates of the plug hole position and the error-proof pin coordinates in , and mark the type of the plug hole position and the type of the error-proof pin.

S62: Perform an image enhancement operation on each image in the plug hole region data set D3 to obtain an enhanced hole region data set D4.

S63 , constructing a deep neural network for hole position identification, establishing a training target of the hole position identification network model, returning the plug hole position coordinates and error-proofing pin coordinates in the image of the plug hole position area, and classifying the plug hole positions and the types of error-proofing pins in the image .

S64, using the plug hole location area data set D3 and the enhanced hole location area data set D4 to train a hole location identification neural network, and fit the parameters of the network model.

S65. After the training is completed, save the hole position identification network model.

Further, the image enhancement operations in S32 and S62 include five operations of image white balance, image color transformation, image geometric scaling, image rotation transformation, and image random noise.

Further, the plug positioning deep neural network in the S33 and the hole position recognition deep neural network in S63 include a feature extraction network, a feature fusion network and an output network, and the feature extraction network includes four layers of convolution layers and pooling layers. The feature fusion network includes three upsampling layers and cascade layers, the output network predicts the category and position coordinates of the detection target, and a 1×1 convolution filter is connected between the feature extraction network and the feature fusion network. A 3×3 convolution filter is connected between the feature fusion network and the output network.

Preferably, the size constraint in the step S4 is that the aspect ratio of the aviation plug candidate target in the S3 is 0.6-1.4, and the length of the aviation plug candidate target in the S3 is greater than the length of the aviation plug image in the S3. quarter.

Preferably, the array multi-target sorting method in step S7 includes:

S71. According to the coordinates of the error-proofing pins outputted in the S6 Calculate the geometric center point of the plug hole area Take this point as the pole and lead to the coordinates of the error proofing pin with the largest aspect ratio The ray establishes the level coordinate system for the polar axis.

S72: Calculate the polar angle of the coordinates P _i =(x _i , y _i ) of each plug hole position output in the S6 in the polar coordinate system and polar diameter

S73, according to the distance between the pole diameter γ _i and the pole, divide the plug hole position coordinates into M sections from the outside to the inside, and the plug hole position coordinates on the jth section are Polar angle from outside to inside according to the coordinates of plug holes on each layer segment The size of the plug holes is sorted, and the plug holes in the plug hole area are numbered in this order.

Preferably, in the step S8, it is determined whether or not a wire is installed in the hole according to the pixel color of the hole of the plug.

The advantage of the invention is that the camera takes an image of the aviation plug, visually identifies the aviation plug and the plug hole position in the image, sorts and numbers the detected plug hole position, judges the installation state of the plug hole position, and knows The installation result of the aviation plug replaces the manual inspection of the installation result of the aviation plug, which saves labor costs and has high efficiency; the plug hole size of the aviation plug is small, and it is difficult to directly identify the plug hole position in the aviation plug image. First, train the plug positioning network model. Detect the plug hole area from the image, crop the plug hole area from the image to form a new image, and then train the hole position recognition network model to identify the plug hole position from the plug hole area. Detecting aviation plugs and plug hole positions in the background image improves the accuracy of the recognition results; the four-layer convolutional layer and pooling layer of the feature extraction network in the plug positioning deep neural network and hole position recognition deep neural network can effectively extract aviation The feature fusion network can fuse the extracted features of different scales in the image, which improves the performance of the network; the size constraints of the candidate target of the aviation plug can filter the noise target in the image and ensure that the predicted candidate target is The aviation plug improves the reliability of the positioning of the aviation plug; combined with the array features of the error-proof pins and the plug hole positions, the multi-objective sorting and numbering of the plug hole positions in the polar coordinate system is simple and easy to implement and has strong adaptability; The hole position of the plug is brass, which is obviously different from the hole position of the plug without the wire. The pixel color is used to determine whether the wire is installed in the plug hole, which is efficient and accurate.

Description of drawings

Fig. 1 is the air plug hole position identification flow chart of the present invention;

2 is a schematic structural diagram of a typical aviation plug of the present invention;

3 is a schematic diagram of the distribution of holes of a typical aviation plug of the present invention;

4 is a schematic diagram of the identification process of the air plug hole position of the present invention;

5 is a schematic structural diagram of a deep neural network of the present invention;

6 is a schematic diagram of a method for sorting the hole positions of a single-layer aviation plug according to the present invention;

FIG. 7 is a schematic diagram of the method for numbering holes of multi-layer aviation plugs according to the present invention.

In Figure 2, 1. the hole position of the plug, 2. the front of the plug, and 3. the error-proof pin.

In Fig. 4, 31, the coordinates of the plug, 61, the coordinates of the error-proof pin, and 62, the coordinates of the hole position of the plug.

The shape and structure of a typical aviation plug can be seen from Figure 2.

Figure 3 shows the arrangement of the plug holes and error-proof pins of a typical aviation plug.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments.

In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inside", " The orientation or positional relationship indicated by "outside" is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must have a specific orientation, so as to The specific orientation configuration and operation are therefore not to be construed as limitations of the present invention.

Example 1:

Referring to Figure 1-2, an aviation plug hole position identification method based on deep learning and multi-target distribution sorting adopts the following workflow:

S1. Start the camera and initialize it, set the focus mode of the camera, and adjust the brightness of the camera flash.

S2. After the camera is focused, refer to Figure 3 to capture an image of the front 2 of the aviation plug in an industrial scene.

S3. Send the captured air plug image into the trained plug positioning network model. Referring to FIG. 4, output the type of the air plug and the plug coordinates 31 in the image coordinate system as the candidate target of the air plug.

S4. Determine whether the candidate target of the aviation plug satisfies the size constraint. If not, repeat steps S2 and S3 to recapture the image of the aviation plug.

S5. If the candidate target of the aviation plug satisfies the size constraints in the S4, referring to FIG. 4, crop the plug hole area from the image according to the plug coordinates 31 output by the plug positioning network model in the S3.

S6. Send the plug hole position area into the trained hole position identification network model, referring to FIG. 4 , output the plug hole position coordinates 62 and the error-proof pin coordinates 61 in the plug hole position area, and complete the plug hole position in the plug hole position area. Class recognition and position detection for bit 1 and error proofing pin 3.

S7. According to the plug hole position coordinates 62 and the error-proof pin coordinates 61, referring to FIG. 6-7, perform array multi-object sorting on the plug hole position 1 and the error-proof pin 3, and number the plug hole position 1.

S8. Determine the installation state of each plug hole 1, determine whether a wire is installed in the plug hole 1, and then know the installation result of the entire aviation plug.

S9. Match the installation result of the aviation plug with the installation result in the database, output the installation inspection result of the aviation plug, and determine whether the installation result of the aviation plug is consistent with the database.

In order to solve the problem that it is difficult to directly identify the plug hole position 1 from the aviation plug image, refer to Figure 1, respectively train the plug positioning network model and the hole position identification network model, refer to Figure 4, first use the plug positioning network model to detect the aviation plug in the image. According to the plug coordinate 31, the plug hole area is cut out from the image and sent to the hole identification network model according to the plug coordinate 31, and the hole position coordinate 62 and the error-proof pin coordinate 61 are detected. The training method of the plug positioning network model includes collecting a large number of images of the front face 2 of aviation plugs of different types in a live scene, labeling the aviation plug category in each image and the plug coordinates 31 in the image coordinate system, and making the aviation plug. Image data set D1; image enhancement operation is performed on each image in the aviation plug image data set D1 to obtain an enhanced aviation plug image data set; a plug positioning deep neural network is constructed, a training target of the plug positioning network model is established, and the aviation plug image is returned The plug coordinate 31 in the classification image is used to classify the aviation plug type in the image; use the aviation plug image data set D1 and the enhanced aviation plug image data set D2 to train the plug positioning neural network, and fit the parameters of the network model; after the training is completed, save the plug positioning network. Model. The method for training a network model for hole position identification includes: cropping the aviation plug in each image as a new image according to the plug coordinates 31 marked in the aviation plug image data set D1, making a plug hole position area data set D3, and labeling the plug. The plug hole position coordinates 62 and the error-proofing pin coordinates 61 in each image in the hole position area dataset D3 are marked with the category of the plug hole position 1 and the category of the error-proof pin 3; for each image in the plug hole position area data set D3 Perform image enhancement operations on the first image to obtain the enhanced hole location data set D4; build a hole location recognition deep neural network, establish the training target of the hole location recognition network model, and return the plug hole location coordinates 62 and error-proof pins in the image of the plug hole location area. Coordinate 61, classify the types of plug hole position 1 and error-proof pin 3 in the image; use the plug hole position area data set D3 and the enhanced hole position area data set D4 to train the hole position recognition neural network, and fit the parameters of the network model; training After completion, save the hole location recognition network model. The image enhancement operations include five operations: image white balance, image color transformation, image geometric scaling, image rotation transformation, and image random noise. Referring to FIG. 5 , the deep neural network for plug positioning and the deep neural network for hole position recognition include a feature extraction network, a feature fusion network and an output network, and the feature extraction network includes four convolution layers and a pooling layer. The network includes three layers of upsampling layers and cascade layers. The output network predicts the category and position coordinates of the detection target. A 1×1 convolution filter is connected between the feature extraction network and the feature fusion network. A 3×3 convolution filter is connected between the feature fusion network and the output network.

The size constraint in the step S4 is that the aspect ratio of the aviation plug candidate target in the S3 is 0.6-1.4, and the length of the aviation plug candidate target in the S3 is greater than one quarter of the length of the aviation plug image in the S3. one.

In order to solve the multi-objective sorting problem of the array of the plug hole position 1, referring to FIG. 6, according to the coordinates of the error-proof pin 3 output in the S6 Calculate the geometric center point of the plug hole area Taking this point as the pole, lead to the coordinates of the error-proof pin 3 with the largest aspect ratio The ray establishes a level coordinate system for the polar axis. Calculate the polar angle of the coordinates P _i =(x _i , y _i ) of each plug hole position 1 output in the S6 in the polar coordinate system and polar diameter Referring to FIG. 7 , according to the distance of the pole diameter γ _i from the pole, the plug hole position coordinates 62 are divided into M sections from the outside to the inside, and the plug hole position coordinates 62 on the jth section are: Polar angle according to plug hole position coordinates 62 on each layer segment from outside to inside The size of the plug hole 1 is sorted, and the plug hole 1 in the plug hole area is numbered in this order.

The plug hole position 1 with the wire installed is brass, and the color difference is obvious from that of the plug hole position 1 without the wire. In the step S8, it is determined whether the wire is installed in the hole position according to the pixel color of the plug hole position 1, and the operation is simple. Convenient and worthy of widespread promotion.

The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited to this. The equivalent replacement or change of the inventive concept thereof shall be included within the protection scope of the present invention.

Claims (7)

1. a kind of aviation plug hole location recognition methods based on deep learning Yu multiple target distribution sorting, which is characterized in that the party Method includes:

S1, starting camera simultaneously initialize, and the focal modes of camera are arranged, adjust the brightness of camera flash-light；

After the completion of S2, camera focusing, aviation plug direct picture is captured under industrial scene at the scene；

S3, the aviation plug image of capture is sent into the plug positioning network model that training is completed, exports the classification of aviation plug And the plug coordinate under image coordinate system, as aviation plug candidate target；

S4, judge whether aviation plug candidate target meets size constraint, if not satisfied, repeating step S2 and S3, again Capture aviation plug image；

If S5, aviation plug candidate target meet the size constraint in the S4, network is positioned according to plug in the S3 The plug coordinate of model output cuts plug hole location region from image；

S6, plug hole location region is sent into the hole location identification network model that training is completed, the plug in output plug hole location region Hole location coordinate and mistake proofing pin coordinate complete classification identification and the position of the plug hole location and mistake proofing pin in plug hole location region Detection；

S7, according to plug hole location coordinate and mistake proofing pin coordinate, array multicriterion scheduling is carried out to plug hole location and mistake proofing pin, Plug hole location is numbered；

S8, the installation condition for determining each plug hole location, judge in the plug hole location whether conducting wire, and then learn entire boat The installation results of blind plug；

S9, by the installation results in the installation results matching database of aviation plug, export aviation plug and inspection result be installed, sentence Whether other aviation plug installation results are consistent with database.

2. a kind of aviation plug hole location identification side based on deep learning Yu multiple target distribution sorting as described in claim 1 Method, which is characterized in that the plug in the S3 step positions network model training method and includes:

S31, a large amount of different types of aviation plug direct pictures are acquired in scene at the scene, the aviation marked in every image is inserted Head classification and the plug coordinate under image coordinate system, make aviation plug image data set D1；

S32, image enhancement operation is carried out to every image in aviation plug image data set D1, obtains enhancing aviation plug figure As data set D2；

S33, building plug emplacement depth neural network, establish the training objective of plug positioning network model, return aviation plug Plug coordinate in image, classify image in aviation plug type；

S34, nerve net is positioned using aviation plug image data set D1 and enhancing aviation plug image data set D2 training plug Network is fitted the parameter of network model；

After the completion of S35, training, saves plug and position network model.

3. a kind of aviation plug hole location identification side based on deep learning Yu multiple target distribution sorting as described in claim 1 Method, which is characterized in that the plug in the S6 step positions network model training method and includes:

S61, will be under aviation plug in every image cuts according to the plug coordinate marked in aviation plug image data set D1 As new image, plug hole location area data collection D3 is made, is marked in plug hole location area data collection D3 in every image Plug hole location coordinate and mistake proofing pin coordinate, mark plug hole location classification and mistake proofing pin classification；

S62, image enhancement operation is carried out to every image in plug hole location area data collection D3, obtains enhancing hole location number of regions According to collection D4；

S63, building hole location identify deep neural network, establish the training objective of hole location identification network model, return plug hole location Plug hole location coordinate and mistake proofing pin coordinate in area image, classify image in plug hole location and mistake proofing pin type；

S64, neural network is identified using plug hole location area data collection D3 and enhancing hole location area data collection D4 training hole location, intend Close the parameter of network model；

After the completion of S65, training, saves hole location and identify network model.

4. such as claim 2 and a kind of aviation plug based on deep learning Yu multiple target distribution sorting as claimed in claim 3 Hole location recognition methods, it is characterised in that: the image enhancement operation in the S32 and S62 includes image white balance, color of image change It changes, image geometry is flexible, image rotation converts and five kinds of image random noise operations, plug emplacement depth nerve in the S33 Hole location identification deep neural network includes feature extraction network, Fusion Features network and output network, the spy in network and S63 It includes four layers of convolutional layer and pond layer that sign, which extracts network, and the Fusion Features network includes three layers of up-sampling layer and cascading layers, institute The classification and position coordinates for stating output neural network forecast detection target, between the feature extraction network and the Fusion Features network It is connected with 1 × 1 Convolution Filter, 3 × 3 convolution filter is connected between the Fusion Features network and the output network Device.

5. a kind of aviation plug hole location identification side based on deep learning Yu multiple target distribution sorting as described in claim 1 Method, it is characterised in that: the size constraint in the S4 step is that the length-width ratio of aviation plug candidate target in the S3 exists The length of aviation plug candidate target is greater than a quarter of aviation plug image length in the S3 in 0.6-1.4, the S3.

6. a kind of aviation plug hole location identification side based on deep learning Yu multiple target distribution sorting as described in claim 1 Method, which is characterized in that the array multicriterion scheduling method in the S7 step includes:

S71, according to the coordinate of the mistake proofing pin exported in the S6In the geometry for calculating plug hole location region Heart pointUsing the point as pole, the maximum mistake proofing pin of length-width ratio is guided into The coordinate of nailRay be polar axis establish grade coordinate system；

S72, the coordinate P for calculating each plug hole location exported in the S6_i=(x_i, y_i) polar angle in polar coordinate systemAnd polar diameter

S73, according to polar diameter γ_iPlug hole location coordinate is divided into M section by distance apart from pole from the outside to the core, on j-th of section Plug hole location coordinate be P_i ^J, from the outside to the core according to the polar angle of the plug hole location coordinate on each layer of sectionSize is to plug Hole location is ranked up, and the plug hole location in plug hole location region is numbered in sequence according to this.

7. a kind of aviation plug hole location identification side based on deep learning Yu multiple target distribution sorting as described in claim 1 Method, it is characterised in that: in the S8 step according to the pixel color of plug hole location differentiate hole location in whether conducting wire.