CN114693939B - Method for extracting depth features of transparent object detection under complex environment - Google Patents

Abstract

The invention discloses a method for extracting transparent object detection depth features in a complex environment, which comprises the steps of obtaining preliminary features through a composite backbone network, processing the preliminary features by utilizing a receptive field enhancement feature pyramid module, carrying out receptive field enhancement processing on a first pyramid stage feature map by fusing contextual feature information, and fusing the enhanced features with a second pyramid stage feature map, so that the depth features of transparent objects such as glassware and the like which are relatively robust in the complex environment can be extracted, and the effect of transparent object detection algorithms such as glassware and the like can be improved.

Description

A deep feature extraction method for transparent object detection in complex environments

Technical Field

The present invention belongs to the field of image depth feature extraction, and in particular relates to a method for extracting depth features of transparent object detection in a complex environment.

Background technique

The transparent object detection algorithm can be applied to a variety of real-life scenarios. For example, a food delivery robot in a hotel needs to detect glass doors on its route to avoid collisions, and a home robot needs to identify wine bottles, glasses and other fragile transparent glass products. Therefore, the transparent object detection algorithm has high practical value.

Transparent objects widely exist in the real world, such as windows, cups, wine bottles, etc. And the detection of transparent objects can be applied to a variety of practical scenarios. For example, robots used in factory logistics need to identify glass doors on the route to avoid collisions, and robots used in smart homes need to identify transparent and fragile objects such as glasses and wine bottles. In addition, transparent object detection has a wide range of application scenarios in glassware factories, chemical laboratories, and other scenes with a large number of transparent objects.

The target detection task is a classic computer vision task, which aims to identify the target category in the image and locate the position of the target. It has been widely used in many fields such as robot navigation, autonomous driving, industrial inspection, etc., and has very important practical value. In recent years, the target detection algorithm based on deep learning has made great improvements in detection speed and detection accuracy. However, compared with non-transparent targets, transparent objects such as glassware are more susceptible to factors such as background and lighting, and the background of actual application scenarios is often relatively complex, and the ambient lighting is variable and not fixed. At present, most of the existing target detection algorithms are designed for non-transparent general targets such as vehicles and pedestrians. General target detection algorithms are more likely to have problems such as false detection and missed detection for transparent targets, which limits the application of target detection algorithms in scenes with a large number of transparent objects. At present, the feature extraction methods of target detection algorithms based on deep learning are basically designed for general targets. For transparent targets such as glassware that are easily disturbed by the background, such methods often lack targeted design, resulting in the extracted features not being able to well express the transparent target information, thereby affecting the effect of target detection algorithms.

Glossary:

concat feature map group fusion operation: It is a very important operation in network structure design, which is used to combine features, fuse features extracted by multiple convolutional feature extraction frameworks, or fuse the information of the output layer.

Summary of the invention

In order to solve the problem that the features extracted by the current mainstream deep learning-based target detection network feature extraction network cannot well extract transparent target features, the present invention provides a deep feature extraction method for transparent object detection in a complex environment.

To achieve the above object, the technical solution of the present invention is:

A method for extracting depth features of transparent object detection in a complex environment comprises the following steps:

Step 1: Extract initial features using a composite backbone network;

S1.1: The composite backbone network includes a first sub-network and a second sub-network;

S1.2: The first subnetwork includes N layers connected in series, and the second subnetwork includes N layers connected in series; the output features of the i-th layer of the first subnetwork are input to the i+1-th layer of the first subnetwork, and the i+1-th layer of the first subnetwork extracts features from the output features of the i-th layer of the first subnetwork and outputs the output features of the i+1-th layer of the first subnetwork; the output feature maps of the i-th layer, the i+1-th layer...the N-th layer of the first subnetwork are all subjected to 1x1 convolution to unify the number of channels, and enter the i-1-th first feature fusion module after being uniformly sized through nearest neighbor upsampling to obtain an output auxiliary feature map; the auxiliary feature map and the output of the i-1-th layer of the second subnetwork are added point by point as the input of the i-th layer of the second subnetwork, and the i-th layer of the second subnetwork extracts features from the input of the i-th layer of the second subnetwork to obtain the preliminary features of the i-th layer output of the second subnetwork; the set of preliminary features of the outputs of the last m layers of the second subnetwork is the initial features, and the initial features include m preliminary features, 0<m<n; i＞1;

Step 2: The receptive field enhancement feature pyramid module processes the initial features to obtain the output features of the second feature fusion module;

S2.1: Process the initial features into the first stage of the pyramid and output the first pyramid stage feature map;

S2.2: The feature map of the first pyramid stage is processed by receptive field enhancement to obtain the receptive field enhancement feature of the first pyramid stage:

Among them, F _out2 represents the receptive field enhancement feature of the first pyramid stage, ε _n represents the nth type of dilated convolution with different dilation rates, Represents the concat feature map group fusion operation of two feature maps in the feature map channel dimension, _Fin represents the first pyramid stage feature map input to the receptive field enhancement module, and σ() represents a 1×1 convolution;

S2.3: Process the first pyramid stage feature map into the second pyramid stage, and output the second pyramid stage feature map;

S2.4: Input the first pyramid stage receptive field enhancement feature and the second pyramid stage feature map together into the second feature fusion module to obtain the output feature of the second feature fusion module:

F _out = weight*F ₁ + (1-weight)*F ₂

Among them, weight represents the number of learned weights, _F1 represents the feature map of the second pyramid stage, and _F2 represents the receptive field enhancement feature of the first pyramid stage.

As a further improvement, the preliminary features of the second sub-network are:

in, is the preliminary feature of the lth level in the second sub-network,/> represents the preliminary features of the output of the l-1th level in the second sub-network, n represents the number of levels of each sub-network in the composite backbone network, l represents the level of the current preliminary features, /> represents the output feature map of the i-th level of the first subnetwork input to the second subnetwork, and ε represents the feature fusion operation.

As a further improvement, the first stage of pyramid processing includes the following steps:

A1: All preliminary features in the obtained initial features are subjected to 1x1 convolution respectively to unify the number of channels and dimensions to obtain feature maps of unified dimensions;

A2: The feature maps of unified dimension are arranged in order from deep layers to shallow layers. The deepest layer m-th layer feature map is upsampled by nearest neighbor to obtain the m-th layer upsampled feature map. The size of the m-th layer upsampled feature map is the same as that of the m-1-th layer feature map. The m-th layer upsampled feature map and the m-1-th layer feature map are added point by point to obtain a new feature map in the upsampling process. The new feature map in the upsampling process is retained as the new feature map of the m-1-th layer. Then, the new feature map of the m-1-th layer in the upsampling process is repeatedly upsampled, layer by layer, until all the feature maps of unified dimension are processed.

A3: All new feature maps obtained in the upsampling process in step A2 and the feature map of the mth level are subjected to 3×3 convolution to reduce the aliasing effect, and the feature map of the first pyramid stage is output.

As a further improvement, the second stage of pyramid processing includes the following steps:

B1: The feature maps of the first pyramid stage are arranged from shallow to deep levels, and the feature map of the shallowest level 1 is downsampled to obtain the downsampled feature map of the first level. The size of the downsampled feature map of the first level is the same as the size of the feature map of the second level of the adjacent next deep level. The downsampled feature map of the first level is added point by point to the feature map of the second level to obtain a new feature map in the downsampling process. The new feature map in the downsampling process is retained as the new feature map of the second level;

B2: Continue to downsample the new feature map of the second level, and operate layer by layer until all the feature maps of the first pyramid stage are processed;

B3: Perform 3×3 convolution on the new feature map from all downsampling processes in step B2 and the first-level feature map to output the feature map of the second pyramid stage.

As a further improvement, the ε feature fusion operation is:

ε←upsample(f(F _a ))

Among them, upsample() is the nearest neighbor upsampling algorithm, f() represents 1x1 convolution, and _Fa represents the output feature map of the first sub-network.

As a further improvement, both the first sub-network and the second sub-network are Res2Net101 networks pre-trained on the ImageNet dataset.

As a further improvement, the second feature fusion module is an AM attention module.

Further improvement, the weight parameter weight is:

weight = sigmoid(σ(F))

Among them, σ() is a 1x1 convolution, sigmoid() represents the activation function, and F represents the feature map output by the receptive field enhancement module.

In a further improvement, the N is 5.

As a further improvement, the feature extraction includes ResNet feature extraction and Res2Net feature extraction.

Advantages of the present invention:

1. By fusing contextual feature information, the feature map of the first pyramid stage is enhanced using the receptive field and the enhanced features are fused with the feature map of the second pyramid stage. This can extract the deep features of transparent targets such as glassware in complex environments that are more robust, which is beneficial to improving the effect of the detection algorithm for transparent targets such as glassware.

2. The receptive field of the preliminary features is enhanced through the receptive field enhancement module, so that the extracted features can better express the global features of transparent targets rather than local features, and are fused with the output features of the second-stage feature pyramid through the attention feature fusion module to reduce the interference of background noise, which is conducive to improving the effect of the detection algorithm for transparent targets such as glassware.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of a composite backbone network;

FIG2 is a schematic diagram of the feature fusion process of the first sub-network in the composite backbone network;

FIG3 is a schematic diagram of a feature pyramid based on shallow feature enhancement and receptive field enhancement;

FIG4 is a schematic diagram of a receptive field enhancement module;

FIG5 is a schematic diagram of a feature fusion module based on an attention mechanism;

FIG6 is a schematic diagram showing the comparison of input and output before and after feature extraction.

in:

The first layer of the first subnetwork 1, the second layer of the first subnetwork 2, the third layer of the first subnetwork 3, the fourth layer of the first subnetwork 4, the fifth layer of the first subnetwork 5, the first layer of the second subnetwork 6, the second layer of the second subnetwork 7, the third layer of the second subnetwork 8, the fourth layer of the second subnetwork 9, the fifth layer of the second subnetwork 10, the first feature fusion module a11, the second feature fusion module b12, the third feature fusion module c13, the fourth feature fusion module d14, the input image 15, the output feature map A of the third layer of the first subnetwork, the output feature map B of the fourth layer of the first subnetwork, the output feature map of the fifth layer of the first subnetwork Figure C, feature map A1 of the third layer of the first subnetwork after 1x1 convolution and nearest neighbor upsampling, feature map B1 of the fourth layer of the first subnetwork after 1x1 convolution and nearest neighbor upsampling, feature map C1 of the fifth layer of the first subnetwork after 1x1 convolution and nearest neighbor upsampling, auxiliary feature map H, feature map 16 output by the receptive field enhancement module, feature map 17 output by the second stage feature pyramid, weight matrix 18 learned by the network for the feature map of the receptive field enhancement module, feature map 19 output by the weighted receptive field enhancement module, feature map 20 output by the weighted second stage feature pyramid, output feature 21 of the second feature fusion module.

Detailed ways

The present invention is further described below in conjunction with the accompanying drawings and embodiments.

Example 1

As shown in Figures 1 to 4, a schematic diagram of a method for extracting depth features of transparent object detection in a complex environment, the specific implementation method is as follows

First: Composite backbone network

Two Res2Net101 networks pre-trained on the ImageNet dataset were selected as sub-networks of the composite backbone network. One was used as the assistant backbone of the first sub-network, and the other was used as the lead backbone of the second sub-network. The last fully connected layer was removed from both, and the pre-trained weight stem part was removed from the lead backbone of the second sub-network, which included a convolutional layer and a maximum pooling layer. During the training process, the first two layers of the assistant backbone of the first sub-network and the lead backbone of the second sub-network were frozen, and the parameters of the frozen parts did not participate in the training process.

Each level output feature map of the first sub-network assistant backbone is transmitted to the second sub-network lead backbone. In each feature map with a lower level than the first sub-network assistant backbone feature map, the output feature map first passes through 1x1 convolution to unify the number of channels and then passes through nearest neighbor upsampling to unify the size. Then all feature maps are added point by point to obtain the auxiliary feature map of the first sub-network assistant backbone input to the second sub-network lead backbone. Finally, the auxiliary feature map and the feature map of the current level of the second sub-network lead backbone are added point by point again to complete the feature fusion of one level. For features of other levels, the processing method is exactly the same, and the only difference is the number of sub-feature maps of the fused auxiliary feature map.

Preliminary characteristics of the mth level in the second sub-network of the composite backbone network have:

in, represents the preliminary features of the output of the m-1th level in the second sub-network, n represents the number of levels of each sub-network in the composite backbone network, m represents the level of the current preliminary features, /> represents the output feature map of the first subnetwork at the i-th level input to the second subnetwork, and ε represents the feature fusion operation. For the feature fusion operation, there are:

ε←upsample(f(F _a ))

Where upsample() is the nearest neighbor upsampling algorithm, and f() represents 1x1 convolution.

Second: Feature pyramid based on shallow feature enhancement and receptive field enhancement

The module consists of three parts: the first-stage feature pyramid from bottom to top, the second-stage feature pyramid from top to bottom, and the receptive field enhancement module and the feature fusion module based on the attention mechanism, as shown in Figure 3, where RFM is the receptive field enhancement module, and its detailed structure is shown in Figure 4, and AM is the feature fusion module based on attention, and its detailed structure is shown in Figure 5. Upsampling refers to converting a small image into a large image, and downsampling refers to converting a large image into a small image.

After obtaining the feature maps of each stage of the composite backbone network, each feature map first uses 1x1 convolution to unify the number of channels of the feature map and reduce the dimension of the high-dimensional semantic feature map. After unifying the dimension of the feature map, the last layer of feature map is upsampled to the size of the feature map of the previous layer, and the upsampled feature map is added point by point to the feature map of the previous layer to obtain a new feature map. The operation is then repeated for the new feature map, layer by layer until all feature maps are processed. Finally, all fused feature maps are subjected to 3x3 convolution to reduce the aliasing effect. This step can be used to fuse the feature maps of different levels in the composite backbone network. The new feature map obtained has richer information than the original feature map. This feature map is the result of the first stage feature pyramid.

The output of the first-stage feature pyramid starts from the shallowest feature, is downsampled and added point by point with the feature map of the next layer to obtain a new feature map, and the above process is repeated until all feature maps are processed. Finally, all the fused feature maps are subjected to 3x3 convolution to obtain the output features of the second-stage feature pyramid.

For the features of each level extracted by the composite backbone network, after the 1x1 convolution in the first stage feature pyramid, the number of channels of each feature map remains consistent. Each feature map is passed through the receptive field enhancement module and input to the features after the receptive field enhancement module:

Among them, F _out2 represents the receptive field enhancement feature of the first pyramid stage, ε _n represents the nth type of dilated convolution with different dilation rates, It represents the concat feature map group fusion operation of two feature maps in the feature map channel dimension, _Fin represents the first pyramid stage feature map input to the receptive field enhancement module, σ() represents 1×1 convolution, the number of feature map channels input to σ convolution is n*channels( _Fin ), and the output feature map channel is the number of _Fin channels. After obtaining the receptive field enhancement feature, the feature is input into the feature fusion module together with the second stage feature pyramid. The output features of the second feature fusion module are:

F _out = weight*F ₁ + (1-weight)*F ₂

Where weight is the learned weight parameter, _F1 and _F2 represent the output features of the second-stage feature pyramid and the original features after the receptive field enhancement, respectively. For the weight parameter weight, we have:

weight = sigmoid(σ(F))

Among them, σ() is a 1x1 convolution, and its output channel number is 1, which can reduce the dimension of the feature map of size cxhxw to hxw. Finally, all eigenvalues are normalized to the interval [0,1] through the sigmoid() activation function, and finally a matrix of size hxw with all values in the interval [0,1] is obtained. The value of each point on the matrix is the weight of the corresponding feature point in the final feature fusion process.

Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the embodiments. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and shown here.

Claims (10)

1. A method for extracting depth features of transparent object detection in a complex environment, characterized by comprising the following steps: Step 1: Extract initial features using a composite backbone network; S1.1: The composite backbone network includes a first sub-network and a second sub-network; S1.2: The first sub-network includes N layers connected in series, and the second sub-network includes N layers connected in series; the output features of the i-th layer of the first sub-network are input to the i+1-th layer of the first sub-network, and the i+1-th layer of the first sub-network extracts the output features of the i-th layer of the first sub-network and outputs the output feature map of the i+1-th layer of the first sub-network; the output feature maps of the i-th layer, the i+1-th layer...the N-th layer of the first sub-network are all convolved with 1x1 to unify the number of channels, and are unified by nearest neighbor upsampling. After the size is increased, it enters the i-1th first feature fusion module for feature fusion operation to obtain an output auxiliary feature map; the auxiliary feature map and the output of the i-1th layer of the second sub-network are added point by point as the input of the i-th layer of the second sub-network, and the i-th layer of the second sub-network performs feature extraction on the input of the i-th layer of the second sub-network to obtain the preliminary features of the i-th layer output of the second sub-network; the collection of the preliminary features of the outputs of the m layers after the second sub-network is the initial feature, and the initial feature includes m preliminary features, 0<m<n; i＞1; Step 2: The receptive field enhancement feature pyramid module processes the initial features to obtain the output features of the second feature fusion module; S2.1: Process the initial features into the first stage of the pyramid and output the first pyramid stage feature map; S2.2: The feature map of the first pyramid stage is processed by receptive field enhancement to obtain the receptive field enhancement feature of the first pyramid stage: Among them, F _out2 represents the receptive field enhancement feature of the first pyramid stage, ε _n represents the nth type of dilated convolution with different dilation rates, Represents the concat feature map group fusion operation of two feature maps in the feature map channel dimension, _Fin represents the first pyramid stage feature map input to the receptive field enhancement module, and σ() represents a 1×1 convolution; S2.3: Process the first pyramid stage feature map into the second pyramid stage, and output the second pyramid stage feature map; S2.4: Input the first pyramid stage receptive field enhancement feature and the second pyramid stage feature map together into the second feature fusion module to obtain the output feature of the second feature fusion module: F _out = weight*F ₁ + (1-weight)*F ₂ Among them, weight represents the number of learned weights, _F1 represents the feature map of the second pyramid stage, and _F2 represents the receptive field enhancement feature of the first pyramid stage. 2. A method for extracting deep features for transparent object detection in a complex environment as claimed in claim 1, characterized in that the preliminary features in the second sub-network are: in, is the preliminary feature of the lth level in the second sub-network,/> represents the preliminary features of the output of the l-1th level in the second sub-network, n represents the number of levels of each sub-network in the composite backbone network, l represents the level of the current preliminary features, /> represents the output feature map of the i-th level of the first subnetwork input to the second subnetwork, and ε represents the feature fusion operation. 3. The method for extracting depth features of transparent object detection in a complex environment according to claim 1, wherein the first stage of pyramid processing comprises the following steps: A1: All preliminary features in the obtained initial features are subjected to 1x1 convolution respectively to unify the number of channels and dimensions to obtain feature maps of unified dimensions; A2: The feature maps of unified dimension are arranged in order from deep layers to shallow layers. The deepest layer m-th layer feature map is upsampled by nearest neighbor to obtain the m-th layer upsampled feature map. The size of the m-th layer upsampled feature map is the same as that of the m-1-th layer feature map. The m-th layer upsampled feature map and the m-1-th layer feature map are added point by point to obtain a new feature map in the upsampling process. The new feature map in the upsampling process is retained as the new feature map of the m-1-th layer. Then, the new feature map of the m-1-th layer in the upsampling process is repeatedly upsampled, layer by layer, until all the feature maps of unified dimension are processed. A3: All new feature maps obtained in the upsampling process in step A2 and the feature map of the mth level are subjected to 3×3 convolution to reduce the aliasing effect, and the feature map of the first pyramid stage is output. 4. The method for extracting depth features of transparent object detection in a complex environment according to claim 1, wherein the second stage of pyramid processing comprises the following steps: B1: The feature maps of the first pyramid stage are arranged from shallow to deep levels, and the feature map of the shallowest level 1 is downsampled to obtain the downsampled feature map of the first level. The size of the downsampled feature map of the first level is the same as the size of the feature map of the second level of the adjacent next deep level. The downsampled feature map of the first level is added point by point to the feature map of the second level to obtain a new feature map in the downsampling process. The new feature map in the downsampling process is retained as the new feature map of the second level; B2: Continue to downsample the new feature map of the second level in the downsampling process, and operate layer by layer until all the feature maps of the first pyramid stage are processed; B3: Perform 3×3 convolution on the new feature map from all downsampling processes in step B2 and the first-level feature map to output the feature map of the second pyramid stage. 5. The method for extracting depth features of transparent object detection in a complex environment according to claim 1, wherein the feature fusion operation ε is: ε←upsample(f(F _a )) Among them, upsample() is the nearest neighbor upsampling algorithm, f() represents 1x1 convolution, and _Fa represents the output feature map of the first sub-network. 6. A deep feature extraction method for transparent object detection in a complex environment as described in claim 1, characterized in that the first sub-network and the second sub-network are both Res2Net101 networks pre-trained on the ImageNet dataset. 7. A deep feature extraction method for transparent object detection in a complex environment as described in claim 1, characterized in that the second feature fusion module is an AM attention module. 8. The method for extracting depth features of transparent object detection in a complex environment according to claim 1, wherein the weight parameter weight is: weight = sigmoid(σ(F)) Among them, σ() is a 1x1 convolution, sigmoid() represents the activation function, and F represents the feature map output by the receptive field enhancement module. 9. A method for extracting depth features for transparent object detection in a complex environment as described in claim 1, characterized in that N is 5. 10. A method for extracting deep features for transparent object detection in a complex environment as described in claim 1, characterized in that the feature extraction method includes ResNet feature extraction and Res2Net feature extraction.