CN111126385A - Deep learning intelligent identification method for deformable living body small target - Google Patents
Deep learning intelligent identification method for deformable living body small target Download PDFInfo
- Publication number
- CN111126385A CN111126385A CN201911284570.7A CN201911284570A CN111126385A CN 111126385 A CN111126385 A CN 111126385A CN 201911284570 A CN201911284570 A CN 201911284570A CN 111126385 A CN111126385 A CN 111126385A
- Authority
- CN
- China
- Prior art keywords
- deformable
- convolution
- offset
- dimensional
- pooling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 26
- 238000013135 deep learning Methods 0.000 title claims abstract description 16
- 238000011176 pooling Methods 0.000 claims abstract description 96
- 238000001514 detection method Methods 0.000 claims abstract description 46
- 238000005070 sampling Methods 0.000 claims abstract description 24
- 230000000694 effects Effects 0.000 claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 11
- 238000013527 convolutional neural network Methods 0.000 claims description 61
- 238000004422 calculation algorithm Methods 0.000 claims description 25
- 238000010586 diagram Methods 0.000 claims description 21
- 230000004927 fusion Effects 0.000 claims description 21
- 230000006872 improvement Effects 0.000 claims description 15
- 230000004913 activation Effects 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 10
- 238000010606 normalization Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 5
- 150000001875 compounds Chemical class 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 claims description 2
- 238000009795 derivation Methods 0.000 claims description 2
- 238000007499 fusion processing Methods 0.000 claims description 2
- 238000013461 design Methods 0.000 description 4
- 230000003190 augmentative effect Effects 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011897 real-time detection Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000003416 augmentation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000000280 densification Methods 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000013095 identification testing Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
本发明涉及一种可变形活体小目标的深度学习智能识别方法，属于机器人视觉及其智能识别技术领域；本发明将可变形卷积模块和可变形ROI池化模块与Faster R‑CNN进行合理的结合，使用可变形网络的可变形卷积模块和可变形ROI池化模块对模型进行改进，对标准卷积的空间取样点和普通ROI池化加入二维甚至高维偏移，使得卷积的取样点发生形状变化，提高改进后模型的可变形特性，改进模型提高了对可变形目标的检测识别效果；考虑对不同层的特征图进行融合，对底层特征图池化处理降低分辨率，对高层特征逆卷积处理提高分辨率，然后融合低、中、高三层特征图；同时考虑增加一组小尺度预选框，增加小目标预选框的生成数量，改进模型提高了对小目标的检测识别效果。
The invention relates to a deep learning intelligent identification method for deformable living body small targets, belonging to the technical field of robot vision and its intelligent identification. In combination, the deformable convolution module and deformable ROI pooling module of the deformable network are used to improve the model, and two-dimensional or even high-dimensional offsets are added to the spatial sampling points of standard convolution and ordinary ROI pooling, so that the convolutional The shape of the sampling point changes, the deformable characteristics of the improved model are improved, and the improved model improves the detection and recognition effect of deformable targets; consider fusing the feature maps of different layers, and reduce the resolution by pooling the underlying feature maps. High-level feature deconvolution processing improves resolution, and then fuses low, medium, and high-level feature maps; at the same time, consider adding a set of small-scale pre-selection boxes, increasing the number of small target pre-selection boxes, and improving the model to improve detection and recognition of small targets Effect.
Description
Technical Field
The invention relates to a deep learning intelligent identification method of a deformable living body small target, belonging to the technical field of robot vision and intelligent identification thereof.
Background
The vision and intelligent recognition technology of the robot is one of the main means for acquiring external information by the robot, and is widely used in the fields of detection, target tracking, operation and the like in the robot field at present. However, with the revolution of technology and the need for improving system performance, the visual intelligent technology requires the robot to detect and identify not only small-scale targets in different scenes, but also deformable living targets. Two main solutions are currently available for the problem of difficulty in detecting deformable living targets. The first category is to build a training set with enough varying shapes for the target, which is mainly achieved by augmenting the existing data. The method mainly achieves robust detection of deformable targets by consuming a large amount of training and complex model parameters. The second category of methods uses features and algorithms with transform invariance, and this category of methods contains many classical algorithms such as SIFT, scale invariant feature transform, and sliding window based object detection paradigm.
However, the above-mentioned method suffers from both of these disadvantages. First, the geometric transformation is fixed and known, and this a priori knowledge is used to design augmentation data and design features and algorithms, however, for live objects, the shape transformation has many forms, and the augmented object morphology is limited, so this approach cannot deal with the unknown geometric transformations for those morphologies that are not augmented. Second, for overly complex transforms, even if the transforms are known, artificially designing invariant features and algorithms is difficult to implement and infeasible.
Disclosure of Invention
The invention aims to provide a deep learning intelligent identification method of a deformable living small target for improving the detection effect of the deformable target.
The invention aims to realize the method, and the method for intelligently recognizing the deep learning of the deformable living small target specifically comprises the following steps:
and 4, in the fast R-CNN network, the RPN network is used for generating preselected frames, then an algorithm classifies and regresses the preselected frames, the mechanism of the anchor point is improved, and a group of small-scale preselected frames are added in the anchor point, so that the RPN can generate more small target preselected frames, and the detection and identification effects on small targets are improved.
The invention also includes such structural features:
1. the deformable convolution network comprises a deformable convolution module, a deformable ROI pooling module and a deformable position-sensitive ROI pooling module; the convolution and the feature map in a convolutional neural network are both three-dimensional, the deformable convolution operates in a two-dimensional spatial domain, and the deformable convolution operation is the same between different channel dimensions.
2. Step 1 is a two-dimensional operation description of deformable convolution, and specifically comprises adding two-dimensional or even high-dimensional offset to spatial sampling points of standard convolution to enable the shape of the sampling points of the convolution to change; the offset is obtained by performing convolution operation on the same input feature map, and the convolution kernel of the convolution operation and the previous convolution layer keep the same resolution and expansion value; the output offset domain and the input feature map have the same spatial resolution, the number of channels of the offset domain is twice that of the input feature map, which corresponds to the two-dimensional offset of each sampling position of convolution, in training, a convolution kernel for generating the output feature map and a convolution kernel for generating the offset domain are learned at the same time, and for learning to obtain the offset domain, the gradient is obtained by bilinear operation inverse operation of the following two formulas:
where p denotes the position of an arbitrary sample point, the gradient relative offset Δ p in the deformable ROI pooling moduleijIn the formula (c), p is p0+pn+ΔpnQ represents an input feature diagram InAll integer spatial traversal points in (a), (b), G (a, b) max (0,1- | a-b);
in the deformable convolution formula, the gradient is relative to the offset Δ pnThe calculation formula of (2) is as follows:
in the formula (I), the compound is shown in the specification,
can be represented by the formula G (q, p) ═ G (q)x,px)·g(qy,py) Derivation gives, notice of Δ pnIs a two-dimensional quantity, which we use for simplicity
To replace
And
3. the step 2 is that the deformable ROI pooling operation of the ROI pooling layer is operated in a two-dimensional space domain, the deformable ROI pooling operation is the same among different channel dimensions, and the deformable ROI pooling operation specifically comprises that a two-dimensional or even high-dimensional offset is added to the position of each square of the common ROI pooling, so that the deformable capability of the convolutional neural network is improved, and the detection and identification capability of the convolutional neural network on a deformable target is improved; firstly, obtaining a pooled feature map by using ROI pooling operation; then, a full connection layer is connected behind the characteristic graph to obtain normalized offset; finally this normalized offset is calculated by multiplying the elements of the width and height of the region of interest; the normalization of the offset is essential for learning the invariance of the offset to the size of the region of interest, and the parameters of the subsequent full-link layer are obtained through a back propagation algorithm; in the deformable ROI pooling Module, the relative gradient offset Δ pijThe value of (d) can be calculated as:
4. the deformable convolution network can be improved on a Faster R-CNN network, the improvement is divided into two stages, the first stage is that a full convolution network generates a feature map for an input picture, and a modified VGG16 network removes a maximum pooling layer, two 4096 unit full connection layers and a 1000 unit full connection layer which follow a convolution unit in order to extract features; the deformable convolution is applied to the last convolution unit, i.e., the three convolution layers conv5_1, conv5_2, and conv5_ 3. The second phase is that a lightweight task-based network generates results based on the input feature map; the classification regression part of the Faster R-CNN network mainly uses an RPN network to generate a preselected frame, then the preselected frame and a feature map are input into the Fast R-CNN network, firstly, an ROI pooling layer performs ROI pooling on a frame to obtain features, two 1024-dimensional full-connected layers are added, and finally, two parallel branches are connected, and target regression and classification are respectively performed to obtain a final result.
5. Step 3, the improvement of the Faster R-CNN model by using a structure based on inverse convolution specifically comprises the steps of inserting an inverse pooling layer in a convolution neural network; in order to apply the inverse pooling layer, the position of the maximum activation value is first recorded at the time of the pooling operation; then, returning the activation value to the position of the activation value during pooling during anti-pooling, and setting the rest positions to be zero; finally, the output characteristic diagram of the inverse convolution needs to be clipped, so that the resolution of the characteristic diagram after the inverse convolution processing is consistent with the resolution of the inverse pooling output characteristic diagram.
6. Step 3, the improvement of the Faster R-CNN model by using the multi-layer feature fusion structure specifically comprises the steps of firstly carrying out fusion processing on features aiming at the condition of insufficient feature information, and then carrying out ROI pooling on a plurality of regions of interest, so that only one-time feature fusion and one-time normalization are needed, and the time for repeated calculation is saved; secondly, aiming at the condition that the region of interest is small, the last layer of features is subjected to inverse convolution processing, the third layer of features is subjected to maximum pooling processing, and finally the three feature graphs are fused.
Compared with the prior art, the invention has the beneficial effects that: the invention designs a deep learning intelligent identification method of a deformable living body small target, which reasonably combines a deformable convolution module and a deformable ROI pooling module with fast R-CNN according to the characteristics of the deformable living body small target, wherein the deformable convolution module is used for replacing a basic convolution unit, the deformable ROI pooling module is used for replacing an ROI pooling layer, and the sampling of a detection model can be changed along with the change of the shape of the detection target by introducing the deformable convolution and the deformable ROI pooling module, so that the detection effect of the deformable target is improved. The fast R-CNN model is improved by using inverse convolution and multi-layer feature fusion, the information quantity obtained by a small target preselection frame is richer by using the inverse convolution and the multi-layer feature fusion, and the improvement of an anchor point mechanism is that RPN can generate more small target preselection frames. Meanwhile, the method based on inverse convolution and multi-layer feature fusion has strong semantic information of high-layer features and combines the advantage of high resolution of low-layer features.
Drawings
FIG. 1 is a schematic diagram of a 3 × 3 deformable convolution;
FIG. 2 is a schematic 3X 3 deformable ROI pooling;
FIG. 3 is a schematic diagram of a modification of the deformable convolution, deformable ROI pooling to Faster R-CNN;
FIG. 4 is a schematic diagram of the deconvolution and inverse pooling operations;
FIG. 5 is a schematic of multi-layer feature fusion;
FIG. 6 is a schematic diagram of improved multi-layer feature fusion;
fig. 7 is a schematic diagram of the structure of an RPN network;
FIG. 8 is a result of deformable convolution, deformable ROI pooling real-time online identification of video frames;
FIG. 9 is a visualization of the original Faster R-CNN (left) and improved Faster R-CNN (right) sea creature target detection.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
The invention designs a method for intelligently recognizing the deep learning of a deformable living body small target, which reasonably combines a deformable convolution module and a deformable ROI pooling module with an Faster R-CNN according to the characteristics of the deformable living body small target, wherein the deformable convolution module is used for replacing a basic convolution unit, the deformable ROI pooling module is used for replacing an ROI pooling layer, meanwhile, an inverse convolution and multilayer feature fusion are used for improving an Faster R-CNN model, the inverse convolution and multilayer feature fusion enable a small target preselection frame to obtain richer information quantity, and the improvement on an anchor point mechanism enables an RPN to generate more small target preselection frames.
The method is based on the advanced achievement of the inventor in the aspect of artificial intelligence research, and the method is accurate in identification of the deformable living body small target. The introduction of the deformable convolution and the deformable ROI pooling module can enable the sampling of the detection model to change along with the change of the shape of the detection target, thereby improving the detection effect of the deformable target. The method based on the inverse convolution and the multi-layer feature fusion has strong semantic information of high-layer features and combines the advantage of high resolution of low-layer features.
The invention is realized as follows:
a. a deep learning intelligent identification method of a deformable living body small target mainly comprises the following steps: firstly, adding two-dimensional or even high-dimensional offset to spatial sampling points of standard convolution to enable the shape of the sampling points of the convolution to change; secondly, adding a two-dimensional or even high-dimensional offset to the position of each square block of the ordinary ROI (region of interest) pooling so as to improve the deformable capability of the convolutional neural network, thereby improving the detection and identification capability of the convolutional neural network on a deformable target. Aiming at the detection and identification of small targets, a structure based on inverse convolution and multi-layer feature fusion is firstly used for improving a Faster R-CNN model, so that the information quantity obtained by a small target preselection frame is richer; second, an improvement to the anchor point mechanism allows the RPN to generate more small target pre-selection boxes. Thereby improving the detection and identification effects on small targets.
b. The deformable convolution network comprises a deformable convolution module, a deformable ROI pooling module and a deformable position-sensitive ROI pooling module. The convolution and the feature map in a convolutional neural network are both three-dimensional, the deformable convolution operates in a two-dimensional spatial domain, and the deformable convolution operation is the same between different channel dimensions. Without loss of generality, we will next describe the two-dimensional operation of the model, extending to three dimensions, exactly the same, in order to simplify the problem.
Adding two-dimensional or even high-dimensional offset to the spatial sampling points of the standard convolution to enable the shape of the sampling points of the standard convolution to change; the offset is obtained by performing a convolution operation on the same input feature map, the convolution kernel of the convolution operation maintaining the same resolution and dilation values as the previous convolution layer. The output offset field has the same spatial resolution as the input signature, and the number of channels in the offset field is twice the number of channels in the input signature, which corresponds to the two-dimensional offset of convolving each sample location. In training, the convolution kernel that generates the output feature map and the convolution kernel that generates the offset field are learned simultaneously. To learn the offset domain, the gradient is obtained by inverse operation of bilinear operations in equations (1) and (2).
In the formula, p represents an arbitrary sampling point position (in the formula (4-3), p is p0+pn+Δpn) Q represents an input feature diagram InAll integer space traversal points in (a), (b) represent a bilinear interpolation kernel, G (a, b) ═ max (0,1- | a-b |).
In the deformable convolution formula, the gradient is relative to the offset Δ pnThe calculation formula of (a) is as follows:
in the formula (I), the compound is shown in the specification,
can be derived by the formula (2). Note Δ pnIs a two-dimensional quantity, which we use for simplicity
To replace
And
c. likewise, the deformable ROI pooling operation is also operated in the two-dimensional spatial domain, and is the same between different channel dimensions. Without loss of generality, we will next describe the two-dimensional operation of the model, extending to three dimensions, exactly the same, in order to simplify the problem.
A two-dimensional or even high-dimensional offset is added to the position of each square block for the common ROI (region of interest) pooling, so that the deformable capability of the convolutional neural network is improved, and the detection and identification capability of the convolutional neural network on a deformable target is improved. First, a pooled feature map is obtained using an ROI pooling operation. The signature is then followed by a fully connected layer to get the normalized offset. Finally, this normalized offset is multiplied by the elements of the width and height of the region of interest. Normalization of the offset is essential for the offset learning to be invariant to the region of interest size, and the parameters of the subsequent fully-connected layer will be obtained by the back-propagation algorithm.
In the deformable ROI pooling Module, the relative gradient offset Δ pijThe value of (d) can be calculated as follows:
d. for Faster R-CNN, the network is intended to be divided into two phases. In the first stage, a full convolution network generates a feature map for an input picture. In the second phase, a lightweight task-based network generates results based on the input feature map. We mainly refined these two parts with deformable convolution and deformable ROI pooling.
The first stage of the improvement of deformable network to the fast R-CNN network is that a full convolution network generates a characteristic diagram for an input picture. A modified version of the VGG16 network removed one maximum pooling layer, two 4096-cell full-link layers, and one 1000-cell full-link layer following the convolution cell in order to extract features. The method of claim 1 is applied to the last convolution unit, namely the three convolution layers conv5_1, conv5_2 and conv5_ 3.
A classification regression part of a light task-based network generating a result Faster R-CNN network based on an input feature map, mainly using an RPN network to generate a preselected frame, inputting the preselected frame and the feature map into the Fast R-CNN network, performing ROI pooling on a frame by an ROI pooling layer to obtain features, adding two 1024-dimensional full-connected layers, and finally connecting two parallel branches to respectively perform target regression and classification to obtain a final result.
e. Aiming at the detection and identification of small targets, a structure based on inverse convolution and multi-layer feature fusion is designed. The fast R-CNN model is first improved, and an inverse pooling layer is inserted into a convolutional neural network. To apply the inverse pooling layer, first, at the time of the pooling operation, the location of the maximum activation value is recorded. Then, when the pool is reversed, the activation value is returned to the position when the pool is reversed, and the rest positions are all set to be zero. Finally, we need to clip the deconvolved output feature map so that the resolution of the deconvolved feature map is consistent with the resolution of the inverse pooled output feature map.
In the aspect of multi-layer feature fusion, firstly, the feature is fused under the condition of insufficient feature information, and then ROI pooling is carried out on a plurality of interested regions, so that only one-time feature fusion and one-time normalization are needed, and the time of repeated calculation is saved. Secondly, aiming at the condition that the region of interest is small, the last layer of features is subjected to inverse convolution processing, the third layer of features is subjected to maximum pooling processing, and finally the three feature graphs are fused. The resolution of the final used feature map is improved.
f. The anchor point mechanism in the RPN network is modified, and a group of small-scale pre-selection frames are added in the anchor point, so that the pre-selection frames of the small targets contained in the pre-selection frames extracted last by the RPN network are more, and the detection and the identification of the small targets are facilitated.
In the Faster R-CNN network, the RPN network is used to generate preselected boxes, which are then classified and regressed by an algorithm. Therefore, if the RPN is able to generate a more appropriate pre-selection frame, the detection recognition result will also be improved.
The present invention will be described in detail with reference to the drawings:
the first implementation mode comprises the following steps: FIG. 1 is a schematic diagram of a deformable convolution in which a sample point becomes an irregular and offset point by applying an offset to a conventional sampling grid, typically a fractional number, for sampling on an input feature map, typically by bilinear interpolation. The offset is obtained by performing convolution operation on the same input feature map, the convolution kernel of the convolution operation maintains the same resolution and expansion value as the previous convolution layer, the output offset domain has the same spatial resolution as the input feature map, and the number of channels of the offset domain is twice the number of channels of the input feature map, which corresponds to the two-dimensional offset (offset in the x-axis direction and offset in the y-axis direction) of each sampling position of convolution.
The second embodiment: FIG. 2 is a schematic diagram of deformable ROI pooling. First, a pooled feature map is obtained using an ROI pooling operation. Then, the feature map is followed by a fully connected layer to obtain a normalized offset
Finally, this normalized offset
By multiplication of elements with the width and height of the region of interest, e.g. formula
The offset Δ p used in the following equation is obtainedij. Empirically, this amount is usually set to γ of 0.1. Normalization of the offset is essential for the invariance of the offset to the size of the region of interest. The parameters of the subsequent fully-connected layer are obtained by a back propagation algorithm.
The third embodiment is as follows: FIG. 3 is a schematic diagram of a modification of the deformable convolution, deformable ROI pooling to Faster R-CNN. The characteristic extraction part of the Faster R-CNN network uses a modified VGG16 network as a basic network to extract characteristics, and the modified VGG16 network removes a maximum pooling layer, two 4096 unit full-link layers and a 1000 unit full-link layer which follow a convolution unit in order to extract characteristics. Experiments have shown that better results can be obtained when the last convolution unit is used for the deformable convolution. Therefore, consider applying the deformable convolution to the last convolution unit, i.e., the three convolution layers conv5_1, conv5_2, conv5_ 3.
The classification regression part of the Faster R-CNN network mainly uses an RPN network to generate a preselected frame, then the preselected frame and a feature map are input into the Fast R-CNN network, firstly, an ROI pooling layer performs ROI pooling on a frame to obtain features, two 1024-dimensional full-connected layers are added, and finally, two parallel branches are connected, and target regression and classification are respectively performed to obtain a final result. In the Fast R-CNN part, we replace the ROI pooling layer with a deformable ROI pooling layer.
The fourth embodiment: FIG. 4 is a schematic diagram of the deconvolution and inverse pooling operations: first, at the time of the pooling operation, the location of the maximum activation value is recorded. Then, when the pool is reversed, the activation value is returned to the position when the pool is reversed, and the rest positions are all set to be zero. The deconvolution operation subjects the output profile of the inverse pooling operation to densification by using a multi-layer convolution-like operation to generate a dense profile. However, in contrast to convolutional layer one convolution operation, which convolves multiple inputs to obtain one output, one input is deconvoluted to obtain multiple outputs. Finally, we need to clip the deconvolved output feature map so that the resolution of the deconvolved feature map is consistent with the resolution of the inverse pooled output feature map.
The fifth embodiment: the combination of global features and local features, such as multi-scale, is used to enhance the acquisition of global texture and local information by the fast R-CNN network to improve the robustness of target detection, and fig. 5 is a multi-layer feature fusion, and the combination of global features and local features, such as multi-scale, is used to enhance the acquisition of global texture and local information by the fast R-CNN network to improve the robustness of target detection. In order to enhance the detection capability of the network, a shallow feature map, such as conv3 or conv4, is considered and then ROI pooling is performed, so that the network can detect a feature containing more low-level features within the region of interest, as shown in the figure.
Embodiment six: consider that the high-level information is deconvoluted to the same resolution as the low-level information, and then the multi-level features of the same resolution are fused. FIG. 6 is a schematic diagram of improved multi-layer feature fusion. Firstly, the output characteristic maps of the three layers conv3, conv4 and conv5 are taken. Then, ROI pooling is carried out on areas corresponding to conv3, conv4 and conv5 by using the region of interest, pooled features are normalized and merged by using L2 in one layer, and the number of the merged feature channels is reduced to be consistent with the output features of conv 5. And finally, connecting a target classification layer and a target regression layer. Since the three feature maps need to be combined, the features of different layers are normalized, such as L2 normalization, and then combined.
Embodiment seven: fig. 7 is a network configuration diagram of the RPN. Nine pre-selected boxes are generated at each sliding window in the original RPN network, which are the metrics 1282,2562,5122]And aspect ratio [1:1,1:2,2:1]A random combination of (a). This scale and aspect ratio selection gave the best test results for the pascal voc data set. Adding a set 64 of small target objects2Of (4), i.e. a preselected box size of [642,1282,2562,5122]. Thus, 12 pre-selection frames are generated at each sliding window and the pre-selection frames tend to tilt towards small target detection, ultimately improving the detection efficiency of small targets.
The eighth embodiment: FIG. 8 is the result of deformable convolution, deformable ROI pooling real-time online identification of video frames, we performed online identification experiments using the improved Faster R-CNN model, with the detection rate of the improved algorithm in the experiments being 12 frames per second.
Table 1 is the test results of our online identification. The predicted value is a target value of each kind of result obtained by algorithm prediction, and the true value is a value obtained by manually labeling the real-time detection video. As can be seen from Table 1, the predicted value of the algorithm is close to the true value, which shows that the improved algorithm has better detection robustness to the deformation problem of marine organisms encountered in real-time detection. FIG. 8 is a test result of identifying certain frames in a video online. As can be seen from fig. 8, the detection result is stable, which indicates that the improved algorithm has a better detection performance for the disturbance deformable target in the unstable imaging environment.
TABLE 1 Online identification test results
The ninth embodiment: detection results of original Faster R-CNN algorithm and improved Faster R-CNN algorithm on targets with different scales of marine biological data
TABLE 2
As can be seen from Table 2, the improved Faster R-CNN improves the detection results of targets with different scales, and the improvement effect of the small target detection effect is obvious. The detection results of the original Faster R-CNN algorithm and the improved Faster R-CNN algorithm on the small target are respectively mAP (IOU threshold value minus 0.5)35.45 and 42.95, the improvement is 21.16%, and the improvement of the improved Faster R-CNN algorithm on the small target detection is obvious compared with the original Faster R-CNN algorithm. Under a stricter evaluation index, namely, the IOU threshold is taken to be 0.7, the detection results mAP of the original Faster R-CNN algorithm and the improved Faster R-CNN algorithm on the small target are respectively 22.40 and 29.78, the improvement is 32.94%, and the improvement of the algorithm on the small target detection effect is further explained.
In summary, the invention introduces the deformable network, improves the model by using the deformable convolution module and the deformable ROI pooling module of the deformable network, adds two-dimensional or even high-dimensional offset to the spatial sampling point of the standard convolution and the common ROI (region of interest) pooling, enables the sampling point of the convolution to generate shape change, improves the deformable characteristic of the improved model, and improves the detection and identification effect of the deformable target by improving the model. And considering fusion of feature maps of different layers, performing pooling processing on the feature maps of the bottom layer to reduce resolution, performing deconvolution processing on the features of the high layer to improve resolution, and then fusing the feature maps of the low layer, the middle layer and the high layer. Meanwhile, a group of small-scale preselection frames is added, the generation number of the small target preselection frames is increased, and the detection and identification effect of the small target is improved by improving the model.
Claims (7)
1. A deep learning intelligent identification method for a deformable living body small target is characterized by specifically comprising the following steps:
step 1, replacing a basic convolution unit by a deformable convolution module: adding two-dimensional or even high-dimensional offset to the spatial sampling points of the standard convolution to change the shapes of the sampling points of the standard convolution;
step 2. the deformable ROI pooling module replaces the ROI pooling layer: adding a two-dimensional or even high-dimensional offset to the position of each square grid of the common ROI (Region of Interest) pooling so as to improve the deformable capability of the convolutional neural network, obtain the deformable convolutional network and improve the detection and identification capability of the convolutional neural network on a deformable target;
step 3, aiming at the detection and identification of the small target, a structure based on inverse convolution and multi-layer feature fusion is used for improving the FasterR-CNN model, so that the information quantity obtained by a small target preselection frame is richer;
and 4, in the fast R-CNN network, the RPN network is used for generating preselected frames, then an algorithm classifies and regresses the preselected frames, the mechanism of the anchor point is improved, and a group of small-scale preselected frames are added in the anchor point, so that the RPN can generate more small target preselected frames, and the detection and identification effects on small targets are improved.
2. The intelligent deep learning identification method for the deformable living small target as claimed in claim 1, characterized in that: the deformable convolution network comprises a deformable convolution module, a deformable ROI pooling module and a deformable position-sensitive ROI pooling module; the convolution and the feature map in a convolutional neural network are both three-dimensional, the deformable convolution operates in a two-dimensional spatial domain, and the deformable convolution operation is the same between different channel dimensions.
3. The intelligent deep learning identification method for the deformable living small target as claimed in claim 1, characterized in that: step 1 is a two-dimensional operation description of deformable convolution, and specifically comprises adding two-dimensional or even high-dimensional offset to spatial sampling points of standard convolution to enable the shape of the sampling points of the convolution to change; the offset is obtained by performing convolution operation on the same input feature map, and the convolution kernel of the convolution operation and the previous convolution layer keep the same resolution and expansion value; the output offset domain and the input feature map have the same spatial resolution, the number of channels of the offset domain is twice that of the input feature map, which corresponds to the two-dimensional offset of each sampling position of convolution, in training, a convolution kernel for generating the output feature map and a convolution kernel for generating the offset domain are learned at the same time, and for learning to obtain the offset domain, the gradient is obtained by bilinear operation inverse operation of the following two formulas:
G(q,p)＝g(qx,px)·g(qy,py)
where p denotes the position of an arbitrary sample point, the gradient relative offset Δ p in the deformable ROI pooling moduleijIn the formula (c), p is p0+pn+ΔpnQ represents an input feature diagram InAll integer space traversal points in (a), (b), G (a, b) denotes a bilinear interpolation kernel, G max (0,1- | a-b |);
in the deformable convolution formula, the gradient is relative to the offset Δ pnThe calculation formula of (2) is as follows:
in the formula (I), the compound is shown in the specification,
can be represented by the formula G (q, p) ═ G (q)x,px)·g(qy,py) Derivation gives, notice of Δ pnIs a two-dimensional quantity, which we use for simplicity
To replace
And
4. the intelligent deep learning identification method for the deformable living small target as claimed in claim 1, characterized in that: the step 2 is that the deformable ROI pooling operation of the ROI pooling layer is operated in a two-dimensional space domain, the deformable ROI pooling operation is the same among different channel dimensions, and the deformable ROI pooling operation specifically comprises that a two-dimensional or even high-dimensional offset is added to the position of each square of the common ROI pooling, so that the deformable capability of the convolutional neural network is improved, and the detection and identification capability of the convolutional neural network on a deformable target is improved; firstly, obtaining a pooled feature map by using ROI pooling operation; then, a full connection layer is connected behind the characteristic graph to obtain normalized offset; finally this normalized offset is calculated by multiplying the elements of the width and height of the region of interest; the normalization of the offset is essential for learning the invariance of the offset to the size of the region of interest, and the parameters of the subsequent full-link layer are obtained through a back propagation algorithm; in the deformable ROI pooling Module, the relative gradient offset Δ pijThe value of (d) can be calculated as:
5. the intelligent deep learning identification method for the deformable living small target as claimed in claim 1, characterized in that: the deformable convolution network can be improved on a Faster R-CNN network, the improvement is divided into two stages, the first stage is that a full convolution network generates a feature map for an input picture, and a modified VGG16 network removes a maximum pooling layer, two 4096 unit full connection layers and a 1000 unit full connection layer which follow a convolution unit in order to extract features; applying the deformable convolution to the last convolution unit, i.e. the three convolution layers conv5_1, conv5_2 and conv5_ 3; the second phase is that a lightweight task-based network generates results based on the input feature map; the classification regression part of the Faster R-CNN network mainly uses an RPN network to generate a preselected frame, then the preselected frame and a feature map are input into the Fast R-CNN network, firstly, an ROI pooling layer performs ROI pooling on a frame to obtain features, two 1024-dimensional full-connected layers are added, and finally, two parallel branches are connected, and target regression and classification are respectively performed to obtain a final result.
6. The intelligent deep learning identification method for the deformable living small target as claimed in claim 1, characterized in that: step 3, the improvement of the Faster R-CNN model by using a structure based on inverse convolution specifically comprises the steps of inserting an inverse pooling layer in a convolution neural network; in order to apply the inverse pooling layer, the position of the maximum activation value is first recorded at the time of the pooling operation; then, returning the activation value to the position of the activation value during pooling during anti-pooling, and setting the rest positions to be zero; finally, the output characteristic diagram of the inverse convolution needs to be clipped, so that the resolution of the characteristic diagram after the inverse convolution processing is consistent with the resolution of the inverse pooling output characteristic diagram.
7. The intelligent deep learning identification method for the deformable living small target as claimed in claim 1, characterized in that: step 3, the improvement of the Faster R-CNN model by using the multi-layer feature fusion structure specifically comprises the steps of firstly carrying out fusion processing on features aiming at the condition of insufficient feature information, and then carrying out ROI pooling on a plurality of regions of interest, so that only one-time feature fusion and one-time normalization are needed, and the time for repeated calculation is saved; secondly, aiming at the condition that the region of interest is small, the last layer of features is subjected to inverse convolution processing, the third layer of features is subjected to maximum pooling processing, and finally the three feature graphs are fused.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911284570.7A CN111126385A (en) | 2019-12-13 | 2019-12-13 | Deep learning intelligent identification method for deformable living body small target |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911284570.7A CN111126385A (en) | 2019-12-13 | 2019-12-13 | Deep learning intelligent identification method for deformable living body small target |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111126385A true CN111126385A (en) | 2020-05-08 |
Family
ID=70498812
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911284570.7A Pending CN111126385A (en) | 2019-12-13 | 2019-12-13 | Deep learning intelligent identification method for deformable living body small target |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111126385A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111709307A (en) * | 2020-05-22 | 2020-09-25 | 哈尔滨工业大学 | A small target detection method in remote sensing images based on resolution enhancement |
| CN111815510A (en) * | 2020-09-11 | 2020-10-23 | 平安国际智慧城市科技股份有限公司 | Image processing method based on improved convolutional neural network model and related equipment |
| CN111860171A (en) * | 2020-06-19 | 2020-10-30 | 中国科学院空天信息创新研究院 | A method and system for detecting irregularly shaped targets in large-scale remote sensing images |
| CN112651346A (en) * | 2020-12-29 | 2021-04-13 | 青海三新农电有限责任公司 | Streaming media video identification and detection method based on deep learning |
| CN112733672A (en) * | 2020-12-31 | 2021-04-30 | 深圳一清创新科技有限公司 | Monocular camera-based three-dimensional target detection method and device and computer equipment |
| CN113177486A (en) * | 2021-04-30 | 2021-07-27 | 重庆师范大学 | Dragonfly order insect identification method based on regional suggestion network |
| CN114155246A (en) * | 2022-02-10 | 2022-03-08 | 国网江西省电力有限公司电力科学研究院 | A Defect Detection Method for Transmission Tower Pins Based on Deformable Convolution |
| CN116205967A (en) * | 2023-04-27 | 2023-06-02 | 中国科学院长春光学精密机械与物理研究所 | Medical image semantic segmentation method, device, equipment and medium |
| CN116824333A (en) * | 2023-06-21 | 2023-09-29 | 中山大学附属第一医院 | A nasopharyngeal cancer detection system based on deep learning model |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107229904A (en) * | 2017-04-24 | 2017-10-03 | 东北大学 | A kind of object detection and recognition method based on deep learning |
| CN107316001A (en) * | 2017-05-31 | 2017-11-03 | 天津大学 | Small and intensive method for traffic sign detection in a kind of automatic Pilot scene |
| CN107766818A (en) * | 2017-10-18 | 2018-03-06 | 哈尔滨工程大学 | A kind of didactic submerged structure environment line feature extraction method |
| US20180137642A1 (en) * | 2016-11-15 | 2018-05-17 | Magic Leap, Inc. | Deep learning system for cuboid detection |
| CN109766873A (en) * | 2019-02-01 | 2019-05-17 | 中国人民解放军陆军工程大学 | pedestrian re-identification method based on hybrid deformable convolution |
| CN110163275A (en) * | 2019-05-16 | 2019-08-23 | 西安电子科技大学 | SAR image objective classification method based on depth convolutional neural networks |
-
2019
- 2019-12-13 CN CN201911284570.7A patent/CN111126385A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180137642A1 (en) * | 2016-11-15 | 2018-05-17 | Magic Leap, Inc. | Deep learning system for cuboid detection |
| CN107229904A (en) * | 2017-04-24 | 2017-10-03 | 东北大学 | A kind of object detection and recognition method based on deep learning |
| CN107316001A (en) * | 2017-05-31 | 2017-11-03 | 天津大学 | Small and intensive method for traffic sign detection in a kind of automatic Pilot scene |
| CN107766818A (en) * | 2017-10-18 | 2018-03-06 | 哈尔滨工程大学 | A kind of didactic submerged structure environment line feature extraction method |
| CN109766873A (en) * | 2019-02-01 | 2019-05-17 | 中国人民解放军陆军工程大学 | pedestrian re-identification method based on hybrid deformable convolution |
| CN110163275A (en) * | 2019-05-16 | 2019-08-23 | 西安电子科技大学 | SAR image objective classification method based on depth convolutional neural networks |
Non-Patent Citations (1)
| Title |
|---|
| 周浩: "样本不足条件下水下机器人小目标检测识别研究", 《网页在线公开：HTTPS://D.WANFANGDATA.COM.CN/THESIS/CHJUAGVZAXNOZXDTMJAYMTEYMDESCFKZNTUXMDU0GGHYAG10EWXWNW%3D%3D》 * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111709307B (en) * | 2020-05-22 | 2022-08-30 | 哈尔滨工业大学 | Resolution enhancement-based remote sensing image small target detection method |
| CN111709307A (en) * | 2020-05-22 | 2020-09-25 | 哈尔滨工业大学 | A small target detection method in remote sensing images based on resolution enhancement |
| CN111860171A (en) * | 2020-06-19 | 2020-10-30 | 中国科学院空天信息创新研究院 | A method and system for detecting irregularly shaped targets in large-scale remote sensing images |
| CN111815510A (en) * | 2020-09-11 | 2020-10-23 | 平安国际智慧城市科技股份有限公司 | Image processing method based on improved convolutional neural network model and related equipment |
| CN111815510B (en) * | 2020-09-11 | 2020-12-22 | 平安国际智慧城市科技股份有限公司 | Image processing method based on improved convolutional neural network model and related equipment |
| CN112651346A (en) * | 2020-12-29 | 2021-04-13 | 青海三新农电有限责任公司 | Streaming media video identification and detection method based on deep learning |
| CN112733672A (en) * | 2020-12-31 | 2021-04-30 | 深圳一清创新科技有限公司 | Monocular camera-based three-dimensional target detection method and device and computer equipment |
| CN113177486A (en) * | 2021-04-30 | 2021-07-27 | 重庆师范大学 | Dragonfly order insect identification method based on regional suggestion network |
| CN113177486B (en) * | 2021-04-30 | 2022-06-03 | 重庆师范大学 | Identification method of Odonata insects based on region proposal network |
| CN114155246B (en) * | 2022-02-10 | 2022-06-14 | 国网江西省电力有限公司电力科学研究院 | Deformable convolution-based power transmission tower pin defect detection method |
| CN114155246A (en) * | 2022-02-10 | 2022-03-08 | 国网江西省电力有限公司电力科学研究院 | A Defect Detection Method for Transmission Tower Pins Based on Deformable Convolution |
| CN116205967A (en) * | 2023-04-27 | 2023-06-02 | 中国科学院长春光学精密机械与物理研究所 | Medical image semantic segmentation method, device, equipment and medium |
| CN116824333A (en) * | 2023-06-21 | 2023-09-29 | 中山大学附属第一医院 | A nasopharyngeal cancer detection system based on deep learning model |
| CN116824333B (en) * | 2023-06-21 | 2024-08-16 | 中山大学附属第一医院 | A nasopharyngeal carcinoma detection system based on deep learning model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111126385A (en) | Deep learning intelligent identification method for deformable living body small target | |
| Zhang et al. | SCSTCF: spatial-channel selection and temporal regularized correlation filters for visual tracking | |
| Truong et al. | Pdc-net+: Enhanced probabilistic dense correspondence network | |
| Christiansen et al. | Unsuperpoint: End-to-end unsupervised interest point detector and descriptor | |
| Rong et al. | Radial lens distortion correction using convolutional neural networks trained with synthesized images | |
| Garg et al. | Unsupervised cnn for single view depth estimation: Geometry to the rescue | |
| Xu et al. | Learning low-rank and sparse discriminative correlation filters for coarse-to-fine visual object tracking | |
| US20160196672A1 (en) | Graph image representation from convolutional neural networks | |
| CN112541491B (en) | End-to-end text detection and recognition method based on image character region perception | |
| CN114419732B (en) | HRNet human posture recognition method based on attention mechanism optimization | |
| Song et al. | Deep novel view synthesis from colored 3d point clouds | |
| US20220335571A1 (en) | Methods and systems for super resolution for infra-red imagery | |
| CN116188999B (en) | Small target detection method based on visible light and infrared image data fusion | |
| Tan et al. | ECO-TR: Efficient correspondences finding via coarse-to-fine refinement | |
| CN113052755A (en) | High-resolution image intelligent matting method based on deep learning | |
| US20200379481A1 (en) | Localising a vehicle | |
| Mueller et al. | Image-to-image translation for enhanced feature matching, image retrieval and visual localization | |
| Vaquero et al. | Tracking more than 100 arbitrary objects at 25 FPS through deep learning | |
| Wu et al. | Sc-wls: Towards interpretable feed-forward camera re-localization | |
| CN112329662B (en) | Multi-view saliency estimation method based on unsupervised learning | |
| Zhang et al. | A new deep spatial transformer convolutional neural network for image saliency detection | |
| Lee et al. | Connectivity-based convolutional neural network for classifying point clouds | |
| Guo et al. | Salient object detection from low contrast images based on local contrast enhancing and non-local feature learning | |
| CN113807354A (en) | Image semantic segmentation method, device, equipment and storage medium | |
| Dou et al. | Background subtraction based on circulant matrix |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200508 |
|
| RJ01 | Rejection of invention patent application after publication |