CN112070784A - Perception edge detection method based on context enhancement network - Google Patents

Abstract

The invention discloses a perception edge detection method based on a context enhancement network, which is characterized by comprising the following steps: 1) acquiring an image training data set training edge detection model; 2) extracting characteristics; 3) reducing the dimension and weighting; 4) bidirectional recursion; 5) classifying; 6) and (6) outputting. The method improves the accuracy of edge detection by capturing the internal relation between the multi-scale context information and has the advantage of high speed.

Description

A Context-Enhanced Network-Based Perceptual Edge Detection Method

technical field

The invention relates to the field of image processing, in particular to a perceptual edge detection method based on a context enhancement network.

Background technique

Image edge refers to the set of pixels whose surrounding pixels have step change and roof change, and is one of the most basic features of the image. Image edges often carry most of the information of an image. Edge detection plays an important role in computer vision, image processing and other applications, and is an important part of image analysis and recognition. Therefore, image edge detection has always been a hot topic of research. .

Traditional edge detection methods usually use features such as color and brightness to find edge points of images, such as Canny and Sobel operators. However, due to the complexity of natural images, it is difficult to achieve accurate edge detection only by using features such as gradient and color. Some scholars try to use various low-level features such as gradient, color and brightness for edge detection in a data-driven way, such as gPb+UCM. , StrucutredEdge and other methods, although this type of method has achieved a certain improvement compared with the gradient-based method, but because this type of method only utilizes the low-level features of the image, it is difficult to achieve robust detection in special scenarios. In order to capture the high-level features of images, some scholars try to use convolutional neural networks to extract image block features, and propose some classic edge detection methods, such as N4Fields, DeepEdge, etc., thanks to the powerful feature extraction capabilities of convolutional neural networks, This type of method has been improved in detection accuracy, but the processing method of image blocks is not conducive to capturing the global features of the image. Based on this, Xie et al. proposed an end-to-end learning edge detection method. , multi-scale features, so that the edge detection accuracy has been greatly improved. On this basis, Liu et al. improved the expressive ability of each scale and improved the accuracy of edge detection by fusing the receptive field features of different sizes in each scale.

Although a series of edge detection methods based on multi-scale combination proposed in recent years have achieved a certain improvement in detection accuracy, these methods generally suffer from less large-scale feature semantic information, more misclassification, less small-scale spatial information, and poor localization. However, the above problems cannot be well solved by weighting the output of different scales. How to complement the information of the multi-scale features of the image and enhance the classification accuracy of each scale is a difficulty at present.

SUMMARY OF THE INVENTION

Aiming at the problems of the existing edge detection methods based on multi-scale structure that the features of each scale are independent of each other, the large-scale features have less semantic information, more misclassification, less small-scale spatial information, and serious loss of small target information, the present invention provides an A perceptual edge detection method based on context-augmented networks. This method improves the accuracy of edge detection by capturing the intrinsic connection between multi-scale contextual information, and has the advantage of being fast.

The technical scheme that realizes the object of the present invention is:

A method for perceptual edge detection based on a context-enhanced network, comprising the following steps:

(1) Obtain the image training data set and train the edge detection model: the edge detection model includes a feature extraction process, a neural network two-way recursive process and a classification process, specifically:

其中CSU模块里包含横向和纵向两个支路，每个CSU模块的纵向支路数N从前到后分别为{2,2,3,3,3}，纵向支路负责提取高维图像特征，横向支路负责进行特征的聚合以及上采样，过程可以通过公式(1)、公式(2)进行表示：Feature extraction process: map the sample image x into 5 groups of d-dimensional features. The feature extraction network consists of 5 CSU modules. As a trainable feature extraction network, the input sample image x is mapped into 5 groups of 21-dimensional features. feature, that is

The CSU module contains two branches, horizontal and vertical. The number of vertical branches N of each CSU module is {2, 2, 3, 3, 3} from front to back. The vertical branch is responsible for extracting high-dimensional image features. The lateral branch is responsible for feature aggregation and upsampling, and the process can be expressed by formula (1) and formula (2):

Among them, i is the i-th CSU module, n is the n-th longitudinal branch convolution in the CSU module, W is the longitudinal convolution kernel parameter, and the size of the convolution kernel is uniformly 3×3, the same below, dW is the horizontal convolution Kernel parameters, the size of the convolution kernel is uniformly 1×1, Φ is the Relu activation function, up( ) is a bilinear interpolation function, and the aggregation operation is implemented by 1×1 convolution;

和后向递归神经网络的5个输出

其中递归神经网络由5个CLSTM单元串联组成，CLSTM单元内部设有三部分，分别为输入单元i_t、输出单元o_t、遗忘单元f_t，递归过程如下所示：(2) Input the five feature groups [x ₁ , x ₂ , x ₃ , x ₄ , x ₅ ] into two recurrent neural networks in an orderly manner according to the forward and reverse sequence directions, and obtain five features of the forward recurrent neural network output

and the 5 outputs of the backward recurrent neural network

The recurrent neural network is composed of 5 CLSTM units connected in series. There are three parts inside the CLSTM unit, which are the input unit i _t , the output unit _ot , and the forgetting unit ft _. The recursive process is as follows:

Input: memory feature c _t-1 of the previous CLSTM unit, input x _t at the current moment, output h _t-1 of the previous CLSTM unit,

2-1) The memory feature c _t-1 of the previous CLSTM unit is screened by the forget gate f _t , and the output c' is formula (3):

c'=f _t ·c _t-1 (3),

f _t =σ(W _f *[x _t ,h _t-1 ]+b _f ) (4),

where W _f is the 1×1 convolution weight, b _f is the bias term, x _t is the input at the current moment, h _t-1 is the output of the previous CLSTM unit, σ is the sigmoid activation function, and c _t-1 is the previous CLSTM The memory features of the unit, the forget gate is designed to select and filter the memory features of the previous unit. The previous unit fuses the feature information of x _t , h _t-1 through 1×1 convolution, and generates a weight through the sigmoid activation function. map, and use the weight map to select and filter memory features;

2-2) The input feature is screened by the input gate i _t , and weighted with the screened memory feature c' to obtain the memory feature c _t of the current CLSTM unit as formula (5):

c _t =c'+(1+i _t )×Φ(W _x *[x _t ,h _t-1 ]+b _c ) (5),

i _t =σ(W _i *[x _t ,h _t-1 ]+b _i ) (6),

where W _x and Wi are the convolution weights, _bi and b _c are the bias terms, Φ is the _Relu function, σ is the Sigmoid function, * is the convolution operation, and the input gate is designed to model the importance of the input features , generate a feature weight matrix, and selectively enhance the input features. The current CLSTM unit fuses the feature information of x _t and h _t-1 through 1×1 convolution, and generates a weight map through the Sigmoid activation function. The weight map selectively enhances the fused input feature information;

2-3) The memory feature is selectively enhanced by the output gate o _t , and the current CLSTM output h _t is obtained as formula (7):

h _t =(o _t +1)×Φ(c _t ) (7),

o _t =σ(W _o *[x _t ,h _t-1 ]+b _o ) (8),

Among them, Φ is the Relu function, σ is the Sigmoid function, * is the convolution operation, W _o is the convolution weight, and b _o is the bias term;

Output: the memory feature c _t of the current CLSTM unit, the output h _t of the current CLSTM unit;

以及后向支路特征张量

分别经过卷积操作将特征通道数降至1，得到

以及

过程如公式(9)、公式(10)所示：(3) The obtained forward branch feature tensor

and the backward branch feature tensor

After the convolution operation, the number of feature channels is reduced to 1, and we get

as well as

The process is shown in formula (9) and formula (10):

在维度1进行通道拼接，得到张量so，采用1×1卷积对张量so进行加权融合，经过Sigmoid激活函数得到最终输出sout为公式(11)：(4) The feature tensor

Perform channel splicing in dimension 1 to obtain the tensor so, use 1×1 convolution to weight the tensor so, and obtain the final output sout through the Sigmoid activation function as formula (11):

sout=Wso* _so (11),

分别经过Sigmoid激活函数，得到10个支路输出

(5) will

After passing through the Sigmoid activation function respectively, 10 branch outputs are obtained

(6) Calculate the loss of all outputs and labels by using the cross entropy _lw (X _i ), and optimize the overall network by minimizing the loss of _Lw . Since the labels in the data set are labelled by many people, this technical solution performs a weighted average of multiple labels. , and divide the label according to the threshold θ, take the pixel points with y _i <θ as the ambiguous point, do not calculate the loss, y _i =θ is the non-edge point, y _i ≥ θ is the edge point:

in,

|Y ⁺ | and |Y ^- | represent the sum of the number of positive samples and the total number of negative samples in each batch, respectively, the hyperparameter λ is the balance parameter of positive and negative samples, X represents the network output, and _yi is the label pixel. Therefore, in the end, The loss function of is formula (16):

为前向递归神经网络第k阶段的输出，

为后向递归神经网络第k阶段的输出，sout是最终网络的输出值，|I|为图像I的所有像素点数量；

is the output of the kth stage of the forward recurrent neural network,

is the output of the kth stage of the backward recurrent neural network, sout is the output value of the final network, |I| is the number of all pixels in the image I;

(7) Repeat steps (1) to (6) until the network converges.

This technical solution uses a bidirectional recurrent neural network to enhance the internal connection between multi-scale contexts, and solves the problems of insufficient utilization of each scale information based on the current multi-scale method and limited edge detection accuracy. The characteristics of time sequence from small to large, the multi-scale context information is analyzed and modeled from the time dimension, using the idea of recurrent neural network, through the recurrent neural network branches from top to bottom and bottom to top to gradually capture different directions and The context connection in different ranges completes the enhancement of the expression ability of the feature pyramid, which has the advantages of high detection accuracy and fast speed.

This method improves the accuracy of edge detection by capturing the intrinsic connection between multi-scale contextual information, and has the advantage of being fast.

Description of drawings

1 is a schematic diagram of the internal structure of a CLSTM unit in an embodiment;

2 is a schematic diagram of an overall network structure in an embodiment;

FIG. 3 is a schematic flowchart of the method in the embodiment.

Detailed ways

The content of the present invention will be further elaborated below in conjunction with the accompanying drawings and embodiments, but it is not intended to limit the present invention.

Example:

Referring to Figures 2 and 3, a method for perceptual edge detection based on a context-enhanced network, comprising the following steps:

(1) Obtain the image training data set and train the edge detection model: the edge detection model includes a feature extraction process, a neural network two-way recursive process and a classification process, specifically:

其中CSU模块里包含横向和纵向两个支路，每个CSU模块的纵向支路数N从前到后分别为{2,2,3,3,3}，纵向支路负责提取高维图像特征，横向支路负责进行特征的聚合以及上采样，过程可以通过公式(1)、公式(2)进行表示：Feature extraction process: map the sample image x into 5 groups of d-dimensional features. The feature extraction network consists of 5 CSU modules. As a trainable feature extraction network, the input sample image x is mapped into 5 groups of 21-dimensional features. feature, that is

The CSU module contains two branches, horizontal and vertical. The number of vertical branches N of each CSU module is {2, 2, 3, 3, 3} from front to back. The vertical branch is responsible for extracting high-dimensional image features. The lateral branch is responsible for feature aggregation and upsampling, and the process can be expressed by formula (1) and formula (2):

Among them, i is the i-th CSU module, n is the n-th longitudinal branch convolution in the CSU module, W is the longitudinal convolution kernel parameter, and the size of the convolution kernel is uniformly 3×3, the same below, dW is the horizontal convolution Kernel parameters, the size of the convolution kernel is uniformly 1×1, Φ is the Relu activation function, up( ) is a bilinear interpolation function, and the aggregation operation is implemented by 1×1 convolution;

和后向递归神经网络的5个输出

其中递归神经网络由5个CLSTM单元串联组成，CLSTM单元内部设有三部分，分别为输入单元i_t、输出单元o_t、遗忘单元f_t，CLSTM单元内部结构如图1所示，递归过程如下所示：(2) Input the five feature groups [x ₁ , x ₂ , x ₃ , x ₄ , x ₅ ] into two recurrent neural networks in an orderly manner according to the forward and reverse sequence directions, and obtain five features of the forward recurrent neural network output

and the 5 outputs of the backward recurrent neural network

The recurrent neural network is composed of 5 CLSTM units in series. There are three parts inside the _CLSTM unit, namely the input unit it, the output unit ot, and the forgetting unit ft _. The internal structure of the _CLSTM unit is shown in Figure 1. The recursive process is as follows Show:

Input: memory feature c _t-1 of the previous CLSTM unit, input x _t at the current moment, output h _t-1 of the previous CLSTM unit,

2-1) The memory feature c _t-1 of the previous CLSTM unit is screened by the forget gate f _t , and the output c' is formula (3):

c'=f _t ·c _t-1 (3),

f _t =σ(W _f *[x _t ,h _t-1 ]+b _f ) (4),

where W _f is the 1×1 convolution weight, b _f is the bias term, x _t is the input at the current moment, h _t-1 is the output of the previous CLSTM unit, σ is the sigmoid activation function, and c _t-1 is the previous CLSTM The memory features of the unit, the forget gate is designed to select and filter the memory features of the previous unit. The previous unit fuses the feature information of x _t , h _t-1 through 1×1 convolution, and generates a weight through the sigmoid activation function. map, and use the weight map to select and filter memory features;

2-2) The input feature is screened by the input gate i _t , and weighted with the screened memory feature c' to obtain the memory feature c _t of the current CLSTM unit as formula (5):

c _t =c'+(1+i _t )×Φ(W _x *[x _t ,h _t-1 ]+b _c ) (5),

i _t =σ(W _i *[x _t ,h _t-1 ]+b _i ) (6),

where W _x and Wi are the convolution weights, _bi and b _c are the bias terms, Φ is the _Relu function, σ is the Sigmoid function, * is the convolution operation, and the input gate is designed to model the importance of the input features , generate a feature weight matrix, and selectively enhance the input features. The current CLSTM unit fuses the feature information of x _t and h _t-1 through 1×1 convolution, and generates a weight map through the Sigmoid activation function. The weight map selectively enhances the fused input feature information;

2-3) The memory feature is selectively enhanced by the output gate o _t , and the current CLSTM output h _t is obtained as formula (7):

h _t =(o _t +1)×Φ(c _t ) (7),

o _t =σ(W _o *[x _t ,h _t-1 ]+b _o ) (8), where Φ is the Relu function, σ is the Sigmoid function, * is the convolution operation, and W _o is the convolution weight, b _o is the bias term;

Output: the memory feature c _t of the current CLSTM unit, the output h _t of the current CLSTM unit;

以及后向支路特征张量

分别经过卷积操作将特征通道数降至1，得到

以及

过程如公式(9)、公式(10)所示：(3) The obtained forward branch feature tensor

and the backward branch feature tensor

After the convolution operation, the number of feature channels is reduced to 1, and we get

as well as

The process is shown in formula (9) and formula (10):

在维度1进行通道拼接，得到张量so，采用1×1卷积对张量so进行加权融合，经过Sigmoid激活函数得到最终输出sout为公式(11)：(4) The feature tensor

Perform channel splicing in dimension 1 to obtain the tensor so, use 1×1 convolution to weight the tensor so, and obtain the final output sout through the Sigmoid activation function as formula (11):

sout=Wso* _so (11),

分别经过Sigmoid激活函数，得到10个支路输出

(5) will

After passing through the Sigmoid activation function respectively, 10 branch outputs are obtained

(6) Calculate the loss of all outputs and labels by using the cross entropy _lw (X _i ), and optimize the overall network by minimizing the loss of _Lw . Since the labels in the data set are labelled by many people, this technical solution performs a weighted average of multiple labels. , and divide the label according to the threshold θ, take the pixel points with y _i <θ as the ambiguous point, do not calculate the loss, y _i =θ is the non-edge point, y _i ≥ θ is the edge point:

in,

|Y ⁺ | and |Y ^- | represent the sum of the number of positive samples and the total number of negative samples in each batch, respectively, the hyperparameter λ is the balance parameter of positive and negative samples, X represents the network output, and _yi is the label pixel. Therefore, in the end, The loss function of is Equation (16):

为前向递归神经网络第k阶段的输出，

为后向递归神经网络第k阶段的输出，sout是最终网络的输出值，|I|为图像I的所有像素点数量；

is the output of the kth stage of the forward recurrent neural network,

is the output of the kth stage of the backward recurrent neural network, sout is the output value of the final network, |I| is the number of all pixels in the image I;

(7) Repeat steps (1) to (6) until the network converges.

Claims (1)

1. a perceptual edge detection method based on context enhancement network, is characterized in that, comprises the steps: (1) Obtain the image training data set and train the edge detection model: the edge detection model includes a feature extraction process, a neural network two-way recursive process and a classification process, specifically: Feature extraction process: map the sample image x into 5 groups of d-dimensional features. The feature extraction network consists of 5 CSU modules. As a trainable feature extraction network, the input sample image x is mapped into 5 groups of 21-dimensional features. feature, that is

i∈{1～5}, in which the CSU module contains two branches, horizontal and vertical. The number of vertical branches N of each CSU module is {2,2,3,3,3} from front to back, respectively. The path is responsible for extracting high-dimensional image features, and the lateral branch is responsible for feature aggregation and upsampling. The process is represented by formula (1) and formula (2):

Among them, i is the i-th CSU module, n is the n-th longitudinal branch convolution in the CSU module, W is the longitudinal convolution kernel parameter, and the size of the convolution kernel is uniformly 3×3, the same below, dW is the horizontal convolution Kernel parameters, the size of the convolution kernel is uniformly 1×1, Φ is the Relu activation function, up( ) is a bilinear interpolation function, and the aggregation operation is implemented by 1×1 convolution; (2) Input the five feature groups [x ₁ , x ₂ , x ₃ , x ₄ , x ₅ ] into two recurrent neural networks in an orderly manner according to the forward and reverse sequence directions, and obtain five features of the forward recurrent neural network output

and the 5 outputs of the backward recurrent neural network

The recurrent neural network is composed of 5 CLSTM units connected in series. There are three parts inside the CLSTM unit, which are the input unit i _t , the output unit _ot , and the forgetting unit ft _. The recursive process is as follows: Input: memory feature c _t-1 of the previous CLSTM unit, input x _t at the current moment, output h _t-1 of the previous CLSTM unit, 2-1) The memory feature c _t-1 of the previous CLSTM unit is screened by the forget gate f _t , and the output c' is formula (3): c'=f _t ·c _t-1 (3), f _t =σ(W _f *[x _t ,h _t-1 ]+b _f ) (4), where W _f is the 1×1 convolution weight, b _f is the bias term, x _t is the input at the current moment, h _t-1 is the output of the previous CLSTM unit, σ is the sigmoid activation function, and c _t-1 is the previous CLSTM The memory features of the unit, the forget gate is designed to select and filter the memory features of the previous unit. The previous unit fuses the feature information of x _t , h _t-1 through 1×1 convolution, and generates a weight through the sigmoid activation function. map, and use the weight map to select and filter memory features; 2-2) The input feature is screened by the input gate i _t , and weighted with the screened memory feature c' to obtain the memory feature c _t of the current CLSTM unit as formula (5): c _t =c'+(1+i _t )×Φ(W _x *[x _t ,h _t-1 ]+b _c ) (5), i _t =σ(W _i *[x _t ,h _t-1 ]+b _i ) (6), where W _x and Wi are the convolution weights, _bi and b _c are the bias terms, Φ is the _Relu function, σ is the Sigmoid function, * is the convolution operation, and the input gate is designed to model the importance of the input features , generate a feature weight matrix, and selectively enhance the input features. The current CLSTM unit fuses the feature information of x _t and h _t-1 through 1×1 convolution, and generates a weight map through the Sigmoid activation function. The weight map selectively enhances the fused input feature information; 2-3) The memory feature is selectively enhanced by the output gate o _t , and the current CLSTM output h _t is obtained as formula (7): h _t =(o _t +1)×Φ(c _t ) (7), o _t =σ(W _o *[x _t ,h _t-1 ]+b _o ) (8), where Φ is the Relu function, σ is the Sigmoid function, * is the convolution operation, and W _o is the convolution weight, b _o is the bias term; Output: the memory feature c _t of the current CLSTM unit, the output h _t of the current CLSTM unit; (3) The obtained forward branch feature tensor

and the backward branch feature tensor

After the convolution operation, the number of feature channels is reduced to 1, and we get

as well as

The process is shown in formula (9) and formula (10):

(4) The feature tensor

Perform channel splicing in dimension 1 to obtain the tensor so, use 1×1 convolution to weight the tensor so, and obtain the final output sout through the Sigmoid activation function as formula (11): sout=Wso* _so (11),

(5) will

After passing through the Sigmoid activation function respectively, 10 branch outputs are obtained

(6) Calculate the loss of all outputs and labels by using the cross entropy _lw (X _i ), optimize the overall network by minimizing the loss of Lw, perform a weighted average of multiple labels, and divide the labels according to the threshold θ, with _y Pixels with _i < θ are ambiguous points, no loss is calculated, _yi = θ is a non-edge point, y _i ≥ θ is an edge point:

in,

|Y ⁺ | and |Y ^- | represent the total number of positive samples and the total number of negative samples in each batch, respectively, the hyperparameter λ is the balance parameter of positive and negative samples, X represents the network output, _yi is the label pixel, the final loss The function is formula (16):

is the output of the kth stage of the forward recurrent neural network,

is the output of the kth stage of the backward recurrent neural network, sout is the output value of the final network, |I| is the number of all pixels in the image I; (7) Repeat steps (1) to (6) until the network converges.

Images