CN110852316A - Image tampering detection and positioning method adopting convolution network with dense structure - Google Patents

Abstract

The invention provides an image tampering detection and positioning method using a dense structure convolutional network, which includes inputting an image to be tested, performing preprocessing with spatial enrichment SRM convolution to obtain a preprocessed image, and constructing a densely connected volume. The product network extracts the tampered image feature of the preprocessed image, obtains the binary classification information of the image to be tested, and completes the detection of image tampering; constructs a deconvolution network symmetrical with the connected convolutional network structure, and takes the binary classification information as input. ; According to the obtained image tampering area, the image after positioning is completed by the deconvolution network. The method provided by the present invention applies deep learning technology to image tampering detection and positioning, is suitable for a variety of tampering methods, and has good robustness and practicability; provides a unified framework for detection and positioning, not only multi-image Whether the prediction has been made by tampering, it can also predict the tampered area, give accurate pixel-by-pixel annotations, and obtain detailed object contour boundaries.

Description

An Image Tampering Detection and Localization Method Using Densely Structured Convolutional Networks

technical field

The invention relates to the technical field of image blind forensics, and more particularly, to an image tampering detection and positioning method using a dense structure convolutional network.

Background technique

In the information age, images are one of the main ways of disseminating information. Because of their intuitive representation of things and thinking orientation, this way of information dissemination has been fully integrated into human life. However, the development of image tampering technology is also advancing by leaps and bounds, and the major threat to the security of multimedia content cannot be ignored. Current image tampering techniques are mainly divided into methods based on artificial feature extraction and methods based on deep learning.

The method based on artificial feature extraction is to perform various transformations on the image, and then use threshold or machine learning methods to classify image features, but this method relies on the modeling of image features by researchers, and is usually only applicable to one type of Image tampering recognition, that is, although it has a good effect on one tampering method, it has poor applicability to other tampering methods, and has poor robustness and scalability; deep learning-based methods usually only focus on detection and localization. One of them, although it can achieve high accuracy in detection, cannot exert the superior performance of deep learning in target detection, and does not make full use of the connection between detection and positioning.

SUMMARY OF THE INVENTION

The invention provides an image tampering detection and localization method using a dense structure convolutional network in order to overcome the technical defect that the existing image tampering recognition technology cannot realize the detection and location of image tampering at the same time.

For solving the above-mentioned technical problems, the technical scheme of the present invention is as follows:

A method for image tampering detection and localization using densely structured convolutional networks, comprising the following steps:

S1: Input the image to be tested, perform preprocessing with spatial enrichment SRM convolution to obtain the preprocessed image;

S2: Construct a densely connected convolutional network to extract the tampered image features of the preprocessed image, obtain the binary classification information of the image to be tested, and complete the detection of image tampering;

S3: Construct a deconvolutional network that is symmetrical with the connected convolutional network structure, and uses the binary information of the image to be tested as input to locate the image tampering area;

S4: According to the obtained image tampering area, the image after positioning is completed by the deconvolution network, and the positioning of the image tampering is completed.

Wherein, in the step S2, the densely connected convolutional network includes a pooling layer, a dense layer, a transition layer, a global average pooling layer and a fully connected layer; wherein:

The pooling layer performs a convolution and maximum pooling operation on the preprocessed image, and inputs the result into the dense layer;

The dense layer and the transition layer are provided with multiple layers, and the output results of each dense layer are input into the corresponding transition layer, and finally the tampered image feature map obtained by the last transition layer is input into the global average pooling layer. middle;

The global average pooling layer performs average pooling on the tampered image feature map, and the fully connected layer calculates and outputs two probability values, which represent the probability of tampering and non-tampering respectively, and obtains the binary classification information of the image to be tested.

Wherein, the dense layer includes a plurality of basic structure layers, each basic structure layer is composed of two consecutive convolutional layers, wherein the input of each basic structure layer is merged with the output of the previous layer, which is A locally dense version of the residual structure.

The densely connected convolutional network is provided with four dense layers, including 5, 10, 20, and 12 basic structure layers, respectively.

The transition layer includes a convolution layer, which convolves the feature map input to the dense layer once, and then performs average pooling to reduce the size of the image.

Wherein, the fully connected layer calculates and outputs two probability values through the softmax function, and the specific calculation formula is:

代表网络在i类别上的输出值，y_i表示样本在i类别上的真实值，a_i代表i类别的权重。where i represents the two categories of tampering/non-tampering,

Represents the output value of the network on category i, _yi represents the real value of the sample on category i, and a _i represents the weight of category i.

In the above scheme, in order to better capture the tampering noise features of the image, an SRM convolution is performed on the RGB three channels of the input image, and the convolution kernel is initialized with the normalized SRM, and the three channels of a convolution kernel are all Use the same model to assign values to get 30 filters, and the convolutional output is combined with the RGB three channels in series.

In the above scheme, after the pooling operation is performed by the pooling layer, the dense layer and transition layer of the densely connected convolutional network are used to construct a deep network to facilitate the extraction of the features of the tampered image. Two consecutive convolutional layers form a base structure layer, and a dense layer can include multiple base structure layers, and in a dense layer, the input of each base structure layer is composed of all outputs of the previous layers after combining operations. , such a structure is a locally dense version of the residual structure and can be beneficial for training deeper networks without overfitting. One of the convolutional networks uses four dense layers, including 5, 10, 20, and 12 basic structures, respectively. The transition layer is a convolutional layer. The input feature map is convolved once, the depth is reduced, and then averaged Pooling, downsizing. The pooling used in the network is all 2x2 pooling, and the feature map after the last dense layer should be one-thirtieth of the original size. The global average pooling layer averages the feature maps and only retains the depth. After the fully connected layer, two values are output, which are converted into probability values by the softmax function, representing the probability of tampering and non-tampering respectively, and the maximum value is the final judgment result. .

In the above scheme, there are batch normalization and relu activation function layers after each convolution layer to prevent gradient explosion or dispersion and introduce nonlinear models.

Wherein, in the step S3, the deconvolution network includes a fully connected layer, a dense layer and a corresponding deconvolution transition layer; firstly, the fully connected layer is used to calculate the tampered image features point by point, and then the dense layer And the corresponding deconvolution transition layer continues to restore the image layer by layer to locate the image tampering area.

Wherein, in the step S4, according to the image tampering area, the binary image after the location of the image to be tested is output by the deconvolution network, so as to complete the location of the image tampering.

In the above scheme, the present invention constructs a deconvolutional network by using a structure that is as symmetrical as possible to the convolutional network. First, the global pooling layer in the convolutional network is removed to retain the complete feature map, and the corresponding fully connected layer can be used. The operation is performed point by point on the feature map, which is equivalent to a 1x1 convolution, followed by three dense layers containing 12, 6, and 3 basic structure layers and a deconvolution transition layer. The deconvolution transition layer is the transition layer. The improvement of the average pooling is replaced by a 2x deconvolution layer, thereby doubling the size of the feature map.

In the above scheme, in order to better supplement the details of the output image, the present invention inputs the output of the previous convolutional network into the layer of the subsequent deconvolutional network through direct connection, 2x deconvolution, and 4x deconvolution operations. A multi-size feature splicing, the feature maps obtained by merging in series can study multi-size context information, thereby helping the network learn how to accurately predict the boundary, contour and size of the tampered area; The output of the connection layer is successively deconvolved by 2x and then connected to the following layers to increase its importance in network decision-making. The present invention considers that the output of the fully connected layer is a kind of effective spatial decision information, because the fully connected layer is trained for the tampered binary classification in the detection task, so this information is used more.

Wherein, the training process of the densely connected convolutional network and the deconvolutional network is as follows:

Collect training image data and perform preprocessing;

Divide the preprocessed image data into training set and test set;

Use the training set to pre-train 128x128 images, and calculate the gradient update parameters;

The weights of the densely connected convolutional network are obtained by training the full size image according to the gradient update parameters;

Pre-train the deconvolution network with 128x128 images according to the weight of the densely connected convolutional network, and calculate the gradient update parameters;

Complete the training of the deconvolution network by training the full-size image according to the calculated gradient update parameters;

Use the test set to evaluate and adjust the deconvolution network, and finally output the densely connected convolution network and deconvolution network corresponding to the weights.

Among them, in the training and adjustment process of the densely connected convolutional network and the deconvolutional network, the five-fold cross-validation method is used for adjustment, and each time one fifth of the preprocessed image data is taken as the test set, and four fifths as the test set. The training set, through five training evaluations, takes the average training result as the final result.

In the above scheme, the present invention saves the window containing the tampered area as a new image by sliding the image with a 128x128 small window, and selects scientific and reasonable samples from the window based on the strategy of the size of the tampered area. First, in order not to make the tampered area too large, only the windows with the tampered area accounting for no more than 40% are kept; secondly, in order to avoid the tampered area being too small, the windows with the area of the tampered area less than 150 pixels are discarded. This can prevent the unreasonable area of the sample image tampering area from appearing, which is helpful for the network to learn image tampering detection. At the same time, the multi-angle rotation of the image is carried out by means of data augmentation to enhance the rotation invariance of the model.

In the above scheme, the training revolves around first detection and then positioning, first training the binary convolutional network used to detect whether the image has been tampered, and then retaining the weights of the convolutional network, training with the target of locating the tampered area, and only using the tampered training samples to update Convolutional and Deconvolutional Networks.

In the above scheme, 128x128 small images are used to calculate the gradient update parameters during training, and the video memory of the graphics processor is fully utilized, and the gradients of multiple samples can be calculated by forward propagation; Uneven, a forward pass can only calculate the gradient of one sample. In order to make the loss decrease smoothly, the present invention uses a program to set up a gradient accumulator, averages multiple gradients, and then updates the parameters again.

Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

The invention provides an image tampering detection and positioning method using a densely structured convolutional network. The deep learning technology is applied to the image tampering detection and positioning, and the network is trained to learn the characteristics of the tampered image, which is suitable for dealing with various tampering methods. It can continue to update parameters to improve performance under the premise of new data sets, and has good robustness and practicability; compared with other deep learning methods, the invention realizes a unified framework for detection and positioning, not only can multiple images be tampered or not. It can also predict the tampered area, give accurate pixel-by-pixel annotations, and obtain detailed object contour boundaries.

Description of drawings

Fig. 1 is the flow chart of the method steps of the present invention;

Figure 2 is a schematic diagram of the structure of a convolutional network and a deconvolutional network;

Fig. 3 is the training flow chart of convolutional network and deconvolutional network;

FIG. 4 is a schematic diagram of the results of the positioning test sample.

Detailed ways

The accompanying drawings are for illustrative purposes only, and should not be construed as limitations on this patent;

In order to better illustrate this embodiment, some parts of the drawings are omitted, enlarged or reduced, which do not represent the size of the actual product;

It will be understood by those skilled in the art that some well-known structures and their descriptions may be omitted from the drawings.

The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

Example 1

As shown in Figure 1, an image forgery detection and localization method using a densely structured convolutional network includes the following steps:

S1: Input the image to be tested, perform preprocessing with spatial enrichment SRM convolution to obtain the preprocessed image;

S2: Construct a densely connected convolutional network to extract the tampered image features of the preprocessed image, obtain the binary classification information of the image to be tested, and complete the detection of image tampering;

S3: Construct a deconvolutional network that is symmetrical with the connected convolutional network structure, and uses the binary information of the image to be tested as input to locate the image tampering area;

S4: According to the obtained image tampering area, the image after positioning is completed by the deconvolution network, and the positioning of the image tampering is completed.

More specifically, as shown in Figure 2, in the step S2, the densely connected convolutional network includes a pooling layer, a dense layer, a transition layer, a global average pooling layer and a fully connected layer; wherein:

The pooling layer performs a convolution and maximum pooling operation on the preprocessed image, and inputs the result into the dense layer;

The dense layer and the transition layer are provided with multiple layers, and the output results of each dense layer are input into the corresponding transition layer, and finally the tampered image feature map obtained by the last transition layer is input into the global average pooling layer. middle;

The global average pooling layer performs average pooling on the tampered image feature map, and the fully connected layer calculates and outputs two probability values, which represent the probability of tampering and non-tampering respectively, and obtains the binary classification information of the image to be tested.

More specifically, the dense layer includes a plurality of basic structure layers, each basic structure layer is composed of two consecutive convolutional layers, wherein the input of each basic structure layer is formed by combining the outputs of the previous layer. , is a locally dense version of the residual structure.

More specifically, the densely connected convolutional network is provided with four dense layers, including 5, 10, 20, and 12 basic structure layers, respectively.

More specifically, the transition layer includes a convolution layer, which convolves the feature map input to the dense layer once, and then performs average pooling to reduce the size of the image.

More specifically, the fully connected layer calculates and outputs two probability values through the softmax function, and the specific calculation formula is:

代表网络在i类别上的输出值，y_i表示样本在i类别上的真实值，a_i代表i类别的权重。where i represents the two categories of tampering/non-tampering,

Represents the output value of the network on category i, _yi represents the real value of the sample on category i, and a _i represents the weight of category i.

In the specific implementation process, in order to better capture the tampering noise features of the image, an SRM convolution is performed on the RGB three channels of the input image, and the convolution kernel is initialized with the normalized SRM. The channels are all assigned with the same model, and 30 filters are obtained, and the convolutional output is combined with the RGB three channels in series.

In the specific implementation process, after the pooling operation is performed by the pooling layer, the dense layer and transition layer of the densely connected convolutional network are used to construct a deep network to facilitate the extraction of the features of the tampered image. Two consecutive convolutional layers form a base structure layer, and a dense layer can include multiple base structure layers, and in a dense layer, the input of each base structure layer is composed of all outputs of the previous layers after combining operations. , such a structure is a locally dense version of the residual structure and can be beneficial for training deeper networks without overfitting. One of the convolutional networks uses four dense layers, including 5, 10, 20, and 12 basic structures, respectively. The transition layer is a convolutional layer. The input feature map is convolved once, the depth is reduced, and then averaged Pooling, downsizing. The pooling used in the network is all 2x2 pooling, and the feature map after the last dense layer should be one-thirtieth of the original size. The global average pooling layer averages the feature maps and only retains the depth. After the fully connected layer, two values are output, which are converted into probability values by the softmax function, representing the probability of tampering and non-tampering respectively, and the maximum value is the final judgment result. .

In the specific implementation process, there are batch normalization and relu activation function layers after each convolutional layer, so as to prevent gradient explosion or dispersion and introduce nonlinear models.

More specifically, as shown in FIG. 2 , in the step S3, the deconvolution network includes a fully connected layer, a dense layer and a corresponding deconvolution transition layer; first, the tampered image features are gradually updated through the fully connected layer. Points are calculated, and then the image is restored layer by layer through the dense layer and the corresponding deconvolution transition layer to locate the image tampering area.

More specifically, in the step S4, according to the image tampering area, the deconvolution network outputs the binary image after the location of the image to be tested, so as to complete the location of the image tampering.

In the specific implementation process, the present invention constructs a deconvolutional network by using a structure that is as symmetrical as possible to the convolutional network. First, the global pooling layer in the convolutional network is removed so that the complete feature map can be retained, and the corresponding fully connected The layer can be operated point by point on the feature map, which is equivalent to a 1x1 convolution, followed by three dense layers containing 12, 6, and 3 basic structure layers and a deconvolution transition layer. The improvement of the transition layer replaces the average pooling with a 2x deconvolution layer, thereby doubling the size of the feature map.

In the specific implementation process, in order to better supplement the details of the output image, the present invention inputs the output of the previous convolutional network into the layer of the subsequent deconvolution network through direct connection, 2x deconvolution, and 4x deconvolution operations. In this way, a multi-size feature splicing is formed, and the feature maps obtained by concatenating and merging can study multi-size context information, thereby helping the network learn how to accurately predict the boundary, contour and size of the tampered area; The output of the fully connected layer is deconvolved step by step 2x and connected to the following layers to increase its importance in network decision-making. The present invention considers that the output of the fully connected layer is a kind of effective spatial decision information, because the fully connected layer is trained for the tampered binary classification in the detection task, so this information is used more.

Example 2

More specifically, on the basis of Embodiment 1, as shown in FIG. 3 , the training process of the densely connected convolutional network and the deconvolutional network is as follows:

Collect training image data and perform preprocessing;

Divide the preprocessed image data into training set and test set;

Use the training set to pre-train 128x128 images, and calculate the gradient update parameters;

The weights of the densely connected convolutional network are obtained by training the full size image according to the gradient update parameters;

Pre-train the deconvolution network with 128x128 images according to the weight of the densely connected convolutional network, and calculate the gradient update parameters;

Complete the training of the deconvolution network by training the full-size image according to the calculated gradient update parameters;

Use the test set to evaluate and adjust the deconvolution network, and finally output the densely connected convolution network and deconvolution network corresponding to the weights.

More specifically, in the training and adjustment process of the densely connected convolutional network and the deconvolutional network, the five-fold cross-validation method is used for adjustment, and one fifth of the preprocessed image data is taken as the test set each time, and the fifth is used as the test set. Four is used as the training set. After five training evaluations, the average training result is taken as the final result.

In the specific implementation process, the present invention saves the window containing the tampered area as a new image by sliding the image with a 128x128 small window, and selects scientific and reasonable samples from the window based on the strategy of the size of the tampered area. First, in order not to make the tampered area too large, only the windows with the tampered area accounting for no more than 40% are kept; secondly, in order to avoid the tampered area being too small, the windows with the area of the tampered area less than 150 pixels are discarded. This can prevent the unreasonable area of the sample image tampering area from appearing, which is helpful for the network to learn image tampering detection. At the same time, the multi-angle rotation of the image is carried out by means of data augmentation to enhance the rotation invariance of the model.

In the specific implementation process, the training revolves around first detection and then positioning. First, the binary convolutional network for detecting whether the image has been tampered is trained, and then the weight of the convolutional network is reserved. The target training is to locate the tampered area, and only the tampering training is used. Sample update convolutional and deconvolutional networks.

In the specific implementation process, 128x128 small images are used to calculate the gradient update parameters during training, and the video memory of the graphics processor is fully utilized, and the gradients of multiple samples can be calculated by forward propagation. Due to the uneven size, a forward pass can only compute the gradient of one sample. In order to make the loss decrease smoothly, the present invention uses a program to set up a gradient accumulator, averages multiple gradients, and then updates the parameters again.

Example 3

In the specific implementation process, the network proposed by the present invention is built using the Tensorflow deep learning framework, which can be trained on a Geforce GTX 1080Ti GPU (graphics processing unit). When the sample size is 128x128, updating the parameters once can use 128 images for one iteration. On test sets ranging in size from 240×160 to 1000×1000 pixels, it takes an average of 17.75 ms to detect whether an image has been tampered with, and an average of 99.84 ms to locate a tampered image.

In the specific implementation process, the present invention uses multiple public datasets for training and testing, including four commonly used datasets, CASIAv1.0, CASIA v2.0, NC 2016 and Columbia Umcompressed. The average results are obtained after five training tests are performed on the model. Table 1 shows the average classification accuracy, average pixel classification accuracy, and average intersection ratio on the test set. Accuracy is the ratio of the number of correctly classified samples to the total number of samples. The intersection and union ratio refers to the ratio of the intersection and union of the real tampered area and the predicted tampered area.

Table 1

In the specific implementation process, the results of some positioning test samples are shown in Figure 4, and the white pixels represent the tampered areas. Due to the removal of the global pooling layer of the convolutional network, the fully connected layer outputs a 2-channel feature map, which is visualized as the fourth column. It can be seen that this layer outputs effective spatial decision information and roughly gives the predicted position. Deconvolutional networks use this information and through dense connections to shallow networks, further accurate predictions. The dense structure convolutional neural network proposed by the present invention can effectively identify the tampering means of splicing, copying, moving, and removing, and output the result of pixel-by-pixel classification, which can accurately predict the tampered object, size, and shape, and is close to the real annotation.

Obviously, the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those of ordinary skill in the art, changes or modifications in other different forms can also be made on the basis of the above description. There is no need and cannot be exhaustive of all implementations here. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (10)

1. an image tampering detection and positioning method using a dense structure convolutional network, is characterized in that, comprises the following steps: S1: Input the image to be tested, perform preprocessing with spatial enrichment SRM convolution to obtain the preprocessed image; S2: Construct a densely connected convolutional network to extract the tampered image features of the preprocessed image, obtain the binary classification information of the image to be tested, and complete the detection of image tampering; S3: Construct a deconvolutional network that is symmetrical with the connected convolutional network structure, and uses the binary information of the image to be tested as input to locate the image tampering area; S4: According to the obtained image tampering area, the image after positioning is completed by the deconvolution network, and the positioning of the image tampering is completed. 2. A method for image tampering detection and localization using a densely structured convolutional network according to claim 1, wherein in the step S2, the densely connected convolutional network comprises a pooling layer, a dense layer, transition layer, global average pooling layer, and fully connected layer; where: The pooling layer performs a convolution and maximum pooling operation on the preprocessed image, and inputs the result into the dense layer; The dense layer and the transition layer are provided with multiple layers, and the output results of each dense layer are input into the corresponding transition layer, and finally the tampered image feature map obtained by the last transition layer is input into the global average pooling layer. middle; The global average pooling layer performs average pooling on the tampered image feature map, and the fully connected layer calculates and outputs two probability values, which represent the probability of tampering and non-tampering respectively, and obtains the binary classification information of the image to be tested. 3. An image tampering detection and localization method using a dense structure convolutional network according to claim 2, wherein the dense layer comprises a plurality of basic structure layers, and each basic structure layer consists of two consecutive layers. It consists of convolutional layers, in which the input of each basic structure layer is merged with the output of the previous layer, which is a locally dense version of the residual structure. 4. a kind of image tampering detection and positioning method adopting dense structure convolutional network according to claim 3, is characterized in that, described densely connected convolutional network is provided with four dense layers, respectively comprises 5, 10, 20, 12 basic structural layers. 5. An image tampering detection and localization method using a densely structured convolutional network according to claim 4, wherein the transition layer comprises a convolutional layer, which pre-rolls the feature map input by the dense layer Product once, and then perform average pooling to reduce the size of the image. 6. a kind of image tampering detection and positioning method adopting dense structure convolutional network according to claim 5, is characterized in that, described fully connected layer calculates and outputs two probability values by softmax function, and concrete calculation formula is:

where i represents the two categories of tampering/non-tampering,

Represents the output value of the network on category i, _yi represents the real value of the sample on category i, and a _i represents the weight of category i. 7. The image tampering detection and localization method using a densely structured convolutional network according to claim 2, wherein in the step S3, the deconvolutional network comprises a fully connected layer, a dense layer and a The corresponding deconvolution transition layer; firstly, the fully connected layer is used to calculate the tampered image features point by point, and then the image is restored layer by layer through the dense layer and the corresponding deconvolution transition layer to locate the image tampering area. 8. A method for image tampering detection and positioning using a densely structured convolutional network according to claim 7, wherein in the step S4, according to the image tampering area, the output to be tampered with is output by the deconvolution network. The binary image after image location is measured to complete the location of image tampering. 9. a kind of image tampering detection and positioning method adopting dense structure convolutional network according to claim 8 is characterized in that, the training process of described densely connected convolutional network and deconvolutional network is specifically: Collect training image data and perform preprocessing; Divide the preprocessed image data into training set and test set; Use the training set to pre-train 128x128 images, and calculate the gradient update parameters; The weights of the densely connected convolutional network are obtained by training the full size image according to the gradient update parameters; Pre-train the deconvolution network with 128x128 images according to the weight of the densely connected convolutional network, and calculate the gradient update parameters; Complete the training of the deconvolution network by training the full-size image according to the calculated gradient update parameters; Use the test set to evaluate and adjust the deconvolution network, and finally output the densely connected convolution network and deconvolution network corresponding to the weights. 10. The image tampering detection and positioning method using a densely structured convolutional network according to claim 9, characterized in that, in the training and adjustment process of the densely connected convolutional network and the deconvolutional network, a five-fold cross is used. The verification method is adjusted. Each time, one-fifth of the preprocessed image data is taken as the test set and four-fifths as the training set. After five training evaluations, the average training result is taken as the final result.

Images