CN111027576B - Co-saliency detection method based on co-saliency generative adversarial network - Google Patents

Abstract

The present invention proposes a collaborative saliency detection method based on a collaborative saliency generative adversarial network. The steps are as follows: constructing a collaborative saliency generative adversarial network model; performing two-stage training on the collaborative saliency generative adversarial network model: in the first step In the first training stage, the collaborative saliency generative adversarial network is pre-trained with a labeled salient target database. After this stage of training, the collaborative saliency generative adversarial network has the ability to detect the saliency of a single image; in the second In the training stage, based on the model parameters trained in the first stage, the synergistic saliency generative adversarial network is trained by using the labeled synergistic saliency data group whose synergistic saliency targets belong to the same category. After this stage of training, the trained adversarial network can be directly used The model performs class co-saliency detection. The invention has the advantages of simple training process, high detection efficiency, strong versatility and high accuracy.

Description

Co-saliency detection method based on co-saliency generative adversarial network

technical field

The invention relates to the technical fields of computer vision and machine learning, and in particular to a collaborative saliency detection method based on a synergistic saliency generative confrontation network.

Background technique

With the advent of the era of big data, the emergence of websites and mobile storage devices that can provide various information resources, a large number of digital information of images and videos are flooding our lives, how to enable computers to quickly and accurately acquire and retain more effective information? Ability appears to be necessary. Collaborative saliency detection is based on the human biological visual attention mechanism. By detecting multiple related scene images and common saliency targets in videos, it can extract effective target information that can represent such images, automatically filter redundancy and noise in images, and reduce algorithms. The time and space complexity is reduced, so as to realize the preferential allocation of computing resources and improve the execution efficiency of subsequent image tasks.

There are many existing collaborative saliency detection methods. The extraction of saliency features of a single image and the capture of similarity cues between multiple images are the key links involved in this task. With the development of deep learning, existing collaborative saliency methods can be divided into two categories according to whether the method adopts deep learning techniques or not. Non-deep learning-based methods often perform collaborative saliency detection based on some hand-designed features and artificially set similarity measurement criteria, resulting in the extracted features and similarity information that limit the detection performance and have a greater impact on the detection accuracy. Another type of deep learning-based collaborative saliency detection method is more effective than the traditional collaborative saliency detection method, and the information extracted by the deep model is more effective, which greatly improves the performance of collaborative saliency detection. However, the existing annotated collaborative saliency data is limited in size, which restricts the application of deep learning technology.

SUMMARY OF THE INVENTION

Aiming at the technical problem that the existing deep learning-based collaborative saliency detection method is limited by the insufficient amount of training data, which greatly affects the detection accuracy, the present invention proposes a collaborative saliency detection method based on a collaborative saliency generative adversarial network. , which effectively utilizes the saliency of a single image and the internal correlation information of the same category of images to perform collaborative saliency detection between images of the same category. The training and detection process is simple and the detection efficiency is high.

In order to achieve the above purpose, the technical solution of the present invention is achieved as follows: a collaborative saliency detection method based on a collaborative saliency generative adversarial network, the steps of which are as follows:

Step 1: Build a synergistic saliency generative adversarial network model: Design the network architecture of the generator and discriminator in the collaborative saliency generative adversarial network according to the characteristics of the collaborative saliency detection task, and build a synergistic saliency generative adversarial network model ;

Step 2: Two-stage training of the synergistic saliency generative adversarial network model: In the first training stage, the labeled salient target database is used to pre-train the synergistic saliency generative adversarial network. In the first stage, the trained model parameters are used to train the synergistic saliency generative adversarial network using the synergistic saliency data set whose synergistic salient targets belong to the same category;

Step 3: Category collaborative saliency detection: The generator of the collaborative saliency generative adversarial network model trained in step 2 is used as the category collaborative saliency detector, and images belonging to the same category are used as the category collaborative saliency detector. Input, direct end-to-end output of the collaborative saliency map corresponding to the same category of images.

In the first step, the generator network adopts the U-Net network structure, which is a fully convolutional network, wherein the convolution kernel size, step size and padding value of the convolutional layer and the deconvolutional layer are symmetrically set, and the last three-layer volume is set. The product layer and the first three deconvolution layers are equipped with a Dropout operation; the network of the discriminator is also a full convolution network. After the multi-layer convolution operation, a two-dimensional probability matrix is output, and the discriminator is determined according to the two-dimensional probability matrix. The input image is subjected to Patch-level true and false discrimination; the generator learns the mapping relationship between the original image and the collaborative saliency map, thereby generating the collaborative saliency map, and the discriminator compares the collaborative saliency map generated by the generator with the ground truth map. Distinguish between true and false.

In the first training stage and the second training stage in the second step: when training the generator, the network parameters of the discriminator are fixed, the probability that the image generated by the generator is judged to be true by the discriminator is increased, and the parameters of the generator are updated; When the generator is used, the parameters of the generator are fixed, so that the discriminator increases the probability that the real sample is judged to be true, reduces the probability that the generated pseudo sample is judged to be true, and updates the parameters of the discriminator.

In the first training stage and the second training stage, the loss function of the generator is:

LG=LG ₁ +λ·LG ₂ (1)

Among them, LG ₁ is the adversarial loss of the generator, LG ₂ is the pixel loss of the generator, λ is the coefficient for adjusting the loss weight; θ ^G is the network model parameter of the generator; the adversarial loss LG ₁ of the generator is:

LG ₁ =BCE(D(G(I _m ),I _m ),A _real ) (3)

BCE(x,y)=y·logx+(1-y)·log(1-x) (4)

The pixel loss LG ₂ of the generator is:

LG ₂ =||S _m -G(I _m )|| ₁ (5)

Among them, Im and S _m respectively represent the input _m -th original image and its corresponding saliency target ground truth map, G( ) represents the pseudo saliency map generated by the generator, D( , ) represents the output of the discriminator Two-dimensional probability matrix, A _real is a two-dimensional matrix whose elements are all 1, and the size is consistent with the probability matrix D(·,·); the function BCE(·,·) is used to calculate the two-dimensional probability matrix D(·,· ) and the binary cross-entropy of the two-dimensional matrix A _real , the expression of the function BCE(·,·) is shown in formula (4), where x and y are the independent variables of the function BCE(·,·); LG ₂ is significant 1-norm loss of target ground-truth map and generated image;

The loss function of the discriminator is expressed as:

LD=BCE(D(S _m ,I _m ),A _real )+BCE(D(G(I _m ),I _m ),A _fake ) (6)

Among them, θ ^D represents the network model parameters of the discriminator, A _fake is a two-dimensional matrix whose elements are all 0, and the size is consistent with the size of the probability matrix D(·,·).

In the first training stage, the labeled salient target database is used as a training sample set to train the synergistic saliency generative adversarial network, and the mapping relationship between the original image and the salient target truth map is automatically learned. The specific implementation method is as follows:

The images with pixel-level labels in the salient target database PASCAL-1500, HKU-IS and DUTS are used as training data, and all original images and the corresponding salient target ground-truth maps are resized to the input size of the generator, and the original images are input The generator obtains a pseudo saliency map of the same size. The pseudo saliency map and the saliency target ground truth map are compared at the pixel level, and the pseudo saliency map and the original image are spliced according to the number of channels as pseudo samples, and the saliency target ground truth map is spliced with the original image. As real samples, they are respectively sent to the discriminator; wherein, by minimizing the loss function, the Adam optimization algorithm is used to iteratively update the generator network model parameters θ ^G and the discriminator network model parameters θ ^D .

In the second step, the image groups are divided according to the categories of the common salient objects contained in the images, and based on the model parameters trained in the first stage, the corresponding image groups are used for a certain category to perform the first step on the synergistic saliency generative adversarial network. The second-stage training enables the collaborative saliency generative adversarial network to learn the mapping relationship between the original image and the collaborative saliency ground-truth map. The specific implementation method is as follows:

Using CoSal2015, iCoseg and MSRC-A, a total of 3 public co-saliency detection databases that have been grouped by salient object categories, adjust the size of all original images and the corresponding co-saliency ground-truth maps as the input of the generator size, directly and randomly select 50% of the image data in each group as the category training samples, and train the synergistic saliency generative adversarial network trained in the first stage, so that the generator automatically learns to extract the public saliency information in the category samples , to train a co-saliency detection model for a single class of images.

For any image, if it belongs to a certain category of the training sample categories used in the second-stage training, it will be sent to the generator of the co-saliency detection model of the corresponding category that has been trained, and it needs to be entered before input. The size is adjusted to the input size of the generator, and the output image of the generator is the co-saliency map of the input image:

CoS _m = G ^* (I _m ) (8)

Among them, G ^* ( ) represents the generator of the co-saliency generative adversarial network trained in two stages, and CoS _m is the co-saliency map finally generated by the image I _m to be detected.

Compared with the prior art, the present invention has the beneficial effects: based on the synergistic saliency generative adversarial network, a two-stage training mechanism is used to detect the synergistic saliency. Insufficient volume can cause a series of training problems. 2) A two-stage training mechanism is adopted, and the salient target data is used as the first-stage training data to ensure that the network trained in this stage has the ability to detect the saliency of a single image. Using the memory function of the network, the network has reserved a single salient target detection. On the basis of the ability, the collaborative saliency image group is used as the second stage training data, so that the network has the ability to capture the correlation information between the images of the same category, and the saliency detection information of a single image and the correlation between the images in the same group. The information is fused, so that the network finally has the ability of category collaborative saliency detection. It can be seen from experiments that the training and detection process of the present invention is simple and the detection efficiency is high, the versatility and accuracy of the collaborative saliency detection can be significantly improved, and the rapid acquisition of key information is of great significance.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

FIG. 1 is a flow chart of the present invention.

FIG. 2 is a comparison diagram of subjective results between the present invention and the existing algorithm on the CoSal2015 database.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

As shown in Figure 1, a collaborative saliency detection method based on collaborative saliency generative adversarial network uses the trained collaborative saliency generative adversarial network to achieve collaborative saliency detection. The steps are as follows:

Step 1: Build a synergistic saliency generative adversarial network model: Design the network architecture of the generator and discriminator in the collaborative saliency generative adversarial network according to the characteristics of the collaborative saliency detection task, and build a synergistic saliency generative adversarial network model .

The synergistic saliency generative adversarial network model includes a generator network model and a discriminator network model. An appropriate generator network model is designed to learn the mapping relationship between the original image and the co-saliency map, so that the co-saliency map generated by the generator is as close as possible to the co-saliency ground-truth map. Design a suitable discriminator network model to make it possible to distinguish the true and false of the co-saliency map generated by the generator from the co-saliency ground truth map, so as to assist the training of the generator.

The generator network model adopts the U-Net network structure as a whole, which is a fully convolutional network, including 8 convolutional layers and 8 deconvolutional layers symmetrically set to ensure that the image output by the generator is the same size as the input image, The convolution kernel size, stride and padding value of the convolutional layer and deconvolutional layer are set symmetrically. The last three convolutional layers and the first three deconvolutional layers have a Dropout operation with a value of 0.5. The activation output is randomly 50% zeroed, which can effectively prevent overfitting. In the generator network model architecture, on the basis of the symmetry of the encoder-decoder network structure, short connections are added, and feature maps of the same size are cascaded. O, Fischer P, Brox T, et al. U-Net: Convolutional Networks for Biomedical ImageSegmentation. in Proc. Medi. Image Comput. and Comput. Assis., 2015, pp. 234-241). The purpose of the cascade is to ensure that the generated image retains details such as object edges. Therefore, the network structure of generator G is shown in Table 1.

Table 1 The network structure of the generator

The discriminator network module is designed with an encoder structure, which is also a fully convolutional network. The network structure of discriminator D is shown in Table 2. The output of the discriminator is a 28×28 two-dimensional matrix, and each element on the matrix corresponds to its The probability that the image block of the input image is judged to be true, that is, the patch-level judgment of the input image is performed, see the literature (Phillip Isola, Jun-YanZhu, Tinghui Zhou, et.al,. Image-to-Image Translation with Conditional Adversarial Networks .arXiv preprint arXiv:1611.07004).

Table 2 The network structure of the discriminator

Network layer enter convolution kernel size step size fill value output Convolutional layer 1 256×256×6 4 2 1 128×128×64 Convolutional layer 2 128×128×64 4 2 1 64×64×128 Convolutional layer 3 64×64×128 4 2 1 32×32×256 Convolutional layer 4 32×32×256 3 1 0 30×30×512 Convolutional layer 5 30×30×512 3 1 0 28×28×1

Step 2: Two-stage training of the synergistic saliency generative adversarial network model: In the first training stage, a labeled salient target database with abundant data is used to pre-train the synergistic saliency generative adversarial network, and in the second In the training phase, based on the model parameters trained in the first stage, the collaborative saliency data set with the collaborative saliency targets belonging to the same category is used to train the collaborative saliency generative adversarial network for class collaborative saliency detection.

The present invention designs a two-stage training mechanism. In the first training stage, pre-training enables the generator in the synergistic saliency generative confrontation network to have the preliminary single-image salient target detection ability, and the second training stage further enables the generator to detect multiple images. The co-salient object capability of this class of images.

During the two-stage training process, the loss function of the generator is expressed as:

LG=LG ₁ +λ·LG ₂ (1)

The loss function of the generator consists of an adversarial loss and a pixel loss. LG ₁ and LG ₂ represent the adversarial loss and pixel loss of the generator in the two-stage training, respectively. λ is a coefficient for adjusting the loss weight. The present invention sets λ=100, θ ^G is the network model parameter of the generator G. When training the generator, the parameters of the discriminator are fixed, and the probability that the image generated by the generator is judged to be true by the discriminator is increased as much as possible, so as to update the parameters of the generator. Among them, the adversarial loss of the generator is expressed as:

LG ₁ =BCE(D(G(I _m ),I _m ),A _real ) (3)

BCE(x,y)=y·logx+(1-y)·log(1-x) (4)

The pixel loss of the generator is expressed as:

LG ₂ =||S _m -G(I _m )|| ₁ (5)

Among them, Im and S _m represent the input _{m-th original image and its corresponding saliency ground truth map, respectively. Im and S m in the} first stage come from the saliency target database required for training, and I in the second stage _m and S _m come from the collaborative saliency database required for training, G(·,·) represents the pseudo-saliency map generated by the generator, and the original image _Im and G(·,·) pseudo-saliency map are spliced together as a probability moment D The input of , D(·,·) represents the probability matrix of the output of the discriminator. The parameter θ ^D of the probability matrix D(·,·) is the weight and bias of all neurons between the convolutional networks in the trained discriminator network, which is determined by the network structure, and the output of the discriminator is a two-dimensional probability matrix. The input image size of the discriminator is 256×256×3, and the output is a two-dimensional probability matrix of 28×28. Then each element on the matrix is a probability value, which corresponds to a size of about 9×9 according to the position. The image block is judged as a true probability value by the discriminator, and the probability value range is between [0, 1]. A _real is a two-dimensional matrix whose elements are all 1, and its size is consistent with the probability matrix D(·,·), where each element corresponds to an image block in the input sample, and represents the corresponding image block discriminator The probability of being true is all 1. The function BCE(·,·) is used to calculate the binary cross-entropy between the probability matrix D(·,·) and the two-dimensional matrix A _real , and its function expression is shown in formula (4), where x and y are the function BCE(· , ) independent variable. LG ₂ is the 1-norm loss of the salient object ground-truth map and the generated image.

In the two-stage training process, the discriminator loss function is expressed as:

LD(θ ^D )=BCE(D(S _m ,θ ^D ),A _real )+BCE(D(G(I _m ,θ ^G ),θ ^D ),A _fake ) (6)

In the above formula (6), θ ^D represents the network model parameters of the discriminator, A _real is a two-dimensional matrix whose elements are all 1, and A _fake is a two-dimensional matrix whose elements are all 0. Its size is related to the probability matrix D(· , ) are the same size. When training the discriminator, the parameters of the generator are fixed, so that the discriminator can increase the probability that the real sample is judged to be true as much as possible, and reduce the probability that the generated pseudo sample is judged to be true, so as to update the parameters of the discriminator.

In the first training stage, the collaborative saliency generative adversarial network is trained in the first stage using the salient target database with abundant data and labels as the sample set, so that the model can learn the mapping between the original image and the salient target ground-truth map. relationship, so that it firstly has the ability to detect salient objects for a single image. Based on the model parameters trained in the first stage, the model inherits the ability to detect the saliency of a single image, divides image groups according to the categories of common salient objects contained in the image, and uses the corresponding image group for a certain category to synergistically saliency The second-stage training of the generative adversarial network. The co-saliency generative adversarial network learns the mapping relationship between the original image and the co-saliency ground-truth map, so that the trained class co-saliency generative adversarial network has the ability to detect common salient objects among multiple images.

The first stage of training: A total of 21,517 images with pixel-level labels in the salient target database PASCAL-1500, HKU-IS and DUTS are used as training data. Resize all original images and the corresponding saliency object ground-truth maps to 256×256×3 size, and input the original images into the generator. The size of the generated image is 256 × 256 × 3, the generated image is compared with the salient target ground truth map for pixel-level image, and the generated image and the original image are spliced according to the number of channels as a pseudo sample, and the saliency target ground truth map and the original image are spliced as a pseudo sample. The real samples are sent to the discriminator respectively. Among them, the Adam optimization algorithm is used to iteratively update the model parameters, and the Batchsize, learning rate, Dropout rate and Epoch are set to 1, 0.0002, 0.5 and 100, respectively. Batchsize refers to the number of samples used to update network parameters once during the training process, learning rate refers to the magnitude of each model parameter training update during the training process, and Epoch refers to the number of times all training samples are traversed during the training process. The dropout operation is only added in the 6-8 convolutional layers and the 1-3 deconvolutional layers, which makes the generator network structure have a certain robustness.

The second stage of training: On the basis of the parameters of the synergistic saliency generative adversarial network model trained in the first stage, the parameters trained in the first stage are used as the initial parameters of the second stage training, and the training model has the ability to detect the category synergistic saliency . Model training was performed using three public collaborative saliency detection databases, CoSal2015, iCoseg and MSRC-A. Before training, the image size of all original images and the corresponding salient target ground-truth maps are adjusted to 256×256. The images in these three databases have been grouped according to whether the collaborative salient targets belong to the same class, so the group is directly selected randomly. 50% of the data are trained in the second stage. During the training of the same category of samples, the model is prompted to learn to extract the common saliency information in the category of samples, so as to train a collaborative saliency detection model for a single category of images. The training involves the same parameter settings as the first-stage training, except that the parameter for Epoch is 400.

Step 3: Category collaborative saliency detection: Use the generator of the collaborative saliency generative adversarial network model trained in step 2 as a category collaborative saliency detector, and take images belonging to the same category as the training samples in the second stage as the category The input of the co-saliency detector directly outputs the co-saliency map corresponding to the image end-to-end.

The generator in the two-stage training co-saliency generative adversarial network model is directly used as the category co-saliency detector. For any image, if it belongs to one of the training sample categories used in the second-stage training, Then it is sent to the generator of the co-saliency detection model of the corresponding category that has been trained. Before input, the image size is unified to 256×256×3, and the image output by the generator is the co-saliency map of the input image. The size is also 256×256×3. The image generated by the generator is directly used as a collaborative saliency map, as shown in the following formula:

CoS _m = G ^* (I _m ) (8)

G ^* ( ) represents the generator of the two-stage trained co-saliency generative adversarial network, and CoS _m is the co-saliency map finally generated by the image I _m to be detected.

So far, the collaborative saliency detection of a group of target image groups containing the same category is completed, that is, the collaborative saliency detection of images is completed.

The present invention has carried out experiments on workstations whose hardware environment is Intel(R) XeonE5-2650 v3@2.30Hz×20CPU, NVIDIA GTXTITAN-XPGPU, and 128G video memory, and the running software environment is: Ubuntu16.04 and deep learning framework Pytorch1.0 .

In order to verify the detection performance and efficiency of the present invention, the method of the present invention and 6 collaborative saliency detection methods were compared on the CoSal2015 database to compare the detection time of each image and the subjective results. On the CoSal2015 database, under the same hardware environment, the detection time of the four methods of SACS-R, SACS, CBCS and ESMG of the public code and the present invention are compared, as shown in Table 3.

Table 3. Comparison of detection time of existing algorithms on CoSal2015 database

algorithm SACS-R SACS CBCS ESMG this invention code type MATLAB MATLAB MATLAB MATLAB Python Detection time 8.873 seconds 2.652 seconds 1.688 seconds 1.723 seconds 0.562 seconds

The methods involved in subjective comparison are derived from literature (D.Zhang, J.Han, C.Li et al.Co-saliency detection via looking deep and wide.in Proc.IEEE Conf.Comput.Vis.PatternRecognit., 2015, pp. 2994-3002) LDAW, from the literature (X.Cao, Z.Tao, B.Zhang et al.Self-Adaptively Weighted Co-Saliency Detection via Rank Constraint.IEEETrans.Image Process.vol.23,no.9, pp.4175-4186, 2014) of SACS-R and SACS, from the literature (J.Han, G.Cheng, Z.Li et al.A Unified Metric Learning-Based Framework for Co-Saliency Detection.IEEE Trans.on AUM of Cir.and Sys.for Vid.Tech., vol.28, no.10, pp.2473-2483, 2018), from the literature (H.Fu, X.Cao, and Z.Tu, Cluster-Based Co-Saliency Detection. IEEE Trans. Image Process., vol. 22, no. 10, pp. 3766-3778, 2013) CBCS and from the literature (A. Joulin, K. Tang, and L. Feifei. Efficient Image and VideoCo-localization with Frank-Wolfe Algorithm.in Proc.IEEE Eur.Conf.Comput.Vis., 2014, pp.253-268) ESMG, CoSal2015 database upper grouping -- subjective comparison results of starfish, frog and rubber as shown in picture 2.

It can be seen from FIG. 2 and Table 3 that the present invention has the shortest time for collaborative saliency detection for an image, that is, the highest efficiency. Compared with other existing methods, the collaborative saliency map obtained by the present invention is the closest to the true value map.

The invention includes the construction of a synergistic saliency generative adversarial network model and two-stage model training. In the first training stage, a marked salient target data set with a relatively rich amount of data is used as a training data set to alleviate the problem of the collaborative visual saliency field. At the same time, the trained network has the ability to detect the saliency of a single image. In the second stage, the model is trained with annotated collaborative saliency image group, and the model memory function is used to train the model. The model is further equipped with the ability to detect the saliency of a single image on the basis of the ability to detect the saliency of a single image; the generator trained in two stages is used as a collaborative saliency detector, and the collaborative saliency map of the category of images is output end-to-end. . The invention effectively utilizes the memory function of the network, the saliency of a single image and the internal correlation information of the same category of image groups to perform collaborative saliency detection among multiple images, with simple training and detection processes and high detection efficiency.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims (5)

1. a collaborative saliency detection method based on synergistic saliency generative adversarial network, is characterized in that, its steps are as follows: Step 1: Build a collaborative saliency generative adversarial network model: Design the network architecture of the generator and discriminator in the collaborative saliency generative adversarial network according to the characteristics of the collaborative saliency detection task, and build a collaborative saliency generative adversarial network model ; Step 2: Two-stage training of the synergistic saliency generative adversarial network model: In the first training stage, the labeled salient target database is used to pre-train the synergistic saliency generative adversarial network. In the first stage, the trained model parameters are used to train the synergistic saliency generative adversarial network using the synergistic saliency data set whose synergistic salient targets belong to the same category; Step 3: Category collaborative saliency detection: The generator of the collaborative saliency generative adversarial network model trained in step 2 is used as the category collaborative saliency detector, and images belonging to the same category are used as the category collaborative saliency detector. Input, direct end-to-end output of the collaborative saliency map corresponding to the same category of images; In the first training stage, the labeled salient target database is used as a training sample set to train the synergistic saliency generative adversarial network, and the mapping relationship between the original image and the salient target truth map is automatically learned. The specific implementation method is as follows: The images with pixel-level labels in the salient target database PASCAL-1500, HKU-IS and DUTS are used as training data, and all original images and the corresponding salient target ground-truth maps are resized to the input size of the generator, and the original images are input The generator obtains a pseudo saliency map of the same size. The pseudo saliency map and the saliency target ground truth map are compared at the pixel level, and the pseudo saliency map and the original image are spliced according to the number of channels as pseudo samples, and the saliency target ground truth map is spliced with the original image. As real samples, they are respectively sent to the discriminator; wherein, by minimizing the loss function, the Adam optimization algorithm is used to iteratively update the generator network model parameters θ ^G and the discriminator network model parameters θ ^D ; In the second step, the image groups are divided according to the categories of the common salient objects contained in the images, and based on the model parameters trained in the first stage, the corresponding image groups are used for a certain category to perform the first step on the synergistic saliency generative adversarial network. The second-stage training enables the collaborative saliency generative adversarial network to learn the mapping relationship between the original image and the collaborative saliency ground-truth map. The specific implementation method is as follows: Using CoSal2015, iCoseg and MSRC-A, a total of 3 public co-saliency detection databases that have been grouped by salient object categories, adjust the size of all original images and the corresponding co-saliency ground-truth maps as the input of the generator size, directly and randomly select 50% of the image data in each group as the category training samples, and train the synergistic saliency generative adversarial network trained in the first stage, so that the generator automatically learns to extract the public saliency information in the category samples , to train a co-saliency detection model for a single class of images. 2. the collaborative saliency detection method based on synergistic saliency generative confrontation network according to claim 1, is characterized in that, in described step 1, the network of generator adopts U-Net network structure, is full convolution network, The convolution kernel size, stride and padding value of the convolutional layer and deconvolutional layer are set symmetrically, and the last three convolutional layers and the first three deconvolutional layers are equipped with Dropout operations; the network of the discriminator is also It is a fully convolutional network. After multi-layer convolution operation, a two-dimensional probability matrix is output, and according to the two-dimensional probability matrix, the image input by the discriminator is subjected to Patch-level true and false discrimination; the generator learns the original image and the collaborative significant true value. The mapping relationship between the graphs to generate a collaborative saliency map, and the discriminator distinguishes between the true and false values of the collaborative saliency map generated by the generator and the ground truth map. 3. The collaborative saliency detection method based on collaborative saliency generative adversarial network according to claim 1 or 2, characterized in that, in the first training stage and the second training stage in the step 2: when training the generator , fix the network parameters of the discriminator, improve the probability that the image generated by the generator is judged to be true by the discriminator, and update the parameters of the generator; when training the discriminator, fix the parameters of the generator, so that the discriminator improves the real sample to be judged as true Probability, reduce the probability that the generated pseudo samples are judged to be true, and update the parameters of the discriminator. 4. The collaborative saliency detection method based on collaborative saliency generative adversarial network according to claim 3, characterized in that, in the first training stage and the second training stage, the loss function of the generator is: LG=LG ₁ +λ·LG ₂ (1)

Among them, LG ₁ is the adversarial loss of the generator, LG ₂ is the pixel loss of the generator, λ is the coefficient for adjusting the loss weight; θ ^G is the network model parameter of the generator; the adversarial loss LG ₁ of the generator is: LG ₁ =BCE(D(G(I _m ),I _m ),A _real ) (3) BCE(x,y)=y·logx+(1-y)·log(1-x) (4) The pixel loss LG ₂ of the generator is: LG ₂ =||S _m -G(I _m )|| ₁ (5) Among them, Im and S _m respectively represent the input _m -th original image and its corresponding saliency target ground truth map, G( ) represents the pseudo saliency map generated by the generator, D( , ) represents the output of the discriminator Two-dimensional probability matrix, A _real is a two-dimensional matrix whose elements are all 1, and the size is consistent with the probability matrix D(·,·); the function BCE(·,·) is used to calculate the two-dimensional probability matrix D(·,· ) and the binary cross-entropy of the two-dimensional matrix A _real , the expression of the function BCE(·,·) is shown in formula (4), where x and y are the independent variables of the function BCE(·,·); The loss function of the discriminator is expressed as: LD=BCE(D(S _m ,I _m ),A _real )+BCE(D(G(I _m ),I _m ),A _fake ) (6)

Among them, θ ^D represents the network model parameters of the discriminator, A _fake is a two-dimensional matrix whose elements are all 0, and the size is consistent with the size of the probability matrix D(·,·). 5. The collaborative saliency detection method based on collaborative saliency generative adversarial network according to claim 1, characterized in that, for any image, if it belongs to a certain category of the training sample categories used in the second stage training , then it is sent to the generator of the co-saliency detection model of the corresponding category that has been trained, and its size needs to be adjusted to the input size of the generator before input, and the image output by the generator is the co-saliency map of the input image. : CoS _m = G ^* (I _m ) (8) Among them, G ^* ( ) represents the generator of the co-saliency generative adversarial network trained in two stages, and CoS _m is the co-saliency map finally generated by the image I _m to be detected.

Images