CN111091492B - Face image illumination migration method based on convolutional neural network - Google Patents
Face image illumination migration method based on convolutional neural network Download PDFInfo
- Publication number
- CN111091492B CN111091492B CN201911335861.4A CN201911335861A CN111091492B CN 111091492 B CN111091492 B CN 111091492B CN 201911335861 A CN201911335861 A CN 201911335861A CN 111091492 B CN111091492 B CN 111091492B
- Authority
- CN
- China
- Prior art keywords
- illumination
- image
- face
- migration
- data set
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
A face image illumination migration method based on a convolutional neural network realizes illumination migration of a face image by using a convolutional neural network CNN. The method is mainly realized by two parts: the method comprises the following steps of illumination model training, illumination classification, illumination matching and illumination migration realized on the basis of image style migration reference. Firstly, the illumination classification is completed on a Yale Face data set and a PIE Face data set by combining a convolutional neural network VGG19 and VGG16, and a model capable of classifying the illumination of Face images is obtained; then, the model is utilized to realize illumination matching of a single face image, and an image similar to the illumination of the given face image can be obtained from the illumination data set of the face image; and finally, extracting and processing related illumination characteristics of the given reference face illumination image through an illumination classification model so as to be convenient for transferring to the input face image, thereby realizing the integral transfer of the illumination of the single face image.
Description
Technical Field
The invention discloses a face image illumination migration method based on a convolutional neural network, and belongs to the field of computer vision.
Background
The lighting effect of images is a research hotspot in a plurality of research directions in the field of computer vision, and is also important for human images. The effect of light and shadow has wide application requirements in the aspects of art design such as modern digital film and television production, portrait photography beautification, advertisement and the like. One of the difficulties in movie production is how to capture the best lens performance of actors, and the illumination effect in the scene is the most ideal effect. The problem of improving the shooting process is solved at high cost, and the problem of post-production is solved at complex technical processing means. In the aspect of beautifying portrait photography, the similar problem of movie and television production is faced, the cost of setting ideal photography illumination scenes for professional photography is too high and is also influenced by factors such as the personal ability of a photographer, and the later use of professional image processing software also needs time and labor cost and is very complex.
The illumination migration of the face image provides a simple solution to the above problem. In the aspect of image illumination, the method can transfer the illumination effect to the target image only by giving the image with the ideal illumination effect, so that the target image can obtain the ideal illumination effect. In the video image problem, it requires an image or video given a desired lighting effect, and the lighting is migrated to a target image or video, one or both of which are video files. The illumination migration aims at generating the target image or video with the ideal illumination effect in one step only by providing the ideal illumination effect without complex operation means and modes, and time, labor and resource cost are saved. The existing human face image illumination transfer method only changes the illumination of the human face image face and does not process the illumination of the neck and background parts of the human face image. In an actual application scene, the face part and the non-face part are inseparable, and only the illumination of the face part is changed, so that the actual application requirement is not completely met. Aiming at the problem, the invention researches a method capable of carrying out illumination migration on the whole human face image. The method can enable the target image to obtain an ideal illumination effect and simultaneously be closer to an actual illumination migration application scene.
Disclosure of Invention
The technical problem of the invention is solved: the face image illumination migration based on the convolutional neural network is proposed on the basis of the traditional face makeup and style migration. After the deep neural network is combined, the illumination migration effect is better and closer to the illumination effect of a real image, the process is simple, and the function is stronger.
The technical solution of the invention is as follows: a face image illumination migration method based on a convolutional neural network comprises the following steps:
step 1, preparing and establishing a data set;
step 2, preparing a pre-training model;
step 3, obtaining an illumination classification model;
step 4, light matching based on the light classification model;
step 5, light migration based on the light classification model is specifically as follows:
step 1, preparation and establishment of a data set: the current illumination classification detailed data sets comprise a YaleFace data set and a PIE data set. The Yale Face data set B includes 10 photographic subjects, 9 males and 1 female, which are black and white images. Each photographic subject contained 9 poses, 64 lighting effects in each pose, and a total of 5760 human face lighting images containing backgrounds. The two data sets total 21888 human face images. The PIE face dataset is a color face dataset, and contains a total of 68 photographic subjects, including male, female, and all races. The human face illumination images without background light in 13 postures and the human face illumination images with background light in 3 postures respectively comprise 21 illuminations in each posture, and are classified according to the positions of the camera in the x direction, the y direction and the z direction. The face illumination image portion of the data set consisted of 22848 pictures, with 1428 pictures that included a backlight and were oriented facing the camera.
The above two data sets are both small, and for the Yale Face data set, the Face data of 28 photographic objects are used for training, and the Face data of 10 photographic objects are used for testing. For the PIE face data set, face data of 50 photographic subjects were used for training and face data of 18 photographic subjects were used for testing. The background illumination of all images containing background light in the PIE data set is close, interference is caused to learning classification face illumination, and therefore image data are subjected to matting processing.
Step 2, preparation of a pre-training model: there are two methods for training classification tasks, one is to start training from an initial state, and the other is to perform transfer learning. On the basis of a small data set, the model effect obtained by using the pre-training model migration learning is better than that of random initialization. Therefore, the invention trains the illumination classification model in a transfer learning mode. Aiming at the Yale Face data set, a pre-training model of an object classification data set ImageNet provided by VGG19 convolutional neural network and Matconvnet is adopted. For the PIE Face data set, a pre-training model of a Face recognition model Vgg Face provided by a convolutional neural network VGG16, Matconvnet is used. Matconvnet is a tool kit which can use a convolutional neural network in Matlab, and has abundant pre-training model resources.
And 3, obtaining an illumination classification model: the method comprises the steps of firstly removing the last full-link layer of an original network according to a classification task, adding a new softmax layer, a top1-error layer and a top5-error layer for training, adding a new full-link layer according to the category number of a data set, wherein a Yale Face data set is 64 types, and a PIE data set is 21 types. Then define the whole network, fine-tune the weights of the first few layers of the network, and set the learning rate to 1 × 10-4And iterating for 300 rounds to finish the task of illumination classification of the face image. The average accuracy of the illumination classification model trained by the VGG16 network and the VGG Face pre-training model on the PIE Face data set test set is 62.8571%. The average accuracy of the illumination classification model trained by the VGG19 network and the ImageNet pre-training model on the Yale Face data set test set is 94.375%, and the classification accuracy of 48 types of illumination effects in 64 types of illumination is 100%.
And 4, light matching based on the light classification model: and (3) illumination matching, namely, giving a human face illumination input image, searching an image with the same illumination effect as the input image in the human face illumination image data set, and outputting the image with the same illumination. The key to illumination matching is how to compute illumination information representing an image. The VGG neural network adopted by the invention is composed of a convolutional layer, a full connection layer, a pooling layer and a softmax layer. The output of the convolutional layer operation is image characteristic information with large data, and it is difficult to extract illumination information from the image characteristic information, and the connotation of information obtained after the convolutional neural network processes the image cannot be clarified. And the output data dimension of the last full connection layer of the VGG network is consistent with the category number of the image, and the association with the image illumination information is easy to find out. Therefore, on the basis of the illumination classification model, the model trained by the YaleFace data set with high illumination classification accuracy and detailed classes is used for searching the association between the illumination information and the output of the last full-connection layer of the model, and the illumination matching of the face illumination image is completed. By observing the output of a large number of images through a network full-connection layer and the labels of image types, a certain rule exists between the output of the large number of images and the labels of the image types, and the dimension serial number corresponding to the maximum value in the output 64-dimensional data is the same as the serial number of the labels of the image types. Therefore, the illumination matching algorithm based on the illumination classification model is creatively proposed by the invention as shown in fig. 3.
And 5, light migration based on the light classification model: on the basis of an illumination classification model, the end-to-end single face image illumination migration is completed by combining a quotient image method of the traditional illumination migration research under the enlightenment of Neural Style. The traditional Neural Style adopts an input content graph and a Style graph, calculates the loss of the content graph and the Style graph respectively, and finally linearly combines the loss of the content graph and the Style graph to obtain a final Style transition graph. Based on the thought, the invention provides illumination migration based on an illumination classification model, and also provides an input image and a reference image required by the illumination migration, wherein the input image is generally a uniform front illumination image, the reference image is generally an image with obvious light and shadow difference, the input image and the reference image are input into an illumination migration network (VGG19) to obtain characteristic matrix values of the two images, and then an illumination quotient is solved according to the characteristic matrix values. And returning the shadow quotient and the expected result image to a migration network to minimize a migration loss function, and outputting the result image after 1000 iterations. In a specific experiment, it is found that the Style migration part in the traditional Neural Style cannot migrate the learning illumination information, so that only the content migration part is finally used for migrating the illumination information of the image.
Further, the shadow quotient solving process is as follows:
the ratio of a reference image and a target image after VGG network learning is calculated before a light quotient is calculated by a light migration method based on a light classification model, and the formula is as follows:
wherein, Fl[I]Is a feature matrix of the input image I at the convolution layer of the layer I, Fl[E]Is the characteristic matrix of the reference image E at the convolution layer of the layer I; a constant value is set to 0.0001, and the eigenvalue of the input image is divided by the eigenvalue of the eigen matrix value of the reference image to obtain a sum of the reference image and the sum of the eigenvalues of the input imageRatio S of target image after VGG network learninglMultiplying the ratio by the input image feature matrix to obtain a light and shadow quotient, wherein the formula is as follows:
Fl[M]＝Fl[I]×Sl
wherein, the invention is to Fl[M](Fl[M]Is the light-shadow quotient of the image M at the convolution layer of the layer l) is formulated as
The constraints shown improve. Wherein the constraint value rijExperiments prove that the illumination information can be better migrated within the constraint range and the structure and the content of the reference image can not be excessively migrated to the input image when the illumination information is 0.4 and 5.
Further, the migration loss function calculation process is as follows:
neural Style creatively proposes content loss and Style loss of pictures to represent migration loss of the content and Style of convolutional Neural network learning images, and the formula is as follows:
wherein l represents the presence of dimension D at the first layer convolution layerlN of vectorized feature mappinglA filter (D)lIs the number of elements in the filter response). Fl[.]∈RNl×DlIs the obtained feature matrix, and (i, j) is the index of the feature matrix. (F)l[I],Fl[E],Fl[O]) L is the total number of layers in the inspected network, L is the feature matrix of the input image I, the feature matrix of the reference painterly image E, and the feature matrix of the desired output result image O of the L-th convolutional layer, respectively (α)l，βl) Is a configured weight parameter. Is a weight that reconciles between the integrity of the input image content and the amount of painterly migration.
The migration loss function of the migration method of the present invention is:
Fl[O]for the desired resulting image, Fl[M]Calculating the distance between the light and shadow quotient and continuously reducing the difference between the light and shadow quotient to obtain natural light migration. After the experiment, the content migration part in the Neural Style is reserved to migrate the illumination information of the image. The convolution layers for illumination migration are determined to be convolution 1_2 layers and convolution 2_1 layers through experiments, and the lower convolution layer is sensitive to the content and structure information of the image and can retain the content information of the input image. The illumination migration method based on the illumination classification model can migrate illumination in multiple directions on the same camera object, comprises a positive light source, a left light source and a right light source, can migrate illumination, and has natural illumination.
Compared with the prior art, the invention has the advantages that:
(1) the rise of deep learning brings breakthrough to many researches in the field of computer vision. The style migration is typical, and the invention provides a method for performing illumination migration by using a convolutional neural network under the inspiration of style migration, compared with the traditional method, the method for performing face makeup migration and style migration has better effect and enables the illumination migration effect of a face image to be closer to the result of the illumination effect of a real image.
(2) The invention adopts a pre-training model to perform the illumination classification model training of transfer learning on the Yale Face data set and the PIE data set which are more detailed in illumination classification. Different pre-training models are used for different data sets, the neural structure of the pre-training models is improved, and the pre-training models are trained after fine adjustment so as to achieve a better illumination classification model.
(3) The invention carries out illumination matching on the basis of an illumination classification model, for a given face illumination input image, searches images with the same illumination effect as the input image in a face illumination image data set, and outputs the images with the same illumination. In order to obtain the correlation between the illumination information in the neural network and the output of the last full-connection layer of the model through a large number of experiments, the illumination matching of the face illumination image is completed, and an illumination matching algorithm is proposed.
(4) The invention provides illumination migration by matching illumination of the illumination classification model on the basis of obtaining a well-trained illumination classification model and combining with the innovation of image style migration, improves the loss value function of the traditional style migration, and only takes the content migration part to migrate the illumination information of the image, thereby finally obtaining the facial image illumination migration method based on the convolutional neural network.
Drawings
FIG. 1 is a block diagram of the present invention;
FIG. 2 is a network architecture diagram of convolutional neural networks VGG16 and VGG19 used in the present invention;
FIG. 3 is a flowchart of an illumination matching algorithm based on an illumination classification model according to the present invention;
FIG. 4 is a flowchart of an illumination migration algorithm based on an illumination classification model according to the present invention.
Detailed Description
For a better understanding of the invention, some basic concepts will be explained below.
A convolutional neural network: the conditional neural Networks, CNNs, are a class of neural Networks that contain convolution computations and are commonly used for deep learning.
And (3) rolling layers: extracting image features through convolution calculation;
a pooling layer: compressing the input characteristic image to simplify the network computation complexity;
full connection layer: connecting all the characteristics, and sending an output value to a softmax layer;
transfer learning: the weights in the network model which can complete a certain classification task after being trained are migrated to a brand new network model which is trained by another target classification instead of being trained from the initial state.
Illumination matching: and (3) illumination matching, namely, giving a human face illumination input image, searching an image with the same illumination effect as the input image in the human face illumination image data set, and outputting the image with the same illumination.
Matconvnet: a tool box of a convolutional neural network can be used in Matlab, and pre-training model resources are abundant.
Referring to fig. 1, the whole implementation process of the invention is as follows:
(1) firstly, preparing and establishing a data set, and selecting a Yale Face data set and a PIE data set. For the YaleFace face data set, face data of 28 photographic objects are used for training, and face data of 10 photographic objects are used for testing. For the PIE face data set, face data of 50 photographic subjects were used for training and face data of 18 photographic subjects were used for testing.
(2) Different pre-training models are used for the two data sets selected. The Yale Face data set uses a convolutional neural network VGG19, the Matconvnet provides a pre-training model for ImageNet, the PIE data set uses a convolutional neural network VGG16, the Matconvnet provides a Vgg Face pre-training model, and the schematic diagram of the neural network structures of VGG16 and VGG19 is shown in FIG. 2.
(3) And obtaining an illumination classification model. Modifying the network structures of VGG16 and VGG19, removing the last full connection layer of the original network according to classification tasks, adding a new softmax layer, a top1-error layer and a top5-error layer for training, adding a new full connection layer according to the category number of the data set, wherein the Yale Face data set is 64 types, and the PIE data set is 21 types. Then fine tuning the whole network, fine tuning the weights of the previous layers of the network, setting the learning rate as 300 iterations, and completing the illumination classification task of the face image. The average accuracy of the illumination classification model trained by the VGG16 network and the VGG Face pre-training model on the PIE Face data set test set is 62.8571%. The average accuracy of the illumination classification model trained by the VGG19 network and the ImageNet pre-training model on the Yale Face data set test set is 94.375%, and the classification accuracy of 48 types of illumination effects in 64 types of illumination is 100%.
(4) And carrying out illumination matching on the basis of an illumination classification model. The key to performing illumination matching is how to compute the illumination information representing the image. After a large number of experiments, it is found that a certain rule exists between the output of the image passing through the network full connection layer and the label of the image category, and the dimension serial number corresponding to the maximum value in the output 64-dimensional data is the same as the serial number of the label of the image category. Therefore, an illumination matching algorithm based on the illumination classification model is proposed, and a specific algorithm flow is shown in fig. 3.
(5) Inspired by Neural Style, the loss function is modified by combining a quotient image method of the traditional illumination migration research, only the content migration part is reserved to migrate the illumination information of the image, and the end-to-end illumination migration of the single face image is completed, wherein the specific algorithm flow is shown in fig. 4.
The concrete implementation process of the steps is as follows:
the structure of the convolutional neural network VGG16 and the convolutional neural network VGG19 used in the step (1) is shown in fig. 2:
(1.1) the VGG16 contains 16 weight layers and the VGG19 contains 19 weight layers, wherein the weight layers contain convolutional layers and fully-connected layers;
(1.2) VGG16 and VGG19 input 224 × 224RGB images;
(1.3) VGG16 contains 13 convolutional layers, each represented by conv3-xxx, where 3 represents the convolution kernel size 3 x 3 and-xxx represents the number of convolutions; 3 full-junction layers, each with FC-xxx; 5 pooling layers, denoted maxpool; 1 sofmax layer;
(1.4) VGG19 contains 16 convolutional layers, each represented by conv3-xxx, where 3 represents the convolution kernel size 3 x 3 and-xxx represents the number of convolutions; 3 full-junction layers, each with FC-xxx; 5 pooling layers, denoted maxpool; 1 sofmax layer.
The illumination matching specific algorithm process based on the illumination classification model in the step (4)
The light matching is realized on the basis of obtaining a light classification model through training, and the flow of the light matching algorithm is shown in figure 3:
(4.1) start;
(4.2) inputting the face illumination image into an illumination classification model: the method comprises the steps of firstly removing the last full-link layer of an original network according to a classification task, adding a new softmax layer, a top1-error layer and a top5-error layer for training, adding a new full-link layer according to the category number of a data set, wherein a Yale Face data set is 64 types, and a PIE data set is 21 types. Then fine tuning the whole network, fine tuning the weights of the previous layers of the network, setting the learning rate as iteration for 300 rounds, and completing the illumination classification task of the face image;
(4.3) taking out the output value of the last full-connection layer of the model: the VGG neural network adopted by the invention is composed of a convolutional layer, a full connection layer, a pooling layer and a softmax layer. The output of the convolutional layer operation is image characteristic information with large data, and it is difficult to extract illumination information from the image characteristic information, and the connotation of information obtained after the convolutional neural network processes the image cannot be clarified. And the output data dimension of the last full connection layer of the VGG network is consistent with the category number of the image, and the association with the image illumination information is easy to find out. Therefore, on the basis of the illumination classification model, the model trained by using the YaleFace data set with high illumination classification accuracy and detailed classes is used for searching the association between the illumination information and the output of the last full-connection layer of the model so as to complete illumination matching on the face illumination image;
(4.4) the maximum value and the number of the dimension in which the maximum value is located are obtained: by observing the output of a large number of images through a network full-connection layer and the labels of image types, a certain rule exists between the output of the large number of images and the labels of the image types, and the dimension serial number corresponding to the maximum value in the output 64-dimensional data is the same as the serial number of the labels of the image types, so that the obtained maximum value and the serial number of the dimension are the same as the dimension serial number of the input image;
(4.5) searching images with the same dimension sequence number as the input images in the face data set: based on the steps, finding out the images with the same dimensionality serial numbers as the input images in the face data set so as to achieve the matching of illumination;
(4.6) comparing the maximum values of the images with the same dimension number: comparing the maximum value images of all images with the same dimensionality sequence number in the images with the same dimensionality sequence number as the input image, which are found in the face data set, to obtain a maximum value image, namely an image with the closest matching of the images in the face data set and the input image illumination;
(4.7) outputting an image having a maximum value closest to the input image as a matching image;
and (4.8) finishing.
The illumination migration specific algorithm process based on the illumination classification model in the step 5
On the basis of an illumination classification model, end-to-end single face image illumination migration is completed by combining a quotient image method of the traditional illumination migration research under the enlightenment of neural style. The traditional Neural Style adopts an input content graph and a Style graph, calculates the loss of the content graph and the Style graph respectively, and finally linearly combines the loss of the content graph and the Style graph to obtain a final Style transition graph. Based on the thought, the invention provides illumination migration based on an illumination classification model, and the implementation process of a specific algorithm is shown in FIG. 4:
(5.1) starting;
(5.2) inputting the reference image and the input image into the migration network: the input image is generally a uniform front illumination image, the reference image is generally an image with obvious light and shadow difference, and the input image and the reference image are input into an illumination migration network (VGG 19);
(5.3) deriving feature values of the input image and the reference image: the input image and the reference image are calculated through an illumination migration network to obtain a characteristic value of the image, so that the light and shadow quotient of the image can be calculated conveniently;
(5.4) solving the light and shadow quotient: before calculating the shadow quotient, calculating the ratio of the reference image and the target image after being learnt by the VGG network, wherein the formula is as follows:
wherein, Fl[I]Is a feature matrix of the input image I at the convolution layer of the layer I, Fl[E]Is the characteristic matrix of the reference image E at the convolution layer of the layer I; a constant is set to 0.0001, the characteristic matrix value of the reference image divides the characteristic value of the input image to obtain the ratio S of the reference image and the target image after VGG network learninglMultiplying the ratio by the input image feature matrix to obtain a light and shadow quotient, wherein the formula is as follows:
Fl[M]＝Fl[I]×Sl
wherein the invention is to Fl[M](Fl[M]Is the light-shadow quotient of the image M at the convolution layer of the layer l) is formulated as
The constraints shown improve. Wherein the constraint value rijExperiments show that the illumination information can be better migrated within the constraint range and the structure and the content of the reference image can not be excessively migrated to the input image when the illumination information is 0.4 and 5;
(5.5) obtaining a result image through a migration loss function: neural Style creatively proposes content loss and Style loss of pictures to represent migration loss of contents and styles of convolutional Neural network learning images, and the formula is as follows:
wherein l represents the presence of dimension D at the first layer convolution layerlN of vectorized feature mappinglA filter (D)lIs the number of elements in the filter response). Fl[.]∈RNl×DlIs the obtained feature matrix, and (i, j) is the index of the feature matrix. (F)l[I],Fl[E],Fl[O]) L is the total number of layers in the inspected network, L being (α L, β) among whichl) Is a configured weight parameter. Is a weight that reconciles between the integrity of the input image content and the amount of painterly migration.
The migration loss function of the illumination migration method is:
Fl[O]for the desired resulting image, Fl[M]Calculating the distance between the light and shadow quotient and continuously reducing the distance for the light and shadow quotientThe difference between the two is to obtain natural light migration. After the experiment, the content migration part in the Neural Style is reserved to migrate the illumination information of the image. The convolution layers for illumination migration are determined to be convolution 1_2 layers and convolution 2_1 layers through experiments, and the lower convolution layer is sensitive to the content and structure information of the image and can retain the content information of the input image. The illumination migration method based on the illumination classification model can migrate illumination in multiple directions on the same camera object, comprises a positive light source, a left light source and a right light source, can migrate the illumination, and has natural illumination;
(5.6) returning a result image;
and (5.7) finishing.
Although illustrative embodiments of the present invention have been described above to facilitate the understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the embodiments, but various changes may be apparent to those skilled in the art, and it is intended that all inventive concepts utilizing the inventive concepts set forth herein be protected without departing from the spirit and scope of the present invention as defined and limited by the appended claims.
Claims (7)
1. A face image illumination migration method based on a convolutional neural network is characterized by comprising the following steps:
step 1, preparing and establishing a data set;
step 2, preparing a pre-training model;
step 3, obtaining an illumination classification model;
step 4, light matching based on the light classification model;
step 5, light migration based on the light classification model;
the illumination matching based on the illumination classification model in the step (4) specifically comprises the following steps:
(4.1) start;
(4.2) inputting the face illumination image into an illumination classification model: the training illumination classification model firstly removes the last full connection layer of the original network according to the classification task and adds a new one for trainingAdding a new full-connection layer according to the category number of the data set, wherein the softmax layer, the top1-error layer and the top5-error layer are respectively of 64 types, and the PIE data set is of 21 types; then fine tuning the whole network, fine tuning the weights of the first layers of the network, setting the learning rate to 1 x 10-4Iteration is carried out for 300 rounds, and the task of face image illumination classification is completed;
(4.3) taking out the output value of the last full-connection layer of the model: the adopted VGG neural network consists of a convolution layer, a full connection layer, a pooling layer and a softmax layer, wherein the characteristic information of an output image calculated by the convolution layer is trained by using a Yale Face data set on the basis of an illumination classification model, and the correlation between the illumination information and the output of the last full connection layer of the model is searched to complete the illumination matching of a Face illumination image;
(4.4) the maximum value and the number of the dimension in which the maximum value is located are obtained: the image passes through the label of the output and image category of the network full connection layer, the dimension serial number corresponding to the maximum value in the output 64-dimensional data is the same as the serial number of the label of the image category, and the maximum value and the serial number of the dimension where the maximum value is located are obtained;
(4.5) searching images with the same dimensionality sequence number as the input image in the face data set by using the maximum value obtained in the step (4.4) and the sequence number of the dimensionality: finding out images with the same dimensionality serial number as the input images in the face data set so as to achieve illumination matching;
(4.6) comparing the maximum values of the images with the same dimension number: in the images which are found in the face data set and have the same dimension serial number as the input images, comparing the maximum values of the images with the same dimension serial number to obtain a maximum value image, namely an image which is the most matched with the illumination of the images in the face data set and the input images;
(4.7) outputting the maximum value image as a matching image of an image closest to the input image;
and (4.8) finishing.
2. The illumination migration method for the face image based on the convolutional neural network as claimed in claim 1, wherein the step (1) of preparing and establishing the data set comprises the following steps:
for the Yale Face data set, adopting Face data of 28 photographic objects for training, and Face data of 10 photographic objects for testing;
for the PIE face data set, face data of 50 photographic objects are used for training, and face data of 18 photographic objects are used for testing; and performing cutout processing on the image data.
3. The illumination migration method for the face image based on the convolutional neural network as claimed in claim 1, wherein the step (2) of preparing the pre-training model comprises:
aiming at the Yale Face data set, adopting a pre-training model of an object classification data set ImageNet provided by VGG19 convolutional neural network and Matconvnet;
for the PIE face data set, a pre-training model of a face recognition model VggFace provided by a convolutional neural network VGG16, Matconvnet is used.
4. The illumination migration method for the face image based on the convolutional neural network as claimed in claim 1, wherein the illumination classification model obtained in step (3) is as follows:
the method comprises the steps that a training illumination classification model firstly removes the last full connection layer of an original network according to classification tasks, adds a new softmax layer, a top1-error layer and a top5-error layer for training, adds a new full connection layer according to the category number of a data set, and comprises 64 classes of Yale Face data sets and 21 classes of PIE data sets; then define the whole network, fine-tune the weights of the first few layers of the network, and set the learning rate to 1 × 10-4And iterating for 300 rounds, completing a human face image illumination classification task, and training an illumination classification model by a VGG16 network and a VGGFace pre-training model.
5. The illumination migration method for the face image based on the convolutional neural network as claimed in claim 1, wherein the step (4) of illumination matching based on the illumination classification model is as follows:
searching an image with the same illumination effect as the input image in the human Face illumination image data set, outputting the image with the same illumination, and finishing illumination matching on the human Face illumination image by using a model trained by a Yale Face data set on the basis of an illumination classification model.
6. The illumination migration method for the face image based on the convolutional neural network as claimed in claim 1, wherein the step (5) is based on the illumination migration of the illumination classification model:
firstly, preparing an input image and a reference image required by illumination migration, wherein the input image is a uniform front illumination image, the reference image is an image with obvious light and shadow difference, inputting the input image and the reference image into an illumination migration network to obtain characteristic matrix values of the two images, and then solving a light and shadow quotient according to the characteristic matrix values; and returning the shadow quotient and the expected result image to the migration network to minimize the migration loss function, and outputting the result image after iterating for multiple rounds.
7. The illumination migration method for the face image based on the convolutional neural network as claimed in claim 6, wherein the step (5) of the illumination migration based on the illumination classification model specifically comprises:
(5.1) starting;
(5.2) inputting the reference image and the input image into the migration network: the input image is a uniform front illumination image, the reference image is an image with obvious light and shadow difference, and the input image and the reference image are input into an illumination migration network VGG 19;
(5.3) deriving feature values of the input image and the reference image: the input image and the reference image are calculated through an illumination migration network to obtain a characteristic value of the image, so that the light and shadow quotient of the image can be calculated conveniently;
(5.4) solving the light and shadow quotient: before calculating the shadow quotient, calculating the ratio of the reference image and the target image after being learnt by the VGG network, wherein the formula is as follows:
wherein, Fl[I]Is a feature matrix of the input image I at the convolution layer of the layer I, Fl[E]Is the characteristic matrix of the reference image E at the convolution layer of the layer I; a constant is set to 0.0001, the characteristic matrix value of the reference image divides the characteristic value of the input image to obtain the ratio S of the reference image and the target image after VGG network learninglMultiplying the ratio by the input image feature matrix to obtain a light and shadow quotient, wherein the formula is as follows:
Fl[M]＝Fl[I]×Sl
Fl[M]is the light quotient of image M at the first layer convolution layer, where F is the pairl[M]The constraint improvement is as follows:
wherein the constraint value rijIn the range of 0.4 to 5;
(5.5) obtaining a result image through a migration loss function: the content loss content and Style loss of the picture in the Neural Style method are used for representing the migration loss of the content and Style of the convolutional Neural network learning image, and the formula is as follows:
wherein l represents the presence of dimension D at the first layer convolution layerlN of vectorized feature mappinglA filter, DlIs the number of elements in the filter response, Fl[.]∈RNl×DlIs the obtained feature matrix, (i, j) is the index of the feature matrix, Fl[I],Fl[E],Fl[O]Respectively, of the feature matrix of the input image I, of the reference painterly image E and of the result image O of the expected output of the first-layer convolutional layer, L is the total number of layers in the network under examination, of which αl，βlIs a configured weight parameter; is a weight that reconciles between the integrity of the input image content and the amount of painterly migration;
the migration loss function of the illumination migration method is:
Fl[O]for the desired resulting image, Fl[M]Calculating the distance between the light and shadow quotient of the two light and shadow quotient, and continuously reducing the difference between the two light and shadow quotient to obtain natural light migration;
(5.6) returning a result image;
and (5.7) finishing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911335861.4A CN111091492B (en) | 2019-12-23 | 2019-12-23 | Face image illumination migration method based on convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911335861.4A CN111091492B (en) | 2019-12-23 | 2019-12-23 | Face image illumination migration method based on convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111091492A CN111091492A (en) | 2020-05-01 |
| CN111091492B true CN111091492B (en) | 2020-09-04 |
Family
ID=70395905
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911335861.4A Active CN111091492B (en) | 2019-12-23 | 2019-12-23 | Face image illumination migration method based on convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111091492B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111951373B (en) * | 2020-06-30 | 2024-02-13 | 重庆灵翎互娱科技有限公司 | Face image processing method and equipment |
| CN114155368A (en) * | 2020-08-17 | 2022-03-08 | 田英杰 | A Mapping Algorithm Based on Attention Transfer Network |
| CN114972006A (en) * | 2021-02-25 | 2022-08-30 | 阿里巴巴集团控股有限公司 | Image generation method, device, equipment and storage medium |
| CN114255498A (en) * | 2021-12-13 | 2022-03-29 | 厦门美图之家科技有限公司 | A CNN-based portrait face transfer method and device and device |
| CN114495201A (en) * | 2021-12-28 | 2022-05-13 | 杭州魔点科技有限公司 | Multi-task face recognition method, system, device and medium including liveness detection |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103186774A (en) * | 2013-03-21 | 2013-07-03 | 北京工业大学 | Semi-supervised learning-based multi-gesture facial expression recognition method |
| CN106599863A (en) * | 2016-12-21 | 2017-04-26 | 中国科学院光电技术研究所 | Deep face recognition method based on transfer learning technology |
| CN108629380A (en) * | 2018-05-11 | 2018-10-09 | 西北大学 | A kind of across scene wireless signal cognitive method based on transfer learning |
| CN110427965A (en) * | 2019-06-25 | 2019-11-08 | 重庆邮电大学 | Convolutional neural networks structural reduction and image classification method based on evolution strategy |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2532075A (en) * | 2014-11-10 | 2016-05-11 | Lego As | System and method for toy recognition and detection based on convolutional neural networks |
| CN107527334A (en) * | 2017-08-28 | 2017-12-29 | 北京小米移动软件有限公司 | Human face light moving method and device |
| CN110570435B (en) * | 2018-09-10 | 2020-06-26 | 阿里巴巴集团控股有限公司 | Method and device for carrying out damage segmentation on vehicle damage image |
| CN110458844B (en) * | 2019-07-22 | 2022-02-15 | 大连理工大学 | A Semantic Segmentation Method for Low Light Scenes |
| CN110503711B (en) * | 2019-08-22 | 2023-02-21 | 三星电子（中国）研发中心 | Method and device for rendering virtual object in augmented reality |
-
2019
- 2019-12-23 CN CN201911335861.4A patent/CN111091492B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103186774A (en) * | 2013-03-21 | 2013-07-03 | 北京工业大学 | Semi-supervised learning-based multi-gesture facial expression recognition method |
| CN106599863A (en) * | 2016-12-21 | 2017-04-26 | 中国科学院光电技术研究所 | Deep face recognition method based on transfer learning technology |
| CN108629380A (en) * | 2018-05-11 | 2018-10-09 | 西北大学 | A kind of across scene wireless signal cognitive method based on transfer learning |
| CN110427965A (en) * | 2019-06-25 | 2019-11-08 | 重庆邮电大学 | Convolutional neural networks structural reduction and image classification method based on evolution strategy |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111091492A (en) | 2020-05-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111091492B (en) | Face image illumination migration method based on convolutional neural network | |
| Laffont et al. | Transient attributes for high-level understanding and editing of outdoor scenes | |
| Qi et al. | Geonet: Geometric neural network for joint depth and surface normal estimation | |
| CN110209859B (en) | Method and device for recognizing places and training models of places and electronic equipment | |
| Radenovic et al. | Deep shape matching | |
| Wang et al. | Deep cropping via attention box prediction and aesthetics assessment | |
| CN110084121A (en) | Implementation method based on the human face expression migration for composing normalized circulation production confrontation network | |
| CN111625667A (en) | Three-dimensional model cross-domain retrieval method and system based on complex background image | |
| CN110827304B (en) | Traditional Chinese medicine tongue image positioning method and system based on deep convolution network and level set method | |
| CN115393225A (en) | A low-light image enhancement method based on multi-level feature extraction and fusion | |
| CN110852276A (en) | Pedestrian re-identification method based on multitask deep learning | |
| CN110276753B (en) | Object Adaptive Hiding Method Based on Feature Space Statistical Information Mapping | |
| WO2023151529A1 (en) | Facial image processing method and related device | |
| CN116189292B (en) | A video action recognition method based on dual-stream network | |
| CN112329662A (en) | Multi-view saliency estimation method based on unsupervised learning | |
| CN113887353A (en) | Visible light-infrared pedestrian re-identification method and system | |
| CN102013020B (en) | Method and system for synthesizing human face image | |
| Shu et al. | Eyeopener: Editing eyes in the wild | |
| CN111160119B (en) | A multi-task deep discriminative metric learning model building method for makeup face verification | |
| CN116434010A (en) | Multi-view pedestrian attribute identification method | |
| Huo et al. | Efficient photorealistic style transfer with multi-order image statistics | |
| CN113222953B (en) | Natural image enhancement method based on depth gamma transformation | |
| Li et al. | Facial age estimation by deep residual decision making | |
| CN113469006A (en) | Pedestrian attribute identification method based on graph convolution | |
| CN112860936A (en) | Visual pedestrian re-identification method based on sparse graph similarity migration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |