CN111862061A - Methods, systems, devices and media for assessing the aesthetic quality of pictures - Google Patents

Abstract

The invention discloses an evaluation method, system, equipment and medium of picture aesthetic quality, wherein the evaluation method of picture aesthetic quality includes the following steps: integrating and labeling picture data sets; extracting deep learning features of pictures according to the picture data sets; Character features and puzzle features; Semantic segmentation of pictures, and extraction of centroid features and angle features according to the results of semantic segmentation; Obtaining the color saturation of pictures; Saturation is trained as a training feature to obtain a prediction model; the aesthetic quality of the picture is evaluated according to the prediction model. The invention improves the accuracy of aesthetic quality evaluation.

Description

Methods, systems, devices and media for assessing the aesthetic quality of pictures

technical field

The invention belongs to the technical field of picture aesthetic quality assessment, and in particular relates to a picture aesthetic quality assessment method, system, device and medium.

Background technique

With the development of the times and the advancement of science and technology, taking pictures has become a thing that ordinary people can do anytime, anywhere, and the development of the Internet age has also enabled photos to be disseminated in all corners of the world in a digital form. Then selecting a good-looking photo and filtering out some unsightly photos is an application direction, especially in the travel photography interface of the app (application). By selecting good-looking pictures to filter out unsightly pictures, and placing good-looking pictures on the homepage, it is possible to show visitors some attractive scenery or food, and then attract visitors to these places for consumption. The correctness of pushing pictures and the efficiency of filtering pictures determine the work efficiency of the app.

In fact, the study of image aesthetics has a popular saying in the field of computer vision or machine learning, that is, computer aesthetics. It sounds like it is a relatively subjective thing to let a computer do aesthetics on pictures. It seems unrealistic to let a computer do such a thing that humans are a little uncertain about, but in reality, there are indeed many topics on this subject. Research has been carried out in this regard, and has achieved good results.

Computer aesthetics means that the computer aesthetics the picture, which can output a score, or it can be divided into good-looking or bad-looking. There is another topic called image quality assessment that is also related to computer aesthetics. In a broad sense, image quality assessment is also a part of computer aesthetics. The image quality evaluation mainly evaluates the noise, saturation and other factors of the image, and the computer aesthetics score is compared with the existing score of the data set. In photography, we pay attention to symmetry and the law of golden ratio. The law of golden ratio is to consider the position of the object in the picture at the golden ratio point, so that the photos taken are in line with people's aesthetics. In 2010, Pere Obrador's research (person's name) extracted relevant features such as the centroid of the image and built a model, and the highest accuracy rate could only reach about 70%. With the popularity of convolutional neural networks, aesthetic scores began to gradually break through the bottleneck. A topic jointly researched by the University of Buffalo and Tianjin University in 2017, they published an article called A-Lamp (article name) by randomly deducting patches (spots) in the picture to extract features, using The convolutional network VGG (a model) builds the model, and the final highest accuracy rate that can be achieved is about 81%. In 2018, Hossein Talebi (person's name) built a model called NIMA (Neural Image Assessment), which uncharacteristically starts with the data set and selects the AVA (a data set) data set, in which each image in the data set is classified. Scored for 10 levels and scored by more than 250 annotators, each person can choose an appropriate score from 1 to 10 points, and the score of each image can finally be represented by a histogram, they are in loss The function is improved, and the convolutional neural network is still used in the model construction, and the final highest accuracy rate can reach about 82%.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a method, system, device and medium for evaluating the aesthetic quality of a picture in order to overcome the defect of low accuracy in evaluating the aesthetic quality of a picture in the prior art.

The present invention solves the above-mentioned technical problems through the following technical solutions:

The present invention provides a method for evaluating the aesthetic quality of a picture, comprising the following steps:

Integrate and label image datasets;

Extract the deep learning features of pictures according to the picture dataset;

Extract the embossed features and puzzle features of the pictures;

Semantic segmentation is performed on the image, and centroid features and angle features are extracted according to the results of semantic segmentation;

Get the color saturation of the picture;

Train deep learning features, embossing features, puzzle features, centroid features, angle features, and color saturation as training features to obtain a prediction model;

Assess the aesthetic quality of the image based on the predictive model.

Preferably, the step of extracting the deep learning feature of the picture according to the picture data set includes:

Extract deep learning features based on Resnet (a model) model, xception (a model) model, and VGG model;

The steps of extracting the embossed features and puzzle features of the pictures include:

The embossed and puzzle features are extracted based on the yolo (one model) model and the resnet (one model) model.

Preferably, performing semantic segmentation on the picture includes:

Semantic segmentation of images based on pspnet (a model) model.

Preferably, the deep learning feature further includes the mean and deviation of prediction scores of pictures extracted based on the NIMA (a kind of model) model.

The present invention also provides an evaluation system for image aesthetic quality, including a data set generation unit, a feature extraction unit, a training unit, and an evaluation unit;

The dataset generation unit is used to integrate and label image datasets;

The feature extraction unit is used to extract the deep learning features of the picture according to the picture data set;

The feature extraction unit is also used to extract the embossing feature and puzzle feature of the picture;

The feature extraction unit is also used to perform semantic segmentation on the image, and extract the centroid feature and the angle feature according to the result of the semantic segmentation;

The feature extraction unit is also used to obtain the color saturation of the picture;

The training unit is used to train deep learning features, embossing features, puzzle features, centroid features, angle features and color saturation as training features to obtain a prediction model;

The evaluation unit is used to evaluate the aesthetic quality of the picture according to the prediction model.

Preferably, the feature extraction unit is also used to extract deep learning features based on the Resnet model, the xception model, and the VGG model;

The feature extraction unit is also used to extract embossed and puzzle features based on the yolo model and the resnet model.

Preferably, the feature extraction unit is further configured to perform semantic segmentation on the picture based on the pspnet model.

Preferably, the deep learning feature further includes the mean and deviation of prediction scores of pictures extracted based on the NIMA model.

The present invention also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and running on the processor, when the processor executes the computer program, the method for evaluating the aesthetic quality of a picture of the present invention is implemented.

The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the method for evaluating the aesthetic quality of a picture of the present invention.

The positive progressive effect of the present invention is that the present invention improves the accuracy of aesthetic quality assessment.

Description of drawings

FIG. 1 is a flowchart of a method for evaluating the aesthetic quality of a picture according to Embodiment 1 of the present invention.

FIG. 2 is a schematic structural diagram of a picture aesthetic quality evaluation system according to Embodiment 2 of the present invention.

FIG. 3 is a schematic structural diagram of an electronic device according to Embodiment 3 of the present invention.

Detailed ways

The present invention is further described below by way of examples, but the present invention is not limited to the scope of the described examples.

Example 1

This embodiment provides a method for evaluating the aesthetic quality of a picture. Referring to Figure 1, the evaluation method for the aesthetic quality of the picture includes the following steps:

Step S101, integrating and labeling the image data set.

Step S102, extracting the deep learning feature of the picture according to the picture data set.

Step S103, extracting embossing features and puzzle features.

Step S104 , perform semantic segmentation on the picture, and extract centroid features and angle features according to the results of the semantic segmentation.

Step S105, acquiring the color saturation of the picture.

Step S106 , using deep learning features, embossing features, puzzle features, centroid features, angle features, and color saturation as training features for training to obtain a prediction model.

Step S107: Evaluate the aesthetic quality of the picture according to the prediction model.

During specific implementation, in step S101, the image data set is integrated and marked. In an optional embodiment, the image dataset comes from a computer network. The pictures included in the picture dataset are travel pictures.

As an optional implementation manner, in step S102, deep learning features are extracted using models such as Resnet, xception, and VGG.

As an optional implementation manner, in step S103, yolo and resnet are used to extract embossing and puzzle features.

As an optional implementation manner, in step S104, semantic segmentation is performed using pspnet, and features such as centroid and angle are extracted using mathematical formulas using the results of the semantic segmentation. Use the extracted centroid position to calculate the distance from the golden section point. A small distance indicates that it conforms to the golden section law of aesthetics and is considered to be a picture with higher quality. Compare the importance of the centroid and delete the features with low importance and many null values. features to simplify the model training process.

In an optional implementation manner, the original picture is directly processed, and the color saturation of the picture is calculated by using a mathematical formula, and it is considered that a picture with high color saturation is of higher quality.

In an optional embodiment, the deep learning feature further includes the mean and deviation of prediction scores of pictures extracted based on the NIMA model. NIMA is a deep learning model that evaluates image quality with high quality in recent years.

In step S106, use Resnet to train features and perform feature screening to obtain an optimal prediction model, and perform multiple parameter adjustments to obtain optimal parameters. In specific implementation, the penultimate layer features of the pre-training model are extracted by means of finetune. Test the model effect and select the optimal model as the prediction model.

In step S107, the aesthetic quality of the picture is evaluated according to the prediction model.

In the prior art, there are great differences in the type and quality of pictures taken by travel and open-source picture datasets (such as AVA), and the effect of directly migrating the models constructed from the open-source datasets is not good. In the method for evaluating the aesthetic quality of pictures in this embodiment, the travel photos are manually marked, the model is retrained, and the model results trained on the open source data set are transferred as the features of the training model.

In the prior art, when extracting deep learning image features offline, the amount of artificially constructed data sets is small and cannot provide effective features well. The method for evaluating the aesthetic quality of an image in this embodiment uses a finetune method to extract the features of the penultimate layer of the pre-training model.

In the prior art, objects in a picture have different colors, and it is difficult to directly calculate the centroid of an object by using pixels. The method for evaluating the aesthetic quality of a picture in this embodiment calls the PSP net model to perform semantic segmentation, so that the colors of the objects in the picture are unified, and the position and angle of the centroid of the object are calculated as the features of the training model.

In the prior art, there are some pictures in categories such as embossed characters and jigsaw puzzles in the travel pictures, which need to be specially processed. The method for evaluating the aesthetic quality of a picture in this embodiment establishes embossing and jigsaw models, and discriminates these categories of pictures to obtain a discriminant score, which is used as a feature of the training model.

The image aesthetic quality evaluation method of this embodiment combines the two focuses of accurate extraction of traditional features and deep learning of convolutional neural networks, and uses a database for model training, striving to break through the existing bottleneck and improve the classification and prediction accuracy of the model. The accuracy rate of the method for evaluating the aesthetic quality of a picture in this embodiment can reach up to 92.4%.

Example 2

This embodiment provides an evaluation system for the aesthetic quality of pictures. Referring to FIG. 2 , the image aesthetic quality evaluation system includes a data set generation unit 201 , a feature extraction unit 202 , a training unit 203 , and an evaluation unit 204 .

The dataset generating unit 201 is used for integrating and labeling the image dataset;

The feature extraction unit 202 is used to extract the deep learning feature of the picture according to the picture data set;

The feature extraction unit 202 is also used to extract the embossing feature and the puzzle feature of the picture;

The feature extraction unit 202 is also used to perform semantic segmentation on the picture, and extract the centroid feature and the angle feature according to the result of the semantic segmentation;

The feature extraction unit 202 is also used to obtain the color saturation of the picture;

The training unit 203 is used to train the deep learning feature, embossing feature, puzzle feature, centroid feature, angle feature and color saturation as training features to obtain a prediction model;

The evaluation unit 204 is configured to evaluate the aesthetic quality of the picture according to the prediction model.

During specific implementation, the data set generating unit 201 integrates and annotates the image data set. In an optional embodiment, the image dataset comes from a computer network. The pictures included in the picture dataset are travel pictures.

As an optional implementation manner, the feature extraction unit 202202 extracts deep learning features based on models such as Resnet, xception, and VGG.

As an optional implementation manner, the feature extraction unit 202202 extracts embossed and puzzle features based on yolo and resnet.

As an optional implementation manner, the feature extraction unit 202202 performs semantic segmentation based on pspnet, and uses the result of the semantic segmentation to extract features such as centroid and angle using mathematical formulas. Use the extracted centroid position to calculate the distance from the golden section point. A small distance indicates that it conforms to the golden section law of aesthetics and is considered to be a picture with higher quality. Compare the importance of the centroid and delete the features with low importance and many null values. features to simplify the model training process.

In an optional implementation manner, the original picture is directly processed, and the color saturation of the picture is calculated by using a mathematical formula, and it is considered that a picture with high color saturation is of higher quality.

In an optional embodiment, the deep learning feature further includes the mean and deviation of prediction scores of pictures extracted based on the NIMA model. NIMA is a deep learning model that evaluates image quality with high quality in recent years.

The training unit 203 uses Resnet to train features and perform feature screening to obtain an optimal prediction model, and to obtain optimal parameters after performing multiple parameter adjustments. In specific implementation, the penultimate layer features of the pre-training model are extracted by means of finetune. Test the model effect and select the optimal model as the prediction model.

The evaluation unit 204 evaluates the aesthetic quality of the picture according to the prediction model.

In the prior art, there are great differences in the type and quality of pictures taken by travel and open-source picture datasets (such as AVA), and the effect of directly migrating the models constructed from the open-source datasets is not good. In the method for evaluating the aesthetic quality of pictures in this embodiment, the travel photos are manually marked, the model is retrained, and the model results trained on the open source data set are transferred as the features of the training model.

In the prior art, when extracting deep learning image features offline, the amount of artificially constructed data sets is small and cannot provide effective features well. The method for evaluating the aesthetic quality of an image in this embodiment uses a finetune method to extract the features of the penultimate layer of the pre-training model.

In the prior art, objects in a picture have different colors, and it is difficult to directly calculate the centroid of an object by using pixels. The method for evaluating the aesthetic quality of a picture in this embodiment calls the PSP net model to perform semantic segmentation, so that the colors of the objects in the picture are unified, and the position and angle of the centroid of the object are calculated as the features of the training model.

In the prior art, there are some pictures in categories such as embossed characters and jigsaw puzzles in the travel pictures, which need to be specially processed. The method for evaluating the aesthetic quality of a picture in this embodiment establishes embossing and jigsaw models, and discriminates these categories of pictures to obtain a discriminant score, which is used as a feature of the training model.

The image aesthetic quality evaluation system of this embodiment combines the two focuses of accurately extracting traditional features and deep learning convolutional neural networks, and uses a database for model training, striving to break through existing bottlenecks and improve the accuracy of model classification and prediction. The accuracy rate of the image aesthetic quality evaluation system of this embodiment can reach up to 92.4%.

Example 3

FIG. 3 is a schematic structural diagram of an electronic device provided in this embodiment. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for evaluating the aesthetic quality of a picture in Embodiment 1 is implemented. The electronic device 30 shown in FIG. 3 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present invention.

The electronic device 30 may take the form of a general-purpose computing device, which may be, for example, a server device. Components of the electronic device 30 may include, but are not limited to, the above-mentioned at least one processor 31 , the above-mentioned at least one memory 32 , and a bus 33 connecting different system components (including the memory 32 and the processor 31 ).

The bus 33 includes a data bus, an address bus and a control bus.

Memory 32 may include volatile memory, such as random access memory (RAM) 321 and/or cache memory 322 , and may further include read only memory (ROM) 323 .

The memory 32 may also include a program/utility 325 having a set (at least one) of program modules 324 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, which An implementation of a network environment may be included in each or some combination of the examples.

The processor 31 executes various functional applications and data processing by running the computer program stored in the memory 32, such as the evaluation method for the aesthetic quality of the picture in Embodiment 1 of the present invention.

The electronic device 30 may also communicate with one or more external devices 34 (eg, keyboards, pointing devices, etc.). Such communication may take place through input/output (I/O) interface 35 . Also, the model-generating device 30 may also communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network such as the Internet) through a network adapter 36 . As shown, the network adapter 36 communicates with the other modules of the model generation device 30 via the bus 33 . It should be understood that, although not shown in the figures, other hardware and/or software modules may be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk) array) systems, tape drives, and data backup storage systems.

It should be noted that although several units/modules or sub-units/modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. Indeed, the features and functions of two or more units/modules described above may be embodied in one unit/module according to embodiments of the present invention. Conversely, the features and functions of one unit/module described above may be further subdivided to be embodied by multiple units/modules.

Example 4

This embodiment provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the steps of the method for evaluating the aesthetic quality of a picture in Embodiment 1.

Wherein, the readable storage medium may include, but is not limited to, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any of the above suitable combination.

In a possible implementation manner, the present invention can also be implemented in the form of a program product, which includes program codes, when the program product runs on a terminal device, the program code is used to cause the terminal device to execute the implementation Steps of the method for evaluating the aesthetic quality of pictures in Example 1.

Wherein, the program code for executing the present invention can be written in any combination of one or more programming languages, and the program code can be completely executed on the user equipment, partially executed on the user equipment, as an independent The software package executes on the user's device, partly on the user's device, partly on the remote device, or entirely on the remote device.

Although the specific embodiments of the present invention are described above, those skilled in the art should understand that this is only an illustration, and the protection scope of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principle and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims (10)

1. A method for evaluating the aesthetic quality of a picture is characterized by comprising the following steps:

integrating and labeling the picture data set;

extracting deep learning features of the picture according to the picture data set;

extracting the character pressing feature and the picture splicing feature of the picture;

performing semantic segmentation on the picture, and extracting a centroid feature and an angle feature according to a result of the semantic segmentation;

acquiring the color saturation of the picture;

training the deep learning feature, the character pressing feature, the picture splicing feature, the centroid feature, the angle feature and the color saturation as training features to obtain a prediction model;

and evaluating the aesthetic quality of the picture according to the prediction model.

2. The method of claim 1, wherein the step of extracting the deep learning features of the picture according to the picture data set comprises:

extracting the deep learning features based on a Resnet model, an xception model and a VGG model;

the steps of extracting the character pressing feature and the picture splicing feature of the picture comprise:

and extracting the embossed characters and the jigsaw puzzle features based on a yolo model and a resnet model.

3. The method of claim 1, wherein semantically segmenting the picture comprises:

and performing semantic segmentation on the picture based on a pspnet model.

4. The method for evaluating the aesthetic quality of a picture according to claim 1, wherein the deep learning features further comprise a prediction score mean and a deviation of the picture extracted based on a NIMA model.

5. The system for evaluating the aesthetic quality of the picture is characterized by comprising a data set generation unit, a feature extraction unit, a training unit and an evaluation unit;

the data set generating unit is used for integrating and labeling the picture data set;

the feature extraction unit is used for extracting deep learning features of the picture according to the picture data set;

the feature extraction unit is also used for extracting the character pressing feature and the picture splicing feature of the picture;

the feature extraction unit is also used for carrying out semantic segmentation on the picture and extracting a centroid feature and an angle feature according to the result of the semantic segmentation;

the feature extraction unit is further used for acquiring the color saturation of the picture;

the training unit is used for training the deep learning feature, the character pressing feature, the picture splicing feature, the centroid feature, the angle feature and the color saturation as training features to obtain a prediction model;

the evaluation unit is used for evaluating the aesthetic quality of the picture according to the prediction model.

6. The system for evaluating the aesthetic quality of a picture according to claim 5, wherein the feature extraction unit is further configured to extract the deep learning feature based on a Resnet model, an xception model, and a VGG model;

the feature extraction unit is further used for extracting the embossed characters and the jigsaw feature based on a yolo model and a resnet model.

7. The system for evaluating the aesthetic quality of a picture according to claim 5, wherein the feature extraction unit is further configured to semantically segment the picture based on a pspnet model.

8. The system for evaluating the aesthetic quality of a picture according to claim 5, wherein the deep learning features further comprise a prediction score mean and deviation of the picture extracted based on a NIMA model.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method for assessing the aesthetic quality of a picture according to any one of claims 1 to 4 when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for assessing the aesthetic quality of a picture according to any one of claims 1 to 4.

Images