CN108574803B - Image selection method and device, storage medium and electronic equipment - Google Patents

Abstract

The present application discloses an image selection method, device, storage medium and electronic device. The method comprises: obtaining multiple frames of images to be processed containing human faces; performing expression recognition on the facial images of each user in the images to be processed to obtain expression recognition results of each user; and selecting a basic image from the images to be processed according to the expression recognition results of each user. This embodiment can improve the flexibility of the terminal in selecting images for processing from multiple frames of images.

Description

Image selection method, device, storage medium and electronic device

technical field

The present application belongs to the field of image technology, and in particular, relates to an image selection method, device, storage medium and electronic device.

Background technique

Taking pictures is a basic function of the terminal. With the continuous improvement of hardware and image processing algorithms such as camera modules, the shooting functions of terminals are becoming more and more powerful. Users also use the terminal to take pictures more and more frequently, for example, the user often uses the terminal to take pictures of people and so on. In the related art, a terminal may collect multiple frames of images, and then select an image for processing from the multiple frames of images. However, when an image for processing is selected from multiple frames of images, the terminal has poor flexibility in selecting an image.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an image selection method, apparatus, storage medium, and electronic device, which can improve the flexibility of a terminal when selecting images for processing from multiple frames of images.

The embodiment of the present application provides a method for selecting an image, including:

Get multiple frames of images to be processed that contain faces;

Perform facial expression recognition on the face image of each user in the image to be processed to obtain the facial expression recognition result of each user;

According to the expression recognition result of each user, a base image is selected from the to-be-processed images.

An embodiment of the present application provides an image selection device, including:

an acquisition module, used to acquire multiple frames of images to be processed containing faces;

A recognition module, for performing facial expression recognition on the face image of each user in the image to be processed, to obtain the facial expression recognition result of each user;

The selection module is configured to select a basic image from the to-be-processed images according to the expression recognition result of each user.

Embodiments of the present application provide a storage medium on which a computer program is stored, and when the computer program is executed on a computer, causes the computer to execute the steps in the image selection method provided by the embodiments of the present application.

Embodiments of the present application further provide an electronic device, including a memory and a processor, where the processor is configured to execute steps in the image selection method provided by the embodiments of the present application by invoking a computer program stored in the memory.

In this embodiment, when a basic image needs to be selected from multiple frames of images to be processed, the terminal may first perform expression recognition on the face image of each user in the to-be-processed image, and then perform expression recognition on the facial image of each user according to the expression recognition result of each user. , and determine the base image from the to-be-processed image. That is, in this embodiment, the basic image can be selected from the images to be processed according to the user's expression. Therefore, this embodiment can improve the flexibility of the terminal when selecting images for processing from multiple frames of images. In addition, this embodiment can also improve the imaging effect of the photo taken by the terminal.

Description of drawings

The technical solutions of the present invention and its beneficial effects will be apparent through the detailed description of the specific embodiments of the present invention below in conjunction with the accompanying drawings.

FIG. 1 is a schematic flowchart of an image selection method provided by an embodiment of the present application.

FIG. 2 is another schematic flowchart of an image selection method provided by an embodiment of the present application.

FIG. 3 to FIG. 5 are schematic diagrams of scenarios of an image selection method provided by an embodiment of the present application.

FIG. 6 is a schematic structural diagram of an image selection apparatus provided by an embodiment of the present application.

FIG. 7 is another schematic structural diagram of an image selection apparatus provided by an embodiment of the present application.

FIG. 8 is a schematic structural diagram of a mobile terminal provided by an embodiment of the present application.

FIG. 9 is another schematic structural diagram of a mobile terminal provided by an embodiment of the present application.

Detailed ways

Please refer to the drawings, in which the same reference numerals represent the same components, and the principles of the present invention are exemplified by being implemented in a suitable computing environment. The following description is based on illustrated embodiments of the invention and should not be construed as limiting other embodiments of the invention not detailed herein.

It can be understood that the execution body of the embodiment of the present application may be a terminal device such as a smart phone or a tablet computer.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of a method for selecting an image provided by an embodiment of the present application. The process may include:

In step S101, multiple frames of images to be processed containing human faces are acquired.

Taking pictures is a basic function of the terminal. With the continuous improvement of hardware and image processing algorithms such as camera modules, the shooting functions of terminals are becoming more and more powerful. Users also use the terminal to take pictures more and more frequently, for example, the user often uses the terminal to take pictures of people and so on. However, in the related art, the imaging effect of the image collected by the terminal is poor.

For example, the terminal may first acquire multiple frames of images to be processed that contain human faces. For example, the terminal acquires six frames of images A, B, C, D, E, and F to be processed.

In step S102, the facial image of each user in the to-be-processed image is subjected to facial expression recognition to obtain the facial expression recognition result of each user.

For example, after acquiring the six frames of images A, B, C, D, E, and F to be processed, the terminal can perform expression recognition on the face image of each user in the six frames of images to be processed, so as to obtain each user's facial expression. expression recognition results.

For example, if the above-mentioned six frames of images A, B, C, D, E, and F are single-person images of the same user A that are continuously and rapidly collected by the terminal, the terminal can analyze the person of user A in each frame to be processed The facial image is used for facial expression recognition, so as to obtain the facial expression recognition result of user A.

For another example, if the above-mentioned six frames of images A, B, C, D, E, and F are group photo images of multiple people collected continuously and rapidly by the terminal, for example, the above-mentioned six frames of images are group photo images of four persons A, B, C, and D. , then the terminal can first perform expression recognition on the face image of user A in the six frames of images, and then perform expression recognition on the face images of user B, user C, and user D in turn, so as to obtain user A in the six frames of images respectively. , B, C, D four people's facial expression recognition results.

In one embodiment, the terminal may recognize the expression of the face image in the image to be processed by the following method: first determine the expression key points in the face image, and the expression key points may be, for example, eyes, eyebrows, Mouth, cheeks, etc. Then, the terminal extracts a partial image from the face image according to these expression key points. After that, the terminal can input these extracted partial images into the trained expression recognition algorithm model for expression recognition.

In step S103, a base image is selected from the to-be-processed images according to the expression recognition result of each user.

For example, after obtaining the expression recognition result of each user in the image to be processed, the terminal may select one frame of images from the to-be-processed image as the base image according to the expression recognition result of each user.

It can be understood that, in this embodiment, when a basic image needs to be selected from multiple frames of images to be processed, the terminal can first perform expression recognition on the face image of each user in the to-be-processed image, and then perform expression recognition according to each The user's facial expression recognition result, and the basic image is determined from the to-be-processed image. That is, in this embodiment, the basic image can be selected from the images to be processed according to the user's expression. Therefore, this embodiment can improve the flexibility of the terminal when selecting images for processing from multiple frames of images.

Please refer to FIG. 2. FIG. 2 is another schematic flowchart of a method for selecting an image according to an embodiment of the present application. The process may include:

In step S201, the terminal acquires multiple frames of images to be processed that include human faces.

In an implementation manner, the images collected by the camera of the terminal can be stored in a buffer queue of a certain length, and then the terminal can acquire images from the buffer queue when it needs to acquire images.

For example, in this embodiment, if the user uses the terminal camera to take pictures of people, then the most recent images collected by the terminal can be stored in the cache queue, and when the user presses the camera button, the terminal can obtain multiple frames from the cache queue Images that contain faces, which are the images to be processed.

For example, the terminal acquires six frames of H, I, J, K, L, and M images to be processed. The six frames of images to be processed are all group photos of four people, A, B, C, and D.

In step S202, the terminal performs expression recognition on the face image of each user in the image to be processed, and obtains the expression recognition result of each user.

For example, after acquiring the group photo images H, I, J, K, L, and M of four persons A, B, C, and D, the terminal can express expressions on the face images of each user in the 6 frames of images to be processed. recognition, so as to obtain the facial expression recognition result of each user.

For example, the terminal can first perform expression recognition on the face image of user A in the six frames of images, and then perform expression recognition on the face images of user B, user C, and user D in sequence, so as to obtain H, I, J, K respectively. The facial expression recognition results of users A, B, C, and D in the six frames of images , L and M.

After obtaining the facial expression recognition result of each user in the image to be processed, the terminal may determine whether the facial expression of the user in the to-be-processed image has changed according to the facial expression recognition result of each user.

If it is determined according to the facial expression recognition result of each user that the facial expressions of all users in the to-be-processed image have not changed, then proceed to step S203.

If it is determined according to the facial expression recognition result of each user that the facial expressions of some users in the to-be-processed image have changed, step S206 is entered.

In step S203, if it is determined that the expressions of all users in the to-be-processed image have not changed according to the expression recognition result of each user, the terminal obtains the eye value of each user in each of the to-be-processed images. The part value is a numerical value representing the size of the eye.

For example, the terminal determines that the expression of user A has not changed in the six frames of images H, I, J, K, L, and M to be processed according to the expression recognition result of user A, that is, the terminal determines that user A is in the six frames to be processed. The expressions in the processed images are consistent, or the changes are minimal. Similarly, the terminal determines that the expressions of users B, C, and D have not changed in the 6 frames of images to be processed according to the expression recognition results of each user. In this case, the terminal can acquire the eye value of each user in each image to be processed. Wherein, the eye value may be a numerical value used to represent the size of the eye. For example, the eye value may be a numerical value representing the area size of the eye, or the eye value may be a numerical value representing the height of the eye in the vertical direction.

In step S204, according to the eye value of each user in each of the images to be processed, the terminal determines the target face image of each user, and the target face image corresponds to the maximum value of the user's eye values Image.

For example, the eye values of user A in the images H, I, J, K, L, and M to be processed are 70, 72, 75, 80, 78, and 79, respectively. The numerical values of the eye size of B in the images H, I, J, K, L, and M to be processed are 80, 80, 81, 82, 85, and 82, respectively. The numerical values of the eye size of C in the images H, I, J, K, L, and M to be processed are 80, 82, 82, 50, 30, and 0, respectively. The numerical values of Ding's eye size in the images H, I, J, K, L, and M to be processed are 82, 83, 84, 88, 85, and 81, respectively. Among them, from the change of the size of C's eyes in the image to be processed, it can be known that C's eyes keep getting smaller from the image J, which can be considered that C is blinking. In the image M, C is in a closed eye state (eye size is 0).

After obtaining the eye value of each user in each frame of the to-be-processed image, the terminal may determine the target face image of each user from the to-be-processed image. Wherein, the target face image of each user may be the face image corresponding to the maximum value among the eye values of the user.

For example, the eye value 80 of user X in the image K to be processed is the maximum value among the eye values of X in all the images to be processed, so the terminal can determine the face image of user X in the image K to be processed as user X target face image.

Similarly, the terminal can determine the face image of user B in the image to be processed L as the target face image of user B, and determine the face image of user C in the image to be processed I or J as the target face image of user C. , and the face image of user D in the image K to be processed is determined as the target face image of user D.

In step S205, the terminal selects the image to be processed that contains the largest number of target face images as the base image.

For example, after determining the target face image of each user, the terminal may select the image to be processed that contains the largest number of target face images as the base image.

For example, since images I and J to be processed contain one target face image (target face image of user C), and image K to be processed contains two target face images (target face images of users A and D), The image L to be processed includes a target face image (the target face image of user B), and the other images to be processed do not include the target face image. Therefore, the terminal can select the to-be-processed image K as the base image.

In step S206, if it is determined according to the expression recognition result of each user that there is a face image of a user whose expression changes in the to-be-processed image, the terminal determines the user whose expression changes as a target user.

In step S207, for each target user, the terminal determines a face image whose expression meets the preset condition as the target face image of the target user.

In step S208, for each non-target user, the terminal obtains the eye value of each non-target user in each image to be processed, the eye value is a numerical value used to indicate the size of the eye, and the eye value is The image corresponding to the maximum value of is determined as the target face image of the non-target user.

In step S209, the terminal selects the image to be processed that contains the largest number of target face images as the base image.

For example, steps S206, S207, S208 and S209 may include:

According to the facial expression recognition results of each user, the terminal determines that there are facial images of users whose expressions have changed in the image to be processed, that is, the terminal determines that the expressions of some users in the image to be processed have changed. For example, the images to be processed are O, P, Q, R, S, T. According to the expression recognition result, the terminal determines that the expression of user C in the six frames of images ranges from not containing a smile to including a smile. The eye values of user C in the images O, P, Q, R, S, and T to be processed are 82, 80, 60, 50, 30, and 0, respectively. That is, from the image O to the image T, the eyes of the user C gradually become smaller, but the expression of the user C changes from not containing a smile to containing a smile. Moreover, user C is a girl, and when she smiles, her eyes also smile, making the eyes appear curved. It is precisely because user C's eyes are bent when smiling that user C's eyes gradually become smaller. In this case, the terminal may determine the user C whose expression changes as the target user.

For the target user, the terminal may determine the face image whose expression meets the preset condition as the target face image of the target user. For example, the preset condition can be the expression that the user smiles the brightest.

For example, since the terminal detects that there is a face image containing a smiling expression in the face image of user C, the terminal may determine the face image of user C who smiles most brilliantly among the face images containing the smiling expression as the face image of user C target face image. For example, if the terminal detects that the expression of user C in the image T to be processed is the brightest smile, the terminal may determine the face image of user C in the image T as the target face image of user C.

For each non-target user (that is, a user whose expression has not changed), the terminal may acquire the eye value of each non-target user in each image to be processed. Wherein, the eye value may be a numerical value used to represent the size of the eye. Then, the terminal may determine the face image corresponding to the maximum value of the eye values of each user as the target face image of the non-target user.

For example, the expressions of users A, B, and D in the image to be processed have not changed or the changes have been extremely small, so the terminal determines users A, B, and D as non-target users. Among them, the eye values of user A in the images O, P, Q, R, S, and T to be processed are 70, 71, 72, 72, 72, and 73, respectively, and the eye values of user B in the images O, P, Q, The eye values in R, S, and T are 80, 81, 81, 83, 82, and 82, respectively, and the eye values of user D in the images O, P, Q, R, S, and T to be processed are 82, 82, and 82, respectively. 83, 83, 84, 85, 84.

Then, the terminal can determine the face image of user A in the image to be processed T as the target face image of user A, and the face image of user B in the image to be processed R as the target face image of user B, The face image of user D in the processed image S is determined as the target face image of user D.

Then, the terminal may select the image to be processed that contains the largest number of target face images as the base image. For example, since the to-be-processed image T contains two target face images (target face images of users A and C), and the to-be-processed image R contains one target face image (the target face image of user B), the The processed image S contains a target face image (the target face image of user D). Therefore, the terminal can determine the image T to be processed as the base image.

In step S210, the terminal determines the face image to be replaced from the basic image, and the face image to be replaced is a non-target face image of the user.

In step S211, from the to-be-processed image, the terminal acquires a target face image for replacing each to-be-replaced face image, and each to-be-replaced face image and its corresponding target face image are people of the same user face image.

In step S212, the terminal uses the corresponding target face image to perform image replacement processing on each face image to be replaced to obtain a basic image that has undergone image replacement processing.

For example, steps S210, S211 and S212 may include:

After determining the base image, the terminal may determine the face image to be replaced from the base image. The terminal may determine the non-target face image in the base image as the face image to be replaced.

For example, in the base image T, the face images of user B and user D are not their respective target face images, so the terminal may determine the face images of user B and user D in the base image T as the face images to be replaced.

Then, the terminal may acquire, from images other than the base image to be processed, a target face image for replacing each face image to be replaced. It can be understood that each face image to be replaced and its corresponding target face image are face images of the same user.

For example, the face image of user B in the image to be processed R is the target face image of user B, and the face image of user D in the image to be processed S is the target face image of user D, then the terminal can obtain the image to be processed The face image of user B in R and the face image of user D in the image S to be processed.

After obtaining the target face image of each face image to be replaced, the terminal can use the corresponding target face image to perform image replacement processing on each face image to be replaced, thereby obtaining a basic image subjected to image replacement processing.

For example, in the base image T, the terminal may replace the face image of user B to be replaced in the base image T with the target face image of user B in the image R to be processed, and the face image to be replaced of user D in the base image T The image is replaced with the target face image of the user D in the image S to be processed, so as to obtain a basic image T that has undergone image replacement processing.

It can be understood that the face image of each user in the basic image T after image replacement processing is the target face image of the user. For example, after image replacement processing, in the base image T, the face images of users A, B, and D are the face images corresponding to the maximum values of their respective eye values, and the face image of user C is the face image of user C who smiles the most Brilliant face images too.

In one embodiment, the terminal may also numerically represent the expression of each user. For example, for a positive expression, the terminal may assign a positive value, and for a negative expression, the terminal may assign a negative value. For positive expressions, the terminal can assign different positive values according to the degree of expression of the expressions. For example, the terminal may assign a small positive number to a smiling expression with a small smile range, and the terminal may assign a large positive number to a smiling expression with a large smile range. In this case, the user's expressions are represented by expression values.

Then, the terminal can comprehensively determine the user's target face image from the images to be processed according to the expression value and the eye value. For example, for a face image of a user whose expression changes, the terminal may assign a larger weight to the expression value and a smaller weight to the eye value. For example, the expression value is weighted 85%, and the eye value is weighted 15%.

Taking the face image of user C as an example, the eye values of user C in the images O, P, Q, R, S, and T to be processed are 82, 80, 60, 50, 30, and 0, respectively. The eye values of user C in the images O, P, Q, R, S, and T to be processed are 10, 20, 30, 40, 50, and 60, respectively. Then, the comprehensive value of user C in the image O to be processed is 20.8 (82*15%+10*85%). Similarly, the comprehensive values of the images P, Q, R, S, and T to be processed are 29, 34.5, 41.5, 47, and 51 in sequence. Then, since the comprehensive value of the to-be-processed image T is the largest, the terminal may determine the face image of user C in the to-be-processed image T as the target face image of user C. Then, the terminal determines the to-be-processed image with the largest number of target face images as the base image.

In one embodiment, before the step of acquiring multiple frames of images to be processed containing human faces, the following steps may also be included:

When collecting an image including a human face, the terminal determines the target frame number according to the at least two frames of images collected.

Then, the step that the terminal acquires multiple frames of images to be processed including faces may include: from the multiple frames of images collected, the terminal acquires the number of images to be processed that is the target number of frames.

For example, after entering the camera preview interface, if it is detected that the terminal is collecting images containing faces, the terminal may determine a target frame number according to at least two collected images containing faces. In one embodiment, the target frame number may be greater than or equal to 2.

For example, when the terminal collects four frames of images containing a human face, the terminal can detect whether the position of the human face in the four frames of images is displaced. If there is no displacement or the displacement is very small, it can be considered that the face image in the image is relatively stable, that is, the user does not shake or turn his head in a large range. If displacement occurs, it can be considered that the face image is unstable, that is, the user shakes or turns his head, and the amplitude is large.

In one embodiment, whether the face in the image is displaced can be detected by the following method: after acquiring the four frames of images collected, the terminal can generate a coordinate system, and then the terminal can use the same method to convert each frame The image is placed into this coordinate system. After that, the terminal can acquire the coordinates of the feature points of the face image in each frame of images in the coordinate system. After obtaining the coordinates of the feature points of the face image in each frame of images in the coordinate system, the terminal can compare whether the coordinates of the feature points of the same face image in different images are the same. If they are the same, it can be considered that the face image in the image is not displaced. If they are different, it can be considered that the face image in the image is displaced. If it is detected that the face image is displaced, the terminal can obtain a specific displacement value. If the specific displacement value is within the preset value range, it can be considered that the displacement of the face image in the image is small. If the specific displacement value is outside the preset value range, it can be considered that the displacement of the face image in the image is relatively large.

In one embodiment, for example, if the face image is displaced, the target number of frames may be determined to be 4 frames. If the face image is not displaced, the target frame number can be determined to be 6 frames or 8 frames.

After the user presses the photographing button, the terminal may acquire images to be processed whose number is the target number of frames from the recently collected images.

In one embodiment, after the step of obtaining the basic image that has undergone image replacement processing, the following steps may also be included:

According to the image to be processed, the terminal performs image noise reduction processing on the basic image that has undergone image replacement processing.

For example, after obtaining the basic image subjected to the image replacement processing, the terminal may perform image noise reduction processing on the basic image subjected to the image replacement processing according to the image to be processed. For example, the terminal may acquire a set of continuously collected images including the base image, and perform multi-frame noise reduction processing on the base image after image replacement processing according to the set of images.

For example, since the base image is image T, the terminal may acquire images Q, R, and S to be processed, and perform multi-frame noise reduction processing on the base image T after image replacement processing according to images Q, R, and S.

In one embodiment, when performing multi-frame noise reduction, the terminal may first align the images Q, R, S, and T, and obtain the pixel value of each group of aligned pixels in the image. If the pixel values of the aligned pixels in the same group are similar, the terminal may calculate the average pixel value of the aligned pixels in the group, and then replace the pixel value of the corresponding pixel of the image T with the average pixel value. If the pixel values of the same group of aligned pixels are quite different, the pixel values in the image T may not be adjusted.

For example, pixel P1 in image Q, pixel P2 in image R, pixel P3 in image S, and pixel P4 in image T are a set of mutually aligned pixels, where the pixel value of P1 is 101, and the pixel value of P2 is 101. 102, the pixel value of P3 is 103, and the pixel value of P4 is 104, then the average pixel value of this group of aligned pixels is 102.5, then the terminal can adjust the pixel value of the P4 pixel in the image T from 104 to 102.5, Thereby, noise reduction processing is performed on the P4 pixel in the image T. If the pixel value of P1 is 80, the pixel value of P2 is 83, the pixel value of P3 is 90, and the pixel value of P4 is 103, then because their pixel values are quite different, the terminal may not adjust the pixel value of P4 at this time. That is, the pixel value of P4 remains unchanged at 104.

Please refer to FIG. 3 to FIG. 5 . FIG. 3 to FIG. 5 are schematic diagrams of scenes of a method for selecting an image provided by an embodiment of the present application.

In this embodiment, after entering the preview interface of the camera, if it is detected that the terminal is collecting face images, the terminal can collect the current environment parameters, and determine a target frame number according to the collected at least two frames of face images . The ambient parameter may be ambient light brightness.

If the terminal determines that the face in the image is not displaced (or the displacement is small) according to the collected face images in at least two frames, and the terminal is currently in a bright light environment, the terminal can determine the target frame number as 8 frames. If the terminal determines that the face in the image is not displaced (or the displacement is small) according to the collected face images in at least two frames, and is currently in a dark light environment, the terminal can determine the target frame number as 6 frames. If the terminal determines that the face in the images is displaced according to the collected face images in at least two frames, the terminal may determine the target frame number as 4 frames.

The terminal can save the collected images to the buffer queue. The buffer queue may be a fixed-length queue, for example, the buffer queue may store 10 frames of images newly collected by the terminal.

For example, five people, A, B, C, D, and E, go out to play and prepare to take pictures next to a landscape. Among them, A uses the terminal to take pictures of B first, as shown in Figure 3. For example, after entering the preview interface of the camera, the terminal collects a frame of image every 50 milliseconds according to the currently collected environmental parameters. Before A presses the camera button of the camera, the terminal can first obtain the 4 frames of images collected from the cache queue. It is understandable that these 4 frames of images all contain B's face image. Then, the terminal can detect whether the position of B's face image in the picture is displaced in these 4 frames of images. If there is no displacement or the displacement is small, it can be considered that B's face image is relatively stable, that is, B does not shake or turn his head in a large range. If displacement occurs, it can be considered that B's face image is unstable, that is, B shakes or turns his head, and the amplitude is large. For example, in this embodiment, the terminal detects that the position of the face image of B in the above-mentioned four frames of images does not shift in the picture.

Then, the terminal can acquire the current brightness of the ambient light, and judge whether it is currently in a dark light environment according to the brightness of the ambient light. For example, the terminal determines that it is currently in a dark light environment.

Afterwards, the terminal can determine a target frame number according to the information obtained above: the position of B's face image in the picture has not shifted, and it is currently in a dark light environment. For example, the determined target frame number is 6 frames.

After that, when A presses the camera button, the terminal can obtain 6 frames of collected images about B. For example, the terminal may acquire the most recently collected 6 frames of images about B from the cache queue. For example, in chronological order, the 6 frames of images are A, B, C, D, E, and F respectively. It can be understood that the six frames of images A, B, C, D, E, and F are the images to be processed obtained by the terminal.

After acquiring 6 frames of images, the terminal can perform facial expression recognition on the 6 frames of images, and detect the eye value of the face image in the image, where the eye value is a numerical value used to indicate the size of the eye. For example, the eye values of B in images A, B, C, D, E, and F are 80, 82, 83, 84, 85, and 84, respectively. For example, the terminal detects that the expression of user B has not changed in these 6 frames of images.

In the case where it is determined that the expression of user B in the image to be processed has not changed, since the 6 frames of images are the single-person images of B, the terminal can determine the frame of the image with the largest eye value among the 6 frames of images as The base image, ie, the image E, is determined as the base image.

After the image E is determined as the base image, the terminal may perform multi-frame noise reduction processing on the image E according to the images C, D, and F. After performing multi-frame noise reduction processing on the image E, the terminal may store the image E after the noise reduction processing in an album to become a photo. It can be understood that the image E is the big eye photo of B taken.

After that, Wu took a group photo for A, B, C, and D. For example, after entering the preview interface of the camera, the terminal detects that the positions of the faces of the four persons A, B, C, and D in the collected 4-frame images have not shifted in the screen, and they are currently in a dark environment. Based on this, the terminal determines that the target frame number is 6 frames.

After that, when E presses the photo button, the terminal can obtain 6 frames of collected images about A, B, C, and D, as shown in FIG. 4 . For example, the terminal may acquire the most recently acquired 6 frames of images about AED3 from the buffer queue. For example, according to time sequence, the six frames of images are O, P, Q, R, S, and T, respectively. It can be understood that these 6 frames of images are the images to be processed.

Afterwards, the terminal may perform expression recognition on the face image of each user in the 6 frames of images to be processed, so as to obtain the expression recognition result of each user. After obtaining the facial expression recognition result of each user in the image to be processed, the terminal may determine whether the facial expression of the user in the to-be-processed image has changed according to the facial expression recognition result of each user.

For example, according to the expression recognition result, the terminal determines that the expression of user C in the six frames of images ranges from not containing a smile to including a smile. The eye values of user C in the images O, P, Q, R, S, and T to be processed are 82, 80, 60, 50, 30, and 0, respectively. That is, from the image O to the image T, the eyes of the user C gradually become smaller, but the expression of the user C changes from not containing a smile to containing a smile. In this case, the terminal may determine the user C whose expression changes as the target user. For example, the face images of user C from image O to image T are shown in FIG. 5 .

Since the terminal detects that there is a face image containing a smiling expression in the face image of user C, the terminal can determine the face image of user C who smiles the brightest among the face images containing the smiling expression as the target person of user C face image. For example, if the terminal detects that the expression of user C in the image T to be processed is the brightest smile, the terminal may determine the face image of user C in the image T as the target face image of user C.

In addition, the expressions of users A, B, and D in the to-be-processed image do not change or the changes are extremely small, so the terminal determines users A, B, and D as non-target users. Among them, the eye values of user A in the images O, P, Q, R, S, and T to be processed obtained by the terminal are 70, 71, 72, 72, 72, and 73, respectively. User B is in the images O, P to be processed. , Q, R, S, and T eye values are 80, 81, 81, 83, 82, 82, respectively, and the eye values of user D in the images O, P, Q, R, S, and T to be processed are respectively 82, 83, 83, 84, 85, 84.

Then, the terminal can determine the face image of user A in the image to be processed T as the target face image of user A, and the face image of user B in the image to be processed R as the target face image of user B, The face image of user D in the processed image S is determined as the target face image of user D.

Then, the terminal may select the image to be processed that contains the largest number of target face images as the base image. For example, since the to-be-processed image T contains two target face images (target face images of users A and C), and the to-be-processed image R contains one target face image (the target face image of user B), the The processed image S contains a target face image (the target face image of user D). Therefore, the terminal can determine the image T to be processed as the base image.

In the base image T, since the face images of user B and user D are not their respective target face images, the terminal may determine the face images of user B and user D in the base image T as the face images to be replaced.

For example, the face image of user B in the image to be processed R is the target face image of user B, and the face image of user D in the image to be processed S is the target face image of user D, then the terminal can obtain the image to be processed The face image of user B in R and the face image of user D in the image S to be processed.

Then, the terminal can replace the face image to be replaced of user B in the base image T with the target face image of user B in the image to be processed R, and replace the face image to be replaced of user D in the base image T with the image to be processed The target face image of user D in S, so as to obtain the basic image T after image replacement processing.

It can be understood that the face image of each user in the basic image T after image replacement processing is the target face image of the user. For example, after image replacement processing, in the base image T, the face images of users A, B, and D are the face images corresponding to the maximum values of their respective eye values, and the face image of user C is the face image of user C who smiles the most Brilliant face images too.

Afterwards, the terminal may acquire the images Q, R, and S to be processed, and perform multi-frame noise reduction processing on the basic image T that has undergone image replacement processing according to the images Q, R, and S. Then, the terminal may store the image T subjected to noise reduction processing in an album as a photo.

It can be understood that in the photos stored in the album, the four persons A, B, C, and D are either in the big-eyed state or in the best expression (smile) state. Therefore, this embodiment can improve the imaging effect of the photo taken by the terminal and improve the user experience.

Please refer to FIG. 6 , which is a schematic structural diagram of an image selection apparatus provided by an embodiment of the present application. The image selection apparatus 300 may include: an acquisition module 301 , an identification module 302 , and a selection module 303 .

The acquiring module 301 is configured to acquire multiple frames of images to be processed including human faces.

The recognition module 302 is configured to perform facial expression recognition on the face image of each user in the to-be-processed image to obtain the facial expression recognition result of each user.

The selection module 303 is configured to select a basic image from the to-be-processed images according to the expression recognition result of each user.

In one embodiment, the selection module 303 may be used to:

If it is determined that the expressions of all users in the to-be-processed image have not changed according to the expression recognition result of each user, the eye value of each user in each of the to-be-processed images is obtained, and the eye value is A numerical value used to represent the size of the eye;

According to the eye value of each user in each of the to-be-processed images, a base image is selected from the to-be-processed images.

In one embodiment, the selection module 303 may be used to:

Determine the target face image of each user according to the eye value of each user in each of the to-be-processed images, and the target face image is the image corresponding to the maximum value of the user's eye values;

The to-be-processed image containing the largest number of target face images is selected as the base image.

In one embodiment, the selection module 303 may be used to:

If it is determined according to the expression recognition result of each user that there is a face image of a user whose expression has changed in the to-be-processed image, then the user whose expression has changed is determined as a target user;

For each target user, determining a face image whose expression meets the preset condition as the target face image of the target user;

For each non-target user, obtain the eye value of each non-target user in each of the images to be processed, the eye value is a numerical value used to represent the size of the eye, and the maximum value of the eye values The corresponding image is determined as the target face image of the non-target user;

The to-be-processed image containing the largest number of target face images is selected as the base image.

Please also refer to FIG. 7 . FIG. 7 is another schematic structural diagram of an image selection apparatus provided by an embodiment of the present application. In an embodiment, the image selection apparatus 300 may further include: a collection module 304 and a processing module 305 .

The collection module 304 is configured to determine the target frame number according to the at least two frames of images collected when collecting an image containing a human face.

Then, the obtaining module 301 may be configured to obtain, from the collected multi-frame images, the to-be-processed images whose number is the target number of frames.

The processing module 305 is configured to: after the step of selecting a base image from the to-be-processed image:

Determine the face image to be replaced from the base image, and the face image to be replaced is the non-target face image of the user;

From the image to be processed, obtain a target face image for replacing each face image to be replaced, and each face image to be replaced and its corresponding target face image are the face images of the same user;

Perform image replacement processing on each face image to be replaced by using the corresponding target face image to obtain a basic image that has undergone image replacement processing.

In one embodiment, the processing module 305 may be further configured to: perform image noise reduction processing on the basic image that has undergone image replacement processing according to the to-be-processed image.

An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed on a computer, causes the computer to execute the steps in the image selection method provided by this embodiment .

An embodiment of the present application further provides an electronic device, including a memory and a processor, where the processor is configured to execute the steps in the image selection method provided by this embodiment by calling a computer program stored in the memory.

For example, the above-mentioned electronic device may be a mobile terminal such as a tablet computer or a smart phone. Please refer to FIG. 8 , which is a schematic structural diagram of a mobile terminal according to an embodiment of the present application.

The mobile terminal 400 may include components such as a camera module 401, a memory 402, a processor 403, and the like. Those skilled in the art can understand that the structure of the mobile terminal shown in FIG. 8 does not constitute a limitation on the mobile terminal, and may include more or less components than shown, or combine some components, or arrange different components.

The camera module 401 may include a single camera module and a dual camera module.

Memory 402 may be used to store applications and data. The application program stored in the memory 402 contains executable code. Applications can be composed of various functional modules. The processor 403 executes various functional applications and data processing by executing the application programs stored in the memory 402 .

The processor 403 is the control center of the mobile terminal, uses various interfaces and lines to connect various parts of the entire mobile terminal, and executes the mobile terminal by running or executing the application program stored in the memory 402 and calling the data stored in the memory 402. various functions and processing data, so as to monitor the mobile terminal as a whole.

In this embodiment, the processor 403 in the mobile terminal loads the executable code corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 403 executes the executable code stored in the memory The application in 402, thus implementing the steps:

Acquiring multiple frames of images to be processed that include faces; performing facial expression recognition on the facial images of each user in the images to be processed, to obtain an expression recognition result for each user; according to the facial expression recognition results for each user, from A base image is selected from the to-be-processed images.

Embodiments of the present invention also provide an electronic device. The above electronic device includes an image processing circuit, and the image processing circuit may be implemented by hardware and/or software components, and may include various processing units that define an ISP (Image Signal Processing, image signal processing) pipeline. FIG. 9 is a schematic structural diagram of an image processing circuit in one embodiment. As shown in FIG. 9 , for the convenience of description, only various aspects of the image processing technology related to the embodiments of the present invention are shown.

As shown in FIG. 9 , the image processing circuit includes an image signal processor 540 and a control logic 550 . Image data captured by imaging device 510 is first processed by image signal processor 540 , which analyzes the image data to capture image statistics that can be used to determine and/or control one or more parameters of imaging device 510 . Imaging device 510 may include a camera having one or more lenses 511 and an image sensor 512 . Image sensor 512 may include an array of color filters (eg, Bayer filters), image sensor 512 may acquire light intensity and wavelength information captured with each imaging pixel of image sensor 512 and provide a set of raw image data. The sensor 520 may provide raw image data to the image signal processor 540 based on the sensor 520 interface type. The sensor 520 interface may utilize a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

The image signal processor 540 processes raw image data pixel by pixel in various formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the image signal processor 540 may perform one or more image processing operations on the raw image data, collecting statistical information about the image data. Among them, the image processing operations can be performed with the same or different bit depth precision.

Image signal processor 540 may also receive pixel data from image memory 530 . For example, the raw pixel data is sent from the sensor 520 interface to the image memory 530, and the raw pixel data in the image memory 530 is provided to the image signal processor 540 for processing. The image memory 530 may be a part of a memory device, a storage device, or an independent dedicated memory in an electronic device, and may include a DMA (Direct Memory Access, direct memory access) feature.

When receiving raw image data from the sensor 520 interface or from the image memory 530, the image signal processor 540 may perform one or more image processing operations, such as temporal filtering. The processed image data may be sent to image memory 530 for additional processing before being displayed. The image signal processor 540 receives processed data from the image memory 530 and performs image data processing in the original domain and in the RGB and YCbCr color spaces on the processed data. The processed image data may be output to the display 570 for viewing by a user and/or further processed by a graphics engine or a GPU (Graphics Processing Unit, graphics processor). In addition, the output of the image signal processor 540 may also be sent to the image memory 530 , and the display 570 may read image data from the image memory 530 . In one embodiment, image memory 530 may be configured to implement one or more frame buffers. Also, the output of the image signal processor 540 may be sent to the encoder/decoder 560 for encoding/decoding the image data. The encoded image data may be saved and decompressed prior to display on the display 570 device. The encoder/decoder 560 may be implemented by a CPU or GPU or a coprocessor.

Statistics determined by image signal processor 540 may be sent to control logic 550 . For example, the statistics may include image sensor 512 statistics such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, lens 511 shading correction, and the like. The control logic 550 may include a processor and/or microcontroller executing one or more routines (eg, firmware) that may determine control parameters of the imaging device 510 and control of the imaging device 510 based on the received statistics parameter. For example, control parameters may include sensor 520 control parameters (eg, gain, integration time for exposure control), camera flash control parameters, lens 511 control parameters (eg, focal length for focus or zoom), or a combination of these parameters. ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (eg, during RGB processing), and lens 511 shading correction parameters.

The following are the steps of implementing the image processing method provided by this embodiment by using the image processing technology in FIG. 9 :

Acquiring multiple frames of images to be processed that include faces; performing facial expression recognition on the facial images of each user in the images to be processed, to obtain an expression recognition result for each user; according to the facial expression recognition results for each user, from A base image is selected from the to-be-processed images.

In one embodiment, when the electronic device performs the step of selecting a basic image from the to-be-processed image according to the expression recognition result of each user, it may perform: if it is determined according to the expression recognition result of each user The expressions of all users in the images to be processed have not changed, then the eye value of each user in each of the images to be processed is obtained, and the eye value is a numerical value used to indicate the size of the eye; An eye value of each user in the to-be-processed image, and a base image is selected from the to-be-processed image.

In one embodiment, when the electronic device performs the step of selecting a basic image from the images to be processed according to the eye value of each user in each of the images to be processed, the electronic device may perform: according to each of the images to be processed The eye value of each user in the image to be processed, and the target face image of each user is determined, and the target face image is the image corresponding to the maximum value of the user's eye value; the target face image will be included The image to be processed with the largest number is selected as the base image.

In one embodiment, when the electronic device performs the step of selecting a basic image from the to-be-processed image according to the expression recognition result of each user, it may perform: if it is determined according to the expression recognition result of each user If there is a face image of a user whose expression has changed in the image to be processed, the user whose expression has changed is determined as a target user; for each target user, a face image whose expression meets preset conditions is determined. is the target face image of the target user; for each non-target user, obtain the eye value of each non-target user in each of the to-be-processed images, and the eye value is used to represent the size of the eye. value, the image corresponding to the maximum value of the eye value is determined as the target face image of the non-target user; the to-be-processed image containing the largest number of target face images is selected as the base image.

In one embodiment, before the step of acquiring multiple frames of images to be processed containing human faces, the electronic device may further perform: when acquiring images containing human faces, determine the target according to the at least two frames of the acquired images number of frames;

Then, when the electronic device performs the step of acquiring multiple frames of images to be processed that include a human face, it may perform: from the multiple frames of images collected, the number of images to be processed is the target number of frames acquired;

After the step of selecting a base image from the image to be processed, the electronic identification can also perform: determining a face image to be replaced from the base image, and the face image to be replaced is a non-target face of the user Image; From the described image to be processed, obtain the target face image for replacing each face image to be replaced, and each face image to be replaced and its corresponding target face image are the face images of the same user; Perform image replacement processing on each face image to be replaced by using the corresponding target face image to obtain a basic image that has undergone image replacement processing.

In an embodiment, after the step of obtaining the basic image subjected to the image replacement processing, the electronic device may further perform: according to the to-be-processed image, perform image noise reduction processing on the basic image subjected to the image replacement processing .

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the detailed description of the image selection method above, which will not be repeated here.

The image selection apparatus provided in the embodiment of the present application and the image selection method in the above embodiments belong to the same concept, and any of the image selection method provided in the image selection apparatus can be executed on the image selection apparatus The specific implementation process is described in the image selection method embodiment, which will not be repeated here.

It should be noted that, for the image selection method described in the embodiments of the present application, those of ordinary skill in the art can understand that all or part of the process for implementing the image selection method described in the embodiments of the present application can be controlled by a computer program. The computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process can include, for example, the implementation of the image selection method. example process. The storage medium may be a magnetic disk, an optical disk, a read only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), and the like.

For the image selection apparatus of the embodiments of the present application, each functional module may be integrated in one processing chip, or each module may exist physically alone, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, etc. .

The method, device, storage medium, and electronic device for selecting an image provided by the embodiments of the present application have been described in detail above. The principles and implementations of the present invention are described with specific examples in this paper. It is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific embodiments and application scope. The contents of the description should not be construed as limiting the present invention.

Claims (6)

1. the selection method of an image, is characterized in that, comprises: When collecting an image containing a face, determine the target frame number according to whether the face in the collected at least two frames of the face image is displaced and the brightness of the ambient light; From the collected multiple frames of images containing faces, obtain images to be processed whose number is the target number of frames; Perform facial expression recognition on the face image of each user in the image to be processed to obtain the facial expression recognition result of each user; If it is determined that the expressions of all users in the to-be-processed image have not changed according to the expression recognition result of each user, the eye value of each user in each of the to-be-processed images is obtained, and the eye value is A numerical value used to represent the size of the eye; according to the eye value of each user in each of the images to be processed, determine the target face image of each user, and the target face image is the value of the user's eye. The image corresponding to the maximum value of ; the to-be-processed image containing the largest number of target face images is selected as the base image; Or, if it is determined that there is a face image of a user whose expression changes in the to-be-processed image according to the expression recognition result of each user, the user whose expression changes are determined as the target user; for each target user The user determines the face image whose expression meets the preset conditions as the target face image of the target user; for each non-target user, obtains the eye value of each non-target user in each of the images to be processed, The eye value is a numerical value used to represent the size of the eye, and the image corresponding to the maximum value in the eye value is determined as the target face image of the non-target user; The image is selected as the base image. 2. The method for selecting an image according to claim 1, characterized in that, before the step of obtaining the multi-frame images to be processed comprising a human face, further comprising: When collecting an image containing a human face, determine the target frame number according to the at least two frames of images collected; The step of obtaining multiple frames of images to be processed including faces includes: from the collected multiple frames of images, obtaining images to be processed whose number is the target number of frames; After the step of selecting a base image from the to-be-processed image, the method further includes: Determine the face image to be replaced from the base image, and the face image to be replaced is the non-target face image of the user; From the image to be processed, obtain a target face image for replacing each face image to be replaced, and each face image to be replaced and its corresponding target face image are the face images of the same user; Perform image replacement processing on each face image to be replaced by using the corresponding target face image to obtain a basic image that has undergone image replacement processing. 3. The selection method of image according to claim 2, is characterized in that, after the described step of obtaining the basic image processed through image replacement, also comprises: According to the to-be-processed image, image noise reduction processing is performed on the basic image that has undergone image replacement processing. 4. an image selection device, is characterized in that, comprises: The acquisition module is used to determine the number of target frames according to whether the faces in the collected at least two frames of face images are displaced and the brightness of the ambient light when collecting images containing faces; In the image of , obtain the to-be-processed image whose quantity is the target frame number; A recognition module, for performing facial expression recognition on the face image of each user in the image to be processed, to obtain the facial expression recognition result of each user; The selection module is used to obtain the eye value of each user in each of the to-be-processed images if it is determined that the expressions of all users in the to-be-processed image have not changed according to the expression recognition result of each user, and the The eye value is a numerical value used to represent the size of the eye; according to the eye value of each user in each of the to-be-processed images, the target face image of each user is determined, and the target face image is the user The image corresponding to the maximum value in the eye value; the to-be-processed image containing the largest number of target face images is selected as the basic image; If there is a face image of the user whose expression changes, the user whose expression changes is determined as the target user; for each target user, the face image whose expression meets the preset conditions is determined as the target user's face image. target face image; for each non-target user, obtain the eye value of each non-target user in each of the to-be-processed images, the eye value is a numerical value used to represent the size of the eye, and the eye value is The image corresponding to the maximum value in the target face image is determined as the target face image of the non-target user; the to-be-processed image containing the largest number of target face images is selected as the basic image. 5. A storage medium on which a computer program is stored, characterized in that, when the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 3. 6. An electronic device, comprising a memory and a processor, wherein the processor is configured to execute the method according to any one of claims 1 to 3 by invoking a computer program stored in the memory.

Images