CN109766759A - Emotion recognition method and related products - Google Patents

Abstract

Embodiments of the present application provide an emotion recognition method and related products. The method includes: acquiring video clips and audio clips for a target user in a specified time period; analyzing the video clips to obtain the target user's target Body behavior and target facial expression; perform parameter extraction on the audio segment to obtain the target voice feature parameters of the target user; determine the target user according to the target body behavior, the facial expression and the voice feature parameters target emotion. Through the embodiments of the present application, body behavior, facial expressions, and voice features can be parsed from the video. The above three dimensions all reflect the user's emotions to a certain extent. Furthermore, the user's emotions can be jointly determined through the three dimensions, which can accurately Identify user emotions.

Description

Emotion recognition method and related products

technical field

The present application relates to the technical field of video surveillance, in particular to an emotion recognition method and related products.

Background technique

With the rapid development of economy, society and culture, the influence at home and abroad is increasing day by day, and more and more migrants are flowing to cities. While accelerating the process of urbanization, these population increases also bring greater challenges to urban management. , Video surveillance provides technical support for urban security. At present, cameras have been deployed in the city, and cameras can effectively monitor the security of the city and provide effective help for the security of relevant institutions. However, in life, people's expressions are mainly recognized by obtaining image information of people through cameras, and then people's emotions are analyzed. However, the accuracy of emotions recognized by this single judgment method is low.

SUMMARY OF THE INVENTION

The embodiments of the present application provide an emotion recognition method and related products, which can accurately recognize the user's facial expression.

A first aspect of the embodiments of the present application provides an emotion recognition method, including:

Obtain the video clips and audio clips of the target user in the specified time period;

The video clip is analyzed to obtain the target limb behavior and the target facial expression of the target user;

Perform parameter extraction on the audio segment to obtain the target speech feature parameters of the target user;

The target emotion of the target user is determined according to the target body behavior, the facial expression and the speech feature parameters.

Optionally, the method further includes:

authenticate the target user;

After the target user's identity verification is passed, obtain an indoor map;

Marking the location of the target user in the indoor map to obtain the target area where the target user is located;

According to a preset mapping relationship between emotions and control parameters, determine a target control parameter corresponding to the target emotion for controlling at least one smart home device corresponding to the target area;

The at least one smart home device is adjusted according to the target control parameter.

Further optionally, the performing identity verification on the target user includes:

obtaining the first face image of the target user;

Perform image quality evaluation on the first face image to obtain a target image quality evaluation value;

According to the mapping relationship between the preset image quality evaluation value and the matching threshold, determine the target matching threshold corresponding to the target image quality evaluation value;

performing contour extraction on the first face image to obtain a first peripheral contour;

performing feature point extraction on the first face image to obtain a first feature point set;

Matching the first peripheral contour with the second peripheral contour of the preset face template to obtain a first matching value;

The first feature point set is matched with the second feature point set of the preset face template to obtain a second matching value;

determining a target matching value according to the first matching value and the second matching value;

When the target matching value is greater than the target matching threshold, it is confirmed that the target user's identity verification is passed.

A second aspect of the embodiment of the present application provides an emotion recognition device, comprising:

an acquisition unit for acquiring a video clip and an audio clip for the target user in a specified time period;

an analysis unit, configured to analyze the video clip to obtain the target limb behavior and the target facial expression of the target user;

an extraction unit, configured to perform parameter extraction on the audio segment to obtain the target speech feature parameter of the target user;

A decision unit, configured to decide the target emotion of the target user according to the target body behavior, the facial expression and the speech feature parameter.

In a third aspect, embodiments of the present application provide an emotion recognition device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured by The above-mentioned processor is executed, and the above-mentioned program includes instructions for executing the steps in the first aspect of the embodiments of the present application.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program causes a computer to execute the computer program as described in the first embodiment of the present application. Some or all of the steps described in an aspect.

In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute as implemented in the present application. Examples include some or all of the steps described in the first aspect. The computer program product may be a software installation package.

Implementing the embodiments of the present application has the following beneficial effects:

It can be seen that, through the emotion recognition method and related products of the embodiments of the present application, video clips and audio clips for the target user in a specified time period are obtained, and the video clips are analyzed to obtain the target body behavior and target facial expressions of the target user. Extract the parameters of the audio clip to obtain the target voice feature parameters of the target user, and decide the target user's target emotion according to the target body behavior, facial expression and voice feature parameters. The above three dimensions all reflect the user's emotions to a certain extent, and then, the user's emotions can be jointly determined through the three dimensions, and the user's emotions can be accurately identified.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1A is a schematic flowchart of an embodiment of an emotion recognition method provided by an embodiment of the present application;

FIG. 1B is a schematic diagram illustrating another emotion recognition method provided by an embodiment of the present application;

2 is a schematic flowchart of an embodiment of another emotion recognition method provided by an embodiment of the present application;

3 is a schematic structural diagram of an embodiment of an emotion recognition device provided by an embodiment of the present application;

FIG. 4 is a schematic structural diagram of an embodiment of another emotion recognition apparatus provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

The terms "first", "second", "third" and "fourth" in the description and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order . Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes For other steps or units inherent to these processes, methods, products or devices.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification is not necessarily all referring to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

The emotion recognition device described in the embodiments of this application may include smart phones (such as Android phones, iOS phones, Windows Phone phones, etc.), tablet computers, PDAs, notebook computers, video matrixes, monitoring platforms, and mobile Internet devices (MID, Mobile Internet Devices) or wearable devices, etc., the above are only examples, not exhaustive, including but not limited to the above devices, of course, the above emotion recognition device can also be a server.

It should be noted that the emotion recognition device in this embodiment of the present application may be connected to multiple cameras, each camera may be used to capture video images, and each camera may have a corresponding position mark, or may have a corresponding number. Usually, cameras can be installed in public places, such as schools, museums, intersections, pedestrian streets, office buildings, garages, airports, hospitals, subway stations, stations, bus stops, supermarkets, hotels, entertainment venues, and so on. After the camera captures the video image, the video image can be saved to the memory of the system where the emotion recognition device is located. There may be multiple image banks stored in the memory, and each image bank may contain different video images of the same person. Of course, each image bank may also be used to store a video image of an area or a video image captured by a specified camera.

Further optionally, in the embodiment of the present application, each frame of video image captured by the camera corresponds to one attribute information, and the attribute information is at least one of the following: the shooting time of the video image, the position of the video image, the attribute parameters of the video image ( format, size, resolution, etc.), the number of the video image, and the character attributes of the person in the video image. The character attributes in the above-mentioned video image may include, but are not limited to: the number of characters in the video image, the position of the character, the angle of the character, the age, the image quality, and the like.

It should be further noted that the video image collected by each camera is usually a dynamic face image. Therefore, in the embodiment of the present application, the angle information of the face image may be planned, and the above-mentioned angle information may include but is not limited to: horizontal rotation angle. , pitch angle or inclination. For example, it can be defined that the dynamic face image data requires that the distance between the eyes be not less than 30 pixels, and more than 60 pixels is recommended. The pan angle does not exceed ±30°, the pitch angle does not exceed ±20°, and the tilt angle does not exceed ±45°. It is recommended that the pan angle does not exceed ±15°, the pitch angle does not exceed ±10°, and the tilt angle does not exceed ±15°. For example, it can also screen whether the face image is blocked by other objects. Usually, accessories should not block the main area of the face, such as dark sunglasses, masks and exaggerated jewelry. Of course, the camera may also be covered with dust. , causing the face image to be occluded. The image formats of the video images in the embodiments of the present application may include but are not limited to: BMP, JPEG, JPEG2000, PNG, etc., and the size may be between 10-30KB, and each video image may also correspond to a shooting time, and Information such as the unified camera number of the video image, the link to the large panorama image corresponding to the face image (the face image and the global image establish a feature correspondence relationship file).

Please refer to FIG. 1A , which is a schematic flowchart of an embodiment of an emotion recognition method provided by an embodiment of the present application. The emotion recognition method described in this embodiment includes the following steps:

101. Acquire a video clip and an audio clip for a target user in a specified time period.

The above specified time period may be set by the user or the system defaults. The above-mentioned video clips and audio clips may be for the same physical scene and the same specified time period. The target user can be any user, and the target user can be tracked and photographed to obtain video clips of the target user, and at the same time, the shooting site can be photographed to obtain audio clips.

102. Analyze the video clip to obtain the target body behavior and the target facial expression of the target user.

The video clips are a series of body movements of the target user. Therefore, these body movements can be identified to obtain the target body behavior. The target body behavior can be at least one behavior. In the embodiment of the present application, the body behavior can be at least one of the following Types: walking, akimbo, running, raising hands, typing on a keyboard, thinking, holding hands, fighting, etc., which are not limited here. Of course, a human face may also be included in the video clip, and by performing facial expression recognition on the human face, the target facial expression can be obtained. Surprise, embarrassment, excitement, nervousness, etc., are not limited here.

Optionally, in the above step 102, analyzing the video clip to obtain the physical behavior and facial expression of the target user, which may include the following steps:

21. Analyze the video clip to obtain multiple frames of video images;

22. Perform image segmentation on the multi-frame video images to obtain a plurality of target images, each target image being a human body image of the target user;

23. Perform behavior recognition according to the multiple target images to obtain the target limb behavior;

24. Perform face recognition on the multiple target images to obtain multiple face images;

25. Perform facial expression recognition on the plurality of face images to obtain a plurality of expressions, and use the facial expression that appears the most frequently among the plurality of facial expressions as the target facial expression.

Among them, in the specific implementation, video clips can be analyzed to obtain multiple frames of video images. Of course, not every frame of video images includes human images. Therefore, image segmentation of multiple frames of video images can be performed to obtain multiple target images. Each target image includes the entire human body image of the user, and further, behavior recognition can be performed on multiple target images to obtain the behavior of the target limbs. Specifically, multiple target images can be input into a preset neural network model to obtain at least limbs. Behavior, the preset neural network model can be set by the user or the system defaults, for example, the convolutional neural network model can also perform face recognition on multiple target images, obtain multiple face images, and express expressions on the multiple face images. Recognition, since different face images can correspond to different expressions, multiple expressions can be obtained, and then the most frequent expression among the multiple expressions can be used as the target facial expression. In this way, a series of video images can be parsed out. Body behavior can also accurately identify the user's facial expressions.

103. Perform parameter extraction on the audio segment to obtain target speech feature parameters of the target user.

Wherein, in the embodiment of the present application, the speech feature parameters may include at least one of the following: speech rate, intonation, keywords, timbre, etc., which are not limited here, and timbre is related to the frequency of the sound. Since some information is included in the speech, the user's mood can be reflected from the side. For example, if the user says "beautiful", it can indicate that his mood is joy or joy. For another example, if the user says "embarrassed", it can indicate that the user's emotion is embarrassing.

Optionally, the target speech feature parameters include at least one of the following: target speech rate, target intonation, and at least one target keyword; in the above step 103, parameter extraction is performed on the audio segment to obtain the target user's target speech The characteristic parameters can include the following steps:

31. Perform semantic analysis on the audio segment to obtain a plurality of characters;

32. Determine the pronunciation duration corresponding to the multiple characters, and determine the speech rate according to the multiple characters and the pronunciation duration;

or,

33. Perform character segmentation on the multiple characters to obtain multiple keywords;

34. Matching the multiple keywords with a preset keyword set to obtain the at least one target keyword;

or,

35. Extract the waveform diagram of the voice segment;

36. Analyze the waveform diagram to obtain the target intonation.

Among them, in the specific implementation, the audio clip can be semantically analyzed to obtain a plurality of characters, and each character in the plurality of characters can correspond to a pronunciation duration, and further, the pronunciation duration corresponding to the plurality of characters can be determined, and the speech rate=multiple Pronunciation duration corresponding to each character/total number of multiple characters. Further, multiple characters can be divided into multiple characters to obtain multiple keywords, and each keyword can be understood as a word. In the embodiment of the present application, it is also possible to Pre-store a preset keyword set, the preset keyword set may include at least one keyword, and then match the keyword with the preset keyword set to obtain at least one target keyword that is successfully matched. , the waveform diagram of the speech segment can be extracted, and the waveform diagram can be analyzed to obtain the target intonation. For example, the average amplitude can be used as the target intonation. In this way, the feature parameters of multiple dimensions can be obtained by analyzing the speech segment.

104. Determine the target emotion of the target user according to the target body behavior, the facial expression, and the speech feature parameter.

Among them, since body behavior, facial emotion, and speech feature parameters all reflect the user's emotion to a certain extent, the final target emotion can be obtained by making decisions through these three dimensions.

Optionally, in the above step 104, the target emotion of the target user is determined according to the target body behavior, the target facial expression and the target voice feature parameter, which may include the following steps:

41. Determine a first emotion set corresponding to the target limb behavior, the first emotion set includes at least one emotion, and each emotion corresponds to a first emotion probability value;

42. Determine the second emotion set corresponding to the target facial expression, the second emotion set includes at least one emotion, and each emotion corresponds to a second emotion probability value;

43. Determine a third emotion set corresponding to the target speech feature parameter, the third emotion set includes at least one emotion, and each emotion corresponds to a third emotion probability value;

44. Obtain the first weight corresponding to the body behavior, the second weight corresponding to the facial expression, and the third weight corresponding to the speech feature parameter;

45. Determine each type of emotion according to the first weight, the second weight, the third weight, the first emotion set, the second emotion set, and the third emotion set. Score value, get multiple score values;

46. Select the maximum value from the plurality of score values, and use the emotion corresponding to the maximum value as the target emotion.

1B shows that each physical behavior can correspond to at least one emotion, each facial expression can correspond to at least one emotion, each speech feature parameter can also correspond to at least one emotion, and each emotion can also correspond to at least one emotion. Corresponding to an emotion probability value, the final target emotion can be determined from all the above emotions and the emotion probability value.

Specifically, different limb behaviors may correspond to different emotions. Therefore, the emotion recognition device may pre-store a mapping relationship between limb behaviors and emotions, and then, according to the mapping relationship, at least one emotion corresponding to the target limb behavior may be determined, and the The at least one emotion may constitute a first emotion set. Of course, each emotion may correspond to a first emotion probability value, and the first emotion probability value may be preset or system default. Of course, different facial expressions can also correspond to different emotions. Therefore, a mapping relationship between facial expressions and emotions can be pre-stored in the emotion recognition device, and at least one emotion corresponding to the target facial expression can be determined through the mapping relationship. One emotion may constitute a second emotion set, each emotion in the second emotion set may correspond to a second emotion probability value, and the second emotion probability value may be preset or system default. In addition, each speech feature parameter may also correspond to a different emotion. Therefore, the emotion recognition device may also pre-store a mapping relationship between the speech feature parameter and emotion, and then determine at least one corresponding to the target speech feature parameter according to the mapping relationship. Emotion, the at least one emotion may constitute a third emotion set, each emotion in the third emotion set may correspond to a third emotion probability value, and the third emotion probability value may be preset or system default. Further, the first weight corresponding to the body behavior, the second weight corresponding to the facial expression, and the third weight corresponding to the speech feature parameter can also be obtained. Of course, the first weight, the second weight, the third weight The weight can be preset, the first weight + the second weight + the third weight=1. The score value of each emotion = emotion value * emotion probability value, thus, the score value of each emotion can be obtained, and then the score value corresponding to each type of emotion can be determined, and the corresponding score value of each type can be obtained, from The maximum value is selected from each type of score value, and the emotion corresponding to the maximum value is used as the target emotion. In this way, the final emotion can be determined through the three dimensions of body behavior, facial expression, and voice, and the error of single-dimensional emotion recognition is large. , and multiple dimensions can weaken the recognition error caused by a single dimension and improve the accuracy of emotion recognition.

The above-mentioned first probability value, second probability value, and third probability value can all be obtained from big data. Specifically, a large number of users' physical behaviors, facial expressions, and voices are collected, and the probability values of emotions corresponding to their physical behaviors are analyzed. .

Further optionally, the target limb behavior includes at least one limb behavior; the above step 23, performing behavior recognition according to the multiple target images, to obtain the target limb behavior, can be implemented as follows:

Inputting the multiple target images into a preset neural network model to obtain the at least one limb behavior and the recognition probability corresponding to each limb behavior;

Then, in the above step 41, determine the first emotion set corresponding to the target limb behavior, the first emotion set includes at least one emotion, and each emotion corresponds to a first emotion probability value, which may include the following steps:

411. Determine the emotion corresponding to each limb behavior in the at least one limb behavior according to the preset mapping relationship between the limb behavior and the emotion, and obtain at least one emotion, and each emotion corresponds to a preset probability value;

412. Perform an operation according to the identification probability corresponding to each limb behavior, and each emotion in the at least one emotion corresponds to a preset probability value to obtain at least one first emotion probability value, and each emotion corresponds to a first emotion probability. value.

Among them, the above-mentioned target limb behavior may include at least one limb behavior. Due to behavior recognition, it is possible to identify one limb action as multiple limb behaviors, and each limb behavior may correspond to a recognition probability. The image is input into the preset neural network model, and the preset neural network model performs behavior recognition on each target image, and at least one limb behavior can be obtained, and each limb behavior can correspond to a recognition probability. The emotion recognition device can pre-store a preset mapping relationship between body behaviors and emotions, and then, according to the mapping relationship, determine the emotion corresponding to at least one body behavior, and obtain at least one emotion, and each emotion can correspond to a preset probability value. , the preset probability value can be preset, and then, according to the recognition probability corresponding to each limb behavior and the preset probability value corresponding to each emotion in the at least one emotion, at least one first emotion probability value is obtained, that is, the first emotion probability value is obtained. Emotion probability value=recognition probability*preset probability value, in this way, the final emotion probability value can be accurately determined through the recognition accuracy of the body behavior and the emotion probability value, which is conducive to the accurate analysis of the user's emotion.

Of course, in the above steps 42-43, a face image or voice can be input into a neural network model, and corresponding emotions can be obtained through the neural network model.

For example, the emotion recognition device can use a camera to capture the information of the test object, including a snapshot image based on a capture camera and a video containing the test object within a certain period of time, wherein the snapshot image is used to identify the facial expression information of the test object, and the video content can be used for Identify the physical behavior of the test object, and also test the facial expression of the object. Specifically, it can be obtained by analyzing the face information captured by the camera, and the physical behavior of the test object in the video clip can be analyzed. Specifically, using The deep learning pre-training model takes facial expression information and body behavior as input at the same time, and outputs the multi-category probability of emotion, which is used as the emotion prediction of the test object. Of course, the categories of emotions can be determined according to specific situations, for example, including happiness, excitement, tension, anger, etc., which are not limited here. In addition, the emotion recognition device can use a recording device (such as a speaker) to obtain the audio clips of the test object at the same time, and the audio content is used to identify the test object's voice, speech rate and intonation. Finally, the test object's voice, speech rate can be used. And intonation, input into the pre-training model based on deep learning, and output the multi-classification probability of emotion, so as to predict the emotional result based on the audio of the test object. , and output the final comprehensive emotion judgment result.

Further optionally, after the above step 104, the following steps may also be included:

A1. Authenticate the target user;

A2. After the target user's identity verification is passed, obtain an indoor map;

A3. Mark the location of the target user in the indoor map to obtain the target area where the target user is located;

A4. Determine the target control parameter corresponding to the target emotion for controlling at least one smart home device corresponding to the target area according to the preset mapping relationship between the emotion and the control parameter;

A5. Adjust the at least one smart home device according to the target control parameter.

The emotion recognition device may pre-store the mapping relationship between preset emotions and control parameters, and the above-mentioned smart home equipment may be at least one of the following: smart air conditioners, smart humidifiers, smart speakers, smart lights, smart curtains, smart Massage chairs, smart TVs, etc., are not limited here. In the specific implementation, the target user can be authenticated. After the identity verification is passed, the indoor map of the current scene can be obtained, and the location of the target user can be marked in the indoor map to obtain the target area where the target user is located. The above-mentioned pre-stored mapping relationship between preset emotions and control parameters can determine the target control parameter corresponding to the target emotion for controlling at least one smart home device corresponding to the target area, and the above-mentioned control parameter can be at least one of the following: temperature Adjustment parameters, humidity adjustment parameters, speaker playback parameters (for example, volume, track, sound effects, etc.), brightness or color temperature adjustment parameters, curtain control parameters (closeness parameter), massage chair control parameters (massage method, massage duration), TV Control parameters (volume, surface, brightness, color temperature, etc.), and then adjust at least one smart home device according to the target control parameter, so that different environments can be adjusted according to different emotions, which helps to ease the user's emotions, Improve user experience.

Further optionally, the above-mentioned step A1, performing identity verification on the target user, may include the following steps:

A11. Obtain the first face image of the target user;

A12. Perform image quality evaluation on the first face image to obtain a target image quality evaluation value;

A13. According to the mapping relationship between the preset image quality evaluation value and the matching threshold, determine the target matching threshold corresponding to the target image quality evaluation value;

A14, performing contour extraction on the first face image to obtain a first peripheral contour;

A15. Perform feature point extraction on the first face image to obtain a first feature point set;

A16, matching the first peripheral contour with the second peripheral contour of the preset face template to obtain a first matching value;

A17, matching the first feature point set with the second feature point set of the preset face template to obtain a second matching value;

A18. Determine a target matching value according to the first matching value and the second matching value;

A19. When the target matching value is greater than the target matching threshold, confirm that the target user's identity verification is passed.

The emotion recognition device may pre-store a preset face template. In the face recognition process, success or failure largely depends on the quality of the face image. Therefore, in this embodiment of the present application, a dynamic matching threshold may be considered, That is, if the quality is good, the matching threshold can be increased, and if the quality is poor, the matching threshold can be lowered. Since the captured image may not be of good quality in a scotopic environment, the matching threshold can be adjusted appropriately. The emotion recognition device can also store the mapping relationship between the preset image quality evaluation value and the matching threshold, and then determine the target matching threshold corresponding to the target image quality evaluation value according to the mapping relationship. Perform contour extraction on the face image to obtain a first peripheral contour, perform feature point extraction on the first face image to obtain a first feature point set, and match the first peripheral contour with the second peripheral contour of the preset face template to obtain For the first matching value, the first feature point set is matched with the second feature point set of the preset face template to obtain the second matching value, and then the target matching value is determined according to the first matching value and the second matching value, for example , the mapping relationship between the environmental parameter and the weight value pair can be stored in advance, and the first weight coefficient corresponding to the first matching value and the second weight coefficient corresponding to the second matching value can be obtained. Target matching value=first matching value* The first weight coefficient + the second matching value * the second weight coefficient, and finally, when the target matching value is greater than the target matching threshold, confirm that the first face image is successfully matched with the preset face template, otherwise, confirm the face recognition Failure, in this way, the dynamic adjustment of the face matching process is conducive to improving the efficiency of face recognition for specific environments.

In addition, the contour extraction algorithm may be at least one of the following: Hough transform, canny operator, etc., which is not limited here, and the feature point extraction algorithm may be at least one of the following: Harris corner point, scale-invariant feature extraction Transformation (scaleinvariant feature transform, SIFT), etc., is not limited here.

Optionally, in the above step A12, image quality evaluation is performed on the first face image to obtain a target image quality evaluation value, which can be implemented as follows:

Perform image quality evaluation on the first face image by using at least one image quality evaluation index to obtain a target image quality evaluation value.

Wherein, the image quality evaluation indicators may include but are not limited to: average gray level, mean square error, entropy, edge retention, signal-to-noise ratio, and so on. It can be defined that the larger the obtained image quality evaluation value, the better the image quality.

It can be seen that, through the emotion recognition method of the embodiment of the present application, video clips and audio clips of the target user in a specified time period are obtained, the video clips are analyzed, the target body behavior and target facial expressions of the target user are obtained, and the audio clips are analyzed. Parameter extraction, obtain the target voice feature parameters of the target user, and decide the target user's target emotion according to the target body behavior, facial expression and voice feature parameters. In this way, the body behavior, facial expression and voice features are parsed from the video. The above three All dimensions reflect the user's emotions to a certain extent, and then, the user's emotions can be jointly determined through the three dimensions, and the user's emotions can be accurately identified.

Consistent with the above, please refer to FIG. 2 , which is a schematic flowchart of an embodiment of an emotion recognition method provided by an embodiment of the present application. The emotion recognition method described in this embodiment includes the following steps:

201. Acquire a video clip and an audio clip for a target user in a specified time period.

202. Analyze the video clip to obtain the target body behavior and the target facial expression of the target user.

203. Perform parameter extraction on the audio segment to obtain target speech feature parameters of the target user.

204. Determine the target emotion of the target user according to the target body behavior, the facial expression, and the speech feature parameter.

205. Authenticate the target user.

206. After the target user's identity verification is passed, acquire an indoor map.

207. Mark the location of the target user on the indoor map to obtain a target area where the target user is located.

208. Determine, according to a preset mapping relationship between emotions and control parameters, a target control parameter corresponding to the target emotion for controlling at least one smart home device corresponding to the target area.

209. Adjust the at least one smart home device according to the target control parameter.

Wherein, for the emotion recognition method described in the above steps 201 to 209, reference may be made to the corresponding steps of the emotion recognition method described in FIG. 1A .

It can be seen that, through the emotion recognition method of the embodiment of the present application, video clips and audio clips of the target user in a specified time period are obtained, the video clips are analyzed, the target body behavior and target facial expressions of the target user are obtained, and the audio clips are analyzed. Parameter extraction, obtain the target voice feature parameters of the target user, decide the target user's target emotion according to the target body behavior, facial expression and voice feature parameters, authenticate the target user, and obtain the indoor map after the target user's identity verification is passed. Mark the location of the target user in the indoor map, obtain the target area where the target user is located, and determine at least one smart home corresponding to the target emotion for controlling the target area according to the preset mapping relationship between emotions and control parameters The target control parameters of the device, adjust at least one smart home device according to the target control parameters. In this way, the body behavior, facial expressions and voice features are parsed from the video. The above three dimensions reflect the user's emotions to a certain extent, and then Through these three dimensions, the user's emotions can be jointly determined, which can accurately identify the user's emotions, and can also adjust the smart home devices in the environment according to the emotions, which helps to ease the user's emotions and improve the user experience.

Consistent with the above, the following is a device for implementing the above emotion recognition method, and the details are as follows:

Please refer to FIG. 3 , which is a schematic structural diagram of an embodiment of an emotion recognition apparatus according to an embodiment of the present application. The emotion recognition device described in this embodiment includes: an acquisition unit 301, an analysis unit 302, an extraction unit, and a decision unit 304, and the details are as follows:

Obtaining unit 301, for obtaining the video clip and audio clip for the target user in a specified time period;

An analysis unit 302, configured to analyze the video clip to obtain the target limb behavior and the target facial expression of the target user;

An extraction unit 303, configured to perform parameter extraction on the audio segment to obtain the target voice feature parameter of the target user;

The decision unit 304 is configured to decide the target emotion of the target user according to the target body behavior, the facial expression and the speech feature parameter.

The acquisition unit 301 can be used to implement the method described in the above step 101, the analysis unit 302 can be used to implement the method described in the above step 102, the extraction unit 303 can be used to implement the method described in the above step 103, and the decision unit 304 can be used to implement the method described in the above step 103. It can be used to implement the method described in the above step 104, and so on.

It can be seen that the emotion recognition device in the embodiment of the present application is applied to the emotion recognition device to obtain video clips and audio clips of the target user in a specified time period, analyze the video clips, and obtain the target body behavior and target face of the target user. Expression, extract the parameters of the audio clip to obtain the target voice feature parameters of the target user, and decide the target user's target emotion according to the target body behavior, facial expression and voice feature parameters. In this way, the body behavior, facial expression and Voice features, the above three dimensions all reflect the user's emotions to a certain extent, and then the user's emotions can be jointly determined through the three dimensions, which can accurately identify the user's emotions.

In a possible example, in the aspect of analyzing the video clip to obtain the physical behavior and facial expression of the target user, the analyzing unit 302 is specifically configured to:

Analyzing the video clip to obtain multiple frames of video images;

Image segmentation is performed on the multi-frame video images to obtain a plurality of target images, each target image being a human body image of the target user;

performing behavior recognition according to the multiple target images to obtain the target limb behavior;

performing face recognition on the multiple target images to obtain multiple face images;

Performing expression recognition on the multiple face images to obtain multiple expressions, and using the most frequently occurring expression among the multiple expressions as the target facial expression.

In a possible example, in the aspect of determining the target emotion of the target user according to the target body behavior, the target facial expression and the target voice feature parameter, the decision unit 304 is specifically configured to:

determining a first emotion set corresponding to the target limb behavior, the first emotion set includes at least one emotion, and each emotion corresponds to a first emotion probability value;

determining a second emotion set corresponding to the target facial expression, the second emotion set including at least one emotion, and each emotion corresponding to a second emotion probability value;

determining a third emotion set corresponding to the target speech feature parameter, the third emotion set includes at least one emotion, and each emotion corresponds to a third emotion probability value;

Obtain the first weight corresponding to the body behavior, the second weight corresponding to the facial expression, and the third weight corresponding to the speech feature parameter;

Determine the score of each type of emotion according to the first weight, the second weight, the third weight, the first emotion set, the second emotion set, and the third emotion set value, get multiple score values;

The maximum value is selected from the plurality of score values, and the emotion corresponding to the maximum value is used as the target emotion.

In a possible example, the target limb behavior includes at least one limb behavior;

In the aspect of performing behavior recognition according to the multiple target images to obtain the target limb behavior, the analyzing unit 302 is specifically configured to:

Inputting the multiple target images into a preset neural network model to obtain the at least one limb behavior and the recognition probability corresponding to each limb behavior;

In the aspect of determining the first emotion set corresponding to the target limb behavior, the first emotion set includes at least one emotion, and each emotion corresponds to a first emotion probability value, the decision-making unit is specifically used for:

According to the preset mapping relationship between the limb behavior and the emotion, the emotion corresponding to each limb behavior in the at least one limb behavior is determined, and at least one emotion is obtained, and each emotion corresponds to a preset probability value;

According to the identification probability corresponding to each limb behavior, and each emotion in the at least one emotion corresponding to a preset probability value, at least one first emotion probability value is obtained, and each emotion corresponds to a first emotion probability value.

In a possible example, the target limb behavior includes at least one limb behavior;

In the aspect of performing behavior recognition according to the multiple target images to obtain the target limb behavior, the analyzing unit 302 is specifically configured to:

Inputting the multiple target images into a preset neural network model to obtain the at least one limb behavior and the recognition probability corresponding to each limb behavior;

Determining the first emotion set corresponding to the target limb behavior, the first emotion set includes at least one emotion, and each emotion corresponds to a first emotion probability value, including:

According to the preset mapping relationship between the limb behavior and the emotion, the emotion corresponding to each limb behavior in the at least one limb behavior is determined, and at least one emotion is obtained, and each emotion corresponds to a preset probability value;

According to the identification probability corresponding to each limb behavior, and each emotion in the at least one emotion corresponding to a preset probability value, at least one first emotion probability value is obtained, and each emotion corresponds to a first emotion probability value.

In a possible example, the target speech feature parameters include at least one of the following: target speech rate, target intonation, and at least one target keyword;

In the aspect of performing parameter extraction on the audio segment to obtain the target speech feature parameters of the target user, the extraction unit 303 is specifically configured to:

Semantic analysis is performed on the audio clip to obtain a plurality of characters;

Determine the pronunciation duration corresponding to the multiple characters, and determine the speech rate according to the multiple characters and the pronunciation duration;

or,

performing character segmentation on the multiple characters to obtain multiple keywords;

Matching the multiple keywords with a preset keyword set to obtain the at least one target keyword;

or,

extracting a waveform diagram of the speech segment;

The waveform diagram is analyzed to obtain the target intonation.

It can be understood that the functions of each program module of the emotion recognition apparatus in this embodiment can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can refer to the relevant descriptions of the above method embodiments, which will not be repeated here.

Consistent with the above, please refer to FIG. 4 , which is a schematic structural diagram of an embodiment of an emotion recognition apparatus provided in an embodiment of the present application. The emotion recognition apparatus described in this embodiment includes: at least one input device 1000; at least one output device 2000; at least one processor 3000, such as a CPU; It is connected to the memory 4000 through the bus 5000.

The above-mentioned input device 1000 may specifically be a touch panel, a physical button or a mouse.

The above-mentioned output device 2000 may specifically be a display screen.

The above-mentioned memory 4000 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as a magnetic disk memory. The above-mentioned memory 4000 is used to store a set of program codes, and the above-mentioned input device 1000, output device 2000 and processor 3000 are used to call the program codes stored in the memory 4000, and perform the following operations:

The above processor 3000 is used for:

Obtain the video clips and audio clips of the target user in the specified time period;

The video clip is analyzed to obtain the target limb behavior and the target facial expression of the target user;

Perform parameter extraction on the audio segment to obtain the target speech feature parameters of the target user;

The target emotion of the target user is determined according to the target body behavior, the facial expression and the speech feature parameters.

It can be seen that, through the emotion recognition device of the embodiment of the present application, the video clips and audio clips of the target user in the specified time period are obtained, the video clips are analyzed, the target body behavior and the target facial expression of the target user are obtained, and the audio clips are analyzed. Parameter extraction, obtain the target voice feature parameters of the target user, and decide the target user's target emotion according to the target body behavior, facial expression and voice feature parameters. In this way, the body behavior, facial expression and voice features are parsed from the video. The above three All dimensions reflect the user's emotions to a certain extent, and then, the user's emotions can be jointly determined through the three dimensions, and the user's emotions can be accurately identified.

In a possible example, in the aspect of analyzing the video clip to obtain the physical behavior and facial expression of the target user, the processor 3000 is specifically configured to:

Analyzing the video clip to obtain multiple frames of video images;

Image segmentation is performed on the multi-frame video images to obtain a plurality of target images, each target image being a human body image of the target user;

performing behavior recognition according to the multiple target images to obtain the target limb behavior;

performing face recognition on the multiple target images to obtain multiple face images;

Performing expression recognition on the multiple face images to obtain multiple expressions, and using the most frequently occurring expression among the multiple expressions as the target facial expression.

In a possible example, in the aspect of determining the target emotion of the target user according to the target body behavior, the target facial expression and the target voice feature parameters, the processor 3000 is specifically configured to:

determining a first emotion set corresponding to the target limb behavior, the first emotion set includes at least one emotion, and each emotion corresponds to a first emotion probability value;

determining a second emotion set corresponding to the target facial expression, the second emotion set including at least one emotion, and each emotion corresponding to a second emotion probability value;

determining a third emotion set corresponding to the target speech feature parameter, the third emotion set includes at least one emotion, and each emotion corresponds to a third emotion probability value;

Obtain the first weight corresponding to the body behavior, the second weight corresponding to the facial expression, and the third weight corresponding to the speech feature parameter;

Determine the score of each type of emotion according to the first weight, the second weight, the third weight, the first emotion set, the second emotion set, and the third emotion set value, get multiple score values;

The maximum value is selected from the plurality of score values, and the emotion corresponding to the maximum value is used as the target emotion.

In a possible example, the target limb behavior includes at least one limb behavior;

In the aspect of performing behavior recognition according to the multiple target images to obtain the target limb behavior, the processor 3000 is specifically configured to:

Inputting the multiple target images into a preset neural network model to obtain the at least one limb behavior and the recognition probability corresponding to each limb behavior;

In the aspect of determining the first emotion set corresponding to the target limb behavior, the first emotion set includes at least one emotion, and each emotion corresponds to a first emotion probability value, the processor 3000 is specifically configured to:

According to the preset mapping relationship between the limb behavior and the emotion, the emotion corresponding to each limb behavior in the at least one limb behavior is determined, and at least one emotion is obtained, and each emotion corresponds to a preset probability value;

According to the identification probability corresponding to each limb behavior, and each emotion in the at least one emotion corresponding to a preset probability value, at least one first emotion probability value is obtained, and each emotion corresponds to a first emotion probability value.

In a possible example, the target speech feature parameters include at least one of the following: target speech rate, target intonation, and at least one target keyword;

In the aspect of performing parameter extraction on the audio segment to obtain the target voice feature parameters of the target user, the processor 3000 is specifically configured to:

Semantic analysis is performed on the audio clip to obtain a plurality of characters;

Determine the pronunciation duration corresponding to the multiple characters, and determine the speech rate according to the multiple characters and the pronunciation duration;

or,

performing character segmentation on the multiple characters to obtain multiple keywords;

Matching the multiple keywords with a preset keyword set to obtain the at least one target keyword;

or,

extracting a waveform diagram of the speech segment;

The waveform diagram is analyzed to obtain the target intonation.

An embodiment of the present application further provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program includes part or all of the steps of any emotion recognition method described in the above method embodiments.

It should be understood by those skilled in the art that the embodiments of the present application may be provided as a method, an apparatus (apparatus), or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. The computer program is stored/distributed in a suitable medium, provided with or as part of other hardware, or may take other forms of distribution, such as over the Internet or other wired or wireless telecommunication systems.

Although the application has been described in conjunction with specific features and embodiments thereof, it will be apparent that various modifications and combinations can be made therein without departing from the spirit and scope of the application. Accordingly, this specification and drawings are merely exemplary illustrations of the application as defined by the appended claims, and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of this application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (10)

1. A method of emotion recognition, comprising:

acquiring a video clip and an audio clip aiming at a target user in a specified time period;

analyzing the video segments to obtain target limb behaviors and target facial expressions of the target user;

extracting parameters of the audio clip to obtain target voice characteristic parameters of the target user;

and deciding the target emotion of the target user according to the target limb behaviors, the facial expression and the voice characteristic parameters.

2. The method of claim 1, wherein analyzing the video segments for the target user's limb behavior and facial expression comprises:

analyzing the video clips to obtain multi-frame video images;

carrying out image segmentation on the multi-frame video image to obtain a plurality of target images, wherein each target image is a human body image of the target user;

performing behavior recognition according to the target images to obtain the target limb behaviors;

carrying out face recognition on the target images to obtain a plurality of face images;

and performing expression recognition on the plurality of facial images to obtain a plurality of expressions, and taking the expression with the largest occurrence frequency in the plurality of expressions as the target facial expression.

3. The method of claim 1 or 2, wherein said deciding a target emotion of the target user based on the target limb behavior, the target facial expression and the target voice feature parameters comprises:

determining a first emotion set corresponding to the target limb behavior, wherein the first emotion set comprises at least one emotion, and each emotion corresponds to a first emotion probability value;

determining a second emotion set corresponding to the target facial expression, wherein the second emotion set comprises at least one emotion, and each emotion corresponds to a second emotion probability value;

determining a third emotion set corresponding to the target voice characteristic parameter, wherein the third emotion set comprises at least one emotion, and each emotion corresponds to a third emotion probability value;

acquiring a first weight corresponding to the body behavior, a second weight corresponding to the facial expression and a third weight corresponding to the voice characteristic parameter;

determining score values of each type of emotion according to the first weight, the second weight, the third weight, the first emotion set, the second emotion set and the third emotion set to obtain a plurality of score values;

and selecting a maximum value from the plurality of score values, and taking the emotion corresponding to the maximum value as the target emotion.

4. The method of claim 3, wherein the target limb behavior comprises at least one limb behavior;

the behavior recognition according to the target images to obtain the target limb behaviors comprises the following steps:

inputting the target images into a preset neural network model to obtain at least one limb behavior and the identification probability corresponding to each limb behavior;

the determining a first emotion set corresponding to the target limb behavior, the first emotion set comprising at least one emotion, each emotion corresponding to a first emotion probability value, includes:

determining the emotion corresponding to each limb behavior in the at least one limb behavior according to a preset mapping relation between the limb behavior and the emotion to obtain at least one emotion, wherein each emotion corresponds to a preset probability value;

and calculating according to the recognition probability corresponding to each body behavior and a preset probability value corresponding to each emotion in the at least one emotion to obtain at least one first emotion probability value, wherein each emotion corresponds to one first emotion probability value.

5. The method according to any of claims 1-4, wherein the target speech feature parameters comprise at least one of: target speed of speech, target intonation, at least one target keyword;

the parameter extraction of the audio clip to obtain the target voice characteristic parameter of the target user comprises:

performing semantic analysis on the audio clip to obtain a plurality of characters;

determining pronunciation duration corresponding to the characters, and determining the speed of speech according to the characters and the pronunciation duration;

or,

performing character segmentation on the characters to obtain a plurality of keywords;

matching the plurality of keywords with a preset keyword set to obtain at least one target keyword;

or,

extracting a waveform image of the voice segment;

and analyzing the oscillogram to obtain the target intonation.

6. An emotion recognition apparatus, comprising:

the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a video segment and an audio segment aiming at a target user in a specified time period;

the analysis unit is used for analyzing the video segments to obtain target limb behaviors and target facial expressions of the target users;

the extraction unit is used for extracting parameters of the audio clips to obtain target voice characteristic parameters of the target user;

and the decision unit is used for deciding the target emotion of the target user according to the target limb behaviors, the facial expression and the voice characteristic parameters.

7. The apparatus according to claim 6, wherein in the analyzing the video segments to obtain the body behaviors and facial expressions of the target user, the analyzing unit is specifically configured to:

analyzing the video clips to obtain multi-frame video images;

carrying out image segmentation on the multi-frame video image to obtain a plurality of target images, wherein each target image is a human body image of the target user;

performing behavior recognition according to the target images to obtain the target limb behaviors;

carrying out face recognition on the target images to obtain a plurality of face images;

and performing expression recognition on the plurality of facial images to obtain a plurality of expressions, and taking the expression with the largest occurrence frequency in the plurality of expressions as the target facial expression.

8. The apparatus according to claim 6 or 7, wherein in said deciding a target emotion of the target user based on the target limb behavior, the target facial expression and the target speech feature parameters, the deciding unit is specifically configured to:

determining a first emotion set corresponding to the target limb behavior, wherein the first emotion set comprises at least one emotion, and each emotion corresponds to a first emotion probability value;

determining a second emotion set corresponding to the target facial expression, wherein the second emotion set comprises at least one emotion, and each emotion corresponds to a second emotion probability value;

determining a third emotion set corresponding to the target voice characteristic parameter, wherein the third emotion set comprises at least one emotion, and each emotion corresponds to a third emotion probability value;

acquiring a first weight corresponding to the body behavior, a second weight corresponding to the facial expression and a third weight corresponding to the voice characteristic parameter;

determining score values of each type of emotion according to the first weight, the second weight, the third weight, the first emotion set, the second emotion set and the third emotion set to obtain a plurality of score values;

and selecting a maximum value from the plurality of score values, and taking the emotion corresponding to the maximum value as the target emotion.

9. The apparatus of claim 8, wherein the target limb behavior comprises at least one limb behavior;

in the aspect of performing behavior recognition according to the target images to obtain the target limb behaviors, the analysis unit is specifically configured to:

inputting the target images into a preset neural network model to obtain at least one limb behavior and the identification probability corresponding to each limb behavior;

in the determining of the first emotion set corresponding to the target limb behavior, where the first emotion set includes at least one emotion, and each emotion corresponds to a first emotion probability value, the decision unit is specifically configured to:

determining the emotion corresponding to each limb behavior in the at least one limb behavior according to a preset mapping relation between the limb behavior and the emotion to obtain at least one emotion, wherein each emotion corresponds to a preset probability value;

and calculating according to the recognition probability corresponding to each body behavior and a preset probability value corresponding to each emotion in the at least one emotion to obtain at least one first emotion probability value, wherein each emotion corresponds to one first emotion probability value.

10. A computer-readable storage medium storing a computer program for execution by a processor to implement the method of any one of claims 1-5.