CN118587757A - AR-based emotional data processing method, device and electronic device - Google Patents

Abstract

The present application provides an AR-based emotional data processing method, device and electronic device, which relates to the field of data processing. In this method, the emotional data for the user sent by AR glasses is obtained, and the user wears AR glasses, and the emotional data includes facial expression data and user voice data; emotional recognition is performed on the facial expression data to obtain a first recognition result; voice recognition is performed on the user voice data to obtain a second recognition result; if it is determined that both the first recognition result and the second recognition result indicate that the user's emotional state is a negative emotion, the current moment data is obtained; according to the current moment data, a corresponding processing strategy is generated, and the AR glasses are controlled to execute the processing strategy to alleviate the user's negative emotions. Implementing the technical solution provided by the present application is convenient for improving the accuracy of emotion recognition.

Description

AR-based emotional data processing method, device and electronic device

Technical Field

The present application relates to the technical field of data processing, and specifically to an AR-based emotion data processing method, device and electronic device.

Background Art

Under the existing technical background, the recognition and response of users' emotional states, especially in augmented reality (AR) technology, still faces many challenges. With the rapid development of AR technology, wearable devices such as AR glasses have gradually become important tools in people's daily life and work.

At present, AR glasses in the relevant technology mainly rely on facial recognition technology in the dimension of emotion recognition. However, this technology is often limited by factors such as lighting conditions, user's facial occlusion and the naturalness of facial expressions, resulting in low accuracy in emotion recognition.

Therefore, there is an urgent need for an AR-based emotion data processing method, device and electronic device.

Summary of the invention

The present application provides an AR-based emotion data processing method, device, and electronic device to facilitate improving the accuracy of emotion recognition.

In a first aspect of the present application, an AR-based emotion data processing method is provided, the method comprising: obtaining emotion data for a user sent by AR glasses, the user wearing the AR glasses, the emotion data comprising facial expression data and user voice data; performing emotion recognition on the facial expression data to obtain a first recognition result; performing voice recognition on the user voice data to obtain a second recognition result; if it is determined that both the first recognition result and the second recognition result indicate that the user's emotional state is negative, obtaining current moment data; generating a corresponding processing strategy based on the current moment data, and controlling the AR glasses to execute the processing strategy to alleviate the user's negative emotions.

By adopting the above technical solution, by combining facial expression data and user voice data, this system can more comprehensively analyze the user's emotional state. Because emotional expression is usually multifaceted, including facial expressions and voices, the integration of this information can significantly improve the accuracy of emotion recognition. When the system recognizes that the user is in a negative emotion, it can immediately obtain the data at the current moment and generate a corresponding treatment strategy based on it. This real-time response capability is crucial to alleviating the user's negative emotions because it can provide timely intervention before the emotions worsen. Based on the data at the current moment, the system can generate personalized treatment strategies. This strategy is to provide comfort information, recommend relaxation activities, etc. to meet the needs of users in different situations. By timely identifying and alleviating the user's negative emotions, this system can significantly improve the user's experience when using AR glasses. Users will feel that the device is not only a tool to provide information, but also an intelligent partner that can understand and respond to their emotional needs. Negative emotions may be related to mental health problems. By monitoring the user's emotional state for a long time, the system can provide users with more comprehensive health care, and even in some cases, it can serve as an early warning system to remind users or medical professionals to pay attention to possible psychological problems. As a result, through more accurate facial recognition and voice recognition, the accuracy of emotion recognition can be improved.

Optionally, obtaining the emotion data for the user sent by the AR glasses specifically includes: receiving original facial expression data and original user voice data sent by the AR glasses; preprocessing the original facial expression data and the original user voice data to obtain the emotion data, and the preprocessing includes denoising, filtering and normalization.

By adopting the above technical solutions, denoising can eliminate the noise components in the data, which may come from environmental interference, equipment failure or data transmission errors. By denoising, the clarity and accuracy of the data can be improved, laying a solid foundation for subsequent emotion recognition. Filtering can smooth high-frequency fluctuations in the data and make the data more stable. This helps to reduce misjudgments caused by short-term changes in user expressions or voice fluctuations, and improve the stability of emotion recognition. Normalization can convert data from different sources and different dimensions to the same dimension, making different data comparable. In emotion recognition, normalization can ensure that facial expression data and user voice data are treated equally in terms of weight, improving the fairness and accuracy of emotion recognition. The preprocessed data has a better data structure and feature representation, which helps the subsequent emotion recognition algorithm to find key information in the data faster. Therefore, preprocessing can improve the computational efficiency of emotion recognition and enable the system to respond to the user's emotional state more quickly. Because preprocessing can eliminate noise and fluctuations in the original data, the system is less sensitive to input data. This helps to enhance the robustness of the system and enable the system to maintain stable performance in different environments. The preprocessed sentiment data has higher quality and better comparability, which provides a better foundation for subsequent sentiment analysis, model training and other steps. By conducting in-depth analysis and mining of the preprocessed data, the accuracy and reliability of sentiment recognition can be further improved.

Optionally, the emotion recognition is performed on the facial expression data to obtain a first recognition result, which specifically includes: identifying at least one positioning point included in the facial expression data according to a preset positioning point numbering sequence; determining a target positioning point among the at least one positioning point whose positioning point value is greater than a positioning point threshold, and adding the target positioning point to the target facial feature positioning points; searching for a symmetrical positioning point of the target positioning point, and if the positioning point value of the symmetrical positioning point is greater than the positioning point threshold, adding the symmetrical positioning point to the target facial feature positioning points; determining a target positioning point among the at least one positioning point whose next positioning point value is greater than the positioning point threshold, and executing the step of determining a target positioning point among the at least one positioning point whose positioning point value is greater than the positioning point threshold; until the number of the target facial feature positioning points reaches a preset number threshold; determining the target facial expression of the user according to the correspondence between different target facial feature positioning points and different target facial expressions, wherein the first recognition result includes the target facial expression of the user, and the target facial expression is used to indicate the emotional state of the user.

By adopting the above technical solution, facial feature recognition is performed by presetting the sequence of positioning point numbers, which can ensure that the system focuses on key areas of the face, which usually play a key role in emotional expression. Therefore, this method can improve the accuracy of emotion recognition. The process not only focuses on single facial feature points, but also considers symmetrical positioning points. When the values of symmetrical positioning points all exceed the threshold, they are added to the target facial feature positioning points, which helps to identify subtle facial expressions, such as smiles, frowns, etc., so as to more accurately reflect the user's emotional state, which enables the system to adapt to different expressions and accurately identify the user's emotional state. Once the system identifies a sufficient number of target facial feature positioning points (reaching the preset number threshold), the user's facial expression can be immediately determined. This fast response capability enables the system to respond to the user's emotional state in a timely manner and provide timely feedback or processing. The process performs emotion recognition based on clear facial feature points, which makes the recognition results interpretable and credible. When the system recognizes an emotion, it can clearly indicate which facial feature points are used to make the judgment, which increases the user's trust in the system. The process can adapt to different application scenarios and user needs by adding or reducing positioning points, adjusting thresholds, or updating the correspondence between facial feature points and facial expressions. This makes the system more adaptable and scalable. By comprehensively considering multiple facial feature points and symmetrical positioning points, the system can reduce the false alarm rate caused by misjudgment of a single feature point. Only when multiple key feature points meet the conditions will the system make an emotion recognition judgment, which improves the accuracy of recognition.

Optionally, performing speech recognition on the user voice data to obtain a second recognition result specifically includes: performing feature recognition on the user voice data to obtain user semantic features, user speaking rate features and user pitch features; inputting the user semantic features, the user speaking rate features and the user pitch features into a preset recognition model to obtain target acoustic features of the user, the preset recognition model is a model pre-built and trained based on deep learning, and the second recognition result includes the target acoustic features, and the target acoustic features are used to indicate the emotional state of the user.

By adopting the above technical solutions, speech recognition is not limited to simple text conversion, but deeply analyzes multiple dimensions of user speech, including semantics, speaking speed and intonation. This multi-dimensional feature analysis can capture the user's emotional state more comprehensively and improve the accuracy of emotion recognition. Using a preset recognition model based on deep learning means that the system is able to handle complex speech data and pattern recognition tasks. The deep learning model can automatically learn and optimize the feature extraction and classification process, thereby improving the efficiency and accuracy of speech recognition. Deep learning models usually have strong adaptability and generalization capabilities. This means that the system can handle the speech data of different users, including different accents, speaking speeds and intonations, without having to train a model for each user separately. This greatly reduces development costs and time. Due to the optimization of the deep learning model and the improvement of computing power, the system is able to process and analyze user speech data in real time. This ensures that the system can respond immediately when the user expresses emotions and provide timely feedback. Through in-depth analysis and processing of user speech data, the system can more accurately identify the user's emotional state. This provides users with a more personalized and caring service experience, and also provides valuable data support for research in fields such as sentiment analysis and sentiment computing. Because the system performs emotion recognition based on clear voice features and deep learning models, the recognition results are interpretable and credible. When the system recognizes the user's emotional state, it can clearly indicate which voice features and models are used to make the judgment, which increases the user's trust in the system. The process can be expanded and customized by updating or adjusting the deep learning model, adding new voice features, or adjusting feature weights. This enables the system to adapt to different application scenarios and user needs, providing more flexible and diverse services.

Optionally, before inputting the user semantic features, the user speaking rate features and the user pitch features into the preset recognition model, the preset recognition model is trained; the training of the preset recognition model specifically includes: using CNN to extract global spatiotemporal features from the user's historical voice data, the global spatiotemporal features including user historical semantic features, user historical speaking rate features and user historical pitch features; using RNN to perform sequence modeling on the global spatiotemporal features to obtain time-dependent features; using a Softmax function to classify the time-dependent features and output emotion labels, one emotion label corresponds to one target acoustic feature, and the emotion labels include positive emotion labels, neutral emotion labels and negative emotion labels.

By adopting the above technical solution, CNN is used to extract global spatiotemporal features from the user's historical voice data, which can capture the complex structure and pattern in the voice. This global spatiotemporal feature not only includes semantic information, but also includes key emotional expression elements such as speech rate and pitch, providing a solid foundation for subsequent emotion recognition. RNN is used to perform sequence modeling on global spatiotemporal features because voice data has temporal sequence. RNN can process dependencies in sequence data and capture the association between different time points in voice, which is crucial for understanding the user's emotional state. The Softmax function is used to classify the temporal dependency features output by RNN and output the emotional label. The Softmax function can convert the output of the model into a probability distribution so that each emotional label has a corresponding probability value. By selecting the emotional label with the highest probability value as the prediction result, the accuracy and efficiency of the classification can be ensured. Emotional labels include positive emotions, neutral emotions, and negative emotions, which can clearly indicate the user's emotional state. This clear emotional label helps the system better understand the user's emotional needs and provide corresponding services and feedback. By using the user's historical voice data for model training, the model can learn the voice characteristics and emotional expressions of different users. This enables the model to have strong generalization capabilities, and it can process voice data from different users and accurately identify their emotional states. The trained preset recognition model can immediately perform emotion recognition when user voice data is input, achieving real-time response. This helps the system to capture the user's emotional changes in a timely manner and provide corresponding feedback and services.

Optionally, the corresponding processing strategy is generated based on the current time data, specifically including: if the current time data indicates that the current time is daytime, a first processing strategy is generated, the first processing strategy includes a user outdoor activity planning strategy, and the user outdoor activity planning strategy is displayed on the AR glasses; if the current time data indicates that the current time is night, a second processing strategy is generated, the second processing strategy includes displaying preset text and playing preset audio strategies.

By adopting the above technical solution, by judging the current time data, the system can intelligently identify whether it is daytime or nighttime, and generate corresponding processing strategies according to the characteristics of different time periods. This intelligent adaptability makes the service closer to the actual needs of users. During the day, the user's outing activity planning strategy generated by the system can recommend suitable activity locations, routes, etc. to users based on the user's historical behavior, preferences and other information, thereby enhancing the user's travel experience. This personalized service not only improves user satisfaction, but also increases the use value of the system. When the system determines that it is nighttime, the generated processing strategy focuses on displaying preset text and playing preset audio. This night mode not only reduces the interference of screen brightness on users, but also avoids excessive visual interaction at night, thereby improving the user's comfort. By displaying the processing strategy on AR glasses, users can more conveniently obtain the required information without additional equipment or operation. This interactive method not only improves the user's convenience of use, but also enhances the practicality of AR glasses as wearable devices. Different processing strategies are generated according to different time periods, so that users can get a coherent and consistent service experience when using AR glasses at different times. This consistency helps to enhance users' trust and dependence on the system. The strategy of lowering screen brightness and reducing visual interaction at night can also help save energy and reduce emissions, lower the energy consumption of equipment, and comply with the concept of green environmental protection.

Optionally, the method also includes: if it is determined that the first recognition result indicates that the user's emotional state is a positive emotion, and the second recognition result indicates that the user's emotional state is a negative emotion, or if it is determined that the first recognition result indicates that the user's emotional state is a negative emotion, and the second recognition result indicates that the user's emotional state is a positive emotion, then generating a consulting strategy and controlling the AR glasses to execute the consulting strategy, and the consulting strategy is used to determine the user's emotional state.

By adopting the above technical solutions, text analysis and speech recognition are two different ways of emotion recognition, which may produce inconsistent results for various reasons. When this happens, generating a consulting strategy can be used as a "verification" step to confirm the user's true emotional state by directly interacting with the user, thereby improving the accuracy of emotion recognition. By generating and executing consulting strategies, the system can ensure that inappropriate services or feedback are not provided due to misjudgment of the user's emotional state. For example, if the system mistakenly identifies the user's negative emotions as positive emotions and provides inappropriate advice or information based on this, it may make the user feel troubled or dissatisfied. The use of consulting strategies can avoid this situation and optimize the user experience. When the system finds that there is a conflict in the emotion recognition results, it can actively generate consulting strategies and interact with the user, which reflects the high intelligence and interactivity of the system. This interaction not only helps to solve the current emotion recognition problem, but also enhances the user's trust and dependence on the system. The consulting strategy can provide personalized support based on the user's actual emotional state. For example, if the system determines that the user is currently in a negative emotion, it can guide the user to vent emotions through consulting strategies, provide advice or resources for emotion regulation, etc., so as to help the user better manage his or her emotions. By executing consulting strategies and interacting with users, the system can collect more data about the user's emotional state. This data can be used to further train and optimize the emotion recognition model, improving the accuracy and efficiency of the model in identifying user emotions in the future. By generating and executing consulting strategies to verify emotion recognition results, the risk of misjudgment can be reduced, ensuring that the service or feedback provided by the system is more reliable and secure.

The present application also provides an AR-based emotion data processing device, which includes an acquisition module and a processing module, wherein the acquisition module is used to acquire emotion data for a user sent by AR glasses, wherein the user wears the AR glasses, and the emotion data includes facial expression data and user voice data; the processing module is used to perform emotion recognition on the facial expression data to obtain a first recognition result; the processing module is also used to perform voice recognition on the user voice data to obtain a second recognition result; the processing module is also used to acquire current moment data if it is determined that both the first recognition result and the second recognition result indicate that the user's emotional state is a negative emotion; the processing module is also used to generate a corresponding processing strategy based on the current moment data, and control the AR glasses to execute the processing strategy to alleviate the user's negative emotions.

In the third aspect of the present application, an electronic device is provided, which includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs the method described above.

In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, the method described above is executed.

In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

1. By combining facial expression data and user voice data, the system can more comprehensively analyze the user's emotional state. Because emotional expression is usually multifaceted, including facial expressions and voices, combining this information can significantly improve the accuracy of emotion recognition. When the system recognizes that the user is in a negative emotion, it can immediately obtain the current data and generate a corresponding treatment strategy based on it. This real-time response capability is crucial to alleviating the user's negative emotions because it can provide timely intervention before the emotions worsen. Based on the current data, the system can generate personalized treatment strategies. This strategy is to provide comfort information, recommend relaxing activities, etc. to meet the needs of users in different situations. By promptly identifying and alleviating the user's negative emotions, the system can significantly improve the user's experience when using AR glasses. Users will feel that the device is not only a tool to provide information, but also an intelligent partner that can understand and respond to their emotional needs. Negative emotions may be related to mental health problems. By monitoring the user's emotional state for a long time, the system can provide users with more comprehensive health care, and in some cases, it can even serve as an early warning system to remind users or medical professionals to pay attention to possible psychological problems. Therefore, through more accurate facial recognition and voice recognition, the accuracy of emotion recognition can be improved;

2. Once the system identifies a sufficient number of target facial feature points (reaching a preset number threshold), the user's facial expression can be immediately determined. This rapid response capability enables the system to respond to the user's emotional state in a timely manner and provide timely feedback or processing. The process is based on clear facial feature points for emotion recognition, which makes the recognition results interpretable and credible. When the system recognizes an emotion, it can clearly indicate which facial feature points are used to make the judgment, which increases the user's trust in the system. The process can adapt to different application scenarios and user needs by adding or reducing points, adjusting thresholds, or updating the correspondence between facial feature points and facial expressions. This makes the system more adaptable and scalable. By comprehensively considering multiple facial feature points and symmetrical positioning points, the system can reduce the false alarm rate caused by misjudgment of a single feature point. The system will only make an emotion recognition judgment when multiple key feature points meet the conditions, which improves the accuracy of recognition.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of an AR-based emotion data processing method provided in an embodiment of the present application.

FIG2 is another flow chart of an AR-based emotion data processing method provided in an embodiment of the present application.

FIG3 is a module diagram of an AR-based emotion data processing device provided in an embodiment of the present application.

FIG. 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

Explanation of the reference numerals: 31, acquisition module; 32, processing module; 41, processor; 42, communication bus; 43, user interface; 44, network interface; 45, memory.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

In the current technological context, although augmented reality (AR) technology is changing with each passing day, the accurate recognition and timely response of users' emotional states, especially in the application of AR technology, is still a difficult problem to be solved. With the popularization of wearable devices such as AR glasses, they have gradually become a powerful assistant in our daily life and work.

However, the current ability of these AR glasses in emotion recognition mainly relies on facial recognition technology. However, this technology is often subject to many limitations in practical applications, such as changes in lighting conditions, occlusion of the user's face, and the naturalness of facial expressions, which may greatly reduce the accuracy of emotion recognition.

In order to solve the above technical problems, the present application provides a flowchart of an AR-based emotion data processing method. The emotion data processing method is applied to a server and includes steps S110 to S150, which are as follows:

S110. Acquire emotion data for the user sent by the AR glasses, where the user wears the AR glasses, and the emotion data includes facial expression data and user voice data.

Specifically, a server is a computer or device used to store, process or forward data in a network. In this scenario, the server is responsible for receiving emotional data sent from AR glasses. AR glasses are an augmented reality (AR) device that allows users to see the combination of virtual elements and the real world through the display screen of the glasses. AR glasses not only have display functions, but may also contain various sensors and cameras to capture user movements, positions, expressions and other information. By wearing AR glasses, users can experience the interaction and immersion brought by augmented reality. At the same time, AR glasses are also constantly collecting data about users, including users' facial expressions and voice data. Emotional data refers to data that can reflect the user's emotional state. In an embodiment of the present application, emotional data includes two parts: facial expression data is user facial expression information captured by the camera on AR glasses. The camera of AR glasses can track the changes in the user's facial expressions in real time and convert these changes into data. For example, when a user smiles, the camera can capture the curvature of the mouth and the changes in the shape of the eyes, which are facial expression data. User voice data is the sound data generated by the user through the microphone or other audio input device of AR glasses. Voice data contains information such as the user's tone, speaking speed, and volume, which can reflect the user's emotional state. For example, when a user is angry, the speaking speed may increase and the volume may rise, and these changes can be captured through voice data.

In a possible implementation, obtaining emotion data for a user sent by the AR glasses specifically includes: receiving original facial expression data and original user voice data sent by the AR glasses; preprocessing the original facial expression data and the original user voice data to obtain emotion data, wherein the preprocessing includes denoising, filtering, and normalization.

Specifically, the server first receives the raw facial expression data and raw user voice data sent from the AR glasses. These data are unprocessed and captured directly from the camera and microphone of the AR glasses. Preprocessing is a series of operations on the raw data to eliminate noise, improve data quality, and make it more suitable for subsequent analysis and processing. Among them, preprocessing includes denoising, filtering, and normalization. Due to various interference factors that may exist in the environment (such as light changes, background noise, etc.), the raw data often contains noise. The denoising operation aims to reduce or eliminate these noises to make the data purer. The filtering operation further smoothes the data, removes high-frequency noise or unnecessary frequency components, and makes the data smoother and easier to analyze. Normalization is to scale the data to a certain range (such as between 0 and 1) so that data from different sources and different scales can be compared and analyzed at the same scale. In emotional data analysis, normalization helps to ensure that the weights between different features are equal, thereby avoiding some features from excessively affecting the results due to their large numerical range.

For example, suppose a user is using an AR glasses. The application uses the AR glasses to capture the user's facial expressions and voice data to analyze the user's emotional state and adjust the teaching strategy accordingly. Among them, the camera of the AR glasses continuously captures the user's facial expression data, such as the movement and position changes of the eyes, eyebrows, mouth, etc. At the same time, the microphone also records the user's voice data in real time, such as intonation, speaking speed, etc. These raw data are sent to the server in real time. After receiving the raw facial expression data, the server finds that it contains shadows and reflections caused by light changes. In order to eliminate these noises, the server uses image processing algorithms (such as Gaussian blur, median filtering, etc.) to denoise the image. For voice data, the server uses audio processing algorithms (such as spectrum analysis, wavelet transform, etc.) to remove background noise. For facial expression data, the server may use edge detection, corner detection and other algorithms to further extract facial feature points, and filter these feature points to remove high-frequency noise and jitter. For voice data, the server may use bandpass filters, low-pass filters, etc. to further smooth the voice signal. For subsequent emotional analysis, the server needs to normalize data from different sources. For example, for different feature points in facial expression data (such as eye width, mouth height, etc.), the server can scale them to the range of 0 to 1; for different features in voice data (such as fundamental frequency, volume, etc.), the server can also perform similar normalization processing.

S120: Perform emotion recognition on the facial expression data to obtain a first recognition result.

Specifically, first, the camera of AR glasses will take images or videos of the user's face and send this data to a server or local processor for analysis. Once the facial expression data is obtained, the next step is to extract emotion-related features. These features may include the shape of the eyebrows, the degree of eye opening and closing, the degree of mouth opening, etc. Feature extraction usually involves image processing and computer vision techniques such as edge detection, shape recognition, texture analysis, etc. After the emotion-related features are extracted, these features can be used for emotion recognition. Emotion recognition is a classification task that matches the input facial expression data with pre-defined emotion categories. These emotion categories may include happiness, sadness, anger, surprise, etc. To complete this classification task, various machine learning algorithms can be used, such as support vector machines (SVM), neural networks (NN), deep learning models (such as convolutional neural networks CNN), etc. The result of emotion recognition is one or more emotion categories that represent the system's estimation of the user's current emotional state.

In a possible implementation, emotion recognition is performed on facial expression data to obtain a first recognition result, which specifically includes: identifying at least one positioning point included in the facial expression data according to a preset positioning point numbering sequence; determining a target positioning point whose positioning point value is greater than a positioning point threshold among at least one positioning point, and adding the target positioning point to the target facial feature positioning points; searching for a symmetrical positioning point of the target positioning point, and if the positioning point value of the symmetrical positioning point is greater than the positioning point threshold, adding the symmetrical positioning point to the target facial feature positioning points; determining a target positioning point whose next positioning point value is greater than the positioning point threshold among at least one positioning point, and executing the step of determining a target positioning point whose positioning point value is greater than the positioning point threshold among at least one positioning point; until the number of target facial feature positioning points reaches a preset number threshold; determining the target facial expression of the user according to the correspondence between different target facial feature positioning points and different target facial expressions, the first recognition result includes the target facial expression of the user, and the target facial expression is used to indicate the emotional state of the user.

Specifically, first, the server identifies at least one anchor point in the facial expression data according to a preset anchor point number sequence. These anchor points are usually key points on the face, such as the corners of the eyes, the corners of the mouth, the specific positions of the eyebrows, etc. Then, the server checks the value of each anchor point, which can be a pixel intensity. If the value of a certain anchor point is greater than the set anchor point threshold, then this anchor point is considered to be a target anchor point and added to the target facial feature anchor point set. In order to capture more comprehensive facial features, the server will also find symmetrical anchor points of the target anchor point. If the value of the symmetrical anchor point is also greater than the anchor point threshold, then it will also be added to the target facial feature anchor point set. After determining one or a pair of target anchor points, the server will continue to check other anchor points and repeat the above steps until the number of target facial feature anchor points reaches the preset number threshold. After collecting a sufficient number of target facial feature anchor points, the server will determine the user's target facial expression based on the preset correspondence between these anchor points and different target facial expressions. This correspondence may be a trained model or a mapping table. Finally, the server outputs a first recognition result, that is, the user's target facial expression. This facial expression is used to indicate the user's emotional state.

For example, suppose a user is wearing AR glasses, which judge the user's emotions by analyzing the user's facial expressions. The server first starts to identify key points on the user's face in a preset order (such as from left to right, from top to bottom), such as the left corner of the eye, the right corner of the eye, the left corner of the mouth, and the right corner of the mouth. When checking these key points, the server finds that the user's left corner of the mouth is obviously upturned, and its value is greater than the set threshold. Therefore, the server uses this left corner of the mouth point as the target positioning point and adds it to the target facial feature positioning point set. Next, the server looks for the symmetrical point of the left corner of the mouth - the right corner of the mouth. If the right corner of the mouth also shows an upward trend and its value is greater than the threshold, the server will also add it to the target facial feature positioning point set. After checking these two corners of the mouth points, the server may also need to check other key points, such as eyebrows, eyes, etc., to determine whether there are other target facial feature positioning points. After collecting enough target facial feature positioning points, the server determines that the user's target facial expression is "smile" based on the correspondence between these positioning points and different facial expressions (such as "upturned corner of the mouth" corresponds to "smile"). Finally, the server outputs a first recognition result, namely, the user's facial expression is "smiling", which is used to indicate that the user's emotional state is positive or happy.

S130: Perform voice recognition on the user voice data to obtain a second recognition result.

Specifically, speech recognition is a technology that can convert human speech into machine-readable text or commands. In this process, the server receives the user's voice data (usually a sound wave signal), and then uses a speech recognition algorithm or model to analyze and process the voice signal, and finally converts it into the corresponding text or command. The "second recognition result" mentioned here is relative to the "first recognition result" mentioned above (usually the result obtained by emotion recognition of facial expression data). The second recognition result refers to the output obtained by the server after speech recognition of the user's voice data, usually a piece of text or command.

In one possible implementation, speech recognition is performed on the user voice data to obtain a second recognition result, which specifically includes: performing feature recognition on the user voice data to obtain user semantic features, user speaking rate features, and user pitch features; inputting the user semantic features, user speaking rate features, and user pitch features into a preset recognition model to obtain the user's target acoustic features, the preset recognition model is a model pre-built and trained based on deep learning, and the second recognition result includes the target acoustic features, which are used to indicate the user's emotional state.

Specifically, user semantic features usually refer to the meaning expressed by words, phrases or sentences recognized from the user's voice data. This needs to be achieved through natural language processing (NLP) technology, such as word segmentation, part-of-speech tagging, named entity recognition, etc. Speech rate refers to the speed at which the user speaks, which can be measured by counting the number of syllables or words uttered per second. Speech rate features can reflect the user's psychological state, such as tension, excitement or relaxation. Pitch refers to the high and low changes in the voice, which is usually closely related to the emotional state. For example, a high pitch may indicate excitement or surprise, while a low pitch may indicate frustration or calmness. The preset recognition model is a deep learning-based model that is pre-built and trained to recognize acoustic features in user voice. Deep learning models, such as recurrent neural networks (RNN), long short-term memory networks (LSTM) or Transformers, are usually used to process sequence data, such as speech or text. In this process, the model learns how to extract acoustic features related to the target emotional state from user semantic features, speech rate features and pitch features. This is the output obtained after processing the user's voice data through the preset recognition model. These features are used to indicate the user's emotional state, such as happiness, anger, surprise, frustration, etc. The second recognition result includes the above-mentioned target acoustic features, which not only provide the text content of the speech, but also provide important clues about the user's emotional state. For example, if the user utters the voice exclamation "Alas" in succession, the server can determine that the user's emotional state is negative.

S140: If it is determined that both the first recognition result and the second recognition result indicate that the user's emotional state is negative, then obtain current moment data.

Specifically, the first recognition result is a recognition result obtained by an emotion recognition algorithm based on the user's facial expression data. This recognition result usually indicates the user's emotional state, such as positive, negative, neutral, etc. The second recognition result is a recognition result obtained by a speech recognition and speech emotion analysis algorithm based on the user's voice data. This recognition result also indicates the user's emotional state. The server will compare the first recognition result and the second recognition result to determine whether the user's emotional state is consistent. If both recognition results indicate that the user is in a negative emotion (such as frustration, anger, anxiety, etc.), then the server will consider the user's current emotional state to be negative. Similarly, if both recognition results indicate that the user is in a positive emotion (such as happiness, joy, etc.), then the server will consider the user's current emotional state to be positive. Among them, the current moment data represents the time when the user's emotional state occurs.

S150: Generate a corresponding processing strategy based on the current moment data, and control the AR glasses to execute the processing strategy to alleviate the user's negative emotions.

Specifically, the server will generate one or more processing strategies based on the current moment data, combined with preset algorithms and models. These strategies are designed to help alleviate the user's negative emotions, and may include providing personalized suggestions, displaying relevant content or information, playing relaxing music, adjusting environmental settings, etc. Once the server generates the processing strategy, it will send the instruction to the AR glasses through wireless communication technology (such as Wi-Fi, Bluetooth, etc.). After receiving the instruction, the AR glasses will perform corresponding operations according to the strategy, such as displaying specific content, playing music, vibrating reminders, etc. By executing these processing strategies, AR glasses attempt to create a more positive and comfortable environment for users, or provide information and suggestions that help improve emotions. This helps alleviate users' negative emotions and may improve their satisfaction and experience.

For example, the server analyzes the data and finds that the user's emotional state is negative, so the server generates a processing strategy, including playing relaxing music on the display of the AR glasses and providing users with search prompts for comforting and healing movies. The server sends the processing strategy instructions to the AR glasses. After receiving the instructions, the AR glasses begin to perform the corresponding operations. Through the execution of these processing strategies, the user hears soothing music and gets help in the search process. These operations help improve the user's emotional state and make them feel more satisfied and comfortable.

In one possible implementation, a corresponding processing strategy is generated based on the current time data, specifically including: if the current time data indicates that the current time is daytime, a first processing strategy is generated, and the first processing strategy includes a user outdoor activity planning strategy, and the user outdoor activity planning strategy is displayed on the AR glasses; if the current time data indicates that the current time is night, a second processing strategy is generated, and the second processing strategy includes displaying preset text and playing preset audio strategies.

Specifically, the server can determine whether it is daytime or nighttime based on the current time data. This is based on a preset time period, such as sunrise to sunset for daytime, and sunset to sunrise the next day for nighttime. If the current time data indicates that the current time is daytime, the server will generate a first processing strategy. This strategy includes a user's outdoor activity planning strategy, which may be a series of outdoor activities planned for users based on user interests, location, weather and other factors, such as hiking, picnics, visiting attractions, etc. These activity plans will be displayed to the user through AR glasses. If the current time data indicates that the current time is night, the server will generate a second processing strategy. This strategy may be more inclined to indoor activities or relaxation methods, including displaying preset text (which may be chicken soup for the soul, inspirational mottos, etc.) and playing preset audio (which may be light music, natural sounds, etc.). These texts and audio will be displayed and played to the user through AR glasses.

For example, suppose the user is frustrated for some reason. At this time, the server is generating a processing strategy for the user through the AR glasses based on the current time data. If the current time data indicates that it is 10 am (daytime), the server will generate the first processing strategy. This strategy may be a user's outdoor activity plan, such as: "Today's weather is fine and suitable for outdoor activities. I recommend a nearby forest park where you can go hiking and picnic." This plan will be displayed on the user's AR glasses in the form of AR, and the user can intuitively see the map, route, nearby facilities and other information of the forest park. If the current time data indicates that it is 8 pm (night), the server will generate the second processing strategy. This strategy may include displaying a preset text on the screen of the AR glasses, such as: "Every night is a new beginning. Let go of the fatigue of the day, and tomorrow will be better." At the same time, the server will also control the AR glasses to play a preset light music or natural sound, such as the sound of waves and forest insects, to help users relax and fall asleep.

In a possible implementation, refer to FIG2 , which is another flow chart of an AR-based emotional data processing method provided in an embodiment of the present application. Before inputting the user semantic features, user speech rate features, and user pitch features into the preset recognition model, train the preset recognition model; training the preset recognition model includes steps S210 to S230, and the above steps are as follows: S210, using CNN to extract global spatiotemporal features from the user's historical voice data, and the global spatiotemporal features include user historical semantic features, user historical speech rate features, and user historical pitch features; S220, using RNN to perform sequence modeling on the global spatiotemporal features to obtain time-dependent features; S230, using the Softmax function to classify the time-dependent features and output an emotion label, one emotion label corresponds to one target acoustic feature, and the emotion label includes a positive emotion label, a neutral emotion label, and a negative emotion label.

Specifically, in order for the preset recognition model to accurately recognize emotions, it needs to be trained with data with known labels. In this process, the model learns how to extract features from the input data and map these features to corresponding emotion labels. CNN (Convolutional Neural Network) is used to process one-dimensional time series data (such as speech signals). Here, CNN is used to extract global spatiotemporal features from the user's historical speech data. These features include user historical semantic features (semantic information extracted from the speech content), user historical speech rate features (the rate or rhythm of the speech), and user historical pitch features (the pitch or pitch of the speech). RNN (Recurrent Neural Network) is particularly good at processing sequence data because it can capture the temporal dependencies in the data. Here, RNN is used to perform sequence modeling on the global spatiotemporal features extracted by CNN, thereby obtaining features containing temporal dependency information. The Softmax function is a multi-classification function that converts the output of the model into a probability distribution, with each probability corresponding to a possible emotion label. In this example, the Softmax function maps the temporal dependency features output by the RNN to three emotion labels: positive emotion emotion label, neutral emotion emotion label, and negative emotion emotion label.

In a possible implementation, the method also includes: if it is determined that the first recognition result indicates that the user's emotional state is a positive emotion, and the second recognition result indicates that the user's emotional state is a negative emotion, or, if it is determined that the first recognition result indicates that the user's emotional state is a negative emotion, and the second recognition result indicates that the user's emotional state is a positive emotion, then generating a consulting strategy and controlling the AR glasses to execute the consulting strategy, and the consulting strategy is used to determine the user's emotional state.

Specifically, if the two recognition results are inconsistent, such as one indicating that the user is in a positive emotional state and the other indicating that the user is in a negative emotional state, the server needs to further determine the user's true emotional state. In the case of inconsistent recognition results, the server generates a consultation strategy. This strategy includes displaying some questions or prompts to the user through AR glasses, asking the user to actively confirm his or her emotional state, or interacting with the user through other means (such as voice prompts) to obtain more accurate emotional information. Once the consultation strategy is generated, the server controls the AR glasses to execute this strategy. This can include displaying questions or prompts on the display of the AR glasses, or playing voice prompts through the audio output function of the AR glasses. By executing the consultation strategy, the server can obtain direct feedback from the user about his or her emotional state, thereby more accurately determining the user's emotional state.

For example, suppose a user is wearing AR glasses, and the server identifies the user's emotional state through two different methods. The first recognition result may indicate that the user's voice is full of enthusiasm and vitality, so the user is judged to be in a positive emotional state. The second recognition result may find that the user's facial expression appears frustrated and lost, so the user is judged to be in a negative emotional state. The server detects that there is an inconsistency between the two recognition results, that is, one is judged as a positive emotion and the other is judged as a negative emotion. The server will generate a consulting strategy that asks the user about the current emotional state by displaying a message on the display of the AR glasses, such as: "Are you feeling happy or unhappy?" At the same time, the server may also play the corresponding voice prompt through the audio output function of the AR glasses. The AR glasses execute the consulting strategy according to the instructions of the system, displaying the message on the display and playing the voice prompt through the audio output. After seeing and hearing the prompts of the AR glasses, the user interacts with the AR glasses through voice or gestures to confirm his true emotional state. The server finally determines the user's emotional state based on the user's feedback and adopts the corresponding processing strategy based on this state.

The present application also provides an AR-based emotion data processing device, with reference to FIG3, which is a module diagram of an AR-based emotion data processing device provided in an embodiment of the present application. The emotion data processing device is a server, and the server includes an acquisition module 31 and a processing module 32, wherein the acquisition module 31 acquires the emotion data for the user sent by the AR glasses, the user wears the AR glasses, and the emotion data includes facial expression data and user voice data; the processing module 32 performs emotion recognition on the facial expression data to obtain a first recognition result; the processing module 32 performs voice recognition on the user voice data to obtain a second recognition result; if the processing module 32 determines that both the first recognition result and the second recognition result indicate that the user's emotional state is negative, then the current moment data is acquired; the processing module 32 generates a corresponding processing strategy based on the current moment data, and controls the AR glasses to execute the processing strategy to alleviate the user's negative emotions.

In a possible implementation, the acquisition module 31 acquires the emotion data for the user sent by the AR glasses, specifically including: the acquisition module 31 receives the original facial expression data and the original user voice data sent by the AR glasses; the processing module 32 preprocesses the original facial expression data and the original user voice data to obtain the emotion data, and the preprocessing includes denoising, filtering and normalization.

In a possible implementation, the processing module 32 performs emotion recognition on the facial expression data to obtain a first recognition result, which specifically includes: the processing module 32 identifies at least one positioning point included in the facial expression data according to a preset positioning point numbering sequence; the processing module 32 determines a target positioning point whose positioning point value is greater than a positioning point threshold among at least one positioning point, and adds the target positioning point to the target facial feature positioning point; the processing module 32 searches for a symmetrical positioning point of the target positioning point, and if the positioning point value of the symmetrical positioning point is greater than the positioning point threshold, the symmetrical positioning point is added to the target facial feature positioning point; the processing module 32 determines a target positioning point whose next positioning point value is greater than the positioning point threshold among at least one positioning point, and executes the step of determining a target positioning point whose positioning point value is greater than the positioning point threshold among at least one positioning point; the processing module 32 performs the step of processing until the number of target facial feature positioning points reaches a preset number threshold; the processing module 32 determines the target facial expression of the user based on the correspondence between different target facial feature positioning points and different target facial expressions, and the first recognition result includes the target facial expression of the user, and the target facial expression is used to indicate the emotional state of the user.

In one possible implementation, the processing module 32 performs speech recognition on the user voice data to obtain a second recognition result, which specifically includes: the processing module 32 performs feature recognition on the user voice data to obtain user semantic features, user speaking rate features, and user pitch features; the processing module 32 inputs the user semantic features, user speaking rate features, and user pitch features into a preset recognition model to obtain the user's target acoustic features, the preset recognition model is a model pre-built and trained based on deep learning, and the second recognition result includes the target acoustic features, which are used to indicate the user's emotional state.

In a possible implementation, the processing module 32 trains a preset recognition model before inputting the user semantic features, the user speaking rate features and the user pitch features into the preset recognition model; the training of the preset recognition model specifically includes: the processing module 32 uses CNN to extract global spatiotemporal features from the user's historical voice data, and the global spatiotemporal features include the user's historical semantic features, the user's historical speaking rate features and the user's historical pitch features; the processing module 32 uses RNN to perform sequence modeling on the global spatiotemporal features to obtain time-dependent features; the processing module 32 uses the Softmax function to classify the time-dependent features and output an emotion label, where one emotion label corresponds to one target acoustic feature, and the emotion label includes a positive emotion label, a neutral emotion label and a negative emotion label.

In one possible implementation, the processing module 32 generates a corresponding processing strategy based on the current time data, specifically including: if the current time data indicates that the current time is daytime, the processing module 32 generates a first processing strategy, the first processing strategy includes a user outdoor activity planning strategy, and the user outdoor activity planning strategy is displayed on the AR glasses; if the current time data indicates that the current time is night, the processing module 32 generates a second processing strategy, the second processing strategy includes displaying preset text and playing preset audio strategies.

In one possible implementation, if the processing module 32 determines that the first recognition result indicates that the user's emotional state is a positive emotion and the second recognition result indicates that the user's emotional state is a negative emotion, or if the processing module 32 determines that the first recognition result indicates that the user's emotional state is a negative emotion and the second recognition result indicates that the user's emotional state is a positive emotion, then a consulting strategy is generated and the AR glasses are controlled to execute the consulting strategy, where the consulting strategy is used to determine the user's emotional state.

It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

The present application also provides an electronic device, referring to Figure 4, which is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include: at least one processor 41, at least one network interface 44, a user interface 43, a memory 45, and at least one communication bus 42.

The communication bus 42 is used to realize the connection and communication between these components.

The user interface 43 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 43 may also include a standard wired interface and a wireless interface.

The network interface 44 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

Among them, the processor 41 may include one or more processing cores. The processor 41 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 45, and calling data stored in the memory 45. Optionally, the processor 41 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 41 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 41, but may be implemented separately through a chip.

Among them, the memory 45 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 45 includes a non-transitory computer-readable storage medium. The memory 45 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 45 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 45 may also be at least one storage device located away from the aforementioned processor 41. As shown in Figure 4, the memory 45 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for an AR-based emotional data processing method.

In the electronic device shown in FIG4 , the user interface 43 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 41 can be used to call an application program storing an AR-based emotion data processing method in the memory 45. When executed by one or more processors, the electronic device executes one or more methods in the above-mentioned embodiments.

It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors, the electronic device executes one or more of the methods described in the above embodiments.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, it will be easy for those skilled in the art to think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims (10)

1. An AR-based emotion data processing method, the method comprising:

the method comprises the steps of obtaining emotion data sent by AR glasses and aiming at a user, wherein the user wears the AR glasses, and the emotion data comprises facial expression data and user voice data;

carrying out emotion recognition on the facial expression data to obtain a first recognition result;

performing voice recognition on the user voice data to obtain a second recognition result;

If the first recognition result and the second recognition result are determined to indicate that the emotion state of the user is negative emotion, acquiring current moment data;

and generating a corresponding processing strategy according to the current time data, and controlling the AR glasses to execute the processing strategy so as to relieve the negative emotion of the user.

2. The AR-based emotion data processing method of claim 1, wherein the acquiring emotion data for a user sent by AR glasses specifically includes:

Receiving original facial expression data and original user voice data sent by the AR glasses;

Preprocessing the original facial expression data and the original user voice data to obtain the emotion data, wherein the preprocessing comprises denoising, filtering and normalization.

3. The AR-based emotion data processing method of claim 1, wherein said emotion recognition of said facial expression data to obtain a first recognition result specifically includes:

Identifying at least one locating point included in the facial expression data according to a preset locating point numbering sequence;

determining a target positioning point with one positioning point value larger than a positioning point threshold value in the at least one positioning point, and adding the target positioning point into a target facial feature positioning point;

searching a symmetrical locating point of the target locating point, and if the locating point value of the symmetrical locating point is larger than the locating point threshold value, adding the symmetrical locating point into the target facial feature locating point;

Determining a target positioning point with a next positioning point value larger than a positioning point threshold value in the at least one positioning point, and executing the step of determining the target positioning point with the next positioning point value larger than the positioning point threshold value in the at least one positioning point;

until the number of the target facial feature positioning points reaches a preset number threshold;

And determining the target facial expression of the user according to the corresponding relation between different target facial feature positioning points and different target facial expressions, wherein the first recognition result comprises the target facial expression of the user, and the target facial expression is used for indicating the emotional state of the user.

4. The AR-based emotion data processing method of claim 1, wherein said performing speech recognition on said user speech data to obtain a second recognition result specifically includes:

performing feature recognition on the user voice data to obtain user semantic features, user speech speed features and user tone features;

inputting the user semantic features, the user speech speed features and the user tone features into a preset recognition model to obtain target acoustic features of the user, wherein the preset recognition model is a model which is built and trained in advance based on deep learning, and the second recognition result comprises the target acoustic features which are used for indicating the emotional state of the user.

5. The AR-based emotion data processing method of claim 4, wherein said preset recognition model is trained prior to said inputting said user semantic features, said user speech rate features, and said user pitch features into said preset recognition model; the training of the preset recognition model specifically comprises the following steps:

extracting global space-time features from the historical voice data of the user by adopting CNN, wherein the global space-time features comprise user historical semantic features, user historical speech speed features and user historical tone features;

performing sequence modeling on the global space-time features by adopting RNNs to obtain time sequence dependent features;

and classifying the time sequence dependent features by adopting a Softmax function, outputting emotion labels, wherein one emotion label corresponds to one target acoustic feature, and the emotion labels comprise positive emotion labels, neutral emotion labels and negative emotion labels.

6. The AR-based emotion data processing method of claim 1, wherein the generating a corresponding processing policy according to the current time data specifically includes:

If the current time data indicates that the current time is daytime, a first processing strategy is generated, wherein the first processing strategy comprises a user outgoing activity planning strategy, and the user outgoing activity planning strategy is displayed on the AR glasses;

And if the current time data indicates that the current time is night, generating a second processing strategy, wherein the second processing strategy comprises displaying a preset text and playing a preset audio strategy.

7. The AR-based emotion data processing method of claim 1, further comprising:

if it is determined that the first recognition result indicates that the emotional state of the user is a positive emotion, the second recognition result indicates that the emotional state of the user is a negative emotion, or,

If the first recognition result indicates that the emotion state of the user is a negative emotion, the second recognition result indicates that the emotion state of the user is a positive emotion, a consultation strategy is generated, the AR glasses are controlled to execute the consultation strategy, and the consultation strategy is used for determining the emotion state of the user.

8. An AR-based emotion data processing device, characterized in that the AR-based emotion data processing device comprises an acquisition module (31) and a processing module (32), wherein,

The acquiring module (31) is configured to acquire emotion data sent by the AR glasses and specific to a user, where the user wears the AR glasses, and the emotion data includes facial expression data and user voice data;

The processing module (32) is used for carrying out emotion recognition on the facial expression data to obtain a first recognition result;

the processing module (32) is further used for performing voice recognition on the user voice data to obtain a second recognition result;

the processing module (32) is further configured to acquire current time data if it is determined that the first recognition result and the second recognition result both indicate that the emotional state of the user is a negative emotion;

The processing module (32) is further configured to generate a corresponding processing policy according to the current time data, and control the AR glasses to execute the processing policy so as to alleviate the negative emotion of the user.

9. An electronic device, characterized in that the electronic device comprises a processor (41), a memory (45), a user interface (43) and a network interface (44), the memory (45) being arranged to store instructions, the user interface (43) and the network interface (44) being arranged to communicate to other devices, the processor (41) being arranged to execute the instructions stored in the memory (45) to cause the electronic device to perform the method according to any one of claims 1 to 7.

10. A computer readable storage medium storing instructions which, when executed, perform the method of any one of claims 1 to 7.