CN109542230B - Image processing method, device, electronic device and storage medium - Google Patents

Abstract

Embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium, which relate to the technical field of image processing. The method includes: in the process of the user participating in the interactive activity, obtaining the face image of the user; obtaining the facial expression parameter of the user based on the facial image; judging the interactive state of the user's next operation of the interactive activity based on the facial expression parameter; generating and outputting according to the interactive state The prompt information corresponding to the interactive state. Since the user's interaction state for the next step of the interactive activity can be determined based on the user's demeanor parameters, it is possible to generate and output a prompt for the user when it is determined according to the user's interaction state that the user's decision at this time needs to be improved. Help users quickly improve the level of decision-making in interactive activities.

Description

Image processing method, device, electronic device and storage medium

technical field

The present application relates to the technical field of image processing, and in particular, to an image processing method, apparatus, electronic device, and storage medium.

Background technique

At present, users can participate in many interactive activities, such as chess interactive activities or card-based interactive activities, etc., but in the process of interactive activities, it is difficult for users to actively discover which decisions need to be improved, which makes users prone to encountering To the bottleneck can not continue to improve the level of decision-making in interactive activities.

SUMMARY OF THE INVENTION

The present application is to provide an image processing method, apparatus, electronic device and storage medium, so as to provide prompts for the user's decision-making to be improved, so as to help the user to quickly improve the decision-making level in interactive activities.

In order to achieve the above purpose, the embodiments of the present application are achieved in the following ways:

In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

Obtain the user's face image during the user's participation in interactive activities;

Based on the facial image, obtain the demeanor parameter of the user;

Based on the demeanor parameter, determine the interactive state of the user's next operation of the interactive activity;

Generate and output prompt information corresponding to the interaction state according to the interaction state.

With reference to the first aspect, in some possible implementations, the facial images include M facial images obtained within a first preset time period before the current moment, and the facial parameters include M corresponding to the M facial images The attitude parameter, M is an integer greater than 1. Based on the attitude parameter, the interactive state of the user's next operation of the interactive activity is judged, including:

Obtaining M focus points of the user's sight on the interactive interface of the interactive activity based on the M attitude parameters;

Based on the M focus points, it is determined whether the interaction state of the user's next operation of the interactive activity is in the focus focus state.

With reference to the first aspect, in some possible implementations, judging whether the interaction state of the user's next operation of the interactive activity is in the line-of-sight state based on the M focus points, including:

judging whether the number of sight focal points located in the same area of at least two areas on the interactive interface among the M sight focal points is greater than or equal to a first preset number;

If so, it indicates that the user's interaction state of the next operation of the interaction activity is in the focus state.

With reference to the first aspect, in some possible implementations, obtaining M focus points of the user's sight on the interactive interface of the interactive activity based on the M facial parameters, including:

Determine every two gaze directions of the user corresponding to each of the M attitude parameters;

The sight focus formed on the interactive interface of the interactive activity is determined for every two sight directions, and the M sight focus points are determined in total.

With reference to the first aspect, in some possible implementations, the facial image further includes N facial images obtained within a second preset time period before the current moment, where N is an integer greater than 1, and the facial parameter includes the N facial images. N attitude parameters corresponding to a face image, after judging the interaction state of the user's next operation of the interactive activity based on the attitude parameters, and after generating and outputting the interaction state according to the interaction state Before the corresponding prompt information, the method further includes:

After it is determined that the user is in the focused state of sight, based on the N attitude parameters, determine the emotional type corresponding to each of the N attitude parameters, and there are N emotional types in total;

Based on the N emotion types, determine whether the interaction state of the user is in a non-positive emotional state;

If yes, perform the step of generating and outputting prompt information corresponding to the interaction state according to the interaction state.

With reference to the first aspect, in some possible implementations, based on the N emotion types, determining whether the interaction state of the user is in a non-positive emotional state includes:

Determine whether the number of non-positive emotions in the N emotion types is greater than or equal to a second preset number, wherein the number of non-positive emotions greater than or equal to the second preset number indicates the interaction of the user The state is in a non-positive emotional state.

With reference to the first aspect, in some possible implementations, based on the N state parameters, the emotion type corresponding to each state parameter in the N state parameters is determined, and there are N emotion types in total, including:

The facial emotion analysis model is used to analyze each demeanor parameter of the N deity parameters, and each demeanor parameter output by the facial emotion analysis model is obtained as each emotion type to be determined among the various emotion types to be determined. The probability;

According to the probability of each emotion type to be determined, the emotion type to be determined with the highest probability among the multiple emotion types to be determined is determined, wherein, the emotion type to be determined with the highest probability of each state parameter is the emotion type corresponding to each state parameter. Emotion type.

With reference to the first aspect, in some possible implementations, generating and outputting prompt information corresponding to the interaction state according to the interaction state, including:

determining the object in the interactive activity contained in the same area according to the user's interactive state in the line-of-sight focused state;

Generate prompt information related to the next operation according to the object, and output the prompt information.

With reference to the first aspect, in some possible implementations, prompt information related to the next operation is generated according to the object, and the prompt information is output, including:

According to the object, determine whether the object is an entity in the interactive activity or a background in the interactive activity;

If the object is an entity in the interactive activity, increase the weight used to calculate the entity in the current evaluation function from a first value to a second value to obtain a currently adjusted evaluation function, based on the current adjustment The subsequent evaluation function generates prompt information of the next operation related to the entity; if the object is the background in the interactive activity, the prompt information of the next operation is generated based on the current evaluation function.

With reference to the first aspect, in some possible implementations, the facial images include M facial images obtained within a first preset time period before the current moment and N faces obtained within a second preset time period before the current moment. image, the state parameters include: M state parameters corresponding to the M facial images and N state parameters corresponding to the N facial images, where M and N are integers greater than 1, and based on the state parameters, determine The interaction state of the user's next operation of the interactive activity, including:

Obtaining M focus points of the user on the interactive interface of the interactive activity based on the M attitude parameters, and determining the emotion type corresponding to each of the N attitude parameters based on the N attitude parameters , a total of N emotion types;

Based on the M line-of-sight focal points, determine whether the number of line-of-sight focal points located in the same area of at least two areas on the interactive interface is greater than or equal to a first preset number among the M line-of-sight focal points; and Emotion type, determine whether the number of non-positive emotions in the N emotion types is greater than or equal to a second preset number;

When it is determined that the number of sight-focus points in the same area satisfies the first preset number, it is determined that the interaction state of the user's next operation of the interactive activity is in the sight-focus state; when determining the N emotion types When the number of non-positive emotions meets the second preset number, it is determined that the interaction state of the user is in a non-positive emotion state.

With reference to the first aspect, in some possible implementations, the method further includes:

When it is determined that the facial image does not contain at least part of the facial features of the user, an image acquisition angle adjustment prompt is generated and output, wherein the facial features of the user include the facial features of the user.

In a second aspect, an embodiment of the present application provides an image processing apparatus, and the apparatus includes:

The image obtaining module is used to obtain the face image of the user during the user's participation in the interactive activity.

The facial expression obtaining module is used to obtain the facial expression parameters of the user based on the facial image.

An operation judging module, configured to judge the interactive state of the next operation of the interactive activity by the user based on the facial expression parameter.

A prompt output module is configured to generate and output prompt information corresponding to the interaction state according to the interaction state.

With reference to the second aspect, in some optional implementations, the facial images include M facial images obtained within a first preset time period before the current moment, and the facial parameters include M corresponding to the M facial images. A state parameter, M is an integer greater than 1,

The operation judging module is further configured to obtain M focus points of the user on the interactive interface of the interactive activity based on the M attitude parameters; Whether the interaction state of the next operation is in the focus state.

In conjunction with the second aspect, in some optional implementations,

The operation judging module is further configured to judge whether the number of sight focal points located in the same area of at least two areas on the interactive interface among the M sight focal points is greater than or equal to a first preset number; The interactive state of the user's next operation of the interactive activity is in the focus state.

In conjunction with the second aspect, in some optional implementations,

The operation judging module is further configured to determine every two line of sight directions of the user corresponding to each of the M state parameters; and determine that every two line of sight directions are formed on the interactive interface of the interactive activity. The M line-of-sight focal points are determined in total.

With reference to the second aspect, in some optional implementation manners, the facial image further includes N facial images obtained within a second preset time period before the current moment, where N is an integer greater than 1, and the demeanor parameter includes the The N state parameters corresponding to the N facial images, based on the state parameters, the device further includes:

An emotion determination module, configured to determine, after determining that the user is in the focus state, based on the N state parameters, determine the emotion type corresponding to each state parameter in the N state parameters, a total of N emotion types ;

A prompt determination module, configured to judge whether the interaction state of the user is in a non-positive emotional state based on the N emotion types;

The prompt execution module is configured to, if yes, perform the step of: generating and outputting prompt information corresponding to the interaction state according to the interaction state.

In conjunction with the second aspect, in some optional implementations,

The prompt determination module is further configured to determine whether the number of non-positive emotions in the N emotion types is greater than or equal to a second preset number, wherein the number of non-positive emotions is greater than or equal to the second preset number The number indicates that the interaction state of the user is in a non-positive emotional state.

In conjunction with the second aspect, in some optional implementations,

The emotion determination module is further configured to analyze each demeanor parameter in the N deity parameters through a facial emotion analysis model, and obtain each demeanor parameter output by the facial emotion analysis model as a plurality of types to be determined. The probability of each to-be-determined emotion type in the emotion type; according to the probability of each to-be-determined emotion type, determine the to-be-determined emotion type with the highest probability among the multiple to-be-determined emotion types, wherein the probability of each demeanor parameter is the highest The emotion type to be determined is the emotion type corresponding to each facial expression parameter.

In conjunction with the second aspect, in some optional implementations,

The prompt output module is further configured to determine the object in the interactive activity included in the same area according to the user's interactive state in the line-of-sight focused state; prompt information for one-step operation, and output the prompt information.

In conjunction with the second aspect, in some optional implementations,

The prompt output module is further configured to determine, according to the object, whether the object is an entity in the interactive activity or a background in the interactive activity; if the object is an entity in the interactive activity, the The weight used to calculate the entity in the current evaluation function is increased from the first value to the second value, the currently adjusted evaluation function is obtained, and the lower evaluation function related to the entity is generated based on the currently adjusted evaluation function. Prompt information for one step operation; if the object is the background in the interactive activity, generate prompt information for the next step operation based on the current evaluation function.

In combination with the second aspect, in some optional implementations, the facial images include M facial images obtained within a first preset time period before the current moment and N face images obtained within a second preset time period before the current moment. facial images, the facial parameters include: M facial parameters corresponding to the M facial images and N facial parameters corresponding to the N facial images, where M and N are integers greater than 1,

The operation judgment module is further configured to obtain M focus points of the user on the interactive interface of the interactive activity based on the M attitude parameters, and determine the N attitude parameters based on the N attitude parameters The emotion type corresponding to each demeanor parameter in , there are a total of N emotion types; based on the M sight focus, determine the number of sight focus located in the same area in at least two areas on the interactive interface among the M sight focus Whether it is greater than or equal to a first preset number; and based on the N emotion types, judging whether the number of non-positive emotions in the N emotion types is greater than or equal to a second preset number; When the number of focus satisfies the first preset number, it is determined that the interaction state of the user's next operation of the interactive activity is in the focus state; when it is determined that the number of non-positive emotions in the N emotion types satisfies all When the second preset number is reached, it is determined that the interaction state of the user is in a non-positive emotional state.

In conjunction with the second aspect, in some optional implementations, the apparatus further includes:

The angle prompting module is configured to generate and output an image acquisition angle adjustment prompt when it is determined that the facial image does not contain at least part of the facial features of the user, wherein the facial features of the user include the user's facial features. facial features.

In a third aspect, an embodiment of the present application provides an electronic device, the electronic device includes: a processor, a memory, a bus, and a communication interface; the processor, the communication interface, and the memory are connected through the bus. The memory is used to store programs. The processor is configured to execute the image processing method according to the first aspect and any one of the implementation manners of the first aspect by calling the program stored in the memory.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer-executable non-volatile program code, the program code causing the computer to execute the first aspect and any one of the first aspect The image processing method described in the embodiment.

The beneficial effects of the embodiments of the present application are:

Since the user's interaction state for the next step of the interactive activity can be determined based on the user's attitude parameters, it is possible to generate and output a prompt for the user when it is determined according to the user's interaction state that the user's decision at this time needs to be improved. Help users to quickly improve the level of decision-making in interactive activities.

In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the preferred embodiments are exemplified below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following drawings will briefly introduce the drawings that need to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore do not It should be regarded as a limitation of the scope, and for those of ordinary skill in the art, other related drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 shows a structural block diagram of an electronic device provided by a first embodiment of the present application;

FIG. 2 shows a first flowchart of an image processing method provided by a second embodiment of the present application;

FIG. 3 shows the first sub-flow chart of step S130 in the first flow chart of an image processing method provided by the second embodiment of the present application;

FIG. 4 shows a second flowchart of an image processing method provided by the second embodiment of the present application;

FIG. 5 shows a second sub-flow chart of step S130 in the first flow chart of an image processing method provided by the second embodiment of the present application;

FIG. 6 shows a structural block diagram of an image processing apparatus provided by a third embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Thus, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the application as claimed, but is merely representative of selected embodiments of the application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures. The terms "first", "second", etc. are only used to differentiate the description and should not be construed to indicate or imply relative importance.

first embodiment

Referring to FIG. 1, an embodiment of the present application provides an electronic device 10, and the electronic device 10 may be a terminal device or a server. The terminal device can be a personal computer (persoMalcomputer, PC), tablet computer, smart phone, personal digital assistant (persoMal digital assistaMt, PDA), etc.; the server can be a network server, a database server, a cloud server, or a server composed of multiple sub-servers Server integration, etc.

In this embodiment, the electronic device 10 may include: a memory 11 , a communication interface 11 , a bus 13 and a processor 14 . The processor 14 , the communication interface 11 and the memory 11 are connected through the bus 13 . The processor 14 is used to execute executable modules, such as computer programs, stored in the memory 11 . The components and structures of the electronic device 10 shown in FIG. 1 are only exemplary and not limiting, and the electronic device 10 may also have other components and structures as required.

The memory 11 may include a high-speed random access memory (RaMdom Access Memory RAM), and may also include a non-volatile memory (MoM-volatile memory), such as at least one disk memory. In this embodiment, the memory 11 stores programs required to execute the image processing method.

The bus 13 may be an ISA bus, a PCI bus, an EISA bus, or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one bidirectional arrow is shown in FIG. 1, but it does not mean that there is only one bus or one type of bus.

The processor 14 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 14 or an instruction in the form of software. The above-mentioned processor 14 may be a general-purpose processor, including a central processing unit (CeMtral ProcessiMg UMit, referred to as CPU), a network processor (Metwork Processor, referred to as MP), etc.; may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. Various methods, steps, and logical block diagrams disclosed in the embodiments of the present invention can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art.

The flow process disclosed in any of the embodiments of the present invention or the method executed by the defined apparatus may be applied to the processor 14 or implemented by the processor 14 . After the processor 14 receives the execution instruction and calls the program stored in the memory 11 through the bus 13, the processor 14 controls the communication interface 11 through the bus 13 to execute the method flow of the image processing method.

In addition, in some cases, if the electronic device 10 is a terminal device, the electronic device 10 may further have a camera 15, and the camera 15 may be a conventional high-definition line camera. The camera 15 can be connected to the bus 13 , and the camera 15 can be used to capture images including objects, so that the processor 14 of the electronic device 10 obtains images captured by the camera 15 based on the bus 13 and executes the method flow of the image processing method.

In addition, in other cases, if the electronic device 10 is a server, the electronic device 10 can obtain the user's image with the terminal that collects the user's image, so that the electronic device 10 can execute the method flow of the image processing method based on the obtained image. .

Second Embodiment

This embodiment provides an image processing method. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although shown in the flowchart A logical order is presented, but in some cases steps shown or described may be performed in an order different from that herein. This embodiment will be described in detail below.

Referring to FIG. 2 , in the image processing method provided in this embodiment, the image processing method includes: step S110 , step S120 , step S130 and step S140 .

Step S110: During the process of the user participating in the interactive activity, the facial image of the user is obtained.

Step S120: Based on the facial image, obtain the facial parameter of the user.

Step S130: Based on the demeanor parameter, determine the interactive state of the user's next operation of the interactive activity.

Step S140: Generate and output prompt information corresponding to the interaction state according to the interaction state.

Each step of the present application will be described in detail below with reference to FIG. 2 and FIG. 3 .

Step S110: During the process of the user participating in the interactive activity, the facial image of the user is obtained.

Interactive activities can be interactive applications, and the types of interactive applications can be, for example, web-based cloud applications where small programs are installed on terminal devices and most of them are in the cloud, or most programs are installed and run Traditional applications on end devices.

When the electronic device is a terminal device, the electronic device can provide an interactive interface for the user by running the interactive application, so that the user can participate in the interactive activities of the interactive application based on the interactive interface. Optionally, the interactive activity may be a chess and card activity, for example, may be: Gobang, Go, Chinese chess, international chess, playing cards, or mahjong, etc., which is not limited in this embodiment.

In order to increase the real experience of users after participating in these interactive activities, the interactive interface can display the real game interface of these chess and card activities. For example, the interactive interface can display the game interface of Chinese chess. For example, The interactive interface can also display the game interface of the poker.

Furthermore, after the user participates in the interactive activity, the opponent played by the user in the interactive activity can be an AI computer (Artificial Intelligence, artificial intelligence), or the opponent played by the user in the interactive activity can be another user. For example, a user can play Chinese chess with an AI computer based on an electronic device, but a user can also play Chinese chess with other users based on the electronic device. That is to say, although the opponents played by the user may be different, the game interface presented on the electronic device may be the same.

Furthermore, the game interface of the electronic device may provide the user with an option of whether to enable the operation prompt. If the user chooses to enable the operation prompt based on this option, then the electronic device can start the execution of the image processing method in response to the user's opening operation. ; On the contrary, the execution of the image processing method is not started, and the user is allowed to participate in the interactive activities in the traditional mode.

After the electronic device starts to execute the image processing method, the electronic device can photograph the user's face based on the camera, and obtain a video stream of the user's face photographed by the camera. Since the video stream may be composed of multiple frames of consecutive user's face images, obtaining the video of the user's face by the electronic device can also be understood that the electronic device obtains multiple frames of the user's face images.

In the case where the electronic device is a server, the electronic device can implement the image processing method without directly interacting with the user. In this case, the interactive application can be run on the user terminal used by the user. When the user terminal used by the user participates in the interactive activities of the interactive application, the electronic device can interact with the user terminal used by the user. Thus, the execution of the image processing method is realized. Therefore, the electronic device can obtain the video stream of the user's face captured by the camera of the user terminal from the user terminal, and in this way, the electronic device also obtains multiple frames of facial images of the user.

As an optional way to store the video stream, since the electronic device can continuously obtain the video stream of the user's face during the process from the start of the execution of the image processing method to the end of the user's participation in the interactive activity, the video stream of the user's face in the electronic device In the case of limited storage space, the electronic device can update the stored video stream in real time, so as to store the video stream within a period of time before the current moment in real time. For example, during the entire interactive activity, the electronic device can ensure real-time storage of the video stream 1-5 minutes before the current moment by updating the stored video stream.

After the electronic device obtains the multiple frames of facial images of the user, the electronic device may continue to perform step S120.

Step S120: Based on the facial image, obtain the facial parameter of the user.

It can be understood that in the process of the user participating in the interactive activity, it is more appropriate if the user does not know or is not sure how to operate the next step of the interactive activity. For example, in the process of Chinese chess, the user does not know which piece should go Where is the best way to go, in this case, based on physiological responses, users often appear in a state of thinking, that is, the user's eyes will focus on a certain place to facilitate thinking. Therefore, the electronic device can determine whether the user does not know the next appropriate action based on this physiological response.

When the user is in a thinking state, it is a procedural behavior that the user's eyes will focus on a certain place, that is, the user's eyes can focus on a certain place for a period of time, so that he can concentrate on thinking during this time. Therefore, in the user's multi-frame facial images, it is difficult for a single frame of facial image to accurately reflect the user's thinking state, and it is necessary to use multiple facial images within a period of time to more accurately reflect the user's thinking state. . Therefore, the electronic device can process the plurality of facial images to determine whether the user is in a thinking state of not knowing the next appropriate operation of the interactive activity by processing the plurality of facial images.

Optionally, since the time in which the user is in the thinking state is not too long in most cases, if the time corresponding to the multiple facial images processed by the electronic device is too long, the obtained results may be inaccurate. The user is in a state of thinking for 3 seconds, and then is in a state of distraction for the next 10 seconds. If multiple facial images covering the entire 13 seconds are processed, it is very likely that the processing structure that the user is in a state of thinking will not be obtained. Therefore, the electronic device may process the M facial images obtained within a first preset time period that is shorter from the current moment to ensure the accuracy of the obtained results, where M may be an integer greater than 1, and the first preset The duration can be 1-3 seconds, but is not limited.

In this embodiment, as a way to obtain M facial images, the way for the electronic device to obtain the M facial images may be that the electronic device extracts the most recent 1-3 seconds of video from the stored video stream of 1-5 minutes , and take the multiple frames of facial images contained in this video as M facial images.

As another way to obtain M facial images, at the level of each frame, since it is difficult for the facial expressions corresponding to every two adjacent frames of facial images to change abruptly, this principle can be used to ensure the accuracy of the results. At the same time, the calculation amount of the electronic device is reduced, that is, the electronic device can extract a part of the frames from the 1-3 seconds of this video as the obtained M facial images, for example, from every two consecutive frames of facial images or consecutive One frame of face image is extracted from the three frames of face images, but it is not limited.

It can also be understood that, since the user's thinking state can generally be reflected as the user's eyes are focused, the electronic device can determine whether the user is in a thinking state based on processing and analyzing the image of the eye part in the user's facial image.

In this embodiment, a trained facial emotion analysis model is preset in the electronic device, so that the electronic device can call the facial emotion analysis model, and can input each facial image of the M facial images to the person The facial emotion analysis model, so that the facial emotion analysis model can perform matting processing on each facial image based on the deep neural network, and determine the image of the eye part of the user's face from each facial image. Therefore, the electronic device can obtain the image parameters corresponding to the images of the eyes in each facial image output by the facial emotion analysis model, and obtain M image parameters in total.

It can be understood that, in this embodiment, the function of the image of the binocular part can be used to determine the user's attitude, that is, to determine whether the user's attitude is in a state of thinking, therefore, the image data corresponding to the image of the binocular part can be the user's attitude parameter. . In this way, when the electronic device obtains the M image parameters of the user, it can obtain the M demeanor parameters of the user.

After obtaining the M state parameters, the electronic device may continue to perform step S130.

Step S130: Based on the demeanor parameter, determine the interactive state of the user's next operation of the interactive activity.

As shown in FIG. 3, in this embodiment, the sub-flow of step S130 may include: step S131 and step S132.

Step S131: Obtain M focus points of the user's sight on the interactive interface of the interactive activity based on the M attitude parameters.

Step S132: Based on the M sight focus points, determine whether the interaction state of the user's next operation of the interactive activity is in the sight focus state.

The flow of steps S131 and S132 will be described in detail below.

Step S131: Obtain M focus points of the user's sight on the interactive interface of the interactive activity based on the M attitude parameters.

Since what the electronic device can determine is whether the user's gaze is focused, the electronic device can determine the user's gaze focus based on the M facial parameters.

In detail, the electronic device may also continue to call the facial emotion analysis model, so as to input M facial expression parameters into the facial emotion analysis model. The facial emotion analysis model can calculate each of the M facial parameters based on the deep neural network, so as to determine every two gaze directions of the user corresponding to each facial parameter.

It should be noted that, in general, the user can watch through both eyes, and the gaze direction of each eyeball in the eyes can correspond to a line of sight direction, and since each facial parameter corresponds to the image of the user's eyes, each Every two gaze directions of the user can be obtained by using a facial expression parameter. When the directions of the user's eyes are different, the images reflected in the user's eyes may be different positions of the eyeballs. Therefore, when the user gazes at different positions, two different gaze directions can be determined based on the different facial parameters corresponding to the different positions.

In this embodiment, in order to easily determine which position the user is looking at, after obtaining every two gaze directions, the electronic device can input each two gaze directions into the facial emotion analysis model for calculation. The calculation of each two gaze directions by the sentiment analysis model can predict the gaze focus formed by each two gaze directions on the interactive interface of the interactive activity. In this way, the electronic device can obtain M focus points of sight formed on the interactive interface of the interactive activity.

Therefore, the electronic device can continue to perform step S132 through the M line-of-sight focal points.

Step S132: Based on the M sight focus points, determine whether the interaction state of the user's next operation of the interactive activity is in the sight focus state.

In order to accurately determine the position of the user's gaze on the interactive interface, the electronic device can divide the interactive interface into at least two areas in advance, such as dividing the interactive interface into 20 areas, but not limited, so that the user can be measured by the area. Gaze location criteria.

Since the essence of determining the M line-of-sight focal points may be determining the coordinates of each line-of-sight focus on the interactive interface among the M line-of-sight focal points, the electronic device may determine each line-of-sight focus based on the coordinates of each line-of-sight focus on the interactive interface Which of the at least two regions is located. In this way, the electronic device can determine the number of line-of-sight focal points in the same area of the at least two areas.

In this embodiment, if the number of focus points in the same area is greater, it may indicate that the user gazes at an area for a long time, and it is more likely to indicate that the user is in a state of thinking focus. Therefore, a first preset number may be preset in the electronic device, and the first preset number may represent the lower limit of the user's line of sight focus state.

In this way, the electronic device can determine, based on the determined number of sight focal points in the same area of the at least two areas, whether the number of sight focal points located in the same area of the at least two areas on the interactive interface among the M sight focal points is greater than or equal to the first Preset quantity. For example, the first preset number may be set to 30-60.

If the number of line-of-sight focus points in the same area is not greater than or equal to the first preset number, it means that the user's line-of-sight focus within the first preset time period is not concentrated, so it can indicate that the user is interested in the interactive activities within the first preset time period. The interaction state of the next operation is not in the focus state, that is, it can be determined that the user knows the next appropriate operation of the interactive activity. In this way, the electronic device can terminate the execution of the subsequent procedures in the current execution of the image processing method, so as to wait for the next round of training to execute the image processing method.

It can be known that, after the electronic device is turned on to execute the image processing method, and the user chooses to turn off the operation prompt based on whether to turn on the option of the operation prompt or at the end of the interactive activity process, the electronic device can turn the For example, the electronic device establishes a process based on M facial images to execute the image processing method, and if the electronic device obtains M facial images, 5 facial images are newly obtained. When the facial images are updated, the electronic device may continue to execute the original process, and at the same time, establish a new process based on the M facial images updated with the 5 facial images to execute the image processing method.

If the number of line-of-sight focus in the same area is greater than or equal to the first preset number, it means that the user's line-of-sight focus within the first preset time period is concentrated, so it can be indicated that the user is interested in the interactive activity within the first preset time period. The interaction state of the one-step operation is in the focus state, that is, it can be determined that the user does not know the next appropriate operation of the interactive activity.

When it is determined that the user does not know the next appropriate operation, the electronic device may continue to perform step S140.

Step S140: Generate and output prompt information corresponding to the interaction state according to the interaction state.

In order to improve the user's experience so that the generated prompt information can be related to the user's unknown or uncertain next operation as much as possible, in this embodiment, the electronic device can generate the prompt information based on the object that the user pays attention to on the interactive interface.

Based on the electronic device determining the same area where the focus of sight is greater than the first preset number, that is, based on determining that the user's interactive state is in the focused state of sight, the electronic device may also analyze the image of the same area in the interactive interface, Then, the objects in the interactive activities contained in the same area are determined, wherein the objects may include: entities in the interactive activities or backgrounds in the interactive activities. Taking Chinese chess as an example, the entities in the interactive activity can be the pieces in Chinese chess, and the background in the interactive activity can be other areas in the interactive interface except the pieces; The entity of the poker can be the card face belonging to the user in the playing card, and the background in the interactive activity can be other areas in the interactive interface except the card face belonging to the user.

It can be understood that, in order to ensure the accuracy of determining whether the object is an entity or a background, the content contained in each area divided into at least two areas can either be an entity in the interactive activity, or vice versa. Background, it is not recommended that each area contain both the entity in the interactive activity and the background in the interactive activity.

Therefore, the electronic device determines the object in the interactive activity contained in the same area, and the electronic device can determine whether the object is the entity in the interactive activity or the background in the interactive activity.

If it is determined that the object is an entity in the interactive activity, it can be determined that the user's gaze may be, for example, a certain piece or a certain card.

As an optional method, the electronic device may generate the prompt information as follows: a trained interactive activity evaluation model is preset in the electronic device, and the interactive activity evaluation model may include an evaluation function, and the evaluation function includes the The parameters of each entity in the interactive activity can be determined by the situation of each entity in the interactive activity, for example, the evaluation function can determine the parameters of each piece based on the position of each piece in the current game situation. In this way, the interactive activity evaluation model calculates the evaluation function based on the parameters of each entity, so that prompt information that conforms to the current game situation can be determined.

Therefore, after the electronic device determines that the object is an entity in the interactive activity, the electronic device can increase the weight used to calculate the entity in the current evaluation function from the first value to the second value, and obtain the currently adjusted evaluation function , so that the entity can have a greater influence on the final result in the currently adjusted evaluation function. In this way, the interactive activity evaluation model performs calculation based on the currently adjusted evaluation function to generate prompt information for the next operation related to the entity.

After calculating the prompt information, when the electronic device is a server, the electronic device can output the prompt information to the user terminal used by the user, so that the user terminal can display the prompt information. In the case where the electronic device is a terminal device, the electronic device may display the prompt information in the form of animation or text. Accordingly, the user can receive prompt information corresponding to the operation that he is uncertain or does not know, so that the user can have a better user experience.

Taking Chinese chess as an example, if it is determined that the object is the "Pao" in Chinese chess, it means that the user wants to perform the next operation based on the "Pao" piece, but the user is not sure that in the current game situation, How to operate the chess piece "Cannon" is the most appropriate. Therefore, the electronic device can increase the weight of the entity "cannon" in the evaluation function to calculate the prompt information, so that the electronic device can calculate the prompt information related to the operation of the "cannon". Based on this, the prompt information displayed by the electronic device can be: three guns into five.

If it is determined that the object is the background in the interactive activity, it can be determined that the user's gaze may be, for example, a certain checkerboard.

In this way, the electronic device does not need to adjust the weight of each entity in the current evaluation function, so that the interactive evaluation model can perform calculations based on the current evaluation function, and then can generate a prompt for the next step related to the current game situation. information. Therefore, the electronic device can also output the prompt information to the user terminal or display the prompt information.

As an optional method, in order to avoid false prompts, after the electronic device finishes executing step S130 and before starting to execute step S140, the electronic device can determine whether the user has performed the next step based on analyzing the image of the interactive interface operation. If it is determined that the user has performed the next operation, the electronic device may terminate the execution of the subsequent process in the current process of executing the image processing method; otherwise, the electronic device continues to perform step S140.

Referring to FIG. 4 , as an optional implementation in this embodiment, after the electronic device performs step S130 and before the electronic device starts to perform step S140, the electronic device may further perform steps S101 and S102.

Step S101: After it is determined that the user is in the focus state, based on the N state parameters, determine the emotion type corresponding to each state parameter in the N state parameters, and there are N emotion types in total.

Step S102: Based on the N emotion types, determine whether the interaction state of the user is in a non-positive emotional state.

The flow of steps S101 and S102 will be described in detail below.

It can be understood that although it is determined that the user does not know the next appropriate operation of the interactive activity, in order to improve the user's experience, it is not necessary to generate prompt information immediately and push it to the user, but to continue to detect the user's emotions. The user can only be prompted when the emotion has been in a non-positive emotional state for a period of time.

Therefore, before step S101 is performed, in this embodiment, the electronic device can also correspondingly extract the facial image required for detecting emotion from the multi-frame facial images corresponding to the stored video stream, which is used for detecting the facial image required for emotion. The images may be N facial images obtained by the electronic device within a second preset time period before the current moment, where the second preset time period may be 10-30 seconds.

In this embodiment, as a way to obtain N facial images, the electronic device can extract the most recent 10-30 seconds of video from the stored video stream of 1-5 minutes, and extract the multiple frames contained in this video. face images as N face images.

As another way to obtain N facial images, at the level of each frame, since it is difficult for the facial expressions corresponding to every two adjacent frames of facial images to mutate, it can also be based on this principle to ensure the accuracy of the results. At the same time, the computational load of the electronic device is reduced, that is, the electronic device can extract a part of the frames from this video of 10-30 seconds as the obtained N facial images, for example, also from the continuous five frames of facial images or One frame of face image is extracted from every six consecutive frames of face images, but it is not limited.

The electronic device can also call the preset facial emotion analysis model, and can input each facial image of N facial images into the facial emotion analysis model, so that the facial emotion analysis model can analyze each facial emotion based on the deep neural network. The face images are cut out, and the background other than the user's face is removed from each face image, so that a face-only image can be obtained. In this way, the electronic device can obtain image parameters corresponding to each face-only image output by the facial emotion analysis model, and obtain N image parameters in total.

It can be understood that, in this embodiment, the function of only the face image can also be to determine the user's attitude, that is, to determine whether the user's attitude indicates whether the user's interaction state is still in a non-positive emotional state, therefore, only the face image is used. The image data of the image may also be the facial parameter of the user. In this way, when the electronic device obtains N images of the user's face only, it can obtain N facial parameters of the user.

After obtaining the N state parameters, the electronic device may execute step S101.

Step S101: After it is determined that the user is in the focus state, based on the N state parameters, determine the emotion type corresponding to each state parameter in the N state parameters, and there are N emotion types in total.

The electronic device can also process and analyze the N facial expression parameters based on the facial emotion analysis model, that is, the electronic device can also call the facial emotion analysis model, so that each facial expression parameter of the N facial emotion parameters can be input to the person. Face sentiment analysis model. Various emotion types are preset in the facial emotion analysis model, and the various emotion types may include, for example, happy, optimistic, neutral, anxious, and sad.

In this way, the facial emotion analysis model processes and analyzes each demeanor parameter based on multiple emotion types, and can determine the probability of each to-be-determined emotion type among the multiple to-be-determined emotion types represented by each demeanor parameter. In this way, according to the probability of each emotion type to be determined, the electronic device can obtain the emotion type to be determined with the highest probability from a plurality of emotion types to be determined based on each facial parameter, and the emotion type to be determined with the highest probability of each facial parameter is determined. The emotion type is the emotion type corresponding to each facial expression parameter. For example, for an expression parameter, the probability that the determined emotional type to be determined is happy is 0.05, the probability that the determined emotional type to be determined is optimistic is 0.05, the probability that the determined emotional type to be determined is neutral is 0.01, If the determined probability that the emotion type to be determined is anxiety is 0.7 and the determined probability that the emotion type to be determined is sad is 0.1, then the electronic device can determine that the emotion type corresponding to the demeanor parameter is anxiety.

Therefore, the electronic device can determine the emotion type corresponding to each of the N expression parameters, thereby obtaining N emotion types in total.

Step S102: Based on the N emotion types, determine whether the interaction state of the user is in a non-positive emotional state.

In this embodiment, the electronic device may classify various emotion types as positive emotions and non-positive emotions. For example, happy and optimistic may be classified as positive emotions, while neutral, anxious, and sad may be classified as non-positive emotions. positive emotions.

Then, in the N emotion types, if the number of non-positive emotions is greater, it can mean that the user has been in a non-positive emotional state for a longer time. Therefore, a second preset number may also be preset in the electronic device, and the second preset number may represent the lower limit value of the user's interaction state being a non-positive emotional state.

In this way, the electronic device can also determine, based on the determined N emotion types, whether the number of non-positive emotions in the N emotion types is greater than or equal to the second preset number. For example, the second preset number may be set to 70-90.

If the number of non-positive emotions in the N emotion types is less than the second preset number, it means that the user's interaction state is not in a non-positive emotional state within the second preset time period, that is, it can be determined that the current state of the user does not require any prompt. In this way, the electronic device can also terminate the execution of the subsequent process in the execution of the image processing method this time, so as to wait for the next polling execution of the image processing method.

If the number of non-positive emotions in the N emotion types is greater than or equal to the second preset number, it means that the user's interaction state is in a non-positive emotional state within the second preset time period, that is, it can be determined that the user's current state is relatively anxious , you may need to prompt the user for the next operation.

Then, after it is determined that the next operation needs to be prompted, and the electronic device also determines that the user has not performed the next operation based on the analysis of the image of the interactive interface, the electronic device may perform step S140 to prompt the user.

As some optional methods in this embodiment, the electronic device may also analyze the facial image. If it is determined that the facial image does not contain at least some of the facial features of the user, the facial features of the user include the facial features of the user. . The electronic device can generate and output an image capture angle adjustment prompt, so that the user can adjust his posture based on the image capture angle adjustment prompt, so that his face can be located within the capture range of the camera.

As shown in FIG. 5 , as another possible implementation manner of step S130 in this embodiment, step S130 may include: step S1301 , steps S1302 and S1303 .

Step S1301: Obtain M focus points of the user's sight on the interactive interface of the interactive activity based on the M attitude parameters, and determine, based on the N attitude parameters, corresponding to each of the N attitude parameters. There are N emotional types in total.

Step S1302: Based on the M line-of-sight focuses, determine whether the number of line-of-sight focuses located in the same area of at least two areas on the interactive interface is greater than or equal to a first preset number among the M line-of-sight focuses; and The N emotion types are selected, and it is determined whether the number of non-positive emotions in the N emotion types is greater than or equal to a second preset number.

Step S1303: when it is determined that the number of sight-focus points in the same area meets the first preset number, it is determined that the interaction state of the user's next operation of the interactive activity is in the sight-focus state; When the number of non-positive emotions in each emotion type meets the second preset number, it is determined that the interaction state of the user is in a non-positive emotional state.

That is, the electronic device can take both being in a focused state of sight and being in a non-positive emotional state as conditions for judging whether the user needs to be prompted, then in either state, the electronic device can determine that the user needs to be prompted.

It can be understood that the detailed implementation process of step S1301 , step S1302 and S1303 may refer to the aforementioned implementation manner, and will not be repeated here.

Third Embodiment

Referring to FIG. 6, an embodiment of the present application provides an image processing apparatus 100. The image processing apparatus 100 can be applied to electronic equipment. The image processing apparatus 100 includes:

The image obtaining module 110 is configured to obtain the facial image of the user during the user's participation in the interactive activity.

The facial expression obtaining module 120 is configured to obtain facial expression parameters of the user based on the facial image.

The operation judging module 130 is used for judging the interactive state of the next operation of the interactive activity by the user based on the facial expression parameter.

The prompt output module 140 is configured to generate and output prompt information corresponding to the interaction state according to the interaction state.

It should be noted that, as those skilled in the art can clearly understand, for the convenience and brevity of the description, for the specific working process of the above-described systems, devices and units, reference may be made to the corresponding processes in the foregoing method embodiments. No longer.

Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

To sum up, the embodiments of the present application provide an image processing method, an apparatus, an electronic device, and a storage medium. The method includes: in the process of the user participating in the interactive activity, obtaining the facial image of the user; obtaining the facial expression parameter of the user based on the facial image; judging the interactive state of the user's next operation of the interactive activity based on the facial expression parameter; generating and outputting according to the interactive state The prompt information corresponding to the interactive state.

Since the user's interaction state for the next step of the interactive activity can be determined based on the user's attitude parameters, it is possible to generate and output a prompt for the user when it is determined according to the user's interaction state that the user's decision at this time needs to be improved. Help users to quickly improve the level of decision-making in interactive activities.

The above are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application. It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, and should cover within the scope of protection of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (13)

1. an image processing method, is characterized in that, described method comprises: Obtain the user's face image during the user's participation in interactive activities; Based on the facial image, obtain the facial expression parameters of the user, where the facial expression parameters are image data corresponding to the images of the user's eyes; Based on the demeanor parameter, determine the interactive state of the user's next operation of the interactive activity; Generate and output prompt information corresponding to the interaction state according to the interaction state; wherein, the facial image includes M facial images obtained within a first preset time period before the current moment, and the demeanor parameter includes the M facial images M attitude parameters corresponding to a face image, where M is an integer greater than 1, and based on the attitude parameters, determine the interactive state of the user's next operation of the interactive activity, including: Obtaining M focus points of the user's sight on the interactive interface of the interactive activity based on the M attitude parameters; Based on the M focus points, it is determined whether the interaction state of the user's next operation of the interactive activity is in the focus focus state. 2. The image processing method according to claim 1, wherein, based on the M focus points, judging whether the interactive state of the user's next operation of the interactive activity is in a focus focus state, comprising: judging whether the number of sight focal points located in the same area of at least two areas on the interactive interface among the M sight focal points is greater than or equal to a first preset number; If so, it indicates that the user's interaction state of the next operation of the interaction activity is in the focus state. 3. The image processing method according to claim 2, wherein obtaining M focus points of the user's sight on the interactive interface of the interactive activity based on the M facial parameters, comprising: Determine every two gaze directions of the user corresponding to each of the M attitude parameters; The sight focus formed on the interactive interface of the interactive activity is determined for every two sight directions, and the M sight focus points are determined in total. 4. The image processing method according to claim 1, wherein the facial image further comprises N facial images obtained within a second preset time period before the current moment, where N is an integer greater than 1, and the facial expression The parameters include N state parameters corresponding to the N facial images, after judging the interaction state of the user's next operation of the interactive activity based on the state parameters, and after generating and outputting according to the interaction state Before the prompt information corresponding to the interaction state, the method further includes: After it is determined that the user is in the focused state of sight, based on the N attitude parameters, determine the emotional type corresponding to each of the N attitude parameters, and there are N emotional types in total; Based on the N emotion types, determine whether the interaction state of the user is in a non-positive emotional state; If yes, perform the step of generating and outputting prompt information corresponding to the interaction state according to the interaction state. 5. The image processing method according to claim 4, wherein, based on the N emotion types, judging whether the interaction state of the user is in a non-positive emotional state, comprising: Determine whether the number of non-positive emotions in the N emotion types is greater than or equal to a second preset number, wherein the number of non-positive emotions greater than or equal to the second preset number indicates the interaction of the user The state is in a non-positive emotional state. 6. The image processing method according to claim 4, wherein, based on the N state parameters, the emotion type corresponding to each state parameter in the N state parameters is determined, and there are N emotion types in total, include: The facial emotion analysis model is used to analyze each demeanor parameter of the N deity parameters, and each demeanor parameter output by the facial emotion analysis model is obtained as each emotion type to be determined among the various emotion types to be determined. The probability; According to the probability of each emotion type to be determined, the emotion type to be determined with the highest probability among the multiple emotion types to be determined is determined, wherein, the emotion type to be determined with the highest probability of each state parameter is the emotion type corresponding to each state parameter. Emotion type. 7. The image processing method according to claim 2, wherein generating and outputting prompt information corresponding to the interaction state according to the interaction state, comprising: determining the object in the interactive activity contained in the same area according to the user's interactive state in the line-of-sight focused state; Generate prompt information related to the next operation according to the object, and output the prompt information. 8. The image processing method according to claim 7, wherein generating prompt information related to the next operation according to the object, and outputting the prompt information, comprising: According to the object, determine whether the object is an entity in the interactive activity or a background in the interactive activity; If the object is an entity in the interactive activity, increase the weight used to calculate the entity in the current evaluation function from a first value to a second value to obtain a currently adjusted evaluation function, based on the current adjustment The subsequent evaluation function generates prompt information of the next operation related to the entity; if the object is the background in the interactive activity, the prompt information of the next operation is generated based on the current evaluation function. 9. The image processing method according to claim 1, wherein the facial images comprise M facial images obtained within a first preset time period before the current moment and M face images obtained within a second preset time period before the current moment. The obtained N facial images, the facial parameters include: M facial parameters corresponding to the M facial images and N facial parameters corresponding to the N facial images, M and N are integers greater than 1, based on the The expression parameters are used to determine the interaction state of the user's next operation of the interactive activity, including: Obtaining M focus points of the user on the interactive interface of the interactive activity based on the M attitude parameters, and determining the emotion type corresponding to each of the N attitude parameters based on the N attitude parameters , a total of N emotion types; Based on the M line-of-sight focal points, determine whether the number of line-of-sight focal points located in the same area of at least two areas on the interactive interface is greater than or equal to a first preset number among the M line-of-sight focal points; and Emotion type, determine whether the number of non-positive emotions in the N emotion types is greater than or equal to a second preset number; When it is determined that the number of sight-focus points in the same area satisfies the first preset number, it is determined that the interaction state of the user's next operation of the interactive activity is in the sight-focus state; when determining the N emotion types When the number of non-positive emotions meets the second preset number, it is determined that the interaction state of the user is in a non-positive emotion state. 10. The image processing method according to any one of claims 1-9, wherein the method further comprises: When it is determined that the facial image does not contain at least part of the facial features of the user, an image acquisition angle adjustment prompt is generated and output, wherein the facial features of the user include the facial features of the user. 11. An image processing device, wherein the device comprises: The image acquisition module is used to acquire the user's facial image during the user's participation in the interactive activity; an attitude obtaining module, configured to obtain an attitude parameter of the user based on the facial image, where the attitude parameter is the image data corresponding to the image of the user's eyes; an operation judging module, used for judging the interactive state of the next operation of the interactive activity by the user based on the expression parameter; a prompt output module, configured to generate and output prompt information corresponding to the interactive state according to the interactive state; Wherein, the facial images include M facial images obtained within a first preset time period before the current moment, and the expression parameters include M expression parameters corresponding to the M facial images, where M is an integer greater than 1, so The operation judgment module is used to obtain M focus points of the user's sight on the interactive interface of the interactive activity based on the M attitude parameters; Based on the M focus points, it is determined whether the interaction state of the user's next operation of the interactive activity is in the focus focus state. 12. An electronic device, characterized in that the electronic device comprises: a processor, a memory, a bus and a communication interface; the processor, the communication interface and the memory are connected through the bus; the memory for storing programs; The processor is configured to execute the image processing method according to any one of claims 1-10 by calling a program stored in the memory. 13. A computer-readable storage medium having a computer-executable non-volatile program code, wherein the program code enables the computer to execute the image processing method according to any one of claims 1-10 .

Images