CN113176827B - AR interaction method and system based on expressions, electronic device and storage medium - Google Patents

Abstract

The present invention proposes an AR interaction method, system, electronic device and storage medium based on expressions. The technical solution of the method includes collecting real-time image data of a real object and a first character in a real scene, and simultaneously collecting the voice data of the first character and the real-time image data of the real scene. Real-time image data of the environment; according to the real-time image data of the physical object, the first person and the environment, generate an AR screen including virtual imaging of the physical object, the first person and the environment, and superimpose expressions on the virtual imaging of the physical object in the AR screen elements, and at the same time generate an agent image in the AR screen, and the AR screen is displayed on the screen; the agent image is based on the virtual imaging of the physical object superimposed with expression elements, and is based on the voice data of the first character and the first character in the real scene. To interact; the interaction includes the agent avatar having a dialogue with the first character according to the preset corpus. The present invention solves the problem that the existing VR-based or AR-based interactive systems fail to take into account the group of autistic children.

Description

Expression-based AR interaction method, system, electronic device and storage medium

technical field

The invention belongs to the technical field of augmented reality, and in particular relates to an expression-based AR interaction method, system, electronic equipment and storage medium.

Background technique

In the emotional intervention of autistic children, the existing technology will be carried out through virtual reality (VR), and the existing virtual reality technology products need to wear virtual helmets or wearable devices, the interaction space is not only limited, and children are prone to misuse , especially for autistic children, who don't like wearing devices on their bodies. However, in the existing augmented reality (AR) technology, augmented reality helmets or handheld digital devices are required to achieve stereoscopic imaging effects, resulting in high equipment costs, low efficiency and poor recognition accuracy due to the inability to free hands; It meets the technical development needs of autistic children as a user group. The operation is complicated and the operation process is cumbersome. It is not suitable for children, especially autistic children.

SUMMARY OF THE INVENTION

The embodiment of the present application provides an expression-based AR interaction method, system, electronic device, and storage medium to at least solve the problem that existing VR-based or AR-based interactive systems fail to take autistic children into consideration.

In the first aspect, the embodiment of the present application provides an expression-based AR interaction method, including: 1. An expression-based AR interaction method, characterized in that it includes: a real data collection step, collecting a physical object and a first person in a real scene Real-time image data of the real-time image data of the first person, and simultaneously collect the sound data of the first person and the real-time image data of the environment of the real scene; the AR picture generation step, according to the real-time images of the entity, the first person and the environment Image data, generating an AR picture including the virtual imaging of the entity, the first character and the environment, superimposing an expression element on the virtual imaging of the entity in the AR picture, and simultaneously An agent image is generated in the AR image, and the AR image is displayed on a screen; in the AR intelligent interaction step, the agent image is based on the virtual imaging of the physical object superimposed with the expression elements, and according to the The voice data of the first character and the real-time image data interact with the first character in the real scene; the interaction includes the agent image having a dialogue with the first character according to a preset corpus.

Preferably, the method further includes: an operation intervention step, if the preset corpus cannot support the dialogue between the agent image and the first character, intervene in the interaction through a second character.

Preferably, the real data collection step further includes: an expression training step, training an expression recognition and classification model according to a human facial expression data set through a CNN neural network; The real-time image data is recognized, the facial expression data of the first person is extracted, and input into the expression recognition classification model for classification.

Preferably, an identification image is covered on the surface of the entity, and the identification image includes a two-dimensional figure with pattern and color, and is used for collecting the real-time image data of the entity.

In the second aspect, the embodiment of the present application provides an expression-based AR interaction system, which is applicable to the above-mentioned expression-based AR interaction method, including: a real data collection module, which collects real-time image data of an entity object and a first person in a real scene, And simultaneously collect the voice data of the first person and the real-time image data of the environment of the real scene; the AR picture generation module generates a real-time image data according to the real object, the first person and the environment An AR picture including virtual imaging of the physical object, the first character, and the environment, superimposing an expression element on the virtual imaging of the physical object in the AR picture, and simultaneously displaying an expression element in the AR picture Generate an intelligent body image, the AR picture is displayed on a screen; the AR intelligent interaction module, the intelligent body image is based on the virtual imaging of the physical object superimposed with the expression elements, and according to the voice of the first person data and the real-time image data to interact with the first character in the real scene; the interaction includes the agent image having a dialogue with the first character according to a preset corpus.

In some of these embodiments, the system further includes: an operation intervention module, if the preset corpus cannot support the dialogue between the agent image and the first character, then a second character will interact with the Intervene.

In some of these embodiments, the real data acquisition module further includes: an expression training unit, which trains an expression recognition and classification model according to a facial expression data set through a CNN neural network; The real-time image data of the first person is recognized, the facial expression data of the first person is extracted, and input into the expression recognition classification model for classification.

In some of the embodiments, an identification image is covered on the surface of the entity, and the identification image includes a two-dimensional figure with pattern and color, and is used for collecting the real-time image data of the entity.

In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the computer program, Realize the expression-based AR interaction method as described in the first aspect above.

In a fourth aspect, the embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the expression-based AR interaction method as described in the first aspect above is implemented.

Compared with related technologies, the embodiments of the present application collect characters and environments in real scenes and directly display them on the screen. In addition, design and collect physical objects for interaction, and superimpose expression elements when generating AR images. Based on the above Elements can create a suitable environment experience for children with autism to interact with the AR system, and this embodiment of the application can collect and classify and recognize the expressions of characters. By designing interactive games, children with autism can interact with AR based on expressions.

Description of drawings

The drawings described here are used to provide a further understanding of the application and constitute a part of the application. The schematic embodiments and descriptions of the application are used to explain the application and do not constitute an improper limitation to the application. In the attached image:

Fig. 1 is the flow chart of the AR interaction method based on expression of the present invention;

Fig. 2 is the step-by-step flowchart of step S1 in Fig. 1;

Fig. 3 is the frame diagram of the AR interactive system based on expression of the present invention;

Fig. 4 is the frame diagram of electronic equipment of the present invention;

Fig. 5 is an effect diagram of the physical object of the embodiment of the present application;

FIG. 6 is an effect diagram of an agent image in an embodiment of the present application;

FIG. 7 is an interactive effect diagram of the embodiment of the present application;

FIG. 8 is another interactive effect diagram of the embodiment of the present application;

In the picture above:

1. Realistic data acquisition module; 2. AR screen generation module; 3. AR intelligent interaction module; 4. Operation intervention intervention module; 11. Expression training unit; 12. Expression classification unit; 60. Bus; 61. Processor; 62 . Memory; 63. Communication interface.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application, and those skilled in the art can also apply the present application to other similar scenarios. In addition, it can also be understood that although such development efforts may be complex and lengthy, for those of ordinary skill in the art relevant to the content disclosed in this application, the technology disclosed in this application Some design, manufacturing or production changes based on the content are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

Reference in this application to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood explicitly and implicitly by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

Unless otherwise defined, the technical terms or scientific terms involved in the application shall have the usual meanings understood by those with ordinary skill in the technical field to which the application belongs. Words such as "a", "an", "an" and "the" involved in this application do not indicate a limitation on quantity, and may indicate singular or plural numbers. The terms "comprising", "comprising", "having" and any variations thereof involved in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or process that includes a series of steps or modules (units). The apparatus is not limited to the listed steps or units, but may further include steps or units not listed, or may further include other steps or units inherent to the process, method, product or apparatus.

Below, describe the embodiment of the present invention in detail in conjunction with accompanying drawing:

Fig. 1 is the flow chart of the AR interaction method based on facial expressions of the present invention, see Fig. 1, the AR interactive method based on facial expressions of the present invention comprises the following steps:

S1: Collect real-time image data of a physical object and a first person in a real scene, and simultaneously collect voice data of the first person and real-time image data of the environment of the real scene. Optionally, the surface of the solid object is covered with an identification image, the identification image includes a two-dimensional figure with patterns and colors, and is used for collecting the real-time image data of the entity object.

In a specific implementation, a physical object is designed as an external physical object of the augmented reality system, and the physical object conforms to the size and scope of the first person's hand. Optionally, building blocks can be used as the physical object. In a specific implementation, two-dimensional graphics that can be recognized by a computer vision system are pasted on the physical object, and the complexity of patterns and colors of the two-dimensional graphics is designed to improve the accuracy of computer recognition technology.

In a specific implementation, the real-time image of the physical object is collected through the camera, the real-time image of the first person is collected through the camera, and the voice data of the first person is collected at the same time. Optionally, the real-time image of the first person includes the face of the first person expression. In addition, the camera also collects real-time image data of the current real environment.

Optionally, Fig. 2 is a step-by-step flowchart of step S1 in Fig. 1, please refer to Fig. 2:

S11: Using a CNN neural network, train an expression recognition classification model according to a facial expression data set;

S12: Recognize the collected real-time image data of the first person through the OpenCV interface, extract the facial expression data of the first person, and input it into the expression recognition classification model for classification.

In the specific implementation, use the Fer2013 facial expression data set, use the CNN neural network for training, collect the real-time image data of the first person, call the OpenCV interface to recognize the facial expression, and transfer the facial expression to the trained model Classify facial expressions.

Please continue to see Figure 1:

S2: According to the real-time image data of the physical object, the first person and the environment, generate an AR picture including virtual imaging of the physical object, the first person and the environment, in the AR An expression element is superimposed on the virtual imaging of the physical object in the screen, and an agent image is generated in the AR screen at the same time, and the AR screen is displayed through a screen.

In the specific implementation, the real-time image data collected in step S1 is mirrored and displayed on the display screen, and the physical object is identified, and an expression element is superimposed on each physical object through augmented reality technology, that is, an expression element is superimposed on the physical object. Emoticon, the emoticon moves with the movement of the entity. Fig. 5 is the effect diagram of the physical object of the embodiment of the present application, please refer to Fig. 5, the building blocks in the real scene are used as the physical object, and the two-dimensional graphics including patterns and colors are pasted on the physical object, through the augmented reality technology An expression element is superimposed on the building block for interaction with the first character.

In addition, use three-dimensional design software to design a virtual intelligent body image, and use augmented reality technology to superimpose on the screen. Optionally, the image of the virtual agent is a character image. FIG. 6 is an effect diagram of a physical object in the embodiment of the present application. Referring to FIG. 6 , a virtual agent image of a human figure is designed and three-dimensionally processed.

In the specific implementation, the action and voice of the agent image are controlled by the game engine Unity.

In a specific implementation, a wake-up password is preset, and the first character activates the agent image by speaking the wake-up password.

S3: The agent image is based on the virtual imaging of the physical object superimposed with the expression elements, and according to the voice data of the first character and the real-time image data and the first character in the real scene The characters interact; the interaction includes the agent image having a dialogue with the first character according to a preset corpus.

In the specific implementation, the Vuforia AR engine interface is used to locate the position of the target entity, so that the first person can control the virtual expression on the screen by manipulating the entity.

In the specific implementation, Fig. 7 is an interactive effect diagram of the embodiment of the present application, please refer to Fig. 7, the mirror image of the real scene is displayed in the screen, and a virtual agent is generated by augmented reality technology, and the real-time images of the building blocks in the real scene are collected. Image data, and display it on the screen after superimposing expression elements through augmented reality technology, as shown in Figure 7, the embodiment of the present application provides the first interaction rule: the first character completes the memory game by talking with the virtual agent , the augmented reality engine scans the building blocks as physical objects and generates virtual expressions. In the screen, the virtual expression will automatically change the angle and position, allowing the first person to guess and find out the specified virtual expression. Optionally, the first character can find out the building block corresponding to the virtual expression in reality through voice or action.

In the specific implementation, Fig. 8 is another interactive effect diagram of the embodiment of the present application, please refer to Fig. 8, the mirror image of the real scene is displayed in the screen, and a virtual agent is generated by augmented reality technology, and the image of the building blocks in the real scene is collected Real-time image data, and display it on the screen after superimposing expression elements through augmented reality technology. In addition, a virtual whiteboard is generated in the generated augmented reality screen, and the virtual whiteboard presents a two-dimensional cartoon, as shown in Figure 8. The embodiment of the application provides the second interaction rule: a virtual whiteboard is generated in the generated augmented reality screen, the virtual whiteboard presents a two-dimensional cartoon, the virtual agent asks questions to the first character, and the first character needs to answer the character in the cartoon What expression should be made at this time. In a specific implementation, optionally, the first character inputs information by lifting a building block, and the virtual agent judges whether it is correct and gives a prompt. In a specific implementation, a social scene is preset as the content of the cartoon.

In specific implementation, the embodiment of the present application provides a third interaction rule: directly display the image of the first character in the augmented reality screen, that is, the appearance of the first character will appear on the screen, and the first character should respond to the requirements of the virtual agent Make a specified expression, collect the facial expression of the first person, input it into the expression classification model for classification and detection, and optionally calculate the duration of the expression, and display the completed effect in a visual way.

In a specific implementation, the dialogue between the virtual agent and the first person is supported by an artificial intelligence corpus, and the corresponding reply sentence is retrieved according to the keywords of the first person's conversation.

S4: If the preset corpus cannot support the dialogue between the agent image and the first character, intervene in the interaction through a second character.

In the specific implementation, another operating terminal is designed to be operated by a second person other than the first person. If the agent image cannot complete the dialogue with the first person based on the existing corpus, the second person will intervene. Manipulate the intelligent body to conduct a conversation; optionally, the second person can also control the progress of the interaction and other emergencies that cannot be dealt with by any preset rules.

In a specific implementation, the second character can control the dialogue of the virtual agent, control the progress of the interaction, and trigger subsequent interaction events through the UDP network protocol.

Fig. 3 is a frame diagram of the expression-based AR interactive system according to the present invention, please refer to Fig. 3, including:

Reality data collection module 1: collect real-time image data of a physical object and a first person in a real scene, and simultaneously collect voice data of the first person and real-time image data of the environment of the real scene. Optionally, the surface of the solid object is covered with an identification image, the identification image includes a two-dimensional figure with patterns and colors, and is used for collecting the real-time image data of the entity object.

In a specific implementation, a physical object is designed as an external physical object of the augmented reality system, and the physical object conforms to the size and scope of the first person's hand. Optionally, building blocks can be used as the physical object. In a specific implementation, two-dimensional graphics that can be recognized by a computer vision system are pasted on the physical object, and the complexity of patterns and colors of the two-dimensional graphics is designed to improve the accuracy of computer recognition technology.

In a specific implementation, the real-time image of the physical object is collected through the camera, the real-time image of the first person is collected through the camera, and the voice data of the first person is collected at the same time. Optionally, the real-time image of the first person includes the face of the first person expression. In addition, the camera also collects real-time image data of the current real environment.

Optionally, the real data acquisition module 1 also includes:

Expression training unit 11: training an expression recognition and classification model according to a facial expression data set through the CNN neural network;

Expression classification unit 12: recognize the collected real-time image data of the first person through the OpenCV interface, extract the facial expression data of the first person, and input it into the expression recognition classification model for classification.

In the specific implementation, use the Fer2013 facial expression data set, use the CNN neural network for training, collect the real-time image data of the first person, call the OpenCV interface to recognize the facial expression, and transfer the facial expression to the trained model Classify facial expressions.

AR picture generating module 2: according to the real-time image data of the physical object, the first person and the environment, generate an AR picture including virtual imaging of the physical object, the first person and the environment, An emoticon element is superimposed on the virtual imaging of the physical object in the AR picture, and an agent image is generated in the AR picture at the same time, and the AR picture is displayed through a screen.

In the specific implementation, the real-time image data collected by the real data acquisition module 1 is mirrored and displayed on the display screen, and the physical objects are identified, and an expression element is superimposed on each physical object through augmented reality technology, that is, the physical object An emoticon is superimposed on it, and the emoticon moves with the movement of the physical object. Fig. 5 is the effect diagram of the physical object of the embodiment of the present application, please refer to Fig. 5, the building blocks in the real scene are used as the physical object, and the two-dimensional graphics including patterns and colors are pasted on the physical object, through the augmented reality technology An expression element is superimposed on the building block for interaction with the first character.

In addition, use three-dimensional design software to design a virtual intelligent body image, and use augmented reality technology to superimpose on the screen. Optionally, the image of the virtual agent is a character image. FIG. 6 is an effect diagram of a physical object in the embodiment of the present application. Referring to FIG. 6 , a virtual agent image of a human figure is designed and three-dimensionally processed.

In the specific implementation, the action and voice of the agent image are controlled by the game engine Unity.

In a specific implementation, a wake-up password is preset, and the first character activates the agent image by speaking the wake-up password.

AR intelligent interaction module 3: the image of the intelligent body is based on the virtual imaging of the physical object superimposed with the expression elements, and according to the voice data of the first person and the real-time image data and the real scene The first character interacts; the interaction includes the agent image having a dialogue with the first character according to a preset corpus.

In the specific implementation, the Vuforia AR engine interface is used to locate the position of the target entity, so that the first person can control the virtual expression on the screen by manipulating the entity.

In the specific implementation, Fig. 7 is an interactive effect diagram of the embodiment of the present application, please refer to Fig. 7, the mirror image of the real scene is displayed in the screen, and a virtual agent is generated by augmented reality technology, and the real-time images of the building blocks in the real scene are collected. Image data, and display it on the screen after superimposing expression elements through augmented reality technology, as shown in Figure 7, the embodiment of the present application provides the first interaction rule: the first character completes the memory game by talking with the virtual agent , the augmented reality engine scans the building blocks as physical objects and generates virtual expressions. In the screen, the virtual expression will automatically change the angle and position, allowing the first person to guess and find out the specified virtual expression. Optionally, the first character can find out the building block corresponding to the virtual expression in reality through voice or action.

In the specific implementation, Fig. 8 is another interactive effect diagram of the embodiment of the present application, please refer to Fig. 8, the mirror image of the real scene is displayed in the screen, and a virtual agent is generated by augmented reality technology, and the image of the building blocks in the real scene is collected Real-time image data, and display it on the screen after superimposing expression elements through augmented reality technology. In addition, a virtual whiteboard is generated in the generated augmented reality screen, and the virtual whiteboard presents a two-dimensional cartoon, as shown in Figure 8. The embodiment of the application provides the second interaction rule: a virtual whiteboard is generated in the generated augmented reality screen, the virtual whiteboard presents a two-dimensional cartoon, the virtual agent asks questions to the first character, and the first character needs to answer the character in the cartoon What expression should be made at this time. In a specific implementation, optionally, the first character inputs information by lifting a building block, and the virtual agent judges whether it is correct and gives a prompt. In a specific implementation, a social scene is preset as the content of the cartoon.

In specific implementation, the embodiment of the present application provides a third interaction rule: directly display the image of the first character in the augmented reality screen, that is, the appearance of the first character will appear on the screen, and the first character should respond to the requirements of the virtual agent Make a specified expression, collect the facial expression of the first person, input it into the expression classification model for classification and detection, and optionally calculate the duration of the expression, and display the completed effect in a visual way.

In a specific implementation, the dialogue between the virtual agent and the first person is supported by an artificial intelligence corpus, and the corresponding reply sentence is retrieved according to the keywords of the first person's conversation.

Operational intervention module 4: If the preset corpus cannot support the dialogue between the agent image and the first character, intervene in the interaction through a second character.

In the specific implementation, another operating terminal is designed to be operated by a second person other than the first person. If the agent image cannot complete the dialogue with the first person based on the existing corpus, the second person will intervene. Manipulate the intelligent body to conduct a conversation; optionally, the second person can also control the progress of the interaction and other emergencies that cannot be dealt with by any preset rules.

In a specific implementation, the second character can control the dialogue of the virtual agent, control the progress of the interaction, and trigger subsequent interaction events through the UDP network protocol.

In addition, the expression-based AR interaction method described in conjunction with FIG. 1 and FIG. 2 can be implemented by electronic devices. FIG. 4 is a block diagram of the electronic device of the present invention.

The electronic device may include a processor 61 and a memory 62 storing computer program instructions.

Specifically, the processor 61 may include a central processing unit (CPU), or an Application Specific Integrated Circuit (ASIC for short), or may be configured to implement one or more integrated circuits in the embodiments of the present application.

Wherein, the memory 62 may include a large-capacity memory for data or instructions. For example and not limitation, the memory 62 may include a hard disk drive (Hard Disk Drive, referred to as HDD), a floppy disk drive, a solid state drive (SolidState Drive, referred to as SSD), flash memory, optical disk, magneto-optical disk, tape or universal serial bus (Universal Serial Bus, referred to as USB) drive or a combination of two or more of the above. Memory 62 may comprise removable or non-removable (or fixed) media, where appropriate. Memory 62 may be internal or external to the data processing arrangement, where appropriate. In a particular embodiment, memory 62 is a non-volatile (Non-Volatile) memory. In a specific embodiment, the memory 62 includes a read-only memory (Read-Only Memory, ROM for short) and a random access memory (Random Access Memory, RAM for short). In appropriate cases, the ROM can be mask programmed ROM, programmable ROM (Programmable Read-Only Memory, referred to as PROM), erasable PROM (Erasable Programmable Read-Only Memory, referred to as EPROM), electrically erasable In addition to PROM (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), electrically rewritable ROM (Electrically Alterable Read-Only Memory, referred to as EAROM) or flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM may be a static random-access memory (Static Random-Access Memory, referred to as SRAM) or a dynamic random-access memory (Dynamic Random-Access Memory, referred to as DRAM), wherein the DRAM may be a fast page Mode Dynamic Random Access Memory (Fast Page Mode Dynamic Random Access Memory, referred to as FPMDRAM), Extended Data Output Dynamic Random Access Memory (Extended Date Out Dynamic Random Access Memory, referred to as EDODRAM), Synchronous Dynamic Random Access Memory (Synchronous Dynamic Random Access Memory) Random-Access Memory, referred to as SDRAM) and so on.

The memory 62 can be used to store or cache various data files required for processing and/or communication, as well as possible computer program instructions executed by the processor 61 .

The processor 61 reads and executes the computer program instructions stored in the memory 62 to implement any expression-based AR interaction method in the above embodiments.

In some of these embodiments, the electronic device may further include a communication interface 63 and a bus 60 . Wherein, as shown in FIG. 4 , the processor 61 , the memory 62 , and the communication interface 63 are connected through the bus 60 and complete mutual communication.

The communication port 63 can realize data communication with other components such as external devices, image/data acquisition equipment, database, external storage, and image/data processing workstations.

Bus 60 includes hardware, software, or both, and couples the components of the electronic device to each other. The bus 60 includes but not limited to at least one of the following: a data bus (Data Bus), an address bus (Address Bus), a control bus (Control Bus), an expansion bus (Expansion Bus), and a local bus (Local Bus). For example without limitation, the bus 60 may include an Accelerated Graphics Port (AGP for short) or other graphics bus, an Enhanced Industry Standard Architecture (EISA for short) bus, a Front Side Bus (FrontSide Bus, Referred to as FSB), Hyper Transport (Hyper Transport, referred to as HT) interconnection, Industry Standard Architecture (Industry Standard Architecture, referred to as ISA) bus, wireless bandwidth (InfiniBand) interconnection, low pin count (Low Pin Count, referred to as LPC) bus, memory bus, Micro Channel Architecture (Micro Channel Architecture, referred to as MCA) bus, peripheral component interconnect (Peripheral Component Interconnect, referred to as PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology A Serial Advanced Technology Attachment (SATA for short) bus, a Video Electronics Standards Association Local Bus (VLB for short) bus or other suitable buses or a combination of two or more of these. Bus 60 may comprise one or more buses, where appropriate. Although the embodiments of this application describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

The electronic device can execute the expression-based AR interaction method in the embodiment of the present application.

In addition, in combination with the expression-based AR interaction method in the foregoing embodiments, the embodiments of the present application may provide a computer-readable storage medium for implementation. Computer program instructions are stored on the computer-readable storage medium; when the computer program instructions are executed by a processor, any expression-based AR interaction method in the foregoing embodiments may be implemented.

The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ReadOnly Memory, abbreviated as ROM), random access memory (Random Access Memory, abbreviated as RAM), magnetic disk or optical disc, etc., which can store program codes. medium.

The technical features of the above-mentioned embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, should be considered as within the scope of this specification.

The above-mentioned embodiments only represent several implementation modes of the present application, and the description thereof is relatively specific and detailed, but it should not be construed as limiting the scope of the patent for the invention. It should be noted that, for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent application should be based on the appended claims.

Claims (10)

1. The AR interaction method based on the expression is characterized by comprising the following steps:

a real data acquisition step, which is to acquire real object and real-time image data of a first person in a real scene and simultaneously acquire sound data of the first person and real-time image data of the environment of the real scene; wherein the real-time image data of the first person includes facial expression data of the first person;

an AR picture generation step of generating an AR picture displayed on a screen according to the real-time image data of the object, the first person, and the environment, the AR picture including: superimposing an expression element on the virtual image of the solid object in the AR picture according to the identification image of the solid object, designing an intelligent object image of the human object by using three-dimensional design software, and simultaneously superimposing the intelligent object image in the AR picture by using an augmented reality technology, wherein the screen comprises a computer screen;

an AR intelligent interaction step, wherein the intelligent agent image asks a question for the first person in the AR picture, the first person answers by finding out an entity corresponding to the corresponding virtual expression in reality, and the intelligent agent image judges whether the first person is correct or not and gives a prompt; or; the first person makes a designated expression according to the requirement of the intelligent agent image, expression duration is calculated according to the collected real-time image data of the first person, and the real-time image of the first person is visually displayed in an AR picture.

2. The expression-based AR interaction method of claim 1, wherein the method further comprises:

and an intervention step, namely, if the preset corpus can not support the conversation between the intelligent image and the first person, intervening the interaction through a second person.

3. The expression-based AR interaction method of claim 1, wherein the step of collecting reality data further comprises:

an expression training step, namely training an expression recognition classification model according to a face expression data set through a CNN neural network;

and an expression classification step, namely identifying the acquired real-time image data of the first person through an OpenCV interface, extracting facial expression data of the first person, and inputting the facial expression data into the expression identification classification model for classification.

4. The AR interaction method based on the expression as claimed in claim 1, wherein a recognition image is covered on the surface of the object, the recognition image includes a two-dimensional graph with patterns and colors, and is used for collecting the real-time image data of the object.

5. AR interactive system based on expression, characterized by comprising:

the real data acquisition module is used for acquiring real-time image data of an entity and a first person in a real scene and simultaneously acquiring sound data of the first person and real-time image data of the environment of the real scene; wherein the real-time image data of the first person includes facial expression data of the first person;

the AR picture generation module generates an AR picture displayed through a screen according to the real object, the first person and the real-time image data of the environment, and the AR picture comprises: superimposing an expression element on the virtual image of the entity, the real-time image of the environment and the real-time image of the first person according to the identification image of the entity in the AR picture, designing an intelligent body image of the person image by using three-dimensional design software, and simultaneously superimposing the intelligent body image in the AR picture by using an augmented reality technology, wherein the screen comprises a computer screen;

the AR intelligent interaction module is used for asking a question for the first person in the AR picture by the intelligent body image, the first person answers the question by finding out an entity corresponding to the corresponding virtual expression in reality, and the intelligent body image judges whether the question is correct or not and gives a prompt; or; the first person makes a designated expression according to the requirement of the intelligent agent image, expression duration is calculated according to the collected real-time image data of the first person, and the real-time image of the first person is visually displayed in an AR picture.

6. The expression-based AR interaction system of claim 5, wherein the system further comprises:

and operating an intervention module, and if the preset corpus can not support the dialog between the intelligent body image and the first person, intervening the interaction through a second person.

7. The expression-based AR interaction system of claim 5, wherein the reality data collection module further comprises:

the expression training unit is used for training an expression recognition classification model according to a face expression data set through a CNN neural network;

and the expression classification unit is used for identifying the acquired real-time image data of the first person through an OpenCV interface, extracting the facial expression data of the first person, and inputting the facial expression data into the expression identification classification model for classification.

8. The AR interaction system based on expressions according to claim 5, characterized in that the surface of the physical object is covered with an identification image, the identification image comprises a two-dimensional graph with patterns and colors, and the real-time image data of the physical object is acquired.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the expression-based AR interaction method of any one of claims 1 to 4.

10. A computer-readable storage medium on which a computer program is stored, the program, when executed by a processor, implementing the expression-based AR interaction method according to any one of claims 1 to 4.

Images