CN114299940A - Display device and voice interaction method - Google Patents

Abstract

The embodiment of the application provides a display device and a voice interaction method, wherein the display device comprises a display used for presenting a user interface; a controller connected with the display, the controller configured to: acquiring user identity information of a target person, and acquiring a voice real-time instruction, wherein the target person comprises a person who sends the awakening instruction or a registered user; detecting face information in an image acquired by a camera; if the face information is the face information of the target person, face tracking and lip movement detection are carried out on the target person, and if the face of the target person moves, and the voice real-time instruction comprises the voice of the target person, the voice real-time instruction is responded; and if the human face of the target person does not have lip movement or the voice real-time instruction does not comprise the voice of the target person, the voice real-time instruction is not responded. The technical problem that voice interaction experience is poor is solved.

Description

Display device and voice interaction method

technical field

The present application relates to the technical field of voice interaction, and in particular, to a display device and a voice interaction method.

Background technique

With the rise of smart homes, controlling home devices such as smart TVs through voice interaction has become an increasingly popular control method. Wake-up rate and speech recognition accuracy are two important indicators that affect the user experience of voice interaction. In the early stage of the development of voice interaction technology, voice interaction is usually near-field interaction. In near-field voice interaction scenarios, the human-machine distance is small, the impact of noise interference is small, and the wake-up rate and voice recognition accuracy are high. However, when people watch TV, they are usually far away from the TV, and near-field interaction cannot meet people's needs. In order to improve the convenience of voice interaction, far-field voice interaction technology emerges as the times require. In the far-field voice interaction scenario, the distance between the human and the machine is large, and the impact of noise interference also increases, resulting in a decrease in the wake-up rate and voice recognition accuracy, resulting in a poor voice interaction experience.

SUMMARY OF THE INVENTION

In order to solve the technical problem of poor voice interaction experience, the present application provides a display device and a voice interaction method.

In a first aspect, the present application provides a display device, the display device comprising:

a display for presenting the user interface;

camera, for capturing images;

a controller, connected to the display, the controller being configured to:

Collect voice wake-up commands;

In response to the voice wake-up command, obtain the user identity information of the target person, and collect the voice real-time command, and the target person includes the person who issued the wake-up command or a registered user;

Detect face information in the image captured by the camera;

If the face information of the target person is detected, face tracking and lip movement detection are performed on the target person. If the face of the target person has lip movement, and the voice real-time instruction includes the target person voice, and respond to the voice real-time command;

If there is no lip movement on the face of the target person, or the real-time voice command does not include the voice of the target person, no response is made to the real-time voice command.

In some embodiments, detecting the face information in the image collected by the camera includes:

Perform sound source localization on the voice wake-up command to obtain the wake-up sound source position;

Rotate the camera toward the position of the wake-up sound source. During the rotation process, face information is detected in the image collected by the camera. If the face information of the target person is detected, the camera is controlled to stop rotating.

In some embodiments,

Perform face tracking and lip motion detection on the target person, including:

Obtain the real-time coordinate range of the face of the target person in the image captured by the camera;

Control the camera to rotate according to the change trend of the real-time coordinate range, so that the face of the target person is located in a preset area in the image collected by the camera;

Perform lip motion detection on the image of the target person's face.

In a second aspect, the present application provides a voice interaction method, the method comprising:

Collect voice wake-up commands;

In response to the voice wake-up command, obtain the user identity information of the target person, and collect the voice real-time command, and the target person includes the person who issued the wake-up command or a registered user;

Detect face information in the image captured by the camera;

If the face information of the target person is detected, face tracking and lip movement detection are performed on the target person. If the face of the target person has lip movement, and the voice real-time instruction includes the target person voice, and respond to the voice real-time command;

If there is no lip movement on the face of the target person, or the real-time voice command does not include the voice of the target person, no response is made to the real-time voice command.

The beneficial effects of the display device and the voice interaction method provided by the present application include:

The display device provided by this application, after receiving the voice wake-up command, obtains the user identity information of the wake-up source, and performs face tracking on the target person corresponding to the user identity information, so that the interference of non-target persons can be excluded; After the face of the person, according to the target person's lip movement, and the received voice real-time command includes the target person's voice, and then respond to the voice real-time command, if the target person does not have lip movement, or the voice real-time command The command does not include the voice of the target person, and does not respond to the voice real-time command, which reduces the probability of noise collection, improves the accuracy of voice recognition, and improves user experience.

Description of drawings

In order to illustrate the technical solutions of the present application more clearly, the accompanying drawings that need to be used in the embodiments will be briefly introduced below. Other drawings can also be obtained from these drawings.

FIG. 1 exemplarily shows a schematic diagram of an operation scene between a display device and a control apparatus according to some embodiments;

FIG. 2 exemplarily shows a hardware configuration block diagram of the control apparatus 100 according to some embodiments;

FIG. 3 exemplarily shows a hardware configuration block diagram of the display device 200 according to some embodiments;

FIG. 4 exemplarily shows a schematic diagram of software configuration in the display device 200 according to some embodiments;

FIG. 5 exemplarily shows a schematic diagram of the principle of voice interaction according to some embodiments;

FIG. 6 exemplarily shows a scene diagram of voice interaction according to some embodiments;

FIG. 7 exemplarily shows a schematic diagram of signal processing for voice interaction according to some embodiments;

FIG. 8 exemplarily shows a timing diagram of voice interaction according to some embodiments;

FIG. 9 exemplarily shows the overall flow diagram of the voice interaction method according to some embodiments;

FIG. 10 exemplarily shows a schematic flowchart of a method for processing a voice wake-up instruction according to some embodiments;

FIG. 11 exemplarily shows a schematic flowchart of a processing method when a single human face is detected in a voice interaction process according to some embodiments;

FIG. 12 exemplarily shows a schematic flowchart of a method for processing when multiple human faces are detected in a voice interaction process according to some embodiments.

Detailed ways

In order to make the purpose and implementation of the present application clearer, the exemplary embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only the Some embodiments are claimed, but not all embodiments.

It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the embodiments described below, rather than intended to limit the embodiments of the present application. Unless otherwise specified, these terms are to be understood according to their ordinary and ordinary meanings.

The terms "first", "second", "third", etc. in the description and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and are not necessarily meant to limit specific Sequential or sequential, unless otherwise noted. It is to be understood that the terms so used are interchangeable under appropriate circumstances.

The terms "comprising" and "having", and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that incorporates a series of components is not necessarily limited to all components explicitly listed, but may include no explicit other components listed or inherent to these products or devices.

The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware or/and software code capable of performing the functions associated with that element.

FIG. 1 is a schematic diagram of an operation scenario between a display device and a control apparatus according to an embodiment. As shown in FIG. 1 , a user can operate the display device 200 through the smart device 300 or the control device 100 .

In some embodiments, the control apparatus 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, and the display device 200 is controlled wirelessly or wiredly. The user can control the display device 200 by inputting user instructions through keys on the remote control, voice input, control panel input, and the like.

In some embodiments, a smart device 300 (eg, a mobile terminal, a tablet computer, a computer, a notebook computer, etc.) can also be used to control the display device 200 . For example, the display device 200 is controlled using an application running on the smart device.

In some embodiments, the display device 200 can also be controlled in a manner other than the control apparatus 100 and the smart device 300. For example, the module for acquiring voice commands configured inside the display device 200 can directly receive the user's voice command for control. , the user's voice command control can also be received through a voice control device provided outside the display device 200 device.

In some embodiments, the display device 200 is also in data communication with the server 400 . The display device 200 may be allowed to communicate via local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 . The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

FIG. 2 exemplarily shows a configuration block diagram of the control apparatus 100 according to an exemplary embodiment. As shown in FIG. 2 , the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply. The control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and play an intermediary role between the user and the display device 200 .

FIG. 3 is a block diagram showing a hardware configuration of the display apparatus 200 according to an exemplary embodiment.

In some embodiments, display device 200 includes tuner 210, communicator 220, detector 230, external device interface 240, controller 250, display 260, audio output interface 270, memory, power supply, user interface at least one.

In some embodiments the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an nth interface for input/output.

In some embodiments, the display 260 includes a display screen component for presenting a picture, and a driving component for driving the image display, for receiving the image signal output from the controller, for displaying the video content, the image content and the menu manipulation interface Components and user manipulation UI interface.

In some embodiments, the display 260 may be a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and projection screen.

In some embodiments, communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 may establish transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220 .

In some embodiments, the user interface may be used to receive control signals from the control device 100 (eg, an infrared remote control, etc.).

In some embodiments, the detector 230 is used to collect signals from the external environment or interaction with the outside. For example, the detector 230 includes a light receiver, a sensor for collecting ambient light intensity; alternatively, the detector 230 includes an image collector, such as a camera, which can be used to collect external environmental scenes, user attributes or user interaction gestures, or , the detector 230 includes a sound collector, such as a microphone, for receiving external sound.

In some embodiments, the external device interface 240 may include, but is not limited to, the following: High Definition Multimedia Interface (HDMI), Analog or Data High Definition Component Input (Component), Composite Video Input (CVBS), USB Input (USB) , RGB port, etc. any one or more interfaces. It may also be a composite input/output interface formed by a plurality of the above-mentioned interfaces.

In some embodiments, the tuner demodulator 210 receives broadcast television signals through wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or cable broadcast television signals.

In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box Wait.

In some embodiments, the controller 250, through various software control programs stored on the memory, controls the operation of the display device and responds to user operations. The controller 250 controls the overall operation of the display apparatus 200 . For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 may perform an operation related to the object selected by the user command.

In some embodiments, the object may be any of the selectable objects, such as hyperlinks, icons, or other operable controls. The operations related to the selected object include: displaying operations connected to hyperlinked pages, documents, images, etc., or executing operations of programs corresponding to the icons.

In some embodiments, the controller includes a central processing unit (Central Processing Unit, CPU), a video processor, an audio processor, a graphics processing unit (Graphics Processing Unit, GPU), RAM (Random Access Memory, RAM), ROM (Read-Only). Memory, ROM), at least one of the first interface to the nth interface for input/output, a communication bus (Bus), and the like.

CPU processor. It is used to execute the operating system and application program instructions stored in the memory, and to execute various application programs, data and contents according to various interactive instructions received from external input, so as to finally display and play various audio and video contents. CPU processor, which can include multiple processors. For example, it includes a main processor and one or more sub-processors.

In some embodiments, the graphics processor is used to generate various graphic objects, such as: icons, operation menus, and user input instructions to display graphics, etc. The graphics processor includes an operator, which performs operations by receiving various interactive instructions input by the user, and displays various objects according to the display attributes; it also includes a renderer, which renders various objects obtained based on the operator, and the rendered objects are used for rendering. displayed on the display.

In some embodiments, the video processor is used to decompress, decode, scale, reduce noise, convert frame rate, convert resolution, and synthesize images according to the standard codec protocol of the received external video signal. After processing, a signal directly displayed or played on the displayable device 200 can be obtained.

In some embodiments, the video processor includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, and the like. The demultiplexing module is used for demultiplexing the input audio and video data stream. The video decoding module is used to process the demultiplexed video signal, including decoding and scaling. The image synthesizing module, such as an image synthesizer, is used for superimposing and mixing the GUI signal generated by the graphics generator according to the user's input or itself, and the zoomed video image, so as to generate an image signal that can be displayed. The frame rate conversion module is used to convert the input video frame rate. The display formatting module is used to convert the received frame rate into the video output signal, and change the signal to conform to the display format signal, such as outputting the RGB data signal.

In some embodiments, the audio processor is configured to receive an external audio signal, and perform decompression and decoding according to a standard codec protocol of the input signal, as well as noise reduction, digital-to-analog conversion, and amplification, etc. The sound signal played in the loudspeaker.

In some embodiments, the user may input user commands on a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input commands through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

In some embodiments, a "user interface" is a medium interface for interaction and information exchange between an application program or an operating system and a user, which enables conversion between an internal form of information and a form acceptable to the user. A commonly used form of user interface is a graphical user interface (Graphic User Interface, GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be an icon, window, control and other interface elements displayed on the display screen of the electronic device, wherein the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. visual interface elements.

In some embodiments, the system of the display device may include a kernel (Kernel), a command parser (shell), a file system and an application program. Together, the kernel, shell, and file system make up the basic operating system structures that allow users to manage files, run programs, and use the system. After power-on, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, etc., runs and maintains virtual memory, scheduler, signals and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. An application is compiled into machine code after startup, forming a process.

The system of the display device may include a kernel (Kernel), a command parser (shell), a file system, and an application program. Together, the kernel, shell, and file system make up the basic operating system structures that allow users to manage files, run programs, and use the system. After power-on, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, etc., runs and maintains virtual memory, scheduler, signals and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. An application is compiled into machine code after startup, forming a process.

Referring to FIG. 4 , in some embodiments, the system is divided into four layers, from top to bottom, the applications layer (referred to as “application layer”), the application framework layer (referred to as “framework layer”) "), the Android runtime (Android runtime) and the system library layer (referred to as "system runtime library layer"), and the kernel layer.

In some embodiments, at least one application program runs in the application program layer, and these application programs may be a Window program, a system setting program, or a clock program that comes with the operating system; they may also be developed by third-party developers. s application. During specific implementation, the application package in the application layer is not limited to the above examples.

The framework layer provides an application programming interface (application programming interface, API) and a programming framework for the application. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center, which decides to let the applications in the application layer take action. The application program can access the resources in the system and obtain the services of the system during execution through the API interface.

As shown in FIG. 4 , the application framework layer in the embodiment of the present application includes managers (Managers), content providers (Content Provider), etc., wherein the manager includes at least one of the following modules: the activity manager (ActivityManager) is used with Interact with all activities running in the system; Location Manager is used to provide system services or applications with access to system location services; Package Manager is used to retrieve files currently installed on the device. Various information related to the application package; Notification Manager (Notification Manager) is used to control the display and clearing of notification messages; Window Manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers and desktops on the user interface part.

In some embodiments, the activity manager is used to manage the life cycle of the various applications and general navigation rollback functions, such as controlling the exit, opening, rollback, etc. of the application. The window manager is used to manage all window programs, such as obtaining the size of the display screen, judging whether there is a status bar, locking the screen, taking screenshots, and controlling the change of the display window (such as reducing the display window, shaking display, distorting display, etc.), etc.

In some embodiments, the system runtime layer provides support for the upper layer, that is, the framework layer. When the framework layer is used, the Android operating system will run the C/C++ library included in the system runtime layer to implement the functions to be implemented by the framework layer.

In some embodiments, the kernel layer is the layer between hardware and software. As shown in Figure 4, the kernel layer at least includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensors, etc.), and power drives, etc.

The hardware or software architecture in some embodiments may be based on the introduction in the foregoing embodiments, and may be based on other similar hardware or software architectures in some embodiments, and the technical solutions of the present application may be implemented.

To clearly illustrate the embodiments of the present application, a speech recognition network architecture provided by the embodiments of the present application is described below with reference to FIG. 5 .

Referring to FIG. 5, FIG. 5 is a schematic diagram of a speech recognition network architecture provided by an embodiment of the present application. In FIG. 5 , the smart device is used to receive the input information and output the processing result of the information. The speech recognition service device is an electronic device deployed with a speech recognition service, the semantic service device is an electronic device deployed with a semantic service, and the business service device is an electronic device deployed with a business service. The electronic device here may include a server, a computer, etc. Here, the speech recognition service, the semantic service (also referred to as a semantic engine) and the business service are web services that can be deployed on the electronic device, wherein the speech recognition service is used to convert audio Recognized as text, the semantic service is used for semantic analysis of the text, and the business service is used to provide specific services such as the weather query service of Moji Weather and the music query service of QQ Music. In one embodiment, in the architecture shown in FIG. 5, there may be multiple entity service devices deployed with different business services, or one or more functional services may be aggregated in one or more entity service devices.

In some embodiments, the following describes the process of processing the information input to the smart device based on the architecture shown in FIG. 5 . Taking the information input to the smart device as an example of a query sentence input by voice, the above process may include the following three processes: :

[Speech Recognition]

After receiving the query sentence input by voice, the smart device can upload the audio of the query sentence to the voice recognition service device, so that the voice recognition service device can recognize the audio as text through the voice recognition service and return it to the smart device. In one embodiment, before uploading the audio of the query sentence to the speech recognition service device, the smart device may perform denoising processing on the audio of the query sentence, where the denoising processing may include steps such as removing echoes and ambient noise.

[Semantic understanding]

The smart device uploads the text of the query sentence recognized by the speech recognition service to the semantic service device, so that the semantic service device performs semantic analysis on the text through the semantic service to obtain the business field and intent of the text.

[semantic response]

According to the semantic analysis result of the text of the query sentence, the semantic service device sends a query instruction to the corresponding business service device to obtain the query result given by the business service. The smart device can obtain and output the query result from the semantic service device. As an embodiment, the semantic service device may also send the semantic parsing result of the query statement to the smart device, so that the smart device outputs the feedback statement in the semantic parsing result.

It should be noted that the architecture shown in FIG. 5 is only an example, and does not limit the protection scope of the present application. In the embodiments of the present application, other architectures may also be used to implement similar functions. For example, all or part of the three processes may be completed by an intelligent terminal, which will not be repeated here.

In some embodiments, the smart device shown in FIG. 5 can be a display device, such as a smart TV, the function of the voice recognition service device can be realized by a sound collector and a controller set on the display device, the semantic service device and the business service device The functions of the display device can be implemented by the controller of the display device, or by the server of the display device.

In some embodiments, a query sentence or other interactive sentence input by the user to the display device through voice may be referred to as a voice command.

In some embodiments, what the display device obtains from the semantic service device is the query result given by the business service, and the display device can analyze the query result, generate response data for the voice command, and then control the display device to execute the corresponding response data according to the response data. Actions.

In some embodiments, what the display device obtains from the semantic service device is the semantic parsing result of the voice command, the display device can analyze the semantic parsing result, generate response data, and then control the display device to perform corresponding actions according to the response data.

In some embodiments, a voice control button may be provided on the remote control of the display device. After the user presses the voice control button on the remote control, the controller of the display device may control the display of the display device to display a voice interaction interface and control sound A collector, such as a microphone, picks up the sound around the display device. At this time, the user may input a voice instruction to the display device.

In some embodiments, the display device may support a voice wake-up function, and the sound collector of the display device may be in a state of continuously collecting sounds. After the user speaks the wake-up word, the display device performs speech recognition on the voice command input by the user, and after recognizing that the voice command is the wake-up word, it can control the display of the display device to display the voice interaction interface. At this time, the user can continue to input voice to the display device. instruction. The wake-up word may be called a voice wake-up command, and the voice command that the user continues to input may be called a voice real-time command.

In some embodiments, after the user inputs a voice command, during the process that the display device obtains the response data of the voice command or the display device responds according to the response data, the sound collector of the display device can keep the sound collecting state, and the user can Press and hold the voice control button on the remote control at any time to re-enter the voice command, or say the wake-up word. At this time, the display device can end the last voice interaction process, and start a new voice interaction process according to the new voice command input by the user. This ensures the real-time nature of voice interaction.

In some embodiments, when the current interface of the display device is a voice interaction interface, after the display device performs voice recognition on the voice command input by the user, the text corresponding to the voice command is obtained, and the display device itself or the server of the display device performs the text processing on the text. After semantic understanding, the user intent is obtained, the user intent is processed to obtain a semantic parsing result, and response data is generated according to the semantic parsing result.

Exemplarily, in the voice interaction mode in which the display device starts a voice dialogue according to the received voice wake-up command, the user who issues the voice wake-up command may be referred to as the target person.

In some embodiments, the target person can also be a registered user on the display device. When the user registers on the display device, the user can enter voiceprint information and a face image on the display device.

In some voice interaction scenarios, due to environmental noise interference and non-target people's voice interference, the display device may be awakened by mistake, or the target person's intention cannot be accurately recognized based on the collected audio, which will seriously affect the user's experience. Voice interaction experience.

In order to solve the above problem, the embodiment of the present application shows a voice interaction solution, which can effectively improve the voice interaction experience in complex voice interaction scenarios by combining audio signal processing and video signal processing methods.

In some embodiments, in order to collect video signals required in the process of voice interaction, the display device may be provided with a camera, or an external camera may be connected.

Taking a display device provided with a camera as an example, see FIG. 6 , which is a schematic diagram of a voice interaction scenario according to some embodiments. As shown in FIG. 6, in some embodiments, the display device 200 may be provided with a camera 201, and the camera 201 may capture images. If the camera is fixed on the display device 200, it can only capture images within a certain field of view, such as the image within the area of A area in FIG. When the user is standing in the A area, the camera 201 can photograph the user, but when the user is standing in the B area and the C area, the camera 201 cannot photograph the user, wherein the B area is the left area of the A area, and the C area is the A area Right area of the area.

In some embodiments, in order to expand the field of view of the camera 201 , the camera 201 may be provided with a pan/tilt or other structures capable of adjusting the field of view of the camera. The PTZ can adjust the field of view of the camera 201, and the controller of the display device can be connected to the camera, and the PTZ can control the dynamic change of the field of view of the camera, so that the field of view of the camera 201 can reach 0-180 degrees after rotation, so that it can shoot To the user within the range of 0-180 degrees, where 0 degrees means that the user is standing on the left side of the display device 200 and is on the same plane as the display device 200, and 180 degrees means that the user is standing on the right side of the display device 200 and is on the same plane as the display device 200. The devices 200 are in the same plane. It can be seen that by rotating the camera, the camera can capture the users located in the B area and the C area, so that as long as the user is standing in front of the display device 200 , the user can be captured by rotating the camera 201 .

In some embodiments, the display device performs image acquisition through an external camera, and the external camera may be a camera provided with a pan/tilt head, so as to realize image acquisition of a dynamic field of view.

In some embodiments, the camera 201 itself does not have a pan/tilt, and can be installed on a pan/tilt that is communicatively connected to the display device, and the image capture of the dynamic field of view may also be achieved through the display device's control of the pan/tilt.

In some embodiments, for a method of combining audio signal processing and video signal processing by a display device, reference may be made to FIG. 7 , which is a schematic diagram of signal processing in a voice interaction process according to some embodiments.

As shown in FIG. 7 , in some embodiments, the display device obtains an audio signal through audio collected by a microphone, and the processing of the audio signal includes sound source localization, user attribute recognition, voiceprint recognition, speech recognition, and noise reduction enhancement.

The sound source localization may include determining the angle between the audio source and the display device. Referring to Figure 6, if α is 90 degrees, when the user is located in the B area, the wake-up angle is between 0 and 45 degrees, and when the user is located in the A area , the wake-up angle is between 45-135 degrees, and when the user is in the C area, the wake-up angle is between 135-180 degrees. Sound source localization can be achieved by a variety of algorithms, such as time difference of arrival method, beamforming method and so on. In order to achieve sound source localization, the display device can be provided with a microphone array. The microphone array includes a plurality of microphones arranged at different positions of the display device. Each microphone is connected to the controller of the display device. For multi-channel audio signals, the wake-up angle is obtained by comprehensive analysis of the multi-channel audio signals. For example, in the time difference of arrival method, the display device can calculate the position of the audio source relative to the display device according to the time difference between the audio signals received by the multiple microphones and the relative positional relationship between the multiple microphones. In the beamforming method, the display device can filter and weight the audio signals collected by each microphone to form a sound pressure distribution beam. According to the distribution characteristics of the sound pressure beam, the position of the audio source relative to the display device can be obtained. Based on the position of the audio source relative to the display device, the angle between the audio source and the display device is obtained.

User attributes determined by user attribute identification may include user gender and user age, where the age may be an age group, such as 1-10 years old, 11-20 years old, 20-40 years old, 40-60 years old and so on. User attribute recognition can be implemented based on pre-trained models. By collecting a large number of audio samples with different user attributes, a model capable of predicting user attributes can be trained based on a neural network. After inputting audio signals into the model, user attributes can be obtained.

The noise reduction enhancement may include enhancing the target person's voice in the audio signal and performing noise reduction processing on the non-target person's audio. Noise reduction enhancement can be achieved through speech-directed enhancement techniques. Through the speech enhancement beam, it is dynamically adjusted to an enhancement beam centered on the target person, which enhances the target person's speech and suppresses the sound outside the beam.

It can be seen that the processing of the audio signal realizes the determination of the user's position, the determination of the user's identity and the determination of the audio content. After the user's position is determined, the camera can be controlled to rotate to quickly locate the target person. After the user's identity is determined, the target of the voice interaction can be distinguished from other people. After the audio content is determined, the user's intention can be obtained.

As shown in FIG. 7 , in some embodiments, the display device obtains a video signal through an image captured by a camera, and the processing of the video signal includes face detection and tracking, face recognition, lip movement detection and lip language recognition.

Among them, in the process of face detection and tracking, the camera can be controlled to rotate to ensure that the target person is always within the field of view of the camera.

Lip motion detection can be implemented in the face captured by the camera to determine whether the lips have changed. If the lips have changed, it can be determined that the person is speaking. If the lips have not changed, it can be determined that the person is not speaking. If a person is speaking, it can be combined with face recognition technology to determine whether the person who is speaking is the target person. If the person who is speaking includes the target person, it can be determined that the received audio signal contains the target person's voice. The person does not include the target person, and it can be determined that the received audio signal does not contain the target person's voice.

Lip recognition can identify the speech content of the person who is speaking, and the speech content can be used for comparison and analysis with the audio content obtained from the speech recognition of the audio signal. If they are consistent or roughly consistent, the audio content of the audio signal can be considered to come from the camera. On the contrary, if the difference between the people in the captured images is relatively large, it can be considered that the audio content of the audio signal comes from people or environments other than the images captured by the camera. In some embodiments, lip language recognition may not be performed, and only lip motion detection is used to determine whether the audio content of the audio signal comes from a person in the image captured by the camera, which can reduce the resource occupation of video signal processing and improve processing efficiency. .

It can be seen that the processing of the video signal realizes the tracking of the user's position and the determination of whether the user is speaking.

As shown in FIG. 7 , in some embodiments, after obtaining the processing result of the audio signal and the processing result of the video signal, the display device may also obtain application scene information. The application scene information may include interactive control information preset by the foreground running application, and the interactive control information may include audio capture control parameters and video capture control parameters. Exemplarily, the value of the audio capture control parameter is 1, indicating that audio data can be currently processed. Collection, a value of 0 indicates that the current audio data collection cannot be performed, a value of the video capture control parameter value of 1 indicates that the current video data collection can be performed, and a value of 0 indicates that the current video data collection cannot be performed.

For example, when the application running in the foreground is a video chat application, the audio capture control parameter and the video capture control parameter are both 1, and the fusion decision engine can process the audio signal according to the audio capture control parameter and the video capture control parameter are both 1. And the processing result of the video signal is analyzed comprehensively to get the recognition result.

When the application running in the foreground is an online teaching application, if the display device is the student terminal, the students may not be allowed to speak at some moments. In this case, the audio acquisition control parameter can be 0. The fusion decision engine can determine whether to adopt the processing result of the audio signal and the processing result of the video signal according to the audio acquisition control parameters and the video acquisition control parameters to obtain the identification result, or determine whether to control the display device to acquire audio data and video data. .

In some embodiments, the processing result of the audio signal, the processing result of the video signal and the application scene information are input into the feature fusion decision engine, and the multimodal recognition result can be output through the feature fusion decision engine. The multimodal recognition result may include the speech content of the audio signal, the person who is speaking, the person who is not speaking, and the correspondence between the person and the speech content. According to the corresponding relationship, it can be determined whether the target person has spoken, and if the target person has spoken, the content of his speech can be determined, thereby improving the accuracy of speech recognition and reducing the probability of false awakening.

In some embodiments, the processing of the audio signal and the processing of the video signal shown in FIG. 7 can be implemented locally by the display device, or the display device can send the audio signal and the video signal to the server, and the server processes it in the cloud, The processing result is then returned to the display device, and part of the functions can also be implemented locally by the display device and part of the functions implemented by the server.

In some embodiments, the user's voiceprint information and face information are private information, and the display device can be configured to display privacy functions such as voiceprint recognition and face recognition when the user is powered on for navigation and when the user uses the voice assistant function for the first time After viewing the prompt, the user can click the option control and choose to enable the above privacy function to improve the voice interaction effect, or choose not to trigger the option control, thus Improve privacy security.

To further describe the signal processing process of the voice interaction process shown in FIG. 7 , FIG. 8 shows a sequence diagram of the voice interaction process according to some embodiments. It should be noted that this timing diagram is only an exemplary timing diagram of the signal processing process shown in FIG. 7 , and in an actual embodiment, the signal processing process shown in FIG. 7 may further include other timing sequences.

Referring to FIG. 8 , in the process of voice interaction, the display device can interact with the user and the server respectively to provide voice control services for the user.

In FIG. 8 , the user may be a target person who wants to control the display device. In addition to the target person, the microphone of the display device may also collect other users' voices or ambient noise.

In some embodiments, the wake-up word input by the user to the display device may be a voice wake-up command. After receiving the voice wake-up command, the display device may locate the sound source according to the voice wake-up command, calculate the position of the wake-up sound source, and obtain the wake-up sound source. Position information, the position information of the wake-up sound source may include the wake-up angle.

In some embodiments, after calculating the wake-up angle, the display device can turn on the camera, rotate the camera toward the wake-up sound source according to the wake-up angle, and continue to shoot during the rotation to obtain images in the dynamic field of view. The wake-up sound source is the target person. Of course, if the user does not input a wake-up word to the display device, that is, the display device is woken up by mistake, the wake-up sound source may not actually exist.

In some embodiments, after the display device collects the voice wake-up command, in order to prevent the voice wake-up command from being a false wake-up voice command, the display device can obtain the user identity information of the voice wake-up command, and then display the user identity information of the voice wake-up command in the collected dynamic field of view. The target person is detected in the image. If the target person can be located in the collected image, it can be determined that the wake-up is not a false wake-up. If the target person cannot be located, it can be determined that the wake-up is a false wake-up.

In some embodiments, in order to obtain the user identity information, the display device may generate a wake-up word recognition request including the wake-up word voice, and send the wake-up word recognition request to the server. After receiving the wake-up word recognition request, the server extracts the wake-up word voice from the request, performs user attribute recognition and voiceprint recognition on the wake-up word voice, and obtains the user identity information of the target person.

In some embodiments, the server may obtain voiceprint features when performing user attribute identification and voiceprint identification, and then matches the voiceprint features with the voiceprint features in the database, and obtains the user identity information of the voice wake-up command according to the matching result , wherein the voiceprint feature in the database can be obtained according to the audio data pre-entered by the user. The user identity information may include the target person's voiceprint mark U1, gender US1 and age UA1, where gender US1 and age UA1 may be user attributes, and age UA1 may represent an age group, such as 1-10 years old, 11-20 years old, 20-40 years old, 40-60 years old and so on.

In some embodiments, user attribute recognition and voiceprint recognition can also be implemented locally by the display device, the user can pre-record audio on the display device, and the display device can perform voice wake-up instruction with the audio pre-recorded by the user through voiceprint recognition By comparing, the user identity information corresponding to the voice wake-up command is obtained.

In some embodiments, the user identity information may also include a face image of the target person to facilitate subsequent face recognition. The avatar may be stored in the server and/or the display device, and the display device is acquiring the user identity corresponding to the voice wake-up command. information, the face image can be obtained.

In some embodiments, in the process of requesting the user identity information from the server, the display device may directly control the camera to capture images without adjusting the field of view of the camera, and locally perform face recognition on the images captured by the camera. At this time, if the target person is standing in the current field of view of the camera, the target person can be photographed, and if the user is outside the current field of view of the camera, the target person cannot be photographed. In order to reduce the resource consumption of the display device, during face recognition, only the age feature and gender feature of the face can be detected. After receiving the user identity information from the server, the gender US1 and age UA1 in the user identity information can be extracted and compared with the results of face recognition. If no face is detected, or the age of the detected face Or the gender does not match the user's identity information, adjust the field of view of the camera, and re-take the image for face detection. Of course, if the user identity information includes a face image, during face recognition, it can also be judged whether the face in the image collected by the camera is consistent with the face image in the user identity information. Improve the accuracy of target person recognition, but may lead to a decrease in the speed of target person recognition.

In some embodiments, if the display device cannot identify the target person from the image captured by the camera in the current field of view, after obtaining the wake-up angle, the display device may adjust the field of view of the camera according to the wake-up angle to capture the target person .

In some embodiments, the display device may adjust the camera's field of view to cover the wake-up angle. For example, if the wake-up angle is 30 degrees and the current field of view of the display device is 40-140 degrees, the camera can be rotated more than 10 degrees to the left, so that the target person can be located in the captured image. During the rotation process, the camera can continuously take pictures and perform face recognition. If a face matching the user's identity information is recognized, the target person can be located. At this time, the camera can stop rotating.

In some embodiments, after the display device adjusts the field of view of the camera according to the wake-up angle, the captured image may still fail to recognize the face matching the user's identity information, and it may be determined that the wake-up angle is incorrectly calculated or a false wake-up occurs. In order to solve the problem of incorrect calculation of the wake-up angle, the display device can control the rotation of the camera to cover the maximum field of view, that is, the field of view of 0-180 degrees. If the target person cannot be recognized within the field of view of 0-180 degrees, confirm Accidental wake-up during voice wake-up command. Alternatively, the display device can re-measure the angle of the target person relative to the display device according to the voice real-time command received after the voice wake-up command, and then adjust the field of view of the camera according to the re-measured angle. The target person can determine that a false wake-up has occurred, and at this time, the display device can exit the voice interaction process.

In some embodiments, in the image captured by the camera, after the display device recognizes the target person, the face tracking of the target person can be performed. Since the target person may move back and forth, the field of view of the camera is dynamically adjusted through face tracking, so that the target person is always within the field of view of the camera. During face tracking, a preset area can be determined in the image captured by the camera, and then the real-time coordinate range of the target face in the image captured by the camera can be obtained, and the camera can be controlled to rotate according to the change trend of the real-time coordinate range. If the real-time coordinate range is within the preset area, the camera does not need to be rotated. If a boundary of the real-time coordinate range coincides with the boundary of the preset area, the camera can be rotated according to the change trend of the real-time coordinate range, so that the face is located in the preset area again. For example, if the change trend of the real-time coordinate range is to translate to the left, control the camera to rotate to the left.

In some embodiments, the real-time voice command received by the display device after the voice wake-up command may include any one or more of the target person's voice, other people's voice, and ambient noise. In the image captured by the camera, by detecting the lip movement of the target person, it can be judged whether the target person is speaking. If the target person is speaking, the real-time voice command can be semantically recognized. If the target person is not speaking, the voice real-time command can be ignored. Semantic recognition.

In some embodiments, the display device may generate a speech recognition request containing real-time speech and send the request to the server. After receiving the speech recognition request, the server performs semantic recognition on the real-time speech in the speech recognition request, and returns the semantic recognition result to the display device.

In some embodiments, the server may first perform voiceprint recognition on the real-time voice, and if the result of the voiceprint recognition determines that the real-time voice contains the target person's voice, then perform semantic recognition on the real-time voice, and return the semantic recognition result to the display The device, or the server can perform voiceprint recognition and semantic recognition at the same time, and return the voiceprint recognition result and the semantic recognition result to the display device.

In some embodiments, when the server performs semantic recognition, in order to improve the accuracy of semantic recognition, the server may perform noise reduction processing on the real-time voice command.

In some embodiments, if there are other people in addition to the target person in the image captured by the camera, the noise reduction process may be implemented by a human voice separation mechanism. Separate the single-channel human voice based on the mixed human voice, perform voiceprint recognition on the single-channel audio, determine whether the single-channel audio belongs to the target speaker's voice, then perform semantic recognition on the target speaker's voice, and discard the non-target speaker's voice . The human voice separation mechanism adopts the voiceprint features based on the identity information of the target person, and based on noise source modeling, to achieve the enhancement of the target voice and the suppression of sounds other than the target speaker's voice to achieve the goal of this scenario. The effect of human voice recognition is optimized.

In some embodiments, after receiving the identification result from the server, the display device may respond according to the identification result. Exemplarily, the recognition result may include: R={U=U1, V=turn up the volume}, where R represents the recognition result, U represents the voiceprint, U1 represents the voiceprint of the target person, and the voice content of the target person is "Turn up the volume". The display device may increase the volume of the display device according to the recognition result.

In some embodiments, the specific flow of the speech recognition method of the present application can be referred to FIG. 9 to FIG. 12 , and the technical solution of the present application will be introduced below with reference to the specific flow of the speech recognition method.

Referring to FIG. 9 , it is a schematic diagram of an overall flow of a voice interaction method according to some embodiments. After the user inputs the voice wake-up command to the display device, as shown in Figure 9, the display device can collect the user's voice wake-up command, calculate the wake-up angle according to the voice wake-up command, and then rotate the camera according to the wake-up angle to make the camera face the wake-up angle, thereby Make the acquisition range of the camera cover the wake-up angle.

After the display device controls the camera to face the wake-up angle, it can perform face detection and face recognition on the image captured by the camera, so as to determine whether a target person is detected in the image captured by the camera.

If the target person is not detected in the image captured by the camera, the wake-up angle is recalculated according to the real-time recorded audio to realize real-time sound source localization, and then according to the result of the sound source localization, the camera is rotated to make the camera face the recalculated wake-up angle. , and then detect the face from the image captured by the camera, and perform face recognition on the detected face to determine whether it is the target person.

If the target person is detected in the image captured by the camera, the display device controls the camera to track the face of the target person, and performs lip movement detection and recognition on the image captured by the camera. The image captured by the camera may include the face of the target person and the face of the non-target person, or only the face of the target person.

If it is detected in the image captured by the camera that the person with lip movement is the target person, a real-time recording, that is, a real-time voice command collected by the audio input device of the display device, can be obtained. The display device can use the beam forming method to process the real-time recording beam, and the processing content can include enhancing the voice beam of the target person and suppressing other voice beams. location is determined.

If a non-target person is detected in the image captured by the camera, the real-time recording may include one or more of noise and non-target vocals. Sound suppression, in order to improve the recognition accuracy of the next real-time recording, of course, the display device can also directly delete the noise or non-target vocals.

Wherein, before performing face detection and recognition, the display device may determine the target person in advance according to the voice wake-up instruction, and the target person may be the user who issued the voice wake-up instruction. Referring to FIG. 10 , it is a schematic flowchart of a method for processing a voice wake-up instruction according to some embodiments. As shown in Figure 10, when the display device collects the wake-up audio, it determines that the wake-up audio includes a voice wake-up instruction through wake-up word detection, calculates the wake-up angle according to the wake-up audio, saves the wake-up audio to a preset path, and then The wake-up audio of the set path is uploaded to the cloud, so that the server in the cloud can process the wake-up audio. The server's processing of the wake-up audio may include voiceprint recognition and user attribute recognition. Through the voiceprint recognition, the user's voiceprint of the wake-up audio can be determined, and the user's gender and age group corresponding to the wake-up audio can be obtained through the user attribute recognition. Finally, the wake-up angle calculated locally by the display device and the voiceprint and user attributes recognized by the server in the cloud are determined as the recognition result of the wake-up audio.

As can be seen from Figure 10, after the display device collects the voice wake-up command, it calculates the wake-up angle according to the voice wake-up command. However, controlling the camera to rotate toward the wake-up angle can increase the probability of detecting the target person in the captured image.

In some embodiments, the images captured by the camera may include single-person faces and multiple-person faces. In order to analyze these two situations specifically, FIG. 11 shows that a single person is detected during the voice interaction process according to some embodiments. Fig. 12 shows a schematic flowchart of a processing method when multiple human faces are detected in a voice interaction process.

Referring to Figure 11, if the display device detects a single human face within the wake-up angle, it can collect real-time recordings and perform lip movement detection on the detected face. Among them, the processing of real-time recording includes real-time sound source positioning and cloud recognition. Real-time sound source positioning includes calculating the real-time sound source angle of real-time recording. Cloud recognition includes voice recognition and voiceprint recognition. During voiceprint recognition, the voiceprint of the voice wake-up command and the real-time recording can be uploaded to the server, so that after the server obtains the voiceprint of the real-time recording, it can confirm whether the voiceprint and the voiceprint of the voice wake-up command belong to the same user. voiceprint, and include the confirmation result into the voiceprint recognition result. According to the speech recognition results and voiceprint recognition results returned by the server, as well as the real-time sound source localization results obtained locally by the display device, the display device can obtain the real-time sound source angle, voiceprint, attributes, voice content and other information corresponding to the real-time recording.

The display device can verify whether the real-time recording is the voice of the target speaker according to the voiceprint recognition result. If the real-time recording is not the voice of the target speaker, based on the real-time sound source angle determined by the real-time sound source localization, re-rotate the camera to make the camera face the real-time sound source angle, and then perform face detection in the image captured by the camera. If the real-time recording is the voice of the target speaker, face tracking is performed on the face detected by the camera, and then the lip movement detection result of the face is obtained. If the result of the lip movement detection is that the lip movement has occurred, the voice interaction is performed according to the real-time recording.

Referring to Figure 12, if the display device detects multiple faces within the wake-up angle, it can collect real-time recordings and perform lip motion detection on the detected faces. Among them, the processing of real-time recording includes real-time sound source positioning and cloud recognition. Real-time sound source positioning includes calculating the real-time sound source angle of real-time recording. Cloud recognition includes voice recognition and voiceprint recognition. During voiceprint recognition, the voiceprint of the voice wake-up command and the real-time recording can be uploaded to the server, so that after the server obtains the voiceprint of the real-time recording, it can confirm whether the real-time recording contains the voiceprint of the voice wake-up command. The voiceprint of the same user, that is, whether the voiceprint of the target person is included, and the confirmation result is included in the voiceprint recognition result. According to the speech recognition results and voiceprint recognition results returned by the server, as well as the real-time sound source localization results obtained locally by the display device, the display device can obtain the real-time sound source angle, voiceprint, attributes, voice content and other information corresponding to the real-time recording.

The display device can verify whether the real-time recording is the voice of the target speaker according to the voiceprint recognition result. If the real-time recording is not the voice of the target speaker, based on the real-time sound source angle determined by the real-time sound source localization, re-rotate the camera to make the camera face the real-time sound source angle, and then perform face detection in the image captured by the camera. If the real-time recording contains the voice of the target speaker, locate the target person from the face detected by the camera, then perform face tracking on the target person, and then obtain the lip movement detection result of the target person's face. If the result of the lip movement detection is that the lip movement has occurred, the voice interaction is performed according to the real-time recording.

In the above-mentioned embodiment, the camera built in the display device or the camera connected to the display device can be rotated to obtain a larger field of view angle, thereby realizing face detection, tracking and lip motion detection within a larger field of view. , which improves the probability of successful target positioning in voice interaction, thereby improving the wake-up rate of voice interaction and the accuracy of voice recognition. However, in some embodiments, the camera provided with the display device or the camera connected to the display device is not set with a pan/tilt, and the camera cannot be rotated. Perform face detection, tracking and lip movement detection. At this time, face tracking can be used to obtain the real-time coordinate range of the target person's face, and the real-time coordinate range can be a coordinate range of a rectangular area containing the target person's face. The detection is to perform lip motion detection on the image within the real-time coordinate range. Through face tracking, the range of lip movement detection can be narrowed and the response speed of voice interaction can be improved.

It can be seen from the above embodiments that the present application can effectively improve the voice interaction experience in complex voice interaction scenarios by integrating technologies such as sound source localization, face tracking, voiceprint recognition, lip movement detection, and noise reduction processing.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application. scope.

For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussions are not intended to be exhaustive or to limit implementations to the specific forms disclosed above. Numerous modifications and variations are possible in light of the above teachings. The above embodiments are chosen and described to better explain the principles and practical applications, so as to enable those skilled in the art to better utilize the described embodiments and various modified embodiments suitable for specific use considerations.

Claims (10)

1. A display device, comprising:

a display for presenting a user interface;

the camera is used for collecting images;

a controller connected with the display, the controller configured to:

collecting a voice awakening instruction;

responding to the voice awakening instruction, acquiring user identity information of a target person, and acquiring a voice real-time instruction, wherein the target person comprises a person who sends the awakening instruction or a registered user;

detecting face information in an image acquired by the camera;

if the face information of the target person is detected, carrying out face tracking and lip movement detection on the target person, and if the face of the target person has lip movement and the voice real-time instruction comprises the voice of the target person, responding to the voice real-time instruction;

and if the human face of the target person does not have lip movement or the voice real-time instruction does not comprise the voice of the target person, not responding to the voice real-time instruction.

2. The display device according to claim 1, wherein detecting face information in the image captured by the camera comprises:

carrying out sound source positioning on the voice awakening instruction to obtain an awakening sound source position;

and rotating the camera towards the awakening sound source position, detecting face information in an image acquired by the camera in the rotating process, and controlling the camera to stop rotating if the face information of the target person is detected.

3. The display device according to claim 1, wherein performing face tracking and lip movement detection on the target person comprises:

acquiring a real-time coordinate range of the face of a target person in an image shot by the camera;

and carrying out lip movement detection on the image in the real-time coordinate range.

4. The display device according to claim 1, wherein performing face tracking and lip movement detection on the target person comprises:

acquiring a real-time coordinate range of the face of a target person in an image shot by the camera;

controlling the camera to rotate according to the variation trend of the real-time coordinate range, so that the face of the target person is positioned in a preset area in the image acquired by the camera;

and carrying out lip movement detection on the image of the face of the target person.

5. The display device of claim 1, wherein the controller is further configured to:

if the face of the target person cannot be detected, carrying out sound source positioning on the voice real-time instruction to obtain a real-time sound source position;

and rotating a camera towards the real-time sound source position, and detecting the face of the target person corresponding to the user identity information in the image acquired by the camera.

6. The display device according to claim 1, wherein obtaining the user identity information corresponding to the voice wake-up instruction comprises:

and carrying out voiceprint recognition on the voice awakening instruction to obtain user identity information, wherein the user identity information comprises voiceprint information of the target person.

7. The display device of claim 1, wherein responding to the voice real-time instruction comprises:

if the image shot by the camera only comprises a single human face of the target person, performing directional voice enhancement on the voice corresponding to the awakening sound source position;

and responding to the voice real-time instruction after the directional voice enhancement.

8. The display device of claim 1, wherein responding to the voice real-time instruction comprises:

if the voice real-time instruction comprises multi-path voice, separating the single-path voice of the target person from the voice real-time instruction;

and responding according to the single-channel voice of the target person.

9. The display device of claim 1, wherein the controller is further configured to:

after the voice real-time instruction is collected, the voice real-time instruction is sent to a server, so that the server performs voiceprint recognition, voice recognition and semantic recognition on the voice real-time instruction to obtain a recognition result;

and receiving the recognition result of the server to the voice real-time instruction.

10. A method of voice interaction, comprising:

collecting a voice awakening instruction;

responding to the voice awakening instruction, acquiring user identity information of a target person, and acquiring a voice real-time instruction, wherein the target person comprises a person who sends the awakening instruction or a registered user;

detecting face information in an image acquired by a camera;

if the face information of the target person is detected, carrying out face tracking and lip movement detection on the target person, and if the face of the target person has lip movement and the voice real-time instruction comprises the voice of the target person, responding to the voice real-time instruction;

and if the human face of the target person does not have lip movement or the voice real-time instruction does not comprise the voice of the target person, not responding to the voice real-time instruction.

Images