CN114627864A - Display device and voice interaction method - Google Patents

Abstract

The embodiment of the application provides a display device and a voice interaction method, wherein when first voice data input by a user is received, candidate user intentions corresponding to the first voice data are determined; when the first voice data corresponds to a plurality of candidate user intentions, generating an inquiry statement according to the candidate user intentions, and feeding back the inquiry statement to the user for prompting the user to select one user intention from the candidate user intentions; receiving second voice data input by a user, and determining a target user intention corresponding to the first voice data in the candidate user intentions according to the second voice data; and outputting the associated information associated with the target user intention. According to the method and the device, the accuracy of understanding the user intention in the voice interaction process can be effectively improved.

Description

Display device and voice interaction method

technical field

The embodiments of the present application relate to the technical field of voice interaction, and in particular, to a display device and a voice interaction method.

Background technique

At this stage, due to the development of voice technology, there are more and more intelligent voice interaction devices, and voice interaction has become a very important way of human-computer interaction, especially the popularity of voice assistants in recent years, from mobile terminals to some smart home appliances. The service can be obtained through voice interaction.

Existing voice interaction systems usually first understand the user's intention according to the sentence input by the user, and then provide relevant services to the user according to the user's intention.

However, in the face of cross-service decisions or ambiguous user intentions, it is difficult for existing voice interaction systems to make accurate intention understanding and decision-making. For example, when the search result corresponding to the query input by the user includes both songs and videos, the existing voice interaction system will score the two search results, if the score of the searched "song" is higher than that of the "video" If the score of the searched "video" is higher than that of "song", the user will be given priority to feed back the searched video, which may lead to the final decision not conforming to the the real intent of the user.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a display device and a voice interaction method, which can improve the accuracy of user intention understanding during the voice interaction process.

In some embodiments, embodiments of the present application provide a display device, the display device comprising:

A voice collection device for collecting voice data;

An audio processor for processing the collected voice data;

a display screen for displaying images;

A controller that is configured to:

Receive the first voice data input by the user, and determine the candidate user intent corresponding to the first voice data;

When the first voice data corresponds to multiple candidate user intentions, generate an inquiry sentence according to the multiple candidate user intentions, and send the inquiry sentence to the display screen for display, where the inquiry sentence is used to prompt the user selecting a user intent from the plurality of candidate user intents;

receiving second voice data input by the user, and determining, according to the second voice data, a target user intent corresponding to the first voice data among the plurality of candidate user intents;

Output association information associated with the target user intent.

In a possible design manner, when the first voice data corresponds to a single candidate user intent, output associated information associated with the candidate user intent.

In one possible design, the controller is configured to:

Obtain the historical user intent determined during the voice interaction process;

According to the first voice data and the historical user intent, the intent recognition model is used to determine a candidate user intent corresponding to the first voice data.

In one possible design, the controller is configured to:

Determine whether the first voice data and the historical user intent belong to the same dialogue sequence based on the dialogue state tracking model;

When the first voice data and the historical user intent belong to the same dialogue sequence, use the intent recognition model to determine the initial user intent corresponding to the first voice data, and update the first voice data according to the historical user intent an initial user intent corresponding to the voice data, obtaining a candidate user intent corresponding to the first voice data;

When the first voice data and the historical user intent do not belong to the same dialogue sequence, the intent recognition model is used to determine a candidate user intent corresponding to the first voice data.

In one possible design, the controller is configured to:

According to each candidate user intent corresponding to the first voice data, the score of each output module in the dialogue strategy learning model is determined; the dialogue strategy learning model includes at least one of the following output modules: a rewriting module, which refers to a resolution module , vertical domain intent analysis module, task multi-round response module, question and answer module, news search module, chat module, recommendation module and candidate intent analysis module;

When the output module with the highest score in the dialogue strategy learning model is the candidate intent parsing module, the query sentence is generated according to the score of each candidate user intent.

In one possible design, the controller is configured to:

When the score of the first candidate user intent corresponding to the first voice data is less than the first preset threshold, the score of the second candidate user intent corresponding to the first voice data is greater than the second preset threshold, and the first When the difference between the score of the candidate user intent and the score of the second candidate user intent is less than a preset interval threshold, the query sentence is generated based on the first candidate user intent and the second candidate user intent; wherein, the The first candidate user intent is the candidate user intent with the highest score corresponding to the first voice data, the second candidate user intent is the candidate user intent with the second highest score corresponding to the first voice data, and the first candidate user intent The preset threshold is greater than the second preset threshold.

In some embodiments, the embodiments of the present application further provide a voice interaction method, the method comprising:

Receive the first voice data input by the user, and determine the candidate user intent corresponding to the first voice data;

When the first voice data corresponds to a plurality of candidate user intentions, a query sentence is generated according to the plurality of candidate user intentions, and the query sentence is output to the user, where the query sentence is used to prompt the user to select from the plurality of user intentions. select a user intent from among the candidate user intents;

receiving second voice data input by the user, and determining, according to the second voice data, a target user intent corresponding to the first voice data among the plurality of candidate user intents;

Output association information associated with the target user intent.

In a possible design manner, when the first voice data corresponds to a single candidate user intent, output associated information associated with the candidate user intent.

In a possible design manner, the determining the candidate user intent corresponding to the first voice data based on the intent recognition model includes:

Obtain the historical user intent determined during the voice interaction process;

According to the first voice data and the historical user intent, the intent recognition model is used to determine a candidate user intent corresponding to the first voice data.

In a possible design manner, determining the candidate user intent corresponding to the first voice data by using the intent recognition model according to the first voice data and the historical user intent includes:

Determine whether the first voice data and the historical user intent belong to the same dialogue sequence based on the dialogue state tracking model;

When the first voice data and the historical user intent belong to the same dialogue sequence, use the intent recognition model to determine the initial user intent corresponding to the first voice data, and update the first voice data according to the historical user intent an initial user intent corresponding to the voice data, obtaining a candidate user intent corresponding to the first voice data;

When the first voice data and the historical user intent do not belong to the same dialogue sequence, the intent recognition model is used to determine a candidate user intent corresponding to the first voice data.

In a possible design manner, the generating a query sentence according to the multiple candidate user intentions includes:

According to each candidate user intent corresponding to the first voice data, the score of each output module in the dialogue strategy learning model is determined; the dialogue strategy learning model includes at least one of the following output modules: a rewriting module, which refers to a resolution module , vertical domain intent analysis module, task multi-round response module, question and answer module, news search module, chat module, recommendation module and candidate intent analysis module;

When the output module with the highest score in the dialogue strategy learning model is the candidate intent parsing module, the query sentence is generated according to the score of each candidate user intent.

In a possible design manner, the generating the query sentence according to the scores of each of the candidate user intentions includes:

When the score of the first candidate user intent corresponding to the first voice data is greater than a first preset threshold, the score of the second candidate user intent corresponding to the first voice data is greater than a second preset threshold, and the first When the difference between the score of the candidate user intent and the score of the second candidate user intent is less than a preset interval threshold, the query sentence is generated based on the first candidate user intent and the second candidate user intent; wherein, the The first candidate user intent is the candidate user intent with the highest score corresponding to the first voice data, the second candidate user intent is the candidate user intent with the second highest score corresponding to the first voice data, and the first candidate user intent The preset threshold is greater than the second preset threshold.

In the display device and the voice interaction method provided by the embodiment of the present application, when the first voice data input by the user is received, the candidate user intent corresponding to the first voice data is determined; when the first voice data corresponds to multiple candidate user intents , generate a query statement according to multiple candidate user intentions, and output the query statement to the user, which is used to prompt the user to select a user intention from the above-mentioned multiple candidate user intentions; The voice data determines a target user intent corresponding to the first voice data among the plurality of candidate user intents; and outputs response information associated with the target user intent. In this application, when there are two or more candidate user intentions corresponding to the user input sentence, the voice interaction system actively feeds back the query sentence to the user through an anthropomorphic interactive way, and then according to the user input response sentence Determining the real intention of the user can effectively improve the accuracy of understanding the user's intention in the process of voice interaction.

Description of drawings

FIG. 1 exemplarily shows a schematic diagram of an operation scene between a display device and a control device according to an embodiment;

FIG. 2 exemplarily shows a block diagram of the hardware configuration of the display device 200 according to the exemplary embodiment;

FIG. 3 exemplarily shows a block diagram of the configuration of the control device 1001 according to an exemplary embodiment;

4 is a schematic diagram of a software system of a display device provided by the present application;

5 is a schematic diagram of an application program that can be provided by the display device provided by the present application;

6 is a schematic diagram of an application of a display device in a voice interaction scenario;

FIG. 7 is a schematic flowchart of a display device being applied to a voice interaction scenario;

FIG. 8 is a schematic diagram of an application scenario exemplarily shown in an embodiment of the present application;

FIG. 9 is another schematic flowchart of a display device applied in a voice interaction scenario;

FIG. 10 is a schematic diagram of an identification model issued by a supplier of an identification model;

Fig. 11 is a kind of schematic flow chart that the server 400 obtains the recognition model;

Fig. 12 is a kind of schematic flow chart that the server updates the recognition model;

13 is a schematic flowchart 1 of a voice interaction method provided in an embodiment of the application;

14 is a second schematic flowchart of a voice interaction method provided in an embodiment of the application;

15a to 15d are schematic diagrams of voice interaction of a display device in an embodiment of the present invention;

16a to 16d are schematic diagrams of another voice interaction of a display device in an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application. Furthermore, although the disclosures in this application have been presented in terms of illustrative example or instances, it should be understood that various aspects of this disclosure may also constitute a complete embodiment in isolation.

It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the embodiments described below, rather than intended to limit the embodiments of the present application. Unless otherwise specified, these terms are to be understood according to their ordinary and ordinary meanings.

The terms "first", "second" and the like in the description and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence. unless otherwise noted. It should be understood that the terms so used are interchangeable under appropriate circumstances, eg, can be implemented in an order other than those given in the illustration or description of the embodiments of the present application.

Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover but not exclusively include, for example, a product or device incorporating a series of components is not necessarily limited to those explicitly listed, but may include No other components are expressly listed or inherent to these products or devices.

The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware or/and software code capable of performing the function associated with that element.

As used in this application, the term "remote control" refers to a component of an electronic device, such as the display device disclosed in this application, that can wirelessly control the electronic device, usually over a short distance. Generally, infrared and/or radio frequency (RF) signals and/or Bluetooth are used to connect with electronic devices, and functional modules such as WiFi, wireless USB, Bluetooth, and motion sensors may also be included. For example, a hand-held touch remote control replaces most of the physical built-in hard keys in a general remote control device with a user interface in a touch screen.

FIG. 1 exemplarily shows a schematic diagram of an operation scenario between a display device and a control apparatus according to an embodiment. As shown in FIG. 1 , the user can operate the display apparatus 200 through the mobile terminal 1002 and the control apparatus 1001 .

In some embodiments, the control device 1001 may be a remote control, and the communication between the remote control and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, etc., and controls the display device 200 by wireless or other wired methods. The user can control the display device 200 by inputting user instructions through keys on the remote control, voice input, control panel input, and the like. For example, the user can control the display device 200 by inputting corresponding control commands through the volume up/down key, channel control key, up/down/left/right movement keys, voice input key, menu key, power-on/off key, etc. on the remote control. function.

In some embodiments, mobile terminals, tablet computers, computers, notebook computers, and other smart devices may also be used to control the display device 200 . For example, the display device 200 is controlled using an application running on the smart device. The app can be configured to provide users with various controls in an intuitive user interface (UI) on the screen associated with the smart device.

In some embodiments, the mobile terminal 1002 can install a software application with the display device 200 to realize connection communication through a network communication protocol, so as to achieve the purpose of one-to-one control operation and data communication. For example, a control command protocol can be established between the mobile terminal 1002 and the display device 200, the remote control keyboard can be synchronized to the mobile terminal 1002, and the function of controlling the display device 200 can be realized by controlling the user interface on the mobile terminal 1002. It is also possible to transmit the audio and video content displayed on the mobile terminal 1002 to the display device 200 to implement a synchronous display function.

As also shown in FIG. 1 , the display device 200 also performs data communication with the server 400 through various communication methods. The display device 200 may be allowed to communicate via local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 . For example, the display device 200 interacts by sending and receiving information, and electronic program guide (EPG), receiving software program updates, or accessing a remotely stored digital media library. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers. Other network service contents such as video-on-demand and advertising services are provided through the server 400 .

The display device 200 may be a liquid crystal display, an OLED display, or a projection display device. The specific display device type, size and resolution are not limited. Those skilled in the art can understand that the display device 200 can make some changes in performance and configuration as required.

The display device 200 may additionally provide a smart IPTV function that provides computer-supported functions, including but not limited to, IPTV, smart TV, Internet Protocol TV (IPTV), and the like, in addition to the broadcast receiving TV function.

FIG. 2 exemplarily shows a block diagram of the hardware configuration of the display device 200 according to the exemplary embodiment.

In some embodiments, the display device 200 includes a controller 250 , a tuner 210 , a communicator 220 , a detector 230 , an input/output interface 255 , a display screen 275 , an audio output interface 285 , a memory 260 , and a power supply 290 , at least one of the user interface 265 and the external device interface 240 .

In some embodiments, the display screen 275 is used to receive the image signal from the output of the first processor, and to perform components for displaying video content and images and a menu manipulation interface.

In some embodiments, the display screen 275 includes a display screen component for presenting a picture, and a driving component for driving image display.

In some embodiments, the video content displayed may be from broadcast television content or various broadcast signals that may be received via wired or wireless communication protocols. Alternatively, various image contents sent from the network server side can be displayed and received from the network communication protocol.

In some embodiments, display screen 275 is used to present a user-manipulated UI interface generated in display device 200 and used to control display device 200 .

In some embodiments, depending on the type of display screen 275, a driving component for driving the display is also included.

In some embodiments, the display screen 275 is a projection display screen, and may also include a projection device and a projection screen.

In some embodiments, communicator 220 is a component for communicating with external devices or external servers according to various communication protocol types. For example, the communicator may include at least one of a Wifi chip, a Bluetooth communication protocol chip, a wired Ethernet communication protocol chip and other network communication protocol chips or a near field communication protocol chip, and an infrared receiver.

In some embodiments, the display device 200 may establish control signal and data signal transmission and reception between the communicator 220 and the external control device 100 or the content providing device.

In some embodiments, the user interface 265 may be used to receive infrared control signals from a control device (eg, an infrared remote control, etc.).

In some embodiments, the detector 230 is a signal used by the display device 200 to collect the external environment or interact with the outside.

In some embodiments, the detector 230 includes a light receiver, a sensor for collecting ambient light intensity, and can adaptively display parameter changes and the like by collecting ambient light.

In some embodiments, the detector 230 may also include an image collector, such as a camera, a camera, etc., which can be used to collect external environment scenes, and used to collect user attributes or interactive gestures with the user, and can adaptively change display parameters, User gestures can also be recognized to implement functions that interact with users.

In some embodiments, detector 230 may also include a temperature sensor or the like, such as by sensing ambient temperature.

In some embodiments, the display device 200 can adaptively adjust the display color temperature of the image. For example, when the temperature is relatively high, the display device 200 can be adjusted to display a relatively cool color temperature of the image, or when the temperature is relatively low, the display device 200 can be adjusted to display a warmer color of the image.

In some embodiments, the detector 230 may also be a sound collector or the like, such as a microphone, which may be used to receive the user's voice. Exemplarily, a voice signal including a control instruction of the user to control the display device 200, or collecting ambient sounds, is used to identify the type of the environment scene, so that the display device 200 can adaptively adapt to the ambient noise.

In some embodiments, as shown in FIG. 2 , the input/output interface 255 is configured to enable data transfer between the controller 250 and other external devices or other controllers 250 . Such as receiving video signal data and audio signal data of external equipment, or command instruction data, etc.

In some embodiments, the external device interface 240 may include, but is not limited to, the following: any one or more of a high-definition multimedia interface HDMI interface, an analog or data high-definition component input interface, a composite video input interface, a USB input interface, an RGB port, etc. interface. It is also possible to form a composite input/output interface by a plurality of the above-mentioned interfaces.

In some embodiments, as shown in FIG. 2 , the tuner and demodulator 210 is configured to receive broadcast television signals through wired or wireless reception, and can perform modulation and demodulation processing such as amplification, frequency mixing, and resonance, and can perform modulation and demodulation processing from multiple wireless receivers. Or demodulate the audio and video signal from the cable broadcast TV signal, the audio and video signal may include the TV audio and video signal carried in the frequency of the TV channel selected by the user, and the EPG data signal.

In some embodiments, the frequency demodulated by the tuner-demodulator 210 is controlled by the controller 250, and the controller 250 can send a control signal according to the user's selection, so that the modem responds to the user-selected TV signal frequency and modulates and demodulates the frequency. The frequency carried by the television signal.

In some embodiments, broadcast television signals may be classified into terrestrial broadcast signals, cable broadcast signals, satellite broadcast signals, or Internet broadcast signals, etc. according to different broadcast formats of the television signals. Or according to different modulation types, it can be divided into digital modulation signal, analog modulation signal, etc. Or, it can be divided into digital signals, analog signals, etc. according to different types of signals.

In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box Wait. In this way, the set-top box outputs the modulated and demodulated television audio and video signals of the received broadcast television signals to the main device, and the main device receives the audio and video signals through the first input/output interface.

In some embodiments, the controller 250, through various software control programs stored on the memory, controls the operation of the display device and responds to user operations. The controller 250 may control the overall operation of the display apparatus 200 . For example, in response to receiving a user command for selecting a UI object to be displayed on the display screen 275, the controller 250 may perform an operation related to the object selected by the user command.

As shown in FIG. 2 , the controller 250 includes a random access memory 251 (Random Access Memory, RAM), a read-only memory 252 (Read-Only Memory, ROM), a video processor 270, an audio processor 280, and other processors 253 (For example: at least one of a graphics processing unit (Graphics Processing Unit, GPU), a central processing unit 254 (Central Processing Unit, CPU), a communication interface (Communication Interface), and a communication bus 256 (Bus). Wherein, the communication bus connects various parts.

In some embodiments, RAM 251 is used to store temporary data for the operating system or other running programs

In some embodiments, ROM 252 is used to store various system startup instructions.

In some embodiments, ROM 252 is used to store a basic input output system, referred to as a Basic Input Output System (BIOS). It is used to complete the power-on self-check of the system, the initialization of each functional module in the system, the driver program of the basic input/output of the system, and the boot operating system.

In some embodiments, when a power-on signal is received, the power supply of the display device 200 starts, and the CPU executes the system start-up instruction in the ROM 252, and copies the temporary data of the operating system stored in the memory to the RAM 251, so as to facilitate the start-up or running operation system. After the startup of the operating system is completed, the CPU copies the temporary data of various application programs in the memory to the RAM 251, so as to facilitate the startup or operation of various application programs.

In some embodiments, the CPU processor 254 executes operating system and application program instructions stored in memory. And various application programs, data and content are executed according to various interactive instructions received from the external input, so as to finally display and play various audio and video content.

In some exemplary embodiments, CPU processor 254 may include multiple processors. The plurality of processors may include a main processor and one or more sub-processors. The main processor is used to perform some operations of the display device 200 in the pre-power-on mode, and/or an operation of displaying a picture in the normal mode. One or more sub-processors for an operation in a state such as standby mode.

In some embodiments, the graphics processor 253 is used to generate various graphic objects, such as: icons, operation menus, and user input instructions to display graphics and the like. It includes an operator, which performs operations by receiving various interactive instructions input by the user, and displays various objects according to the display properties. and includes a renderer, which renders various objects obtained based on the operator, and the rendered objects are used for displaying on the display screen.

In some embodiments, the video processor 270 is configured to receive the external video signal and perform decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc. according to the standard codec protocol of the input signal. After video processing, a signal that can be directly displayed or played on the display device 200 can be obtained.

In some embodiments, the video processor 270 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, and the like.

Wherein, the demultiplexing module is used for demultiplexing the input audio and video data stream. For example, if MPEG-2 is input, the demultiplexing module demultiplexes it into video signals and audio signals.

The video decoding module is used to process the demultiplexed video signal, including decoding and scaling.

The image synthesizing module, such as an image synthesizer, is used for superimposing and mixing the GUI signal generated by the graphics generator according to the user's input or itself, and the zoomed video image, so as to generate an image signal that can be displayed.

The frame rate conversion module is used to convert the input video frame rate, such as converting 60Hz frame rate to 120Hz frame rate or 240Hz frame rate. The usual format is implemented by means of frame insertion.

The display formatting module is used for converting the received frame rate into the video output signal, and changing the signal to conform to the display format signal, such as outputting the RGB data signal.

In some embodiments, the graphics processor 253 may be integrated with the video processor, or may be separately configured. When integrated, the graphics signal processing output to the display screen may be performed, and when separated, different functions may be performed separately. , such as GPU+FRC (Frame Rate Conversion)) architecture.

In some embodiments, the audio processor 280 is configured to receive an external audio signal, perform decompression and decoding, and noise reduction, digital-to-analog conversion, and amplification processing according to a standard codec protocol of the input signal to obtain a The sound signal played in the speaker.

In some embodiments, the video processor 270 may comprise one or more chips. The audio processor may also include one or more chips.

In some embodiments, the video processor 270 and the audio processor 280 may be separate chips, or may be integrated into one or more chips together with the controller.

The power supply 290, under the control of the controller 250, provides power supply support for the display device 200 with the power input from the external power supply. The power supply 290 may include a built-in power supply circuit installed inside the display device 200 , or may be an external power supply installed in the display device 200 to provide an external power supply interface in the display device 200 .

The user interface 265 is used for receiving user input signals, and then sending the received user input signals to the controller 250 . The user input signal may be a remote control signal received through an infrared receiver, and various user control signals may be received through a network communication module.

In some embodiments, the user inputs user commands through the control device or the mobile terminal, the user input interface is based on the user's input, and the display device 200 responds to the user's input through the controller 250 .

In some embodiments, the user may input user commands on a graphical user interface (GUI) displayed on the display 275, and the user input interface receives the user input commands through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

In some embodiments, a "user interface" is a medium interface for interaction and information exchange between an application program or an operating system and a user, which enables conversion between an internal form of information and a form acceptable to the user. A commonly used form of user interface is a graphical user interface (Graphic User Interface, GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be an icon, window, control and other interface elements displayed on the display screen of the electronic device, wherein the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. visual interface elements.

The memory 260 includes storing various software modules for driving the display device 200 . For example, various software modules stored in the first memory include at least one of a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules.

The basic module is used for signal communication between various hardwares in the display device 200, and is a low-level software module that sends processing and control signals to the upper-layer module. The detection module is a management module used to collect various information from various sensors or user input interfaces, perform digital-to-analog conversion, and analyze and manage.

For example, the speech recognition module includes a speech parsing module and a speech instruction database module. The display control module is a module used to control the display to display image content, and can be used to play information such as multimedia image content and UI interface. Communication module, a module for control and data communication with external devices. The browser module is a module for performing data communication between browsing servers. Service modules are used to provide various services and modules including various applications. At the same time, the memory 260 is also used to store and receive external data and user data, images of various items in various user interfaces, and visual effect diagrams of focus objects, and the like.

FIG. 3 exemplarily shows a block diagram of the configuration of the control device 1001 according to an exemplary embodiment. As shown in FIG. 3, the control device 1001 includes a controller 110, a communication interface 130, a user input/output interface, a memory, and a power supply.

The control device 1001 is configured to control the display device 200 , and can receive the user's input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, so as to play an interactive role between the user and the display device 200 . For example, the user operates the channel addition and subtraction keys on the control device 1001, and the display device 200 responds to the channel addition and subtraction operations.

In some embodiments, the control device 1001 may be a smart device. For example, the control device 1001 can install various applications for controlling the display device 200 according to user requirements.

In some embodiments, as shown in FIG. 1 , the mobile terminal 1002 or other intelligent electronic device can perform a similar function of controlling the device 1001 after installing the application for operating the display device 200 . For example, the user can install the application, various function keys or virtual buttons of the graphical user interface available on the mobile terminal 1002 or other intelligent electronic devices, so as to realize the function of controlling the physical keys of the device 1001 .

The controller 110 includes a processor 112 and RAM 113 and ROM 114, a communication interface 130, and a communication bus. The controller is used to control the operation and operation of the control device 1001, as well as communication and cooperation between internal components and external and internal data processing functions.

The communication interface 130 realizes the communication of control signals and data signals with the display device 200 under the control of the controller 110 . For example, the received user input signal is sent to the display device 200 . The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131 , a Bluetooth module 132 , and an NFC module 133 .

The user input/output interface 140, wherein the input interface includes at least one of other input interfaces such as a microphone 141, a touch panel 142, a sensor 143, and a key 144. For example, the user can implement the user command input function through actions such as voice, touch, gesture, pressing, etc. The input interface converts the received analog signal into a digital signal, and converts the digital signal into a corresponding command signal, and sends it to the display device 200.

The output interface includes an interface for transmitting received user instructions to the display device 200 . In some embodiments, it can be an infrared interface or a radio frequency interface. For example, when the infrared signal interface is used, the user input instruction needs to be converted into an infrared control signal according to the infrared control protocol, and sent to the display device 200 through the infrared sending module. For another example, when a radio frequency signal interface is used, the user input command needs to be converted into a digital signal, and then modulated according to the radio frequency control signal modulation protocol, and then sent to the display device 200 by the radio frequency transmission terminal.

In some embodiments, the control device 1001 includes at least one of a communication interface 130 and an input-output interface 140 .

The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 1001 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

The power supply 180 is used to provide operating power support for each element of the control device 1001 under the control of the controller. Can battery and related control circuit.

In some embodiments, a system may include a kernel (Kernel), a command parser (shell), a file system, and applications. Together, the kernel, shell, and file system make up the basic operating system structures that allow users to manage files, run programs, and use the system. After power-on, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, etc., runs and maintains virtual memory, scheduler, signals and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. An application is compiled into machine code after startup, forming a process.

FIG. 4 is a schematic diagram of a software system of a display device provided by the present application. Referring to FIG. 4 , in some embodiments, the system is divided into four layers, which are respectively an application layer (referred to as “application layer”) from top to bottom. , the Application Framework layer (referred to as the "framework layer"), the Android runtime (Android runtime) and the system library layer (referred to as the "system runtime layer"), and the kernel layer.

In some embodiments, at least one application program runs in the application program layer, and these application programs may be a Window program, a system setting program, a clock program, a camera application, etc. built into the operating system; they may also be developed by a third party The application programs developed by the author, such as the Hijian program, the K song program, the magic mirror program, etc. During specific implementation, the application package in the application layer is not limited to the above examples, and may actually include other application packages, which is not limited in this embodiment of the present application.

The framework layer provides an application programming interface (application programming interface, API) and a programming framework for the applications of the application layer. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center, which decides to let the applications in the application layer take action. Through the API interface, the application can access the resources in the system and obtain the services of the system during execution

As shown in FIG. 4 , the application framework layer in the embodiment of the present application includes managers (Managers), content providers (Content Provider), etc., wherein the manager includes at least one of the following modules: the activity manager (ActivityManager) is used with Interact with all activities running in the system; Location Manager is used to provide system services or applications with access to system location services; Package Manager is used to retrieve files currently installed on the device. Various information related to the application package; Notification Manager (NotificationManager) is used to control the display and clearing of notification messages; Window Manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers and desktop components on the user interface .

In some embodiments, the activity manager is used to: manage the life cycle of each application and the usual navigation and fallback functions, such as controlling the exit of the application (including switching the user interface currently displayed in the display window to the system desktop), opening the , back (including switching the currently displayed user interface in the display window to the upper-level user interface of the currently displayed user interface), and the like.

In some embodiments, the window manager is used to manage all window programs, such as obtaining the size of the display screen, judging whether there is a status bar, locking the screen, taking screenshots, and controlling the change of the display window (for example, reducing the display window to display, shaking display, twisting deformation display, etc.), etc.

In some embodiments, the system runtime layer provides support for the upper layer, that is, the framework layer. When the framework layer is used, the Android operating system will run the C/C++ library included in the system runtime layer to implement the functions to be implemented by the framework layer.

In some embodiments, the kernel layer is the layer between hardware and software. As shown in Figure 4, the kernel layer at least includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, touch sensors, pressure sensors, etc.), etc.

In some embodiments, the kernel layer further includes a power driver module for power management.

In some embodiments, software programs and/or modules corresponding to the software architecture in FIG. 4 are stored in the first memory or the second memory shown in FIG. 2 or FIG. 3 .

In some embodiments, taking the magic mirror application (photography application) as an example, when the remote control receiving device receives the input operation of the remote control, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the input operation into the original input event (including the value of the input operation, the timestamp of the input operation and other information). Raw input events are stored at the kernel layer. The application framework layer obtains the original input event from the kernel layer, identifies the control corresponding to the input event according to the current position of the focus, and regards the input operation as a confirmation operation, and the control corresponding to the confirmation operation is the control of the magic mirror application icon. The mirror application calls the interface of the application framework layer, starts the mirror application, and then starts the camera driver by calling the kernel layer to capture still images or videos through the camera.

In some embodiments, for a display device with a touch function, taking a split-screen operation as an example, the display device receives an input operation (such as a split-screen operation) performed by the user on the display screen, and the kernel layer can generate corresponding input operations according to the input operation. Enter an event and report the event to the application framework layer. The window mode (such as multi-window mode) and window position and size corresponding to the input operation are set by the activity manager of the application framework layer. The window management of the application framework layer draws the window according to the settings of the activity manager, and then sends the drawn window data to the display driver of the kernel layer, and the display driver displays the corresponding application interface in different display areas of the display screen.

In some embodiments, FIG. 5 is a schematic diagram of an application program that can be provided by the display device provided by the present application. As shown in FIG. 5 , the application program layer includes at least one application program that can display corresponding icon controls on the display, such as: Live TV application icon control, video on demand application icon control, media center application icon control, application center icon control, game application icon control, etc.

In some embodiments, the live TV application may provide live TV from different sources. For example, a live TV application may provide a TV signal using input from cable, over-the-air, satellite services, or other types of live TV services. And, the live TV application may display the video of the live TV signal on the display device 200 .

In some embodiments, a video-on-demand application may provide video from various storage sources. Unlike live TV applications, video-on-demand provides a display of video from certain storage sources. For example, video-on-demand can come from the server side of cloud storage, from local hard disk storage containing existing video programs.

In some embodiments, the media center application may provide various multimedia content playback applications. For example, a media center may provide services other than live TV or video-on-demand, where users can access various images or audio through a media center application.

In some embodiments, the application center may provide storage of various applications. An application can be a game, an application, or some other application that is related to a computer system or other device but can be run on a Smart TV. The application center can obtain these applications from various sources, store them in local storage, and then run them on the display device 200 .

More specifically, in some embodiments, any of the aforementioned display devices 200 in the present application may have the function of voice interaction, so as to improve the intelligence of the display device 200 and improve the user experience of the display device 200 .

In some embodiments, FIG. 6 is a schematic diagram of an application of a display device in a voice interaction scenario, in which the user 1 can speak the command that the display device 200 wishes to execute through voice, then the display device 200 can collect voice data in real time, and Identify the instruction of user 1 included in the voice data, and directly execute the instruction after recognizing the instruction of user 1. During the whole process, user 1 does not actually operate the display device 200 or other devices, but simply commanded.

In some embodiments, when the display device 200 shown in FIG. 2 is applied in the scenario shown in FIG. 6 , the display device 200 can collect voice data in real time through its sound collector 231 , and then the sound collector 231 will collect voice data in real time. The obtained voice data is sent to the controller 250, and finally the controller 250 recognizes the instructions included in the voice data.

In some embodiments, FIG. 7 is a schematic flowchart of the application of the display device in the voice interaction scenario, which can be executed by the device in the scenario shown in FIG. 6 . Specifically, in S11, the sound collector 231 in the display device 200 is real-time The voice data in the surrounding environment where the display device 200 is located is collected, and the collected voice data is sent to the controller 250 for identification.

In some embodiments, in S12 as shown in FIG. 7 , after receiving the voice data, the controller 250 recognizes the instructions included in the voice data. For example, if the voice data includes an instruction of “increase the brightness” from the user 1, after the controller 250 recognizes the instruction included in the voice data, the controller 250 can execute the identified instruction to control the display 275 increases brightness. It can be understood that, in this case, the controller 250 recognizes each received voice data, and it may happen that there is no instruction in the recognized voice data.

In other embodiments, the model based on instruction recognition is relatively large and the operation efficiency is low, and it can also be specified that user 1 adds a keyword before speaking the instruction, such as "ABCD", then the user needs to say "ABCD, increase the “Brightness” instruction, so that in S12 as shown in FIG. 7 , after receiving the voice data, the controller 250 firstly identifies whether there is a keyword of “ABCD” in each voice data, and after identifying that there is a keyword After that, the instruction recognition model is used to identify the specific instruction corresponding to "increase the brightness" in the speech data.

In some embodiments, after receiving the speech data, the controller 250 may further perform denoising on the speech data, including removing echoes and ambient noise, processing it into clean speech data, and recognizing the processed speech data.

In some embodiments, FIG. 7 is a schematic diagram of another application of the display device in a voice interaction scenario, wherein the display device 200 can be connected to the server 400 through the Internet, and after the display device 200 collects the voice data, the voice data can be Sent to the server 400 through the Internet, the server 400 recognizes the instructions included in the voice data, and sends the recognized instructions back to the display device 200, so that the display device 200 can directly execute the received instructions. Compared with the scene shown in FIG. 6 , this scenario reduces the requirement on the computing power of the display device 200 , and a larger recognition model can be set on the server 400 to further improve the accuracy of command recognition in the voice data.

In some embodiments, when the display device 200 shown in FIG. 2 is applied in the scenario shown in FIG. 6 , the display device 200 can collect voice data in real time through its sound collector 231 , and then the sound collector 231 will collect voice data in real time. The obtained voice data is sent to the controller 250, and the controller 250 sends the voice data to the server 400 through the communicator 220. After the server 400 recognizes the instructions included in the voice data, the display device 200 receives the server 400 through the communicator 220. sent instructions, and finally the received instructions are executed by the controller 250 .

In some embodiments, FIG. 9 is another schematic flowchart of the application of the display device in the voice interaction scenario, which can be executed by the device in the scenario as shown in FIG. 8 , wherein, in S21 , the sound collection in the display device 200 is displayed. The device 231 collects the voice data in the surrounding environment where the display device 200 is located in real time, and sends the collected voice data to the controller 250. The controller 250 further sends the voice data to the server 400 through the communicator 220 in S22, and the server The instruction included in the voice data is recognized in S23, and then, the server 400 sends the recognized instruction back to the display device 200 in S24. Correspondingly, the display device 200 receives the instruction through the communicator 220 and sends it to the controller 250, and finally controls the The controller 250 may directly execute the received instructions.

In some embodiments, in S23 shown in FIG. 7 , after receiving the voice data, the server 400 recognizes the instructions included in the voice data. For example, the voice data includes an instruction of "increase the brightness" given by the user 1 . However, since the model of instruction recognition is relatively large, and the server 400 recognizes each received voice data, it may happen that there is no instruction in the recognized voice data. Therefore, in order to reduce the invalid recognition performed by the server 400 and reduce the number of display devices 200 and servers The amount of communication interaction data between 400 and 400, in the specific implementation, it can also be specified that user 1 adds a keyword before speaking the command, such as "ABCD", then the user needs to speak the command "ABCD, increase brightness", then, In S22, the controller 250 of the display device 200 first identifies whether the keyword "ABCD" exists in the speech data through a keyword recognition model with a smaller model and a lower computational load. If no keyword is identified in the voice data, the controller 250 will not send the voice data to the server 400; if a keyword is identified in the voice data currently being processed by the controller 250, the controller 250 will then send all the voice data to the server 400. , or the part after the keyword in the speech data is sent to the server 400, and the server 400 recognizes the received speech data. Since the voice data received by the controller 250 includes keywords at this time, the voice data sent to the server 400 for recognition is more likely to include the user's instruction, so the invalid recognition calculation of the server 400 can be reduced, and the display can be reduced. Invalid communication between device 200 and server 400.

In some embodiments, in order to enable the display device 200 to have the function of recognizing instructions in the voice data in a specific scenario as shown in FIG. 6 , or to enable the display device 200 to have the function as shown in FIG. 6 or FIG. 8 In a specific scenario of , the recognition function of keywords in the voice data, as a supplier of the voice interaction function of the display device 200, also needs to make a machine learning model that can be used to identify instructions or identify keywords, such as textcnn, transform, etc. Deep learning models. These models are stored in the display device 200 and used by the display device 200 when performing identification.

In some embodiments, FIG. 10 is a schematic diagram of an identification model issued by a supplier of the identification model, wherein, after the server 400 set by the supplier obtains the identification model (which may be an instruction identification model or a keyword identification model), The recognition model may be sent to each display device 200 . The process shown in FIG. 10 may be performed during the production of the display device 200, and the server 400 sends the server to each display device 200; or, after the display device 200 starts to use, the server 400 will identify The model is sent to the display device 200 .

In some embodiments, the server 400 may obtain the recognition model by collecting speech data and learning based on the machine learning model. For example, FIG. 11 is a schematic flowchart of a recognition model obtained by the server 400, wherein, in S31, each display device (take display device 1-display device N, N as an example) collects voice data 1-N, In S32, the collected voice data 1-N are sent to the server 400. Subsequently, in S33, the staff of the supplier can mark each voice data and the instructions or keywords included in the voice data by manually marking, and then use the voice data itself and the marking information corresponding to the voice data as The data is sent into the machine learning model, and the server learns. When the learned recognition model is used later, when a voice data to be recognized is input, the recognition model compares the voice data with the learned voice data, and then The probability of each label information is output, and the label information corresponding to the final maximum probability can be used as the recognition result of the speech data to be recognized. In S34, the server 400 may send the calculated recognition model to each display device.

In some embodiments, the server 400 may not use the voice data actually collected by the display devices 1-N in the embodiment shown in FIG. 11 to calculate the recognition model, but may directly input different voice data by the staff, and each The annotation information of each voice data is sent to each display device after the recognition model is obtained by calculation.

In some embodiments, as shown in FIG. 11 , the display device 1-N that collects voice data and sends it to the server, and the server calculates the recognition model and sends it to the display device 1-N can be two independent processes, That is to say, in S32, the server receives the voice data collected by the N display devices, and in S34, the server can send the trained recognition model to the other N display devices. The N display devices in the two processes may be the same or different, or may be partially the same.

In some embodiments, due to the limited number of samples used when obtaining the recognition model, it is impossible for the recognition model set by the display device 200 to achieve 100% accurate recognition. Therefore, the supplier can also use the server 400 at any time. The voice data collected by each display device 200 during actual use is collected, and the recognized recognition model is updated according to the collected voice data, so as to further improve the recognition accuracy of the recognition model.

For example, FIG. 12 is a schematic flowchart of the server updating the recognition model. It can be understood that, before the embodiment shown in FIG. 12 is executed, each display device is set as shown in FIG. Identify the model. Then in S31 as shown in FIG. 12 , each display device (taking display device 1-display device N, a total of N as an example) collects voice data 1-N, and sends the collected voice data 1-N in S32. to server 400. Subsequently, in S33, the staff of the supplier can mark each voice data and the instructions or keywords included in the voice data by manually marking, and then use the voice data itself and the marking information corresponding to the voice data as The data is sent into the machine learning model, and the server updates the recognition model that has been calculated according to the new voice data received, and in S34, the server 400 can re-send the updated recognition model to each display device 200, This enables each display device 200 to be updated using the updated recognition model. Among them, for any one of the N display devices, since the new learning model adopts the voice data collected by the display device 200, it can effectively improve the follow-up response to the voice data collected by the display device 200. The accuracy of the identification of the data.

In some embodiments, each display device as shown in FIG. 12 can send the voice data to the server after receiving it, or send the voice data collected during the fixed period of time after the end of the fixed period of time. To the server, or, after collecting a certain amount of voice data, send it to the server uniformly, or, according to the instructions of the user of the display device, or according to the instructions of the staff of the server, the received voice can be sent to the server .

In some embodiments, the N display devices shown in FIG. 12 may simultaneously send voice data to the server at the same appointed time, and the server updates the recognition model according to the N voice data received; or, N Each display device can also send voice data to the server respectively, and after the server receives more than N voice data, the server can start to update the recognition model according to the received voice data.

When building a voice interaction system, in addition to the speech recognition and text-to-speech modules, four layers of processing capability are required. The first is the basic feature processing layer, which mainly includes word segmentation, semantic labeling, emotion recognition, etc.; the second layer is the intent understanding layer, which is used to convert user questions into machine-understandable and structured user intent representations. At this time, the user There may be many possibilities for the intent; the third layer is the dialog management layer, which is used to further clarify the user's intent based on the context information, update the dialog state, and make decisions according to the dialog strategy. This process will call the data provided by the fourth layer of business service layer. Services (such as access to media assets) and response plans. For example, for a search of the same title, if the search result of music is obviously better than that of film and television, the voice interaction system will feed back the search result of music to the user after obtaining the feedback information from the business service layer.

In some embodiments, after receiving the speech input by the user, the display device generally performs speech recognition on the speech input by the user, determines the user's intent through a semantic understanding engine, and then provides the user with related services according to the user's intent. Among them, the current semantic understanding engine performs multiple rounds of interaction and omission completion in specific scenarios through limited domain multi-round modeling and referential resolution modeling. The results of the semantic processing module make sorting decisions to complete the positioning of user intentions. However, in more conventional fields, factors such as dialogue history and user scenarios are not considered, and in the face of cross-business decisions or the user's intentions are ambiguous, accurate intention understanding and positioning decisions cannot be made.

In order to solve the above technical problems, an embodiment of the present application provides a display device, when the user input sentence corresponds to two or more candidate user intentions, the display device actively feeds back to the user in an anthropomorphic interactive manner Inquiry sentence, and then determine the user's true intention according to the response sentence input by the user, which can effectively improve the accuracy of the user's intention understanding in the process of voice interaction. Detailed description is given below by using detailed embodiments.

Based on the display device 200 described in the foregoing embodiment, in a feasible implementation manner, the voice acquisition device of the display device 200 may be a microphone array corresponding to the display device 200, or the voice acquisition device may also be a display device 200 corresponds to the microphone on the control device.

In this embodiment, after collecting the user's voice data, the above-mentioned voice collecting device sends the collected voice data to the audio processor in the display device, and the audio processor preprocesses the voice data and sends it to the display device in the controller.

After receiving the first voice data input by the user, the above-mentioned controller determines the candidate user intent corresponding to the first voice data based on the intent recognition model.

In some embodiments, the above-mentioned intent recognition model can classify sentences or queries into corresponding intent categories by means of classification. Among them, different user intentions will correspond to different domain dictionaries, such as book titles, song titles, commodity names, and so on. Judgment can be made according to the degree of matching or coincidence between the query and the dictionary. In which domain the query and the dictionary have a high degree of coincidence, the query is judged to which domain.

For example, when the query entered by the user is "song of the sunset", the intent of the query is music intent, and when the query entered by the user is "news broadcast", the intent of the query is news search intent.

In some embodiments, the query input by the user may correspond to various user intentions. For example, when the user queries "Resident Evil", because "Resident Evil" not only has related games, but also related movies, in order to be more accurate To locate the user's intent, in a feasible implementation manner of the present application, when the query input by the user corresponds to multiple user intents, a query sentence is generated according to multiple determined candidate user intents, such as "Do you want to watch a movie, or Want to play a game?", then send the query sentence to the display screen for display, and/or send it to the speaker for playback, and continue to collect the user's voice data.

Exemplarily, when the user's voice data is subsequently collected as "watching a movie", a movie related to "Resident Evil" is searched and pushed to the display screen for display or playback; when the user's voice data collected subsequently is " When playing games", search for "Resident Evil" related games, and push them to the display for display, or directly start the game.

That is, the display device provided in the embodiment of the present application, when faced with cross-service decisions or ambiguous user intentions, actively interacts with the user through an anthropomorphic interaction mode, and then determines the user's status according to the conversation with the user. Real intent can save users' search clicks, shorten search time, and greatly improve the accuracy of user intent understanding in the process of voice interaction.

Based on the descriptions in the above embodiments, in some embodiments of the present application, after receiving the first voice data, the controller of the above display device may also acquire the historical user intent determined during the voice interaction, and then, according to the first voice data With the above historical user intent, the intent recognition model is used to determine the candidate user intent corresponding to the first voice data. For example, for the same query sentence, if the user has been searching for news media resources for a period of time, the query should be understood as a news search service rather than an encyclopedia service.

Among them, the above-mentioned intent recognition model can handle the recognition of the general user response to the central control action, such as the recognition of the general user response such as "the first", "movie", "I don't want to watch this". The processing steps of the entire model are as follows:

1. Query normalization processing: Traverse the set of candidate user intents and replace the intent names that appear in the query. The intent name here will support a certain generalization. For example, the generalization of music search is "Tinggeer", "song", etc. This part of the data will add some more streamlined and free based on the generalization of intent in language generation. statement.

2. Intent parsing: Traverse the general intent classification rule set predefined in the database (each rule includes rule content, rule matching method, intent decision result, priority, and rule matching is sorted by priority), if a rule is matched, the The intent decision result of the rule is added to the final parsing result, and then the matched part of the query is replaced with a placeholder, and subsequent rules are traversed until all rules are traversed.

3. Result verification: Combined with the analytical integrity, determine whether the intent recognition result can be output. If no output is available, the result object is set to null.

For better understanding of the present application, this embodiment assumes that the query input by the user is "spring flowers bloom", wherein "spring flowers bloom" contains both related TV dramas and related songs. Then, when the historical user intent determined in the acquired voice interaction process is "music", it can be considered that the user is currently listening to music, so that the candidate user intent corresponding to the above query is determined to be "music"; when the acquired voice interaction process When the historical user intent determined in the query is "TV series", it can be considered that the user is currently watching a TV series, so that the candidate user intent corresponding to the above query is determined to be "TV series".

In some feasible implementations, the above-mentioned historical user intent may be the user intent determined in the last round of voice interaction.

That is, the display device provided in the embodiment of the present application locates the user's intent on the currently received voice data by combining the historical user intent determined in the voice interaction process, which is helpful for quickly locating the user's true intent. .

In some embodiments of the present application, after the controller of the display device receives the first voice data and acquires the historical user intent in the voice interaction process, it determines the relationship between the first voice data and the historical user based on a conversation state tracking model. Whether the intent belongs to the same dialogue sequence.

The above dialogue state tracking model includes a session segmentation module and a dialogue state update module. Among them, the session segmentation module is used to determine whether the current user reply should open a new session, and the segmentation can be performed based on time interval and text semantics. Among them, the segmentation based on time interval is based on the assumption that the user's intention does not change in a relatively short period of time, and through the analysis of user logs, different segmentation standards are formulated according to the difference of the previous round of services. The segmentation based on text semantics focuses on dealing with speech recognition errors, insufficient semantic parsing capabilities, and strong correlations between services. The segmentation is performed by calculating text similarity and correlation matrix between services.

In some implementation manners, when the interval duration between the current reception time of the first voice data and the determination time of the above-mentioned historical user intention is greater than a preset duration threshold, it can be considered that the currently received first voice data and the above-mentioned historical User intent data does not belong to the same dialogue sequence, otherwise, it belongs to the same dialogue sequence.

For example, assuming that a user chooses to play a song through voice interaction during a certain period of time, and after a period of time, the user wakes up the voice assistant again and performs voice input, it is understandable that when the above mentioned When the interval is small, the probability of the user continuing to choose to listen to the song will be relatively high. It is likely that he wants to choose to play another song. Therefore, it can be considered that the user's current input voice and the user's intention determined during the last voice interaction belong to the same dialogue. sequence. On the contrary, when the above interval is longer, the probability of the user continuing to choose to listen to songs will be relatively small. It may be that he wants to change the entertainment method, such as watching TV. Therefore, it can be considered that the user's current input voice interacts with the last voice interaction. The user intent determined at the same time does not belong to the same dialogue sequence.

Exemplarily, it is assumed that the user wakes up the voice assistant in a certain period of time and says "play the song of the sunset". At this time, the display device recognizes that the user's intention is music and plays the song "song of the sunset". After a period of time, the user wakes up the voice assistant again and says "Spring flowers are blooming", then when the above interval is less than the preset duration threshold, it can be considered that "playing the song of the sunset" and "Spring flowers are blooming" "belongs to the same dialogue sequence, and the user's intention is music; when the above interval is greater than or equal to the preset duration threshold, it can be considered that "playing the song of the sunset" and "spring flowers blooming" do not belong to the same dialogue sequence, and the user The intent may be music or TV series.

In some embodiments, when the similarity between the parsed text semantics of the current first voice data and the text semantics of the above historical user intent is less than a preset similarity threshold, it can be considered that the currently received first voice data is the same as the The above historical user intent data does not belong to the same dialogue sequence, otherwise, it belongs to the same dialogue sequence.

For example, it is assumed that a user selects to play a song through voice interaction during a certain period of time, and after a period of time, the user wakes up the voice assistant again and performs voice input. It can be understood that, if the similarity between the text semantics of the input voice data after parsing and the text semantics of the user's intention determined in the previous round of voice interaction is relatively large, it can be considered that the user's current intention has not changed. , the user's current input voice and the user's intent determined during the last voice interaction belong to the same dialogue sequence. Conversely, if the similarity between the parsed text semantics of the voice data input by the user and the text semantics of the user's intention determined in the previous round of voice interaction is small, it can be considered that the user's current intention has changed, and it can be considered that the user's current intention has changed. The input speech does not belong to the same dialogue sequence as the user intent determined during the last speech interaction.

Exemplarily, it is assumed that the user wakes up the voice assistant in a certain period of time and says "play the song of the sunset". At this time, the display device recognizes that the user's intention is music and plays the song "song of the sunset". After a period of time, the user wakes up the voice assistant again and says "play a game", because the similarity between the current input "play a game" and the user's intention "music" determined during the last round of voice interaction is relatively high. If it is small, it can be considered that "playing the song of the sunset" and "playing the game" do not belong to the same dialogue sequence.

The above-mentioned dialogue state updating module is configured to create or update the dialogue state of the current user according to the segmentation result of the above-mentioned session segmentation module.

In this embodiment, when it is determined that the first voice data and the historical user intent belong to the same dialogue sequence, the candidate user intent of the first voice data can be determined by combining the first voice data and the historical user intent; when When it is determined that the first voice data and the historical user intent do not belong to the same dialogue sequence, a new dialogue sequence is established, and the candidate user intent of the first voice data is determined only based on the first voice data.

Based on the descriptions in the above embodiments, in some embodiments of the present application, each output module in the dialogue strategy learning model may be scored based on each candidate user intent corresponding to the first speech data, and according to the score of each output module, output Discretized dialog actions that interact with the user in a dialog.

Among them, the dialogue strategy learning model includes a rewriting module, a referential resolution module, a vertical domain intent parsing module, a task multi-round response module, a question-and-answer module, a news search module, a chat module, a recommendation module, and a candidate intent parsing module.

In a feasible implementation, the following strategies can be adopted to select the output module:

Step 1. Rank each output module according to the score of each output module. If the score of the output module ranked first is greater than or equal to a given threshold value 1, go to Step 4; otherwise, go to Step 2.

Step 2: Determine whether the score of the output module ranked first is less than the given threshold 2, if so, output the default action; otherwise, go to Step 3. Among them, the given threshold 2 is less than the given threshold 1.

Step 3. If the score of the second output module in the ranking is greater than or equal to the given threshold 2, and the difference between the scores of the two output modules in the first two rankings is less than the given threshold 3, the select action is output, and the select action It is used to provide a selection option, and the selection option includes the output module ranked first and the output module ranked second; otherwise, a confirm action is output, and the confirm action includes the output module ranked first.

Step 4: Determine whether the first-ranked module is a non-candidate intent parsing module, and if so, directly output an inform (notification) action, which includes the first-ranked output module; otherwise, go to step 5.

Step 5. Sort the multiple candidate user intents according to the scores of each candidate user intent. If the score of the candidate user intent with the highest score among the multiple candidate user intents is greater than or equal to a given threshold of 4, or the candidate user intent is unique and If the score is greater than or equal to the given threshold of 5, the inform action is directly output, and the inform action includes the candidate user intent with the highest score, indicating that the candidate user intent with the highest score is the target user intent; otherwise, go to step 5.1.

Step 5.1, determine whether the score of the candidate user intent with the highest score is less than the given threshold 6, if so, directly output the default action; if not, go to step 5.2.

Step 5.2. Determine whether the score of the candidate user intent with the second highest score is greater than or equal to the given threshold 6, if the score of the candidate user intent with the second highest score is greater than or equal to the given threshold 6, and the candidate user intent with the highest score is equal to or equal to the given threshold 6. If the difference between the scores of the candidate user intent with the second highest score is less than the given interval threshold, the select action is output, otherwise the confirm action is output. Wherein, the select action may provide selection options, and the selection options include the candidate user intent with the highest score and the candidate user intent with the second highest score; the confirm action includes the candidate user intent with the highest score.

The display device provided in the embodiment of the present application, by adding a general intent understanding model, a dialogue state tracking model, and a dialogue strategy learning model, combined with system dialogue actions to generate corresponding system responses, supports the voice interaction system through rich dialogue action types, Active interaction with users to complete the user's true intention positioning can effectively improve the user experience.

Based on the descriptions in the foregoing embodiments, an embodiment of the present application further provides a voice interaction method. Referring to FIG. 13 , FIG. 13 is a schematic flowchart 1 of a voice interaction method provided in the embodiment of the present application. In the embodiment of , the above-mentioned voice interaction method includes:

S1301. Receive first voice data input by a user, and determine a candidate user intent corresponding to the first voice data.

S1302. When the first voice data corresponds to multiple candidate user intents, generate a query statement according to the multiple candidate user intents, and feed back the query statement to the user.

Wherein, the above query sentence is used to prompt the user to select a user intent from the plurality of candidate user intents.

S1303. Receive the second voice data input by the user, and determine, according to the second voice data, a target user intent corresponding to the first voice data among the plurality of candidate user intents.

S1304. Output associated information associated with the target user's intention.

That is, in the voice interaction method provided by this application, when the candidate user intentions corresponding to the user input sentence are two or more, the voice interaction system can actively feed back the query sentence to the user through an anthropomorphic interactive mode, and then according to The response sentence input by the user is used to determine the real intention of the user, which can effectively improve the accuracy of understanding the user's intention in the process of voice interaction.

For a better understanding of the embodiments of the present application, refer to FIG. 14 , which is a second schematic flowchart of a voice interaction method provided in the embodiments of the present application.

In Figure 14, assuming that the voice data input by the user includes "toy train", and "toy train" contains both related songs and related movies, the display device will generate the query sentence "" after receiving the voice data Do you want to watch a movie or listen to music", if the received voice data input by the user again includes "listen to music", then query songs related to "toy train", and display on the display interface of the display device "In progress for You query the song".

For a better understanding of the embodiments of the present application, refer to FIGS. 15a to 15d , which are schematic diagrams of voice interaction of a display device in an embodiment of the present invention.

When the user needs to perform voice interaction with the display device 200, a voice wake-up instruction may be sent to the display device 200 through a preset wake-up mode. For example, in a feasible implementation manner, the user can speak a preset wake-up keyword, such as "Hi, Xiaoju", etc. through the voice collection device of the display device 200; at this time, the voice collection of the display device 200 The device will send the collected voice information to the controller, and the controller will recognize the received voice information, and if the recognition result includes the above-mentioned wake-up keyword, it will control the display device to enter the voice interaction state.

In some embodiments, after the display device enters the voice interaction state, a reminder message may be displayed on the display screen to remind the user that the display device 200 has entered the voice interaction state. As shown in FIG. 15a, the display screen may display “You How can I help you?".

After the display device enters the voice interaction state, the above-mentioned voice collecting device starts to collect the voice information input by the user, and sends the collected voice information to the controller. For example, in a feasible implementation, the user can say "toy train" through the microphone of the display device 200; at this time, the microphone of the display device 200 will send the collected voice information to the controller. After receiving the voice information, the controller performs voice recognition on the voice information, and determines the candidate user intent corresponding to the voice information.

When the candidate user intent corresponding to the above voice information includes both movies and music, the language generation module is used to generate a query sentence and display it on the display screen, as shown in Figure 15b, the display screen can display "Do you want to Watching movies or listening to music?".

During the process of displaying the query sentence on the display device, the voice collecting device continues to collect the voice information input by the user. When the voice information input by the user is "listen to music", the server will search for songs related to "toy train"; when the voice information input by the user is "watching a movie", the server will search for songs related to "toy train". Movie. As shown in Figure 15c, after it is determined that the voice information input by the user is "listen to music", it can be displayed on the display interface: Searching for songs related to "toy train" for you...

After the display device completes the search task, the search results can be displayed on the display. As shown in Figure 15d, it is displayed on the display screen: The following song is searched for you: "Toy Train.MP3".

Referring to FIGS. 16a to 16d , FIGS. 16a to 16d are schematic diagrams of another voice interaction of the display device in the embodiment of the present invention.

After the display device enters the voice interaction state, the above-mentioned voice collecting device starts to collect the voice information input by the user, and sends the collected voice information to the controller. For example, in a feasible implementation manner, the user can say "Transformers" through the microphone of the display device 200; at this time, the microphone of the display device 200 will send the collected voice information to the controller. After receiving the voice information, the controller performs voice recognition on the voice information, and determines the candidate user intent corresponding to the voice information. At the same time, the voice recognition result is displayed on the display interface, as shown in Figure 16a, and "Transformers" can be displayed on the display screen.

When the candidate user intent corresponding to the above voice information includes both movies and commodities, use the language generation module to generate a query sentence and display it on the display screen, as shown in Figure 16b, the display screen can display "Do you want to Movies or shopping?".

During the process of displaying the query sentence on the display device, the voice collecting device continues to collect the voice information input by the user. When the voice information input by the user is "shopping", the server will search for products related to "Transformers"; when the voice information input by the user is "watching movies", the server will search for movies related to "Transformers" . As shown in Figure 16c, after it is determined that the voice information input by the user is "shopping", it can be displayed on the display interface: Searching for products related to "Transformers" for you...

After the display device completes the search task, the search results can be displayed on the display. As shown in Fig. 16d, the commodity links of the searched commodities are displayed on the display screen.

It can be understood that, the voice interaction method described in the above embodiments may be executed by a server. For example, when the display device detects the user's input operation, it obtains the voice data input by the user, and then sends the voice data input by the user to the server. After the server performs voice recognition on the voice data input by the user, the corresponding voice data is determined. target user intent, and feedback the response information associated with the target user intent to the display device.

In some embodiments, the above server may perform data interaction with the display device through a network, or the above server may also be integrated in the display device and perform data interaction with the display device through a communication bus in the display device.

In addition, the voice interaction method described in the above embodiments can also be performed by a display device. For example, when the display device detects the user's input operation, it acquires the voice data input by the user, and then performs voice recognition on the voice data input by the user. , determine the target user's intention corresponding to the voice data, and send the response information associated with the target user's intention to the display for display.

It can be understood that the voice interaction method described in the above-mentioned embodiments can be applied not only to the above-mentioned display devices, but also to other electronic devices with voice interaction functions, such as smart speakers, smart homes, wearable devices, Children's toys, learning machines, etc., are not limited in the embodiments of the present application.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.

Claims (12)

1. A display device, characterized in that the display device comprises:

the voice acquisition device is used for acquiring voice data;

the audio processor is used for processing the collected voice data;

a display screen for displaying an image;

a controller configured to:

receiving first voice data input by a user, and determining candidate user intentions corresponding to the first voice data;

when the first voice data corresponds to a plurality of candidate user intentions, generating an inquiry statement according to the candidate user intentions, and sending the inquiry statement to the display screen for displaying, wherein the inquiry statement is used for prompting a user to select one user intention from the candidate user intentions;

receiving second voice data input by the user, and determining a target user intention corresponding to the first voice data in the candidate user intentions according to the second voice data;

outputting association information associated with the target user intent.

2. The display device according to claim 1, wherein the controller is configured to:

when the first voice data corresponds to a single candidate user intention, outputting associated information associated with the candidate user intention.

3. The display device according to claim 1, wherein the controller is configured to:

acquiring historical user intentions determined in a voice interaction process;

and determining candidate user intentions corresponding to the first voice data by utilizing the intention recognition model according to the first voice data and the historical user intentions.

4. The display device according to claim 3, wherein the controller is configured to:

determining whether the first speech data and the historical user intent belong to the same dialog sequence based on a dialog state tracking model;

when the first voice data and the historical user intention belong to the same dialogue sequence, determining an initial user intention corresponding to the first voice data by using the intention recognition model, and updating the initial user intention corresponding to the first voice data according to the historical user intention to obtain a candidate user intention corresponding to the first voice data;

when the first voice data and the historical user intention do not belong to the same dialog sequence, determining candidate user intentions corresponding to the first voice data by using the intention recognition model.

5. The display device according to claim 1, wherein the controller is configured to:

determining scores of all output modules in a dialogue strategy learning model according to the intentions of all candidate users corresponding to the first voice data; the dialogue strategy learning model comprises at least one of the following output modules: the rewriting module is used for referring to a resolution module, a vertical field intention analysis module, a task multi-turn response module, a question-answering module, a news search module, a chat module, a recommendation module and a candidate intention analysis module;

and when the output module with the highest score in the dialogue strategy learning model is the candidate intention analysis module, generating the inquiry statement according to the score of each candidate user intention.

6. The display device according to claim 5, wherein the controller is configured to:

when the score of a first candidate user intention corresponding to the first voice data is smaller than a first preset threshold, the score of a second candidate user intention corresponding to the first voice data is larger than a second preset threshold, and the difference between the score of the first candidate user intention and the score of the second candidate user intention is smaller than a preset interval threshold, generating the query sentence based on the first candidate user intention and the second candidate user intention; the first candidate user intention is a candidate user intention with the highest score corresponding to the first voice data, the second candidate user intention is a candidate user intention with the second highest score corresponding to the first voice data, and the first preset threshold is larger than the second preset threshold.

7. A method of voice interaction, the method comprising:

receiving first voice data input by a user, and determining candidate user intentions corresponding to the first voice data;

when the first voice data corresponds to a plurality of candidate user intentions, generating a query statement according to the candidate user intentions, and feeding back the query statement to the user, wherein the query statement is used for prompting the user to select one user intention from the candidate user intentions;

receiving second voice data input by the user, and determining a target user intention corresponding to the first voice data in the candidate user intentions according to the second voice data;

outputting association information associated with the target user intent.

8. The method of claim 7, further comprising:

when the first voice data corresponds to a single candidate user intention, outputting associated information associated with the candidate user intention.

9. The method of claim 7, wherein the determining the candidate user intent corresponding to the first speech data based on the intent recognition model comprises:

acquiring historical user intentions determined in a voice interaction process;

and determining candidate user intentions corresponding to the first voice data by utilizing the intention recognition model according to the first voice data and the historical user intentions.

10. The method of voice interaction according to claim 9, wherein determining, using the intent recognition model, the candidate user intent corresponding to the first speech data based on the first speech data and the historical user intent comprises:

determining whether the first speech data and the historical user intent belong to the same dialog sequence based on a dialog state tracking model;

when the first voice data and the historical user intention belong to the same dialogue sequence, determining an initial user intention corresponding to the first voice data by using the intention recognition model, and updating the initial user intention corresponding to the first voice data according to the historical user intention to obtain a candidate user intention corresponding to the first voice data;

when the first voice data and the historical user intention do not belong to the same dialog sequence, determining candidate user intentions corresponding to the first voice data by using the intention recognition model.

11. The method of claim 7, wherein generating a query statement according to the plurality of candidate user intents comprises:

determining scores of all output modules in a dialogue strategy learning model according to the intentions of all candidate users corresponding to the first voice data; the dialogue strategy learning model comprises at least one of the following output modules: the rewriting module is used for referring to a resolution module, a vertical field intention analysis module, a task multi-turn response module, a question-answering module, a news search module, a chat module, a recommendation module and a candidate intention analysis module;

and when the output module with the highest score in the dialogue strategy learning model is the candidate intention analysis module, generating the query sentence according to the score of each candidate user intention.

12. The method of claim 11, wherein generating the query statement based on the score of each of the candidate user intentions comprises:

when the score of a first candidate user intention corresponding to the first voice data is smaller than a first preset threshold, the score of a second candidate user intention corresponding to the first voice data is larger than a second preset threshold, and the difference between the score of the first candidate user intention and the score of the second candidate user intention is smaller than a preset interval threshold, generating the query sentence based on the first candidate user intention and the second candidate user intention; the first candidate user intention is a candidate user intention with the highest score corresponding to the first voice data, the second candidate user intention is a candidate user intention with the second highest score corresponding to the first voice data, and the first preset threshold is larger than the second preset threshold.

Images