DE112022003216T5 - Early call for context data processing - Google Patents

Abstract

A speech processing system uses context data to determine the specific domains, subdomains, and applications appropriate to take actions in response to spoken commands and other utterances. The system may use signals and other context data associated with an utterance, such as location signals, content catalog data, data regarding historical usage patterns, data regarding content visually presented on a display screen of a computing device at the time of an utterance, other data, or combinations thereof.

Description

ADDITION BY REFERENCE

The following applications are incorporated herein by reference and made a part of this specification: US Application No. title Registration date 17/304,714 DYNAMIC CONTEXT-BASED ROUTING OF SPEECH PROCESSING 24 June 2021 17/304,720 EARLY INVOCATION FOR CONTEXTUAL DATA PROCESSING 24 June 2021

GENERAL STATE OF THE ART

Electronic devices, such as voice-assisted electronic devices, are capable of performing various functions. For example, a person may issue a command to activate such a device, and in response, the device may perform various functions and/or cause one or more actions to be taken. Some voice-enabled electronic devices can communicate with a network-accessible system to process spoken commands, perform functions, and the like.

BRIEF DESCRIPTION OF THE DRAWINGS

• 1 ist ein Blockdiagramm, das Datenflüsse und Interaktionen zwischen Systemen und Komponenten einer veranschaulichenden vernetzten Sprachverarbeitungsumgebung gemäß einigen Ausführungsformen zeigt.
• 2 ist ein Diagramm einer veranschaulichenden Architektur der in 1 gezeigten Systeme gemäß einigen Ausführungsformen.
• 3 ist ein Diagramm veranschaulichender Datenflüsse und Interaktionen zwischen Komponenten eines domäneninternen Routing-Systems und anderen Komponenten eines Sprachverarbeitungssystems gemäß einigen Ausführungsformen.
• 4 ist ein Ablaufdiagramm eines veranschaulichenden Prozesses für domäneninternes Routing unter Verwendung von Kontextdaten gemäß einigen Ausführungsformen.
• 5 ist ein Diagramm einer veranschaulichenden mehrschichtigen Domänenkonfiguration gemäß einigen Ausführungsformen.
• 6 ist ein Diagramm veranschaulichender Datenflüsse und Interaktionen zwischen einem Kontextdatenverwaltungssystem und anderen Komponenten eines Sprachverarbeitungssystems gemäß einigen Ausführungsformen.
• 7 ist ein Ablaufdiagramm eines veranschaulichenden Prozesses zum Verwalten von Kontextdaten gemäß einigen Ausführungsformen.
• 8 ist ein Diagramm veranschaulichender Datenflüsse und Interaktionen zwischen einem Kontextdatenverwaltungssystem und anderen Komponenten eines Sprachverarbeitungssystems gemäß einigen Ausführungsformen.

Embodiments of various inventive features will now be described with reference to the following drawings: In the drawings, reference numerals may be reused to indicate correspondence between designated elements. The drawings are provided to illustrate exemplary embodiments described herein and are not intended to limit the scope of the present disclosure.

• 1 is a block diagram showing data flows and interactions between systems and components of an illustrative networked speech processing environment, according to some embodiments.
• 2 is a diagram of an illustrative architecture of the 1 shown systems according to some embodiments.
• 3 is a diagram illustrating data flows and interactions between components of an intra-domain routing system and other components of a speech processing system, according to some embodiments.
• 4 is a flow diagram of an illustrative process for intra-domain routing using context data, according to some embodiments.
• 5 is a diagram of an illustrative multi-tier domain configuration, according to some embodiments.
• 6 is a diagram illustrating data flows and interactions between a context data management system and other components of a language processing system, according to some embodiments.
• 7 is a flow diagram of an illustrative process for managing context data, according to some embodiments.
• 8th is a diagram illustrating data flows and interactions between a context data management system and other components of a language processing system, according to some embodiments.

DETAILED DESCRIPTION

The present disclosure is directed to a system that uses context data to determine the specific domains, subdomains, and intention processing applications appropriate for taking actions in response to spoken commands and other utterances. In some In embodiments, the system may use signals and other contextual data associated with an utterance, such as location signals, content catalog data, data related to historical usage patterns, data related to content visually presented on a display screen of a computing device at the time of an utterance, other data, or combinations thereof . By incorporating contextual data into routing decisions for responding to - or otherwise taking action on - potentially ambiguous spoken utterances, a speech processing system can provide an improved user experience in terms of user-perceived latency and success in achieving interaction goals the user provides.

Some speech processing systems process utterances by generating transcripts or other textual representations of the utterances using automatic speech recognition ("ASR") and then analyzing the textual representations to determine their meaning using natural language understanding ("NLU") ) to determine. The systems then take one or more actions based on the specific meanings of the utterances. However, some statements may be ambiguous and may reasonably be interpreted as requests to perform any number of different actions, and some actions may be appropriately performed in any number of different ways and/or through any number of applications. Some language processing systems attempt to account for such ambiguities by using contextual data to help determine the meaning of an utterance. The utterance can then be forwarded to an application to take an action associated with the particular meaning. However, such systems may not have access to all relevant contextual data at the time of determining the meaning of the initial utterance or may otherwise be unable to take into account all relevant contextual data while determining the meaning of the utterance. For these reasons, among others, the systems may not accurately determine the most appropriate actions to be taken or the otherwise most appropriate applications for carrying out the actions. Additionally, new measures and applications may not be integrated into the system and cannot accommodate new contextual data without making changes to the underlying NLU processing.

Some aspects of the present disclosure address the foregoing issues, including by providing context-aware routing of utterances to specific domains and domain-specific applications for processing. In some embodiments, a speech processing system implements utterance processing using a set of domains. In general, a domain is a collection of applications (also referred to as “experience provider services,” “experience providers,” or “services”), each configured to respond to utterances related to a specific subject of the utterance, such as are assigned to a specific subset of voice commands and other intentions that the voice processing system can understand. A response to an utterance may include automatically performing a function, engaging in spoken dialogue with multiple turns, other actions, or combinations thereof. For example, a domain might be a purchasing domain configured to handle all types of purchasing-related commands and queries. Another domain may be a media content management service through which a user accesses media content that they own, have licensed, or access third-party media. Yet another domain may be a communications domain configured to initiate telephone calls and generate emails, etc. The specific actions taken by the domains in response to utterances may be processed by specialized applications, as combinations of Software and hardware are implemented. Separating the functionality of the language processing system into domains can allow for easier addition and modification to the universe of commands and queries that the language processing system can accept. However, in such a configuration, routing utterances to the correct domain - or to the correct application within a given domain - becomes an important task because applications in one domain may not be able to route utterances associated with the topic of another domain must be processed appropriately. In addition, some domains can be large and organized into subdomains in any number of layers. In such cases, routing an utterance to the correct subdomain can present additional complexity.

To improve the routing of utterances to specific domains, subdomains and applications, an intra-domain routing system may use any number of subdomain-specific and/or application-specific confidence providers to obtain judgments about which Subdomains and/or applications (if any) are appropriate targets for a concrete utterance. For example, an ASR system may generate text data representing a transcript or a series of words in an utterance, and an NLU system may then generate an N-best list (where N is a positive integer) of interpretations of the utterance that the language processing system is configured to process. The interpretations of the utterance may also be referred to as "intentions." The domains associated with one or more of the N-best intentions may be identified and asked to indicate whether they are capable of appropriately handling the intention(s). Within a concrete domain, the subdomain and/or application-specific confidence providers associated with the intention(s) may be asked to provide an assessment of the likelihood that the corresponding subdomains and/or applications are the correct entity to respond to the utterance.

The confidence providers' assessments may be based not only on the utterance itself (e.g., the NLU-generated intent, ASR-generated text data, etc.), but also on contextual data associated with the utterance. For example, a domain may be a shopping domain configured to process all kinds of shopping-related commands and queries. The shopping domain may have any number of intent processing applications, which may be divided into any number of subdomains, including a first subdomain associated with searching for products in a concrete store and a second subdomain associated with purchasing products online. If the user says "Find product X for me", the correct entity to act on that utterance may depend on contextual data associated with the utterance, such as the user's geographic location: if the user is in a specific store, then the correct entity may be the first subdomain and/or application, while if the user is at home, the correct entity may be the second subdomain and/or application. Confidence providers may be provided with contextual data indicating the user's geographic location, and confidence providers may create different assessments for different users and/or different utterances depending on the user's geographic location at the time of the utterance.

Advantageously, the consideration of contextual information during routing confidence assessments is separate from the consideration of contextual information (if any) during NLU intent determinations and may include additional contextual information and/or completely different contextual information.

Additional aspects of the present disclosure relate to proactively obtaining and managing contextual information during utterance processing for use in subsequent routing determinations, response generation operations, and other downstream processes. A context data management system can be integrated at various points in the process of receiving and processing an utterance to generate intent. As data is generated, retrieved, or otherwise becomes available during utterance processing, the context data management system can identify and obtain the context data to be used for routing confidence assessments. The acquired context data can be stored, processed, used to generate and/or obtain additional context data, etc. In some embodiments, the context data management system may include a register of context data items to be obtained, the integration points at which the context data items are to be obtained, the processing/storage/etc. to be performed using the context data, which routing confidence scores the context data items use, etc. administer. For example, the register may indicate that a context data item may become available at a concrete point during utterance processing and should be used as input to a calculation or evaluated by a model to produce a concrete value to be considered in routing confidence assessment shall be. As another example, the register may indicate that at another point during utterance processing, another piece of contextual data may become available and should be stored for consideration during further routing confidence assessment. The registry may define any number and combination of such context data item collection processes, and the context data management system may use the registry to capture the context data items and make them available for routing confidence assessments. In some embodiments, the register may associate context data items with specific confidence providers.

Various aspects of the disclosure will now be described with respect to specific examples and embodiments intended to illustrate, but not limit, the disclosure. Even if While aspects of some embodiments described in the disclosure focus on specific examples of utterances, intentions, applications, routing determinations, and contextual data elements for purposes of illustration, the examples are for illustrative purposes only and are not intended to be limiting. In some embodiments, the techniques described herein may be applied to additional or alternative utterances, intents, applications, routing determinations and contextual data elements, and the like.

Speech processing environment

1 is a schematic diagram of an illustrative network environment in which a user 104 makes an utterance 106, one or more voice-assisted devices 102 recognize the utterance 106, and a speech processing system 100 determines the meaning of the utterance 106 - and takes an action in response.

In some embodiments, the voice-assisted device 102 may include one or more microphones used to capture the user's speech, such as utterance 106, one or more speakers used to reproduce speech (e.g., computer synthesized dialogue), or others Content used, one or more advertisements used to display content, etc. The voice-assisted device 102 may also be referred to as a user computing device or user device. In some embodiments, the voice-assisted device 102 may be portable or mobile. For example, the voice-enabled device 102 may be a handheld device or other mobile device, such as a cell phone or tablet computer. In some embodiments, the voice-assisted device 102 may be designed to operate from a relatively fixed location. The voice-enabled device may, for example, be a speaker configured with audio input capabilities and network access (e.g., a “smart speaker”), a display configured with audio input capabilities and network access (e.g., a “smart display”), or be another electronic device. The voice processing system 100 can process audio signals received from the voice-assisted device 102 and formulate responses for the user 104. The speech processing system 100 may include various components for providing the features described herein. In some embodiments, the speech processing system 100 may include an ASR system 202 to process audio signals or other audio data and generate text data representing user utterances. The speech processing system 100 may also include an NLU system 204 to process text data and generate semantic representations of the user's utterances. The speech processing system 100 may also include a system of domains 206, each containing or managing one or more applications 282 to respond to or otherwise respond to user statements, such as providing requested content, performing requested operations, and the like. Advantageously, individual domains 206 (or certain subsets thereof) may be assigned to a corresponding intra-domain routing system 212 to determine whether specific utterance meaning hypotheses generated by the NLU system 204 can be processed by applications 282 of the corresponding domain 206. The speech processing system 100 may also include a context data management system 214 to manage the collection, generation, and provision of context data to the intra-domain routing system(s) 212 for use in routing decisions, generating responses, etc. The decisions made by the intra-domain routing system 212 may be used by a cross-domain routing system 210 to manage the routing of utterances to individual domains 206 or applications 282.

The exemplary systems and components of the 1 are for illustrative purposes only and are not intended to be limiting. In some embodiments, a speech processing system 100 may include fewer, additional, and/or alternative components and data storage. A specific, detailed exemplary embodiment of the speech processing system 100 is shown in 2 and is described in more detail below.

The voice processing system 100 may be implemented on one or more physical server computing devices that provide computing services and resources to end user devices, such as voice-enabled devices 102. In some embodiments, the speech processing system 100 (or individual components thereof, such as the ASR system 202, the NLU system 204, the domain systems 206, the cross-domain routing system 210, the intra-domain routing systems 212, the context data management system 214, etc .) be implemented on one or more host devices, such as blade servers, midrange computing devices, mainframe computers, desktop computers computers or any other computing device configured to provide computing services and resources. For example, a single host device may execute one or more ASR systems 202, NLU systems 204, domain systems 206, cross-domain routing system 210, intra-domain routing systems 212, context data management system 214, or combinations thereof, etc. The speech processing system 100 may include any number of such hosts.

In some embodiments, the features and services provided by the voice processing system 100 may be implemented as web services that may be consumed over one or more communications networks. In further embodiments, the speech processing system 100 (or individual components thereof) is provided by one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more rapidly provisioned and shared computing resources, such as computing devices, networking devices, and/or storage devices. A hosted computing environment can also be referred to as a “cloud” computing environment.

As in 1 , a user 104 may interact with a voice-enabled device 102 using an utterance 106. The voice-enabled device 102 may detect, via one or more microphones, sounds corresponding to the user's utterance 106. In certain implementations, the utterance 106 may include, or may precede the utterance, a wake word or other triggering phrase or event (e.g., "computer!") spoken by the user 104 to indicate that the user's subsequent utterance (e.g., "Where's the coffee?") is a device-directed utterance (e.g., a utterance to be received and processed by the voice-enabled device 102 and/or the speech processing system 100). The voice-enabled device 102 may detect the wake word and begin streaming audio signals to the speech processing system 100. In some cases, the voice-assisted device 102 may operate in a low-functionality mode and analyze sounds using ASR processing. When the wake word is detected (e.g., using ASR, certain keywords (keyword spotting), etc.), the voice-assisted device 102 may begin streaming the audio signal (and optionally other data) to the speech processing system 100. The wake word may be a reserved keyword that is detected locally by the voice-assisted device 102, such as through the use of an expression detector that analyzes audio signals generated by the microphones of the voice-assisted device 102 to detect the wake word, which may generally be a predefined word, phrase, or other sound. Such an expression detector may be implemented, for example, using keyword spotting technology.

In the 1 In the example illustrated, following an activation word, the user 104 issues an utterance 106 that the voice-assisted device 102 captures. The voice-assisted device 102 may generate an audio signal representing the utterance 106. As shown, the voice-assisted device 102 may send the audio signal to the speech processing system 100. In some embodiments, the voice-assisted device 102 may further determine and/or generate and send to the speech processing system 100 additional metadata that may be used to determine various terms in the utterance 106. The types of metadata may include, for example, data regarding information currently displayed on a display component (or other display), sensor data representing the current location and/or environment of the voice-assisted device 102, snapshots indicating device states of the voice-assisted device 102, data about the voice-assisted device 102 (e.g., unique identifier, version), etc. Data regarding the currently displayed information may include identifiers of content items currently displayed on the screen, identifiers of items in a list, etc. Sensor data may include data generated by various sensors and other components of the voice-enabled device, such as data representing geolocation, ambient temperature, ambient lighting, device orientation, device movement, and the like. A snapshot may include device states indicating current operations of the voice-enabled device 102 before, during, and/or after the utterance 106 is recognized by the voice-enabled device 102. Device states of the voice-enabled device 102 may represent actions such as making a phone call, sounding an alarm, streaming audio (e.g., music, audiobooks, etc.), engaging in dialogue with the user 104, performing online searches, controlling devices in a home, or any other type of activity that a voice-enabled device 102 may be configured to perform. Data about the voice-assisted device 102 may include a device identifier, a version identifier, data regarding Components and/or capabilities of the voice-enabled device 102 (e.g., whether the device has a display screen, a touchscreen, a speaker, a microphone, a keyboard, etc.), data regarding a network connection available to the voice-enabled device 102, geolocation or local location data regarding the location of the voice-enabled device 102, etc.

While the snapshots of device states may be sent to the speech processing system 100 when an utterance 106 is recognized, in other examples, the speech processing system 100 may also store indications of device states rather than receiving them each time an audio signal is received. For example, the speech processing system 100 may receive an indication that the user 104 has set an alarm and know when the alarm will sound. Therefore, once the alarm sounds, the speech processing system 100 may already have stored the device state and may not receive the snapshots each time an utterance 106 is recognized. Similarly, some or all of the metadata may be stored in the speech processing system 100 before the utterance 106 is captured by the voice-assisted device 102.

Referring to an illustrative example, the user 104 may make an utterance 106 such as "Where's coffee?" At [A], an audio signal representing the user's utterance 106 may be transmitted to the speech processing system 100 for processing and response. Based solely on the words spoken, the utterance 106 may be ambiguous: it could be a request for a physical cafe nearby; it could be a query about the location of coffee to a physical store; it could be a query for an online coffee purchase. Any of these interpretations, and possibly others, may be plausible. Traditional systems can rank the possible interpretations, select the top ranked interpretation, and pass the query to an application to take an associated action. However, without considering the contextual data associated with the utterance, routing the query to an application may not be appropriate. For example, if a user is in a store that sells coffee, the second interpretation (it is a query about the location of coffee in a physical store) may be the most likely correct interpretation, while the third interpretation (it is a Query about purchasing coffee online) may be the most likely correct interpretation when the user is at home and viewing a shopping website. Different applications can be used to take actions in response to these different interpretations. By taking into account contextual information, such as the user's current location, the user's current activities and/or interactions with the voice-enabled device 102, etc., the system 100 can make dynamic routing determinations that vary from user to user and even by query to query, even if they are received from the same user.

At [B], the ASR system 202 may generate ASR results using the audio signal. As an illustration, the ASR results may include one or more transcripts or other sets of text data representing words in the utterance 106. In some examples, the ASR system 202 may generate ASR confidence score data representing the likelihood that a specific set of words of the text data matches the words spoken in the utterance 106. For example, the ASR system 202 may determine a score representing a confidence or probability that a specific word matching the sounds would be included at the specified location in the sentence (e.g., using a language or grammar model). Thus, each possible textual interpretation of the spoken utterance 106 (hypothesis) is associated with an ASR confidence score. The ASR system 202 may send the ASR results to the NLU system 204 at [C].

At [D], the NLU system 204 may generate NLU result data, such as a semantic representation of the utterance 106, using the acquired ASR results. In the present example, as described above, the utterance "Where's coffee?" Query for the location of a physical cafe, as a query for the location of coffee in a physical store, as a query for coffee in an online store, etc. The NLU system 204 may generate an N-best list of interpretations, including some or all of these interpretations and possibly others. The NLU system 204 may provide the NLU results to the cross-domain routing system 210 at [E]. At [B'], the context data management system 214 may acquire and/or generate context data at various integration points during the ingest, ASR, and NLU processes. Contextual data collection is shown using the label [B' - F'] to indicate the parallel or otherwise asynchronous operation with respect to other shown and processes described, including those labeled [B] - [F]. For example, the context data management system 214 may obtain geolocation data representing the current geographic location of the voice-enabled device 102. This data may be acquired by the system 100 with the audio signal at [A] and then provided to the context data management system 214, requested by the system 100 upon receipt of the audio signal and then provided to the context data management system 214, or otherwise obtained by the context data management system 214. In some embodiments, the geolocation data may be processed by the context data management system 214 in parallel with ASR processing at [B], provision of ASR results at [C], NLU processing at [D], and provision of NLU results at [E]. or obtained at another time (e.g. the data may be available from the processing of a previous statement). The context data management system 214 may include other context data items before, in parallel with, or after ASR processing at [B], providing ASR results at [C], NLU processing at [D], providing NLU results at [E ] and/or at various other times (e.g. during processing of a previous utterance). For example, the context data management system 214 may include data regarding content presented by the voice-enabled device 102 when the utterance 106 is made (e.g., web pages, list items, etc.), partial or full ASR results, partial or full NLU results, etc . gain. As another example, the context data management system 214 may perform operations based on the ASR and/or NLU results, such as determining a difference in score between highest ranked NLU results, performing an initial classification of the utterance 106 as having a previous interaction with the system 100 related or unrelated, generating encodings of portions of NLU partial results, etc. These contextual data elements may be stored or otherwise made available to routing confidence providers for use in routing determinations and/or generating a response to the utterance .

At [F], the cross-domain routing system 210 may determine the domain(s) from which to generate a routing confidence assessment request (e.g., a request to determine whether the domain or a application of which is the correct entity to respond to the utterance). This request may be referred to as a “confidence request” or a “fulfillment request.”

In some embodiments, the cross-domain routing system 210 may maintain or otherwise access a mapping of which domain(s) is/are configured to handle or otherwise associated with the intent(s) that the NLU system 204 is configured to generate. The cross-domain routing system 210 may receive from the NLU system 204 a list of one or more intentions for the current utterance 106, such as an N-best list. The cross-domain routing system 210 may traverse the N-best list or a portion thereof and use the mapping to identify the domain(s) associated with the intention(s). The inter-domain routing system 210 may then generate a confidence query for each identified domain and send the query to the intra-domain routing system 212 associated with each identified domain. A confidence query may include intent data representing one or more intents associated with the domain to which the query is sent. The query may include additional information, such as score or ranking data indicating the relative likelihood that the intent is the correct intent, as assessed by the NLU system 204.

In the present example, an N-best list of three intents may be provided to an intra-domain routing system 212 for a shopping domain: an intent corresponding to a query for a location of a physical cafe, an intent corresponding to a query for the location of coffee in a physical store, and an intent corresponding to a query for coffee in an online store. Additional intents may be provided to one or more other intra-domain routing systems 212 for other domains. For example, the N-best list of intents received by the inter-domain routing system 210 at [E] may include the three intents described above for the shopping domain, an intent for a hospitality domain (e.g., to find out if there is coffee in a hotel room), an intent for a video content domain (e.g., to find a video with the word "coffee" in the title). The cross-domain routing system 210 may generate confidence requests for these two other domains in addition to the confidence request generated for the shopping domain. In some embodiments, the confidence requests sent to the intra-domain routing systems 212 for the different domains may be made and/or processed serially, or in some embodiments, the requests may be made and/or processed in parallel or otherwise asynchronously.

In [G], the intra-domain routing system 212 may obtain context data for use in routing confidence evaluations upon receipt of each confidence request. The context data may include, but is not limited to, data regarding previous utterances of the current session, data regarding utterances of previous sessions, data regarding items presented on a screen when the current utterance was made, data regarding the domain(s) associated with items presented on a screen during the current utterance, data regarding entities recognized during NLU processing, data regarding entities resolved during NLU processing, data regarding user feedback during the current or previous session, data regarding the user's purchase history, data regarding the user's interaction history, data regarding specified or learned user preferences, data regarding the user's accessibility history and/or preferences, geolocation data, data regarding device capabilities, data regarding device characteristics, data regarding the user's digital content library items, data regarding user subscriptions, data regarding the user's purchase history, directory data regarding the presence and/or absence of domain terms, other data, or combinations thereof. The example context data elements described herein are for illustrative purposes only and are not intended to be limiting, required, or exhaustive.

In the present example, the intra-domain routing system 212 for the shopping domain may obtain location data representing the geographic location of the voice-enabled device 102. As another example, the intra-domain routing system 212 for the shopping domain may obtain data representing content that was displayed on a screen of the voice-enabled device 102 when the utterance 106 was made.

At [H], the intra-domain routing system 212 may generate one or more routing confidence scores upon receipt of each confidence request. In some embodiments, an intra-domain routing system 212 may employ any number of application-specific confidence providers to make judgments about which subdomains (e.g., applications) are appropriate targets for a particular utterance. For example, the intra-domain routing system 212 for the shopping domain may request routing confidence scores from each of three application-specific routing confidence providers: a first routing confidence provider for the physical store location application, a second confidence provider for a product location application Store and a third routing confidence provider for an online shopping application. Requests for routing confidence assessments may be made and/or processed serially, or in some embodiments, the requests may be made and/or processed in parallel or otherwise asynchronously. In some embodiments, requests for routing confidence scores may include the various contextual data elements, if any, used by the respective application-specific routing confidence providers. For example, the routing confidence provider for the physical store location and the in-store product location application may receive contextual data representing the geographic location of the voice-enabled device 102, while the routing confidence provider for the online shopping application may receive contextual data. representing the geographic location and also the content (if any) that was displayed by the voice-enabled device 102 when the utterance 106 was made.

In some embodiments, a routing confidence score may be a confidence score that represents the respective provider's confidence that the application associated with the provider is the correct application to handle the intent. Scoring may be determined using deterministic rules, a statistical model, or a combination thereof. The routing confidence scores may be provided to the cross-domain routing system 210 at [I].

In this example, the routing confidence provider for the store aisle location application may apply a set of deterministic rules and/or a statistical model to the associated intent (e.g., "find aisle location" intent) and contextual data (e.g., . Geographical data indicating that the user is in a specific store that offers voice-based product location). This routing confidence provider can generate a relatively high confidence score, indicating high confidence that a store aisle location application is the correct application to handle the intent. The routing confidence provider for the application Locating a physical store can apply a set of deterministic rules and/or a statistical model to the associated intent (e.g., an intent to locate a store in transit) and contextual data (e.g., geographic data indicating that the user is located in the aforementioned specific store). This routing confidence provider can generate a relatively medium confidence score, indicating a medium confidence that a physical store location application is the correct application to handle the intent. The routing confidence provider for the online shopping application may apply a set of deterministic rules and/or a statistical model to the associated intent (e.g. a “browse online items” or “purchase items” intent) and contextual data (e.g. B. geographical data indicating that the user is in the aforementioned specific store and content other than an online shopping site or application was displayed by the voice-activated device 102 when the utterance 106 was made). This routing confidence provider can generate a relatively low confidence score, indicating a low confidence that an online shopping application is the correct application to handle the intent.

In some embodiments, the intra-domain routing system 212 may compare or otherwise analyze the various routing confidence scores (e.g., confidence scores) generated by the routing confidence providers. Based on the analysis, the intra-domain routing system 212 may determine which application in the domain (if any) is likely to be the best application to handle the intent. In this example, the intra-domain routing system 212 may determine that the store aisle location application is the best application in the shopping domain to serve the intent.

At [J], the inter-domain routing system 210 may determine a domain, sub-domain, and/or application assignment based on the routing confidence scores received from one or more intra-domain routing systems 212. In the present example, the spoken question "Where is coffee?" will be routed to the store aisle location application based on the relatively high confidence score of this application compared to the other confidence scores for other applications.

At [K], the associated application 282 may generate a response to the utterance 106. The response may include performing a function, generating a synthesized spoken response, generating a visual response, taking another action, or any combination thereof. For example, performing a function may include initiating a purchase transaction. Generating a synthesized response may include providing requested information in audio form. Generating a visual response may include displaying requested information on a visual display. In the present example, the response may be a synthesized spoken response (e.g., "Coffee is in aisle 20"), a visual response (e.g., a map of the store with the location of the coffee highlighted), or any combination thereof.

At [L], the voice processing system 100 may transmit the generated response, if any, to the voice-assisted device 102, and the voice-assisted device may present the response as needed.

Referring now to 2 Various examples of components of an embodiment of the speech-based device 102 and an embodiment of the architecture of the speech processing system 100 of 1 A voice-enabled device 102 may correspond to any suitable type of electronic device, including, but not limited to, desktop computers, mobile computers (e.g., laptops, ultrabooks), cell phones, smartphones, tablets, televisions, set-top boxes, smart TVs, personal displays, large format displays (e.g., billboards, street signs, etc.), personal digital assistants (“PDAs”), gaming consoles and/or devices, smart furniture, smart home appliances (e.g., refrigerators, microwaves, etc.), smart vehicles (e.g., cars, trucks, motorcycles, etc.), smart transportation devices (e.g., boats, ships, trains, airplanes, etc.), wearable devices (e.g., watches, pins/brooches, headphones, glasses, headsets, etc.), and/or smart accessories (e.g., light bulbs, light switches, electrical switches, etc.). In some embodiments, a voice-assisted device 102 may be relatively simple or basic, including no or only a minimal number of mechanical input options (e.g., keyboard, mouse, trackpad) or touch input(s) (e.g., touchscreen, buttons). For example, a voice-assisted device 102 may be capable of receiving and outputting audio, and may include power, processing capabilities, data storage/memory capabilities, and communication capabilities. However, in other embodiments, a voice-assisted device 102 may include one or more components for receiving mechanical or touch inputs, such as a touch screen and/or one or more buttons.

A voice-enabled device 102, in one embodiment, may include a minimal number of input mechanisms (e.g., a power switch) such that the functionality of a voice-enabled device 102 is solely or primarily through audio input and audio output. For example, a voice-enabled device 102 may include or communicate with one or more microphones that listen for an activation word by continuously monitoring local audio. In response to detecting the activation word, a voice-enabled device 102 may connect to the speech processing system 100, send audio data to the speech processing system 100, and wait for or receive a response from the speech processing system 100. However, in some embodiments, non-voice/sound-enabled devices may also communicate with the speech processing system 100. For example, in response to pressing a button or touch screen, or pressing and holding a button or touch screen, a microphone associated with a voice-enabled device 102 may begin recording local audio, connect to the speech processing system 100, send audio data representing the captured audio to the speech processing system 100, and wait for or receive a response from the speech processing system 100 and/or wait for an action to occur.

Voice-enabled device 102 may include one or more processors 220, data storage/memory 222, communication circuitry 224, one or more microphones 226 or other audio input devices (e.g., transducers), one or more speakers 228 or other audio output devices, one or more cameras 230 or other image capture components, and a display component 232. However, one or more additional components may be included in a voice-enabled device 102 and/or one or more components may be omitted. For example, a voice-enabled device 102 may also include a power supply or a bus connector. As yet another example, a voice-enabled device 102 may include one or more additional input and/or output mechanisms, such as one or more sensors, one or more buttons, or one or more switches or knobs. Moreover, while a voice-enabled device 102 may include multiple instances of one or more components, for simplicity, only one of each component has been shown.

In some embodiments, a voice-assisted device 102 may correspond to or include the functionality of a manually activated device. A manually activated device, as described herein, may correspond to a device that can be activated in response to a manual input (e.g., pressing a button, touching a portion of a touch screen, performing an action on a device). For example, a tap-to-talk device is a type of manually activated device. For example, such tap-to-talk devices are capable of acquiring and outputting audio data in response to the pressing of a key.

The processor(s) 220 may include any suitable processing circuitry capable of controlling operations and functionality of a voice-enabled device 102 as well as facilitating communication between various components within a voice-enabled device 102. In some embodiments, the processor(s) 220 may include a central processing unit (“CPU”), a graphic processing unit (“GPU”), one or more microprocessors, a digital signal processor, or any other type of processor, or any combination thereof. In some embodiments, the functionality of the processor(s) 202 may be performed by one or more hardware logic components, including, but not limited to, field-programmable gate arrays (“FPGAs”), application specific integrated circuits (“ASICs”), application specific standard products (“ASSPs”), system-on-chip systems (“SOCs”), and/or complex programmable logic devices (“CPLDs”). In addition, each of the processor(s) 220 may include its own local memory that may store program systems, program data, and/or one or more operating systems. However, processor(s) 220 may run an operating system (“OS”) for voice-enabled device 102 and/or one or more firmware applications, media applications, and/or resident applications. In some embodiments, processor(s) 220 may run a local client script to read and render content received from one or more websites. For example, processor(s) 220 may (en) 220 execute a local JavaScript client to render HTML or XHTML content received from a particular URL accessed by a voice-enabled device 102. Data storage/memory 222 may include one or more types of storage media, such as any volatile or non-volatile memory or removable or non-removable memory suitably implemented to store data for a voice-enabled device 102. For example, data may be stored using computer-readable instructions, data structures, and/or program systems. Various types of data storage/memory may include, but are not limited to, hard disks, solid state drives, flash memory, non-volatile memory (e.g., ROM), electronically erasable programmable read-only memory ("EEPROM"), CD-ROMs, digital versatile disk ("DVD") or other optical storage media, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, RAID storage systems, or other types of storage or combinations thereof. Additionally, storage/memory 222 may be implemented as computer-readable storage media (“CRSM”), which may be any available physical medium that can be accessed by processor(s) 220 to execute one or more instructions stored in storage/memory 222. In some embodiments, one or more applications (e.g., games, music, video, calendar, lists, etc.) may be executed by processor(s) 220 and stored in memory 222.

In some embodiments, the data store/memory 222 may store voice biometric data associated with one or more individuals. For example, a person operating a voice-enabled device 102 may have a registered user account or other profile information in the voice processing system 100 (e.g., in a user data store 216). In some embodiments, a voice-enabled device 102 may be associated with a group account, and different people may have user accounts that operate under the group account's rules and configurations. As an illustrative example, a voice-enabled device 102 may be associated with a first group account in the voice processing system 100, where the first group account is for a family that lives in a household in which a voice-enabled device 102 is located. Each family member may also have a user account associated with the first group account (e.g., a parent, a child, etc.), and therefore each user account may gain some or all of the rights of the first group account. For example, a voice-enabled device 102 may have a first group account in the voice processing system 100 registered to a specific family or group, and each parent and child of the family may have their own user account registered under the parent's registered account. In an illustrative embodiment, voice biometric data for each individual may be stored by that individual's corresponding user account. The voice biometric data may, for example, correspond to a “voice profile” or “voice model” of a specific person, which may be a graphical representation of a person's voice including a frequency decomposition of that person's voice. The communication circuit 224 may include any circuitry that allows or enables one or more components of a voice-enabled device 102 to communicate with each other and/or with one or more additional devices, servers, and/or systems. For example, communication circuitry 224 may facilitate communication between a voice-enabled device 102 and the voice processing system 100. As an illustrative example, audio data representing an utterance (e.g., Utterance 106 in 1 ), over a network 150, such as the Internet, to the voice processing system 100 using any number of communication protocols, such as Transfer Control Protocol and Internet Protocol ("TCP/IP") (e.g., any of the protocols, used in each of the TCP/IP layers), Hypertext Transfer Protocol (“HTTP”), WebRTC, SIP, Wireless Application Protocol (“WAP”), etc. The communication circuit 224 may use any communication protocol, such as any of the foregoing mentioned exemplary communication protocols. In some embodiments, a voice-enabled device 102 may include one or more antennas to facilitate wireless communication with a network using various wireless technologies (e.g., Wi-Fi, Bluetooth, radio frequency, etc.). In another embodiment, a voice-enabled device 102 may include one or more Universal Serial Bus ("USB") ports, one or more Ethernet or broadband ports, and/or any other type of hard-wired access port such that the communications circuitry 224 of a voice-enabled device 102 enables communication with one or more communication networks.

A voice-assisted device 102 may also include one or more microphones 226 and/or transducers. Additionally or alternatively, one or more microphones located in a separate device may communicate with a voice-assisted device 102 to capture sounds for a voice-assisted device 102. The microphone(s) 226 may be any suitable component that audio signals. For example, the microphone(s) 226 may include one or more sensors for generating electrical signals and circuitry capable of processing the generated electrical signals. In some embodiments, the microphone(s) 226 may include multiple microphones capable of detecting different frequency levels. As an illustrative example, a voice-enabled device 102 may include multiple microphones (e.g., four, seven, ten, etc.) placed at various locations around a voice-enabled device 102 to monitor/capture any audio output in the environment in which a voice-enabled device 102 is located. The various microphones 226 may include some microphones optimized for distant sounds, while some microphones optimized for sounds occurring in the near range of a voice-enabled device 102. In some embodiments, the microphone(s) 226 may begin detecting audio signals only in response to manual input to a voice-enabled device 102. For example, a manually activated device may begin collecting audio data using the microphone(s) 226 in response to user input, such as pressing a button, tapping a touchscreen, or providing a touch input gesture to a touch input component. A voice-enabled device 102 may include one or more speakers 228. In addition, a voice-enabled device 102 may communicate with one or more speakers 228. The speaker(s) 228 may correspond to any suitable mechanism for outputting audio signals. For example, the speaker(s) 228 may include one or more speaker units, speaker enclosures, transducers, speaker assemblies, and/or transducer assemblies that may be capable of emitting audio signals and/or audio content to an environmental area in which the voice-enabled device 102 may be located. In some embodiments, the speaker(s) 228 may include headphones or earbuds that may be wirelessly or wired connected to a voice-assisted device 102 that may be capable of transmitting audio directly to a person.

In some embodiments, one or more microphones 226 may serve as input devices for receiving audio input. A voice-assisted device 102 in the previously mentioned embodiment may then also include one or more speakers 228 to output acoustic responses. In this way, a voice-enabled device 102 may function solely via voice or audio, without the use or need of any input mechanisms or displays, but this is merely exemplary.

The display component 232 may correspond to a display device and/or touch screen that may be of any size and/or shape and may be located on any portion of a voice-enabled device 102. Various types of displays may include, but are not limited to, liquid crystal displays (“LCD”), monochrome displays, color graphics adapter (“CGA”) displays, enhanced graphics adapter (“EGA”) displays, variable graphics array (“VGA”) displays, or any other type of display or any combination thereof. Additionally, in some embodiments, a touch screen may correspond to a display device having capacitive sensor pads that can detect touch inputs thereon. For example, the display component 232 may correspond to a projected capacitive touch (“PCT”). In some embodiments, the display component 232 may be an optional component for a voice-enabled device 102. For example, a voice-enabled device 102 may not include a display component 232. Such devices, sometimes referred to as "monitorless" devices, may output audio or communicate with a display device to output visible content.

In some embodiments, the content displayed on the display component 232 may be formatted such that contextual entities and lists may be analyzed by the language processing system 100 for list resolution and/or anaphora resolution. Context associated with the displayed content may include entities associated with a voice-enabled device 102, including, but not limited to, foreground entities (e.g., lists of items, detail pages), background entities (e.g., songs, audiobooks), and notification entities. The context data can be structured into context entity slots, list metadata, and any other additional data available. For example, contextual entity slots can correspond to data used for list resolution and/or anaphora resolution. The contextual entity slots can be specified in domain definitions with corresponding values. The list metadata may include list identifiers, element identifiers for elements of a list and absolute positions of the list for a specific element (e.g. a first element of a list, a second element of a list, etc.). Such additional data can include unique identifiers associated with an object, item prices, quantities, and the like.

In some embodiments, a voice-enabled device 102 may include one or more cameras 230 corresponding to one or more suitable image capture components or components capable of capturing one or more images and/or videos. The camera(s) 230 may, in some embodiments, be configured to capture photos, photo sequences, rapid-fire shots (e.g., multiple photos taken in succession during a relatively short period of time), videos, or any other type of image, or any combination thereof. In some embodiments, a voice-enabled device 102 may include multiple cameras 230, such as one or more front-facing cameras and/or one or more rear-facing cameras. In addition, the camera(s) 230 may be configured to detect far-field images (e.g., objects located at a great distance from a voice-enabled device 102) or near-field images (e.g., objects located at a relatively close distance from a voice-enabled device 102). In some embodiments, the camera(s) may be high-definition (“HD”) cameras capable of acquiring images and/or videos at a substantially high resolution (e.g., 720p, 1080p, 1080i, 4K, etc.). In some embodiments, the camera(s) 230 may be optional for a voice-enabled device 102. For example, the camera(s) 230 may be external to and in communication with a voice-enabled device 102. For example, an external camera may be capable of capturing images and/or videos that may then be provided to a voice-enabled device 102 for display and/or processing.

In some embodiments, display component 232 and/or camera(s) 230 may be optional for voice-enabled device 102. For example, voice-enabled device 102 may operate using audio inputs and may output audio in response or initiate one or more actions in response, and therefore display component 232 and/or camera(s) 230 may not be included. Additionally, in some embodiments, voice-enabled device 102 may not include display component 232 and/or camera(s) 230, but may instead communicate with display component 232 and/or camera(s) 230. For example, voice-enabled device 102 may be connected to a display screen via a Wi-Fi connection (e.g., 802.11 protocol) such that visual content sent to voice-enabled device 102 may be sent to and output through the display screen.

In some embodiments, context data may be obtained through computer vision analysis of an object detected by the camera(s) 230. For example, in response to uttering the utterance "buy this," a voice-enabled device 102 may cause the camera(s) 230 to capture an image.

This image can be analyzed to determine what the object is, and the context data associated with identifying those objects can be determined. For example, if a person is holding up a ketchup bottle, the computer vision analysis may be able to determine a product name, barcode, and/or other attribute of the ketchup bottle from the captured image and populate a context data structure specifying the determined values (e.g., 'Item Name' slot: "Ketchup").

The voice-assisted device 102 may communicate with the voice processing system 100 via one or more networks 150. The one or more networks 150 may include any suitable communications technology, such as wired and/or wireless modalities and/or technologies. The networks 150 may be any combination of Personal Area Networks (“PANs”), Local Area Networks (“LANs”), Campus Area Networks (“CANs”), Metropolitan Area Networks (“CANs”). MANs"), extranets, intranets, the Internet, short-range wireless communications networks (e.g. ZigBee, Bluetooth, etc.), wide area networks ("WANs") - centralized and/or distributed - and/or any Include combination, permutation and/or aggregation thereof.

As in 2 As shown, a speech processing system 100 may include various subsystems, components and/or modules, including, among others, an ASR system 202, an NLU system 204, a domain system 206, a TTS system 208, a cross-domain routing system 210, an intra-domain Routing system 212, a context data management system 214, a user data store 216 and a processing data store 218. In some embodiments, voice ver Processing system 100 also include an orchestrator system (not shown) capable of orchestrating one or more processes to be executed by one or more of the ASR system 202, the NLU system 204, the domain system 206, the TTS System 208, the cross-domain routing system 210, the intra-domain routing system 212 and one or more additional components, devices and / or systems associated therewith. The speech processing system 100 may also include computer-readable media, including, but not limited to, flash memory, random access memory (“ ^R_AM ”), and/or read-only memory (“ROM”). The voice processing system 100 may also include various modules that store software, hardware, logic, instructions and/or commands for the voice processing system 100, such as a speaker identification module (“ID”) or any other module or any combination thereof.

The ASR system 202 may be configured to recognize human speech in recognized audio, such as audio captured by a speech-enabled device 102, which may then be transmitted to the speech processing system 100. The ASR system 202, in one embodiment, may include one or more processors 252, data storage/memory 254, and communication circuitry 256. The processor(s) 252, data storage/memory 254, and communication circuitry 256, in some embodiments, may be substantially similar to the processor(s) 220, data storage/memory 222, and communication circuitry 224 described in more detail above; therefore, the foregoing descriptions apply. Additionally, in some embodiments, the ASR system 202 may include an STT system 280. The STT system 280 may employ various speech-to-text techniques.

The ASR system 202 may transcribe received audio data using the STT system 280 into text data representing the words of speech included in the audio data. The text data may then be used by other components for various purposes, such as executing system commands, entering data, etc. The different ways in which a spoken utterance may be transcribed (i.e., the different hypotheses) may each be assigned a probability or confidence score that indicates the likelihood that a specific set of words matches the words spoken in the utterance. The confidence score may be based on a number of factors, including, for example, the similarity of the sound in the utterance to models of speech sounds (e.g., an acoustic model) and the likelihood that a specific word matching the sounds would be included at the specific location in the sentence (e.g., using a language or grammar model). Thus, each potential textual representation of the spoken utterance (hypothesis) is assigned a confidence score. Based on the factors considered and the assigned confidence score, the ASR system 202 can output the most likely textual representation(s) of the audio data.

The ASR system 202 may generate result data in the form of, for example, a single textual representation of the language, an N-best list including multiple hypotheses and respective scores, and/or a grid, which may be sent to the NLU system 204 for processing, such as Converting the text into commands for execution, either by a voice-enabled device 102, a voice processing system 100, or by another device, such as a separate device or server, that can perform one or more additional functionalities (e.g., a TV that can output video content).

The NLU system 204 may be configured to determine an intent of an utterance based on the received audio data. The NLU system 204 may include processor(s) 252, data storage/memory 254, and communication circuitry 256, which in one embodiment may be substantially similar to the processor(s) 220, data storage/memory 222, and communication circuitry 224 of the electronic device 102; therefore, the previous description applies.

The NLU system 204 may include a named entity recognition (“NER”) system 272 that may be used to identify portions of text that correspond to a named entity that is recognizable by the NLU system 204. A downstream process called named entity resolution 292 may be configured to associate a portion of text with an actual specific known entity. To perform named entity resolution, the system may utilize directory data stored in an entity library store. The directory data can be used for entity resolution, such as matching ASR results with different entities (such as song titles, contact names, etc.). Directories can be associated with a user account or profile of the user account data store 216 and linked to certain domains (e.g. music or shopping) or organized in various other ways.

Generally described, the NLU system 204 may accept text input and attempt to perform a semantic interpretation of the text. That is, the NLU system 204 may be configured to determine a meaning of text data (e.g., based on the individual words represented by the text data) and then implement that meaning. In some embodiments, the NLU system 204 may interpret a text string to derive an intention or desired action of the utterance (e.g., utterance 106) as well as the relevant pieces of information in the text that enable completion of an action. For example, if a spoken utterance is processed by the ASR system 202 and outputs the text "Call Mom," the NLU system 204 may determine that an intention of the utterance is to activate a telephone or telephone feature and place a call to one Initiate contact that matches the entity “Mom”. In some embodiments, the NLU system 204 may process multiple text inputs related to the same utterance. For example, if the ASR system 202 outputs N text segments (as part of an N-best list), the NLU system 204 can process all N outputs.

The NLU system 204 may be configured to analyze and label, mark, or otherwise annotate text. For example, in the text "Call Mom," the word "call" can be marked as a command (e.g., a command to make a phone call), and the word "Mom" can be marked as the specific entity and target of the command (e.g., B. a phone number for the entity that corresponds to “Mom” and is stored in a contact list). Additionally, the NLU system 204 may be used to provide response data in response to queries, for example, using a knowledge base stored in the data store/memory 254 of the NLU system 204 and/or in the data store/memory of the speech processing system 100.

An intent classification (“IC”) system 274 may analyze the query to determine one or more intents, where the intent corresponds to the action to be taken in response to the query. The intentions identified by the IC system 274 can be linked to grammar frameworks that have fields, also known as “slots,” for filling. Each slot can correspond to a section of the query text that the system believes corresponds to an entity. For example, if "play music" is an identified intent, one or more grammar frameworks can support sentence structures such as "Play {artist name}", "Play {album name}", "Play {song name}", "Play {song name} by {artist name}" “ etc. correspond. However, to make resolution more flexible, these frameworks may not be structured as sentences, but rather based on a mapping of slots to grammatical markers. As another example, if "Play 'Song 1'" is an identified intention, a grammar framework may correspond to sentence structures such as "Play {Song 1}".

• „Künstler 1“ analysiert und markiert werden. An diesem Punkt des Prozesses kann „Abspielen“ basierend auf einer Wortdatenbank, die der Musikdomäne zugeordnet ist, als Verb identifiziert werden, wobei das IC-Modul 274 bestimmen kann, dass es der Intention „Musik abspielen“ entspricht. Es wurde keine Bestimmung über die Bedeutung von „Lied 1“ und „Künstler 1“ vorgenommen, aber basierend auf einem Modell, wie etwa einem multimodalen Modell, kann bestimmt werden, dass sich der Text dieser Phrasen auf das grammatikalische Objekt bezieht (d. h. Entität) der Abfrage bezieht.

The NER system 272 may parse the query to identify words as subject, object, verb, preposition, etc. based on grammar rules and/or models before resolving named entities. The identified verb may be used by the IC module 274 to identify the intention, which is then used by the NER system 272 to identify frameworks. A framework for a "play" intention may specify a list of slots/fields/placeholders applicable to placing the identified "object," such as, for example, {artist name}, {album name}, {song name}, {application name}, {anaphoric term}, and any object modifier (e.g., a prepositional phrase). The NER system 272 may then search the appropriate fields in the domain-specific and personalized lexicon(s) and attempt to match words and phrases in the query marked as a grammatical object or object modifier with those identified in the database(s). For example, a query of "Play Song 1` by 'Artist 1'" could be parsed and returned as {verb}: "play", {object}: "song 1", {object preposition} "by", and {object modifier}:

• "Artist 1" may be analyzed and tagged. At this point in the process, "play" may be identified as a verb based on a word database associated with the music domain, where IC module 274 may determine that it corresponds to the intent "play music." No determination has been made about the meaning of "song 1" and "artist 1," but based on a model, such as a multimodal model, it may be determined that the text of these phrases refers to the grammatical object (i.e., entity) of the query.

The frameworks associated with the intent can then be used to determine which database fields to search to determine the meaning of those phrases, such as for example, searching a user's directory for similarity to the framework slots. So, a framework for the intent "play music" might attempt to resolve the identified object for {artist name}, {album name}, {song name}, and {application name}, {anaphoric term}, and another framework for the same intent might attempt to resolve the object modifier based on {artist name} and resolve the object based on {album name} and {song name} associated with the identified {artist name}. If searching the directory does not resolve a slot/field using directory data, the NER system 272 may search the database for generic words associated with the concrete domain. For example, if the query was "play songs by artist 1", the NER system 272 may search the domain vocabulary for the word "songs" after failing to determine an album name or song name named "songs" by "artist 1". For example, using the object "songs" may match some or all of the songs associated with a specific artist (e.g., "artist 1"). Alternatively, generic words may be checked before the directory data, or both could be tried, potentially yielding two different results.

• {Intention}: „Musik abspielen“, {Künstlername}: „Künstler 1“, {Medienart}: „Lied“ und {Liedname}: „Lied 1.“ Als weiteres Beispiel könnte „Lieder von Künstler 1` abspielen“ Folgendes erzeugen: {Intention}: „Musik abspielen“, {Künstlername}:
  • „Künstler 1“ und {Medienart}: Lied. Darüber hinaus könnte „Dies zu meinem Warenkorb hinzufügen“ zu folgendem Ergebnis führen: {Intention} „Artikel hinzufügen“, {Anaphorischer Begriff}: „dies“ und {Listenart} „Warenkorb“.

The results of natural language understanding processing can be marked or labeled to assign meaning to the query. For example, playing 'Song 1' by Artist 1" might produce the following result:

• {Intention}: "Play music", {Artist name}: "Artist 1", {Media type}: "Song" and {Song name}: "Song 1." As another example, "Play songs by Artist 1" might produce: {Intention}: "Play music", {Artist name}:
  • "Artist 1" and {Media Type}: song. Additionally, "Add this to my cart" might result in {Intent} "add item", {Anaphoric Term}: "this", and {List Type} "cart".

• (0,93) Intention: Film abspielen; Titel: „Harry Potter und der Stein der Weisen“
• (0,03) Intention: Film abspielen; Titel: „Harry Potter und die Heiligtümer des Todes Teil 2“
• (0,02) Intention: Film abspielen; Titel: „Harry Potter und die Heiligtümer des Todes Teil 1“
• (0,01) Intention: Musik abspielen; Titel: „Harry Potter Original-Film-Soundtrack“
• (0,01) Intention: Hörbuch abspielen; Titel: „Harry Potter und der Stein der Weisen“

The NLU system 204 may generate multiple intention hypotheses to be considered by downstream processes. In some embodiments, the set of intention hypotheses may be arranged in an N-best list of intentions and slots corresponding to the top choices regarding the meaning of an utterance, along with scores for each item. For example, for ASR output data representing the utterance "Play Harry Potter", the NLU system 204 may generate output in the form of an N-best list s of scored intention hypotheses, such as:

• (0.93) Intention: Play movie; Title: “Harry Potter and the Philosopher’s Stone”
• (0.03) Intention: Play movie; Title: “Harry Potter and the Deathly Hallows Part 2”
• (0.02) Intention: Play movie; Title: “Harry Potter and the Deathly Hallows Part 1”
• (0.01) Intention: Play music; Title: “Harry Potter Original Motion Picture Soundtrack”
• (0.01) Intention: Play audio book; Title: “Harry Potter and the Philosopher’s Stone”

The NLU system 204 may generate scores for intentions and content slots using the text data and other data, such as context data. The scores may indicate how likely it is that individual labels are the correct labels for individual words of the utterance being processed. In the present example, the utterance "Play Harry Potter" includes three words: "Play," "Harry," and "Potter." The individual words may be labeled using a predetermined set of labels, including different labels for the different intentions recognizable by the NLU system 204 and different labels for the content slots corresponding to the different intentions. This process may involve semantic tagging, that is, labeling a word or combination of words according to their type/semantic meaning. The labeling may be performed using a NER model alone or in combination with heuristic grammar rules. The NER model can be built using techniques such as hidden Markov models, maximum entropy models, log-linear models, conditional random fields (“CRFs”), or other models configured to generate label data (e.g., scores, probabilities, etc. for individual labels) using text data and, in some cases, context data.

In some embodiments, the NLU system 204 may include domain-specific recognizers for each of multiple domains, and each of the domain-specific recognizers may include its own NER component 272, IC component 274, etc. For example, can the NLU system 204 includes recognition devices for a shopping domain, a music domain, a video domain and a communication domain.

The NLU system 204 may also include a slot filler component 290. The slot filler component 290 can take text from slots and modify it to make it easier for downstream components to process the text. The operations of the slot filler component 290 are typically low latency operations that do not involve expensive operations such as referencing a knowledge base. The purpose of the slot filler component 290 is to replace words with other words or values that can be more easily understood by downstream components. For example, if a text interpretation represented in the text data 320 includes the word "tomorrow", the slot filler component 290 may replace the word "tomorrow" with an actual date for downstream processing purposes. Likewise, the word “CD” can be replaced with the word “Album” from the words “Compact Disc”. The replaced words can then be included in the cross-domain N-best list data.

N-best list data can then be sent to an entity resolution component 292. The entity resolution component 292 may apply rules or other instructions to standardize labels or tokens from previous phases into an intent/slot representation. The exact transformation may depend on the domain (e.g., for a travel domain, a text mention of “Boston Airport” may be converted to the standard three-letter BOS code referring to the airport). The entity resolution component 292 may refer to an authority source (e.g., a knowledge base) that is used to specifically identify the exact entity referenced in the entity mention identified in each slot included in the cross-domain N -best list data is presented. Specific intention/slot combinations can also be tied to a specific source, which can then be used to resolve the text. In the “Play Songs by the Stones” example, the entity resolution component 292 may reference a personal music catalog, an Amazon Music account, a user account, or the like. The output from the entity resolution component 292 may include modified N-best list data based on the cross-domain N-best list represented in the cross-domain N-best list data, but may also include more detailed data (e.g., entity IDs). include the specific entities mentioned in the slots and/or more detailed slot data that may ultimately be used by an application 282, which may be integrated into the same system components or pipeline, or located in a separate device in communication with the NLU system 204 can be located. There may be multiple entity resolution components 292, where a specific entity resolution component 292 may be specific to one or more domains.

The NLU system 204 may generate NLU output data from the N-best list data described above. The NLU output data may include a top-scoring interpretation from the cross-domain N-best list data, or it may be data representing an N-best list of the highest-scoring interpretations. In some embodiments, the NLU system 204 may re-evaluate, influence, or otherwise modify the N-best list data generated by the entity resolution component 292. To this end, the NLU system 204 may consider not only the N-best list data generated by the entity resolution component 292, but also other data. The other data can include many different data. The other data may include, for example, application rating or popularity data. For example, if an application has a particularly high ranking, the NLU system 204 may increase the score of the results associated with that specific application. The other data may also include data about applications that have been specifically activated by the user (as specified in a user account). NLU output data associated with enabled applications may be valued higher than results associated with non-enabled applications. An example of NLU system processing is shown in US Patent No. 10,515,625 , issued December 24, 2019, which is incorporated herein by reference.

For example, the domain system 206 may correspond to various action-specific applications 282 capable of processing various task-specific actions and/or performing various functionalities related to the user experience. The domain system 206 may further correspond to first-party applications and/or third-party applications capable of performing various tasks or actions or performing various functionalities. For example, based on the context of the audio received from a voice-enabled device 102, the speech processing system 100 may execute a certain application 282 to generate a response or obtain response data, which in turn may be communicated back to a voice-based device 102 and/or to another electronic device (e.g., a television). The domain system 206 may also include processor(s) 252, data storage/memory 254, and communication circuitry 256. In some embodiments, an application 282 of the domain system 206 may be written in various computer languages, such as JavaScript and Java.

The TTS system 208 may employ various text-to-speech techniques to present to a user (e.g., a spoken response to an utterance). The TTS system 208 may also include processor(s) 252, data storage/memory 254, communication circuitry 256, and a speech synthesizer 284.

The cross-domain routing system 210 may manage the routing of utterances to the appropriate domain, as described in more detail below. The cross-domain routing system 210 may also include processor(s) 252, data store/memory 254, communications circuitry 256, and speech synthesizer 284.

The intra-domain routing system 212 may manage the routing of utterances within a particular domain to the appropriate subdomain and/or application 282, as described in more detail below. The intra-domain routing system 212 may also include processor(s) 252, data storage/memory 254, communication circuitry 256, and speech synthesizer 284.

The context data management system 214 may manage the collection of context data to be used in routing determinations, as described in more detail below. The context data management system 214 may also include processor(s) 252, data storage/memory 254, the communication circuitry 256, and the speech synthesizer 284.

The user data store 216 may store data representing or otherwise associated with one or more user accounts or user profiles corresponding to users having an account in the speech processing system 100. In some embodiments, data, settings, and/or preferences for each user profile may be stored by the user data store 216. For example, the user data store 216 may store data regarding past user interactions with the speech processing system 100, such as data regarding actions the speech processing system 100 has taken in response to utterances from a user associated with a particular user profile. As another example, the user data store 216 may store data regarding one or more voice-enabled devices 102 associated with a user profile, such as device identifiers, phone numbers, network addresses, version information, data regarding features and installed applications, and the like. In some embodiments, the user data store 216 may include a list of media items currently stored in an individual's registered account or user profile. For example, a list of music or videos purchased or obtained by a person may be stored in the person's user profile in the user data store 216, which the person may access when he or she wants to listen to a specific song or songs or watch a specific video or videos. The user data store 216 may also include a list of all applications currently enabled for each user profile. In some embodiments, the NLU system 204 may receive indications of which applications are currently enabled for a specific user profile or account so that the NLU system 204 knows what rules and capabilities this speech processing system 100 can perform for the specific user profile or account. In some embodiments, the user data store 216 may store a speech signal, such as biometric speech data, for a specific user profile. This allows speaker identification techniques to be used to match a voice to biometric speech data associated with a specific user profile. The examples of data stored in user data store 216 are illustrative only and are not exhaustive, required, or limiting of the data that may be stored in user data store 216. Processing data store 218 may store data to be used during processing of an utterance, such as context data obtained and/or generated by context data management system 214, as described in more detail above and below. Such data may be used in routing determinations, such as those made by intra-domain routing system 212. In some embodiments, processing data store 218 may store data to be used offline, such as after (or without) processing of an utterance. For example, data related to the utterance, context data items obtained during processing of the utterance obtained, feedback, etc. are stored in the processing data store 218 and used during offline processes, such as training a model used by one or more components of the speech processing system 100.

Although each of the ASR system 202, the NLU system 204, the domain systems 206, the TTS system 208, the cross-domain routing system 210, the intra-domain routing system 212, the context data management system 214, the user data store 216, and the processing data store 218 may each include instances of the processor(s) 252, the data storage/memory 254, and the communication circuitry 256, these instances of the processor(s) 252, the data storage/memory 254, and the communication circuitry 256 may be located within each of the ASR system 202, the NLU system 204, the domain systems 206, the TTS system 208, the cross-domain routing system 210, the intra-domain routing system 212, the Context data management system 214, user data store 216, and processing data store 218. For example, the structure, functionality, and style of the processor(s) 252 in the ASR system 202 may be substantially similar to the structure, functionality, and style of the processor(s) 252 in the NLU system 204, but the actual processor(s) 252 need not be the same entity.

Domain routing

3 is a diagram illustrating data flows and interactions between components of the speech processing system 100 during processing and routing of an utterance to an application for a response or other action. Sections of 3 is with further reference to 4 which is a flow diagram of an illustrative process that may be performed by an intra-domain routing system 212 to generate routing confidence scores for the utterance.

As shown, an utterance may be processed using a set of natural language processing operations, such as those performed by the ASR system 202 and the NLU system 204, to generate one or more intentions, such as an N-best list 300. Before, during, and after the generation of the intention(s), various context data items 302 may be obtained and/or generated by the context data management system 214. An exemplary process for obtaining and/or generating context data items for use by one or more intra-domain routing systems 212 is described in more detail below.

The cross-domain routing system 210 may determine which domains are associated with individual intentions of the N-best list 300. In some embodiments, individual domain systems 206 may be assigned, registered, or otherwise associated with different intentions. The cross-domain routing system 210 may determine which domain systems 206 are associated with each of the intentions in the N-best list 300 (or a subset thereof). The cross-domain routing system 210 may generate and send confidence queries 304 to the intra-domain routing systems 212 for the particular domain systems 206. For example, the N-best list 300 may include five intentions ranked 1-5, where 1 is the highest ranked intention and 5 is the fifth highest ranked intention. A particular domain system 206 may be registered to handle two of the five intentions (e.g., the intentions at rank 1 and 4), another domain system may be registered to handle one of the intentions (e.g., the intention at rank 2), and so on. The cross-domain routing system 210 may generate a confidence request 304 to the intra-domain routing system 212 for the domain 206 registered to handle the intentions at rank 1 and 4. The confidence request 304 may be a request to the intra-domain routing system 212 to evaluate whether the corresponding domain 206 or a specific application 282 thereof is the correct entity to respond to the user's utterance. A confidence request may include the intentions and, in some cases, may include additional information, such as ranking information or the like.

Upon receipt of a confidence request 304, the intra-domain routing system 212 may identify one or more routing confidence providers 320 to generate routing confidence scores that are used to assign a specific application 282, to generate a response to the utterance, or to to specify that no application in the domain should generate a response. Even though the routing confidence providers are 320 in 3 As shown as being external to the intra-domain routing system 212, in some embodiments, some or all of the routing confidence providers 320 may be integrated into the intra-domain routing system 212. For example The intra-domain routing system 212 may include routing confidence providers for each application of the corresponding domain or for a subset thereof. As another example, one or more routing confidence providers may be hosted by a computing system external to the intra-domain routing system 212, such as a computing system on which a corresponding application is implemented. To illustrate, the domain may be configured with an application programming interface (“API”) that allows third parties to deploy applications and corresponding routing confidence providers to a specific domain.

4 is a flowchart of an illustrative process 400 that may be performed by an intra-domain routing system 212 to generate routing confidence scores in response to confidence queries 304. The method 400 begins at block 402. When the process 400 is initiated, a set of executable program instructions stored on one or more non-transitory computer-readable media (e.g., hard drive, flash memory, removable storage, etc.) may be written into memory (e.g. random access memory or “RAM”) of a computing device are loaded and executed. For example, executable instructions may be loaded into the memory 254 of a computing device of the intra-domain routing system 212 and executed by one or more processors 252. In some embodiments, the process 400 or portions thereof may be implemented on multiple processors in series or in parallel. At block 404, the intra-domain routing system 212 may obtain the confidence query 304 from the cross-domain routing system 210. In some embodiments, the confidence query may be obtained by or provided to a specific component of the intra-domain routing system 212, such as a master speechlet 310 for the corresponding domain 206.

At block 406, the master speechlet 310 may use another component of the intra-domain routing system 212, such as the routing arbiter 314, to determine the routing confidence providers 320 from which to obtain routing confidence scores. For example, a specific domain system 206 may be associated with any number of subdomains and/or applications 282. A routing confidence provider 320 may be implemented for each individual subdomain or application 282 or various subsets thereof. Individual routing confidence providers 320 may be configured to generate routing confidence scores (e.g., scores) that indicate the level of confidence that a specific subdomain or application 282 is the appropriate entity to respond to an utterance. The routing decider 314 or another component of the intra-domain routing system 212 may identify the routing confidence providers 320 from which routing confidence scores are to be obtained based on a predetermined or dynamically determined mapping of intentions 300 to routing confidence providers 320. For each intent 300 in the confidence query 304, the routing decider 314 may identify the routing confidence provider(s) 320 from which a routing confidence score is to be obtained.

At block 408, the routing arbiter 314 or another component of the intra-domain routing system 212 may obtain the context data elements 302, if any, to be provided to the routing confidence providers 320. In some embodiments, a mapping may be used to determine the specific context data elements to provide to a given routing confidence provider 320. For example, one routing confidence provider 320 may make a routing confidence assessment using location data associated with the voice-enabled device 102 and capability data representing capabilities of the voice-enabled device, while another routing confidence provider 320 may make a routing confidence assessment based on specific entities recognized in the utterance and concrete content elements that were displayed by the voice-assisted device 102 during the utterance. The routing arbiter 314 may obtain the context data items associated with each of the routing confidence providers 320. In some embodiments, the context data items may be obtained from the processing data store 218, from the context data management system 214, and/or from other sources.

At block 410, the routing arbiter 314 may obtain routing confidence scores from the identified routing confidence providers 320. The routing arbiter 314 may generate requests, commands, or other communications to the routing confidence providers 320 and may include context data items 302 in the same communication or otherwise provide access to the context data items 302 (e.g., by providing a storage or network address of the Location from which the routing confidence providers 320 can obtain the context data elements 302). In some embodiments, the routing confidence providers 320 may obtain the context data items 302 directly from the Processing data store 218, from the context data management system 214 or another source, without receiving or otherwise gaining access to the context data elements 302 from the routing arbiter. Individual routing confidence providers 320 may perform routing confidence assessments in various ways.

In some embodiments, a routing confidence provider 320 may apply a set of one or more deterministic rules. For example, a set of rules may include applying a particular score or factor when the intention being evaluated is an intention, and a different score or factor when the intention is a different intention. As another example, a score or factor may be used depending on the ranking of the intention within the N-best list (e.g., if the ranking is relatively low, the confidence score may be lower than if the ranking were relatively high). The same, additional, and/or alternative scores and/or factors may be included depending on the values of all context data elements 302 considered (e.g., geolocation of the voice-assisted device 102, geographic distance of the voice-assisted device 102 from a point of interest, content displayed by the voice-assisted device 102 during the utterance, individual words or phrases present in the utterance, historical usage by an active user profile, content catalog associated with the active user profile, etc.). The routing confidence provider 320 may apply the various scores, factors, calculations, and the like based on the set of deterministic rules and arrive at an output routing confidence score. The routing confidence score may be a score, for example, a score between a minimum (representing a minimum level of confidence that the utterance will be properly handled by an application associated with the routing confidence provider 320) and a maximum (representing a maximum level of confidence). In some embodiments, the routing confidence score may include generating a classification into one or more possible classes, such as a first class indicating a low level of confidence, a second class indicating a high level of confidence, and a third class indicating a medium level of confidence associated with ambiguity.

In some embodiments, a routing confidence provider 320 may apply a statistical model or other non-deterministic calculation. For example, a statistical model may be derived from a set of observed and/or synthetic data. Applying the statistical model may include obtaining or deriving input values representing the intent to be evaluated, the ranking of the intent within the N-best list, the values of all context data items 302 considered, etc. The routing confidence provider 320 may apply the various scores, factors, calculations, and the like based on the statistical model and arrive at an output routing confidence score. The routing confidence score may be a score, such as a score between a minimum and a maximum, a classification into one or more possible classes, etc.

In some embodiments, a routing confidence provider 320 may use a combination of deterministic rules and a statistical model to arrive at a routing confidence score. In some embodiments, additional and/or alternative routing confidence assessment methods may be used. The example routing confidence assessment procedures described herein are for illustrative purposes only and are not intended to be limiting, required, or exhaustive.

At decision block 412, the routing arbitrator 314 or another component of the intra-domain routing system 212 may determine whether the routing confidence score(s) received from the routing confidence provider(s) 320 indicate ambiguity. If so, the process 400 may proceed to block 414 where a response to the confidence request indicating ambiguity may be generated. Otherwise, if the routing confidence scores do not indicate ambiguity, the process 400 may proceed to decision block 416.

In some embodiments, identification of an ambiguity may be based on whether a confidence score made by a routing confidence provider 320 indicates an ambiguity. For example, if the routing score is a score between two extremes indicating the lowest and highest confidence, respectively, then a score that is at least a threshold distance from both extremes (e.g., in the middle 50% of the range, in the middle 33% of the range, etc.) may trigger identification of an ambiguity. As another example, if the confidence score is one of three classes indicating low, high, and medium confidence, respectively, then a score in the middle class triggers identification of an ambiguity. In some embodiments, identification of an ambiguity may be based on whether a highest confidence score of all confidence scores requested by the routing decider 314 indicates an ambiguity.

At block 414, the routing arbitrator 314 or another component of the intra-domain routing system 212 may generate a response to the confidence request based on the identification of the ambiguity. In some embodiments, the response to the confidence request may indicate that an ambiguity resolution component 312 of the intra-domain routing system 212 is the appropriate entity to respond to the utterance. The ambiguity resolution component 312 may be configured to resolve routing ambiguities that remain after (or are introduced during) ASR, NLU, and other processes. The ambiguity resolution component 312 may resolve ambiguity in interpreting the utterance and determine a target entity (subdomain or application) to respond. For example, the ambiguity resolution component 312 may perform non-interactive user involvement in resolving the ambiguity by providing a relevant hint/suggestion and allowing the user to use a new phrasing. The reworded utterance may then be edited using the utterance processing described above (e.g., edited as if the previous utterance had not occurred) without being passed back to the ambiguity resolution component 312. As another example, the ambiguity resolution component 312 may perform interactive user involvement (e.g., a multi-turn dialogue) to resolve the ambiguity through clarifications and/or confirmations. In this case, the user's responses may be passed back to the ambiguity resolution component 312 as part of the ambiguity resolution component 312's management of a multi-turn dialogue. To illustrate, interactive engagement may involve presenting options to the user, such as two or more possible applications (e.g., corresponding to the two or more highest routing confidence scores), and prompting the user to select one of them. As another example, the ambiguity resolution component 312 may obtain additional context data elements 302 that may not have been available during the initial routing confidence evaluation process, and may use these additional context data elements 302 to resolve the ambiguity without requiring additional user input (e.g., by adjusting or overriding one or more routing confidence scores).

In decision block 416, the routing decider or another component of the intra-domain routing system 212 may determine whether the routing confidence scores indicate a subdomain or application of the domain 206 for which the intra-domain routing system 212 is generating routing confidence scores and the correct one Entity to respond to the utterance is. If so, the process 400 may continue to block 418, where the routing arbiter 314 or another component of the intra-domain routing system 212 may generate a response to the confidence query indicating the subdomain or application of the domain 206, for which there is a high degree of confidence that it is the correct entity to respond to the utterance. Otherwise, the process 400 may continue to block 420, where the routing arbiter 314 or another component of the intra-domain routing system 212 may generate a response to the confidence query indicating that no subdomain or application of the domain 206 has the correct entity for Reacting to the statement is.

At block 422, the intra-domain routing system 212 may provide the output generated as above in response to the confidence request to the inter-domain routing system 210. The process 400 may end at block 424.

5 is a diagram of a multi-tier domain 206 that includes multiple applications 282. The process 400 may determine that each of the applications 282A, 282B, 282C is the most appropriate application to respond to the utterance depending on the context data obtained and used to perform routing confidence assessments. For illustration, the utterance 106 is the same utterance from the example in 1 : "Where can I get coffee?" The three example applications are a physical store location application 282A, a store aisle location application 282B, and an online shopping application 282C. If the utterance 106 is made and processed in conjunction with a first set of context data (e.g., location information indicating that the utterance was made outside of the user's home 104 and outside of a store), then the utterance 282A is a physical store may be the most appropriate application for responding to utterance 106. If utterance 106 is made and processed in conjunction with a second set of contextual data (e.g., location information indicating that the utterance was made in a store), then store aisle locating application 282B may be the most appropriate application for responding to utterance 106. If utterance 106 is made and processed in conjunction with a third set of contextual data (e.g., content presentation information indicating that the user is searching for items to purchase online), then online shopping application 282B may be the most appropriate application for responding to utterance 106.

Early invocation and context data generation

6 is a diagram illustrating data flows and interactions between components of the speech processing system 100 for invoking context data generation actions early during processing of an utterance. Sections of 6 is with further reference to 7 which is a flowchart of an illustrative process that may be performed by the context data management system 214 to generate context data associated with an utterance, and 8th , which is a diagram illustrating data flows and interactions between components of a context data management system 214 during early invocation of context data generation actions. As in 6 As shown and described in more detail above, an utterance may be processed using a set of natural language processing operations, such as those performed by the ASR system 202 and the NLU system 204, to generate one or more intentions. The cross-domain routing system 210 may generate one or more confidence queries 304 based on the intentions.

In some cases, the natural language processing operations performed by the ASR system 202 and the NLU system 204 may incur or experience a relatively high degree of latency. To reduce or eliminate additional user-perceived latency that may be caused by obtaining and/or generating context data items 302 for use in routing confidence assessments, the context data items 302 may be proactively obtained and/or generated without necessarily waiting for the natural language processing operations to complete. For example, before, during, and after the generation of the intent(s), various context data items 302 may be obtained and/or generated by the context data management system 214. The points at which the context data management system 214 receives and/or generates context data items may be referred to as integration points.

6 shows an integration point triggered at [I] that causes the context data management system 214 to perform various data aggregations and/or early processing invocation. When an integration point is triggered, it may cause the context data management system 214 to execute to process or store data or otherwise initiate early invocation of other processes. Advantageously, execution of the context data management system 214 triggered at the integration point may occur in parallel or otherwise asynchronously, with at least a portion of the set of natural language processing actions being performed on the utterance.

In some embodiments, an integration point may be implemented prior to invoking the ASR system 202. For example, an identifier of the voice-enabled device 102, an identifier of an active user profile associated with the utterance, an identifier of a geographic location of the voice-enabled device 102, and/or various other data elements may be obtained with or otherwise associated with the utterance. The speech processing system 100 may trigger the integration point based on obtaining such data.

In some embodiments, an integration point may be implemented during execution of the NLU system 204. For example, the NLU system may generate one or more preliminary semantic representations of the processed utterance, including intentions with a subset of slots populated with entities. The NLU system 204 may initiate an entity resolution process to resolve one or more entities (e.g., to determine whether the entities in question are present in one or more catalogs or other data stores). An integration point may be implemented in the entity resolution phase, and one or more preliminary semantic representations of the Utterances may be provided to the context data management system 214 for further processing, storage, or the like.

The example integration points described herein are for illustrative purposes only and are not intended to be limiting, required, or exhaustive. In some embodiments, integration points may be implemented in other portions of utterance processing, such as at any point where data is generated or acquired (e.g., to be further processed or provided to another downstream process). To illustrate, each time data is generated or acquired during or otherwise related to utterance processing, the context data management system 214 may execute to store the data, acquire new data using the data, process the data, to to generate new data or to initiate an early call to another process. By calling data aggregation functions and other functions early and performing them in parallel with other processing of the utterance, all prerequisite operations for subsequent processes can be completed when the subsequent processes are initiated (e.g., the process of determining routing responsibility for a response to the utterance), thereby reducing the user's perceived latency (e.g. delay in receiving a response to the utterance).

For example, at [II], the context data management system 214 may employ a model evaluation component 600 to use a model, such as a machine learning model or other probabilistic model, to evaluate one or more currently available data items and perform prediction, classification, or the like. As another example, at [III], the context data management system 214 may employ an execution service 602 to pre-compute a context data item. The perfected, pre-computed, or otherwise pre-generated context data items may be stored at [IV] in a processing data store 218 for access by subsequent processes or in an internal data store of the context data management system 214 to be requested during subsequent processes.

7 is a flowchart of an illustrative process 700 that may be performed by the context data management system 214 for data aggregation and/or early processing invocation. The method 700 begins at block 702. When the process 700 is initiated, a set of executable program instructions stored on one or more non-transitory computer-readable media (e.g., hard drive, flash memory, removable storage, etc.) may be written into memory (e.g. random access memory or “RAM”) of a computing device are loaded and executed. For example, executable instructions may be loaded into memory 254 of a computing device of context data management system 214 and executed by one or more processors 252. In some embodiments, the process 700 or portions thereof may be implemented on multiple processors in series or in parallel. At block 704, an integration point may be triggered. Triggering the integration point may cause the context data management system 214 to execute. Execution of the context data management system 214 may be triggered any number of times at any number of integration points during utterance processing, as indicated by the repeat arrow. The processing performed by the context data management system 214 in response to triggered integration points may be serial, parallel, or asynchronous, as appropriate. Additionally, processing for some integration points may occur independently of processing for other integration points.

At block 706, the context data management system 214 may obtain initial data associated with the integration point. The initial data may include data generated or acquired during utterance processing before or after the integration point is reached. For example, if the integration point that was triggered to initiate the current iteration of the process 700 (the “current integration point”) is at utterance capture or otherwise prior to ASR processing, the initial data may be an identifier of the voice-enabled device 102, an identifier an active user profile, etc. As another example, when the current integration point is at the point of entity resolution during NLU processing, the initial data may include preliminary NLU results, such as semantic representations of intent and at least a subset of corresponding entities. At block 708, the context data management system 214 may evaluate the initial data and any data generated or obtained in connection with previous iterations of the process 700 during processing of the current utterance (collectively, the “currently available data”) to determine one or more context data actions to take to determine. In some embodiments, as in 8th shown, the context data ver management system 214 includes a register 802 of contextual data actions to be performed using the currently available data. For example, if the currently available data includes geolocation data and a user profile identifier, the registry may indicate that a contextual data item should be generated that represents whether the user is at home. As another example, if the currently available data includes preliminary NLU results, the registry may indicate that a contextual data item should be generated that represents whether the user is likely to make an utterance that is a continuation of a previous utterance (e.g., a refinement of a search) or is unrelated to a previous statement (e.g. a completely new search).

The mechanism by which registry 802 specifies the contextual data actions to be taken may be based on data or functional signatures, such as those that include annotations. In some embodiments, the register may store a list of function signatures (e.g., Java function signatures) annotated to indicate data on which the functions depend (e.g., data that must be currently available, to execute the function). When an integration point is triggered, an orchestrator component 800 of the context data management system 214 may examine data in register 802 and find any functions that can be performed using the currently available data.

At decision block 710, the orchestrator 800 or another component of the context data management system 214 may determine whether to prefetch data using the currently available data. If so, the process 700 may proceed to block 712. Otherwise, the process 700 proceeds to decision block 714.

At block 712, the orchestrator 800 or another component of the context data management system 214 may initiate prefetching a context data item using the currently available data. For example, the orchestrator 800 may perform a directory search using one or more of the currently available data items. In some embodiments, the results of the search may be stored in an internal data store 804 where they are retained as a currently available data item for future context data generation processes and/or provided to other components of the speech processing system 100 during further processing (e.g., for routing determinations). In some embodiments, the results of the search may be stored in a processing data store 218 external to the context data management system 214 where they are made accessible to other components of the speech processing system 100 during subsequent processing (e.g., for routing determinations).

In some embodiments, process 700 may run asynchronously. For example, after initiating one or more prefetch operations in block 712, process 700 may proceed to decision block 714 without necessarily waiting for the prefetch operations to complete. When the prefetch operation(s) initiated in block 712 are completed, process 700 may proceed to decision block 718, possibly in parallel or asynchronously with the execution of decision block 714, block 716, etc.

In decision block 714, the orchestrator 800 or another component of the context data management system 214 may determine whether to precompute data using the currently available data. If so, process 700 may proceed to block 716. Otherwise, process 700 (or an execution thread thereof) may end at block 720.

At block 716, the context data management system 214 may initiate the precomputing of a context data item using the currently available data. The precomputed context data item may be stored in the internal data storage 804 of the context data management system 214 or in a processing data storage 218 external to the context data management system 214. In some embodiments, a model caller 806 may use a model or cause an external model evaluation component 600 to use a model, such as a machine learning model or other probabilistic model, to evaluate one or more currently available data items and provide a prediction, classification, or to undertake the like. For example, the model caller 806 may generate a classification of the currently available data as an indication of a search refinement or a new search. In some embodiments, a runtime hosting environment 808 may precompute a context data item or cause an external execution service 602 to precompute the context data item. For example, the runtime hosting environment can obtain 808 scores for the top two intentions of an N-best list and a difference between the who Calculate calculations. As another example, an external execution service 602 may pre-compute embeddings, such as Bidirectional Encoder Representations from Transformers ("BERT") embeddings, for use by subsequent processes.

In some embodiments, process 700 may run asynchronously. For example, after initiating one or more pre-computation operations at block 716, a thread of execution of process 700 may terminate at block 720 without necessarily waiting for the pre-computation operation(s) to complete. When the pre-computation operation(s) initiated at block 716 are completed, process 700 may proceed to decision block 718, possibly in parallel or asynchronously with execution of other portions of process 700.

At decision block 718, the orchestrator 800 or another component of the context data management system 214 may determine whether to prefetch and/or precalculate additional context data based on data prefetched and/or precalculated during the current iteration of the process 700. If so, the process 700 may return to block 708. Otherwise, the process 700 may terminate.

In some embodiments, the process 700 may be designed to create a data history in which initial data is obtained, additional data is retrieved, calculated, or otherwise generated based on the available data, and then additional data is retrieved, calculated, or otherwise Way to be generated based on the additional data and so on. In this way, the process 700 can be a recursive process in which a single integration point causes the data history to be expanded to include multiple data elements, levels, etc. In some embodiments, a previous recursive instance of process 700 may still be running when a subsequent integration point is reached during utterance processing and another instance of process 700 is invoked. In this way, the process 700 may be performed in parallel or otherwise asynchronously with respect to other instances of the same process 700. Returning to 6 In response to a confidence request received at [V], an intra-domain routing system 216 may obtain context data items at [VI] for use in generating routing confidence scores, as described in more detail above. Once a routing decision has been made, at [VII] an application 282 may be assigned to generate a response to the utterance. In some embodiments, the application 282 may access contextual data elements at [VIII] as part of the response generation process or otherwise in the course of managing a customer experience (e.g., during a multi-turn dialogue).

Examples of implementation

• Absatz 1. System, das Folgendes umfasst:
  • computerlesbaren Speicher, der ausführbare Anweisungen speichert; und
  • einen oder mehrere Prozessoren in Kommunikation mit dem computerlesbaren Speicher und durch die ausführbaren Anweisungen zu mindestens Folgendem konfiguriert:
  • Empfangen von Audiodaten von einer sprachgestützten Vorrichtung, wobei die Audiodaten eine Äußerung darstellen;
  • Generieren von Textdaten unter Verwendung der Audiodaten und eines Prozesses zur automatischen Spracherkennung („ASR“), wobei die Textdaten mindestens einen Teil der Äußerung darstellen; Generieren von Intentionsdaten unter Verwendung der Textdaten und eines Prozesses zum natürlichen Sprachverständnis („NLU“), wobei die Intentionsdaten eine semantische Darstellung der Äußerung umfassen;
  • an einem Integrationspunkt während des NLU-Prozesses, Bestimmen, ein Kontextdatenelement parallel zu mindestens einem Abschnitt des NLU-Prozesses zu generieren;
  • Erlangen eines oder mehrerer anfänglichen Datenelemente, die dem Integrationspunkt zugeordnet sind;
  • unter Verwendung eines Registers von Kontextdatenelementen und entsprechender Kontextdatenmaßnahmen, Bestimmen einer Kontextdatenmaßnahme, die durchzuführen ist, um das Kontextdatenelement zu generieren;
  • Generieren des Kontextdatenelements zumindest teilweise basierend auf der Durchführung der Kontextdatenmaßnahme parallel zu mindestens einem Abschnitt des NLU-Prozesses, wobei die Kontextdatenmaßnahme das Kontextdatenelement unter Verwendung des einen oder der mehreren anfänglichen Datenelemente generiert;
  • Bestimmen einer Anwendung, an die die Intentionsdaten zur Generierung einer Reaktion auf die Äußerung weiterzuleiten sind, wobei die Anwendung zumindest teilweise basierend auf dem Kontextdatenelement bestimmt wird; und
  • Generieren einer Reaktion auf die Äußerung unter Verwendung der Anwendung und der Intentionsdaten.
• Absatz 2. System nach Absatz 1, wobei das eine oder die mehreren anfänglichen Datenelemente mindestens eines von Folgendem umfassen: ein von der sprachgestützten Vorrichtung empfangenes Datenelement, ein während des ASR-Prozesses generiertes Datenelement oder ein während des NLU-Prozesses generiertes Datenelement.
• Absatz 3. System nach Absatz 1, wobei die Kontextdatenmaßnahme eines von Folgenden umfasst: Bewertung des einen oder der mehreren anfänglichen Datenelemente unter Verwendung eines Modells, um eine Vorhersage bezüglich eines Aspekts der Äußerung zu generieren; Durchführung einer Berechnung unter Verwendung des einen oder der mehreren anfänglichen Datenelemente, um einen Wert vorab zu berechnen; oder Erfassung eines Werts aus einem Datenspeicher unter Verwendung des einen oder der mehreren anfänglichen Datenelemente.
• Absatz 4. System nach Absatz 1, wobei das Kontextdatenelement Daten umfasst, die mindestens eines von Folgendem darstellen: einen Standort der sprachgestützten Vorrichtung, eine Kennung der sprachgestützten Vorrichtung, einen Inhaltskatalog, der einem Benutzerprofil zugeordnet ist, historische Interaktionen, die dem Benutzerprofil zugeordnet sind, eine Vorhersage bezüglich eines Aspekts der Äußerung oder einen Vergleich einer Vielzahl von Interpretationen der Äußerung.
• Absatz 5. Computerimplementiertes Verfahren, das Folgendes umfasst:
  • gemäß Durchführung durch ein Rechensystem, das einen oder mehrere Computerprozessoren umfasst, die dazu konfiguriert sind, spezifische Anweisungen ausführen,
  • Ausführen eines Satzes von Verarbeitungsmaßnahmen für natürliche Sprache an einer Eingabe in natürlicher Sprache, um Intentionsdaten zu generieren, wobei die Eingabe in natürlicher Sprache eine Äußerung eines Benutzers darstellt und wobei die Intentionsdaten eine semantische Darstellung der Äußerung des Benutzers umfassen;
  • an einem Integrationspunkt während des Satzes von Verarbeitungsmaßnahmen für natürliche Sprache, Bestimmen, ein Kontextdatenelement unter Verwendung eines anfänglichen Datenelements zu generieren, das mindestens einer aus dem Satz von Verarbeitungsmaßnahmen für natürliche Sprache zugeordnet ist;
  • Generieren des Kontextdatenelements zumindest teilweise basierend auf dem anfänglichen Datenelement gleichzeitig mit der Durchführung mindestens eines Abschnitts des Satzes von Verarbeitungsmaßnahmen für natürliche Sprache; und Bereitstellen von Zugriff auf das Kontextdatenelement, wobei auf das Kontextdatenelement während der Durchführung von mindestens einem von Folgendem zugegriffen wird: dem Satz von Verarbeitungsmaßnahmen für natürliche Sprache oder einem Satz von Reaktionsbestimmungsmaßnahmen.
• Absatz 6. Computerimplementiertes Verfahren nach Absatz 5, ferner umfassend Bestimmen einer Kontextdatenmaßnahme, die durchzuführen ist, um das Kontextdatenelement zu generieren, unter Verwendung eines Registers, wobei das Register die Kontextdatenmaßnahme und ein oder mehrere anfängliche Datenelemente spezifiziert, die zu verwenden sind, um das Kontextdatenelement zu generieren.
• Absatz 7. Computerimplementiertes Verfahren nach Absatz 5, ferner umfassend:
  • an einem zweiten Integrationspunkt während des Satzes von Verarbeitungsmaßnahmen für natürliche Sprache, Bestimmen, ein zweites Kontextdatenelement unter Verwendung eines zweiten anfänglichen Datenelements zu generieren; und
  • Generieren des zweiten Kontextdatenelements zumindest teilweise basierend auf dem zweiten anfänglichen Datenelement gleichzeitig mit der Durchführung mindestens eines zweiten Abschnitts des Satzes von Verarbeitungsmaßnahmen für natürliche Sprache.
• Absatz 8. Computerimplementiertes Verfahren nach Absatz 5, ferner umfassend:
  • als Reaktion auf das Generieren des Kontextdatenelements, Bestimmen, ein zweites Kontextdatenelement unter Verwendung des Kontextdatenelements zu generieren; und
  • Generieren des zweiten Kontextdatenelements zumindest teilweise basierend auf dem Kontextdatenelement gleichzeitig mit der Durchführung mindestens eines zweiten Abschnitts des Satzes von Verarbeitungsmaßnahmen für natürliche Sprache.
• Absatz 9. Computerimplementiertes Verfahren nach Absatz 5, wobei das Generieren des Kontextdatenelements zumindest teilweise basierend auf dem anfänglichen Datenelement Bewerten des anfänglichen Datenelements unter Verwendung eines Modells umfasst, um eine Vorhersage bezüglich eines Aspekts der Äußerung zu generieren.
• Absatz 10. Computerimplementiertes Verfahren nach Absatz 9, ferner umfassend Generieren einer Reaktion auf die Äußerung zumindest teilweise basierend auf der Vorhersage, wobei sich die Vorhersage auf einen wahrscheinlichen Status der Äußerung als eine von einer anfänglichen Äußerung des Benutzers eines Dialogs mit mehreren Wendungen oder einer nachfolgenden Äußerung des Benutzers eines Dialogs mit mehreren Wendungen bezieht.
• Absatz 11. Computerimplementiertes Verfahren nach Absatz 5, wobei das Bereitstellen von Zugriff auf das Kontextdatenelement Speichern des Kontextdatenelements in einem Datenspeicher umfasst, auf den während der Durchführung des mindestens einen von dem Satz von Verarbeitungsmaßnahmen für natürliche Sprache oder dem Satz von Reaktionsbestimmungsmaßnahmen zugegriffen werden kann.
• Absatz 12. Computerimplementiertes Verfahren nach Absatz 5, wobei das Bereitstellen von Zugriff auf das Kontextdatenelement Reagieren auf eine Anfrage nach dem Kontextdatenelement während der Durchführung des mindestens einen von dem Satz von Verarbeitungsmaßnahmen für natürliche Sprache oder dem Satz von Reaktionsbestimmungsmaßnahmen umfasst.
• Absatz 13. System, das Folgendes umfasst:
  • computerlesbaren Speicher, der ausführbare Anweisungen speichert; und
  • einen oder mehrere Prozessoren in Kommunikation mit dem computerlesbaren Speicher und durch die ausführbaren Anweisungen zu mindestens Folgendem konfiguriert:
  • Ausführen eines Satzes von Verarbeitungsmaßnahmen für natürliche Sprache an einer Eingabe in natürlicher Sprache, um Intentionsdaten zu generieren, wobei die Eingabe in natürlicher Sprache eine Äußerung eines Benutzers darstellt und wobei die Intentionsdaten eine semantische Darstellung der Äußerung des Benutzers umfassen;
  • an einem Integrationspunkt während des Satzes von Verarbeitungsmaßnahmen für natürliche Sprache, Bestimmen, ein Kontextdatenelement unter Verwendung eines anfänglichen Datenelements zu generieren, das mindestens einer aus dem Satz von Verarbeitungsmaßnahmen für natürliche Sprache zugeordnet ist;
  • Generieren des Kontextdatenelements zumindest teilweise basierend auf dem anfänglichen Datenelement gleichzeitig mit der Durchführung mindestens eines Abschnitts des Satzes von Verarbeitungsmaßnahmen für natürliche Sprache; und Bereitstellen von Zugriff auf das Kontextdatenelement, wobei auf das Kontextdatenelement während der Durchführung von mindestens einem von Folgendem zugegriffen wird: dem Satz von Verarbeitungsmaßnahmen für natürliche Sprache oder einem Satz von Reaktionsbestimmungsmaßnahmen.
• Absatz 14. System nach Absatz 13, wobei der eine oder die mehreren Prozessoren durch weitere ausführbare Anweisungen dazu konfiguriert sind, eine Kontextdatenmaßnahme, die durchzuführen ist, um das Kontextdatenelement zu generieren, unter Verwendung eines Registers zu bestimmen, wobei das Register die Kontextdatenmaßnahme und ein oder mehrere anfängliche Datenelemente spezifiziert, die zu verwenden sind, um das Kontextdatenelement zu generieren.
• Absatz 15. System nach Absatz 13, wobei der eine oder die mehreren Prozessoren durch weitere ausführbare Anweisungen zu mindestens Folgendem konfiguriert sind:
  • an einem zweiten Integrationspunkt während des Satzes von Verarbeitungsmaßnahmen für natürliche Sprache, Bestimmen, ein zweites Kontextdatenelement unter Verwendung eines zweiten anfänglichen Datenelements zu generieren; und
  • Generieren des zweiten Kontextdatenelements zumindest teilweise basierend auf dem zweiten anfänglichen Datenelement gleichzeitig mit der Durchführung mindestens eines zweiten Abschnitts des Satzes von Verarbeitungsmaßnahmen für natürliche Sprache.
• Absatz 16. System nach Absatz 13, wobei der eine oder die mehreren Prozessoren durch weitere ausführbare Anweisungen zu mindestens Folgendem konfiguriert sind:
  • als Reaktion auf das Generieren des Kontextdatenelements, Bestimmen, ein zweites Kontextdatenelement unter Verwendung des Kontextdatenelements zu generieren; und
  • Generieren des zweiten Kontextdatenelements zumindest teilweise basierend auf dem Kontextdatenelement gleichzeitig mit der Durchführung mindestens eines zweiten Abschnitts des Satzes von Verarbeitungsmaßnahmen für natürliche Sprache.
• Absatz 17. System nach Absatz 13, wobei der eine oder die mehreren Prozessoren, die dazu konfiguriert sind, das Kontextdatenelement zumindest teilweise basierend auf dem anfänglichen Datenelement zu generieren, durch weitere ausführbare Anweisungen dazu konfiguriert sind, das anfängliche Datenelement unter Verwendung eines Modells zu bewerten, um eine Vorhersage bezüglich eines Aspekts der Äußerung zu generieren. Absatz 18. System nach Absatz 17, wobei der eine oder die mehreren Prozessoren durch weitere ausführbare Anweisungen dazu konfiguriert sind, eine Reaktion auf die Äußerung zumindest teilweise basierend auf der Vorhersage zu generieren, wobei sich die Vorhersage auf einen wahrscheinlichen Status der Äußerung als eine von einer anfängliche Äußerung des Benutzers eines Dialogs mit mehreren Wendungen oder einer nachfolgenden Äußerung des Benutzers eines Dialogs mit mehreren Wendungen bezieht. Absatz 19. System nach Absatz 13, wobei der eine oder die mehreren Prozessoren, die dazu konfiguriert sind, Zugriff auf das Kontextdatenelement bereitzustellen, durch weitere ausführbare Anweisungen dazu konfiguriert sind, das Kontextdatenelement in einem Datenspeicher zu speichern, auf den während der Durchführung des mindestens einen von dem Satz von Verarbeitungsmaßnahmen für natürliche Sprache oder dem Satz von Reaktionsbestimmungsmaßnahmen zugegriffen werden kann.
• Absatz 20. System nach Absatz 13, wobei der eine oder die mehreren Prozessoren, die dazu konfiguriert sind, Zugriff auf das Kontextdatenelement bereitzustellen, durch weitere ausführbare Anweisungen dazu konfiguriert sind, während der Durchführung des mindestens einen von dem Satz von Verarbeitungsmaßnahmen für natürliche Sprache oder dem Satz von Reaktionsbestimmungsmaßnahmen auf eine Anfrage nach dem Kontextdatenelement zu reagieren.
• Absatz 21. System, das Folgendes umfasst:
  • computerlesbaren Speicher, der ausführbare Anweisungen speichert; und
  • einen oder mehrere Prozessoren in Kommunikation mit dem computerlesbaren Speicher und durch die ausführbaren Anweisungen zu mindestens Folgendem konfiguriert:
    • Empfangen von Audiodaten von einer sprachgestützten Vorrichtung, wobei die Audiodaten eine Äußerung darstellen;
    • Ausführen eines Satzes von Verarbeitungsmaßnahmen für natürliche Sprache, die zumindest teilweise auf den Audiodaten basieren, um Intentionsdaten zu generieren, die eine Vielzahl von semantischen Darstellungen der Äußerung umfassen;
    • Bestimmen einer ersten Domäne einer Vielzahl von Domänen zumindest teilweise basierend auf einer Zuordnung der ersten Domäne zur einer Teilmenge der Vielzahl von semantischen Darstellungen, wobei die erste Domäne eine Vielzahl von Anwendungen umfasst, die dazu konfiguriert ist, Reaktionen auf Äußerungen zu generieren;
    • Erlangen eines Satzes von Kontextdatenelementen, die der Äußerung zugeordnet sind;
    • Bestimmen einer ersten Anwendung der Vielzahl von Anwendungen zumindest teilweise basierend auf einer Zuordnung der ersten Anwendung zu einer ersten semantischen Darstellung der Teilmenge von semantischen Darstellungen;
    • Erlangen einer ersten Routing-Konfidenzbewertung zumindest teilweise basierend auf einer ersten Teilmenge des Satzes von Kontextdatenelementen von einem ersten Routing-Konfidenzbereitsteller, der der ersten Anwendung zugeordnet ist, wobei die erste Routing-Konfidenzbewertung einen Grad an Konfidenz darstellt, dass eine Reaktion auf die Äußerung durch die erste Anwendung zu generieren ist;
    • Bestimmen einer zweiten Anwendung der Vielzahl von Anwendungen zumindest teilweise basierend auf einer Zuordnung der zweiten Anwendung zu einer zweiten semantischen Darstellung der Teilmenge von semantischen Darstellungen;
    • Erlangen einer zweiten Routing-Konfidenzbewertung zumindest teilweise basierend auf einer zweiten Teilmenge des Satzes von Kontextdatenelementen von einem zweiten Routing-Konfidenzbereitsteller, der der zweiten Anwendung zugeordnet ist, wobei die zweite Routing-Konfidenzbewertung einen Grad an Konfidenz darstellt, dass eine Reaktion auf die Äußerung durch die zweite Anwendung zu generieren ist; und
    • Zuweisen der ersten Anwendung, um die Reaktion zu generieren, zumindest teilweise basierend auf der ersten Routing-Konfidenzbewertung und der zweiten Routing-Konfidenzbewertung. Absatz 22. System nach Absatz 21, wobei die Intentionsdaten eine Rangliste der Vielzahl von semantischen Darstellungen umfassen, wobei die erste semantische Darstellung einen niedrigeren Rang hat als die zweite semantische Darstellung.
• Absatz 23. System nach Absatz 21, wobei eine erste Teilmenge der Vielzahl von Anwendungen einer ersten Schicht einer mehrschichtigen domäneninternen Hierarchie zugeordnet ist und wobei eine zweite Teilmenge der Vielzahl von Anwendungen einer zweiten Schicht der mehrschichtigen domäneninternen Hierarchie zugeordnet ist.
• Absatz 24. System nach Absatz 21, wobei der erste Routing-Konfidenzbereitsteller die erste Routing-Konfidenzbewertung unter Verwendung eines statistischen Modells generiert und wobei der zweite Routing-Konfidenzbereitsteller die zweite Routing-Konfidenzbewertung unter Verwendung eines Satzes deterministischer Regeln generiert.
• Absatz 25. System nach Absatz 21, wobei der Satz von Kontextdatenelementen Daten umfasst, die mindestens eines von Folgendem darstellen: einen Standort der sprachgestützten Vorrichtung, eine Kennung der sprachgestützten Vorrichtung, einen Inhaltskatalog, der einem Benutzerprofil zugeordnet ist, historische Interaktionen, die dem Benutzerprofil zugeordnet sind, eine Vorhersage bezüglich eines Aspekts der Äußerung oder einen Vergleich einer Vielzahl von Interpretationen der Äußerung.
• Absatz 26. Computerimplementiertes Verfahren, das Folgendes umfasst:
  • gemäß Durchführung durch ein Rechensystem, das einen oder mehrere Computerprozessoren umfasst, die dazu konfiguriert sind, spezifische Anweisungen ausführen,
  • Empfangen einer Anfrage diesbezüglich, ob eine Anwendung einer Domäne dazu konfiguriert ist, eine Reaktion auf eine Äußerung zu generieren, wobei die Anfrage einer Vielzahl von semantischen Darstellungen der Äußerung zugeordnet ist;
  • Erlangen eines Satzes von Kontextdatenelementen, die der Äußerung zugeordnet sind;
  • Identifizieren einer Teilmenge von Anwendungen der Domäne zumindest teilweise basierend auf einer Zuordnung einzelner Anwendungen der Teilmenge zu einzelnen semantischen Darstellungen der Vielzahl von semantischen Darstellungen; Generieren einer Vielzahl von Routing-Konfidenzbewertungen unter Verwendung der Vielzahl von semantischen Darstellungen und des Satzes von Kontextdatenelementen, wobei eine Routing-Konfidenzbewertung einen Grad an Konfidenz darstellt, dass eine Reaktion auf die Äußerung durch eine Anwendung zu generieren ist; und
  • Generieren einer Reaktion auf die Anfrage zumindest teilweise basierend auf der Vielzahl von Routing-Konfidenzbewertungen.
• Absatz 27. Computerimplementiertes Verfahren nach Absatz 26, wobei das Erlangen des Satzes von Kontextdatenelementen mindestens eines von Folgendem umfasst: Abrufen eines Kontextdatenelements aus einem Datenspeicher, Anfordern des Kontextdatenelements von einem Kontextdatenverwaltungssystem oder Erlangen des Kontextdatenelements aus der Anfrage.
• Absatz 28. Computerimplementiertes Verfahren nach Absatz 26, wobei das Identifizieren der Teilmenge von Anwendungen zumindest teilweise auf einem vorbestimmten Mapping von Anwendungen zu Arten semantischer Darstellungen basiert.
• Absatz 29. Computerimplementiertes Verfahren nach Absatz 26, wobei das Generieren der Vielzahl von Routing-Konfidenzbewertungen Generieren einer ersten Routing-Konfidenzbewertung unter Verwendung eines statistischen Modells und einer Teilmenge des Satzes von Kontextdatenelementen umfasst.
• Absatz 30. Computerimplementiertes Verfahren nach Absatz 26, wobei das Generieren der Vielzahl von Routing-Konfidenzbewertungen Generieren einer ersten Routing-Konfidenzbewertung unter Verwendung eines Satzes von deterministischen Regeln und einer Teilmenge des Satzes von Kontextdatenelementen umfasst.
• Absatz 31. Computerimplementiertes Verfahren nach Absatz 26, wobei das Generieren der Vielzahl von Routing-Konfidenzbewertungen Veranlassen einer Ausführung eines Routing-Konfidenzbereitstellers eines Drittanbieters unter Verwendung einer Teilmenge des Satzes von Kontextdatenelementen umfasst.
• Absatz 32. Computerimplementiertes Verfahren nach Absatz 26, wobei das Generieren der Reaktion auf die Anfrage Angeben, dass die Reaktion auf die Äußerung durch eine erste Anwendung der Teilmenge von Anwendungen zu generieren ist, umfasst.
• Absatz 33. Computerimplementiertes Verfahren nach Absatz 26, ferner umfassend Bestimmen, dass eine Routing-Konfidenzbewertung der Vielzahl von Routing-Konfidenzbewertungen eine Mehrdeutigkeit diesbezüglich angibt, ob die Reaktion auf die Äußerung durch eine erste Anwendung der Teilmenge von Anwendungen zu generieren ist, wobei das Generieren der Reaktion auf die Anfrage Identifizieren einer Mehrdeutigkeitsauflösungskomponente als eine Anwendung, die die Reaktion auf die Äußerung generieren soll, umfasst.
• Absatz 34. Computerimplementiertes Verfahren nach Absatz 33, ferner umfassend Ausführen der Mehrdeutigkeitsauflösungskomponente, um die Mehrdeutigkeit aufzulösen, wobei die Mehrdeutigkeitsauflösungskomponente dazu konfiguriert ist, mindestens eines von Folgendem darzustellen: einen Hinweis bezüglich einer zweiten Äußerung, die gemacht werden soll, eine Abfrage nach einer zweiten Äußerung bezüglich einer Klarstellung der Äußerung oder eine Abfrage nach einer zweiten Äußerung, um anzugeben, ob die Reaktion auf die Äußerung durch die erste Anwendung zu generieren ist.
• Absatz 35. System, das Folgendes umfasst:
  • computerlesbaren Speicher, der ausführbare Anweisungen speichert; und
  • einen oder mehrere Prozessoren in Kommunikation mit dem computerlesbaren Speicher und durch die ausführbaren Anweisungen zu mindestens Folgendem konfiguriert:
    • Empfangen einer Anfrage diesbezüglich, ob eine Anwendung einer Domäne dazu konfiguriert ist, eine Reaktion auf eine Äußerung zu generieren, wobei die Anfrage einer Vielzahl von semantischen Darstellungen der Äußerung zugeordnet ist;
    • Erlangen eines Satzes von Kontextdatenelementen, die der Äußerung zugeordnet sind;
    • Identifizieren einer Teilmenge von Anwendungen der Domäne zumindest teilweise basierend auf einer Zuordnung einzelner Anwendungen der Teilmenge zu einzelnen semantischen Darstellungen der Vielzahl von semantischen Darstellungen; Generieren einer Vielzahl von Routing-Konfidenzbewertungen unter Verwendung der Vielzahl von semantischen Darstellungen und des Satzes von Kontextdatenelementen, wobei eine Routing-Konfidenzbewertung einen Grad an Konfidenz darstellt, dass eine Reaktion auf die Äußerung durch eine Anwendung zu generieren ist; und
    • Generieren einer Reaktion auf die Anfrage zumindest teilweise basierend auf der Vielzahl von Routing-Konfidenzbewertungen. Absatz 36. System nach Absatz 35, wobei der eine oder die mehreren Prozessoren, die dazu konfiguriert sind, den Satz von Kontextdatenelementen zu erlangen, durch weitere ausführbare Anweisungen dazu konfiguriert sind, ein Kontextdatenelement von einem der Folgenden zu erlangen: einem Datenspeicher, einem Kontextdatenverwaltungssystem oder der Anfrage.
• Absatz 37. System nach Absatz 35, wobei der eine oder die mehreren Prozessoren, die dazu konfiguriert sind, die Vielzahl von Routing-Konfidenzbewertungen zu generieren, durch weitere ausführbare Anweisungen dazu konfiguriert sind, eine erste Routing-Konfidenzbewertung unter Verwendung eines statistischen Modells und einer Teilmenge des Satzes von Kontextdatenelementen zu generieren.
• Absatz 38. System nach Absatz 35, wobei der eine oder die mehreren Prozessoren, die dazu konfiguriert sind, die Vielzahl von Routing-Konfidenzbewertungen zu generieren, durch weitere ausführbare Anweisungen dazu konfiguriert sind, eine erste Routing-Konfidenzbewertung unter Verwendung eines Satzes von deterministischen Regeln und einer Teilmenge des Satzes von Kontextdatenelementen zu generieren.
• Absatz 39. System nach Absatz 35, wobei der eine oder die mehreren Prozessoren, die dazu konfiguriert sind, die Vielzahl von Routing-Konfidenzbewertungen zu generieren, durch weitere ausführbare Anweisungen dazu konfiguriert sind, die Ausführung eines Routing-Konfidenzbereitstellers eines Drittanbieters unter Verwendung einer Teilmenge des Satzes von Kontextdatenelementen zu veranlassen.
• Absatz 40. System nach Absatz 35, wobei der eine oder die mehreren Prozessoren durch weitere ausführbare Anweisungen dazu konfiguriert sind zu bestimmen, dass eine Routing-Konfidenzbewertung der Vielzahl von Routing-Konfidenzbewertungen eine Mehrdeutigkeit diesbezüglich angibt, ob die Reaktion auf die Äußerung durch eine erste Anwendung der Teilmenge von Anwendungen zu generieren ist, wobei die Reaktion auf die Anfrage eine Mehrdeutigkeitsauflösungskomponente als eine Anwendung identifiziert, die die Reaktion auf die Äußerung generieren soll.

Some inventive aspects of the disclosure are set out in the following paragraphs:

• Paragraph 1. System comprising:
  • computer-readable memory storing executable instructions; and
  • one or more processors in communication with the computer-readable memory and configured by the executable instructions to at least:
  • Receiving audio data from a speech-enabled device, the audio data representing an utterance;
  • Generating text data using the audio data and an automatic speech recognition (“ASR”) process, wherein the text data represents at least a portion of the utterance; Generating intention data using the text data and a natural language understanding (“NLU”) process, wherein the intention data comprises a semantic representation of the utterance;
  • at an integration point during the NLU process, determining to generate a context data element in parallel with at least a portion of the NLU process;
  • Obtaining one or more initial data elements associated with the integration point;
  • using a register of context data elements and corresponding context data actions, determining a context data action to be performed to generate the context data element;
  • generating the context data item based at least in part on performing the context data action in parallel with at least a portion of the NLU process, wherein the context data action generates the context data item using the one or more initial data items;
  • Determining an application to which the intention data is to be forwarded to generate a response to the utterance, the application being determined based at least in part on the context data element; and
  • Generate a response to the utterance using the application and intent data.
• Paragraph 2. The system of paragraph 1, wherein the one or more initial data elements comprise at least one of: a data element received from the voice-assisted device, a data element generated during the ASR process, or a data element generated during the NLU process.
• Paragraph 3. The system of paragraph 1, wherein the contextual data action comprises one of: evaluating the one or more initial data items using a model to generate a prediction regarding an aspect of the utterance; performing a calculation using the one or more initial data items to pre-calculate a value; or extracting a value from a data store using the one or more initial data items.
• Paragraph 4. The system of paragraph 1, wherein the context data element comprises data representing at least one of: a location of the voice-enabled device, an identifier of the voice-enabled device, a catalog of content associated with a user profile, historical interactions associated with the user profile, a prediction regarding an aspect of the utterance, or a comparison of a plurality of interpretations of the utterance.
• Paragraph 5. Computer-implemented procedure comprising:
  • as performed by a computing system comprising one or more computer processors configured to execute specific instructions,
  • performing a set of natural language processing actions on a natural language input to generate intent data, wherein the natural language input represents an utterance of a user and wherein the intent data comprises a semantic representation of the user's utterance;
  • at an integration point during the set of natural language processing actions, determining to generate a context data item using an initial data item associated with at least one of the set of natural language processing actions;
  • generating the context data item based at least in part on the initial data item concurrently with performance of at least a portion of the set of natural language processing actions; and providing access to the context data item, wherein the context data item is accessed during performance of at least one of: the set of natural language processing actions or a set of response determination actions.
• Paragraph 6. The computer-implemented method of paragraph 5, further comprising determining a context data action to be performed to generate the context data item using a register, the register specifying the context data action and one or more initial data items to be used to generate the context data item.
• Paragraph 7. Computer-implemented method according to paragraph 5, further comprising:
  • at a second integration point during the set of natural language processing actions, determining to generate a second context data item using a second initial data item; and
  • Generating the second context data item based at least in part on the second initial data item concurrently with performing at least a second portion of the set of natural language processing actions.
• Paragraph 8. Computer-implemented procedure according to paragraph 5, further comprising:
  • in response to generating the context data element, determining to generate a second context data element using the context data element; and
  • Generating the second context data element based at least in part on the context data element concurrently with performing at least a second portion of the set of natural language processing actions.
• Paragraph 9. The computer-implemented method of paragraph 5, wherein generating the context data item based at least in part on the initial data item comprises evaluating the initial data item using a model to generate a prediction regarding an aspect of the utterance.
• Paragraph 10. The computer-implemented method of paragraph 9, further comprising generating a response to the utterance based at least in part on the prediction, wherein the prediction relates to a likely status of the utterance as one of an initial utterance of the user of a multi-turn dialogue or a subsequent utterance of the user of a multi-turn dialogue.
• Paragraph 11. The computer-implemented method of paragraph 5, wherein providing access to the context data element comprises storing the context data element in a data store that is accessible during performance of at least one of the set of natural language processing actions or the set of response determination actions.
• Paragraph 12. The computer-implemented method of paragraph 5, wherein providing access to the context data element comprises responding to a request for the context data element during performance of at least one of the set of natural language processing actions or the set of response determination actions.
• Paragraph 13. System comprising:
  • computer-readable memory storing executable instructions; and
  • one or more processors in communication with the computer-readable memory and configured by the executable instructions to at least:
  • performing a set of natural language processing actions on a natural language input to generate intent data, wherein the natural language input represents an utterance of a user and wherein the intent data comprises a semantic representation of the user's utterance;
  • at an integration point during the set of natural language processing actions, determining to generate a context data item using an initial data item associated with at least one of the set of natural language processing actions;
  • generating the context data item based at least in part on the initial data item concurrently with performance of at least a portion of the set of natural language processing actions; and providing access to the context data item, wherein the context data item is accessed during performance of at least one of: the set of natural language processing actions or a set of response determination actions.
• Paragraph 14. The system of paragraph 13, wherein the one or more processors are configured by further executable instructions to determine a context data action to be performed to generate the context data item using a register, the register specifying the context data action and one or more initial data items to be used to generate the context data item.
• Paragraph 15. The system of paragraph 13, wherein the one or more processors are configured by further executable instructions to at least:
  • at a second integration point during the set of natural language processing actions, determining to generate a second context data item using a second initial data item; and
  • Generating the second context data item based at least in part on the second initial data item concurrently with performing at least a second portion of the set of natural language processing actions.
• Paragraph 16. The system of paragraph 13, wherein the one or more processors are configured by further executable instructions to at least:
  • in response to generating the context data element, determining to generate a second context data element using the context data element; and
  • Generating the second context data element based at least in part on the context data element concurrently with performing at least a second portion of the set of natural language processing actions.
• Paragraph 17. The system of paragraph 13, wherein the one or more processors configured to generate the context data item based at least in part on the initial data item are configured by further executable instructions to evaluate the initial data item using a model to generate a prediction regarding an aspect of the utterance. Paragraph 18. The system of paragraph 17, wherein the one or more processors configured by further executable instructions to generate a response to the utterance based at least in part on the prediction, wherein the prediction relates to a likely status of the utterance as one of an initial utterance of the user of a multi-turn dialogue or a subsequent utterance of the user of a multi-turn dialogue. Paragraph 19. The system of paragraph 13, wherein the one or more processors configured to provide access to the context data item are configured by further executable instructions to store the context data item in a data store accessible during performance of the at least one of the set of natural language processing actions or the set of response determination actions.
• Paragraph 20. The system of paragraph 13, wherein the one or more processors configured to provide access to the context data item are configured by further executable instructions to respond to a request for the context data item during performance of the at least one of the set of natural language processing actions or the set of response determination actions.
• Paragraph 21. System comprising:
  • computer-readable memory storing executable instructions; and
  • one or more processors in communication with the computer-readable memory and configured by the executable instructions to at least:
    • Receiving audio data from a speech-enabled device, the audio data representing an utterance;
    • performing a set of natural language processing operations based at least in part on the audio data to generate intention data comprising a plurality of semantic representations of the utterance;
    • Determining a first domain of a plurality of domains based at least in part on an association of the first domain with a subset of the plurality of semantic representations, the first domain comprising a plurality of applications configured to generate responses to utterances;
    • Obtaining a set of context data elements associated with the utterance;
    • determining a first application of the plurality of applications based at least in part on an association of the first application with a first semantic representation of the subset of semantic representations;
    • Obtaining a first routing confidence score based at least in part on a first subset of the set of context data items from a first routing confidence provider associated with the first application, the first routing confidence score representing a level of confidence that a response to the utterance is to be generated by the first application;
    • determining a second application of the plurality of applications based at least in part on an association of the second application with a second semantic representation of the subset of semantic representations;
    • Obtaining a second routing confidence score based at least in part on a second subset of the set of context data items from a second routing confidence provider associated with the second application, the second routing confidence score representing a level of confidence that a response to the utterance is to be generated by the second application; and
    • Assigning the first application to generate the response based at least in part on the first routing confidence score and the second routing confidence score. Paragraph 22. The system of paragraph 21, wherein the intent data comprises a ranked list of the plurality of semantic representations, the first semantic representation having a lower rank than the second semantic representation.
• Paragraph 23. The system of paragraph 21, wherein a first subset of the plurality of applications is associated with a first layer of a multi-layered intra-domain hierarchy, and wherein a second subset of the plurality of applications is associated with a second layer of the multi-layered intra-domain hierarchy.
• Paragraph 24. The system of paragraph 21, wherein the first routing confidence provider generates the first routing confidence score using a statistical model and wherein the second routing confidence provider generates the second routing confidence score using a set of deterministic rules.
• Paragraph 25. The system of paragraph 21, wherein the set of context data elements comprises data representing at least one of: a location of the voice-enabled device, an identifier of the voice-enabled device, a catalog of content associated with a user profile, historical interactions associated with the user profile, a prediction regarding an aspect of the utterance, or a comparison of a plurality of interpretations of the utterance.
• Paragraph 26. Computer-implemented procedure comprising:
  • as performed by a computing system comprising one or more computer processors configured to execute specific instructions,
  • Receiving a query regarding whether an application of a domain is configured to generate a response to an utterance, the query associated with a plurality of semantic representations of the utterance;
  • Obtaining a set of context data elements associated with the utterance;
  • Identifying a subset of applications of the domain based at least in part on an association of individual applications of the subset with individual semantic representations of the plurality of semantic representations; generating a plurality of routing confidence scores using the plurality of semantic representations and the set of context data elements, wherein a routing confidence score represents a degree of confidence that a response to the utterance is to be generated by an application; and
  • Generating a response to the request based at least in part on the plurality of routing confidence scores.
• Paragraph 27. The computer-implemented method of paragraph 26, wherein obtaining the set of context data items comprises at least one of: retrieving a context data item from a data store, requesting the context data item from a context data management system, or obtaining the context data item from the request.
• Paragraph 28. The computer-implemented method of paragraph 26, wherein identifying the subset of applications is based at least in part on a predetermined mapping of applications to types of semantic representations.
• Paragraph 29. The computer-implemented method of paragraph 26, wherein generating the plurality of routing confidence scores comprises generating a first routing confidence score using a statistical model and a subset of the set of context data elements.
• Paragraph 30. The computer-implemented method of paragraph 26, wherein generating the plurality of routing confidence scores comprises generating a first routing confidence score using a set of deterministic rules and a subset of the set of context data elements.
• Paragraph 31. The computer-implemented method of paragraph 26, wherein generating the plurality of routing confidence scores comprises causing execution of a third-party routing confidence provider using a subset of the set of context data elements.
• Paragraph 32. The computer-implemented method of paragraph 26, wherein generating the response to the request comprises specifying that the response to the utterance is to be generated by a first application of the subset of applications.
• Paragraph 33. The computer-implemented method of paragraph 26, further comprising determining that a routing confidence score of the plurality of routing confidence scores indicates an ambiguity as to whether the response to the utterance is to be generated by a first application of the subset of applications, wherein generating the response to the request comprises identifying an ambiguity resolution component as an application to generate the response to the utterance.
• Paragraph 34. The computer-implemented method of paragraph 33, further comprising executing the ambiguity resolution component to resolve the ambiguity, wherein the ambiguity resolution component is configured to present at least one of: an indication regarding a second utterance to be made, a query for a second utterance regarding clarification of the utterance, or a query for a second utterance to indicate whether the response to the utterance is to be generated by the first application.
• Paragraph 35. System comprising:
  • computer-readable memory storing executable instructions; and
  • one or more processors in communication with the computer-readable memory and configured by the executable instructions to at least:
    • Receiving a query regarding whether an application of a domain is configured to generate a response to an utterance, the query associated with a plurality of semantic representations of the utterance;
    • Obtaining a set of context data elements associated with the utterance;
    • Identifying a subset of applications of the domain based at least in part on an association of individual applications of the subset with individual semantic representations of the plurality of semantic representations; generating a plurality of routing confidence scores using the plurality of semantic representations and the set of context data elements, wherein a routing confidence score represents a degree of confidence that a response to the utterance is to be generated by an application; and
    • Generating a response to the request based at least in part on the plurality of routing confidence scores. Paragraph 36. The system of paragraph 35, wherein the one or more processors configured to obtain the set of context data items are configured, by further executable instructions, to obtain a context data item from one of the following: a data store, a context data management system, or the request.
• Paragraph 37. The system of paragraph 35, wherein the one or more processors configured to generate the plurality of routing confidence scores are configured by further executable instructions to generate a first routing confidence score using a statistical model and a subset of the set of context data elements.
• Paragraph 38. The system of paragraph 35, wherein the one or more processors configured to generate the plurality of routing confidence scores are configured by further executable instructions to generate a first routing confidence score using a set of deterministic rules and a subset of the set of context data elements.
• Paragraph 39. The system of paragraph 35, wherein the one or more processors configured to generate the plurality of routing confidence scores are supplemented by further executable Instructions configured to cause a third-party routing confidence provider to execute using a subset of the set of context data elements.
• Paragraph 40. The system of paragraph 35, wherein the one or more processors are configured by further executable instructions to determine that a routing confidence score of the plurality of routing confidence scores indicates an ambiguity as to whether the response to the utterance is to be generated by a first application of the subset of applications, the response to the request identifying an ambiguity resolution component as an application that is to generate the response to the utterance.

terminology

Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein may be performed in a different order, may be added, merged, or omitted entirely (e.g., not all of the described acts or events are necessary for the practice of the algorithm). Moreover, in certain embodiments, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors or processor cores, or on other parallel architectures, rather than sequentially.

The various illustrative logical blocks, modules, routines, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or as combinations of electronic hardware and computer software. To illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software implemented on the hardware depends on the particular application and design constraints imposed on the overall system. The functionality described may be implemented in different ways for each particular application, but such implementation choices should not be interpreted as causing a departure from the scope of the disclosure.

Moreover, the various illustrative logical blocks and modules described in connection with the embodiments disclosed herein may be implemented or executed by a machine, such as a general-purpose processor device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof configured to perform the functions described herein. A processor device may be a microprocessor, but alternatively, the processor device may be a controller, a microcontroller, or a state machine, combinations thereof, or the like. A processor device may include electrical circuits configured to process computer-executable instructions. In another embodiment, a processor device includes an FPGA or other programmable device that performs logical operations without processing computer-executable instructions. A processor device may also be implemented as a combination of computing devices, e.g., a microprocessor, a processor core, a memory module, a processor architecture, or a processor architecture. B. as a combination of a DSP and a microprocessor, a plurality of microprocessors, as one or more microprocessors in conjunction with a DSP core, or as any other such configuration. Although described herein primarily in terms of digital technology, a processor device may also include primarily analog components. For example, some or all of the algorithms described herein may be implemented in analog circuits or mixed analog and digital circuits. A computing environment may include any type of computer system, including, but not limited to, a microprocessor-based computer system, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computation engine within a device, to name a few.

The elements of a method, process, routine, or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor device, or in a combination of both. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable storage device, a CD-ROM, or any other form of non-transitory computer-readable storage medium. An exemplary storage medium may be coupled to the processor device such that the processor device can read data from the storage medium and write data to the storage medium. Alternatively, the storage medium can be integrated into the processor device.

The processor device and storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Alternatively, the processor device and storage medium may be located as discrete components in a user terminal. Conditional language as used herein, such as, but not limited to, “may,” “could,” “would,” “might,” “e.g.,” unless specifically stated otherwise or the context used indicates otherwise, is intended to generally convey that certain embodiments include certain features, elements, and/or steps while other embodiments do not include them. Thus, such conditional language is not intended to generally imply that features, elements, and/or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for deciding, with or without other inputs or prompts, whether those features, elements, and/or steps are included or should be performed in a particular embodiment. The terms “comprising,” “including,” “having,” and the like are synonymous and are used inclusively and openly and do not exclude additional elements, features, acts, operations, etc. In addition, the term "or" is used in its inclusive sense (rather than its exclusive sense), so that when the term "or" is used, for example, to join a list of items, it means one, some, or all of the items in the list.

A disjunctive phrase such as the phrase "at least one of X, Y, Z" is, unless explicitly stated otherwise, to be understood in context as generally used to represent that an item, term, etc. can be either X, Y, or Z, or any combination thereof (e.g., X, Y, and/or Z). Thus, such disjunctive language is generally not intended to imply that certain embodiments require that at least one of X, at least one of Y, or at least one of Z be present, respectively.

Unless explicitly stated otherwise, articles such as "a" or "an" should generally be interpreted to include one or more described items. Accordingly, phrases such as "a device configured to" are intended to include one or more named devices. One or more of the named devices may also be configured together to perform the named tasks. For example, "a processor configured to perform tasks A, B, and C" may include a first processor configured to perform task A operating in conjunction with a second processor configured to perform tasks B and C.

While the foregoing detailed description has shown, described, and emphasized new features applied to various embodiments, it is to be understood that various omissions, substitutions, and changes in the form and details of the illustrated devices or algorithms may be made without departing from the spirit of the disclosure. As will be appreciated, certain embodiments described herein may be embodied in a form that does not provide all of the features and advantages set forth herein, since some features may be utilized or practiced separately from others. The scope of certain embodiments disclosed herein is therefore indicated by the appended claims rather than the foregoing description. All changes which come within the meaning and range of equivalence of the claims are to be understood as being included within the scope thereof.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US10515625 [0067]

Claims (15)

Computer-implemented method comprising: as performed by a computing system that includes one or more computer processors configured to execute specific instructions, performing a set of natural language processing actions on a natural language input to generate intent data, the natural language input representing an utterance of a user and the intent data comprising a semantic representation of the user's utterance; at an integration point during the set of natural language processing actions, determining to generate a context data item using an initial data item associated with at least one of the set of natural language processing actions; generating the contextual data item based at least in part on the initial data item concurrently with performing at least a portion of the set of natural language processing actions; and Providing access to the context data item, wherein the context data item is accessed during the performance of at least one of: the set of natural language processing measures or a set of response determination measures. Computer-implemented method Claim 1 , further comprising determining a context data action to be taken to generate the context data item using a register, the register specifying the context data action and one or more initial data items to be used to generate the context data item. Computer-implemented method Claim 2 , wherein determining the contextual data measure includes determining one of the following: evaluating the one or more initial data elements using a model to generate a prediction regarding an aspect of the user's utterance; performing a calculation using the one or more initial data items to precompute a value; or acquiring a value from a data store using the one or more initial data elements. Computer-implemented method Claim 1 , wherein generating the context data element comprises generating data representing at least one of: a location of a voice-enabled device from which the natural language input is received, an identifier of the voice-enabled device, a content catalog associated with a user profile, historical Interactions associated with the user profile, a prediction regarding an aspect of a user's utterance, or a comparison of a plurality of interpretations of the user's utterance. Computer-implemented method Claim 1 , further comprising: at a second integration point during the set of natural language processing actions, determining to generate a second contextual data item using a second initial data item; and generating the second contextual data item based at least in part on the second initial data item concurrently with performing at least a second portion of the set of natural language processing actions. Computer-implemented method according to Claim 1 further comprising: in response to generating the context data item, determining to generate a second context data item using the context data item; and generating the second context data item based at least in part on the context data item concurrently with performing at least a second portion of the set of natural language processing actions. Computer-implemented method Claim 1 , wherein providing access to the context data item includes storing the context data item in a data store accessible during performance of the at least one of the set of natural language processing measures or the set of response determination measures. Computer-implemented method according to Claim 1 , wherein performing the set of natural language processing actions comprises: generating text data using audio data and an automatic speech recognition ("ASR") process, the text data representing at least a portion of the user's utterance; and generating intention data using the text data and a natural language understanding ("NLU") process, the intention data comprising a semantic representation of the user's utterance. A system comprising: computer-readable memory storing executable instructions; and one or more processors in communication with the computer-readable memory and configured by the executable instructions to at least: perform a set of natural language processing actions on a natural language input to generate intent data, wherein the natural language input represents an utterance of a user and wherein the intent data comprises a semantic representation of the user's utterance; at an integration point during the set of natural language processing actions, determining to generate a context data item using an initial data item associated with at least one of the set of natural language processing actions; generating the context data item based at least in part on the initial data item concurrently with performing at least a portion of the set of natural language processing actions; and providing access to the context data element, wherein the context data element is accessed during performance of at least one of: the set of natural language processing actions or a set of response determination actions. System after Claim 9 , wherein the one or more processors are configured by further executable instructions to determine a context data action to be taken to generate the context data item using a register, the register specifying the context data action and one or more initial data items that to be used to generate the context data element. System according to Claim 9 wherein the one or more processors are configured by further executable instructions to at least: at a second integration point during the set of natural language processing actions, determine to generate a second context data item using a second initial data item; and generate the second context data item based at least in part on the second initial data item concurrently with performing at least a second portion of the set of natural language processing actions. System after Claim 9 , wherein the one or more processors are configured by further executable instructions to at least: in response to generating the context data item, determine to generate a second context data item using the context data item; and generating the second context data item based at least in part on the context data item concurrently with performing at least a second portion of the set of natural language processing actions. System after Claim 9 , wherein the one or more processors configured to: generate the context data item based at least in part on the initial data item are configured by further executable instructions to evaluate the initial data item using a model to make a prediction regarding an aspect to generate the utterance; and generate a response to the user's utterance based at least in part on the prediction, the prediction relating to a likely status of the user's utterance as one of an initial user utterance of a multi-turn dialogue or a subsequent user utterance of a dialogue with several twists. System after Claim 9 , wherein the one or more processors configured to provide access to the context data item are configured by further executable instructions to do so during performance of the at least one of the set of natural language processing measures or the set of response determination measures in response to a request to respond according to the context data element. System according to Claim 9 wherein the one or more processors are configured by further executable instructions to: determine an application to which to forward intent data comprising a semantic representation of the user's utterance for generating a response to the user's utterance; and generate a response to the user's utterance using the application and the intent data.

Images