JP6025785B2 - Automatic speech recognition proxy system for natural language understanding - Google Patents

Description

  The present invention relates to the field of interactive response communication systems, and more particularly, selectively routes speech to an automatic speech recognition (ASR) processor, human speech recognition (HSR) resources, or both ASR and HSR mechanisms. To an interactive response communication system.

Related Application This application is a continuation-in-part of US patent application Ser. No. 12 / 985,174, filed Jan. 5, 2011, entitled "Automated Speech Recognition System for Natural Language Understanding." No. 13 / 070,865 (currently US Pat. No. 8,484,031), owned by the owner of this application, named “Automated Speech Recognition Proxy System For Natural Language Understanding”, filed on May 24 And claims priority to this application under Section 120 of the US Patent Act. The contents of the above referenced applications are hereby incorporated by reference as if fully set forth herein.

  Many companies interact with customers by electronic means (most commonly telephone, email and online text chat). Such electronic systems save a lot of money for the company by limiting the number of customer service agents or support agents required. However, the customer experience provided by these electronic systems is generally unsatisfactory. The customer experience may be acceptable in the case of a simple transaction, but if the customer is not proficient at talking to or interacting with a computer, it may be disappointing or totally frustrating Many.

  Such interactive response systems are well known in the art. For example, providing customer service via telephone using an interactive voice response (IVR) system is one such system. An example of a customer service system using IVR technology is described in Patent Document 1. The IVR system typically communicates with the customer using a series of pre-recorded phrases, responds to some verbal input and touch tone signals, and can route or forward the phone. The disadvantage of such IVR systems is that they are usually built around a “menu” structure, which presents only a few valid options to the caller at a time, and the caller Requires a narrow range response from

  Many of these IVR systems now incorporate voice recognition technology. An example of a system incorporating a speech recognition technology is described in Patent Document 2. The robustness of the speech recognition technology used by IVR systems varies, but the responses they are trying to hear and understand are often within a certain range of responses, which means that the end user can Limit your ability to interact with. Thus, the caller often feels forced to talk to the system “as if talking to a computer”. Furthermore, even when interacting with a system that utilizes speech recognition, customer input is often not recognized or incorrectly determined, thereby requiring the customer to contact a human customer service agent as soon as possible.

  Human customer service agents continue to be used for more complex customer service requests. These agents can talk to customers over the phone, respond to customer emails, and chat with customers online. Agents typically answer customer questions or respond to customer requests. The company has customer service groups, which may be outsourced to companies specializing in “customer relationship management”. Such a company operates a center where hundreds of agents are staffed, and these agents spend the entire day of the work on the phone or otherwise interact with the customer. spend. An example of such a system is described in Patent Document 3.

  A typical model for customer service interaction is one agent assisting the customer for the duration of the customer interaction. Sometimes, if a customer needs help with multiple requests, one agent (eg, a technical support representative) may transfer the customer to another agent (such as a sales representative). In general, however, one agent spends his or her time helping the customer for the entire duration of the customer's phone or chat session or solves the customer's problem via email. To concentrate on. Also, most call centers consider that agents take time to log calls (leave a record). The deficiencies of this heavy agent interface model are (1) high agent turnover rates and (2) typically requiring many initial and ongoing agent training, all of which The service will ultimately be a significant expense for these customer service providers.

  To reduce agent-related expenses, some organizations outsource their customer service needs. One recent trend in the United States with the proliferation of high-speed fiber optic voice data networks is to place customer service centers abroad to take advantage of lower labor costs. Such outsourcing requires an overseas customer service agent to speak English fluently. When these agents are used for phone-based support, it is often a challenge that they can clearly understand and speak in English. The unfortunate consequence of outsourcing overseas is the misunderstanding and unsatisfactory customer service experience for those seeking service.

  The improved interactive response system integrates computer-implemented speech recognition with intermittent use of human agents. To some extent, this has been done for years. A system that uses both a human clerk and an automatic voice recognizer is treated (Patent Document 4). Similarly, a system is disclosed in which only a part of a call that requires a user to interpret a user's utterance is presented to a human agent (Patent Document 5). The contents of these patents, as well as all other techniques referred to herein, are hereby incorporated by reference as if fully set forth herein. The benefits of such a system increase if its cost is relatively low, which generally requires only limited human interaction. To achieve such limited human interaction, it would be desirable to have a system that requires minimal initial training and whose results continue to improve over time. In particular, a learning / training system that provides “from day one” performance suitable for production use and that quickly increases in efficiency over time would be particularly valuable.

  Many existing ASR systems have significant training constraints, such as the need to be trained to recognize the voice of each particular user of the system, or the need to strictly limit the recognition vocabulary to provide reasonable results. suffer. Such a system is easily recognized as artificial by the user. Typical human prompt “Please do business”, artificial prompt “Would you like to make a reservation?” “Situation” to check the reservation status, “Cancel” to cancel the reservation. Please say. Please consider the difference between.

  The goal of the speech system with ASR (automatic speech recognition) was to achieve a conversation system for conducting caller dialogue, similar to the HAL in “2001 Space Travel”. In order to improve ASR functionality, voice user interface (VUI) techniques have been developed. This allows prompts to be expressed accurately and compactly in an attempt to reduce the vocabulary used and to give the caller hints about the words that the caller must speak in order to achieve more accurate speech recognition. Is done. Since then, ASR has improved and now deals with open-ended conversation recognition. However, such free answer conversations require more vocabulary, resulting in a much higher speech recognition error rate. As a result, more dissatisfaction and contempt for the IVR system remains for callers. This is based, for example, on over-confirming what was previously stated and understood, making wrong choices, and allowing the caller to return to the previous menu. VUI design attempts to direct the caller to a so-called “directed dialog” in an attempt to narrow the conversation from the general to the specific. ASR and NLU have been more successful when applied to directed dialogs because small regions have a limited vocabulary and a relatively small utterance repertoire. The IVR industry is working to further improve understanding by characterizing the knowledge domain using statistics and "search" with speech recognition. However, these approaches still have a significant number of callers, particularly those with dialects or pronunciation patterns that are difficult to understand using complex techniques such as building a personalized ASR acoustic model. , Do not handle well. With the advent of human-assisted recognition, there is now an opportunity to recognize speech, text, graphics, and video using automation and human understanding, which makes the understanding more accurate and ASR-based IVR Many of the system weaknesses are avoided. The fundamental task of the IVR system is to coordinate filling in the information slots in various business forms corresponding to user requests. In conventional IVR systems, this adjustment is typically performed according to a pre-fixed decision tree, and there is little departure from a limited number of ways to interact with the user. Various recognition strategies have been developed, including variations in VUI design, various criteria that are optimized to successfully identify an accurate understanding, and techniques for understanding and recognizing in the shortest possible time.

  There are many reasons why the system uses various appropriate techniques to make the interaction between the caller and the automated system that uses human-assisted recognition as seamless and natural as possible.

  Humans recognize and interpret meaning with much higher accuracy than automatic speech recognition (ASR), graphics and video processing, and natural language understanding (NLU) techniques. If it can be understood using humans when the accuracy of the automation is insufficient, it is possible to provide a good user experience while automating a large number of user interactions. However, computer resources can be scaled to meet an unusual and unpredictable volume peak, but unlike those computer resources, human resources need to be scheduled and are available in a timely manner to meet the peak It may not be. Therefore, DTMF (dual-tone multi-frequency) is also used when accuracy is not sufficient, so that the system automatically adapts to the amount of HSR required in any particular application, thereby minimizing the use of HSR. It is needed. If human interaction changes during unscheduled peaks, self-service can continue to be implemented in a more traditional manner.

  The goal is now how to combine human assistance and automation to best recognize and interpret the caller's utterances while at the same time achieving the most human-like user experience possible. Traditional techniques used to classify utterances and achieve the highest level of recognition vary in subtle but important ways. Thus, the challenges that are not addressed by existing systems use the most efficient combination of human and automation in a given situation under a given workload, while providing the most successful user experience It is.

  Traditionally, the ASR system begins to “listen” to it as it is spoken. If recognition automation fails, the user will wait for the length of time it takes for a complete utterance to be spoken, after which the HSR will begin to listen and process it. Instead, it would be desirable if the system could try to understand the dialog in near real time. For example, as the user speaks words and describes their meaning (or “intent”), it is first processed by the ASR and then processed by the HSR, resulting in the end of the utterance and the start of the response. A considerable time gap occurs. This time gap can be filled with audio reproduction such as typing sound, for example. This may work for some applications, especially for data gathering applications. In other applications, this time gap makes it difficult to continue a natural conversation with the system. In addition, the longer the story, the lower the recognition quality. The longer the story, the more words are included in the story, and the more the words are connected. In summary, these increase speech recognition errors and reduce the accuracy of understanding.

  Therefore, there is a need for an automatic recognition system that can predict perceived and work well as soon as possible before using human assistance and maintain human-like dialogue. In addition, since human assistance may be required, the automatic recognition system also monitors human assistance staffing and automatically increases confidence in understanding depending on system status load and human assistance skillset capabilities. They also need to be able to adjust and / or proceed to full automation.

  Larger systems, such as the Natural Language Understanding (NLU) system, require a significant machine learning period of tedious manual grammar descriptions to obtain usable results from larger grammars and vocabularies. Especially in environments where the vocabulary may be dynamic (such as a system for taking tickets for new plays or concerts with new music groups), the learning period is too long to produce satisfactory results. It may be too much. The inclusion of accents, dialects, vocabulary, grammatical regional differences, etc., further complicates the task of teaching systems such that such systems can achieve reasonable thresholds of recognition accuracy.

  Currently available ASR systems are effective in recognizing simple verbal utterances, such as numbers, data, and simple grammars (ie, small word sets and expressions composed of them). To date, however, ASR systems do not provide a sufficiently high level of speech recognition performance to create a speech interface that provides free-flowing conversations. In addition, ASR performance is not only degraded by accents and dialects as described above, but also by background noise, adult voices than children's voices, and often female voices more than male voices . ASR performance has improved over time, and some systems use a statistical language model that is intended to recognize a very wide range of responses from the caller, so the caller is very constrained It is possible to be recognized even when speaking naturally rather than speaking in a way of speaking. Even so, ASR performance is still not comparable to actual human interaction, and ASR systems that provide the highest levels of performance are time consuming and can be built and tuned to specific applications ( tune) is expensive.

  Tuning the grammar by taking into account the statistical probabilities of the various answers expected and synonyms is one technique used to improve ASR performance. Another technique is to create a statistical language model, which can require considerable effort to transcribe a recording of the speech of a live telephone conversation with a live operator. ASR performance is fairly acceptable in some applications, but is not yet suitable in other applications, so known ASR-based systems still lack the ability to understand unconstrained natural utterances.

  Accordingly, there remains a need in the art for interactive systems that provide a consistently high quality experience without the limitations of configuring ASR components.

US Pat. No. 6,411,686 US Pat. No. 6,499,013 US Pat. No. 5,987,116 US Pat. No. 5,033,088 US Pat. No. 7,606,718

  An interactive response system mixes the HSR subsystem with the ASR subsystem to facilitate natural language understanding and improve the overall capabilities of the voice user interface. This system allows an incomplete ASR subsystem to use HSR when needed and still reduce the burden on the loaded HSR subsystem. An IVR system is implemented using an ASR proxy, which routes the utterance to only one ASR based on a set of rules, routes the utterance to the HSR in addition to at least one ASR, Routing utterances only to one or more HSR subsystems, rerouting utterances originally sent to the ASR to the HSR, using HSR to assist in pacing and training of one or more ASRs, and , Decide to use multiple ASRs to increase the reliability of the results.

  In one aspect, the ASR proxy comprises a recognition decision engine and a result decision engine. In a related aspect, the two engines facilitate recognition performance, natural language understanding, and recognition and grammatical pacing to accurately fill information slots in various business forms.

  In yet another aspect, the ASR proxy may determine the ASR resource and / or based on one or more of application criteria, recognition confidence prediction, historical results, and recognition experienced in a particular user's voice. Select an HSR resource.

  In yet another aspect, the ASR proxy can be configured based on various parameters, such as maximizing the use of ASR or making the interaction more “human” or “less human”.

  In yet another aspect, the ASR proxy automatically adapts to the HSR system resource capacity to maximize the use of ASR or DTMF.

  In yet another aspect, the ASR proxy uses the results of the evaluation component that analyzes the ASR results to determine an optimal length for length-based testing and an optimal quality metric level for user responses to various prompts. Select one or more of the best classifiers for the various prompts.

  In yet another aspect, the selection of ASR or HSR resources by the ASR proxy is transparent to software applications that request voice recognition from the ASR proxy.

  In yet another aspect, the system maintains a more human experience using a method that predicts successful automatic recognition in near real time when using HSR.

  Those skilled in the art will recognize that the particular configurations targeted in this disclosure can also be implemented in various other ways. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

  The features described above can be used alone or in combination without departing from the scope of the present disclosure. Other features, objects and advantages of the systems and methods disclosed herein will become apparent from the detailed description and figures that follow.

Still other features and various advantages will be apparent from the following detailed description when read in conjunction with the accompanying drawings. Like reference characters refer to like parts throughout the drawings.
1 is a block diagram illustrating one embodiment of an interactive response system architecture. FIG. 2 is a flowchart illustrating one embodiment of a method for communication between a customer, an interactive response system, and a human interface. Figure 3 is a chart illustrating one embodiment of customer / interactive response system interaction in the context of Figure 2; FIG. 3 is a computer screen user interface illustrating one embodiment for capturing customer intent and data in the context of FIG. 2. Figure 3 is a chart illustrating one embodiment of customer / interactive response system interaction in the context of Figure 2; FIG. 3 is a computer screen user interface illustrating one embodiment for capturing customer intent and data in the context of FIG. 2. Figure 3 is a chart illustrating one embodiment of customer / interactive response system interaction in the context of Figure 2; FIG. 3 is a computer screen user interface illustrating one embodiment for capturing customer intent and data in the context of FIG. 2. FIG. 5 is a flow chart for processing an email in the context of an interactive response system. 1 is a block diagram illustrating one embodiment of an interactive response system architecture having a training subsystem. FIG. FIG. 4 is an exemplary process flow 800 for ASR training. FIG. 6 is a high-level block diagram illustrating an example of a computer 900 used as any of the computers / processors referred to herein. FIG. 5 is a timeline representation of recognizing audio stream intent and data by different intent analysts. FIG. 2 is a block diagram of an application that interacts with an ASR proxy and shows the major components of the proxy. 2 is a flow diagram illustrating a recognition decision engine process and decision flow for determining whether to use ASR, HSR, or both. 2 is a flow diagram illustrating a process and decision flow for a results determination engine that uses a single ASR. 2 is a flow diagram illustrating a process and decision flow for a result determination engine that uses multiple ASRs. 2 is a flow diagram illustrating a process and decision flow for a results determination engine that uses both ASR and HSR. 2 is a flow diagram illustrating a process and decision flow for a result determination engine using HSR. It is a time-series figure which shows the response gap by automatic recognition and human assistance recognition. FIG. 2 is a block diagram of an application and interaction with an ASR proxy, showing the main components of the ASR proxy. 5 is a flow diagram illustrating a recognition decision and results decision process and decision flow using recognition statistics and system status information. FIG. 5 is a flow diagram showing recognition decisions and result decisions using ASR statistics and system status. FIG. 6 is a flow diagram showing recognition decisions and result decisions using timer ASR statistics and system status. FIG. FIG. 7 is a flow diagram showing recognition decisions and result decisions using predictor recognition ASR statistics and system status. FIG. 6 is a flow diagram illustrating a process for generating statistics. FIG. 6 shows examples of some recognition optimization criteria for generating statistics.

  First, a description of the operation of the interactive response system and associated machine learning system and process is provided according to FIGS. Thereafter, in accordance with FIGS. 11-16, a description of the operation of the ASR proxy system and associated processes is provided. 17-24 and the corresponding discussion generally relate to the process of optimizing the ASR proxy, the goal is to optimize the combination of computer recognition automation and human assisted recognition while simultaneously improving the user experience. It should be noted that the terms “intent” and “meaning” as used herein refer to the contextual reason corresponding to the utterance (eg, for a caller who makes a new flight reservation), unless expressly specified otherwise. Let the system determine the intent of the request). In contrast, the term “recognize” and its derivatives are generally used herein in the process of converting a sound into a corresponding word.

  Multi-channel and multi-modal systems are realized using a human assisted decision engine. This is the use of HSR even after auto-recognition conflicts, after routing "interactions" to automation, and depending on the prediction results from automation, based on a set of prediction data and capacity factors. decide. In some embodiments, the system automatically accelerates “speech” or “video” to further reduce the time gap between automation and human assistance.

  Interpreting responses to prompts can be viewed as two types of text analysis: information extraction and sense classification. Information extraction is the identification, extraction and normalization of specific pieces of information that are essential to fill a slot in a business form, such as customer ID, phone number, date and time, address, product type, and problem. Sense classification is concerned with identifying two additional types of information: meaning (intent) and response quality. The meaning (intent) has to do with what kind of form needs to be filled in (billing, booking scheduling, complaints, etc.). Response quality has something to do with the response itself (indistinct, noisy, Spanish rather than English, a desire to talk to a live agent, etc.).

  This response interpretation can be done by intent analysis alone (pure HSR), by automation (ASR and intention classification), or by some combination of ASR and HSR. Use ASR automation results to determine when ASR is producing reliable results, with limited quality loss or without quality loss, to ASR automation It is possible to trade off. This means that the combination of the two approaches in a proxy processing system can achieve a greater throughput than if only HSR is used, and can handle peak demand loads with a smaller team of intention analysts.

  FIG. 1 illustrates one embodiment of an architecture that connects an interactive platform 102 to an interactive response system 100 via an interactive router 101 (hereinafter referred to as an “i-router”). As shown in FIG. 1, the interaction platform 102 is connected to a customer 103 via a communication link 104. The interaction platform 102 is also connected to the interactive response system 100 at the irouter 101 via a data link, which in this exemplary embodiment includes a TCP / IP data link. The interaction platform 102 in this exemplary embodiment includes a computer server. The exact configuration of the computer server varies depending on the implementation, but typically an audio board from a vendor such as Dialogic® is used to run an operating system such as Windows® or Linux® It consists of a Pentium (registered trademark) -based server. The interaction platform 102 can also be an email gateway or a web server. Thus, customer input enters the interactive response system 100 via telephone or local calls, and text is entered via email or an interactive chat interface (eg, a web page or a stand-alone application such as Yahoo Messenger). The

  In the architecture of FIG. 1, in various embodiments, each of interaction platform 102 and communication link 104 is implemented using a number of different types of devices. The interaction platform 102 can be implemented by any device that can communicate with the customer 103. For example, the interaction platform 102 is, in one embodiment, a telephone server in the interactive response system 100, where the customer is making a call. The telephone server handles incoming call answering, forwarding and disconnecting. The phone server is also a warehouse for pre-recorded audio clips, so the phone server can play any welcome prompt and other audio clips directed by the i-router 101.

  The telephone server according to this embodiment is assembled from off-the-shelf components, for example, Windows as an operating system, a central processing unit such as a Pentium processor, and an Intel (registered trademark) voice board. Using this architecture, the communication link 104 is implemented by any means that provides an interface between the customer's phone and the phone server. For example, the communication link 104 is a dial-up connection or a two-way wireless communication link in various embodiments.

  In another exemplary embodiment, the interaction platform 102 is a gateway server in the interactive response system 100. According to this exemplary embodiment, the customer interacts with the interactive response server via email, interactive text chat or VOIP. The gateway server executes customized open source email, www server software or SIP. In addition, the gateway server according to this exemplary embodiment is designed to conduct email, interactive text chat or VOIP transactions with customers, as well as transfer and receive data with other elements of the system. Using this architecture, the communication link 104 is implemented by any means that provides an interface between the customer's computer and the gateway server. For example, the communication link 104 is a dedicated interface, a single network, a combination of networks, a dial-up connection or a cable modem in various embodiments.

  Although only one interaction platform 102 is shown in FIG. 1, those skilled in the art will appreciate that multiple interaction platforms 102 can be used in this system after reviewing this specification. If there are multiple interactive platforms 102, the interactive response system can communicate with the customer via voice and text data. In addition, a dedicated interaction platform 102 for each customer base can accommodate multiple customer bases. In this way, a workflow (described in detail below) is selected by determining which of the plurality of interaction platforms 102 has initiated the interaction.

  In the architecture of FIG. 1, the i router 101 includes software that controls the interactive response system 100. iRouter 101 “owns” the interaction with customer 103 from start to finish by coordinating activities between other components and managing transactions. iRouter 101 manages interactions with customer 103 according to one or more programmable scripts (referred to as “workflows” according to this exemplary embodiment). In general, a workflow includes an interaction flow in which a route through the workflow depends on an intention input from a customer. The workflow is pre-programmed by the system engineer and is advantageously "smallly improved" on a regular basis to improve customer satisfaction, speed and accuracy. According to this exemplary embodiment, iRouter 101 is almost always “responsible” to select the next step or path in the workflow.

  The i-router 101 receives the dialogue entered from the dialogue platform 102 in the form of an audio clip, email, text data, or other interaction type depending on the form of customer communication. The iRouter 101 receives this input from one or more human agents 105 (sometimes referred to as “intention analysts” or “IAs”), speech recognition engines or expert systems (collectively 108, and “automatic speech recognizers”). That is, it may be referred to as “ASR”), and the current workflow is advanced using the response. When the input needs to be interpreted (or translated) by a human, iRouter 101 instructs the human agent desktop software to display the appropriate visual context for the current workflow. When the i-router 101 understands the input, the i-router 101 proceeds through the workflow and instructs the dialogue platform 102 to respond appropriately to the customer 103.

  In an exemplary embodiment where the interaction platform 102 includes a telephone server, the iRouter 101 sends a sound clip for playback, a text-voice clip, or both to the customer. Alternatively, the interaction platform 102 can store sound clips, have text-to-speech functionality, or both. In this embodiment, the i-router instructs the interaction platform 102 what to play and when to the customer.

  The iRouter 101 in this exemplary embodiment comprises a networked off-the-shelf commercial processor running an operating system such as Windows or Linux. In addition, the iRouter 101 software includes a modified Open VoiceXML (VXML) browser and VXML script that incorporates objects suitable for a particular application. Those skilled in the art will understand how to construct these objects after reviewing this specification.

  According to the exemplary architecture of FIG. 1, interactive response system 100 includes at least one pool of human agents 105. The pool of human agents 105 is often located at the contact center location. The human agent 105 uses specialized desktop software specific to the system 100 (described further below with respect to FIGS. 3B, 4B, and 5B), according to this embodiment of the present invention, which is a possibility. A collection of intents is presented on those screens (their user interfaces) along with the history or context of customer interaction up to that point. One or more human agents 105 interpret the input and select the appropriate customer intent, data, or both.

  In the case of a telephone conversation, the human agent 105 wears headphones and listens to a sound clip (“utterance”) streamed from the telephone server 102 according to an instruction from the i router 101. According to one aspect of the invention, a single human agent 105 does not handle the entire transaction for customer 103. Rather, the human agent 105 handles several parts of the transaction that are designated by the workflow designer as needing the human 103 to interpret the utterance of the customer 103. The i-router 101 can send the same customer 103 interaction to any number of human agents 105 and can distribute a portion of a given interaction to many different human agents 105.

  According to an exemplary embodiment of the present invention, human agent 105 is preferably off site. Furthermore, the human agent 105 may be present in various geographical areas of the world, such as India, the Philippines, and Mexico. The human agents 105 may be a group in the building or may be working from home. In applications that require 24/7 human agent support, human agents 105 can be deployed throughout the world so that each human agent 105 can work during appropriate business hours.

  The interactive response system 100 of the present invention utilizes custom human agent application software. The human agent 105 uses a custom application developed in Java and running on a standard call center computer network workstation. Generally speaking, the interactive response system 100 requires human intelligence to interpret the customer's 103 input to determine "intention" (what the customer wants) and data (what the customer wants). Applicable to any data). Interpretation typically includes, in this exemplary embodiment, selecting from the list of choices the most correct interpretation of what was said. In an alternative embodiment, computer-aided data entry (eg, text entry or email address entry autocomplete) is used in conjunction with agent processing.

  The workflow server 106 of the present invention, which is an off-the-shelf component, is an archive of workflows used by the interactive router. The workflow server 106, in one embodiment, is built with off-the-shelf hardware using a commercially available processor running a standard server operating system, and in this exemplary embodiment, the workflow document is written in XML. . The workflow server 106 maintains a set of business rules that regulate the behavior of the i router 101.

  The interactive response system 100 utilizes a workflow designer used by a business analyst or process engineer to formulate a workflow. The workflow acts as a map that the iRouter 101 follows in a given interaction with speech recognition or with a human agent. In response to the customer input, the workflow “steers” the i-router 101 along the route in the workflow. The location in the workflow is called the “context” along with the data collected up to that point.

  The workflow designer constructs an instruction for the human agent 105 in the workflow in order to guide interpretation of the intention of the human agent 105. The workflow designer may include a version of the Eclipse software development environment customized to focus on building XML documents. However, after reviewing this specification, one of ordinary skill in the art will be able to develop a workflow designer.

  The performance and interaction archive 107 of the present invention includes a database that can be maintained on any common computer server hardware. The performance and interaction archive 107 is both archive data of system transactions with the customer 103 (ie, a repository of sound clips, emails, chats, etc. from the interaction with the customer 103) and performance data about the human agent 105. including.

  This exemplary embodiment utilizes “reporter” software to generate statistics about groups of interactions or to display the performance ranking of human agents 105. The reporter software can also reconstruct the interaction with the customer 103 from the sound clip, email, or chat text that made up the contact of the customer 103 stored in the interaction archive 107. Reporter software is a series of simple scripts that may be executed on any common server hardware.

  This exemplary embodiment also includes manager / administrator software, which is typically run from the same station as the reporter software. The manager / administrator software sets operating parameters for the interactive response system 100. Such operational parameters include, but are not limited to, business rules for load balancing, uploading changes during a workflow, and other management changes. In one particular embodiment, the manager / administrator software is a small custom Java application that runs on a standard call center computer workstation.

  Support system 108 consists of a number of databases and customer proprietary systems (including off-the-shelf automatic speech recognition (ASR) software such as Nuance®) that can be used in responding to customer 103 requests. For example, the support system 108 may include a database for customer information or knowledge base. The speech recognition software is an off-the-shelf component that is used in this exemplary embodiment to interpret the customer 103 utterances. Support system 108 may also include text-to-speech functionality, which is often off-the-shelf software that reads text to customer 103.

  The company agent 109 of the present invention is a human agent that handles a customer 103 request for which a workflow makes an inquiry. For example, if the customer 103 intends to get help with a company and the outsourced human agent 105 identifies this intention, the workflow will respond interactively to forward the call to the company agent 109. System 100 can be instructed.

  Elements of the interactive response system 100 communicate via a TCP / IP network in this exemplary embodiment. Communication is driven by a workflow that the i-router 101 follows. The “database” in this embodiment can be a flat file database, a relational database, an object database, or any combination thereof.

  Turning now to FIGS. 2-5, these figures illustrate how information is retrieved and processed by the interactive response system 100 when a customer interacts with the interactive response system 100 via the telephone. An example of The example shown in Figure 2 shows that all the necessary hardware, software, networking and system integration is complete, and that the business analyst has used the graphical workflow designer to develop the possible steps in customer interaction. It is assumed that there is. The business analyst has also created text for anything that the interactive response system might say to the customer 103. These are the first prompts (for example, “Thank you for calling. Sound that is sent to the customer while on) and closing words. Either text-to-speech software or voice talent records each server-side voice written by the business analyst. This workflow is loaded into the interactive response system 100 where it can be used by the i-router 101.

  As shown in block 201, the conversation begins when customer 103 calls the company's customer service phone number. The interaction platform 102 (in this case a telephone server) responds to the call, as shown in block 202, (1) the caller's ANI / DNIS information, or (2) other business rules (eg, a phone call came in) The appropriate workflow stored in the workflow database is retrieved based on either the line or the trunk line). The phone server plays the appropriate welcome prompt as shown at block 203 and the customer responds to this prompt (block 204).

  For example, Inter Air, a fictitious airline, provides customer service via an interactive response system according to a call center embodiment of the present invention. Accordingly, the interactive platform 102 is a telephone interface, and the i-router 101 selects a workflow suitable for the internet.

  The exemplary workflow of FIG. 3A shows a first point or context in the workflow. There is no customer utterance, and therefore no intention or data to capture (and respond). The only response is a greeting and a prompt for customer input.

  Processing proceeds to box 204 in the flowchart of FIG. The telephone server starts digitizing the customer's verbal input and connects to the i-router. At this point, the workflow or business rule determines whether the interactive response to the customer needs to be handled by a human agent or by voice recognition software. That is, the i-router selects an appropriate workflow for the phone call from the workflow repository and conducts a conversation with the customer according to the workflow rules.

  To interpret the customer's language, iRouter 101 uses ASR from the support system, as appropriate, or causes the customer's audio to stream to human agent 105 in the contact center, as shown in block 205. If human agent 105 is required by the workflow, iRouter 101 identifies available human agents by applying a load balancing algorithm as shown in block 207 and pops up on their screen. Trigger (as shown in the empty pop-up screen of FIG. 3B, initially), presents several selectable intent options, and begins streaming customer audio to the identified human agent. As will be appreciated by those skilled in the art given this disclosure, this load balance identifies more or fewer human agents to interpret the utterance based on any of a variety of factors at various times. Including doing. As shown in blocks 210 and 211, the human agent listens to the customer's utterance with headphones and the computer software prompts interpretation of the utterance.

  According to the exemplary workflow of FIG. 4A, the customer utterance heard by one or more human agents is “I want to confirm my flight from Chicago to London this afternoon”. As shown in FIG. 4B, the agent screen shows the current context (or point in the workflow). In this illustrative screenshot, there are 12 possible requests (including unanswerable and terminated) that a human agent can select. In operation, there are hundreds of possible interpretations available to agents. This variety of choices gives the agent the flexibility of interpretation, which allows the i-router to jump around in its workflow according to the interpreted intent. Therefore, according to one aspect of the present invention, the i-router can respond appropriately even if the customer changes the subject on the way.

  In each case, each agent selects the one that feels the most appropriate interpretation of the customer utterance in the current context of the workflow. In the example of FIG. 4B, the human agent selects “CFT” (Flight Time Confirmation) and enters the departure city and arrival city (or other pre-programmed information that the customer may speak). Or select from the drop-down menu.

  Note that in blocks 208 and 209, the human agent can decide to apply acceleration to the customer audio clip received at the station to compensate for any response delay (response delay is typically in application setup). Delay time, ie, the time it takes for the human agent desktop software to receive the streaming audio and display the appropriate workflow). Network latency may be around 0.2 seconds, and application delays may be larger in the range of 1+ seconds. To compensate for application delay, the interactive response system accelerates the voice clip (but not to the point where distortion is perceivable). The goal is to strive for a more “real-time” conversational conversation so that the customer does not experience noticeable delays while waiting for a response. Acceleration is applied to words as they flow from the phone server. Acceleration cannot overcome link-specific latencies, but acceleration allows human agents to “recover” any application setup time, and to reduce the amount of latency in the conversation, ideally due to latency in the network. It can be reduced to the limit imposed. However, acceleration is optional, and beginner agents may require slower replay, while more experienced agents can apply acceleration.

  In test 213, the i-router evaluates the accuracy of customer audio interpretation in real time and updates the speed / accuracy profile of each agent. At block 214, the iRouter processes the interpretation to perform the next step in the workflow (eg, a database search based on input data) and then forwards the appropriate response to the customer via the telephone server ( 218) (if the interpretation is considered accurate). If the i-router determines that the interpretation is correct, the i-router may play the response based on the interpretation of the speech recognition software or by applying a key algorithm to the response of one or more human agents. To send to customers. In this example, the response is given in the last block of screen 2 in FIG. 4A.

  To determine accuracy, the i-router compares the interpretations of the two human agents and, if no agreement is reached, seeks further interpretation and plays the customer audio clip to the third human agent ( That is, the “majority rule” determines which is the exact response). Other business rules may be used to determine the correct interpretation. For example, the interpretation from the agent with the best accuracy score can be selected. Alternatively, one of the interpretations can be selected and replayed to the customer ("I understand that ...") and the customer's response was correct To decide. In addition, an interpretation can be selected from known data (eg, only one of the two interpretations of a credit card number can compare two interpretations of email addresses with a database of customer email addresses. Pass the checksum algorithm, etc.).

  The interactive response system allows almost any number of human agents to handle the same customer interaction at a time. That is, the interactive response system can be listened to by two agents during busy hours, or can be listened to by seven human agents during spare time. Furthermore, during times of high phone volume, the “double check” rule can be eliminated to reduce accuracy and maintain fast response time. Agents assigned high trust ranks based on the agent's speed / accuracy profile can be asked to work without double checking. In addition to trading off accuracy for faster system availability, an uninterrupted flow of audio clips flows through each agent, thereby reducing the “lazy” time of human agents.

  Returning to the flowchart of FIG. 2, whether the customer will respond again, as seen in block 204, or will the call be forwarded, as shown in block 215 (depending on the step in the workflow or business rules)? If so instructed by) or the customer to end the call. If the interpretation is deemed inaccurate at block 213, iRouter 101 plays the time earned voice to the customer (block 216) and sends the audio clip to an additional human agent for another interpretation. (Block 217), the accuracy is reevaluated.

  The iRouter uses the workflow as its guide to manage customer interaction until the phone is completed. The i-router can stream customer utterances to human agents for interpretation at many points in the phone call. When the call ends, a snapshot of the customer interaction is stored in the archive database. The human agent's speed / accuracy profile is constantly updated and maintained.

  If no human intervention is required to interpret the customer request, the ASR interprets the audio clip and the i-router determines the appropriate response, as shown in blocks 206 and 214.

  Continuing with the example of Interair, as seen in FIG. 5A, the captured customer utterance has two requests: a food and entertainment query. According to another aspect of the invention, the human agent captures two intentions: a meal and a movie. There is no relevant data to enter. This is because the interactive response system already knows the flight information from the previous data entered in FIG. 4B (this data is visible in FIG. 5B). As seen in FIG. 5B, the human agent inputs “general” and “meal” from an on-screen display of possible intents. The human agent also enters a “movie”. As seen in FIG. 5A, the interactive response system provides an appropriate response. As can be seen in FIG. 5B, the customer may ask for other meals or movies such as “What meals do you have?”, “Is there any special meals?”, “What are the age restrictions for movies?” The appropriate human agent interpretation options are located on the computer screen.

  FIG. 6 shows an example of how information is retrieved and processed by an interactive response system when a customer interacts via email (e-mail generally known in the art). . As shown in block 601, the interaction begins with the customer sending an email to the company's customer service email address. An interaction platform (in this exemplary embodiment, a gateway server) opens an email and, as indicated at 602, based on either (1) customer to / from information and (2) other business rules. Retrieve the appropriate workflow stored in the workflow database. The gateway server sends an appropriate response acknowledgment as shown at 602. The iRouter 101 identifies available human agents for handling email by applying a load balancing algorithm, as shown in block 603, and triggers a pop-up on their screen for interpretation. Indicates possible intent and sends email content to one or more human agents. The human agent interprets the email as shown in blocks 604 and 605. In test 606, iRouter 101 evaluates the accuracy of customer email interpretation in real time and updates the speed / accuracy profile of each agent, but after this test, iRouter 101 processes the interpretation and follows it accordingly. Perform the next step in the workflow. Eventually, iRouter 101 forwards the appropriate email response to the customer via the gateway server, as seen at block 607 (if the interpretation is deemed accurate). As shown in block 608, the email is archived in an appropriate database. If the interpretation is deemed inaccurate, iRouter 101 sends an email to another human agent for another interpretation (block 609) and reevaluates its accuracy. The i-router 101 manages the interaction with the customer up to the e-mail response using the workflow as a guide.

  Considerations for the interactive response system described above with respect to FIGS. 1-6 and the processes that comprise it include the operation of one or more speech recognition and related subsystems 108. Implementation of the IVR system 100 actually requires that such a subsystem 108 be able to recognize a significant portion of customer utterances in order to minimize the need for human interaction.

  Referring now to FIG. 7, a training subsystem 710 is included as part of the IVR system 100. In operation, the training subsystem 710 selectively provides machine learning functions to the real-time ASR in the subsystem 108 so that they can adapt very quickly to new or changed customer interactions. For example, when the IVR system 100 is first installed for a company, the general functionality of the embedded ASR may not be very useful for actual customer interaction, especially when these interactions use industry-specific terminology. That's the case with many (for example, an electrician who calls to order a ground fault circuit breaker would typically use the acronym “GFCI”, but the ASR that easily recognizes this is There will be little). Similarly, when new offerings become available, existing ASR functionality may begin to fail even though it used to work (eg, “iPod®” in past use). ASR that correctly identified can start to fail if another product with a similar name such as “iPad®” is introduced). These changes may not be frequent in some applications, but may occur regularly in other applications. For example, applications for selling rock concert tickets will need to adapt regularly to new customer requirements for band names.

  In one embodiment, training is performed based on the indicated need for such training. For existing systems where the accuracy of ASR is well above the acceptability threshold, training may occur only occasionally, even if it occurs. In such a case, training can be performed, for example, only during periods of very low phone volumes (in which period the IA 105 is normally relatively idle). More training may be required if the system is new, or whenever the success of the ASR falls below acceptable limits, so the training subsystem 710 becomes active more frequently.

  The non-real-time training ASR 711 of the training subsystem 710 receives the customer utterance from the i-router 101 as an input and the corresponding intention from the IA 105 as input. In practice, a plurality of training ASRs 711 can be used as described below.

  As with the real-time production process, the process for non-real-time training includes input from a single IA in some embodiments, and input from multiple IAs in other embodiments. These differences are very useful in training ASR, as the differences in intentions selected by different IAs can indicate particularly subtle utterances that require a great deal of additional training. The business intent can have a small grammar with very few options such as “yes” or “no”, and a pre-packaged understanding of utterances in “yes” and “no” is attached to the ASR In the simplest form, training may consist of building a statistical model that can be used for grammar pacing. In more complex training, domain knowledge is used to assist word recognition in ASR to build a statistical language model of utterances that can be said.

  In one preferred embodiment, the IVR system 100 is implemented using multiple available real-time ASRs in the support system 108. In practice, each ASR is found to have strengths and weaknesses, and success in a particular area can be used by the i-router 101 to determine which ASR to use in a particular situation, and Can be used by the training subsystem 710 to determine which ASRs can benefit from training in a particular situation. Currently available ASRs are from Carnegie Mellon University (Sphinx), Nunance, Dragon, Loquendo (R), Lumenvox, AT & T (R), SRI International, Nexidia, Microsoft (R) and Google (R). Includes ASR. Because only carefully selected ASRs are available at no cost (eg, under an open source license), economic considerations may limit the number of ASRs included in the support system 108. Because iRouter 101 can selectively route production requests to ASRs that are expected to work well in any particular context, and the training subsystem 710 similarly uses real-time ASRs to predict their performance. It can often be advantageous to select a group of ASRs that have performance characteristics that are somewhat orthogonal to each other, since they can be selectively trained based on the improvements made. In this way, one ASR can be expected to make up for the weaknesses of another ASR. For example, an ASR that is optimized for processing telephone words may have significantly different performance characteristics than an ASR that is designed for words from dictation equipment.

  To increase the accuracy of real-time ASR used in IVR system 100, training subsystem 710 provides real-time ASR with training specific to the meaning of each received utterance based on the non-real-time behavior of training ASR 711. Make machine learning easy.

  In general, ASR is trained in several different ways. First, the ASR must be able to categorize the audio stream and each part of the audio stream into components that can help to recognize the spoken words. Typically this is a similar sound class known as “phone”, a sound transition or combination known as “diphone”, and a more complex case commonly called “senone”. It involves identifying a set with a waveform portion in the audio stream. In general, an utterance is split everywhere a silence period is detected. Divide a speech frame (such as a 10 millisecond time frame), and within this time frame, various different features of the audio (such as whether the amplitude and frequency are increasing, constant, or decreasing) Is extracted from the utterance. In the Spinx ASR available from Carnegie Mellon University, 39 features are extracted and the speech is represented as a “feature vector”. Typically, ASR engines involve this aspect that their perception is fixed, and users of such systems change which features are analyzed or how they are analyzed. I can't.

  ASR uses various models to proceed from the raw audio waveform to prediction of the word corresponding to the utterance. The acoustic model determines the most probable feature / feature vector for the received senone. A speech model maps sounds and words, but the words come from a fixed dictionary or come from a vocabulary (or “grammar”) derived by machine learning. The language model restricts candidate word choices based on some context, such as previously recognized words. ASR typically uses a combination of these models to predict which words correspond to utterances. The focus of training in the embodiments discussed below is the latter two models: a speech model and a language model, but the concepts covered here are easily applied to other models used in speech recognition. can do.

  In many cases, ASR training is accomplished by using context from previously recognized words or by using non-real-time processing (ie, words that are later recognized in the same customer discourse). It can be achieved more effectively. Such training is described below.

  First, look at the voice model and consider the user utterance: “I would like to fly roundtrip between Boston and San Diego.” The “off the shelf” ASR may have some difficulty in recognizing some of these words across different speakers. For example, when pronouncing the word “roundtrip”, some speakers may omit the consonant sounds of “d” and “t” into one sound (rountrip), while other speakers These may be pronounced separately (as if they were the two words “round” and “trip”).

  In one embodiment, the training subsystem 710 provides machine learning to the non-real time training ASR 711 by addressing each of these issues. First, the training subsystem 710 selects a target vocabulary based on the business meaning corresponding to the utterance determined by the IA 105 when the utterance was first received. In this case, there is a high possibility that the IA has selected “new reservation” as the message meaning. The word “roundtrip” may have been one of 40,000 words in general grammar and may have a very low statistical incidence, but it is specific to the intention of “new booking” May be one of only a thousand words and may have a much higher statistical incidence. Thus, the training subsystem 710 can increase the probability that the training ASR 711 will accept the word “roundtrip” as spoken by changing the applicable grammar even if the feature vector deviates significantly from the standardized model of this word. Raise significantly. In addition, as additional utterances of “roundtrip” become associated with the intention of “new booking”, these utterances are closer to at least some of the already known instances where “roundtrip” was spoken. It is likely that they will match. Therefore, as time passes, both the possibility of the word “roundtrip” occurring within the intent of “new booking” and the variation in pronunciation of this word will lead to the following two results. (A) More certainty when recognizing a word (this can be propagated to other grammars that contain the same word, such as a grammar related to the intention of “cancel reservation”) And (b) a refined statistic on how often a word is associated with a particular intent can better predict a business intent.

  Returning to the utterance example above, a fast-talker may not be able to pronounce all the sounds clearly, obscuring the distinction between “Boston” and the following word “and”, thereby training. ASR 711 may be trying to analyze the sound “Bostonan”. Similarly, the city name “San Diego” may be pronounced to sound like “Sandy A-go”, depending on the speaker. Again, the statistical possibility that the recognition of “Boston” and “San Diego” can be reliably achieved by selecting a grammar specific to “new booking” rather than a generalized grammar is also dramatic. This is likely to increase. As a further refinement, the training subsystem 710 uses a repetitive path through the utterance of the entire user discourse to further improve training. In the above example, during the discourse, the caller may then say “Boston” at the end of the sentence so that it can be easily recognized by the training ASR 711. This speaker's acoustic signature for "Boston" is included in the ASR mapping so that in the second pass, the same speaker's "Boston" utterance is considered a better match to "Boston" than before It will be. Similarly, the speaker may say “San Diego” the second time, making a more distinction between “San” and “Diego”, so that if it tries to recognize it repeatedly, 1 Learning is provided that leads to a higher likelihood of successfully recognizing the second ambiguous utterance. For long customer discourses, multiple iterations can lead to significant improvements in overall recognition, as the caller's voice characteristics become better understood through words that the system can recognize.

  Referring now also to FIG. 10, in one embodiment, the audio stream is decomposed into separate utterances for recognition (eg, by training ASR 711) using the actual recognition time points by the intent analyst. Specifically, the recognition time (1001, 1004) of the utterance intention “I want to take a flight from”, the recognition time (1002, 1005) of the data portion “Boston”, and the recognition time of the data portion “San Diego” ( 1003, 1006) are all sufficiently different, so the time frame itself can be used to facilitate breaking the audio into separate utterances for recognition. In some cases, the IA may provide recognition before (or after) the utterance is complete (eg, “San Diego” is displayed before the last “o” sound, as shown at 1003 in FIG. Thus, in such cases, the time frame is adjusted to end with an appropriate pause after (or before) the recognition provided by the IA. Possible requirement intentions and the number of typical words used to represent them can be used to narrow down the intention recognition grammar, and the type of data collected (eg city names) Can be used to narrow down the grammar.

  Moving on to the language model, the training system 710 still uses training intent to assist in training. For example, if the IA indicates a business intention of “new reservation”, at least one instance of the word “and” in the utterance will be preceded by one city name followed by another city name. The probability will be very high statistically. Similarly, if the words “from” or “to” are recognized, the probability that these words will be followed by a city name will be statistically very high. In contrast, if the business intention determined by the IA is “seat assignment”, these same words “from” and “to” rarely correlate with adjacent city names, but , Will correlate to a nearby number and letter pair (for example, “I would like to change from seat 39B to seat 11A”).

  Such language model training also allows easy adaptation to the changing language of the user. For example, if an airline initiates service to England, the airline may suddenly begin to receive requests for the same business intent using a different language than was previously used. For example, the previous example of “I would like to fly roundtrip between Boston and San Diego.” Might be spoken by a British customer as “I would like to book a return trip between Boston and London.” Initially, the word “book” will not appear with high probability in the “new booking” grammar, but the statistical use of this word in this grammar is quickly increased by additional British customers. Similarly, the use of the term “return” will change with the addition of the British customer base and the “new booking” grammar will be adjusted accordingly to recognize this.

  The training subsystem 710 also adjusts statistics for recognition candidates based on the combination of business intentions and adjacent recognized words in the discourse. Consider an example where the business intention is determined to be “new booking” and only one utterance in the user's discourse is initially unrecognizable at an available confidence level. If the discourse is recognized as containing only one city name, the probability that the unrecognized utterance is another city name is very high and is the city name supported by the airline using this system. The probability is even higher. Changing the probability for a candidate word in the grammar and performing partial recognition may cause some candidate words to be successfully truncated from further consideration, and only one candidate (possibly a city name) can be used May be at certainty level. In this case, machine learning incorporates this particular user's city pronunciation into the model of ASR so that subsequent instances of similar utterances are more easily recognized.

  Maintaining a separate grammar for each acceptable requirement intention makes it easier for the training subsystem 710 to provide ASR teaching faster than would normally be possible. For example, the utterances “book”, “notebook”, and “Bucharest” have strong audio similarities. The determination of which of these meanings corresponds to the user's utterance is greatly improved by considering the business intention. For example, if the business intent is “Lost and Found”, “book” (for the noun) and “notebook” (as in “notebook computer”) are much more than in other contexts. It will appear with high possibility. If the business intention is “new booking”, then “book” (with its verb meaning) will also appear very likely. Similarly, if the business intention is “new reservation”, “Bucharest” will appear with a higher probability than if the business intention was “seat selection”, for example.

  After the training ASR 711 itself is fully trained, the correlation between the business intention and the language model can be created in a very robust manner. For example, an illustrative part of a mapping of words that sound similar may be as follows:

  Training ASR 711 is particularly well suited for producing language model statistics because it has two advantages over real-time ASR from support system 108. First, since it is not used for production operations, it does not need to operate in real time, and therefore it is at least on a relatively moderate computing platform to perform recognition fast enough to be used for real-time processing. You can use more complex recognition algorithms that wouldn't be possible. This allows training ASR 711 to recognize utterances that real-time ASR in support system 108 would not be able to recognize. Second, the training ASR 711 can use not only deductive information from customer discourse but also inductive information. Thus, it is possible to wait until all utterances in the dialog have been analyzed and then take multiple paths upon recognition, perhaps more likely to succeed in later iterations. As mentioned above, the first user utterance that sounds like "Bostonan" can be recognized much more easily after the second "Boston" utterance.

  The training ASR 711 builds up a series of statistics related to the language elements used with each relevant business intent over time. In one embodiment, multiple training ASRs 711 are used, and each training ASR 711 contributes to the overall statistics. In some embodiments, the statistics include a measure of certainty about recognition that is based on multiple instances of recognition by a single training ASR 711, based on a match between multiple training ASRs 711, or both. Based.

  Statistics generated in this way can be used by any of the real-time ASRs in the support system 108. Each of the various ASRs that can be used for real-time recognition in the support system typically has its own mechanism for training and a corresponding specification on how a language model can be entered into this mechanism for training. Have In a preferred embodiment, the training subsystem 710 formats the statistics it produces for each ASR in the support system 108 so that the statistics generated by the training subsystem 710 are available to each of these ASRs. To. In practice, ASRs vary greatly in the mechanisms they support for training, so the training algorithm 712 is appropriate for each existing ASR as well as each new ASR that may be added to the support system 108. In a manner, it can be easily configured to collect, format, and provide training data to the ASR. Since the performance of real-time ASR improves with training, the quality of its recognition can allow real-time ASR to replace the function of IA 105 in processes 210, 211.

  The training subsystem 710 also works with the functionality of each ASR to ensure that ASR training is maximized for use in the IVR system 100. For example, ASR can support the determination of a threshold for when enough utterances are recognized to be usable for performing statistical analysis, such as using a sentence tree, and training algorithm 712 can It is configured to fit such features to determine the progress of

  Real-time ASR in support system 108 is used in two different ways that require different statistical processing. In the first scheme, they are used to recognize the process after the IA has determined the corresponding business intent. For example, one or more IAs 105 may select “new booking” as a business intention for a sentence spoken by the caller, based on which one or more real-time ASRs in support system 108 are Will try to recognize certain words spoken by the caller.

  In the second method, the business intention is determined using real-time ASR instead of IA. This is a different recognition task than determining the specific words spoken by the caller. For example, determining whether the business intent may be “new reservation” or “seat request” may include the words “from” and “to” for “new reservation”, and It may involve recognizing a small number of likely keywords specific to each intent, such as the words “passage side” and “window side” for “seat reservation”. One type of ASR in support system 108 may be better suited for determining the business intention, and another type of ASR may be better at recognizing words based on that business intention. May be suitable. In one embodiment, the format of the training statistics for each real-time ASR provided by the training subsystem 710 is such that the real-time ASR will be optimized for intent determination or word recognition based on the determined intent. It is adjusted based on what will be optimized.

  Part of the training process involves determining how effective machine learning was for real-time ASR in support system 108. This is called validation. In a preferred embodiment, validation is performed by the training subsystem 710. In an alternative embodiment, the validation is performed by iRouter 101 or a dedicated validation processor (not shown). In validation, ASRs are run in parallel with each other and with the IA to determine how comparable their performance is. Each training instance provides more information that is used to create a statistical model and probability of grammar usage for each subject meaning provided by the IA. In some situations, historical data from the IA also determines the expected level of automation that may be available for the utterance. If the IA always provides multiple meanings for utterances, the ASR will only be usable if significant context training is possible. An ASR with robust context processing may be able to handle such utterances correctly, but an ASR that is not contextually strong can meet the minimum threshold regardless of how much training is provided. It may not be possible. For example, the utterance “IP” may mean “Internet Protocol” or “Intellectual Property”. When used in applications where both meanings are general, an error in processing accuracy will be expected unless ASR is able to derive which of the two meanings is appropriate after training.

  As training progresses, real-time ASR performance improves. When the ASR is statistically stable enough to meet the need to specifically use this ASR within the IVR system 100, the ASR is placed in production operation. For example, an ASR intended to determine the business meaning of an utterance can operate in parallel with the IA in non-production mode until such time as its performance is sufficiently trained to reach that of the IA, When fully trained, the operation is switched to the actual operation, and the burden of IA in the processes 210 and 211 is reduced.

  In an exemplary embodiment, both real-time production processing and training processing provide input from two IAs to two ASRs for increased accuracy. If the input from two IAs for the same utterance in the same user discourse is different, in one embodiment, the utterance is selected based on a third IA (possibly based on the degree of quality of the IA) for semantic determination. Submitted).

  The training process transitions when the ASR reaches an accuracy level higher than a certain threshold, as determined through validation and as determined based on environmental characteristics. In one exemplary embodiment, ASR is used for production processing, but training continues as described above. All training ends in an environment that is less demanding or has fewer resources available. In the third environment, training continues, but priorities are lowered (eg, training processing only when there is a certain amount of processing capacity available, or the ASR performance has degraded to a certain degree. Only when you know.)

  In some embodiments, the validation processor is configured to test the ASRs to determine their performance level. Validation is followed in some embodiments after the training phase, and in other embodiments is performed concurrently with training. Based on the results from the validation, iRouter 101 changes its utterance assignment to ASR and IA. For example, if it is found that the ASR works sufficiently well in comparison with the IA in determining the business meaning, the i-router 101 routes utterances to this ASR much more frequently than to route to the IA. Advantageously, such routing is very adaptable and configurable. In accordance with the example used with respect to FIGS. 3-5, iRouter 101 can choose IA for interpreting the response immediately after the welcome message (FIG. 4B) based on performance statistics (FIG. 4B). The first ASR can be selected for response interpretation (FIG. 5A), the second ASR can be selected for response to seat designation and airplane information, and the other options shown in FIG. 5B can be selected. You can choose. In one embodiment, two ASRs (as in 210, 211) are selected for each specific interpretation area to ensure accuracy. If both ASRs provide the same interpretation, a corresponding response is provided to the user. If the ASR is different, the utterance is provided to the IA, as in 217, and the meaning is selected via judgment.

  As a result, the human IA is needed only in certain cases where the ASR cannot function properly, and processing can return to the ASR immediately after IA intervention, depending on business standards, and the IA is connected to the customer discourse. There is no need to remain. If training can improve ASR, training improves ASR without imposing many additional costs or other overhead on the overall IVR system 100. No human interaction needs to be involved beyond listening to a single user utterance and selecting the user's meaning or intention from a drop-down list of predefined options so that an appropriate automatic response is provided to the user. .

  Referring to FIG. 8, an exemplary process flow 800 for ASR training is shown. If the digitized audio stream containing the user utterance is provided to one or more IAs 105 (801) and the IA can provide a usable intent response as described with respect to FIG. 7, the audio stream is trained. Provided to ASR 711. If the training ASR 711 cannot fully recognize (802) the utterance to convert the audio to its corresponding text, the utterance is discarded and not used for training.

  If ASR 711 can fully recognize the speech (802), a statistical model / pacing grammar (eg, a grammar corresponding to the meaning and data provided by the IA) is constructed (803) as described above with respect to FIG. . For some utterances determined by ASR 711 that are below a certain confidence threshold, an additional verification loop can be utilized for IA to verify the recognition of intent or data by ASR 711. If recognition is confirmed, processing proceeds as described for 803, otherwise the result is discarded.

  Next, a test is performed to determine if the performance of the training ASR 711 is now sufficient (804). The performance threshold may depend on the criticality of the application example. Healthcare applications may be much less error resistant than, for example, a free traveler information service would be resistant to errors. The performance threshold may also depend on the rate at which new words or phrases are added to the statistical model. If the performance is not sufficient, processing returns to prepare for further utterances that can be digitized (801) and used for additional training. If the performance is sufficient, the training results are applied and the real-time ASR of the support system 108 is composed of models obtained from training (805), and these real-time ASRs are validated and, if appropriate, production. Used for processing.

  In some embodiments, the training is then considered complete. The ASR is initially brought online in a provisional mode, ie as an IA shadow. If the ASR meets a quality level as determined by business standards (eg, by comparing the results from the ASR with the results of one or more IAs), the ASR begins to be used fully in production. , Thereby replacing IA in process 210. Similarly, the performance of the second ASR is measured and if this ASR produces sufficient quality in recognition, it is brought online and replaces the second IA in process 211. In other embodiments, a further test 806 is performed at a time determined by the particular environment to determine if the ASR performance has dropped below some applicable minimum threshold. If it falls, the flow returns to 801 for additional training. If the performance is acceptable, the process loops back to 806 and repeats the test at the appropriate time. If performance does not reach an acceptable threshold even after many trials, in some embodiments, training is abandoned.

  FIG. 9 is a high-level block diagram illustrating an example of a computer 900 used as any of the computers / processors referred to herein. Shown is at least one processor 902 coupled to chipset 904. The chipset 904 includes a memory controller hub 920 and an input / output (I / O) controller hub 922. A memory controller hub 920 is coupled with a memory 906 and a graphics adapter 912, and a display device 918 is coupled with the graphics adapter 912. A storage device 908, a keyboard 910, a pointing device 914, and a network adapter 916 are coupled to the I / O controller hub 922. Other embodiments of the computer 900 have a different architecture. For example, in some embodiments, memory 906 is coupled directly to processor 902. In some embodiments, components such as the keyboard 910, graphics adapter 912, pointing device 914, and display device 918 are not used for certain computers 900 (eg, certain server computers) that do not require direct human interaction.

  The storage device 908 is a computer readable storage medium such as a hard drive, CD-ROM, DVD, or solid state memory device. Memory 906 holds instructions and data used by processor 902. Pointing device 914 is a mouse, trackball or other type of pointing device that is used with keyboard 910 to enter data into computer system 900. Graphics adapter 912 displays images and other information on display device 918. Network adapter 916 couples computer system 900 to the Internet 1001. Certain embodiments of the computer 900 have different and / or other components than those shown in FIG.

  Computer 900 is adapted to execute computer program modules for providing the functionality described herein. As used herein, the term “module” refers to computer program instructions and other logic used to provide a specified function. Thus, the module can be implemented in hardware, firmware and / or software. In one embodiment, program modules formed of executable computer program instructions are stored in storage device 908, loaded into memory 906, and executed by processor 902.

  The type of computer 900 used by the components described herein will vary depending on the embodiment and the processing power used by the entity. For example, customer computers 103 typically have limited processing power. In contrast, iRouter 101 may include multiple servers that work together to provide the functionality described herein. For certain applications, a single processor (or set of processors) can implement both real-time ASR in support system 108 and training ASR 711 and other functions of training subsystem 710. In these applications, determining when to do so much training allows a relatively inexpensive and reasonably powerful computer to be used for both training and production ASR processing.

  The systems and methods described above are not only applicable to voice interaction, but may also be used with, for example, video, text, email, chat, photos, and other images in some embodiments. These other embodiments can be used in applications such as online chat, security monitoring, theme park concierge services, and device help. As a specific example, a help mechanism in which open-ended questions are interpreted and processed as described above is described in Apple, Inc. Can be provided to consumer devices such as iPhone (registered trademark) and iPad devices. Similarly, the techniques described above can be used to facilitate video stream and image recognition.

  As is apparent from the above discussion, the ASR subsystem may be more appropriate than the HSR subsystem to handle some of the customer interactions. In order to provide the best possible user experience, when an application program (such as that stored in the workflow repository 106) seeks speech recognition resources, the selection of resources used for such recognition (ie, ASR). Alternatively, benefits can be achieved by optimizing the HSR, as well as the selection of specific ASR / HSR resources that best suit the current recognition task.

  Referring to FIG. 11, a block diagram of the operation of ASR proxy 1102 to achieve such selection of appropriate processing resources is shown. More specifically, the functions described below include, in various embodiments, encapsulation in Media Resource Control Protocol (MRCP) within a Voice Extensible Markup Language (VXML) browser, web services, and application programming interfaces. (API, for example, written in Java or C # language). In one particular embodiment, a common ASR from various vendors uses MRCP as a standard interface to the VXML platform (browser), and in this environment, the ASR proxy 1102 is for software applications 1101 running with the VXML platform. Is configured to be visible to the ASR engine, but instead acts as a proxy between the VXML application and the speech recognition function by providing speech recognition resources from both the ASR subsystem and the HSR subsystem. do.

  As will be described in more detail below, the ASR proxy 1102 may be one or more ASR subsystems 1104 (such as those described above with respect to support system 108 considerations) or HSR subsystem 1106 (as described above with reference to offsite agent 105 considerations). Etc.) is freely selected. Based on the statistical database subsystem 1105, the ASR proxy 1102 is configured with a recognition decision engine 1103 (this operation is further described with respect to FIG. 12) and a result determination engine 1107 (this operation is further described with respect to FIGS. 13-16). Communicate and make a decision regarding which ASR / HSR resources 1104, 1106 to use at any particular time. If any HSR resource is selected for use, corresponding user interface information is provided to the appropriate HSR desktop workstation 1108 as described above with respect to the offsite agent 105.

  ASR proxy 1102 alleviates the need for developers of software application 1101 to consider whether utterances should be recognized by ASR or HSR. Thus, such a software developer can build (and assume its availability) a more human voice user interface than has been conventionally used on computers.

  Referring to FIG. 11 in more detail, in various embodiments, software application 1101 serves various purposes. In one embodiment, software application 1101 is an IVR system for toll-free caller assistance, and in another embodiment is an interactive help application on a tablet computer. The software application 1101 tells the ASR proxy 1102 what to recognize (ie, provides the grammar to the ASR proxy 1102) and utterance (usually an audio file such as a .wav file, or a real-time audio stream (eg, MRCP Instruct the ASR proxy 1102 by providing it with a real-time protocol stream)). As expected, the ASR proxy 1102 responds with a “text” or meaning of what it recognizes, along with a confidence score that indicates the confidence of the ASR that it correctly recognized the utterance.

  Since the ASR proxy 1102 can have different functionality than the traditional ASR, the ASR proxy 1102 may require additional information, for example in grammar meta tags for statistics and decisions. This additional information includes a unique method for identifying prompts and grammars, a unique method for identifying the current session, a unique method for identifying “voices” or users (learning the speaker's acoustic model). And a threshold value for specifying the behavior of the ASR proxy 1102. For some applications, the grammar is predefined or built-in. In other applications, the grammar is not built-in, so meta-information related to the grammar (such as user interface information to frame and guide agent decisions) is provided and possible responses are Better defined (eg in the case of the HSR subsystem).

  When the software application 1101 requests the ASR proxy 1102 to recognize the utterance, the ASR proxy 1102 passes the processing to the recognition determination engine 1103. The recognition decision engine 1103 is responsible for determining how to recognize the utterance. For example, parameters and confidence thresholds provided by software application 1101 may affect this determination. As a specific example, if the application requires very high recognition quality, the recognition decision engine 1103 can instruct that recognition be achieved only by the HSR resource 1106. On the other hand, the application may consider cost as the most important, and as a result, by default only the ASR resource 1104 is instructed to be used, and the use of the HSR resource 1106 is error prone when using ASR You can also keep it alone.

  In one embodiment, the recognition decision engine 1103 automatically and dynamically makes similar decisions by varying the appropriate threshold to meet the specific requirements of the application. Thus, high quality thresholds can be used for wealthy bank customers, while utility billing queries from consumers are given a lower acceptable threshold. In this embodiment, the threshold is based on historical statistics calculated based on past recognition attempts.

  It has been found that beneficial results can be obtained not only by choosing between using ASR resources and HSR resources, but also by allowing such combinations of resources to be selected. For example, one parameter set is best met by submitting utterances to be recognized by multiple ASR resources, and another parameter set is best met by submitting utterances to a single specific ASR. Yet another parameter set may be best met by submitting utterances to a mixture of ASR and HSR resources. In practice, the extent to which the ASR has been trained or tuned (eg, as discussed above for training), whether the ASR has been validated for a particular grammar, whether the cost of multiple recognition paths is acceptable, and history All things such as results are helpful in deciding which resources to apply in any particular situation.

  Similarly, security meta tags related to utterances can help determine the most appropriate recognition resource. For example, a meta tag indicating that the utterance is a social security number can be sent to be processed by an ASR resource to avoid the possibility of a person obtaining personal information about a person.

  Another parameter considered in certain embodiments is the level of activity of various system resources. If there is a large demand on human staff, this outstanding request can be used as a parameter to choose to increase the use of ASR resources.

  In one embodiment, a double check of results is provided using multiple resources, whether of the same type or different types.

  In yet another embodiment, the recognition decision engine 1103 dynamically keeps track of the current audio stream length and compares it to the expected utterance length defined by the corresponding grammar. For example, an utterance is expected to have a grammar consisting of only one of the three colors "red", "green", and "blue", and the actual utterance length is If it is 3 seconds, based on the expectation that the utterance is not one of the expected single syllable colors in the grammar, the previous decision to make the ASR resource aware of the utterance is modified and added to the ASR Alternatively, the HSR resource can be recognized instead of the ASR. Such an approach has been found to minimize the final time to recognize “unexpected” utterances and thus increase the overall efficiency of the ASR proxy 1102.

  As described above, the operation of the ASR proxy 1102 and the corresponding engines 1103, 1107 widely utilizes statistics, thresholds and other unique information for personalizing the system to address the needs of the software application 1101. This information is stored in the statistical database 1105 as shown in FIG. For example, the results of the ASR operation are stored in the database 1105 as confidence score statistics, and the total statistics for this ASR are subject to this ASR under applicable business rules or other rules required by the software application 1101. Is considered as to whether it can be used. In addition, all statistics related to speech, such as speaker, prompt, grammar, application, recognition method (eg ASR, HSR, single ASR, multiple ASR, multiple HSR), confidence, no match or no input, and training / pacing Is stored in the database 1105 by the ASR proxy 1102.

  As described with respect to the previous figure, if the ASR fails to provide a usable result for the utterance, the utterance is sent to the HSR resource for recognition / mismatch resolution. Statistics are maintained for HSR as well as ASR, and statistics are also maintained on an individual speaker basis. Thus, if an ASR is found to be particularly effective in recognizing a particular speaker, statistics are maintained and updated to increase the likelihood that this ASR will be used for later utterances from the same speaker. Is done. Similarly, statistics are also maintained on an individual grammar basis, which again allows the recognition decision engine to choose the appropriate resource to use based on the expected grammar or prompt / grammar combination. Is maximized. For example, the “yes / no” grammar would be more effective at recognizing simple prompts by ASR, such as “Are you John Smith?”, But “I feel better today than the same day last week. More complex questions, such as “Okay?” Would be less effective.

  Generalizing from the above, statistics are generated on a variety of grounds and maintained so that intelligent decisions are made about when to use a particular ASR / HSR resource. Based on the confidence level, a grammar capable of highly reliable ASR recognition can even be used by the software application 1101 more frequently. For example, a “yes” or “no” grammar would be very reliable with simple ASR resources. According to the statistics, a simple confirmation statement such as "Is your phone number (555)123-4567 correct?" Says "Yes if you feel good in the past week" Please. Say “No” if you are not feeling well. Are recorded for prompt / grammar combinations, up to more complex communications.

  The grammar considerations herein can be extended and generalized to a combination of grammars and prompts. For example, one statistic relates to the overall confidence level for a set of utterances (ie across multiple prompts) for the current speaker in the current session. If ASR recognition fails for the speaker regardless of the prompt / grammar combination, this would be better for the ASR proxy 1102 to rely on the HSR rather than attempting ASR for this speaker. It shows that. On the other hand, if the utterance of a particular speaker always shows strong confidence, the ASR proxy uses ASR as the preferred recognition method. To generalize beyond a particular session, a unique speaker reference ID allows the system to recognize a particular speaker (eg, based on the phone number used to connect to the system) Appropriate ASR or HSR resources can be selected.

  Software application 1101 provides thresholds that can be considered appropriate for a particular situation by a software developer and, in some situations, generated over time based on previous cognitive experiences. For example, if statistics can be generated by double checking or confirmation via HSR resources, these statistics are collected and stored in database 1105. Means, standard deviations and mode information from such statistics are applied to various thresholds based on the overall goal of such applications and as determined by the software developer of software application 1101. The

  In addition, statistics can be used to determine when further reliance on ASR resources is no longer effective. For example, if a significant sample size recognition quality for an ASR and a particular grammar indicates that the performance is unlikely to exceed an acceptable recognition threshold, this ASR may be future for this particular recognition task. Excluded from consideration. Although this recognition task may require more training (or pacing), it turns out that multiple training / pacing attempts will not work, so this particular recognition attempt It is permanently excluded from consideration until changes occur, such as adjustments or the use of new or new versions of ASR.

  Statistics can also be used to tune the ASR. Grammar pacing can be purely statistical, such as the percentage when “red” is used in the grammar “red, green, or blue”, or synonyms such as “turquoise” for “blue”. May also contain. In the latter case, pacing is facilitated by using HSR resources for “out of grammar” recognizers (eg, to confirm that “turquoise” should be considered a synonym for “blue” in certain cases). To). Immediately after such pacing, in some applications, tuned ASR is introduced on a “silent” limited test basis rather than production basis to ensure performance is above acceptable thresholds. Would be desirable. In one embodiment, to verify that the ASR can recognize the grammar, and to calculate confidence threshold statistics during the above-described validation, and if the recognition by the ASR is invalid, HSR is used to calculate degree threshold statistics. Even after validation, a random double check with ASR or HSR resources provides a continuous check on the validity of the selected recognition method. The frequency of such checks is based in one embodiment on the statistical deviation between correct and incorrect ASR recognition. As a specific example, consider a situation where the average confidence of correct recognition is 56 and the average confidence of wrong recognition is 36. If the standard deviation is small (e.g. 8), this would suggest that there is little practical confusion between correct and incorrect recognition, so double checking is used less frequently do not have to. However, if the standard deviation is larger (eg, 12), more frequent double checks will be required to fine tune the grammar confidence threshold.

  Over time, the statistics can suggest to the ASR proxy 1102 to change its initial behavior. For example, if very good success is statistically suggested, this can suggest a change from a double check of two ASRs to a check of only one ASR. Alternatively, if success is poor, it can be suggested to stop trying to train or tune for particularly difficult grammars and use only HSR instead.

  Both ASR initial training and subsequent pacing share common characteristics, and these may be implemented as well. However, in many cases, training involves more subtle issues, larger vocabulary and statistical language models than initial pacing, and thus techniques that work well in pacing may not be optimal for training. Training may require a much larger sample size, using more HSR, and relying on out-of-grammar ASR resources.

  Particularly complex grammars may require consistent double checking by two ASRs (from different vendors) with different recognition models, and different results are judged by the HSR. Relying on multiple HSRs (eg, two HSRs and a third HSR that serves to resolve the difference) can in some cases be even more beneficial (eg, the contents of which are described herein). (See Patent Document 5, which is incorporated by reference as if fully described). The ASR proxy 1102 can be configured to address any of these possibilities via the software application 1101.

  Turning to FIG. 12, in one embodiment, the recognition decision engine 1201 operates as follows, depending on historical statistics (eg, for speakers, sessions, grammars and applications) and other factors: And based on various configuration settings, how to process the utterance is determined. In the example shown in FIG. 12, as a first step, the recognition decision engine 1201 can instruct the ASR not to be used until the ASR is trained or tuned. A check 1202 is performed to determine this. If so, a check 1207 is performed to determine if such pacing / training has already been completed. If not so indicated, a quick check 1203 is performed to determine if the grammar is simple enough (eg, the grammar has only a few endings) that does not require training. If the grammar is not simple, the process returns to check 1207 again. If the grammar is simple enough, processing moves to check 1204. The check 1207 described above includes stored statistics on ASR success for this grammar and whether the ASR has been tuned / trained for this grammar previously (whether in the same application 1101 or similar goals and corresponding trust). In other applications that may have a degree threshold). If these statistics indicate that they are fully trained / tuned, then check 1207 passes the processing to check 1204. Otherwise, processing proceeds to HSR processing 1210.

  Check 1204 uses confidence statistics stored in database 1105 and a threshold at which ASR can understand a particular grammar and a second statistic on the confidence in progress of recognizing the speaker in the session. . For simple grammars that are not pacing or trained, the ongoing statistics on how well the ASR is performing the recognition task are compared to the expected recognition confidence threshold provided by the application or the threshold calculated by the proxy Is done. In cases where initial recognition is being implemented, the threshold may be set to be automatically considered as not being met, so that the threshold can be initially calculated by the proxy, forcing it to be recognized by the HSR. To. In some embodiments, the threshold is augmented with historical information about the current grammar. Additionally, if the ASR speaker recognition capability suggests a confidence level higher than the threshold, then ASR processing will be used and processing moves to check 1205. Otherwise, the HSR process 1210 is used. For example, the threshold can be set as the number of times that ASR recognition is less than confidence (or adjusted confidence, eg, a high-value speaker). Depending on the application, this number is set low so that if there is even one ASR recognition with less than confidence, subsequent recognition is performed by the HSR.

  A check 1205 determines whether the use of double checking is required for recognition by the software application 1101 or another component (eg, a requirement for training or validation). If the use of double checking is not required, processing moves to step 1206 where a single ASR is used for recognition.

  If double checking is required, processing moves to check 1208, where double checking can be performed by more than one ASR (eg, trained ASR and other forms of acceptable ASR Determine if there are more than two available). If so, the process moves to step 1209 where recognition is performed by such multiple ASRs. If this is not possible, for example if the ASR is not suitable for recognition or if an ASR validation will be performed, then the process moves to steps 1210 and 1211, so that recognition is performed by both ASR and HSR resources. To be implemented.

  When the ASR or HSR completes recognition, statistics regarding recognition are stored in the statistics database 1105.

  As described above with respect to FIG. 11, ASR proxy 1102 also communicates with result determination engine 1107. The purpose of such an engine is to evaluate the results of the recognition process with ASR / HSR resources. Referring to FIG. 13, an exemplary result determination engine 1301 is shown and this operation will be described as follows. The result determination engine 1301 examines the results of recognition from one or more ASR / HSR resources and determines an appropriate next step. Initially, a check 1302 is performed to determine if the reported confidence level meets the recognition threshold set by the software application 1101 or calculated by the ASR proxy 1102. If so, the validation statistics are updated to reflect the recognition success (1303) and the operation of the result determination engine 1301 is complete. If not, further processing is required and a “filler” prompt is provided to the user (1304). For example, a caller may be told, “Please wait because you are still working.” The particular message provided to the caller may be such a default message, or it may be a more specific message provided and determined by the software application 1101 via some form of reference. .

  The process then moves to recognition 1305 with one or more HSR resources and moves to check 1306 to determine whether the HSR recognition matches the ASR recognition. If they match, the statistics are updated again (1303), but this time the recognition also requires HSR, so the statistics are prorated. In one embodiment, the proportional distribution is a third deduction from the score that would have been provided if the confidence threshold was cleared.

  If the recognition results between the HSR and ASR are different, a check 1308 is performed to determine if a dual HSR has been used. If used, the results from the double HSR are used (1307) and the statistics that track successful ASR recognition are decremented. If not, an additional filler message is played (1309) and additional HSR recognition is attempted (1310). If the HSR results do not match, a third attempt to use HSR is performed (this is done in one embodiment but not in other embodiments). If there is no agreement between the HSRs, a “no match” result is returned. This indicates that no recognizer understands the speaker (thus not showing any deflection to the ASR). Depending on the current load conditions, it may not be practical to implement the second or third HSR, in which case a single HSR result is used, but again there is no deflection to ASR. In such an embodiment, a similar process is used for the operation of the result determination engine, which is also discussed with respect to FIGS. If it is determined that the ASR matches the HSR recognition, the process is completed. Otherwise, processing returns to 1307 to apply HSR recognition and update statistics as described above.

  Note that in one implementation, the ASR need not be selected from the grammar as a result of recognition. The ASR can also return a result of “no match”, “no input” or “noise”, in which case further HSR processing is used as described above, again depending on the criteria established by the application. The

  Referring to FIG. 14, one embodiment of a result determination engine 1401 is shown and this operation is described as follows. The result determination engine 1401 examines the results of recognition from two or more ASR resources and determines an appropriate next step. Initially, a check 1402 is performed to determine if the results from the two ASR resources match. If there is a match, a check 1403 is performed to determine if the reliability is higher than an appropriate threshold. In one embodiment, each ASR has its own threshold, and if any ASR is higher than the confidence threshold, the confidence is considered sufficient. In that case, validation statistics are incremented for recognizers that are higher than the threshold (1404) (if there is an ASR that matches but is less than the threshold, the statistics are not incremented or decremented) and the process is complete.

  If the results do not match, or if the confidence level is not high enough, the filler is played to the caller (1405) and at 1406 the HSR resource is invoked to perform recognition. A check 1407 is then performed to determine if at least one of the ASR results matches the HSR result. If not, a check 1408 is performed to determine if the HSR was a double check HSR. If not, the filler is regenerated again (1409) and additional HSR recognition 1410 is performed. If the HSR matches the ASR, or if the HSR is a double check, or if the second HSR 1410 is implemented, the process moves to use the matching HSR result (1411). This includes decrementing statistics from non-matching ASRs, and decrementing statistics from any ASRs that match but below the threshold (but proportionally distributed, in one embodiment 3 minutes) 1). Next, if there are matching ASR validation statistics above the threshold, they are incremented (1412) and the process is complete.

  FIG. 15 illustrates the processing of the result determination engine when one or more ASR resources are used with one or more HSR resources. The operation of a particular result determination engine 1501 in this case begins by checking (1502) whether all the results match. If there is a match, a check 1503 is performed as described above to determine if the confidence for each ASR is higher than that threshold, and if so, the validation statistics are incremented ( 1504). As described above, if there is an ASR that matches but is less than the threshold, they are incremented by subtraction of proportional distribution. The process is then complete.

  If the results do not match, a check 1505 is made to determine if a double check HSR has been used, and if not, the filler is regenerated (1506) and a second HSR recognition 1507 is performed. . The HSR result is then used (1508) and the statistics for the unmatched ASR are decremented, as described above, assuming the HSR results match. If the HSR results do not match, processing continues as described above with respect to FIG. If there are matching ASRs, the validation statistics are incremented for them either completely or in a proportional manner as described above (1509). The process is then complete.

  Referring to FIG. 16, the processing of one embodiment of a result determination engine 1601 is shown when only HSR resources are used. The first check 1602 determines whether a double check HSR has been used (assuming a double check HSR was required by the calling application). If double checking was not used, the filler is regenerated (1603) and a second HSR recognition 1604 is performed to ensure that the recognition is correct.

  A check 1605 is then performed to determine if the HSR results match. If not, processing is complete and in one embodiment, further processing outside the scope of this process is required, such as a third HSR awareness (not shown), to meet the calling application's requirements. Will be. In such a case, if there is no convergence after the third recognition, a “no match” situation is declared, indicating that the recognition attempt has failed. If there is convergence, then at least two matching HSR results are used.

  If the two HSR results in check 1605 match, the process is complete, for example, the recognized utterance can be added to a group for pacing / training as described above. Interpreting responses to prompts can be viewed as two types of text analysis: information extraction and sense classification. Information extraction is the identification, extraction and normalization of specific pieces of information that are essential to fill a slot in a business form, such as customer ID, phone number, date and time, address, product type, and problem. Sense classification relates to identifying two additional types of information: meaning (intent) and response quality. The meaning (intent) has to do with what kind of form needs to be filled in (billing, booking scheduling, complaints, etc.). Response quality has something to do with the response itself (indistinct, noisy, Spanish rather than English, a desire to talk to a live agent, etc.).

  Referring to FIG. 17, the methods and systems described above can be implemented to maximize the human experience. As shown in the results of predictive optimization 1730 and media acceleration 1734, the overall recognition gap time to respond back to the application from the ASR proxy can be reduced to 1.25 seconds in one example. Considering the specific graph of FIG. 17 in detail, 1710 represents a typical cognitive experience that is not optimized. The media (speech) to be recognized is 3.75 seconds long (1750). In this case, the ASR proxy streams the media in real time, but it usually takes more than a fraction of a second from the end of the media stream to complete automatic recognition (1712). The result determination engine of the ASR proxy determines that an HSR (1860 in FIG. 18 to be described later) is required, but the media (utterance) needs to be processed from the beginning, which adds about 4 more seconds ( 1714), the gap as seen by the user is at least 4.25 seconds. This gap can be filled by the application 1810 in a manner often referred to in the industry as the “filler prompt”, thereby ensuring that the user is aware that the system is still working on the problem. Certainly this filler prompt will not achieve the goal of creating a more human interaction with the caller. Moving to graph 1715, the system can improve by accelerating the media, for example, by 1 second, thereby reducing human assisted understanding to 3 seconds (1719) and reducing the media pause to the recognition gap by 3 Can be shortened to 25 seconds. This is a considerable improvement. As shown at 1730, automatic recognition provides prediction of results in a shorter time using a partial recognition predictor. As shown at 1732, it takes only 2 seconds to determine that the recognition has failed, after which the ASR proxy streams the media for human assistance and accelerates the media. As a result, the overall recognition gap from the end of the media to the success of human assistance has been greatly reduced from 4.25 seconds to 1.25 seconds. As a result, the recognition gap of the ASR proxy is reduced to a range that more closely matches human-like dialogue.

  FIG. 18 illustrates the main system components of the ASR proxy, further illustrating some elements of FIG. 11 and further illustrating the ASR proxy. Although not part of the illustration of FIG. 11, there is a user state management store 1813 within this disclosure, which is specifically shown in FIG. 18 for clarity. User state management 1813 has information about the user (eg, user identification, preferred communication channel and owned equipment). Information important to the user's processing, such as recognition success (automation rather than human assistance), is stored in the statistics store 1830 for future use. The system maintains information about the status of each interaction. This information, on the one hand, consists of information on the availability of intent analysis, and on the other hand extracted from the recognition requests presented, the responses to these requests, the meaning (intentions) of these responses, and these responses. Information about the specific content made and the sequence of what actions the proxy will perform next.

  Based on the specific prompt, the meaning (intent) of the response to this prompt, and the specific information extracted from this response, the proxy processing system determines its action (ie what additional information is requested from the user), And what action to perform next using that information). The system status subsystem 1815 keeps track of HSR capacity or in some embodiments system load and how this affects the use of automatic and human recognition. The remaining elements of FIG. 18 are as described above with respect to the other figures, where the ASR / NLU 1850 is specifically shown with multiple circles to represent multiple available ASR / NLU instances.

  FIG. 19 illustrates the operation of the decision engine, optionally using ASR or DTMF functionality based on system status evaluation (if appropriate based on application). Although these operations are described herein as being handled by a recognition decision engine 1980 and a result decision engine 1990, those skilled in the art will recognize that such engines can be implemented using various memory and processor architectures. Will do. If there are no recognition statistics (1900), no automation is used other than informing the application to automate using the DTMF approach if there is not enough HSR capacity. The DTMF will be made available to the application, and the application will allow DTMF variants to be made available by business rules. In this embodiment, DTMF will be used based on the second recognition request from the application. In various embodiments, the application can choose to ignore subsequent availability and attempt subsequent recognition, or use DTMF for certain recognition requests, with the most difficult items in the HSR. You can choose to leave it to me. For example, data collection of telephone numbers can be easily performed by DTMF, but email addresses are more appropriately handled by HSR.

  The application, in one embodiment, notifies (1900R) and provides some form of human interaction depending on system status 1815 and statistics availability 1830. That is, these dialogues are (1) human-like dialogue 1925 that uses only human-assisted understanding, (2) human-like dialogue 1930 that uses a combination of automation and human assistance in high quality, and (3) different application quality Human interaction 1930, using a combination of automation and human assistance with variable quality depending on load factors, without needing to be able to respond to (4) the application increases validation acceleration to lower automation reliability, A human-like dialogue 1930 or 1940 that uses a combination of automation 1950 and human assistance 1960 in variable quality depending on the load factor, and (5) a dialogue 1940 that is not intended to be human-like, such as a DTMF dialog. In this way, the system presents various types of prompts in response to ASR proxy capabilities and system load. For example, in the case of (5), "Please press 1 for sales. Please press 2 for ...", but the same question is "What is your business?" In other words, this illustrates the case of (1).

  FIG. 20 shows the flow of ASR and HSR processing using statistics, including logic and components as mainly described in FIGS. 18 and 19, and using statistics. Note that FIGS. 21 and 22 are optional concurrent flows. FIG. 20 uses a recognition decision engine 2000 and a results decision engine 2020 that use statistics 1820 with system status information 1815 and optionally accelerates the recognition media (voice, video) (2010), Reduce failover time between automation and human assistance.

  FIG. 21 illustrates an optional parallel flow where timer statistics are combined with recognition and system status 1815 in recognition decision engine 2100. If the media is longer than can normally be recognized well (adjustable according to system load), a timer event is fired and recognition passes to human assistance 1860. The result determination engine 2150 operates as described above.

  FIG. 22 shows an optional parallel flow. In this case, the recognition determination engine 2200 performs recognition prediction for a part of the media in accordance with the system load prediction reliability adjustment, and the recognition is not sufficiently successful. Recognition shifts to human assistance 1860. The result determination engine 2250 still operates as described above.

  FIG. 23 shows a pacing subsystem for collecting data around media and media meaning to build an optimal grammar and classifier for semantic extraction, and also to build an optimal recognition predictor / Shows the flow. FIG. 23 describes use cases where ASR 2310 and classifier automation are applied to a selected subset 2320 of prompts in an application. Although the set of application prompts falls into various categories, some of these are obvious candidates for automation and some are difficult to automate. For example, yes / no prompts and limited option prompts usually have a very limited repertoire of user utterances and very few intent labels. Evaluating and modeling these types of prompts requires a relatively small amount of data, whether for an ASR grammar, a statistical language model, or a machine learning classifier 2340. On the other hand, free answer prompts allow the user to generate a much less constrained utterance set, but free answer prompts are more difficult. These can be augmented by both general and domain specific knowledge bases 2330. These types of prompts require a relatively large amount of data. Even when there is a large amount of data, the diversity may still be too great to produce a reliable model for all types of utterances or intention labels. In other words, in these cases, automation proceeds by establishing linguistic, categorical and statistical characteristics of prompts and driving prompt selection and formulation based on these characteristics. This involves a set of correlated tasks as follows.

-Classify utterances into various categories based on their characteristics.
Identify candidate prompts for a given application with characteristics suitable for ASR and classifier automation.
-Determine predictors for early recognition success or failure.
For each prompt, create, tune and store an acoustic and language model for the ASR for the utterances generated by this prompt and a classifier model for the target intent of this prompt.
Determine selection criteria for deciding when to use or trade off ASR and classifier automation and human intention analysis.

  FIG. 24 is an example of how a timer statistic can be calculated, as well as a very simple predictor, using an example of dividing a North American telephone number into multiple recognition components. Elements 2401 to 2403 represent statistics gathered upon successful recognition of a particular question (prompt). Element 2401 represents an utterance of a type whose length is 2 seconds or less. This length represents 15% of all utterances that have statistics in this example. ASR was determined to be 90% successful for utterances of less than 2 seconds. Element 2402 represents a type of utterance that is longer than 2 seconds and not longer than 3 seconds, with ASR recognition success of 75% and 25% of utterances falling into this group. Element 2403 represents a type of utterance that is longer than 3 seconds and not longer than 4 seconds. This is an example of a use case that may be affected by the system status. If there is enough HSR resources, a timer can be established to interrupt recognition in 3 seconds (2402) and ASR can be used to successfully recognize 32.3% of utterances. Or, if the system load increases, the timer can be adjusted to 4 seconds (2403) to recognize 44.3%. Note that under very high load, the ASR proxy can decide not to use a timer. However, this causes a longer waiting time for the speaker. However, as a result, 55.3% is well recognized.

  Element 2404 represents ASR recognition of a 3-digit area code. Element 2405 represents the ASR recognition of the 3-digit area code and in addition the recognition of the 3-digit switching office. Element 2406 represents ASR recognition of the entire North American telephone number. For example, if it takes about 8 seconds to speak a telephone number, each step 2404, 2405, and 2406 takes more time to process the utterance. The first step 2404 takes about 30% of the time (2.4 seconds), and step 2 takes 60% of the time (4.8 seconds), and any of the three recognition steps is reliable. If it shows less than the result, recognition shifts to human assistance. For example, if the area code is not recognized correctly, the use of HSR may occur within 2.4 seconds while the phone number is spoken, rather than failing only after the entire phone number is spoken. is there.

  In various embodiments and implementations, this response interpretation is done by intent analysts only (pure HSR), by automated ASR (pure automatic speech recognition and intention classification), or some combination of ASR and HSR Can be done by. Trade ASR automation to HSR without quality loss (or with controlled quality loss) by determining when ASR is producing reliable results using confidence in the results of ASR automation It is possible to turn it off. This means that the combination of these two approaches in a proxy processing system can achieve greater throughput than using only HSR and can better meet peak demand loads with a smaller team of intent analysts. means.

  The above subject matter has been described in particular detail with respect to various possible embodiments. Those skilled in the art will appreciate that the subject matter may be practiced in other embodiments. First, the specific naming of components, capitalization of terms, attributes, data structures, or any other programming or structural aspects are not required or significant, and the mechanisms that implement the subject matter or its features are different It may have a name, format or protocol. Furthermore, the system may be implemented through a combination of hardware and software as described, or may be implemented entirely in hardware elements. Also, the specific division of functionality between the various system components described herein is only an example and is not required. Functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.

  In some of the above, the subject features, process steps and instructions are presented in algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. Although these operations are described functionally or logically, these operations may be embodied in software, firmware, or hardware, and when embodied in software, are used by a real-time network operating system. It may be downloaded to exist on and operate from various platforms.

  Furthermore, it has also proven convenient at times to refer to these configurations of operations as modules or by functional names without loss of generality.

  Throughout this description, considerations utilizing terms such as “determine” will be made in computer system memory or registers, or other such information storage, unless otherwise specified or apparent from the above discussion. It should be understood that it refers to actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within a transmission, or display device.

  The subject matter also relates to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes, or selectively activated or reactivated by a computer program stored on a computer readable medium that can be accessed by a computer and executed by a computer processor. It may include a general purpose computer configured. Such a computer program may be stored in a non-transitory computer-readable storage medium, and the non-transitory computer-readable storage medium is not limited to the following, but is a floppy (registered trademark) disk, optical disk, or CD-ROM. Or any type of disk, including magneto-optical disks, ROM, RAM, EPROM, EEPROM, magnetic or optical card, ASIC or any type of medium suitable for storing electronic instructions, etc. Coupled to the computer system bus. Further, the computers referred to herein may comprise a single processor or may be an architecture that utilizes a multiple processor design for increased computing power.

  Also, the subject matter does not describe any particular programming language. That various programming languages may be used to implement the teachings of the subject matter described herein, and that any reference to a particular language is provided for subject matter availability and best mode. I want you to understand.

  The subject is well suited to a wide range of computer network systems that span many topologies. Within this field, large network configuration and management includes storage devices and computers that are communicatively coupled to disparate computers and storage devices via a network, such as the Internet.

  Finally, the language used herein was selected primarily for readability and teaching purposes, and may not have been selected to delineate or limit the subject matter. Please keep in mind. Accordingly, the disclosure herein is intended to illustrate rather than limit the scope of the subject matter.

Claims (20)

A computer-implemented system for processing an interaction, the interaction comprising an utterance that requires recognition before it can be used for further computer-execution processing, the system comprising:
An application configured to provide the utterance, wherein the utterance is received from a customer device over a computer network;
A recognition decision engine configured to receive the utterance for recognition, wherein the recognition decision engine is different from an automatic speech recognition (ASR) subsystem and the automatic speech recognition subsystem and the computer By the application to dynamically select one or more recognizers from a human speech recognition (HSR) subsystem that communicates over a computer network with a device that is remote from the execution system before recognizing the interaction . A recognition decision engine using the provided parameters;
A result determination engine coupled to the one or more recognizers and configured to provide recognition results.   The system further comprises a system status subsystem operatively connected to the recognition decision engine, the recognition decision engine having system load information from the system status subsystem as input for use in the dynamic selection. The system of claim 1.   The subset of the one or more recognizers is configured to provide a confidence metric to the recognition decision engine, the recognition decision engine using the confidence metric in the dynamic selection. 1 system.   The system of claim 3, wherein the reliability metric includes a threshold, and the threshold varies based on resource availability. The system of claim 1, wherein the recognition decision engine is configured to choose to select the automatic speech recognition subsystem for the human speech recognition subsystem based on a recognition cost factor. The system of claim 1, wherein the recognition decision engine is configured to choose to select the automatic speech recognition subsystem for the human speech recognition subsystem based on a human resource availability factor. .   The result determination engine is responsive to a result match between a first recognizer subsystem of the recognizer and a second recognizer subsystem of the recognizer. The system of claim 1, wherein the system is configured to update a confidence threshold associated with the first recognizer subsystem.   The recognition decision engine initially selects a first recognizer subsystem of the automatic speech recognition subsystem and is provided by the first recognizer subsystem of the automatic speech recognition subsystem. Responsive to the first result received, configured to make a subsequent selection of a second recognizer subsystem of the recognizers, the processing of the utterance being performed by the automatic speech recognition sub The system of claim 1, wherein the system is performed before being completed by the first recognizer subsystem of the system. A computer-implemented method executed by a computer system to process an interaction, wherein the interaction includes an utterance that requires recognition before it can be used for further computer-execution processing, the computer-implemented method comprising:
Receiving data representing an utterance from a computer application, the utterance being received from a customer device over a computer network;
Using the parameters provided by the application, an automatic speech recognizer (ASR) and a human speech recognition recognizer that communicates over a computer network with a device that is different from the automatic speech recognizer and remote from the computer system Dynamically selecting one or more recognizers from (HSR) before recognizing the interaction ;
Providing a recognition result in response to a result of processing by the one or more recognizers.   The computer-implemented method of claim 9, wherein the dynamically selecting is responsive to a system load metric.   The computer-implemented method of claim 9, wherein the dynamically selecting is responsive to a reliability metric.   The computer-implemented method of claim 11, wherein the reliability metric includes a threshold, and the threshold varies based on resource availability. The computer-implemented method of claim 9, wherein the dynamically selecting is to select the automatic speech recognizer for the human speech recognition recognizer based on a recognition cost factor. 10. The computer-implemented method of claim 9, wherein the dynamically selecting is to select the automatic speech recognizer for the human speech recognition recognizer based on a human resource availability factor. .   In response to a result match between a first recognizer of the recognizers and a second recognizer of the recognizers, update a confidence threshold associated with the first recognizer of the recognizers. The computer-implemented method of claim 9, further comprising:   First selecting a first recognizer of the automatic speech recognizers and in response to a first result provided by the first recognizer of the automatic speech recognizers, a second of the recognizers 10. The computer implemented method of claim 9, further comprising making a subsequent selection of a recognizer, wherein the subsequent selection is made before processing of the utterance is completed by the first of the automatic speech recognizers. Method. A non-transitory computer readable storage medium storing executable computer program code for processing an interaction, wherein the interaction includes an utterance that requires recognition before it can be used for further computer execution processing, When the computer program code is executed by the computer system,
Receiving data representing an utterance from a computer application, the utterance being received from a customer device over a computer network;
Using the parameters provided by the application, an automatic speech recognizer (ASR) and a human speech recognition recognizer that communicates over a computer network with a device that is different from the automatic speech recognizer and remote from the computer system Dynamically selecting one or more recognizers from (HSR) before recognizing the interaction ;
A non-transitory computer readable storage medium for causing a recognition result to be provided in response to a result of processing by the one or more recognizers.   The non-transitory computer readable storage medium of claim 17, wherein the dynamic selection is responsive to a system load metric.   The non-transitory computer readable storage medium of claim 17, wherein the dynamic selection is responsive to a reliability metric.   The non-transitory computer readable storage medium of claim 19, wherein the reliability metric includes a threshold, and the threshold varies based on resource availability.

Images