CN106409283A - Audio frequency-based man-machine mixed interaction system and method - Google Patents

Abstract

The invention discloses an audio-based human-computer hybrid interactive system. The speech recognition module is connected with the semantic recognition module and transmits text information corresponding to the speech. The exception processing module is connected with the speech recognition module and the semantic recognition module, and the speech recognition module transmits text information. To the exception processing module, the semantic recognition module transmits the semantic analysis result to the exception processing module; the exception processing module is connected with the speech synthesis module and transmits intervention information. The invention also discloses an audio-based human-computer hybrid interaction method. The voice recognition module converts voice information into text information and outputs it to the semantic recognition unit; the semantic recognition unit extracts the user's purpose and corresponding key information from the text information; The processing module judges whether there is an abnormality in the human-computer dialogue according to the text information of the speech recognition module and the semantic information of the semantic recognition module, and processes the message reply for the abnormality. The technical solution of the invention provides a unified man-machine dialogue experience.

Description

Audio-based human-computer hybrid interaction system and method

technical field

The present invention relates to the technical field of information processing, in particular to an audio-based human-computer hybrid interaction system and method.

Background technique

As shown in Figure 1, the current audio-based man-machine dialogue systems all use the machine’s reply as the final reply to present to the user. When the machine decision-making system cannot clarify the user’s intention, most of the dialogue systems choose to present a reply such as “please say it again”. In order to allow users to re-input, some of the man-machine dialogue systems have introduced a manual intervention method based on the call center.

At present, the abnormal handling of the existing human-machine dialogue is mainly realized through the call center. When the machine cannot process the user input audio or when the user clearly expresses the need for manual service, the call center is requested to intervene manually. At this time, a pair is established between the user and the operator. One call connection, the operator communicates directly with the user, learns the user's needs and issues corresponding instructions through the call platform.

The problems existing in the manual intervention method of the existing call center mainly include: low manual efficiency, the interventionist and the user need to establish one-to-one voice communication, and cannot serve other people during the time period waiting for user input; high cost, large-scale The call center needs a series of telecommunication equipment and corresponding service integration. At the same time, due to low efficiency, more interventionists are required to provide intervention services, which indirectly increases labor costs; it is greatly affected by the network environment: using network resources to directly transmit audio requires a stable network Connection, the fluctuation of the network environment will lead to the degradation of audio quality, which will affect the dialogue experience, and even interrupt the process of man-machine dialogue.

Therefore, those skilled in the art are devoting themselves to developing an audio-based human-computer hybrid interaction system and method, which combines manual intervention reply with machine reply, so as to unify the process of man-machine dialogue and improve user experience.

Contents of the invention

In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to improve the efficiency and user experience of man-machine dialogue in the customer service process.

To achieve the above object, the present invention provides an audio-based human-computer hybrid interaction system, including a speech recognition module, a speech synthesis module, a semantic recognition module and an exception processing module, wherein the speech recognition module is configured to communicate with the The semantic recognition module is connected and transmits text information corresponding to the voice, the exception processing module is configured to be connected to the speech recognition module and the semantic recognition module, and the speech recognition module is configured to transmit text information to the exception processing module, the semantic recognition module is configured to transmit semantic analysis results to the exception processing module; the exception processing module is configured to be connected to the speech synthesis module and transmit intervention information.

Further, the speech recognition module includes a signal processing and feature extraction unit, an acoustic model, a language model, and a decoder, wherein the signal processing and feature extraction unit is configured to be connected to the acoustic model and transmit acoustic feature information, The decoder is configured to be connected to the acoustic model and the language model and output a recognition result.

Further, the speech synthesis module includes a text analysis unit, a prosody control unit and a speech synthesis unit, wherein the text analysis unit is configured to receive text information and process the text information, and transmit the processing result to the A prosody control unit and the synthesized speech unit, the prosody control unit is configured to be connected to the synthesized speech unit, and transmit pitch, sound length, sound intensity, pause and intonation information, and the synthesized speech unit is configured to The analysis result of the received text analysis unit and the control parameters of the prosody control unit are synthesized into the output speech.

Further, the semantic recognition module includes a domain labeling unit, an intention judgment unit, and an information extraction unit, wherein the domain labeling unit is configured to be connected to the intention judgment unit and transmit domain information, and the intention judgment unit is configured In order to be connected with the information extraction unit and transmit user intention information, the information extraction unit outputs the result of semantic analysis.

Further, the abnormality processing module includes an abnormality detection unit, a database query unit and an interventionist unit, wherein the abnormality detection unit is configured to receive the output of the speech recognition module and the semantic recognition module, and decide whether to take Intervention measures, the database query unit is configured to receive the intervention signal of the abnormality detection unit, and receive the semantic information of the semantic recognition module, query and output an intervention message, and the interventionist unit is configured to use the interventionist to The intervention message output by the database query unit undergoes necessary optimization and modification, and finally outputs a reply message to the user.

The present invention also provides an audio-based human-computer hybrid interaction method, comprising the following steps:

Step 1, providing a speech recognition module, a speech synthesis module, a semantic recognition module and an exception handling module;

Step 2, the voice recognition module converts voice information into text information and outputs it to the semantic recognition unit;

Step 3, the semantic recognition unit extracts the user purpose and corresponding key information from the text information;

Step 4. The abnormality processing module judges whether there is an abnormality in the human-computer dialogue according to the text information of the speech recognition module and the semantic information of the semantic recognition module, and processes the message reply for the abnormality.

Further, in step 2, the following steps are specifically included:

Step 2.1, extracting features from the input audio stream for acoustic model processing, while reducing the impact of environmental noise, channel and speaker factors on the features;

Step 2.2: The decoder searches for a word string that can output the audio stream with the highest probability based on the acoustic model processing results of the acoustic model and the dictionary as the speech recognition result.

Further, in step 3, the following steps are specifically included:

Step 3.1, using iconic keywords in the text information to mark the field to which the current conversation belongs;

Step 3.2, judging the user's intention based on rules in the field;

Step 3.3, extract specific key information according to the field and the user's intention in combination with rules.

Further, in step 4, the following steps are specifically included:

Step 4.1, the abnormality detection unit judges whether the current man-machine dialogue is abnormal according to the text information of the speech recognition module and the semantic information of the semantic recognition module, and if it is abnormal, the interventionist unit takes over the man-machine dialogue;

Step 4.2, the database query unit queries the database according to the semantic information, and obtains intervention messages with a recommendation degree. If the recommendation degree of the intervention message is high, the intervention message is directly used for intervention. If the recommendation degree is low, the interventionist is requested. perform manual intervention;

Step 4.3. When the machine algorithm fails to find a highly recommended intervention message, the interventionist intervenes to select and modify the intervention message, and then sends the modified intervention message to the client.

Further, the key information includes dialogue fields and dialogue keywords, and the dialogue keywords include content keywords and emotion keywords.

Compared with prior art, technical effect of the present invention comprises:

1. Efficiency improvement: Make full use of the time that the interventionist waits for user input, so that the interventionist can provide intervention services to multiple users at the same time, improving the efficiency of intervention.

2. Cost reduction: There is no need to purchase a series of telecommunication equipment related to the call center, and the intervention platform can be built by using the existing computers and servers.

3. Rich work scenarios: Since the interventionist interface adopts the B/S (Browser/Server browser/server) structure, the interventionist can open the browser and log in to the corresponding website to perform intervention operations, and it is unnecessary to answer the phone at the workstation. Intervention services can be performed on mobile terminals such as PADs, smart phones, and personal notebooks.

4. Low network requirements: The amount of data transmitted by text is small, thereby reducing the requirements for the network. At the same time, the voice heard by the user is synthesized locally and is not affected by the network situation.

5. Unified man-machine dialogue experience: For users, the interventionist is transparent, and the user experience is like talking with a fully intelligent "machine", which can seamlessly connect with the current man-machine dialogue.

The idea, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings, so as to fully understand the purpose, features and effects of the present invention.

Description of drawings

FIG. 1 is a schematic diagram of an intervention mode of an existing traditional call center;

Fig. 2 is a schematic diagram of a system module of the present invention;

Fig. 3 is a schematic flow chart of the system of a preferred embodiment of the present invention;

Fig. 4 is a schematic diagram of a character dialog flow in a preferred embodiment of the present invention.

detailed description

The present invention is achieved through the following technical solutions:

As shown in Fig. 2, the present invention relates to a kind of audio-based man-machine dialogue exception processing system, comprising: a speech recognition module, a speech synthesis module, a semantic recognition module and an exception processing module, wherein: the speech recognition module is connected with the semantic recognition module and The text information corresponding to the voice is transmitted, the speech recognition module and the semantic recognition module are connected to the exception processing module, and respectively transmit text information and semantic analysis results, and the exception processing module is connected to the speech synthesis module and transmits intervention information.

Described speech recognition module comprises: signal processing and feature extraction unit, acoustic model, language model and decoder, wherein: signal processing and feature extraction unit are connected with acoustic model and transmit acoustic characteristic information, decoder and acoustic model and language model Connected to output recognition results to the outside world.

The speech synthesis module includes: a text analysis unit, a prosody control unit and a synthetic speech unit, wherein: the text analysis unit receives the text information and processes it, and transmits the processing result to the prosody control unit and the synthesized speech unit, and the prosody control unit The unit is connected to the synthesized speech unit, and transmits information such as pitch, duration, sound intensity, pause, and intonation of the target. The synthesized speech unit receives the analysis results of the text analysis unit and the control parameters of the prosody control unit, and outputs the synthesized speech to the outside world. .

The semantic recognition module includes: a field labeling unit, an intention judgment unit, and an information extraction unit, wherein: the field labeling unit is connected with the intention judgment unit and transmits domain information, and the intention judgment unit is connected with the information extraction unit and transmits user intention information, information The unit is connected to the outside world and transmits information for semantic analysis.

The abnormal processing module includes: an abnormal detection unit, a database query unit, and an intervention division unit, wherein: the abnormal detection unit receives the output of the speech recognition module and the semantic recognition module, and decides whether to take intervention measures, and the database query unit receives the abnormal detection The intervention signal of the unit, and receive the semantic information of the semantic recognition module, query and output the intervention message, the interventionist unit uses the interventionist to select and modify the intervention message output by the database query unit, and finally outputs the user reply message.

The present invention relates to the man-machine dialog abnormality processing method of the above system, which specifically includes the following steps:

Step 1. Provide a speech recognition module, a speech synthesis module, a semantic recognition module and an exception handling module.

Step 2, the voice recognition module converts the voice information into text information and outputs it to the semantic recognition unit, and the specific steps include:

2.1 The front end processes the audio stream and extracts features from the input signal for processing by the acoustic model. At the same time, the influence of environmental noise, channel, speaker and other factors on the features is reduced as much as possible.

2.2 The decoder looks for the word string that can output the signal with the greatest probability according to the acoustic, linguistic model and dictionary for the input signal, as the speech recognition result.

Step 3, the semantic recognition unit extracts the user's purpose and corresponding key information from the text information, and the specific steps include:

3.1 Use iconic keywords in the text information to mark the field to which the current conversation belongs.

3.2 Judge the user's intention based on the rules in the specific field.

3.3 According to the field and user intentions, combined with rules, such as preset templates, to extract specific key information.

Step 4, the exception processing module judges whether the man-machine dialogue is currently abnormal according to the text information of the speech recognition module and the semantic information of the semantic recognition module, and performs abnormal processing and message reply. The specific steps include:

4.1 The abnormality detection unit judges whether the current man-machine dialogue is abnormal according to the text information of the speech recognition module and the semantic information of the semantic recognition module. If there is no exception, it will be handled by the local client, and if there is an exception, the intervening server will take over the man-machine dialogue.

4.2 The database query unit searches the database according to the semantic information to obtain the recommended intervention message. If the recommendation degree of the intervention message is high, the intervention message is directly used for intervention. If the recommendation degree is low, the interventionist is requested to intervene manually.

4.3 When the machine algorithm fails to find a highly recommended intervention message, the interventionist intervenes to select and modify the intervention message, and then sends the modified intervention message to the client.

In the process of man-machine dialogue exception handling, after the user's voice input passes through the machine's voice recognition and semantic analysis, the voice recognition results and semantic analysis results will be sent to the interventionist in the form of text, and the interventionist will receive the message After that, you can choose to send a dialog message or send a command message. Dialogue messages are transmitted to the machine in the form of text, and then the voice is synthesized by a speech synthesis system (TTS) and played to the user, while the command message is directly executed by the machine.

This embodiment includes the following steps, as shown in Figure 3 and Figure 4, that is, the three steps of user input->intervention message generation->client push intervention message are introduced respectively for technical solutions:

1) User input

During the user's voice input process, the voice recognition system used converts the user's voice input audio into text, and at the same time performs semantic analysis on the sentence (the result of semantic analysis includes the user's current dialogue field, key information of the user's requested service, etc. ), and finally transmit the text and semantic analysis results to the exception handling module in text form through the POST method of the HTTP protocol.

2) Intervention message generation

The exception processing module queries the database according to the text information of the speech recognition and the semantic slot of the semantic recognition under abnormal circumstances, and obtains alternative intervention messages. If the recommendation degree of the intervention message is high, the intervention message is directly used for intervention, and if the recommendation degree is low, the interventionist is requested to intervene manually. Interventionists can see the auxiliary data provided by the exception handling module on the interface, such as the recognition results input by users and the results of semantic analysis, etc. Combining these information, interventionists can more accurately and quickly screen and modify candidate intervention messages. Intervention messages are divided into dialogue messages and command messages, both of which are transmitted in the form of text using a unified Websocket protocol, the difference lies in the difference in the transmission content and the processing method of the machine.

3) The client pushes the intervention message

After the client computer receives the intervention message, it immediately returns the confirmation message of "the message has been received" to the interventionist, and caches the intervention message in the message queue. The client will monitor the current state of the man-machine dialogue and try to take out the message from the message queue and push it to the user under certain conditions. The specific push timing includes: 1. When the intervention message arrives, 2. When the broadcast of the voice message synthesized by TTS is completed; The conditions to be met are 1. the message queue is not empty, and 2. the audio player of the client is currently idle. If the intervention message is successfully pushed, it will return the interventionist's confirmation message "The intervention message has been pushed".

For example:

1. User A issues a voice command "I am going to a fun place".

2. The voice recognition module converts voice input into text.

3. After processing by the semantic analysis module, it is obtained that the user's intention is "navigation", and the label of the navigation destination is "fun".

4. The abnormal detection unit in the abnormal processing module receives the service request from user A, including the complete voice recognition result "I am going to a fun place", and the semantic analysis results "navigation" and "fun", and simultaneously detects There is an exception in the current dialog state.

5. The database query unit in the exception handling module performs database query according to "navigation" and "fun", and gets some alternative messages such as "do you want to go to Suzhou's fun snacks?", "find 5 for you and fun" Relevant locations", the recommendation of these two messages is relatively low, so manual intervention from the interventionist unit is requested. The interventionist selects and modifies the intervention message by using the database query results obtained by the exception handling module, the semantic analysis results, and the text results of speech recognition, and changes the intervention message to "What kind of entertainment do you want?", and sends the message to the user. text message.

6. After receiving the intervention message, the client computer stores it in the message queue, sends a feedback of "the message has been received" to the exception handling module, and tries to push it.

7. After the conditions are met, the speech synthesis system synthesizes and broadcasts the intervention message. The user hears the audio "What kind of entertainment do you want?", and the client computer sends a "message has been pushed" feedback to the exception handling module.

8. The customer makes further voice input "I'm going to sing"

9. ASR system converts speech input into text

10. Semantic analysis shows that the user's intention is "navigation", and the navigation target is "KTV"

11. The anomaly detection unit obtains the specific service requirements of user A, including the complete voice recognition result "I'm going to sing", and the semantic analysis results "navigation" and "KTV".

12. The database query unit searches the database according to "navigation", "KTV" and user related information, and obtains an alternative intervention message "recommend xxx for you, do you want to go?", and because the recommendation is very high, it is bypassed Intervention division unit, send a text message directly to the client computer "recommend xxx for you, would you like to go?"

13. The user confirms to go to

14. The user of the exception handling system pushes an intervention message of the command type, including the command type "navigation" and the POI information of the destination.

15. The client takes out the message of the command type "navigation" and the corresponding POI information from the message queue, and performs the navigation operation. The client sends a feedback of "the message has been pushed" to the exception handling module, and the interaction ends.

The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make many modifications and changes according to the concept of the present invention without creative efforts. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art shall be within the scope of protection defined by the claims.

Claims (10)

1. a kind of man-machine mixing interactive system based on audio frequency is it is characterised in that include sound identification module, phonetic synthesis mould Block, semantics recognition module and exception processing module, wherein, described sound identification module is configured to and described semantics recognition mould Block is connected and transmits the corresponding Word message of voice, and described exception processing module is configured to and described sound identification module and institute Predicate justice identification module is connected, and described sound identification module is configured to transmit Word message to described exception processing module, institute Predicate justice identification module is configured to transmit semantic analysis result to described exception processing module；Described exception processing module is joined It is set to and be connected with described voice synthetic module and transmit intervention information.

2. the man-machine mixing interactive system based on audio frequency as claimed in claim 1 is it is characterised in that described sound identification module Including signal processing and feature extraction unit, acoustic model, language model and decoder, wherein, described signal processing and spy Levy extraction unit to be configured to be connected and transmit acoustic featuress information with described acoustic model, described decoder is configured to and institute State acoustic model to be connected with described language model and export recognition result.

3. the man-machine mixing interactive system based on audio frequency as claimed in claim 1 is it is characterised in that described voice synthetic module Including text analysis unit, prosodic control unit and synthesis voice unit, wherein, described text analysis unit is configured to connect Receive text message and described text message is processed, result is transferred to described prosodic control unit and described synthesis Voice unit, described prosodic control unit be configured to described synthesis voice unit be connected, and transmit pitch, the duration of a sound, loudness of a sound, Pause and prosody information, described synthesis voice unit be configured to by described receive text analysis unit analysis result with described The voice of the control parameter synthesis output of prosodic control unit.

4. the man-machine mixing interactive system based on audio frequency as claimed in claim 1 is it is characterised in that described semantics recognition module Including field mark unit, intention determination unit, information extraction unit, wherein, described field mark unit is configured to and institute State intention determination unit to be connected and transmission field information, described intention determination unit is configured to and described information extraction unit phase Connect and transmit user intent information, described information extraction unit exports the result of semantic analysis.

5. the man-machine mixing interactive system based on audio frequency as claimed in claim 1 is it is characterised in that described exception processing module Including abnormality detecting unit, data base querying unit and intervention Shi Danyuan, wherein, described abnormality detecting unit is configured to connect Receive described sound identification module and the output of described semantics recognition module, and decide whether to take intervening measure, described data base Query unit is configured to receive the intervention signal of described abnormality detecting unit, and receives the semantic letter of described semantics recognition module Breath, inquires about and exports intervention message, and it is defeated to described data base querying unit that described intervention Shi Danyuan is configured to, with intervention teacher The described intervention message going out carry out necessary preferentially and modification, final output replying message to user.

6. a kind of man-machine mixing exchange method based on audio frequency is it is characterised in that comprise the following steps：

Step 1, offer sound identification module, voice synthetic module, semantics recognition module and exception processing module；

Voice messaging is converted to Word message and exports to described semantics recognition unit by step 2, described sound identification module；

Step 3, described semantics recognition unit extract customer objective and corresponding key message from Word message；

Step 4, described exception processing module are according to the Word message of described sound identification module and described semantics recognition module Semantic information judge whether human computer conversation reply that is abnormal and being directed to abnormality processing message currently.

7. the man-machine mixing exchange method based on audio frequency as claimed in claim 6 it is characterised in that in step 2, specifically wraps Include following steps：

Step 2.1, extract feature from the audio stream of input and supply acoustic model to process, reduce environment noise, channel and say simultaneously The impact that words people's factor causes to described feature；

, according to acoustics, linguistic model and dictionary, the result to described acoustic model, searching can for step 2.2, decoder Export the word string of described audio stream with maximum of probability, as the recognition result of voice.

8. the man-machine mixing exchange method based on audio frequency as claimed in claim 6 it is characterised in that in step 3, specifically wraps Include following steps：

Step 3.1, using the significant field belonging to keyword tag current session in Word message；

Step 3.2, rule-based in described field user view is judged；

Step 3.3, according to described field and described user view, binding rule, specific key message is extracted.

9. the man-machine mixing exchange method based on audio frequency as claimed in claim 6 it is characterised in that in step 4, specifically wraps Include following steps：

Step 4.1, abnormality detecting unit are according to the Word message of described sound identification module and described semantics recognition module Semantic information judges whether current human computer conversation exception, if abnormal, takes over human computer conversation by intervening Shi Danyuan；

Step 4.2, data base querying unit carry out the inquiry of data base according to semantic information, and the intervention obtaining having recommendation degree disappears Breath, if the recommendation degree of intervention message is higher, is directly intervened using this intervention message, if recommendation degree is relatively low, please Seek intervention Shi Jinhang manpower intervention；

Step 4.3, when machine algorithm cannot find the intervention message of high recommendations degree, intervene teacher and intervene the choosing carrying out intervention message Select and change, subsequently send amended intervention message to client.

10. the man-machine mixing exchange method based on audio frequency as described in claim 6 or 8 is it is characterised in that described key message Including dialogue field, dialogue key word, described dialogue key word includes content keyword and emotion key word.