CN106409283B - Man-machine mixed interaction system and method based on audio - Google Patents

Abstract

The invention discloses a human-computer hybrid interaction system based on audio. A speech recognition module is connected with a semantic recognition module and transmits text information corresponding to the speech. An exception processing module is connected with the speech recognition module and the semantic recognition module, and the speech recognition module transmits the text information. To the exception processing module, the semantic recognition module transmits the semantic analysis result to the exception processing module; the exception processing module is connected with the speech synthesis module and transmits the intervention information. The invention also discloses a human-computer hybrid interaction method based on audio. The speech recognition module converts the speech information into text information and outputs it to the semantic recognition unit; the semantic recognition unit extracts the user purpose and corresponding key information from the text information; abnormality The processing module judges whether the current human-machine dialogue is abnormal according to the text information of the speech recognition module and the semantic information of the semantic recognition module, and processes the reply to the message according to the abnormality. The technical solution of the present invention provides a unified human-machine dialogue experience.

Description

Audio-based human-computer hybrid interaction system and method

technical field

The present invention relates to the technical field of information processing, and in particular, to an audio-based human-computer hybrid interaction system and method.

Background technique

As shown in Figure 1, the current audio-based human-machine dialogue systems all use the machine reply as the final reply to present to the user. When the machine decision-making system cannot clarify the user's intention, most dialogue systems choose to present a reply such as "please say it again". In order to allow users to re-enter, some of the man-machine dialogue systems have introduced manual intervention methods based on call centers.

At present, the existing abnormal handling of human-machine dialogue is mainly implemented in the form of a call center. When the machine cannot process the user's input audio or when the user clearly indicates that manual service is required, the manual call center is requested to intervene. At this time, the user and the operator establish a pair of Once the call is connected, the operator communicates directly with the user, learns the user's needs and issues corresponding instructions through the call platform.

The main problems of the existing manual intervention method of the call center are: low labor efficiency, the interventionist and the user need to establish one-to-one voice communication, and cannot serve other people during the time period waiting for the user's input; high cost, large-scale The call center requires a series of telecommunication equipment and corresponding service integration. At the same time, due to low efficiency, more interventionists are required to provide intervention services, which indirectly increases labor costs; it is greatly affected by the network environment: using network resources to directly transmit audio requires a stable network Connection, fluctuations in the network environment can cause audio quality to degrade, affecting the conversation experience, and even interrupting the human-machine conversation flow.

Therefore, those skilled in the art are devoted to developing an audio-based human-machine hybrid interaction system and method, which combines human intervention response with machine response, so as to unify the process of human-machine dialogue and improve user experience.

SUMMARY OF THE INVENTION

In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to improve the efficiency of human-machine dialogue and user experience in the customer service process.

In order to achieve the above object, the present invention provides an audio-based human-computer hybrid interaction system, including a speech recognition module, a speech synthesis module, a semantic recognition module and an exception processing module, wherein the speech recognition module is configured to The semantic recognition module is connected to and transmits text information corresponding to the voice, the exception processing module is configured to be connected to the voice recognition module and the semantic recognition module, and the voice recognition module is configured to transmit text information to the exception processing module, the semantic recognition module is configured to transmit the semantic parsing result to the exception processing module; the exception processing module is configured to be connected with the speech synthesis module and transmit intervention information.

Further, the speech recognition module includes a signal processing and feature extraction unit, an acoustic model, a language model and a decoder, wherein the signal processing and feature extraction unit is configured to be connected with the acoustic model and transmit acoustic feature information, The decoder is configured to connect with the acoustic model and the language model and output a recognition result.

Further, the speech synthesis module includes a text analysis unit, a prosody control unit and a synthesized speech unit, wherein the text analysis unit is configured to receive text information and process the text information, and transmit the processing result to the a prosody control unit and the synthesized speech unit, the prosody control unit configured to be connected to the synthesized speech unit and to transmit pitch, length, intensity, pause and intonation information, the synthesized speech unit configured to be The output speech is synthesized by combining the analysis result of the received text analysis unit and the control parameters of the prosody control unit.

Further, the semantic recognition module includes a domain labeling unit, an intention judging unit, and an information extraction unit, wherein the domain labeling unit is configured to be connected to the intention judging unit and transmit domain information, and the intention judging unit is configured In order to connect with the information extraction unit and transmit user intention information, the information extraction unit outputs the result of semantic analysis.

Further, the abnormality processing module includes an abnormality detection unit, a database query unit and an interventionist unit, wherein the abnormality detection unit is configured to receive the output of the speech recognition module and the semantic recognition module, and decide whether to take the Intervention measures, the database query unit is configured to receive the intervention signal of the abnormality detection unit, and to receive the semantic information of the semantic recognition module, query and output the intervention message, and the interventionist unit is configured to use the interventionist to The intervention message output by the database query unit is selected and modified as necessary, and finally a reply message is output to the user.

The present invention also provides an audio-based human-computer hybrid interaction method, comprising the following steps:

Step 1. Provide a speech recognition module, a speech synthesis module, a semantic recognition module and an exception handling module;

Step 2, the speech recognition module converts the speech information into text information and outputs it to the semantic recognition unit;

Step 3, the semantic recognition unit extracts the user purpose and the corresponding key information from the text information;

Step 4: The abnormality processing module judges whether there is currently an abnormality in the human-machine dialogue according to the text information of the speech recognition module and the semantic information of the semantic recognition module, and processes a reply to the abnormality message.

Further, in step 2, the following steps are specifically included:

Step 2.1. Extract features from the input audio stream for processing by the acoustic model, while reducing the impact of environmental noise, channel and speaker factors on the features;

Step 2.2: The decoder searches for the word string that can output the audio stream with the greatest probability according to the acoustic and linguistic models and the dictionary and the processing result of the acoustic model, as the speech recognition result.

Further, in step 3, the following steps are specifically included:

Step 3.1, use the iconic keywords in the text information to mark the field to which the current conversation belongs;

Step 3.2, judge the user's intention based on the rules in the field;

Step 3.3: Extract specific key information according to the domain and the user's intention and in combination with the rules.

Further, in step 4, it specifically includes the following steps:

Step 4.1, the abnormality detection unit judges whether the current man-machine dialogue is abnormal according to the text information of the speech recognition module and the semantic information of the semantic recognition module, and if abnormal, the interventionist unit takes over the man-machine dialogue;

Step 4.2, the database query unit queries the database according to the semantic information, and obtains the intervention message with the recommendation degree. If the recommendation degree of the intervention message is high, the intervention message is directly used for intervention, and if the recommendation degree is low, the interventionist is requested. carry out manual intervention;

Step 4.3. When the machine algorithm cannot find a highly recommended intervention message, the interventionist steps in to select and modify the intervention message, and then sends the modified intervention message to the client.

Further, the key information includes dialogue fields and dialogue keywords, and the dialogue keywords include content keywords and emotion keywords.

Compared with the prior art, the technical effects of the present invention include:

1. Efficiency improvement: The time spent by the interventionist waiting for user input is fully utilized, so that the interventionist can provide intervention services to multiple users at the same time, thereby improving the efficiency of intervention.

2. Cost reduction: There is no need to purchase a series of telecommunications equipment related to the call center, and an intervention platform can be built using existing computers and servers.

3. Rich work scenarios: Since the interventionist interface adopts the B/S (Browser/Server browser/server) structure, the interventionist can open the browser and log in to the corresponding website to perform intervention operations, and it is not necessary to answer the phone at the workstation. Intervention services can be performed on mobile terminals such as PADs, smartphones, and personal notebooks.

4. Low network requirements: The amount of data transmitted by text is very small, so the requirements for the network are reduced. At the same time, the voice heard by the user is synthesized locally and is not affected by the network situation.

5. Unified human-machine dialogue experience: For the user, the interventionist is transparent, and the user's experience is like a dialogue with a fully intelligent "machine", which can seamlessly connect the current human-machine dialogue mode.

The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings, so as to fully understand the purpose, characteristics and effects of the present invention.

Description of drawings

1 is a schematic diagram of an intervention mode of an existing traditional call center;

2 is a schematic diagram of a system module of the present invention;

3 is a schematic flow chart of a system according to a preferred embodiment of the present invention;

FIG. 4 is a schematic diagram of a character dialogue flow according to a preferred embodiment of the present invention.

Detailed ways

The present invention is achieved through the following technical solutions:

As shown in Figure 2, the present invention relates to an audio-based human-machine dialogue exception processing system, including: a speech recognition module, a speech synthesis module, a semantic recognition module and an exception processing module, wherein: the speech recognition module is connected with the semantic recognition module and The text information corresponding to the voice is transmitted. The speech recognition module and the semantic recognition module are connected to the exception processing module, and respectively transmit text information and semantic analysis results. The exception processing module is connected to the speech synthesis module and transmits intervention information.

The speech recognition module includes: a signal processing and feature extraction unit, an acoustic model, a language model and a decoder, wherein: the signal processing and feature extraction unit is connected with the acoustic model and transmits acoustic feature information, and the decoder is connected with the acoustic model and the language model. connected, and output the recognition result to the outside world.

The speech synthesis module includes: a text analysis unit, a rhythm control unit and a synthesized speech unit, wherein: the text analysis unit receives and processes the text information, and transmits the processing result to the rhythm control unit and the synthesized speech unit, and the rhythm control unit The unit is connected with the synthetic speech unit, and transmits information such as the pitch, length, intensity, pause and intonation of the target. The synthetic speech unit receives the analysis results of the text analysis unit and the control parameters of the prosody control unit, and outputs the synthesized speech to the outside world. .

The semantic recognition module includes: a domain labeling unit, an intent judging unit, and an information extraction unit, wherein: the domain labeling unit is connected with the intent judging unit and transmits domain information, the intent judging unit is connected with the information extraction unit and transmits user intent information, information Units are connected to the outside world and transmit information for semantic analysis.

The abnormality processing module includes: an abnormality detection unit, a database query unit, and an interventionist unit, wherein: the abnormality detection unit receives the output of the speech recognition module and the semantic recognition module, and decides whether to take intervention measures, and the database query unit receives the abnormality detection unit. The intervention signal of the unit, and receive the semantic information of the semantic recognition module, query and output the intervention message, the interventionist unit uses the interventionist to perform necessary selection and modification of the intervention message output by the database query unit, and finally outputs the user reply message.

The present invention relates to the abnormal processing method of the human-machine dialogue of the above-mentioned system, which specifically includes the following steps:

Step 1. Provide a speech recognition module, a speech synthesis module, a semantic recognition module and an exception handling module.

Step 2, the speech recognition module converts the speech information into text information and outputs it to the semantic recognition unit, and the specific steps include:

2.1 The front end processes the audio stream and extracts features from the input signal for processing by the acoustic model. At the same time, the influence of environmental noise, channel, speaker and other factors on the features is minimized.

2.2 The decoder searches for the word string that can output the signal with the greatest probability according to the acoustic, linguistic model and dictionary for the input signal, as the recognition result of the speech.

Step 3, the semantic recognition unit extracts the user purpose and the corresponding key information from the text information, and the specific steps include:

3.1 Use the iconic keywords in the text information to mark the field to which the current conversation belongs.

3.2 Judging the user's intention based on rules in specific fields.

3.3 According to the domain and user intent, combined with rules, such as preset templates, extract specific key information.

Step 4. The abnormality processing module judges whether the human-machine dialogue is abnormal at present according to the text information of the speech recognition module and the semantic information of the semantic recognition module, and performs abnormal processing and message reply. The specific steps include:

4.1 The abnormality detection unit judges whether the current man-machine dialogue is abnormal according to the text information of the speech recognition module and the semantic information of the semantic recognition module. If it is not abnormal, it will be handled by the local client, and if it is abnormal, the intervening server will take over the man-machine dialogue.

4.2 The database query unit queries the database according to the semantic information, and obtains the recommended intervention message. If the recommendation degree of the intervention message is high, the intervention message is directly used for intervention, and if the recommendation degree is low, the interventionist is requested to perform manual intervention.

4.3 When the machine algorithm cannot find a highly recommended intervention message, the interventionist will intervene to select and modify the intervention message, and then send the modified intervention message to the client.

In the process of abnormal processing of human-machine dialogue, after the user's voice input is passed through the machine's voice recognition and semantic analysis, the results of speech recognition and semantic analysis will be sent to the interventionist in the form of text, and the interventionist will receive the message. You can then choose to send a conversation message or issue a command message. Conversational messages are transmitted to the machine in the form of text, and then speech is synthesized and played back to the user through a speech synthesis system (TTS), while command messages execute commands directly through the machine.

This embodiment includes the following steps, as shown in FIG. 3 and FIG. 4 , that is, the three steps of user input --> intervention message generation --> client pushes the intervention message respectively introduce technical solutions:

1) User input

In the process of the user's voice input, the used voice recognition system converts the user's voice input audio into text, and at the same time performs semantic analysis on the text (the results of the semantic analysis include the user's current dialogue field, the key information of the user's request for services, etc. ), and finally the text and semantic analysis results are transmitted to the exception handling module in the form of text through the POST method of the HTTP protocol.

2) Intervention message generation

The exception handling module queries the database according to the text information of the speech recognition and the semantic slot of the semantic recognition under abnormal conditions, and obtains an alternative intervention message. If the recommendation degree of the intervention message is high, the intervention message is directly used for intervention, and if the recommendation degree is low, the interventionist is requested to perform manual intervention. The interventionist can see the auxiliary data provided by the exception handling module on the interface, such as the recognition result input by the user and the result of semantic analysis, etc. The interventionist can screen and modify the candidate intervention messages more accurately and quickly by combining this information. Intervention messages are divided into dialogue messages and command messages, both of which are transmitted in the form of text using the unified Websocket protocol.

3) The client pushes the intervention message

After receiving the intervention message, the client immediately returns to the interventionist "message has been received" confirmation, and caches the intervention message in the message queue. The client will monitor the current man-machine dialogue state and try to push the message from the message queue to the user under certain conditions. The specific push timings include: 1. When the intervention message arrives, 2. When the TTS synthesized voice message broadcast is completed; The conditions that need to be met are 1. The message queue is not empty, and 2. The audio player of the client is currently idle. If the intervention message is successfully pushed, it will return the confirmation message of the interventionist that "the intervention message has been pushed".

E.g:

1. User A issues a voice command "I'm going to a fun place".

2. The speech recognition module converts the speech input into text.

3. After processing, the semantic analysis module obtains that the user's intention is "navigation", and the label of the navigation destination is "fun".

4. The anomaly detection unit in the anomaly processing module receives the service request from user A, including the complete speech recognition result "I'm going to a fun place", and the results of semantic analysis "navigation" and "fun". The current conversation state is abnormal.

5. The database query unit in the exception handling module performs database query according to "navigation" and "fun", and gets some alternative messages such as "do you want to go to Suzhou's fun snacks?", "to find 5 and fun for you. Relevant places", the recommendation degree of these two messages is relatively low, so the manual intervention of the interventionist unit is requested. The interventionist selects and modifies the intervention message by using the database query results obtained by the exception processing module, the results of semantic analysis and the text results of speech recognition, and changes the intervention message to "What kind of entertainment do you want?", and sends the message to the user. text message.

6. After the client receives the intervention message, it stores it in the message queue, sends a "message received" feedback to the exception handling module, and tries to push it.

7. After the conditions are met, the speech synthesis system of the intervention message is synthesized and broadcasted. The user hears the audio "What kind of entertainment do you want?", and the client sends a "message has been pushed" feedback to the exception handling module.

8. The customer makes further voice input "I'm going to sing"

9. ASR system converts speech input into text

10. Semantic analysis obtains that the user's intention is "navigation", and the goal of navigation is "KTV"

11. The abnormality detection unit obtains the specific service requirements of user A, including the complete speech recognition result "I'm going to sing", and the results of semantic analysis "navigation" and "KTV".

12. The database query unit searches the database according to "navigation", "KTV", and the relevant information of the user, and obtains the alternative intervention message "Recommend xxx for you, do you want to go?". At the same time, due to the high degree of recommendation, bypass Interventionist unit, send a text message directly to the client "I recommend xxx for you, do you want to go?"

13. The user confirms to go to

14. The exception handling system pushes the intervention message of the command type by the user, including the command type "navigation" and the POI information of the destination.

15. The client takes out the message of the command type "navigation" and the corresponding POI information from the message queue, and performs the navigation operation. The client sends the "message has been pushed" feedback to the exception handling module, and the interaction ends.

The preferred embodiments of the present invention have been described in detail above. It should be understood that many modifications and changes can be made according to the concept of the present invention by those skilled in the art without creative efforts. Therefore, all technical solutions that can be obtained by those skilled in the art through logical analysis, reasoning or limited experiments on the basis of the prior art according to the concept of the present invention shall fall within the protection scope determined by the claims.

Claims (8)

1. A man-machine mixed interaction system based on audio is characterized by comprising a voice recognition module, a voice synthesis module, a semantic recognition module and an exception handling module, wherein the voice recognition module is connected with the semantic recognition module and transmits text information corresponding to voice, the exception handling module is connected with the voice recognition module and the semantic recognition module, the voice recognition module is configured to transmit the text information to the exception handling module, and the semantic recognition module is configured to transmit a semantic parsing result to the exception handling module; the exception handling module is configured to connect with the speech synthesis module and transmit intervention information; the voice synthesis module is configured to convert the intervention information transmitted by the exception handling module into voice, send and play the voice to a user, and wait for the user to further feed back;

the abnormality processing module comprises an abnormality detection unit, a database query unit and an intervener unit, wherein the abnormality detection unit is configured to receive the outputs of the voice recognition module and the semantic recognition module and decide whether to take intervention measures, the database query unit is configured to receive an intervention signal of the abnormality detection unit, receive semantic information of the semantic recognition module, query and output the intervention information with high recommendation degree to the voice synthesis module; the intervener unit is configured to perform necessary preference and modification on the intervention information with low recommendation degree output by the database query unit by using an intervener, and then transmit the intervention information to the voice synthesis module to obtain a reply message to be further fed back by the user.

2. The audio-based human-computer hybrid interaction system according to claim 1, wherein the speech recognition module comprises a signal processing and feature extraction unit, an acoustic model, a language model, and a decoder, wherein the signal processing and feature extraction unit is configured to be connected to the acoustic model and to transmit acoustic feature information, and the decoder is configured to be connected to the acoustic model and the language model and to output a recognition result.

3. The audio-based human-computer hybrid interactive system according to claim 1, wherein the speech synthesis module comprises a text analysis unit, a prosody control unit and a synthesized speech unit, wherein the text analysis unit is configured to receive and process text information, transmit the processing result to the prosody control unit and the synthesized speech unit, the prosody control unit is configured to be connected to the synthesized speech unit and transmit pitch, duration, intensity, pause and intonation information, and the synthesized speech unit is configured to receive the analysis result of the text analysis unit and the control parameters of the prosody control unit to synthesize the output speech.

4. The audio-based human-computer hybrid interaction system according to claim 1, wherein the semantic recognition module comprises a domain labeling unit, an intention judging unit, and an information extracting unit, wherein the domain labeling unit is configured to be connected with the intention judging unit and transmit domain information, the intention judging unit is configured to be connected with the information extracting unit and transmit user intention information, and the information extracting unit outputs a result of semantic analysis.

5. A man-machine mixed interaction method based on audio is characterized by comprising the following steps:

step 1, providing a voice recognition module, a voice synthesis module, a semantic recognition module and an exception handling module;

step 2, the voice recognition module converts voice information into character information and outputs the character information to the semantic recognition module;

step 3, the semantic recognition module extracts the user purpose and corresponding key information from the character information;

step 4, the exception handling module judges whether the man-machine conversation is abnormal at present according to the character information of the voice recognition module and the semantic information of the semantic recognition module and replies to exception handling information;

wherein, in the step 4, the method specifically comprises the following steps:

step 4.1, the abnormality detection unit judges whether the current man-machine conversation is abnormal according to the character information of the voice recognition module and the semantic information of the semantic recognition module, and if the current man-machine conversation is abnormal, the interventionalist unit takes over the man-machine conversation;

step 4.2, the database query unit queries the database according to the semantic information to obtain intervention information with recommendation degree, if the recommendation degree of the intervention information is higher, intervention is directly performed by using the intervention information, the intervention information is sent to a client, and the step 2 is entered to wait for further feedback of the user; if the recommendation degree is low, requesting an interventionalist to perform manual intervention;

and 4.3, when the intervention information with high recommendation degree cannot be found by the machine algorithm, an intervention teacher intervenes to select and modify the intervention information, then the modified intervention information is sent to a client, and the step 2 is entered to wait for further feedback of the user.

6. The audio-based human-computer hybrid interaction method according to claim 5, wherein in the step 2, the method specifically comprises the following steps:

2.1, extracting features from the input audio stream for processing by an acoustic model, and simultaneously reducing the influence of environmental noise, channels and speaker factors on the features;

and 2.2, searching a word string capable of outputting the audio stream with the maximum probability as a voice recognition result by the decoder according to the acoustic model, the linguistic model and the dictionary.

7. The audio-based human-computer hybrid interaction method according to claim 5, wherein in step 3, the method specifically comprises the following steps:

step 3.1, marking the field to which the current conversation belongs by using the symbolic key words in the character information;

step 3.2, judging the user intention based on rules in the field;

and 3.3, extracting specific key information according to the field and the user intention and by combining rules.

8. An audio-based human-computer hybrid interaction method according to claim 5 or 7, wherein the key information includes a dialogue domain, dialogue keywords including content keywords and emotion keywords.

Images