CN114360535B - Voice dialog generation method, device, electronic device and storage medium - Google Patents

Abstract

The present disclosure provides a method and an apparatus for generating a voice dialog, an electronic device, and a storage medium, which relate to the field of computer technologies, and in particular, to the technical fields of artificial intelligence, such as voice technology, natural language processing, and computer vision. The specific implementation scheme is as follows: performing voice recognition on the acquired input voice to determine a first text corresponding to the input voice; performing audio feature extraction on input voice to determine a first audio feature corresponding to the input voice; determining a second text and a second audio characteristic corresponding to the reply sentence to be generated according to the first audio characteristic and the first text; based on the second audio feature and the second text, a response speech is generated. Therefore, the second text and the second audio characteristics are determined according to the first audio characteristics and the first text corresponding to the input voice, so that the accuracy of the determined second text is improved, and the generated reply voice is more fit to the emotion of the speaker corresponding to the input voice.

Description

Voice dialog generation method, device, electronic device and storage medium

technical field

The present disclosure relates to the field of computer technology, in particular to the field of artificial intelligence technology such as speech technology, natural language processing, and computer vision, and in particular to a method, device, electronic device, and storage medium for generating a voice dialogue.

Background technique

With the continuous development and improvement of artificial intelligence technology, it has played an extremely important role in various fields related to human daily life. For example, artificial intelligence has made remarkable progress in the field of spoken dialogue. In related technologies, speech information can be converted into text, and semantic analysis is performed on the text to determine the reply text. Since the relevant technology only determines the answer text based on the single feature of the text contained in the voice information, which may lead to a lower accuracy of the final answer text, therefore, how to improve the accuracy of the answer sentence has become a key research direction .

Contents of the invention

The disclosure provides a method, device, electronic equipment and storage medium for generating a voice dialogue.

According to a first aspect of the present disclosure, a method for generating a voice dialogue is provided, including:

performing speech recognition on the acquired input speech to determine the first text corresponding to the input speech;

performing audio feature extraction on the input speech to determine a first audio feature corresponding to the input speech;

According to the first audio feature and the first text, determine the second text and the second audio feature corresponding to the reply sentence to be generated;

A reply speech is generated based on the second audio feature and the second text.

According to a second aspect of the present disclosure, an apparatus for generating a voice dialogue is provided, including:

A first determining module, configured to perform voice recognition on the acquired input voice, so as to determine the first text corresponding to the input voice;

The second determination module is used to perform audio feature extraction on the input speech, so as to determine the first audio feature corresponding to the input speech;

A third determining module, configured to determine a second text and a second audio feature corresponding to the reply sentence to be generated according to the first audio feature and the first text;

A generating module, configured to generate a reply voice based on the second audio feature and the second text.

According to a third aspect of the present disclosure, an electronic device is provided, including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor, so that the at least one processor can perform the generation of the voice dialogue as described in the first aspect method.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, the computer instructions are used to cause the computer to execute the method for generating a speech dialogue as described in the first aspect.

According to a fifth aspect of the present disclosure, there is provided a computer program product, including computer instructions, and when executed by a processor, the computer instructions implement the steps of the method for generating a speech dialog as described in the first aspect.

The voice dialog generation method, device, electronic equipment, and storage medium provided by the present disclosure have the following beneficial effects:

In the embodiment of the present disclosure, speech recognition is first performed on the acquired input speech to determine the first text corresponding to the input speech, and then audio feature extraction is performed on the input speech to determine the first audio feature corresponding to the input speech, and then according to the first The audio feature and the first text determine the second text and the second audio feature corresponding to the answer sentence to be generated, and finally generate the answer voice based on the second audio feature and the second text. Thus, according to the first audio feature and the first text corresponding to the input voice, the second text and the second audio feature corresponding to the answer sentence are determined, thereby not only improving the accuracy of the determined second text, but also corresponding to the input sentence. The emotional feature of the response sentence determines the emotional feature of the reply sentence, so that the generated reply speech is more suitable for the emotion of the speaker corresponding to the input speech.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

Description of drawings

The accompanying drawings are used to better understand the present solution, and do not constitute a limitation to the present disclosure. in:

FIG. 1 is a schematic flowchart of a method for generating a speech dialogue according to an embodiment of the present disclosure;

Fig. 2 is a schematic flowchart of a method for generating a voice dialog according to another embodiment of the present disclosure;

Fig. 3 is a schematic flowchart of a method for generating a speech dialogue according to another embodiment of the present disclosure;

Fig. 4 is a schematic structural diagram of an apparatus for generating a voice dialogue according to an embodiment of the present disclosure;

Fig. 5 is a block diagram of an electronic device used to implement the method for generating a speech dialogue according to an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Embodiments of the present disclosure relate to artificial intelligence technology fields such as computer vision and deep learning.

Artificial Intelligence (Artificial Intelligence), the English abbreviation is AI. It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

The key technologies of speech technology in the computer field include automatic speech recognition technology (Automatic SpeechRecognition, ASR) and speech synthesis technology (Text To Speech, TTS). Enabling computers to hear, see, speak, and feel is the development direction of human-computer interaction in the future. Among them, voice will become the most promising human-computer interaction method in the future, and voice has more advantages than other interactive methods.

Natural language processing is the use of computers to process, understand and use human languages (such as Chinese, English, etc.). It is an interdisciplinary subject between computer science and linguistics, and is often called computational linguistics. Because natural language is the fundamental symbol that distinguishes human beings from other animals. Without language, human thinking is impossible to talk about, so natural language processing embodies the highest task and state of artificial intelligence, that is to say, only when the computer has the ability to process natural language, the machine can be regarded as realizing real intelligence .

Computer vision refers to machine vision that uses cameras and computers instead of human eyes to identify, track and measure targets, and further performs graphics processing to make computer processing images that are more suitable for human eyes to observe or sent to instruments for detection.

In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

FIG. 1 is a schematic flowchart of a method for generating a speech dialogue according to an embodiment of the present disclosure;

Wherein, it should be noted that the voice dialogue generation method of this embodiment is executed by a voice dialogue generation device, which can be implemented by software and/or hardware, and which can be configured in an electronic device. It may include but not limited to terminals, servers, etc.

As shown in Figure 1, the method for generating the voice dialogue includes:

S101: Perform voice recognition on the acquired input voice to determine a first text corresponding to the input voice.

Wherein, the acquired input speech may be a speech that needs to generate a corresponding reply text according to the content contained in the speech. The input speech may be a continuous speech, such as a sentence, a paragraph, and so on.

Optionally, the input voice can be acquired through a voice collection device, such as a microphone, a sound sensor, etc., or read from a storage space where the voice is stored. This embodiment does not limit the way of acquiring the input voice.

Wherein, the first text refers to the text contained in the input voice, that is, the content contained in the input voice is displayed in the form of text.

In the embodiment of the present disclosure, the voice recognition is used to convert the voice signal corresponding to the input voice into the corresponding first text. Optionally, a hidden Markov model (Hidden Markov Model, HMM) can be used to perform voice recognition on the input voice to determine the first text corresponding to the input voice; Perform comparison to find the same voice, and then obtain the utterance text corresponding to the voice in the voice database as the first text corresponding to the input voice. The present disclosure does not limit this.

S102: Perform audio feature extraction on the input speech to determine a first audio feature corresponding to the input speech.

Wherein, the first audio feature may be information such as frequency and amplitude of a speech signal corresponding to the input speech.

It should be noted that the frequency, amplitude and other features of the voice signal can reflect the emotional information of the speaker corresponding to the input voice. For example, if the frequency of the voice signal corresponding to the input voice is high, it means that the speaker speaks faster and may be more irritable; when the frequency of the voice signal is normal, it means that the speaker may be in a relaxed mood. When the amplitude of the speech signal is higher, it means that the speaker's voice is louder and the emotion may be higher. When the amplitude corresponding to the voice signal is low, it means that the speaker's voice is low and the mood may be depressed.

Optionally, fast Fourier transform may be used to perform audio feature extraction on the input speech, so as to determine the frequency, amplitude, etc. corresponding to the input speech. Alternatively, you can also use the max function in the matlab tool to extract the amplitude corresponding to the input voice, and use the pitch function to extract the frequency in the input voice. The present disclosure does not limit this.

S103: According to the first audio feature and the first text, determine the second text and the second audio feature corresponding to the answer sentence to be generated.

It should be noted that, in the embodiment of the present disclosure, when determining the reply sentence according to the first audio feature of the input voice and the first text, not only the second text corresponding to the reply sentence can be determined, but also the second text of the reply sentence can be determined at the same time. Audio characteristics, which can determine the mood when the reply sentence is played.

Wherein, the second text may be a text generated according to the first audio feature and the first text, and used to reply the input voice.

Wherein, the second audio feature may be an emotional feature when playing the second text determined according to the first audio feature. For example, the first audio feature is higher in frequency and higher in amplitude, indicating that the speaker corresponding to the input sentence is in a more irritable mood. Therefore, the second audio feature corresponding to the reply sentence can be moderate in frequency and moderate in amplitude, that is, using Play the second text in a more soothing tone.

Optionally, the first audio feature and the first text may be input into a preset dialogue model, so as to obtain the second text and the second audio feature corresponding to the reply sentence to be generated.

Alternatively, the keywords contained in the first text may also be extracted first, and then the second text and the second audio features corresponding to the answer sentences to be generated are determined according to the keywords contained in the first text and the first audio features.

In the embodiment of the present disclosure, according to the first audio feature and the first text corresponding to the input voice, the second text of the reply sentence is determined, so that when the first text corresponding to the input voice is the same, if the first audio corresponding to the input voice The characteristics are different, and the second text corresponding to the generated reply sentence is also different, which not only improves the accuracy of the reply sentence, but also makes the determined reply sentence more suitable for the emotion of the speaker corresponding to the input voice, and improves the diversity of the reply text .

S104: Generate a reply voice based on the second audio feature and the second text.

Wherein, the reply voice is the voice obtained by playing the second text by using the second audio feature.

Optionally, a speech synthesis technology (Text to Speech, TTS) may be used to combine the second text and the second audio feature to generate a reply speech.

In the embodiment of the present disclosure, speech recognition is first performed on the acquired input speech to determine the first text corresponding to the input speech, and then audio feature extraction is performed on the input speech to determine the first audio feature corresponding to the input speech, and then according to the first The audio feature and the first text determine the second text and the second audio feature corresponding to the answer sentence to be generated, and finally generate the answer voice based on the second audio feature and the second text. Thus, according to the first audio feature and the first text corresponding to the input voice, the second text and the second audio feature corresponding to the answer sentence are determined, thereby not only improving the accuracy of the determined second text, but also corresponding to the input sentence. The emotional feature of the response sentence determines the emotional feature of the reply sentence, so that the generated reply speech is more suitable for the emotion of the speaker corresponding to the input speech.

Fig. 2 is a schematic flowchart of a method for generating a voice dialog according to yet another embodiment of the present disclosure. As shown in Figure 2, the generation method of the speech dialogue includes:

S201: Perform voice recognition on the acquired input voice to determine a first text corresponding to the input voice.

Wherein, for the specific implementation form of step S201, reference may be made to the detailed descriptions in other embodiments of the present disclosure, and details will not be repeated here.

S202: Perform audio feature extraction on the input speech to determine a first audio feature corresponding to the input speech.

Wherein, the first audio feature may include an amplitude feature and a frequency feature.

Optionally, the second amplitude value corresponding to the input voice may be determined first according to the first amplitude value corresponding to each frame of voice in the input voice, and then the amplitude characteristic corresponding to the input voice may be determined according to the range to which the second amplitude value belongs.

Wherein, the first amplitude may be the maximum value among the amplitudes corresponding to each frame of speech.

Wherein, the second amplitude may be the maximum value among the first amplitudes corresponding to each frame of speech. That is, the maximum amplitude corresponding to the input voice is used as the second amplitude corresponding to the input voice.

Wherein, the amplitude feature may include: high amplitude, medium amplitude, low amplitude, etc., which is not limited in the present disclosure. It should be noted that each amplitude feature corresponds to a different amplitude range. In the embodiment of the present disclosure, the amplitude feature corresponding to the input speech may be determined according to the range to which the second amplitude belongs.

In the embodiment of the present disclosure, before obtaining the first amplitude value corresponding to each frame of the input voice, the input voice may be divided into frames, that is, the first audio data is divided into small segments of fixed length.

Optionally, any preferred manner may be used to determine the first amplitude corresponding to each frame of speech in the input speech, which is not limited in the present disclosure. For example, Fourier transform may be performed on each frame of speech in the input speech to obtain the first amplitude value corresponding to each frame of speech.

In the embodiment of the present disclosure, the second amplitude corresponding to the input voice is first determined according to the first amplitude corresponding to each frame of voice in the input voice, and then the input voice is determined to be high amplitude, medium amplitude, or medium according to the range to which the second amplitude belongs. Amplitude, or low amplitude, so that multiple amplitudes corresponding to the input audio can be represented by one amplitude feature, thereby reducing the calculation amount of subsequent processing without affecting the accuracy of subsequent acquisition of the second text.

Optionally, pitch detection may be performed on the input speech to determine the frequency value corresponding to the speech signal, and then determine the frequency feature corresponding to the input speech according to the range to which the frequency value belongs.

Wherein, the frequency feature may include: high frequency, intermediate frequency, low frequency, etc., which is not limited in the present disclosure. It should be noted that each frequency feature corresponds to a different frequency range. In the embodiment of the present disclosure, the frequency feature corresponding to the input voice may be determined according to the range to which the frequency value belongs.

Optionally, pitch detection may be performed on the input speech to obtain the maximum frequency corresponding to the input speech, and the maximum frequency may be used as the frequency value corresponding to the speech signal. Alternatively, the average frequency corresponding to the input voice may also be used as the frequency value corresponding to the voice signal. The present disclosure does not limit this.

In the embodiment of the present disclosure, the pitch detection is first performed on the input voice to determine the frequency value corresponding to the voice signal, and then according to the range of the frequency value, it is determined whether the input voice is high frequency, medium frequency or low frequency, so that an amplitude can be used The features represent multiple amplitudes corresponding to the input audio, thereby reducing the amount of computation for subsequent processing without affecting the accuracy of subsequent acquisition of the second text.

S203: According to the first audio feature and the first text, determine the second text corresponding to the answer sentence to be generated and the emoticons contained in the second text.

Wherein, the emoticon may include: happy, surprised, afraid, worried, kind, etc., which is not limited in the present disclosure.

It should be noted that there may be one or more emoticons contained in the second text, which is not limited in the present disclosure. For example, the first sentence in the second text corresponds to an emoticon, the second sentence and the third sentence correspond to an emoticon, and so on.

For example, if the amplitude feature contained in the first audio feature is low frequency, and the frequency feature is also low frequency, and then it is determined that the emotion of the speaker corresponding to the input voice is sad, then the emoticon contained in the generated second text can be Worry.

It should be noted that the above example is only a simple illustration, and cannot be used as a specific limitation of the first audio feature and the emoticons contained in the second text in the embodiment of the present disclosure.

S204: Display the second text and the emoticon on the display screen of the interactive device.

Wherein, the interactive device is an electronic device capable of interacting with a user. The interaction device can receive the user's interaction request and process the interaction request to generate a result corresponding to the interaction request, and then display the result to the user in the form of voice or text.

In the embodiment of the present disclosure, after determining the second text and the emoticons contained in the reply sentence corresponding to the input voice, the second text and the emoticons can be displayed on the display screen of the interactive device, so that the user can Combining with the emoticons included in the display interface, reading the second text not only answers the user's request, but also realizes multi-angle effective communication with the user.

Optionally, in this embodiment of the present disclosure, not only can the second text and emoticons be displayed on the display screen of the interactive device, but also the interactive device can play the second text in the tone corresponding to the emoticon.

For example, if the emoticon contained in the second text is worry, the interactive device can not only display the second text and the emoticon of worry on the display screen, but also play the second text in a comforting tone.

In the embodiment of the present disclosure, speech recognition is first performed on the acquired input speech to determine the first text corresponding to the input speech, and then audio feature extraction is performed on the input speech to determine the first audio feature corresponding to the input speech, and then according to the first The audio features and the first text determine the second text corresponding to the answer sentence to be generated and the emoticons contained in the second text, and finally display the second text and the emoticons on the display screen of the interactive device. Thus, according to the first audio feature and the first text corresponding to the input voice, the second text corresponding to the reply sentence and the emoticons contained in the second text are determined, so that the determined second text can be more accurate, and it can also be The second sentence and the corresponding emoticons are displayed on the display screen of the interactive device, thereby realizing effective communication with the user from multiple angles.

Fig. 3 is a schematic flowchart of a method for generating a voice dialog according to yet another embodiment of the present disclosure. As shown in Figure 3, the method for generating the speech dialogue includes:

S301: Perform voice recognition on the acquired input voice to determine a first text corresponding to the input voice.

S302: Perform audio feature extraction on the input speech to determine a first audio feature corresponding to the input speech.

Wherein, for the specific implementation forms of step S301 and step S302, reference may be made to the detailed descriptions in other embodiments of the present disclosure, and details will not be repeated here.

S303: According to the first audio feature and the first text, determine the second text and the second audio feature corresponding to the reply sentence to be generated.

In the embodiment of the present disclosure, the first audio feature and the first text may be input into a preset dialogue model, so as to obtain the second text and the second audio feature corresponding to the reply sentence to be generated.

Optionally, the specific steps of obtaining a preset dialogue model may include: obtaining a training sample set, wherein the training sample set contains input text and corresponding audio features, and annotated answer text corresponding to the input text and corresponding audio feature labels, and then Input the input text and corresponding audio features into the initial dialogue model to obtain the predicted answer text and corresponding predicted audio features output by the initial dialogue model, and then predict the audio features according to the difference between the predicted answer text and the marked answer text The difference between the audio feature labels and the initial dialogue model is corrected to generate the preset dialogue model.

Optionally, the training sample set can be obtained in the following ways: First, a large amount of text dialogue data is automatically mined from network information, and the text dialogue data is manually dubbed, and then the audio feature extraction is performed on the dubbed sample speech data to obtain the text The input text and corresponding audio features included in the dialogue data, and the answer text and corresponding audio feature labels are marked.

Wherein, the audio features may include frequency features and amplitude features. Amplitude features may include: high amplitude, medium amplitude, and low amplitude; frequency features may include: high frequency, middle frequency, and low frequency.

In the embodiment of the present disclosure, after audio feature analysis is performed on the sample speech data obtained from dubbing, the obtained frequencies and amplitudes can be sorted in descending order, and then the frequencies within the first threshold range can be marked as high frequency , mark the frequency within the second threshold range as intermediate frequency, mark the frequency within the third threshold range as low frequency; mark the amplitude within the fourth threshold range as high amplitude, and mark the amplitude within the fifth threshold range as The middle amplitude and the amplitude within the sixth threshold range are marked as low amplitude.

For example, if the frequency range corresponding to all sample speech data is [a, b], then the first threshold range may be [b-10%*(b-a), b], that is, the highest 10% of the frequency range The frequency is marked as high frequency, and the second threshold range can be [a+10%*(b-a), b-10%*(b-a)], that is, the frequency within the frequency range of 10%-90% is marked as intermediate frequency, and the third threshold The range may be [a, a+10%*(b-a)], that is, the lowest 10% of frequencies in the frequency range are marked as low frequencies.

For example, if the amplitude range corresponding to all sample speech data is [c, d], the fourth threshold range may be [d-10%*(d-c), d], that is, the highest 10% of the amplitude range The amplitude of is marked as a high amplitude, and the fifth threshold range can be [c+10%*(d-c), d-10%*(d-c)], that is, the amplitude of 10%-90% within the amplitude range is marked as For medium amplitude, the sixth threshold range may be [c, c+10%*(d-c)], that is, the lowest 10% of the amplitude within the amplitude range is marked as low amplitude.

It should be noted that the above examples are simply illustrations and cannot be used as the first threshold range, the second threshold range, the third threshold range, the fourth threshold range, the fifth threshold range, the sixth threshold range, etc. in the embodiments of the present disclosure. specific limitations.

The amplitude within the fourth threshold range is marked as high amplitude, the amplitude within the fifth threshold range is marked as medium amplitude, and the amplitude within the sixth threshold range is marked as low amplitude.

It can be understood that since the preset dialogue model cannot learn the frequencies or amplitudes of all values, in the embodiment of the present disclosure, the frequencies or amplitudes can be divided into different levels according to the range, so that the dialogue model can be improved. generalization ability.

S304: Acquire a scene image corresponding to the input voice.

Wherein, the scene image may include the scene where the speaker who inputs the voice is located, for example, a classroom, a restaurant, a playground, and the like. Optionally, the scene image may include the face image of the speaker corresponding to the input voice, or may not include the face image not including the speaker, which is not limited in the present disclosure.

Optionally, when it is detected that the collected voice data contains the user's voice, the image acquisition component is started to acquire the scene image corresponding to the input voice.

Wherein, the image acquisition component may be a component in the interactive device that has the function of taking pictures. For example, camera components included in mobile phone devices and tablet devices with interactive functions.

Or, according to the acquisition time of the input voice, the scene image corresponding to the input voice is intercepted from the collected video stream.

Optionally, the video capture device included in the interactive device is used to capture the video stream in real time, and store the captured video stream in the memory, and then according to the acquisition time of the input voice, intercept the scene corresponding to the input voice from the captured video stream image.

In the embodiment of the present disclosure, when it is detected that the collected voice data contains the user's voice, the scene image corresponding to the input voice is acquired; or, according to the acquisition time of the input voice, the scene image corresponding to the input voice is intercepted from the collected video stream. The scene image, so that the acquired scene image can accurately reflect the scene where the speaker corresponding to the input voice is located.

S305: Perform visual feature extraction on the scene image to determine visual features corresponding to the scene image.

Wherein, the visual feature may be a scene feature included in the scene image. For example, teachers, restaurants, basketball courts, etc.

Optionally, target detection can be performed on the scene image first to obtain the type and location information of the objects contained in the scene image, and then determine the scene described by the scene image according to the type and location information of each object, that is, the corresponding object of the scene image. visual features.

Alternatively, the scene image can be automatically segmented to divide the objects or color regions contained in the scene image, and then features are extracted for each image sub-block, and an index is established to obtain the spatial relationship features corresponding to each object in the scene image , and then determine the scene described by the scene image based on the type and spatial relationship of each object.

S306: Correct the second text and/or the second audio feature according to the visual feature.

It can be understood that after the visual features are determined, the second text or the second audio features can be corrected according to the visual features, so that the corrected second text or the second audio features are more accurate and fit The emotion of the speaker corresponding to the input speech.

S307: Generate a reply voice based on the modified second audio feature and the second text.

Wherein, for the specific implementation form of step S307, reference may be made to the detailed descriptions in other embodiments of the present disclosure, and details will not be repeated here.

In the embodiment of the present disclosure, speech recognition is first performed on the acquired input speech to determine the first text corresponding to the input speech, and then audio feature extraction is performed on the input speech to determine the first audio feature corresponding to the input speech, and then according to the first Audio features and the first text, determine the second text and the second audio features corresponding to the reply sentence to be generated, and then modify the second text and the second audio features according to the visual features corresponding to the scene image, and finally based on the corrected The second audio feature and the second text generate a reply voice. Thus, based on the visual features corresponding to the scene image, the second text and the second audio features generated according to the first audio features and the first text are corrected, thereby further improving the performance of the corrected second text and the second audio features. The accuracy further improves the accuracy of the generated answering voice, so that the answering voice is more suitable for the emotion of the speaker corresponding to the input voice.

FIG. 4 is a schematic structural diagram of an apparatus for generating a voice dialogue according to another embodiment of the present disclosure. As shown in FIG. 4 , the apparatus for generating a voice dialogue 400 includes: a first determination module 410 and a second determination module 420 , a third determining module 430 and a generating module 440 .

The first determination module 410 is configured to perform voice recognition on the acquired input voice, so as to determine the first text corresponding to the input voice;

The second determination module 420 is used to extract the audio features of the input speech, so as to determine the first audio feature corresponding to the input speech;

The third determination module 430 is used to determine the second text and the second audio feature corresponding to the reply sentence to be generated according to the first audio feature and the first text;

The generating module 440 is configured to generate a reply voice based on the second audio feature and the second text.

Optionally, the second determining module 420 is specifically used for:

According to the first amplitude corresponding to each frame of voice in the input voice, determine the second amplitude corresponding to the input voice;

According to the range to which the second amplitude belongs, the amplitude feature corresponding to the input speech is determined.

Optionally, the second determining module 420 is specifically used for:

Perform pitch detection on the input voice to determine the frequency value corresponding to the voice signal;

According to the range to which the frequency value belongs, the frequency feature corresponding to the input speech is determined.

Optionally, generating module 440, including:

A first acquisition unit, configured to acquire a scene image corresponding to the input voice;

The first determination unit is configured to perform visual feature extraction on the scene image to determine visual features corresponding to the scene image;

a correction unit, configured to correct the second text and/or the second audio feature according to the visual feature;

A generating unit, configured to generate a reply voice based on the modified second audio feature and the second text.

Optionally, the first acquisition unit is specifically used for:

In response to detecting that the collected voice data contains the user's voice, start the image acquisition component to obtain the scene image corresponding to the input voice;

Or, according to the acquisition time of the input voice, the scene image corresponding to the input voice is intercepted from the collected video stream.

Optionally, also include:

The fourth determination module is used to determine the second text corresponding to the answer sentence to be generated and the emoticons contained in the second text according to the first audio feature and the first text;

The display module is used to display the second text and emoticons on the display screen of the interactive device.

It should be noted that the foregoing explanations on the method for generating a voice dialogue are also applicable to the apparatus for generating a voice dialogue in this embodiment, and details are not repeated here.

In the embodiment of the present disclosure, speech recognition is first performed on the acquired input speech to determine the first text corresponding to the input speech, and then audio feature extraction is performed on the input speech to determine the first audio feature corresponding to the input speech, and then according to the first The audio feature and the first text determine the second text and the second audio feature corresponding to the answer sentence to be generated, and finally generate the answer voice based on the second audio feature and the second text. Thus, according to the first audio feature and the first text corresponding to the input voice, the second text and the second audio feature corresponding to the answer sentence are determined, thereby not only improving the accuracy of the determined second text, but also corresponding to the input sentence. The emotional feature of the response sentence determines the emotional feature of the reply sentence, so that the generated reply speech is more suitable for the emotion of the speaker corresponding to the input speech.

According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

FIG. 5 shows a schematic block diagram of an example electronic device 500 that may be used to implement embodiments of the present disclosure. Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 5 , the device 500 includes a computing unit 501 that can execute according to a computer program stored in a read-only memory (ROM) 502 or loaded from a storage unit 508 into a random-access memory (RAM) 503. Various appropriate actions and treatments. In the RAM 503, various programs and data necessary for the operation of the device 500 can also be stored. The computing unit 501 , ROM 502 and RAM 503 are connected to each other through a bus 504 . An input/output (I/O) interface 505 is also connected to the bus 504 .

Multiple components in the device 500 are connected to the I/O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc. ; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 509 allows the device 500 to exchange information/data with other devices over a computer network such as the Internet and/or various telecommunication networks.

The computing unit 501 may be various general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of computing units 501 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The calculation unit 501 executes various methods and processes described above, such as the generation of voice dialogues. For example, in some embodiments, the generation of the spoken dialogue may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 508 . In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 500 via the ROM 502 and/or the communication unit 509 . When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the above-described generation of the voice dialogue may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured in any other suitable way (for example, by means of firmware) to perform the generation of the voice dialogue.

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide for interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the problem of traditional physical host and VPS service ("Virtual Private Server", or "VPS") Among them, there are defects such as difficult management and weak business scalability. The server can also be a server of a distributed system, or a server combined with a blockchain.

In this embodiment, speech recognition is first performed on the acquired input speech to determine the first text corresponding to the input speech, and then audio feature extraction is performed on the input speech to determine the first audio feature corresponding to the input speech, and then according to the first audio feature and the first text, determine the second text and the second audio feature corresponding to the answer sentence to be generated, and finally generate the answer voice based on the second audio feature and the second text. Thus, according to the first audio feature and the first text corresponding to the input voice, the second text and the second audio feature corresponding to the answer sentence are determined, thereby not only improving the accuracy of the determined second text, but also corresponding to the input sentence. The emotional feature of the response sentence determines the emotional feature of the reply sentence, so that the generated reply speech is more suitable for the emotion of the speaker corresponding to the input speech.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present disclosure can be achieved, no limitation is imposed herein.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined. In the description of the present disclosure, the words "if" and "if" as used may be interpreted as "at" or "when" or "in response to a determination" or "under circumstances".

The specific implementation manners described above do not limit the protection scope of the present disclosure. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims (13)

1. A method for generating a speech dialogue, comprising: performing speech recognition on the acquired input speech to determine the first text corresponding to the input speech; performing audio feature extraction on the input speech to determine a first audio feature corresponding to the input speech; According to the first audio feature and the first text, determine the second text and the second audio feature corresponding to the answer sentence to be generated, wherein the first audio feature and the first text are input into a preset dialogue model In order to obtain the second text and the second audio feature corresponding to the reply sentence to be generated; generating a reply voice based on the second audio feature and the second text; The method for generating the preset dialogue model includes: obtaining a training sample set, wherein the training sample set includes input text and corresponding audio features, and the input text corresponds to annotated answer text and corresponding audio feature labels. The input text and corresponding audio features are input into the initial dialogue model to obtain the predicted answer text output by the initial dialogue model and the corresponding predicted audio features, according to the difference between the predicted answer text and the marked answer text, and The difference between the predicted audio feature and the audio feature label corrects the initial dialogue model to generate the preset dialogue model, wherein the audio feature includes a frequency feature and an amplitude feature; Said generating a reply voice based on said second audio feature and said second text includes: Acquiring a scene image corresponding to the input voice, the scene image including the scene where the speaker of the input voice is located; Performing visual feature extraction on the scene image to determine visual features corresponding to the scene image, wherein the visual feature is a scene feature contained in the scene image; modifying the second text and/or second audio features based on the visual features; generating a reply voice based on the modified second audio feature and the second text; The determining the visual feature corresponding to the scene image includes: Automatically segment the scene image, divide the object or color area contained in the scene image, extract features for each image sub-block, and establish an index, so as to obtain the corresponding spatial relationship of each object in the scene image feature, based on the type and spatial relationship of each object, determine the visual feature corresponding to the scene image. 2. The method according to claim 1, wherein the audio feature extraction is performed on the input speech to determine the first audio feature corresponding to the input speech, comprising: Determine a second amplitude corresponding to the input voice according to the first amplitude corresponding to each frame of voice in the input voice; Determine the amplitude feature corresponding to the input speech according to the range to which the second amplitude belongs. 3. The method according to claim 2, wherein said performing audio feature extraction on said input speech, to determine the first audio feature corresponding to said input speech, comprising: performing pitch detection on the input voice to determine a frequency value corresponding to the voice signal; According to the range to which the frequency value belongs, the frequency feature corresponding to the input voice is determined. 4. The method according to claim 1, wherein said obtaining the scene image corresponding to said input voice comprises: In response to detecting that the collected voice data contains the user's voice, start the image acquisition component to obtain the scene image corresponding to the input voice; Or, according to the acquisition time of the input voice, intercept the scene image corresponding to the input voice from the collected video stream. 5. The method according to any one of claims 1-3, wherein, after determining the first audio feature corresponding to the input voice, further comprising: According to the first audio feature and the first text, determine the second text corresponding to the reply sentence to be generated and the emoticons contained in the second text; On the display screen of the interactive device, the second text and the emoticon are displayed. 6. A device for generating a speech dialogue, comprising: A first determining module, configured to perform voice recognition on the acquired input voice, so as to determine the first text corresponding to the input voice; The second determination module is used to perform audio feature extraction on the input speech, so as to determine the first audio feature corresponding to the input speech; The third determination module is used to determine the second text and the second audio feature corresponding to the answer sentence to be generated according to the first audio feature and the first text, wherein the first audio feature and the first Text input into the preset dialogue model to obtain the second text and the second audio feature corresponding to the reply sentence to be generated; A generating module, configured to generate a reply voice based on the second audio feature and the second text; The method for generating the preset dialogue model includes: obtaining a training sample set, wherein the training sample set includes input text and corresponding audio features, and the input text corresponds to annotated answer text and corresponding audio feature labels. The input text and corresponding audio features are input into the initial dialogue model to obtain the predicted answer text output by the initial dialogue model and the corresponding predicted audio features, according to the difference between the predicted answer text and the marked answer text, and The difference between the predicted audio feature and the audio feature label corrects the initial dialogue model to generate the preset dialogue model, wherein the audio feature includes a frequency feature and an amplitude feature; Wherein, the generating module includes: A first acquiring unit, configured to acquire a scene image corresponding to the input voice, where the scene image includes the scene where the speaker of the input voice is located; The first determination unit is configured to perform visual feature extraction on the scene image, so as to determine the visual feature corresponding to the scene image, wherein the visual feature is a scene feature contained in the scene image; a correction unit, configured to correct the second text and/or the second audio feature according to the visual feature; A generating unit, configured to generate a reply voice based on the modified second audio feature and the second text; The determining the visual feature corresponding to the scene image includes: Automatically segment the scene image, divide the object or color area contained in the scene image, extract features for each image sub-block, and establish an index, so as to obtain the corresponding spatial relationship of each object in the scene image feature, based on the type and spatial relationship of each object, determine the visual feature corresponding to the scene image. 7. The device according to claim 6, wherein the second determining module is specifically configured to: Determine a second amplitude corresponding to the input voice according to the first amplitude corresponding to each frame of voice in the input voice; Determine the amplitude feature corresponding to the input speech according to the range to which the second amplitude belongs. 8. The device according to claim 7, wherein the second determination module is specifically configured to: performing pitch detection on the input voice to determine a frequency value corresponding to the voice signal; According to the range to which the frequency value belongs, the frequency feature corresponding to the input voice is determined. 9. The device according to claim 6, wherein the first acquiring unit is specifically configured to: In response to detecting that the collected voice data contains the user's voice, start the image acquisition component to obtain the scene image corresponding to the input voice; Or, according to the acquisition time of the input voice, intercept the scene image corresponding to the input voice from the collected video stream. 10. The device according to any one of claims 6-8, further comprising: The fourth determination module is used to determine the second text corresponding to the answer sentence to be generated and the emoticons contained in the second text according to the first audio feature and the first text; A display module, configured to display the second text and the emoticons on the display screen of the interactive device. 11. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can perform any one of claims 1-5. Methods. 12. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1-5. 13. A computer program product comprising computer instructions, which implement the steps of the method according to any one of claims 1-5 when executed by a processor.

Images