JP7495220B2 - Voice recognition device, voice recognition method, and voice recognition program - Google Patents

Description

The present invention relates to a voice recognition device, a voice recognition method, and a voice recognition program.

Advances in technological innovation have improved the accuracy of technologies that convert input voice into text, making communication methods more scalable.

JP 2012-510088 A JP 2019-60921 A JP 2012-003162 A

However, voice input is highly dependent on the environment at the time of input, and may impede necessary communication. For example, voice input may not be recognized accurately in the following cases: 1. Voice input in an environment where the user cannot speak loudly, such as a public place; 2. Voice input by a user who is unwell due to a cold or whose voice is difficult to hear; 3. Voice input in places with loud ambient noise, such as a main road or event venue; 4. Voice input by a user with a speech disorder. As a result, there are cases where advanced communication tools cannot be utilized.

Therefore, the objective of the present invention is to solve the above problems and improve the accuracy of voice recognition.

In order to solve the above-mentioned problems, the present invention is characterized by comprising a first acquisition unit that acquires speech information including information indicating the movement of the user's mouth when the user speaks in a whisper and voice information of the user, and an output unit that uses a model created by learning using the speech information acquired by the first acquisition unit and the speech content indicated by the speech information, inputs the speech information to be recognized, and outputs a recognition result of the speech content indicated by the speech information.

The present invention can improve the accuracy of voice recognition.

FIG. 1 is a diagram showing an example of the configuration of a system including a speech-reading device. FIG. 2 is a diagram for explaining an outline of the system. FIG. 3 is a diagram showing an example of the configuration of a speech reading device. FIG. 4 is a diagram illustrating an example of the configuration of a terminal device. FIG. 5 is a flowchart showing an example of a processing procedure of the speech reading apparatus. FIG. 6 is a sequence diagram showing an example of a processing procedure of the system. FIG. 7 is a diagram illustrating an example of a computer that executes a voice recognition program.

Below, a description will be given of a form (embodiment) for carrying out the present invention with reference to the drawings. The present invention is not limited to the embodiment described below.

The outline of a system including a speech-reading device (voice recognition device) 10 of this embodiment will be described with reference to Figures 1 and 2. For example, as shown in Figure 1, the system includes a speech-reading device 10 and a user's terminal device 20. The speech-reading device 10 recognizes what the user has said based on the mouth movements and voice of the user when whispering, which are acquired from the terminal device 20. The speech-reading device 10 then transmits the recognition result (e.g., text information) to the terminal device 20. In the following description, a whisper is defined as a voice (unvoiced sound) uttered by the user without vibrating the vocal cords.

The terminal device 20 is a mobile phone, a smartphone, a tablet terminal, a personal computer, etc. The terminal device 20 and the speech reading device 10 are communicatively connected via a network such as the Internet. Note that the number of terminal devices 20 and speech reading devices 10 installed in the system is not limited to the number shown in FIG. 2.

Next, the outline of the system will be described with reference to FIG. 2. For example, the terminal device 20 first acquires the change between each coordinate of the outline of the mouth when the user speaks in a whisper using a camera or the like, and also acquires the audio waveform of the whisper using a microphone or the like. Next, the terminal device 20 creates a multiplexed digital signal by multiplexing, for example, information indicating the acquired change in the outline of the mouth (speech reading information) and information indicating the audio waveform of the whisper (audio information), and transmits the multiplexed digital signal to the speech reading device 10. The speech reading device 10 identifies the contents of the user's speech based on the multiplexed digital signal transmitted from the terminal device 20. This makes it possible for the speech reading device 10 to easily identify the contents of the user's speech even in cases where the contents of the user's speech cannot be identified (recognized) using only the speech reading information or only the audio information. For example, by using the above audio information, the speech reading device 10 can identify divisions of consonants, words, phrases, etc. in the speech, which are difficult to identify using only the speech reading information. As a result, the speech reading device 10 can improve the accuracy of identifying the contents of the user's speech.

The speech reading device 10 also learns speech reading information and voice information when the user speaks in a whisper, and uses the learning results to identify the content of the user's speech. This can further improve the accuracy of identifying the content of the user's speech. For example, even in cases where the content of the user's speech cannot be identified even by combining the speech reading information and voice information, the learning results can be used to make it easier to identify the content of the user's speech.

[Speech reading device]
Next, a configuration example of the speech-reading device 10 will be described with reference to Fig. 3. As shown in Fig. 3, the speech-reading device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

The communication unit 11 is realized, for example, by a network interface card (NIC). The communication unit 11 is connected to the network via a wired or wireless connection, and transmits and receives information to and from the terminal device 20.

The storage unit 12 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 12 stores information referenced by the control unit 13 when performing various processes, and information created by various processes. For example, the storage unit 12 stores a model learned (created) by the learning unit 134. This model is created for each user of the terminal device 20. Details of the model will be described later.

The control unit 13 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs (corresponding to an example of a voice recognition program) stored in a storage device inside the speech reading device 10 using the RAM as a working area.

The control unit 13 includes a speech reading processing unit 131 and a speech-to-text conversion unit (text conversion unit) 132.

The speech-reading processor 131 recognizes the contents of a user's speech based on information indicating the movement of the user's mouth when the user speaks and audio information of the user's whispering voice (collectively referred to as "speech information"). For example, the speech-reading processor 131 generates audio data indicating the contents of a user's speech based on the user's speech information. The speech-reading processor 131 then outputs the generated audio data to the speech-to-text converter 132. The speech-to-text converter 132 converts the audio data output from the speech-reading processor 131 into text information.

The speech reading processing unit 131 will be described in detail. The speech reading processing unit 131 includes an utterance information acquisition unit (first acquisition unit) 133, a learning unit 134, an output processing unit (output unit) 135, and a correction information acquisition unit (second acquisition unit) 136.

The speech information acquisition unit 133 acquires user speech information from the terminal device 20. For example, the speech information acquisition unit 133 acquires speech information from the terminal device 20 when the user speaks text for initial setup, and speech information of the user to be recognized.

The learning unit 134 performs learning using the user's utterance information acquired by the utterance information acquisition unit 133 and the utterance content indicated by the utterance information. For example, the learning unit 134 learns the user's utterance information and the utterance content indicated by the utterance information, and creates a model for outputting the recognition result of the utterance content indicated by the user's utterance information.

Here is an example. For example, the learning unit 134 first registers information that associates utterance information obtained when the user reads out the text for initial setup from the utterance information acquisition unit 133 with the content of the text for initial setup as initial information for the model.

After that, when the correction information acquisition unit 136 (described below) receives correction information from the terminal device 20 regarding the recognition result of the utterance content using the model after the initial information is registered, the learning unit 134 corrects the model of the user based on the correction information. Also, when the correction information acquisition unit 136 receives correction information from the terminal device 20 regarding the recognition result of the utterance content using the above-mentioned corrected user model, the learning unit 134 corrects the model of the user based on the correction information. By repeating such processing, the learning unit 134 can create a model that can accurately recognize the user's utterance content.

When the learning unit 134 performs learning based on the user's speech information, it learns homonyms using the words and sounds before and after the word of interest. For example, the user's mouth movements are almost the same when pronouncing the words "tools," "roots," and "cool." Therefore, for example, the learning unit 134 learns homonyms using the words and sounds before and after the words "tools," "roots," and "cool" in the following sentence:

・Tools invented by Japanese people are excellent. ・The roots of the Japanese people are the Jomon and Yayoi people. ・Everything the Japanese use is cool.

By doing this, the learning unit 134 can create a model that can accurately recognize homonyms among the words spoken by the user.

The output processing unit 135 receives as input the speech information of the user to be recognized, and outputs the recognition result of the speech content indicated by the speech information of the user using the learning result by the learning unit 134 (e.g., the above model). For example, the output processing unit 135 receives as input the speech information of the user to be recognized, and generates voice data indicating the speech content of the user using the above model, and outputs it to the speech text conversion unit 132. After that, when the output processing unit 135 receives text information of the voice data from the speech text conversion unit 132, it transmits the text information to the terminal device 20 of the user.

The correction information acquisition unit 136 acquires correction information related to the recognition result of the user's utterance content from the terminal device 20. For example, the correction information acquisition unit 136 receives correction information related to text data indicating the user's utterance content from the terminal device 20. Then, the correction information acquisition unit 136 outputs the correction information to the learning unit 134.

[Terminal Device]
Next, a configuration example of the terminal device 20 will be described with reference to Fig. 4. As shown in Fig. 4, the terminal device 20 includes a communication unit 21, a storage unit 22, a microphone 23, a camera 24, an input unit 25, an output unit 26, and a control unit 27.

The communication unit 21 is an interface such as a NIC that communicates with the speech reading device 10 that is communicatively connected to the network.

The memory unit 22 is realized, for example, by a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk.

The microphone 23 acquires audio. For example, the microphone 23 acquires audio when the user speaks in a whisper. The camera 24 captures images (video or still images). For example, the camera 24 captures images of the movement of the user's mouth when the user speaks in a whisper.

The input unit 25 is an input device that accepts various operations from a user. For example, the input unit 25 is realized by a keyboard, a mouse, operation keys, etc. The output unit 26 is a display device for displaying various information. For example, the output unit 26 is realized by a liquid crystal display, etc. Note that when a touch panel is adopted for the terminal device 20, the input unit 25 and the output unit 26 are integrated.

The control unit 27 is a controller, and is realized, for example, by a CPU, an MPU, or the like, executing various programs (speech reading programs) stored in a storage device inside the terminal device 20 using the RAM as a working area.

The control unit 27 has an utterance receiving unit 271, an utterance information transmitting unit 272, a text receiving unit 273, a display unit 274, and a correction information transmitting unit 275.

The speech receiving unit 271 acquires information indicating the movement of the user's mouth when speaking using the camera 24, and also acquires voice information of the user when speaking using the microphone 23.

The speech information sending unit 272 creates speech information including information indicating the user's mouth movements when speaking and the user's voice information acquired by the speech receiving unit 271, and sends the information to the speech reading device 10. For example, the speech information sending unit 272 converts the information indicating the user's mouth movements when speaking and the user's voice information acquired by the speech receiving unit 271 into a digital signal, and sends the digital signal to the speech reading device 10 as speech information.

The text receiving unit 273 receives text information indicating the content of the user's speech from the speech reading device 10. The display unit 274 displays various information on the output unit 26. For example, the display unit 274 displays text information for initial setup on the output unit 26, and displays text information received by the text receiving unit 273 on the output unit 26.

For example, when the speech reading program of the terminal device 20 is started, the display unit 274 displays a message on the output unit 26 prompting the user to whisper the text information for initial setup. Then, when the user reads the text information in a whisper, the speech receiving unit 271 acquires information indicating the movement of the user's mouth when reading aloud by the camera 24, and also acquires voice information when reading aloud by the microphone 23. Then, the speech information sending unit 272 creates speech information including the information indicating the movement of the user's mouth when reading the text for initial setup and the voice information acquired by the speech receiving unit 271, and sends it to the speech reading device 10.

The correction information sending unit 275 sends correction information of the text information received from the speech reading device 10 to the speech reading device 10. For example, if the display unit 274 displays the text information received from the speech reading device 10 on the output unit 26 and then receives correction information for the text information from the input unit 25, the correction information sending unit 275 sends the correction information to the speech reading device 10.

[Processing Procedure]
Next, an example of a processing procedure of the speech-reading apparatus 10 will be described with reference to FIG.

Although not illustrated in FIG. 5, the speech-reading device 10 identifies which user's terminal device 20 is the accessing terminal device 20 by, for example, performing user authentication on the terminal device 20. This allows the speech-reading device 10 to manage models for each user.

First, the speech information acquisition unit 133 of the speech reading device 10 acquires the user's speech information for initial setup from the terminal device 20 (S1). For example, the speech information acquisition unit 133 acquires the speech information when the user reads the text for initial setup in a whisper. Then, the learning unit 134 registers the user's speech information for initial setup acquired in S1 as initial information of the model (S2: Registration of user's speech information for initial setup). For example, the learning unit 134 registers information that associates the speech information when the user reads the text for initial setup in a whisper with the content of the text for initial setup as initial information of the model.

After S2, when the speech information acquisition unit 133 of the speech reading device 10 acquires speech information of the user whose speech content is to be recognized from the terminal device 20 (S3), the output processing unit 135 uses the model of the user to output the recognition result of the speech content indicated by the speech information (S4). For example, when the output processing unit 135 acquires speech information of the user whose speech content is to be recognized from the terminal device 20, it generates voice data indicating the speech content indicated by the speech information using the model of the user. Then, the output processing unit 135 outputs the generated voice data to the speech text conversion unit 132. Then, the output processing unit 135 receives text information of the voice data from the speech text conversion unit 132. Then, the output processing unit 135 transmits the received text information to the terminal device 20 of the user.

After that, if the correction information acquisition unit 136 acquires correction information for the recognition result output in S4 from the terminal device 20 of the user (Yes in S5), the learning unit 134 corrects the model for the user based on the correction information (S6) and returns to S3. On the other hand, if the correction information acquisition unit 136 does not acquire correction information for the recognition result output in S4 from the terminal device 20 of the user (No in S5), the process returns to S3.

By repeating the above steps S3 to S6, the speech reading device 10 can learn the mouth movements and voice characteristics when the user speaks in a whisper. As a result, the speech reading device 10 can accurately recognize the content of the user's whispered speech.

[Example of processing procedure]
Next, an example of a processing procedure of a system including the speech-reading apparatus 10 will be described with reference to Fig. 6. The processing procedure can be divided into the following phases, for example: (1) initial information registration, (2) use of a voice recognition service, and (3) utilization of the recognition results.

(1) Initial Information Registration For example, when the speech-reading application of the terminal device 20 is started, the terminal device 20 displays a text sentence for initial setup on the output unit 26. Then, the user of the terminal device 20 whispers the text sentence for initial setup to the terminal device 20 (S11). At this time, the speech receiving unit 271 acquires information indicating the movement of the user's mouth when the text sentence for initial setup is read out by the camera 24, and also acquires voice information of the user when the text sentence for initial setup is read out by the microphone 23. After that, the speech information transmitting unit 272 converts the acquired information indicating the movement of the mouth and the voice information into digital signals and transmits them to the speech-reading device 10 (S12).

After S12, when the speech information acquisition unit 133 of the speech reading device 10 receives a digital signal from the terminal device 20, the learning unit 134 compares the received digital signal with the text for initial setup and registers it in the model (S13). In other words, the learning unit 134 registers the initial information of the user's speech in the model.

(2) Use of Voice Recognition Service Next, the user whispers into the terminal device 20 (S21). The speech receiving unit 271 then obtains information indicating the user's mouth movements during the utterance using the camera 24, and obtains the user's voice information during the utterance using the microphone 23. After that, the speech information transmitting unit 272 converts the obtained information indicating the mouth movements and voice information into digital signals and transmits them to the speech reading device 10 (S22).

After S22, when the speech information acquisition unit 133 of the speech reading device 10 receives the digital signal from the user's terminal device 20, the output processing unit 135 converts the received digital signal into a voice signal using a model in which the user's initial information is registered, and outputs the signal to the voice text conversion unit 132 (S23). The voice text conversion unit 132 then converts the output voice signal into text information, and outputs the text information to the output processing unit 135 (S24: voice to text conversion). The output processing unit 135 transmits the text information converted in S24 to the user's terminal device 20 (S25).

After S25, when the text receiving unit 273 of the terminal device 20 receives the text information from the speech-reading device 10, the display unit 274 displays the received text information on the output unit 26 (S26: display text). Next, the user checks the text information displayed on the terminal device 20 (S27), and if there is a part of the text information that needs correction, inputs correction information for the text information using the input unit 25 or the like. Then, the correction information sending unit 275 sends the correction information for the input text information to the speech-reading device 10 (S28: text correction). After that, when the correction information acquisition unit 136 of the speech-reading device 10 receives correction information for the text information from the user's terminal device 20, the learning unit 134 uses the correction information to correct the model for that user (S29: correct model using correction information).

Although not explained here, if the system receives the user's speech again after S29, the speech-reading device 10 executes the process from S23 onwards again based on the corrected model of the user. By repeating the above process, the speech-reading device 10 can create a model that achieves highly accurate conversion customized for the user.

In addition, in S27, the user may check the text information displayed on the terminal device 20, and if there is no part that needs correction, input to the terminal device 20 that no correction is necessary. In that case, the terminal device 20 transmits information to the speech reading device 10 that the text information does not need correction.

(3) Utilization of Recognition Results The terminal device 20 may utilize the text information (the recognition results of the speech content) received from the speech-reading device 10 for other applications or services. For example, the terminal device 20 may use the received text information to perform text linking to a communication application such as e-mail or chat.

[others]
In the above embodiment, the speech-reading device 10 converts the user's speech content into text information and transmits the converted text information to the terminal device 20. However, the present invention is not limited to this. For example, the speech-reading device 10 may transmit voice data indicating the user's speech content to the terminal device 20.

Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. For example, the functions of the speech reading device 10 may be equipped in the terminal device 20.

The above-described embodiments and variations can be combined as appropriate to the extent that they do not cause inconsistencies in the processing content.

Although several embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the disclosure section of the invention.

[program]
The speech-reading device 10 can be implemented by installing a program that realizes the functions of the speech-reading device 10 described in the above embodiment in a desired information processing device (computer). For example, the above program provided as package software or online software can be executed by the information processing device to cause the information processing device to function as the speech-reading device 10. The information processing device referred to here includes desktop or notebook personal computers, rack-mounted server computers, and the like. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as PDAs (Personal Digital Assistants), and the like. The speech-reading device 10 may also be implemented in a cloud server.

An example of a computer that executes the above program (voice recognition program) will be described with reference to FIG. 7. As shown in FIG. 7, the computer 1000 has, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

The memory 1010 includes a read only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

As shown in FIG. 7, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. The various data and information described in the above embodiments are stored, for example, in the hard disk drive 1090 or memory 1010.

Then, the CPU 1020 reads the program module 1093 and program data 1094 stored in the hard disk drive 1090 into the RAM 1012 as necessary, and executes each of the procedures described above.

The program module 1093 and program data 1094 relating to the above-mentioned voice recognition program are not limited to being stored in the hard disk drive 1090, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 relating to the above-mentioned program may be stored in another computer connected via a network such as a LAN or WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

10 speech reading device 11, 21 communication unit 12, 22 storage unit 13, 27 control unit 20 terminal device 23 microphone 24 camera 25 input unit 26 output unit 131 speech reading processing unit 132 speech text conversion unit 133 speech information acquisition unit 134 learning unit 135 output processing unit 136 correction information acquisition unit 271 speech reception unit 272 speech information transmission unit 273 text reception unit 274 display unit 275 correction information transmission unit

Claims (6)

a first output unit that outputs a message encouraging a user to whisper a predetermined utterance content; a first acquisition unit that acquires speech information including information indicating a mouth movement of the user when the user whispers the predetermined utterance content and voice information of the user;
a learning unit that creates a model that outputs a recognition result of the utterance content indicated by the utterance information by learning using the utterance information acquired by the first acquisition unit and the utterance content indicated by the utterance information; and a second output unit that uses the created model to input the utterance information to be recognized and outputs a recognition result of the utterance content indicated by the utterance information,
When the predetermined speech content includes a homonym, the learning unit creates the model by learning the homonym included in the predetermined speech content using words before and after the homonym and speech information during learning using the speech information and the speech content.
1. A speech recognition device comprising: A text conversion unit converts the recognition result of the utterance content into text information,
The second output section is
2. The speech recognition device according to claim 1, further comprising: a text conversion unit that converts the speech into text information, the text information being a recognition result of the speech content. The recognition result of the speech content is
The speech data is
The text conversion unit is
3. The speech recognition device according to claim 2, further comprising: a step of converting the speech data of the utterance into text information. The speech recognition device includes:
A second acquisition unit that acquires correction information for text information of the recognition result of the utterance content input by the user,
The learning unit is
3. The speech recognition device according to claim 2, further comprising: a processor for modifying the model using modification information for the text information of the recognition result of the speech content, the modification information being acquired by the second acquisition unit. A speech recognition method executed by a speech recognition device, comprising:
outputting a message prompting the user to whisper a predetermined utterance content;
acquiring speech information including information indicating a mouth movement of the user when the user whispers the predetermined speech content and voice information of the user;
creating a model that outputs a recognition result of the speech content indicated by the utterance information by learning using the acquired utterance information and the speech content indicated by the utterance information;
and using the created model, receiving utterance information to be recognized as an input, and outputting a recognition result of the utterance content indicated by the utterance information ,
In the creating step, when the predetermined speech content includes a homonym, the model is created by learning the homonym included in the predetermined speech content using words before and after the homonym and speech information during learning using the speech information and the speech content.
13. A speech recognition method comprising: outputting a message prompting the user to whisper a predetermined utterance content;
acquiring speech information including information indicating a mouth movement of the user when the user whispers the predetermined speech content and voice information of the user;
creating a model that outputs a recognition result of the speech content indicated by the utterance information by learning using the acquired utterance information and the speech content indicated by the utterance information;
and a step of inputting utterance information to be recognized using the created model and outputting a recognition result of the utterance content indicated by the utterance information ;
In the creating step, when the predetermined speech content includes a homonym, the model is created by learning the homonym included in the predetermined speech content using words before and after the homonym and speech information during learning using the speech information and the speech content.
A speech recognition program comprising:

Images