JP7541474B2 - Speech evaluation system - Google Patents

Description

The present invention relates to a speech evaluation system.

Literature 1 discloses a speech recognition result evaluation device that determines whether the speech recognition result is correct by comparing the result of a speaker's speech being transcribed by a human with the result of speech recognition of the speaker's speech by a specified speech recognition unit. Such a device makes it possible to detect recognition errors in the speech recognition result.

JP 2001-67096 A

Here, with regard to speech recognition of speech, there is a desire from users to detect pronunciation errors in speech in order to accurately grasp the meaning of the recognized utterance. However, the conventional devices described above are unable to properly detect pronunciation errors when there is a pronunciation error in the speech itself of the speaker who is the target of speech recognition. For example, when an error in speech is detected from the results of speech recognition of an utterance, typically there is no distinction between the speaker's misuse of words and the speaker's pronunciation error, and both are detected as errors in speech.

The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a speech evaluation system that can appropriately detect pronunciation errors in a speaker's speech.

The speech evaluation system according to one aspect of the present invention includes a conversion unit that acquires the result of speech recognition of a speaker's speech and converts the result of speech recognition into a character string representing a pronunciation; a calculation unit that calculates an edit distance between an expected character string, which is a character string representing the pronunciation of a word expected to appear in the speaker's speech, and one or more spoken character strings included in the character string representing the pronunciation converted by the conversion unit; a detection unit that detects, among the one or more spoken character strings, a spoken character string whose edit distance is equal to or less than a predetermined value and is not identical to the expected character string as a mispronunciation character string; and an output unit that outputs the mispronunciation character string. The calculation unit calculates the edit distance taking into account the pronunciation classification of consonants included in the expected character string and the spoken character string.

In a speech evaluation system according to one aspect of the present invention, a speaker's speech is converted into a character string representing a pronunciation, an edit distance between an expected character string, which is a character string representing the pronunciation of a word expected to appear in the speaker's speech, and one or more speech character strings included in the converted character string representing the pronunciation is calculated, and among the one or more speech character strings, a speech character string whose edit distance is equal to or less than a predetermined value and which is not identical to the expected character string is detected as a mispronunciation character string. Here, in detecting errors in speech, for example, when the result of speech recognition is converted into a sentence and an error in speech is detected, a speaker's misuse of a word and a speaker's mispronunciation are not usually distinguished from each other, and both are detected as errors in speech. In such a detection process, it is not possible to appropriately detect mispronunciations in the speaker's speech. In this regard, in the speech evaluation system according to one aspect of the present invention, the speaker's speech is converted into a character string representing the pronunciation, the edit distance between the speech string included in the character string and the expected character string is derived, and the speech string with the edit distance equal to or less than a predetermined value is detected as a speech error string. Therefore, a speech string estimated to have a small degree of error and a high probability of being a mere pronunciation error rather than a misuse of a word can be appropriately detected as a mispronounced string. Furthermore, in the speech evaluation system according to one aspect of the present invention, the edit distance between the expected character string and the speech string is calculated taking into account the pronunciation classification of the consonant. With this configuration, for example, the edit distance is calculated so that the edit distance between the expected character string and the speech string becomes smaller as the pronunciation of the pronunciation string becomes more similar to that of the expected character string. As a result, the detection unit can detect a speech string with a pronunciation close to that of the expected character string (a string that is "high probability of being a mere pronunciation error rather than a misuse of a word") as a mispronounced string. As described above, the speech evaluation system according to one aspect of the present invention can appropriately detect pronunciation errors in the speaker's speech.

The present invention provides a speech evaluation system that can appropriately detect pronunciation errors in a speaker's speech.

FIG. 1 is a diagram illustrating an overview of an utterance evaluation system according to an embodiment of the present invention. 1 is a block diagram showing a functional configuration of an utterance evaluation system according to an embodiment of the present invention; FIG. 2 is a diagram showing an example of pronunciation classification of consonants taken into consideration in the calculation of the edit distance in FIG. 1 . FIG. 4 is a diagram showing an example of calculation of the edit distance in FIG. 1 taking into account the pronunciation classification of the consonants in FIG. 3 . 4 is a flowchart showing a process performed by the speech evaluation system according to the present embodiment. 2 is a diagram comparing the conventional technique and the technique according to the present embodiment with respect to the calculation of the edit distance in FIG. 1; FIG. 1A and 1B are diagrams illustrating speech evaluation according to the conventional technology and the technology of the present embodiment. 2 is a diagram showing the hardware configuration of a communication terminal and an utterance evaluation server included in the utterance evaluation system according to the present embodiment. FIG.

Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements will be designated by the same reference numerals, and duplicate descriptions will be omitted.

The speech evaluation system 1 shown in Figures 1 and 2 is a system that recognizes the speech of a speaker S and evaluates the speech recognition result. Evaluating the speech recognition result means, for example, evaluating whether or not a character string of the speech content that is the speech recognition result is appropriate. As an example, the speech evaluation system 1 is a system used for language tests, etc., and evaluates whether or not a character string of the speech content of a speaker S in response to a question (speech recognition result) is appropriate as a character string indicating an answer to the question.

More specifically, the speech evaluation system 1 determines whether or not there is a pronunciation error in the speech recognition result, and if there is a pronunciation error, corrects the pronunciation error and evaluates the speech recognition result after the correction. The speech evaluation system 1 converts the speech recognition result into a character string representing the pronunciation, detects and corrects the pronunciation error from the speech recognition result taking into account the pronunciation similarity of the consonants. For example, the speech evaluation system 1 derives the pronunciation similarity of consonants between a character string representing the pronunciation of a word expected to appear in the speech of the speaker S in response to a question and a character string representing the pronunciation related to the speech recognition result, and if the two character strings are not identical and the pronunciation similarity of the consonants contained in both character strings is higher than a predetermined value, it determines that the speech of the speaker S contains a pronunciation error (detects the pronunciation error), corrects the pronunciation error, and evaluates the speech recognition result after the correction. The speech evaluation system 1 includes a communication terminal 10 having a function of collecting the speech of the speaker S, and a speech evaluation server 30 that transmits and receives data and processes data. First, we will provide an overview of the processing performed by the speech evaluation system 1.

In the speech evaluation system 1, the speech evaluation server 30 pre-stores character strings (hereinafter referred to as expected character strings) representing the pronunciation of words expected to appear in the speech of speaker S, and pronunciation classifications of consonants (described later). In the example shown in FIG. 1, the expected character string is "gita-".

For example, when an application is executed on a communication terminal 10 or the like to perform speech evaluation, the communication terminal 10 transmits the speech of speaker S, collected by a microphone or the like, to the speech evaluation server 30 as speech voice data. The speech evaluation server 30 performs voice recognition on the speech of speaker S using the speech voice data acquired from the communication terminal 10. The speech evaluation server 30 converts the result of the voice recognition into a character string representing the pronunciation.

In the example shown in FIG. 1, the voice of speaker S responding to the question "What are your hobbies?" with "My hobby is playing the guitar" is collected by the microphone of the communication terminal 10, and the collected voice is transmitted by the communication terminal 10 to the speech evaluation server 30 as speech voice data. Next, the speech evaluation server 30 performs speech recognition on the speech voice data, thereby performing speech recognition on the speech of speaker S. Then, the speech evaluation server 30 generates the sentence "My hobby is playing the guitar" as a result of the speech recognition. The speech evaluation server 30 then converts the generated sentence into a romanized text that expresses the pronunciation "syuumihabita-wohikukotodesu."

The speech evaluation server 30 acquires an expected string. The speech evaluation server 30 generates one or more strings (hereinafter referred to as spoken strings) with the same number of characters as the expected string from a string representing the pronunciation converted from the results of speech recognition (hereinafter referred to as a converted string). The speech evaluation server 30 calculates the edit distance between the expected string and each spoken string taking into account the pronunciation classification of the consonants (described below). Here, the edit distance refers to the minimum number of steps required to transform a string into another string, with the replacement of one character being one step. Among the spoken strings whose calculated edit distance is less than or equal to a predetermined value, the speech evaluation server 30 detects those that are not identical to the expected string as mispronounced strings.

In the example shown in FIG. 1, the speech evaluation server 30 generates N-grams of five characters, the same number of characters as the expected character string "gita-", from the Roman alphabet sentence "syumihabita-wohikukotodesu.", and identifies each as a spoken character string. Here, an N-gram is a character string of a predetermined number of characters extracted from a converted character string while shifting the first character by one character and allowing overlaps. The speech evaluation server 30 then calculates an error rate, which is the value obtained by dividing the edit distance between the expected character string "gita-" and each spoken character string by the number of characters, 5. Furthermore, the speech evaluation server 30 detects the spoken character string "bita-", whose error rate falls below a threshold as a result of comparison with the expected character string "gita-", as a mispronounced character string.

When the speech evaluation server 30 detects a mispronounced character string, it corrects the mispronounced character string in the converted character string to the assumed character string used to detect the mispronounced character string. The speech evaluation server 30 converts the corrected converted character string (e.g., a Roman alphabet text) into a sentence format (Japanese text) in the language used in the speech of the speaker S (hereinafter, the converted text will be referred to as an evaluation text). The speech evaluation server 30 then converts the mispronounced character string into the language used in the speech of the speaker S (hereinafter, the converted word will be referred to as a mispronounced word). The speech evaluation server 30 scores the evaluation text, and outputs the evaluation text, the mispronounced word, and the scoring result. Note that, when the speech evaluation server 30 does not detect a mispronounced character string, the result of the initial voice recognition is used as the evaluation text.

In the example shown in FIG. 1, the speech evaluation server 30 corrects the speech character string "bita-" detected as a mispronounced character string in the romanized sentence "syumihabita-wohikukotodesu." to the expected character string "gita-". Then, the speech evaluation server 30 converts the romanized sentence "syumihagita-wohikukotodesu." after the pronunciation error correction into the Japanese sentence "My hobby is playing the guitar." (evaluation sentence) after the pronunciation error correction. Furthermore, since there are no word errors in the evaluation sentence "My hobby is playing the guitar.", the speech evaluation server 30 scores the Japanese sentence after the pronunciation error correction as "100 points".

Next, referring to FIG. 2, the functional components of the communication terminal 10 and the speech evaluation server 30 will be described.

The communication terminal 10 is, for example, a terminal configured to be able to communicate with a server. The communication terminal 10 is, for example, a smartphone, a tablet terminal, a PC, etc. When an application is executed, the communication terminal 10 collects the speech of the speaker S using a microphone or the like mounted thereon, and transmits the collected voice to the speech evaluation server 30 as speech voice data. The communication terminal 10 also displays the evaluation sentence, the mispronounced word, and the scoring result received from the speech evaluation server 30 on a screen such as a display mounted thereon. In the example shown in FIG. 1, the communication terminal 10 transmits the speech content "My hobby is playing the guitar" to the speech evaluation server 30 as speech voice data. Then, the communication terminal 10 acquires the evaluation sentence "My hobby is playing the guitar", the mispronounced word "bitter", and the scoring result "100 points" from the speech evaluation server 30, and displays them on the screen. The communication terminal 10 may also accept input of the pronunciation classification of consonants (described below), the native language of speaker S, and an expected character string, and transmit the input results to the speech evaluation server 30.

The speech evaluation server 30 has the following functional components: a memory unit 31, a voice recognition unit 32, a conversion unit 33, a calculation unit 34, a detection unit 35, a correction unit 36, a scoring unit 37, and an output unit 38.

The memory unit 31 stores the pronunciation classification of consonants, the native language of the speaker S, and the expected character string. The pronunciation classification of consonants is a combination of multiple consonants that are estimated to be prone to mispronunciation. In the example shown in FIG. 3, the pronunciation classification of consonants is set. For example, the consonants "r" and "l" are pronounced in a similar manner. Therefore, it is estimated that the consonants "r" and "l" are prone to mispronunciation. For example, a speaker may mistakenly pronounce "l" when trying to pronounce "r", or vice versa. Therefore, "r" and "l" are set to the same pronunciation classification. The pronunciation classification of consonants may be set in advance or may be obtained from the communication terminal 10.

The pronunciation classification of consonants may be set according to the native language of the speaker. Specifically, since the combination of multiple consonants that is estimated to be prone to mispronunciation differs for each language, the pronunciation classification of consonants is changed or added according to the native language of the speaker. In the example shown in FIG. 3, since a native English speaker is prone to mispronounce the consonant "t" as the consonant "s" or the consonant "r", if the native language of the speaker is English, the classifications "t, s" and "t, r" are added to the pronunciation classification of consonants. On the other hand, if the native language of the speaker is an Asian language, the classifications "z, j" and "d, r" are added to the pronunciation classification of consonants, since a native Asian speaker is prone to mispronounce the consonant "z" as the consonant "j" and the consonant "d" as the consonant "r".

The storage unit 31 stores the native languages of all possible speakers as the native language of the speaker S. The storage unit 31 stores each character string assumed as an answer to a predetermined question as an expected character string. Specifically, when speech evaluation is performed, the native language (e.g., English, Chinese, German, etc.) of a person who can be the speaker S is set in advance or acquired from the communication terminal 10 or the like. Since the content of the question in the speech evaluation is predetermined, the character string assumed as an answer to the question is also set in advance or acquired from the communication terminal 10 or the like. In the example shown in FIG. 1, the storage unit 31 stores the Roman alphabet text "gita-" representing the pronunciation of "guitar" as an expected character string as an answer to the predetermined question "My hobby is playing bitters." Other expected character strings in the example shown in FIG. 1 include, for example, the Roman alphabet text "ryokou" representing the pronunciation of "travel" and the Roman alphabet text "yakyu" representing the pronunciation of "baseball."

The voice recognition unit 32 acquires spoken voice data from the communication terminal 10. The voice recognition unit 32 performs voice recognition on the received spoken voice data, and passes the sentence resulting from the voice recognition to the conversion unit 33 and the scoring unit 37. In the example shown in FIG. 1, the voice recognition unit 32 performs voice recognition on the spoken voice data "My hobby is playing bitters" and generates the sentence "My hobby is playing bitters."

The conversion unit 33 acquires the results of speech recognition of the speaker S's speech, and converts the results of speech recognition into a character string representing the pronunciation. Specifically, the conversion unit 33 acquires a sentence that is the result of speech recognition from the speech recognition unit 32, and converts the sentence into a romanized character string representing the pronunciation. The conversion unit 33 passes the converted romanized character string representing the pronunciation to the calculation unit 34 and the correction unit 36. In the example shown in FIG. 1, the conversion unit 33 acquires the sentence "My hobby is playing bitters" that is the result of speech recognition from the speech recognition unit 32, and converts it into a romanized character string representing the pronunciation "syuumihabita-wohikukotodesu."

The calculation unit 34 calculates the edit distance between an expected string, which is a string representing the pronunciation of a word expected to appear in the speech of the speaker S, and one or more utterance strings included in the string representing the pronunciation converted by the conversion unit 33. Specifically, the calculation unit 34 acquires the expected string from the storage unit 31. The calculation unit 34 identifies one or more N-grams as utterance strings from the string representing the pronunciation converted by the conversion unit 33, where the number of characters in the expected string is N. Then, the calculation unit 34 calculates an error rate, which is a value obtained by dividing the edit distance between the expected string and the identified utterance string by the number of characters N. The calculation unit 34 calculates the error rate for each utterance string for all expected strings, and transfers all the error rate values to the detection unit 35. Note that the expected strings acquired by the calculation unit 34 may be all expected strings stored in the storage unit 31, or may be some of the expected strings stored in the storage unit 31 if candidates for the expected string can be narrowed down based on information related to the question or utterance.

1, the calculation unit 34 extracts character strings with the same number of characters as the expected character string "gita-" from the 27-character Roman alphabet sentence "syumihabita-wohikukotodesu." obtained from the conversion unit 33 while shifting the first character by one character and allowing overlaps, thereby generating 23 N-grams, "syumi," "yumih," "umiha," ..., "bita-," ..., "desu.", and identifies them as spoken character strings. The calculation unit 34 calculates the edit distance between the expected character string and each spoken character string in consideration of the pronunciation classification of the consonants, and determines the error rate as the value obtained by dividing the calculated edit distance by the number of characters, 5. For example, the calculation unit 34 calculates an edit distance of "5" between the expected character string "gita-" and the spoken character string "sumi", divides the calculated edit distance by the number of characters "5", and calculates an error rate of "1" for the spoken character string "sumi". Similarly, the calculation unit 34 calculates an error rate of "0.1" for the spoken character string "bita-" taking into account the pronunciation classification of the consonants (described below).

The calculation unit 34 also calculates the edit distance taking into consideration the pronunciation classification of the consonants included in the expected character string and the spoken character string. Specifically, when consonants included in the same pronunciation classification are replaced in the calculation of the edit distance, the calculation unit 34 calculates the edit distance with 0.5 steps per character replacement. Here, one step usually means that when a character string is transformed into another character string, one character replacement is performed once. In the example shown in FIG. 4, the calculation unit 34 calculates the edit distance between the expected character string "gita-" and the spoken character string "mita-", and the edit distance between the expected character string "gita-" and the spoken character string "bita-". To transform the expected character string "gita-" into the spoken character string "mita-", it is necessary to replace only the first character from the consonant "g" to the consonant "m". Therefore, the calculation unit 34 calculates the edit distance between the expected character string "gita-" and the spoken character string "mita-" to be "1". On the other hand, to transform the expected character string "gita-" into the spoken character string "bita-", it is necessary to replace only the first character from "g" to "b". Here, as shown in FIG. 3, the consonants "g" and "b" are in the same category in the consonant pronunciation classification, so the calculation unit 34 calculates the edit distance between the expected character string "gita-" and the spoken character string "bita-" as "0.5". Note that, in the calculation of the edit distance, when consonants in the same pronunciation classification are replaced, the number of steps required for replacing one character may be smaller than in other cases. For example, when consonants in the same pronunciation classification are replaced, the edit distance may be calculated with 0.3 steps for replacing one character.

The detection unit 35 detects, as a mispronounced string, a spoken string that has an edit distance equal to or less than a predetermined value and is not identical to an expected string among one or more spoken strings. Specifically, the detection unit 35 detects, as a mispronounced string, a spoken string that has the smallest error rate among the spoken strings whose error rate with a specific expected string has been calculated and that has an error rate greater than 0, if the error rate is equal to or less than a threshold for determining that the string is a mispronounced string. The detection unit 35 passes the detected mispronounced string to the correction unit 36. The detection unit 35 also converts the mispronounced string into a mispronounced word and passes it to the output unit 38. In the example shown in FIG. 1, among multiple spoken strings whose error rates with the expected string "gita-" have been calculated, the spoken string "bita-" has the smallest error rate, and its error rate of "0.1" is below the threshold value of "0.3" for determining a pronunciation error, so the detection unit 35 detects the spoken string "bita-" as a mispronounced string.

The correction unit 36 corrects the mispronounced character string detected by the detection unit in the character string representing the converted pronunciation to the assumed character string used in calculating the edit distance used to detect the mispronounced character string, and converts the character string representing the converted pronunciation to a sentence in the language used in the utterance. Specifically, the correction unit 36 acquires the converted character string from the conversion unit 33. The correction unit 36 corrects the mispronounced character string in the acquired converted character string to the assumed character string used in calculating the error rate used to detect the mispronounced character string. The correction unit 36 converts the corrected converted character string into a sentence (evaluation sentence) in the language used in the utterance of the speaker S. The correction unit 36 passes the evaluation sentence to the scoring unit 37 and the output unit 38.

In the example shown in FIG. 1, the correction unit 36 corrects the mispronounced character string "bita-" in the converted character string "syumihabita-wohikukotodesu." to the assumed character string "gita-" that was used in calculating the edit distance used to detect the mispronounced character string. The correction unit 36 converts the corrected converted character string "syumihagita-wohikukotodesu." into the Japanese sentence "My hobby is playing the guitar" (evaluation sentence), which is the language used in the utterance of the speaker S.

When the detection unit 35 detects a mispronounced character string, the scoring unit 37 scores the sentence converted by the correction unit 36, and when the detection unit 35 does not detect a mispronounced character string, the scoring unit 37 scores the result of the speech recognition. Specifically, when the detection unit 35 detects a mispronounced character string, the scoring unit 37 obtains the evaluation sentence from the correction unit 36. When the detection unit 35 does not detect a mispronounced character string, the scoring unit 37 obtains the sentence that is the result of the speech recognition from the speech recognition unit 32 and sets it as the evaluation sentence. In both cases, the scoring unit 37 scores the evaluation sentence. The scoring unit 37 passes the scoring result and the evaluation sentence to the output unit 38.

In the example shown in FIG. 1, the detection unit 35 detects a mispronounced character string, and the scoring unit 37 obtains the corrected sentence "My hobby is playing the guitar." from the correction unit 36. Since the corrected sentence "My hobby is playing the guitar." does not contain any errors in words, etc., the scoring unit 37 obtains a scoring result of "100 points."

The output unit 38 acquires the mispronounced word from the detection unit 35. The output unit 38 acquires the scoring result and the evaluation sentence from the scoring unit 37. The output unit 38 outputs the evaluation sentence, the mispronounced word, and the scoring result to an external device such as the communication terminal 10. In the example shown in FIG. 1, the output unit 38 outputs the mispronounced word "bitter", the scoring result "100 points", and the evaluation sentence "My hobby is playing the guitar."

Next, the process performed by the speech evaluation system 1 according to this embodiment, specifically, the process of correcting pronunciation errors in the speech of the speaker S acquired by the communication terminal 10 and scoring the speech, will be described with reference to FIG. 5. FIG. 5 is a flowchart showing the process performed by the speech evaluation system 1.

As shown in FIG. 5, in the speech evaluation system 1, the speech recognition unit 32 acquires speech data. The speech recognition unit 32 then performs speech recognition on the speech data, and generates a sentence that is the speech recognition result (step S101). The conversion unit 33 then acquires the speech recognition result and converts it into a character string that represents the pronunciation (step S102). Specifically, the conversion unit 33 acquires the sentence that is the speech recognition result and converts it into a romanized character string that represents the pronunciation.

Then, the calculation unit 34 generates a spoken string from the string representing the pronunciation (step S103). Specifically, the calculation unit 34 acquires an expected string from the storage unit 31. Then, the calculation unit 34 identifies one or more N-grams as a spoken string from the converted string, where N is the number of characters in the expected string. Next, the calculation unit 34 derives an edit distance between the spoken string and the expected string (step S104). Specifically, the calculation unit 34 divides the edit distance between the expected string and each spoken string by the number of characters N in the expected string, taking into account the pronunciation classification of the consonants included in the expected string and the spoken string, to calculate multiple error rates.

Then, the detection unit 35 determines whether or not there is a mispronounced character string in the multiple spoken character strings (step S105). Specifically, the detection unit 35 detects a spoken character string as a mispronounced character string when the error rate of the spoken character string with the smallest error rate and the error rate greater than 0 among the spoken character strings whose error rates with a specific assumed character string have been calculated is equal to or less than a threshold for determining that the character string is a mispronounced character string.

If the detection unit 35 detects a mispronounced character string (step S105: YES), the correction unit 36 corrects the mispronounced character string in the converted character string to the assumed character string used to calculate the error rate used to detect the mispronounced character string (step S106). The correction unit 36 then converts the corrected converted character string into a sentence in the language used in the utterance of the speaker S (evaluation sentence) (step S107). On the other hand, if the detection unit 35 does not detect a mispronounced character string (step S105: No), the scoring unit 37 obtains a sentence that is the result of the speech recognition from the speech recognition unit 32 and sets it as the evaluation sentence (step S108).

Then, the scoring unit 37 scores the evaluation sentence (step S109). Then, the output unit 38 outputs the evaluation sentence, the mispronounced words, and the scoring results (step S110).

Next, we will explain the effects of the speech evaluation system according to this embodiment.

The speech evaluation system 1 according to this embodiment includes a conversion unit 33 that acquires the results of speech recognition of a speaker S's speech and converts the results of speech recognition into a character string representing a pronunciation, a calculation unit 34 that calculates an edit distance between an expected character string, which is a character string representing the pronunciation of a word expected to appear in the speaker's speech, and one or more spoken character strings included in the character string representing the pronunciation converted by the conversion unit, and divides the edit distance by an error rate, a detection unit 35 that detects, among the one or more spoken character strings, a spoken character string whose error rate is equal to or less than a predetermined value and which is not identical to the expected character string, as a mispronunciation character string, and an output unit 38 that outputs the mispronunciation character string, and the calculation unit calculates the edit distance taking into account the pronunciation classification of consonants included in the expected character string and the spoken character string.

In the speech evaluation system 1 according to the present embodiment, the speech of the speaker S is converted into a character string representing a pronunciation, an edit distance between an expected character string, which is a character string representing the pronunciation of a word expected to appear in the speech of the speaker S, and one or more speech character strings included in the converted character string representing the pronunciation is calculated, an error rate is calculated from the edit distance, and among the one or more speech character strings, a speech character string whose error rate is equal to or less than a predetermined value and is not identical to the expected character string is detected as a mispronunciation character string. Here, in detecting an error in speech, for example, when the result of speech recognition is converted into a sentence and an error in speech is detected, a misuse of a word by the speaker and an error in pronunciation by the speaker are usually not distinguished from each other, and both are detected as errors in speech. In such a detection process, it is not possible to appropriately detect only the pronunciation error in the speaker's speech. In this regard, in the speech evaluation system 1 according to the present embodiment, the speech of the speaker S is converted into a character string representing the pronunciation, the edit distance between the speech string included in the character string and the expected character string is calculated, the error rate is calculated from the edit distance, and the speech string with the error rate equal to or less than a predetermined value is detected as the speech erroneous character string, so that the speech string estimated to have a small degree of error and a high probability of being a mere pronunciation error rather than a misuse of a word can be appropriately detected as the mispronunciation character string. Furthermore, in the speech evaluation system 1 according to the present embodiment, the edit distance between the expected character string and the speech string is calculated taking into consideration the pronunciation classification of the consonant. According to this configuration, for example, the edit distance is calculated so that the more similar the pronunciation string is to the expected character string in pronunciation, the smaller the edit distance between the expected character string and the speech string becomes, so that the more similar the pronunciation string is to the expected character string in pronunciation, the smaller the error rate between the expected character string and the speech string becomes. As a result, the detection unit can detect the speech string whose pronunciation is close to the expected character string (the string that is "highly likely to be a mere pronunciation error rather than a misuse of a word") as the mispronunciation character string. As described above, the speech evaluation system 1 according to this embodiment can appropriately detect pronunciation errors in a speaker's speech.

In the example shown in FIG. 6, the conventional technology and the technology according to the present embodiment are shown for calculating the edit distance between the expected word "guitar" and the spoken words "motah," "mitah," and "bitter." The expected word is a word that is expected to appear in an utterance, and the spoken word is a word that is spoken by speaker S. When the conventional technology is used, the edit distance between the expected word and each spoken word is all "1." Here, when the expected word is converted into an expected character string, and the spoken word is converted into a spoken character string, and the edit distance between each spoken character string and the expected character string "gita-" is calculated, the edit distance is "2" for the spoken character string "mоta-" and "1" for the spoken character strings "mita-" and "bita-." In this way, by taking into account the pronunciation of each word, the edit distance between the expected character string and the pronunciation character string that is more similar in pronunciation to the expected character string is smaller. Furthermore, when the technology according to the present embodiment is used to calculate the edit distance between each spoken string and the expected string "gita-" while taking into account the pronunciation classification of the consonants, the edit distance is "2" for the spoken string "mоta-", "1" for the spoken string "mita-", and "0.5" for the spoken string "bita-". In this way, by taking into account the pronunciation similarity of the consonants in the pronunciation of each word, the edit distance between the expected string and the expected string becomes smaller as the pronunciation string is more similar to the expected string in the pronunciation. As described above, when the technology according to the present embodiment is used, the edit distance is calculated such that the edit distance between the expected string and the pronunciation string is smaller as the pronunciation is more similar to the expected string. As a result, the error rate between the expected string and the pronunciation string is smaller as the pronunciation is more similar to the expected string, so that pronunciation errors in the speaker's speech can be appropriately detected.

The calculation unit 34 calculates the edit distance in consideration of the pronunciation classification of the consonants set according to the native language of the speaker S, and calculates the error rate from the edit distance. Here, the pronunciation similarity of the consonants differs depending on the native language of the speaker S. Therefore, according to this configuration, for example, the pronunciation similarity of the consonants according to the native language of the speaker S is taken into consideration, and the error rate is calculated so that the error rate with the expected string becomes smaller as the pronunciation string becomes more similar to the expected string. For example, when the native language of the speaker S is English, the consonants "r" and "t", which are similar in pronunciation to the speaker S whose native language is English, are added to the pronunciation classification of the consonants as additional pronunciation classifications, and the edit distance and error rate are calculated. In this way, it is possible to detect a spoken string whose pronunciation is close to the expected string (a string that is "more likely to be a mere pronunciation error rather than a misuse of a word") as a mispronounced string in consideration of the native language of the speaker.

The calculation unit 34 identifies one or more N-grams as spoken strings from the string representing the pronunciation converted by the conversion unit, where the number of characters in the expected string is N. Regarding detection of pronunciation errors, for example, if the string representing the converted pronunciation is divided into multiple strings of a predetermined number of characters to be used as spoken strings, the pronunciation error may not be properly detected if the pronunciation error spans multiple strings. In this regard, in the configuration according to the present embodiment, one or more N-grams are identified as spoken strings from the string representing the converted pronunciation. As a result, since the N-gram is a string of a predetermined number of characters extracted from the string representing the converted pronunciation while shifting the first character by one character and allowing overlaps, it is possible to detect all pronunciation errors in the speaker's speech.

In the character string representing the converted pronunciation, the mispronunciation character string detected by the detection unit 35 is replaced with the assumed character string used to calculate the edit distance used to detect the mispronunciation character string, and the character string representing the converted pronunciation is converted into a sentence in the language used in the utterance. If the detection unit 35 detects a mispronunciation character string, the scoring unit 37 scores the converted sentence in the correction unit 36, and if the detection unit 35 does not detect a mispronunciation character string, the scoring unit 37 scores the result of the speech recognition in the speech recognition unit 32. In this configuration, in the converted pronunciation character string, a character string that is "more likely to be a mere pronunciation error than a misuse of a word" is corrected to an assumed character string that is "a character string that is estimated to have been uttered if the pronunciation error had not occurred," and the sentence in which only the pronunciation error has been corrected is scored. This makes it possible to distinguish between a case in which the speaker S misused a word in an utterance and a case in which the speaker mispronounced a word despite using the word correctly, and to score the result.

The example shown in FIG. 7 shows the correction results and scoring results of the speech content in the conventional technology and the technology according to this embodiment. First, in the conventional technology, both "bitter" and "mottar" are detected as errors and are corrected. On the other hand, in the technology according to this embodiment, only "bitter" is estimated to be a mispronunciation of "guitar" and is corrected. Therefore, compared to the conventional technology, in the technology according to this embodiment, the speech of speaker S is scored more accurately because the mispronunciation is not mistaken for a misuse of the word.

The present invention is not limited to the above embodiment. Specifically, the language evaluated by the speech evaluation system 1 may be any language composed of vowels and consonants, and is not limited to Japanese. The character string converted by the conversion unit 33 may be any character string that represents a pronunciation, and may be expressed, for example, in phonetic symbols or the alphabet.

Next, the hardware configuration of the communication terminal 10 and the speech evaluation server 30 included in the speech evaluation system 1 will be described with reference to FIG. 8. The communication terminal 10 and the speech evaluation server 30 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

In the following description, the term "apparatus" may be interpreted as a circuit, device, unit, etc. The hardware configuration of the communication terminal 10 and the speech evaluation server 30 may be configured to include one or more of the devices shown in the figure, or may be configured to exclude some of the devices.

The functions of the communication terminal 10 and the speech evaluation server 30 are realized by loading specific software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations and control communication by the communication device 1004 and the reading and/or writing of data in the memory 1002 and storage 1003.

The processor 1001, for example, operates an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the control functions of the voice recognition unit 32 of the speech evaluation server 30, etc. may be realized by the processor 1001.

The processor 1001 also reads out programs (program codes), software modules, and data from the storage 1003 and/or the communication device 1004 into the memory 1002, and executes various processes according to these. The programs used are those that cause a computer to execute at least some of the operations described in the above embodiments.

For example, the control functions of the speech recognition unit 32 of the speech evaluation server 30 may be realized by a control program stored in the memory 1002 and running on the processor 1001, and the other functional blocks may be realized in a similar manner. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented in one or more chips. The program may be transmitted from a network via a telecommunications line.

The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), a RAM (Random Access Memory), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a wireless communication method according to one embodiment of the present invention.

Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including memory 1002 and/or storage 1003.

The communication device 1004 is hardware (transmitting/receiving device) for communicating between computers via a wired and/or wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc.

The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one configuration (e.g., a touch panel).

In addition, each device, such as the processor 1001 and memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured as a single bus, or may be configured as different buses between the devices.

The communication terminal 10 and the speech evaluation server 30 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these pieces of hardware.

Although the present embodiment has been described in detail above, it is clear to those skilled in the art that the present embodiment is not limited to the embodiment described in this specification. The present embodiment can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended as an illustrative example and does not have any restrictive meaning with respect to the present embodiment.

For example, the speech evaluation system 1 has been described as including a communication terminal 10 and a speech evaluation server 30, but is not limited to this, and each function of the speech evaluation system 1 may be realized only by the speech evaluation server 30.

Each aspect/embodiment described herein may be applied to systems utilizing LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), or other suitable systems and/or next generation systems based on and enhanced thereon.

The steps, sequences, flow charts, etc. of each aspect/embodiment described herein may be reordered unless inconsistent. For example, the methods described herein present elements of various steps in an example order and are not limited to the particular order presented.

The input and output information may be stored in a specific location (e.g., memory) or may be managed in a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be sent to another device.

The determination may be based on a value represented by one bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., with a predetermined value).

Each aspect/embodiment described in this specification may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Software, instructions, etc. may also be transmitted and received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies, such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber line (DSL), and/or wireless technologies, such as infrared, radio, and microwave, these wired and/or wireless technologies are included within the definition of a transmission medium.

The information, signals, etc. described herein may be represented using any one of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

Note that terms explained in this specification and/or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

In addition, the information, parameters, etc. described in this specification may be expressed as absolute values, as relative values from a predetermined value, or as corresponding other information.

A communications terminal may also be referred to by those skilled in the art as a mobile communications terminal, subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communications device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

As used herein, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

When designations such as "first," "second," and the like are used herein, any reference to that element is not intended to generally limit the quantity or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.

To the extent that the terms "include," "including," and variations thereof are used herein in the specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Further, the term "or" as used herein is not intended to be an exclusive or.

In this specification, multiple devices are included unless the context or technical basis clearly indicates that only one device exists.

Throughout this disclosure, the plural is intended to be included unless the context clearly indicates the singular.

S...speaker, 1...speech evaluation system, 10...communication terminal, 30...speech evaluation server, 31...storage unit, 32...speech recognition unit, 33...conversion unit, 34...calculation unit, 35...detection unit, 36...correction unit, 37...scoring unit, 38...output unit.

Claims (5)

a conversion unit that acquires a result of speech recognition of a speaker's speech and converts the result of speech recognition into a character string representing a pronunciation;
a calculation unit that calculates an edit distance between an expected character string that is a character string representing the pronunciation of a word that is expected to appear in the speech of the speaker and one or more spoken character strings included in a character string representing the pronunciation converted by the conversion unit;
a detection unit that detects, from among the one or more spoken character strings, a spoken character string whose edit distance is equal to or smaller than a predetermined value and which is not identical to the expected character string, as a mispronounced character string;
an output unit that outputs the mispronounced character string,
The calculation unit specifies, as the spoken character string, one or more N-grams from a character string representing the pronunciation converted by the conversion unit, where N is the number of characters in the expected character string ;
The calculation unit calculates the edit distance in consideration of pronunciation classifications of consonants included in the expected character string and the spoken character string. the calculation unit calculates the edit distance in consideration of a pronunciation classification of the consonants that is set according to the native language of the speaker.
The speech evaluation system according to claim 1 . a correction unit that corrects the mispronounced character string detected by the detection unit in the character string representing the converted pronunciation to the expected character string used in calculating the edit distance used to detect the mispronounced character string, and converts the character string representing the converted pronunciation into a sentence in a language used in the utterance,
The output unit outputs the sentence.
The speech evaluation system according to claim 1 or 2 . a scoring unit that, when the detection unit detects the mispronounced character string, scores the sentence converted by the correction unit, and, when the detection unit does not detect the mispronounced character string, scores a result of the speech recognition,
The output unit further outputs the scoring result in the scoring unit.
The speech evaluation system according to claim 3 . a conversion unit that acquires a result of speech recognition of a speaker's speech and converts the result of speech recognition into a character string representing a pronunciation;
a calculation unit that calculates an edit distance between an expected character string that is a character string representing the pronunciation of a word that is expected to appear in the speech of the speaker and one or more spoken character strings included in a character string representing the pronunciation converted by the conversion unit;
a detection unit that detects, from among the one or more spoken character strings, a spoken character string whose edit distance is equal to or smaller than a predetermined value and which is not identical to the expected character string, as a mispronounced character string;
a correction unit that corrects the mispronounced character string detected by the detection unit in the character string representing the converted pronunciation to the assumed character string used in calculating the edit distance used to detect the mispronounced character string, and converts the character string representing the converted pronunciation into a sentence in the language used in the utterance ;
an output unit that outputs the sentence ,
The calculation unit calculates the edit distance in consideration of pronunciation classifications of consonants included in the expected character string and the spoken character string.

Images