CN114882904A - Two-stage voice alignment method, electronic device, and storage medium - Google Patents

Abstract

The embodiment of the invention relates to a two-stage voice alignment method, electronic equipment and a storage medium, wherein the two-stage voice alignment method comprises the following steps: acquiring voice data and a word-level transcript; inputting the two into a word-level alignment model to obtain an initial word-level alignment result; traversing each frame of word-level frame data, merging adjacent frames belonging to the same word to obtain first merged frame data, and recording the starting time and the ending time of the first merged frame data; merging the first-class placeholder and the word-level frame data corresponding to the word adjacent to the first-class placeholder to obtain second merged frame data; updating the starting time and the ending time of the word to obtain a word-level alignment transcript; acquiring a first frequency spectrum characteristic sequence and an information vector; inputting the two and the word-level alignment transcript into a phoneme-level alignment model to obtain an initial phoneme-level alignment result; and traversing each frame of phoneme-level frame data, merging the second placeholder and the phoneme-level frame data corresponding to the phoneme unit, and acquiring a phoneme-level alignment transcript. In this way, no error occurs at the start position or the end position of the transcript.

Description

Two-level speech alignment method, electronic device, and storage medium

technical field

Embodiments of the present invention relate to the field of computer technologies, and in particular, to a two-level speech alignment method, an electronic device, and a storage medium.

Background technique

With the rapid development of artificial intelligence technology, the related technologies in the field of speech are becoming more and more mature, and the application of speech technology is becoming more and more extensive. As an important link in speech processing, speech alignment technology can provide important support for speech recognition, speech synthesis, speech evaluation and other technologies. The accuracy of speech alignment boundaries determines the accuracy of speech recognition and speech evaluation results, as well as the quality of speech synthesized audio.

In the current technology, due to the high sensitivity of the phoneme-level alignment model, the start position of the speech alignment result may be misplaced or the end position may be smeared. As a result, the final obtained transcript corresponding to the speech cannot be compared with the speech. There may be some errors in its complete correspondence.

SUMMARY OF THE INVENTION

The present application provides a two-level speech alignment method, electronic device, and storage medium to solve some or all of the above technical problems in the prior art.

In a first aspect, the present application provides a two-level speech alignment method, the method comprising:

Obtain speech data, and word-level transcripts corresponding to the speech data;

The speech data and the word-level transcript corresponding to the speech data are input into a pre-built word-level alignment model, and an initial word-level alignment result corresponding to the speech data is obtained, wherein the initial word-level alignment result includes multiple frames of word-level frame data, And each frame of word-level frame data corresponds to a word in the word-level transcript, or corresponds to a first-class placeholder, and a word in the word-level transcript corresponds to at least one frame of word-level frame data;

Traverse each frame of word-level frame data, merge adjacent frames belonging to the same word, obtain the first merged frame data, and record the start time and end time of the first merged frame data;

According to the first preset rule, the first type of placeholder is combined with the word-level frame data corresponding to the word immediately adjacent to the first type of placeholder to obtain the second combined frame data;

Update the start time and end time of the second merged frame data, and finally obtain the word-level alignment transcript corresponding to the voice data;

Obtain the first spectral feature sequence corresponding to the voice data, and the information vector corresponding to the voice data;

Input the word-level alignment transcript, the first spectral feature sequence and the information vector into the pre-built phoneme-level alignment model, and obtain the initial phoneme-level alignment result corresponding to the speech data. The initial phoneme-level alignment result includes multiple frames of phoneme-level frame data , each frame of phoneme-level frame data corresponds to a phoneme unit, or corresponds to a second-type placeholder;

Traverse each frame of phoneme-level frame data, merge at least one second-type placeholder adjacent to the phoneme-level frame data corresponding to the phoneme unit in the initial phoneme-level alignment result with the phoneme-level frame data corresponding to the phoneme unit, and obtain the first Three merged frame data;

The start time and end time of the third merged frame data are recorded as the start time and end time corresponding to the phoneme unit, and finally a phoneme-level aligned transcript corresponding to the speech data is obtained.

Optionally, according to the first preset rule, the first type of placeholder is merged with the word-level frame data corresponding to the word immediately adjacent to the first type of placeholder to obtain the second merged frame data, which specifically includes:

When the left side of the first type of placeholder is the word-level frame data corresponding to the word, the first type of placeholder is merged with the word-level frame data corresponding to the word on the left, and the end of the first type of placeholder is Time is the end time of the second merged frame, and the start time of the word-level frame data of the word on the left is the start time of the second merged frame;

Or, when the first type of placeholder is the word-level frame data corresponding to the word on the right side, the first type of placeholder is merged with the word-level frame data corresponding to the right side of the word, and the first type of placeholder is combined. The start time of the second merged frame is taken as the start time of the second merged frame, and the right side is the end time of the word-level frame data corresponding to the word as the end time of the second merged frame.

In a second aspect, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

memory for storing computer programs;

The processor is configured to implement the steps of the two-level speech alignment method of any one of the embodiments of the first aspect when executing the program stored in the memory.

A third aspect provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the two-level speech alignment method according to any one of the embodiments of the first aspect.

Compared with the prior art, the above-mentioned technical solutions provided in the embodiments of the present application have the following advantages:

The method provided by the embodiment of the present application is used for speech data and word-level transcripts corresponding to the speech data. Then the speech data and the word-level transcript corresponding to the speech data are input into the pre-built word-level alignment model, and then the initial word-level alignment result is obtained. Traverse each frame of word-level frame data in the initial word-level alignment result, and merge the frame data corresponding to the word in the word-level frame data and the first type of placeholders respectively. Placeholder for non-voice data. Allocate the time of the placeholder in the speech data to the adjacent words to ensure that the speech of each word in the speech data can correspond to the words in the transcript, and achieve word-level transcript alignment.

On this basis, each word in the word-level transcript is used as the phoneme transcript, the first spectral feature sequence corresponding to the speech data, and the information vector corresponding to the speech data are input into the pre-built phoneme-level alignment model to obtain the initial Phoneme-level alignment results. Then, each frame of phoneme-level frame data is traversed, and a process of merging the phoneme-level frame data is performed, so as to merge the second-type placeholder and the frame data corresponding to the adjacent phonemes to obtain a merged frame. The second type of placeholders are also phoneme-level placeholders corresponding to non-speech data. The start time and end time of the merged frame are used as the start time and end time of the phoneme, and in this way, the phoneme-level transcript alignment is achieved. Above, because the time corresponding to all non-speech data has been reasonably allocated to the phoneme-level frame data corresponding to the adjacent phoneme-level transcript, that is, the time corresponding to the non-speech data is allocated to the adjacent phoneme-level frame data. In terms of phonemes, in this way, the obtained phoneme-level transcripts will not be misplaced at the beginning of the speech alignment result or tailed at the end position, and it is completely guaranteed that the final phoneme-level transcripts obtained can be compared with the speech itself. Basically corresponds to the maximum possible elimination error.

Description of drawings

1 is a schematic flowchart of a two-level speech alignment method provided by an embodiment of the present invention;

2 is a schematic diagram of the effect of a word-level aligned transcript provided by an embodiment of the present invention;

3 is a schematic flowchart of a third method for merging frame data according to an embodiment of the present invention;

4 is a schematic diagram of the effect of obtaining a phoneme-level aligned transcript provided by an embodiment of the present invention;

5 is a schematic diagram of the overall flow of the method for constructing a word-level alignment model provided by the present invention;

6 is a schematic diagram of the overall flow of the method for constructing a phoneme-level alignment model provided by the present invention;

FIG. 7 is a schematic structural diagram of a two-level speech alignment apparatus according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In order to facilitate the understanding of the embodiments of the present invention, further explanation will be given below with specific embodiments in conjunction with the accompanying drawings, and the embodiments do not constitute limitations to the embodiments of the present invention.

In response to the technical problems mentioned in the background art, an embodiment of the present application provides a two-level speech alignment method, as shown in FIG. 1 . FIG. 1 is a schematic flowchart of a two-level speech alignment method provided by an embodiment of the present invention. , the method steps include:

Step 110: Acquire voice data and a word-level transcript corresponding to the voice data.

Specifically, the voice data is audio data collected through a sound pickup device. Word-level transcripts can be pre-set or post-annotated transcripts. Pre-set, for example, a book, or a paragraph of lines, etc., any transcript will do. Then, find the corresponding personnel to record the transcripts such as books or lines to obtain the voice data.

Step 120: Input the speech data and the word-level transcript corresponding to the speech data into a pre-built word-level alignment model, and obtain an initial word-level alignment result corresponding to the speech data.

Specifically, the word-level alignment model is a pre-built model, such as a pre-built end-to-end neural network model. In the following, the construction process of the word-level alignment model will be described in detail, and will not be explained too much here.

After the word-level alignment model, the obtained initial word-level alignment result can include multiple frames of word-level frame data, and each frame of word-level frame data corresponds to a word in the word-level transcript, or corresponds to a first-class Placeholder, a word in the word-level transcript corresponds to at least one frame of word-level frame data.

For example, the voice data is "non-voice data + hello". For non-speech data, such as noise, the corresponding word-level transcript is that the first to fifth frames of data are noise, and the noise data is represented in the form of the first type of placeholders. The word "you" corresponds to the 6th frame data to the 7th frame data, and the word "good" corresponds to the 8th frame data to the 9th frame data. In order to distinguish it from the phoneme-level frame data below, the frame data output by the word-level alignment model is defined as word-level frame data. In essence, it is just a frame of data. In addition, in the initial word-level alignment results, a word-level placeholder can be inserted at the beginning of the transcript, at the end, and between words. The reason for adding word-level placeholders in these positions is to consider that in the initial word-level alignment results that may be obtained, not every frame of data can correspond to the corresponding word, and some places may not correspond to the pronunciation position, so it is necessary to In the form of word-level placeholders.

Step 130: Traverse each frame of word-level frame data, merge adjacent frames belonging to the same word, acquire first merged frame data, and record the start time and end time of the first merged frame data.

Step 140 , according to the first preset rule, combine the first type of placeholder with the word-level frame data corresponding to the word immediately adjacent to the first type of placeholder to obtain the second combined frame data.

Step 150: Update the start time and end time of the second merged frame data, and finally obtain a word-level aligned transcript corresponding to the speech data.

Specifically, as described above, the initial word-level alignment result includes multiple frames of word-level frame data. Each frame of word-level frame data is mapped to a word or word-level placeholder. According to the decoding rules of CTC (Connectionist TemporalClassification), multi-frame input corresponds to one output, for example, the word "you" corresponds to the 6th frame data and the 7th frame data. Therefore, it is necessary to traverse each frame of word-level frame data, merge adjacent frames belonging to the same word, and obtain the first merged frame data.

At the same time, it is also necessary to combine the first type of placeholder with the word-level frame data corresponding to the word immediately adjacent to the first type of placeholder according to the first preset rule, to obtain the second combined frame data.

Specifically, according to the first preset rule, the first type of placeholder is merged with the word-level frame data corresponding to the word immediately adjacent to the first type of placeholder, and the implementation of obtaining the second merged frame data is as follows:

When the left side of the first type of placeholder is the word-level frame data corresponding to the word, the first type of placeholder is merged with the word-level frame data corresponding to the word on the left, and the end of the first type of placeholder is Time is the end time of the second merged frame, and the start time of the word-level frame data of the word on the left is the start time of the second merged frame;

Or, when the first type of placeholder is the word-level frame data corresponding to the word on the right side, the first type of placeholder is merged with the word-level frame data corresponding to the right side of the word, and the first type of placeholder is combined. The start time of the second merged frame is taken as the start time of the second merged frame, and the right side is the end time of the word-level frame data corresponding to the word as the end time of the second merged frame.

For example, as mentioned above, frames 1 to 5 are placeholders of the first type, and frame 6 is the word "you" (assuming that the two frames of word-level frame data corresponding to the current word "you" have been merged , which constitutes the first combined frame, and the first combined frame is currently the 6th frame data).

In an optional embodiment, the first type placeholder of the fifth frame may be first merged with the word-level frame data corresponding to the word "you" in the first merged frame. Then, merge the first-type placeholders of the fourth frame with the word-level frame data corresponding to the current "you" word, and so on, merge all the first-type placeholders with the "you" word.

In the second optional embodiment, the word-level placeholders corresponding to the first five frames can be combined into one frame of word-level frame data, and then the combined word-level frame data can be combined with the "word-level frame data" of the first combined frame. The word-level frame data corresponding to the "you" word is merged.

In the third optional example, it is assumed that the sixth frame and the seventh frame in the initial word-level alignment result have not been merged, that is, the word "you" still currently corresponds to two frames of word-level frame data. The first 5 frames are all first-class placeholders.

Then, the first 5 frames of the first type of placeholders can be merged with the first type of placeholders as described in either of the first or second type described above, and then merged with the sixth frame " The word-level frame data corresponding to the word "you" is combined, and then the combined frame is combined with the word-level frame data of the seventh frame to obtain the final word-level frame data corresponding to the word "you".

Considering that there is a first type of placeholder between the word-level frame data corresponding to the word "you" and the word-level frame data corresponding to the word "good".

Then this first type of placeholder can be set to merge with the word-level frame data corresponding to the "word" on the left, or can be set to merge with the word-level frame data corresponding to the "word" on the right. The specific merging method can be set according to the actual situation. Similarly, suppose that the word "好" also has a first-class placeholder, and there are no other words after the word "好", so the first-class placeholder after the word "好" is the word corresponding to the word "好" Level frame data merging.

In addition, while merging the word-level frame data, it is also necessary to determine the start time and the end time of the word-level frame data after merging. For example, the start time corresponding to the word "you" mentioned above is the start time corresponding to the first frame of word-level frame data. The end time is the end time of the last frame of word-level frame data corresponding to the word "you" (at this moment, it is assumed that the first type of placeholder behind the word-level frame data corresponding to the word "you" is the word corresponding to the word "good" level frame data merging).

It should be noted that the execution order of step 140 and step 150 is not fixed. It can be switched freely according to the actual situation. For details on how to switch, please refer to the several situations described above.

In a word, in this way, after combining all the word-level frame data in the above-mentioned manner, the word-level aligned transcript corresponding to the speech data can be obtained.

FIG. 2 shows the word-level aligned transcript obtained according to the above method, see FIG. 2 for details. Figure 2 takes "noise + cooling mode + noise" as an example for description. Therefore, the first type of word-level placeholders corresponding to the first half of the noise are combined with the word-level frame data corresponding to the "system" word to obtain the start time and end time of the "system" word. For others, please refer to the introduction above, and will not repeat them here.

Step 160: Obtain a first spectral feature sequence corresponding to the speech data and an information vector corresponding to the speech data.

Specifically, after the voice data is obtained, feature extraction may be performed on the voice data to obtain a first spectral feature sequence. The specific extraction process is the prior art. In simple terms, it is to first extract the time-frequency signal corresponding to the speech signal and convert it into a frequency-domain signal, and then perform feature extraction on the frequency-domain signal to obtain the first spectral feature, which is not detailed here. illustrate. The information vector corresponding to the speech data is also obtained through the prior art, and details are not repeated here.

Step 170: Input the word-level alignment transcript, the first spectral feature sequence and the information vector into a pre-built phoneme-level alignment model to obtain an initial phoneme-level alignment result corresponding to the speech data.

Specifically, similar to the initial word-level alignment result, the initial phone-level alignment result includes multiple frames of phone-level frame data, and each frame of phone-level frame data corresponds to a phone unit or a second-type placeholder. However, the specific construction process of the pre-built phoneme-level alignment model is also described in detail below, and will not be explained too much here. Currently, just to introduce its functionality, the phoneme-level alignment model can be, for example, a Gaussian mixture model.

Step 180, traverse each frame of phoneme-level frame data, and combine at least one second-type placeholder adjacent to the phoneme-level frame data corresponding to the phoneme unit in the initial phoneme-level alignment result with the phoneme-level frame data corresponding to the phoneme unit. , to obtain the third merged frame data.

Step 190: Record the start time and end time of the third merged frame data as the start time and end time corresponding to the phoneme unit, and finally obtain the phoneme-level alignment transcript corresponding to the speech data.

Specifically, in an optional example, each frame of phoneme-level frame data is traversed, and at least one second-type placeholder adjacent to the phoneme-level frame data corresponding to the phoneme unit in the initial phoneme-level alignment result is compared with the phoneme. The phoneme-level frame data corresponding to the unit is merged, and the specific implementation process of acquiring the third merged frame data can be referred to as shown in Figure 3, and the method steps include:

Step 310, when the phoneme-level frame data is adjacent to at least one second-type placeholder, merge the at least one second-type placeholder, obtain the fourth merged frame data, and determine the start time of the fourth merged frame data and end time.

Step 320: Combine the phoneme-level frame data and the fourth combined frame data to obtain the third combined frame data.

Specifically, the second type of placeholder is actually an ordinary placeholder similar to the first type of placeholder. Its function is to distinguish phoneme-level (or word-level) frame data corresponding to speech data and non-speech data.

In an optional example, the second type of placeholders may include phoneme-level placeholders and/or silence markers.

Then, when the phoneme-level frame data is adjacent to at least one second-type placeholder, the at least one second-type placeholder is merged to obtain the fourth merged frame data, which may specifically include the following situations.

First, when the second type of placeholders only include phoneme-level placeholders, after converting each phoneme-level placeholder into silence marks, the silence marks are merged to obtain fourth merged frame data.

Second, when the second type of placeholders include phoneme-level placeholders and silence placeholders in sequence (the two placeholders are adjacent), the phoneme-level placeholders are combined with their adjacent silence placeholders. , merge all silence placeholders, and obtain the fourth merged frame data.

Further optionally, record the start time and end time of the third merged frame data as the start time and end time corresponding to the phoneme unit, specifically including the following method steps:

When the fourth merged frame data is on the left side of the phoneme-level frame data, the start time of the fourth merged frame data is used as the start time of the third merged frame data, and the end time of the phoneme-level frame data is used as the third merge. The end time of the frame data;

Or, when the fourth merged frame data is on the right side of the phoneme-level data, the start time of the phoneme-level data is used as the start time of the third merged frame data, and the end time of the fourth merged frame data is used as the third merged data. End time of frame data.

In the following examples, the second type of placeholders include phoneme-level placeholders and mute marks as an example for description.

In a specific example, it is assumed that when the second type of placeholders include phoneme-level placeholders and silence markers, and the combination order is phoneme-level placeholders and silence markers, the phoneme-level placeholders need to be converted into silences first. mark and then merge it with the right adjacent silence into a single silence marker, along with the boundary time.

In another specific example, it is assumed that when the second type of placeholders include phoneme-level placeholders and silence markers, and the combination order is silence markers and phoneme-level placeholders, the phoneme-level placeholders need to be converted into Silence marker, then merges it with the left-adjacent silence into a single silence marker, also merging the boundary times.

In another specific example, it is assumed that when the second type of placeholders include phoneme-level placeholders and silence markers, and the combination sequence is silence markers, phoneme-level placeholders, and silence markers, the phoneme-level placeholders need to be placed first. The symbol is converted into a silence marker, which is then merged with the left-adjacent silence marker, the right-adjacent silence marker into a single silence marker, and the boundary times are merged at the same time.

In the above example, although steps are involved in converting phoneme-level placeholders to silence markers. But in practice, this is not actually necessary. For example, the second type of placeholder only includes phoneme-level placeholders, then the phoneme-level placeholders are directly merged, and then the phoneme-level frame data corresponding to the adjacent phoneme units can be merged (including the merging of time) without conversion. Mark for silence. Only when there are two types of phoneme-level placeholders and silence markers, it is better to unify them into the same type, and then combine them with the phoneme-level frame data corresponding to the phoneme unit, which is more efficient in terms of the calculation process.

Of course, it is not necessary to convert phoneme-level placeholders to silence markers, and vice versa. In short, the purpose of the above operations is to improve the efficiency of merging frame data. Therefore, whether it is to convert the phoneme-level placeholders to silence markers first, and then merge all silence markers, or directly merge the phoneme-level placeholders with phoneme-level frame data, and then merge the phoneme-level frame data and silence markers, also Or merge them in order. For example, phoneme-level alignment transcripts include: phoneme-level placeholders, phoneme-level placeholders, phoneme-level frame data, silence marks, and phoneme-level placeholders. Then, the frame data at the phoneme level is merged, then it is merged with the silence marker, and finally it is merged with the placeholder at the phoneme level.

The above and so on are all within the practicable scope of this embodiment, and the specific implementation manners will not be illustrated one by one here.

FIG. 4 illustrates an effect diagram of obtaining a phoneme-level aligned transcript by using the above method steps of the embodiment of the present application. Specifically, referring to FIG. 4 , in FIG. 4 , “non-voice data + cooling mode + non-voice data” is also used as voice data, and the effect diagram obtained after executing the above operation steps. It can be seen from the whole process that phoneme-level alignment can be achieved.

Further optionally, after obtaining the phoneme-level aligned transcript corresponding to the speech data, the method may further include:

The word-level aligned transcripts corresponding to the phonetic data are optimized using the phoneme-level aligned transcripts corresponding to the phonetic data.

Specifically, after the phoneme-level alignment transcript is obtained, in fact, for the speech data, each word in the speech data and the frame data corresponding to its pinyin are composed of the frame data corresponding to the phoneme. It is equivalent to having completed the alignment operation in a finer granularity. Then, using the phoneme combination to complete the pinyin alignment operation will be more accurate, that is, the optimization of the word-level alignment transcript is completed. The word-level alignment operation is optimized by taking the start time of the first phoneme corresponding to each pinyin as the start time of the pinyin, and the end time of the last phoneme of the pinyin as the end time of the pinyin.

Further optional, as can be seen from the above description, when acquiring the word-level alignment transcript, the speech data and the word-level transcript corresponding to the speech data need to be input into the pre-built word-level alignment model. When obtaining the phoneme-level alignment transcript, the word-level alignment transcript, the first spectral feature sequence, and the information vector need to be input into a pre-built phoneme-level alignment model.

In the following, how to build a word-level alignment model and a phoneme-level alignment model will be explained in detail. See Figure 5 and Figure 6 for details. Figure 5 shows the process of building a word-level alignment model, and Figure 6 shows the process of building a phoneme-level alignment model.

First, as shown in Figure 5, the method steps include:

Step 510: Acquire multiple pieces of speech sample data, and acquire a word-level transcript corresponding to each piece of speech sample data.

Specifically, the multiple pieces of voice sample data are voice sample data of multiple people, and each person corresponds to at least two pieces of voice sample data. For the specific acquisition process, refer to the above process of acquiring the voice data and the word-level transcript corresponding to the voice data, which will not be repeated here.

Step 520: Perform feature vector extraction on a plurality of pieces of speech sample data of the ith person respectively to obtain the ith information vector.

Step 530: Perform frequency domain feature extraction on a plurality of pieces of speech sample data of the ith person respectively, and obtain a second spectral feature sequence corresponding to each piece of speech sample data.

Specifically, the process of acquiring the feature vector, and the process of extracting the frequency domain feature can be acquired through the prior art, which will not be described here.

Step 540, according to each second spectral feature sequence, the word-level transcript corresponding to each piece of speech sample data, and the i-th information vector, iteratively train the word-level alignment model until the word-level alignment model reaches the first preset. When standard, get the final obtained word-level alignment model.

Specifically, the model can be trained according to the iterative training process of the conventional neural network model. However, considering the influence of reducing the channel and additive noise, before the iterative training of the word-level alignment model, each second spectral feature sequence can be normalized separately to obtain the same value as each second spectral feature sequence. The third spectral feature sequence corresponding to the sequence conforming to the normal distribution;

Then, sort the third spectral feature sequence according to the sequence length; finally, according to the sorting order, during the jth training, the third spectral feature sequence in the jth position and the word level corresponding to each piece of speech data are sorted. The transcript, and the i-th information vector are input into the word-level alignment model to complete the j-th training of the word-level alignment model.

In a specific example, the third spectral feature sequence needs to be sorted in descending order. The reason why the third spectral feature sequence is sorted according to the sequence length, and then the jth training is performed in sequence according to the sorting order, is to provide the model with a step-by-step process and improve the robustness of the model. The reason why the speaker's information vector is added is to improve the generalization of the model.

In an optional example, determining whether the word-level alignment model meets the first preset standard may be determined according to a preconfigured loss function. The loss gets smaller as the time and the number of iterations increase. When the loss is less than a preset threshold, the standard is reached. Alternatively, after the number of iterations reaches the preset number of times, the training is stopped. After the preset number of training times is reached by default, the obtained model is the final desired word-level alignment model, where i and j are both positive integers.

Figure 6 shows the construction process for the phoneme-level alignment model, and the details are as follows:

Step 610: Acquire multiple pieces of speech sample data, and acquire a word-level transcript corresponding to each piece of speech sample data.

The word-level transcript is obtained according to the method steps corresponding to FIG. 6 . The multiple pieces of voice sample data are voice sample data of multiple people, and each person corresponds to at least two pieces of voice sample data.

Step 620: Perform frequency domain feature extraction on each piece of speech sample data, to obtain a fourth spectral feature sequence corresponding to each piece of speech sample data.

Specifically, the feature extraction process is in the prior art, and details are not repeated here.

Step 630: Perform normalization processing on each fourth spectral feature sequence, respectively, to obtain a fifth spectral feature sequence corresponding to each fourth spectral feature sequence and conforming to the normal distribution.

Specifically, the purpose of normalizing the fourth spectral feature sequence is also to reduce the influence of the channel and additive noise.

Step 640 , constructing a data set from each fifth spectral feature sequence, the word-level transcript and phoneme-level transcript corresponding to each fifth spectral feature sequence.

For example, these three kinds of data are bound to generate unique ID information.

In step 650, all the data sets are randomly divided into a plurality of subsets, and each subset includes a data set corresponding to each of the plurality of persons.

Specifically, each subset includes a data set corresponding to each of the multiple persons, in order to ensure that the model is more robust.

Step 660, using multiple subsets to iteratively train the phoneme-level alignment model in turn, and ending when the phoneme-level alignment model reaches the second preset standard, and obtain the final phoneme-level alignment model, that is, a pre-built phoneme-level alignment model.

Specifically, the subset training generated model in the previous step is used as the initialization model for the next subset training, but the model in the first step is randomly initialized. The final step is iterative training using all the training data. According to the subset data from small to large, it is entered into the phoneme-level alignment model for alignment training, so that the model has a step-by-step process and improves its complexity.

In addition to the training process, it also includes the testing process of conventional neural network models, which will not be repeated here.

Further optionally, in addition to the above operations, the method may further include: constructing a word-level pronunciation dictionary and a phoneme-level pronunciation dictionary. Among them, when building a word-level pronunciation dictionary, the transcripts in the speech sample data can be split into word-level transcripts to generate word-level transcripts. After de-duplicating all the words in the transcripts, a word-level pronunciation dictionary can be generated. This process does not require Word segmentation is required.

When building a phoneme pronunciation dictionary, since each word may have multiple pronunciations, each word may map multiple phoneme combinations. The word-level transcript corresponding to the speech template data can generate phoneme-level transcripts according to semantics and pronunciation rules. In a specific scenario, each word basically has its fixed pronunciation and corresponding meaning, and each word corresponds to a unique pair of phonemes Combination, in which the phonemes are separated by spaces, which is convenient for program processing.

When using the word-level alignment model to generate the initial word-level alignment results above, it can be assisted by the word-level pronunciation dictionary. Similarly, when using the phoneme-level alignment model to generate the initial phoneme-level alignment results, it can also be assisted by the phoneme-level pronunciation dictionary.

In the above-mentioned initial word-level alignment results and phoneme-level alignment results, the word-level frame data or phoneme-level frame data involving tones also carry corresponding tonal annotations. For example, "Hello" in the phoneme-level alignment result is n i3 h ao3, and the 3 in it is the tone.

The two-level speech alignment method provided by the embodiment of the present invention is for speech data and word-level transcripts corresponding to the speech data. Then the speech data and the word-level transcript corresponding to the speech data are input into the pre-built word-level alignment model, and then the initial word-level alignment result is obtained. Traverse each frame of word-level frame data in the initial word-level alignment result, and merge the frame data corresponding to the word in the word-level frame data and the first type of placeholders respectively. Placeholder for non-voice data. Allocate the time of the placeholder in the speech data to the adjacent words to ensure that the speech of each word in the speech data can correspond to the words in the transcript, and achieve word-level transcript alignment.

On this basis, each word in the word-level transcript is used as the phoneme transcript, the first spectral feature sequence corresponding to the speech data, and the information vector corresponding to the speech data are input into the pre-built phoneme-level alignment model to obtain the initial Phoneme-level alignment results. Then, each frame of phoneme-level frame data is traversed, and a process of merging the phoneme-level frame data is performed, so as to merge the second-type placeholder and the frame data corresponding to the adjacent phonemes to obtain a merged frame. The second type of placeholders are also phoneme-level placeholders corresponding to non-speech data. The start time and end time of the merged frame are used as the start time and end time of the phoneme, and in this way, the phoneme-level transcript alignment is achieved. Above, because the time corresponding to all non-speech data has been reasonably allocated to the phoneme-level frame data corresponding to the adjacent phoneme-level transcript, that is, the time corresponding to the non-speech data is allocated to the adjacent phoneme-level frame data. In terms of phonemes, in this way, the obtained phoneme-level transcripts will not be misplaced at the beginning of the speech alignment result or tailed at the end position, and it is completely guaranteed that the final phoneme-level transcripts obtained can be compared with the speech itself. Basically corresponds to the maximum possible elimination error.

The above are several method embodiments of the two-level speech alignment provided by the present application, and the following describes other embodiments of the two-level speech alignment provided by the present application. For details, refer to the following.

FIG. 7 is a two-level speech alignment apparatus provided by an embodiment of the present invention. The apparatus includes: an acquisition module 701 , a processing module 702 , a traversal module 703 , a recording module 704 , and an update module 705 .

The acquiring module 701 is configured to acquire speech data and word-level transcripts corresponding to the speech data.

The processing module 702 is used to input the speech data and the word-level transcript corresponding to the speech data into a pre-built word-level alignment model, and obtain an initial word-level alignment result corresponding to the speech data, wherein the initial word-level alignment result includes multiple Frame word-level frame data, and each frame of word-level frame data corresponds to a word in the word-level transcript, or corresponds to a first-class placeholder, and a word in the word-level transcript corresponds to at least one frame of word-level frame data;

Traversing module 703, for traversing each frame of word-level frame data;

The processing module 702 is further configured to merge adjacent frames belonging to the same word to obtain the first merged frame data;

Recording module 704, for recording the start time and end time of the first merged frame data;

The processing module 702 is further configured to merge the first type of placeholder with the word-level frame data corresponding to the word immediately adjacent to the first type of placeholder according to the first preset rule, and obtain the second merged frame data;

The update module 705 is used to update the start time and the end time of the second merged frame data, and finally obtains the word-level alignment transcript corresponding to the voice data;

The obtaining module 701 is further configured to obtain the first spectral feature sequence corresponding to the voice data and the information vector corresponding to the voice data;

The processing module 702 is also used to input the word-level alignment transcript, the first spectral feature sequence and the information vector into the pre-built phoneme-level alignment model, and obtain the initial phoneme-level alignment result corresponding to the speech data, and the initial phoneme-level alignment result. Including multiple frames of phoneme-level frame data, each frame of phoneme-level frame data corresponds to a phoneme unit, or corresponds to a second-type placeholder;

Traversing module 703, further configured to traverse each frame of phoneme-level frame data;

The processing module 702 is further configured to combine at least one second type placeholder adjacent to the phoneme-level frame data corresponding to the phoneme unit in the initial phoneme-level alignment result with the phoneme-level frame data corresponding to the phoneme unit, and obtain a third combination frame data;

The recording module 704 is further configured to record the start time and end time of the third merged frame data as the start time and end time corresponding to the phoneme unit, and finally obtain the phoneme-level aligned transcript corresponding to the speech data.

Optionally, the apparatus further includes: an optimization module 706, configured to optimize the word-level aligned transcript corresponding to the speech data by using the phoneme-level aligned transcript corresponding to the speech data.

Optionally, the processing module 702 is specifically configured to: when the phoneme-level frame data is adjacent to at least one second-type placeholder, merge the at least one second-type placeholder, obtain the fourth merged frame data, and determine The start time and end time of the fourth merged frame data;

The phoneme-level frame data and the fourth merged frame data are merged to obtain the third merged frame data.

Optionally, the recording module 704 is specifically configured to use the start time of the fourth merged frame data as the start time of the third merged frame data when the fourth merged frame data is on the left side of the phoneme-level frame data, and use the phoneme as the start time of the third merged frame data. The end time of the level frame data, as the end time of the third merged frame data;

Or, when the fourth merged frame data is on the right side of the phoneme-level data, the start time of the phoneme-level data is used as the start time of the third merged frame data, and the interpretation time of the fourth merged frame data is used as the third merged time. End time of frame data.

Optionally, the second type of placeholders include phoneme-level placeholders and/or silence markers; the processing module 702 is specifically used for:

When the second type of placeholders all include phoneme-level placeholders, after converting each phoneme-level placeholder into silence marks, merge the silence marks to obtain the fourth merged frame data;

Or, when the second type of placeholders includes phoneme-level placeholders and silence placeholders, after merging the phoneme-level placeholders with their adjacent silence placeholders, merge all silence placeholders to obtain the fourth Merge frame data.

Optionally, the obtaining module 701 is further configured to obtain multiple pieces of voice sample data, and obtain word-level transcripts corresponding to each piece of voice sample data; wherein, the multiple pieces of voice sample data are voice sample data of multiple people, and each One person corresponds to at least two voice sample data;

The processing module 702 is also used to extract the feature vector respectively for the multiple pieces of speech sample data of the ith person to obtain the ith information vector;

Perform frequency domain feature extraction on multiple pieces of speech sample data of the i-th person respectively, and obtain a second spectral feature sequence corresponding to each piece of speech sample data;

According to each second spectral feature sequence, the word-level transcript corresponding to each piece of speech sample data, and the i-th information vector, the word-level alignment model is iteratively trained until the word-level alignment model reaches the first preset standard. The word-level alignment model finally obtained, that is, the pre-built word-level alignment model, where i is a positive integer.

Optionally, the processing module 702 is further configured to perform normalization processing on each second spectral feature sequence respectively, and obtain a third spectral feature sequence corresponding to each second spectral feature sequence and conforming to the normal distribution;

sorting the third spectral feature sequence according to the sequence length;

According to the sorting order, in the jth training, the third spectral feature sequence sorted in the jth position, the word-level transcript corresponding to each piece of speech data, and the i-th information vector are input into the word-level alignment model, Used to complete the jth training of the word-level alignment model, where j is a positive integer.

Optionally, the obtaining module 701 is further configured to obtain multiple pieces of voice sample data, and obtain word-level transcripts and phoneme-level transcripts corresponding to each piece of voice sample data respectively; wherein, the multiple pieces of voice sample data are voice samples of multiple people. data, and each person corresponds to at least two voice sample data;

The processing module 702 is further configured to perform frequency domain feature extraction on each piece of speech sample data respectively, and obtain a fourth spectral feature sequence corresponding to each piece of speech sample data;

Perform normalization processing on each fourth spectral feature sequence respectively, and obtain a fifth spectral feature sequence corresponding to each fourth spectral feature sequence and conforming to the normal distribution;

constructing a data set from each fifth spectral feature sequence, the word-level transcript and phoneme-level transcript corresponding to each fifth spectral feature sequence;

Randomly split all data sets into multiple subsets, each subset including a respective data set corresponding to each of the multiple persons;

Using multiple subsets, the phoneme-level alignment model is iteratively trained in sequence, and the end when the phoneme-level alignment model reaches the second preset standard, and the final phoneme-level alignment model, that is, a pre-built phoneme-level alignment model, is obtained.

The functions performed by each component in the two-level speech alignment apparatus provided in the embodiment of the present invention have been described in detail in any of the above method embodiments, and thus are not repeated here.

The embodiment of the present invention provides a two-level speech alignment device for speech data and word-level transcripts corresponding to the speech data. Then the speech data and the word-level transcript corresponding to the speech data are input into the pre-built word-level alignment model, and then the initial word-level alignment result is obtained. Traverse each frame of word-level frame data in the initial word-level alignment result, and merge the frame data corresponding to the word in the word-level frame data and the first type of placeholders respectively. Placeholder for non-voice data. Allocate the time of the placeholder in the speech data to the adjacent words to ensure that the speech of each word in the speech data can correspond to the words in the transcript, and achieve word-level transcript alignment.

On this basis, each word in the word-level transcript is used as the phoneme transcript, the first spectral feature sequence corresponding to the speech data, and the information vector corresponding to the speech data are input into the pre-built phoneme-level alignment model to obtain the initial Phoneme-level alignment results. Then, each frame of phoneme-level frame data is traversed, and a process of merging the phoneme-level frame data is performed, so as to merge the second-type placeholder and the frame data corresponding to the adjacent phonemes to obtain a merged frame. The second type of placeholders are also phoneme-level placeholders corresponding to non-speech data. The start time and end time of the merged frame are used as the start time and end time of the phoneme, and in this way, the phoneme-level transcript alignment is achieved. Above, because the time corresponding to all non-speech data has been reasonably allocated to the phoneme-level frame data corresponding to the adjacent phoneme-level transcript, that is, the time corresponding to the non-speech data is allocated to the adjacent phoneme-level frame data. In terms of phonemes, in this way, the obtained phoneme-level transcripts will not be misplaced at the beginning of the speech alignment result or tailed at the end position, and it is completely guaranteed that the final phoneme-level transcripts obtained can be compared with the speech itself. Basically corresponds to the maximum possible elimination error.

As shown in FIG. 8 , an embodiment of the present application provides an electronic device, including a processor 111 , a communication interface 112 , a memory 113 and a communication bus 114 , wherein the processor 111 , the communication interface 112 , and the memory 113 are completed through the communication bus 114 communication with each other.

a memory 113 for storing computer programs;

In an embodiment of the present application, the processor 111 is configured to implement the two-level speech alignment method provided by any one of the foregoing method embodiments when executing the program stored in the memory 113, including:

Obtain speech data, and word-level transcripts corresponding to the speech data;

The speech data and the word-level transcript corresponding to the speech data are input into a pre-built word-level alignment model, and an initial word-level alignment result corresponding to the speech data is obtained, wherein the initial word-level alignment result includes multiple frames of word-level frame data, And each frame of word-level frame data corresponds to a word in the word-level transcript, or corresponds to a first-class placeholder, and a word in the word-level transcript corresponds to at least one frame of word-level frame data;

Traverse each frame of word-level frame data, merge adjacent frames belonging to the same word, obtain the first merged frame data, and record the start time and end time of the first merged frame data;

According to the first preset rule, the first type of placeholder is combined with the word-level frame data corresponding to the word immediately adjacent to the first type of placeholder to obtain the second combined frame data;

Update the start time and end time of the second merged frame data, and finally obtain the word-level alignment transcript corresponding to the voice data;

Obtain the first spectral feature sequence corresponding to the voice data, and the information vector corresponding to the voice data;

Input the word-level alignment transcript, the first spectral feature sequence and the information vector into the pre-built phoneme-level alignment model, and obtain the initial phoneme-level alignment result corresponding to the speech data. The initial phoneme-level alignment result includes multiple frames of phoneme-level frame data , each frame of phoneme-level frame data corresponds to a phoneme unit, or corresponds to a second-type placeholder;

Traverse each frame of phoneme-level frame data, merge at least one second-type placeholder adjacent to the phoneme-level frame data corresponding to the phoneme unit in the initial phoneme-level alignment result with the phoneme-level frame data corresponding to the phoneme unit, and obtain the first Three merged frame data;

The start time and end time of the third merged frame data are recorded as the start time and end time corresponding to the phoneme unit, and finally a phoneme-level aligned transcript corresponding to the speech data is obtained.

Optionally, after obtaining the phoneme-level aligned transcript corresponding to the speech data, the method further includes:

The word-level aligned transcripts corresponding to the phonetic data are optimized using the phoneme-level aligned transcripts corresponding to the phonetic data.

Optionally, according to the first preset rule, the first type of placeholder is merged with the word-level frame data corresponding to the word immediately adjacent to the first type of placeholder to obtain the second merged frame data, which specifically includes:

When the left side of the first type of placeholder is the word-level frame data corresponding to the word, the first type of placeholder is merged with the word-level frame data corresponding to the word on the left, and the end of the first type of placeholder is Time is the end time of the second merged frame, and the start time of the word-level frame data of the word on the left is the start time of the second merged frame;

Or, when the first type of placeholder is the word-level frame data corresponding to the word on the right side, the first type of placeholder is merged with the word-level frame data corresponding to the right side of the word, and the first type of placeholder is combined. The start time of the second merged frame is taken as the start time of the second merged frame, and the right side is the end time of the word-level frame data corresponding to the word as the end time of the second merged frame.

Optionally, traverse each frame of phoneme-level frame data, and place at least one second-type placeholder adjacent to the phoneme-level frame data corresponding to the phoneme unit in the initial phoneme-level alignment result, and the phoneme-level frame data corresponding to the phoneme unit. Merge to obtain the third merged frame data, including:

When the phoneme-level frame data is adjacent to at least one second-type placeholder, the at least one second-type placeholder is merged, the fourth merged frame data is obtained, and the start time and end time of the fourth merged frame data are determined ;

The phoneme-level frame data and the fourth merged frame data are merged to obtain the third merged frame data.

Optionally, record the start time and end time of the third merged frame data as the start time and end time corresponding to the phoneme unit, specifically including:

When the fourth merged frame data is on the left side of the phoneme-level frame data, the start time of the fourth merged frame data is used as the start time of the third merged frame data, and the end time of the phoneme-level frame data is used as the third merge. The end time of the frame data;

Or, when the fourth merged frame data is on the right side of the phoneme-level data, the start time of the phoneme-level data is used as the start time of the third merged frame data, and the interpretation time of the fourth merged frame data is used as the third merged time. End time of frame data.

Optionally, the second-type placeholder includes a phoneme-level placeholder and/or a silence mark; when the phoneme-level frame data is adjacent to at least one second-type placeholder, at least one second-type placeholder is merged , and obtain the fourth merged frame data, which specifically includes: when the second type of placeholders all include phoneme-level placeholders, converting each phoneme-level placeholder into a silence mark, merging the silence marks, and obtaining the fourth merged frame data;

Or, when the second type of placeholders includes phoneme-level placeholders and silence placeholders, after merging the phoneme-level placeholders with their adjacent silence placeholders, merge all silence placeholders to obtain the fourth Merge frame data.

Optionally, before inputting the speech data and the word-level transcript corresponding to the speech data into the pre-built word-level alignment model, the method further includes:

Acquiring multiple pieces of voice sample data, and obtaining word-level transcripts corresponding to each piece of voice sample data; wherein, the multiple pieces of voice sample data are voice sample data of multiple people, and each person corresponds to at least two pieces of voice sample data;

Extracting feature vectors from multiple pieces of speech sample data of the ith person to obtain the ith information vector;

Perform frequency domain feature extraction on multiple pieces of speech sample data of the i-th person respectively, and obtain a second spectral feature sequence corresponding to each piece of speech sample data;

According to each second spectral feature sequence, the word-level transcript corresponding to each piece of speech sample data, and the i-th information vector, the word-level alignment model is iteratively trained until the word-level alignment model reaches the first preset standard. The word-level alignment model finally obtained, that is, the pre-built word-level alignment model, where i is a positive integer.

Optionally, the word-level alignment model is iteratively trained according to each second spectral feature sequence, the word-level transcript corresponding to each piece of speech sample data, and the i-th information vector, specifically including:

Perform normalization processing on each second spectral feature sequence respectively, and obtain a third spectral feature sequence corresponding to each second spectral feature sequence and conforming to the normal distribution;

sorting the third spectral feature sequence according to the sequence length;

According to the sorting order, in the jth training, the third spectral feature sequence sorted in the jth position, the word-level transcript corresponding to each piece of speech data, and the i-th information vector are input into the word-level alignment model, Used to complete the jth training of the word-level alignment model, where j is a positive integer.

Optionally, before inputting the word-level alignment transcript, the first spectral feature sequence and the information vector into the pre-built phoneme-level alignment model, the method further includes:

Acquire multiple pieces of voice sample data, and obtain word-level transcripts and phoneme-level transcripts corresponding to each piece of voice sample data; wherein, the multiple pieces of voice sample data are voice sample data of multiple people, and each person corresponds to at least two voices sample;

Perform frequency domain feature extraction on each piece of speech sample data respectively, and obtain a fourth spectral feature sequence corresponding to each piece of speech sample data;

Perform normalization processing on each fourth spectral feature sequence respectively, and obtain a fifth spectral feature sequence corresponding to each fourth spectral feature sequence and conforming to the normal distribution;

constructing a data set from each fifth spectral feature sequence, the word-level transcript and phoneme-level transcript corresponding to each fifth spectral feature sequence;

Randomly split all data sets into multiple subsets, each subset including a respective data set corresponding to each of the multiple persons;

Using multiple subsets, the phoneme-level alignment model is iteratively trained in sequence, and the end when the phoneme-level alignment model reaches the second preset standard, and the final phoneme-level alignment model, that is, a pre-built phoneme-level alignment model, is obtained.

Embodiments of the present application further provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the two-level speech alignment method provided by any one of the foregoing method embodiments.

It should be noted that, in this document, relational terms such as "first" and "second" etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these There is no such actual relationship or sequence between entities or operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article, or device that includes the element.

The above are only specific embodiments of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims (10)

1. A two-stage speech alignment method, the method comprising:

acquiring voice data and a word-level transcript corresponding to the voice data;

inputting the voice data and the word-level transcript corresponding to the voice data into a pre-constructed word-level alignment model, and obtaining an initial word-level alignment result corresponding to the voice data, wherein the initial word-level alignment result comprises multiple frames of word-level frame data, each frame of the word-level frame data corresponds to one word in the word-level transcript, or corresponds to one first-type placeholder, and one word in the word-level transcript corresponds to at least one frame of the word-level frame data;

traversing each frame of the word-level frame data, merging adjacent frames belonging to the same word to obtain first merged frame data, and recording the starting time and the ending time of the first merged frame data;

according to a first preset rule, combining the first type of placeholder with word-level frame data corresponding to the word adjacent to the first type of placeholder to obtain second combined frame data;

updating the starting time and the ending time of the second merged frame data, and finally obtaining a word-level alignment transcript corresponding to the voice data;

acquiring a first spectrum characteristic sequence corresponding to the voice data and an information vector corresponding to the voice data;

inputting the word-level alignment transcript, the first spectrum feature sequence and the information vector into a pre-constructed phoneme-level alignment model, and acquiring an initial phoneme-level alignment result corresponding to the voice data, wherein the initial phoneme-level alignment result comprises multi-frame phoneme-level frame data, and each frame of the phoneme-level frame data corresponds to a phoneme unit or corresponds to a second-class placeholder;

traversing each frame of the phoneme-level frame data, merging at least one second-class placeholder adjacent to the phoneme-level frame data corresponding to the phoneme unit in the initial phoneme-level alignment result with the phoneme-level frame data corresponding to the phoneme unit, and acquiring third merged frame data;

and recording the starting time and the ending time of the third merged frame data as the starting time and the ending time corresponding to the phoneme unit, and finally obtaining the phoneme-level aligned transcript corresponding to the voice data.

2. The method of claim 1, wherein after obtaining the phone-level aligned transcript corresponding to the speech data, the method further comprises:

and optimizing the word-level aligned transcript corresponding to the voice data by utilizing the phoneme-level aligned transcript corresponding to the voice data.

3. The method of claim 1, wherein traversing each frame of the phone-level frame data, merging at least one second type placeholder in the initial phone-level alignment result adjacent to the phone-level frame data corresponding to the phone unit with the phone-level frame data corresponding to the phone unit to obtain third merged frame data comprises:

when the phoneme-level frame data is adjacent to at least one second-class placeholder, merging the at least one second-class placeholder to obtain fourth merged frame data, and determining the starting time and the ending time of the fourth merged frame data;

and merging the phoneme-level frame data and the fourth merged frame data to obtain third merged frame data.

4. The method according to claim 3, wherein recording a start time and an end time of the third merged frame data as a start time and an end time corresponding to the phoneme unit specifically includes:

when the fourth merged frame data is on the left side of the phoneme-level frame data, taking the start time of the fourth merged frame data as the start time of the third merged frame data, and taking the end time of the phoneme-level frame data as the end time of the third merged frame data;

or, when the fourth merged frame data is on the right side of the phoneme-level data, the start time of the phoneme-level data is used as the start time of the third merged frame data, and the interpretation time of the fourth merged frame data is used as the end time of the third merged frame data.

5. The method of claim 3 or 4, wherein the second type of placeholder comprises a phone-level placeholder and/or a mute flag; when the phoneme-level frame data is adjacent to at least one second-class placeholder, merging the at least one second-class placeholder to obtain fourth merged frame data, specifically including:

when the second-class placeholders comprise the phoneme-level placeholders, converting each phoneme-level placeholder into a mute marker, merging the mute markers, and acquiring fourth merged frame data;

or when the second type of placeholder includes a phoneme-level placeholder and a silence placeholder, merging the phoneme-level placeholder and the silence placeholder adjacent to the phoneme-level placeholder, and merging all the silence placeholders to obtain fourth merged frame data.

6. The method of any of claims 1-4, wherein prior to inputting the speech data and the word-level transcript corresponding to the speech data into the pre-constructed word-level alignment model, the method further comprises:

acquiring a plurality of pieces of voice sample data, and acquiring word-level copies corresponding to each piece of voice sample data respectively; the voice sample data of a plurality of persons are obtained, and each person corresponds to at least two pieces of voice sample data;

respectively extracting characteristic vectors of a plurality of pieces of voice sample data of an ith person to obtain an ith information vector;

respectively extracting frequency domain features of a plurality of pieces of voice sample data of an ith person to obtain a second frequency spectrum feature sequence corresponding to each piece of voice sample data;

and performing iterative training on the word-level alignment model according to each second frequency spectrum characteristic sequence, the word-level transcript corresponding to each voice sample data and the ith information vector, and obtaining a finally obtained word-level alignment model until the word-level alignment model reaches a first preset standard, namely the pre-constructed word-level alignment model, wherein i is a positive integer.

7. The method according to claim 6, wherein iteratively training a word-level alignment model according to each of the second spectral feature sequences, the word-level transcript corresponding to each of the voice sample data, and the ith information vector comprises:

respectively carrying out normalization processing on each second frequency spectrum characteristic sequence to obtain a third frequency spectrum characteristic sequence which corresponds to each second frequency spectrum characteristic sequence and accords with the positive distribution;

sequencing the third spectral feature sequence according to the sequence length;

and inputting a third spectral feature sequence which is sequenced at a jth position, word-level transcripts which respectively correspond to each piece of voice data and the ith information vector into the word-level alignment model according to a sequencing sequence when training for the jth time in sequence to finish the jth training of the word-level alignment model, wherein j is a positive integer.

8. The method of any of claims 1-4, wherein before inputting the word-level aligned transcript, the first sequence of spectral features, and the information vector into a pre-constructed phoneme-level alignment model, the method further comprises:

acquiring a plurality of pieces of voice sample data, and acquiring a word-level transcript and a phoneme-level transcript corresponding to each piece of voice sample data respectively; the voice sample data of a plurality of persons are obtained, and each person corresponds to at least two pieces of voice sample data;

respectively extracting frequency domain characteristics of each piece of voice sample data to obtain a fourth frequency spectrum characteristic sequence corresponding to each piece of voice sample data;

respectively carrying out normalization processing on each fourth frequency spectrum characteristic sequence to obtain a fifth frequency spectrum characteristic sequence which corresponds to each fourth frequency spectrum characteristic sequence and accords with positive distribution;

constructing a data set by using each fifth frequency spectrum characteristic sequence, and the word-level transcript and the phoneme-level transcript corresponding to each fifth frequency spectrum characteristic sequence;

randomly splitting all data sets into a plurality of subsets, each subset including a data set corresponding to each of a plurality of persons respectively;

and performing iterative training on the phoneme level alignment model in sequence by using the plurality of subsets, and obtaining a final phoneme level alignment model, namely the pre-constructed phoneme level alignment model, when the phoneme level alignment model reaches a second preset standard.

9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;

a memory for storing a computer program;

a processor for implementing the steps of the two-stage speech alignment method of any of claims 1-8 when executing a program stored in a memory.

10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the two-stage speech alignment method according to any one of claims 1 to 8.

Images