CN115312078A - Method, apparatus, device and storage medium for determining quality of voice data - Google Patents

Abstract

Embodiments of the present disclosure relate to a method, apparatus, device, and storage medium for determining the quality of voice data. The method includes determining characteristics of the acquired speech data. The method also includes obtaining, based on the feature, a first quality level and a second quality level for the speech data, the first quality level being related to the first language, and the second quality level being related to the second language. The method also includes determining a target quality level for the speech data based on the first quality level and the second quality level, the target quality level indicating the speech quality of the speech data. Through the method, the accuracy and efficiency of evaluating the quality of speech data across fields can be improved, the utilization rate of data is improved, and the user experience is improved.

Description

Method, device, device and storage medium for determining the quality of speech data

technical field

Embodiments of the present disclosure generally relate to the field of speech data processing, and specifically relate to a method, device, device and storage medium for determining the quality of speech data.

Background technique

With the development of computer technology, the level of speech processing is also improving rapidly. The use of computing devices to synthesize or convert speech data is also increasingly used in various devices and applications. For these voice data, the voice quality can be used for evaluation, because the voice quality is the main index reflecting the system performance through speech synthesis, speech conversion, etc. The Mean Opinion Score (MOS) is the subjective evaluation score of the voice quality of the audio after the annotator conducts a listening test on the synthesized audio. Since the traditional MOS scoring requires the participation of a large number of annotators, this subjective evaluation process will lead to high costs and long time consumption. Therefore, there are still many problems to be solved in the processing of voice data.

Contents of the invention

Embodiments of the present disclosure provide a method, device, device and storage medium for determining the quality of voice data.

According to a first aspect of the present disclosure, a method of determining the quality of speech data is provided. The method includes determining features of the acquired speech data. The method also includes obtaining, based on the feature, a first quality level for the speech data and a second quality level, the first quality level being related to the first language and the second quality level being related to the second language. The method also includes determining a target quality level for the speech data based on the first quality level and the second quality level, the target quality level indicating a speech quality of the speech data.

In a second aspect of the present disclosure, an apparatus for determining quality of speech data is provided. The device includes a feature determination module configured to determine features of the acquired voice data; a quality level acquisition module configured to acquire a first quality level and a second quality level for the voice data based on the features, the first quality level and the second quality level The first language is related, the second quality level is related to the second language; and the target quality level determination module is configured to determine the target quality level for the voice data based on the first quality level and the second quality level, the target quality level indicates the voice Data voice quality.

In a third aspect of the present disclosure, an electronic device is provided, including at least one processor; and a storage device for storing at least one program, when the at least one program is executed by the at least one processor, the at least one processor realizes A method according to the first aspect of the present disclosure.

In a fourth aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, the program implements the method according to the first aspect of the present disclosure when executed by a processor.

It should be understood that the content described in this summary is not intended to limit the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

Description of drawings

The above and other objects, features and advantages of the present disclosure will become more apparent by describing the exemplary embodiments of the present disclosure in more detail with reference to the accompanying drawings, wherein, in the exemplary embodiments of the present disclosure, the same reference numerals generally represent same parts.

FIG. 1 illustrates a schematic diagram of an example environment 100 in which devices and/or methods of embodiments of the present disclosure may be implemented;

FIG. 2 illustrates a flowchart of a process 200 for determining the quality of speech data according to an embodiment of the present disclosure;

FIG. 3 illustrates a schematic diagram of an example 300 for generating a score for speech data according to an embodiment of the present disclosure;

FIG. 4 illustrates a schematic diagram of a process 400 for training a decoder and a speech representation model according to an embodiment of the present disclosure;

FIG. 5 illustrates a schematic diagram of an example 500 of training a decoder and speech representation model according to an embodiment of the present disclosure;

Fig. 6 illustrates a schematic block diagram of an apparatus 600 for determining the quality of speech data according to an embodiment of the present disclosure;

FIG. 7 illustrates a schematic block diagram of an example device 700 suitable for implementing embodiments of the present disclosure.

In the respective drawings, the same or corresponding reference numerals denote the same or corresponding parts.

Detailed ways

It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the user should be informed of the type, scope of use, and use scenarios of the personal information involved in the present disclosure in an appropriate manner in accordance with relevant laws and regulations, and the authorization of the user should be obtained. .

For example, in response to receiving the user's active request, send prompt information to the user to clearly remind the user that the requested operation will require the acquisition and use of the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as electronic devices, application programs, servers, or storage media that perform the operations of the technical solution of the present disclosure according to the prompt information.

As an optional but non-limiting implementation, in response to receiving the active request from the user, the prompt information may be sent to the user, for example, in the form of a pop-up window, and the prompt information may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

It can be understood that the above process of notifying and obtaining user authorization is only illustrative and does not limit the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

It can be understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of data) should comply with the requirements of corresponding laws and regulations and relevant regulations.

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

In the description of the embodiments of the present disclosure, the term "comprising" and its similar expressions should be interpreted as an open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be read as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same object. Other definitions, both express and implied, may also be included below.

As mentioned above, using annotators to score the speech quality of speech data would result in high cost and time-consuming. Therefore, the MOS automatic scoring system is proposed, which mainly uses the machine to score the synthesized audio, replacing the subjective evaluation of the annotator, so as to save time and resources. However, there are at least two challenges in using machines for MOS scoring. The first challenge is the problem of data sparsity. There is not a lot of data used to train the MOS scoring system, which will limit the performance of the scoring system. The second challenge is the scoring of cross-domain synthetic audio (synthetic audio in different languages). In the MOS scoring system, the synthetic audio of various languages may not be able to give an accurate score due to the lack of training data of the corresponding language. .

Automatic scoring systems are designed in some traditional schemes. The system uses pre-trained encoders for automatic MOS scoring. This scheme can achieve better scoring results on data in the same domain (the language of the training set and the test set are the same).

However, this traditional scheme has many problems in cross-domain tasks. For example, when the training sets of different domains are used separately, the cross-domain training sets are very rare, which will limit the performance of the encoder; when the training sets of different domains are mixed, due to the different language backgrounds of some training sets and test sets, the system Synthetic audio that does not fit well to the test set. This leads to the problem of inaccurate scoring by the MOS system.

In addition, these traditional solutions do not consider the connection between different tasks, such as the association between automatic speech recognition tasks and MOS automatic scoring. MOS scoring data is sparse, but the data of automatic speech recognition is massive.

To address at least the above and other potential problems, embodiments of the present disclosure propose a method of determining the quality of speech data. In the method, a computing device determines characteristics of acquired speech data. Then using the obtained features, a first quality level and a second quality level for the voice data are obtained. These quality levels are related to the kind of language. The computing device then determines a target quality level for the voice data based on the first quality level and the second quality level. Through the method, the accuracy and efficiency of cross-domain evaluation of the quality of voice data can be improved, the utilization rate of data can be improved, and user experience can be improved.

Embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings, wherein FIG. 1 shows an example environment 100 in which the devices and/or methods of the embodiments of the present disclosure can be implemented.

Included in the environment 100 is a computing device 104 for processing speech data 102 to determine a target quality level 112 in the speech data 102 .

Examples of computing device 104 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics , a minicomputer, a mainframe computer, a distributed computing environment including any one of the above-mentioned systems or devices, and the like.

Speech data 102 received by computing device 104 is speech data that includes a speaker. The speech data includes, but is not limited to, synthesized speech data or converted speech data. For synthesized speech data, it includes speech synthesized by a computing device simulating human speech. For converted voice data, it includes converting the voice spoken by a speaker into another speaker's voice or converting the content spoken by a speaker into another language. FIG. 1 shows computing device 104 receiving voice data 102 . It is only an example and does not limit the present disclosure. The computing device 104 may also generate speech data or store the speech data in local memory of the computing device 104 .

After the computing device 104 obtains the speech data 102 , it can process the speech data 102 to obtain the features 106 of the speech data. In one example, a computing device may run a trained wav2vec2 model to extract contextual representations of speech data as features 106 . In another example, the computing device runs other speech representation models to obtain the features of the speech data, such as any suitable speech representation model that converts speech information into a vector representation can be used to obtain the features of the speech data. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure. Those skilled in the art can use any suitable model to obtain the features of speech data.

Computing device 104 may also obtain the first quality level and the second quality level for the speech data based on characteristics 106 . Wherein the first quality level is related to the first language, and the second quality level is related to the second language. In one example, the first language is English and the second language is Chinese. In another example, the first language is Chinese and the second language is English. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure. Those skilled in the art can set the specific languages of the first language and the second language according to needs. In some embodiments, the first quality level is generated using an encoder associated with the first language and the second quality level is generated using an encoder associated with the second language.

Computing device 104 is shown obtaining quality level 108 and quality level 110 in FIG. 1 , which is merely an example and not a specific limitation of the present disclosure. Computing device 104 may also obtain any number of quality levels for any number of languages as desired.

Computing device 104 determines a target quality level for voice data 102 from first quality level 108 and second quality level 110 to indicate the voice quality of voice data 102 . In examples where more than two quality levels are generated by computing device 104, computing device 104 selects a target quality level from among the plurality of quality levels. Alternatively or additionally, the computing device 104 uses the language of the speech data to select the quality level corresponding to the language as the target quality level.

In one example, the target quality rating is a score. In another example, the target quality level is different levels, such as good, medium, poor or A, B, C, D. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure. Those skilled in the art can set the content of the target quality level according to needs.

Through the method, the accuracy and efficiency of cross-domain evaluation of the quality of voice data can be improved, the utilization rate of data can be improved, and user experience can be improved.

A block diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is described above in connection with FIG. 1 . A flow chart of a process 200 for determining the quality of speech data according to an embodiment of the present disclosure is described below with reference to FIG. 2 . Process 200 may be performed at computing device 104 of FIG. 1 .

At block 202, characteristics of the acquired speech data are determined. For example, computing device 104 captures speech data and then processes the speech data to determine characteristics of the captured speech data.

In some embodiments, the voice data acquired by the computing device 104 may be voice data generated through speech synthesis. For example, speech data generated by converting text to speech. In some embodiments, the voice data captured by computing device 104 is generated by voice conversion. For example, converting a speaker speaking Chinese into speech data of a speaker speaking English. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure. Any suitable speech data may be used to determine the quality of speech. In this way, the quality of the synthesized or transformed speech can be quickly determined.

In some embodiments, the speech data is characterized by vectors obtained for the speech data. In some embodiments, the speech data is characterized as a contextual representation of the speech data. In some embodiments, computing device 104 obtains a fine-tuned speech representation model. Features are then obtained by applying the fine-tuned speech representation model to the speech data. In some embodiments, computing device 104 applies the adjusted speech representation model to the speech data to obtain features. The adjusted speech representation model is obtained by adjusting the fine-tuned speech representation model when training the speech data evaluation model as a whole by using sample speech data and corresponding quality levels. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure. Those skilled in the art can use any suitable model method to obtain the features of speech data. Alternatively or additionally, the contextual representation is related to the text corresponding to the speech data. The training of the model will be further described below with reference to FIG. 4 and FIG. 5 .

At block 204, based on the characteristics, a first quality level is obtained for the speech data, the first quality level is related to the first language, and the second quality level is related to the second language. For example, computing device 104 processes the feature to obtain first quality level 108 and second quality level 110 for speech data 102 .

In some embodiments, computing device 104 obtains the first quality level by applying a trained first decoder to the feature, and obtains the second quality level by applying a trained second decoder to the feature. The trained first decoder and the trained second decoder can be used to process the features of the speech data to obtain corresponding quality levels. The first decoder is trained using speech data related to the first language, and the second decoder is trained using speech data related to the second speech. In this way, different decoders can be used to determine the quality level of speech data, so that the quality detection of speech data in various languages can be realized.

Alternatively or additionally, when using the first decoder or the second decoder to obtain the quality grade, it is also necessary to input the identification ID of the provider providing the quality evaluation, so as to obtain the quality grade given by the provider. During inferential use of either the first decoder or the second decoder, the provider is the virtual scorer. The training process of the decoder will be further described below in conjunction with FIGS. 4 and 5 .

Alternatively or additionally, the computing device 104 also has within the computing device 104 decoders for other languages. For example, based on the characteristics, the computing device may also obtain a third quality level for speech data, the third quality level being related to the third language.

At block 206, based on the first quality level and the second quality level, a target quality level for the speech data is determined, the target quality level indicating the speech quality of the speech data. Computing device 104 utilizes the first quality level and the second quality level to determine the target quality level.

In some embodiments, the computing device 104 selects a quality level corresponding to the language of the voice data from the first quality level and the second quality level as the target quality level.

In some embodiments, if computing device 104 also computes a third quality level. Therefore, determining the target quality level for the voice data also includes determining the target quality level based on the first quality level, the second quality level and the third quality level.

Through the method, the accuracy and efficiency of cross-domain evaluation of the quality of voice data can be improved, the utilization rate of data can be improved, and user experience can be improved.

The flow chart of the process 200 for determining the quality of voice data according to an embodiment of the present disclosure is described above with reference to FIG. 2 . A specific example 300 of determining the voice quality of voice data is described below with reference to FIG. 3 . In the example depicted in Figure 3, the speech quality of the data is represented by a score, and the speech representation model is a wav2vec2 model.

As shown in FIG. 3 , synthesized speech data 302 is input to a trained wav2vec2 model 304 . In one example, the trained wav2vec2 model 304 is obtained by fine-tuning a pre-trained wav2vec2 model using a set of speech data and corresponding text. In another example, the trained wav2vec2 model 304 is fine-tuned on the pre-trained wav2vec2 model, and needs to be further adjusted by using the sample speech data and the sample quality level when training the decoder later.

The pre-trained wav2vec2 model is obtained by using a large amount of speech data for self-learning. The pre-trained wav2vec2 consists of three parts: an encoder composed of a convolutional neural network, a context processor composed of a Transformer, and a quantizer. Its input is the original speech signal. The encoder can encode the speech signal with a sampling rate of 16kHz and encode the 25ms segment every 20ms into a hidden vector. The context processor can then consider the entire speech from other segments on the current segment. The information of the segment, and the latent vector is further processed into a context-dependent representation of the segment. The quantizer is only used during wav2vec2 pre-training. The pre-trained wav2vec2 model may be provided by another provider, or may be trained by the user himself using voice data.

When fine-tuning the pre-trained wav2vec2 model, the pre-trained wav2vec2 will retain the parameters of the encoder of the convolutional neural network and the context processor of the Transformer, and then add a fully connected layer with initialization parameters. When fine-tuning, the pre-trained wav2vec2 is trained using sample speech data and corresponding text, and the parameters of wav2vec2 are adjusted using the connectionist temporal classification CTC loss function. After fine-tuning wav2vec2 in the way of voice recognition, the fine-tuned wav2vec2 has the ability to recognize pronunciation, and the determined context representation is related to the text content corresponding to the voice data. The fine-tuned wav2vec2 will remove the last fully connected layer added during training when inferring in this solution.

A trained wav2vec2 model 304 processes synthesized speech data 303 to obtain context vectors 306 . The obtained context vectors 306 are then input to decoder A 310 and decoder B 314 respectively. Decoder A 310 is used to score English voice data, and decoder B is used to score Chinese voice data. When scoring, the decoder A 310 also needs to input the English virtual scorer ID 308 , and the decoder B 314 also needs to input the Chinese virtual scorer ID 312 . Then decoder A and decoder B respectively obtain the scores of the corresponding virtual scorers. Then, in the score selection module 316 , according to the language of the synthesized voice data 303 , the score corresponding to the language is selected as the score 318 for evaluating the synthesized voice data 303 . For example, if the synthesized speech data is in English, the score of decoder A 310 is chosen, and if the synthesized speech data is in Chinese, the score of decoder B 314 is chosen. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure. The number of decoders and the corresponding languages can be set by the user according to actual needs. For example, multiple decoders can be trained with speech data in multiple languages when training the decoders.

Through the method, the accuracy and efficiency of cross-domain evaluation of the quality of voice data can be improved, the utilization rate of data can be improved, and user experience can be improved.

The reasoning process using the speech representation model and the decoder is described above, and a schematic diagram of a process 400 for training a decoder and a speech representation model according to an embodiment of the present disclosure is described below in conjunction with FIG. 4 . Process 400 may be performed by the computing device in FIG. 1 or other suitable computing devices.

The training process is by using sample speech data and sample scores to obtain an adjusted speech representation model, a trained first decoder and a trained second decoder.

At block 402, the computing device 104 first obtains a fine-tuned speech representation model, a first decoder, and a second decoder. In one example column, the fine-tuned speech representation model, first decoder and second decoder are received from other devices. In another example, the fine-tuned speech representation model, the first decoder, and the second decoder are obtained from the computing device 104 .

At block 404, computing device 104 obtains a first set of sample speech data and corresponding sample quality levels. The computing device obtains a sample quality level corresponding to the set of sample data. For example, each sample speech data has four sample quality levels, three of which are provided by the real provider, the fourth is provided by the virtual provider, and the fourth is the quality of the other three Average quality rating. The above examples are only used to describe the disclosure, rather than to specifically limit the disclosure. Those skilled in the art can set the number of quality levels and the composition of the quality levels according to needs.

At block 406, the computing device 104 then determines a first set of identifications for providers for providing sample quality ratings. These providers include the identity of a virtual provider. For example, there are four provider IDs, three are real provider IDs and one is a dummy provider ID.

At block 408, the computing device 104 obtains the experienced speech representation model by jointly training the fine-tuned speech representation model and the first decoder or the second decoder using the first set of sample speech data, the corresponding sample quality level, and the first set of identifications. An adapted speech representation model and either a trained first decoder or a trained second decoder.

During the training process, the computing device 104 obtains sample speech data from a set of sample speech data, and then obtains corresponding features through a fine-tuned speech representation model. This feature is then input to a decoder corresponding to the language in the sample speech data. In addition, the identifier of the provider of the sample quality level for the speech data also needs to be input in the decoder. For example, if there are four providers, the decoder will output four scores, and then adjust the fine-tuned speech representation model and the parameters of the corresponding decoder according to the four output scores and the scores given by the four providers.

The joint training of fine-tuned speech representation models and decoders is described above. In some embodiments, the first decoder and the second decoder can be trained independently without training together with the fine-tuned speech representation model. At this time, the fine-tuned speech representation model is regarded as a trained speech representation model. Computing device 104 utilizes the fine-tuned speech representation model to determine sample features corresponding to the second set of sample speech data. Computing device 104 then determines a second set of identifications for providers that are used to provide sample quality ratings. The computing device 104 then trains the first decoder or the second decoder using the sample features, the second group identifier and the sample quality level to obtain a trained first decoder or the trained second decoder.

In some embodiments, the fine-tuned speech representation model may be trained by computing device 104, but may also be trained by other computing devices. When obtaining the fine-tuned speech representation model, first obtain the pre-trained speech representation model. Then, the third group of sample voice data and corresponding sample text are acquired. Next, the fine-tuned speech representation model is generated by using the third set of sample speech data and corresponding sample texts to train the pre-trained speech representation model. In this way, a fine-tuned speech representation model related to the text of the speech data can be obtained, thereby improving the accuracy of evaluation.

The pre-trained speech representation model may be trained by the computing device 104, or may be trained by other computing devices. When generating the pre-trained speech representation model, the fourth set of sample speech data is obtained first. Then, a pre-trained speech representation model is generated by using the fourth set of sample speech data to train the speech representation model. The speech representation model is a neural network model that converts speech signals into corresponding context representations.

Alternatively, the process of training the first encoder and the second encoder on the computing device 104 is shown above, and the first encoder and the second encoder may also be trained on other computing devices.

In some embodiments, the first decoder is a decoder for Chinese, and the second decoder is a decoder for English.

An example training process is described above in conjunction with FIG. 4 , and an example 500 for training a model is described below in conjunction with FIG. 5 .

As shown in FIG. 5 , when training the speech quality evaluation model, the model includes three parts, fine-tuned wav2vec2, decoder A 512 and decoder B 514 . At this time, the fine-tuned wav2vec2 removed the last fully connected layer. During training, the English synthesized audio data 502 or the Chinese synthesized audio data 504 is input into the fine-tuned wav2vec2 model 506, and then the corresponding context vector 508 is generated, and the context vector 508 is input into the decoder A 512 or the decoder B 514. If the context vector 508 is English synthesized audio data, then enter decoder A 512 . At the same time, decoder A 512 also inputs a set of scorer IDs 510 of English synthetic data, one of which is a virtual scorer ID, and the score corresponding to the virtual scorer ID is the average of the scores of other scorers in the group. The scores corresponding to the different raters are then obtained 518 . The obtained scores are then compared with the sample scores for this speech data to adjust the parameters of the fine-tuned wav2vec2 and decoder A.

If this context vector 508 is Chinese synthetic audio data, then enter decoder B 514, simultaneously decoder B514 also imports the scorer ID 516 of a group of Chinese synthetic data, one of the bases is the ID of the virtual scorer, and the virtual scorer ID corresponds to The score is the average of the scores of the other scorers in the group. The scores corresponding to the different raters are then obtained 520 . The obtained scores are then compared to the sample scores for the speech data to adjust the fine-tuned wav2vec2 model 506 and decoder B 514 parameters.

Figure 5 depicts wav2vec2, decoder A 512 and decoder B 514 fine-tuned simultaneously using sample speech data and sample scores. In some embodiments, the fine-tuned wav2vec2 parameters may not be adjusted, and the fine-tuned wav2vec2 is used to generate the context vector for the sample speech data during training, and then the parameters of decoder A and decoder B are adjusted in combination with the sample scores to realize the decoding Training of devices A and B.

Fig. 6 shows a schematic block diagram of an apparatus for determining a speaker in a text according to an embodiment of the present disclosure. As shown in FIG. 6 , the device 600 includes a feature determination module 602 configured to determine the features of the acquired voice data; a quality level acquisition module 604 configured to acquire the first quality level and the second quality level for the voice data based on the features. Quality level, the first quality level is related to the first language, and the second quality level is related to the second language; and the target quality level determination module 606 is configured to determine the target quality level for the voice data based on the first quality level and the second quality level The target quality level indicates the voice quality of the voice data.

In some embodiments, the quality level obtaining module 604 includes: a first quality level obtaining module configured to obtain the first quality level by applying the trained first decoder to the feature; and a second quality level obtaining module, configured to obtain the second quality level by applying the trained second decoder to the features.

In some embodiments, the feature determination module includes: a first feature acquisition module configured to apply the adjusted speech representation model to the speech data to obtain features.

In some embodiments, a joint training module is also included, configured to obtain the adjusted speech representation model, the trained first decoder, and the trained second decoder by: a model and decoder acquisition module, which is Configured to acquire the fine-tuned speech representation model, the first decoder and the second decoder; the sample quality level acquisition module is configured to acquire the first set of sample speech data and the corresponding sample quality level; the first set of identification determination module is configured configured to determine a first set of identifications for providers, the provider is used to provide the sample quality level; the training acquisition module is configured to use the first set of sample speech data, the corresponding sample quality level and the The first set of identifiers jointly trains the fine-tuned speech representation model and the first decoder or the second decoder to obtain the adjusted speech representation model and the trained first decoder or The trained second decoder.

In some embodiments, the feature determination module includes: a fine-tuned speech representation model acquisition module configured to obtain a fine-tuned speech representation model; a second feature acquisition module configured to apply the fine-tuned speech representation model to the The speech data is used to obtain the features.

In some embodiments, the decoder obtaining module is configured to further include obtaining the trained first decoder and the trained second decoder by: a sample feature determination module configured to determine the same The sample features corresponding to the second group of sample speech data; the second group identification determination module is configured to determine the second group identification for the provider, and the provider is used to provide the sample quality level; the decoder training module, configured to train the first decoder or the second decoder using the sample features, the second set of identifications and the sample quality level to obtain the trained first decoder or the The trained second decoder described above.

In some embodiments, obtaining the fine-tuned speech representation model includes: a pre-training model acquisition module configured to obtain a pre-trained speech representation model; a corresponding sample text acquisition module configured to obtain a third group of sample speech data and corresponding sample text; and a first generation module configured to generate the fine-tuned speech representation by using the third set of sample speech data and the corresponding sample text to train the pre-trained speech representation model Model.

In some embodiments, the pre-training model acquisition module includes: a fourth group of sample speech data acquisition module configured to obtain a fourth group of sample speech data; and a pre-trained speech representation model generation module configured to use the The fourth group of sample voice data is used to train the voice representation model to generate the pre-trained voice representation model.

In some embodiments, the speech representation model is a neural network model that converts speech signals into corresponding features.

In some embodiments, wherein the features are related to text corresponding to the speech data.

In some embodiments, the apparatus 600 further includes: a third quality level acquiring module configured to acquire a third quality level for the voice data based on the feature, the third quality level being related to a third language; And the target quality level determination module further includes: a determination module based on three quality levels, configured to determine the target quality level based on the first quality level, the second quality level and the third quality level.

In some embodiments, the target quality level determination module includes: a selection module configured to select a quality level corresponding to the language of the voice data from the first quality level and the second quality level as the selected target quality level.

In some embodiments, the apparatus 600 further includes: a speech data generation module configured to generate the speech data through speech synthesis or speech conversion.

FIG. 7 shows a schematic block diagram of an example device 700 that may be used to implement embodiments of the present disclosure. Computing device 104 in FIG. 1 may be implemented with device 700 . As shown, the device 700 includes a central processing unit (CPU) 701 that can be programmed according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 into a random-access memory (RAM) 703 program instructions to perform various appropriate actions and processes. In the RAM 703, various programs and data necessary for the operation of the device 700 can also be stored. The CPU 701 , ROM 702 , and RAM 703 are connected to each other via a bus 704 . An input/output (I/O) interface 705 is also connected to the bus 704 .

Multiple components in the device 700 are connected to the I/O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage page 708, such as a magnetic disk, an optical disk, etc. ; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information/data with other devices over a computer network such as the Internet and/or various telecommunication networks.

The various procedures and processes described above, such as the methods 200 and 400 , can be executed by the processing unit 701 . For example, in some embodiments, methods 200 and 400 may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708 . In some embodiments, part or all of the computer program may be loaded and/or installed on the device 700 via the ROM 702 and/or the communication unit 709 . When the computer program is loaded into RAM 703 and executed by CPU 701, one or more actions of methods 200 and 400 described above may be performed.

The present disclosure may be a method, apparatus, system and/or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for carrying out various aspects of the present disclosure.

A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.

Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or Source or object code written in any combination, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as the “C” language or similar programming languages. Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as via the Internet using an Internet service provider). connect). In some embodiments, an electronic circuit, such as a programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA), can be customized by utilizing state information of computer-readable program instructions, which can Various aspects of the present disclosure are implemented by executing computer readable program instructions.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer readable program instructions may be provided to a processing unit of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processing unit of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions into a computer, other programmable data processing device, or other equipment, so that a series of operational steps are performed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , so that instructions executed on computers, other programmable data processing devices, or other devices implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, a portion of a program segment, or an instruction that includes one or more Executable instructions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or action , or may be implemented by a combination of dedicated hardware and computer instructions.

Having described various embodiments of the present disclosure above, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the various embodiments, practical applications or technical improvements over technologies in the market, or to enable other persons of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims (16)

1. A method of determining the quality of speech data, comprising:

determining characteristics of the acquired voice data;

based on the features, obtaining a first quality level and a second quality level for the voice data, the first quality level being related to a first language and the second quality level being related to a second language; and

determining a target quality level for the voice data based on the first quality level and the second quality level, the target quality level indicating a voice quality of the voice data.

2. The method of claim 1, obtaining the first quality level and the second quality level comprises:

obtaining the first quality level by applying a trained first decoder to the feature; and

obtaining the second quality level by applying a trained second decoder to the feature.

3. The method of claim 2, wherein the determining a characteristic of the acquired voice data comprises:

applying the adapted speech representation model to the speech data to obtain the feature.

4. The method of claim 3, further comprising:

obtaining a fine-tuned speech representation model, a first decoder and a second decoder;

acquiring a first group of sample voice data and corresponding sample quality grades;

determining a first set of identities for providers that are to provide the sample quality rating;

obtaining the adjusted speech representation model and the trained first decoder or the trained second decoder by training the fine-tuned speech representation model and the first decoder or the second decoder using the first set of sample speech data, corresponding sample quality levels, and the first set of identifications.

5. The method of claim 2, wherein the determining characteristics of the acquired speech data comprises:

acquiring a fine-tuned voice representation model;

obtaining the feature by applying the fine-tuned speech representation model to the speech data.

6. The method of claim 5, further comprising:

determining sample features corresponding to the second set of sample speech data;

determining a second set of identities for providers that are to provide the sample quality rating;

training the first decoder or the second decoder using the sample features, the second set of identifications, and the sample quality levels to obtain the trained first decoder or the trained second decoder.

7. The method of claim 4 or 5, wherein obtaining the fine-tuned speech representation model comprises:

acquiring a pre-trained voice representation model;

acquiring a third group of sample voice data and corresponding sample texts; and

generating the fine-tuned speech representation model by training the pre-trained speech representation model using the third set of sample speech data and the corresponding sample text.

8. The method of claim 7, wherein obtaining the pre-trained speech representation model comprises:

acquiring a fourth group of sample voice data; and

generating the pre-trained speech representation model by training a speech representation model using the fourth set of sample speech data.

9. The method of claim 8, wherein the speech representation model is a neural network model that converts speech signals into corresponding features.

10. The method of claim 1, wherein the feature relates to text corresponding to the speech data.

11. The method of claim 1, further comprising:

based on the features, obtaining a third quality level for the speech data, the third quality level being related to a third language; and is provided with

Determining a target quality level for the voice data further comprises:

determining the target quality level based on the first quality level, the second quality level, and the third quality level.

12. The method of claim 1, wherein determining the target quality level comprises:

selecting a quality class corresponding to a language of the voice data from the first quality class and the second quality class as the target quality class.

13. The method of claim 1, further comprising:

the voice data is generated by voice synthesis or voice conversion.

14. An apparatus for determining quality of speech data, comprising:

a feature determination module configured to determine a feature of the acquired voice data;

a quality level obtaining module configured to obtain a first quality level and a second quality level for the voice data based on the feature, the first quality level being related to a first language, the second quality level being related to a second language; and

a target quality level determination module configured to determine a target quality level for the voice data based on the first quality level and the second quality level, the target quality level indicating a voice quality of the voice data.

15. An electronic device, comprising:

at least one processor; and

storage means for storing at least one program which, when executed by the at least one processor, causes the at least one processor to carry out the method according to any one of claims 1-13.

16. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-13.

Images