CN114333790A - Data processing method, device, equipment, storage medium and program product - Google Patents

Abstract

The application provides a data processing method, a device, equipment, a storage medium and a program product, wherein the method comprises the following steps: acquiring voice data to be recognized and acquiring a target syllable sequence of a reference keyword; calling a keyword detection model to process the voice data to be recognized, determining a syllable sequence to be detected of the voice data to be recognized, and determining a keyword detection result of the voice data to be recognized according to the syllable sequence to be detected and the target syllable sequence; the keyword detection model is obtained by training by using a training data set, wherein the training data set comprises sample voice data of one or more language categories; the keyword detection model may perform keyword detection on speech data in any of one or more language categories. The method and the device can be applied to various scenes such as cloud technology, artificial intelligence, a smart platform and vehicle-mounted Internet, can realize multi-language keyword detection, realize automation and intellectualization of keyword detection, and improve the efficiency of keyword detection.

Description

Data processing method, apparatus, equipment, storage medium and program product

technical field

The present application relates to the field of computer technology, in particular to the field of artificial intelligence technology, and in particular to a data processing method, a data processing apparatus, computer equipment, a computer-readable storage medium, and a computer program product.

Background technique

With the continuous development and application of computer technology, more and more scenarios require the use of data processing technology, such as keyword detection on voice data through data processing technology to wake up smart devices and detect the frequency of keyword occurrences. But how to realize keyword detection is the current research hotspot.

SUMMARY OF THE INVENTION

The present application provides a data processing method, data processing device, computer equipment, computer-readable storage medium and computer program product, which can realize keyword detection in multiple languages, realize automation and intelligence of keyword detection, and improve keyword detection. s efficiency.

The application provides a data processing method, the method includes: acquiring speech data to be recognized, and acquiring a target syllable sequence of a reference keyword;

Invoke the keyword detection model to process the above-mentioned speech data to be recognized, determine the syllable sequence to be detected of the above-mentioned speech data to be recognized, and determine the keyword detection result of the above-mentioned speech data to be recognized according to the above-mentioned syllable sequence to be detected and the above-mentioned target syllable sequence;

Wherein, the above keyword detection model is obtained by using a training data set, and the above training data set includes sample speech data of one or more language categories; the above keyword detection model can be used for any language in the above one or more language categories Category speech data for keyword detection.

The application provides a data processing device, the device includes:

an acquisition module, used for acquiring the speech data to be recognized, and acquiring the target syllable sequence of the reference keyword;

The processing module is used to call the keyword detection model to process the above-mentioned speech data to be recognized, determine the syllable sequence to be detected of the above-mentioned speech data to be recognized, and determine the key of the above-mentioned speech data to be recognized according to the above-mentioned syllable sequence to be detected and the above-mentioned target syllable sequence word detection result;

Wherein, the above keyword detection model is obtained by using a training data set, and the above training data set includes sample speech data of one or more language categories; the above keyword detection model can be used for any language in the above one or more language categories Category speech data for keyword detection.

The present application provides a computer device, comprising: a memory and a processor, wherein the memory stores a data processing program, and when the data processing program is executed by the processor, the steps of the above data processing method are implemented.

The present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program includes program instructions, and the program instructions are executed by a processor to execute the steps of the above-mentioned data processing method.

The present application provides a computer program product, wherein the computer program product includes a computer program or computer instructions, and the computer program or computer instructions are executed by a processor to implement the above-mentioned data processing method.

The application first obtains the speech data to be recognized and the target syllable sequence of the reference keyword, and then calls the keyword detection model to process the speech data to be recognized, determines the syllable sequence to be detected of the speech data to be recognized, and then determines the syllable sequence to be detected according to the syllable sequence to be detected and the target syllable. The sequence determines the keyword detection results of the speech data to be recognized, which can realize the automation and intelligence of keyword detection and improve the efficiency of keyword detection; the keyword detection model proposed in this application can be obtained by training sample speech data in multiple languages. Yes, keyword detection can be performed on speech data of multiple language categories, which improves the applicability of keyword detection and further improves the intelligence of keyword detection.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, for those of ordinary skill in the art, without creative efforts On the premise, other drawings can also be obtained according to these drawings.

1 is a schematic diagram of the architecture of a data processing system provided by an exemplary embodiment of the present application;

2 is a schematic flowchart of a data processing method provided by an exemplary embodiment of the present application;

3 is a schematic flowchart of a keyword detection provided by an exemplary embodiment of the present application;

4 is a schematic flowchart of another data processing method provided by an exemplary embodiment of the present application;

5 is a schematic structural diagram of a factorized time-delay neural network provided by an exemplary embodiment of the present application;

6 is a schematic diagram of the structure and process flow of a syllable recognition network provided by an exemplary embodiment of the present application;

7 is a schematic flowchart of another keyword detection provided by an exemplary embodiment of the present application;

8 is a schematic block diagram of a data processing apparatus provided by an exemplary embodiment of the present application;

FIG. 9 is a schematic block diagram of a computer device provided by another exemplary embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

The embodiment of the present application provides a data processing method, so as to realize the automation and intelligence of keyword detection and improve the efficiency of keyword detection. The data processing methods provided in the embodiments of the present application may be implemented by one or more technologies in artificial intelligence technologies.

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes natural language processing technology, computer vision technology, machine learning/deep learning and other major directions. The solutions provided by the embodiments of the present application involve technologies such as natural language processing and machine learning under artificial intelligence technology, and the natural language processing technology and machine learning technology will be described below.

Natural language processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between humans and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, the language that people use on a daily basis, so it is closely related to the study of linguistics. Natural language processing technology usually includes speech recognition, text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies. This application mainly relates to speech recognition technology in natural language processing technology. Specifically, the terminal device obtains the speech data to be detected, and uses the speech recognition technology to convert the speech data into the corresponding syllable sequence form (that is, the to-be-detected speech data in this application). detection syllable sequence and target syllable sequence). Subsequently, a keyword detection operation may be performed based on the syllable sequence to be detected and the target syllable sequence.

Machine Learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. Machine learning specializes in how computers simulate or realize human learning behaviors to acquire new knowledge or skills, and to reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications are in all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other techniques. The present application mainly relates to artificial neural networks in machine learning technology. Specifically, the terminal equipment trains the initialized neural network through the training data set to obtain a keyword detection model, and then uses the keyword detection model to process the speech data to be recognized, and obtains The sequence of syllables to be detected. The keyword detection is performed based on the syllable sequence to be detected and the target syllable sequence of the keyword, which improves the detection efficiency and intelligence of the keyword.

With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones , robots, intelligent medical care, intelligent customer service, Internet of Vehicles, autonomous driving, intelligent transportation, etc. With the development of technology, artificial intelligence technology will be applied in more fields and play more and more important value.

Keyword detection (Spoken Term Detection) is a sub-field of speech recognition, and its purpose is to detect all occurrences of a specified word in a speech signal. Existing keyword detection techniques are generally based on keyword/stuffing detection methods. The main idea of this method is to use a keyword model to output keyword detection results, while a padding model is used to absorb non-keyword speech. In order to build a complete set of speech keyword detection system based on keyword/filling, the traditional method selects phoneme as the modeling unit, and then uses various deep neural networks as acoustic models, and then guides the training of the model through the loss function.

The main disadvantage of the keyword/stuffing-based keyword detection method is that the acoustic model parameters of keyword detection are many and large, resulting in a slow inference speed. In addition, the traditional keyword/stuffing-based keyword detection methods are all done in a single language, and do not support multilingual keyword detection tasks.

The present invention proposes a multilingual keyword detection method based on a factorized time delay neural network (Factorized Time Delay Neural Network, TDNN-F). The method firstly mixes the corpus of each language in proportion, uses the multi-task learning method to train the Acoustic model (AM), and then uses the Weighted Finite State Translator (WFST) method to decode and output the keyword results . During multi-task learning, we use syllables as output units, and each language has a separate output. The multi-task learning method can treat each language as a separate branch, and the weight of the encoder part of each branch is shared, but the final normalization layer (Softmax layer) is independent. Therefore, it can not only learn common information between languages, but also map the encoder output to the corresponding languages through the Softmax layer. This method can be effectively applied to multilingual keyword detection tasks, while the previous keyword detection methods are generally done in a single language and do not support multilingual keyword detection tasks.

In addition, the present application proposes to add a singular value decomposition layer (The Singular Value Decomposition, SVD) to the TDNN, which effectively reduces the training parameters of the acoustic model, and significantly speeds up the inference speed when the effect is not significantly different. Experiments show that the effect of this method exceeds that of language recognition and monolingual system cascading methods, and the number of correctly reported keywords under the same accuracy rate is increased by 1.2%-3.6%.

The keyword detection method proposed in this application can be applied to the fields of smart device wake-up interaction and voice keyword detection of audio and video files. In order to cope with the limited computing resources of smart devices and the processing requirements of massive audio and video files, we usually need the detection speed of keywords to be fast enough, and then we hope that the detection results are accurate enough on this basis, and finally we hope that the detection method has more functions. , more flexibility.

This application can be applied to various scenarios such as cloud technology, artificial intelligence, smart platforms, and in-vehicle Internet. In the field of cloud technology, the application can store the speech data to be recognized, the syllable sequence to be detected, the reference keyword, the target syllable sequence and the keyword detection result on the cloud server, which is convenient for data management and reuse. Results and other data can be obtained directly from the cloud server; in the field of artificial intelligence, the keyword detection method can be used to wake up smart devices and other operations, making smart devices more user-friendly. Intelligent application services; in the field of intelligent platforms, after obtaining the user's consent, the keyword detection method can be used to perform statistics and analysis on the voice data input by the user, determine different types of user classification tags for different keywords, and conduct correlation based on user tags. Recommendations to ensure a good user experience; in the field of in-vehicle Internet, the keyword detection method can be used to wake up the in-vehicle equipment, and it can also collect the voice data of the passengers in the car in real time, and conduct intelligent interaction according to the detected keywords. For example, when the keywords "tired, sleepy, bored" are detected, the in-vehicle device can play music, and when the keywords "smoking, hot, stuffy" are detected, the in-vehicle device can open the window.

The application will be specifically described by the following examples:

Please refer to FIG. 1. FIG. 1 is a schematic structural diagram of a data processing system provided by an exemplary embodiment of the present application. As shown in FIG. 1 , the data processing system may specifically include a terminal device 101 and a server 102, and the terminal device 101 and the server 102 are connected through a network, for example, through a wireless network connection. Based on the data processing method proposed in this application, the terminal device 101 can collect the speech data to be recognized, and the terminal device 101 performs processing such as feature extraction, syllable recognition, and keyword matching, and in the process of processing the collected speech data to be recognized The data, keyword detection results, and intermediate data in the processing process are sent to the server 102, which is convenient for the server 102 to carry out subsequent data management; the server 102 can also perform feature extraction, syllable recognition, and keyword matching processing. When the server 102 executes At this time, the terminal device 101 can collect the voice data to be recognized, send the voice data to be recognized to the server 102 for the above processing, and obtain the keyword detection result, and the server 102 returns the keyword detection result to the terminal device 101, and then performs subsequent operations .

In this embodiment of the present application, the terminal device 101 may acquire the voice data to be recognized and the reference keywords; the terminal device 101 may send the voice data to be recognized and the reference keywords to the server 102; the server 102 determines the voice data to be recognized according to the voice data to be recognized The syllable sequence to be detected is determined, and the target syllable sequence of the reference keyword is determined according to the reference keyword; the server 102 invokes the keyword detection model to perform matching processing on the syllable sequence to be detected and the target syllable sequence of the speech data to be recognized to obtain the key of the speech data to be recognized. The word detection result is obtained, and the keyword detection result is sent to the terminal device 101; the terminal device 101 displays the keyword detection result on the user interface according to the received keyword detection result.

The terminal equipment 101 is also referred to as a terminal (Terminal), user equipment (UE), an access terminal, a subscriber unit, a mobile device, a user terminal, a wireless communication device, a user agent, or a user equipment. Terminal devices can be smart home appliances, handheld devices with wireless communication functions (such as smart phones, tablet computers), computing devices (such as personal computers (PCs), in-vehicle terminals, intelligent voice interaction devices, wearable devices, or other smart devices. device, etc., but not limited to this.

The server 102 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, Cloud servers for basic cloud computing services such as middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

It can be understood that the schematic diagram of the architecture of the system described in the embodiments of the present application is to more clearly illustrate the technical solutions of the embodiments of the present application, and does not constitute a limitation on the technical solutions provided by the embodiments of the present application. For example, in this application, the terminal device 101 may include more than three devices in addition to the three devices shown in FIG. It may be composed of multiple servers (ie, server clusters), and the server 102 may also be a local server established on the terminal device 101 . Those of ordinary skill in the art know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

Please refer to FIG. 2. FIG. 2 is a schematic flowchart of a data processing method provided by an exemplary embodiment of the present application, and the method is applied to the terminal device (referring to the above-mentioned terminal device 101) in FIG. equipment to describe) as an example to illustrate, the method may include the following steps:

S201. Acquire the speech data to be recognized, and acquire the target syllable sequence of the reference keyword.

In the embodiment of the present application, the speech data to be recognized is the target speech that needs to be subjected to keyword detection, the reference keyword is the keyword to be detected that needs to be subjected to keyword detection, and the target syllable sequence of the reference keyword is obtained to obtain the keyword detection. Raw data, and subsequent keyword detection methods are based on the raw data. This embodiment of the present application is to determine whether the target speech includes a reference keyword.

In an embodiment, the voice data to be recognized in the present application may be voice data obtained by the terminal device through a sound pickup device (such as a microphone) configured by the terminal device, or may be obtained by the terminal device from other devices in the network through the Internet. The voice data may also be voice data obtained directly from a local storage device (such as a U disk).

In one embodiment, the speech data to be recognized can be either the speech data of the official language (such as Chinese, English) used in any country, or the local language (such as Cantonese, Hokkien, Sichuan) voice data.

In one embodiment, the reference keywords of the present application are preset, and it is necessary to perform keyword detection on the reference keywords in the speech data to be recognized to determine whether the reference keywords exist in the speech data to be recognized. Reference keywords can be keywords in official languages (such as Chinese, English, and Russian) used in any country, or keywords in local languages (such as Cantonese, Fujian, Sichuan) used in any region in any country. Keywords; in addition, the reference keyword can be one or more than one.

In one embodiment, the method of obtaining the target syllable sequence of the reference keyword may be:

(1) Obtain reference keywords, and obtain the target language category of the speech data to be recognized.

The reference keywords may be in different languages, and the speech data to be recognized may also be in different languages. Therefore, it is necessary to perform language matching processing on the reference keywords and the to-be-recognized speech data, so that in the subsequent steps, according to the reference keywords The syllable sequence and the syllable sequence of the speech data to be recognized are subjected to keyword detection.

(2) Obtain the syllable sequence of the reference keyword that matches the target language category, and determine the syllable sequence of the reference keyword that matches the target language category as the target syllable sequence.

In one embodiment, the language category (that is, the target language category) of the speech data to be recognized can be obtained first, then the reference keyword is obtained, and the reference keyword is subjected to speech conversion processing to obtain the reference keyword corresponding to the target language category. Reference keyword syllable sequence (ie, target syllable sequence).

In one embodiment, the target syllable sequence of the reference keyword can be generated through a pronunciation dictionary model. The pronunciation dictionary model can include multiple branch networks, and can perform text conversion operations on reference keywords in different languages to generate syllable sequences in the target language. Exemplarily, when the language category of the speech data to be recognized is language A and the language category of the reference keyword is language B, the reference keyword of language B can be converted into a syllable sequence of language A according to the pronunciation dictionary model. It should be noted that the pronunciation dictionary model can also first translate the reference keywords of language B into language A, and then perform a syllable recognition operation on the translation result to obtain the target syllable sequence.

S202: Invoke a keyword detection model to process the to-be-recognized speech data, determine a to-be-detected syllable sequence of the to-be-recognized speech data, and determine the to-be-recognized speech data according to the to-be-detected syllable sequence and the target syllable sequence keyword detection results.

In the embodiment of the present application, the keyword detection model can obtain the syllable sequence of the speech data to be recognized (that is, the syllable sequence to be detected) through operations such as feature extraction and acoustic modeling. When the syllable sequence to be detected and the target syllable sequence are obtained , based on the idea of similarity judgment, detect the segment with the highest similarity with the target syllable sequence in the syllable sequence to be detected, and when the similarity of the segment reaches the preset condition, it can be judged that there is a target syllable sequence in the syllable sequence to be detected ( That is, it is determined that there is a reference keyword in the speech data to be recognized), and a keyword detection result is obtained.

In one embodiment, if the reference keyword includes multiple sub-keywords, that is, the target syllable sequence includes multiple sub-syllable sequences, the keyword detection model may perform keyword detection on the multiple sub-syllable sequences included in the target syllable sequence, respectively, and output multiple sub-syllable sequences. The sub-keywords in the sub-syllable sequence that meet the preset conditions are used as the keyword detection result.

In one embodiment, the speech data to be recognized may also include a time sequence of syllables corresponding to the sequence of syllables to be detected. The syllable time sequence corresponding to the syllable sequence is detected, and the occurrence time of the sub-keywords satisfying the preset condition is determined; then the sub-keywords satisfying the preset condition and the occurrence time are output.

In one embodiment, the keyword detection model is obtained by training a training data set, and the training data set includes sample speech data of one or more language categories; the keyword detection model can be used for any language in one or more language categories. Category speech data for keyword detection.

The keyword detection model may include a feature extraction network and a syllable recognition network. The keyword detection model performs keyword detection through the syllable sequence to be detected and the target syllable sequence, mainly based on the syllable recognition network in the keyword detection model. The method of training the keyword detection model may include the following steps:

(1) Acquire a training data set, the training data set includes one or more training data subsets, and each training data subset includes sample speech data of any language category in one or more language categories and corresponding sample speech data. The reference syllable sequence of .

When the training data set includes a subset of training data (that is, the training data set only includes sample speech data of one language category), the syllable recognition network of the keyword detection model can be trained based on the sample speech data of this language category ; When the training data set includes multiple training data subsets (that is, the training data set includes sample speech data in multiple languages), the corpus of each language can be mixed in proportion, for example, referring to the language distribution ratio of online speech data. The training corpus of each language is mixed to make the model perform better in actual business.

It should be noted that in the training process of the keyword detection model, the weight occupied by each language has a great influence on the bias of the final model. In order to make the final model not biased towards any one language, the language weights can be equally divided. This method has better training effect when the amount of training data is similar, but in actual business, the training corpus of each language is quite different. . Therefore, the language weight needs to be adjusted according to the number of languages, actual training data and business needs.

(2) Perform feature extraction processing on sample speech data in one or more training data subsets to obtain sample speech features of the sample speech data.

(3) Using the obtained sample speech features and the corresponding reference syllable sequence, train the initial syllable recognition network to obtain the trained syllable recognition network.

In the embodiment of the present application, the sample speech features of the sample speech data are input into the initial syllable recognition network to obtain the predicted syllable sequence, and then based on the predicted syllable sequence and the reference syllable sequence, the network parameters of the initial syllable recognition network are adjusted in combination with the loss function, When the predicted syllable sequence of the subsequent sample speech features satisfies the preset condition, it can be judged that the training of the initial syllable recognition network is completed, and the trained syllable recognition network is obtained.

In one embodiment, the model training method may be as follows: using a loss function to perform backpropagation, updating the weights of each node in the initialized syllable recognition network, and when the loss function satisfies a preset condition, determine each node in the initialization model. The final weight of the node, where the loss function can use one or more of the 0-1 loss function, the absolute value loss function, the mean squared loss function, the logarithmic loss function and the exponential loss function.

(4) Generate a trained keyword detection model according to the trained syllable recognition network.

After the training of the syllable recognition network is completed, a trained keyword detection model can be generated based on the feature extraction network, the syllable recognition network, and the like. Among them, the feature extraction network can also be trained by using the training data set to achieve more accurate feature extraction.

In one embodiment, the training keyword detection model can use the same training data set to jointly train the feature extraction network and the syllable recognition network, and the way to jointly train the feature extraction network and the syllable recognition network can be:

The feature extraction network and the syllable recognition network each contain an optimizer. Each optimizer corresponds to the network parameters of the network. By fusing the two network parameters, joint training can be performed to obtain the trained feature extraction network and syllable recognition network. . It should be noted that joint training is a method to improve the prediction accuracy of the model, and the prediction accuracy of the model is also affected by many aspects, such as the quality of training samples. Therefore, in the use of the method provided in this application, an appropriate training method should be selected in combination with the actual situation and the prediction result, so as to achieve a higher model prediction accuracy.

Exemplarily, the keyword detection model includes a feature extraction network, a syllable identification network, and a keyword matching network; the syllable identification network includes one or more identification sub-networks, and each identification sub-network is used to perform a voice data analysis on a specified language category. Syllable recognition, the specified language category is contained in one or more language categories.

Each recognition sub-network included in the syllable recognition network can perform syllable recognition on the speech data of a specified language category, which includes the following two cases. identification sub-network.

In the first case, when the syllable recognition network includes a recognition sub-network, the syllable recognition network can perform syllable recognition on the speech data of the language category corresponding to the syllable recognition network, and obtain the syllable sequence corresponding to the speech data (that is, the syllable sequence to be detected). ).

In the second case, when the syllable identification network includes multiple identification sub-networks, the network parameters of any two identification sub-networks in the multiple identification sub-networks match. That is, the weights of the coding layers of the recognition sub-network corresponding to one or more language categories are shared. The weight sharing of neural network means that the information learned from a local area is applied to other places of the data to be processed, so as to reduce the amount of calculation and improve the training speed of the model. It should be noted that, in the multiple identification sub-networks, the weights of any two identification sub-networks may be shared, or the weights of all identification sub-networks in the multiple identification sub-networks may be shared.

Please refer to FIG. 3. FIG. 3 is a schematic flowchart of a keyword detection. First, the speech data to be recognized (for example: hello, world) is input into the feature extraction network to obtain the speech features of the speech data to be recognized; The speech features of the speech data are input into the syllable recognition network to obtain the syllable sequence to be detected of the speech data to be recognized; the reference keyword is used to generate a vocabulary for the pronunciation dictionary (for example: the syllable sequence of "Hello" is 'ni3 hao3', The syllable sequence of "world" is 'shi4 jie4', where '3' or '4' represent the corresponding pitch of the syllable respectively), and the target syllable sequence corresponding to the reference keyword is obtained; then the keyword matching network is used to compare the target syllable sequence and The syllable sequence to be detected is matched to obtain an output result (that is, a keyword matching result).

Among them, the feature extraction network converts continuous speech information into discrete vector representations through digital signal processing algorithms. These vectors can effectively represent relevant speech features and are beneficial to subsequent speech tasks; the syllable recognition network is mainly based on the given Feature sequence and state sequence to train the model, in which the state output probability of the model can be determined by Graph Neural Network (GNN), Deep Neural Networks (DNN), Convolutional Neural Networks (Convolutional Neural Networks, CNN) , Recurrent Neural Network (RNN), Time Delay Neural Network (TDNN) and other modeling, and the state transition probability of the model can be determined by Hidden Markov Model (Hidden Markov Model, HMM) etc. to model; the vocabulary generation operation is to generate the syllable sequence corresponding to the keyword through the dictionary; the keyword matching network mainly searches all possible state spaces through the trained model, and then finds the most likely state sequence of the input speech features, so that The output has the highest probability. The keyword detection model can detect the keywords in the speech and the starting and ending positions of the keywords by cascading the above networks and methods.

Exemplarily, the present application can use the multi-task learning method to train each language as a separate branch, the weights of the encoder parts of each branch are shared, and the final Softmax layer is independent. Therefore, through the above method, the common information between each language can be learned, and the encoder output can also be mapped to the corresponding language through the Softmax layer.

Exemplarily, after obtaining the weight of the model as a floating-point number, the relevant clustering method is used to divide into different categories according to the degree of similarity in size. Each category only needs to save the weight of one cluster center and the corresponding cluster index. (Index), which can greatly reduce the amount of data. Backpropagation is performed when the weights are updated, the gradient of each weight is calculated, the gradients of the same category of the previous clustering are accumulated, and the weights are updated in combination with the learning rate (LR).

The application first obtains the speech data to be recognized and the target syllable sequence of the reference keyword, and then calls the keyword detection model to process the speech data to be recognized, determines the syllable sequence to be detected of the speech data to be recognized, and then determines the syllable sequence to be detected according to the syllable sequence to be detected and the target syllable. The sequence determines the keyword detection results of the speech data to be recognized, which can realize the automation and intelligence of keyword detection and improve the efficiency of keyword detection; the keyword detection model proposed in this application can be obtained by training sample speech data in multiple languages. It can perform keyword detection on speech data of multiple language categories, and can match the reference keyword with the language category of the speech data to be recognized, which improves the applicability of keyword detection and further improves the accuracy of keyword detection. Intelligent; in the training of the keyword detection model, the weight sharing is used to improve the training speed of the model, and the training data set is mixed according to the distribution ratio of the language of the online voice data, which improves the detection accuracy of the keyword detection model.

Please refer to FIG. 4. FIG. 4 is a schematic flowchart of a data processing method provided by another exemplary embodiment of the present application. Taking the method applied to the terminal device in FIG. 1 as an example, the method may include the following steps:

S401. Acquire the speech data to be recognized, and acquire the target syllable sequence of the reference keyword.

For the specific implementation of step S401, refer to the relevant description of step S201 in the foregoing embodiment, which will not be repeated here.

S402 , invoking the feature extraction network of the keyword detection model to process the speech data to be recognized to obtain speech features of the speech data to be recognized.

In one embodiment, the above step S402 may be implemented by the following methods:

(1) Call the feature extraction network to perform frame processing on the speech data to be recognized, and obtain the time domain features of the speech data to be recognized.

In the embodiment of the present application, since the duration of the sound that can be heard by the human ear is at least 10 ms, the digital signal needs to be divided into a plurality of audible blocks, that is, divided into frames. For example, the 10s-long speech data to be recognized may be divided into a frame of 25ms to obtain 400 speech frames, and the 400 speech frames are used as the time domain features of the speech data to be recognized.

(2) Call the feature extraction network to perform windowing and frequency domain transformation processing on the time-domain features to obtain frequency-domain features.

The frequency domain feature is to add a window on the basis of the time domain feature, and then perform frequency domain transform processing (typical methods such as Fourier transform processing). Windowing is to use a window function to process each frame, eliminating the samples at both ends of a frame, so that a period of periodic signal can be generated. If the frequency domain transformation is performed directly without windowing, it will cause spectrum leakage ( spectral leakage).

(3) The frequency domain feature is used as the voice feature of the voice data to be recognized.

After obtaining the frequency domain feature of the voice data to be recognized, the frequency domain feature can be used as the voice feature of the voice data to be recognized.

S403 , invoking the syllable recognition network of the keyword detection model to process the speech feature, and obtain the to-be-detected syllable sequence of the speech data to be recognized.

In the embodiment of the present application, acquiring the syllable sequence to be detected of the speech data to be recognized is to perform keyword matching between the syllable sequence to be detected of the speech data to be recognized and the target syllable sequence of the reference keyword, so as to obtain a keyword detection result. The keyword detection model includes a syllable recognition network, and the function of the syllable recognition network is to generate a syllable sequence corresponding to the input speech data.

In one embodiment, the above step S403 can be implemented by the following methods:

(1) The syllable recognition network of the keyword detection model is invoked to process the speech features, and the target language category of the speech data to be recognized is determined according to the speech features.

In the embodiment of the present application, the syllable recognition network of the keyword detection model first determines the language category corresponding to the speech feature, so as to use the recognition sub-network corresponding to the language category to process the speech feature and improve the processing accuracy.

In one embodiment, the syllable recognition network includes a language recognition network, and the keyword detection model calls the language recognition network to process the speech data to be recognized to obtain the target language category of the speech data to be recognized. Language identification network can use CV syllable division method, linear prediction residual and other methods for language identification.

In one embodiment, the language recognition network is independent of the syllable recognition network, before the syllable recognition network and after the feature extraction network. The language recognition network first obtains the speech features output by the feature extraction network, processes the speech features, and obtains the target language category of the speech data to be recognized, and then sends the speech features of the speech data to be recognized and the target language category to the syllable recognition network for processing.

(2) Call the recognition sub-network corresponding to the target language category in the syllable recognition network to process the speech feature, and obtain the syllable distribution probability of the speech feature.

According to the syllable characteristics of each language, the syllable category corresponding to the language category can be preset. For example, there are 1000 syllable categories in language A, then given a piece of speech in the target language category, each frame in the speech corresponds to the target language category The identification sub-network of can get the distribution probability of 1000 syllable categories (that is, the syllable distribution probability).

(3) Determine the to-be-detected syllable sequence of the speech data to be recognized according to the syllable distribution probability.

In one embodiment, after calling the recognition sub-network corresponding to the target language category in the syllable recognition network to process each frame of the speech features of the speech data to be recognized, the syllable distribution probability corresponding to each frame of speech is obtained respectively, and the maximum The syllable corresponding to the probability is used as the target syllable; when the syllable distribution probability of all speech features is determined, that is, multiple target syllables are obtained, and the multiple target syllables are combined as the to-be-detected syllable sequence of the speech data to be recognized.

In one embodiment, the syllable recognition network is constructed and generated based on a time-delay neural network and a singular value decomposition network.

In the embodiments of the present application, the time-delay neural network (ie, TDNN) is used to solve the problem that the traditional method Hidden Markov Model (HMM) in speech recognition cannot adapt to dynamic time domain changes in speech signals. TDNN has fewer structural parameters, and does not need to align phonetic symbols and audio on the timeline in advance for speech recognition. Experiments show that TDNN performs better than HMM. Two obvious features of TDNN are dynamic adaptation to temporal feature changes and fewer parameters. The input layer of traditional deep neural network is connected to the hidden layer one by one. TDNN has changed here, that is, the features of the hidden layer are not only the same as the current one. The input of the time is related to the input of the future time, so TDNN has the ability to express the relationship of speech features in time, and can facilitate the learning of the TDNN network through the method of weight sharing.

Singular value decomposition (ie SVD) is a method of decomposing a matrix, but unlike the eigenvalue decomposition method, eigenvalue decomposition is only applicable to square matrices, but SVD does not require the matrix to be decomposed to be a square matrix, and can be applied to Decomposition of arbitrary matrices. Using SVD, the original data set can be represented by a smaller data set, thereby removing noise and redundant information, greatly reducing the amount of data, and improving the operation speed.

The present invention selects syllables as modeling units, constructs SVD network based on SVD method, combines TDNN and uses factorization time delay neural network (ie TDNN-F) as syllable recognition network. The main role of the syllable recognition network is to convert the speech input into an acoustically represented syllable output. Specifically, for each speech frame, the probability distribution of the syllable to which it belongs is calculated.

Please refer to FIG. 5. FIG. 5 is a schematic structural diagram of the TDNN-F network proposed by the present application. The TDNN-F network is a multi-layer network, each layer has a strong ability to abstract speech features, and it has a wider contextual field of view, which can capture a wider range of contextual information, and builds speech timing dependent information. The mold ability is stronger. In addition to the common layer structure included in the TDNN-F network (the network structure composed of black solid circles in Figure 5), an SVD layer can be inserted between some TDNN layers (the network structure framed by SVD in Figure 5). Exemplarily, the TDNN-F model has N layers, and each TDNN layer and SVD layer have M and K nodes, respectively, where K is much smaller than M. Then the number of model parameters D1 of TDNN is:

D1=N*M*M

The number of model parameters D2 of TDNN-F is:

D2=N*(M+M)*K

It can be seen from the above two model parameter calculation formulas that the number of model parameters D1 of TDNN is much larger than the number of model parameters D2 of TDNN-F. Compared with the TDNN network, the TDNN-F network can effectively reduce the number of The training parameters of the model to speed up the inference speed of the model.

It should be noted that a network model with larger parameters can achieve better results, but at the same time, it will slow down the inference speed of the model. Therefore, in order to achieve a better balance between speed and effect, the network model usually needs to optimize the parameters. In the TDNN-F network proposed in this application, the main parameters affecting the network effect are the number of network layers, the number of nodes in each layer, the number of SVD nodes, and the width of the context view.

According to the proposed TDNN-F network above, a syllable recognition network based on the TDNN-F network can be constructed. Please refer to FIG. 6. FIG. 6 is a schematic diagram of the structure and flow of a syllable recognition network proposed by the present application. The syllable recognition network is based on the TDNN-F network and adopts the Chain Model method (a method for sequence discrimination training) and Multi-task learning methods treat each language category as an independent branch. The specific process of the syllable recognition network is as follows: take the speech features of the speech data to be recognized as the input of the syllable recognition network, process the speech features of different language categories through Chain Model network processing, and then output multiple language categories (such as language 1 and The syllable sequence corresponding to language N) (that is, the syllable sequence to be detected).

Among them, the Chain Model network is a multi-layer network structure (the network layer where the black solid circle is located in Figure 6 is the input layer, the layer where the hollow circle is located is the middle layer, and the layer where the slashed circle is located is the output layer). It also includes a multi-layer TDNN-F network. Each TDNN-F network consists of the i-th layer in the Chain Model network (i refers to any layer in the middle layer), the i-1 layer, and the Dropout layer (the neuron discarding layer, with It is used to set the probability of the specified neural network unit to zero in the use of the neural network), TDNN-F layer composition and multiple channels; multiple channels include direct channels from the i-1 layer to the i layer (that is, the skip layer connection ), also including the ordinary channel from the i-1 layer to the i layer through the TDNN-F layer and the Dropout layer; the model training is more stable through the skip layer connection and the Dropout method;

In one embodiment, the TDNN-F network-based syllable recognition network constructed in this application can use the sub-sampling layer and the method of sharing weights to reduce the amount of calculation, and can also use the frame-level cross-entropy (Cross-entropy) loss function and On this basis, the LF-MMI loss function is added to guide the error transfer of the TDNN-F model; in the network output stage, the syllable recognition network proposed in this application will perform keyword matching operations on the branches of each language category. When the matching result of the keyword satisfies the preset condition, output the keyword that satisfies the condition, and the predicted probability and start and end position of the keyword.

According to the syllable recognition network based on TDNN-F network, this application proposes a method for detecting multilingual speech keywords based on TDNN-F, and the specific implementation is as follows:

Please refer to FIG. 7. FIG. 7 is a schematic flowchart of the method for detecting speech keywords based on TDNN-F multilingual proposed by the present application. First, the speech data to be recognized (for example: hello, world) is input into the feature extraction network to obtain the speech data to be recognized. The speech features of the speech data; then input the speech features of the speech data to be recognized into the syllable recognition network to obtain the syllable sequence to be detected of the speech data to be recognized. The syllable recognition network is based on TDNN-F and multi-task learning. The speech features of the speech data to be recognized are separately processed to obtain syllable sequences to be detected corresponding to multiple languages (eg, language 1 and language N). Then, the method based on Hidden Markov (HMM) and Lattic-FreeMaximum Mutual Information (LF-MMI) is used to realize the state jump of the model. The multilingual speech key based on TDNN-F proposed in this application is The word detection method is based on HMM, which describes a Markov process with hidden unknown parameters. The Markov process contains visible states (such as syllables in speech data, such as S1, S2... in Figure 7) and implicit states (such as The contextual relationship between the syllables in the speech data, such as the arrow pointing to itself above S2 in Figure 7). By this method, the syllable sequence of the speech data to be detected and the implicit state between the syllable sequences can be fully utilized, and the utilization rate of the data is improved.

In this embodiment of the present application, a multilingual pronunciation dictionary can be used to generate a vocabulary for the reference keywords (that is, the multilingual keywords in the figure) (for example, the syllable sequence of the keyword "hello" includes 'ni3 hao3', 'nei5' hou2', etc., the syllable sequence of "world" includes 'shi4 jie4', 'sai3 jaai3', etc., where '2', '3' or '4' represent the tones corresponding to the syllable, respectively, to obtain multiple reference keywords corresponding to After the target syllable sequence of the language type, the keyword matching network is used to match the target syllable sequence and the syllable sequence to be detected to obtain an output result (that is, a keyword matching result).

S404 , invoking the keyword matching network of the keyword detection model to process the syllable sequence to be detected and the target syllable sequence to obtain a keyword detection result of the speech data to be recognized.

In one embodiment, the syllable sequence to be detected may include one or more syllable elements arranged in a first order, and the first order may be arranged according to the sequence of the syllables in the speech data to be recognized; the target syllable sequence includes one or more syllable elements. The plurality of syllable elements are arranged in a second order, and the second order may be arranged according to the sequence of the syllables in the keyword; it may also be adjusted and sorted according to the components of the keyword. For example, when the keyword is "hello, world", it can be sorted according to the syllable order of the keyword, that is, multiple syllable elements based on "hello world" are obtained; or according to the composition of the keyword Make adjustments (the keywords include "hello" and "world"), get the syllable order of "hello world", and get multiple syllable elements based on "hello world".

In one embodiment, the above step S404 can be implemented by the following methods:

(1), call the keyword matching network of the keyword detection model to process the syllable sequence to be detected and the target syllable sequence, and detect whether there is a matching syllable element that matches the first syllable element in the target syllable sequence, and the first syllable element is to be Detect any syllable element in a syllable sequence.

In the embodiment of the present application, the first syllable element can be taken from any syllable element in the syllable sequence to be detected, and the above steps can be understood as: calling the keyword matching network of the keyword detection model, and respectively matching the target syllable sequence with the syllable to be detected Each syllable element in the sequence (in each matching process, the selected syllable element in the syllable sequence to be detected can be used as the first syllable element) is matched, and then it is detected whether there is a match with the first syllable element in the target syllable sequence of matching syllable elements.

In one embodiment, the method for judging whether there is a matching syllable element that matches the first syllable element in the target syllable sequence may be: calling a keyword matching network to carry out each syllable element in the target syllable sequence and the first syllable element respectively. Match, obtain multiple matching probabilities of each syllable element in the corresponding target syllable sequence, obtain the maximum matching probability among the multiple matching probabilities, if the maximum matching probability is greater than the syllable matching probability threshold, then judge the syllable element corresponding to the maximum matching probability as Matching the syllable element, that is, judging that there is a matching syllable element matching the first syllable element in the target syllable sequence.

(2), if there is a matching syllable element that matches the first syllable element, then detect whether there is a matching syllable element that matches the second syllable element in the target syllable sequence, and the second syllable element is the syllable sequence to be detected. The syllable element after the first syllable element.

In one embodiment, after judging that there is a matching syllable element that matches the first syllable element in the target syllable sequence, the next syllable element of the first syllable element can be used as the second syllable element, according to above-mentioned steps (1. ), the keyword matching network is called to perform matching processing on the target syllable sequence and the second syllable element.

(3) If there is a matching syllable element that matches the second syllable element, determine the keyword detection result of the speech data to be recognized according to the first syllable element and the second syllable element.

In one embodiment, the syllable elements of the reference keyword are 2, and the method based on steps (1) and (2) carries out syllable matching, if there is a matching syllable element matching the first syllable element in the target syllable sequence, and the target If there is a matching syllable element in the syllable sequence that matches the second syllable element (in this case, the second syllable element includes one syllable element), it is determined that the speech data to be recognized has the target syllable sequence.

In an embodiment, the syllable elements of the reference keyword are K (K is greater than 2), then the second syllable element may include K-1 syllable elements (wherein the K-1 syllable elements may be sorted according to the second order) , after judging that there is a matching syllable element that matches the first syllable element in the target syllable sequence according to the method provided by step (1), to the first of the K-1 syllable elements according to the method of the above-mentioned step (1) Perform keyword matching, and then perform keyword judgment on the subsequent syllable elements in the K-1 syllable elements in turn when the matching is successful. Recognize that the speech data exists in the target syllable sequence, and output the keyword detection result of the speech data to be recognized.

In one embodiment, outputting the keyword detection result of the speech data to be recognized may specifically include outputting the target keyword judged to exist in the speech data to be recognized, the start and end time of the target keyword, and the predicted probability of the target keyword.

In an embodiment, the predicted probability of the target keyword refers to the matching probability of all syllable elements in the target keyword obtained by fusion, for example, the target keyword contains K (for example, K=3) syllable elements, and the syllable matching probability threshold is V. (eg V=0.9), the matching probabilities of the K syllable elements contained in the target keyword are all greater than the syllable matching probability threshold (for example, the matching probabilities of the three syllable elements are 0.96, 0.98, and 0.94, respectively). The average value of matching probabilities (eg, 0.96) is used as the predicted probability of the target keyword, and the product of matching probabilities of three syllable elements (eg, 0.88) can also be used as the predicted probability of the target keyword.

The keyword matching network may include multiple matching strategies, and the result of the keyword is determined through the matching probability obtained from the multiple matching strategies, which improves the accuracy of keyword detection. The matching strategy can include methods such as dynamic programming, longest sequence first, and exhaustive exhaustion to obtain the optimal path. The matching strategy can be implemented based on one or more of the above methods. When performing keyword matching based on multiple matching strategies, the specific steps are: as follows:

(1) Call multiple matching strategies in the keyword matching network, process the syllable sequence to be detected (including the first syllable element and the second syllable element) and the target syllable sequence respectively, and obtain multiple sets of posterior probability, each The posterior probability set includes a plurality of predicted probabilities of a reference keyword obtained corresponding to one matching strategy among the multiple matching strategies.

Exemplarily, the keyword matching network includes Z (for example, Z=3) matching strategies, and the Z matching strategies are used to process the to-be-detected syllable sequence and the target syllable sequence to obtain Z sets of posterior probabilities (such as each posterior). The probability set includes 3 posterior probabilities, the posterior probability sets are [0.94, 0.96, 0.98], [0.88, 0.94, 0.96] and [0.90, 0.94, 0.92]).

(2) Determine the maximum predicted probability of multiple predicted probabilities in each posterior probability set as the strategy probability, and determine the predicted probability of the reference keyword according to the multiple strategy probabilities.

Exemplarily, the maximum predicted probability of multiple predicted probabilities is determined according to the posterior probability set (for example, the maximum predicted probability corresponding to the first matching strategy is 0.98, the maximum predicted probability corresponding to the second matching strategy is 0.96, and the third matching strategy corresponds to 0.96. The maximum prediction probability is 0.94), and then the prediction probability of the reference keyword is determined according to the maximum prediction probability of multiple prediction probabilities (for example, the three maximum prediction probabilities corresponding to the three matching strategies are 0.98, 0.96, and 0.94, respectively. The probability is averaged to obtain a predicted probability of 0.96 for the reference keyword).

(3) Determine the keyword detection result of the speech data to be recognized according to the predicted probability of the reference keyword.

Exemplarily, if the syllable matching probability threshold is V (for example, V=0.9), and the predicted probability of the reference keyword is 0.96, which is greater than the syllable matching probability threshold, the reference keyword is determined as the target keyword, and then the keyword detection result is output.

In the process of keyword detection, if the voice data to be detected is not accurate (for example, the user's pronunciation is not standard, then the sequence of syllables to be detected identified by the voice data to be detected is relatively ambiguous), the keyword detection method proposed in this application can be used. A correction network is added to the model (it can be before the feature extraction network, or after the syllable recognition network), and the speech data or the syllable sequence to be detected is processed through the built-in grammar rules or syllable pronunciation rules based on multiple languages. Correction to obtain a more accurate syllable sequence to be detected, which can improve the accuracy of subsequent key matching detection.

In addition, the method proposed in this application generates a fuzzy syllable sequence while generating the target syllable sequence according to the reference keyword. The fuzzy syllable sequence is a non-standard form of the target syllable sequence (for example, the reference keyword is "world", the The target syllable sequence of the keyword can be "shi4 jie4", and the fuzzy syllable sequence of the keyword can be "si4jie4"). Detection, so that the non-standardized keyword speech data in the speech data to be detected can be detected, and the intelligence and applicability of the method are improved. keyword detection) can improve the user experience.

When performing keyword matching based on the fuzzy syllable sequence, the target syllable sequence, and the syllable sequence to be detected, it can be implemented in the following ways:

The keyword matching network of the keyword detection model is called to process the syllable sequence to be detected, the target syllable sequence and the fuzzy syllable sequence to obtain the keyword detection result of the speech data to be recognized. According to the method provided in the above step S404, the first predicted probability of the reference keyword (that is, the target syllable sequence) and the second predicted probability of the fuzzy syllable sequence can be obtained respectively. Identify the keyword detection results of speech data.

Exemplarily, the first predicted probability of the target syllable sequence is 0.88, the second predicted probability of the fuzzy syllable sequence is 0.92, and the first syllable matching probability threshold is 0.9. By analyzing the first predicted probability of the target syllable sequence is less than the syllable matching probability threshold (in this case, it means that there is no standardized syllable sequence based on the keyword in the speech data to be detected), but the second predicted probability of the fuzzy syllable sequence is greater than the syllable matching probability threshold (in this case, it means that there is a non-standard syllable sequence based on the keyword in the speech data to be detected), then it is judged that there is a non-standard form of the keyword in the speech data to be detected, and further judgment is made. For example, a second syllable matching probability threshold P (for example, P is 0.85) can be set. If the first predicted probability of the target syllable sequence is less than the first syllable matching probability threshold, the first predicted probability of the target syllable sequence is greater than the second syllable matching. probability threshold, and the second predicted probability of the fuzzy syllable sequence is greater than the syllable matching probability threshold, it is determined that the speech data to be recognized exists in the keyword.

This application tests the proposed keyword detection method based on TDNN-F. First, two target language categories (including language 1 and language 2) are selected, and 27.5 hours of corpus is selected as the test data in each language category. In order to test the performance of the keyword detection model based on TDNN-F, a set based on language recognition + monolingual keyword detection cascade is also built as a baseline for comparison.

See the table below:

method Accuracy recall Harmonic mean Language recognition + monolingual keyword detection cascade 91.6% 64.9% 76.0% Keyword Detection Based on TDNN-F 91.7% 68.5% 78.4% boost effect +0.1% +3.6% +2.45%

The table lists the effect of keyword detection in language 1. In language 1, the accuracy rate of language identification + monolingual keyword detection cascade method is 91.6%, the recall rate is 64.9%, and the harmonic mean is 76.0%; the accuracy rate of keyword detection method based on TDNN-F is 91.7% , the recall rate is 68.5%, and the harmonic mean is 78.4%. Compared with the language recognition + monolingual keyword detection cascade method, the accuracy rate of the keyword detection method based on TDNN-F is calculated to be 0.1%, and the recall rate is improved to 3.6%, and the harmonic mean improvement is 2.45%.

It can be seen from the results that, compared with the cascading method of language recognition + monolingual keyword detection, the keyword detection method based on TDNN-F proposed in this application has a coverage rate increased by 3.6% under the condition that the accuracy is basically the same. The results show that the keyword detection system based on TDNN-F can effectively increase the number of recalled keywords, and the effect is better.

See the table below:

method Accuracy recall Harmonic mean Language recognition + monolingual keyword detection cascade 84.2% 78.5% 81.1% Keyword Detection Based on TDNN-F 84.4% 79.4% 81.82% boost effect +0.2% +1.2% +0.73%

The table lists the effect of keyword detection in language 2. In language 2, the accuracy rate of language identification + monolingual keyword detection cascade method is 84.2%, the recall rate is 78.5%, and the harmonic mean is 81.1%; the accuracy rate of keyword detection method based on TDNN-F is 84.4% , the recall rate is 79.4%, and the harmonic mean is 81.82%. Compared with the TDNN-F-based keyword detection method, the accuracy rate is improved to 0.2%, and the recall rate is improved to 1.2%, and the harmonic mean lift is 0.73%.

It can be seen from the results that, compared with the cascading method of language recognition + monolingual keyword detection, the keyword detection method based on TDNN-F proposed in this application has the same accuracy, and the coverage is increased by 1.2% , this structure verifies the effectiveness of the multilingual keyword detection method based on TDNN-F again.

The syllable recognition network in the keyword detection model proposed in this application includes a language recognition network, which firstly performs language recognition on the speech to be detected, and then calls the recognition sub-network corresponding to the target language category in the syllable recognition network to process the speech features, and determines the speech to be recognized. The syllable sequence of the data to be detected enables the keyword detection model to perform targeted processing on the speech data of different languages, which improves the processing efficiency; this application selects the syllable as the modeling unit, and uses the singular value decomposition method and the time-delay neural network to construct a The factorization delay neural network keyword detection model improves the keyword detection accuracy and detection efficiency, and uses the method of skip layer connection and neuron discarding to further improve the detection efficiency and model training stability; Sort the syllable sequence of the reference keyword and generate the fuzzy syllable sequence corresponding to the reference keyword, so that the keyword detection model can detect different syllable sequences and non-standardized keywords, which improves the applicability and intelligence; Multiple matching strategies are preset in the matching network, and keyword detection results are determined based on keyword prediction probabilities obtained from the multiple matching strategies, thereby further improving the accuracy of keyword detection.

Please refer to FIG. 8. FIG. 8 is a schematic block diagram of a data processing apparatus provided by an embodiment of the present application. Wherein, the data processing device may specifically include:

An acquisition module 801, configured to acquire the speech data to be recognized, and acquire the target syllable sequence of the reference keyword;

The processing module 802 is configured to call a keyword detection model to process the speech data to be recognized, determine the syllable sequence to be detected of the speech data to be recognized, and determine the syllable sequence to be detected according to the syllable sequence to be detected and the target syllable sequence. The keyword detection result of the speech data to be recognized;

Wherein, the keyword detection model is obtained by training using a training data set, and the training data set includes sample speech data of one or more language categories; the keyword detection model can be used for the one or more language categories. The speech data of any language category is used for keyword detection.

In one embodiment, the keyword detection model includes a feature extraction network, a syllable identification network, and a keyword matching network; the syllable identification network includes one or more identification sub-networks, and each identification sub-network is used to identify a specified sub-network. The syllable recognition is performed on the speech data of the language category, and the specified language category is included in the one or more language categories; when the syllable recognition network includes multiple identification sub-networks, any two of the multiple identification sub-networks are included. The network parameters that identify the sub-network are matched.

Optionally, when the acquisition module 801 is used to acquire the target syllable sequence of the reference keyword, it is specifically used for:

Obtain reference keywords, and obtain the target language category of the speech data to be recognized;

Acquire the syllable sequence of the reference keyword that matches the target language category, and determine the syllable sequence of the reference keyword that matches the target language category as the target syllable sequence.

Optionally, the processing module 802 is used to call the keyword detection model to process the speech data to be recognized, and determine the syllable sequence to be detected of the speech data to be recognized, according to the syllable sequence to be detected and the syllable sequence to be detected. When the target syllable sequence determines the keyword detection result of the to-be-recognized speech data, it is specifically used for:

Call the feature extraction network of the keyword detection model to process the to-be-recognized speech data to obtain the speech features of the to-be-recognized speech data;

Invoke the syllable recognition network of the keyword detection model to process the speech features to obtain the to-be-detected syllable sequence of the to-be-recognized speech data;

The keyword matching network of the keyword detection model is invoked to process the syllable sequence to be detected and the target syllable sequence to obtain a keyword detection result of the speech data to be recognized.

Optionally, when the processing module 802 is used for invoking the syllable recognition network of the keyword detection model to process the speech feature to obtain the to-be-detected syllable sequence of the to-be-recognized speech data, it is specifically used for:

Call the syllable recognition network of the keyword detection model to process the voice feature, and determine the target language category of the voice data to be recognized according to the voice feature;

Invoke the recognition sub-network corresponding to the target language category in the syllable recognition network to process the speech feature, and obtain the syllable distribution probability of the speech feature;

The to-be-detected syllable sequence of the to-be-recognized speech data is determined according to the syllable distribution probability.

Optionally, the syllable sequence to be detected includes one or more syllable elements arranged in a first order, the target syllable sequence includes one or more syllable elements arranged in a second order, the processing module 802, in When the keyword matching network used for invoking the keyword detection model processes the syllable sequence to be detected and the target syllable sequence, and obtains the keyword detection result of the speech data to be recognized, it is specifically used for:

Invoke the keyword matching network of the keyword detection model to process the to-be-detected syllable sequence and the target syllable sequence, and detect whether there is a matching syllable element that matches the first syllable element in the target syllable sequence. The first syllable element is any syllable element in the to-be-detected syllable sequence;

If there is a matching syllable element that matches the first syllable element, it is detected whether there is a matching syllable element matching the second syllable element in the target syllable sequence, and the second syllable element is the syllable to be detected. a syllable element arranged one digit after the first syllable element in the sequence;

If there is a matching syllable element matching the second syllable element, the keyword detection result of the speech data to be recognized is determined according to the first syllable element and the second syllable element.

In one embodiment, the syllable recognition network is constructed and generated based on a time-delay neural network and a singular value decomposition network.

Optionally, the processing module 802 is further configured to:

Obtain the training data set, the training data set includes one or more training data subsets, each training data subset includes sample speech data of any language category in the one or more language categories and the same as the one or more language categories. the reference syllable sequence corresponding to the sample speech data;

Perform feature extraction processing on the sample voice data in the one or more training data subsets to obtain sample voice features of the sample voice data;

Using the obtained sample speech features and the corresponding reference syllable sequence, the initial syllable recognition network is trained, and the trained syllable recognition network is obtained;

A trained keyword detection model is generated from the trained syllable recognition network.

It should be noted that the functions of each functional module of the image processing apparatus in the embodiments of the present application may be specifically implemented according to the methods in the above method embodiments, and the specific implementation process may refer to the relevant descriptions of the above method embodiments, which will not be repeated here. .

Please refer to FIG. 9 , which is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device in the embodiment of the present application as shown in the figure may include: a processor 901 , a storage device 902 , and a network interface 903 . Data interaction can be performed among the above-mentioned processor 901 , storage device 902 and network interface 903 .

The above-mentioned storage device 902 may include volatile memory (volatile memory), such as random-access memory (random-access memory, RAM); the storage device 902 may also include non-volatile memory (non-volatile memory), such as flash memory (flash memory), solid-state drive (solid-state drive, SSD), etc.; the above-mentioned storage device 902 may also include a combination of the above-mentioned types of memories.

The above-mentioned processor 901 may be a central processing unit (central processing unit, CPU). In one embodiment, the above-mentioned processor 901 may also be a graphics processor (Graphics Processing Unit, GPU). The above-mentioned processor 901 may also be a combination of a CPU and a GPU. In one embodiment, the above-mentioned storage device 902 is used to store program instructions, and the above-mentioned processor 901 can call the above-mentioned program instructions to perform the following operations:

Obtain the speech data to be recognized, and obtain the target syllable sequence of the reference keyword;

Invoke a keyword detection model to process the to-be-recognized speech data, determine the to-be-detected syllable sequence of the to-be-recognized speech data, and determine the key of the to-be-recognized speech data according to the to-be-detected syllable sequence and the target syllable sequence word detection result;

Wherein, the keyword detection model is obtained by training using a training data set, and the training data set includes sample speech data of one or more language categories; the keyword detection model can be used for the one or more language categories. The speech data of any language category is used for keyword detection.

In one embodiment, the keyword detection model includes a feature extraction network, a syllable identification network, and a keyword matching network; the syllable identification network includes one or more identification sub-networks, and each identification sub-network is used to identify a specified sub-network. The syllable recognition is performed on the speech data of the language category, and the specified language category is included in the one or more language categories; when the syllable recognition network includes multiple identification sub-networks, any two of the multiple identification sub-networks are included. The network parameters that identify the sub-network are matched.

Optionally, when the processor 901 is used to obtain the target syllable sequence of the reference keyword, it is specifically used for:

Obtain reference keywords, and obtain the target language category of the speech data to be recognized;

Acquire the syllable sequence of the reference keyword that matches the target language category, and determine the syllable sequence of the reference keyword that matches the target language category as the target syllable sequence.

Optionally, the processor 901 is used to call a keyword detection model to process the to-be-recognized speech data, and to determine the to-be-detected syllable sequence of the to-be-recognized speech data, according to the to-be-detected syllable sequence and the When the target syllable sequence determines the keyword detection result of the to-be-recognized speech data, it is specifically used for:

Call the feature extraction network of the keyword detection model to process the to-be-recognized speech data to obtain the speech features of the to-be-recognized speech data;

Invoke the syllable recognition network of the keyword detection model to process the speech features to obtain the to-be-detected syllable sequence of the to-be-recognized speech data;

The keyword matching network of the keyword detection model is invoked to process the syllable sequence to be detected and the target syllable sequence to obtain a keyword detection result of the speech data to be recognized.

Optionally, when the processor 901 is used for invoking the syllable recognition network of the keyword detection model to process the speech feature to obtain the to-be-detected syllable sequence of the to-be-recognized speech data, it is specifically used for:

Call the syllable recognition network of the keyword detection model to process the voice feature, and determine the target language category of the voice data to be recognized according to the voice feature;

Invoke the recognition sub-network corresponding to the target language category in the syllable recognition network to process the speech feature, and obtain the syllable distribution probability of the speech feature;

The to-be-detected syllable sequence of the to-be-recognized speech data is determined according to the syllable distribution probability.

Optionally, the syllable sequence to be detected includes one or more syllable elements arranged in a first order, the target syllable sequence includes one or more syllable elements arranged in a second order, the processor 901, in When the keyword matching network used for invoking the keyword detection model processes the syllable sequence to be detected and the target syllable sequence, and obtains the keyword detection result of the speech data to be recognized, it is specifically used for:

Invoke the keyword matching network of the keyword detection model to process the to-be-detected syllable sequence and the target syllable sequence, and detect whether there is a matching syllable element that matches the first syllable element in the target syllable sequence. The first syllable element is any syllable element in the to-be-detected syllable sequence;

If there is a matching syllable element that matches the first syllable element, it is detected whether there is a matching syllable element matching the second syllable element in the target syllable sequence, and the second syllable element is the syllable to be detected. a syllable element arranged one digit after the first syllable element in the sequence;

If there is a matching syllable element matching the second syllable element, the keyword detection result of the speech data to be recognized is determined according to the first syllable element and the second syllable element.

In one embodiment, the syllable recognition network is constructed and generated based on a time-delay neural network and a singular value decomposition network.

Optionally, the processor 901 is further configured to:

Obtain the training data set, the training data set includes one or more training data subsets, each training data subset includes sample speech data of any one of the one or more language categories and the the reference syllable sequence corresponding to the sample speech data;

Perform feature extraction processing on the sample voice data in the one or more training data subsets to obtain sample voice features of the sample voice data;

Using the obtained sample speech features and the corresponding reference syllable sequence, the initial syllable recognition network is trained, and the trained syllable recognition network is obtained;

A trained keyword detection model is generated from the trained syllable recognition network.

In a specific implementation, the processor 901, the storage device 902, and the network interface 903 described in the embodiments of the present application may perform the implementations described in the related embodiments of the data processing methods provided in FIG. 2 or FIG. 4 in the embodiments of the present application, The implementation manner described in the related embodiment of the data processing apparatus provided in FIG. 8 in this embodiment of the present application can also be executed, and details are not described herein again.

In the several embodiments provided in this application, it should be understood that the disclosed method, apparatus and system may be implemented in other manners. For example, the device embodiments described above are only illustrative; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

In addition, it should be pointed out here that the embodiments of the present application further provide a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the aforementioned image processing apparatus, and the computer program includes The program instructions, when the processor executes the above program instructions, can execute the methods in the foregoing embodiments corresponding to FIG. 2 and FIG. 4 , and therefore will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments involved in the present application, please refer to the description of the method embodiments of the present application. By way of example, program instructions may be deployed on one computer device, or executed on multiple computer devices located at one site, or alternatively, distributed across multiple sites and interconnected by a communications network, Multiple computer devices distributed in multiple locations and interconnected by a communication network can form a blockchain system.

According to one aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device can execute the methods in the embodiments corresponding to FIG. 2 and FIG. Repeat.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing the relevant hardware through a computer program. The above programs can be stored in a computer-readable storage medium, and when the program is executed , which may include the processes of the above-mentioned method embodiments. The above-mentioned storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM) or the like.

The above disclosure is only a part of the embodiments of the present application, of course, the scope of the rights of the present application cannot be limited by this. Those of ordinary skill in the art can understand that all or part of the procedures for realizing the above-mentioned embodiments are implemented according to the claims of the present application. The equivalent changes of the invention still belong to the scope covered by the invention.

Claims (12)

1. A method of data processing, the method comprising:

acquiring voice data to be recognized and acquiring a target syllable sequence of a reference keyword;

calling a keyword detection model to process the voice data to be recognized, determining a syllable sequence to be detected of the voice data to be recognized, and determining a keyword detection result of the voice data to be recognized according to the syllable sequence to be detected and the target syllable sequence;

the keyword detection model is obtained by training by utilizing a training data set, wherein the training data set comprises sample voice data of one or more language categories; the keyword detection model may perform keyword detection on speech data in any of the one or more language categories.

2. The method of claim 1, wherein the keyword detection model comprises a feature extraction network, a syllable recognition network, a keyword matching network; the syllable recognition network comprises one or more recognition sub-networks, each recognition sub-network is used for carrying out syllable recognition on the voice data of a specified language category, and the specified language category is contained in the one or more language categories; when the syllable recognition network includes a plurality of recognition subnetworks, network parameters of any two recognition subnetworks of the plurality of recognition subnetworks match.

3. The method of claim 1, wherein obtaining the target syllable sequence of the reference keyword comprises:

acquiring a reference keyword and acquiring a target language category of the voice data to be recognized;

and acquiring the syllable sequence of the reference keyword matched with the target language category, and determining the syllable sequence of the reference keyword matched with the target language category as a target syllable sequence.

4. The method according to claim 2, wherein the invoking the keyword detection model to process the voice data to be recognized, determining a syllable sequence to be detected of the voice data to be recognized, and determining a keyword detection result of the voice data to be recognized according to the syllable sequence to be detected and the target syllable sequence comprises:

calling a feature extraction network of a keyword detection model to process the voice data to be recognized to obtain voice features of the voice data to be recognized;

calling a syllable recognition network of the keyword detection model to process the voice characteristics to obtain a syllable sequence to be detected of the voice data to be recognized;

and calling a keyword matching network of the keyword detection model to process the syllable sequence to be detected and the target syllable sequence to obtain a keyword detection result of the voice data to be recognized.

5. The method according to claim 4, wherein the step of processing the voice feature by calling the syllable recognition network of the keyword detection model to obtain the syllable sequence to be detected of the voice data to be recognized comprises:

calling a syllable recognition network of the keyword detection model to process the voice characteristics, and determining the target language category of the voice data to be recognized according to the voice characteristics;

calling an identifier network corresponding to the target language category in the syllable recognition network to process the voice features to obtain syllable distribution probability of the voice features;

and determining the syllable sequence to be detected of the voice data to be recognized according to the syllable distribution probability.

6. The method of claim 4, wherein the syllable sequence to be detected comprises one or more syllable elements arranged in a first order, and the target syllable sequence comprises one or more syllable elements arranged in a second order; the step of processing the syllable sequence to be detected and the target syllable sequence by calling the keyword matching network of the keyword detection model to obtain a keyword detection result of the voice data to be recognized comprises the following steps:

calling a keyword matching network of the keyword detection model to process the syllable sequence to be detected and the target syllable sequence, and detecting whether a matching syllable element matched with a first syllable element exists in the target syllable sequence, wherein the first syllable element is any one syllable element in the syllable sequence to be detected;

if the matched syllable element matched with the first syllable element exists, detecting whether a matched syllable element matched with a second syllable element exists in the target syllable sequence, wherein the second syllable element is a syllable element which is arranged one bit behind the first syllable element in the syllable sequence to be detected;

and if the matched syllable element matched with the second syllable element exists, determining a keyword detection result of the voice data to be recognized according to the first syllable element and the second syllable element.

7. The method of claim 2, wherein the syllable recognition network is generated based on a time-lapse neural network and a singular value decomposition network construction.

8. The method of claim 1, further comprising:

obtaining the training data set, wherein the training data set comprises one or more training data subsets, and each training data subset comprises sample voice data of any one of the one or more language categories and a reference syllable sequence corresponding to the sample voice data;

performing feature extraction processing on the sample voice data in the one or more training data subsets to obtain sample voice features of the sample voice data;

training the initial syllable recognition network by using the obtained sample voice characteristics and the corresponding reference syllable sequence to obtain a trained syllable recognition network;

and generating a trained keyword detection model according to the trained syllable recognition network.

9. A data processing apparatus, characterized in that the apparatus comprises:

the acquisition module is used for acquiring voice data to be recognized and acquiring a target syllable sequence of a reference keyword;

the processing module is used for calling a keyword detection model to process the voice data to be recognized, determining a syllable sequence to be detected of the voice data to be recognized, and determining a keyword detection result of the voice data to be recognized according to the syllable sequence to be detected and the target syllable sequence;

the keyword detection model is obtained by training by utilizing a training data set, wherein the training data set comprises sample voice data of one or more language categories; the keyword detection model may perform keyword detection on speech data in any of the one or more language categories.

10. A computer device, comprising: the device comprises a memory and a processor, wherein the memory stores a data processing program, and the data processing program is used for realizing the data processing method according to any one of claims 1-8 when being executed by the processor.

11. A computer-readable storage medium device, characterized in that the computer-readable storage medium stores a computer program comprising program instructions which are executed by a processor for implementing the data processing method according to any one of claims 1 to 8.

12. A computer program product, characterized in that it comprises a computer program or computer instructions which, when executed by a processor, implement a data processing method according to any one of claims 1 to 8.

Images