CN115910066A - Intelligent dispatching command and operation system for regional power distribution network - Google Patents

Abstract

The invention discloses an intelligent dispatching command and operation system for a regional power distribution network, which comprises an intelligent voice navigation module, an automatic generation module of a planned dispatching instruction ticket, an information monitoring and diagnosis module and an intelligent commuting recognition module, wherein the intelligent voice navigation module comprises a voice recognition engine unit and a semantic analysis engine unit, and the voice recognition engine unit is used for directly converting human voice into corresponding texts so as to facilitate a processing module to understand and generate corresponding operations, thereby realizing natural voice interaction between a human and a machine and converting audio stream data into a recognition result of character stream data in real time. The invention discloses an intelligent scheduling command and operation system for a regional power distribution network, which integrates artificial intelligence with a production operation system, realizes intelligent sensing, analysis, control and interaction of production operation services, and truly realizes transformation development of intelligent production operation.

Description

Intelligent dispatching command and operation system for regional distribution network

technical field

The invention belongs to the technical field of production and operation of regional distribution networks, and in particular relates to an intelligent dispatching command and operation system for regional distribution networks.

Background technique

With the further expansion of the distribution network scale and the increasingly complex grid structure, the requirements for the operation and management level of the distribution network are getting higher and higher. The normal operation of the distribution network directly affects the user level and power supply quality of the enterprise, and is the main factor in evaluating the company's ability to fulfill its responsibilities.

As a distribution network business hub, the distribution network production and operation center, first, involves more and more information systems. The island phenomenon among information systems is serious. Second, the production and operation points of the distribution network are many and wide, and the production and operation need to carry out repeated process docking with multiple sites at the same time, resulting in the "hub congestion" effect of the distribution network production and operation business. Third, the massive data information of the production and operation system needs to be further excavated. Fourth, the existing production and operation auxiliary system is constructed using traditional expert systems, and manual editing is used for knowledge acquisition and maintenance. The scalability is poor, and the degree of automation and intelligence of knowledge reasoning is not enough. Fifth, the intelligent interaction mode of the production operation system is single, inefficient, and lacks voice, touch and other interactive modes. Therefore, it is urgent to improve the efficiency of distribution network production and operation business and process reengineering.

Contents of the invention

The main purpose of the present invention is to provide an intelligent dispatching command and operation system for the regional distribution network, which integrates artificial intelligence with the production and operation system to realize the intelligent perception, analysis, control and interaction of the production and operation business, and truly realize the For the transformation and development of smart production and operation, intelligent analysis relies on deep learning technology to intelligently judge power grid faults, assist production operators to carry out intelligent decision-making, and realize fast and accurate judgment of complex faults; intelligent control is to intelligently control the operation mode of the power grid based on the analysis and judgment results and the status of power grid equipment, and comprehensively improve the control ability of the power grid; intelligent interaction is to realize information interaction with production operators, emergency repair operators, operation and maintenance personnel and users through comprehensive display, intelligent information push, intelligent voice interaction and other technical means.

In order to achieve the above objectives, the present invention provides an intelligent dispatching command and operation system for regional distribution network, which is used to ensure the safe operation of the production and operation system of the distribution network, including intelligent voice navigation module, automatic generation of planning dispatching instruction tickets module, information monitoring and diagnosis module and intelligent shift identification module, among which:

(Based on hybrid intelligence and human-machine coupling technology) intelligent voice navigation module includes a voice recognition engine unit and a semantic analysis engine unit, the voice recognition engine unit is used to directly convert human voice into corresponding text, so that the processing module can understand and generate Corresponding operations, so as to realize the natural voice interaction between human and machine, and based on the deep full sequence convolutional neural network framework, the audio stream data is converted into the recognition result of the text stream data in real time; the semantic analysis engine unit is used for speech recognition Semantic analysis of the recognition results obtained by the engine unit (to make the human-computer interaction process smoother);

The planning scheduling instruction ticket automatic generation module automatically generates the planning scheduling instruction ticket based on information extraction, and the planning scheduling instruction ticket automatic generation module includes the learning phase unit and the extraction phase unit;

The information monitoring and diagnosis module performs intelligent monitoring, diagnosis and early warning of the information of the production and operation system based on the text recognition unit and the speech synthesis unit;

The intelligent shift recognition module is used to identify the user's voiceprint and integrate it into the production and operation system, realize voiceprint-based identity authentication through voiceprint registration and analysis, and then identify the shift-over user through voiceprint.

As a further preferred technical solution of the above technical solution, the speech recognition engine unit includes a Chinese punctuation intelligent prediction unit, a file format intelligent conversion unit, a front-end speech processing unit and a back-end recognition processing unit, wherein;

The Chinese punctuation intelligent prediction unit is used to perform intelligent prediction of dialogue speech on the recognition result obtained by the recognition of the speech recognition engine unit through the language model, thereby providing prediction of intelligent sentence segmentation and punctuation marks;

The file format intelligent conversion unit formats the parameters including numbers, dates and times appearing in the recognition results and generates regular text;

The front-end speech processing unit detects the input speech and pre-processes the noise reduction by means of signal processing, so as to obtain the speech that best matches the speech recognition engine unit. The front-end speech processing unit includes an endpoint detection subunit and a noise elimination subunit. The endpoint detection The subunit is used to analyze the voice in the form of the input audio stream to determine the start and end of the user's speech. When it is detected that the user starts to speak, the voice starts to flow to the speech recognition engine unit until the end of the user's speech is detected;

The back-end recognition processing unit includes a personalized speech recognition subunit, a confidence output subunit, a multi-result recognition subunit, a speaker adaptive subunit, and a semantic context self-correction subunit. The personalized speech recognition subunit is based on the user's voice characteristics , collect and upload words (hot words) with high usage rate, and establish a personalized entry language model from a business perspective to adjust recognition parameters, modify the weight of uploaded hot words and continuously optimize recognition; the confidence output subunit is used in The confidence of the recognition result is carried when the recognition result is returned, so that the analysis and subsequent processing can be carried out through the confidence result; during the recognition process, the multi-result recognition subunit returns the multi-results that meet the conditions to the application program through the result of the confidence output subunit A recognition result, not the only result, provides a list of possible recognition results, and arranges them from high to low according to the confidence results; the speaker adaptive subunit is used when the user conducts multiple conversations with the speech recognition engine unit , extracting the voice features of the call online, automatically adjusting the recognition parameters, so that the recognition effect can be continuously optimized; the semantic context self-correction subunit is used for dynamic correction in combination with the context, and dynamically corrects the result according to the speech recognition result and the context to make the result more in line with the current language territory;

The semantic analysis engine unit includes a rule understanding unit and a model understanding unit. The rule understanding unit is used for matching rules. The model understanding unit includes a semantic model training subunit, a semantic feature extraction subunit, a semantic similarity evaluation subunit, and a semantic model training subunit. The unit is used to model the text, and the text with similar semantic expression is mapped to the similar vector in the semantic space; the semantic feature extraction subunit is used to extract the feature of the text information, after completing the training of the semantic model training subunit, through The hidden layer vector specific to the extraction model is obtained; the semantic similarity evaluation subunit is used to evaluate the similarity between the user sentence vector extracted by deep learning and the sentence vector extracted offline in the library.

As a further preferred technical solution of the above technical solution, for the learning stage unit:

In the learning phase, there is a labeled data set, each sample contains a sequence of text units and a sequence of labels, as follows:

Among them, represents the i-th sample, the front x part represents the text unit sequence of the i-th sample, and the following y part represents the label sequence of the i-th sample;

Construct a learning model based on existing annotations and represent it with a conditional probability distribution;

The extraction stage unit uses different models as classifiers, so as to obtain the extraction methods of text information that do not pass, and correct the extracted data obtained by each classifier.

As a further preferred technical solution of the above technical solution, the text recognition unit introduces contextual sequence information into the text recognition, and performs recognition through a neural network of temporal relationships.

As a further preferred technical solution of the above technical solution, for the speech synthesis unit:

Full-language semantic model, based on the unsupervised text pre-training model of multi-language fusion, extracts text semantic information, and builds independent text prediction information branches for each language based on the unified pre-training model;

Pronunciation system model, which automatically constructs a unified unit pronunciation dictionary through the semi-supervised clustering method, and modularizes language-related technologies, enabling rapid customization of the synthesis system under the condition of limited language resources;

Based on the multilingual mixed model of auditory quantization coding, the auditory quantization coding is performed on different attribute information in speech, and the acoustic parameters of speech are predicted

Based on the high-quality speech generation model of generative confrontation network, high-quality multilingual speech synthesis is performed by modeling the overall structure of the spectral envelope and the local details of the spectral envelope.

As a further preferred technical solution of the above technical solution, the intelligent shift recognition module includes a model training phase unit and a model prediction phase unit:

The model training phase unit is used to train the model using audio and video synchronization and data sets with voice and text annotations;

The application of the model in the model prediction stage unit inputs the voices of multiple people and the lip movement video of a specific target person, recognizes the text sequence, and at the same time separates the speaker's voice that matches the content of the lip movement video.

In order to achieve the above object, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor. Intelligent dispatching command and operation system of distribution network.

In order to achieve the above object, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the intelligent dispatching command and operation for the regional distribution network is realized. system.

The beneficial effects of the present invention are:

First, it is conducive to further improving the cognitive level of power grid operation characteristics. Based on technologies such as video collection and OCR text recognition, it realizes intelligent monitoring, diagnosis and early warning of SOE information, and realizes self-diagnosis of four types of states, including reclosing failure, reclosing success, trip non-reclosing, and ground fault. And through TTS broadcasting technology, important information is broadcasted in the production operation command center to ensure that no important information is missed.

Second, it is conducive to further improving the efficiency of production operation control and task completion. Based on technologies such as simulated clicks, semantic understanding, and text parsing, the automatic generation of power outage planning dispatching order tickets is realized, which improves the efficiency of dispatching ticket generation and completion of dispatching tasks, further reduces the time of planned power outages and fault power outages, and improves the self-healing of the power grid level. At the same time, the intelligent voice control and navigation of the production operation system is realized based on voice, and the control efficiency of the production operation system is improved.

Third, it is conducive to further improving the security level of the production and operation system. Realize the voiceprint authentication function of the production operation system based on voiceprint recognition technology, clarify the operation authority of production operators, ensure smooth shift handover, and build a safe and efficient intelligent production operation system.

Description of drawings

Fig. 1 is a schematic diagram of an intelligent dispatching command and operation system for a regional distribution network according to the present invention.

Detailed ways

The following description serves to disclose the present invention to enable those skilled in the art to carry out the present invention. The preferred embodiments described below are only examples, and those skilled in the art can devise other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents and other technical solutions without departing from the spirit and scope of the present invention.

In the preferred embodiment of the present invention, those skilled in the art should note that the distribution network and the like involved in the present invention can be regarded as prior art.

preferred embodiment.

The invention discloses an intelligent dispatching command and operation system for a regional distribution network, which is used to ensure the safe operation of the production and operation system of the distribution network. Diagnosis module and intelligent shift identification module, in which:

(Based on hybrid intelligence and human-machine coupling technology) intelligent voice navigation module includes a voice recognition engine unit and a semantic analysis engine unit, the voice recognition engine unit is used to directly convert human voice into corresponding text, so that the processing module can understand and generate Corresponding operations, so as to realize the natural voice interaction between human and machine, and based on the deep full sequence convolutional neural network framework, the audio stream data is converted into the recognition result of the text stream data in real time; the semantic analysis engine unit is used for speech recognition Semantic analysis of the recognition results obtained by the engine unit (to make the human-computer interaction process smoother);

The automatic planning scheduling instruction ticket generation module is based on information extraction to automatically generate the planning scheduling instruction ticket. The planning scheduling instruction ticket automatic generation module includes the learning stage unit and the extraction stage unit (based on the information extraction technology to automatically generate the planning scheduling instruction ticket. The power outage plan information In the details page, the contents of the five key fields are analyzed, and according to the analyzed content, a scheduling instruction ticket is generated according to the template);

The information monitoring and diagnosis module performs intelligent monitoring, diagnosis and early warning on the information of the production and operation system based on the text recognition unit and the speech synthesis unit (based on OCR text recognition technology, NLP technology and voice outbound technology to realize intelligent monitoring, diagnosis and early warning of SOE information .Based on SOE text information and relationships, realize fault self-diagnosis; including accident information, abnormal information, displacement information, notification information, etc.; combined with the context of SOE information, automatic diagnosis of reclosing failure, reclosing success, trip non-reclosing, ground fault Four types of abnormalities. At the same time, when the system identifies fault information that needs attention, it will broadcast in the production operation room through speech synthesis, and the original information (time, content) and classification of the information, etc. The short message platform of the Internet sends the information to the external network server. The external network server receives the alarm content of the short message and parses it into the corresponding text, automatically dials the outbound call to the on-site operator and the commander-in-chief, and records whether the call is dialed. At the same time, When the production and operation personnel are monitoring the large screen, OCR can be used to recognize the text on the large screen. After matching the content of the production operator's speech, the system voice will broadcast "consistent", otherwise it will broadcast "inconsistent");

The intelligent shift recognition module is used to identify the user's voiceprint and integrate it into the production and operation system, realize voiceprint-based identity authentication through voiceprint registration and analysis, and then use the voiceprint to identify the user of the shift (production and operation personnel log in When using the system, you need to say the specified password; the system will identify whether you have permission to operate according to the voiceprint recognition of the password. During the shift change, the system will judge the identity of the two handover personnel based on the voiceprint recognition, and prompt the handover is successful).

Specifically, the speech recognition engine unit includes a Chinese punctuation intelligent prediction unit, a file format intelligent conversion unit, a front-end speech processing unit and a back-end recognition processing unit, wherein;

ASR (Speech Recognition Engine Unit) is a technology that realizes the conversion from "sound" to "text", by directly converting human speech into corresponding text so that the computer can understand and generate corresponding operations, and finally realize the human and machine Natural voice interaction between. Based on the deep full sequence convolutional neural network framework, the audio stream data is converted into text stream data in real time.

Aiming at problems such as dialect accents and background noise in speech recognition applications, based on the massive speech data collected in the actual business system covering different dialects and different types of background noise, speech modeling is carried out through advanced discriminative training methods, so that speech The recognizer has good performance in complex application environments.

The Chinese punctuation intelligent prediction unit is used to perform intelligent prediction of dialogue speech on the recognition result obtained by the recognition of the speech recognition engine unit through the language model, thereby providing prediction of intelligent sentence segmentation and punctuation marks;

The file format intelligent conversion unit formats the parameters including numbers, dates and times appearing in the recognition results and generates regular text;

The front-end speech processing unit detects the input speech and pre-processes the noise reduction by means of signal processing, so as to obtain the speech that best matches the speech recognition engine unit. The front-end speech processing unit includes an endpoint detection subunit and a noise elimination subunit. The endpoint detection The subunit is used to analyze the voice in the form of the input audio stream to determine the start and end of the user's speech. When it is detected that the user starts to speak, the voice starts to flow to the speech recognition engine unit until the end of the user's speech is detected;

The back-end recognition processing unit includes a personalized speech recognition subunit, a confidence output subunit, a multi-result recognition subunit, a speaker adaptive subunit, and a semantic context self-correction subunit. The personalized speech recognition subunit is based on the user's voice characteristics , collect and upload words (hot words) with high usage rate, and establish a personalized entry language model from a business perspective to adjust recognition parameters, modify the weight of uploaded hot words and continuously optimize recognition; the confidence output subunit is used in The confidence of the recognition result is carried when the recognition result is returned, so that the analysis and subsequent processing can be carried out through the confidence result; during the recognition process, the multi-result recognition subunit returns the multi-results that meet the conditions to the application program through the result of the confidence output subunit A recognition result, not the only result, provides a list of possible recognition results, and arranges them from high to low according to the confidence results; the speaker adaptive subunit is used when the user conducts multiple conversations with the speech recognition engine unit , extracting the voice features of the call online, automatically adjusting the recognition parameters, so that the recognition effect can be continuously optimized; the semantic context self-correction subunit is used for dynamic correction in combination with the context, and dynamically corrects the result according to the speech recognition result and the context to make the result more in line with the current language territory;

The semantic analysis engine unit includes a rule understanding unit and a model understanding unit. The rule understanding unit is used for matching rules. The model understanding unit includes a semantic model training subunit, a semantic feature extraction subunit, a semantic similarity evaluation subunit, and a semantic model training subunit. The unit is used to model the text, and the text with similar semantic expression is mapped to the similar vector in the semantic space; the semantic feature extraction subunit is used to extract the feature of the text information, after completing the training of the semantic model training subunit, through The hidden layer vector specific to the extraction model is obtained; the semantic similarity evaluation subunit is used to evaluate the similarity between the user sentence vector extracted by deep learning and the sentence vector extracted offline in the library.

Further, for learning phase units:

In the learning phase, there is a labeled data set, each sample contains a sequence of text units and a sequence of labels, as follows:

Among them, represents the i-th sample, the front x part represents the text unit sequence of the i-th sample, and the following y part represents the label sequence of the i-th sample;

Construct a learning model based on existing annotations and represent it with a conditional probability distribution;

The extraction stage unit uses different models as classifiers, so as to obtain the extraction methods of text information that do not pass, and correct the extracted data obtained by each classifier.

More specifically, the text recognition unit introduces contextual sequence information in text recognition, and recognizes through the neural network of temporal relations.

Commonly used models for text information extraction are: hidden Markov model, maximum entropy Markov model, conditional random field, and voting perceptron model. These models are concrete implementations of the general text information extraction mentioned above.

First, each label is regarded as independent of the sequence and other labels of the sequence. For each text unit sequence and its labels, such a conditional probability distribution can be obtained;

Such a conditional probability distribution is actually a classifier model, which obtains the best annotation of each unit sequence of the sample sequence.

On this basis, using different models as classifiers, different text information extraction methods can be obtained.

For example, use the maximum entropy model as a classifier, and assume that there is a first-order Markov property between different labels;

Each conditional probability distribution is a new classifier model, but this classifier is based on the classifier under the conditions determined by the previous label. At this time, the model becomes the maximum entropy Markov model.

It should be noted that the maximum entropy Markov model is a local model, because there is a first-order Markov property between labels, and local data is used for training, but when extracting global information, the effect may not be very good, and it will produce Labeling bias problem. Therefore, some complex global models can be used to solve the labeling bias problem. Among them, the most commonly used global model is the conditional random field. Each label in the conditional probability distribution will depend on other labels except itself. The conditional random field can accurately describe the global labeling situation. The accuracy is due to the maximum entropy Markov model, but the training time is also more.

Even further, for the speech synthesis unit:

Full-language semantic model, based on the unsupervised text pre-training model of multi-language fusion, extracts text semantic information, and builds a text prediction information branch for each language independently based on the unified pre-training model (BERT unsupervised text based on multi-language fusion The pre-training model extracts text semantic information and reduces the difficulty and quantity of manual text labeling. Specifically, on the issue of small language construction, the multi-language fusion massive text BERT pre-training method extracts semantic change information and improves the weak representation ability of text input The full name of BERT is Bidirectional EncoderRepresentations from Transformers, which is a way of pre-training language representation. It has achieved industry-leading results in a variety of NLP tasks. It has high adaptability to the problem of small language construction in this project. The massive text pre-training method of multi-language integration extracts semantic change information, improves the reality of weak text input ability, and improves the performance of speech synthesis. On the basis of the unified BERT pre-training model, the text prediction information branch of each language independent situation is constructed. Such as Chinese polyphonic characters, Japanese tune-up, Arabic vowel recovery and other issues, to achieve multi-task task learning of shared information);

Pronunciation system model, which automatically constructs a unified unit pronunciation dictionary through the semi-supervised clustering method, and modularizes language-related technologies, enabling rapid customization of the synthesis system under the condition of limited language resources;

The construction process of Global Phone is based on International Phonetic Alphabet (IPA). According to the laws of physical pronunciation, the design covers all pronunciation systems, including isolated language, agglutinative language, inflectional language, and distinguishing syllable prominent language (toned, atonal), non-syllable Speech pronunciation types in various major language families such as prominent languages (accented, unaccented). Pre-defined by Global Phone and manually confirmed annotations in data-rich languages,

In order to achieve multilingual speech synthesis in a limited time, a unified framework for multilingual synthesis based on Global Phone is proposed. The pronunciation dictionary of a unified unit is automatically constructed through the VAE semi-supervised clustering method, and the language-related technology is modularized. In the language Under the condition of limited resources, the rapid customization of the synthesis system can be realized, which solves the needs of the speech system of resource-limited languages such as minority languages or dialects.

Firstly, the system modules are divided into language-related and language-independent modules. The language-independent module mainly includes the general modules of speech synthesis engine and speech recognition engine. The similarities between different languages allow us to observe speech phenomena from a unified perspective.

Language-related modules are mainly related to text processing and sound library processing, including speech unit definition and classification, text regularization, word-to-sound conversion, corpus design, word segmentation, prosodic structure prediction, speech database annotation, and other modules required for speech synthesis, among which speech recognition can be shared Definition and classification of speech units, text regularization, word-to-sound conversion, corpus design, speech data processing, etc. In the construction of language-related information such as dictionaries, parts of speech, and prosodic levels, the Global Phone unified framework is used to uniformly model small languages with the same system, reducing the difficulty of building a single language.

The construction process of Global Phone is based on the International Phonetic Alphabet (IPA). Consonants are mainly distinguished according to the pronunciation position and pronunciation method; Duration, nasalization, and tightness are used as secondary distinguishing dimensions to build a unified Global Phone system in various languages. For languages with tones, a unified Global tone label is carried out according to the tone class, tone value, and tone domain.

In languages with insufficient data resources and expert resources, semi-supervised VAE clustering supported by labeled data can automatically learn the sound bank pronunciation sequence and pronunciation dictionary under the unified system for small languages with insufficient resources, and finally realize a set of general Global Phone multilingual pronunciation unified construction method.

Based on the multilingual mixed model of auditory quantitative coding, the auditory quantitative coding is performed on different attribute information in the speech, and the acoustic parameters of the speech are predicted;

Since it is difficult to collect data recordings in small languages, it is necessary to consider the mixed use of multi-lingual, multi-person and multi-style data from the very beginning. The speech synthesis system is constructed by using the acoustic model based on auditory quantization coding. First, the auditory quantization coding is performed on different attribute information in the speech, and then the acoustic parameters of the speech are predicted. Specifically, by manually defining the speaker, language, emotional style and other attribute codes of multi-person mixed speech, and introducing residual codes to describe the changes in the pronunciation of speakers due to different states such as emotions, environments, and time differences when recording voice data Information, combined with text information, realizes the prediction of speech acoustic parameters through a fully connected feed-forward (FF) network and a long-short-term memory network (LSTM-RNN). For new speakers with less corpus, it is planned to adopt the transfer learning method to improve the effect of speech synthesis with a small amount of data.

Assume that there are multiple people and multilingual mixed data sets. The traditional acoustic modeling method for speech synthesis is to use neural network to model directly. In order to decompose the information in the speech by controlling the speaker, language, and speaking style of the speech synthesis, four kinds of auditory quantization coding are defined: speaker coding, language coding, emotional style coding, and residual coding. Then explicitly quantify the speaker, language, and emotional style in the speech, and directly model the joint distribution of these quantized codes and acoustic parameters;

The whole model is implemented using a neural network, which mainly includes two parts: (1) the main network part, which predicts the acoustic parameters when given text and auditory quantization encoding, and shares the same main network with all data, so that the model can learn different speech The impact of person, language, and style coding on synthesized speech; (2) collateral network, when given text, language, and speaker, predicts the emotional coding, which is also implemented using neural networks.

During model training, the speaker and language are given, and the emotional quantization coding is defined by manual marking. Residual coding describes the changes in the pronunciation of the speaker due to emotions, environments, time differences, etc. when recording voice data, which cannot be manually marked. Therefore, each sentence is represented by a separate residual coding. The residual The encoding is initialized randomly and updated through model training. The entire model is trained using the minimum mean square error criterion and gradient updates are performed using a stochastic gradient descent algorithm.

For new languages and new speakers with less corpus, this project proposes the use of transfer learning methods to improve the effect of speech synthesis with a small amount of data. Firstly, the weight of the trained multi-person mixed model is used as the initialization of the new speaker model, and then fine-tuning is performed using a small amount of data of the new speaker, so as to achieve fast modeling of the new speaker with a small amount of data.

Based on the high-quality speech generation model of generative confrontation network, high-quality multilingual speech synthesis is performed by modeling the overall structure of the spectral envelope and the local details of the spectral envelope.

In order to build a high-quality multilingual speech synthesis system, it is planned to divide the spectral modeling of speech into two stages: 1) modeling the overall structure of the spectral envelope; 2) modeling the local details of the spectral envelope. The GAN model is proposed to model the local details of the spectral envelope in a language-independent manner under the condition that the acoustic model predicts the overall framework of the spectral envelope. Specifically, a convolutional neural network is used to construct an acoustic model to predict the mapping from the spectrum to the spectrum envelope of natural speech, and the GAN criterion is used for model training. The generation network in GAN uses noise, text features and low-dimensional Mel cepstrum as input to predict spectral envelope features; the discriminant network uses text features as conditions to distinguish between the spectral envelope predicted by the generative network and the natural spectral envelope . Finally, in the case of a limited amount of speaker data in a specific language, the fine structure in the spectral envelope can be restored through the cross-language GAN model, and the subjective quality of speech can be improved.

The overall framework of the spectral envelope determines the correctness of speech pronunciation, and the specific details of the spectral envelope determine the sound quality of the speech. The Mel cepstrum is a decorrelated feature extracted from the spectral envelope. Its low dimension represents the overall structure of the spectral envelope, and its high dimension describes the local details of the spectral envelope. In speech synthesis, the high dimension of the Mel cepstrum has a great influence on the sound quality. In the global variance of the Mel cepstrum of the natural speech and the Mel cepstrum of the synthetic speech, the low-dimensional global variance of the Mel cepstrum has a similar distribution. , while the global variance of the high-dimensional part has a large difference, and the high-dimensional Mel cepstrum of the synthesized speech is usually too smooth, resulting in impaired speech quality. This shows that the overall structure modeling of the spectrum envelope in the current statistical parametric speech synthesis is relatively accurate, but the local detail modeling of the spectrum envelope has relatively large defects. Considering that the samples generated by the GAN model are closer to natural samples than the traditional generative model based on maximum likelihood training, it is proposed to use the GAN model to construct the local details of the spectral envelope when the overall framework of the spectral envelope is given. mold. Since CNN has a strong modeling ability for local structures, and the speech spectral envelope features have important local structures on the time and frequency axes, this project chooses CNN as the discriminant network and generation network of the GAN model, and selects the spectral envelope features as prediction samples. Given the spectral envelope feature s of the speech, the corresponding text feature c, and the corresponding low-dimensional Mel cepstrum feature m, the conditional distribution P(s|c) of the time spectral envelope feature of the given text is not directly constructed Instead, the conditional distribution P(s|c,m) of the spectral envelope is modeled when given text features and low-dimensional Mel cepstral features;

The generation network in GAN uses noise, text features and low-dimensional Mel cepstrum as input to predict spectral envelope features; the discriminant network uses text features as conditions to distinguish between the spectral envelope predicted by the generative network and the natural spectral envelope .

In the training phase, the natural Mel cepstrum low-dimensional and text are used as conditional input. During the test process, since there is no Mel cepstrum parameter, another model needs to be used to predict it. Traditional statistical modeling methods or GAN-based modeling approach. This project plans to use the RNN-LSTM model to model the low-dimensional parameters of the Mel cepstrum. The GAN model is then used to predict spectral envelope parameters conditioned on text features and Mel cepstrum low-dimensionality. Finally, the final speech is synthesized by combining the duration and fundamental frequency predicted by the traditional statistical parameter speech synthesis system. This spectrum generation process can be regarded as a two-stage generation method. First, the low-dimensional Mel cepstrum parameters are predicted by the model trained based on the MSE criterion, the basic framework of the spectrum envelope is determined, and then the spectrum envelope is predicted using the GAN model. fine structure in the network. Through this two-level generation model, it is expected to significantly improve the quality of speech generation and improve the experience effect.

Preferably, the intelligent shift recognition module includes a model training phase unit and a model prediction phase unit:

The model training phase unit is used to train the model using audio and video synchronization and data sets with voice and text annotations;

The application of the model in the model prediction stage unit inputs the voices of multiple people and the lip movement video of a specific target person, recognizes the text sequence, and at the same time separates the speaker's voice that matches the content of the lip movement video.

The specific implementation of the model training steps and testing steps is as follows:

Step 1: Data Preprocessing

Data preprocessing mainly includes image preprocessing, voice preprocessing, and text annotation preprocessing. The preprocessing of the image is to detect the face feature points of the speaker in the video, scale the face to the same scale according to the feature points, and then take the mouth area image of a fixed size with the center point of the mouth as the center to obtain the mouth image video In this example, the size of the mouth area image obtained in this example is 80*80, the number of channels is RGB three-channel, and the frame rate of the video is 25fps; the voice preprocessing is divided into two types: (1) The recognized voice feature extraction is sliding The window extracts the 40-dimensional fbank feature of the speech. In this example, the window length is set to 25ms, and the frame shift is 10ms. (2) The separated speech feature is extracted by performing short-time Fourier transform (STFT) on the speech signal to form 2 channels The spectrogram of the text annotation; the preprocessing of the text annotation is to use force alignment to align the phonemes of the text pronunciation to the speech signal, and every 4 frames of the speech signal correspond to a triphone, so that the text annotation is actually converted into a triphone annotation, and the annotation frame rate is 25fps , synchronized with the video frame rate, is a quarter of the audio frame rate.

Step 2: Joint training of speech recognition model and separation model

After obtaining enough training data, a neural network model for speech recognition and a neural network model for lip language recognition will be built separately for joint training of multimodal speech recognition and separation.

The input of the multimodal speech recognition and separation neural network model is the audio and video with a fixed duration of 3s.

The audio feature input is a 100fps 40-dimensional speech frame fbank feature vector sequence. There are 300*40 fbank feature maps in 3s. After the feature extraction module, it will be sampled 4 times in the time dimension to obtain a 75*512-dimensional speech feature vector sequence. ;The video image input is a 25fps image sequence, the image size is an 80*80 RGB three-channel image, and there are 75*3*80*80 image sequences in 3s duration, and a 75*512-dimensional video feature vector is obtained after the feature extraction module Sequence; the multimodal information fusion module fuses the 512-dimensional speech feature vector extracted by the two modules with the 512-dimensional image feature vector. The fusion can use feature splicing, and then generate a new 75*512 through a small fusion neural network. Dimensional fused feature vectors.

For the multi-modal recognition task, the 75*512-dimensional fusion feature is directly sent to the recognition module, and after softmax classification, it corresponds to the marked triphone phoneme label one by one, and the loss function used is CrossEntry Loss.

The multimodal speech separation task uses the U_Net network structure, and the original input of the model is a 3s speech signal undergoing STFT transformation to form a 2*298*257 spectrogram. The U_Net structure will first downsample and then upsample the feature map. The intermediate feature map with the same size in the downsampling and upsampling modules will have a feature splicing operation; on the smallest feature vector in the middle, the previous 75 *The 512-dimensional fusion feature vectors are spliced in to form intermediate conditions and guide U_Net to perform spectral map reconstruction. The reconstructed spectrogram and the label spectrogram are used for L2 Loss. Gradient backpropagation not only updates the U_Net module, but also updates the feature fusion module, audio feature extraction module, and video feature extraction module to form a joint training effect with the CrossEntry Loss of multimodal recognition.

During this step of training, the task type is similar to Multi_task, and the speech separation and recognition are performed synchronously. The two tasks are performed synchronously, but the same fusion features are used, so that the feature extraction and fusion modules can learn closer multi-modal joint information. . Finally, L2 Loss and CrossEntry Loss are weighted and summed to form the final Loss, and the entire network is trained through the gradient backpropagation optimization algorithm.

Step 3: Model testing and use

After the model is trained, it can be tested for multi-modal speech separation and recognition. The preprocessing of speech and images during use is the same as during training, and will not be repeated here. First, the audio and video features are formed into a fusion feature through the fusion module, and then the phoneme state is decoded into a text sequence through the Viterbi algorithm after identifying the triphone phoneme state through the recognition module to achieve the final recognition effect. The fusion features can also be spliced to the intermediate features of the separated and reconstructed U_Net network to reconstruct a separated spectrogram, and the spectrogram can be restored into a separated speech signal after ISTFT transformation.

The invention also discloses an electronic device, which includes a memory, a processor, and a computer program stored on the memory and operable on the processor. Intelligent dispatch command and operation system.

The invention also discloses a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the intelligent dispatching command and operation system for the regional power distribution network is realized.

It is worth mentioning that the distribution network and other technical features involved in the patent application of the present invention should be regarded as prior art. Conventional selection is sufficient, and should not be regarded as the invention point of the patent of the present invention, and the patent of the present invention will not be further elaborated in detail.

For those skilled in the art, it is still possible to modify the technical solutions described in the foregoing embodiments, or to perform equivalent replacements for some of the technical features. Any modifications made within the spirit and principles of the present invention, Equivalent replacements, improvements, etc., should all be included in the protection scope of the present invention.

Claims (8)

1. The utility model provides an intelligent scheduling commander and operation system for regional distribution network for the safe operation of the production operation system of guarantee distribution network, its characterized in that, including intelligent voice navigation module, the automatic generation module of plan dispatch instruction ticket, information monitoring diagnosis module and intelligent shift identification module, wherein:

the intelligent voice navigation module comprises a voice recognition engine unit and a semantic analysis engine unit, wherein the voice recognition engine unit is used for directly converting human voice into corresponding texts so that the processing module can understand and generate corresponding operations, thereby realizing natural voice interaction between a human and a machine, and converting audio stream data into a recognition result of character stream data in real time based on a deep full-sequence convolutional neural network framework; the semantic analysis engine unit is used for performing semantic analysis on the recognition result obtained by the voice recognition engine unit;

the automatic generation module of the scheduling instruction ticket extracts and automatically generates the scheduling instruction ticket based on the information, and the automatic generation module of the scheduling instruction ticket comprises a learning stage unit and an extraction stage unit;

the information monitoring and diagnosing module carries out intelligent monitoring, diagnosis and early warning on the information of the production operation system based on the text recognition unit and the voice synthesis unit;

the intelligent shift-to-shift identification module is used for identifying the voiceprint of the user and fusing the voiceprint to a production operation system, identity authentication based on the voiceprint is achieved through voiceprint registration and analysis, and then the shift-to-shift user is identified through the voiceprint.

2. The intelligent dispatching command and operation system for the regional power distribution network according to claim 1, wherein the voice recognition engine unit comprises a Chinese punctuation intelligent prediction unit, a file format intelligent conversion unit, a front-end voice processing unit and a back-end recognition processing unit, wherein;

the Chinese punctuation intelligent prediction unit is used for intelligently predicting dialogue voice for the recognition result obtained by the recognition of the voice recognition engine unit through a language model so as to provide prediction of intelligent punctuation and punctuation;

the file format intelligent conversion unit formats parameters including numbers, dates and times appearing in the recognition result and generates a regular text;

the front-end voice processing unit detects and performs noise reduction pretreatment on the input voice through a signal processing method so as to obtain the voice which is most matched with the voice processed by the voice recognition engine unit, the front-end voice processing unit comprises an endpoint detection subunit and a noise elimination subunit, the endpoint detection subunit is used for analyzing the input voice in the form of audio stream and determining the start and the end of the user speaking, and when the user starts speaking is detected, the voice starts to flow to the voice recognition engine unit until the end of the user speaking is detected;

the rear-end recognition processing unit comprises an individualized voice recognition subunit, a confidence coefficient output subunit, a multi-result recognition subunit, a speaker self-adaption subunit and a semantic context self-correction subunit, wherein the individualized voice recognition subunit collects and uploads words with high utilization rate based on the voice characteristics of a user, and establishes an individualized entry language model from a business perspective so as to adjust recognition parameters, modify the weight of uploaded words and continuously optimize recognition; the confidence coefficient output subunit is used for carrying the confidence coefficient of the recognition result when the recognition result is returned, so that analysis and subsequent processing are carried out according to the confidence coefficient result; in the identification process of the multi-result identification subunit, returning a plurality of identification results meeting the conditions to the application program through the results of the confidence coefficient output subunit instead of unique results, providing a possible identification result list, and arranging according to the confidence coefficient results from high to low; the speaker self-adaptive subunit is used for extracting the voice characteristics of the call on line and automatically adjusting the recognition parameters in the process of multiple conversations between the user and the voice recognition engine unit so as to continuously optimize the recognition effect; the semantic context self-correction subunit is used for dynamically correcting according to the speech recognition result and the context pair in combination with the context dynamic correction, so that the result is more in line with the current context;

the semantic analysis engine unit comprises a rule understanding unit and a model understanding unit, the rule understanding unit is used for matching rules, the model understanding unit comprises a semantic model training subunit, a semantic feature extraction subunit and a semantic similarity evaluation subunit, the semantic model training subunit is used for modeling texts, and mapping the texts with similar semantic expressions to similar vectors in a semantic space; the semantic feature extraction subunit is used for extracting features of the text information and obtaining the text information by extracting a specific hidden vector of the model after the training of the semantic model training subunit is completed; and the semantic similarity evaluation subunit is used for evaluating the similarity between the user sentence vector extracted by deep learning and the offline sentence vector extracted from the library.

3. The intelligent dispatching command and operation system for the regional distribution network of claim 2, wherein for the learning phase unit:

in the learning stage, a labeled data set is provided, and each sample comprises a word unit sequence and a labeled sequence, as follows:

the method comprises the following steps that (1) an ith sample is represented, the front part x represents a character unit sequence of the ith sample, and the rear part y represents a labeling sequence of the ith sample;

constructing a learning model according to the existing labels, and expressing the learning model by using a conditional probability distribution;

the extraction stage unit takes different models as classifiers, so that the extraction mode of the text information which does not pass is obtained, and the extraction data obtained by each classifier is mutually corrected.

4. The system according to claim 3, wherein the text recognition unit introduces context sequence information into the text recognition and performs the recognition through a neural network of a time sequence relationship.

5. The intelligent dispatching command and operation system for the regional distribution network of claim 4, wherein for the speech synthesis unit:

the system comprises a whole-language semantic model, a non-supervision text pre-training model based on multi-language fusion, a text semantic information extraction module and a text prediction information branch construction module, wherein the text prediction information branch is constructed under each language independent condition on the basis of the unified pre-training model;

the pronunciation system model automatically constructs a pronunciation dictionary of a unified unit through a semi-supervised clustering method, and realizes the quick customization of a synthesis system by a language correlation technique modularization method under the condition of limited language resources;

based on a multi-language mixed model of the listening quantization coding, the listening quantization coding is carried out on different attribute information in the voice, and the acoustic parameters of the voice are predicted

Based on generating a high quality speech generating model of the countermeasure network, high quality multilingual speech synthesis is performed by modeling the overall structure of the spectral envelope and modeling the local details of the spectral envelope.

6. The intelligent dispatching command and operation system for the regional distribution network of claim 5, wherein the intelligent shift recognition module comprises a model training phase unit and a model prediction phase unit:

the model training stage unit is used for training a model by using a data set which is synchronous with audio and video and has a voice character label;

and the model prediction stage unit is applied to the model, the voice of a plurality of people with aliasing and the lip movement video of a specific target person are input, the character sequence is recognized, and the voice of the speaker corresponding to the lip movement video content is separated.

7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the intelligent scheduling commanding and operating system for regional distribution network according to any one of claims 1 to 6.

8. A non-transitory computer readable storage medium, having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the intelligent dispatch command and operation system for regional distribution networks as claimed in any one of claims 1 to 6.

Images