CN117975968B - Remote patrol system control method and system based on sound and language model - Google Patents

Abstract

The invention discloses a remote patrol system control method and a remote patrol system control system based on a sound and language model, relates to the field of intelligent patrol of transformer substations, and solves the problems that the existing remote patrol system is complex in operation, easy to make mistakes in inquiry and untimely in result notification; comprising the following steps: acquiring voice data, and converting the voice data into text data according to the fine-tuned voice recognition model; parsing intention data from the text data according to the remote patrol expert model, the intention data comprising: API interface and interface parameter of the service to be called; analyzing interface parameters in the intention data, and enhancing the interface parameters to obtain enhanced intention data; generating a service call request according to the enhanced intention data, and calling a remote inspection system to execute a service through the service call request to obtain a service result; inputting the service result into a remote patrol expert model to generate reply text data; converting the reply text data into reply voice data and outputting the reply voice data; the remote patrol system can be controlled based on sound.

Description

A remote patrol system control method and system based on sound and language model

Technical Field

The present invention relates to the field of intelligent inspection of transformer substations, and more specifically, to a remote inspection system control method and system based on sound and language models.

Background Art

The substation remote inspection system is a system planned by the State Grid Corporation of China that uses robots, drones, voiceprint devices, cameras, etc. as perception layer devices, and generates inspection results after intelligent analysis of collected data through the algorithm host. The system mainly performs inspection tasks of substation equipment by manually initiating inspections, setting periodic inspection tasks, or triggering alarm signals from main and auxiliary systems. The system carries out inspection tasks around inspection points. Inspection points refer to business checkpoints, and there are generally multiple inspection points for one device. The minimum business monitoring point is usually constructed by one or more camera, drone or robot preset positions to observe whether there are defects in the components of each inspection point. The system is designed with a retrieval module based on point attributes, which is used to query the inspection points and perform equipment monitoring and viewing, inspection task initiation, inspection equipment settings, and other operations.

However, due to the numerous business attributes of patrol points, such as substation area, substation interval, equipment name, component name, phase name, point name, point code, patrol type, importance level, recognition algorithm, perception layer equipment, etc., the query accuracy is not high during manual retrieval, resulting in cumbersome operation. When an emergency occurs, it is impossible to check the relevant conditions of the equipment in the first time, especially for substations without primary and auxiliary systems or patrol points not covered by the primary and auxiliary systems. On the other hand, since multiple patrol points are involved, the patrol process lasts for a long time, and the patrol results may not be observed by on-site staff in the first time after they are generated.

Summary of the invention

The purpose of this application is to provide a remote patrol system control method and system based on sound and language models, so as to solve the problems of cumbersome operation, error-prone query and untimely notification of results of existing remote patrol systems; to expand the existing substation remote patrol system, realize remote patrol system control through sound and language large models, and enable staff to communicate with the remote patrol system through voice input, guide the remote patrol system to quickly complete patrol-related work, and obtain voice feedback on patrol task results, so as to simplify the operation process of the remote patrol system, reduce the difficulty of operation, and facilitate staff to know the task results in a timely manner.

The present application first provides a remote patrol system control method based on sound and language models, including: acquiring voice data, and converting the voice data into text data according to a fine-tuned voice recognition model; parsing intent data from the text data according to a remote patrol expert model, the intent data including: an API interface and interface parameters of a service to be called, the remote patrol expert model being a SOTA large language model obtained through training; parsing the interface parameters in the intent data, and enhancing the interface parameters to obtain enhanced intent data; generating a service call request according to the enhanced intent data, and calling the remote patrol system to execute the service through the service call request to obtain a service result; inputting the service result into the remote patrol expert model to generate reply text data; and converting the reply text data into reply voice data for output.

By adopting the above technical solution, the service interface of the remote patrol system can be called by obtaining the voice data of the staff, and the remote patrol system can be controlled to perform services. The results can be displayed by voice, and the remote patrol system can be controlled to perform services by voice. The service results are also returned in the form of voice, which simplifies the operation process of the staff. In addition, accurate calling can be achieved by enhancing the intent of the interface parameters, reducing the requirements for the staff.

In a possible implementation, the fine-tuned speech recognition model is obtained in the following manner: a speech recognition SOTA model is acquired, and the speech recognition SOTA model is fine-tuned according to proprietary speech text data in the electric power field to obtain a fine-tuned speech recognition model.

In a possible implementation, the remote patrol expert model is obtained in the following manner: obtaining the API interface and annotations of the remote patrol system to form an intention implementation seed library; supplementing the context and placeholder of the remote patrol system information for each API interface in the intention implementation seed library to generate a placeholder intention expression; inputting the placeholder intention expression into a general large language model for expansion to form a placeholder intention expression data set; manually reviewing the placeholder intention expression data set to filter out erroneous placeholder intention expressions; replacing the placeholders in the placeholder intention expression data set with the power equipment information, point information, and perception layer equipment information in the remote patrol system to obtain an intention expression data set; training the large language model based on the intention expression data set to obtain a remote patrol expert model.

In a possible implementation, the enhanced intent data is obtained in the following manner: synchronizing the relational data and entity data in the remote patrol system to generate a document database, vectorizing the entity data in the remote patrol system through a word embedding model to generate a vector database; retrieving the interface parameters in the document database and the vector database respectively, and performing an intersection operation on the retrieval results to obtain enhanced interface parameters; and combining the enhanced interface parameters and the API interface of the service to be called into enhanced intent data.

In a possible implementation, the method further includes: acquiring an alarm text of a remote patrol system, and converting the alarm text into a voice output.

The present application also provides a remote patrol system control system based on sound and language models, including: a speech recognition module, used to obtain speech data, and convert the speech data into text data according to a fine-tuned speech recognition model; a large language model module, used to parse intent data from the text data according to a remote patrol expert model, the intent data including: an API interface and interface parameters of a service to be called, the remote patrol expert model is a SOTA large language model obtained through training; a retrieval enhancement module, used to parse the interface parameters in the intent data, and enhance the interface parameters to obtain enhanced intent data; an intent call module, used to generate a service call request according to the enhanced intent data, and call the remote patrol system to execute the service through the service call request to obtain a service result; the large language model module is also used to input the service result into the remote patrol expert model to generate reply text data; a speech generation module, used to convert the reply text data into reply voice data output.

In a possible implementation, the speech recognition module is also used to obtain a speech recognition SOTA model, and fine-tune the speech recognition SOTA model according to proprietary speech text data in the power field to obtain a fine-tuned speech recognition model.

In a possible implementation, the large language model module is also used to: obtain the API interface and annotations of the remote patrol system to form an intention realization seed library; supplement the context and placeholder of the remote patrol system information for each API interface in the intention realization seed library to generate a placeholder intention expression; input the placeholder intention expression into a general large language model for expansion to form a placeholder intention expression data set; manually review the placeholder intention expression data set to screen out erroneous placeholder intention expressions; replace the placeholders in the placeholder intention expression data set with the power equipment information, point information, and perception layer equipment information in the remote patrol system to obtain an intention expression data set; train the large language model based on the intention expression data set to obtain a remote patrol expert model.

In a possible implementation, the retrieval enhancement module is also used to: synchronize relational data and entity data in the remote patrol system to generate a document database, vectorize the entity data in the remote patrol system through a word embedding model to generate a vector database; retrieve the interface parameters in the document database and the vector database respectively, and perform intersection operations on the retrieval results to obtain enhanced interface parameters; and combine the enhanced interface parameters and the API interface of the service to be called into enhanced intent data.

In a possible implementation, the voice generation module is further used to obtain an alarm text of the remote patrol system and convert the alarm text into a voice output.

Compared with the prior art, the present application has the following beneficial effects: the present application collects the voice data of the staff, converts the voice data into text data through the voice recognition model, parses the intention data through the remote patrol expert model, and then calls the corresponding service of the remote patrol system through the intention data, and converts the structured service results into semantically coherent text through the remote patrol expert model, and finally outputs them through voice; the staff can control the remote patrol system to perform tasks through voice, and finally return the task results in voice form, which simplifies the operation process of the remote patrol system and facilitates timely acquisition of task results;

This application enhances the interface parameters in the intent data, combines the document database and the vector database to retrieve the expression closest to the interface parameters, ensures accurate control, avoids control errors caused by non-standard expressions of staff, improves fault tolerance, and reduces operational requirements for staff;

This application uses API interfaces and annotations to form an intent seed library, generates placeholder intent expressions by supplementing the context and placeholders of the remote patrol system information, expands the training corpus through a general large language model, obtains an intent expression dataset by replacing placeholders with entities, and trains a remote patrol expert model through the intent expression dataset. By inputting a piece of text data, the API interface and interface parameters of the service to be called can be parsed, thereby controlling the remote patrol system to perform tasks.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

FIG1 is a flow chart of a remote patrol system control method based on sound and language models;

FIG2 is a flow chart of training a remote patrol expert model;

FIG3 is a schematic diagram of the process of enhancing interface parameter retrieval;

FIG4 is a schematic diagram of the structure of the remote patrol system control system based on sound and language models.

DETAILED DESCRIPTION

In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with examples and drawings. The illustrative implementation scheme of the present application and its description are only used to explain the present application and are not intended to limit the present application.

Please refer to Figure 1, which is a flow chart of a remote patrol system control method based on sound and language models, the method comprising: S1, acquiring voice data, and converting the voice data into text data according to a fine-tuned speech recognition model; S2, parsing intent data from the text data according to a remote patrol expert model, the intent data comprising: an API interface and interface parameters of a service to be called, the remote patrol expert model being a SOTA large language model obtained through training; S3, parsing the interface parameters in the intent data, and enhancing the interface parameters to obtain enhanced intent data; S4, generating a service call request according to the enhanced intent data, and calling the remote patrol system to execute the service through the service call request to obtain a service result; S5, inputting the service result into the remote patrol expert model to generate reply text data; S6, converting the reply text data into reply voice data output.

Specifically, the existing remote patrol system is improved by acquiring the voice data of the staff through audio acquisition equipment, converting the voice data into text data through a speech recognition model, parsing the intention data through a remote patrol expert model, retrieving the interface parameters to accurately and enhance the intention data, and then calling the remote patrol system through the intention data to execute the corresponding service task. The returned task results are converted into semantically coherent text through the remote patrol expert model and finally output through voice.

The improvement of this solution is that by obtaining the voice data of the staff, the service interface of the remote patrol system can be called, the remote patrol system can be controlled to perform services, and the results can be displayed in the form of voice, so that the voice control remote patrol system can perform services and the results can be notified in time in the form of voice, simplifying the operation process of the staff. In addition, accurate calling can be achieved by retrieving the interface parameters, reducing the requirements for the staff.

Step S1 obtains voice data, and converts the voice data into text data according to the fine-tuned voice recognition model. In order to improve the voice recognition model's recognition ability for specialized vocabulary in the power field, the voice recognition model is fine-tuned using pre-recorded voice text data specific to the power field. For example, the whisper open sourced by OpenAI is used as a voice recognition model, and the voice recognition model is fine-tuned (Fine-Tune) in the power field through pre-recorded voice and corresponding text data specific to the power field, so as to improve the voice recognition model's recognition accuracy for specialized vocabulary in the power industry. Furthermore, in order to reduce the computing power cost of fine-tuning training, Lora or QLora can be used for fine-tuning.

Step S2 parses the intent data from the text data according to the remote patrol expert model, and the intent data includes: the API interface and interface parameters of the service to be called. The purpose of the remote patrol expert model is to perform intent analysis on the text data generated by the speech recognition large model and simulate human natural language conversations. An open source large language model can be selected for intent recognition training to obtain a remote patrol expert model. For example, the ChatGLM3-6B model. In the traditional way, intent recognition training for a large language model requires manual writing of a large amount of corpus data. In order to reduce the cost of labor input, this solution uses a general large language model to write training corpus data for intent recognition training to obtain a remote patrol expert model.

Please refer to Figure 2, which is a flow chart of training a remote patrol expert model. The remote patrol expert model is obtained by:

S21. Obtain the API interface and annotations of the remote patrol system to form an intent realization seed library. Specifically, after collecting the relevant API interface source codes that require voice interaction in the existing remote patrol system, manually supplement and adjust the parts of the source code where the annotations are missing or unclear, and remove the API interface implementation part of the source code. Each pair of API interface and annotation constitutes an API interface pair, and multiple API interface pairs constitute an intent realization seed library.

S22. Supplement the context and placeholder of the remote patrol system information for each API interface in the intent implementation seed library to generate a placeholder intent expression; S23. Input the placeholder intent expression into a general large language model for expansion to form a placeholder intent expression data set. Specifically, the placeholder intent expression is a piece of text, which indicates that the API interface needs to be called to perform an operation on the placeholder. The placeholder intent expression is input into the general large language model LLM to perform the intent expression generation task, and a placeholder intent expression data set consisting of multiple placeholder intent expressions is obtained. The general large language model here can select a commercial or open source large language model, such as ChatGPT, ChatGLM, Wenxin Yiyan, Llama 2, etc.

For example, use the ChatGPT 3.5-Turbo model to perform the intent expression generation task. The following lists a possible input and output, where the input is a placeholder intent expression and the output is multiple expanded placeholder intent expressions, which are used to form a placeholder intent expression dataset. For example:

Enter the general large language model LLM:

"The substation remote inspection system developed based on Springboot has the following interfaces (the specific implementation part is hidden)

###

@ApiOperation(value = "Turn the device-related camera to the corresponding preset point. These video preset points are usually facing a certain component of the device")

@PostMapping("/cameraFoucsDevice")

public WrappedResult<Boolean>cameraFoucsDevice(

@ApiParam(value = "Name of substation equipment", name = "deviceName",required = true) @RequestParam("deviceName") String deviceName,

@ApiParam(value = "device component name", name = "partName",required = false) @RequestParam("partName") String partName

){}

###

You are a professional Chinese linguist. Please help me list the expressions you know. The intentions expressed by these sentences can be realized by calling the corresponding interfaces.

The sentences need to meet the following requirements:

1. According to the definition of the function, the sentence needs to combine the function parameters. In the sentence you generate, strictly use the parameter name as the placeholder of the entity, such as {deviceName}

2. Pay attention to optional parameters. If a parameter is optional, the sentence can exclude the parameter. If the parameter is excluded, there cannot be other entities referring to the parameter in the sentence. The sentence must be grammatically correct.

3. The sentence should be expressed as the user role of the system and should conform to the human expression, such as: I need to see the monitoring video of the transformer

4. Try to list different expressions of the same intent as much as possible, cover all parameters of the interface, and ensure that the sentences are not repeated

The return format for each sentence is:

Intent: sentence Action: name of the interface to be called Params: parameter list".

Accordingly, the general large language model LLM returns:

“Intent: I want to move the camera corresponding to the {deviceName} device to the preset position Action:cameraFoucsDevice Params: deviceName

Intent: Can you help me focus my camera on {deviceName}? Action: cameraFoucsDevice Params: deviceName

Intent: I need to focus the camera on {deviceName} device Action:cameraFoucsDevice Params: deviceName

Intent: Focus the camera on {deviceName} device Action: cameraFoucsDeviceParams: deviceName

Intent: Focus the camera on the part of the {deviceName} device Action: cameraFoucsDeviceParams: deviceName, partName

Intent: Can you help me turn the camera to focus on {partName} of {deviceName}? Action: cameraFoucsDevice Params: deviceName, partName

Intent: Can you adjust the camera to the {partName} component, which belongs to the {deviceName} device? Action: cameraFoucsDevice Params: deviceName, partName

Intent: I need to adjust the focus of the camera to the {partName} component of the {deviceName} device Action: cameraFoucsDevice Params: deviceName, partName".

S24, manually review the placeholder intention expression data set to filter out incorrect placeholder intention expressions. Specifically, the placeholder intention expression data set generated by the general large language model may contain inappropriate expressions, which staff will not use. Therefore, it is necessary to manually filter out incorrect placeholder intention expressions.

S25. Replace the placeholders in the placeholder intent expression data set with the power equipment information, point information, and perception layer equipment information in the remote patrol system to obtain the intent expression data set. For example, replace the component "oil pillow" of the equipment "main transformer" with the placeholder in the placeholder intent expression "zoom in on the {partName} part of {deviceName}, I need to see it clearly", and generate: "zoom in on the oil pillow part of the main transformer, I need to see it clearly", and add it to the intent expression data set.

Specifically, the API Schema of the API interface in step S21 is obtained using the Swagger document, and the placeholder intent expression dataset generated in step S23 is mixed with the API Schema through a Python script to obtain an intent expression dataset that can be used for intent training and verification of a large language model (for example, the ChatGLM3-6B model). The following is an expression of the intent expression dataset:

"{

"tools": [

"cameraFoucsDevice: Move the device-related camera to the corresponding preset point. These video preset points are generally facing a certain component of the device\nParameters: {\"deviceName\": \"Required. string. The name of the substation equipment\", \"partName\": \"Optional. string. The name of the device component.\"}\nOutput: boolean[True,False]\n"

],

"conversations": [

{

"role": "user",

"content": "I want to move the camera corresponding to the main transformer device to the preset point"

},

{

"role": "assistant",

"content": "I need to use cameraFoucsDevice to turn the camera to the main transformer area."

},

{

"role": "tool",

"name": "cameraFoucsDevice",

"parameters": {

"deviceName": "Main Transformer"

},

"observation": "True"

},

{

"role": "assistant",

"content": "The operation returns True, the camera has been successfully transferred to the main transformer"

}

]

}".

S26. Train the large language model based on the intention expression data set to obtain a remote patrol expert model. Specifically, the remote patrol expert model can identify the remote patrol system API interface and interface parameters that should be called based on the text data, and pass it in the form of Python function execution source code to generate a service call request. For example:

Input the remote patrol expert model: "I want to move the camera corresponding to the main transformer equipment to the preset point."

Remote patrol expert model output: "dispath_request("cameraFoucsDevice","main transformer")".

Step S3 parses the interface parameters in the intent data and enhances the interface parameters to obtain enhanced intent data. The purpose of parsing and enhancing the interface parameters in the intent data is to ensure that the interface parameters of the API interface call are accurate and to improve the robustness of the interface call. For example, the data authority of the remote patrol system staff is a certain substation. After being parsed by the remote patrol expert model, the interface parameters displayed in the intent data are "Main Transformer No. 1" or "Main Transformer", but there is only one transformer named "Main Transformer No. 1" in the substation. After enhancing the interface parameters, it will return to "Main Transformer No. 1".

Among them, the enhanced intent data is obtained in the following manner: synchronizing the relational data and entity data in the remote patrol system to generate a document database, vectorizing the entity data in the remote patrol system through a word embedding model to generate a vector database; searching the interface parameters in the document database and the vector database respectively, and performing an intersection operation on the search results to obtain enhanced interface parameters; combining the enhanced interface parameters and the API interface of the service to be called into enhanced intent data.

Specifically, the relational data and entity data (patrol points, power equipment, personnel information, etc.) in the remote patrol system are synchronized to the document database, and when synchronizing data, the entity data is vectorized and expressed through the word embedding model (Embeding Model) and stored in the vector database. Milvus can be used as the vector database, Elasticsearch as the document database, and bge-large-zh as the word embedding model. For an interface parameter, on the one hand, the interface parameter is searched by keywords in the document database (Elasticsearch), specifically, the _search interface (provided by Elasticsearch) is used for fuzzy query to obtain a list of entities similar to the interface parameter. On the other hand, the interface parameter is vectorized through the Embeding model, and the list of entities with semantics similar to the interface parameter is queried in the vector database (Milvus). Specifically, the Euclidean distance indicator is used in the vector database to query the nearest entity list. Perform "intersection operation" on the two entity lists mentioned above: Specifically, in order to be compatible with Chinese homophones, the two lists are first merged and the Chinese in the list is converted into pinyin. The pypinyin library can be used to perform this operation. Finally, the input interface parameter is converted into pinyin, and then the edit distance (Edit Distance) similarity is calculated between it and the pinyin in the list. The search result with the highest similarity is obtained from the list as the final output. As shown in Figure 3, Figure 3 is a schematic diagram of the process of interface parameter retrieval enhancement.

Step S4 generates a service call request according to the enhanced intention data, and calls the remote patrol system to execute the service through the service call request to obtain the service result. Specifically, the enhanced intention data in the form of Python function source code transmitted by the remote patrol expert model is executed, and an HTTP request for calling the remote patrol system API interface is initiated. The remote patrol system calls the service corresponding to the API interface according to the interface parameters and returns the service result.

Step S5: input the service result into the remote inspection expert model to generate reply text data. Specifically, the returned service result is generated into reply text data expressed in natural language by the remote inspection expert model.

Step S6 converts the reply text data into reply voice data for output. Specifically, the reply text data is passed to a speech generation model (PaddleSpeech, OpenTTS, eSpeak model, etc.) to be converted into reply voice, and then output through the audio speaker of the remote patrol system.

In a possible implementation, step S6 further includes: acquiring an alarm text of the remote patrol system, converting the alarm text into a reply voice, and outputting the reply voice through an audio speaker of the remote patrol system.

It can be understood that the remote patrol system control method based on sound and language models provided in this solution, on the one hand, calls the API interface service of the remote patrol system through voice dialogue, such as calling cameras, drones, and robots to perform patrol tasks, obtains service results, and simplifies the operation process; on the other hand, during and after the patrol, patrol alarms, patrol results, etc. are converted into voice broadcasts to promptly notify on-site staff.

Please refer to Figure 4, which is a structural diagram of a remote patrol system control system based on sound and language models. The system is used to implement the remote patrol system control method based on sound and language models as described above. The system includes: a speech recognition module, which is used to obtain speech data and convert the speech data into text data according to a fine-tuned speech recognition model; a large language model module, which is used to parse intent data from the text data according to a remote patrol expert model, and the intent data includes: an API interface and interface parameters of a service to be called, and the remote patrol expert model is a SOTA large language model obtained through training; a retrieval enhancement module, which is used to parse the interface parameters in the intent data and enhance the interface parameters to obtain enhanced intent data; an intent call module, which is used to generate a service call request according to the enhanced intent data, and to call the remote patrol system to execute the service through the service call request to obtain a service result; the large language model module is also used to input the service result into the remote patrol expert model to generate reply text data; and a speech generation module is used to convert the reply text data into reply speech data output.

In a possible implementation, the speech recognition module is also used to obtain a speech recognition SOTA model, and fine-tune the speech recognition SOTA model according to proprietary speech text data in the power field to obtain a fine-tuned speech recognition model.

In a possible implementation, the large language model module is also used to: obtain the API interface and annotations of the remote patrol system to form an intention realization seed library; supplement the context and placeholder of the remote patrol system information for each API interface in the intention realization seed library to generate a placeholder intention expression; input the placeholder intention expression into a general large language model for expansion to form a placeholder intention expression data set; manually review the placeholder intention expression data set to screen out erroneous placeholder intention expressions; replace the placeholders in the placeholder intention expression data set with the power equipment information, point information, and perception layer equipment information in the remote patrol system to obtain an intention expression data set; train the large language model based on the intention expression data set to obtain a remote patrol expert model.

In a possible implementation, the retrieval enhancement module is also used to: synchronize relational data and entity data in the remote patrol system to generate a document database, vectorize the entity data in the remote patrol system through a word embedding model to generate a vector database; retrieve the interface parameters in the document database and the vector database respectively, and perform intersection operations on the retrieval results to obtain enhanced interface parameters; and combine the enhanced interface parameters and the API interface of the service to be called into enhanced intent data.

In a possible implementation, the voice generation module is further used to obtain an alarm text of the remote patrol system and convert the alarm text into a voice output.

Compared with the prior art, first, this application collects the voice data of the staff, converts the voice data into text data through the voice recognition model, parses the intention data through the remote patrol expert model, and then calls the corresponding service of the remote patrol system through the intention data, and converts the structured service results into semantically coherent text through the remote patrol expert model, and finally outputs it through voice; the staff can control the remote patrol system to perform tasks through voice, and finally return the task results in voice form, simplifying the operation process of the remote patrol system, and facilitating timely learning of the task results. Second, this application enhances the interface parameters in the intention data, combines the document database and the vector database to retrieve the expression closest to the interface parameters, ensures precise control, avoids control errors caused by the non-standard expression of the staff, improves the fault tolerance rate, and reduces the operation requirements for the staff. Third, this application forms an intention seed library through API interfaces and annotations, generates placeholder intention expressions by supplementing the context and placeholders of the remote patrol system information, expands the training corpus through the general large language model, obtains the intention expression data set by replacing the placeholders with entities, trains the remote patrol expert model through the intention expression data set, and inputs a piece of text data to parse the API interface and interface parameters of the service to be called, thereby controlling the remote patrol system to perform tasks.

The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims (6)

1. A method for controlling a remote patrol system based on a sound and language model, comprising:

Acquiring voice data, and converting the voice data into text data according to a fine-tuned voice recognition model;

Parsing intention data from the text data according to a remote patrol expert model, the intention data comprising: the remote patrol expert model is obtained by training an SOTA large language model;

Analyzing interface parameters in the intention data, and enhancing the interface parameters to obtain enhanced intention data;

Generating a service call request according to the enhanced intention data, and calling a remote patrol system to execute a service through the service call request to obtain a service result;

inputting the service result into a remote patrol expert model to generate reply text data;

converting the reply text data into reply voice data to be output;

The remote patrol expert model is obtained by the following steps: acquiring an API interface and comments of a remote patrol system to form an intention realization seed library; supplementing the context and placeholder of the remote patrol system information for each API interface in the intention realization seed library to generate a placeholder intention expression; the occupation intention expression is input into a general large language model to be expanded, so that a occupation intention expression data set is formed; manually inspecting the occupation intention expression data set, and screening out wrong occupation intention expressions; replacing placeholders in the placeholder intention expression data set with power equipment information, point location information and perception layer equipment information in a remote inspection system to obtain an intention expression data set; training the SOTA large language model based on the intention expression data set to obtain a remote patrol expert model;

The enhancement intention data is obtained by the following steps: synchronizing the relational data and the entity data in the remote patrol system to generate a document database, and vectorizing the entity data in the remote patrol system through a word embedding model to generate a vector database; searching the interface parameters in the document database and the vector database respectively, and performing intersection operation on the search result to obtain enhanced interface parameters; combining the enhanced interface parameters and an API interface of the service to be called into enhanced intention data;

Searching the interface parameters in the document database and the vector database respectively, and performing intersection operation on the search result to obtain enhanced interface parameters; comprising the following steps: on one hand, the interface parameters are subjected to keyword retrieval of a document database, and an entity list similar to the interface parameters is obtained; on the other hand, after the interface parameters are expressed in a vector mode through Embeding, an entity list similar to the interface parameter semantics is inquired in a vector database; in order to be compatible with the situation of Chinese harmonic words, the two lists are combined, chinese in the lists is converted into pinyin, the input interface parameters are converted into pinyin, edit distance similarity calculation is carried out on the input interface parameters and the pinyin in the lists, and the search result with the maximum similarity is obtained from the lists and is finally output.

2. A method of controlling a remote tour system according to claim 1, wherein the fine-tuned speech recognition model is obtained by: and acquiring a voice recognition SOTA model, and performing fine adjustment on the voice recognition SOTA model according to the special voice text data in the electric power field to obtain a fine-adjusted voice recognition model.

3. A method of controlling a remote tour system based on a sound and language model according to claim 1, further comprising: and acquiring an alarm text of the remote patrol system, and converting the alarm text into voice output.

4. A voice and language model based remote patrol system control system for performing a voice and language model based remote patrol system control method as claimed in any one of claims 1-3, comprising:

The voice recognition module is used for acquiring voice data and converting the voice data into text data according to the fine-tuned voice recognition model;

The large language model module is used for analyzing intention data from the text data according to a remote patrol expert model, and the intention data comprises: the remote patrol expert model is obtained by training an SOTA large language model;

the retrieval enhancement module is used for analyzing the interface parameters in the intention data and enhancing the interface parameters to obtain enhanced intention data;

the intention calling module is used for generating a service calling request according to the enhanced intention data, and calling a remote inspection system execution service through the service calling request to obtain a service result;

The large language model module is also used for inputting the service result into a remote patrol expert model to generate reply text data;

the voice generation module is used for converting the reply text data into reply voice data and outputting the reply voice data;

Wherein, the big language model module is further used for: acquiring an API interface and comments of a remote patrol system to form an intention realization seed library; supplementing the context and placeholder of the remote patrol system information for each API interface in the intention realization seed library to generate a placeholder intention expression; the occupation intention expression is input into a general large language model to be expanded, so that a occupation intention expression data set is formed; manually inspecting the occupation intention expression data set, and screening out wrong occupation intention expressions; replacing placeholders in the placeholder intention expression data set with power equipment information, point location information and perception layer equipment information in a remote inspection system to obtain an intention expression data set; training a large language model based on the intention expression data set to obtain a remote patrol expert model;

The retrieval enhancement module is further configured to: synchronizing the relational data and the entity data in the remote patrol system to generate a document database, and vectorizing the entity data in the remote patrol system through a word embedding model to generate a vector database; searching the interface parameters in the document database and the vector database respectively, and performing intersection operation on the search result to obtain enhanced interface parameters; combining the enhanced interface parameters and an API interface of the service to be called into enhanced intention data;

Searching the interface parameters in the document database and the vector database respectively, and performing intersection operation on the search result to obtain enhanced interface parameters; comprising the following steps: on one hand, the interface parameters are subjected to keyword retrieval of a document database, and an entity list similar to the interface parameters is obtained; on the other hand, after the interface parameters are expressed in a vector mode through Embeding, an entity list similar to the interface parameter semantics is inquired in a vector database; in order to be compatible with the situation of Chinese harmonic words, the two lists are combined, chinese in the lists is converted into pinyin, the input interface parameters are converted into pinyin, edit distance similarity calculation is carried out on the input interface parameters and the pinyin in the lists, and the search result with the maximum similarity is obtained from the lists and is finally output.

5. The voice and language model based remote tour system control system according to claim 4, wherein the voice recognition module is further configured to obtain a voice recognition SOTA model, and fine tune the voice recognition SOTA model according to the voice text data specific to the electric power domain, so as to obtain a fine tuned voice recognition model.

6. The voice and language model based remote tour system control system according to claim 4, wherein the voice generating module is further configured to obtain an alert text of the remote tour system, and convert the alert text into a voice output.