CN119341898A

CN119341898A - Large model and multi-modal RAG user equipment fault location method and storage

Info

Publication number: CN119341898A
Application number: CN202411229605.8A
Authority: CN
Inventors: 翟臻
Original assignee: Shanghai Youka Network Technology Co ltd
Current assignee: Shanghai Youka Network Technology Co ltd
Priority date: 2024-09-03
Filing date: 2024-09-03
Publication date: 2025-01-21

Abstract

The present invention relates to the field of Internet of Things technology, and specifically to a user equipment fault location method and storage device of a large model and multi-modal RAG, including: when receiving fault feedback information of a user equipment, obtaining a graphical interface display image of the user equipment; extracting a target graphic screenshot from the graphical interface display image; and inputting a diagnostic large model according to the target graphic screenshot to obtain feedback content corresponding to the fault type. In view of the problem that the operation and maintenance system in the prior art is prone to being unable to diagnose fault information due to unclear descriptions during the communication process, in this solution, the graphical interface display image of the user equipment and specific UI icons are introduced as the main diagnostic information, and the diagnostic large model is used to analyze the interface icon, thereby achieving effective diagnosis of the fault information.

Description

User equipment fault positioning method and memory for large model and multi-mode RAG

Technical Field

The invention relates to the technical field of the Internet of things, in particular to a user equipment fault positioning method and a memory of a large model and a multi-mode RAG.

Background

The internet of things refers to a network composed of entity devices, vehicles, electric appliances and other entity objects, and sensors, software and network connections are embedded in the entity objects, so that data can be collected and shared. The internet of things has a wide range of devices, including simple 'smart home' devices such as smart thermostats, wearable devices such as smart watches and RFID-enabled clothing, and complex industrial machinery and transportation systems. Technical specialists are even thinking of the whole "smart city" based on internet of things technology. The internet of things enables these smart devices to communicate with each other and with other internet-enabled devices. Just like smartphones and gateways, a vast network of interconnected devices is created that can exchange data and autonomously perform various tasks. The Internet of vehicles, namely Internet of things technology applied to an automobile platform, is a common application field. Retrieval enhancement generation (RETRIEVAL-augmented Generation), abbreviated as RAG, is one of the leading edge technologies of the current large models. The search enhancement generation model combines a language model and information search technology. Specifically, when the model needs to generate text or answer questions, the model can firstly retrieve relevant information from a huge document set, and then use the retrieved information to guide the generation of the text, so that the quality and accuracy of prediction are improved.

In the prior art, in order to enable a user side device in an internet of things device to realize a predetermined design function, an operation and maintenance step and an after-sale step are generally required to help a user to remove a device fault. For example, chinese patent CN201810837086.1 discloses an automatic operation and maintenance management system, which comprises an instant communication module for receiving user side information from a mobile communication device, a semantic analysis module for analyzing the user side information to obtain semantic information and matching an operation and maintenance command corresponding to the semantic information, and a result feedback module for feeding back the result of executing the operation and maintenance command, wherein the instant communication module, the semantic analysis module and the result feedback module form data interaction, and the user side information is input by a user in the mobile communication device. The method has the advantage of providing a method for realizing automatic operation and maintenance management of the mobile terminal by the user through natural language.

However, in the practical implementation process, the inventor finds that the implementation process of the technical scheme is mainly realized by means of communication between the user and after-sales operation and maintenance personnel, and when the user cannot describe the fault information correctly, the problem of reduced diagnosis accuracy is easily caused.

Disclosure of Invention

Aiming at the problems in the prior art, a method and a memory for locating faults of user equipment of a large model and a multi-mode RAG are provided.

The specific technical scheme is as follows:

A user equipment fault location method of a large model and a multi-mode RAG comprises the following steps:

Step S1, when fault feedback information of user equipment is received, a graphical interface display image of the user equipment is obtained;

S2, extracting a target graphic screenshot from the graphic interface display image;

And step S3, inputting a diagnosis large model according to the target graph screenshot to acquire feedback content corresponding to the fault type.

On the other hand, the step S1 includes:

step S11, when the fault feedback information is received, a question-answer model is adopted to prompt a user to shoot a display interface of the user equipment so as to acquire a real-time display image;

step S12, collecting a screen area in the real-time display image to obtain a screen image;

and step S13, desensitizing the screen image to obtain the graphical interface display image.

On the other hand, the step S12 includes:

Step S121, acquiring a screen illumination area from the real-time display image;

step S122, matching the screen illumination areas by adopting a preconfigured screen image template to obtain a screen matching template;

and step S123, intercepting and correcting the real-time display image by adopting the screen matching template to obtain the screen image.

On the other hand, in the step S122, the corresponding screen image template is called according to the model of the user equipment.

On the other hand, the step S2 includes:

s21, positioning in the graphical interface display image to obtain a status bar area;

step S22, searching the status bar area by adopting an icon template to obtain an icon area;

and S23, intercepting the graphical interface display image based on the icon area to obtain the target graphic screenshot.

On the other hand, the step S3 includes:

Step S31, processing the target graphic screenshot to obtain graphic description information corresponding to the target graphic screenshot;

and S32, inputting the graphic description information and the target graphic screenshot into the large diagnosis model to acquire the feedback content.

On the other hand, the step S31 includes:

step S311, dividing the target graph screenshot to obtain a divided icon;

step S312, performing template matching on the split icons to obtain the graphic description information corresponding to each split icon.

On the other hand, after executing the step S3, the method further includes:

And S4, returning the feedback content to the user equipment to instruct a user to adjust the user equipment, collecting new fault feedback information, and then returning to the step S1 until the fault is removed.

On the other hand, before executing the step S1, a model fine tuning process is further included, where the model fine tuning process includes:

step S01, collecting fault diagnosis data corresponding to the user equipment;

step S02, constructing a training corpus by adopting the fault diagnosis data;

and S03, training the general large model by using the training corpus to obtain the diagnosis large model.

The memory comprises computer instructions which, when read and executed by a computer device, perform the user device fault location method described above.

The technical scheme has the following advantages or beneficial effects:

Aiming at the problem that the operation and maintenance system in the prior art is easy to cause unclear description in the communication process and can not diagnose fault information, in the scheme, a graphical interface display image of user equipment and a specific UI (user interface) icon are introduced as main information for diagnosis, and meanwhile, a large diagnosis model is adopted for analyzing the interface icon, so that effective diagnosis of the fault information is realized.

Drawings

Embodiments of the present invention will now be described more fully with reference to the accompanying drawings. The drawings, however, are for illustration and description only and are not intended as a definition of the limits of the invention.

FIG. 1 is an overall schematic of an embodiment of the present invention;

FIG. 2 is a schematic diagram of step S1 according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of step S12 according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of step S2 according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of step S3 according to an embodiment of the present invention;

FIG. 6 is a diagram illustrating a step S31 according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of step S4 according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of a model fine tuning process according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

It should be noted that, without conflict, the embodiments of the present invention and features of the embodiments may be combined with each other.

The invention is further described below with reference to the drawings and specific examples, which are not intended to be limiting.

The invention comprises the following steps:

a method for locating faults of user equipment of a large model and a multi-mode RAG is shown in figure 1, and comprises the following steps:

S2, extracting a target graphic screenshot from a graphic interface display image;

Specifically, aiming at the problem that the operation and maintenance system in the prior art is easy to be unclear in description and failure information can not be diagnosed in the communication process, in the embodiment, a graphical interface display image of user equipment and a specific UI (user interface) icon are introduced as main information for diagnosis, and meanwhile, a large diagnosis model is adopted to analyze the interface icon, so that effective diagnosis of the failure information is realized.

In the implementation process, the fault location method of the user equipment is mainly configured in a computer system as a software embodiment, such as a user equipment operation and maintenance system, receives an operation and maintenance work order input from the outside, performs diagnostic analysis and feeds back to a user to a corresponding fault removal method.

In one embodiment, as shown in fig. 2, step S1 includes:

step S11, when fault feedback information is received, a question-answer model is adopted to prompt a user to shoot a display interface of user equipment so as to acquire a real-time display image;

And step S13, desensitizing the screen image to obtain a graphical interface display image.

Specifically, in order to achieve a better information acquisition effect and avoid the problem of inaccurate description in the user input process, in this embodiment, for fault feedback information, a question-answer model is firstly adopted to interact with a user so as to instruct the user to open corresponding rights, and other devices are adopted to shoot a display interface of the user device so as to obtain a real-time display image, wherein the part of the real-time display image is an external image displayed by the user device through a display screen. And then, collecting a screen area in the real-time display image, and removing irrelevant information of a background part to obtain a screen image. Meanwhile, considering that the screen image also comprises personal information of part of users, an application display interface and the like in the screen image are required to be covered so as to achieve a desensitization effect, and finally a graphical interface display image is obtained.

In one embodiment, as shown in fig. 3, step S12 includes:

step S121, acquiring a screen illumination area in a real-time display image;

Step S122, matching the screen illumination area by adopting a preconfigured screen image template to obtain a screen matching template;

Step S123, intercepting and correcting the real-time display image by adopting a screen matching template to obtain a screen image;

In step S122, the corresponding screen image template is invoked according to the model of the user device.

Specifically, in order to achieve a better recognition effect on the screen area, in this embodiment, firstly, the screen illumination area is obtained from the real-time display image by means of gray value detection. Since the illumination of the screen area itself is derived from the light emission of the backlight module or the device, there is a certain difference in gray scale with respect to the ambient light, such as extremely bright in dark or extremely dark in sunlight, so that the screen illumination area is easily obtained by gray scale value matching. Then, the screen illumination areas are matched by adopting a pre-configured screen image template to obtain a screen matching template. The screen image template is determined according to metadata of the user equipment, such as a mobile phone with a specific model number, a car equipment and the like, and the aspect ratio of the screen is easy to obtain. The screen illumination areas are matched according to the aspect ratio, thereby calculating a rough bevel angle size to obtain a corresponding screen matching template. On the basis, a screen matching template is adopted to intercept the real-time display image, and perspective, inclination and the like are corrected to obtain a screen image.

In one embodiment, as shown in fig. 4, step S2 includes:

s21, positioning in a graphical interface display image to obtain a status bar area;

Step S22, searching an icon area from the status bar area by adopting an icon template;

And S23, intercepting the graphical interface display image based on the icon area to obtain a target graphic screenshot.

Specifically, in order to achieve a better matching effect, in this embodiment, after a graphical interface display image is obtained, all height pixels in the image are converted into relative coordinate values, and then the edge of an upper status bar area is determined according to the duty ratio of the relative coordinate values and the change of the color values, so that the status bar area is obtained by positioning and dividing in the graphical interface display image. Then, for the status bar area, an icon template corresponding to a different UI icon is used to search for the status bar area, thereby obtaining an icon area corresponding to the position of the icon. Finally, a plurality of icons in the icon area are identified and intercepted to ultimately determine a target graphical screenshot associated with the status diagnostic.

In one embodiment, as shown in fig. 5, step S3 includes:

Step S31, processing the target graph screenshot to obtain graph description information corresponding to the target graph screenshot;

and S32, inputting the graphic description information and the target graphic screenshot into a diagnosis large model to acquire feedback content.

Specifically, in order to achieve a better input effect, in this embodiment, the target graphic screenshot is first classified according to different icon status types, such as high, medium, low, no signal, etc. of the network signal strength icon, then graphic description information corresponding to the target graphic screenshot is generated according to the classification result, and then the graphic description information and the target graphic screenshot are input together into the diagnosis large model to obtain feedback content.

In one embodiment, as shown in fig. 6, step S31 includes:

step S311, dividing the target graph screenshot to obtain a division icon;

step S312, template matching is performed on the split icons to obtain graphic description information corresponding to each split icon.

Specifically, in order to achieve a better description effect, in this embodiment, a target graphic screenshot corresponding to an entire icon area is first segmented to obtain segmented icons, then template matching is performed on each segmented icon to obtain graphic description information corresponding to each segmented icon, where the graphic description information includes icons that appear in the form of triangles, continuously heightened histograms or continuous dots, and corresponding icon contents, and then the graphic description information is generated.

In one embodiment, as shown in fig. 7, after performing step S3, the method further includes:

and S4, returning the feedback content to the user equipment to instruct the user to adjust the user equipment, collecting new fault feedback information, and returning to the step S1 until the fault is removed.

In one embodiment, as shown in fig. 8, before performing step S1, the method further includes a model fine tuning process, where the model fine tuning process includes:

Step S01, collecting fault diagnosis data corresponding to user equipment;

Step S02, constructing a training corpus by adopting fault diagnosis data;

and S03, training the general large model by using training corpus to obtain a diagnosis large model.

Specifically, in order to achieve a better diagnostic effect on the equipment state, in this embodiment, a fine tuning process for a large model is also introduced between actually starting the inspection. Specifically, fault diagnosis data corresponding to the user equipment, including equipment appearance pictures, equipment parameters, diagnosis results, fault information and the like, are collected in advance for various types of user equipment that may be involved in the after-sales process. Then, a training corpus is constructed based on the operation and maintenance sample data, and the training corpus is assembled according to a specific template to form an input vector. And training the universal large model by using training corpus, and fine-tuning the model to obtain the large model of the Internet of things so as to realize better diagnosis effect.

The storage comprises computer instructions, and when the computer instructions are read and executed by the computer equipment, the operation and maintenance method of the Internet of things equipment is executed.

Those of ordinary skill in the art will appreciate that aspects of the invention, or the possible implementations of aspects, may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention, or the possible implementations of aspects, may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, etc.) or an embodiment combining software and hardware aspects all generally referred to herein as a "circuit," module "or" system. Furthermore, aspects of the invention, or possible implementations of aspects, may take the form of a computer program product, which refers to computer instructions stored in a memory.

The memory may be a computer readable signal medium or a computer readable storage medium. The computer-readable storage medium includes, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing, such as Random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable read-only memory (CD-ROM).

The computer instructions stored in the memory are readable by a processor of the computer, which can cause the processor to perform the functions specified in each step or combination of steps in the flowchart, and produce a device that implements the functions specified in each block of the block diagrams, or combination of blocks.

It should be appreciated that a processor in a computer may be understood as one or more Application Specific Integrated Circuits (ASICs), DSPs, programmable Logic Devices (PLDs), complex Programmable Logic Devices (CPLDs), field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, micro-controllers (MCUs), microprocessors (microprocessors), or other electronic element implementations for executing the aforementioned computer instructions.

The computer instructions may execute entirely on the user's local computer, partly on the user's local computer, as a stand-alone software package, partly on the user's local computer and partly on a remote computer or entirely on the remote computer or server. It should also be noted that in some alternative implementations, the functions noted in the flowchart steps or blocks in the block diagrams may occur out of the order noted in the figures. For example, two steps or blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

The foregoing is merely illustrative of the preferred embodiments of the present invention and is not intended to limit the embodiments and scope of the present invention, and it should be appreciated by those skilled in the art that equivalent substitutions and obvious variations may be made using the description and illustrations of the present invention, and are intended to be included in the scope of the present invention.

Claims

1. A method for locating user equipment faults in a large model and multi-modal RAG, characterized by comprising:

Step S1, when receiving fault feedback information of a user equipment, obtaining a graphical interface display image of the user equipment;

Step S2, extracting a target graphic screenshot from the graphical interface display image;

Step S3, inputting the target graphic screenshot into the diagnosis model to obtain feedback content corresponding to the fault type.

2. The method for locating a fault of a user equipment according to claim 1, wherein step S1 comprises:

Step S11, when the fault feedback information is received, a question-answer model is used to prompt the user to take a picture of the display interface of the user device to obtain a real-time display image;

Step S12, collecting the screen area in the real-time display image to obtain a screen image;

Step S13, desensitizing the screen image to obtain the graphical interface display image.

3. The method for locating a fault of a user equipment according to claim 2, wherein step S12 comprises:

Step S121, acquiring a screen lighting area in the real-time display image;

Step S122, using a pre-configured screen image template to match the screen lighting area to obtain a screen matching template;

Step S123: using the screen matching template to intercept and correct the real-time display image to obtain the screen image.

4. The user equipment fault locating method according to claim 3 is characterized in that, in the step S122, the corresponding screen image template is retrieved according to the model of the user equipment.

5. The method for locating a fault of a user equipment according to claim 1, wherein step S2 comprises:

Step S21, locating a status bar area in the graphical interface display image;

Step S22, searching and obtaining an icon area from the status bar area using an icon template;

Step S23: intercepting the graphical interface display image based on the icon area to obtain the target graphic screenshot.

6. The method for locating a fault of a user equipment according to claim 1, wherein step S3 comprises:

Step S32: input the graphic description information and the target graphic screenshot into the diagnosis macro model to obtain the feedback content.

7. The method for locating a fault of a user equipment according to claim 6, wherein step S31 comprises:

Step S311, segmenting the target graphic screenshot to obtain segmented icons;

Step S312: performing template matching on the segmented icons to obtain the graphic description information corresponding to each segmented icon.

8. The method for locating a fault of a user equipment according to claim 1, characterized in that after executing step S3, it further comprises:

Step S4, returning the feedback content to the user equipment to instruct the user to adjust the user equipment, and collecting new fault feedback information, and then returning to step S1 until the fault is eliminated.

9. The method for locating a fault of a user equipment according to claim 1, characterized in that before executing the step S1, it further comprises a model fine-tuning process, wherein the model fine-tuning process comprises:

Step S01, collecting fault diagnosis data corresponding to the user equipment;

Step S02, constructing a training corpus using the fault diagnosis data;

Step S03: Use the training corpus to train the general large model to obtain the diagnosis large model.

10. A memory comprising computer instructions, characterized in that when a computer device reads and executes the computer instructions, the user equipment fault locating method according to any one of claims 1 to 9 is executed.