CN119152405A - Two-way video fusion interactive maintenance AR glasses system and maintenance method - Google Patents
Two-way video fusion interactive maintenance AR glasses system and maintenance method Download PDFInfo
- Publication number
- CN119152405A CN119152405A CN202411616639.2A CN202411616639A CN119152405A CN 119152405 A CN119152405 A CN 119152405A CN 202411616639 A CN202411616639 A CN 202411616639A CN 119152405 A CN119152405 A CN 119152405A
- Authority
- CN
- China
- Prior art keywords
- image data
- image
- sketching
- information
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000011521 glass Substances 0.000 title claims abstract description 76
- 230000004927 fusion Effects 0.000 title claims abstract description 48
- 230000002452 interceptive effect Effects 0.000 title claims abstract description 45
- 238000000034 method Methods 0.000 title claims abstract description 34
- 238000012423 maintenance Methods 0.000 title claims abstract description 21
- 230000003190 augmentative effect Effects 0.000 claims abstract description 42
- 238000004364 calculation method Methods 0.000 claims abstract description 26
- 230000009471 action Effects 0.000 claims description 44
- 238000012545 processing Methods 0.000 claims description 32
- 230000003993 interaction Effects 0.000 claims description 27
- 230000002457 bidirectional effect Effects 0.000 claims description 15
- 238000007781 pre-processing Methods 0.000 claims description 13
- 230000005540 biological transmission Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 7
- 230000002787 reinforcement Effects 0.000 claims 1
- 238000005516 engineering process Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000005728 strengthening Methods 0.000 description 6
- 238000007689 inspection Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
Landscapes
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a bi-directional video fusion interactive overhaul AR (augmented reality) glasses system and an overhaul method, and relates to the technical field of AR glasses, wherein the overhaul AR glasses system comprises augmented reality glasses, an edge computing server and a video fusion server, wherein the augmented reality glasses are used for acquiring field image data and transmitting the field image data to the edge computing server; the system comprises a target operation terminal, an edge calculation server and an augmented reality glasses, wherein the target operation terminal is used for acquiring sketching information and gesture image data, transmitting the sketching information and the gesture image data to the edge calculation server, and the edge calculation server is used for fusing the field image data, the sketching information and the gesture image data to generate a fused interactive video and transmitting the fused interactive video to the augmented reality glasses and the target operation terminal for display. The invention solves the technical problems that the AR glasses in the related technology have poor interactivity and cannot meet the accurate maintenance requirement.
Description
Technical Field
The invention relates to the technical field of AR (augmented reality) glasses, in particular to a bi-directional video fusion interactive overhaul AR glasses system and an overhaul method.
Background
Currently, for relay protection maintenance in the transformer substation, maintenance staff can realize maintenance through AR (Augmented Reality, namely augmented reality) glasses.
In the related art, by modeling the overhaul site in advance, flow information of various standardized jobs can be constructed and uploaded to a server, then a client requests the server to download the flow information of the standardized jobs corresponding to the current overhaul job, the flow information of the standardized jobs is played by a loudspeaker of the client or the AR glasses, and the current step of the flow is displayed by a display screen of the AR glasses, so that an overhaul worker can overhaul through the played flow information and the current step displayed by the display screen.
However, the above-mentioned overhaul scheme requires modeling the site in advance, and the workload of model establishment and maintenance is large, and the model needs to be modified once the site changes. Moreover, the current AR glasses working mode cannot realize the foreground-background real-time interaction, cannot meet the field complex interaction requirement, and is poor in user experience. In addition, the current AR glasses are guided according to unidirectional videos, overhaulers and the background interact through voice, relay protection operation requirements are accurate, centimeter-level precision guidance is needed for operations of pressing plates, terminal blocks and the like, and work cannot be completed by means of the current voice and guidance modes.
In view of the above problems, no effective solution has been proposed at present.
Disclosure of Invention
The embodiment of the invention provides a bi-directional video fusion interactive overhaul AR (augmented reality) glasses system and an overhaul method, which are used for at least solving the technical problems that the AR glasses in the related technology are poor in interactivity and cannot meet the accurate overhaul requirement.
According to one aspect of the embodiment of the invention, a bi-directional video fusion interactive overhaul AR (augmented reality) glasses system is provided, and comprises augmented reality glasses, a target operation terminal and an edge calculation server, wherein the augmented reality glasses are used for collecting field image data and transmitting the field image data to the edge calculation server, the target operation terminal is used for obtaining sketching information and gesture image data and transmitting the sketching information and the gesture image data to the edge calculation server, and the edge calculation server is used for fusing the field image data, the sketching information and the gesture image data to generate fusion interactive videos and transmitting the fusion interactive videos to the augmented reality glasses and the target operation terminal for display, and the fusion interactive videos are used for guiding overhaul objects wearing the augmented reality glasses to execute overhaul operation.
Further, the augmented reality glasses comprise a camera module used for collecting the field image data, a data transmission module used for transmitting the field image data to the edge computing server and the target operation terminal, and a display module used for displaying the fusion interactive video transmitted by the edge computing server.
The target operation terminal further comprises a touch screen and an action capturing module, wherein the touch screen is used for displaying the field image data and receiving the sketching information, the field image data comprise a plurality of field images, the sketching information is obtained by sketching on the field image displayed by the touch screen, and the action capturing module is used for capturing the gesture image data for guiding the field image.
The edge computing server further comprises a data processing module and a bidirectional video fusion module, wherein the data processing module is used for preprocessing the received live image data to obtain target image data and extracting guiding action images from the gesture image data, and the bidirectional video fusion module is used for fusing the guiding action images, the sketching information and the live images in the live image data to generate the fusion interactive video, wherein each guiding action image corresponds to one live image, and the sketching information corresponds to one live image.
Further, the step of preprocessing the received live image data to obtain target image data comprises the steps of performing deblurring processing on each live image in the live image data to obtain a first image, performing strengthening processing on each first image to obtain a second image, performing denoising processing on each second image to obtain a third image, and performing error processing on each third image to obtain the target image data.
The method comprises the steps of acquiring a scene image data, acquiring a guiding action image, acquiring sketching information, acquiring the guiding action image data, acquiring the sketching information, acquiring the guiding action image data, and fusing the guiding action image, the sketching information and the scene image data in the scene image data, wherein the guiding action image data comprises a touch area determined when the sketching information is sketched, matching the touch area with an image area on the scene image corresponding to the sketching information, marking the sketching information on a target image area on the scene image when the area position of the touch area is consistent with the target area position of the target image area, and determining the image area range indicated by the guiding action image and covering the guiding action image in the image area range on the scene image.
According to another aspect of the embodiment of the invention, a maintenance method of the maintenance AR glasses system using the bidirectional video fusion interaction is further provided, and the maintenance method comprises the steps of collecting field image data, obtaining sketching information and gesture image data corresponding to each field image of the field image data, preprocessing the field image data to obtain target image data, extracting guide action images from the gesture image data, and fusing the guide action images, the sketching information and the target field images in the target image data to generate fusion interaction videos, wherein the fusion interaction videos are used for guiding maintenance operation of maintenance objects wearing the augmented reality glasses.
Further, before the gesture image data is acquired, a capturing background area of the motion capturing module is determined, and a target object with a preset color is covered on the capturing background area.
According to another aspect of the embodiment of the present invention, there is also provided a computer readable storage medium, where the computer readable storage medium includes a stored computer program, where when the computer program runs, a device where the computer readable storage medium is controlled to execute any one of the above-mentioned maintenance methods.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device including one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement any one of the above-mentioned methods of servicing.
The overhaul AR glasses system comprises augmented reality glasses, a target operation terminal and an edge calculation server, wherein the augmented reality glasses are used for collecting field image data and transmitting the field image data to the edge calculation server, the target operation terminal is used for obtaining sketching information and gesture image data and transmitting the sketching information and the gesture image data to the edge calculation server, and the edge calculation server is used for fusing the field image data, the sketching information and the gesture image data to generate fused interactive videos and transmitting the fused interactive videos to the augmented reality glasses and the target operation terminal for display.
According to the invention, through overhauling the AR glasses system, the on-site image data collected by the augmented reality glasses and the sketching information and gesture image data generated by the target operation terminal according to the on-site image data can be transmitted to the edge calculation server for processing, so that a fusion interactive video is generated, the fusion interactive video augmented reality glasses and the target operation terminal are displayed, the interactivity before the front end and the rear end can be improved, the on-site accurate overhauling is realized, and the technical problem that the AR glasses in the related art are poor in interactivity and cannot meet the accurate overhauling requirement is solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:
FIG. 1 is a schematic diagram of an alternative bi-directional video fusion interactive service AR eyewear system in accordance with an embodiment of the present invention;
FIG. 2 is a schematic illustration of an alternative fused interactive image in accordance with an embodiment of the present invention;
FIG. 3 is a schematic illustration of an alternative overhaul process based on an overhaul AR eyeglass system, in accordance with an embodiment of the present invention;
FIG. 4 is a flow chart of an alternative method of servicing an AR eyewear system employing two-way video fusion interactions in accordance with an embodiment of the present invention;
fig. 5 is a block diagram of a hardware structure of an electronic device (or mobile device) for an inspection method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The invention provides a relay protection overhaul AR (augmented reality) glasses system with bidirectional video fusion interaction, which can realize bidirectional real-time interaction guidance in a bidirectional video fusion mode and effectively solve the problem that the current AR application only adopts sketching and voice low interaction.
The invention does not need to realize AR modeling and model maintenance, and can directly start to guide on-site operation based on the on-site actual image. Based on the overhauling AR glasses system, an expert can directly guide front-end operation by adopting gestures and voices in the background, interaction is direct and visual, and immersive interaction is realized. In addition, the overhaul AR glasses system can also realize background sketching and marking, and overhaul objects can intuitively see the content of expert guidance on site.
The present invention will be described in detail with reference to the following examples.
Example 1
According to the embodiment of the invention, an embodiment of a bi-directional video fusion interactive overhaul AR glasses system is provided.
FIG. 1 is a schematic diagram of an alternative bi-directional video fusion interactive service AR eyewear system according to an embodiment of the present invention, as shown in FIG. 1, comprising augmented reality eyewear, a target task terminal, and an edge calculation server.
The augmented reality glasses are used for collecting field image data and transmitting the field image data to the edge computing server;
The target operation terminal is used for acquiring the sketching information and the gesture image data and transmitting the sketching information and the gesture image data to the edge calculation server;
The edge computing server is used for fusing the field image data, the sketching information and the gesture image data to generate a fused interactive video, and sending the fused interactive video to the augmented reality glasses and the target operation terminal for display, wherein the fused interactive video is used for guiding maintenance objects wearing the augmented reality glasses to execute maintenance operation.
In the embodiment of the invention, the overhauling object can wear the augmented reality glasses (namely the AR glasses) to collect the field image data (namely the real-time image of the overhauling site), the field image data can be displayed on the target operation terminal (namely the expert operation terminal) in real time, so that a background expert can generate guide information (namely sketching information and gesture image data) on the basis of the field image data through the target operation terminal, and meanwhile, the field image data can be transmitted to the edge calculation server, so that the edge calculation server can process the field image data, and the field image data and the guide information are fused to generate a fused interactive video for guiding the overhauling object to overhaul.
In the embodiment of the invention, a background expert can carry out sketching and marking and gesture guidance according to the real-time image of the overhaul scene displayed in real time through the target operation terminal so as to generate sketching information and gesture image data, and the target operation terminal can transmit the generated sketching information and gesture image data to the edge calculation server so as to facilitate the edge calculation server to fuse the sketching information and gesture image data with the real-time image, thereby realizing real-time interactive guidance of an overhaul object.
In the embodiment of the invention, the edge computing server can fuse the field image data, the sketching information and the gesture image data, so as to generate the fused interactive video (the fused interactive video is used for guiding the overhauling object wearing the augmented reality glasses to execute overhauling operation), and the fused interactive video is sent to the augmented reality glasses and the target operation terminal to be displayed, so that real-time interactive guiding overhauling of the front end and the rear end is realized.
In some alternative embodiments, the service object may be a field staff member, a robot or a mechanical device, etc.
Optionally, the augmented reality glasses comprise a camera module, a data transmission module and a display module, wherein the camera module is used for acquiring field image data, the data transmission module is used for transmitting the field image data to the edge computing server and the target operation terminal, and the display module is used for displaying the fusion interactive video transmitted by the edge computing server.
The augmented reality glasses comprise a camera module, a data transmission module and a display module, wherein the camera module can acquire relay protection operation scene image data (namely, the camera module is used for acquiring the scene image data), the data transmission module can send the scene image data to an edge computing server and a target operation terminal through a 5G or wireless network (namely, the data transmission module is used for transmitting the scene image data to the edge computing server and the target operation terminal), and the display module can display the fused interaction video result (namely, the fused interaction video) (namely, the display module is used for displaying the fused interaction video transmitted by the edge computing server).
The target operation terminal comprises a touch screen and an action capturing module, wherein the touch screen is used for displaying field image data and receiving sketching information, the field image data comprises a plurality of field images, the sketching information is obtained by sketching on the field images displayed by the touch screen, and the action capturing module is used for capturing gesture image data for guiding the field images.
In the embodiment of the invention, the target operation terminal comprises a touch screen and a motion capture module, wherein the touch screen is used for displaying the field image and the fused interactive video result, receiving operation information such as sketching of an expert on the touch screen, and the motion capture module can extract the operation gesture of the expert from the background. Alternatively, to ensure extraction, a green table mat may be used for the background.
Optionally, the edge computing server comprises a data processing module and a bidirectional video fusion module, wherein the data processing module is used for preprocessing received live image data to obtain target image data and extracting guiding action images from gesture image data, and the bidirectional video fusion module is used for fusing the guiding action images, sketching information and the live images in the live image data to generate fusion interaction videos, wherein each guiding action image corresponds to one live image, and sketching information corresponds to one live image.
In the embodiment of the invention, the edge computing server comprises a data processing module and a bidirectional video fusion module, can realize the bidirectional video fusion function and respectively sends the bidirectional video fusion function to the display module of the augmented reality glasses and the touch screen of the target operation terminal.
In this embodiment, because the relay protection operation scene is complex, the data processing module may perform preprocessing on the received live image data to obtain the target image data. In addition, because the environment where the target operation terminal is located is stable, but a clear action image needs to be captured, a green table pad can be adopted to obtain a clear expert guiding action image (namely, the data processing module is used for preprocessing the received on-site image data to obtain target image data and extracting guiding action images from gesture image data). And then, the two-way video fusion module is used for sending the result after the two-way video is overlapped to the display ends on two sides (namely the display module of the augmented reality glasses and the touch screen of the target operation terminal).
In this embodiment, each instruction action image corresponds to one live image, and the sketching information corresponds to one live image.
In this embodiment, the edge computing server adopts different image processing strategies for the complex field end (i.e., the augmented reality glasses) and the stable expert end (i.e., the target operation terminal), so that the computing amount can be effectively reduced. And the edge calculation server can finish the fusion of the two-way video by adopting an edge calculation mode, so that the data transmission quantity can be effectively reduced.
Optionally, preprocessing the received live image data to obtain target image data, wherein the step of performing deblurring processing on each live image in the live image data to obtain first images, performing strengthening processing on each first image to obtain second images, performing denoising processing on each second image to obtain third images, and performing error processing on each third image to obtain target image data.
In the embodiment of the invention, the data processing module of the edge computing server can respectively perform deblurring, weak light strengthening, noise removing and error processing operations on the field image data to obtain a clear image (namely, the field image is subjected to deblurring processing to obtain a first image, the first image is subjected to strengthening processing to obtain a second image, the second image is subjected to denoising processing to obtain a third image, and the third image is subjected to error processing to obtain target image data).
The method comprises the steps of determining a touch area when the sketching information is received, matching the touch area with an image area on a live image corresponding to the sketching information, marking the sketching information on a target image area on the live image when the area position of the touch area is consistent with the target area position of the target image area, determining an image area range indicated by the guiding action image, and covering the guiding action image in the image area range on the live image.
In the embodiment of the invention, the edge calculation server matches the touch area of the touch screen (namely, the touch area when an expert sketches) with the camera acquisition area of the AR glasses (namely, the image area on the field image), and if the area position of the touch area is consistent with the target area position of the target image area, the sketching information is marked on the target image area on the field image so as to realize the fusion of the sketching information and the field image. And the image area range indicated by the guiding action image can be determined, and the guiding action image is covered in the image area range on the live image, for example, the image area range on the image is operated by the gesture of the expert, which is captured by the action capturing module, so that the edge computing server fuses the gesture operation in the image area range on the live image, and a fused interaction video is generated and sent to the display module of the AR glasses and the touch screen of the expert operation terminal for display.
Fig. 2 is a schematic diagram of an alternative fused interactive image, as shown in fig. 2, showing an image of a fused expert's gesture and a live operator's gesture, with both hands from the expert and the live operator, respectively, according to an embodiment of the present invention.
FIG. 3 is a schematic diagram of an alternative overhaul process based on an overhaul AR eyeglass system according to an embodiment of the present invention, as shown in FIG. 3, comprising the following processes:
(1) When the relay protection site staff needs to call an expert through the AR glasses, the interaction guidance of the two parties can be realized after the relay protection site staff is connected;
(2) The data transmission module of the AR glasses transmits the data acquired by the cameras of the AR glasses to the edge calculation server;
(3) The expert operation terminal acquires touch information of an expert on the touch screen and transmits gesture data acquired by the motion capture module to the edge calculation server;
(4) The edge computing server sequentially carries out deblurring treatment, weak light strengthening treatment, noise removing treatment and error treatment on the image data transmitted by the data transmission module, so as to obtain clear video data of the scene;
(5) The edge computing server carries out background clearing on gesture data of the expert operation terminal so as to send the gesture data after the greenware is removed to the bidirectional video fusion module;
(6) The edge computing server matches the touch area of the touch screen with the camera acquisition area of the AR glasses, and sends the sketching operation record on the touch screen to the bidirectional video fusion module;
(7) The bidirectional video fusion module fuses the clear video data, gesture data and the sketching operation record to generate a video fusion result, and sends the video fusion result to the display module of the AR glasses and the touch screen of the expert operation terminal through the data transmission module;
(8) If the remote guidance is finished, entering (9), otherwise returning to (3);
(9) Hanging up the remote guidance interaction, uploading the remote guidance process data to an edge computing server, and ending the expert guidance.
In the embodiment of the invention, AR modeling and model maintenance are not required to be realized, and the on-site operation can be directly guided based on the on-site actual image. Moreover, based on the overhaul AR glasses system, an expert can directly guide front-end operation by adopting gestures and voices in the background, interaction is direct and visual, and immersive interaction is realized. In addition, the overhaul AR glasses system can also realize background sketching and marking, and an overhaul object can intuitively see the content of expert guidance on site.
Example two
According to an embodiment of the present invention, there is provided an embodiment of an inspection method for inspecting an AR glasses system using two-way video fusion interaction, it should be noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and although a logical order is illustrated in the flowchart, in some cases, the steps illustrated or described may be performed in an order different from that herein.
FIG. 4 is a flowchart of an alternative method for servicing an AR eyewear system using two-way video fusion interactions, according to an embodiment of the present invention, as shown in FIG. 4, comprising the steps of:
step S401, acquiring live image data, and acquiring sketching information corresponding to each live image of the live image data and gesture image data.
Step S402, preprocessing the field image data to obtain target image data, and extracting a guiding action image from the gesture image data.
Step S403, fusing the guiding action image, the sketching information and the target site image in the target image data to generate a fused interactive video, wherein the fused interactive video is used for guiding the overhauling object wearing the augmented reality glasses to carry out overhauling operation.
Through the steps, the field image data can be acquired, the sketching information and the gesture image data corresponding to each field image of the field image data are acquired, the field image data are preprocessed to obtain the target image data, the guiding action image is extracted from the gesture image data, and the guiding action image and the sketching information are fused with the target field image in the target image data to generate the fused interactive video. In the embodiment of the invention, the acquired field image data can be preprocessed, the guiding action image is extracted from the acquired gesture image data, and then the guiding action image, the sketching information and the target field image are fused to generate the fusion interaction video, so that the real-time interaction between an expert and an overhauling object is realized, the interactivity before the front end and the rear end can be improved, the on-site accurate overhauling is realized, and the technical problems that the AR glasses in the related technology are poor in interactivity and cannot meet the accurate overhauling requirement are solved.
In the embodiment of the invention, the overhauling object can wear the augmented reality glasses to collect the field image data (namely, the real-time image of the overhauling scene), the field image data can be displayed on the target operation terminal in real time, so that a background expert can generate guide information (namely, sketching information and gesture image data) on the basis of the field image data through the target operation terminal, and meanwhile, the field image data can be transmitted to the edge computing server, so that the edge computing server can process the field image data, and the field image data and the guide information are fused to generate a fused interactive video for guiding the overhauling object to overhaul.
In the embodiment of the invention, the preprocessing of the field image data comprises blurring processing, weak light strengthening processing, noise removing processing, error processing and the like, so that a clear field image can be obtained.
Optionally, before acquiring the gesture image data, determining a capturing background area of the motion capturing module, and covering a target object with a preset color on the capturing background area.
In the embodiment of the invention, the capturing background area of the motion capturing module can be determined first, and a target object (for example, a table mat, a curtain, etc.) with a preset color (i.e., green) is covered on the capturing background area, so as to obtain a clear guiding motion image.
In the embodiment of the invention, an expert can directly guide front-end operation by adopting gestures, sketching marks and voices in the background, and can realize immersive interaction. In addition, the on-site operation can be directly guided based on the on-site actual image, AR modeling and model maintenance are not needed, and the workload is effectively reduced.
The invention also provides a computer program product which is suitable for executing a program initialized with the following method steps when being executed on a data processing device, wherein the program is used for collecting field image data, acquiring sketching information and gesture image data corresponding to each field image of the field image data, preprocessing the field image data to obtain target image data, extracting guiding action images from the gesture image data, fusing the guiding action images and the sketching information with the target field images in the target image data, and generating a fused interactive video.
According to another aspect of the embodiment of the present invention, there is also provided a computer readable storage medium, including a stored computer program, where the computer program when executed controls a device in which the computer readable storage medium is located to perform the above-described maintenance method.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device including one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the above-described inspection method.
Fig. 5 is a block diagram of a hardware structure of an electronic device (or mobile device) for an inspection method according to an embodiment of the present invention. As shown in fig. 5, the electronic device may include one or more processors (e.g., processor 502a, processor 502b, etc., of fig. 5, processor 502n, etc., which may include, but are not limited to, a microprocessor MCU, programmable logic device FPGA, etc.), memory 504 for storing data. Among other things, a display, an input/output interface (I/O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the I/O interface), a network interface, a keyboard, a power supply, and/or a camera may be included. It will be appreciated by those of ordinary skill in the art that the configuration shown in fig. 5 is merely illustrative and is not intended to limit the configuration of the electronic device described above. For example, the electronic device may also include more or fewer components than shown in FIG. 5, or have a different configuration than shown in FIG. 5.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
The embodiments or examples of the present disclosure are not intended to be exhaustive, but rather are merely illustrative of some of the embodiments or examples, and are not intended to limit the scope of the disclosure in any way. Each step in a certain implementation manner or embodiment may be implemented as an independent embodiment, and the steps may be arbitrarily combined, for example, a scheme after removing part of the steps in a certain implementation manner or embodiment may be implemented as an independent embodiment, and the sequence of the steps in a certain implementation manner or embodiment may be arbitrarily exchanged, further, an optional manner or optional embodiment in a certain implementation manner or embodiment may be arbitrarily combined, further, the implementation manner or embodiment may be arbitrarily combined, for example, part or all of the steps of different implementation manners or embodiments may be arbitrarily combined, and a certain implementation manner or embodiment may be arbitrarily combined with an optional manner or optional embodiment of other implementation manners or embodiments.
In the foregoing embodiments of the present invention, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
In the several embodiments provided in the present invention, it should be understood that the disclosed technology may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, for example, may be a logic function division, and may be implemented in another manner, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. The storage medium includes a U disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, etc. which can store the program code.
The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.
Claims (10)
1. An interactive maintenance AR glasses system of two-way video fusion, its characterized in that, maintenance AR glasses system includes:
the augmented reality glasses are used for collecting field image data and transmitting the field image data to the edge computing server;
the target operation terminal is used for acquiring sketching information and gesture image data and transmitting the sketching information and the gesture image data to the edge calculation server;
The edge computing server is used for fusing the field image data, the sketching information and the gesture image data to generate a fused interactive video, and sending the fused interactive video to the augmented reality glasses and the target operation terminal for display, wherein the fused interactive video is used for guiding an overhauling object wearing the augmented reality glasses to execute overhauling operation.
2. The service AR glasses system of claim 1 wherein the augmented reality glasses comprise:
The camera module is used for collecting the field image data;
the data transmission module is used for transmitting the field image data to the edge calculation server and the target operation terminal;
and the display module is used for displaying the fusion interactive video transmitted by the edge computing server.
3. The service AR glasses system according to claim 1, wherein the target work terminal comprises:
The touch screen is used for displaying the field image data and receiving the sketching information, wherein the field image data comprises a plurality of field images, and the sketching information is obtained by sketching on the field images displayed by the touch screen;
And the motion capture module is used for capturing the gesture image data for guiding the live image.
4. The service AR glasses system of claim 1 wherein the edge calculation server comprises:
The data processing module is used for preprocessing the received on-site image data to obtain target image data and extracting a guiding action image from the gesture image data;
and the bidirectional video fusion module is used for fusing the guiding action image, the sketching information and the field image in the field image data to generate the fusion interaction video, wherein each guiding action image corresponds to one field image, and the sketching information corresponds to one field image.
5. The service AR glasses system according to claim 4, wherein the step of preprocessing the received live image data to obtain target image data comprises:
Performing defuzzification processing on each live image in the live image data to obtain a first image;
performing reinforcement treatment on each first image to obtain a second image;
denoising each second image to obtain a third image;
and carrying out error processing on each third image to obtain the target image data.
6. The service AR glasses system according to claim 4, wherein the step of fusing the guide action image, the sketching information and a live image in the live image data comprises:
determining a touch area when the sketching information is sketched, and matching the touch area with an image area on the field image corresponding to the sketching information;
Marking the sketching information on a target image area on the field image under the condition that the area position of the touch area is consistent with the target area position of the target image area;
and determining an image area range indicated by the guide action image, and covering the guide action image in the image area range on the live image.
7. A method of servicing an AR eyewear system using the two-way video fusion interaction of claim 1, comprising:
acquiring live image data, and acquiring sketching information and gesture image data corresponding to each live image of the live image data;
preprocessing the field image data to obtain target image data, and extracting a guiding action image from the gesture image data;
and fusing the guiding action image, the sketching information and the target site image in the target image data to generate a fused interactive video, wherein the fused interactive video is used for guiding an overhaul object wearing the augmented reality glasses to carry out overhaul operation.
8. The method of claim 7, wherein prior to acquiring the gesture image data, determining a capture background area of a motion capture module and overlaying a target object of a preset color on the capture background area.
9. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored computer program, wherein the computer program, when run, controls a device in which the computer readable storage medium is located to perform the service method according to any one of claims 7 to 8.
10. An electronic device comprising one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of servicing of any of claims 7-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411616639.2A CN119152405A (en) | 2024-11-13 | 2024-11-13 | Two-way video fusion interactive maintenance AR glasses system and maintenance method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411616639.2A CN119152405A (en) | 2024-11-13 | 2024-11-13 | Two-way video fusion interactive maintenance AR glasses system and maintenance method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN119152405A true CN119152405A (en) | 2024-12-17 |
Family
ID=93812940
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411616639.2A Pending CN119152405A (en) | 2024-11-13 | 2024-11-13 | Two-way video fusion interactive maintenance AR glasses system and maintenance method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN119152405A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106339094A (en) * | 2016-09-05 | 2017-01-18 | 山东万腾电子科技有限公司 | Interactive remote expert cooperation maintenance system and method based on augmented reality technology |
| CN108509026A (en) * | 2018-02-06 | 2018-09-07 | 西安电子科技大学 | Tele-Maintenance Support System and method based on enhancing interactive mode |
| CN114119443A (en) * | 2021-11-28 | 2022-03-01 | 特斯联科技集团有限公司 | Image fusion system based on multispectral camera |
| CN115469745A (en) * | 2022-09-09 | 2022-12-13 | 南方电网调峰调频发电有限公司信息通信分公司 | A computer room equipment fault handling method and system based on AR technology |
| CN117135146A (en) * | 2023-08-16 | 2023-11-28 | 国网上海市电力公司 | A remote guidance system based on AR technology |
-
2024
- 2024-11-13 CN CN202411616639.2A patent/CN119152405A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106339094A (en) * | 2016-09-05 | 2017-01-18 | 山东万腾电子科技有限公司 | Interactive remote expert cooperation maintenance system and method based on augmented reality technology |
| CN108509026A (en) * | 2018-02-06 | 2018-09-07 | 西安电子科技大学 | Tele-Maintenance Support System and method based on enhancing interactive mode |
| CN114119443A (en) * | 2021-11-28 | 2022-03-01 | 特斯联科技集团有限公司 | Image fusion system based on multispectral camera |
| CN115469745A (en) * | 2022-09-09 | 2022-12-13 | 南方电网调峰调频发电有限公司信息通信分公司 | A computer room equipment fault handling method and system based on AR technology |
| CN117135146A (en) * | 2023-08-16 | 2023-11-28 | 国网上海市电力公司 | A remote guidance system based on AR technology |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108986189B (en) | Method and system for capturing and live broadcasting of real-time multi-person actions based on three-dimensional animation | |
| CN111526118B (en) | Remote operation guiding system and method based on mixed reality | |
| CN110599603A (en) | Mechanical equipment visual interaction and equipment state monitoring system and method based on augmented reality | |
| KR101195446B1 (en) | A portable terminal providing maintenance guide based on augmented reality and the method of providing maintenance guide using the same | |
| EP3872766A2 (en) | Method and device for processing image, related electronic device and storage medium | |
| CN105759960A (en) | Augmented reality remote guidance method and system in combination with 3D camera | |
| CN106374382A (en) | Power grid equipment troubleshooting method and system | |
| CN105429989A (en) | Simulative tourism method and system for virtual reality equipment | |
| CN107845143A (en) | A kind of finishing outpost display system based on virtual reality | |
| CN103455657A (en) | Kinect based field operation simulation method and Kinect based field operation simulation system | |
| CN113407035A (en) | Communication facility maintenance operation system based on MR mixed reality technology | |
| CN105630175A (en) | Virtual reality equipment and virtual reality image generation method | |
| CN110751734B (en) | Mixed reality assistant system suitable for job site | |
| CN115469745A (en) | A computer room equipment fault handling method and system based on AR technology | |
| CN119152405A (en) | Two-way video fusion interactive maintenance AR glasses system and maintenance method | |
| CN114187392A (en) | Virtual even image generation method and device and electronic equipment | |
| CN206021226U (en) | A kind of camera recognizes laser interactive system | |
| CN113110733A (en) | Virtual field interaction method and system based on remote duplex | |
| CN118354033A (en) | Substation maintenance guidance video generation method, device and processor | |
| CN111510679A (en) | Control method of intelligent safety helmet, intelligent safety helmet equipment and readable storage medium | |
| CN116842591A (en) | XR anti-disclosure visualization system and method based on digital twin technology | |
| CN113068003A (en) | Data display method and device, intelligent glasses, electronic equipment and storage medium | |
| CN101727181A (en) | Method for realizing computer input and output through 3D technology | |
| CN119363958A (en) | Image stabilization method and device based on remote guidance of AR glasses and electronic equipment | |
| CN110211238A (en) | Display methods, device, system, storage medium and the processor of mixed reality |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |