CN110928418A - A kind of MR-based aviation cable auxiliary assembly method and system - Google Patents
A kind of MR-based aviation cable auxiliary assembly method and system Download PDFInfo
- Publication number
- CN110928418A CN110928418A CN201911266672.6A CN201911266672A CN110928418A CN 110928418 A CN110928418 A CN 110928418A CN 201911266672 A CN201911266672 A CN 201911266672A CN 110928418 A CN110928418 A CN 110928418A
- Authority
- CN
- China
- Prior art keywords
- assembly
- information
- virtual
- real
- glasses
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- User Interface Of Digital Computer (AREA)
- Processing Or Creating Images (AREA)
Abstract
本发明公开了一种基于MR的航空线缆辅助装配方法及系统，通过MR眼镜实施装配工艺流程，通过增强现实技术实现待装配零部件的三维模型叠加在真实的装配场景中，并通过双目立体、虚实动画的方式展示给操作工人，通过UI界面展示相应的装配信息辅助工人高效的完成装配任务。增强现实技术就是通过将待装配零部件的三维模型通过MR眼镜以及虚实定位功能，将虚拟的模型按照真实的装配形式，叠加到真实的装配场景中，从而使工人能够更加直观的感受到待装配零部件的装配形式和装配位置。本发明能实现辅助工人进行飞机线缆的装配，提高飞机线缆装配效率和质量。
The invention discloses an MR-based auxiliary assembly method and system for aviation cables. The assembly process is implemented through MR glasses, and the three-dimensional model of the parts to be assembled is superimposed on the real assembly scene through the augmented reality technology. The three-dimensional, virtual and real animations are displayed to the operators, and the corresponding assembly information is displayed through the UI interface to assist the workers to efficiently complete the assembly tasks. Augmented reality technology is to superimpose the virtual model into the real assembly scene according to the real assembly form by passing the three-dimensional model of the parts to be assembled through the MR glasses and the virtual-real positioning function, so that the workers can feel the assembly to be assembled more intuitively. Assembly form and assembly location of components. The invention can assist workers to assemble the aircraft cable, and improve the assembly efficiency and quality of the aircraft cable.
Description
Technical Field
The invention relates to the field of augmented reality and wearable equipment-assisted assembly, in particular to an aviation wire assisted assembly method and system based on a mixed reality technology.
Background
The assembly operation of aerospace products depends on manpower in a large number, so that the problems of high requirements on experience and proficiency of workers, low assembly speed and the like exist, the improvement of the assembly efficiency is severely restricted, and the rapid production requirement of the current aerospace products is difficult to adapt.
At present, in the process of assembling an aircraft cable, the problems of poor legibility of the assembling process and the like by using the traditional paper process file and CAD industrial software still exist, and the situations of assembling errors and reworking caused by wrong understanding or misoperation of workers are difficult to avoid. Therefore, the assembly period of the aviation cable is long, the labor cost is increased, the development period of the product is long, the manufacturing cost is high, and the like.
The appearance of wearable equipment provides a new idea for people to reduce the problem of assembly quality by using an assembly auxiliary means, and the assembly guide is provided for workers in real time through the wearable equipment, so that the informatization level of the assembly process can be improved, and the realization of efficient human-computer interaction in the whole production is facilitated. The supplementary guide of assembly based on wearable equipment can effectively liberate workman's both hands, promotes assembly efficiency to reduce the degree of dependence of assembly operation to the skilled level of workman, effectively prevent the emergence of assembly mistake.
The augmented reality technology is a technology for mixing a virtual scene and a real scene, and with the continuous improvement of the computing capability of wearable equipment, an application platform of the augmented reality technology is gradually transferred to the wearable equipment from a traditional computer. The method has the advantages that the assembly assistance is carried out by using the augmented reality technology, the virtual three-dimensional model of the assembly object can be directly projected at the position to be assembled, compared with the traditional assembly assistance means, the method is more intuitive and easy to understand, and the possibility of errors in the assembly can be reduced.
Currently, many aerospace manufacturers at home and abroad have begun to try to apply wearable technology and augmented reality technology to the assembly of aerospace products. Realizing the popularization of wearable technology and enhanced display technology in enterprise production is an important direction for the development of the manufacturing industry today. In particular for aircraft cable assembly, the relevant assembly assistance techniques are still not mature. Because aircraft cable has the characteristics such as the structure is complicated, a great variety, easily cause workman's judgement error in the assembling process, lead to the appearance of assembling problems such as neglected loading, misloading or dress reversal then. And the augmented reality technology based on wearable equipment can project a corresponding three-dimensional model at the position where the cable is to be assembled in real time, help workers to understand the assembly requirement intuitively, and reduce the possibility of worker judgment errors, so that the visual guide of the aircraft cable assembly by using the wearable equipment and the augmented reality technology has important significance for improving the assembly quality and the assembly efficiency of the aircraft cable.
Disclosure of Invention
The invention aims to provide an aviation cable auxiliary assembly method based on MR, and the other aim of the invention is to provide an aviation cable auxiliary assembly system based on MR, and the method and the system can help workers quickly and accurately understand assembly requirements under complex environments, so that assembly efficiency is improved, and assembly errors are reduced.
In order to achieve the purpose, the invention provides an aviation cable auxiliary assembly method based on MR, which specifically comprises the following steps:
s1: according to the current assembly task, importing a corresponding assembly process file by using the MR glasses and a network communication module at a computer end;
s2: scanning the marker map in the assembly scene through a virtual and real positioning module at the MR spectacle end, and realizing space positioning in the assembly scene by means of the marker map;
s3: controlling the wearable equipment by using gestures or voice according to the assembly guiding information of the MR glasses end through man-machine interaction operation, and sending an operation instruction;
s4: after the MR glasses end receives the operation instruction, the virtual-real display and virtual-real positioning module is utilized to generate assembly operation information in the corresponding step in the assembly operation, and the assembly operation information is displayed to an operator through binocular stereo, a UI interface and virtual-real animation;
s5: the operation worker carries out corresponding assembly operation according to corresponding assembly prompt information of the MR spectacle end;
s6: after the current assembly steps are completed, the steps of S3-S5 are repeated until all assembly tasks are completed.
Further, in S1, the paper assembly process is recorded into the MR spectacle end through an XML file, and the guidance information in the process file is product information, process information, and process step information.
Further, the product information mainly comprises a product name and a product number; the process information comprises tool information, material information, process numbers and process names required for completing the process; the step information includes guide information for the step, including step number, step name, text description information of the step, picture information of the step, and three-dimensional model information of the step.
Further, in S2, providing auxiliary process information in the form of characters, pictures, virtual animation and real animation for an assembler, and projecting the three-dimensional model of the component onto a position to be assembled, the specific implementation steps include:
1) selecting a product to be assembled;
2) reading the assembly auxiliary file, and loading assembly auxiliary information: text auxiliary information, picture auxiliary information, model auxiliary information;
3) selecting a current process step;
4) displaying information of characters and pictures;
in an assembly scene, a world coordinate system, a camera coordinate system, a display screen coordinate system and a virtual camera coordinate system are unified through the identification graph, so that space positioning is realized.
Further, the virtual-real display in S4 specifically includes:
1) establishing a three-dimensional model of the assembled parts by using three-dimensional modeling software, and simplifying the model so as to meet the requirements of loading and rendering the model of the AR glasses and process the shielding relation between the augmented reality information and the real physical scene;
2) establishing a data file interface of the assembly element model, reading the three-dimensional coordinate information of the assembly element frame by frame from the data file, and driving the position of the corresponding three-dimensional model in a Unity3D graphic engine; amplifying a three-dimensional model in a virtual scene in a set proportion to meet the watching requirement;
3) determining the position and the posture of a virtual viewpoint in a virtual scene according to viewpoint position posture information output by the AR glasses positioning module, and associating the position of the assembly element in the virtual scene with the position information of the actual scene to ensure that the assembly element can be matched when being superposed;
4) generating coordinates of binocular viewpoints according to the viewpoint position in the current virtual scene and the position relation with the observed assembly part, and rendering and outputting images of the virtual model according to the current binocular coordinates respectively;
5) and displaying the model information and the character description information of the assembled parts through the animation display of the assembled model and the UI dialog box prompt information.
An auxiliary assembly system of the MR-based aviation cable for implementing the method comprises:
computer-side assembly of an auxiliary subsystem: the network communication module is used for carrying out network communication of configuration files and interactive instructions on the MR glasses;
MR glasses end assembly auxiliary subsystem: the method is used for virtual and real positioning, man-machine interaction and virtual and real display;
the virtual and real positioning realizes positioning of an assembly scene through SLAM positioning and identification positioning, man-machine interaction realizes interaction of workers and an auxiliary assembly system through functions of gestures, voice and staring, virtual and real display realizes assembly operation guidance through functions of binocular stereo, UI display and animation display, so that a three-dimensional model of the part to be assembled is superposed in a real assembly scene, and the visual assembly form and assembly position of the part to be assembled are realized.
Further, the virtual and real positioning: the method is used for scanning the identification map of the appointed assembly operation through the MR glasses before the assembly operation, and acquiring the coordinate positioning of an assembly operation scene, wherein the SLAM positioning is performed through the depth camera of the MR glasses and the real-time scene without the need of identification positioning after the identification positioning.
Further, the virtual and real displays: the binocular stereo system realizes three-dimensional display of parts to be assembled through MR glasses, UI display realizes that a UI interface assisting operation of workers and prompting the workers is provided in an assembly scene, and animation display realizes that virtual-real linkage is realized in a real assembly scene by utilizing three-dimensional animation display and the required assembly action is visually displayed for the workers.
The system further comprises a worker end and an assembly site end, wherein the worker end is used for executing an assembly task by utilizing the MR glasses according to an assembly operation sequence, and needs to import an assembly process file, scan a marking diagram, send an assembly interaction instruction to the MR glasses end and perform assembly operation through assembly guide information of the MR glasses end; the assembling site end is a real scene in an assembling task, and a worker can display parts to be assembled in the real assembling scene according to the virtual and real display function in the MR glasses and guide the worker to intuitively perform assembling operation.
Further, the assembly process file of the worker end is the assembly process file processed by the computer end assembly auxiliary subsystem, the interactive instruction comprises interactive actions such as voice, gestures and staring involved in assembly operation of the worker, and the assembly guidance information is information presented by functions such as binocular stereo, UI display and animation display provided by a virtual and real display function of the MR glasses end.
Compared with the current technology, the MR-based aviation cable auxiliary assembly method and system disclosed by the invention have the following advantages.
(1) The method provided by the invention adopts a method combining wearable equipment and an augmented reality technology, provides a new convenient and visual assembly idea for aircraft cable assembly, and can effectively solve the problem of a traditional operation mode which seriously depends on the experience and proficiency of workers in the aircraft cable assembly process.
(2) The aviation cable auxiliary assembly method based on the MR can realize that assembly operation guidance can be visually provided for operators by superposing virtual parts to be assembled in a real assembly scene and displaying through binocular stereo, UI interfaces and virtual-real animations.
(3) The aviation cable auxiliary assembly method based on the MR changes the original assembly operation mode of the traditional paper assembly process file and the experience of workers, and greatly improves the assembly operation efficiency and the assembly quality of the aircraft cable.
Drawings
FIG. 1 is a system framework of the present invention;
FIG. 2 is a flow chart of establishing aircraft cable assembly process information;
FIG. 3 is an E-R diagram of product information, process information, and process step information;
FIG. 4 is a schematic diagram of a position relationship of an augmented reality system;
FIG. 5 is a communication diagram between the MR smart glasses and the PC terminal;
FIG. 6 is a flow chart of occlusion processing based on model reconstruction.
Detailed Description
The present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
For ease of description, spatially relative terms, such as "upper," "lower," "left," "right," and the like, may be used herein to describe one element or feature's relationship to another element or feature as illustrated in the figures. It will be understood that the spatial terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" other elements or features would then be oriented "above" the other elements or features. Thus, the exemplary term "lower" can encompass both an upper and a lower orientation. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
As shown in fig. 1 to 6, the invention provides an aviation cable auxiliary assembly method and system based on MR, the essence of the method is that an assembly process flow is implemented through MR glasses, a three-dimensional model of a part to be assembled is superimposed in a real assembly scene through an enhanced display technology, the three-dimensional model is displayed to an operator in a binocular stereo and virtual-real animation mode, and corresponding assembly information is displayed through a UI interface to assist the operator to efficiently complete an assembly task. The augmented reality technology is that a virtual model is superposed to a real assembly scene according to a real assembly form by a three-dimensional model of the part to be assembled through MR glasses and a virtual-real positioning function, so that a worker can feel the assembly form, the assembly position and the like of the part to be assembled more intuitively.
The invention provides an aviation cable auxiliary assembly system based on MR, which comprises: the system comprises a computer end assembly auxiliary subsystem, an MR glasses end assembly auxiliary subsystem, a worker end and an assembly field.
The computer-side assembly auxiliary subsystem is used for configuration files and network communication, wherein the configuration files are used for acquiring process file information, product process information and tool material information required by assembly, and the network communication is used for performing functions of instruction processing of assembly site workers, coordinate information processing of operation scenes, interactive information processing of external systems and the like.
The MR glasses end assembly auxiliary subsystem is used for virtual and real positioning, man-machine interaction and virtual and real display, wherein the virtual and real positioning can realize positioning of an assembly scene through an SLAM mode and an identification mode, the MR glasses are used for scanning an identification diagram of appointed assembly operation before the assembly operation to acquire coordinate positioning of the assembly operation scene, and the SLAM positioning is that after the identification positioning is carried out, the identification positioning is not needed any more, and the real-time positioning can be carried out through a depth camera and a real-time scene of the MR glasses. The human-computer interaction can realize the interaction between a worker and an auxiliary assembly system through functions of gestures, voice, staring and the like, the virtual-real display can realize assembly operation guidance through functions of binocular stereo, UI display, animation display and the like, wherein the binocular stereo can realize the three-dimensional display of parts to be assembled through MR glasses, the UI display can realize the UI interface which assists the operation of the worker and prompts the worker in an assembly scene, the animation display can realize the virtual-real linkage in a real assembly scene by utilizing the three-dimensional animation display, and the required assembly action can be visually displayed for the worker.
And the worker end is used for executing the assembly work task by utilizing the MR glasses according to the assembly work sequence, and the assembly work is carried out by importing an assembly process file, scanning a marking diagram, sending an assembly interaction instruction to the MR glasses end and through the assembly guide information of the MR glasses end. The assembly process file is processed by a computer-end assembly auxiliary subsystem, and the interactive instruction comprises interactive actions such as voice, gestures, staring and the like related to a worker in the assembly operation; the assembly guidance information is information presented by functions of binocular stereo, UI display, animation display and the like provided by the virtual and real display function of the MR glasses.
And the assembly site end is a real scene in an assembly task, a worker can display the parts to be assembled in the real assembly scene according to the virtual and real display function in the MR glasses, and can intuitively guide the worker to perform the assembly operation by utilizing the binocular stereo, UI interface and virtual and real animation functions.
The invention relates to an aviation cable auxiliary assembly method based on MR, which adopts the technical scheme that the method comprises the following steps:
s1: an assembly process file is established in the computer-end auxiliary assembly subsystem according to the assembly task, an assembly process file in an xml format is generated and is sent to the MR glasses-end auxiliary assembly subsystem through the network communication module;
s2: in order to correspond to the process files of the assembly task, workers need to import corresponding assembly process files by using MR glasses according to the assembly task, and assembly operation guidance is carried out;
s3: in order to determine the spatial position of the component to be assembled in the subsequent step, a virtual and real positioning module at the MR spectacle end is used for scanning a marking image in an assembly scene, and spatial positioning is realized in the assembly scene by means of the marking image;
s4: a worker makes corresponding human-computer interaction operation, controls the wearable equipment by using gestures or voice according to the assembly guiding information of the MR glasses end, and sends an operation instruction; the MR glasses end receives a worker operation instruction, returns the currently required assembly information and assembly parts after the instruction processing, generates assembly operation information in the corresponding step in the assembly operation by using a virtual-real display and virtual-real positioning module, and displays the assembly operation information to an operator through binocular stereo, a UI interface and virtual-real animation;
s5: an operator performs assembly operation according to the corresponding assembly prompt information of the MR spectacle end, the prompted assembly process information and the corresponding assembly part information;
s6: after the current assembly steps are completed, the steps of S4-S5 are repeated until all assembly tasks are completed.
The specific implementation method is described in detail below with reference to fig. 1 to 6.
S1: the method is based on MR aviation cable auxiliary assembly, so that an aviation cable assembly process must be established, the traditional operation mode is that a paper assembly process is not completely suitable for a wearable device auxiliary assembly system, and therefore the paper assembly process is recorded into the MR aviation cable auxiliary assembly system through an XML (extensive markup language) file in a computer-end auxiliary assembly subsystem (shown in figure 1).
S1.1: for this case, the information contained in the conventional assembly process file is refined, classified and organized herein, and its representation model is built. The information which has great guiding significance to the assembly process through inspection mainly comprises detailed descriptions of the assembly auxiliary tools, materials and assembly steps. The two parts in the assembly AO can be abstracted into three types of information, namely product information, process information and process step information, and a strict hierarchical relationship exists between the two parts. FIG. 3 is an E-R diagram of product information, process information, and process step information.
As shown in fig. 3, the product information mainly includes a product name and a product number; the process information mainly comprises tool information, material information, process numbers and process names required for completing the process; the step information mainly includes guide information for the step, including step number, step name, text description information of the step, picture information of the step, and three-dimensional model information of the step.
Further, the relationship between the product information and the process information is a "one-to-many" relationship, that is, a product of one model includes a plurality of assembly processes. Similarly, the relationship between the process information and the process step information is also a "one-to-many" relationship, that is, one assembly process includes a plurality of assembly process steps.
Aiming at the characteristics of the assembly process information, the product information is abstractly represented by using the attributes of 'productID' and 'productName'; abstract representation is carried out on the process information by using attributes of 'processID', 'processName', 'Tools' and 'Material'; the abstract representation is made using the "stepName", "textInf", "picInf", "ARmodInf" attributes.
In view of the 'one-to-many' inclusion relationship among product information, process information and process step information, the method for realizing the airplane assembly process information management by using the XML file is provided.
S1.2: as shown in fig. 2, the process for establishing the aircraft cable assembly process information may be divided into two main stages, where the first stage is to fill the assembly process information on a computer by a craft worker according to the aircraft cable assembly process, and the second stage is to generate an assembly process information file based on XML according to the filled assembly process information, and the obtained process file based on the XML format may provide support for the MR glasses end assembly auxiliary subsystem.
S2: in order to correspond to the process files of the assembly task, a worker needs to import the corresponding assembly process files by using the MR glasses according to the assembly task, and then assembly operation guidance is performed.
Visual assembly guidance is realized by using the augmented reality technology of the MR glasses end assembly auxiliary subsystem, auxiliary process information in the forms of characters, pictures and virtual-real animations is provided for assembly workers, and the information can be obtained through the assembly auxiliary XML file generated in the step S1 as follows:
through the augmented reality technology with spare part three-dimensional model projection to waiting to assemble the position on, can provide directly perceived and accurate assembly guide for the workman in real time, concrete implementation steps include:
1) and selecting a product to be assembled.
2) The assembly auxiliary file is read and the assembly auxiliary information (character auxiliary information, picture auxiliary information, model auxiliary information) is loaded.
3) And selecting the current process step.
4) And displaying the information of the characters and the pictures.
In order to determine the spatial position of the component to be assembled in the subsequent step, the virtual and real positioning module at the MR spectacle end is used for scanning the marker map in the assembly scene, and as shown in FIG. 4, the world coordinate system, the camera coordinate system, the coordinate system of the display screen and the virtual camera coordinate system are unified through the marker map in the assembly scene, so that spatial positioning is realized;
assuming that a world coordinate system is denoted by W, a camera coordinate system is denoted by C, a coordinate system of a display screen of the optical see-through head-mounted display is denoted by S, and a human eye and the screen together constitute a virtual camera, the coordinate system is denoted by V. If the coordinate of any point P in the space is Pw, Pc under the camera coordinate system and Ps on the screen, then the equations (1) and (2) are obtained,
Pc＝[Rwc|Twc；0 0 0 1]Pw(1)
Ps＝K[Rcv|Tcv]Pc＝GPc(2)
wherein, Pw, Pc are homogeneous coordinates of the three-dimensional position of the point in the world coordinate system, and therefore are all four-dimensional vectors. Ps is the homogeneous coordinate of the two-dimensional position of the point in the screen coordinate system, and is therefore a three-dimensional vector. R is an optional matrix of 3 x 3, T is a translation column vector of 3 x 1, and K is an internal reference matrix of the virtual camera including human eyes, and is a 3 x 3 matrix.
Let Ps ═ u v 1] T, Pc ═ xc zc 1] T, G ═ G11G 12G 13G 14; g21 g22 g23 g 24; g31 g32 g33 g34], then:
the matrix G is a mapping relationship from the camera coordinate system Pc to the screen coordinate system Ps.
The same approach can result in the F, K, M matrix shown in fig. 4.
S3: a worker makes corresponding human-computer interaction operation, controls the wearable equipment by using gestures or voice according to the assembly guiding information of the MR glasses end, and sends an operation finger;
s4: the MR glasses end receives a worker operation instruction, returns the currently required assembly information and assembly parts after the instruction processing, generates assembly operation information in the corresponding step in the assembly operation by using a virtual-real display and virtual-real positioning module, and displays the assembly operation information to an operator through binocular stereo, a UI interface and virtual-real animation;
s4.1: workers make corresponding human-computer interaction operation and send operation instructions, and the method is based on three modes of gazing, gestures and voice developed by an interaction mode of MR glasses;
(1) staring at
The MR glasses use the head movement factor as the main control mode of the Gaze interaction, the input form of the user instruction is named as 'size input', and the main marker of the Gaze function of the MR glasses is a pointer. The pointer is a reference object for helping the user accurately know the gaze point, and functions like a cursor in a conventional computer. The pointer has various forms, the default Cursor provided in the HoloToolkit is used in the system, the form of the Cursor is a light purple solid light spot under a normal state, and the form of the Cursor can become a blue purple hollow aperture after an object is selected.
(2) Gesture
The MR glasses support recognized interactive gestures in three main categories: pinch gestures, tap gestures, blossom gestures. The pinch gesture and the click gesture are mainly used in interactive cooperation with the gazing, and are mainly used for placing and starting applications, triggering response events of various controls in the UI, and then achieving the operation of the user on the MR glasses, and the blooming gesture is mainly used for exiting the current application and returning to a starting menu in the MR glasses.
TABLE 1 gesture interaction of MR glasses
(3) Speech sound
The user speaks corresponding voice information according to the prompt on the current MR glasses, the MR glasses capture audio signals in real time through the microphone and conduct processing and analysis to complete semantic recognition, then the code segment corresponding to the command is called, and the processing result is displayed to the user through the holographic image.
In order to realize the voice interaction function of the aircraft cable assembly auxiliary system, corresponding keywords are required to be specified in advance as trigger signals for command execution. Because the MR glasses study the fashionable and non-published Chinese voice recognition function in the invention, English words are selected as the voice recognition keywords. In this document, 13 operation keys are defined according to the actual requirements of the assembly site, as shown in table 2.
S4.2: after the MR glasses end receives the operation instruction, the virtual-real display and virtual-real positioning module is utilized to generate assembly operation information in the corresponding step in the assembly operation, and the assembly operation information is displayed to an operator through binocular stereo, a UI interface and virtual-real animation;
the method mainly realizes the interaction and display of the assembly operation guidance information according to the network communication function and the virtual and real display function.
TABLE 2 Voice interaction keywords and their execution operations
(1) Network communication function
The MR glasses and the computer end are communicated by Socket. Fig. 5 shows a main mechanism for realizing communication between the glasses and the computer based on Socket, and the transmission of data such as instruction information, position information, text/picture/video, etc. between the glasses and the computer is realized by using a network communication function. The experience of supplementary assembly of MR glasses is promoted.
(2) Virtual and real display function
1) Establishing three-dimensional models such as assembly parts by using three-dimensional modeling software such as 3dmax, appropriately simplifying the models to meet the requirements of model loading and rendering of AR glasses, and processing the shielding relation between augmented reality information and a real physical scene;
the AR virtual-real fusion display technology is also called occlusion processing technology, and in early augmented reality, a real scene appears as a background of a virtual object, so that no matter how an occlusion object in the real scene moves, the occlusion object appears behind the virtual object. With the continuous development of related technologies (such as graphic image technology, three-dimensional rendering technology, etc.), researchers have begun to solve the problem of virtual and real occlusion from various angles. Generally these methods are divided into depth-based computation and model-based reconstruction.
In the application scenario of the project, virtual-real fusion display of the AR needs to be realized in the real-time SLAM process. Because the scene is a known scene, the occlusion processing is carried out by adopting a method based on model reconstruction, thereby obtaining the best virtual-real fusion effect. The specific flow is shown in FIG. 6
In the invention, the Unity3D engine is adopted as an AR virtual scene rendering tool, so that the specific virtual and real occlusion processing is selected to be processed on the Unity3D scene editing tool, and most of the occlusion calculation and illumination consistency processing work can be omitted based on the Unity3D occlusion processing.
2) Establishing a data file interface of the assembly element model, reading the three-dimensional coordinate information of the assembly element frame by frame from the data file, and driving the position of the corresponding three-dimensional model in a Unity3D graphic engine; carrying out appropriate scale amplification on the three-dimensional model in the virtual scene to meet the watching requirement;
3) determining the position and the posture of a virtual viewpoint in a virtual scene according to viewpoint position posture information output by the AR glasses positioning module, and associating the position of the assembly element in the virtual scene with the position information of the actual scene to ensure that the assembly element can be matched when being superposed;
4) generating coordinates of binocular viewpoints according to the viewpoint position in the current virtual scene and the position relation with the observed assembly part, and rendering and outputting images of the virtual model according to the current binocular coordinates respectively;
5) and displaying information such as animation display, UI dialog box prompt and the like of the assembly model, and displaying information such as model information, character description and the like of the assembly parts.
S5: an operator performs assembly operation according to the corresponding assembly prompt information of the MR spectacle end, the prompted assembly process information and the corresponding assembly part information;
s6: after the current assembly steps are completed, the steps of S4-S5 are repeated until all assembly tasks are completed.
Through the steps, the aviation cable auxiliary assembly method and system based on the MR can be realized, can assist workers in assembling airplane cables, and improves the airplane cable assembly efficiency and quality.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911266672.6A CN110928418A (en) | 2019-12-11 | 2019-12-11 | A kind of MR-based aviation cable auxiliary assembly method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911266672.6A CN110928418A (en) | 2019-12-11 | 2019-12-11 | A kind of MR-based aviation cable auxiliary assembly method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110928418A true CN110928418A (en) | 2020-03-27 |
Family
ID=69858965
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911266672.6A Pending CN110928418A (en) | 2019-12-11 | 2019-12-11 | A kind of MR-based aviation cable auxiliary assembly method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110928418A (en) |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112183223A (en) * | 2020-09-07 | 2021-01-05 | 北京劳动保障职业学院 | Underground pipe network teaching auxiliary system and method based on MR glasses |
| CN112330818A (en) * | 2020-11-03 | 2021-02-05 | 北京卫星环境工程研究所 | Unmanned aerial vehicle part manual layering auxiliary system and method based on augmented reality |
| CN112365574A (en) * | 2020-11-06 | 2021-02-12 | 北京市商汤科技开发有限公司 | Method, device, equipment and storage medium for displaying augmented reality AR information |
| CN112509149A (en) * | 2020-12-08 | 2021-03-16 | 太原理工大学 | Three-surface six-point shaft diameter measurement process guidance system and method based on mixed reality |
| CN112515275A (en) * | 2020-11-09 | 2021-03-19 | 义乌市输变电工程有限公司 | Hardware interface device |
| CN112634467A (en) * | 2021-01-05 | 2021-04-09 | 东方电气集团科学技术研究院有限公司 | AR guidance method in equipment manufacturing and assembling process |
| CN112686399A (en) * | 2020-12-24 | 2021-04-20 | 国网上海市电力公司 | Distribution room fire emergency repair method and system based on augmented reality technology |
| CN112734945A (en) * | 2021-03-30 | 2021-04-30 | 上海交大智邦科技有限公司 | Assembly guiding method, system and application based on augmented reality |
| CN112764548A (en) * | 2021-02-24 | 2021-05-07 | 北京计算机技术及应用研究所 | AR auxiliary assembly system |
| CN112764406A (en) * | 2021-01-26 | 2021-05-07 | 三一重机有限公司 | Intelligent auxiliary assembly system and method |
| CN113298003A (en) * | 2021-06-03 | 2021-08-24 | 北京安达维尔科技股份有限公司 | AR-based aviation cable assembly system and method |
| CN113393535A (en) * | 2021-06-25 | 2021-09-14 | 深圳市拓普智造科技有限公司 | Projection type operation guiding method and system |
| WO2022000713A1 (en) * | 2020-06-28 | 2022-01-06 | 南京翱翔信息物理融合创新研究院有限公司 | Augmented reality self-positioning method based on aviation assembly |
| CN114299519A (en) * | 2021-12-28 | 2022-04-08 | 中国商用飞机有限责任公司 | Auxiliary flight method based on XML format electronic flight manual |
| CN114323000A (en) * | 2021-12-17 | 2022-04-12 | 中国电子科技集团公司第三十八研究所 | Cable AR guiding assembly system and method |
| CN114723817A (en) * | 2022-03-24 | 2022-07-08 | 华中科技大学 | Auxiliary assembly method and system based on aviation knowledge graph |
| CN115130979A (en) * | 2022-05-17 | 2022-09-30 | 杭州书微信息科技有限公司 | Smoke machine auxiliary inspection device and method based on MR |
| CN115936414A (en) * | 2022-10-27 | 2023-04-07 | 中国工程物理研究院激光聚变研究中心 | Micro-device flexible assembly method based on assembly process and track self-planning |
| CN116778119A (en) * | 2023-06-26 | 2023-09-19 | 中国信息通信研究院 | Man-machine cooperative assembly system based on augmented reality |
| WO2024156172A1 (en) * | 2023-01-29 | 2024-08-02 | 中国电子科技集团公司第三十八研究所 | Ar-based cable connection assembly quality analysis system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150347849A1 (en) * | 2014-06-02 | 2015-12-03 | Tesa Sa | Method for supporting an operator in measuring a part of an object |
| CN107730591A (en) * | 2017-09-14 | 2018-02-23 | 北京致臻智造科技有限公司 | A kind of assembling bootstrap technique and system based on mixed reality equipment |
| CN108022264A (en) * | 2016-11-01 | 2018-05-11 | 狒特科技（北京）有限公司 | Camera pose determines method and apparatus |
| CN109491497A (en) * | 2018-10-19 | 2019-03-19 | 华中科技大学 | A kind of human assistance assembly application system based on augmented reality |
| CN109782905A (en) * | 2018-12-27 | 2019-05-21 | 佛山科学技术学院 | A kind of augmented reality assembly bootstrap technique and system suitable for AGV steering wheel |
| CN110333775A (en) * | 2019-05-16 | 2019-10-15 | 上海精密计量测试研究所 | An augmented reality maintenance guidance system and method for aerospace equipment |
-
2019
- 2019-12-11 CN CN201911266672.6A patent/CN110928418A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150347849A1 (en) * | 2014-06-02 | 2015-12-03 | Tesa Sa | Method for supporting an operator in measuring a part of an object |
| CN108022264A (en) * | 2016-11-01 | 2018-05-11 | 狒特科技（北京）有限公司 | Camera pose determines method and apparatus |
| CN107730591A (en) * | 2017-09-14 | 2018-02-23 | 北京致臻智造科技有限公司 | A kind of assembling bootstrap technique and system based on mixed reality equipment |
| CN109491497A (en) * | 2018-10-19 | 2019-03-19 | 华中科技大学 | A kind of human assistance assembly application system based on augmented reality |
| CN109782905A (en) * | 2018-12-27 | 2019-05-21 | 佛山科学技术学院 | A kind of augmented reality assembly bootstrap technique and system suitable for AGV steering wheel |
| CN110333775A (en) * | 2019-05-16 | 2019-10-15 | 上海精密计量测试研究所 | An augmented reality maintenance guidance system and method for aerospace equipment |
Non-Patent Citations (1)
| Title |
|---|
| 何汉武等: "《增强现实交互方法与实现》", 31 December 2018 * |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022000713A1 (en) * | 2020-06-28 | 2022-01-06 | 南京翱翔信息物理融合创新研究院有限公司 | Augmented reality self-positioning method based on aviation assembly |
| CN112183223A (en) * | 2020-09-07 | 2021-01-05 | 北京劳动保障职业学院 | Underground pipe network teaching auxiliary system and method based on MR glasses |
| CN112330818B (en) * | 2020-11-03 | 2021-06-22 | 北京卫星环境工程研究所 | Auxiliary system and method for manual layup of UAV parts based on augmented reality |
| CN112330818A (en) * | 2020-11-03 | 2021-02-05 | 北京卫星环境工程研究所 | Unmanned aerial vehicle part manual layering auxiliary system and method based on augmented reality |
| CN112365574A (en) * | 2020-11-06 | 2021-02-12 | 北京市商汤科技开发有限公司 | Method, device, equipment and storage medium for displaying augmented reality AR information |
| CN112515275A (en) * | 2020-11-09 | 2021-03-19 | 义乌市输变电工程有限公司 | Hardware interface device |
| CN112509149A (en) * | 2020-12-08 | 2021-03-16 | 太原理工大学 | Three-surface six-point shaft diameter measurement process guidance system and method based on mixed reality |
| CN112686399A (en) * | 2020-12-24 | 2021-04-20 | 国网上海市电力公司 | Distribution room fire emergency repair method and system based on augmented reality technology |
| CN112634467A (en) * | 2021-01-05 | 2021-04-09 | 东方电气集团科学技术研究院有限公司 | AR guidance method in equipment manufacturing and assembling process |
| CN112764406A (en) * | 2021-01-26 | 2021-05-07 | 三一重机有限公司 | Intelligent auxiliary assembly system and method |
| CN112764548A (en) * | 2021-02-24 | 2021-05-07 | 北京计算机技术及应用研究所 | AR auxiliary assembly system |
| CN112734945A (en) * | 2021-03-30 | 2021-04-30 | 上海交大智邦科技有限公司 | Assembly guiding method, system and application based on augmented reality |
| CN113298003A (en) * | 2021-06-03 | 2021-08-24 | 北京安达维尔科技股份有限公司 | AR-based aviation cable assembly system and method |
| CN113298003B (en) * | 2021-06-03 | 2024-05-03 | 北京安达维尔科技股份有限公司 | AR-based aviation cable assembly system and method |
| CN113393535A (en) * | 2021-06-25 | 2021-09-14 | 深圳市拓普智造科技有限公司 | Projection type operation guiding method and system |
| CN114323000A (en) * | 2021-12-17 | 2022-04-12 | 中国电子科技集团公司第三十八研究所 | Cable AR guiding assembly system and method |
| CN114323000B (en) * | 2021-12-17 | 2023-06-09 | 中国电子科技集团公司第三十八研究所 | Cable AR guide assembly system and method |
| CN114299519A (en) * | 2021-12-28 | 2022-04-08 | 中国商用飞机有限责任公司 | Auxiliary flight method based on XML format electronic flight manual |
| CN114723817A (en) * | 2022-03-24 | 2022-07-08 | 华中科技大学 | Auxiliary assembly method and system based on aviation knowledge graph |
| CN115130979A (en) * | 2022-05-17 | 2022-09-30 | 杭州书微信息科技有限公司 | Smoke machine auxiliary inspection device and method based on MR |
| CN115936414A (en) * | 2022-10-27 | 2023-04-07 | 中国工程物理研究院激光聚变研究中心 | Micro-device flexible assembly method based on assembly process and track self-planning |
| WO2024156172A1 (en) * | 2023-01-29 | 2024-08-02 | 中国电子科技集团公司第三十八研究所 | Ar-based cable connection assembly quality analysis system |
| CN116778119A (en) * | 2023-06-26 | 2023-09-19 | 中国信息通信研究院 | Man-machine cooperative assembly system based on augmented reality |
| CN116778119B (en) * | 2023-06-26 | 2024-03-12 | 中国信息通信研究院 | Man-machine cooperative assembly system based on augmented reality |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110928418A (en) | A kind of MR-based aviation cable auxiliary assembly method and system | |
| Wang et al. | A comprehensive review of augmented reality-based instruction in manual assembly, training and repair | |
| Evans et al. | Evaluating the Microsoft HoloLens through an augmented reality assembly application | |
| Ong et al. | Augmented reality applications in manufacturing: a survey | |
| US20190251750A1 (en) | Systems and methods for using a virtual reality device to emulate user experience of an augmented reality device | |
| Wang et al. | A comprehensive survey of augmented reality assembly research | |
| US20110316845A1 (en) | Spatial association between virtual and augmented reality | |
| CN106843150A (en) | A kind of industry spot simulation method and device | |
| CN108334199A (en) | The multi-modal exchange method of movable type based on augmented reality and device | |
| CN110909439A (en) | AR-based assembly method and device and terminal | |
| CN103488292B (en) | The control method of a kind of three-dimensional application icon and device | |
| CN113270006B (en) | HoloLens-based printing machine operation training system and method | |
| Sheldon et al. | Putting the AR In (AR) chitecture | |
| CN115661412A (en) | A system and method for auxiliary assembly of aero-engine based on mixed reality | |
| CN116569545A (en) | Remote assistance method and device | |
| CN113393535A (en) | Projection type operation guiding method and system | |
| Ganlin et al. | Information integration and instruction authoring of augmented assembly systems | |
| Bellalouna | VR Based Design Process of Industrial Products | |
| Pusch et al. | Augmented reality for operator training on industrial workplaces–Comparing the Microsoft hololens vs. small and big screen tactile devices | |
| CN108961420A (en) | Machine-building teaching auxiliary system and development approach based on AR cloud storage technology | |
| Friedrich et al. | Combining Gesture and Voice Control for Mid-air Manipulation of CAD Models in VR Environments. | |
| Li et al. | Instruction manual for product assembly process based on augmented visualization | |
| Baratoff et al. | Developing and applying AR Technology in design, production, service and training | |
| CN112198961B (en) | Auxiliary guide system for equipment maintenance operation | |
| CN116645247A (en) | Panoramic view-based augmented reality industrial operation training system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200327 |