CN107741785A - A kind of remote guide method and system for protecting front end safety - Google Patents
A kind of remote guide method and system for protecting front end safety Download PDFInfo
- Publication number
- CN107741785A CN107741785A CN201710946561.4A CN201710946561A CN107741785A CN 107741785 A CN107741785 A CN 107741785A CN 201710946561 A CN201710946561 A CN 201710946561A CN 107741785 A CN107741785 A CN 107741785A
- Authority
- CN
- China
- Prior art keywords
- information
- equipment
- augmented reality
- expert
- rear end
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/20—Administration of product repair or maintenance
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440227—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by decomposing into layers, e.g. base layer and one or more enhancement layers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
- H04N21/4622—Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Economics (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Quality & Reliability (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Entrepreneurship & Innovation (AREA)
- Processing Or Creating Images (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a kind of remote guide method and system for protecting front end safety, front end augmented reality equipment and rear end expert's equipment pass through the information exchange link interactive information with 3 transport layers；According to the selected of user, front end augmented reality equipment can forbid the transmission of video layer and/or audio layer, but must assure that the transmission of interaction data.The posture of object and instruction animation and displaying visual angle are instructed in rear end expert's equipment dummy model stored according to itself and the interaction data received, reconstruct；Rear end expert's equipment receives the tutorial message of expert's input, and tutorial message is synchronized into front end augmented reality equipment, is shown by front end augmented reality equipment combination dummy model.Using the present invention, it can still complete to instruct task under conditions of limited, protect the information security of front end, and the clear and definite information of front and back end expert equipment both sides refers to, and improves the validity of remote guide.
Description
Technical field
The present invention relates to the technical field of augmented reality, and in particular to it is a kind of protect front end safety remote guide method and
System.
Background technology
Augmented reality remote guide system is that can have as the staff one at scene to solve to allow the expert of distal end
Sample experience on the spot in person, and give the guidance that field personnel most directly, most accurately combines live outdoor scene.
When work in front end, the insurmountable problem of front-end person who can be many times run into, is just needed this when
Rear end expert consults.It can be realized at present by interactive voice (phone), or video+interactive voice (real-time video)
Target.But both modes can have very much limitation：
One, front and back end personnel interact can be very inconvenient, and information refers to indefinite.Expert is difficult to allow front end only by language
User clearly knows that oneself should how to operate.
Secondly, the scheme for being transmitted by Voice ＆ Video, equal to by all unreserved exhibition of all situations of front end
Show to rear end.If front end user scene is related to privacy concern, or is related to business secret, state secret, it has not been convenient to which rear end is special
When family is seen, a large amount of front end user privacies will be caused compromised, cause user security hidden danger.
The content of the invention
In view of this, can be limited the invention provides a kind of remote guide method and system for protecting front end safety
Under conditions of still complete instruct task, protect the information security of front end, and the front and back end clear and definite information of expert equipment both sides refers to
In generation, improve the validity of remote guide.
In order to solve the above-mentioned technical problem, the present invention is realized in.
A kind of remote guide method for protecting front end safety, front end augmented reality equipment and rear end expert's equipment by with
The information exchange link interactive information of 3 transport layers, 3 transport layers are video layer, audio layer and data Layer respectively, are respectively used to
Transmission of video information, audio-frequency information and interaction data；This method comprises the following steps：
Step 1: the dummy model for instructing object and instruction animation is added on real object by front end augmented reality equipment
Display；Video information, audio-frequency information and the interaction data of front end augmented reality equipment collection real scene；Front end augmented reality
The interaction data of equipment includes characterizing the information at front end augmented reality equipment visual angle, and characterizes the information of dummy model change；
Step 2: the transport layer selected according to user, 1 in the layer information exchange link of front end augmented reality equipment utilization 3
Layer or more than 1 layer carry out information transfer；Selected transport layer comprises at least data Layer；
Step 3: rear end expert's equipment dummy model stored according to itself and the interaction data received, reconstruct refer to
Posture and the displaying visual angle of object and instruction animation are led, is allowed to consistent with the scene that front end augmented reality equipment user sees；Such as
Fruit receives sound, video, instructs object and instruction animation to be added in video reconstruct, plays audio；
Step 4: rear end expert's equipment receives the tutorial message of expert's input, it is existing that tutorial message is synchronized into front end enhancing
Real equipment, it is shown by front end augmented reality equipment combination dummy model.
Preferably, the athletic posture of the information including augmented reality equipment for characterizing front end augmented reality equipment visual angle and
Camera zoom information；It is described to characterize position, rotation and the scalability information that the information that dummy model changes is dummy model.
Preferably, the tutorial message includes voice guidance information and digital tutorial message, the numeric guide packet
Include markup information and/or the conversion to dummy model；The voice guidance information is synchronized to front end augmented reality by voice layer
Equipment plays out；The numeric guide information is synchronized to front end augmented reality equipment by data Layer.
Preferably, front end augmented reality equipment and rear end expert's device configuration have identical guiding step；Guiding step is pressed
Order performs, and each guiding step corresponds to different instruction animations, and front end augmented reality equipment and rear end expert's equipment pass through number
According to the synchronous current guiding step of layer, it is synchronous play corresponding to instruction animation.
Present invention also offers a kind of remote guide system for protecting front end safety, including front end augmented reality equipment is with after
Expert's equipment, front end augmented reality equipment and rear end expert's equipment is held to be interacted by the information exchange link with 3 transport layers
Information, 3 transport layers are video layer, audio layer and data Layer respectively, are respectively used to transmission of video information, audio-frequency information and interaction
Data；
The front end augmented reality equipment, for the dummy model for instructing object and instruction animation to be added to real object
Upper display；Video information, audio-frequency information and the interaction data of real scene are gathered, the interaction data includes characterizing front end enhancing
Real world devices visual angle and the information of dummy model change；The transport layer selected according to user, using in 3 layer information exchange links
1 layer or more than the 1 layer transmission for being acquired information, selected transport layer comprises at least data Layer；Rear end is come from when receiving
After the tutorial message of expert's equipment, it is shown with reference to dummy model；
The rear end expert equipment, for according to the interaction data received, and the dummy model of itself storage, reconstruct
The posture of object and instruction animation and displaying visual angle are instructed, is allowed to consistent with the scene that front end augmented reality equipment user sees；
If receiving sound, video, instruct object and instruction animation to be added in video reconstruct, play audio；Receive expert
The tutorial message of input, it is synchronized to front end augmented reality equipment.
Preferably, the athletic posture of the information including augmented reality equipment for characterizing front end augmented reality equipment visual angle and
Camera zoom information；It is described to characterize position, rotation and the scalability information that the information that dummy model changes is dummy model；
The tutorial message includes voice guidance information and digital tutorial message, and the numeric guide information includes mark and believed
Breath and/or the conversion to dummy model.
Preferably, video acquisition module, front audio collection are configured with the front end augmented reality equipment and receives mould
Block, front end data acquisition module, front end synchronization module and transport layer selecting module；
The dummy model for instructing object and instruction animation is added on real object and shown by front end augmented reality equipment；
Video acquisition module, for gathering the video information of real scene, rear end expert's equipment is sent to by video layer；
Front audio gathers and receiving module, for gathering the audio-frequency information of real scene, after being sent to by audio layer
Hold expert's equipment；Play the voice guidance information from rear end expert's equipment；
Front end data acquisition and receiving module, for gathering the state of front end augmented reality equipment, sent by data Layer
Give rear end expert's equipment；
Transport layer selecting module, transport layer is selected for receiving user, selected transport layer comprises at least data Layer；
Forbid the video acquisition module and the front audio gathers and receiving module sends out the data of the unselected transport layer of user
Give rear end expert's equipment；
Front end synchronization module, for the Current Transform posture of dummy model in the augmented reality equipment of front end to be passed through into data Layer
Rear end expert's equipment is synchronized to, the markup information of data-guiding information and/or right is used as using what rear end expert's equipment synchronously came
The conversion of dummy model is added on dummy model and shown.
Preferably, the rear end expert equipment include rear end video reception module, rear end audio collection and receiving module, after
End data acquisition module, backend synchronization module and rear end display module；
Rear end video reception module, for the video information by video layer receiving front-end augmented reality equipment；
Rear end audio collection and receiving module, for being received and being played by audio layer from front end augmented reality equipment
Audio-frequency information, the voice guidance information of expert is gathered, front end augmented reality equipment is sent to by audio layer；
Back end data acquisition module, for gathering the data-guiding information of expert, including markup information and/or to virtual mould
The conversion of type, it is sent to backend synchronization module；
Backend synchronization module, set for the data-guiding information of expert to be synchronized into front end augmented reality by data Layer
Standby, according to the interaction data from front end augmented reality equipment, and the dummy model of itself storage, reconstruct are instructed object and referred to
Posture and the displaying visual angle of animation are led, instructs object and instruction animation to be sent to rear end display module reconstruct；
Rear end display module, for by reconstruct instruct object and the instruction animation rear end video reception module that is added to receives
Video information in show；If not receiving video, reconstruct content is directly displayed.
Preferably, it is stored with identical guiding step in front end synchronization module and backend synchronization module；Each guiding step
By performing in order, each guiding step corresponds to different instruction animations, and front end synchronization module and backend synchronization module pass through number
According to the synchronous current guiding step of layer, front end augmented reality equipment and rear end expert's equipment synchronously play corresponding to instruction animation.
Preferably, augmented reality equipment in front end further comprises user's guidance panel module, for receiving user to instructing
The selection of step, user is selected to be synchronized to rear end expert's equipment by data Layer.
Beneficial effect：
(1) the information exchange link design of front-end and back-end is 3 layers by the present invention, video layer, audio layer and data Layer.When
When there is privacy concern in front end user scene, user can actively close video or audio transmission channel, only pass through
Data Layer carries out the transmission of content.In the case of no video or audio transmission, due to there is a data transfer, rear end is special
Although family does not know that the outdoor scene content of front end, but can utilize dummy model and the first visual angle, it is known that front-end person who
With the relation between the relation of periphery three-dimensional world, and the subject matter to be instructed, even if can't see outdoor scene, rear end expert also may be used
By the mark in three-dimensional world, to continue to instruct front-end person who, task can be still completed under conditions of limited.
(2) present invention adds data in the interaction content of front and back end, and using interactive data, rear end expert's equipment can
It also can correctly be superimposed upon real scene with front end augmented reality equipment identical visual angle, expert tutorial message to obtain and work as
In, so as to make the interaction of both sides more accurate, solve front and back end expert's equipment personnel's mutual inconvenience, information refers to indefinite ask
Topic.
Brief description of the drawings
Fig. 1 is present system structural representation.
Fig. 2 is the situation that head end video and data are transferred to rear end.
Fig. 3 is the situation that front end data is not delivered to rear end.
Fig. 4 is a front end first person for treating maintenance of equipment operation.
Embodiment
The invention provides a kind of remote guide scheme for protecting front end safety, its basic thought is：It is remote in augmented reality
While journey instructs, when the content of secrecy in need in front end user scene, actively it can be transmitted by three-layer network
Module closes video or audio transmission channel, so as to ensure the privacy of user, and reaches the stable mesh for carrying out remote guide
's.
In order to ensure in the case of closing video and audio transmission channel data, expert still is able to realize and instructed.The present invention
On the basis of video and interactive voice, data interaction is added again, before rear end expert's equipment can be rebuild according to interaction data
The model information of scene is held, and is matched completely with outdoor scene so that rear end expert's equipment also possesses and front end user equipment phase
Same first person information.Expert can be operated or be increased markup information, operation knot in rear end on dummy model
Fruit and markup information can be transmitted back to headend equipment, and show, so as to help user clearly to know the guidance of expert, solve limitation
Under the conditions of remote guide problem, ensure privacy of user, while also solve front and back end personnel's mutual inconvenience, information refers to not clear
The problem of true.
The present invention will now be described in detail with reference to the accompanying drawings and examples.This method comprises the following steps：
Step 1: information of the structure with 3 transport layers is handed between front end augmented reality equipment and rear end expert's equipment
Mutual link, 3 transport layers are video layer, audio layer and data Layer respectively, are respectively used to transmission of video information, audio-frequency information and friendship
Mutual data.User can be selected 3 transport layers according to the situation of presence, can close audio layer and/or video layer, extremely
Retain a data Layer less.
Step 2: the dummy model for instructing object and instruction animation is added on real object by front end augmented reality equipment
Display.Object is wherein instructed to refer to the object to be needed repairing such as air-conditioning, automobile, printer.Instruction animation refers to animation
Screwdriver, the arrow etc. of form displaying are used for the animation for instructing user to operate., being capable of position pair when dummy model is varied from
That answers is reacted on real object, is easy to the understanding and operation of user.
Dummy object overlaps with the object in real scene, image recognition technology can be used to realize.In real scene
A mark point is placed, the mark point can be an image with obvious textural characteristics.Front end augmented reality equipment can be with
The image of the mark point is gathered by camera and is identified, so as to by the coordinate system of the Virtual Space where dummy object with
Coordinate system in reality scene overlaps, to reach the purpose that dummy object can be overlapped correctly with the object in real scene.
Step 3: the front end augmented reality equipment collection video information of real scene, audio-frequency information, and before obtaining sign
The information at augmented reality equipment visual angle is held, and characterizes the information of dummy model change, both information are all used as interaction data.
Then, video information, audio-frequency information and interaction data are transmitted to rear end expert's equipment.
For video acquisition, in this step, video acquisition module is deployed in the augmented reality equipment of front end, collection is true
Video information in scene, then H.265 the video information collected is encoded.By the video information after coding with byte
Form using HTTP/TCP by the video layer in 3 layer information exchange links, rear end is transferred to by front end augmented reality equipment
Expert's equipment.Possess the tutorial program same with front end augmented reality equipment in the expert's equipment of rear end, when video is increased by front end
When strong real world devices pass to rear end expert's equipment by video layer, video is included in rear end expert's equipment by decoding
On, so that rear end expert possesses the video pictures same with front end user.Only so not enough, because video flowing is only
The video information of front end, that is, the information in the real scene of front end are transmitted, but the virtual information of front end does not pass
Pass, as shown in figure 3, being solved in the description of this problem later.
For audio collection, in this step, audio collection and receiving module are deployed in front end augmented reality equipment respectively
And in the expert's equipment of rear end, the audio-frequency information of front end augmented reality equipment and rear end expert's equipment is gathered, then will be collected
Audio-frequency information encoded.By the audio layer in 3 layer information exchange links, it is delivered to the audio collection at both ends and receives mould
Decoding broadcasting is carried out in block, realizes the dialogue of front end user and rear end expert.
Above-mentioned audio layer and video layer all can decide whether closing transmission by user, need exist for explanation,
What user closed is one-way transmission, that is, limits the transmission of front end rear end, has no effect on the passback that voice is instructed in rear end.
For interaction data, the interaction data that front end is sent to rear end includes the letter for characterizing front end augmented reality equipment visual angle
Breath and the information for characterizing dummy model change.Wherein, the information for characterizing front end augmented reality equipment visual angle is existing for front end enhancing
The state of real equipment, including the athletic posture of augmented reality equipment and camera zoom information.In rear end, expert's equipment places one
The virtual video camera of individual posture identical with headend equipment, the first of user can be simulated using the state of augmented reality equipment
Claim visual angle.Characterize the Current Transform state that the information that dummy model changes is dummy model, including the position of dummy model, rotation
And scalability information.These status informations can help rear end expert's equipment reconstructed object and its relation between three-dimensional environment.
And in order to solve the problems, such as that virtual information together transmits not over video flowing, in rear end expert's its upper side administration with before
Identical dummy model in augmented reality equipment is held, the synchronization of model pose, size is carried out by data interaction, and be not required to
Want interaction models in itself, such transinformation content is fewer.
Specifically, the video content that rear end expert equipment only possesses front end user is far from being enough, because not knowing
Road front end user is far and near apart from object, the information such as direction of counterbody, therefore the virtual content of rear end expert's equipment can not
Correctly superposition on video content, can cause to show mistake.Therefore, a virtual video camera is affixed one's name in rear end expert's its upper side
Attitude information for AFE(analog front end) augmented reality equipment, that is to say, that simulate the head pose of front-end operations person.Front end increases
Strong real world devices collect the equipment posture of front end user, that is, move and rotate, be transferred in the form of byte by Transmission Control Protocol
Rear end expert's equipment, after rear end expert's equipment receives attitude information, the equipment posture of front end user is gone out by the information simulation,
Virtual content is set correctly to be superimposed upon among video, as shown in Figure 2.
Step 4: front end augmented reality equipment is selected according to user to transport layer, information transfer is carried out.Forbid user
The data of unselected transport layer are sent to rear end expert's equipment.
Step 5: rear end expert's equipment is according to the interaction data received, and the void of rear end expert's equipment itself storage
Analog model, reconstruct instruct the posture of object and instruction animation and displaying visual angle, are allowed to see with front end augmented reality equipment user
Scene it is consistent.During display, if video information transmits, then by reconstruct instruct object and instruction animation is added to video
In, while audio is played, so as to build one and actual identical three-dimensional environment.If without video, reconstruct is directly displayed
Virtual content.
Step 6: rear end expert's equipment receives the tutorial message of expert's input, tutorial message, which can include voice guidance, to be believed
Cease with data tutorial message, data-guiding information such as markup information (mark point of maintenance) and/or to instructing object, instructing to move
The conversion (mobile etc.) of picture, by tutorial message by 3 layer information exchange link synchronizations to front end augmented reality equipment, wherein, language
Sound tutorial message walks audio layer, and data-guiding information walks data Layer.
In the expert's equipment of rear end, expert can be indicated currently by the model that rear end is shown for front end augmented reality equipment
Operating procedure, such as a mark point is added in the position to need repairing, and tell front end enhancing existing by mobility model
What how real equipment operated.
Step 5: front end augmented reality equipment receives tutorial message, it is shown with reference to dummy model.
In this step, for voice guidance information, play out；For data-guiding information, it is applied on dummy model
It is shown, so, front end user is it is seen that expert directs.
This programme is by the design of layer three information interactive link it can be seen from above scheme, when in front end user scene
When the content of secrecy in need, user can actively close video or audio transmission channel, only be carried out by data Layer
The transmission of content, so as to ensure the privacy of user, and reach the stable purpose for carrying out remote guide.It is additionally, since and adds
The design of data interaction so that expert's equipment reference in front and back end is clear, and validity is instructed in raising.
In order to allow untrained personnel also to use the present invention, the present invention is in front end augmented reality equipment and rear end
Expert's device configuration identical tutorial program.There are multiple guiding steps for needing to perform successively in tutorial program.Each refer to
Lead step and correspond to different instruction animations, such as the first step is to turn on screw, second step is to open lid.Front end augmented reality is set
Each standby operation can be synchronized in the expert's equipment equipment of rear end, and operation of the rear end expert's equipment to dummy model also may be used
To be synchronized among the augmented reality equipment of front end to instruct front end user.Therefore, front end augmented reality equipment and rear end expert set
It is standby to need by the synchronous current guiding step of data Layer so that front and back ends synchronously play corresponding to instruction animation.
Specifically, user is carrying out when guidance of stepwise using augmented reality equipment, and synchronization module should
Preceding step instruction is passed in the expert's equipment of rear end by the data Layer of 3 layer information exchange links.It is same in the expert's equipment of rear end
Possess the guidance content of front end augmented reality equipment, when the synchronization module of rear end expert's equipment is received from front end augmented reality
After the data command that equipment is sent, the content of corresponding step is performed in the expert's equipment of rear end according to instruction.Similar, rear end is special
Synchronization module is also deployed with family's equipment, when expert is operated when rear end, such as expert is to the virtual interior of current procedures
Hold the operation that is pulled, synchronization module can send synchronic command be transferred to by data Layer it is complete among the augmented reality equipment of front end
Into synchronization.So it ensure that front end augmented reality equipment is synchronous with the content of rear end expert's equipment.
In order to realize a kind of remote guide method of the present invention, present invention also offers a kind of long-range finger as shown in Figure 1
Guiding systems.The remote guide system includes a front end augmented reality equipment and rear end expert's equipment.
It is described below for front end augmented reality equipment.
Front end augmented reality equipment is provided with video acquisition module, front end except the augmented reality function with itself
Audio collection and receiving module, front end data acquisition module, front end synchronization module and transport layer selecting module；
The dummy model for instructing object and instruction animation is added on real object and shown by front end augmented reality equipment；
Video acquisition module, for gathering the video information of real scene, rear end expert's equipment is sent to by video layer；
Front audio gathers and receiving module, for gathering the audio-frequency information of real scene, after being sent to by audio layer
Hold expert's equipment；Play the voice guidance information from rear end expert's equipment；
Front end data acquisition and receiving module, for gathering the state of front end augmented reality equipment, sent by data Layer
Give rear end expert's equipment；
Transport layer selecting module, transport layer is selected for receiving user, selected transport layer comprises at least data Layer；
Forbid the video acquisition module and the front audio gathers and receiving module sends out the data of the unselected transport layer of user
Give rear end expert's equipment；
Front end synchronization module, for the Current Transform posture of dummy model in the augmented reality equipment of front end to be passed through into data Layer
Rear end expert's equipment is synchronized to, the markup information of data-guiding information and/or right is used as using what rear end expert's equipment synchronously came
The conversion of dummy model is added on dummy model and shown.
Rear end expert's equipment includes rear end video reception module, rear end audio collection and receiving module, Back end data gathers
Module, backend synchronization module, rear end display module and network environment adaptation module.
Rear end video reception module, for the video information by video layer receiving front-end augmented reality equipment；
Rear end audio collection and receiving module, for being received and being played by audio layer from front end augmented reality equipment
Audio-frequency information, the voice guidance information of expert is gathered, front end augmented reality equipment is sent to by audio layer；
Back end data acquisition module, for gathering the data-guiding information of expert, including markup information and/or to virtual mould
The conversion of type, it is sent to backend synchronization module；
Backend synchronization module, set for the data-guiding information of expert to be synchronized into front end augmented reality by data Layer
Standby, according to the interaction data from front end augmented reality equipment, and the dummy model of itself storage, reconstruct are instructed object and referred to
Posture and the displaying visual angle of animation are led, instructs object and instruction animation to be sent to rear end display module reconstruct；
Rear end display module, for by reconstruct instruct object and the instruction animation rear end video reception module that is added to receives
Video information in show；If not receiving video, reconstruct content is directly displayed.
Preferably, it is stored with identical guiding step in front end synchronization module and backend synchronization module；Each guiding step
By performing in order, each guiding step corresponds to different instruction animations, and front end synchronization module and backend synchronization module pass through number
According to the synchronous current guiding step of layer, front end augmented reality equipment and rear end expert's equipment synchronously play corresponding to instruction animation.
For the ease of operation, front end additionally provides user's guidance panel, and the panel can be virtual or actual, use
Family carries out the advance of guiding step, retrogressing by guidance panel, beginning, terminates, the operation such as skips.The operation of user is also by number
Rear end is synchronized to according to layer.
Name a concrete operations example：
As shown in figure 4, user needs to carry out maintenance work to equipment.Front end user uses the maintenance in augmented reality equipment
Program can be step by step the prompting repaired to the equipment to need repairing instruct.First, program will recognise that real field
Scape, virtual information is correctly superimposed upon among real scene by augmented reality equipment.Then, user breathes out virtual menu, point
Different step information can be watched by hitting virtual push button, and among each step, virtual information can all make corresponding change, example
Such as, screwdriver, which can be moved to, needs the position that dismounts to prompt user to carry out disassembling operations to current location using screwdriver, together
When user perspective in can show current step information.Can be next to call rear end expert when user has a question to maintenance
It is instructed, when button is instructed in click, headend equipment, which is converted to order by data transmission module, can transmit word
Section form, rear end is transferred to by the data transmission channel in three-layer network transport module.Rear end instructs equipment to receive order,
The communication connection of video and audio is established with headend equipment.Then, front end will collect in the augmented reality equipment of front user
Video information rear end is transferred to by video layer.Meanwhile after equipment attitude information is transferred to by headend equipment using data Layer
Carry out the first person of AFE(analog front end) user among the virtual video camera of end.It is a set of as front end due to having in rear end equipment
Step procedure is repaired, therefore, in real scene content and virtual scene that rear end will be simulated in the visual angle of front end user first
Content.Now, rear end expert can be to instruct front end user.The operation that can specifically carry out is：Expert can be right
The operation that the object in virtual scene in rear end equipment is dragged, rotated and scaled, these operations can trigger rear end
Synchronization module, these operations are transferred among headend equipment in the form of byte by data Layer, and will dragging, rotation or
The operation of scaling is applied on the dummy object specified, and front end user is can see the operation behavior of rear end expert.After and
End expert can add mark point to prompt front end user in virtual scene.When front end user is carrying out the same of step change
When, synchronization module can be also triggered, current procedures are transferred among rear end equipment by data Layer.Rear end equipment receives accordingly
The step of and perform given step in the tutorial program of oneself and reach synchronous with front end.
When user terminal environmental restrictions can not transmit voice, audio layer transmission is closed, only transmits video and interaction data, from
And ensure front end privacy.
In summary, presently preferred embodiments of the present invention is these are only, is not intended to limit the scope of the present invention.
Within the spirit and principles of the invention, any modification, equivalent substitution and improvements made etc., it should be included in the present invention's
Within protection domain.
Claims (10)
- A kind of 1. remote guide method for protecting front end safety, it is characterised in that front end augmented reality equipment and rear end expert set Standby by the information exchange link interactive information with 3 transport layers, 3 transport layers is video layer, audio layer and data respectively Layer, is respectively used to transmission of video information, audio-frequency information and interaction data；This method comprises the following steps：Show Step 1: the dummy model for instructing object and instruction animation is added on real object by front end augmented reality equipment Show；Video information, audio-frequency information and the interaction data of front end augmented reality equipment collection real scene；Front end augmented reality is set Standby interaction data includes characterizing the information at front end augmented reality equipment visual angle, and characterizes the information of dummy model change；Step 2: the transport layer selected according to user, 1 layer or 1 in the layer information exchange link of front end augmented reality equipment utilization 3 More than layer carry out information transfer；Selected transport layer comprises at least data Layer；Step 3: rear end expert's equipment dummy model stored according to itself and the interaction data received, reconstruct guidance pair As the posture with instruction animation and displaying visual angle, it is allowed to consistent with the scene that front end augmented reality equipment user sees；If connect Sound, video are received, instructs object and instruction animation to be added in video reconstruct, plays audio；Step 4: rear end expert's equipment receives the tutorial message of expert's input, tutorial message is synchronized to front end augmented reality and set It is standby, it is shown by front end augmented reality equipment combination dummy model.
- 2. the method as described in claim 1, it is characterised in that the information for characterizing front end augmented reality equipment visual angle includes The athletic posture and camera zoom information of augmented reality equipment；The information for characterizing dummy model change is dummy model Position, rotation and scalability information.
- 3. method as claimed in claim 2, it is characterised in that the tutorial message includes voice guidance information and numeric guide Information, the numeric guide information include markup information and/or the conversion to dummy model；The voice guidance information passes through language Sound-absorbing layer is synchronized to front end augmented reality equipment and played out；The numeric guide information is synchronized to front end enhancing now by data Layer Real equipment.
- 4. the method as described in claim 1, it is characterised in that front end augmented reality equipment and rear end expert's device configuration have phase Same guiding step；Guiding step is performed in order, and each guiding step corresponds to different instruction animations, and front end augmented reality is set Standby and rear end expert's equipment by the synchronous current guiding step of data Layer, it is synchronous play corresponding to instruction animation.
- 5. a kind of remote guide system for protecting front end safety, including front end augmented reality equipment and rear end expert's equipment, it is special Sign is, front end augmented reality equipment and rear end expert's equipment by the information exchange link interactive information with 3 transport layers, 3 transport layers are video layer, audio layer and data Layer respectively, are respectively used to transmission of video information, audio-frequency information and interaction data；The front end augmented reality equipment, show for the dummy model for instructing object and instruction animation to be added on real object Show；Video information, audio-frequency information and the interaction data of real scene are gathered, the interaction data includes characterizing front end augmented reality Equipment visual angle and the information of dummy model change；The transport layer selected according to user, utilize 1 in 3 layer information exchange links Layer or more than the 1 layer transmission for being acquired information, selected transport layer comprise at least data Layer；Rear end expert is come from when receiving After the tutorial message of equipment, it is shown with reference to dummy model；The rear end expert equipment, for according to the interaction data received, and the dummy model of itself storage, reconstruct are instructed The posture of object and instruction animation and displaying visual angle, are allowed to consistent with the scene that front end augmented reality equipment user sees；If Sound, video are received, then instructs object and instruction animation to be added in video reconstruct, plays audio；Receive expert's input Tutorial message, be synchronized to front end augmented reality equipment.
- 6. system as claimed in claim 5, it is characterised in that the information for characterizing front end augmented reality equipment visual angle includes The athletic posture and camera zoom information of augmented reality equipment；The information for characterizing dummy model change is dummy model Position, rotation and scalability information；The tutorial message includes voice guidance information and digital tutorial message, and the numeric guide information includes markup information And/or the conversion to dummy model.
- 7. system as claimed in claim 5, it is characterised in that be configured with video acquisition mould in the front end augmented reality equipment Block, front audio collection and receiving module, front end data acquisition module, front end synchronization module and transport layer selecting module；The dummy model for instructing object and instruction animation is added on real object and shown by front end augmented reality equipment；Video acquisition module, for gathering the video information of real scene, rear end expert's equipment is sent to by video layer；Front audio gathers and receiving module, and for gathering the audio-frequency information of real scene, it is special to be sent to rear end by audio layer Family's equipment；Play the voice guidance information from rear end expert's equipment；Front end data acquisition and receiving module, for gathering the state of front end augmented reality equipment, after being sent to by data Layer Hold expert's equipment；Transport layer selecting module, transport layer is selected for receiving user, selected transport layer comprises at least data Layer；Forbid The data of the unselected transport layer of user are sent to by the video acquisition module and front audio collection and receiving module Rear end expert's equipment；Front end synchronization module, for the Current Transform posture of dummy model in the augmented reality equipment of front end is synchronous by data Layer To rear end expert's equipment, using rear end expert's equipment synchronously come as the markup information of data-guiding information and/or to virtual The conversion of model is added on dummy model and shown.
- 8. system as claimed in claim 6, it is characterised in that the rear end expert equipment include rear end video reception module, Rear end audio collection and receiving module, Back end data acquisition module, backend synchronization module and rear end display module；Rear end video reception module, for the video information by video layer receiving front-end augmented reality equipment；Rear end audio collection and receiving module, for the audio from front end augmented reality equipment to be received and played by audio layer Information, the voice guidance information of expert is gathered, front end augmented reality equipment is sent to by audio layer；Back end data acquisition module, for gathering the data-guiding information of expert, including markup information and/or to dummy model Conversion, is sent to backend synchronization module；Backend synchronization module, for the data-guiding information of expert to be synchronized into front end augmented reality equipment, root by data Layer According to the interaction data from front end augmented reality equipment, and the dummy model of itself storage, reconstruct are instructed object and instructed dynamic The posture of picture and displaying visual angle, instruct object and instruction animation to be sent to rear end display module reconstruct；Rear end display module, for by reconstruct instruct object and what instruction animation was added to that rear end video reception module receives regards Shown in frequency information；If not receiving video, reconstruct content is directly displayed.
- 9. system as claimed in claim 8, it is characterised in that be stored with phase in front end synchronization module and backend synchronization module Same guiding step；For each guiding step by performing in order, each guiding step corresponds to different instruction animations, front end synchronization mould Synchronously current guiding step, front end augmented reality equipment and rear end expert's equipment are synchronous by data Layer for block and backend synchronization module Instruction animation corresponding to broadcasting.
- 10. system as claimed in claim 9, it is characterised in that front end augmented reality equipment further comprises user's operating surface Plate module, for receiving selection of the user to guiding step, user is selected to be synchronized to rear end expert's equipment by data Layer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710946561.4A CN107741785B (en) | 2017-10-12 | 2017-10-12 | Remote guidance method and system for protecting front end safety |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710946561.4A CN107741785B (en) | 2017-10-12 | 2017-10-12 | Remote guidance method and system for protecting front end safety |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107741785A true CN107741785A (en) | 2018-02-27 |
| CN107741785B CN107741785B (en) | 2021-01-29 |
Family
ID=61237481
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710946561.4A Expired - Fee Related CN107741785B (en) | 2017-10-12 | 2017-10-12 | Remote guidance method and system for protecting front end safety |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107741785B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109034167A (en) * | 2018-06-08 | 2018-12-18 | 上海数圃信息科技有限公司 | Visual indicia method, system, electric terminal and computer readable storage medium |
| CN109814709A (en) * | 2018-12-27 | 2019-05-28 | 北京航天智造科技发展有限公司 | Operating Guideline method and device based on AR system |
| CN110609913A (en) * | 2019-09-18 | 2019-12-24 | 中国工商银行股份有限公司 | Information processing method and device, electronic device and medium |
| CN110728756A (en) * | 2019-09-30 | 2020-01-24 | 亮风台（上海）信息科技有限公司 | Remote guidance method and device based on augmented reality |
| CN112150507A (en) * | 2020-09-29 | 2020-12-29 | 厦门汇利伟业科技有限公司 | Method and system for synchronously reproducing 3D model of object posture and displacement |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5521616A (en) * | 1988-10-14 | 1996-05-28 | Capper; David G. | Control interface apparatus |
| CN1741050A (en) * | 2004-08-23 | 2006-03-01 | 扬兴人工智慧研发股份有限公司 | Intelligent Decision Support System |
| CN1845064A (en) * | 2005-04-08 | 2006-10-11 | 佳能株式会社 | Information processing method and apparatus |
| US20080157996A1 (en) * | 2006-12-28 | 2008-07-03 | Tain-Rein Chen | Universal remote controller |
| CN101547346A (en) * | 2008-03-24 | 2009-09-30 | 展讯通信（上海）有限公司 | Method and device for receiving and transmitting description of scene in rich media TV |
| CN101866215A (en) * | 2010-04-20 | 2010-10-20 | 复旦大学 | Human-computer interaction device and method using gaze tracking in video surveillance |
| US20130271404A1 (en) * | 2012-04-12 | 2013-10-17 | Lg Electronics Inc. | Remote controller equipped with touch pad and method for controlling the same |
| CN103996314A (en) * | 2014-05-22 | 2014-08-20 | 南京奥格曼提软件科技有限公司 | Teaching system based on augmented reality |
| CN105739704A (en) * | 2016-02-02 | 2016-07-06 | 上海尚镜信息科技有限公司 | Remote guidance method and system based on augmented reality |
| CN106534672A (en) * | 2016-11-02 | 2017-03-22 | 深圳亿维锐创科技股份有限公司 | Medical operation direct broadcast system |
-
2017
- 2017-10-12 CN CN201710946561.4A patent/CN107741785B/en not_active Expired - Fee Related
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5521616A (en) * | 1988-10-14 | 1996-05-28 | Capper; David G. | Control interface apparatus |
| CN1741050A (en) * | 2004-08-23 | 2006-03-01 | 扬兴人工智慧研发股份有限公司 | Intelligent Decision Support System |
| CN1845064A (en) * | 2005-04-08 | 2006-10-11 | 佳能株式会社 | Information processing method and apparatus |
| US20080157996A1 (en) * | 2006-12-28 | 2008-07-03 | Tain-Rein Chen | Universal remote controller |
| CN101547346A (en) * | 2008-03-24 | 2009-09-30 | 展讯通信（上海）有限公司 | Method and device for receiving and transmitting description of scene in rich media TV |
| CN101866215A (en) * | 2010-04-20 | 2010-10-20 | 复旦大学 | Human-computer interaction device and method using gaze tracking in video surveillance |
| US20130271404A1 (en) * | 2012-04-12 | 2013-10-17 | Lg Electronics Inc. | Remote controller equipped with touch pad and method for controlling the same |
| CN103996314A (en) * | 2014-05-22 | 2014-08-20 | 南京奥格曼提软件科技有限公司 | Teaching system based on augmented reality |
| CN105739704A (en) * | 2016-02-02 | 2016-07-06 | 上海尚镜信息科技有限公司 | Remote guidance method and system based on augmented reality |
| CN106534672A (en) * | 2016-11-02 | 2017-03-22 | 深圳亿维锐创科技股份有限公司 | Medical operation direct broadcast system |
Non-Patent Citations (2)
| Title |
|---|
| SAMIR BENBELKACEM; NADIA ZENATI-HENDA; HAYET BELGHIT; ABDELKADER: "Extended web services for remote collaborative manipulation in distributed augmented reality", 《2015 3RD INTERNATIONAL CONFERENCE ON CONTROL, ENGINEERING & INFORMATION TECHNOLOGY (CEIT)》 * |
| 张茜雷勇: "基于虚拟仪器技术的网络化远程实验室系统", 《实验室研究与探索》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109034167A (en) * | 2018-06-08 | 2018-12-18 | 上海数圃信息科技有限公司 | Visual indicia method, system, electric terminal and computer readable storage medium |
| CN109814709A (en) * | 2018-12-27 | 2019-05-28 | 北京航天智造科技发展有限公司 | Operating Guideline method and device based on AR system |
| CN110609913A (en) * | 2019-09-18 | 2019-12-24 | 中国工商银行股份有限公司 | Information processing method and device, electronic device and medium |
| CN110728756A (en) * | 2019-09-30 | 2020-01-24 | 亮风台（上海）信息科技有限公司 | Remote guidance method and device based on augmented reality |
| CN110728756B (en) * | 2019-09-30 | 2024-02-09 | 亮风台（上海）信息科技有限公司 | Remote guidance method and device based on augmented reality |
| CN112150507A (en) * | 2020-09-29 | 2020-12-29 | 厦门汇利伟业科技有限公司 | Method and system for synchronously reproducing 3D model of object posture and displacement |
| CN112150507B (en) * | 2020-09-29 | 2024-02-02 | 厦门汇利伟业科技有限公司 | 3D model synchronous reproduction method and system for object posture and displacement |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107741785B (en) | 2021-01-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107645651A (en) | A kind of remote guide method and system of augmented reality | |
| CN107741785A (en) | A kind of remote guide method and system for protecting front end safety | |
| CN108074431B (en) | System and method for applying VR (virtual reality) technical lecture practical training | |
| EP0586487B1 (en) | Personalized instructional aid | |
| US10049496B2 (en) | Multiple perspective video system and method | |
| KR20190088545A (en) | Systems, methods and media for displaying interactive augmented reality presentations | |
| US20030227453A1 (en) | Method, system and computer program product for automatically creating an animated 3-D scenario from human position and path data | |
| CN105938665A (en) | Remote audio and video operation demonstration system | |
| KR101839122B1 (en) | A multi-access virtual space provisioning system linked with a multi-user real space location system and a method for providing 3D virtual reality images in a multiple access virtual space | |
| CN109118855A (en) | A kind of net work teaching system of huge screen holography reduction real scene | |
| WO2008115334A9 (en) | System and methods for mobile videoconferencing | |
| CN104883557A (en) | Real time holographic projection method, device and system | |
| CN110969905A (en) | Remote teaching interaction and teaching aid interaction system for mixed reality and interaction method thereof | |
| CN107274743A (en) | Transmission line of electricity visual simulating training system based on VR technologies | |
| CN104866261B (en) | A kind of information processing method and device | |
| CN114842704B (en) | Training system and training method | |
| JP2002200206A (en) | Method for retrieving instruction for way of moving body in sports using mobile object image communication | |
| CN112509401A (en) | Remote real-practice teaching method and system based on augmented reality projection interaction | |
| CN105320282A (en) | Image recognition solution based on augmented reality | |
| CN110351514A (en) | A kind of method that dummy model passes through remote assistance mode and video flowing simultaneous transmission | |
| CN208985412U (en) | Wisdom education physical environment system | |
| CN111403022B (en) | Application method of medical movable split type control system | |
| CN107872655A (en) | A kind of method and system for determining hidden danger point | |
| CN106060528B (en) | Augmented reality method and system based on mobile phone terminal and electronic whiteboard | |
| JPH10208073A (en) | Virtual reality creation device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210129 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |