CN102148818B - Method and system for realizing distributed virtual reality and visualization on mobile device - Google Patents

Abstract

The invention discloses a method and system for realizing distributed virtual reality and visualization on a mobile device, belonging to the technical field of graphics and visualization. The method is as follows: 1) Establishing a communication connection between the mobile device and the virtual reality and visualization server; 2) The mobile device parses the interactive operation request command into an interactive command between the virtual reality and the visualization system; 3) The mobile device sends the interactive command to the virtual reality 4) The virtual reality and visualization system performs corresponding calculations according to interactive commands, and sends the execution results to the mobile device; 5) The mobile device decodes the received data and displays the virtual scene and visualization results. The system includes a client on a mobile device and a server on a virtual reality and visualization server; the mobile device and the virtual reality and visualization server are connected through a network. The invention can perform high-precision browsing and roaming of three-dimensional scenes on common mobile devices, and greatly improves the realism of drawing.

Description

Method and system for realizing distributed virtual reality and visualization on mobile devices

technical field

The invention relates to a method for realizing distributed virtual reality and visualization on a mobile device, and belongs to the technical fields of virtual reality and technology, graphics and visualization technology, and network communication.

Background technique

In today's society, mobile devices and various application software on the mobile devices are becoming more and more popular. At present, mobile computing chips on mobile devices are like the latest version of the latest embedded ARM architecture, the Cortex-A9 version, but the maximum memory capacity is 512M, and the core frequency is up to 2GHZ, but its static storage and dynamic capabilities and computing performance are comparable to those traditionally based on X86 The computer and server of the frame are far apart, especially the special model with the most powerful GPU performance under this architecture is Mali-T604, its parameters support OpenGL ES 1.1/2.0, the limit of geometric processing capacity is tens of millions per second, and pixel filling The rate limit is hundreds of megabits per second, which can only support some lightweight 3D digital entertainment, but in virtual reality and visualization systems, a virtual scene may contain tens of millions of meshes, and may also contain a large number of Physical models, vector line drawings, or other virtual reality content such as lighting, waves, sky, etc., the calculation and rendering of these huge data has far exceeded the computing and rendering capabilities of mobile devices, and how to store these massive data is also facing challenges.

Therefore, current mobile devices are completely powerless for some advanced 3D graphics rendering and computing applications, not to mention real-time rendering and related visualization applications for virtual reality environments with massive graphics data (including geometry, texture, images, etc.).

Contents of the invention

Aiming at the limitations of three-dimensional graphics rendering, virtual reality and visualization on mobile devices to computing power, storage capacity, and three-dimensional graphics processing capabilities, the purpose of the present invention is to propose a method and system for realizing virtual reality and visualization functions on mobile devices.

The computing function, storage function and three-dimensional graphics processing function of the virtual reality and visualization system on the mobile device are realized by distributing on the virtual reality and visualization high-performance GPU cluster server, while the mobile device itself only provides user interactive graphical interface and interactive functions, Functions such as the analysis and sending of user interaction instructions, the receiving and decoding of streaming media drawn by virtual reality and visualization, etc., have very low requirements on the computing performance of mobile devices, and almost only need to have the function of displaying images. In this way, the computing load of the entire virtual reality and visualization application is almost completely borne by the GPU cluster server. The GPU cluster server is responsible for completing all functions of the entire virtual reality calculation and rendering, and is responsible for receiving and executing user control commands sent by mobile devices. And the results of 3D virtual scene rendering and visualization are compressed and coded and fed back to the mobile device with streaming media as the carrier. Based on this framework, we have developed a prototype system. The test results show that the system realizes flexible and efficient virtual reality and visualization system functions on mobile devices, far exceeding the computing and storage functions that a single mobile device can possess and complete. The interactive control can also be realized in real time. In addition, a series of visualization-related applications such as medical imaging visualization: digital tomography, earth science visualization: oil and gas exploration, computational biology: molecular dynamics simulation, etc. can also be implemented on mobile devices under this framework. application. Google, the industry leader in the development and application of digital earth systems, has tried to transplant Google Map to mobile devices according to the current model, and launched GoogleMap Mobile (Google Map mobile version), users can use mobile devices such as PocketPC to browse global images online , but its implementation architecture is completely different from our approach.

Technical scheme of the present invention is:

A method for realizing distributed virtual reality and visualization on a mobile device, the steps of which are:

1) Establishing a communication connection between the mobile device and the virtual reality and visualization server; wherein, the mobile device includes a graphical user interface and a client for virtual reality and visualization display, and the virtual reality and visualization server includes a virtual reality and visualization system;

2) The mobile device receives an interactive operation request command through the graphical user interface, and parses it into an interactive command between the virtual reality and the visualization system;

3) The mobile device packs the parsed interactive command into a data packet and sends it to the virtual reality and visualization server;

4) The virtual reality and visualization server parses out the interactive command from the data packet, and sends it to the virtual reality and visualization system;

5) The virtual reality and visualization system performs corresponding calculations according to interactive commands, then performs virtual rendering and compresses and encodes the execution results as streaming media and sends them to mobile devices;

6) The mobile device decodes the received data and displays the virtual scene and visualization results.

Further, if the interactive operation request command is a command other than the three-dimensional main window view of the graphical user interface, the virtual reality and visualization server parses the command according to the set graphical interface command message response function.

Further, if the interactive operation request command is a command within the three-dimensional main window view of the graphical user interface, and its command format is the coordinates of the two-dimensional view, then the virtual reality and visualization server maps the two-dimensional view coordinate format command to Corresponding to the objects in the three-dimensional space and according to the contextual association of the actions in the window, the operation commands for the three-dimensional objects are analyzed.

Further, the virtual reality and visualization system includes a parallel virtual rendering subsystem; after all virtual reality and visualization computing tasks are completed this time, the parallel virtual rendering subsystem performs virtual rendering of three-dimensional graphics.

Further, the parallel virtual rendering subsystem is constructed by a GPU cluster; the parallel virtual rendering subsystem adopts a sort-first strategy for three-dimensional graphics rendering.

Further, the user image interface includes: menus, buttons, toolbars, dialog boxes, shortcut keys, three-dimensional navigation graphic interfaces, and containers for interactive means in three-dimensional scenes.

A system for realizing distributed virtual reality and visualization on a mobile device, characterized in that it includes a client on the mobile device, a server on the virtual reality and visualization server; the mobile device and the virtual reality and visualization server communicate through Network connection; wherein, the client includes a graphical user interface, a command parsing and sending module, a streaming media decoding module, a streaming media drawing module; the server end includes a virtual reality and visualization system, an order receiving module, a streaming media compression coding module;

The graphical user interface is configured to receive an interactive operation request command;

The command parsing and sending module is used to parse the interactive operation request command into an interactive command of the virtual reality and visualization system, and pack it into a data packet and send it to the command receiving module;

The command receiving module is used to store and analyze the received data packets, and send the parsed interactive commands to the virtual reality and visualization system;

The virtual reality and visualization system is used to perform corresponding calculations and virtual renderings according to interactive commands, and send the execution results to the streaming media compression coding module;

The streaming media compression encoding module is used to compress and encode the execution result as streaming media, and send it to the streaming media decoding module;

The streaming media decoding module is used to decode the received data;

The streaming media rendering module is used for rendering and displaying virtual scenes and visualization results on the decoded data.

Further, the virtual reality and visualization system includes a parallel virtual rendering subsystem, which is used to perform virtual rendering of three-dimensional graphics on the data after all virtual reality and visualization computing tasks are completed this time.

Further, the parallel virtual rendering subsystem is constructed by a GPU cluster; the parallel virtual rendering subsystem adopts a sort-first strategy for three-dimensional graphics rendering.

Further, the server adopts an independent server architecture or a distributed server architecture.

Compared with prior art, positive effect of the present invention is:

The mobile version of Google Maps only supports general GIS functions, such as map browsing, location search and navigation, etc. It does not have the high interactivity similar to virtual reality systems, the operability and controllability of 3D scenes, and it does not have 3D virtual reality. Functions of the reality system (including editing construction, modification of 3D virtual scene, virtual simulation, etc.). And its three-dimensional function is very weak, only supports very limited pseudo three-dimensional browsing mode, can't carry out high-precision browsing and roaming of three-dimensional scenes, and the drawing of three-dimensional scenes has poor realism.

The innovation of the present invention is embodied in: solving the problem of completing the same functions as the real-time virtual reality and visualization system that can be realized on the existing common personal computer, high-performance graphics workstation and server on the mobile computing device with relatively poor hardware performance, In the background, GPU clusters are used as high-performance computing servers to realize virtual reality and visualization functions, so that the user's experience of virtual reality functions on mobile devices is even more perfect than that on ordinary personal computers and high-performance graphics workstations.

Description of drawings

Fig. 1, the operation flowchart of mobile distributed virtual reality and visualization system of the present invention;

Fig. 2 is a structural diagram of the distributed virtual reality and visualization system on the mobile device of the present invention.

Detailed ways

Below in conjunction with accompanying drawing, the present invention is described in further detail:

The operation process of the mobile distributed virtual reality and visualization system of the present invention is as shown in Figure 1:

1) The general overview of the system is as follows: the client on the mobile device connects with the virtual reality and visualization application server, and sends calculation and operation requests to the virtual reality application system on the virtual reality and visualization application server, and the application system sends the corresponding calculation and execution results are sent to the client on the mobile device, and a virtual reality and visualization application server can provide support and services for multiple mobile devices;

2) According to the results received and displayed by the client on the mobile device, the user interactively controls various functions in the entire virtual reality and visualization system through the graphical user interface. The user interface is divided into two parts: 1) menu, button, tool Graphical interface (or called window main frame) outside the three-dimensional main window view such as bar (2) Interactive interface within the three-dimensional main window view. The above interface is generated by the client on the mobile device, and there is no interactive graphical interface on the server. The interactive interface includes browsing the 3D virtual scene in the virtual reality system through keyboard, mouse, menu graphic interface and other interactive ways, selecting and picking up 3D virtual objects, adding, deleting, modifying 3D objects and their related states and attributes, Various forms of motion control of 3D objects, adjustment of various rendering attributes of the 3D environment (such as lighting, material, shadow, realistic rendering level), visualization of 3D massive data collections, etc. A series of items related to virtual reality systems function and operation;

This system is different from remote control (such as remote desktop connection). The interactive graphical interface of this system is completely in charge of the client on the mobile device. The server only provides background services and does not provide any interactive interface to remote mobile users. The terminal will not generate any graphical user interface, and will not display any interface content on the server. Users on the server can freely use the server to perform other various computing and display tasks, and do not feel the existence of the remote mobile client at all. In addition, the remote control belongs to the technology of the system level, but the present invention belongs to the technology of the application level.

3) Various computing and drawing functions in the virtual reality system are jointly completed by the virtual reality and visualization application server and various server clusters in the background.

● Among them, the data server is responsible for transferring data to the application server, and the cooperation and scheduling between the data servers are realized through the coordination server. In the distributed virtual reality platform, the data server is responsible for the storage and distribution of massive terrain data, visualization data, model data, etc. The task of the data server is heavy, so there can be multiple data servers in the platform, and the data server balances the load by coordinating the servers. When the virtual environment data requested by the application server is not in a certain server currently connected, the coordinating server will calculate the location of the data in the distributed environment according to the request of the user to browse or visualize graphics, images, multimedia and other data (ie On which data server), the application server establishes a connection with the data server storing the data, and acquires and transmits the data.

●Because in the distributed virtual reality system, there will be multiple clients accessing and controlling the whole virtual reality at the same time, the coordination of the actions of the clients in the virtual reality environment is handled by the central server (coordination, synchronization, load balancing server).

●The virtual reality and visualization application server mainly completes the management and calculation of the virtual reality system and the rendering of the 3D environment. When completing the above functions, if the data to be used is not on the current server, it will send a request to the data server for relevant data , after the data is obtained, calculation and drawing are performed. For the rendering of the 3D environment, the hardware structure of the application server can be configured with multiple GPUs on one host, or multiple hosts (one or more GPUs on each host) can be used as rendering nodes to form an asynchronous GPU cluster , the software environment adopts a parallel drawing strategy, and generally adopts a sort-first parallel drawing strategy. The drawing action performed on the virtual reality and visualization application server is "virtual drawing", which means that the drawing will not have any image output displayed on the display terminal (such as a monitor, projector, etc.), but output to a virtual image Space (such as FBO frame buffer object, etc.), the virtual drawn image results are combined on the application server, and then compressed and encoded by the application server and sent to the mobile client.

The system module structure of the distributed virtual reality and visualization system on the mobile device of the present invention is shown in FIG. 2 . The whole system consists of two parts, namely the mobile device client and several virtual reality and visualization application servers. The mobile device client runs on the mobile device, and the virtual reality and visualization server runs on the virtual reality and visualization application server.

1) Graphical user interface: The graphical interface part provides various interfaces for users to interact on the mobile client, including menus, buttons, toolbars, dialog boxes, shortcut keys, 3D navigation graphical interfaces, and various interactions in 3D scenes Through the above-mentioned various interactive interface tools, users can freely manipulate various objects in the virtual reality and visualization system and the three-dimensional virtual scene. (The graphical interface is developed for the user and can be developed using any mobile phone graphical user interface development tool).

2) User interaction command parsing and sending module: This module is first responsible for receiving the user's operation actions and parsing them into interactive commands for the entire virtual reality system. Since the mobile client itself does not actually execute these commands, the operations in these virtual reality systems The execution of commands is actually completed by the virtual reality and visualization application server, so this module is responsible for packaging these interactive commands into data packets and sending them to the virtual reality and visualization application server through the wireless network.

3) User command receiving module: there is a dedicated data buffer storage area on the server side for receiving interactive control commands sent by the mobile client.

4) Window main frame user command parsing module: According to the different types of user commands, if the command is a command on the main window frame of the graphical interface of the mobile client, then there is a corresponding graphical interface command message response function in this module , parse the command, and transfer the execution task of the command to the distributed virtual reality and visualization system.

5) 2D to 3D mapping and analysis module of the 3D main window: if the user's interactive control command is within the view of the 3D main window, the format of the transmitted command is the coordinates of the 2D view (for example, if the click is on the main window A certain point within the view, then transmit the pixel-level coordinates of the point), using the function of the 2D to 3D inverse mapping, you can obtain the corresponding object in the 3D space, and according to the context of the action in the window , analyze the actual operation actions on the three-dimensional object (such as dragging, moving, etc.), and after analyzing the corresponding actions, through the message passing mechanism, the distributed virtual reality and visualization system responds to the message and executes the relevant operation command.

5) Distributed virtual reality and visualization system: the real-time operation system executes relevant instructions to complete various computing tasks related to the virtual reality system (including scene management, distributed data management, system resource management, IO management, physical engine , collision detection system, scripting engine, computer animation, artificial intelligence, network engine, etc.)

6) GPU cluster parallel virtual rendering subsystem:

The rendering subsystem is a part of the virtual reality and visualization system, and the GPU cluster is used for rendering acceleration processing in the present invention.

Since the task of drawing 3D graphics in the virtual reality and visualization system is the heaviest and often becomes the bottleneck of the system operation, we will separate the drawing step from the virtual reality and visualization system, and wait for all the virtual reality and visualization After the computing task is completed, the parallel virtual rendering subsystem constructed by the GPU cluster completes the entire rendering work. The parallel virtual rendering subsystem adopts the sort-first strategy, that is, first divides the scene into 4 sub-window areas according to the corresponding display window areas (the size of the 4 sub-windows is not evenly divided, and is dynamically adjusted according to the size of the display task. ), assign the drawing tasks of the 4 sub-windows to 4 GPUs for parallel drawing, so that the amount of drawing tasks undertaken by each GPU will be reduced, and the overall drawing efficiency will be improved. At the same time, according to the different performances of the four GPU hardware itself, it is also possible to dynamically allocate the corresponding GPU to perform the drawing task by dividing the geometric drawing task amount in the window area according to the adaptive adjustment strategy.

7) Post-processing module for image generation by parallel drawing: After the images are generated by the 4 GPUs, it is necessary to perform recycling work in the virtual reality and visualization system to retrieve all the scattered images and splicing them into a complete image. (Known that it can be realized)

8) Streaming media compression and sending module: This module is responsible for compressing and encoding the image into a frame of streaming media (the compression standard can adopt various standard encoding forms such as H.263, H.264 or MPEG-4), after passing through the frame After intra-compression and inter-frame compression, it is sent to the mobile device client via a communication connection.

9) Streaming media reception and decompression module: After receiving the video frame, the module on the mobile device side decodes and restores a frame of image according to the standard decoding format.

10) Streaming media rendering module: a frame of image is displayed in real time by means of rendering, so that the user can see the same virtual scene and visualization results in real time as the virtual reality system calculated and displayed by the high-performance computer platform.

Claims (5)

1. realize distributed virtual reality and visualization method on a mobile device, the steps include:

1) sets up the communication connection of mobile device and virtual reality and visualization server; Wherein, mobile device comprises a graphical interface of user and carries out the client of virtual reality and visualization display, virtual reality and visualization server comprise a virtual reality and visualization system, and described graphical interface of user comprises: the container of interactive means in menu, button, toolbar, dialog box, shortcut, three-dimensional navigation graphical interfaces and the three-dimensional scenic;

2) mobile device receives the interactive operation request command by described graphical interface of user, and it is resolved to the interactive command of virtual reality and visualization system;

3) mobile device is packaged into Packet Generation to virtual reality and visualization server with the interactive command of resolving;

4) virtual reality and visualization server parse interactive command from packet, and send it to described virtual reality and visualization system;

5) virtual reality and visualization system are carried out corresponding calculating according to interactive command, then carry out virtually drawing and are that Streaming Media sends to mobile device with the execution result compressed encoding; Described virtual reality and visualization system comprise one by the parallel virtually drawing subsystem of GPU cluster structure; After these all virtual realities and Visual calculation task are finished, described parallel virtually drawing subsystem adopts the sort-first strategy to carry out the 3-D graphic virtually drawing, this virtually drawing outputs to the virtual image space with image result, and the image result that virtually drawing goes out is through making up in virtual reality and visualization server;

6) mobile device shows virtual scene and visualization result to receiving decoding data.

2. the method for claim 1, it is characterized in that then virtual reality and visualization server are resolved this order according to the graphical interfaces command messages response function of setting if the interactive operation request command is the order outside the three-dimensional main window view of described graphical interface of user.

3. the method for claim 1, it is characterized in that if the interactive operation request command is the order within the three-dimensional main window view of described graphical interface of user, its command format is the coordinate of two dimension view, then virtual reality and visualization server with this two dimension view coordinate format command mapping be in the corresponding three-dimensional space object and according to action forward-backward correlation within this window, parse the operational motion order to three dimensional object.

4. realize distributed virtual reality and visual system on a mobile device, it is characterized in that comprising the server end on client, virtual reality and the visualization server on the mobile device; Described mobile device be connected virtual reality and be connected by communication network with visualization server; Wherein, described client comprises a graphical interface of user, a command analysis and sending module, stream media decoding module, Streaming Media drafting module; Described server end comprises a virtual reality and visualization system, orders receiver module, Streaming Media compressed encoding module;

Described graphical interface of user is used for receiving the interactive operation request command; Described graphical interface of user comprises: the container of interactive means in menu, button, toolbar, dialog box, shortcut, three-dimensional navigation graphical interfaces and the three-dimensional scenic;

Described command analysis and sending module are used for the interactive operation request command is resolved to the interactive command of virtual reality and visualization system, and it is packaged into Packet Generation to described order receiver module;

Described order receiver module is used for the packet that receives is stored and resolved, and the interactive command that parses is sent to described virtual reality and visualization system;

Described virtual reality and visualization system are used for carrying out corresponding calculating and virtually drawing according to interactive command, and execution result are sent to Streaming Media compressed encoding module; Described virtual reality and visualization system comprise one by the parallel virtually drawing subsystem of GPU cluster structure; After these all virtual realities and Visual calculation task are finished, described parallel virtually drawing subsystem adopts the sort-first strategy to carry out the 3-D graphic virtually drawing, this virtually drawing outputs to the virtual image space with image result, and the image result that virtually drawing goes out is through making up in virtual reality and visualization server;

Described Streaming Media compressed encoding module, being used for execution result is carried out compressed encoding is Streaming Media, and sends it to described stream media decoding module;

Described stream media decoding module is used for receiving decoding data;

Described Streaming Media drafting module is used for decoded data are drawn demonstration virtual scene and visualization result.

5. system as claimed in claim 4 is characterized in that described server end adopts separate server framework or distributed server architecture.

Images