CN116055540A

CN116055540A - Virtual content display system, method, apparatus and computer readable medium

Info

Publication number: CN116055540A
Application number: CN202310062280.8A
Authority: CN
Inventors: 周清会; 杨迪; 杨婷; 于丽莎; 翁开宇
Original assignee: Beijing Manheng Digital Technology Co ltd
Current assignee: Beijing Manheng Digital Technology Co ltd
Priority date: 2023-01-13
Filing date: 2023-01-13
Publication date: 2023-05-02
Anticipated expiration: 2043-01-13
Also published as: CN116055540B

Abstract

The application provides a display system, a method, a server, a client and a medium based on virtual reality, wherein the system comprises the server and the client; the server is used for creating a memory space and a mapping pointer of the memory space; the client is used for acquiring the mapping pointer, creating a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, and binding the texture pointer to a shader resource; the server side is used for obtaining the texture pointer through the mapping pointer, directly writing the obtained texture data into the texture pointer, so that the client side draws, renders and displays the texture data, and the technical problems that a desktop all-in-one machine is displayed in a plug flow mode in the related art, so that serious performance waste, easiness in clamping and unsmooth pictures are caused can be at least solved.

Description

Virtual content display system, method, apparatus and computer readable medium

Technical Field

The present disclosure relates to the field of information technologies, and in particular, to a system, a method, an apparatus, and a computer readable medium for displaying virtual content.

Background

The desktop all-in-one machine is a computer which combines a main case of a traditional desktop computer with a display, and has the appearance of only one display and auxiliary equipment such as a mouse, a keyboard and the like. In order to enrich the content resources of the desktop all-in-one machine, the desktop all-in-one machine can be made to be compatible with the content of the head-mounted display equipment, such as the content of a VR helmet at a PC end.

Specifically, a SteamVR platform is installed in the desktop all-in-one machine, and after the desktop all-in-one machine calculates data and renders graphics, the display and real-time interaction of the 3D stereoscopic effect are completed by means of a plug flow mode.

However, the inventors found that there are at least the following technical problems in the related art:

by adopting a plug flow mode, data is received and sent in the desktop all-in-one machine, the occupancy rate of the CPU and the GPU exceeds 90%, the plug flow process causes serious performance waste, the interactive operation process of the desktop all-in-one machine is easy to cause clamping, and the stereoscopic picture displayed by the desktop all-in-one machine is not smooth. .

Disclosure of Invention

An object of the present application is to provide a display system, a method, a server, a client and a medium based on virtual reality, which are at least used for solving the technical problems of serious performance waste, unsmooth picture and easy blocking caused by displaying by using a push stream mode of a desktop integrated machine in the related art.

To achieve the above object, some embodiments of the present application provide a virtual reality-based display system, which includes a server side and a client side; the server is used for creating a memory space and a mapping pointer of the memory space; the client is used for acquiring the mapping pointer, creating a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, and binding the texture pointer to a shader resource; the server side is used for obtaining the texture pointer through the mapping pointer, and directly writing the obtained texture data into the texture pointer so that the client side draws, renders and displays the texture data.

Some embodiments of the present application further provide a server, where the server is configured to create a memory space and a mapping pointer of the memory space; the server side is further configured to obtain the mapping pointer at the client side, create a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, bind the texture pointer to a shader resource, obtain the texture pointer through the mapping pointer, and directly write the obtained texture data into the texture pointer, so that the client side draws, renders and displays the texture data.

Some embodiments of the present application further provide a client, where the client is configured to, after the server creates a memory space and a mapping pointer of the memory space, obtain the mapping pointer, create a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, and bind the texture pointer to a shader resource.

Some embodiments of the present application further provide a virtual reality-based display method, applied to the display system described above, the method including: creating a memory space and a mapping pointer of the memory space; acquiring the mapping pointer, creating a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, and binding the texture pointer to a shader resource; and acquiring the texture pointer through the mapping pointer, and directly writing the acquired texture data into the texture pointer so that the client draws, renders and displays according to the texture data.

Some embodiments of the present application also provide a virtual reality-based display device, the device comprising: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the method as described above.

Some embodiments of the present application also provide a computer readable medium having stored thereon computer program instructions executable by a processor to implement the described display method.

Compared with the prior art, the scheme provided by the embodiment of the application omits a plug flow process in the related art, and can directly acquire texture data from the client and display a 3D stereoscopic effect. The results of the inventor after running the same project cases show that compared with the system provided by the related technology, the system provided by the embodiment of the application has the advantages that the occupancy rate of the CPU and the GPU is obviously reduced, so that the resource occupancy is saved, and the performance is obviously improved; the interactive operation process of the desktop all-in-one machine is not easy to cause clamping and is higher in FPS frame rate of the display picture of the desktop all-in-one machine, and the picture is smoother and smoother.

Drawings

FIG. 1 is an exemplary interaction diagram of a virtual content display system according to an embodiment of the present application;

FIG. 2 is an exemplary interactive schematic diagram of a display system of virtual content in the related art;

fig. 3 is a process flow diagram of a method for displaying virtual content according to an embodiment of the present application;

fig. 4 is a schematic structural connection diagram of a device for virtual content according to an embodiment of the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

The following terms are used herein.

Plug flow refers to the process of transmitting the packaged content of the acquisition phase to the server.

A pointer, essentially a mapping of addresses to units.

The SteamVR plug-in is used for tracking the device location. The handle and the helmet are both provided with the assembly.

Texture data is a two-dimensional array of image data. The individual data elements of a texture are called texels, which are determined by the basic format and data type.

Shared memory refers to allowing two unrelated processes to access the same memory.

FFmpeg technology is a set of open source computer programs that can be used to record, convert digital audio, video, and convert it into streams.

FPS, english acronym Frames Per Second, number of frames per second transmitted. Refer to the number of frames per second transmitted from a picture surface, and refer to the number of pictures of an animation or video in popular terms.

Example 1

The embodiment of the application provides a display system based on virtual reality, which comprises a server side and a client side as shown in fig. 1;

the server is used for creating a memory space and a mapping pointer of the memory space;

the client is used for acquiring the mapping pointer, creating a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, and binding the texture pointer to a shader resource;

the server side is used for obtaining the texture pointer through the mapping pointer, and directly writing the obtained texture data into the texture pointer so that the client side draws, renders and displays the texture data.

The server is provided with a SteamVR plug-in, and is rendering equipment, and can be a desktop integrated machine.

The client is a display end and can be a desktop integrated machine, and the client can display a 3D stereoscopic effect.

Specifically, in the step of creating the memory space and the mapping pointer of the memory space at the server, when the server is initialized, a part of memory space with a fixed size is created through a memory sharing mechanism, so as to obtain the mapping pointer corresponding to the memory space, where the mapping pointer is capable of being shared across processes, and thus, the client can also read the mapping pointer.

In some examples, the server may initialize by default an interface program provided by the OpenVR driver after the stepvr plug-in installed on the server is started.

Specifically, after the server creates the memory space and the mapping pointer of the memory space, the client may directly obtain the mapping pointer, then create a texture pointer for sharing texture data in the memory space corresponding to the mapping pointer, and bind the texture pointer to the shader resource.

The created texture pointer for sharing the texture data is used for storing the shared texture data. In some examples, the texture pointer may include specific data that is used to make the display on the display end a fixed, system default screen, such as a blue screen, a black and white striped screen, and so forth.

Specifically, after the client acquires the mapping pointer, creates a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, and binds the texture pointer to a shader resource, the server may acquire the texture pointer through the mapping pointer.

Further, in some examples, the server may poll and read texture data in the stepvr plug-in according to the texture pointer, and when detecting a state change of the texture data in the stepvr plug-in, obtain the changed texture data in the stepvr plug-in, and may copy and write the texture data directly into the texture pointer. At this time, since the texture pointer is bound to the shader resource, in the executed client program, the GPU of the client will default to retrieve the texture data corresponding to the shader resource, so as to draw, render and display, where the effect of the display is a 3D stereoscopic effect.

The texture data refers to image data seen by a user, such as image frames seen by left and right eyes of the user after wearing the VR headset.

In some examples, after the server directly copies and writes the texture data to the texture pointer, the system converts the default picture into a 3D stereoscopic picture which is drawn, rendered and displayed according to the texture data.

As will be appreciated by those skilled in the art, in the related art, it is often necessary to complete the display of the 3D stereoscopic effect by means of plug flow, as can be seen in fig. 2.

The display system in the related art comprises a server and a client, wherein a SteamVR plug-in is installed on the server. The server is rendering equipment, such as a desktop all-in-one machine. The client is a display end and can be a desktop integrated machine.

Specifically, first, a UDP network connection is established between the server and the client. After successful connection, the client is equivalent to a helmet and handle that simulates a VR. Then, the server acquires texture data in the SteamVR, encodes and compresses the texture data in the RGBA format into a video stream through the FFmpeg technology, converts the texture data in the RGBA format into the texture data in the YUV format at the moment, and then sends the video stream to the client.

And finally converting the texture data in the YUV format into the texture data in the RGBA format by a GPU programming mode, wherein the texture data in the RGBA format can be displayed on the client, so that the 3D stereoscopic effect can be rendered and displayed based on the texture data in the RGBA format.

Therefore, the whole plug flow process in the related technology comprises the steps of UDP network data transmission, encoding of texture data by a server side, receiving of the texture data by a client side, decoding and the like, the occupancy rate of the CPU and the GPU exceeds 90%, serious performance waste is caused, and the interactive operation process of the desktop all-in-one machine is easy to cause clamping and blocking, so that pictures displayed by the desktop all-in-one machine are not smooth.

Compared with the related art, the virtual reality-based display system provided by the embodiment of the application omits a plug flow process in the related art, and can directly acquire texture data from the GPU of the client and display a 3D stereoscopic effect. The results of the inventor after running the same project cases show that compared with the system provided by the related technology, the system provided by the embodiment of the application has the advantages that the occupancy rate of the CPU and the GPU is obviously reduced, so that the resource occupancy is saved, and the performance is obviously improved; the interactive operation process of the desktop all-in-one machine is not easy to cause clamping and is higher in FPS frame rate of pictures displayed by the desktop all-in-one machine, and the pictures are smoother and smoother.

Example two

In some embodiments of the present application, the server is specifically configured to create, according to instruction information, a memory space having a preset memory size and a mapping pointer of the memory space.

In some examples, the memory space of the predetermined memory size is a memory space of 256 bytes.

Specifically, the instruction information may be issued by an associated worker to specify the size of the memory space. The size of the memory space can be flexibly configured according to actual needs, which is not particularly limited in the embodiment of the present application.

Example III

In some embodiments of the present application, the client is specifically configured to obtain the mapping pointer, create a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, where a content of the texture pointer is null, and bind the texture pointer whose content is null to a shader resource.

That is, after the server creates the memory space and the mapping pointer of the memory space, the client may directly obtain the mapping pointer, and then create a texture pointer for sharing texture data in the GPU based on the memory space corresponding to the mapping pointer, where the content of the texture pointer is Null (Null).

Example IV

In some embodiments of the present application, as shown in fig. 1, the client is further configured to receive operation data of a user, and write the operation data into a memory space corresponding to the mapping pointer after receiving the operation data of the user;

the server is further configured to obtain the operation data through the mapping pointer.

Specifically, after the client writes the real-time operation data into the memory space corresponding to the mapping pointer, the server can acquire the operation data through the mapping pointer, and complete the simulation operation according to the operation data. For example, the simulated operation may be a VR helmet and handle based simulated operation.

In some examples, the operational data may include, but is not limited to, user-based operational information of a keyboard, mouse, handle, etc.

It is readily appreciated that embodiments of the present application may be understood as embodiments of the interaction phase after the system is displayed. According to the virtual reality-based display system provided by the embodiment of the application, due to the fact that a plug flow process in the related technology is omitted, texture data can be directly obtained from the GPU of the client and the 3D stereoscopic effect is displayed, real-time interaction is performed based on the display effect, the performance is obviously improved, the clamping and the stopping are not easy to occur in the interactive operation process of the desktop all-in-one machine, the FPS frame rate of a picture displayed by the desktop all-in-one machine is higher, the picture is smoother, and the interactive experience of a user is facilitated to be improved.

It should be noted that, the first embodiment may be implemented in cooperation with one or more of the second to fourth embodiments, which is not specifically limited herein.

In summary, the solution provided by the embodiment of the present application omits the plug flow process in the related art, and may directly obtain texture data from the GPU of the client and display the 3D stereoscopic effect. The results of the inventor after running the same project cases show that compared with the system provided by the related technology, the system provided by the embodiment of the application has the advantages that the occupancy rate of the CPU and the GPU is obviously reduced, so that the resource occupancy is saved, and the performance is obviously improved; the interactive operation process of the desktop all-in-one machine is not easy to cause clamping and is higher in FPS frame rate of pictures displayed by the desktop all-in-one machine, and the pictures are smoother and smoother.

Example five

The embodiment of the application provides a server side, which is used for creating a memory space and a mapping pointer of the memory space;

the server side is further configured to obtain the mapping pointer at the client side, create a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, bind the texture pointer to a shader resource, obtain the texture pointer through the mapping pointer, and directly write the obtained texture data into the texture pointer, so that the client side draws, renders and displays the texture data.

It is to be noted that, in this embodiment of the present application, an embodiment corresponding to a server in a display system based on virtual reality, specific implementation details based on the server have been described in detail in the foregoing embodiment, and in order to avoid repetition, details are not repeated here.

Example six

The embodiment of the application provides a client which is used for acquiring a mapping pointer after a memory space and the mapping pointer of the memory space are created by a server, creating a texture pointer for sharing texture data in the memory space corresponding to the mapping pointer, and binding the texture pointer to a shader resource.

It is to be noted that, in this embodiment of the present application, an embodiment corresponding to a client in a display system based on virtual reality, specific implementation details based on the client have been described in detail in the foregoing embodiment, and in order to avoid repetition, details are not repeated here.

Example seven

The embodiment of the application also provides a display method based on virtual reality, which is applied to the display system described in any one of the embodiments, as shown in fig. 3, and the method may include the following steps:

step S101, creating a memory space and a mapping pointer of the memory space;

step S102, obtaining the mapping pointer, creating a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, and binding the texture pointer to a shader resource;

step S103, the texture pointer is obtained through the mapping pointer, and the obtained texture data is directly written into the texture pointer, so that the client draws, renders and displays according to the texture data.

It is to be noted that, in this embodiment of the present application, the implementation details described in the first to fourth embodiments are equally applicable here, and are not repeated here for avoiding repetition.

Example eight

The embodiment of the application further provides a display device based on virtual reality, the structure of which is shown in fig. 4, the device includes a memory 11 for storing computer readable instructions and a processor 12 for executing the computer readable instructions, wherein the computer readable instructions, when executed by the processor, trigger the processor to execute the method for distributing virtual content.

The methods and/or embodiments of the present application may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. The above-described functions defined in the method of the present application are performed when the computer program is executed by a processing unit.

It should be noted that, the computer readable medium described in the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

In the present application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present application may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The flowchart or block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more computer readable instructions executable by a processor to implement the steps of the methods and/or techniques of the various embodiments of the present application described above.

In a typical configuration of the present application, the terminals, the devices of the services network each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer-readable media include both permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices, or any other non-transmission medium which can be used to store information that can be accessed by a computing device.

In addition, the embodiment of the application also provides a computer program which is stored in the computer equipment, so that the computer equipment executes the method for executing the control code.

It should be noted that the present application may be implemented in software and/or a combination of software and hardware, for example, using Application Specific Integrated Circuits (ASIC), a general purpose computer or any other similar hardware device. In some embodiments, the software programs of the present application may be executed by a processor to implement the above steps or functions. Likewise, the software programs of the present application (including associated data structures) may be stored on a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. In addition, some steps or functions of the present application may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions.

It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the apparatus claims can also be implemented by means of one unit or means in software or hardware. The terms first, second, etc. are used to denote a name, but not any particular order.

Claims

1. A display system based on virtual reality, which is characterized by comprising a server and a client;

2. The display system according to claim 1, wherein the server is specifically configured to create a memory space having a preset memory size and a mapping pointer of the memory space according to the instruction information.

3. The display system according to claim 1, wherein the client is specifically configured to obtain the mapping pointer, create a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, wherein a content of the texture pointer is null, and bind the texture pointer whose content is null to a shader resource.

4. A display system according to any one of claims 1 to 3,

the client is further configured to receive operation data of a user, and write the operation data into a memory space corresponding to the mapping pointer after receiving the operation data of the user;

5. The server is characterized in that the server is used for creating a memory space and a mapping pointer of the memory space;

6. The client is characterized in that the client is used for acquiring a mapping pointer after a memory space and the mapping pointer of the memory space are created by a server, creating a texture pointer for sharing texture data in the memory space corresponding to the mapping pointer, and binding the texture pointer to a shader resource.

7. A virtual reality based display method applied to the display system of any one of claims 1 to 4, the method comprising:

creating a memory space and a mapping pointer of the memory space;

acquiring the mapping pointer, creating a texture pointer for sharing texture data in a memory space corresponding to the mapping pointer, and binding the texture pointer to a shader resource;

and acquiring the texture pointer through the mapping pointer, and directly writing the acquired texture data into the texture pointer so that the client draws, renders and displays according to the texture data.

8. A virtual reality-based display device, the device comprising:

one or more processors; and

a memory storing computer program instructions that, when executed, cause the processor to perform the method of claim 7.

9. A computer readable medium having stored thereon computer program instructions executable by a processor to implement the method of claim 7.