CN111913674B - Virtual content display method, device, system, terminal equipment and storage medium - Google Patents

Abstract

The embodiment of the application discloses a method, a device, a system, terminal equipment and a storage medium for displaying virtual content, and relates to the technical field of display. The method for displaying the virtual content is applied to terminal equipment, the terminal equipment is in communication connection with an interaction device, the interaction device comprises an interaction area, and the method for displaying the virtual content comprises the following steps: acquiring content to be displayed, wherein the content to be displayed comprises non-interactive content and interactive content; acquiring first content data corresponding to non-interactive content; acquiring relative spatial position information between an interaction device and terminal equipment; generating virtual extension content according to the relative spatial position information and the first content data; when the interactive picture corresponding to the interactive content is displayed, the virtual extended content is displayed, the first display area of the interactive picture corresponds to the interactive area, and the second display area of the virtual extended content corresponds to the set area outside the interactive area. The method can improve the display effect of the display content.

Description

Virtual content display method, device, system, terminal device and storage medium

Technical Field

The present application relates to the field of display technology, and more specifically, to a method, apparatus, system, terminal device and storage medium for displaying virtual content.

Background technique

At present, with the rapid development of multimedia technology, more and more smart mobile terminals (such as PDAs, smart phones, smart watches, etc.) have entered people's lives, and they are deeply loved by people due to their small size and easy to carry. However, in the use of traditional smart mobile devices, the content displayed by smart mobile devices is usually limited by the screen size of the smart mobile devices, making the displayed content not rich and complete enough.

Summary of the invention

The embodiments of the present application provide a method, apparatus, system, terminal device and storage medium for displaying virtual content, which can improve the display effect of the displayed content.

In a first aspect, an embodiment of the present application provides a method for displaying virtual content, which is applied to a terminal device, wherein the terminal device is communicatively connected to an interactive device, and the interactive device includes an interactive area. The method includes: obtaining content to be displayed, wherein the content to be displayed includes non-interactive content and interactive content; obtaining first content data corresponding to the non-interactive content; obtaining relative spatial position information between the interactive device and the terminal device; generating virtual extended content based on the relative spatial position information and the first content data; when an interactive screen corresponding to the interactive content is displayed, displaying the virtual extended content, wherein the first display area of the interactive screen corresponds to the interactive area, and the second display area of the virtual extended content corresponds to a set area outside the interactive area.

In a second aspect, an embodiment of the present application provides a display device for virtual content, which is applied to a terminal device, wherein the terminal device is communicatively connected to an interactive device, and the interactive device includes an interactive area. The device includes: a content acquisition module, a data acquisition module, a position acquisition module, a content generation module, and a display control module, wherein the content acquisition module is used to acquire content to be displayed, and the content to be displayed includes non-interactive content and interactive content; the data acquisition module is used to acquire first content data corresponding to the non-interactive content; the position acquisition module is used to acquire relative spatial position information between the interactive device and the terminal device; the content generation module is used to generate virtual extended content based on the relative spatial position information and the first content data; the display control module is used to display the virtual extended content when the interactive screen corresponding to the interactive content is displayed, the first display area of the interactive screen corresponds to the interactive area, and the second display area of the virtual extended content corresponds to a set area outside the interactive area.

In a third aspect, an embodiment of the present application provides a virtual content display system, the system comprising a terminal device and an interactive device, the terminal device being communicatively connected to the interactive device, the interactive device comprising an interactive area, wherein the interactive device is used to obtain first content data corresponding to non-interactive content in the content to be displayed according to the content to be displayed in the interactive area, the content to be displayed including non-interactive content and interactive content, send the first content data to the terminal device, and control the interactive area to display an interactive screen corresponding to the interactive content; the terminal device is used to obtain relative spatial position information between the interactive device and the terminal device, receive the first content data, generate virtual extended content according to the relative spatial position information and the first content data, and display the virtual extended content, the display area of the virtual extended content corresponds to a set area outside the interactive area.

In a fourth aspect, an embodiment of the present application provides a terminal device, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by one or more processors, and the one or more programs are configured to execute the virtual content display method provided in the first aspect above.

In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method for displaying virtual content provided in the first aspect above.

In the solution provided by the embodiment of the present application, the final draft device obtains content to be displayed, which includes non-interactive content and interactive content, and then obtains first content data corresponding to the non-interactive content, and obtains relative spatial position information between the interactive device and the terminal device, so as to generate virtual extended content according to the relative spatial position information and the first content data. When the interactive screen corresponding to the interactive content is displayed, the virtual extended content is displayed, the first display area of the interactive screen corresponds to the interactive area, and the second display area of the virtual extended content corresponds to the set area outside the interactive area of the interactive device, so that the virtual extended content corresponding to the non-interactive content in the displayed content is displayed in the virtual space according to the spatial position of the interactive device, so that the user can see the effect of the virtual extended content corresponding to the non-interactive content superimposed outside the interactive area, and the displayed interactive content corresponds to the interactive area, which expands the display space of the displayed content, improves the display effect of the displayed content, and is more conducive to the user to interact with the displayed content.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 shows a schematic diagram of an application environment suitable for an embodiment of the present application.

FIG. 2 shows a schematic diagram of another application scenario applicable to an embodiment of the present application.

FIG. 3 shows a flow chart of a method for displaying virtual content according to an embodiment of the present application.

4A-4B show a schematic diagram of a display effect according to an embodiment of the present application.

FIG. 5 shows a flow chart of a method for displaying virtual content according to another embodiment of the present application.

FIG. 6 shows a schematic diagram of a display effect according to an embodiment of the present application.

FIG. 7 shows another schematic diagram of display effect according to an embodiment of the present application.

FIG. 8 shows a schematic diagram of another display effect according to an embodiment of the present application.

FIG. 9 shows a block diagram of a device for displaying virtual content according to an embodiment of the present application.

FIG. 10 is a block diagram of a terminal device for executing a method for displaying virtual content according to an embodiment of the present application.

FIG. 11 is a block diagram of an interactive device for executing a method for displaying virtual content according to an embodiment of the present application.

FIG. 12 is a storage unit according to an embodiment of the present application for storing or carrying program codes for implementing a method for displaying virtual content according to an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

With the rapid improvement of technology and living standards, mobile terminals (such as smart phones, tablet computers, etc.) have become popular. Mobile terminals are popular because of their small size and portability. When used by users, mobile terminals usually display content on the touch screen, such as multimedia images, application interfaces, file contents, etc. Since the size of the mobile terminal screen is limited, the content displayed is also limited by the size of the screen.

After long-term research, the inventors have proposed a method, device, system, terminal device and storage medium for displaying virtual content in the embodiments of the present application. According to the spatial position of the interactive device, the non-interactive content in the display content is converted into virtual extended content and displayed in the virtual space through augmented reality, so that the user can see the effect of the virtual extended content displayed outside the interactive area of the interactive device, thereby improving the display effect of the display content corresponding to the interactive area.

The following introduces the application scenarios of the method for displaying virtual content provided in the embodiments of the present application.

Please refer to FIG. 1 , which shows a virtual content display system 10 provided in an embodiment of the present application, including a terminal device 100 and an interaction apparatus 200 , wherein the terminal device 100 is in communication connection with the interaction apparatus 200 .

In the embodiment of the present application, the terminal device 100 may be a head-mounted display device, or a mobile device such as a mobile phone or a tablet. When the terminal device 100 is a head-mounted display device, the head-mounted display device may be an integrated head-mounted display device. The terminal device 100 may also be a smart terminal such as a mobile phone connected to an external/accessible head-mounted display device, that is, the terminal device 100 may be used as a processing and storage device for the head-mounted display device, inserted into or accessed into an external head-mounted display device, and display virtual content through the head-mounted display device.

In the embodiment of the present application, the interactive device 200 may be an electronic device provided with a marker 201. The number of markers 201 provided on the interactive device 200 is not limited, and the number of markers 201 may be one or more. The specific morphological structure and size of the interactive device 200 are not limited, and may be of various shapes, such as square, circle, or various forms, such as plane.

In some embodiments, the marker 201 may be attached or integrated on the interactive device 200, or may be provided on a protective cover of the interactive device 200, or may be an external marker that can be inserted into the interactive device 200 via a USB (Universal Serial Bus) or headphone jack when in use. If a display screen is provided on the interactive device 200, the marker 201 may also be displayed on the display screen of the interactive device 200.

In some embodiments, the interactive device 200 can interact with the terminal device 100 for information and instructions. The terminal device 100 and the interactive device 200 can be connected via wireless communication methods such as Bluetooth, WiFi (Wireless-Fidelity), ZigBee (Zifeng Technology), or can be connected via a USB interface for wired communication. Please refer to Figure 2. When the terminal device 100 is a head-mounted display device and the interactive device 200 is a mobile terminal or a tablet computer, the head-mounted display device is connected to the tablet computer and the mobile terminal for wired communication via a USB interface. Of course, the connection method between the terminal device 100 and the interactive device 200 is not limited in the embodiments of the present application.

When the terminal device 100 and the interactive device 200 are used together, the marker 201 can be located within the visual range of the image sensor on the terminal device 100 to capture an image containing the marker 201, and then the marker 201 is identified and tracked to achieve positioning tracking of the interactive device 200.

In some embodiments, the terminal device 100 can also realize the positioning tracking of the interactive device 200 according to the light spots set on the interactive device 200, wherein the light spots can be light spot arrays.

In some embodiments, at least one interactive area 202 is provided on the interactive device 200, and the user can perform relevant control and interaction through the interactive area 202. Among them, the interactive area 202 may include a button, a touch pad, or a touch screen. The interactive device 200 can generate a control instruction corresponding to the control operation through the control operation detected by the interactive area 202, and perform relevant control. The interactive device 200 can also send the control instruction to the terminal device 100, or the interactive device 200 generates operation data according to the operation detected in the interactive area, and sends the operation data to the terminal device 100. When the terminal device 100 receives the control instruction sent by the interactive device 200, it can control the display of the virtual content according to the control instruction (for example, controlling the rotation and displacement of the virtual content, etc.).

In a specific embodiment, the terminal device 100 is a head-mounted display device. By wearing the head-mounted display device, the user can observe the virtual chat interface 301 superimposed on the interactive area 202 of the interactive device 200 in the real space, and can also observe the non-interactive content (such as historical chat records 302) in the virtual chat interface 301 displayed outside the interactive area 202, so that the user can view the historical chat records 302 while chatting with the chat object on the virtual chat interface 301, so that the user can view the content through the interactive device 200 without being limited by the size of the interactive area, thereby realizing interactive display of the displayed content.

The following is an introduction to a specific method for displaying virtual content.

Referring to FIG. 3 , an embodiment of the present application provides a method for displaying virtual content, which can be applied to the above-mentioned terminal device, wherein the terminal device is communicatively connected with an interactive device, and the interactive device includes an interactive area. The method may include:

Step S110: Obtain content to be displayed, where the content to be displayed includes non-interactive content and interactive content.

In an embodiment of the present application, the pre-set display area of the above-mentioned content to be displayed corresponds to the interactive area. In one embodiment, the interactive area includes a touch screen, and the content to be displayed may be the content that the interactive device needs to display through the touch screen. In another embodiment, the content to be displayed may also be the content that the terminal device needs to display. When the terminal device displays the content to be displayed, the terminal device may convert the content to be displayed into corresponding virtual content and superimpose it on the interactive area. The user can see the corresponding virtual content superimposed on the interactive device in the real world through the head-mounted display device, so that the display area of the content to be displayed corresponds to the interactive area.

Since the size of the interactive area is limited, the corresponding content to be displayed will be limited by the size of the area, resulting in poor display effect of the content to be displayed. Therefore, the non-interactive content in the above-mentioned content to be displayed can be superimposed and displayed outside the interactive area through augmented reality technology. It is only necessary to make the display area of the interactive content in the content to be displayed correspond to the interactive area, while ensuring the complete display of the content to be displayed, and improving the display effect of the interactive content corresponding to the interactive area. In an embodiment of the present application, the terminal device can obtain the content to be displayed to determine the non-interactive content based on the content to be displayed. Among them, the content to be displayed includes non-interactive content and interactive content.

In some embodiments, the content to be displayed may be content that the interactive device needs to display, and the terminal device obtains the content to be displayed by transmitting the content to be displayed to the terminal device after the interactive device obtains the content to be displayed on the touch screen, so that the terminal device can obtain the content to be displayed on the touch screen. The content to be displayed may be stored in the interactive device, or may be obtained by the interactive device from a server or other electronic device, which is not limited here.

In other embodiments, the content to be displayed may be content that the terminal device needs to display. The content to be displayed may be stored in the terminal device, or may be obtained by the terminal device from a server or other electronic device, or may be obtained by identifying a marker on an interactive device. Of course, the above method of obtaining the content to be displayed is only an example, and the specific content to be displayed may not be limited in the embodiments of the present application. For example, the content to be displayed may also be obtained based on a scene image of the environment in which the terminal device is located that is collected.

In the embodiment of the present application, the above-mentioned content to be displayed can be any content and is not limited here. For example, it can be an application desktop, an application interface (such as a chat interface, a game interface), a multi-level menu page (such as a settings page, a file management page), etc. The above-mentioned non-interactive content is the content that does not need to be interacted with and only needs to be viewed in the current content to be displayed, such as signal strength, power, time, documents, pictures, videos, personal information, preview information, historical chat content, historical email content, historical interactive pages (such as upper menu pages), lower-level interactive pages, etc. The interactive content can be the current interactive interface, operation controls, etc. Of course, the specific content to be displayed, non-interactive content and interactive content can be not limited in the embodiment of the present application.

Step S120: Acquire first content data corresponding to the non-interactive content.

In an embodiment of the present application, the terminal device may obtain the first content data corresponding to the non-interactive content in the content to be displayed based on the above-mentioned content to be displayed, so as to determine the content in the content to be displayed whose display area does not correspond to the interactive area. Among them, when the content to be displayed is displayed by the interactive device, the content to be displayed obtained by the terminal device from the interactive device is the image data of the content to be displayed, and therefore, the first content data corresponding to the non-interactive content may be the image data of the non-interactive content. When the content to be displayed is displayed by the terminal device, the first content data corresponding to the non-interactive content obtained by the terminal device may be the three-dimensional model data corresponding to the non-interactive content, and the three-dimensional model data may include the color, model vertex coordinates, model contour data, etc. of the model corresponding to the three-dimensional model.

In some embodiments, when the content to be displayed is the content that the interactive device needs to display, the terminal device can directly obtain the first content data corresponding to the non-interactive content from the interactive device. As a method, the terminal device can send a data request to the interactive device, and the interactive device can transmit the first content data of the non-interactive content to the terminal device according to the data request, so that the terminal device can obtain the first content data corresponding to the non-interactive content.

Step S130: Obtain the relative spatial position information between the interaction device and the terminal equipment.

In an embodiment of the present application, the terminal device may obtain relative spatial position information between the interactive apparatus and the terminal device, so that the terminal device displays corresponding virtual content according to the relative spatial position information.

As an implementation method, the terminal device can collect images of markers on the interactive device through an image sensor, identify and track the markers in the image, and obtain relative spatial position information between the interactive device and the terminal device. The relative spatial position information between the interactive device and the terminal device includes: relative position information and posture information between the interactive device and the terminal device, and the posture information can be the orientation and rotation angle of the interactive device relative to the terminal device.

In some embodiments, the marker is a pattern with a topological structure, and the topological structure refers to the connectivity relationship between sub-markers and feature points in the marker.

In some embodiments, the interactive device may also be provided with a light spot and an inertial measurement unit (IMU). The terminal device may collect the light spot image on the interactive device through an image sensor and obtain measurement data through the inertial measurement unit. The relative spatial position information between the interactive device and the terminal device may be determined based on the light spot image and measurement data, thereby realizing the positioning and tracking of the interactive device. The light spot provided on the interactive device may be a visible light spot or an infrared light spot, and the number of light spots may be one or a sequence of light spots consisting of multiple light spots.

Of course, the specific method of obtaining the relative spatial position information between the interactive device and the terminal device may not be limited in the embodiments of the present application.

Step S140: Generate virtual extended content according to the relative spatial position information and the first content data.

In an embodiment of the present application, after the terminal device obtains the relative spatial position information between the interactive device and the terminal device, and the first content data corresponding to the above-mentioned non-interactive content, the terminal device can generate virtual extended content corresponding to the above-mentioned non-interactive content based on the relative spatial position information and the first content data, so as to subsequently display the virtual extended content.

In some embodiments, the terminal device can obtain a set area where the virtual extended content needs to be displayed, and then obtain the rendering position of the virtual extended content based on the relative spatial position information between the above-mentioned interactive device and the terminal device, and the relative position relationship between the set area and the interactive device, and then render the virtual content based on the rendering position. The set area refers to the area superimposed in the real space when the virtual extended content is displayed.

Specifically, the terminal device can obtain the position information of the setting area relative to the terminal device according to the relative spatial position information between the interactive device and the terminal device, and the relative position relationship between the setting area and the interactive device, thereby obtaining the spatial position coordinates of the setting area in the real space, and converting the spatial position coordinates into spatial coordinates in the virtual space, wherein the virtual space refers to a three-dimensional space for rendering and displaying virtual extended content. The spatial coordinates of the setting area in the virtual space can be used as the rendering coordinates of the virtual extended content in the virtual space, that is, the rendering position of the virtual extended content is obtained, so as to render the virtual extended content at the rendering position. Among them, the above-mentioned rendering coordinates refer to the three-dimensional spatial coordinates of the virtual extended content in the virtual space. In some embodiments, the above-mentioned setting area can be any area or a preset area outside the interactive area. For example, the setting area can be an area outside the interactive area but adjacent to the interactive area, and the setting area can also be an area outside the interactive area and a set distance from the interactive area. Of course, the specific setting area is not limited in the embodiment of the present application.

It is understandable that after the terminal device obtains the rendering coordinates for rendering the virtual extended content in the virtual space, the terminal device may construct the virtual extended content according to the first content data corresponding to the acquired non-interactive content and the rendering coordinates.

Step S150: when the interactive screen corresponding to the interactive content is displayed, the virtual extended content is displayed, the first display area of the interactive screen corresponds to the interactive area, and the second display area of the virtual extended content corresponds to the set area outside the interactive area.

In the embodiment of the present application, when the interactive screen corresponding to the interactive content is displayed, the terminal device can display the virtual extended content, wherein the first display area of the interactive screen corresponds to the interactive area, and the second display area of the virtual extended content corresponds to the set area outside the interactive area. By superimposing the non-interactive content in the above-mentioned content to be displayed outside the interactive area through augmented reality display technology, the display area of the interactive content of the content to be displayed corresponds to the interactive area, which ensures the complete display of the content to be displayed and improves the display effect of the interactive content corresponding to the interactive area.

Specifically, after the terminal device constructs and renders the virtual extended content, it can convert the virtual extended content into a virtual screen and obtain display data of the virtual screen, which may include the RGB value of each pixel in the display screen and the corresponding pixel coordinates, etc. The terminal device can generate a display screen according to the display data, and project the display screen to display the virtual extended content. Since the terminal device generates the virtual extended content according to the relative position relationship between the above-mentioned set area and the interactive device, the relative spatial position information and the above-mentioned first content data, the virtual extended content is superimposed and displayed on the above-mentioned set area, that is, the display superimposed area of the virtual extended content is the set area outside the interactive area. The user can see the virtual extended content superimposed and displayed on the set area outside the interactive area of the interactive device in the real world through the display lenses of the head-mounted display device, thereby achieving an augmented reality effect.

Since the display area of the virtual extended content is a set area outside the interactive area, the display position of the virtual extended content will not conflict with other content corresponding to the interactive area, so that the user can control the interactive content corresponding to the interactive area while viewing the virtual extended content outside the interactive area, thereby achieving complete display of the content to be displayed and improving the display effect and operation experience of the interactive content corresponding to the interactive area.

In some embodiments, the interactive screen corresponding to the above-mentioned interactive content is displayed, and the terminal device may convert the interactive content into an interactive screen and display it superimposed on the interactive area. Specifically, the terminal device may obtain the second content data corresponding to the interactive content to construct the interactive screen corresponding to the interactive content according to the second content data; the terminal device may obtain the rendering coordinates of the interactive screen in the virtual space according to the relative spatial position information between the interactive device and the terminal device; the terminal device may construct the interactive screen according to the rendering coordinates, and superimpose it on the interactive area; the user may see the interactive screen superimposed on the interactive area through the head-mounted display device. As an embodiment, the terminal device may render and display the virtual extended content and the interactive screen at the same time, so that the user can see the interactive screen superimposed on the interactive area through the head-mounted display device, and the virtual extended content superimposed on the set area outside the interactive area.

For example, the interactive device 200 in Figure 1 is a touch panel and does not have a display function. The user observes the virtual chat interface 301 superimposed on the interactive area 202 of the interactive device 200 in the real space through the head-mounted display device worn by the user, and can observe that the non-interactive content (such as historical chat records 302) in the virtual chat interface 301 is displayed in the form of virtual content at a position outside the interactive area 202. Therefore, the user can chat with the chat object through the interactive area 202 in the virtual chat interface 301, and can also view the historical chat records 302. This allows the user to view content through the interactive device 200 without being restricted by the size of the interactive area, thereby achieving interactive display of the content to be displayed, improving the display effect, and making it more conducive for the user to interact with the displayed content.

In other embodiments, the interactive screen corresponding to the interactive content is displayed, or the interactive device may display the interactive screen corresponding to the interactive content through the touch screen of the interactive area. The terminal device may display the virtual extended content when it detects that the touch screen displays the interactive screen. Please refer to Figures 4A and 4B. The interactive device 200 is a smart phone. Figure 4A is the screen display of a traditional smart phone. The terminal device can obtain the non-interactive content 203 (such as signal strength, battery power, time, etc.) that only needs to be displayed by the smart phone and display it as virtual extended content in the area outside the touch screen of the smart phone. Please refer to Figure 4B. When the touch screen of the smart phone displays the interactive content, the user can see the virtual extended content 303 displayed in the area outside the touch screen of the smart phone in the real world through the head-mounted display device.

The method for displaying virtual content provided by the embodiment of the present application obtains content to be displayed, the content to be displayed includes non-interactive content and interactive content, then obtains first content data corresponding to the non-interactive content, obtains relative spatial position information between an interactive device and a terminal device, and generates virtual extended content according to the relative spatial position information and the first content data. When an interactive screen corresponding to other content is displayed, the virtual extended content is displayed, a first display area of the interactive screen corresponds to an interactive area, and a second display area of the virtual extended content corresponds to a set area outside the interactive area, so that the virtual extended content corresponding to the non-interactive content in the content to be displayed is displayed in the virtual space according to the spatial position of the interactive device, so that the user can see the effect of the virtual extended content being displayed outside the interactive area, and the displayed interactive content corresponds to the interactive area, thereby expanding the display space of the displayed content, improving the display effect, and being more conducive to the user interacting with the displayed content.

Referring to FIG. 5 , another embodiment of the present application provides a method for displaying virtual content, which can be applied to a terminal device, wherein the terminal device is communicatively connected to an interactive device, and the interactive device includes an interactive area. The method may include:

Step S210: Obtain content to be displayed, where the content to be displayed includes non-interactive content and interactive content.

In some embodiments, the interactive area of the interactive device may include a touch screen, that is, the interactive area may display content. When the interactive device displays content, the above-mentioned acquisition of the content to be displayed may include: sending a data request to the interactive device, the data request is used to instruct the interactive device to acquire the content to be displayed on the touch screen; receiving the content to be displayed sent by the interactive device.

When the terminal device needs to obtain the content to be displayed, it can send a data request to the interactive device to obtain the content to be displayed on the touch screen of the interactive device. When the interactive device receives the data request, it can obtain the content to be displayed on the touch screen from the server or from the local storage. Then, the data of the content to be displayed is transmitted to the terminal device, so that the terminal device can receive the data of the content to be displayed.

In some embodiments, before sending a data request to the interactive device, the terminal device can first establish a communication connection with the interactive device via wired or wireless means, so that the above data request can be successfully sent to the interactive device, and the data of the content to be displayed sent by the interactive device can also be successfully received.

In some embodiments, the non-interactive content may correspond to part of the interactive content. For example, when the part is an operation control, the non-interactive content may be the interface content associated with the operation control. For another example, when the part is the nine-key Pinyin or twenty-six-key English in the input method interface, the non-interactive content may be a character input box, a character display interface, character candidate content corresponding to the key, etc.

Step S220: Acquire first content data corresponding to the non-interactive content.

In some application scenarios, due to the limitation of the display area size of the interactive area, it is often necessary to page the content to be displayed. When viewing the current menu page, the previous menu page is usually hidden, and you need to return to view the content of other menu pages. Moreover, when viewing the current menu page, you cannot directly see the next menu page, and you need to select a menu item to enter before you can see it. Therefore, in an embodiment of the present application, the terminal device can use the previous menu page or the next menu page of the current menu page as non-interactive content, and convert it into virtual extended content and overlay it in a set area outside the interactive area, so that the user can view the previous menu page and the current menu page corresponding to the interactive area at the same time, or the user can view the current menu page corresponding to the interactive area and the next menu page at the same time.

Specifically, in some embodiments, the above-mentioned content to be displayed may include multiple menu pages, and the above-mentioned obtaining of the first content data corresponding to the non-interactive content may include: obtaining the first data of the second menu page corresponding to the first menu page according to the multiple menu pages, and using the first data as the first content data corresponding to the non-interactive content in the content to be displayed. The second menu page is the upper-level menu page or the lower-level menu page of the first menu page, and the first data of the second menu page may be the image data of the second menu page. Thus, the terminal device may use the first data of the upper-level menu page or the lower-level menu page of the current-level menu page as the first content data corresponding to the non-interactive content in the content to be displayed, so as to subsequently generate virtual extended content according to the first content data.

In some embodiments, the first menu page may be a menu page that is currently required to be operated among multiple menu pages to be displayed. It is understandable that when the first menu page is a first-level menu page among multiple menu pages, the second menu page may only be a next-level menu page of the first menu page. For example, when the first menu page is a setting page of a mobile phone, the second menu page may be a sub-page of Bluetooth settings, a sub-page of connectable wireless accounts, etc. When the first menu page is a menu page of another level among multiple menu pages except the first-level menu page, the second menu page may be a next-level menu page of the first menu page, or may be a previous-level menu page of the first menu page. For example, when the first menu page is a sub-page of power settings, the second menu page may be a corresponding previous-level page, such as a setting page of a mobile phone, or may be a corresponding next-level page, such as a power saving settings page, a sleep settings page, etc.

Since the first menu page includes multiple menu items, the first menu page may correspond to multiple second menu pages. Therefore, as an implementation method, the terminal device may obtain the first data corresponding to the second menu page corresponding to the first menu item in the menu item list by default, and use the first data as the first content data corresponding to the above non-interactive content. For example, when the first menu page currently to be operated is a file management page, the terminal device may obtain the data corresponding to the file content in the first folder by default.

As another implementation, the terminal device may also obtain the first data corresponding to the second menu page corresponding to the plurality of menu items in the menu item list, that is, the terminal device may use the plurality of second menu pages as non-interactive content. For example, when the first menu page is a game interface, the terminal device may obtain the data of the second menu page corresponding to the backpack menu item, the map menu item, etc.

Step S230: Obtain the relative spatial position information between the interaction device and the terminal equipment.

In the embodiment of the present application, step S230 can refer to the contents of the above embodiment and will not be repeated here.

Step S240: obtaining a first relative position relationship between a setting area and the interactive device, wherein the setting area is a corresponding superimposed area in a real environment when the virtual extended content is displayed.

In an embodiment of the present application, the terminal device can obtain a display area where the virtual extended content needs to be displayed, so as to subsequently generate the virtual extended content according to the display area. Specifically, the terminal device can obtain a first relative position relationship between the set area and the interactive device, wherein the set area refers to the corresponding superimposed area where the terminal device superimposes the virtual extended content on the real environment, and can also be understood as the area of the virtual extended content in the real environment that the user sees through the terminal device.

In some embodiments, the obtaining of the first relative position relationship between the set area and the interactive device may include: determining the set area corresponding to the non-holding area outside the interactive area according to the non-holding area of the interactive device, and obtaining the first relative position relationship between the set area and the interactive device, wherein the non-holding area is an area in the edge area of the interactive device that is not held. Thus, the terminal device may set the display area of the virtual extended content to the area outside the edge of the interactive device that is not held according to the holding area of the interactive device held by the user.

In some embodiments, the set area outside the interactive area may correspond to the non-holding area of the interactive device, that is, the non-holding area corresponding to different edge sides of the interactive device held by the user is different. The terminal device may determine the set areas corresponding to different non-holding areas based on the correspondence between the set areas and the non-holding areas, so that the display areas of the generated virtual extended content may also be different. For example, when the user holds the interactive device on the left or right side relative to the terminal device, the display area corresponding to the virtual extended content may be the upper area or the lower area outside the interactive area relative to the terminal device, so as to avoid occlusion between the virtual extended content and the user's hand. Among them, the correspondence between the set area and the non-holding area may be stored in the terminal device or downloaded from the server.

As a specific implementation, the above-mentioned determining the set area outside the interaction area corresponding to the non-holding area according to the non-holding area of the interaction device, and obtaining the first relative position relationship between the set area and the interaction device, may include: when it is detected that the interaction device is in a held state, collecting a gesture image; determining the non-holding area of the interaction device according to the gesture image; obtaining the set area outside the interaction area corresponding to the non-holding area, and obtaining the first relative position relationship between the set area and the interaction device. Among them, the terminal device can collect the user's gesture image through an image sensor to determine the non-held area in the edge area of the interaction device according to the gesture image, so as to obtain the set area outside the interaction area corresponding to the non-holding area according to the corresponding relationship between the set area and the non-holding area. Therefore, the terminal device can determine the non-holding area of the interaction device by detecting the posture of the user holding the interaction device.

In some embodiments, the terminal device can determine whether the interactive area of the interactive device is within the field of view of the terminal device's image sensor based on the position information, rotation direction, and rotation angle of the interactive device relative to the terminal device, thereby determining whether the interactive area is in a held state. The terminal device can also directly determine whether the interactive device is in a held state based on the sensing data of the sensor of the interactive device. The sensor can be an acceleration sensor, a gravity sensor, etc.

As another specific implementation, the interactive device may be provided with a temperature sensor or a pressure sensor, and the interactive device may determine the edge of the interactive device held by the user based on the corresponding sensing data, thereby obtaining the non-holding area of the interactive device. The interactive device then transmits the data of the non-holding area to the terminal device.

In other embodiments, the above-mentioned obtaining the first relative position relationship between the setting area and the interactive device may also include: determining the setting area corresponding to the first manipulation action parameter outside the interactive area according to the first manipulation action parameter detected in the interactive area, and obtaining the first relative position relationship between the setting area and the interactive device. Among them, the terminal device may determine the setting area selected by the user according to the first manipulation action parameter detected in the interactive area sent by the interactive device. Thus, the display area where the virtual extended content needs to be displayed can also be selected by the user himself to meet the needs of different users.

In some embodiments, the above-mentioned first manipulation action parameter may include parameters such as the touch position corresponding to the touch operation, the type of touch operation, the number of fingers of the touch operation, the finger pressing pressure, and the duration of the touch operation. Among them, the touch position corresponding to the touch operation may refer to the position of the manipulated area on the interactive area, for example, it may be the touch coordinates in the plane coordinate system of the plane where the interactive area is located. The type of touch operation may include a click operation, a slide operation, a long press operation, etc. The number of fingers of the touch operation refers to the number of fingers performing the operation there, that is, the number of areas pressed when the sensor of the interactive area detects the touch operation, for example, the number is 1, and for another example, the number is 2. The finger pressing pressure refers to the pressing pressure of performing the touch operation, that is, the size of the pressure detected by the sensor of the interactive area, for example, the pressing pressure is 0.5N (Newton). The duration of the touch operation is the time when the finger detected by the interactive area is in contact with the interactive area, for example, the duration is 1S (seconds). Of course, the specific first manipulation action parameter may not be limited in the embodiment of the present application, and the first manipulation action parameter may also include other touch parameters, for example, sliding trajectory, click frequency of click operation, etc.

In some embodiments, the terminal device determines the set area outside the interactive area corresponding to the first manipulation action parameter based on the first manipulation action parameter. This may be when a selection list of the set area is displayed in the interactive area, and the selection list may include multiple set areas. It may also be when a sliding bar for setting the distance of the set area relative to the interactive area is displayed in the interactive area. After the interactive area detects the touch operation on the above-mentioned selection list and the sliding bar, the interaction device can send the corresponding manipulation action parameter to the terminal device, so that the terminal device can determine the setting area option selected by the user based on the manipulation action parameter. For example, the touch position when the user currently selects the setting area option is determined according to the first manipulation action parameter, and the current sliding length of the user on the sliding bar is determined according to the first manipulation action parameter.

In some embodiments, the obtaining of the first relative position relationship between the set area and the interactive device may further include: reading a pre-stored first relative position relationship between the set area outside the interactive area and the interactive device. It is understandable that the position relationship between the display area where the virtual extended content needs to be displayed and the interactive device may be fixed, for example, the set area is located in the left area, right area, etc. outside the interactive area.

Of course, the specific method of obtaining the first relative position relationship between the set area and the interactive device may not be limited in the embodiment of the present application.

Step S250: Acquire a second relative position relationship between the set area and the terminal device according to the relative spatial position information and the first relative position relationship.

In an embodiment of the present application, the terminal device can obtain the second relative position relationship between the set area and the terminal device based on the relative spatial position information between the interaction device and the terminal device and the first relative position relationship between the set area and the interaction device, taking the terminal device as a reference, thereby obtaining the spatial position information of the set area in the real space.

Step S260: Generate virtual extended content according to the second relative position relationship and the first content data.

In the embodiment of the present application, after determining the set area where the virtual extended content needs to be displayed, the terminal device can obtain the rendering position of the virtual extended content in the virtual space according to the second relative position relationship between the set area and the terminal device, and then render the virtual content according to the rendering position. The specific steps of generating the virtual extended content can refer to the content of the above embodiment, which will not be repeated here.

Step S270: when the interactive screen corresponding to the interactive content is displayed, the virtual extended content is displayed, the first display area of the interactive screen corresponds to the interactive area, and the second display area of the virtual extended content corresponds to the set area outside the interactive area.

In some embodiments, when the content to be displayed includes multiple menu pages, and the non-interactive content is the previous menu page or the next menu page of the first menu page, the above-mentioned display of virtual extended content when the interactive screen corresponding to the interactive content is displayed may include: when the interactive screen corresponding to the first menu page is displayed, displaying virtual extended content, the virtual extended content includes a second menu page. Thus, while the display area of the interactive screen corresponding to the first menu page corresponds to the interactive area, the user can see the second menu page superimposed on the set area outside the interactive area through the head-mounted display device, so that the user can view the previous menu page and the current menu page corresponding to the interactive area at the same time, or the user can view the current menu page corresponding to the interactive area and the next menu page at the same time. Among them, the interactive screen corresponding to the first menu page can be superimposed on the interactive area by the terminal device, or it can be displayed by the interactive device through the touch screen of the interactive area.

For example, see FIG6 , the interactive device 200 is a smart phone, and when the smart phone enters the wireless LAN menu page 204, the terminal device can convert the corresponding upper-level menu page into virtual extended content 303 and display it in the area outside the touch screen of the smart phone, so that the user can view the upper-level menu page while viewing the wireless LAN menu page. For another example, see FIG7 , when the content to be displayed is a game interface, the terminal device can convert the lower-level menu page corresponding to the game screen, backpack options, map options, etc. as non-interactive content into virtual extended content, and display it in the area outside the touch screen of the smart phone, so that the user can see the virtual game screen 304, virtual backpack content 305, and virtual map content 306 displayed in the area outside the touch screen of the smart phone in the real world through the head-mounted display device.

In some embodiments, when the terminal device displays virtual extended content in a set area outside the interactive area, the terminal device may display an icon corresponding to the virtual extended content on the interactive area to identify and distinguish the virtual extended content. Specifically, when the interactive screen corresponding to the interactive content is displayed, before displaying the virtual extended content, the method for displaying the virtual content may further include:

Acquire second content data corresponding to the interactive content; acquire icon data corresponding to the non-interactive content; and generate an interactive screen according to the second content data, the icon data, and the relative spatial position information.

In some embodiments, when the interactive device does not have a display function, the above-mentioned interactive content and non-interactive content can be displayed by the terminal device. The terminal device can obtain the icon data of the icon corresponding to the non-interactive content according to the non-interactive content in the above-mentioned content to be displayed, so as to generate an interactive screen containing the icon and the above-mentioned interactive content according to the second content data, icon data and the above-mentioned relative spatial position information corresponding to the interactive content. The icon data may include model data for rendering icons, and the second content data may include model data for rendering interactive content. The terminal can also obtain the position information of the icon and the interactive content at the designated position in the interactive area on the interactive device, so as to obtain the display position of the icon and the interactive content according to the relative spatial position information and the position information, and generate an interactive screen containing the icon and the interactive content for display. Among them, the method of generating the interactive screen can refer to the method of generating virtual extended content in the above-mentioned embodiment, which will not be repeated here.

As a method, when the non-interactive content is a previous level menu page, each icon in the corresponding icon list corresponds to each menu item in the previous level menu page, and the virtual extended content generated according to the previous level menu page can be the explanation, description, name and other information content of each icon. It can be understood that the icon data corresponding to the non-interactive content can be stored in the terminal device or downloaded from the server, which is not limited here.

In some embodiments, the icons corresponding to the non-interactive content may be associated with the controls. The icons may be directly displayed on top of the controls. The controls may be button controls, input box controls, list controls, etc., which may be directly displayed below the controls and may be used to control the display and cancellation of the virtual extended content, and may also be used to control the movement of the virtual extended content and adjust the scale of the virtual extended content, which is not limited here. Thus, the user may trigger the display control of the virtual extended content by operating the icons.

For example, see FIG8 , the interactive area 202 of the interactive device displays a virtual chat page 301 of the selected chat object, and the terminal device can generate a virtual icon option list 307 with a user avatar, and display it superimposed on one side of the interactive area. The user selects different icons in the virtual icon option list to switch virtual chat pages of different chat objects through touch operation on the interactive area. At the same time, the terminal device can obtain information corresponding to each user avatar, such as personal information (account, name, profile, etc.), or historical chat content, generate virtual extended content 303, and display the virtual extended content 303 on the outside of the interactive area. The personal information of each chat object can be displayed with the corresponding icon. When the user drags the virtual icon option list, such as sliding up, the user avatar in the virtual icon option list will change accordingly, and the virtual extended content 303 corresponding to the user avatar will also change accordingly.

Similarly, when the non-interactive content is a next-level menu page, each icon in the corresponding icon list may correspond to each menu item in the next-level menu page. In some embodiments, there are some menu items that do not have a corresponding next-level menu page. For example, GPS settings only have two states, on and off, and there is no next-level menu. The terminal device can display an icon and a button control corresponding to the option on the interactive area, and the icon is superimposed on the button control, so that the user can turn the GPS setting on and off through a touch action on the interactive area (such as clicking an icon).

Further, in some embodiments, when an interactive screen is generated according to the second content data, the icon data, and the relative spatial position information, the user can control the display of the virtual extended content by operating the icon. Specifically, when the interactive screen corresponding to the interactive content is displayed, displaying the virtual extended content may include:

The interactive screen is displayed; when it is determined that the icon in the interactive screen is operated according to the third manipulation action parameter detected in the interactive area, the virtual extended content corresponding to the operated icon is displayed, and the second display area of the virtual extended content corresponds to the set area matching the operated icon outside the interactive area.

After the terminal device generates an interactive screen containing icons and the above-mentioned interactive content, the interactive screen can be displayed so that the user can see the interactive screen containing icons and the above-mentioned interactive content superimposed in the interactive area. When it is determined that the user operates the icon in the interactive screen according to the third manipulation action parameter detected in the interactive area, the terminal device can display the virtual extended content corresponding to the operated icon, and the second display area of the virtual extended content corresponds to the set area matched by the operated icon outside the interactive area. Thus, the user controls the display of the virtual extended content corresponding to the set area outside the interactive area by performing a touch operation on the icon in the interactive area. In some embodiments, operating the icon in the interactive screen may include clicking, moving, and other operations on the icon, which are not limited here. Among them, the above-mentioned setting area matches the icon, and the setting area can be set on the side of the icon according to the position of the icon in the interactive area, so that the user can quickly find the corresponding virtual extended content according to the icon. For example, please refer to Figure 6 again. When the icon list 205 is on the left side of the interactive area, the setting area outside the interactive area corresponding to the virtual extended content 303 is also on the left side of the icon list 205.

In some embodiments, when the interactive device includes a touch screen, the interactive device can display content through the touch screen. Therefore, when the terminal device displays virtual extended content in a set area outside the interactive area, the interactive device can also display the above-mentioned interactive screen containing icons and the above-mentioned interactive content through the touch screen. When the interactive area detects an operation on the icon, the interactive device can send a control instruction for instructing the terminal device to control the virtual extended content to the terminal device, and the terminal device can perform the above-mentioned display control on the virtual extended content according to the control instruction. For example, please refer to Figure 6 again. When the menu page of the wireless local area network is displayed on the screen of the mobile terminal, the terminal device displays the virtual extended content 303 (the upper level menu page) outside the touch screen of the smart phone, and at the same time, the screen of the mobile terminal can generate and display an icon list 205 corresponding to the menu items of the upper level menu page. When the user clicks to select different icon options, such as clicking the WiFi icon in the icon list 205, it can directly jump to the wireless local area network menu page 204 corresponding to the icon option.

In some implementations, when the interactive screen corresponding to the interactive content is displayed, the terminal device may directly display the virtual extended content, thereby realizing automatic display of the virtual extended content without manual control by the user. After displaying the virtual extended content, the terminal device may also perform display control such as moving and scaling the virtual extended content according to the user's manipulation action parameters on the icon in the interactive area. For example, when the icon is moved from the left side of the interactive area to the right side of the interactive area, the second display area of the virtual extended content may move from the left area outside the interactive area to the right area outside the interactive area along with the movement of the icon.

Furthermore, since the non-interactive content in the content to be displayed is displayed by the terminal device outside the interactive area, the display area of the interactive content can correspond to the entire interactive area, so that the position and proportion of the interactive content can be appropriately optimized. For example, in a game application, in order not to block too much screen space, the control controls are sometimes designed to be relatively small. Therefore, when the non-interactive content in the game screen is displayed outside the interactive area, the interactive area can only display the control controls. At this time, the control controls can be designed to be relatively large, which is convenient for users to click and improve the user's gaming experience. In some embodiments, the above-mentioned generation of the interactive screen based on the second content data, the icon data, and the relative spatial position information may include:

Acquire control data from the second content data; determine a first layout position and a first proportional size of each control in the interactive screen according to the control data; determine a second layout position and a second proportional size of each icon in the interactive screen according to the icon data; generate an interactive screen containing controls and icons according to the relative spatial position relationship and the first layout position and the first proportional size, the second layout position and the second proportional size.

Among them, the interactive content may include controls for controlling the display of non-interactive content. The controls may be video "play", "pause", "progress bar" and other controls, or they may be skill controls, displacement controls, equipment setting controls, etc. in the game interface, which are not limited here.

In some embodiments, the terminal device can obtain the control data corresponding to the control in the interactive content, re-determine the first arrangement position and the first scale size of each control in the interactive screen according to the control data, determine the second arrangement position and the second scale size of each icon in the interactive screen according to the above icon data, and generate an interactive screen containing controls and icons according to the relative spatial position relationship and the first arrangement position and the first scale size, the second arrangement position and the second scale size. Among them, the method of generating an interactive screen containing controls and icons can refer to the method of generating virtual extended content in the above embodiment, which will not be repeated here. For example, referring to Figure 8, the terminal device can display the control for triggering the display of the chat page of the corresponding chat object under the above-mentioned virtual icon option list 307 with the user avatar, so that the user can select different icons in the virtual icon option list through touch operation on the interactive area to trigger the corresponding control, so as to switch the virtual chat page of different chat objects.

In other embodiments, when the interactive area includes a touch screen, the above-mentioned interactive screen containing controls and icons can also be generated by the interactive device. For example, please refer to Figure 6 again. The mobile terminal can display the controls corresponding to each menu item in the upper menu page under the generated icon list 205. The controls correspond to the icons one by one, so that when the user clicks to select different icon options, the controls corresponding to the icons can be directly triggered to implement the corresponding functions. For example, clicking the WiFi icon in the icon list 205 can directly trigger the corresponding controls to jump to the corresponding wireless local area network menu page 204. For another example, please refer to Figure 7. In a game application, the terminal device displays the virtual game screen 304 in 3D on the upper side of the smartphone screen, and the virtual backpack content 305, virtual map content 306 and other viewing contents are displayed on the left and right sides of the screen. The mobile terminal can display the corresponding operation controls (including the backpack controls for triggering the display of the virtual backpack content 305 and the map controls for the user to trigger the display of the virtual map content 306) through the touch screen, which is convenient for the user to click and improve the user's gaming experience.

Furthermore, when generating an interactive screen including controls and icons, after displaying the virtual extended content, it also includes:

When it is determined to operate the control in the interactive picture according to the fourth manipulation action parameter detected in the interactive area, a control instruction is generated according to the operated control, and the virtual object in the virtual extended content is controlled according to the control instruction.

In some embodiments, the controls in the content may have a corresponding relationship with the non-interactive content. For example, the non-interactive content in the game interface is the game screen, and the control is a displacement control. The user can switch the game screen by manipulating the displacement control. Therefore, when the virtual extended content is generated according to the non-interactive content, the terminal device can control the virtual extended content according to the manipulation action parameters detected in the interactive area. Specifically, when it is determined to operate the controls in the interactive screen according to the fourth manipulation action parameters detected in the interactive area, the terminal device can generate a control instruction according to the operated control, and control the virtual object in the virtual extended content according to the control instruction, thereby realizing the interaction between the interactive device and the terminal device. For example, please refer to Figure 7 again. The terminal device can switch the content display of the virtual game screen 304 in real time according to the user's manipulation of the displacement control on the smartphone screen.

Furthermore, when the terminal device displays the virtual extended content, it can sort the display area by priority. For example, it can sort the priority from low to high according to the time before and after the display trigger of the virtual extended content, that is, the earlier the virtual extended content is triggered, the lower the priority. As an implementation method, the terminal device can display the virtual extended content with high priority on the upper side of the interactive area, and display the virtual extended content with low priority on the left and right sides of the interactive area.

In some implementations, the terminal device may store the display order of the virtual extended content, so that when the display of the current virtual extended content is exited, the previous virtual extended content of the current virtual extended content may be automatically displayed. As an implementation, a progress bar of multiple virtual extended contents may also be displayed on the interactive area, and the user may switch between the multiple virtual extended contents displayed previously by sliding the progress bar.

The method for displaying virtual content provided by the embodiment of the present application obtains first content data corresponding to non-interactive content in the content to be displayed, then obtains relative spatial position information between the interactive device and the terminal device, obtains a first relative position relationship between a set area and the interactive device, and sets the area as an overlay area corresponding to the virtual extended content in a real environment when the virtual extended content is displayed, so as to obtain a second relative position relationship between the set area and the terminal device according to the relative spatial position information and the first relative position relationship, thereby generating virtual extended content according to the second relative position relationship and the first content data, and displaying the virtual extended content when the interactive screen corresponding to the interactive content is displayed, so as to realize displaying the virtual extended content corresponding to the non-interactive content in the content to be displayed in the virtual space according to the spatial position of the interactive device, so that the user can see the effect that the virtual extended content corresponding to the non-interactive content is displayed outside the interactive area, and the displayed interactive content corresponds to the interactive area, thereby expanding the display space of the displayed content, improving the display effect, and being more conducive to the user interacting with the displayed content.

Please refer to FIG9, which shows a block diagram of a virtual content display device 500 provided in an embodiment of the present application, which is applied to a terminal device, and the terminal device is connected to an interactive device in communication, and the interactive device includes an interactive area. The device may include: a content acquisition module 510, a data acquisition module 520, a position acquisition module 530, a content generation module 540, and a display control module 550. Among them, the content acquisition module 510 is used to acquire the content to be displayed, and the content to be displayed includes non-interactive content and interactive content; the data acquisition module 520 is used to acquire the first content data corresponding to the non-interactive content; the position acquisition module 530 is used to acquire the relative spatial position information between the interactive device and the terminal device; the content generation module 540 is used to generate virtual extended content according to the relative spatial position information and the first content data; the display control module 550 is used to display the virtual extended content when the interactive screen corresponding to the interactive content is displayed, the first display area of the interactive screen corresponds to the interactive area, and the second display area of the virtual extended content corresponds to the set area outside the interactive area.

In some embodiments, the content generation module 540 can be specifically used to: obtain a first relative position relationship between a set area and an interactive device, the set area being an overlay area corresponding to when the virtual extended content is displayed in a real environment; obtain a second relative position relationship between the set area and a terminal device based on the relative spatial position information and the first relative position relationship; generate virtual extended content based on the second relative position relationship and the first content data.

Furthermore, the content generation module 540 obtains the first relative position relationship between the set area and the interaction device, which may include: determining, based on the non-holding area of the interaction device, a set area outside the interaction area corresponding to the non-holding area, and obtaining the first relative position relationship between the set area and the interaction device, the non-holding area being an area in the edge area of the interaction device that is not held; or determining, based on the first manipulation action parameter detected in the interaction area, a set area outside the interaction area corresponding to the second manipulation action, and obtaining the first relative position relationship between the set area and the interaction device; or reading a pre-stored first relative position relationship between the set area outside the interaction area and the interaction device.

Furthermore, the content generation module 540 determines a set area outside the interaction area corresponding to the non-holding area based on the non-holding area of the interaction device, and obtains a first relative position relationship between the set area and the interaction device. The non-holding area is an area in the edge area of the interaction device that is not held. It may include: when it is detected that the interaction device is in a held state, collecting a gesture image; determining the non-holding area of the interaction device based on the gesture image; obtaining a set area outside the interaction area corresponding to the non-holding area, and obtaining a first relative position relationship between the set area and the interaction device.

In some embodiments, the above-mentioned content to be displayed includes multiple menu pages, and the data acquisition module 520 can be specifically used to: obtain the first data of the second menu page corresponding to the first menu page according to the multiple menu pages, and use the first data as the first content data corresponding to the non-interactive content, wherein the second menu page is the upper level menu page or the lower level menu page of the first menu page. The display control module 550 can be specifically used to: when the interactive screen corresponding to the first menu page is displayed, display the virtual extended content, and the virtual extended content includes the second menu page.

Furthermore, the second menu page is a next-level menu page of the first menu page, and the virtual content display device 500 may further include: a content switching module. The content switching module is used to determine the menu item in the first menu page that is in a selected state according to the second manipulation action parameter detected in the interactive area, and obtain the second data of the new next-level menu page corresponding to the menu item in the selected state; generate new virtual extended content according to the relative spatial position information and the second data; and switch the currently displayed virtual extended content to the new virtual extended content, and the new virtual extended content includes a new next-level menu page.

In some embodiments, the interactive area includes a touch screen, and the content acquisition module 510 can be specifically used to: send a data request to the interactive device, the data request is used to instruct the interactive device to obtain the content to be displayed on the touch screen; receive the content to be displayed sent by the interactive device.

In some embodiments, the virtual content display device 500 may further include: a screen generation module. The screen generation module is used to obtain second content data corresponding to the interactive content; obtain icon data corresponding to the non-interactive content; and generate an interactive screen according to the second content data, the icon data, and the relative spatial position information. The display control module 550 may be specifically used to: display the interactive screen; when it is determined to operate the icon in the interactive screen according to the third manipulation action parameter detected in the interactive area, display the virtual extended content corresponding to the operated icon, and the second display area of the virtual extended content corresponds to the set area that matches the operated icon outside the interactive area.

In some embodiments, the above-mentioned screen generation module generates an interactive screen according to the second content data, icon data and relative spatial position information, which may include: obtaining control data from the second content data; determining the first arrangement position and the first proportional size of each control in the interactive screen according to the control data; determining the second arrangement position and the second proportional size of each icon in the interactive screen according to the icon data; generating an interactive screen containing controls and icons according to the relative spatial position relationship and the first arrangement position and the first proportional size, the second arrangement position and the second proportional size. The display device 500 of the virtual content may also include: a content control module. The content control module is used to generate a control instruction according to the operated control when determining to operate the control in the interactive screen according to the fourth manipulation action parameter detected in the interactive area, and control the virtual object in the virtual extended content according to the control instruction.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.

In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

In summary, an embodiment of the present application provides a display device for virtual content. The terminal device obtains content to be displayed, which includes non-interactive content and interactive content, and then obtains first content data corresponding to the non-interactive content, and obtains relative spatial position information between the interactive device and the terminal device, so as to generate virtual extended content according to the relative spatial position information and the first content data. When the interactive screen corresponding to the interactive content is displayed, the virtual extended content is displayed, the first display area of the interactive screen corresponds to the interactive area of the interactive device, and the second display area of the virtual extended content corresponds to the set area outside the interactive area, so as to realize the display of the virtual extended content corresponding to the non-interactive content in the content to be displayed in the virtual space according to the spatial position of the interactive device, so that the user can see the effect of the virtual extended content being displayed outside the interactive area, and the displayed interactive content corresponds to the interactive area, thereby expanding the display space of the displayed content, improving the display effect, and being more conducive to the user interacting with the displayed content.

Please refer to FIG. 1 again. The embodiment of the present application provides a virtual content display system 10. The system includes a terminal device 100 and an interactive device 200. The terminal device 100 is connected to the interactive device 200. The interactive device 200 includes an interactive area 202, wherein:

The interactive device 200 is used to obtain first content data corresponding to non-interactive content in the content to be displayed according to the content to be displayed in the interactive area 202, the content to be displayed includes non-interactive content and interactive content, send the first content data to the terminal device 100, and control the interactive area 202 to display the interactive screen corresponding to the interactive content;

The terminal device 100 is used to obtain the relative spatial position information between the interactive apparatus 200 and the terminal device 100, receive the first content data, generate virtual extended content according to the relative spatial position information and the first content data, and display the virtual extended content, wherein the display area of the virtual extended content corresponds to the set area outside the interactive area 202.

In some implementations, the terminal device 100 in the above embodiment may be an external/accessible head-mounted display device, and the head-mounted display device is connected to the interactive device. The head-mounted display device may only display interactive virtual content such as virtual extended content and collect marker images. All processing operations such as display and control of the virtual extended content may be completed by the interactive device 200. After generating the virtual extended content, the interactive device 200 transmits the display screen corresponding to the virtual extended content to the head-mounted display device, thereby completing the display of the virtual extended content.

Please refer to Figure 10, which shows a structural block diagram of a terminal device provided in an embodiment of the present application. The terminal device 100 can be a terminal device such as a smart phone, a tablet computer, a head-mounted display device, etc. that can run applications. The terminal device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, an image sensor 130, and one or more applications, wherein one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and one or more programs are configured to execute the method described in the aforementioned method embodiment.

The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire terminal device 100, and executes various functions and processes data of the terminal device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the terminal device 100 during use, etc.

In the embodiment of the present application, the image sensor 130 is used to capture images of real objects and scene images of target scenes. The image sensor 130 can be an infrared camera or a visible light camera, and the specific type is not limited in the embodiment of the present application.

In one embodiment, the terminal device is a head-mounted display device, which, in addition to the above-mentioned processor, memory and image sensor, may also include one or more of the following components: a display module, an optical module, a communication module and a power supply.

The display module may include a display control unit. The display control unit is used to receive a display image of the virtual content rendered by the processor, and then display and project the display image onto the optical module, so that the user can view the virtual content through the optical module. The display device may be a display screen or a projection device, etc., which can be used to display images.

The optical module may adopt an off-axis optical system or a waveguide optical system, and the display image displayed by the display device can be projected to the user's eyes after passing through the optical module. While the user sees the display image projected by the display device through the optical module. In some embodiments, the user can also observe the real environment through the optical module and experience the augmented reality effect after the virtual content is superimposed on the real environment.

The communication module may be a module such as Bluetooth, WiFi (Wireless-Fidelity), ZigBee (ZigBee Technology), etc., and the head-mounted display device may establish a communication connection with the terminal device through the communication module. The head-mounted display device in communication connection with the terminal device may exchange information and instructions with the terminal device. For example, the head-mounted display device may receive image data sent by the terminal device through the communication module, and generate virtual content of the virtual world for display based on the received image data.

The power supply can supply power to the entire head-mounted display device to ensure the normal operation of each component of the head-mounted display device.

Please refer to Figure 11, which shows a structural block diagram of an interactive device provided in an embodiment of the present application. The interactive device 200 can be an electronic device with an interactive area such as a smart phone, a tablet computer, etc., and the interactive area can include a touch pad or a touch screen. The interactive device 200 may include one or more of the following components: a processor 210, a memory 220, and one or more applications, wherein one or more applications can be stored in the memory 220 and configured to be executed by one or more processors 210, and one or more programs are configured to execute the method described in the aforementioned method embodiment.

Please refer to Figure 12, which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (12)

1. A method for displaying virtual content, applied to a terminal device, the terminal device being communicatively connected to an interaction device, the interaction device including an interaction area, the method comprising:

acquiring content to be displayed, wherein the content to be displayed comprises non-interactive content and interactive content, and the content to be displayed comprises a plurality of menu pages;

acquiring first data of a second menu page corresponding to a first menu page according to the plurality of menu pages, and taking the first data as first content data corresponding to the non-interactive content, wherein the second menu page is a higher-level menu page or a lower-level menu page of the first menu page;

Acquiring relative spatial position information between the interaction device and the terminal equipment;

generating virtual extension content according to the relative spatial position information and the first content data;

And when the interactive picture corresponding to the first menu page is displayed, displaying the virtual extension content, wherein the virtual extension content comprises the second menu page, a first display area of the interactive picture corresponds to the interactive area, and a second display area of the virtual extension content corresponds to a set area outside the interactive area.

2. The method of claim 1, wherein generating virtual extended content from the relative spatial location information and the first content data comprises:

acquiring a first relative position relation between a setting area and the interaction device, wherein the setting area is a corresponding superposition area in a real environment when virtual extension content is displayed;

acquiring a second relative position relation between the setting area and the terminal equipment according to the relative spatial position information and the first relative position relation;

and generating the virtual extension content according to the second relative position relation and the first content data.

3. The method of claim 2, wherein the obtaining a first relative positional relationship between the setting area and the interaction device comprises:

According to a non-holding area of the interaction device, a setting area outside the interaction area and corresponding to the non-holding area is determined, and a first relative position relation between the setting area and the interaction device is obtained, wherein the non-holding area is an area which is not held in an edge area of the interaction device; or alternatively

According to the first control action parameters detected by the interaction area, determining a setting area outside the interaction area and corresponding to the first control action parameters, and acquiring a first relative position relation between the setting area and the interaction device; or alternatively

And reading a pre-stored first relative position relation between the set area outside the interaction area and the interaction device.

4. The method according to claim 3, wherein the determining a set area outside the interactive area corresponding to the non-holding area according to the non-holding area of the interactive device, and acquiring a first relative positional relationship between the set area and the interactive device, comprises:

when the interaction device is detected to be in a held state, acquiring a gesture image;

Determining a non-holding area of the interaction device according to the gesture image;

And acquiring a setting area outside the interaction area and corresponding to the non-holding area, and acquiring a first relative position relationship between the setting area and the interaction device.

5. The method of claim 1, wherein the second menu page is a next level menu page of the first menu page, and wherein the virtual extension content is displayed when the interactive screen corresponding to the first menu page is displayed, and wherein the virtual extension content includes the second menu page, the method further comprises:

determining a menu item in a selected state in the first menu page according to the second control action parameter detected by the interaction area, and acquiring second data of a new next-stage menu page corresponding to the menu item in the selected state;

generating new virtual extension content according to the relative spatial position information and the second data;

And switching the currently displayed virtual extension content into the new virtual extension content, wherein the new virtual extension content comprises the new next-stage menu page.

6. The method according to any one of claims 1-5, wherein the interaction area includes a touch screen, and the obtaining the content to be displayed includes:

Transmitting a data request to the interaction device, wherein the data request is used for indicating the interaction device to acquire the content to be displayed of the touch screen;

And receiving the content to be displayed, which is sent by the interaction device.

7. The method according to any one of claims 1 to 5, wherein before the virtual extension is displayed when the interactive screen corresponding to the interactive content is displayed, the method further comprises:

acquiring second content data corresponding to the interactive content;

acquiring icon data corresponding to the non-interactive content;

Generating an interactive picture according to the second content data, the icon data and the relative spatial position information;

And displaying the virtual extension content when the interactive picture corresponding to the interactive content is displayed, wherein the method comprises the following steps:

displaying the interactive picture;

And when the operation of the icons in the interactive picture is determined according to the third control action parameters detected by the interactive area, displaying virtual extension content corresponding to the operated icons, wherein a second display area of the virtual extension content corresponds to a setting area matched with the operated icons outside the interactive area.

8. The method of claim 7, wherein generating an interactive screen from the second content data, the icon data, and the relative spatial location information comprises:

acquiring control data from the second content data;

determining a first arrangement position and a first proportion of each control in the interactive picture according to the control data;

determining a second arrangement position and a second proportion of each icon in the interactive picture according to the icon data;

Generating an interactive picture containing the control and the icon according to the relative spatial position information, the first arrangement position, the first proportional size, the second arrangement position and the second proportional size;

After the displaying of the virtual extension, further comprising:

when the control in the interactive picture is determined to be operated according to the fourth control action parameter detected by the interactive area, a control instruction is generated according to the operated control, and the virtual object in the virtual extension content is controlled according to the control instruction.

9. A display device of virtual content, characterized in that it is applied to a terminal device, said terminal device being communicatively connected to an interaction device, said interaction device comprising an interaction area, said device comprising:

The content acquisition module is used for acquiring content to be displayed, wherein the content to be displayed comprises non-interactive content and interactive content, and the content to be displayed comprises a plurality of menu pages;

The data acquisition module is used for acquiring first data of a second menu page corresponding to a first menu page according to the plurality of menu pages, and taking the first data as first content data corresponding to the non-interactive content, wherein the second menu page is a previous menu page or a next menu page of the first menu page;

The position acquisition module is used for acquiring relative spatial position information between the interaction device and the terminal equipment;

the content generation module is used for generating virtual extension content according to the relative spatial position information and the first content data;

And the display control module is used for displaying the virtual extension content when the interactive picture corresponding to the first menu page is displayed, wherein the virtual extension content comprises the second menu page, a first display area of the interactive picture corresponds to the interactive area, and a second display area of the virtual extension content corresponds to a set area outside the interactive area.

10. A system for displaying virtual content, the system comprising a terminal device and an interaction means, the terminal device being in communication with the interaction means, the interaction means comprising an interaction area, wherein:

The interaction device is configured to obtain, according to content to be displayed in the interaction area, first data of a second menu page corresponding to a first menu page in a plurality of menu pages included in the content to be displayed, and use the first data as first content data corresponding to non-interaction content in the content to be displayed, where the content to be displayed includes the non-interaction content and interaction content, the second menu page is a previous menu page or a next menu page of the first menu page, send the first content data to the terminal device, and control the interaction area to display an interaction picture corresponding to the interaction content;

the terminal equipment is used for acquiring relative spatial position information between the interaction device and the terminal equipment, receiving the first content data, generating virtual extension content according to the relative spatial position information and the first content data, displaying the virtual extension content when an interaction picture corresponding to the first menu page is displayed, wherein the virtual extension content comprises the second menu page, and a display area of the virtual extension content corresponds to a set area outside the interaction area.

11. A terminal device, comprising:

one or more processors;

A memory;

one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to perform the method of any of claims 1-8.

12. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a program code, which is callable by a processor for executing the method according to any one of claims 1-8.