WO2023207174A1 - Display method and apparatus, display device, head-mounted device, and storage medium - Google Patents

Abstract

Embodiments of the present application relate to the technical field of virtual scenes, and disclosed are a display method and apparatus, a display device, a head-mounted device, and a storage medium. The method comprises: detecting a target object in a real scene; according to a form correspondence between the target object and a virtual object to be displayed in a virtual scene, determining a target form corresponding to the virtual object; and performing image rendering according to the real scene and the virtual scene, and displaying the target form corresponding to the virtual object.

Description

Display method, device, display device, head-mounted device and storage medium

Cross-references to related applications

This application is filed based on a Chinese patent application with application number 202210470977.4 and a filing date of April 28, 2022, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference into this application in full. .

Technical field

Embodiments of the present application relate to the field of virtual scene technology, and in particular to a display method, device, display device, head-mounted device and storage medium.

Background technique

With the rapid development of virtual scene technology, technology that overlays and interacts with virtual reality and real scenes is becoming more and more popular. In related technologies, some electronic devices can display multiple virtual scene applications. How to improve the display effect of virtual objects in real scenes is an urgent technical problem that needs to be solved.

Contents of the invention

Embodiments of the present application provide a display method, device, display device, head-mounted device, and storage medium.

In a first aspect, embodiments of the present application provide a display method, including:

Detect target objects in real scenes;

Determine the target form corresponding to the virtual object according to the form correspondence between the target object and the virtual object to be displayed in the virtual scene;

Image rendering is performed based on real scenes and virtual scenes to display the target form corresponding to the virtual object.

In a second aspect, embodiments of the present application provide a display device configured to perform the steps in the method described in the first aspect.

In a third aspect, embodiments of the present application provide a display device configured to be worn by a user, including the display device described in the second aspect.

In a fourth aspect, embodiments of the present application provide a head-mounted device, including a processor, a memory, a display, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the processor. , the program includes instructions configured to perform the steps in the method described in the first aspect above.

In a fifth aspect, embodiments of the present application provide a computer-readable storage medium configured to store a computer program. The computer program is executed by a processor to implement the method described in the first aspect. Some or all of the steps described in .

In a sixth aspect, embodiments of the present application provide a computer program product. The computer program product includes computer instructions. The computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the instructions from the computer-readable storage medium. The computer instruction, the processor executes the computer instruction, so that the computer device performs the method described in the first aspect.

The technical solutions provided by this application may include the following technical effects:

According to the form correspondence between the target object and the virtual object, determine the target form corresponding to the virtual object to be displayed in the virtual scene, perform image rendering according to the real scene and the virtual scene, and display the target form corresponding to the virtual object; in this way, the virtual object The corresponding target form can change following the change of the target object, which enriches the diversity of target forms corresponding to the virtual object, thereby improving the display effect of the virtual object in the real scene.

Description of the drawings

The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application. in:

Figure 1 is a schematic architectural diagram of an intelligent interactive system with augmented reality display function provided by an embodiment of the present application;

Figure 2 is a flowchart 1 of the display method provided by the embodiment of the present application;

Figure 3 is a schematic flowchart of evoking a virtual scene application provided by an embodiment of the present application;

Figure 4 is a schematic diagram 1 of the form of a weather application provided by an embodiment of the present application;

Figure 5 is a schematic diagram 2 of the weather application provided by the embodiment of the present application;

Figure 6 is a schematic diagram three of the weather application provided by the embodiment of the present application;

Figure 7 is a schematic diagram 1 of the intersection relationship between rays and target objects provided by the embodiment of the present application;

Figure 8 is a schematic diagram 2 of the intersection relationship between rays and target objects provided by the embodiment of the present application;

Figure 9 is a flow chart 2 of the display method provided by the embodiment of the present application;

Figure 10 is a schematic diagram of the weather application provided by the embodiment of the present application switching from the first position to the second position;

Figure 11 is a flowchart three of the display method provided by the embodiment of the present application;

Figure 12 is a structural block diagram of a display device provided by an embodiment of the present application;

Figure 13 is a structural block diagram of a display device provided by an embodiment of the present application;

Figure 14 is a structural block diagram of an electronic device used to implement the display method according to the embodiment of the present application.

Detailed ways

Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application and cannot be understood as limiting the present application.

In the description of the present application, it should be understood that the terms “first” and “second” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, features defined as “first” and “second” may explicitly or implicitly include one or more of the described features. In the description of this application, "plurality" means two or more than two, unless otherwise expressly and specifically limited.

FIG. 1 is a schematic architectural diagram of an intelligent interaction system 100 with an augmented reality display function according to an embodiment of the present application. As shown in FIG. 1 , the intelligent interaction system 100 includes an electronic device 101 . The electronic device 101 can be installed with virtual scene applications such as augmented reality (Augmented Reality, AR) or mixed reality (Mixed Reality, MR) applications, and can run the AR or MR application based on the user's operation. Here, the user's operation can include electronic control. The control operations of the virtual scene application on the device 101, such as click, touch, slide, shake, voice control and other operations. The user's operations may also include control operations on a controller that can be paired with the electronic device 101 to use the controller such as a handle. The control of the virtual scene application on the electronic device 101 is completed. The electronic device 101 can collect video images of any objects in the environment through local cameras and/or sensors, and display virtual objects (also called virtual objects) on the display component according to the collected video images. The virtual object may be a virtual object in an AR or MR scene (that is, an object in the virtual environment). In practical applications, the electronic device 101 can independently complete the above interactive operations and realize the display of virtual objects. In practical applications, the electronic device 101 can also display virtual objects by interacting with the terminal device 102 . The terminal device 102 may serve as a platform that provides content and information support for AR or MR applications of the electronic device 101 .

It should be noted that in this embodiment of the present application, the virtual scene application in the electronic device 101 may be an application built into the electronic device 101 itself, or may be an application provided by a third-party service provider installed by the user. There is no limit to this.

It should be noted that in the embodiment of the present application, the electronic device 101 may be various types of devices configured with cameras and display components, including but not limited to electronic devices that provide voice and/or other data transmission interactions for users, such as electronic devices. The device 101 can be a mobile phone and other handheld devices with wireless connection functions, tablet computers, laptops, PDAs, video recorders and other terminal devices; the electronic device 101 can also be a wearable device used for virtual scene interaction, such as a smart watch, Smart glasses and other AR devices or MR interactive devices, etc. The embodiment of the present application does not limit the specific form of the electronic device 101.

In practical applications, the intelligent interaction system 100 may also include a server 103. The server 103 may serve as a platform that provides content and information support for AR or MR applications of the electronic device 101. The electronic device 101 and the server 103 can communicate. For example, the electronic device 101 can communicate with the server 103 through communication methods such as wireless LAN, Bluetooth or mobile network. It should be noted that in this embodiment of the present application, the server 103 may be one or more physical servers (for example, one server is exemplarily shown in Figure 1), a computer cluster, or cloud computing. The virtual machine of the scene is not limited in the embodiment of this application.

The embodiment of the present application provides a display method. Figure 2 is a schematic flowchart of the display method according to the embodiment of the present application. The display method can be applied to a display device. For example, the display device can be deployed in the intelligent interaction system 100 of Figure 1 . In some possible implementations, the method can also be implemented by the processor calling computer-readable instructions stored in the memory. In some possible implementations, the display method can be applied to the electronic device 101 in the intelligent interaction system shown in Figure 1 , where the electronic device 101 can be a wearable device used for virtual scene interaction, such as a smart watch, Smart glasses and other AR devices or MR interactive devices can also be terminal devices installed with virtual scene applications such as AR or MR applications, such as mobile phones and other handheld devices with wireless connection functions, tablets, laptops, and handheld computers. , video recorder, etc. As shown in Figure 2, the display method includes:

S201, detect target objects in real scenes;

S202, determine the target form corresponding to the virtual object according to the form correspondence between the target object and the virtual object to be displayed in the virtual scene;

S203: Perform image rendering based on the real scene and the virtual scene to display the target form corresponding to the virtual object.

In this embodiment of the present application, the target object is an object that preloads a virtual object. Target objects include: target space or target plane.

In some examples, the target space is a space with certain three-dimensional morphological attributes, such as inside a house, outside a house, in the air, etc.

In some examples, the target plane is a plane with certain two-dimensional morphological attributes, such as a desktop, wall, floor, ceiling, etc.

In the embodiment of this application, the morphological correspondence relationship at least includes the following relationships:

The form of the virtual object is adapted to the form of the target object, where the same virtual object can have different forms under different target objects. It should be noted that different virtual objects can have the same shape under the same target object. For example, the form of virtual object a under target object A is a two-dimensional form, and the form of virtual object b under target object A is also a two-dimensional form. Different virtual objects can also have different shapes under the same target object. For example, the form of virtual object a under target object A is a two-dimensional form, and the form of virtual object b under target object A is a three-dimensional form.

Here, the form includes, but is not limited to, two-dimensional form (planar form), three-dimensional form (three-dimensional form), etc.

Here, adaptation can be understood as having the same shape or similar shape.

Here, adaptation can also be understood as matching, and specifically it can be understood as: the combination of virtual and real forms conforms to visual perception, or object cognition, or aesthetic standards, or desired effects.

For example, the form of target object 1 is recorded as form 1, the form of target object 2 is recorded as form 2, and the form of target object 3 is recorded as form 3; the form of the virtual object relative to target object 1 is form 1', The form of the virtual object relative to target object 2 is form 2', and the form of the virtual object relative to target object 3 is form 3'; then form 1' is adapted or matched with form 1, and form 2' is consistent with form 2. Adapt or match, the form 3' is adapted or matched with the form 3, that is to say, virtual objects of different forms are displayed on different target objects. For example, when virtual object 1 is displayed on target object 1, the form of virtual object 1 should be form 1'; when virtual object 2 is displayed on target object 1, the form of virtual object 2 should also be form 1'. In other words, different virtual objects appear in the same form on the same target object. As another example, when virtual object 1 is displayed on target object 1, the form of virtual object 1 should be form 1'; when virtual object 1 is displayed on target object 2, the form of virtual object 1 should be form 2'. In other words, the same virtual object appears in different forms on different target objects, that is, the same virtual object has different target forms for different target objects.

In the embodiment of this application, the virtual scene includes but is not limited to any one of AR scene and MR scene.

In some examples, an AR scene is a scene where virtual reality and real scenes can be overlaid and interacted with.

In some examples, MR scenes are new visual environments generated by merging real scenes and virtual scenes, and set up a comprehensive scene with an interactive feedback information loop between real scenes, virtual scenes and users.

In the embodiment of the present application, the target form corresponding to the virtual object is determined based on the form correspondence between the target object and the virtual object to be displayed in the virtual scene, including: if the virtual object is to be displayed at the first position of the first target object , then it is confirmed that the virtual object to be displayed is the first target form; if the virtual object is to be displayed at the first position of the second target object, then it is confirmed that the virtual object to be displayed is the second target form; wherein, the first target form and the third target form The two targets have different shapes.

Here, the first position is the position where the virtual object is displayed.

In the embodiment of the present application, the interaction method for determining the first position includes but is not limited to head movement interaction, gesture interaction, voice interaction, ray interaction, etc. This application does not impose mandatory limitations on the interaction methods.

In the embodiment of the present application, the target form corresponding to the virtual object is determined according to the form correspondence between the target object and the virtual object to be displayed in the virtual scene, including: if the attribute of the target object is a three-dimensional form, then according to the form correspondence The first form correspondence relationship in the method determines that the target form corresponding to the virtual object is a three-dimensional form.

In some possible implementations, identify the attributes of the target object; when it is recognized that the attributes of the target object are in a three-dimensional form, determine that the target form corresponding to the virtual object is a three-dimensional form according to the first form correspondence in the form correspondence. .

Here, the attributes of the target object include morphological attributes. Among them, the morphological attributes are at least divided into two-dimensional morphology, three-dimensional morphology, etc. The above is only an illustrative description, and does not serve as a limitation on all possible contents included in the morphological attributes, but is not exhaustive here.

In addition, the attributes of the target object may also include semantic attributes. Among them, semantic attributes are at least divided into pendant form, ornament form, etc. For example, when the target object is a wall, the semantic attributes include a pendant-like form; when the target object is a desktop, the semantic attributes include an ornament-like form. The above is only an illustrative description, and does not serve as a limitation on all possible contents included in the semantic attributes, but is not exhaustive here.

Here, the first form correspondence relationship includes: when the attribute of the target object is a three-dimensional form, the form of the virtual object should also be a three-dimensional form.

In the embodiment of the present application, the target form corresponding to the virtual object is determined according to the form correspondence between the target object and the virtual object to be displayed in the virtual scene, including: if the attribute of the target object is a two-dimensional form, then according to the form correspondence The second form correspondence relationship in the relationship determines that the target form corresponding to the virtual object is a two-dimensional form.

In some possible implementations, the attributes of the target object are identified; when it is identified that the attributes of the target object are in a two-dimensional form, according to the second form correspondence in the form correspondence, the target form corresponding to the virtual object is determined to be Two-dimensional form.

Here, the second form correspondence relationship includes: when the attribute of the target object is a two-dimensional form, the form of the virtual object should also be a two-dimensional form.

In the embodiment of the present application, determining the target form corresponding to the virtual object according to the form correspondence between the target object and the virtual object to be displayed in the virtual scene includes: determining the virtual object according to the third form correspondence in the form correspondence. The target shape corresponding to the object is a predefined shape corresponding to the predefined shape.

Here, the third form correspondence includes: the virtual object of the target object takes a predefined form corresponding to the predefined shape. Among them, the predefined shapes corresponding to different target objects are different, and the predefined shapes corresponding to different target objects are also different.

In some possible implementations, identifying the attributes of the target object includes: identifying the attributes of the target object based on characteristics such as location, shape, and size of the target object. In other possible implementations, identifying the attributes of the target object includes: identifying the attributes of the target object through a pre-trained model. It should be understood that the methods for identifying the attributes of the target object are not limited to the above-mentioned ones, and the embodiments of the present application do not limit how to identify the attributes of the target object.

In this way, the target form of the virtual object is adapted to the attributes of the target object. Moreover, the target form of the virtual object can change following changes in the attributes of the target object.

In the embodiment of the present application, the target form corresponding to the virtual object displayed in the first position is the image rendering result of a combination of virtual and real. Here, the virtual object may be a virtual application or specific content of the virtual application. For example, the target object is the desktop, the virtual object is the player application, and the target form corresponding to the player application can be a three-dimensional virtual speaker on the desktop, where the virtual speaker is a target form corresponding to the player application. . In another example, the target object is a wall, the virtual object is a calendar application, and the target form corresponding to the calendar application may be a two-dimensional virtual mural on the wall, where the virtual mural is a target corresponding to the calendar application. form. The above is only an illustrative description, and does not serve as a limitation on all possible types of target forms corresponding to the target object and the virtual object, but is not exhaustive here.

Taking the virtual object as a weather application as an example, Figure 3 shows a schematic process diagram for evoking a virtual scene application. As shown in Figure 3, a virtual ray is emitted through the controller, pointing to a target object in the real scene, and the electronic device receives After triggering the operation to evoke the menu, applications that can be displayed on the target object are displayed; after the electronic device receives a selection operation for any application on the target object, the target form corresponding to the application is displayed on the target object. The target form is adapted to the target object. Continuing to take the virtual object as a weather application as an example, when the weather application is placed on a target object such as a desktop, the target form presented on the desktop is a desktop ornament, as shown in Figure 4. When the weather application is placed on a target object such as a wall, the target shape presented on the wall is a wall pendant, as shown in Figure 5. When the weather application is placed on a target object such as the ceiling, the target shape presented on the ceiling is a floating cloud shape, as shown in Figure 6. In this way, the shape of the virtual object can take on different forms depending on the spatial properties/planar properties of the real scene, and the presentation form of the virtual object is more in line with the physical properties of the real scene.

It should be understood that the applications and display forms shown in Figures 3 to 6 are only illustrative, and those skilled in the art can make various obvious changes and/or replacements based on the examples of Figures 3 to 6.

The technical solution of the embodiment of the present application determines the target form corresponding to the virtual object based on the form correspondence between the target object and the virtual object to be displayed in the virtual scene; performs image rendering based on the real scene and the virtual scene to display the corresponding virtual object target form. In this way, the target form corresponding to the virtual object can change following the change of the target object, making the state of the virtual object more consistent with the nature of the target object, enriching the diversity of the target forms corresponding to the virtual object, and thus improving the performance of the virtual object in the real world. The display effect in the scene makes the combination of virtual and real more flexible and diverse.

In the embodiment of the present application, before displaying the target form corresponding to the virtual object, the above display method may further include: in response to the emission operation of the ray, when the ray intersects with the target object in the real scene, confirming the first shape corresponding to the target object. Location. Here, the first position is the position on the target object where the virtual object is displayed.

In some examples, the rays may be virtual rays. It should be noted that the virtual ray is only visible to the user who wears the electronic device or controls the electronic device. That is, only the user who wears the electronic device or controls the electronic device can see the virtual ray. In some embodiments, before the emitting operation of responding to the ray, the method further includes: emitting the ray.

In some examples, this application does not limit the emission mode of rays. For example, rays can be emitted through the emission button of an electronic device. As another example, rays can be emitted through a controller such as a handle paired with an electronic device. Here, the ray is a ray with a certain length value. The specific length value of the ray can be set or adjusted according to the user's needs. For example, the length value of the ray can be adjusted according to the user's environment.

In some possible implementations, if there is an intersection point between the ray and the target object, the intersection point is confirmed as the first position corresponding to the target object; if there is no intersection point between the ray and the target object, the distance on the ray to the target object is determined. The closest point is confirmed as the first position corresponding to the target object.

Here, the point on the ray that is closest to the target object can be understood as the end point of the ray.

In this way, the specific position where the virtual object is displayed on the target object can be determined. The same virtual object displays the same target shape at different positions on the same target object, thereby making the display of virtual objects at different positions on the same target object more unified. .

In the embodiment of the present application, before determining the target form corresponding to the virtual object according to the form correspondence between the target object and the virtual object to be displayed in the virtual scene, the above display method may also include: in response to the ray emission operation, determining The intersection relationship between the ray and the target object.

Here, the intersection relationship can be understood as whether there is an intersection point between the ray and the target object.

In some examples, in response to the emission operation of the ray, determining the intersection relationship between the ray and the target object includes: when the ray acts on the target object, determining the intersection relationship between the ray and the target object as: intersection.

Here, "acting on" can be understood as the virtual ray can reach the target object. Figure 7 shows the schematic diagram 1 of the intersection relationship between the ray and the target object. As shown in Figure 7, if the virtual ray hits the target object, then there is an intersection point A between the virtual ray and the target object, that is, the virtual ray and the target space or target plane The intersection relationship is: intersection. For example, assuming that the target object is a wall and the length of the virtual ray is 10 meters, if the user is standing in the house and the distance between the user and the wall is 4 meters, the electronic device emits a 10-meter beam towards the wall. In the case of a ray, the ray can touch the wall, that is, the virtual ray can act on the wall.

In some examples, in response to the emission operation of the ray, determining the intersection relationship between the ray and the target object includes: in the case where the ray does not act on the target object, determining that the intersection relationship between the ray and the target object is: disjoint .

Here, "not acting on" can be understood as the virtual ray cannot touch the target object. Figure 8 shows the schematic diagram 2 of the intersection relationship between the ray and the target object. As shown in Figure 8, after the virtual ray is emitted, the virtual ray cannot touch the target object, so there will be no intersection point between the virtual ray and the target object, that is, the virtual ray and the target object The intersection relationship of space or target plane is: disjoint. For example, assuming that the target object is the sky and the length of the virtual ray is 10 meters, if the user stands outdoors and the electronic device emits a 10-meter virtual ray toward the sky, the ray cannot touch the sky, that is, the virtual ray does not reach the sky. Act on the sky.

In the embodiment of the present application, determining the target form corresponding to the virtual object according to the form correspondence between the target object and the virtual object to be displayed in the virtual scene includes: determining the corresponding target form of the virtual object according to the form correspondence and combined with the intersection relationship. target form.

In some possible implementations, determining the target form corresponding to the virtual object based on the form correspondence relationship and the intersection relationship includes: when there is an intersection relationship between the ray and the target object, obtaining the intersection point of the target object and the ray. The intersecting surface where the intersection point is located is identified as a spatial attribute, and the intersecting surface is determined to belong to the target space; according to the fourth form correspondence relationship in the form correspondence relationship, the target form corresponding to the virtual object is determined to be the spatial form corresponding to the target space.

Here, spatial attributes can be understood as spatial properties or spatial geometric characteristics.

In this way, after the spatial properties of the real space are recognized, the target form of the virtual object can feedback the form that is most suitable for the real space based on the spatial properties. Under different spatial forms, the amount of information and presentation form of virtual objects will also change with the changes in the spatial form, which will allow applications/virtual objects/operation methods to display the combination of virtual and real in a more flexible and diverse way, along with the characteristics of the real space. Presenting a suitable appearance enhances the spatial adaptability of applications/virtual objects/operation methods.

In some possible implementations, identifying the intersection surface where the intersection point is located as a spatial attribute includes: if the spatial position, shape, size and other geometric characteristics of the intersection surface conform to the spatial attribute characteristics, then determining that the intersection surface where the intersection point is located is a spatial attribute. In other possible implementations, identifying that the intersection surface where the intersection point is located is a spatial attribute includes: determining that the intersection surface where the intersection point is located is a spatial attribute based on the output value of a pre-trained model. It should be understood that the methods for identifying whether the intersection surface where the intersection point is located is a spatial attribute are not limited to the above-mentioned ones. The embodiments of this application do not limit how to identify whether the intersection surface where the intersection point is located is a spatial attribute.

Here, the fourth form correspondence is the correspondence between the target space and the virtual object.

In some examples, the fourth form correspondence may include a correspondence between the target space and the application icon. Exemplarily, the fourth form of correspondence includes a correspondence between the size of the target space and the size of the application icon. For example, the proportional relationship between the size of the target space and the size of the application icon is 1:x, and x is less than or equal to 1. The specific proportional relationship can be adjusted adaptively according to the spatial properties of the target space, such as the ratio between the size of the indoor space and the size of the application icon. The proportional relationship between the size of the outdoor space and the size of the application icon is different from the proportional relationship between the size of the outdoor space and the size of the application icon. As another example, the fourth form correspondence includes a correspondence between the display style of the target space and the display style of the application icon. For example, the display style of the target space and the display style of the application icon are the same. For example, the target space and the application icon both present a realistic style. For another example, the display style of the target space and the display style of the application icon are different styles. For example, the target space presents a realistic style, and the application icon presents an abstract style. The above is only an illustrative description and does not limit all possible types of correspondence between the target space and the application icon, but is not exhaustive here.

In some examples, the fourth form of correspondence may include a correspondence between the target space and the specific content displayed by the application. Here, the specific content displayed by the application is the specific information presented after opening the application. Exemplarily, the fourth form of correspondence includes a correspondence between the size of the target space and the size of specific content displayed by the application. For example, the size of the target space and the size of the specific content displayed by the application are proportional to 1:y, and y is less than or equal to 1. The specific proportional relationship can be adjusted adaptively according to the spatial properties of the target space and the nature of the specific content displayed by the application. For example, the proportional relationship between the size of the indoor space and the size of the specific content displayed by the application is different from the proportional relationship between the size of the outdoor space and the size of the specific content displayed by the application. For another example, when the nature of the specific content displayed by the application is images or text, the size of the target space is in a certain proportion to the size of the specific content displayed by the application. As another example, the fourth form correspondence includes a correspondence between the display style of the target space and the display style of the specific content displayed by the application. For example, the display style of the specific content displayed by the application and the display style of the target space should be the same style. For example, both the target space and the application icon should be in a realistic style. For another example, the display style of the target space and the display style of the specific content displayed by the application are different styles. For example, the target space is in a realistic style, and the specific content displayed in the application is in an abstract style. The above is only an illustrative description and does not limit all possible types of correspondence between the target space and the specific content displayed by the application, but is not exhaustive here.

In this way, when it is determined based on the intersection point that the intersecting surface belongs to the target space, based on the fourth form correspondence between the target space and the virtual object, the target form corresponding to the virtual object is determined to be the space form corresponding to the target space, so that the virtual object corresponds The target shape is adapted to the corresponding spatial shape of the target space, which in turn helps to improve the matching between virtual objects and target objects in real scenes, thereby improving the display effect of combining virtual and real objects.

In some possible implementations, determining the target form corresponding to the virtual object according to the morphological correspondence relationship and the intersection relationship includes: when there is an intersection relationship between the ray and the target object, obtaining the intersection point of the target object and the ray; identifying The intersecting surface where the intersection point is located belongs to the plane attribute, and it is determined that the intersecting surface belongs to the target plane; according to the fifth form correspondence in the form correspondence, it is determined that the target form corresponding to the virtual object is the planar form corresponding to the target plane.

Here, planar properties can be understood as planar properties or having planar geometric characteristics.

In this way, after the plane properties of the real plane are recognized, the target shape of the virtual object can feed back the shape that is most suitable for the real plane based on the plane properties. Under different plane forms, the amount of information and presentation form of virtual objects will also change with the change of plane form, which will allow applications/virtual objects/operation methods to display the combination of virtual and real in a more flexible and diverse way, and follow the real plane. It presents a suitable appearance based on its characteristics, which enhances the plane adaptability of applications/virtual objects/operation methods.

In some possible implementations, identifying that the intersection surface where the intersection point is located is a plane attribute includes: if the geometric characteristics such as the plane position, shape, and size of the intersection surface conform to the plane attribute characteristics, then determining that the intersection surface where the intersection point is located is a plane attribute. In some other possible implementations, identifying that the intersection surface where the intersection point is located is a planar attribute includes: identifying that the intersection surface where the intersection point is located is a planar attribute based on the output value of a pre-trained model. It should be understood that the method of identifying whether the intersection surface where the intersection point is located is a planar attribute is not limited to the above. The embodiment of the present application does not limit how to identify whether the intersection surface where the intersection point is located is a planar attribute.

Here, the fifth form correspondence is the correspondence between the target plane and the virtual object.

In some examples, the fifth form correspondence may include a correspondence between the target plane and the application icon. Exemplarily, the fifth form correspondence includes a correspondence between the size of the target plane and the size of the application icon. For example, the proportional relationship between the size of the target plane and the size of the application icon is 1:z, and z is less than or equal to 1. The specific proportional relationship can be adjusted adaptively according to the flat properties of the target plane, such as between the size of the desktop and the size of the application icon. The proportional relationship is different from the proportional relationship between the size of the wall and the size of the application icon. As another example, the fifth form correspondence includes a correspondence between the display style of the target plane and the display style of the application icon. For example, the display style of the target plane and the display style of the application icon are the same style. For example, the target plane and the application icon are both in a realistic style. For another example, the display style of the target plane and the display style of the application icon are different styles. For example, the target plane has a cool color style and the application icon has a warm color style. The above is only an illustrative description and does not limit all possible types of correspondence between the target plane and the application icon, but is not exhaustive here.

In some examples, the fifth form correspondence may include a correspondence between the target plane and the specific content displayed by the application. Here, the specific content displayed by the application is the specific information presented after opening the application. Exemplarily, the fifth form of correspondence includes the correspondence between the size of the target plane and the size of the specific content displayed by the application. For example, the size of the target plane and the size of the specific content displayed by the application are in a proportional relationship 1:k,k Less than or equal to 1. Specific proportional relationships can be adapted based on the planar nature of the target plane and the nature of the specific content displayed by the application. For example, the proportional relationship between the size of the wall and the size of the specific content displayed in the application is different from the proportional relationship between the size of the floor and the size of the specific content displayed in the application. For another example, when the nature of the specific content displayed by the application is images or text, the size of the target plane is proportional to the size of the specific content displayed by the application. There is a certain proportional relationship. As another example, the fifth form of correspondence includes the correspondence between the display style of the target plane and the display style of the specific content displayed by the application. For example, the display style of the specific content displayed by the application and the display style of the target plane should be the same. The specific content displayed on the target plane and application is in an abstract style. For another example, the display style of the target plane and the display style of the specific content displayed by the application are different styles. For example, the target plane has a realistic style, and the specific content displayed by the application has an abstract style. The above is only an illustrative description and does not limit all possible types of correspondence between the target plane and the specific content displayed by the application, but is not exhaustive here.

In this way, when it is determined based on the intersection point that the intersecting surface belongs to the target plane, based on the fifth form correspondence between the target plane and the virtual object, the target form corresponding to the virtual object is determined to be the plane form corresponding to the target plane, so that the virtual object corresponds The target shape is adapted to the plane shape corresponding to the target plane, which helps to improve the matching between the virtual object and the target object in the real scene, thereby improving the display effect of combining virtual and real objects.

In some possible implementations, determining the target form corresponding to the virtual object based on the form correspondence relationship and the intersection relationship includes: when the ray does not intersect the target object, based on the sixth form correspondence relationship in the form correspondence relationship, The target form corresponding to the virtual object is determined to be a predefined form corresponding to the predefined shape.

Here, the sixth form correspondence is the correspondence between the virtual object and the predefined shape. Here, virtual objects include but are not limited to application icons, specific content displayed by the application, etc. For example, the virtual object C corresponds to the predefined shape c, and the virtual object D corresponds to the predefined shape d.

Here, the predefined shapes include but are not limited to cloud shapes, rainbow shapes, building shapes, animal shapes, plant shapes, etc.

For example, if the predefined shape is a cloud shape, then the target shape of the virtual object is a cloud shape; if the predefined shape is an animal shape, then the target shape of the virtual object is an animal shape. The above is only an illustrative description, and does not serve as a limitation on all possible types of correspondences between virtual objects and predefined shapes, but is not exhaustive here.

In this way, when there is no target space or target plane in the real scene, according to the sixth form correspondence relationship, the target form corresponding to the virtual object is determined to be the predefined form corresponding to the predefined shape, so that the target form corresponding to the virtual object is consistent with the predefined shape. The defined forms are consistent, which in turn helps to improve the plasticity of virtual objects, thereby improving the display effect of combining virtual and real objects.

In some embodiments, the display method may further include: identifying the display parameters of the target object in the real scene; and adjusting the target form corresponding to the virtual object according to the display parameters. Here, the display parameters may include at least one of a display size and a display effect of the target object.

Here, the display size can be understood as the fixed size used to determine the size of the object. Taking the target object as the desktop as an example, the display size of the target object is the length and width of the desktop. Taking the target object indoors as an example, the display size of the target object is the length, width, and height dimensions of the room. It should be noted that the display size of the target object here is the size of the target object displayed by the display component of the electronic device.

Here, the display effect can be understood as the display style. For example, realistic style, abstract style, cool color style, warm color style, etc. It should be noted that the display effect of the target object here is the display effect of the target object displayed by the display component of the electronic device.

In some examples, adjusting the target form corresponding to the virtual object includes: adjusting at least one of the display size and display effect of the virtual object.

In some examples, adjusting the display size of the virtual object may be based on the display size of the target object (such as the target plane). For example, the display size of the target object has a certain proportional relationship with the display size of the virtual object in the target form. For example, when the target object is the ceiling or the floor, the proportional relationship between the size (or size) of the virtual object and the size (or size) of the target object is 1:1. For another example, when the target object is a desktop or a wall, the proportional relationship between the size of the virtual object and the size of the target object is m:1, and m is less than 1. It should be noted that the proportional relationship can be determined based on the characteristics of the virtual object. For example, when the virtual object is a speaker, the speaker should be smaller than the target object (desktop). For another example, when the virtual object is a calendar, the calendar can be smaller than or equal to the target object (wall).

In some examples, adjusting the display size of the virtual object may also be based on the display size of the target object (such as the target space). For example, the display size of the target object has a certain proportional relationship with the display size of the virtual object in the target form. For example, when the target object is inside a house, the proportional relationship between the display size of the virtual object and the display size of the target object is n1:1, and n1 is less than 1. For another example, when the target object is outside the house, the proportional relationship between the display size of the virtual object and the display size of the target object is n2:1, n2 is less than 1, and n2 is greater than n1. It should be noted that the proportional relationship can be determined based on the characteristics of the virtual object. For example, when the virtual object is a cloud, the cloud should be smaller than the indoor area. For another example, when the virtual object is a building, the building can be smaller than or equal to the outdoor space.

In some examples, adjusting the display effect of the virtual object may be based on the display style of the target object. For example, adjusting the display effect of the virtual object includes: adjusting the display style of the virtual object according to the display style of the target object, so that the display style of the target object is consistent with the display style of the virtual object. For example, if the display style of the target object is a realistic style, then the display style of the virtual object is also adjusted to a realistic style. As another example, adjusting the display effect of the virtual object includes: adjusting the display style of the virtual object according to the display style of the target object, so that the display style of the target object contrasts with the display style of the virtual object. For example, if the display style of the target object is a cool color style, adjust the display style of the virtual object to a warm color style.

In this way, adjusting the target form corresponding to the virtual object according to the display parameters of the target object in the real scene can make the target form corresponding to the virtual object adapt to the display parameters corresponding to the target plane, which in turn helps to improve the relationship between the virtual object and the real scene. The matching degree of the target object, thereby improving the display effect of combining virtual and real objects.

In some embodiments, performing image rendering based on the real scene and the virtual scene includes: collecting the first real scene image in the real scene; performing image rendering on the first real scene image and the virtual object to be displayed in the virtual scene to obtain the virtual object and the virtual scene. The target form corresponding to the first real scene image.

Here, the first real scene image is collected through an image collector of the electronic device, such as a camera component.

In some possible implementations, when the first real-scene image is the target space, the target form of the virtual object corresponding to the first real-scene image is the spatial form. For example, if the first real scene image is outdoors, the virtual object is in a three-dimensional form, such as a cloud form. For example, if the first real-scene image is a table, the virtual object is in a three-dimensional form, such as an ornament.

In some possible implementations, when the first real-scene image is a target plane, the target form of the virtual object corresponding to the first real-scene image is a planar form. For example, if the first real-scene image is a wall, the virtual object is in a planar form, such as a mural form. In another example, if the first real-scene image is the ground, the virtual object is in a plane shape, such as a carpet shape.

In some possible implementations, image rendering is performed on the first real scene image and the virtual object to be displayed in the virtual scene to obtain the target form corresponding to the virtual object and the first real scene image, including: based on the three-dimensional image extracted from the first real scene image. information to determine the three-dimensional information of the virtual object; create a virtual object based on the three-dimensional information of the virtual object, and obtain the target form of the virtual object corresponding to the first real scene image.

In some possible implementations, image rendering is performed on the first real-scene image and the virtual object to be displayed in the virtual scene to obtain the target form corresponding to the virtual object and the first real-scene image, including: obtaining the initial three-dimensional information of the virtual object, creating Virtual object; adjust the size and position of the virtual object according to the three-dimensional information extracted from the first real-scene image, and obtain the target form of the virtual object corresponding to the first real-scene image.

It should be noted that the image rendering method is not limited to the above methods. An image rendering method with an image rendering processing function can be used for image rendering to obtain a virtual object in a target form corresponding to the first real scene image.

In this way, the target form corresponding to the virtual object can be adapted to the first real scene image in the real scene, thereby helping to improve the matching degree between the virtual scene and the real scene, thereby improving the display effect of combining virtual and real scenes.

In some examples, displaying the target form corresponding to the virtual object includes: configuring to display at least one image in a display area of the display device. Here, at least one image includes: a target form corresponding to the virtual object.

Here, the display device may be a device with a virtual and real display function. For example, the display device may be a head-mounted display device, such as AR glasses, AR helmet, etc.

Here, the image includes: at least one of a two-dimensional image and a three-dimensional image.

For example, the two-dimensional image of the virtual object is displayed in the display area, and at the same time, the two-dimensional image or the three-dimensional image of the target object is also displayed.

In another example, the three-dimensional image of the virtual object is displayed in the display area, and at the same time, the two-dimensional image or the three-dimensional image of the target object is also displayed.

In this way, the target form corresponding to the virtual object can be displayed at the first position of the target object in the real scene, so that the target form corresponding to the virtual object can adapt to the target object in the real scene, thereby helping to improve the relationship between the virtual scene and the real scene. The degree of integration improves the display effect of virtual and real combination.

In some embodiments, the display method may further include: in response to receiving a switching instruction to move the virtual object from the first target object to the second target object, changing the target form of the virtual object from the one displayed on the first target object. The first target form is changed to a second target form displayed on the second target object. As shown in Figure 9, the processing steps may include:

S901: In response to the switching instruction of moving the virtual object from the first position of the first target object to the second position of the second target object, determine the second target form corresponding to the virtual object in the virtual scene;

S902: Display the second target form corresponding to the virtual object at the second position of the second target object.

In some examples, the above-mentioned S901 and S902 may be executed after S203.

Here, the switching instruction can be issued through the electronic device. For example, the electronic device outputs the switching instruction after receiving the user's voice switching instruction information. The switching command can also be issued through a controller connected to the electronic device, such as a handle. If the handle is moved from the first direction to the second direction, the switching command is output, and the second direction is different from the first direction.

For example, the first target object is the wall, and the second target object is the tabletop or the ground. In another example, the first target object is indoors, and the second target object is outdoor.

Figure 10 shows a schematic diagram of the weather application switching from the first position to the second position. As shown in Figure 10, when the target object is a wall, the weather application takes on a pendant form; when the weather application is moved from the wall to the desktop , on the desktop, the weather application takes the form of a decoration. It can be seen that after the weather application is moved to different locations, it can change its shape according to the changes in spatial or plane semantics. The presentation of a virtual object or application changes as a result of the plane semantics obtained at the intersection point (such as a desktop or a wall). At the same time, the effect of the presentation (such as size) changes by changing the size of the recognized plane. It should be understood that the switching shown in FIG. 10 is only an exemplary illustration and is not intended to limit the types of switching and presentation forms, but is not exhaustive here.

In this way, when the position of the target object changes, the form of the virtual object also changes accordingly. The form of the virtual object can take on different forms depending on the spatial properties/planar properties of the real world. The presentation form of the virtual object is more in line with the real world. The physical properties of virtual objects not only enrich the presentation form of virtual objects, but also enrich the display diversity.

In some embodiments, the display method may further include: displaying a third target form of the virtual object during the movement of the virtual object from the first target object to the second target object, and the third target form is the first target form. The intermediate form with the second target form. As shown in Figure 11, processing steps may include:

S1101: Generate a third target form of the virtual object, which is an intermediate form between the first target form and the second target form;

S1102: Display the third target form while the virtual object moves from the first position of the first target object to the second position of the second target object.

In some examples, the above-mentioned S1101 and S1102 may be executed after S901 and before S902.

Here, the third target form is another target form different from the first target form and the second target form. For example, the third target form is a form related to the first target form or the second target form, such as a form obtained by deformation based on the first target form or the second target form. As another example, the third target form may also be a form that is not related to the first target form and the second target form, such as a preset form.

In some possible implementations, generating a third target form of the virtual object includes: generating a third target form based on the first target form and the second target form. In this way, the third target form is related to both the first target form and the second target form.

In other possible implementations, generating the third target form of the virtual object includes: deforming based on the first target form to obtain the third target form. In this way, the third target form is deformed based on the first target form, and the second target form is displayed only when it moves to the second position. In other possible implementations, generating the third target form of the virtual object includes: deforming based on the second target form to obtain the third target form. In this way, the third target form is deformed based on the second target form, and the second target form is displayed only when it moves to the second position, so that the form related to the second target form is displayed during the movement.

In still some possible implementations, generating the third target form of the virtual object includes: selecting a third target form set from a preset third target form set as the third target form of the virtual object. In this way, the preset third target form can be quickly obtained and presented.

In this way, the virtual object takes on an intermediate form while moving from the first target object to the second target object, which further enriches the presentation form of the virtual object and also enriches the display diversity.

An embodiment of the present application provides a display device. The display device can be implemented as all or part of the terminal device through software, hardware, or a combination of both. Figure 12 shows a structural block diagram of a display device. As shown in Figure 12, the display device includes: a first determination unit 1210 configured to detect a target object in a real scene; a second determination unit 1220 configured to detect a target object according to the target The form correspondence between the object and the virtual object to be displayed in the virtual scene determines the target form corresponding to the virtual object; the control unit 1230 is configured to perform image rendering according to the real scene and the virtual scene, and display the target form corresponding to the virtual object. .

Here, the target object includes a target space or a target plane.

In some examples, the virtual scene includes any of AR scenes and MR scenes.

In some examples, the second determining unit 1220 is configured to confirm that the virtual object to be displayed is the first target form if the virtual object is to be displayed at the first position of the first target object; if the virtual object is to be displayed at the first position of the first target object, If the second target object is in the first position, it is confirmed that the virtual object to be displayed is the second target form. Among them, the first target form and the second target form are different.

In some examples, the second determination unit 1220 is configured to, if the attribute of the target object is a three-dimensional form, determine that the target form corresponding to the virtual object is a three-dimensional form according to the first form correspondence in the form correspondence.

In some examples, the second determination unit 1220 is configured to determine that the target form corresponding to the virtual object is a two-dimensional form according to the second form correspondence in the form correspondence if the attribute of the target object is a two-dimensional form.

In some examples, the second determining unit 1220 is configured to determine that the target form corresponding to the virtual object is a predefined form corresponding to the predefined shape according to the third form correspondence relationship in the form correspondence relationship.

In some examples, the display device may further include an emission unit 1240 (not shown in the figure) configured to emit rays.

In some examples, the display device may further include: a third determination unit 1250 (not shown in the figure) configured to, in response to the emission operation of the ray, confirm the target when the ray intersects with the target object in the real scene. The first position corresponding to the object.

In some examples, the third determination unit 1250 is further configured to obtain the intersection relationship between the ray and the target object in response to the emission operation of the ray.

In some examples, the third determining unit 1250 is configured to determine that the intersection relationship between the ray and the target object is: intersection when the ray operation is configured as the target object.

In some examples, the third determination unit 1250 is configured to determine that the intersection relationship between the ray and the target object is: no intersection when the ray is not configured as the target object.

In some examples, the second determination unit 1220 is configured to: when the ray intersects the target object, obtain the intersection point of the target object and the ray; identify the intersection surface where the intersection point is located as a spatial attribute, and determine that the intersection surface belongs to Target space; according to the fourth form correspondence relationship in the form correspondence relationship, the target form corresponding to the virtual object is determined to be the three-dimensional form corresponding to the target space.

In some examples, the second determination unit 1220 is configured to: when the ray intersects the target object, obtain the intersection point of the target object and the ray; identify that the intersection surface where the intersection point is located belongs to the plane attribute, and determine the intersection surface Belongs to the target plane; according to the fifth form correspondence in the form correspondence, it is determined that the target form corresponding to the virtual object is a two-dimensional form corresponding to the target plane.

In some examples, the second determination unit 1220 is configured to: when the ray does not intersect the target object, determine that the target form corresponding to the virtual object is the same as the predefined form according to the sixth form correspondence in the form correspondence. The predefined shape corresponding to the shape.

In some examples, the display device further includes: a recognition unit 1260 (not shown in the figure), configured to recognize the display parameters of the target object in the real scene; an adjustment unit 1270, configured to adjust the virtual display parameters according to the display parameters. The target form corresponding to the object. Here, the display parameters include: at least one of a display size and a display effect of the target object.

In some examples, the control unit 1230 includes: a collection subunit configured to collect a first real scene image in a real scene; a rendering subunit configured to combine the first real scene image with a virtual object to be displayed in the virtual scene Image rendering is performed to obtain the target form corresponding to the virtual object and the first real scene image.

In some examples, when the first real-scene image is the target space, the target form of the virtual object corresponding to the first real-scene image is the spatial form.

In some examples, when the first real-scene image is a target plane, the target form of the virtual object corresponding to the first real-scene image is a planar form.

In some examples, the control unit 1230 includes: a configuration subunit configured to display at least one image in a display area of the display device. Here, at least one image includes: a target form corresponding to the virtual object.

In some examples, the control unit 1230 is further configured to, in response to a switching instruction for moving the virtual object from the first target object to the second target object, change the target form of the virtual object from the first target form displayed on the first target object. One target form is changed to a second target form displayed on a second target object.

In some examples, the control unit 1230 is further configured to display a third target form of the virtual object during the movement of the virtual object from the first target object to the second target object, and the third target form is the first target form. The intermediate form with the second target form.

Those skilled in the art should understand that the functions of each processing module in the display device according to the embodiment of the present application can be understood with reference to the relevant description of the foregoing display method. Each processing module in the display device according to the embodiment of the present application can be implemented by implementing the present application. The functions described in the embodiments of this application are realized by using analog circuits, or they can also be realized by running software on electronic devices that perform the functions described in the embodiments of this application.

The display device of the embodiment of the present application enables the displayed target form corresponding to the virtual object to change following the change of the target object, enriches the diversity of the target form corresponding to the virtual object, and thereby improves the performance of the virtual object in the real scene. display effect.

The embodiment of the present application provides a display device that can be worn by users. Figure 13 shows a structural block diagram of the display device. As shown in Figure 13, the display device includes: a first sensing unit 1310, a processing unit 1320, The display unit 1330, wherein the first sensing unit 1310 is configured to emit laser; the processing unit 1320 is configured to determine the virtual object correspondence according to the morphological correspondence between the target object and the virtual object to be displayed in the virtual scene. The target form; the display unit 1330 is configured to perform image rendering according to the real scene and the virtual scene, and display the target form corresponding to the virtual object.

Here, the target object includes a target space or a target plane.

In some examples, first sensing unit 1310 may be a laser sensor.

In some examples, the display device may include at least one of the following: a pair of glasses configured to display an augmented reality scene or a virtual reality scene; a head-mounted display configured to display an augmented reality scene or a virtual reality scene.

In some examples, the display unit 1330 is configured to display at least one virtual object on the display area. Here, the type of at least one virtual object includes: application or application content in an application.

In some examples, the processing unit 1320 is configured to, if the attribute of the target object is a three-dimensional form, determine that the target form corresponding to the virtual object is a three-dimensional form according to the first form correspondence in the form correspondence.

In some examples, the processing unit 1320 is configured to, if the attribute of the target object is a two-dimensional form, determine that the target form corresponding to the virtual object is a two-dimensional form according to the second form correspondence in the form correspondence.

In some examples, the processing unit 1320 is configured to determine, according to the third form correspondence in the form correspondence, that the target form corresponding to the virtual object is a predefined form corresponding to the predefined shape.

In some examples, the first sensing unit 1310 is configured to emit rays in response to a selection operation on the at least one virtual object.

In some examples, the processing unit 1320 is further configured to, in response to the ray emission operation, confirm the first position corresponding to the target object when the ray intersects with the target object in the real scene.

In some examples, the processing unit 1320 is configured to determine the intersection point as the first position corresponding to the target object if there is an intersection point between the ray and the target object; if there is no intersection point between the ray and the target object, Then the point closest to the target object on the ray is confirmed as the first position corresponding to the target object.

In some examples, the processing unit 1320 is configured to determine that the intersection relationship between the ray and the target object is: intersection when the ray operation is configured as the target object.

In some examples, the processing unit 1320 is configured to determine that the intersection relationship between the ray and the target object is: disjoint when the ray is not configured as the target object.

In some examples, the processing unit 1320 is configured to: when the ray intersects the target object, obtain the intersection point of the target object and the ray; if it is identified that the intersection surface where the intersection point is located is a spatial attribute, determine that the intersection surface belongs to Target space; according to the fourth form correspondence in the form correspondence, determine the application or the application content in the application to be a three-dimensional form corresponding to the target space.

In some examples, the processing unit 1320 is configured to: when the ray intersects the target object, obtain the intersection point of the target object and the ray; if it is identified that the intersection surface where the intersection point is located belongs to the plane attribute, determine the intersection surface Belongs to the target plane; according to the fifth form correspondence in the form correspondence, it is determined that the application or the application content in the application is a two-dimensional form corresponding to the target plane.

In some examples, the processing unit 1320 is configured to: when the ray does not intersect the target object, determine that the application or the application content in the application is consistent with the predefined shape according to the sixth morphological correspondence in the morphological correspondence. The corresponding predefined form.

In some examples, the processing unit 1320 is configured to: identify the display parameters of the target object in the real scene; and adjust the target form corresponding to the virtual object according to the display parameters. Here, the display parameters may include: at least one of a display size and a display effect of the target object.

In some examples, the display device further includes: a second sensing unit 1340 (not shown in the figure) configured to collect the first real scene image in the real scene. The display unit 1330 is configured to perform image rendering on the first real scene image and the application to be displayed in the virtual scene or the application content in the application, to obtain a target form corresponding to the application or the application content in the application and the first real scene image.

In some examples, the second sensing unit 1340 may be a collector, such as a camera.

In some examples, when the first real-scene image is the target space, the target form corresponding to the application or the application content in the application and the first real-scene image is the spatial form.

In some examples, when the first real-scene image is a target plane, the target form corresponding to the application or the application content in the application and the first real-scene image is a planar form.

In some examples, the processing unit 1320 is configured to display at least one image in a display area of the display device. Here, at least one image includes: a target form corresponding to the virtual object.

In some examples, the display device further includes a switching unit 1340 configured to: in response to a switching instruction to move the virtual object from a first position of the first target object to a second position of the second target object, Determine the second target form corresponding to the virtual object in the virtual scene. The display unit 1330 is further configured to display the second target form corresponding to the virtual object at the second position of the second target object.

In some examples, the display device may further include: a generating unit 1350 (not shown in the figure) configured to generate a third target form of the virtual object, the third target form being the first target form and the second target form. intermediate form. The display unit 1330 is further configured to display the third target form while the virtual object moves from the first position of the first target object to the second position of the second target object.

The display device of the embodiment of the present application enables the displayed target form corresponding to the virtual object to change following the change of the target object, enriches the diversity of the target form corresponding to the virtual object, and thereby improves the performance of the virtual object in the real scene. display effect.

Embodiments of the present application also provide a head-mounted device, including a processor, a memory, a display, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the processor. The program includes instructions configured to perform steps in the display methods of the above-described embodiments. The head-mounted devices include but are not limited to smart glasses, smart helmets, etc.

Figure 14 shows a structural block diagram of an electronic device used to implement the display method according to the embodiment of the present application. It can be implemented as the electronic device in the above embodiment. The electronic device in this application can include one or more of the following components: a processor 1410 and a memory 1420.

Processor 1410 may include one or more processing cores. The processor 1410 uses various interfaces and lines to connect various parts of the entire electronic device, and executes electronic devices by running or executing instructions, programs, code sets or instruction sets stored in the memory 1420, and calling data stored in the memory 1420. Various functions of the device and processing data. Optionally, the processor 1410 may adopt at least one hardware selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). form to achieve. The processor 1010 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), a neural network processor (Neural-network Processing Unit, NPU), a modem, etc. Among them, the CPU mainly handles the operating system, user interface and applications; the GPU is used to render and draw the content that needs to be displayed on the touch screen; the NPU is used to implement artificial intelligence (Artificial Intelligence, AI) functions; the modem is used to process Wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 1410 and may be implemented by a separate chip.

The memory 1420 may include random access memory (RAM) or read-only memory (Read-Only Memory, ROM). Optionally, the memory 1020 includes non-transitory computer-readable storage medium. Memory 1420 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 1420 may include a program storage area and a data storage area, where the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), Instructions, etc., used to implement each of the following method embodiments; the storage data area can store data created according to the use of electronic devices (such as audio data, phone books), etc.

The electronic device in the embodiment of the present application also includes a camera component 1430, which is used to collect environmental images. In one possible implementation, when the electronic device is a terminal, the camera component 1430 may be a front camera or a rear camera of the terminal; in another possible implementation, when the electronic device is a helmet-mounted display (Head -Mounted Display (HMD)), the camera component 1430 may be a camera disposed on the front of the HMD.

In addition, the electronic device may also include a display component 1440, which may include a display screen for displaying images, or may include a projection device (such as a projector on smart glasses) for image projection.

In addition, those skilled in the art can understand that the structure of the electronic device shown in the above drawings does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than those shown in the figures, or a combination of certain components. components, or different component arrangements. For example, electronic equipment also includes radio frequency circuits, input units, sensors, audio circuits, speakers, microphones, power supplies and other components, which will not be described in detail here.

Embodiments of the present application also provide a computer-readable storage medium that stores at least one instruction, and at least one instruction is used to be executed by a processor to implement the display method as described in the above embodiments.

Embodiments of the present application provide a computer program product or computer program. The computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the display method provided by the above embodiment.

Those skilled in the art should realize that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media can be any available media that can be accessed by a general purpose or special purpose computer.

The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection of the present application. within the range.

Claims (19)

A display method including: Detect target objects in real scenes; Determine the target form corresponding to the virtual object according to the form correspondence between the target object and the virtual object to be displayed in the virtual scene; Image rendering is performed according to the real scene and the virtual scene, and the target form corresponding to the virtual object is displayed. The method according to claim 1, wherein determining the target form corresponding to the virtual object based on the form correspondence between the target object and the virtual object to be displayed in the virtual scene includes: If the virtual object is to be displayed at the first position of the first target object, it is confirmed that the virtual object to be displayed is the first target form; If the virtual object is to be displayed at the first position of the second target object, confirm that the virtual object to be displayed is the second target form; Wherein, the first target form and the second target form are different. The method according to claim 1 or 2, further comprising: In response to the ray emission operation, when the ray intersects with the target object in the real scene, the first position corresponding to the target object is confirmed. The method according to claim 3, wherein in response to the emission operation of the ray, when the ray intersects with the target object in the real scene, confirming the first position corresponding to the target object includes: If there is an intersection point between the ray and the target object, the intersection point is confirmed as the first position corresponding to the target object; If there is no intersection point between the ray and the target object, the point on the ray that is closest to the target object is determined as the first position corresponding to the target object. The method according to claim 1 or 4, wherein determining the target form corresponding to the virtual object based on the form correspondence between the target object and the virtual object to be displayed in the virtual scene includes: If the attribute of the target object is a three-dimensional form, it is determined that the target form corresponding to the virtual object is a three-dimensional form according to the first form correspondence relationship among the form correspondence relationships. The method according to claim 1 or 4, wherein determining the target form corresponding to the virtual object based on the form correspondence between the target object and the virtual object to be displayed in the virtual scene includes: If the attribute of the target object is a two-dimensional form, it is determined that the target form corresponding to the virtual object is a two-dimensional form according to the second form correspondence relationship in the form correspondence relationship. The method according to claim 1 or 4, wherein determining the target form corresponding to the virtual object based on the form correspondence between the target object and the virtual object to be displayed in the virtual scene includes: According to the third form correspondence among the form correspondences, it is determined that the target form corresponding to the virtual object is a predefined form corresponding to a predefined shape. The method according to any one of claims 1-7, wherein the method further includes: Identify the display parameters of the target object in the real scene; According to the display parameters, adjust the target form corresponding to the virtual object; Wherein, the display parameters include: at least one of a display size and a display effect of the target object. The method according to any one of claims 1-7, wherein the image rendering based on the real scene and the virtual scene includes: Collect the first real scene image in the real scene; The first real scene image and the virtual object to be displayed in the virtual scene are image rendered to obtain a target form corresponding to the virtual object and the first real scene image. The method according to claim 9, wherein when the first real-scene image is a target space, the target form of the virtual object corresponding to the first real-scene image is a spatial form; or, When the first real-scene image is a target plane, the target form of the virtual object corresponding to the first real-scene image is a planar form. The method according to any one of claims 1-7, wherein the displaying the target form corresponding to the virtual object includes: configured to display at least one image in the display area of the display device; The at least one image includes: a target form corresponding to the virtual object. The method of claim 1, further comprising: In response to the switching instruction of moving the virtual object from the first target object to the second target object, the target form of the virtual object is changed from the first target form displayed on the first target object to the first target form displayed on the first target object. The secondary target form displayed on the secondary target object. The method of claim 12, further comprising: In the process of moving the virtual object from the first target object to the second target object, a third target form of the virtual object is displayed, and the third target form is the combination of the first target form and the second target form. The intermediate form of the second target form. A display device, wherein the display device is configured to perform the steps of the method of any one of claims 1 to 13. A display device is configured to be worn by a user, wherein the display device includes the display device described in claim 14. The display device according to claim 15, wherein the display device includes at least one of the following: a pair of glasses configured to display an augmented reality scene or to display a virtual reality scene; A head-mounted display configured to display an augmented reality scene or to display a virtual reality scene. A head mounted device, including a processor, a memory, a display, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the The program includes instructions configured to perform the steps of the method of any one of claims 1-13. A computer-readable storage medium, wherein the computer-readable storage medium is configured to store a computer program that, when executed by a processor, implements the method according to any one of claims 1-13. A computer program product, wherein the computer program product includes computer instructions stored in a computer-readable storage medium; and a processor of a computer device reads the computer instructions from the computer-readable storage medium, The processor executes the computer instructions, causing the computer device to perform the method according to any one of claims 1-13.

Images