CN118484117A - Method, device, equipment and storage medium for interaction in a virtual scene - Google Patents

Abstract

According to embodiments of the present disclosure, methods, apparatuses, devices, and storage medium for interacting in a virtual scene are provided. The method includes drawing a virtual object in a virtual scene, the virtual object being generated based on a real object in a physical scene. Predetermined operations made by a user in a physical scene with respect to a real object are detected. And in response to detecting the predetermined operation, manipulating at least one interface element associated with the virtual object in the virtual scene. Therefore, the interaction in the virtual scene of the user is realized by utilizing the real objects in the physical scene, and the interaction experience of the user is improved.

Description

Method, device, equipment and storage medium for interaction in a virtual scene

Technical Field

Example embodiments of the present disclosure generally relate to the field of computers, and more particularly to methods, devices, apparatuses, and computer-readable storage media for interacting in a virtual scene.

Background Art

Virtual Reality (VR), Extended Reality (XR), Augmented Reality (AR), and Mixed Reality (MR) can combine 3D graphics technology, multimedia technology, simulation technology, display technology, servo technology, and other technologies to produce a realistic virtual scene with multiple sensory experiences such as 3D vision, touch, and smell. For example, VR uses a computer to simulate a virtual world in 3D space, providing users with an immersive experience in vision, hearing, touch, and other aspects. AR allows real environments and virtual objects to be superimposed in the same space in real time and exist simultaneously. MR is a new visualization environment that integrates the real world and the virtual world, where objects in the physical real world scene coexist with objects in the virtual world in real time.

In this way, virtual scenes can be used to provide users with an immersive interactive experience.

Summary of the invention

In a first aspect of the present disclosure, a method for interacting in a virtual scene is provided. The method includes: drawing a virtual object in the virtual scene, where the virtual object is generated based on a real object in a physical scene. Detecting a predetermined operation performed by a user in the physical scene on the real object. And in response to detecting the predetermined operation, manipulating at least one interface element associated with the virtual object in the virtual scene.

In a second aspect of the present disclosure, a device for interacting in a virtual scene is provided. The device includes: a drawing unit configured to draw a virtual object in the virtual scene, where the virtual object is generated based on a real object in a physical scene; a detection unit configured to detect a predetermined operation performed by a user in the physical scene on the real object; and a manipulation unit configured to manipulate at least one interface element associated with the virtual object in the virtual scene in response to detecting the predetermined operation.

In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

It should be understood that the contents described in the summary of the present invention are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

FIG2 shows a flow chart of a process of interacting in a virtual scene according to some embodiments of the present disclosure;

FIG3 is a schematic diagram showing an example of determining a drawing size according to some embodiments of the present disclosure;

FIG4 is a schematic diagram showing an example of a set of interface elements according to some embodiments of the present disclosure;

5A, 5B, 5C, and 5D are schematic diagrams respectively showing examples of manipulation interface elements according to some embodiments of the present disclosure;

FIG6 is a schematic diagram showing an example of presentation content of a virtual scene associated with a second user according to some embodiments of the present disclosure;

FIG7 shows a block diagram of an apparatus for interacting in a virtual scene according to some embodiments of the present disclosure;

FIG8 shows a block diagram of a device capable of implementing various embodiments of the present disclosure.

DETAILED DESCRIPTION

It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

It should be noted that the titles of any sections/subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section/subsection. In addition, the embodiments described in any section/subsection may be combined in any manner with any other embodiments described in the same section/subsection and/or different sections/subsections.

In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

In the description of the embodiments of the present disclosure, the term "element", such as "icon element", "interface element", etc., is used to indicate the presentation content in a virtual scene. Usually, such presentation content is presented in the form of pixels. For example, an interface element may be an interface presented in a pixel presentation manner in a virtual scene. A user may view the interface and/or interact with the interface through the presented pixels in a virtual scene to achieve the purpose of obtaining and interacting with content in a virtual scene.

As briefly mentioned above, the virtual scene generation device can generate corresponding virtual scenes corresponding to different virtual scene usage requirements. For example, the virtual scene generation device generates a corresponding virtual conference room as a virtual scene according to the meeting requirements. The virtual conference room generates a virtual avatar of the user after the user connects, and the user can use the virtual avatar specific to himself to interact with the virtual avatars specific to other users.

After the generating device generates a virtual scene, the user is allowed to interact with the generating device through the interactive device. The generating device can determine the control content according to the instruction sent by the interactive device, and adjust the presentation elements in the virtual scene according to the control content to provide interaction corresponding to the user in the virtual scene. Usually, the generating device can generate corresponding indication elements in the virtual scene corresponding to the interactive device, and the user can use the interactive device to control the indication elements to send interactive instructions. For example, the generating device can generate an icon element corresponding to the interactive device including a virtual reality handle in the virtual scene. The user can realize the movement control of the icon element by moving the virtual reality handle in the physical scene, and can issue instructions to the presentation elements of the virtual scene indicated by the cursor by triggering the physical buttons in the virtual reality handle.

However, in such a configuration, the icon elements corresponding to the interactive device exist independently of the presentation elements in the virtual scene, so that the user's interaction perception of using the interactive device is inconsistent with the interaction perception of interacting with the presentation elements, resulting in a poor user interaction experience. For example, when the icon element is a virtual image of a hand, the user triggers the physical buttons in the virtual reality handle to achieve the "hand grabbing action" of the presentation element. The interaction perception obtained by the user based on visual feedback in the virtual scene is different from the interaction perception between the user and the handle, resulting in a poor user interaction experience. For another example, when the head-mounted display obtains the user's instructions by capturing the user's hand movements in the air, the user lacks corresponding interaction feedback in the physical scene, resulting in a poor user interaction experience.

The embodiments of the present disclosure propose a scheme for interaction in a virtual scene. According to various embodiments of the present disclosure, a virtual object generated based on a real object in a physical scene is drawn in a virtual scene. A predetermined interaction made by a user in the physical scene with respect to the real object is detected. And in response to detecting the predetermined interaction, at least one interface element associated with the virtual object in the virtual scene is manipulated. Therefore, the embodiments of the present disclosure utilize the real object in the physical scene to realize the interaction of the user in the virtual scene, thereby improving the user's interactive experience.

Example Environment

FIG1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In the environment 100, a physical scene 110 is included, which is an example of an actual scene in the real world, in which a user 111 is using an object (e.g., a finger) 114 to slide on the surface of a real object 113 (or an interactive surface on which the user can interact with the real object). The real object 113 is, for example, a desktop. Assuming that the user 111 is wearing an XR device 112, such as a head-mounted display, smart glasses, etc., the XR device 112 can present a virtual scene 120 corresponding to the physical scene 110 to the user 111.

The virtual scene 120 includes a virtual object 121 drawn based on a real object 113 in the physical scene 110. The virtual scene 120 also includes icon elements 122 and 123, which correspond to the hands of the user 111 (for example, icon element 122 corresponds to the left hand of the user 111, and icon element 123 corresponds to the right hand of the user 111), and are used to feedback the user's actions to the user 111 in the virtual scene 120. For example, a virtual hand object. In some embodiments, the virtual object includes a surface element corresponding to the interactive surface of the real object. For example, when the real object is a table, the interactive surface of the table can be, for example, a desktop of the table facing the user. Accordingly, the virtual object can include a surface element (for example, a virtual desktop) corresponding to the side of the desktop facing the user.

In the physical scene 110, an electronic device 115 is also included, which is used to obtain an image of the physical scene 110, and determine the relative position between the object 114 (e.g., finger) of the user 111 and the real object 113 (e.g., desktop) based on the image. The electronic device 115 can be a separate device that can communicate with the XR device 112 and/or other image capture devices, such as a server, computing node, etc. for image or data processing, or it can be integrated with the XR device 112 and/or other image capture devices. In some embodiments, the electronic device 115 can be implemented as an XR device 112, that is, in this case, the XR device 112 can implement all the functions of the electronic device 115. It should be understood that the above description of the electronic device 115 is merely exemplary and non-restrictive, and the electronic device 115 can be implemented as a variety of forms, structures or categories of devices, and the embodiments of the present disclosure are not limited to this.

The XR device 112 may adjust the presentation elements in the virtual scene 120 according to the actions performed by the user 111 on the real object 113 (e.g., a desktop) using an object 114 (e.g., a finger) in the physical scene 110. Such an object 114 may include, for example, at least one appropriate body part of the user, such as a finger, a wrist, a palm, etc.

For example, the XR device 112 adjusts the icon elements 122 and 123 used to present the hand motion of the user 111 in the virtual scene 120 according to the tapping motion of the user 111 using the object 114 (e.g., finger) in the physical scene 110. For example, the XR device 112 may present a feedback element 124 corresponding to the finger of the user 111 performing the tapping motion on the icon element 123. For example, in the case of contact where the user taps the real object 113 (e.g., desktop) using four objects 114 (e.g., fingers) of the "right hand" as shown in the example corresponding to FIG. 5A , the XR device 112 may present a set of feedback elements 124 on the virtual object 121 (e.g., virtual desktop).

In some embodiments, when the object 114 is embodied as a finger, the action may include at least one of tapping, sliding, and touching. Accordingly, the XR device 112 may configure feedback elements 124 corresponding to different actions. For example, when the action of the object 114 (e.g., a finger) is sliding, the feedback element 124 presented may be a bright bar that slides along with the icon elements 122 and 113.

In some embodiments, the XR device 112 is pre-configured with scene elements for generating a virtual scene, and the corresponding scene elements can be obtained according to the use requirements of the virtual scene to generate presentation elements to generate a virtual scene 120. For example, the virtual scene is a virtual conference room, and the scene elements may include desktop elements, seat elements, etc.

The real object 113 and the virtual scene 120 shown in FIG1 are merely exemplary and are not intended to limit the scope of the present disclosure. The XR device 112 can be applied to generate any virtual scene from any real object.

It should be understood that the structure and function of the environment 100 are described for exemplary purposes only, and do not imply any limitation on the scope of the present disclosure. Some exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings.

Example of interactive process and effect in virtual scene

FIG. 2 shows a process 200 for interacting in a virtual scene according to some embodiments of the present disclosure. The process 200 is implemented at the XR device 112 for exemplary description. Examples of interacting in a virtual scene according to some embodiments of the present disclosure are shown in FIG. 3 , FIG. 4 , FIG. 5A , FIG. 5B , FIG. 5C , FIG. 5D , and FIG. 6 , respectively. For ease of discussion, the process 200 will be described with reference to the environment 100 of FIG. 1 , and the examples in FIG. 3 , FIG. 4 , FIG. 5A , FIG. 5B , FIG. 5C , FIG. 5D , and FIG. 6 .

In block 210, a virtual object is drawn in the virtual scene, the virtual object being generated based on the real object in the physical scene. According to an embodiment of the present disclosure, the XR device 112 draws the virtual object determined based on the real object in the physical scene in the virtual scene 120. In some embodiments, the XR device 112 determines drawing information based on the type of the real object 113 and the size of the real object 113 in the physical scene 110, and draws the virtual object 121 in the virtual scene 120 accordingly based on the drawing information, so that the real object 113 and the virtual object correspond better.

In some embodiments, the XR device 112 may draw a virtual object according to the object size provided by the user 111 after the user 111 selects the type of virtual object. In this case, the user 111 may measure the size of the real object using the XR device 112 or other devices that can be used to provide measurement functions for the XR device 112, such as a surveying device. For example, in FIG3 , the user 111 (not shown) may use a surveying device 310 for providing a measurement function for the XR device 112 (not shown) to calibrate with the left edge position 320 of the real object 113 (e.g., a desktop) as the starting point, and horizontally drag the surveying device 310 to the right edge position 330 as the end point. In this case, the calibration results of the left edge position 320 and the right edge position 330 determine the size of the real object 113 (e.g., a desktop) in the horizontal direction.

In some embodiments, the user 111 may calibrate only the size of some objects as required, so as to more reasonably determine the operating space of the user 111 in the physical scene 110. For example, after selecting the left edge position 320 as the starting point, the user 111 may select the position 340 in the real object 113 (e.g., the desktop) as the end point, and determine the area between the position 320 and the position 340 as the size of the real object 113 (e.g., the desktop).

In some embodiments, the surveying and mapping device 310 can also preset a direction parameter to generate the above-mentioned size according to the calibration result of the user in a single direction. For example, in FIG. 3 , the surveying and mapping device 310 can preset a length 350 in the vertical direction as a size parameter. In this case, the user 111 can use the surveying and mapping device 310 to calibrate the left edge position 320 of the real object 113 (e.g., desktop), and continuously drag the surveying and mapping device 310 from the left edge to the right edge position 330. The surveying and mapping device 310 uses the size of the closed rectangle formed by the left edge position 320, the right edge position 330, and the length 350 in the vertical direction of the real object 113 (e.g., desktop) as the drawing size of the virtual object 121 (e.g., virtual desktop).

In some embodiments, the selection of the real object may be determined based on the user's instructions. For example, the XR device 112 captures the scene image of the real physical scene 110 in which the user is located and presents it to the user 111. The user 111 may select a real object 113 (e.g., a desktop) in the scene image and then control the XR device 112 to generate a corresponding virtual object 121 (e.g., a virtual desktop).

In box 220, a predetermined operation performed by a user in a physical scene on a real object is detected. According to an embodiment of the present disclosure, the XR device 112 detects a predetermined operation performed by a user in a physical scene on a real object. The XR device detects a predetermined operation performed by a user in a physical scene on a real object. The predetermined operation is a pre-configured corresponding instruction. The XR device can determine the instruction sent by the user based on the predetermined action performed by the user, so that the user can interact with the XR device 112 through the predetermined operation. For example, the predetermined operation includes at least one object 114 (e.g., finger) of the user 111 in contact with a real object 113 (e.g., desktop). In some embodiments of this case, in order to avoid false touches caused by a short contact between the object 114 (e.g., finger) of the user 111 and the real object 113 (e.g., desktop), the predetermined operation can also be configured accordingly to include a specific operation indicating that the contact between at least one finger of the user and the real object reaches a predetermined duration. For another example, the predetermined operation includes whether at least one object 114 (e.g., finger) of the user 111 slides on the real object 113 (e.g., desktop). In some embodiments of this case, in order to avoid false touches caused by finger movements without user interaction intentions, the predetermined operation may be configured to include indicating that at least one object 114 (e.g., finger) of the user 111 slides in a specific direction relative to a real object 113 (e.g., a desktop). This allows distinguishing the user's movements, more clearly understanding the user's interaction intentions, and reducing false triggers. This prevents non-predetermined movements in the physical scene from being detected, and prevents the XR device 112 from interacting with the user by mistake.

In some embodiments, detecting the predetermined operation includes: the XR device detects the relative position of at least one limb of the user and the real object. Further, the XR device detects the predetermined operation performed by the user on the real object in the physical scene based on the relative position. Specifically, as described above, the XR device 112 can communicate with the local and/or other image capture devices to collect images of the user's object 114 and the real object 113 (e.g., desktop) in the physical scene 110. As introduced above, such an object 114 may include, for example, at least one appropriate limb of the user, such as a finger, wrist, palm, etc. Further, after the XR device 112 determines the spatial positions of the object 114 (e.g., finger) and the real object 113 (e.g., desktop) based on the collected images, it determines the relative position of the object 114 (e.g., finger) and the real object 113 (e.g., desktop) based on the spatial position of the object 114 (e.g., finger) and the spatial position of the real object 113 (e.g., desktop).

The XR device 112 may determine the operation performed by the object 114 (e.g., finger) on the real object 113 (e.g., desktop) based on the relative position of the object 114 (e.g., finger) and the real object 113 (e.g., desktop). For example, if the spatial position of the object 114 (e.g., finger) is in the spatial position corresponding to the real object 113 (e.g., desktop), the relative position of the object 114 (e.g., finger) and the real object 113 (e.g., desktop) may be determined as the object 114 (e.g., finger) is placed on the real object 113 (e.g., desktop), that is, the object 114 (e.g., finger) and the real object 113 (e.g., desktop) have an action of "contact". In some embodiments, the XR device 112 may determine the action of the object 114 (e.g., finger) corresponding to the real object 113 (e.g., desktop) based on the relative position of the object 114 (e.g., finger) and the real object 113 (e.g., desktop) determined in the continuously captured images. For example, if it is determined based on the continuously captured images that the object 114 (e.g., finger) has a plurality of "contact" actions with the same contact points relative to the real object 113 (e.g., desktop), the action of the object 114 (e.g., finger) corresponding to the desktop can be determined as "continuous contact". For another example, if it is determined based on the continuously captured images that the object 114 (e.g., finger) has a plurality of "contact" actions with continuous and different contact points relative to the real object 113 (e.g., desktop), the action of the object 114 (e.g., finger) corresponding to the desktop can be determined as "sliding".

In this case, after recording the action of the object 114 (e.g., finger) on the real object 113 (e.g., desktop), the XR device 112 detects whether there is a preset operation in the recorded action, such as the contact between at least one finger and the real object for a predetermined time or the sliding of at least one finger in a specific direction.

In box 230, in response to detecting a predetermined operation, at least one interface element associated with the virtual object in the virtual scene is manipulated. In some embodiments of the present disclosure, the XR device 112 may configure the interface element corresponding to the virtual object 121, and establish an association between the manipulation action of the interface element and the predetermined operation. The interface element is used to provide information to the user and/or interact with the user in the virtual scene, such as prompt information and tool icons. For example, the interface element is used to present a tool icon, and the user 111 can call the tool corresponding to the tool icon in the virtual scene by, for example, "touching" the tool icon in the virtual scene 120, so as to realize the interaction in the virtual scene by using the interface element. The interactive method triggers that when the XR device 112 detects a predetermined operation, the XR device 112 manipulates the interface element accordingly according to the predetermined operation. For example, the XR device 112 calls out and presents the interface element in the virtual scene according to the predetermined operation.

In some embodiments, the presentation position of the interface element is associated with the virtual object. For example, when the virtual object is a virtual desktop, the presentation position of the interface element is set on the virtual desktop.

In some embodiments, when the interface element is used to present a tool icon, the tool icon may be configured accordingly based on the virtual scene. For example, when the virtual scene is a virtual conference room, tool icons for tools such as document calling, file delivery, document sharing, and document recording used in the virtual meeting may be selected as interface elements.

In some embodiments, when the interface element is used to present a tool icon, the XR device 112 may also present a preview interface of the tool pointed to by the tool icon when presenting the interface element, so as to facilitate the user to understand the content pointed to by the tool icon. For example, when the tool icon is a tool pointing to a document call, the document used for calling may be presented in the preview interface.

In some cases, users need to move the icon elements corresponding to the interactive devices in the virtual scene to the interface elements to realize the interaction and triggering of the application. For example, in the above-mentioned virtual conference room, when the user wants to project a document to the virtual conference screen for presentation, the virtual conference screen may be far away from the user's location, and the user cannot directly interact with the virtual conference screen or project the document. For another example, when there are multiple interface elements that are far apart in the virtual space, the user will be required to adjust the position of the icon elements in the virtual scene on a large scale and frequently, which results in excessive user consumption and poor experience.

In some embodiments of the present disclosure, a group of interface elements including a plurality of interface elements may be generated, i.e., presented in the form of a group of interface elements. In some embodiments, the independent interface elements may be visually composed of a group of interface elements by adjusting the display position of each independent interface element, such as a plurality of mutually independent interface elements arranged in sequence horizontally. In some embodiments, the XR device 112 may also combine a plurality of interface elements into a group of interface elements, and a target interface element may be selected from the generated group of interface elements by, for example, sliding, etc. For example, in the case where the interface element corresponds to a tool icon of a tool, a group of interface elements may be correspondingly configured as a toolbar.

Specifically, FIG. 4 shows a schematic diagram of an example 400 in a virtual scene according to some embodiments of the present disclosure, as shown in example 400 in FIG. 4. When the virtual scene 120 is a virtual conference room scene, the XR device 112 may select a tool icon for implementing document call as an interface element 411, a tool icon for implementing document delivery as an interface element 412, and a tool icon for implementing document recording as an interface element 413 corresponding to the virtual conference room scene. The XR device 112 generates a group of interface elements 410 including the interface element 411, the interface element 412, and the interface element 413. The XR device 112 may select a target interface element by sliding in a group of interface elements 410 according to the instructions of the user 111. In some embodiments, the XR device 112 may also present a marking element 420 for identifying the target interface element, so that the user 111 may select the target interface element according to the marking element 420. For example, the user 111 may send a left-slide or right-slide action to the XR device 112 to operate the target interface element selected by the marking element 420. In some embodiments, XR device 112 may present the target interface element in a different manner from other interface elements so as to achieve the same effect as marking element 420 , for example, adjusting the display position of interface element 412 as the target interface element to be higher than the interface elements in a group of interface elements 410 .

In some embodiments, manipulating at least one interface element associated with a virtual object in a virtual scene includes: in response to detecting a first operation performed by a user on a real object, presenting a set of interface elements in association with the virtual object in the virtual scene. Specifically, the first operation is available to the user so that when the user performs the first operation, a set of interface elements is called up. The XR device 112 may store operation parameters in association with the first operation so as to determine whether the user performs the first operation by detecting whether the operation parameters of the user meet the operation parameters of the first operation. Such operation parameters include, for example, whether the user's finger is in contact with the real object, the time of contact, etc. Thus, it is convenient for the user to call up the interface element. In some embodiments, the first operation indicates that at least one finger of the user is in contact with the real object for a predetermined time. For example, the first operation operation may indicate that at least one finger of the user is in contact with the real object for a predetermined time. The predetermined time can usually be set based on the response accuracy, for example, set to 1000ms. This distinguishes the user's operation intention and reduces false touches.

Specifically, based on the set of interface elements 410 illustrated in FIG. 4 , further reference is made to FIG. 5A . FIG. 5A shows a schematic diagram of an example 500A in a virtual scene according to some embodiments of the present disclosure. A pre-configured first operation indicates that at least one object 114 (e.g., finger) of the user 111 is in contact with a real object 113 (e.g., desktop) for more than 1000ms, and the XR device 112 detects that the "left hand" and "right hand" of the user 111 are in motion, and adjusts the icon elements 122 and 123 in the virtual scene 120. And based on the situation that the four objects 114 (e.g., fingers) of the "right hand" of the XR device are in contact with the real object 113 (e.g., desktop) at the same time, a set of feedback elements 124 are presented on the virtual object 121 (e.g., virtual desktop) corresponding to the icon element 123. The XR device 112 detects that the four objects 114 (e.g., fingers) of the "right hand" are in contact with the real object 113 (e.g., desktop) at the same time. When the contact duration between the four objects 114 (e.g., fingers) and the real object 113 (e.g., desktop) meets the preset 1000ms, the XR device 112 presents a group of interface elements 410 associated with the real object 113 (e.g., desktop) in the virtual scene 120. Accordingly, when presenting a group of display elements 410, the XR device 112 adjusts the display position of the target interface element 412 to be higher than the interface elements in the group of interface elements 410 for display. And the XR device 112 presents multiple feedback elements 124 corresponding to the icon element 123 in the virtual object 121 (e.g., virtual desktop).

In some embodiments, when it is detected that the predetermined operation performed by the user on the real object includes a second operation, the display position of a group of interface elements associated with the virtual object is changed to determine the target interface element selected in the group of interface elements. Specifically, similar to the above-mentioned first operation, the XR device 112 can be configured to provide a second operation with different functions, for example, a second operation of changing the display position of the above-mentioned interface element is provided. Accordingly, when the XR device 112 detects that the user performs the second operation, it adjusts the display position of the interface element accordingly. Thus, it is further convenient for the user to determine and adjust the interface element. In some embodiments, the second operation indicates that at least one finger of the user slides in a first direction relative to the real object, wherein the XR device 112 can be set to a first direction, and the selection of the first direction generally corresponds to the arrangement direction of the interface elements in a group of interface elements, for example, when the arrangement direction is horizontal arrangement, the first direction can be determined as horizontal left and horizontal right. In some embodiments, the XR device 112 can set a sliding distance threshold, and detect that the sliding of at least one finger of the user relative to the real object in the first direction that meets the distance threshold is determined as a valid sliding. Among them, the sliding distance can be set according to the size of the real object, the size of the user's hand, etc. For example, the sliding threshold is set to 2 cm to distinguish the user's operation intention and reduce accidental touches.

Specifically, refer to the schematic diagram of example 500B shown in FIG5B. In the case where the pre-configured predetermined operation includes a second operation indicating that at least one object 114 (e.g., finger) of the user 111 slides in a horizontal direction relative to a real object, the XR device 112 detects that the four objects 114 (e.g., fingers) in FIG5A above slide horizontally to the right on the real object 113 (e.g., desktop). When the sliding distance of the four objects 114 (e.g., fingers) exceeds the preset 2 cm, the XR device adjusts the target interface element marked by the marking element 420 from the interface element 412 to the interface element 411.

In some embodiments, it also includes presenting a sub-interface corresponding to the target interface element. Specifically, continue to refer to example 500B in Figure 5B, and take interface element 411 as the target interface element as an example. After determining that the target interface element is interface element 411, the XR device 112 may generate a sub-interface 510 corresponding to the target interface element. For example, the sub-interface 510 can be used to present the control interface of the tool pointed to by the tool icon in the interface element 411 embodied as a tool icon. For example, the sub-interface can also be a preview interface of the tool pointed to by the tool icon described above.

In some embodiments, the XR device 112 may keep presenting the target interface element and the corresponding sub-interface until the next time a predetermined operation is detected. For example, when the XR device 112 cannot detect the operation of the user 111 on the real object 113 in the physical scene 110 or the XR device 112 detects that the action performed by the user 111 in the physical scene 110 is not a predetermined operation, the XR device 112 keeps presenting the interface element 411 embodied as the target interface element and the corresponding sub-interface 510.

In the case of a sub-interface, an operation instruction may be configured accordingly to realize the manipulation of the sub-interface, so that the user can adjust whether to present the sub-interface according to the needs. In some embodiments, manipulating at least one interface element associated with a virtual object in a virtual scene includes: in response to detecting a third operation made by the user on the real object, stopping the presentation of the sub-interface corresponding to the target interface element in a group of interface elements. Specifically, similar to the first operation and the second operation described above, a third operation for controlling the sub-interface (for example, stopping the presentation of the sub-interface) may also be configured. When the XR device 112 detects the third operation made by the user on the real object, the sub-interface corresponding to the target interface element in a group of interface elements is stopped. In some embodiments, the third operation indicates the sliding of at least one finger of the user in a second direction relative to the real object. Among them, the XR device 112 can set a second direction, and the second direction is different from the first direction described above, for example, when the first direction is set to the horizontal left direction or the right direction, the second direction is set to the vertical downward direction accordingly. Specifically, with continued reference to example 500C in FIG. 5C , the XR device 112 may stop presenting the sub-interface 510 upon detecting a vertical downward slide made by the user 111 on the real object 113 (eg, a desktop).

In some embodiments, the fourth operation may be set in the same manner as the third operation indication, and the XR device 112 may operate a group of interface elements 410 accordingly according to the fourth operation performed by the user 111. For example, a group of interface elements 410 may be hidden, the presentation form of a group of interfaces 410 may be adjusted to the "immersive mode", etc. The presentation size of a group of interfaces 410 in the "immersive mode" may be configured to be smaller than that of a group of interfaces 410 in the non-"immersive mode", so that the user 111 can understand the state of a group of interface elements 410 based on vision. In this case, reference may be made to FIG. 5D, which shows an example 500D for interaction in a virtual scene according to some embodiments of the present disclosure. In the example 500D, the XR device 112 may adjust a group of interface elements 410 to the "immersive mode" according to the fourth operation performed by the user 111 to obtain a group of interface elements 530, and the presentation size of the adjusted group of interfaces 530 is smaller than that of the group of interface elements 530. The configuration method, detection method, etc. of the fourth operation may refer to the third operation described above, and will not be repeated here.

The virtual scene can be used for the interaction of multiple different users, in order to take into account the sharing of presentation elements and the personalized presentation element configuration requirements. In some embodiments, the generating device usually sets two communication domains, such as "public domain" and "private domain". Among them, the presentation elements in the public domain can be added and associated to all users in the virtual scene for viewing, and the private domain is configured corresponding to the user, and the presentation elements in the private domain can only be viewed by users specific to the private domain.

In some embodiments, user 111 is a first user, and virtual scene 120 is also associated with a second user, wherein at least one interface element is presented in a display domain specific to the first user in the virtual scene, so as to facilitate the user to perform personalized configuration while avoiding interference to other users in the virtual scene.

FIG6 shows an example 600 for interaction in a virtual scene according to some embodiments of the present disclosure. In example 600, the virtual scene 120 is configured as a virtual conference room, and the virtual avatar 610 of the first user 111 described in the above embodiments and the virtual avatar 620 of the second user associated with the virtual scene 120 are presented in the virtual scene 120. For ease of understanding, in this example, only elements in the "public domain" are presented in the virtual scene 120. The display domain 621 can present the elements in the "public domain" and the elements in the "private domain" specific to the second user. The display domain 611 can present the elements in the "public domain" and the elements in the "private domain" specific to the first user 111. For example, the display domain 611 can simultaneously present a shared document 630 in the "public domain" and a group of interface elements 410. Accordingly, only the shared document 630 in the "public domain" is presented in the display domain 621, and a group of page elements 410 in the display domain 611 specific to the user 111 are not presented.

In some embodiments, if the virtual scene is configured with a "public domain" and a "private domain", in the case of a tool icon included in the interface element, the tool corresponding to the tool icon may be a function in the "public domain" that the user can use, so that the user can use the shortcut to manipulate the presentation content in the "public domain". For example, in the above example 600, a group of interface elements 410 may include a tool icon for a tool for implementing document sharing. After using the document sharing tool, the user 111 can post the document to the "public domain" for presentation, such as a shared document 630. In this case, the user 111 can view the shared document 630 in the "public domain". In some embodiments, in the case where the tool corresponding to the tool icon may be a function in the "public domain" that the user can use, a virtual object that implements the function in the virtual scene may also be configured accordingly. For example, in the example of implementing a shared document, the document sharing tool may determine a virtual object for presenting the shared document 630. For example, a virtual conference screen and a virtual desktop in the "public domain" in a virtual conference room.

In some embodiments, the same virtual object in a virtual scene can be used to present content in both the “public domain” and the “private domain”. For example, a virtual object 121 (e.g., a virtual desktop) can simultaneously present a set of interface elements 410 and a shared document 630 to a user 111.

According to an embodiment of the present disclosure, a virtual object is drawn in a virtual scene, and the virtual object is generated based on a real object in a physical scene. A predetermined operation performed by a user in the physical scene on the real object is detected. And in response to detecting the predetermined operation, at least one interface element associated with the virtual object in the virtual scene is manipulated. Thus, the real object in the physical scene is used to realize the interaction of the user in the virtual scene, thereby improving the user's interactive experience.

Example devices and equipment

7 shows a schematic structural block diagram of an apparatus 700 for interacting in a virtual scene according to some embodiments of the present disclosure. The apparatus 700 may be implemented as or included in the timing XR device 112. Each module/component in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

As shown in the figure, the device 700 includes a drawing unit 710, which is configured to draw a virtual object in a virtual scene, and the virtual object is generated based on a real object in a physical scene. The device 700 also includes a detection unit 720, which is configured to detect a predetermined operation performed by a user on a real object in the physical scene. And the device 700 also includes a manipulation unit 730, which is configured to manipulate at least one interface element associated with the virtual object in the virtual scene in response to detecting the predetermined operation.

In some embodiments, the detection unit 720 includes: a position detection subunit that detects the relative position of at least one body part of the user and the real object; and an operation detection subunit that detects a predetermined operation performed by the user on the real object in the physical scene based on the relative position.

In some embodiments, the manipulation unit 730 is further configured to, in response to detecting a first operation performed by the user on the real object, present a group of interface elements in the virtual scene in association with the virtual object.

In some embodiments, the first operation indicates that at least one finger of the user is in contact with the real object for a predetermined period of time.

In some embodiments, the manipulation unit 730 is further configured to, in response to detecting a second operation performed by the user on the real object, change the display position of a group of interface elements associated with the virtual object to determine a target interface element selected from the group of interface elements.

In some embodiments, the second operation indicates a sliding of at least one finger of the user in the first direction relative to the real object.

In some embodiments, the device 700 also includes a presentation unit configured to present a sub-interface corresponding to the target interface element.

In some embodiments, the manipulation unit 730 is further configured to, in response to detecting a third operation performed by the user on the real object, stop presenting the sub-interface corresponding to the target interface element in the group of interface elements.

In some embodiments, the third operation indicates a sliding of at least one finger of the user in a second direction relative to the real object.

In some embodiments, the virtual object includes: the predetermined operation includes an operation performed on an interactive surface of the real object, and the virtual object includes a surface element corresponding to the interactive surface of the real object.

In some embodiments, the user is a first user, the virtual scene is further associated with a second user, and the at least one interface element is presented in a display field in the virtual scene that is specific to the first user.

FIG8 shows a block diagram of a computing device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the computing device 800 shown in FIG8 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The computing device 800 shown in FIG8 may be used to implement the XR device 112 of FIG1 .

As shown in Figure 8, computing device 800 is in the form of a general-purpose computing device. The components of computing device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 850. Processing unit 810 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of computing device 800.

The computing device 800 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the computing device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 830 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and/or data (e.g., training data for training) and may be accessed within the computing device 800.

The computing device 800 may further include additional removable/non-removable, volatile/non-volatile storage media. Although not shown in FIG. 8 , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

The communication unit 840 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 800 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

The input device 850 may be one or more input devices, such as a mouse, keyboard, tracking ball, etc. The output device 850 may be one or more output devices, such as a display, a speaker, a printer, etc. The computing device 800 may also communicate with one or more external devices (not shown) through the communication unit 840 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with the computing device 800, or communicate with any device that allows the computing device 800 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input/output (I/O) interface (not shown).

According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

Various aspects of the present disclosure are described herein with reference to the flowcharts and/or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and/or block diagram and the combination of each box in the flowchart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions/actions specified in one or more boxes in the flowchart and/or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and/or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and/or flow chart, and the combination of the square boxes in the block diagram and/or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims (14)

1. A method for interacting in a virtual scene, comprising: Drawing a virtual object in a virtual scene, wherein the virtual object is generated based on a real object in a physical scene; detecting a predetermined operation performed by a user in the physical scene on the real object; and In response to detecting the predetermined operation, at least one interface element associated with the virtual object in the virtual scene is manipulated. 2. The method according to claim 1, wherein detecting the predetermined operation comprises: detecting a relative position of at least one body part of the user and the real object; and Based on the relative position, a predetermined operation performed by a user in the physical scene on the real object is detected. 3. The method according to claim 1, wherein manipulating at least one interface element associated with the virtual object in the virtual scene comprises: In response to detecting a first operation performed by the user on the real object, a group of interface elements are presented in the virtual scene in association with the virtual object. 4 . The method according to claim 3 , wherein the first operation indicates that at least one finger of the user is in contact with the real object for a predetermined time period. 5. The method according to claim 1, wherein manipulating at least one interface element associated with the virtual object in the virtual scene comprises: In response to detecting a second operation performed by the user on the real object, a display position of a group of interface elements associated with the virtual object is changed to determine a selected target interface element in the group of interface elements. The method according to claim 5 , wherein the second operation indicates sliding of at least one finger of the user in a first direction relative to the real object. 7. The method according to claim 5, further comprising: The sub-interface corresponding to the target interface element is presented. 8. The method according to claim 7, wherein manipulating at least one interface element associated with the virtual object in the virtual scene comprises: In response to detecting a third operation performed by the user on the real object, stopping presenting the sub-interface corresponding to the target interface element in the group of interface elements. 9 . The method according to claim 8 , wherein the third operation indicates sliding of at least one finger of the user in a second direction relative to the real object. 10. The method according to claim 1, wherein the predetermined operation comprises an operation performed on an interactive surface of the real object, and the virtual object comprises a surface element corresponding to the interactive surface of the real object. 11. The method according to any one of claims 1 to 10, wherein the user is a first user, the virtual scene is further associated with a second user, and wherein the at least one interface element is presented in a display field in the virtual scene that is specific to the first user. 12. A device for interacting in a virtual scene, comprising: A drawing unit, configured to draw a virtual object in a virtual scene, wherein the virtual object is generated based on a real object in a physical scene; a detection unit configured to detect a predetermined operation performed by a user in the physical scene on the real object; and The manipulation unit is configured to manipulate at least one interface element associated with the virtual object in the virtual scene in response to detecting the predetermined operation. 13. An electronic device, comprising: at least one processing unit; and At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit. 14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 11.